LM/SGLang: Difference between revisions
< LM
Jump to navigation
Jump to search
(→TODO) |
No edit summary |
||
| (21 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
'''Engine works fine, but most models cannot loaded in xpu mode.''' | |||
= Rebuild docker image from source = | |||
<syntaxhighlight lang="bash"> | |||
cd sglang | |||
git pull | |||
cd docker | |||
docker build -t sglang-xpu:latest -f xpu.Dockerfile . | |||
</syntaxhighlight> | |||
= Run = | |||
<syntaxhighlight lang="bash"> | |||
docker run \ | |||
-it \ | |||
--privileged \ | |||
--ipc=host \ | |||
--network=host \ | |||
--user root \ | |||
--group-add $(getent group video | cut -d: -f3) \ | |||
--device /dev/dri \ | |||
-v /dev/dri/by-path:/dev/dri/by-path \ | |||
-v /dev/shm:/dev/shm \ | |||
-v ~/.cache/huggingface:/root/.cache/huggingface \ | |||
-p 30000:30000 \ | |||
-e "HF_TOKEN=$HF_TOKEN" \ | |||
sglang-xpu:latest /bin/bash | |||
</syntaxhighlight> | |||
In container ... | |||
<syntaxhighlight lang="bash"> | |||
sglang serve \ | |||
--model-path Intel/gemma-4-12B-it-int4-AutoRound \ | |||
--device xpu \ | |||
--attention-backend intel_xpu | |||
</syntaxhighlight> | |||
= Structure = | |||
<quickmmd name="sgl-xpu"> | |||
flowchart TD | |||
%% 前端與介面層 | |||
subgraph Frontend ["1. 前端與高階語言層 (Frontend & Program)"] | |||
A["使用者輸入 (SGLang DSL / OpenAI API / Python)"] | |||
B["SGLang Router & Radix Cache 管理 (CPU)"] | |||
A --> B | |||
end | |||
%% SRT 引擎層 | |||
subgraph Engine ["2. SGLang Runtime (SRT Engine)"] | |||
C["Model Runner / Forward Engine"] | |||
D{"Platform Abstraction Layer (DeviceMixin)"} | |||
B --> C | |||
C --> D | |||
end | |||
%% 轉接層：Intel XPU 平台分支 | |||
subgraph PyTorch_XPU ["3. PyTorch Backend Branch (Intel Focus)"] | |||
E["PyTorch XPU Backend (torch_xpu)"] | |||
F["Distributed Processing (oneCCL / OneAPI)"] | |||
D -- "檢測到 Intel GPU" --> E | |||
D --> F | |||
end | |||
%% 高效算子與編譯層 | |||
subgraph Kernels ["4. XPU Kernel Execution (sgl-kernel-xpu)"] | |||
G["FlashAttention / FMHA (SYCL)"] | |||
H["MLA & MoE Grouped GEMM (SYCL)"] | |||
I["oneDNN / oneMKL (Intel Libraries)"] | |||
J["SYCL JIT Compiler (Intel icpx)"] | |||
E --> G | |||
E --> H | |||
E --> I | |||
G & H -- "AOT 或 JIT 編譯" --> J | |||
end | |||
%% 硬體實體層 | |||
subgraph Hardware ["5. Intel Hardware Target (XPU)"] | |||
K["Intel Data Center GPU (Max/Flex 系列 / Gaudi)"] | |||
J --> K | |||
I --> K | |||
F --> K | |||
end | |||
%% 樣式設定 | |||
%%classDef frontend fill:#e1f5fe,stroke:#0288d1,stroke-width:2px | |||
%%classDef engine fill:#fff3e0,stroke:#f57c00,stroke-width:2px | |||
%%classDef xpu fill:#e8f5e9,stroke:#388e3c,stroke-width:2px | |||
%%classDef kernel fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px | |||
%%classDef hw fill:#eceff1,stroke:#455a64,stroke-width:2px | |||
%%class A,B frontend; | |||
%%class C,D engine; | |||
%%class E,F xpu; | |||
%%class G,H,I,J kernel; | |||
%%class K hw; | |||
</quickmmd> | |||
= TODO = | = TODO = | ||
* https://docs.sglang.io/docs/hardware-platforms/xpu#xpu | |||
* https://hub.docker.com/r/intel/sglang-dev | * https://hub.docker.com/r/intel/sglang-dev | ||
* https://docs.pytorch.org/docs/2.13/notes/get_start_xpu.html# | * https://docs.pytorch.org/docs/2.13/notes/get_start_xpu.html# | ||
Latest revision as of 09:31, 17 September 2026
Engine works fine, but most models cannot loaded in xpu mode.
Rebuild docker image from source
cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .
Run
docker run \
-it \
--privileged \
--ipc=host \
--network=host \
--user root \
--group-add $(getent group video | cut -d: -f3) \
--device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path \
-v /dev/shm:/dev/shm \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 30000:30000 \
-e "HF_TOKEN=$HF_TOKEN" \
sglang-xpu:latest /bin/bash
In container ...
sglang serve \
--model-path Intel/gemma-4-12B-it-int4-AutoRound \
--device xpu \
--attention-backend intel_xpu
Structure
