LM/SGLang: Difference between revisions

From Fundamental Ramen
< LM
Jump to navigation Jump to search
← Older edit
VisualWikitext
Tag: Manual revert
 
(38 intermediate revisions by the same user not shown)
Line 1: Line 1:
= TODO =
'''Engine works fine, but most models cannot loaded in xpu mode.'''


* https://hub.docker.com/r/intel/sglang-dev
= Rebuild docker image from source =
* https://docs.sglang.io/docs/hardware-platforms/xpu#xpu
 
* https://docs.pytorch.org/docs/2.13/notes/get_start_xpu.html#
<syntaxhighlight lang="bash">
cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .
</syntaxhighlight>
 
= Run =
 
<syntaxhighlight lang="bash">
docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=$HF_TOKEN" \
    sglang-xpu:latest /bin/bash
</syntaxhighlight>
 
<ref>
{{ref web
  |url=https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu
  |title=Official SGLang docker image
  |site=hub.docker.com
}}
</ref>
<ref>
{{ref web
  |url=https://hub.docker.com/r/intel/sglang-dev
  |title=Intel patched SGLang docker image
  |site=hub.docker.com
}}
</ref>
<ref>
{{ref web
  |url=https://docs.sglang.io/docs/hardware-platforms/xpu#xpu
  |title=Hardware Platforms XPU
  |site=docs.sglang.io
}}
</ref>
<ref>
{{ref web
  |url=https://docs.sglang.io/docs/advanced_features/server_arguments#server-arguments
  |title=SGLang server arguments
  |site=docs.sglang.io
}}
</ref>
<ref>
{{ref web
  |url=https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3#3-1-basic-configuration
  |title=SGLang cookbook for Qwen3 + BMG
  |site=docs.sglang.io
}}
</ref>
 
= Structure =
 
<quickmmd name="sgl-xpu">
flowchart TD
    %% 前端與介面層
    subgraph Frontend ["1. 前端與高階語言層 (Frontend & Program)"]
        A["使用者輸入 (SGLang DSL / OpenAI API / Python)"]
        B["SGLang Router & Radix Cache 管理 (CPU)"]
        A --> B
    end
 
    %% SRT 引擎層
    subgraph Engine ["2. SGLang Runtime (SRT Engine)"]
        C["Model Runner / Forward Engine"]
        D{"Platform Abstraction Layer (DeviceMixin)"}
        B --> C
        C --> D
    end
 
    %% 轉接層：Intel XPU 平台分支
    subgraph PyTorch_XPU ["3. PyTorch Backend Branch (Intel Focus)"]
        E["PyTorch XPU Backend (torch_xpu)"]
        F["Distributed Processing (oneCCL / OneAPI)"]
        D -- "檢測到 Intel GPU" --> E
        D --> F
    end
 
    %% 高效算子與編譯層
    subgraph Kernels ["4. XPU Kernel Execution (sgl-kernel-xpu)"]
        G["FlashAttention / FMHA (SYCL)"]
        H["MLA & MoE Grouped GEMM (SYCL)"]
        I["oneDNN / oneMKL (Intel Libraries)"]
        J["SYCL JIT Compiler (Intel icpx)"]
       
        E --> G
        E --> H
        E --> I
        G & H -- "AOT 或 JIT 編譯" --> J
    end
 
    %% 硬體實體層
    subgraph Hardware ["5. Intel Hardware Target (XPU)"]
        K["Intel Data Center GPU (Max/Flex 系列 / Gaudi)"]
        J --> K
        I --> K
        F --> K
    end
 
    %% 樣式設定
    %%classDef frontend fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
    %%classDef engine fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    %%classDef xpu fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    %%classDef kernel fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    %%classDef hw fill:#eceff1,stroke:#455a64,stroke-width:2px
 
    %%class A,B frontend;
    %%class C,D engine;
    %%class E,F xpu;
    %%class G,H,I,J kernel;
    %%class K hw;
</quickmmd>
 
== References ==
 
<div style="column-count: 2; column-gap: 2em;">
<references/>
</div>

Latest revision as of 01:56, 11 November 2026

Engine works fine, but most models cannot loaded in xpu mode.

Rebuild docker image from source

cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .

Run

docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=$HF_TOKEN" \
    sglang-xpu:latest /bin/bash

[1] [2] [3] [4] [5]

Structure

References

  1. ↑ Official SGLang docker image - hub.docker.com
  2. ↑ Intel patched SGLang docker image - hub.docker.com
  3. ↑ Hardware Platforms XPU - docs.sglang.io
  4. ↑ SGLang server arguments - docs.sglang.io
  5. ↑ SGLang cookbook for Qwen3 + BMG - docs.sglang.io
Retrieved from "https://wiki.fundamental-ramen.com/index.php?title=LM/SGLang&oldid=4216"