LM/SGLang: Difference between revisions

From Fundamental Ramen
< LM
Jump to navigation Jump to search
← Older edit
VisualWikitext
Tag: Manual revert
 
(34 intermediate revisions by the same user not shown)
Line 1: Line 1:
'''Engine works fine, but most models cannot loaded in xpu mode.'''
= Rebuild docker image from source =
<syntaxhighlight lang="bash">
cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .
</syntaxhighlight>
= Run =
<syntaxhighlight lang="bash">
docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=$HF_TOKEN" \
    sglang-xpu:latest /bin/bash
</syntaxhighlight>
<ref>
{{ref web
  |url=https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu
  |title=Official SGLang docker image
  |site=hub.docker.com
}}
</ref>
<ref>
{{ref web
  |url=https://hub.docker.com/r/intel/sglang-dev
  |title=Intel patched SGLang docker image
  |site=hub.docker.com
}}
</ref>
<ref>
{{ref web
  |url=https://docs.sglang.io/docs/hardware-platforms/xpu#xpu
  |title=Hardware Platforms XPU
  |site=docs.sglang.io
}}
</ref>
<ref>
{{ref web
  |url=https://docs.sglang.io/docs/advanced_features/server_arguments#server-arguments
  |title=SGLang server arguments
  |site=docs.sglang.io
}}
</ref>
<ref>
{{ref web
  |url=https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3#3-1-basic-configuration
  |title=SGLang cookbook for Qwen3 + BMG
  |site=docs.sglang.io
}}
</ref>
= Structure =
= Structure =


Line 8: Line 74:
         B["SGLang Router & Radix Cache 管理 (CPU)"]
         B["SGLang Router & Radix Cache 管理 (CPU)"]
         A --> B
         A --> B
     End
     end


     %% SRT 引擎層
     %% SRT 引擎層
Line 16: Line 82:
         B --> C
         B --> C
         C --> D
         C --> D
     End
     end


     %% 轉接層：Intel XPU 平台分支
     %% 轉接層：Intel XPU 平台分支
Line 24: Line 90:
         D -- "檢測到 Intel GPU" --> E
         D -- "檢測到 Intel GPU" --> E
         D --> F
         D --> F
     End
     end


     %% 高效算子與編譯層
     %% 高效算子與編譯層
Line 37: Line 103:
         E --> I
         E --> I
         G & H -- "AOT 或 JIT 編譯" --> J
         G & H -- "AOT 或 JIT 編譯" --> J
     End
     end


     %% 硬體實體層
     %% 硬體實體層
Line 45: Line 111:
         I --> K
         I --> K
         F --> K
         F --> K
     End
     end


     %% 樣式設定
     %% 樣式設定
     classDef frontend fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
     %%classDef frontend fill:#e1f5fe,stroke:#0288d1,stroke-width:2px
     classDef engine fill:#fff3e0,stroke:#f57c00,stroke-width:2px
     %%classDef engine fill:#fff3e0,stroke:#f57c00,stroke-width:2px
     classDef xpu fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
     %%classDef xpu fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
     classDef kernel fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
     %%classDef kernel fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
     classDef hw fill:#eceff1,stroke:#455a64,stroke-width:2px
     %%classDef hw fill:#eceff1,stroke:#455a64,stroke-width:2px


     %%class A,B frontend;
     %%class A,B frontend;
Line 61: Line 127:
</quickmmd>
</quickmmd>


= TODO =
== References ==


* https://docs.sglang.io/docs/hardware-platforms/xpu#xpu
<div style="column-count: 2; column-gap: 2em;">
* https://hub.docker.com/r/intel/sglang-dev
<references/>
* https://docs.pytorch.org/docs/2.13/notes/get_start_xpu.html#
</div>

Latest revision as of 01:56, 11 November 2026

Engine works fine, but most models cannot loaded in xpu mode.

Rebuild docker image from source

cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .

Run

docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=$HF_TOKEN" \
    sglang-xpu:latest /bin/bash

[1] [2] [3] [4] [5]

Structure

References

  1. ↑ Official SGLang docker image - hub.docker.com
  2. ↑ Intel patched SGLang docker image - hub.docker.com
  3. ↑ Hardware Platforms XPU - docs.sglang.io
  4. ↑ SGLang server arguments - docs.sglang.io
  5. ↑ SGLang cookbook for Qwen3 + BMG - docs.sglang.io
Retrieved from "https://wiki.fundamental-ramen.com/index.php?title=LM/SGLang&oldid=4216"