LM/SGLang: Difference between revisions
< LM
Tag: Manual revert |
|||
| (33 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
'''Engine works fine, but most models cannot loaded in xpu mode.''' | |||
= Rebuild docker image from source = | |||
<syntaxhighlight lang="bash"> | |||
cd sglang | |||
git pull | |||
cd docker | |||
docker build -t sglang-xpu:latest -f xpu.Dockerfile . | |||
</syntaxhighlight> | |||
= Run = | |||
<syntaxhighlight lang="bash"> | |||
docker run \ | |||
-it \ | |||
--privileged \ | |||
--ipc=host \ | |||
--network=host \ | |||
--user root \ | |||
--group-add $(getent group video | cut -d: -f3) \ | |||
--device /dev/dri \ | |||
-v /dev/dri/by-path:/dev/dri/by-path \ | |||
-v /dev/shm:/dev/shm \ | |||
-v ~/.cache/huggingface:/root/.cache/huggingface \ | |||
-p 30000:30000 \ | |||
-e "HF_TOKEN=$HF_TOKEN" \ | |||
sglang-xpu:latest /bin/bash | |||
</syntaxhighlight> | |||
<ref> | |||
{{ref web | |||
|url=https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu | |||
|title=Official SGLang docker image | |||
|site=hub.docker.com | |||
}} | |||
</ref> | |||
<ref> | |||
{{ref web | |||
|url=https://hub.docker.com/r/intel/sglang-dev | |||
|title=Intel patched SGLang docker image | |||
|site=hub.docker.com | |||
}} | |||
</ref> | |||
<ref> | |||
{{ref web | |||
|url=https://docs.sglang.io/docs/hardware-platforms/xpu#xpu | |||
|title=Hardware Platforms XPU | |||
|site=docs.sglang.io | |||
}} | |||
</ref> | |||
<ref> | |||
{{ref web | |||
|url=https://docs.sglang.io/docs/advanced_features/server_arguments#server-arguments | |||
|title=SGLang server arguments | |||
|site=docs.sglang.io | |||
}} | |||
</ref> | |||
<ref> | |||
{{ref web | |||
|url=https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3#3-1-basic-configuration | |||
|title=SGLang cookbook for Qwen3 + BMG | |||
|site=docs.sglang.io | |||
}} | |||
</ref> | |||
= Structure = | = Structure = | ||
| Line 61: | Line 127: | ||
</quickmmd> | </quickmmd> | ||
= | == References == | ||
<div style="column-count: 2; column-gap: 2em;"> | |||
<references/> | |||
</div> | |||
Latest revision as of 01:56, 11 November 2026
Engine works fine, but most models cannot loaded in xpu mode.
Rebuild docker image from source
cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .
Run
docker run \
-it \
--privileged \
--ipc=host \
--network=host \
--user root \
--group-add $(getent group video | cut -d: -f3) \
--device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path \
-v /dev/shm:/dev/shm \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 30000:30000 \
-e "HF_TOKEN=$HF_TOKEN" \
sglang-xpu:latest /bin/bash
Structure
References
- ↑ Official SGLang docker image - hub.docker.com
- ↑ Intel patched SGLang docker image - hub.docker.com
- ↑ Hardware Platforms XPU - docs.sglang.io
- ↑ SGLang server arguments - docs.sglang.io
- ↑ SGLang cookbook for Qwen3 + BMG - docs.sglang.io