LM/SGLang

Rebuild docker image from source

cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .

Run

docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=$HF_TOKEN" \
    sglang-xpu:latest /bin/bash

In container ...

sglang serve --help

sglang serve                         \
    --model-path <MODEL_ID_OR_PATH>  \
    --trust-remote-code              \
    --disable-overlap-schedule       \
    --device xpu                     \
    --host 0.0.0.0                   \
    --tp 1                           \   # using multi GPUs
    --attention-backend intel_xpu    \   # using intel optimized XPU attention backend
    --page-size                      \   # intel_xpu attention backend supports [32, 64, 128]

Structure

TODO

• https://docs.sglang.io/docs/hardware-platforms/xpu#xpu
• https://hub.docker.com/r/intel/sglang-dev
• https://docs.pytorch.org/docs/2.13/notes/get_start_xpu.html#