LM/SGLang

Engine works fine, but most models cannot loaded in xpu mode.

Rebuild docker image from source

cd sglang
git pull
cd docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .

Run

docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=$HF_TOKEN" \
    sglang-xpu:latest /bin/bash

In container ...

sglang serve                        \
    --model-path Intel/gemma-4-12B-it-int4-AutoRound \
    --device xpu                    \
    --attention-backend intel_xpu

Structure

TODO

• https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu
• https://docs.sglang.io/docs/hardware-platforms/xpu#xpu
• https://docs.pytorch.org/docs/2.13/notes/get_start_xpu.html#
• https://hub.docker.com/r/intel/sglang-dev
• https://docs.sglang.io/docs/advanced_features/server_arguments#server-arguments

SGLang docker image - hub.docker.com