LM/vLLM

docker compose for B70 (TP=2)

docker compose

services:
  vllm-xpu:
    image: intel/llm-scaler-vllm:0.31.0-b1-experimental
    # image: vllm/vllm-openai-xpu:v0.33.0
    container_name: vllm-xpu-qwen38-prod
    ports:
      - "127.0.0.1:55850:5850"
    ipc: host
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - /dev/dri/by-path:/dev/dri/by-path:ro
      - ${HOME_DIR}/.cache/huggingface:/root/.cache/huggingface
    environment:
      - HF_HUB_OFFLINE=1
      - NEOReadDebugKeys=1
      - EnableDeferBacking=0
    command:
      - "Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8"
      - "--port=5850"
      - "--served-model-name=develop"
      - "--gpu-memory-utilization=${VRAM_RESERVATION}"
      - "--dtype=bfloat16"
      - "--max-model-len=${CONTEXT_LENGTH}"
      - "--kv-cache-dtype=fp8"
      - "--tensor-parallel-size=${TP}"
      - "--max-num-seqs=8"
      - "--max-num-batched-tokens=4224"
      - "--enable-auto-tool-choice"
      - "--enable-prefix-caching"
      - "--reasoning-parser=qwen3"
      - "--tool-call-parser=qwen3_xml"
      - '--speculative-config={"method":"mtp","num_speculative_tokens":${SPEC_TOKENS}}'
      - '--compilation-config={"cudagraph_mode":"FULL_DECODE_ONLY"}'
    restart: "no"

Intel 0.31 vs vLLM 0.33

MTP	Intel		vLLM
	PP512	TG128	PP512	TG128
6	3017	419	2994	392
5	3052	439	3048	449
4	3065	440	3027	435
3	3035	470	3009	457

Benchmark Script

Get the sample from the second run of the following script.

HF_HUB_OFFLINE=1 llama-benchy --base-url http://localhost:55850/v1 \
  --model develop \
  --pp 512 --tg 128 --runs 5 --concurrency 8 \
  2>/dev/null | grep "^|" -B1 -A3

Environment

• GPU: ASRock Intel Arc Pro B70
• Bus: PCIe 5.0 x8/x8 (AMD 9600X)
• Tensor Parallelism: 2
• Context Length: 163840 (160K)
• Concurrent: 8
• Engine Image:
  • Intel: intel/llm-scaler-vllm:0.31.0-b1-experimental
  • vLLM: vllm/vllm-openai-xpu:v0.33.0

Install from source

TODO

Verify environments in container

Command	Results in vllm v0.30.0
docker exec vllm-xpu-qwen38-prod bash -c \ "pip list | awk 'NR<3 || tolower(\$0) ~ /torch|intel|triton|vllm|oneccl|mkl|dpc|level/'"	Package Version Editable project location ---------------------------------------- --------------- ------------------------------------- dpcpp-cpp-rt 2026.0.0 intel-cmplr-lib-rt 2026.0.0 intel-cmplr-lib-ur 2026.0.0 intel-cmplr-lic-rt 2026.0.0 intel-opencl-rt 2026.0.0 intel-openmp 2026.0.0 intel-pti 0.17.0 intel-sycl-rt 2026.0.0 mkl 2026.0.0 oneccl 2022.0.0 oneccl-devel 2022.0.0 onemkl-license 2026.0.0 onemkl-sycl-blas 2026.0.0 onemkl-sycl-dft 2026.0.0 onemkl-sycl-lapack 2026.0.0 onemkl-sycl-rng 2026.0.0 onemkl-sycl-sparse 2026.0.0 open_clip_torch 2.32.0 torch 2.13.0+xpu torchaudio 2.11.0+xpu torchcodec 0.16.0+cpu torchvision 0.28.0+xpu triton 3.7.2+xpu triton-xpu 3.7.2 vllm 0.30.0+xpu vllm_test_utils 0.1 /workspace/vllm/tests/vllm_test_utils vllm-xpu-kernels 0.1.14.1
docker exec vllm-xpu-qwen38-prod bash -c 'pip show vllm-xpu-kernels'	Name: vllm-xpu-kernels Version: 0.1.14.1 Summary: A high-throughput and memory-efficient inference and serving engine for LLMs Home-page: https://github.com/vllm-project/vllm-xpu-kernels Author: vLLM Team Author-email: License-Expression: Apache-2.0 Location: /opt/venv/lib/python3.12/site-packages Requires: Required-by: vllm
docker exec vllm-xpu-qwen38-prod bash -c 'lsb_release -a'	Distributor ID: Ubuntu Description: Ubuntu 24.04.5 LTS Release: 24.04 Codename: noble
docker exec -it vllm-xpu-qwen38-prod bash

k8v4

# 1. Create and activate Python virtual environment
python3 -m venv k8v4-env
source k8v4-env/bin/activate

# 2. Install Intel oneAPI (if not already installed)
#    Ref: https://www.intel.com/content/www/us/en/developer/tools/oneapi/index.html
source /opt/intel/oneapi/setvars.sh

# 3. Install vLLM 0.30 (XPU build)
pip install vllm==0.30.0

# 4. Clone k8v4-xpu project
git clone https://github.com/Wrapzii/k8v4-xpu.git
cd k8v4-xpu

# 5. Build KV Cache core library (libxe2_kv.so)
#    Project includes C/C++ core, requires local compilation
pip install -e .
# Or manual build:
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
sudo make install

# 6. Set environment variables
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
export VLLM_KV_CACHE_TYPE=k8v4   # Enable K8/V4 compression

# 7. Launch vLLM service (e.g. Swift-1.5 Qwen3.8-27B model)
vllm serve Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768

XPU Graph

• #60379 - vllm-project/vllm
• #54785 - vllm-project/vllm
• #51600 - vllm-project/vllm
• #48327 - vllm-project/vllm