LM/vLLM: Difference between revisions

From Fundamental Ramen
< LM
Jump to navigation Jump to search
← Older edit
VisualWikitext
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
== compose ==
== docker compose for B70 (TP=2) ==


<syntaxhighlight lang="yaml">
<syntaxhighlight lang="yaml">
Line 86: Line 86:
   "pip list | awk 'NR<3 || tolower(\$0) ~ /torch|intel|triton|vllm|oneccl|mkl|dpc|level/'"
   "pip list | awk 'NR<3 || tolower(\$0) ~ /torch|intel|triton|vllm|oneccl|mkl|dpc|level/'"
</syntaxhighlight>
</syntaxhighlight>
| <pre>
|
<div style="overflow:scroll; max-height:250px;">
<pre>
Package                                  Version        Editable project location
Package                                  Version        Editable project location
---------------------------------------- --------------- -------------------------------------
---------------------------------------- --------------- -------------------------------------
Line 117: Line 119:
vllm-xpu-kernels                        0.1.14.1
vllm-xpu-kernels                        0.1.14.1
</pre>
</pre>
</div>
|-
|-
| valign="top" | <syntaxhighlight lang="bash">
| valign="top" | <syntaxhighlight lang="bash">
Line 222: Line 225:
! vLLM image || released || TP for B70 || pip oneccl || native oneCCL
! vLLM image || released || TP for B70 || pip oneccl || native oneCCL
|-
|-
| 0.32.0 ||  ||  ||  ||
| 0.33.0 ||  ||  ||  ||
|-
| 0.32.0 || 2026-10-18 || ❌ || 2022.1.2 || 2021.18.0
|-
|-
| 0.31.0 || 2026-10-04 || ❌ || 2022.1.2 || 2021.18.0
| 0.31.0 || 2026-10-04 || ❌ || 2022.1.2 || 2021.18.0
Line 243: Line 248:


* [https://docs.vllm.ai/en/stable/getting_started/installation/gpu/#create-a-new-python-environment Install using Python]
* [https://docs.vllm.ai/en/stable/getting_started/installation/gpu/#create-a-new-python-environment Install using Python]
* [https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables HF_* Environments]
* [https://github.com/intel/llvm/blob/sycl/sycl/doc/EnvironmentVariables.md ONEAPI_* SYCL_* Environments]
* [https://github.com/intel/llvm/blob/sycl/sycl/doc/EnvironmentVariables.md ONEAPI_* SYCL_* Environments]
* [https://www.intel.com/content/www/us/en/docs/oneccl/developer-guide-reference/2021-16/environment-variables.html CCL_* Environments]


== References ==
== References ==

Latest revision as of 10:42, 29 October 2026

docker compose for B70 (TP=2)

services:
  # Test vLLM 0.30.0
  vllm-xpu:
    image: vllm/vllm-openai-xpu:0.30.0
    container_name: vllm-xpu-test
    ports:
      - "127.0.0.1:56000:8000"
    # For oneCCL use
    ipc: host
    devices:
      # Share host GPU
      - /dev/dri:/dev/dri
    volumes:
      # Share host GPU
      - /dev/dri/by-path:/dev/dri/by-path:ro
      # Share model cache of Hugging Face 
      - ${HOME_DIR}/.cache/huggingface:/root/.cache/huggingface
      # Compiled cache
      - ${HOME_DIR}/.cache/vllm:/root/.cache/vllm
    environment:
      # TP=2 need them
      CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD: "4294967296"
      CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD: "4294967296"
      CCL_LOG_LEVEL: "info"
      PYTORCH_ALLOC_CONF: "expandable_segments:True"
      # Optimize decoding performance for 0.30.0
      VLLM_XPU_ENABLE_XPU_GRAPH: "1"
      # Block internet access
      HF_HUB_OFFLINE: "1"
    command:
      - "Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8"
      - "--host=0.0.0.0"
      - "--served-model-name=test"
      - "--gpu-memory-utilization=${VRAM_RESERVATION}"
      - "--dtype=bfloat16"
      - "--max-model-len=${CONTEXT_LENGTH}"
      - "--kv-cache-dtype=fp8"
      - "--tensor-parallel-size=${TP}"
      - "--max-num-seqs=8"
      - "--max-num-batched-tokens=8192"
      - "--enable-auto-tool-choice"
      - "--enable-prefix-caching"
      - "--reasoning-parser=qwen3"
      - "--tool-call-parser=qwen3_xml"
    restart: "no"

[1] [2] [3] [4]

Verify environments in container

Command Results in vllm v0.30.0 
docker exec vllm-xpu-qwen38-prod bash -c \
  "pip list | awk 'NR<3 || tolower(\$0) ~ /torch|intel|triton|vllm|oneccl|mkl|dpc|level/'"

Package                                  Version         Editable project location
---------------------------------------- --------------- -------------------------------------
dpcpp-cpp-rt                             2026.0.0
intel-cmplr-lib-rt                       2026.0.0
intel-cmplr-lib-ur                       2026.0.0
intel-cmplr-lic-rt                       2026.0.0
intel-opencl-rt                          2026.0.0
intel-openmp                             2026.0.0
intel-pti                                0.17.0
intel-sycl-rt                            2026.0.0
mkl                                      2026.0.0
oneccl                                   2022.0.0
oneccl-devel                             2022.0.0
onemkl-license                           2026.0.0
onemkl-sycl-blas                         2026.0.0
onemkl-sycl-dft                          2026.0.0
onemkl-sycl-lapack                       2026.0.0
onemkl-sycl-rng                          2026.0.0
onemkl-sycl-sparse                       2026.0.0
open_clip_torch                          2.32.0
torch                                    2.13.0+xpu
torchaudio                               2.11.0+xpu
torchcodec                               0.16.0+cpu
torchvision                              0.28.0+xpu
triton                                   3.7.2+xpu
triton-xpu                               3.7.2
vllm                                     0.30.0+xpu
vllm_test_utils                          0.1             /workspace/vllm/tests/vllm_test_utils
vllm-xpu-kernels                         0.1.14.1

docker exec vllm-xpu-qwen38-prod bash -c 'pip show vllm-xpu-kernels'

Name: vllm-xpu-kernels
Version: 0.1.14.1
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm-xpu-kernels
Author: vLLM Team
Author-email:
License-Expression: Apache-2.0
Location: /opt/venv/lib/python3.12/site-packages
Requires:
Required-by: vllm

docker exec vllm-xpu-qwen38-prod bash -c 'lsb_release -a'

Distributor ID: Ubuntu
Description:    Ubuntu 24.04.5 LTS
Release:        24.04
Codename:       noble

docker exec -it vllm-xpu-qwen38-prod bash

About CCL_SYCL_* variables

vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  118 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  118 |CCL_WARN| value of CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD changed to be 4294967296 (default:4194304)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  118 |CCL_WARN| value of CCL_SYCL_REDUCE_SCATTER_SIMPLE_THRESHOLD changed to be 4294967296 (default:8388608)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  118 |CCL_WARN| value of CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD changed to be 4294967296 (default:1048576)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  118 |CCL_WARN| value of CCL_SYCL_ALLTOALL_TMP_BUF changed to be 1 (default:0)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  117 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  117 |CCL_WARN| value of CCL_SYCL_ALLREDUCE_SIMPLE_THRESHOLD changed to be 4294967296 (default:4194304)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  117 |CCL_WARN| value of CCL_SYCL_REDUCE_SCATTER_SIMPLE_THRESHOLD changed to be 4294967296 (default:8388608)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  117 |CCL_WARN| value of CCL_SYCL_ALLGATHERV_SIMPLE_THRESHOLD changed to be 4294967296 (default:1048576)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  117 |CCL_WARN| value of CCL_SYCL_ALLTOALL_TMP_BUF changed to be 1 (default:0)
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  118 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:40:  117 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:41:  223:[1] |CCL_WARN| no membind support for NUMA node 0, skip thread membind
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:41:  224:[0] |CCL_WARN| no membind support for NUMA node 0, skip thread membind
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:41:  118:[1] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disa
ble topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:41:  117:[0] |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disa
ble topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:41:  118:[1] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode
vllm-xpu-qwen38-prod  | 2026:09:30-07:47:41:  117:[0] |CCL_WARN| pidfd is not supported, fallbacks to drmfd exchange mode

k8v4

# 1. 建立並激活 Python 虛擬環境
python3 -m venv k8v4-env
source k8v4-env/bin/activate

# 2. 安裝 Intel oneAPI（若尚未安裝）
#    參考: https://www.intel.com/content/www/us/en/developer/tools/oneapi/index.html
source /opt/intel/oneapi/setvars.sh

# 疑似 AI 誤導
# 3. 安裝 vLLM 0.30（XPU 版本）
pip install vllm==0.30.0

# 4. 克隆 k8v4-xpu 專案
git clone https://github.com/Wrapzii/k8v4-xpu.git
cd k8v4-xpu

# 疑似 AI 誤導
# 5. 編譯 KV Cache 核心庫（libxe2_kv.so）
#    專案包含 C/C++ 核心，需本機編譯
pip install -e .
# 或手動編譯:
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
sudo make install

# 6. 設定環境變數
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
export VLLM_KV_CACHE_TYPE=k8v4   # 啟用 K8/V4 壓縮

# 7. 啟動 vLLM 服務（以 Swift-1.5 Qwen3.8-27B 模型為例）
vllm serve Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768

CCL Optimization

vLLM and oneccl mapping

vLLM image released TP for B70 pip oneccl native oneCCL
0.33.0
0.32.0 2026-10-18 ❌ 2022.1.2 2021.18.0
0.31.0 2026-10-04 ❌ 2022.1.2 2021.18.0
0.30.0 2026-09-22 ✅ 2022.0.0 2021.18.0 
# pip
sudo docker exec vllm-xpu-test pip show oneccl

# native
sudo docker exec vllm-xpu-test bash -lc '
grep -aoE "oneCCL[^[:cntrl:]]{0,100}|CCL[_ -]?version[^[:cntrl:]]{0,100}|202[0-9]\.[0-9]+(\.[0-9]+)?" \
  /opt/venv/lib/libccl.so.2 | sort -u | head -n 50
'

TODO

References

  1. ↑ Official vLLM image vllm-openai-xpu - hub.docker.com
  2. ↑ vllm serve arguments - docs.vllm.ai
  3. ↑ HF_* Environments - huggingface.co
  4. ↑ CCL_* Environments - www.intel.com
Retrieved from "https://wiki.fundamental-ramen.com/index.php?title=LM/vLLM&oldid=4104"