LM/Qwen 3.8 27B: Difference between revisions

Latest revision as of 17:38, 27 September 2026

vLLM

Tested Models

Model ID	B70 loadable	B70x1 tg128	B70x2 tg128
Launch80/Qwen3.8-27B-GPTQ-Int4-baked-v2	✅	68.68 t/s	86.86 t/s
johannrplaster/Qwen3.8-27B-Uncensored-int4-AutoRound	✅	43.21 t/s	78.66 t/s
mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16	✅	47.55 t/s	75.57 t/s
SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16	✅	39.87 t/s	74.91 t/s
cyankiwi/Qwen3.8-27B-AWQ-INT4	✅	37.71 t/s	74.01 t/s
kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16	✅	49.29 t/s	72.63 t/s
cyankiwi/Swift-Qwen3.8-27B-AWQ-BF16-INT4	⚠️	OOM	60.75 t/s
cyankiwi/Qwen3.8-27B-AWQ-BF16-INT4	⚠️	OOM	58.97 t/s
Frozenlock/Qwen3.8-27B-int4-AutoRound	✅	17.50 t/s	🗑️
CySpiegel/Qwen3.8-27B-Int4-AutoRound	✅	17.44 t/s	🗑️
MKRWW/Qwen3.8-27B-int4-AutoRound	✅	14.66 t/s	🗑️
devan-carlin/Qwen3.8-27B-int4-AutoRound	✅	10.87 t/s	🗑️
HaberstrohSystems/Qwen3.8-27B-int4-g128-sym-autoround	✅	10.47 t/s	🗑️
Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound	❌	🗑️	🗑️
Marcin116/Qwen3.8-27B-Uncensored-W4A16-AutoRound-embed-int4	❌	🗑️	🗑️

Wishlist Models

Launch80/Qwen3.8-27B-GPTQ-Int4-baked-v2
kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16
zrlu/Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70
rwmacy/qwen3.8-27b-mtp-head-v8-b70
arcitech-psp/Tiel-Coder-35B-A3B-W4A16-GPTQ-XPU-MTP

Trouble Shooting

vllm-xpu-qwen38   | (EngineCore pid=82) XMX int8 is not supported on B70 with oneAPI < 2026. Falling back to fp16.

TP Issue

vllm-xpu-qwen38   | (EngineCore pid=91) INFO 09-24 16:10:50 [multiproc_executor.py:153] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.25.99 (local), world_size=2, local_world_size=2
vllm-xpu-qwen38   | [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
vllm-xpu-qwen38   | [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
vllm-xpu-qwen38   | (Worker pid=139) INFO 09-24 16:10:57 [parallel_state.py:1775] world_size=2 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_7f41ab22b776493eb4322bb3b16ba3d2 backend=xccl
vllm-xpu-qwen38   | [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
vllm-xpu-qwen38   | [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
vllm-xpu-qwen38   | (Worker pid=140) INFO 09-24 16:10:57 [parallel_state.py:1775] world_size=2 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_7f41ab22b776493eb4322bb3b16ba3d2 backend=xccl
vllm-xpu-qwen38   | (Worker pid=139) INFO 09-24 16:10:57 [parallel_state.py:2119] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
vllm-xpu-qwen38   | 2026:09:24-16:10:57:  140 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
vllm-xpu-qwen38   | 2026:09:24-16:10:57:  139 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
vllm-xpu-qwen38   | 2026:09:24-16:10:57:  140 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
vllm-xpu-qwen38   | 2026:09:24-16:10:57:  139 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL

llama.cpp

Running

The five options are important, especially --chat-template-kwargs '{"reasoning_effort":"..."}'.

The option reasoning_effort can be set to xhigh, medium, and low, but xhigh causes significant performance degradation.

--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.0
--chat-template-kwargs '{"reasoning_effort":"low"}'

# works
llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.7 -ngld 99
  -fa 1 -c 128000 -ctk f16 -ctv f16
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --chat-template-kwargs '{"reasoning_effort":"low"}'
  -b 2048 -ub 1024
  -ngl 99 -np 1 -t 6
  --load-mode mlock --jinja --reasoning-preserve

Benchmark for me

# control group
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0

| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           pp512 |        951.05 ± 7.72 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           tg128 |         22.17 ± 0.06 |

# Test -ctv q4_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0

# Test -ub 2048
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0 -ub 2048

# Test OpenVINO
GGML_OPENVINO_DEVICE=GPU GGML_OPENVINO_STATEFUL_EXECUTION=1 ./llama.cpp/openvino/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0

Benchmark for community

./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_M -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_S -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -fa 1 -ctk q8_0 -ctv q8_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 1 -ctk q8_0 -ctv q8_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS -fa 0,1

./llama.cpp/sycl-nodnn/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0,1
./llama.cpp/sycl-nodnn/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 0,1
./llama.cpp/sycl-nodnn/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS -fa 0,1

./llama.cpp/sycl-f32/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0,1
./llama.cpp/sycl-f32/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 0,1
./llama.cpp/sycl-f32/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS -fa 0,1

## unsloth/Qwen3.8-27B-GGUF:UD-Q6_K

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q6_K                |  20.46 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        979.33 ± 4.64 |
| qwen35 27B Q6_K                |  20.46 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         21.25 ± 0.05 |
| qwen35 27B Q6_K                |  20.46 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        793.28 ± 1.67 |
| qwen35 27B Q6_K                |  20.46 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         21.15 ± 0.02 |

build: 9723942ad (10711)

## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        970.49 ± 3.90 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         22.30 ± 0.01 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        840.86 ± 4.11 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         22.22 ± 0.01 |

build: 9723942ad (10711)

## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_M

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Medium       |  18.40 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |       858.88 ± 22.25 |
| qwen35 27B Q5_K - Medium       |  18.40 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         22.22 ± 0.03 |
| qwen35 27B Q5_K - Medium       |  18.40 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        815.38 ± 6.21 |
| qwen35 27B Q5_K - Medium       |  18.40 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         22.25 ± 0.05 |

build: 9723942ad (10711)

## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_S

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Small        |  17.37 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        914.77 ± 2.72 |
| qwen35 27B Q5_K - Small        |  17.37 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         22.40 ± 0.02 |
| qwen35 27B Q5_K - Small        |  17.37 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        845.78 ± 1.73 |
| qwen35 27B Q5_K - Small        |  17.37 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         22.33 ± 0.02 |

build: 9723942ad (10711)

## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        811.78 ± 1.40 |
| qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         21.39 ± 0.02 |
| qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        784.04 ± 1.72 |
| qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         21.33 ± 0.02 |

build: 9723942ad (10711)

## F16 unsloth/Qwen3.8-27B-GGUF:Q4_0

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        813.56 ± 1.92 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         27.48 ± 0.02 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        355.72 ± 0.43 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         27.37 ± 0.03 |

build: 9723942ad (10711)

## F16 unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        583.02 ± 0.61 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         16.40 ± 0.03 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        586.82 ± 0.83 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         16.34 ± 0.03 |

build: 9723942ad (10711)

## F16 without DNN unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        948.86 ± 4.32 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         22.31 ± 0.03 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        791.62 ± 1.74 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         22.17 ± 0.02 |

build: 9723942ad (10711)

## F16 without DNN unsloth/Qwen3.8-27B-GGUF:Q4_0

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        792.89 ± 2.42 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         27.53 ± 0.02 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        353.01 ± 1.91 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         27.36 ± 0.01 |

build: 9723942ad (10711)

## F16 without DNN unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        571.95 ± 0.63 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         16.43 ± 0.04 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        560.78 ± 0.38 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         16.40 ± 0.01 |

build: 9723942ad (10711)

## F32 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        311.00 ± 0.43 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         22.26 ± 0.03 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        304.89 ± 0.13 |
| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         22.17 ± 0.02 |

build: 9723942ad (10711)

## F32 unsloth/Qwen3.8-27B-GGUF:Q4_0

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        239.50 ± 2.02 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         27.07 ± 0.04 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        270.79 ± 0.47 |
| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         27.05 ± 0.05 |

build: 9723942ad (10711)

## F32 unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   0 |           pp512 |        274.41 ± 0.40 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   0 |           tg128 |         16.51 ± 0.06 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   1 |           pp512 |        273.20 ± 0.13 |
| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  -1 |   1 |           tg128 |         16.42 ± 0.01 |

build: 9723942ad (10711)

Experiment

# search
hf models ls --search "Qwen3.8-27B" --apps llama.cpp --expand "downloads,likes,lastModified" --sort downloads --no-truncate --limit 10

# unsloth
hf models ls --tree -R -h unsloth/Qwen3.8-27B-GGUF

hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-IQ4_XS*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q5_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K.gguf"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_M*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_L*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q8_K_L*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*Q4_0*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*Q8_0*"
hf download unsloth/Qwen3.8-27B-GGUF --include "mmproj-BF16*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*mtp-Qwen3.8-27B-Q4_0.gguf"

find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf"
find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf" -exec sh -c 'rm -f "$1" "$(readlink -f "$1")"' _ {} \;

tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF

# SergiioB
hf models ls --tree -R -h SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
hf dpwnload SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
tree -h ~/.cache/huggingface/hub/models--SergiioB--Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16

Accuracy

https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fl9piThTmmUsePst5w3F2%252Fimage.png%3Falt%3Dmedia%26token%3D821e88e2-ad4e-40bf-adf2-e13837228e82&width=768&dpr=3&quality=100&sign=dd5f7cb5&sv=2

@@ Line 1: / Line 1: @@
 == vLLM ==
-=== Models ===
+=== Tested Models ===
-{| class="wikitable sortable"
+{| class="wikitable sortable" style="text-align:center;"
 ! Model ID
-! B70 compatible
+! B70 loadable
-! llama-benchy
+! B70x1 tg128
+! B70x2 tg128
 |-
-| HaberstrohSystems/Qwen3.8-27B-int4-g128-sym-autoround
+| align="left" | Launch80/Qwen3.8-27B-GPTQ-Int4-baked-v2
-| ✅ Yes
+| ✅
-| 10.47 t/s
+| 68.68 t/s
+| 86.86 t/s
 |-
-| johannrplaster/Qwen3.8-27B-Uncensored-int4-AutoRound
+| align="left" | johannrplaster/Qwen3.8-27B-Uncensored-int4-AutoRound
-| ✅ Yes
+| ✅
 | 43.21 t/s
+| 78.66 t/s
 |-
-| mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
+| align="left" | [https://github.com/MikeCaldera/intel-arc-pro-b70-qwen38-vllm/blob/main/docs/B70-QUALITY-DEPLOYMENT.md mikeinnyc/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16]
-| ✅ Yes
+| ✅
 | 47.55 t/s
+| 75.57 t/s
 |-
-| [https://huggingface.co/SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16]
+| align="left" | [https://huggingface.co/SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16]
-| ✅ Yes
+| ✅
 | 39.87 t/s
+| 74.91 t/s
+|-
+| align="left" | cyankiwi/Qwen3.8-27B-AWQ-INT4
+| ✅
+| 37.71 t/s
+| 74.01 t/s
+|-
+| align="left" | kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16
+| ✅
+| 49.29 t/s
+| 72.63 t/s
+|-
+| align="left" | cyankiwi/Swift-Qwen3.8-27B-AWQ-BF16-INT4
+| ⚠️
+| OOM
+| 60.75 t/s
 |-
-| devan-carlin/Qwen3.8-27B-int4-AutoRound
+| align="left" | cyankiwi/Qwen3.8-27B-AWQ-BF16-INT4
-| ✅ Yes
+| ⚠️
-| 10.87 t/s
+| OOM
+| 58.97 t/s
+|-
+| align="left" | Frozenlock/Qwen3.8-27B-int4-AutoRound
+| ✅
+| 17.50 t/s
+| 🗑️
 |-
-| Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound
+| align="left" | CySpiegel/Qwen3.8-27B-Int4-AutoRound
-| Cannot be loaded
+| ✅
-| -
+| 17.44 t/s
+| 🗑️
 |-
-| MKRWW/Qwen3.8-27B-int4-AutoRound
+| align="left" | MKRWW/Qwen3.8-27B-int4-AutoRound
-| ✅ Yes
+| ✅
 | 14.66 t/s
+| 🗑️
+|-
+| align="left" | devan-carlin/Qwen3.8-27B-int4-AutoRound
+| ✅
+| 10.87 t/s
+| 🗑️
 |-
-| Marcin116/Qwen3.8-27B-Uncensored-W4A16-AutoRound-embed-int4
+| align="left" | HaberstrohSystems/Qwen3.8-27B-int4-g128-sym-autoround
-| Cannot be loaded
+| ✅
-| -
+| 10.47 t/s
+| 🗑️
 |-
-| cyankiwi/Qwen3.8-27B-AWQ-INT4
+| align="left" | Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound
-| ✅ Yes
+| ❌
-| 37.71 t/s
+| 🗑️
+| 🗑️
 |-
-| cyankiwi/Swift-Qwen3.8-27B-AWQ-BF16-INT4
+| align="left" | Marcin116/Qwen3.8-27B-Uncensored-W4A16-AutoRound-embed-int4
-| OOM
+| ❌
-| -
+| 🗑️
+| 🗑️
 |}
+=== Wishlist Models ===
+* Launch80/Qwen3.8-27B-GPTQ-Int4-baked-v2
+* kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16
+* zrlu/Huihui-Qwen3.8-27B-abliterated-GPTQ-Int4-sym-G128-MTP-BF16-B70
+* rwmacy/qwen3.8-27b-mtp-head-v8-b70
+* arcitech-psp/Tiel-Coder-35B-A3B-W4A16-GPTQ-XPU-MTP
+=== Trouble Shooting ===
+<pre>
+vllm-xpu-qwen38   | (EngineCore pid=82) XMX int8 is not supported on B70 with oneAPI < 2026. Falling back to fp16.
+</pre>
+=== TP Issue ===
+<pre>
+vllm-xpu-qwen38   | (EngineCore pid=91) INFO 09-24 16:10:50 [multiproc_executor.py:153] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.25.99 (local), world_size=2, local_world_size=2
+vllm-xpu-qwen38   | [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
+vllm-xpu-qwen38   | [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
+vllm-xpu-qwen38   | (Worker pid=139) INFO 09-24 16:10:57 [parallel_state.py:1775] world_size=2 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_7f41ab22b776493eb4322bb3b16ba3d2 backend=xccl
+vllm-xpu-qwen38   | [ERROR] `min_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
+vllm-xpu-qwen38   | [ERROR] `max_frames` is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /opt/venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
+vllm-xpu-qwen38   | (Worker pid=140) INFO 09-24 16:10:57 [parallel_state.py:1775] world_size=2 rank=1 local_rank=1 distributed_init_method=file:///tmp/vllm_dist_7f41ab22b776493eb4322bb3b16ba3d2 backend=xccl
+vllm-xpu-qwen38   | (Worker pid=139) INFO 09-24 16:10:57 [parallel_state.py:2119] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
+vllm-xpu-qwen38   | 2026:09:24-16:10:57:  140 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
+vllm-xpu-qwen38   | 2026:09:24-16:10:57:  139 |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
+vllm-xpu-qwen38   | 2026:09:24-16:10:57:  140 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
+vllm-xpu-qwen38   | 2026:09:24-16:10:57:  139 |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
+</pre>
 == llama.cpp ==

LM/Qwen 3.8 27B: Difference between revisions

Latest revision as of 17:38, 27 September 2026

Contents

vLLM

Tested Models

Wishlist Models

Trouble Shooting

TP Issue

llama.cpp

Running

Benchmark for me

Benchmark for community

Experiment

Accuracy

Navigation menu

LM/Qwen 3.8 27B: Difference between revisions

Latest revision as of 17:38, 27 September 2026

vLLM

Tested Models

Wishlist Models

Trouble Shooting

TP Issue

llama.cpp

Running

Benchmark for me

Benchmark for community

Experiment

Accuracy

Navigation menu

Search