LM/Qwen 3.8 27B
< LM
Jump to navigation
Jump to search
vLLM
Models
| Model ID | B70 tested | Published B70 configuration | Quantization / packing | Weight size | MTP / speculative decoding | Model behavior | Primary advantage | Main caution |
|---|---|---|---|---|---|---|---|---|
| Pilcothink/Qwen3.8-27B-MixedInt4-AutoRound | ❔ Not publicly verified | — | Custom mixed-precision INT4 AutoRound |
Not clearly published in the model card summary | Not clearly documented as an XPU-tested MTP recipe | Standard Qwen base behavior | Mixed-precision design prioritizes benchmark recovery / quality retention over a simple uniform INT4 recipe | Published optimization and publicity focus on DGX Spark / CUDA, not Intel XPU; mixed-precision layout needs B70 compatibility testing |
| HaberstrohSystems/Qwen3.8-27B-int4-g128-sym-autoround | ✅ Yes | 1× Arc Pro B70 vLLM XPU |
W4A16 INT4 AutoRound 0.14.2 group size 128 symmetric auto_round:auto_gptq
|
17.69 GiB 19.00 GB on disk |
Not the headline feature; verify actual MTP setup before enabling it | Standard Qwen base behavior Reasoning on by default |
Explicitly tuned for one 32 GB B70; full reproducible XPU-oriented recipe; 131K serving target; vision tower retained at original precision | One-card-oriented recipe; verify your own context, FP8 KV cache, driver and vLLM-XPU image combination |
| johannrplaster/Qwen3.8-27B-Uncensored-int4-AutoRound | ✅ Yes | 1× B70 + MTP3 + FP16 KV: 57.5 tok/s 2× B70 TP=2 + MTP3 + FP16 KV: 81.8 tok/s |
W4A16 INT4 AutoRound 0.14.2 group size 128 symmetric auto_round:auto_gptq
|
~18 GB | ✅ Yes Quantized MTP head included qwen3_5_mtpMTP3 is the published best fresh-request trade-off |
Uncensored / abliterated | Best documented B70 performance recipe; MTP works without a separate draft model; preserves sensitive linear-attention projections in FP16 | Uncensored behavior is unsuitable for many public / enterprise deployments; FP16 KV gives speed but FP8 KV is preferable for 128K-capacity use |
| MKRWW/Qwen3.8-27B-int4-AutoRound | ❔ Not publicly verified | Public benchmark: 2× RTX 3090 TP=2 not B70 / XPU |
W4A16 INT4 AutoRound group size 128 symmetric auto_round:auto_gptq
|
17.45 GB weights ~18 GB repository claim |
Not explicitly packaged or benchmarked as an MTP / speculative-decoding checkpoint | Standard Qwen base behavior Reasoning, tool calling and Omni / vision retained |
Clean conservative baseline; transparent recipe; embeddings and LM head remain FP16; includes practical vLLM reasoning and tool-call settings | CUDA-focused public validation; vision/audio towers, embeddings and LM head remain higher precision, so real B70 VRAM headroom must be measured |
| Marcin116/Qwen3.8-27B-Uncensored-W4A16-AutoRound-embed-int4 | ❔ Not publicly verified | — | W4A16 AutoRound with INT4 embedding emphasis |
Not independently verified from an accessible model card | Not publicly verified for B70 / XPU | Uncensored / abliterated | More aggressive VRAM saving through embedding INT4; potentially useful for longer context, larger KV cache or MTP on one GPU | INT4 embeddings can affect multilingual, rare-token, retrieval and long-context quality; exact LM-head / MTP / vision-tower precision must be checked from files or model card |
| devan-carlin/Qwen3.8-27B-int4-AutoRound |
llama.cpp
Running
The five options are important, especially --chat-template-kwargs '{"reasoning_effort":"..."}'.
The option reasoning_effort can be set to xhigh, medium, and low, but xhigh causes significant performance degradation.
--temp 1.0--top-p 0.95--top-k 20--min-p 0.0--chat-template-kwargs '{"reasoning_effort":"low"}'
# works
llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.7 -ngld 99
-fa 1 -c 128000 -ctk f16 -ctv f16
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--chat-template-kwargs '{"reasoning_effort":"low"}'
-b 2048 -ub 1024
-ngl 99 -np 1 -t 6
--load-mode mlock --jinja --reasoning-preserve
Benchmark for me
# control group
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 951.05 ± 7.72 |
| qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 22.17 ± 0.06 |
# Test -ctv q4_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0
# Test -ub 2048
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0 -ub 2048
# Test OpenVINO
GGML_OPENVINO_DEVICE=GPU GGML_OPENVINO_STATEFUL_EXECUTION=1 ./llama.cpp/openvino/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0
Benchmark for community
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_M -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_S -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -fa 1 -ctk q8_0 -ctv q8_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 0,1
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 1 -ctk q8_0 -ctv q8_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS -fa 0,1
./llama.cpp/sycl-nodnn/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0,1
./llama.cpp/sycl-nodnn/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 0,1
./llama.cpp/sycl-nodnn/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS -fa 0,1
./llama.cpp/sycl-f32/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0,1
./llama.cpp/sycl-f32/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 -fa 0,1
./llama.cpp/sycl-f32/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS -fa 0,1
## unsloth/Qwen3.8-27B-GGUF:UD-Q6_K | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q6_K | 20.46 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 979.33 ± 4.64 | | qwen35 27B Q6_K | 20.46 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 21.25 ± 0.05 | | qwen35 27B Q6_K | 20.46 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 793.28 ± 1.67 | | qwen35 27B Q6_K | 20.46 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 21.15 ± 0.02 | build: 9723942ad (10711) ## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 970.49 ± 3.90 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 22.30 ± 0.01 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 840.86 ± 4.11 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 22.22 ± 0.01 | build: 9723942ad (10711) ## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_M | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q5_K - Medium | 18.40 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 858.88 ± 22.25 | | qwen35 27B Q5_K - Medium | 18.40 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 22.22 ± 0.03 | | qwen35 27B Q5_K - Medium | 18.40 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 815.38 ± 6.21 | | qwen35 27B Q5_K - Medium | 18.40 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 22.25 ± 0.05 | build: 9723942ad (10711) ## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_S | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q5_K - Small | 17.37 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 914.77 ± 2.72 | | qwen35 27B Q5_K - Small | 17.37 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 22.40 ± 0.02 | | qwen35 27B Q5_K - Small | 17.37 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 845.78 ± 1.73 | | qwen35 27B Q5_K - Small | 17.37 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 22.33 ± 0.02 | build: 9723942ad (10711) ## F16 unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q4_K - Medium | 16.34 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 811.78 ± 1.40 | | qwen35 27B Q4_K - Medium | 16.34 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 21.39 ± 0.02 | | qwen35 27B Q4_K - Medium | 16.34 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 784.04 ± 1.72 | | qwen35 27B Q4_K - Medium | 16.34 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 21.33 ± 0.02 | build: 9723942ad (10711) ## F16 unsloth/Qwen3.8-27B-GGUF:Q4_0 | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 813.56 ± 1.92 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 27.48 ± 0.02 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 355.72 ± 0.43 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 27.37 ± 0.03 | build: 9723942ad (10711) ## F16 unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 583.02 ± 0.61 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 16.40 ± 0.03 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 586.82 ± 0.83 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 16.34 ± 0.03 | build: 9723942ad (10711) ## F16 without DNN unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 948.86 ± 4.32 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 22.31 ± 0.03 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 791.62 ± 1.74 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 22.17 ± 0.02 | build: 9723942ad (10711) ## F16 without DNN unsloth/Qwen3.8-27B-GGUF:Q4_0 | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 792.89 ± 2.42 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 27.53 ± 0.02 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 353.01 ± 1.91 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 27.36 ± 0.01 | build: 9723942ad (10711) ## F16 without DNN unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 571.95 ± 0.63 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 16.43 ± 0.04 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 560.78 ± 0.38 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 16.40 ± 0.01 | build: 9723942ad (10711) ## F32 unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 311.00 ± 0.43 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 22.26 ± 0.03 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 304.89 ± 0.13 | | qwen35 27B Q5_K - Medium | 19.43 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 22.17 ± 0.02 | build: 9723942ad (10711) ## F32 unsloth/Qwen3.8-27B-GGUF:Q4_0 | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 239.50 ± 2.02 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 27.07 ± 0.04 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 270.79 ± 0.47 | | qwen35 27B Q4_0 | 14.94 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 27.05 ± 0.05 | build: 9723942ad (10711) ## F32 unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS | model | size | params | backend | ngl | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 0 | pp512 | 274.41 ± 0.40 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 0 | tg128 | 16.51 ± 0.06 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 1 | pp512 | 273.20 ± 0.13 | | qwen35 27B IQ4_XS - 4.25 bpw | 13.26 GiB | 27.32 B | SYCL | -1 | 1 | tg128 | 16.42 ± 0.01 | build: 9723942ad (10711)
Experiment
# search
hf models ls --search "Qwen3.8-27B" --apps llama.cpp --expand "downloads,likes,lastModified" --sort downloads --no-truncate --limit 10
# unsloth
hf models ls --tree -R -h unsloth/Qwen3.8-27B-GGUF
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-IQ4_XS*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q5_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K.gguf"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_M*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_L*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q8_K_L*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*Q4_0*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*Q8_0*"
hf download unsloth/Qwen3.8-27B-GGUF --include "mmproj-BF16*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*mtp-Qwen3.8-27B-Q4_0.gguf"
find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf"
find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf" -exec sh -c 'rm -f "$1" "$(readlink -f "$1")"' _ {} \;
tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF
# SergiioB
hf models ls --tree -R -h SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
hf dpwnload SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
tree -h ~/.cache/huggingface/hub/models--SergiioB--Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16
