LM/Qwen 3.8 27B: Difference between revisions

From Fundamental Ramen
< LM
Jump to navigation Jump to search
← Older editNewer edit →
VisualWikitext
Line 78: Line 78:


=== Running ===
=== Running ===
The five options are important, especially <code>--chat-template-kwargs '{"reasoning_effort":"..."}'</code>.
The option <code>reasoning_effort</code> can be set to <code>xhigh</code>, <code>medium</code>, and <code>low</code>, but <code>xhigh</code> causes significant performance degradation.
* <code>--temp 1.0</code>
* <code>--top-p 0.95</code>
* <code>--top-k 20</code>
* <code>--min-p 0.0</code>
* <code>--chat-template-kwargs '{"reasoning_effort":"low"}'</code>


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 106: Line 96:
# control group
# control group
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0
| model                          |      size |    params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| qwen35 27B Q5_K - Medium      |  19.43 GiB |    27.32 B | SYCL      |  99 |  q8_0 |  q8_0 |  1 |          pp512 |        951.05 ± 7.72 |
| qwen35 27B Q5_K - Medium      |  19.43 GiB |    27.32 B | SYCL      |  99 |  q8_0 |  q8_0 |  1 |          tg128 |        22.17 ± 0.06 |


# Test -ctv q4_0
# Test -ctv q4_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0
# Test -ub 2048
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0 -ub 2048
# Test OpenVINO
GGML_OPENVINO_DEVICE=GPU GGML_OPENVINO_STATEFUL_EXECUTION=1 ./llama.cpp/openvino/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0
</syntaxhighlight>
</syntaxhighlight>


Revision as of 09:01, 22 October 2026

vLLM

Best-Fit Models for B70

The B70 delivers the best performance with W4A8 quantized models.

The underlying rationale can be referenced from TechPowerUp’s spec data, and the listed models show solid performance.

Model Last-Modified W A TP2 / C1 TP2 / C2 TP2 / C4 TP2 / C8 Single
PP 512 TG 128 PP 512 TG 128 PP 512 TG 128 PP 512 TG 128 PP 512 TG 128
cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-BF16-INT4 2026-10-19 4 16 1737 65 2087 97 2121 139 2664 240
cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4 2026-10-19 4 16 1908 79 2062 115 2196 158 2632 269
Launch80/Swift-1.5-Qwen3.8-27B-Uncensored-Int4-baked 2026-10-13 4 16 1899 102 2084 165 2595 241 2758 307
davetha/Huihui-Qwen3.8-27B-abliterated-AutoRound-Int4-baked-embed-int8 2026-10-09 4 16 2026 118 2411 164 2801 266 2785 319
🔥 Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8 🔥 2026-10-01 4 16 2335 110 2429 190 2822 286 2928 506

hf operations

# search latest vllm models
hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B int4"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B int8 int4"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B autoround"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B w4a8" 

# list / download / tree
hf models ls --tree -R -h Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
hf download Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
tree -h ~/.cache/huggingface/hub/models--Wrapzii--Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8

llama.cpp

Running

# works
llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.7 -ngld 99
  -fa 1 -c 128000 -ctk f16 -ctv f16
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --chat-template-kwargs '{"reasoning_effort":"low"}'
  -b 2048 -ub 1024
  -ngl 99 -np 1 -t 6
  --load-mode mlock --jinja --reasoning-preserve

Benchmark for me

# control group
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0

# Test -ctv q4_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0

Experiment

# search
hf models ls --search "Qwen3.8-27B" --apps llama.cpp --expand "downloads,likes,lastModified" --sort downloads --no-truncate --limit 10

# unsloth
hf models ls --tree -R -h unsloth/Qwen3.8-27B-GGUF

hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-IQ4_XS*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q5_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K.gguf"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_M*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_L*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q8_K_L*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*Q4_0*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*Q8_0*"
hf download unsloth/Qwen3.8-27B-GGUF --include "mmproj-BF16*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*mtp-Qwen3.8-27B-Q4_0.gguf"

find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf"
find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf" -exec sh -c 'rm -f "$1" "$(readlink -f "$1")"' _ {} \;

tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF
Retrieved from "https://wiki.fundamental-ramen.com/index.php?title=LM/Qwen_3.8_27B&oldid=3977"