LM/Qwen 3.8 27B: Difference between revisions

Latest revision as of 08:30, 27 October 2026

vLLM

Best-Fit Models for B70

The B70 delivers the best performance with W4A8 quantized models.

The underlying rationale can be referenced from TechPowerUp’s spec data, and the listed models show solid performance.

Model	Last-Modified	W	A	TP2 / C1		TP2 / C2		TP2 / C4		TP2 / C8		Single
Model	Last-Modified	W	A	PP 512	TG 128	PP 512	TG 128	PP 512	TG 128	PP 512	TG 128	PP 512	TG 128
cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-BF16-INT4	2026-10-19	4	16	1737	65	2087	97	2121	139	2664	240
cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4	2026-10-19	4	16	1908	79	2062	115	2196	158	2632	269
Launch80/Swift-1.5-Qwen3.8-27B-Uncensored-Int4-baked	2026-10-13	4	16	1899	102	2084	165	2595	241	2758	307
davetha/Huihui-Qwen3.8-27B-abliterated-AutoRound-Int4-baked-embed-int8	2026-10-09	4	16	2026	118	2411	164	2801	266	2785	319
🔥 Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8 🔥	2026-10-01	4	16	2335	110	2429	190	2822	286	2928	506

Available Requests

/openapi.json, Methods: GET, HEAD
/docs, Methods: GET, HEAD
/docs/oauth2-redirect, Methods: GET, HEAD
/redoc, Methods: GET, HEAD
/load, Methods: GET
/version, Methods: GET
/health, Methods: GET
/metrics, Methods: GET
/tokenize, Methods: POST
/detokenize, Methods: POST
/v1/models, Methods: GET
/ping, Methods: GET
/ping, Methods: POST
/invocations, Methods: POST
/v1/chat/completions, Methods: POST
/v1/chat/completions/batch, Methods: POST
/v1/responses, Methods: POST
/v1/responses/{response_id}, Methods: GET
/v1/responses/{response_id}/cancel, Methods: POST
/v1/completions, Methods: POST
/v1/messages, Methods: POST
/v1/messages/count_tokens, Methods: POST
/generative_scoring, Methods: POST
/scale_elastic_ep, Methods: POST
/is_scaling_elastic_ep, Methods: POST

hf operations

# search latest vllm models
hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B int4"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B int8 int4"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B autoround"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
  --expand "downloads,likes,lastModified" \
  --limit 25 --search "Qwen3.8 27B w4a8" 

# list / download / tree
hf models ls --tree -R -h Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
hf download Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
tree -h ~/.cache/huggingface/hub/models--Wrapzii--Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8

llama.cpp

Running

# works
llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.7 -ngld 99
  -fa 1 -c 128000 -ctk f16 -ctv f16
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
  --chat-template-kwargs '{"reasoning_effort":"low"}'
  -b 2048 -ub 1024
  -ngl 99 -np 1 -t 6
  --load-mode mlock --jinja --reasoning-preserve

Benchmark for me

model	size	params	backend	ngl	type_k	type_v	fa	test	t/s
qwen35 27B Q5_K - Medium	19.43 GiB	27.32 B	SYCL	99	q8_0	q8_0	1	pp512	951.05 ± 7.72
qwen35 27B Q5_K - Medium	19.43 GiB	27.32 B	SYCL	99	q8_0	q8_0	1	tg128	22.17 ± 0.06

# control group
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0

# Test -ctv q4_0
./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0

hf operations

# search
hf models ls --apps llama.cpp --sort downloads --no-truncate \
  --expand "downloads,likes,lastModified" \
  --limit 10 --search "Qwen3.8-27B"

# list remote tree
hf models ls --tree -R -h unsloth/Qwen3.8-27B-GGUF

# download quantization
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q5_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_XL*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*Q4_0*"
hf download unsloth/Qwen3.8-27B-GGUF --include "mmproj-BF16*"
hf download unsloth/Qwen3.8-27B-GGUF --include "*mtp-Qwen3.8-27B-Q4_0.gguf"

# list local tree
find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf"
find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf" -exec sh -c 'rm -f "$1" "$(readlink -f "$1")"' _ {} \;
tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF

LM/Qwen 3.8 27B: Difference between revisions

Latest revision as of 08:30, 27 October 2026

Contents

vLLM

Best-Fit Models for B70

Available Requests

hf operations

llama.cpp

Running

Benchmark for me

hf operations

Navigation menu

@@ Line 1: / Line 1: @@
-== llama.cpp ==
+== vLLM ==
+=== Best-Fit Models for B70 ===
-* https://unsloth.ai/docs/basics/dynamic-3.0-ggufs
+The B70 delivers the best performance with W4A8 quantized models.
-=== Running ===
+The underlying rationale can be referenced from TechPowerUp’s spec data, and the listed models show solid performance.
-<syntaxhighlight lang="bash">
+{| class="wikitable" style="text-align:center;"
-# works
+! rowspan="2" | Model
-./llama.cpp/sycl/bin/llama-server \
+! rowspan="2" | Last-Modified
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_M \
+! rowspan="2" | W
-  --spec-type draft-mtp --spec-draft-n-max 2 -ngld 99 \
+! rowspan="2" | A
-  -c 64000 -ctk q8_0 -ctv q8_0 \
+! colspan="2" | TP2 / C1
-  -fa on -ngl 99 --load-mode mlock -np 1 --jinja --reasoning-preserve -t 8
+! colspan="2" | TP2 / C2
+! colspan="2" | TP2 / C4
+! colspan="2" | TP2 / C8
+! colspan="2" | Single
+|-
+! PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128
+|-
+| style="font-size:10pt; text-align:left;" | cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-BF16-INT4
+| 2026-10-19 || 4 || 16
+| 1737 || 65 || 2087 || 97 || 2121 || 139 || 2664 || 240
+| ||
+|-
+| style="font-size:10pt; text-align:left;" | cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4
+| 2026-10-19 || 4 || 16
+| 1908 || 79 || 2062 || 115 || 2196 || 158 || 2632 || 269
+| ||
+|-
+| style="font-size:10pt; text-align:left;" | Launch80/Swift-1.5-Qwen3.8-27B-Uncensored-Int4-baked
+| 2026-10-13 || 4 || 16
+| 1899 || 102 || 2084 || 165 || 2595 || 241 || 2758 || 307
+| ||
+|-
+| style="font-size:10pt; text-align:left;" | davetha/Huihui-Qwen3.8-27B-abliterated-AutoRound-Int4-baked-embed-int8
+| 2026-10-09 || 4 || 16
+| 2026 || 118 || 2411 || 164 || 2801 || 266 || 2785 || 319
+| ||
+|-
+| style="font-size:10pt; text-align:left;" | 🔥 Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8 🔥
+| 2026-10-01 || 4 || 16
+| 2335 || 110 || 2429 || 190 || 2822 || 286 || 2928 || 506
+| ||
+|}
-# with turboquant
+=== Available Requests ===
-./llama.cpp/vulkan/bin/llama-server \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_M \
-  --spec-type draft-mtp --spec-draft-n-max 2 -ngld 99 \
-  -c 64000 -ctk q8_0 -ctv turbo4 \
-  -fa on -ngl 99 --load-mode mlock -np 1 --jinja --reasoning-preserve -t 8
-# assign draft model (cannot run)
+<pre>
-./llama.cpp/sycl/bin/llama-server \
+/openapi.json, Methods: GET, HEAD
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_M \
+/docs, Methods: GET, HEAD
-  -hfd unsloth/Qwen3.8-27B-GGUF:MTP/mtp-Qwen3.8-27B-Q4_0.gguf \
+/docs/oauth2-redirect, Methods: GET, HEAD
-  --spec-type draft-mtp --spec-draft-n-max 2 -ngld 99 \
+/redoc, Methods: GET, HEAD
-  -c 64000 -ctk q8_0 -ctv q8_0 \
+/load, Methods: GET
-  -fa on -ngl 99 --load-mode mlock -np 1 --jinja --reasoning-preserve -t 8 -lv 5
+/version, Methods: GET
-</syntaxhighlight>
+/health, Methods: GET
+/metrics, Methods: GET
+/tokenize, Methods: POST
+/detokenize, Methods: POST
+/v1/models, Methods: GET
+/ping, Methods: GET
+/ping, Methods: POST
+/invocations, Methods: POST
+/v1/chat/completions, Methods: POST
+/v1/chat/completions/batch, Methods: POST
+/v1/responses, Methods: POST
+/v1/responses/{response_id}, Methods: GET
+/v1/responses/{response_id}/cancel, Methods: POST
+/v1/completions, Methods: POST
+/v1/messages, Methods: POST
+/v1/messages/count_tokens, Methods: POST
+/generative_scoring, Methods: POST
+/scale_elastic_ep, Methods: POST
+/is_scaling_elastic_ep, Methods: POST
+</pre>
-=== Benchmark for me ===
+=== hf operations ===
 <syntaxhighlight lang="bash">
-# control group
+# search latest vllm models
-./llama.cpp/sycl/bin/llama-bench \
+hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
-   -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL \
+   --expand "downloads,likes,lastModified" \
-   -fa on -ctk q8_0 -ctv q8_0 -ngl 99
+   --limit 25 --search "Qwen3.8 27B int4"
-| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
+hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
+  --expand "downloads,likes,lastModified" \
-| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           pp512 |        951.05 ± 7.72 |
+  --limit 25 --search "Qwen3.8 27B int8 int4"
-| qwen35 27B Q5_K - Medium       |  19.43 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           tg128 |         22.17 ± 0.06 |
-# Test Quantization
+hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
-./llama.cpp/sycl/bin/llama-bench \
+   --expand "downloads,likes,lastModified" \
-   -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 \
+   --limit 25 --search "Qwen3.8 27B autoround"
-   -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
+hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
+  --expand "downloads,likes,lastModified" \
-| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           pp512 |        798.17 ± 1.28 |
+  --limit 25 --search "Qwen3.8 27B w4a8"
-| qwen35 27B Q4_0                |  14.94 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           tg128 |         27.30 ± 0.05 |
-./llama.cpp/sycl/bin/llama-bench \
+# list / download / tree
-  -hf unsloth/Qwen3.8-27B-GGUF:Q8_0 \
+hf models ls --tree -R -h Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
+hf download Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
+tree -h ~/.cache/huggingface/hub/models--Wrapzii--Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
+</syntaxhighlight>
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
+== llama.cpp ==
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q8_0                |  27.04 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |       817.25 ± 15.79 |
-| qwen35 27B Q8_0                |  27.04 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         15.48 ± 0.01 |
-./llama.cpp/sycl/bin/llama-bench \
+* https://unsloth.ai/docs/models/qwen3.8
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS \
+* https://unsloth.ai/docs/basics/dynamic-3.0-ggufs
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
+=== Running ===
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        573.51 ± 0.58 |
-| qwen35 27B IQ4_XS - 4.25 bpw   |  13.26 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         16.37 ± 0.05 |
-./llama.cpp/sycl/bin/llama-bench \
+<syntaxhighlight lang="bash">
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_S \
+# works
-   -fa on -ctk q8_0 -ctv q8_0 -ngl 99
+llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
+   --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.7 -ngld 99
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
+  -fa 1 -c 128000 -ctk f16 -ctv f16
-| qwen35 27B Q4_K - Small        |  14.29 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        644.94 ± 1.00 |
+  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
-| qwen35 27B Q4_K - Small        |  14.29 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         19.09 ± 0.08 |
+  --chat-template-kwargs '{"reasoning_effort":"low"}'
+  -b 2048 -ub 1024
+  -ngl 99 -np 1 -t 6
+  --load-mode mlock --jinja --reasoning-preserve
+</syntaxhighlight>
-./llama.cpp/sycl/bin/llama-bench \
+=== Benchmark for me ===
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
+{| class="wikitable"
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
+! model || size || params || backend || ngl || type_k || type_v || fa || test || t/s
-| qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        793.27 ± 0.74 |
+|-
-| qwen35 27B Q4_K - Medium       |  16.34 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         21.37 ± 0.03 |
+! qwen35 27B Q5_K - Medium
+| 19.43 GiB || | 27.32 B || | SYCL || 99 || q8_0 || q8_0 || 1 || pp512 || 951.05 ± 7.72
+|-
+! qwen35 27B Q5_K - Medium
+| 19.43 GiB || 27.32 B || SYCL || 99 || q8_0 || q8_0 || 1 || tg128 || 22.17 ± 0.06
+|}
-./llama.cpp/sycl/bin/llama-bench \
+<syntaxhighlight lang="bash">
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_S \
+# control group
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
+./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q5_K - Small        |  17.37 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        880.03 ± 1.55 |
-| qwen35 27B Q5_K - Small        |  17.37 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         22.35 ± 0.05 |
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_M \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q5_K - Medium       |  18.40 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        927.22 ± 0.90 |
-| qwen35 27B Q5_K - Medium       |  18.40 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         22.43 ± 0.05 |
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q6_K                |  20.46 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        945.67 ± 4.57 |
-| qwen35 27B Q6_K                |  20.46 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         21.06 ± 0.05 |
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_M \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q6_K                |  21.49 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        953.90 ± 5.70 |
-| qwen35 27B Q6_K                |  21.49 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         20.14 ± 0.04 |
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_L \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q6_K                |  22.52 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        950.43 ± 5.17 |
-| qwen35 27B Q6_K                |  22.52 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         19.30 ± 0.03 |
-./llama.cpp/sycl/bin/llama-bench \
+# Test -ctv q4_0
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL \
+./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q6_K                |  23.55 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        923.94 ± 1.96 |
-| qwen35 27B Q6_K                |  23.55 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         18.27 ± 0.02 |
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_L \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | n_ubatch | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q8_0                |  26.11 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           pp512 |        836.06 ± 2.05 |
-| qwen35 27B Q8_0                |  26.11 GiB |    27.32 B | SYCL       |  99 |     2048 |   q8_0 |   q8_0 |   1 |           tg128 |         16.28 ± 0.01 |
-# Test -ctv
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL \
-  -fa on -ctk q8_0 -ctv q4_0 -ngl 99
-# Test -ub
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99 -ub 2048
-# Test -fa off
-./llama.cpp/sycl/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL \
-  -fa off -ngl 99
-# Test bartowski
-./llama.cpp/sycl/bin/llama-bench \
-  -hf bartowski/Qwen3.8-27B-GGUF:Q5_K_L \
-  -fa on -ctk q8_0 -ctv q8_0 -ngl 99
-| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |            test |                  t/s |
-| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
-| qwen35 27B Q5_K - Medium       |  20.05 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           pp512 |        997.64 ± 2.90 |
-| qwen35 27B Q5_K - Medium       |  20.05 GiB |    27.32 B | SYCL       |  99 |   q8_0 |   q8_0 |   1 |           tg128 |         22.08 ± 0.06 |
-# Test OpenVINO
-GGML_OPENVINO_DEVICE=GPU GGML_OPENVINO_STATEFUL_EXECUTION=1 ./llama.cpp/openvino/bin/llama-bench \
-  -hf unsloth/Qwen3.8-27B-GGUF:Q4_0 \
-  -fa on -ngl 99
 </syntaxhighlight>
-=== Benchmark for community ===
+=== hf operations ===
-<syntaxhighlight lang="bash">
-./llama.cpp/sycl/bin/llama-bench -hf bartowski/Qwen3.8-27B-GGUF:Q4_0 -fa 0,1
-./llama.cpp/sycl/bin/llama-bench -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_L -fa 0,1
-</syntaxhighlight>
-== Experiment ==
 <syntaxhighlight lang="bash">
 # search
+hf models ls --apps llama.cpp --sort downloads --no-truncate \
+  --expand "downloads,likes,lastModified" \
+  --limit 10 --search "Qwen3.8-27B"
-hf models ls --search "Qwen3.8-27B" --apps llama.cpp --expand "downloads,likes,lastModified" --sort downloads --no-truncate --limit 10
+# list remote tree
-# unsloth
 hf models ls --tree -R -h unsloth/Qwen3.8-27B-GGUF
-hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-IQ4_XS*"
+# download quantization
 hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q4_K_XL*"
 hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q5_K_XL*"
-hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K.gguf"
-hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_M*"
-hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_L*"
 hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K_XL*"
-hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q8_K_L*"
 hf download unsloth/Qwen3.8-27B-GGUF --include "*Q4_0*"
-hf download unsloth/Qwen3.8-27B-GGUF --include "*Q8_0*"
 hf download unsloth/Qwen3.8-27B-GGUF --include "mmproj-BF16*"
 hf download unsloth/Qwen3.8-27B-GGUF --include "*mtp-Qwen3.8-27B-Q4_0.gguf"
+# list local tree
 find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf"
 find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf" -exec sh -c 'rm -f "$1" "$(readlink -f "$1")"' _ {} \;
 tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF
-# bartowski
-hf models ls --tree -R -h bartowski/Qwen3.8-27B-GGUF
-hf download bartowski/Qwen3.8-27B-GGUF --include "*IQ3_XXS*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*IQ4_XS*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*Q4_0*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*Q4_1*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*Q4_K_L*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*Q5_K_L*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*Q6_K_L*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*Q8_0*"
-hf download bartowski/Qwen3.8-27B-GGUF --include "*mmproj-Qwen3.8-27B-bf16.gguf*"
-tree -h ~/.cache/huggingface/hub/models--bartowski--Qwen3.8-27B-GGUF
 </syntaxhighlight>
-== Accuracy ==
-https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fl9piThTmmUsePst5w3F2%252Fimage.png%3Falt%3Dmedia%26token%3D821e88e2-ad4e-40bf-adf2-e13837228e82&width=768&dpr=3&quality=100&sign=dd5f7cb5&sv=2

LM/Qwen 3.8 27B: Difference between revisions

Latest revision as of 08:30, 27 October 2026

vLLM

Best-Fit Models for B70

Available Requests

hf operations

llama.cpp

Running

Benchmark for me

hf operations

Navigation menu

Search