LM/Devstral Small 2 24B: Difference between revisions
< LM
Jump to navigation
Jump to search
| (3 intermediate revisions by the same user not shown) | |||
| Line 10: | Line 10: | ||
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 972.13 ± 3.02 | | | mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 972.13 ± 3.02 | | ||
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 34.24 ± 0.08 | | | mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 34.24 ± 0.08 | | ||
build: bb4caa754 (10566) | |||
./llama.cpp/sycl/bin/llama-bench \ | ./llama.cpp/sycl/bin/llama-bench \ | ||
| Line 19: | Line 21: | ||
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1073.66 ± 4.43 | | | mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1073.66 ± 4.43 | | ||
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 30.04 ± 0.07 | | | mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 30.04 ± 0.07 | | ||
build: bb4caa754 (10566) | |||
./llama.cpp/sycl/bin/llama-bench \ | ./llama.cpp/sycl/bin/llama-bench \ | ||
| Line 24: | Line 28: | ||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | ||
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | |||
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | |||
| mistral3 14B Q6_K | 18.01 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1199.46 ± 12.20 | | |||
| mistral3 14B Q6_K | 18.01 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 26.85 ± 0.06 | | |||
build: bb4caa754 (10566) | |||
./llama.cpp/sycl/bin/llama-bench \ | ./llama.cpp/sycl/bin/llama-bench \ | ||
| Line 29: | Line 39: | ||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | ||
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | |||
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | |||
| mistral3 14B Q8_0 | 23.33 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1189.47 ± 2.36 | | |||
| mistral3 14B Q8_0 | 23.33 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 18.87 ± 0.01 | | |||
build: bb4caa754 (10566) | |||
./llama.cpp/sycl/bin/llama-bench \ | ./llama.cpp/sycl/bin/llama-bench \ | ||
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q4_K_XL \ | -hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q4_K_XL \ | ||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | ||
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | |||
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | |||
| mistral3 14B Q4_K - Medium | 13.50 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1043.96 ± 2.12 | | |||
| mistral3 14B Q4_K - Medium | 13.50 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 32.33 ± 0.09 | | |||
build: bb4caa754 (10566) | |||
./llama.cpp/sycl/bin/llama-bench \ | ./llama.cpp/sycl/bin/llama-bench \ | ||
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q5_K_XL \ | -hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q5_K_XL \ | ||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | ||
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | |||
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | |||
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1243.84 ± 11.72 | | |||
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 30.08 ± 0.08 | | |||
build: bb4caa754 (10566) | |||
./llama.cpp/sycl/bin/llama-bench \ | ./llama.cpp/sycl/bin/llama-bench \ | ||
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q6_K_XL \ | -hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q6_K_XL \ | ||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | ||
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | |||
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | |||
| mistral3 14B Q6_K | 19.35 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1194.97 ± 10.06 | | |||
| mistral3 14B Q6_K | 19.35 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 24.07 ± 0.04 | | |||
build: bb4caa754 (10566) | |||
</syntaxhighlight> | </syntaxhighlight> | ||
| Line 49: | Line 86: | ||
hf models ls --tree -R -h unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF | hf models ls --tree -R -h unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF | ||
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q4*" | hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q4*" | ||
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q5*" | hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q5*" | ||
| Line 55: | Line 91: | ||
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*Q8_0*" | hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*Q8_0*" | ||
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "mmproj-BF16*" | hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "mmproj-BF16*" | ||
hf models ls --tree -R -h bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF | |||
hf download bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF --include "*Q5_K_L*" | |||
tree -h ~/.cache/huggingface/hub/models--unsloth--Devstral-Small-2-24B-Instruct-2512-GGUF | tree -h ~/.cache/huggingface/hub/models--unsloth--Devstral-Small-2-24B-Instruct-2512-GGUF | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Latest revision as of 04:11, 27 August 2026
Benchmark
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_0 \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 972.13 ± 3.02 |
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 34.24 ± 0.08 |
build: bb4caa754 (10566)
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q5_K_M \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1073.66 ± 4.43 |
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 30.04 ± 0.07 |
build: bb4caa754 (10566)
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q6_K \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q6_K | 18.01 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1199.46 ± 12.20 |
| mistral3 14B Q6_K | 18.01 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 26.85 ± 0.06 |
build: bb4caa754 (10566)
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q8_0 \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q8_0 | 23.33 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1189.47 ± 2.36 |
| mistral3 14B Q8_0 | 23.33 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 18.87 ± 0.01 |
build: bb4caa754 (10566)
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q4_K_XL \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q4_K - Medium | 13.50 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1043.96 ± 2.12 |
| mistral3 14B Q4_K - Medium | 13.50 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 32.33 ± 0.09 |
build: bb4caa754 (10566)
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q5_K_XL \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1243.84 ± 11.72 |
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 30.08 ± 0.08 |
build: bb4caa754 (10566)
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:UD-Q6_K_XL \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q6_K | 19.35 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1194.97 ± 10.06 |
| mistral3 14B Q6_K | 19.35 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 24.07 ± 0.04 |
build: bb4caa754 (10566)
Experiment
hf models ls --search "Devstral" --apps llama.cpp --expand "downloads,likes,lastModified" --sort downloads --no-truncate --limit 10
hf models ls --tree -R -h unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q4*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q5*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q6*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*Q8_0*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "mmproj-BF16*"
hf models ls --tree -R -h bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF
hf download bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF --include "*Q5_K_L*"
tree -h ~/.cache/huggingface/hub/models--unsloth--Devstral-Small-2-24B-Instruct-2512-GGUF