LM/Devstral Small 2 24B: Difference between revisions
< LM
Jump to navigation
Jump to search
| Line 5: | Line 5: | ||
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_0 \ | -hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_0 \ | ||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | ||
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | |||
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | |||
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 972.13 ± 3.02 | | |||
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 34.24 ± 0.08 | | |||
./llama.cpp/sycl/bin/llama-bench \ | ./llama.cpp/sycl/bin/llama-bench \ | ||
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q5_K_M \ | -hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q5_K_M \ | ||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | |||
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | |||
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | |||
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1073.66 ± 4.43 | | |||
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 30.04 ± 0.07 | | |||
./llama.cpp/sycl/bin/llama-bench \ | |||
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q6_K_XL \ | |||
-fa on -ctk q8_0 -ctv q8_0 -ngl 99 | -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Revision as of 02:04, 27 August 2026
Benchmark
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_0 \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 972.13 ± 3.02 |
| mistral3 14B Q4_0 | 12.56 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 34.24 ± 0.08 |
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q5_K_M \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | pp512 | 1073.66 ± 4.43 |
| mistral3 14B Q5_K - Medium | 15.61 GiB | 23.57 B | SYCL | 99 | q8_0 | q8_0 | 1 | tg128 | 30.04 ± 0.07 |
./llama.cpp/sycl/bin/llama-bench \
-hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q6_K_XL \
-fa on -ctk q8_0 -ctv q8_0 -ngl 99
Experiment
hf models ls --search "Devstral" --apps llama.cpp --expand "downloads,likes,lastModified" --sort downloads --no-truncate --limit 10
hf models ls --tree -R -h unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q4*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q5*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*UD-Q6*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "*Q8_0*"
hf download unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF --include "mmproj-BF16*"
tree -h ~/.cache/huggingface/hub/models--unsloth--Devstral-Small-2-24B-Instruct-2512-GGUF