Fundamental Ramen - Recent changes [en]

LM/vLLM

Tacoball — Mon, 26 Oct 2026 09:49:25 GMT

vLLM and oneccl mapping

@@ Line 189: / Line 189: @@
 == CCL Optimization ==
-TODO
+=== vLLM and oneccl mapping ===
 == TODO ==

LM/Flint

Tacoball — Mon, 26 Oct 2026 02:08:52 GMT

hf operations

← Older revision		Revision as of 02:08, 26 October 2026
(One intermediate revision by the same user not shown)
Line 7:		Line 7:
	hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \		hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
	--expand "downloads,likes,lastModified" \		--expand "downloads,likes,lastModified" \
	--limit 25 --search "~~Qwen3.8 27B int4~~"		--limit 25 --search "flint"

	hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \		hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
	--expand "downloads,likes,lastModified" \		--expand "downloads,likes,lastModified" \
	--limit 25 --search "~~Qwen3.8 27B int8~~ int4"		--limit 25 --search "flint int4"

	hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \		hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
	--expand "downloads,likes,lastModified" \		--expand "downloads,likes,lastModified" \
	--limit 25 --search "~~Qwen3.8 27B autoround~~"		--limit 25 --search "flint int8 int4"

	hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \		hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
	--expand "downloads,likes,lastModified" \		--expand "downloads,likes,lastModified" \
	--limit 25 --search "~~Qwen3.8 27B~~ w4a8"		--limit 25 --search "flint autoround"

			hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
			--expand "downloads,likes,lastModified" \
			--limit 25 --search "flint w4a8"

	# list / download / tree		# list / download / tree

LM/Flint

Tacoball — Mon, 26 Oct 2026 01:59:10 GMT

Created page with "== vLLM == === hf operations === <syntaxhighlight lang="bash"> # search latest vllm models hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \ --expand "downloads,likes,lastModified" \ --limit 25 --search "Qwen3.8 27B int4" hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \ --expand "downloads,likes,lastModified" \ --limit 25 --search "Qwen3.8 27B int8 int4" hf models ls --apps vllm --sort last_modified --no-truncate..."

New page

== vLLM ==

=== hf operations ===

<syntaxhighlight lang="bash">
# search latest vllm models
hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
--expand "downloads,likes,lastModified" \
--limit 25 --search "Qwen3.8 27B int4"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
--expand "downloads,likes,lastModified" \
--limit 25 --search "Qwen3.8 27B int8 int4"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
--expand "downloads,likes,lastModified" \
--limit 25 --search "Qwen3.8 27B autoround"

hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
--expand "downloads,likes,lastModified" \
--limit 25 --search "Qwen3.8 27B w4a8"

# list / download / tree
hf models ls --tree -R -h Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
hf download Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
tree -h ~/.cache/huggingface/hub/models--Wrapzii--Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
</syntaxhighlight>

LM

Tacoball — Mon, 26 Oct 2026 01:58:28 GMT

Topics

← Older revision		Revision as of 01:58, 26 October 2026
Line 10:		Line 10:
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|
	: [[LM/hf\|hf (Model Management)]]		: [[LM/hf\|hf (Model Management)]]
	: [[LM/~~Ornith 1.5 35B A3B~~\|~~Ornith 1.5 35B A3B~~]]		: [[LM/Flint\|Flint]]
	: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]		: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]
	: [[LM/Gemma 4\|Gemma 4]]		: [[LM/Gemma 4\|Gemma 4]]
	: [[LM/Bonsai 2 27B\|Bonsai 2 27B]]		: [[LM/Bonsai 2 27B\|Bonsai 2 27B]]
			: [[LM/Ornith 1.5 35B A3B\|Ornith 1.5 35B A3B]]
	: [[LM/Experience\|Experience]]		: [[LM/Experience\|Experience]]
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|

LM/Ornith 1.5 35B A3B

Tacoball — Thu, 22 Oct 2026 09:23:44 GMT

hf operations

← Older revision		Revision as of 09:23, 22 October 2026
Line 28:		Line 28:
	hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \		hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
	--expand "downloads,likes,lastModified" \		--expand "downloads,likes,lastModified" \
	--limit 25 --search "~~Qwen3~~.~~8 27B~~ int4"		--limit 25 --search "Ornith 1.5 35B int4"

			hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
			--expand "downloads,likes,lastModified" \
			--limit 25 --search "Ornith 1.5 35B autoround"

	# list / download / tree		# list / download / tree

LM/Ornith 1.5 35B A3B

Tacoball — Thu, 22 Oct 2026 09:20:51 GMT

Created page with "== vLLM == === Best-Fit Models for B70 === {| class="wikitable" style="text-align:center;" ! rowspan="2" | Model ! rowspan="2" | Last-Modified ! rowspan="2" | W ! rowspan="2" | A ! colspan="2" | TP2 / C1 ! colspan="2" | TP2 / C2 ! colspan="2" | TP2 / C4 ! colspan="2" | TP2 / C8 ! colspan="2" | Single |- ! PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128 |- | style="font-size:10pt; text-align:left;" | TODO | 2026-10-19 ||..."

New page

== vLLM ==

=== Best-Fit Models for B70 ===

{| class="wikitable" style="text-align:center;"
! rowspan="2" | Model
! rowspan="2" | Last-Modified
! rowspan="2" | W
! rowspan="2" | A
! colspan="2" | TP2 / C1
! colspan="2" | TP2 / C2
! colspan="2" | TP2 / C4
! colspan="2" | TP2 / C8
! colspan="2" | Single
|-
! PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128 || PP 512 || TG 128
|-
| style="font-size:10pt; text-align:left;" | TODO
| 2026-10-19 || 4 || 16
| 1737 || 65 || 2087 || 97 || 2121 || 139 || 2664 || 240
| ||
|}

=== hf operations ===

<syntaxhighlight lang="bash">
# search latest vllm models
hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
--expand "downloads,likes,lastModified" \
--limit 25 --search "Qwen3.8 27B int4"

# list / download / tree
hf models ls --tree -R -h Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
hf download Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
tree -h ~/.cache/huggingface/hub/models--Wrapzii--Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
</syntaxhighlight>

LM

Tacoball — Thu, 22 Oct 2026 09:19:34 GMT

Topics

← Older revision		Revision as of 09:19, 22 October 2026
(One intermediate revision by the same user not shown)
Line 10:		Line 10:
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|
	: [[LM/hf\|hf (Model Management)]]		: [[LM/hf\|hf (Model Management)]]
	: [[LM/~~Bonsai 2 27B~~\|~~Bonsai 2 27B~~]]		: [[LM/Ornith 1.5 35B A3B\|Ornith 1.5 35B A3B]]
	: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]		: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]
	: [[LM/Gemma 4\|Gemma 4]]		: [[LM/Gemma 4\|Gemma 4]]
	: [[LM/~~Devstral Small~~ 2 ~~24B~~\|~~Devstral Small~~ 2 ~~24B~~]]		: [[LM/Bonsai 2 27B\|Bonsai 2 27B]]
	: [[LM/Experience\|Experience]]		: [[LM/Experience\|Experience]]
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|

LM/Qwen 3.8 27B

Tacoball — Thu, 22 Oct 2026 09:09:26 GMT

hf operations

← Older revision		Revision as of 09:09, 22 October 2026
(9 intermediate revisions by the same user not shown)
Line 78:		Line 78:

	=== Running ===		=== Running ===

	~~The five options are important, especially <code>--chat-template-kwargs '{"reasoning_effort":"..."}'</code>.~~

	~~The option <code>reasoning_effort</code> can be set to <code>xhigh</code>, <code>medium</code>, and <code>low</code>, but <code>xhigh</code> causes significant performance degradation.~~

	* <code>--temp 1.0</code>
	* <code>--top-p 0.95</code>
	* <code>--top-k 20</code>
	* <code>--min-p 0.0</code>
	* <code>--chat-template-kwargs '{"reasoning_effort":"low"}'</code>

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
Line 102:		Line 92:

	=== Benchmark for me ===		=== Benchmark for me ===

			{\| class="wikitable"
			! model \|\| size \|\| params \|\| backend \|\| ngl \|\| type_k \|\| type_v \|\| fa \|\| test \|\| t/s
			\|-
			! qwen35 27B Q5_K - Medium
			\| 19.43 GiB \|\| \| 27.32 B \|\| \| SYCL \|\| 99 \|\| q8_0 \|\| q8_0 \|\| 1 \|\| pp512 \|\| 951.05 ± 7.72
			\|-
			! qwen35 27B Q5_K - Medium
			\| 19.43 GiB \|\| 27.32 B \|\| SYCL \|\| 99 \|\| q8_0 \|\| q8_0 \|\| 1 \|\| tg128 \|\| 22.17 ± 0.06
			\|}

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	# control group		# control group
	./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0		./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0

	~~\| model \| size \| params \| backend \| ngl \| type_k \| type_v \| fa \| test \| t/s \|~~
	~~\| ------------------------------ \| ---------: \| ---------: \| ---------- \| --: \| -----: \| -----: \| --: \| --------------: \| -------------------: \|~~
	~~\| qwen35 27B Q5_K - Medium \| 19.43 GiB \| 27.32 B \| SYCL \| 99 \| q8_0 \| q8_0 \| 1 \| pp512 \| 951.05 ± 7.72 \|~~
	~~\| qwen35 27B Q5_K - Medium \| 19.43 GiB \| 27.32 B \| SYCL \| 99 \| q8_0 \| q8_0 \| 1 \| tg128 \| 22.17 ± 0.06 \|~~

	# Test -ctv q4_0		# Test -ctv q4_0
	./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0		./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q4_0

	~~# Test -ub 2048~~
	~~./llama.cpp/sycl/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 1 -ctk q8_0 -ctv q8_0 -ub 2048~~

	~~# Test OpenVINO~~
	~~GGML_OPENVINO_DEVICE=GPU GGML_OPENVINO_STATEFUL_EXECUTION=1 ./llama.cpp/openvino/bin/llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL -fa 0~~
	</syntaxhighlight>		</syntaxhighlight>

	== ~~Experiment~~ ==		=== hf operations ===

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	# search		# search
	hf models ls --~~search "Qwen3~~.8-~~27B"~~ --~~apps llama.cpp~~ --expand "downloads,likes,lastModified" --~~sort downloads~~ --no-~~truncate --limit 10~~		hf models ls --apps llama.cpp --sort downloads --no-truncate \
			--expand "downloads,likes,lastModified" \
			--limit 10 --search "Qwen3.8-27B"

	# ~~unsloth~~		# list remote tree
	hf models ls --tree -R -h unsloth/Qwen3.8-27B-GGUF		hf models ls --tree -R -h unsloth/Qwen3.8-27B-GGUF

	hf download ~~unsloth/Qwen3.8-27B-GGUF --include "UD-IQ4_XS"~~		# download quantization
	hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q4_K_XL"		hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q4_K_XL"
	hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q5_K_XL"		hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q5_K_XL"
	~~hf download unsloth/Qwen3.8-27B-GGUF --include "*UD-Q6_K.gguf"~~
	~~hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q6_K_M"~~
	~~hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q6_K_L"~~
	hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q6_K_XL"		hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q6_K_XL"
	~~hf download unsloth/Qwen3.8-27B-GGUF --include "UD-Q8_K_L"~~
	hf download unsloth/Qwen3.8-27B-GGUF --include "Q4_0"		hf download unsloth/Qwen3.8-27B-GGUF --include "Q4_0"
	~~hf download unsloth/Qwen3.8-27B-GGUF --include "Q8_0"~~
	hf download unsloth/Qwen3.8-27B-GGUF --include "mmproj-BF16*"		hf download unsloth/Qwen3.8-27B-GGUF --include "mmproj-BF16*"
	hf download unsloth/Qwen3.8-27B-GGUF --include "*mtp-Qwen3.8-27B-Q4_0.gguf"		hf download unsloth/Qwen3.8-27B-GGUF --include "*mtp-Qwen3.8-27B-Q4_0.gguf"

			# list local tree
	find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf"		find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf"
	find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf" -exec sh -c 'rm -f "$1" "$(readlink -f "$1")"' _ {} \;		find ~/.cache/huggingface/hub -name "*Qwen3.8-27B-UD-Q8_K_L.gguf" -exec sh -c 'rm -f "$1" "$(readlink -f "$1")"' _ {} \;

	tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF		tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF
	</syntaxhighlight>		</syntaxhighlight>

LM/Devstral Small 2 24B

Tacoball — Thu, 22 Oct 2026 08:16:52 GMT

Tacoball deleted page LM/Devstral Small 2 24B content was: "== Benchmark == <syntaxhighlight lang="bash"> ./llama.cpp/sycl/bin/llama-bench \ -hf unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_0 \ -fa on -ctk q8_0 -ctv q8_0 -ngl 99 | model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | ------------------------------ | ---------: | --...", and the only contributor was "Tacoball" (talk)

LM/Qwen 3.8 27B

Tacoball — Thu, 22 Oct 2026 08:11:05 GMT

hf operations

← Older revision		Revision as of 08:11, 22 October 2026
(6 intermediate revisions by the same user not shown)
Line 40:		Line 40:
	\| \|\|		\| \|\|
	\|-		\|-
	\| style="font-size:10pt; text-align:left;" \| Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8		\| style="font-size:10pt; text-align:left;" \| 🔥 Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8 🔥
	\| 2026-10-01 \|\| 4 \|\| 16		\| 2026-10-01 \|\| 4 \|\| 16
	\| 2335 \|\| 110 \|\| 2429 \|\| 190 \|\| 2822 \|\| 286 \|\| 2928 \|\| 506		\| 2335 \|\| 110 \|\| 2429 \|\| 190 \|\| 2822 \|\| 286 \|\| 2928 \|\| 506
Line 50:		Line 50:
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	# search latest vllm models		# search latest vllm models
	hf models ls ~~--search "Qwen3.8 27B w4a8"~~ --apps vllm ~~--expand "downloads,likes,lastModified"~~ --sort last_modified --no-truncate --no-gated ~~--limit 25~~		hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
	~~hf models ls --search "Qwen3.8 27B int8 int4" --apps vllm~~ --expand "downloads,likes,lastModified" ~~--sort last_modified --no-truncate --no-gated~~ --limit 25		--expand "downloads,likes,lastModified" \
	~~hf models ls~~ --search "Qwen3.8 27B int4" ~~--apps vllm --expand "downloads,likes,lastModified" --sort last_modified --no-truncate --no-gated --limit 25~~		--limit 25 --search "Qwen3.8 27B int4"
	~~hf models ls --search "Qwen3.8 27B autoround" --apps vllm --expand "downloads,likes,lastModified" --sort last_modified --no-truncate --no-gated --limit 25~~

	~~# Wrapzii/Swift~~-1.5-Qwen3.8-~~27b~~-~~GPTQ~~-~~Int4~~-~~baked~~-v1-~~embed~~-~~int8~~		hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
			--expand "downloads,likes,lastModified" \
			--limit 25 --search "Qwen3.8 27B int8 int4"

			hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
			--expand "downloads,likes,lastModified" \
			--limit 25 --search "Qwen3.8 27B autoround"

			hf models ls --apps vllm --sort last_modified --no-truncate --no-gated \
			--expand "downloads,likes,lastModified" \
			--limit 25 --search "Qwen3.8 27B w4a8"

			# list / download / tree
	hf models ls --tree -R -h Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8		hf models ls --tree -R -h Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
	hf download Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8		hf download Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8
Line 138:		Line 149:
	tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF		tree -h ~/.cache/huggingface/hub/models--unsloth--Qwen3.8-27B-GGUF
	</syntaxhighlight>		</syntaxhighlight>

	~~== Accuracy ==~~

	https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fl9piThTmmUsePst5w3F2%252Fimage.png%3Falt%3Dmedia%26token%3D821e88e2-ad4e-40bf-adf2-e13837228e82&width=768&dpr=3&quality=100&sign=dd5f7cb5&sv=2

LM

Tacoball — Thu, 22 Oct 2026 07:48:49 GMT

Topics

← Older revision		Revision as of 07:48, 22 October 2026
Line 21:		Line 21:
	: [[LM/llama.cpp for SYCL\|llama.cpp for SYCL]]		: [[LM/llama.cpp for SYCL\|llama.cpp for SYCL]]
	: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]		: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]
	~~: [[LM/llama.cpp for OpenVINO\|llama.cpp for OpenVINO]]~~
	: [[LM/SGLang\|SGLang]]		: [[LM/SGLang\|SGLang]]
	: [[LM/Inference Engine Comparison\|Comparison]]		: [[LM/Inference Engine Comparison\|Comparison]]

LM/llama.cpp for OpenVINO

Tacoball — Thu, 22 Oct 2026 07:48:26 GMT

Tacoball deleted page LM/llama.cpp for OpenVINO content was: "https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md == Build == <syntaxhighlight lang="bash"> sudo apt install ninja-build sudo apt install -y opencl-headers ocl-icd-opencl-dev # 2026-08-31 cmake -B build/openvino \ -G Ninja \ -DGGML_OPENVINO=ON \ -DGGML_NATIVE=ON \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_RPATH="\$ORIGIN/../lib" . #...", and the only contributor was "Tacoball" (talk)

LM

Tacoball — Thu, 22 Oct 2026 07:48:13 GMT

Topics

← Older revision		Revision as of 07:48, 22 October 2026
Line 22:		Line 22:
	: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]		: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]
	: [[LM/llama.cpp for OpenVINO\|llama.cpp for OpenVINO]]		: [[LM/llama.cpp for OpenVINO\|llama.cpp for OpenVINO]]
	~~: [[LM/llama.cpp for TurboQuant\|llama.cpp for TurboQuant]]~~
	: [[LM/SGLang\|SGLang]]		: [[LM/SGLang\|SGLang]]
	: [[LM/Inference Engine Comparison\|Comparison]]		: [[LM/Inference Engine Comparison\|Comparison]]

LM/llama.cpp for TurboQuant

Tacoball — Thu, 22 Oct 2026 07:47:55 GMT

Tacoball deleted page LM/llama.cpp for TurboQuant content was: "== Download source == <syntaxhighlight lang="bash"> sudo apt install git cmake sudo apt install libssl-dev openssl mkdir -p ~/llama.cpp/src git clone https://github.com/TheTom/llama-cpp-turboquant ~/llama.cpp/src-tbq </syntaxhighlight> == List available options == <syntaxhighlight lang="bash"> grep -rn "option(GGML_SYCL" CMakeLists.txt ggml/CMakeLists.txt grep -rn "SYCL...", and the only contributor was "Tacoball" (talk)

LM

Tacoball — Thu, 22 Oct 2026 07:47:07 GMT

Models management

@@ Line 53: / Line 53: @@
 <quickmmd name="models-handling-infra">
-flowchart LR
+flowchart TD
    A(LiteLLM)
    B(vLLM<br>Qwen 3.8 27B)
@@ Line 66: / Line 66: @@
 * llama-swap can launch model, cannot manage users.
 * LiteLLM cannot launch model, can manage users.
 = Investigating llama.cpp on Intel Arc Pro B70 =

LM/Qwen 3.8 27B

Tacoball — Thu, 22 Oct 2026 07:25:26 GMT

Benchmark for community

Show changes

← Older revision		Revision as of 01:58, 26 October 2026
Line 10:		Line 10:
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|
	: [[LM/hf\|hf (Model Management)]]		: [[LM/hf\|hf (Model Management)]]
	: [[LM/~~Ornith 1.5 35B A3B~~\|~~Ornith 1.5 35B A3B~~]]		: [[LM/Flint\|Flint]]
	: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]		: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]
	: [[LM/Gemma 4\|Gemma 4]]		: [[LM/Gemma 4\|Gemma 4]]
	: [[LM/Bonsai 2 27B\|Bonsai 2 27B]]		: [[LM/Bonsai 2 27B\|Bonsai 2 27B]]
			: [[LM/Ornith 1.5 35B A3B\|Ornith 1.5 35B A3B]]
	: [[LM/Experience\|Experience]]		: [[LM/Experience\|Experience]]
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|

← Older revision		Revision as of 09:19, 22 October 2026
(One intermediate revision by the same user not shown)
Line 10:		Line 10:
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|
	: [[LM/hf\|hf (Model Management)]]		: [[LM/hf\|hf (Model Management)]]
	: [[LM/~~Bonsai 2 27B~~\|~~Bonsai 2 27B~~]]		: [[LM/Ornith 1.5 35B A3B\|Ornith 1.5 35B A3B]]
	: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]		: [[LM/Qwen 3.8 27B\|Qwen 3.8 27B]]
	: [[LM/Gemma 4\|Gemma 4]]		: [[LM/Gemma 4\|Gemma 4]]
	: [[LM/~~Devstral Small~~ 2 ~~24B~~\|~~Devstral Small~~ 2 ~~24B~~]]		: [[LM/Bonsai 2 27B\|Bonsai 2 27B]]
	: [[LM/Experience\|Experience]]		: [[LM/Experience\|Experience]]
	\| style="width:200px;vertical-align:top" \|		\| style="width:200px;vertical-align:top" \|

← Older revision		Revision as of 07:48, 22 October 2026
Line 21:		Line 21:
	: [[LM/llama.cpp for SYCL\|llama.cpp for SYCL]]		: [[LM/llama.cpp for SYCL\|llama.cpp for SYCL]]
	: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]		: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]
	~~: [[LM/llama.cpp for OpenVINO\|llama.cpp for OpenVINO]]~~
	: [[LM/SGLang\|SGLang]]		: [[LM/SGLang\|SGLang]]
	: [[LM/Inference Engine Comparison\|Comparison]]		: [[LM/Inference Engine Comparison\|Comparison]]

← Older revision		Revision as of 07:48, 22 October 2026
Line 22:		Line 22:
	: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]		: [[LM/llama.cpp for CUDA\|llama.cpp for CUDA]]
	: [[LM/llama.cpp for OpenVINO\|llama.cpp for OpenVINO]]		: [[LM/llama.cpp for OpenVINO\|llama.cpp for OpenVINO]]
	~~: [[LM/llama.cpp for TurboQuant\|llama.cpp for TurboQuant]]~~
	: [[LM/SGLang\|SGLang]]		: [[LM/SGLang\|SGLang]]
	: [[LM/Inference Engine Comparison\|Comparison]]		: [[LM/Inference Engine Comparison\|Comparison]]