LM: Difference between revisions

From Fundamental Ramen
Jump to navigation Jump to search
← Older edit
VisualWikitext
 
(148 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Topics =
= Topics =
{| class="wikitable"
{| class="wikitable"
! Agent || Model || Runner || Driver
! 🤖 Agent / Framework　 || 🧠 Model　 || 🚀 Engine　 || ⚙️ GPU & Driver　|| 📊 Evaluation
|-
|-
| valign="top" |
| style="width:200px;vertical-align:top" |
* [[LLM/jrswab_axe|axe]]
: [[LM/Codex|Codex]]
* SKILL
: [[LM/AnythingLLM|AnythingLLM]]
* MCP
: [[LM/jrswab_axe|axe]]
| valign="top" |
: [[LM/MCP|MCP]]
* Management
: [[LM/Software Development|Software Development]]
| valign="top" |
| style="width:200px;vertical-align:top" |
* SGLang
: [[LM/hf|hf (Model Management)]]
* llama.cpp for SYCL
: [[LM/Qwen 4 27B|Qwen 4 27B]] 🐣
* llama.cpp for OpenVINO
: [[LM/Qwen 3.8 27B|Qwen 3.8 27B]] ⭐
* lemonade for AMD NPU
: [[LM/Flint 40B|Flint 40B]] 🐣
| valign="top" |
: [[LM/Gemma 4|Gemma 4]]
* xe Driver
: [[LM/Bonsai 2 27B|Bonsai 2 27B]]
* oneAPI
: [[LM/Ornith 1.5 35B A3B|Ornith 1.5 35B A3B]]
* OpenVINO
: [[LM/Experience|Experience]]
| style="width:200px;vertical-align:top" |
: [[LM/vLLM|vLLM]] ⭐
: [[LM/SGLang|SGLang]] 🐣
: [[LM/llama-swap|llama-swap]]
: [[LM/ComfyUI|ComfyUI]]
: [[LM/llama.cpp for SYCL|llama.cpp for SYCL]]
: [[LM/llama.cpp for CUDA|llama.cpp for CUDA]]
: [[LM/Inference Engine Comparison|Comparison]]
| style="width:200px;vertical-align:top" |
: Intel Arc Pro B70
:: [[LM/Install driver|Install driver]]
:: [[LM/Install oneAPI|Install oneAPI (SYCL)]]
:: [[LM/Install OpenVINO|Install OpenVINO]]
:: [[LM/.bashrc for B70|.bashrc for B70]]
:: [[LM/Intel Pro Arc B70|Experiment]]
: NVIDIA RTX 3060
:: [[LM/Install NVIDIA driver|Install driver]]
: AMD Ryzen AI 7 350
:: [[LM/Ryzen AI 7 350|Experiment]]
: Tools
:: [[LM/Install nvtop|Install nvtop]]
:: [[LM/PCIe Trouble Shooting|PCIe Trouble Shooting]]
: Comparison
:: [[LM/GPU Comparison]]
| style="width:200px;vertical-align:top" |
: [[LM/Performance|Performance]]
|}
|}


= News =
= News =


= Decision Tree for tinkering with Intel Arc Pro B70 =
* [https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro?leaderboard_max_params=32B SWE Pro Benchmark for models < 32B]
* [https://storage.openvinotoolkit.org/repositories/openvino/packages/2026.3/linux/ OpenVINO 2026.3 supports Ubuntu 26] - 2026-08-04
 
= Models management =
 
<quickmmd name="models-handling-infra" dump-env="true">
flowchart TD
  A(LiteLLM 2)
  B(vLLM<br>Qwen 3.8 27B)
  C1(Intel Arc Pro B70)
  C2(Intel Arc Pro B70)
 
  A --> |Forward| B
  B --> |Tensor Parallelism| C1 & C2
  %% C1 --- |XCCL| C2
</quickmmd>
 
* llama-swap can launch model, cannot manage users.
* LiteLLM cannot launch model, can manage users.
 
= Investigating vLLM on Intel Arc Pro B70 =
 
<quickmmd name="tinkering-b70-vllm">
flowchart TD
  subgraph Hardware
    A1(Intel Arc Pro B70):::node-okay
    A2(Asus ProArt B850 WiFi NEO):::node-okay
    A3(AMD Ryzen 5 9600X):::node-okay
  end
 
  subgraph OS
    B1(Install<br>Ubuntu 26.04):::node-okay
    A1 & A2 & A3 --> B1
  end
 
  subgraph Driver
    C1(Install PPA driver<br>26.27.39122.14):::node-okay
    B1 --> C1
  end
 
  subgraph Tools
    T1(Install<br>pipx)
    T2(Install<br>hf)
    T3(Install<br>nvtop)
    T4(Install<br>tmux)
    T5(Install<br>docker)
    T1 --> T2
    B1 --> T1
    B1 --> T3
    B1 --> T4
    B1 --> T5
  end
 
  subgraph Models
    M1(Qwen3.8-27B)
    T2 --> M1
  end
 
  subgraph Config
    V1(Optimize<br>.bashrc)
    V2(docker-compose.yaml)
    T2 --> V1
    T4 ----> V1
    T5 --> V2
  end
 
  subgraph Engine
    E1(Docker Image<br>vllm-openai-xpu<br>0.30.0+)
    C1 --> E1
    T5 --> E1
    M1 --> E1
    V2 --> E1
  end
 
  subgraph Service
  end
 
  classDef node-okay fill:#efe,stroke:#393
  classDef node-warn fill:#fff0e0,stroke:#d63
  classDef node-fail fill:#fee,stroke:#d33
  classDef node-todo fill:#eee,stroke:#777
</quickmmd>
 
= Investigating llama.cpp on Intel Arc Pro B70 =


<quickmmd name="tinkering-b70">
<quickmmd name="tinkering-b70">
flowchart LR
flowchart TD
   subgraph hardware
   subgraph Hardware
     A[Got B70]
     A1(Intel Arc Pro B70):::node-okay
    A2(Asus ProArt B850 WiFi NEO):::node-okay
    A3(AMD Ryzen 5 9600X):::node-okay
   end
   end


   subgraph os
   subgraph OS
     B1[Install<br>Ubuntu 24.04]:::node-okay
     B1(Install<br>Ubuntu 26.04):::node-okay
     B2[Install<br>Ubuntu 26.04]:::node-warn
     A1 & A2 & A3 --> B1
    A --> B1
    A --> B2
   end
   end


   subgraph drvier
   subgraph Driver
     C1[Install<br>PPA driver]:::node-okay
     C1(Install PPA driver<br>26.27.39122.14):::node-okay
     C2[Install<br>oneAPI]:::node-okay
     C2(Install oneAPI<br>2026.1.1.20260724):::node-okay
    C3[Install<br>OpenVINO]:::node-okay
     C1 --> C2
    C4[Install<br>PPA driver]:::node-warn
    C5[Cannot install<br>oneAPI]:::node-fail
    C6[Cannot install<br>OpenVINO]:::node-fail
     C1 --> C2 & C3
    C4 --> C5 & C6
     B1 --> C1
     B1 --> C1
    B2 --> C4
   end
   end


   subgraph runtime
   subgraph Runtime
     D1[Build llama.cpp for<br>SYCL]:::node-okay
     D1[Build llama.cpp for<br>SYCL]:::node-okay
    D2[Build llama.cpp for<br>OpenVINO]:::node-okay
    D3[Build SGLang for<br>SYCL]:::node-todo
    D4[Build SGLang for<br>OpenVINO]:::node-todo
     C2 --> D1
     C2 --> D1
    C3 --> D2
    C2 --> D3
    C3 --> D4
   end
   end


   subgraph tools
   subgraph Tools
     T1[Install<br>pipx]
     T1(Install<br>pipx)
     T2[Install<br>hf]
     T2(Install<br>hf)
     T3[Install<br>nvtop]
     T3(Install<br>nvtop)
     T4[Install<br>screen]
     T4(Install<br>tmux)
     T5[Install<br>llama-swap]
     T5(Install<br>lact)
    T6(Install<br>llama-swap)
     T1 --> T2
     T1 --> T2
     B1 --> T1
     B1 --> T1
Line 72: Line 171:
     B1 --> T4
     B1 --> T4
     B1 --> T5
     B1 --> T5
    T5 --> T6
   end
   end


   subgraph models
   subgraph Models
     M1[Muse-Glimmer-30B]
     M1(Qwen3.8-27B)
    M2[Devstral-Small-2]
     T2 --> M1
    M3[Qwen3-Coder-30B]
    M4[Gemma-4-E4B]
     T2 --> M1 & M2 & M3 & M4
   end
   end


   subgraph env
   subgraph config
     V1[Optimize<br>.bashrc]
     V1(Optimize<br>.bashrc)
     V2[Optimize<br>llama-swap.yaml]
     V2(Optimize<br>lmsw.yaml)
     C2 --> V1
     C2 --> V1
    C3 --> V1
     T2 --> V1
     T2 --> V1
     T4 ---> V1
     T4 ----> V1
     T5 ---> V2
     T6 ---> V2
   end
   end


   subgraph service
   subgraph Service
     Z[service]
     Z(service<br>llama-swap \<br>  -config lmsw.yaml \<br>  -listen 0.0.0.0:9876)
     M1 --> Z
     M1 --> Z
     D1 --> Z
     D1 --> Z
Line 104: Line 200:
   classDef node-todo fill:#eee,stroke:#777
   classDef node-todo fill:#eee,stroke:#777
</quickmmd>
</quickmmd>
= Build environment =
== hf (model management) ==
* https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables
* https://www.datalearner.com/en/leaderboards/category/code?benchmark=SWE-bench+Verified&modelSize=34b&licenseType=open
* https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro
{| class="wikitable"
! Purpose || Command
|-
| cache management ||
<syntaxhighlight lang="bash">
hf cache list
hf cache rm <model id>
hf cache prune
</syntaxhighlight>
|-
| fix WiFi problem ||
<syntaxhighlight lang="bash">
HF_XET_FIXED_DOWNLOAD_CONCURRENCY=10 hf download "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF" --include "*UD-Q4_K_XL*"
HF_XET_FIXED_DOWNLOAD_CONCURRENCY=10 hf download "unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF" --include "*UD-Q4_K_XL*"
</syntaxhighlight>
|-
| search ||
<syntaxhighlight lang="bash">
# search by population
hf models ls --search "Muse-Glimmer" --apps llama.cpp --expand "downloads,likes,createdAt,lastModified" --sort downloads --no-truncate --limit 25
# search for latest publish
hf models ls --search "Muse-Glimmer" --apps llama.cpp --expand "downloads,likes,createdAt,lastModified" --sort created_at --no-truncate --limit 25
# search for latest tunning
hf models ls --search "Muse-Glimmer" --apps llama.cpp --expand "downloads,likes,createdAt,lastModified" --sort last_modified --no-truncate --limit 25
</syntaxhighlight>
|-
| optimize Qwen3-Coder ||
<syntaxhighlight lang="bash">
hf models ls --search "coder" --apps llama.cpp --sort downloads --limit 1 --format json | jq .
hf models ls -h "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF"
hf download "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF" --include "*UD-Q4_K_XL*"
</syntaxhighlight>
|-
| optimize gemma-4-E4B ||
<syntaxhighlight lang="bash">
hf models ls -h unsloth/gemma-4-E4B-it-qat-GGUF
hf download unsloth/gemma-4-E4B-it-qat-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/gemma-4-E4B-it-qat-GGUF --include "mmproj-BF16.gguf"
hf download unsloth/gemma-4-E4B-it-qat-GGUF --include "mtp-gemma-4-E4B-it.gguf"
</syntaxhighlight>
|-
| optimize gemma-4-12B ||
<syntaxhighlight lang="bash">
hf models ls -h unsloth/gemma-4-12B-it-qat-GGUF
hf download unsloth/gemma-4-12B-it-qat-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/gemma-4-12B-it-qat-GGUF --include "mmproj-BF16.gguf"
hf download unsloth/gemma-4-12B-it-qat-GGUF --include "mtp-gemma-4-12B-it.gguf"
</syntaxhighlight>
|-
| optimize Muse Glimmer ||
<syntaxhighlight lang="bash">
hf models ls -h unsloth/Muse-Glimmer-30B-GGUF
hf download unsloth/Muse-Glimmer-30B-GGUF --include "*UD-Q4_K_XL*"
hf download unsloth/Muse-Glimmer-30B-GGUF --include "*UD-Q5_K_XL*"
hf download unsloth/Muse-Glimmer-30B-GGUF --include "*UD-Q6_K_XL*"
hf download unsloth/Muse-Glimmer-30B-GGUF --include "dflash-kquant.gguf"
hf download unsloth/Muse-Glimmer-30B-GGUF --include "mmproj-kquant.gguf"
</syntaxhighlight>
|-
|  ||
<syntaxhighlight lang="bash">
tree ~/.cache/huggingface/hub/models--unsloth--Devstral-Small-2-24B-Instruct-2512-GGUF/snapshots
tree ~/.cache/huggingface/hub/models--unsloth--Qwen3-Coder-30B-A3B-Instruct-GGUF/snapshots
</syntaxhighlight>
|}
== Ubuntu ==
* [[LLM/Intel Pro Arc B70]]
* [[LLM/Ryzen AI 7 350]] (XDNA)

Latest revision as of 04:35, 11 November 2026

Topics

🤖 Agent / Framework　 🧠 Model　 🚀 Engine　 ⚙️ GPU & Driver　 📊 Evaluation
Codex
AnythingLLM
axe
MCP
Software Development
hf (Model Management)
Qwen 4 27B 🐣
Qwen 3.8 27B ⭐
Flint 40B 🐣
Gemma 4
Bonsai 2 27B
Ornith 1.5 35B A3B
Experience
vLLM ⭐
SGLang 🐣
llama-swap
ComfyUI
llama.cpp for SYCL
llama.cpp for CUDA
Comparison
Intel Arc Pro B70
Install driver
Install oneAPI (SYCL)
Install OpenVINO
.bashrc for B70
Experiment
NVIDIA RTX 3060
Install driver
AMD Ryzen AI 7 350
Experiment
Tools
Install nvtop
PCIe Trouble Shooting
Comparison
LM/GPU Comparison
Performance

News

Models management

File Path/images/quickmmd/LM-models-handling-infra.svg
File Size3.07 KB
Last modified07:29:34
Time of SVG loading0.00 second(s)
Time of SVG conversion0.19 second(s)
Cache DetectedYes
SVG Enginepele
Node Path/usr/bin/node
Mermaid syntax Referencehttps://mermaid.js.org/intro/syntax-reference.html
QuickMMD Version1.0.0 - About QuickMMD
  • llama-swap can launch model, cannot manage users.
  • LiteLLM cannot launch model, can manage users.

Investigating vLLM on Intel Arc Pro B70

Investigating llama.cpp on Intel Arc Pro B70

Retrieved from "https://wiki.fundamental-ramen.com/index.php?title=LM&oldid=4219"