LM: Difference between revisions
Jump to navigation
Jump to search
| (8 intermediate revisions by the same user not shown) | |||
| Line 10: | Line 10: | ||
| style="width:200px;vertical-align:top" | | | style="width:200px;vertical-align:top" | | ||
: [[LM/hf|hf (Model Management)]] | : [[LM/hf|hf (Model Management)]] | ||
: [[LM/ | : [[LM/Flint|Flint]] | ||
: [[LM/Qwen 3.8 27B|Qwen 3.8 27B]] | : [[LM/Qwen 3.8 27B|Qwen 3.8 27B]] | ||
: [[LM/Gemma 4|Gemma 4]] | : [[LM/Gemma 4|Gemma 4]] | ||
: [[LM/ | : [[LM/Bonsai 2 27B|Bonsai 2 27B]] | ||
: [[LM/Ornith 1.5 35B A3B|Ornith 1.5 35B A3B]] | |||
: [[LM/Experience|Experience]] | : [[LM/Experience|Experience]] | ||
| style="width:200px;vertical-align:top" | | | style="width:200px;vertical-align:top" | | ||
| Line 21: | Line 22: | ||
: [[LM/llama.cpp for SYCL|llama.cpp for SYCL]] | : [[LM/llama.cpp for SYCL|llama.cpp for SYCL]] | ||
: [[LM/llama.cpp for CUDA|llama.cpp for CUDA]] | : [[LM/llama.cpp for CUDA|llama.cpp for CUDA]] | ||
: [[LM/SGLang|SGLang]] | : [[LM/SGLang|SGLang]] | ||
: [[LM/Inference Engine Comparison|Comparison]] | : [[LM/Inference Engine Comparison|Comparison]] | ||
| Line 53: | Line 52: | ||
<quickmmd name="models-handling-infra"> | <quickmmd name="models-handling-infra"> | ||
flowchart | flowchart TD | ||
A(LiteLLM) | A(LiteLLM) | ||
B(vLLM<br>Qwen 3.8 27B) | B(vLLM<br>Qwen 3.8 27B) | ||
| Line 69: | Line 68: | ||
= Investigating vLLM on Intel Arc Pro B70 = | = Investigating vLLM on Intel Arc Pro B70 = | ||
<quickmmd name="tinkering-b70"> | <quickmmd name="tinkering-b70-vllm"> | ||
flowchart TD | flowchart TD | ||
subgraph Hardware | subgraph Hardware | ||
| Line 84: | Line 83: | ||
subgraph Driver | subgraph Driver | ||
C1(Install PPA driver<br>26.27.39122.14):::node-okay | C1(Install PPA driver<br>26.27.39122.14):::node-okay | ||
B1 --> C1 | B1 --> C1 | ||
end | end | ||
| Line 99: | Line 91: | ||
T3(Install<br>nvtop) | T3(Install<br>nvtop) | ||
T4(Install<br>tmux) | T4(Install<br>tmux) | ||
T5(Install<br> | T5(Install<br>docker) | ||
T1 --> T2 | T1 --> T2 | ||
B1 --> T1 | B1 --> T1 | ||
| Line 106: | Line 97: | ||
B1 --> T4 | B1 --> T4 | ||
B1 --> T5 | B1 --> T5 | ||
end | end | ||
| Line 114: | Line 104: | ||
end | end | ||
subgraph | subgraph Config | ||
V1(Optimize<br>.bashrc) | V1(Optimize<br>.bashrc) | ||
V2( | V2(docker-compose.yaml) | ||
T2 --> V1 | T2 --> V1 | ||
T4 ----> V1 | T4 ----> V1 | ||
T5 --> V2 | |||
end | |||
subgraph Engine | |||
E1(Docker Image<br>vllm-openai-xpu<br>0.30.0+) | |||
C1 --> E1 | |||
T5 --> E1 | |||
M1 --> E1 | |||
V2 --> E1 | |||
end | end | ||
subgraph Service | subgraph Service | ||
end | end | ||
| Line 135: | Line 128: | ||
classDef node-todo fill:#eee,stroke:#777 | classDef node-todo fill:#eee,stroke:#777 | ||
</quickmmd> | </quickmmd> | ||
= Investigating llama.cpp on Intel Arc Pro B70 = | = Investigating llama.cpp on Intel Arc Pro B70 = | ||
Latest revision as of 01:58, 26 October 2026
Topics
| 🤖 Agent | 🧠 Model | 🚀 Engine | ⚙️ GPU & Driver | 📊 Evaluation |
|---|---|---|---|---|
|
News
Models management
- llama-swap can launch model, cannot manage users.
- LiteLLM cannot launch model, can manage users.
Investigating vLLM on Intel Arc Pro B70
Investigating llama.cpp on Intel Arc Pro B70
