roger/k3s-cluster
1
0
Fork 1
Code Issues Pull Requests 6 Actions Packages Projects Releases Wiki Activity

deploy qwen3.5 4b

Browse Source
This commit is contained in:
Roger Oriol
2026-08-02 11:54:58 +02:00
parent 171178f7d0
commit d928297b64
4 changed files with 151 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
llamacpp/README.md
View File

@@ -15,6 +15,7 @@ to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}
| Alias | Model | GGUF | Service | Args ref |
|--------------------------|--------------------------------|-------------------------------------------------------|------------------------------------------------------|----------|
| `deepseek-v4-flash-0731` | DeepSeek-V4-Flash-0731 (MoE) | unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB) | `llamacpp-deepseek-v4-flash-0731.llamacpp:80` | [args-deepseek-v4-flash-0731.md](args-deepseek-v4-flash-0731.md) |
| `qwen3.5-4b` | Qwen3.5-4B (MTP) | unsloth/Qwen3.5-4B-MTP-GGUF (`UD-Q4_K_XL`) | `llamacpp-qwen35-4b.llamacpp:80` | — |
DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per
token) with MLA attention, so only a small fraction of the weights is computed
@@ -150,3 +151,8 @@ to VRAM).
directory recursively.)
4. Check the VRAM budget table above — at ~87 GiB this model nearly fills the
90 GiB pool on its own, so co-locating another large model is not possible.
The `qwen3.5-4b` alias is exposed through LiteLLM as a low-latency option for
Home Assistant voice Assist. Select this model in the Home Assistant
conversation/voice Assist provider configuration; the LiteLLM endpoint is
`http://litellm-service.litellm:80/v1` from inside the cluster.