# llama.cpp (llama-server) In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB unified memory: ~90 GiB VRAM / 30 GiB CPU RAM). LiteLLM (`litellm/`) points at these in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint. ## Layout One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`): | Alias | Model | GGUF | Service | Args ref | |--------------------------|--------------------------------|-------------------------------------------------------|------------------------------------------------------|----------| | `deepseek-v4-flash-0731` | DeepSeek-V4-Flash-0731 (MoE) | unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB) | `llamacpp-deepseek-v4-flash-0731.llamacpp:80` | [args-deepseek-v4-flash-0731.md](args-deepseek-v4-flash-0731.md) | | `qwen3.5-4b` | Qwen3.5-4B (MTP) | unsloth/Qwen3.5-4B-MTP-GGUF (`UD-Q4_K_XL`) | `llamacpp-qwen35-4b.llamacpp:80` | — | DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per token) with MLA attention, so only a small fraction of the weights is computed per token. The full ~87 GiB of IQ1_M weights is loaded into the unified memory pool. > **Previously** the NUCBox ran two co-resident Qwen3.6 models (a 27B dense and > a 35B-A3B MoE "flash"). DeepSeek-V4-Flash-0731 (~87 GiB) nearly fills the > whole 90 GiB VRAM pool on its own, so both Qwen models were removed to make > room. Their GGUF files are deleted from the shared PVC by the new pod's > `fetch-model` initContainer on first boot. Their deployment arguments are > still documented for redeployment: > - [args-qwen36-27b.md](args-qwen36-27b.md) — dense 27B (deeper reasoning) > - [args-qwen36-35b-a3b.md](args-qwen36-35b-a3b.md) — MoE 35B-A3B "flash" (fast) Model files are downloaded idempotently by an initContainer into a shared hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots without re-downloading. The UD-IQ1_M GGUF is split across 3 shards (`-00001-of-00003` … `-00003-of-00003`); llama.cpp auto-loads all shards when pointed at the first one. ## GPU / Vulkan The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). The Vulkan backend supports the `IQ1_M` matmul (including the MoE `matmul_id` variant), so the whole model runs on the GPU. `deepseek4` is a brand-new arch (2026-07), so the floating `server-vulkan` tag is used to pull a recent enough build — pin to a specific `server-vulkan-bXXXX` tag once a known-good one is verified. The container mounts `/dev/dri` and runs `privileged: true` — the simplest reliable way to give Vulkan access to the DRM render node on k3s without a device plugin. Tighten later with `supplementalGroups` (the host's `render` group GID) if desired. ### Verify the GPU is actually used ```bash kubectl logs -n llamacpp deploy/llamacpp-deepseek-v4-flash-0731 | grep -iE 'vulkan|gpu|offload|device' ``` If only a CPU device shows up, the container can't see the GPU — check that `/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded. ## VRAM budget (single model) The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead `-ngl 38` offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and fragmentation. **KV cache is f16, not q8_0** — the Vulkan backend has no Flash Attention for the `deepseek4` arch, and quantized V cache requires Flash Attention (llama.cpp hard-errors: *"quantized V cache was requested, but this requires Flash Attention"*). `deepseek4`/MLA models also require K and V cache types to be *identical*, so K cannot be quantized either. f16 MLA KV at 64k is ~5.7 GiB (576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes) — larger than q8_0 would be, which is why `-ngl` is 38 rather than 40. Approximate VRAM usage: | Component | VRAM | |---------------------------------|-------------| | Weights (38 GPU layers) | ~77 GiB | | KV cache (f16, 64k, 1 slot) | ~5.7 GiB | | Vulkan compute buffers | ~2 GiB | | **Total in VRAM** | **~85 GiB** | | **Headroom (of 90 GiB)** | **~5 GiB** | 5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory limit, not VRAM). VRAM is exclusive to this model; the other NUCBox pods only compete for the 30 GiB CPU RAM. Several `deepseek4`-specific fused ops (Lightning Indexer, HC pre/comb/post) are not yet implemented in the Vulkan backend and fall back to CPU (logged as warnings, not fatal). Inference still works; it will speed up once those ops land in a future `server-vulkan` build. ## Tuning The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`): - `-ngl 38` — offload 38 of 43 layers to GPU. The model (~87 GiB) is nearly the whole 90 GiB VRAM pool, so full offload would overflow once the f16 KV cache + compute buffers are added. 5 layers (~10 GiB) on CPU leaves ~5 GiB VRAM headroom. Raise toward 43 if VRAM allows; lower (e.g. 36) if the pod OOMs / Vulkan runs out of device memory. - `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets the full 64k. f16 MLA KV at 64k is ~5.7 GiB; capped at the minimum to maximise VRAM headroom. Raise if headroom allows. - `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request). Extra slots multiply the f16 KV cost (~5.7 GiB/slot); 1 slot keeps headroom maximal. - `--cache-type-k f16 --cache-type-v f16` — **f16 KV cache (NOT quantized).** The Vulkan backend has no Flash Attention for `deepseek4`, and quantized V cache requires Flash Attention. `deepseek4`/MLA models also require K and V cache types to be identical, so K cannot be quantized either. This is the reason `-ngl` is 38 rather than 40. - `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4 recommendation). These are server defaults; clients can override per request via the OpenAI-compatible API. - `--threads 8` — CPU threads for sampling + the 5 CPU-resident layers. ## Memory accounting k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is reserved by firmware and managed by `amdgpu`). The GPU-resident model weights and KV cache live in VRAM and are **not** counted against the container's cgroup memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages for the 5 CPU-resident layers (~10 GiB, resident during inference), and reclaimable page cache during load. If the pod is OOM-killed during model load or inference, raise the memory limit (and/or lower `-ngl` to push more layers to VRAM). ## Adding / replacing a model 1. Copy `deployment-deepseek-v4-flash-0731.yaml` → `deployment-.yaml`; change the `model:` label, GGUF URL/file(s), `--alias`, and Service name. For split GGUFs, point `-m` at the first shard and download all shards in the `fetch-model` initContainer. 2. Point LiteLLM at it in `litellm/litellm.yaml`: ```yaml - model_name: litellm_params: model: openai/ api_base: http://.llamacpp/v1 api_key: "sk-no-auth" ``` 3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole directory recursively.) 4. Check the VRAM budget table above — at ~87 GiB this model nearly fills the 90 GiB pool on its own, so co-locating another large model is not possible. The `qwen3.5-4b` alias is exposed through LiteLLM as a low-latency option for Home Assistant voice Assist. Select this model in the Home Assistant conversation/voice Assist provider configuration; the LiteLLM endpoint is `http://litellm-service.litellm:80/v1` from inside the cluster.