llama.cpp (llama-server)

In-cluster LLM inference via llama.cpp's llama-server, serving a local model on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB unified memory: ~90 GiB VRAM / 30 GiB CPU RAM).

LiteLLM (litellm/) points at these in-cluster Services instead of the external 10.88.20.12:11434 Ollama endpoint.

Layout

One Deployment + Service per model, all in namespace llamacpp, all pinned to the NUCBox (nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}):

Alias	Model	GGUF	Service	Args ref
`deepseek-v4-flash-0731`	DeepSeek-V4-Flash-0731 (MoE)	unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB)	`llamacpp-deepseek-v4-flash-0731.llamacpp:80`	args-deepseek-v4-flash-0731.md

DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per token) with MLA attention, so only a small fraction of the weights is computed per token. The full ~87 GiB of IQ1_M weights is loaded into the unified memory pool.

Previously the NUCBox ran two co-resident Qwen3.6 models (a 27B dense and a 35B-A3B MoE "flash"). DeepSeek-V4-Flash-0731 (~87 GiB) nearly fills the whole 90 GiB VRAM pool on its own, so both Qwen models were removed to make room. Their GGUF files are deleted from the shared PVC by the new pod's fetch-model initContainer on first boot. Their deployment arguments are still documented for redeployment:

args-qwen36-27b.md — dense 27B (deeper reasoning)

args-qwen36-35b-a3b.md — MoE 35B-A3B "flash" (fast)

Model files are downloaded idempotently by an initContainer into a shared hostPath PVC (/data/llamacpp/models on the NUCBox), so pods survive reboots without re-downloading. The UD-IQ1_M GGUF is split across 3 shards (-00001-of-00003 … -00003-of-00003); llama.cpp auto-loads all shards when pointed at the first one.

GPU / Vulkan

The server-vulkan image (ghcr.io/ggml-org/llama.cpp:server-vulkan) bundles the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). The Vulkan backend supports the IQ1_M matmul (including the MoE matmul_id variant), so the whole model runs on the GPU. deepseek4 is a brand-new arch (2026-07), so the floating server-vulkan tag is used to pull a recent enough build — pin to a specific server-vulkan-bXXXX tag once a known-good one is verified.

The container mounts /dev/dri and runs privileged: true — the simplest reliable way to give Vulkan access to the DRM render node on k3s without a device plugin. Tighten later with supplementalGroups (the host's render group GID) if desired.

Verify the GPU is actually used

kubectl logs -n llamacpp deploy/llamacpp-deepseek-v4-flash-0731 | grep -iE 'vulkan|gpu|offload|device'

If only a CPU device shows up, the container can't see the GPU — check that /dev/dri/renderD128 exists on the NUCBox and that the amdgpu module is loaded.

VRAM budget (single model)

The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so it cannot be fully offloaded: -ngl 999 would try to put all 43 layers into VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead -ngl 40 offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and co-resident pods. Approximate VRAM usage:

Component	VRAM
Weights (40 GPU layers)	~81 GiB
KV cache (q8_0, 64k, 1 slot)	~1.6 GiB
Vulkan compute buffers	~2 GiB
Total in VRAM	~85 GiB
Headroom (of 90 GiB)	~5–8 GiB

3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory limit, not VRAM).

KV cache is tiny thanks to DeepSeek-V4's MLA attention (num_kv_heads=1, head_dim=512 + 64 decoupled RoPE ⇒ ~576 elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is cheap — -c is capped at 65536 (the required minimum) to maximise VRAM headroom, not because KV is the constraint.

Tuning

The key knobs (in deployment-deepseek-v4-flash-0731.yaml):

-ngl 40 — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom. Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs out of device memory.
-c 65536 — total KV-cache context (64k, the required minimum). 1 slot gets the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped at the minimum to maximise VRAM headroom. Raise if headroom allows.
-np 1 — 1 parallel slot (the full 64k goes to a single concurrent request). Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal.
--cache-type-k q8_0 --cache-type-v q8_0 — quantize the KV cache to q8_0, halving KV VRAM with ~negligible quality loss. Essential to keep headroom.
--temp 1.0 --top-p 0.95 — default sampling parameters (DeepSeek-V4 recommendation). These are server defaults; clients can override per request via the OpenAI-compatible API.
--threads 8 — CPU threads for sampling + the 3 CPU-resident layers.

Memory accounting

k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is reserved by firmware and managed by amdgpu). The GPU-resident model weights and KV cache live in VRAM and are not counted against the container's cgroup memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. If the pod is OOM-killed during model load, raise the memory limit (and/or lower -ngl).

Adding / replacing a model

Copy deployment-deepseek-v4-flash-0731.yaml → deployment-<new>.yaml; change the model: label, GGUF URL/file(s), --alias, and Service name. For split GGUFs, point -m at the first shard and download all shards in the fetch-model initContainer.

Point LiteLLM at it in litellm/litellm.yaml:

- model_name: <alias>
  litellm_params:
    model: openai/<alias>
    api_base: http://<service>.llamacpp/v1
    api_key: "sk-no-auth"

(No gen-apps.sh change needed — the llamacpp app already syncs the whole directory recursively.)
Check the VRAM budget table above — at ~87 GiB this model nearly fills the 90 GiB pool on its own, so co-locating another large model is not possible.

6.9 KiB Raw Blame History Unescape Escape