llama.cpp (llama-server)

In-cluster LLM inference via llama.cpp's llama-server, serving local models on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified memory: 32 GiB RAM / 96 GiB VRAM).

This replaces the bare-metal Ollama setup for models that benefit from always-loaded weights + tuned batching. LiteLLM (litellm/) points at these in-cluster Services instead of the external 10.88.20.12:11434 Ollama endpoint.

Layout

One Deployment + Service per model, all in namespace llamacpp, all pinned to the NUCBox (nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}):

Alias	Model	GGUF	Service
`qwen3.6-27b`	Qwen3.6-27B (dense)	unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB)	`llamacpp-qwen36-27b.llamacpp:80`
`qwen3.6-35b-a3b`	Qwen3.6-35B-A3B (MoE/flash)	unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB)	`llamacpp-qwen36-35b-a3b.llamacpp:80`

The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so inference is significantly faster than the dense 27B despite more total weights — hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for deeper reasoning.

Model files are downloaded idempotently by an initContainer into a shared hostPath PVC (/data/llamacpp/models on the NUCBox), so pods survive reboots without re-downloading.

GPU / Vulkan

The server-vulkan image (ghcr.io/ggml-org/llama.cpp:server-vulkan) bundles the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full layer offload (-ngl 999) puts model weights entirely in the 96 GiB VRAM pool.

The container mounts /dev/dri and runs privileged: true — the simplest reliable way to give Vulkan access to the DRM render node on k3s without a device plugin. Tighten later with supplementalGroups (the host's render group GID) if desired.

Verify the GPU is actually used

kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b     | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b  | grep -iE 'vulkan|gpu|offload|device'

If only a CPU device shows up, the container can't see the GPU — check that /dev/dri/renderD128 exists on the NUCBox and that the amdgpu module is loaded.

VRAM budget (both models co-resident)

Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:

Model	Weights	KV cache	Subtotal
qwen3.6-27b	~16 GiB	~16 GiB (q8_0, 131k total, 65k/slot)	~32 GiB
qwen3.6-35b-a3b	~20 GiB	~4.5 GiB (q8_0, 131k total, 65k/slot)	~24.5 GiB
Total			~56.5 GiB

~39.5 GiB headroom — still comfortable. Both models' KV caches are quantized to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.

Tuning

The key knobs (in each deployment-*.yaml):

-ngl 999 — offload all layers to GPU. Reduce only if VRAM is tight.
-c 131072 (both) — total KV-cache context. With -np 2 each slot gets 65536 tokens. The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B also uses q8_0 KV to keep 131k affordable (~16 GiB).
-np 2 (both) — parallel slots (concurrent requests). Each extra slot multiplies KV-cache VRAM usage. 2 slots give each cron/request a large context slice (65k); bump higher only if you need more concurrent throughput (both have VRAM headroom, but the 27B's dense KV is the constraint).
--cache-type-k q8_0 --cache-type-v q8_0 (both) — quantize the KV cache to q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense 27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b. Drop to q4_0 for even less VRAM if retrieval quality allows.
--threads 8 — CPU threads for sampling/overhead. Mostly irrelevant under full GPU offload; tune if CPU-bound.

Memory accounting

k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is reserved by firmware and managed by amdgpu). The model weights and KV cache live in VRAM and are not counted against the container's cgroup memory limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages during load. If the pod is OOM-killed during model load, raise the memory limit.

Adding a model

Copy deployment-qwen36-27b.yaml → deployment-<new>.yaml; change the model: label, GGUF URL/file, --alias, and Service name.

Point LiteLLM at it in litellm/litellm.yaml:

- model_name: <alias>
  litellm_params:
    model: openai/<alias>
    api_base: http://<service>.llamacpp/v1
    api_key: "sk-no-auth"

(No gen-apps.sh change needed — the llamacpp app already syncs the whole directory recursively.)
Check the VRAM budget table above — two large models may not coexist.

5.4 KiB Raw Blame History