In-cluster LLM inference via llama.cpp's llama-server, serving local models on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified memory: 32 GiB RAM / 96 GiB VRAM).

This replaces the bare-metal Ollama setup for models that benefit from always-loaded weights + tuned batching. LiteLLM (litellm/) points at these in-cluster Services instead of the external 10.88.20.12:11434 Ollama endpoint.

Layout

One Deployment + Service per model, all in namespace llamacpp, all pinned to the NUCBox (nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}):

Alias	Model	GGUF	Service
`qwen3.6-27b`	Qwen3.6-27B (dense)	unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB)	`llamacpp-qwen36-27b.llamacpp:80`
`qwen3.6-35b-a3b`	Qwen3.6-35B-A3B (MoE/flash)	unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB)	`llamacpp-qwen36-35b-a3b.llamacpp:80`

The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so inference is significantly faster than the dense 27B despite more total weights — hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for deeper reasoning.

Model files are downloaded idempotently by an initContainer into a shared hostPath PVC (/data/llamacpp/models on the NUCBox), so pods survive reboots without re-downloading.

GPU / Vulkan

The server-vulkan image (ghcr.io/ggml-org/llama.cpp:server-vulkan) bundles the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full layer offload (-ngl 999) puts model weights entirely in the 96 GiB VRAM pool.

The container mounts /dev/dri and runs privileged: true — the simplest reliable way to give Vulkan access to the DRM render node on k3s without a device plugin. Tighten later with supplementalGroups (the host's render group GID) if desired.

Verify the GPU is actually used

kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b     | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b  | grep -iE 'vulkan|gpu|offload|device'

If only a CPU device shows up, the container can't see the GPU — check that /dev/dri/renderD128 exists on the NUCBox and that the amdgpu module is loaded.

VRAM budget (both models co-resident)

Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:

Model	Weights	KV cache	Subtotal
qwen3.6-27b	~16 GiB	~16 GiB (q8_0, 131k total, 1 slot)	~32 GiB
qwen3.6-35b-a3b	~20 GiB	~9 GiB (q8_0, 262k total, 131k/slot)	~29 GiB
Total			~61 GiB

~35 GiB headroom — comfortable. Both models' KV caches are quantized to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.

Tuning

The key knobs (in each deployment-*.yaml):

-ngl 999 — offload all layers to GPU. Reduce only if VRAM is tight.
-c — total KV-cache context. The 27B runs -c 131072 -np 1 (single slot gets the full 131k); the a3b runs -c 262144 -np 2 (2 slots × 131k each). The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
-np — parallel slots (concurrent requests). Each extra slot multiplies KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request, dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV). Bump higher only if you need more concurrent throughput.
--cache-type-k q8_0 --cache-type-v q8_0 (both) — quantize the KV cache to q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense 27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b. Drop to q4_0 for even less VRAM if retrieval quality allows.
--threads 8 — CPU threads for sampling/overhead. Mostly irrelevant under full GPU offload; tune if CPU-bound.

Memory accounting

k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is reserved by firmware and managed by amdgpu). The model weights and KV cache live in VRAM and are not counted against the container's cgroup memory limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages during load. If the pod is OOM-killed during model load, raise the memory limit.

Adding a model

Copy deployment-qwen36-27b.yaml → deployment-<new>.yaml; change the model: label, GGUF URL/file, --alias, and Service name.

Point LiteLLM at it in litellm/litellm.yaml:

- model_name: <alias>
  litellm_params:
    model: openai/<alias>
    api_base: http://<service>.llamacpp/v1
    api_key: "sk-no-auth"

(No gen-apps.sh change needed — the llamacpp app already syncs the whole directory recursively.)
Check the VRAM budget table above — two large models may not coexist.

README.md Unescape Escape

llama.cpp (llama-server)