In-cluster LLM inference via llama.cpp's llama-server, serving local models on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified memory: 32 GiB RAM / 96 GiB VRAM).

This replaces the bare-metal Ollama setup for models that benefit from always-loaded weights + tuned batching. LiteLLM (litellm/) points at these in-cluster Services instead of the external 10.88.20.12:11434 Ollama endpoint.

Layout

One Deployment + Service per model, all in namespace llamacpp, all pinned to the NUCBox (nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}):

Alias	Model	GGUF	Service
`qwen3.6-27b`	Qwen3.6-27B (dense)	unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB)	`llamacpp-qwen36-27b.llamacpp:80`
`qwen3.6-36b-a3b`	Qwen3.6-35B-A3B (MoE/flash)	unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB)	`llamacpp-qwen36-36b-a3b.llamacpp:80`

The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so inference is significantly faster than the dense 27B despite more total weights — hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for deeper reasoning.

Model files are downloaded idempotently by an initContainer into a shared hostPath PVC (/data/llamacpp/models on the NUCBox), so pods survive reboots without re-downloading.

GPU / Vulkan

The server-vulkan image (ghcr.io/ggml-org/llama.cpp:server-vulkan) bundles the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full layer offload (-ngl 999) puts model weights entirely in the 96 GiB VRAM pool.

The container mounts /dev/dri and runs privileged: true — the simplest reliable way to give Vulkan access to the DRM render node on k3s without a device plugin. Tighten later with supplementalGroups (the host's render group GID) if desired.

Verify the GPU is actually used

kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b     | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-36b-a3b  | grep -iE 'vulkan|gpu|offload|device'

If only a CPU device shows up, the container can't see the GPU — check that /dev/dri/renderD128 exists on the NUCBox and that the amdgpu module is loaded.

VRAM budget (both models co-resident)

Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:

Model	Weights	KV cache (32k×4)	Subtotal
qwen3.6-27b	~16 GiB	~34 GiB	~50 GiB
qwen3.6-36b-a3b	~20 GiB	~10 GiB	~30 GiB
Total			~80 GiB

~16 GiB headroom — comfortable but not infinite. If VRAM is exhausted ( Vulkan allocation failures in logs), reduce -c on the 27B (its KV cache dominates) or drop -np to 2 on either model.

Tuning

The key knobs (in each deployment-*.yaml):

-ngl 999 — offload all layers to GPU. Reduce only if VRAM is tight.
-c 32768 — total KV-cache context. With -np 4 this is 8192 tokens per concurrent request. The 27B's dense KV cache is the larger consumer (~34 GiB at 32k×4); the MoE's is much smaller (~10 GiB).
-np 4 — parallel slots (concurrent requests). Each extra slot multiplies KV-cache VRAM usage. Bump higher on the flash model if you need more throughput (it has VRAM headroom).
--threads 8 — CPU threads for sampling/overhead. Mostly irrelevant under full GPU offload; tune if CPU-bound.

Memory accounting

k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is reserved by firmware and managed by amdgpu). The model weights and KV cache live in VRAM and are not counted against the container's cgroup memory limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages during load. If the pod is OOM-killed during model load, raise the memory limit.

Adding a model

Copy deployment-qwen36-27b.yaml → deployment-<new>.yaml; change the model: label, GGUF URL/file, --alias, and Service name.

Point LiteLLM at it in litellm/litellm.yaml:

- model_name: <alias>
  litellm_params:
    model: openai/<alias>
    api_base: http://<service>.llamacpp/v1
    api_key: "sk-no-auth"

(No gen-apps.sh change needed — the llamacpp app already syncs the whole directory recursively.)
Check the VRAM budget table above — two large models may not coexist.

README.md Unescape Escape

llama.cpp (llama-server)