llama.cpp (llama-server)

In-cluster LLM inference via llama.cpp's llama-server, serving local models on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified memory: 32 GiB RAM / 96 GiB VRAM).

This replaces the bare-metal Ollama setup for models that benefit from always-loaded weights + tuned batching. LiteLLM (litellm/) points at these in-cluster Services instead of the external 10.88.20.12:11434 Ollama endpoint.

Layout

One Deployment + Service per model, all in namespace llamacpp, all pinned to the NUCBox (nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}):

Model	GGUF	Service	litellm alias
qwen3.6	unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL)	`llamacpp-qwen36.llamacpp:80`	`qwen3.6`

Model files are downloaded idempotently by an initContainer into a shared hostPath PVC (/data/llamacpp/models on the NUCBox), so pods survive reboots without re-downloading.

GPU / Vulkan

The server-vulkan image bundles the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full layer offload (-ngl 999) puts the ~16 GiB Q4 model entirely in the 96 GiB VRAM pool.

The container mounts /dev/dri and runs privileged: true — the simplest reliable way to give Vulkan access to the DRM render node on k3s without a device plugin. Tighten later with supplementalGroups (the host's render group GID) if desired.

Verify the GPU is actually used

kubectl exec -n llamacpp deploy/llamacpp-qwen36 -- \
  llama-server --list-devices -m /models/Qwen3.6-27B-UD-Q4_K_XL.gguf
# or check the startup logs for a "vulkan" device line + ngl offload count
kubectl logs -n llamacpp deploy/llamacpp-qwen36 | grep -iE 'vulkan|gpu|offload|device'

If only a CPU device shows up, the container can't see the GPU — check that /dev/dri/renderD128 exists on the NUCBox and that the amdgpu module is loaded.

Tuning

The key knobs (in deployment-qwen36.yaml):

-ngl 999 — offload all layers to GPU. Reduce only if VRAM is tight (it isn't, with 96 GiB).
-c 32768 — total KV-cache context. With -np 4 this is 8192 tokens per concurrent request. For a 27B model the full 32k×4 KV cache is ~32 GiB of VRAM; raise or lower -c to trade context length for VRAM headroom.
-np 4 — parallel slots (concurrent requests). Matches the requested concurrency. Each extra slot multiplies KV-cache VRAM usage.
--threads 8 — CPU threads for sampling/overhead. Mostly irrelevant under full GPU offload; tune if CPU-bound.

Memory accounting

k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is reserved by firmware and managed by amdgpu). The model weights and KV cache live in VRAM and are not counted against the container's cgroup memory limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages during load. If the pod is OOM-killed during model load, raise the memory limit.

Adding a model

Create deployment-<model>.yaml + service-<model>.yaml (copy the qwen3.6 pair; change model: label, the GGUF URL/file, --alias, and Service name).

Point LiteLLM at it in litellm/litellm.yaml:

- model_name: <alias>            # keep the alias so consumers don't change
  litellm_params:
    model: openai/<alias>
    api_base: http://<service>.llamacpp/v1
    api_key: "sk-no-auth"

(No gen-apps.sh change needed — the llamacpp app already syncs the whole directory recursively.)

TODO

glm-4.7-flash: still served by the external Ollama at 10.88.20.12:11434 in litellm/litellm.yaml. Migrate once a GGUF source is confirmed (add a deployment-glm47-flash.yaml + Service and flip the litellm entry).

3.9 KiB Raw Blame History Unescape Escape