In-cluster LLM inference via llama.cpp's llama-server, serving a local model on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB unified memory).

LiteLLM (litellm/) points at the in-cluster Service instead of the external Ollama endpoint.

Active Model

Alias	Model	Configuration	Service
`qwen3.8-27b`	Qwen3.8-27B with MTP	`Q4_K_M` primary, `Q4_0` draft, 196k context, q8_0 K/V cache	`llamacpp-qwen38-27b.llamacpp:80`

The active Deployment uses llama.cpp's Hugging Face downloader for both model repositories:

Primary: ggml-org/Qwen3.8-27B-GGUF:Q4_K_M
Draft: ggml-org/Qwen3.8-27B-GGUF:Q4_0

The model cache is stored on the shared hostPath PVC at /data/llamacpp/models on the NUCBox. The server is configured with --spec-default --spec-type draft-mtp, --reasoning-preserve, --fit off, and --agent.

Retired Models

The following deployments are no longer active, but their argument references are retained for future redeployment:

GPU / Vulkan

The server-vulkan image (ghcr.io/ggml-org/llama.cpp:server-vulkan) bundles the Mesa/RADV Vulkan driver for the Radeon 8060S. The container mounts /dev/dri and runs privileged, which is the current way to provide Vulkan access on k3s without a device plugin.

Verify GPU use with:

kubectl logs -n llamacpp deploy/llamacpp-qwen38-27b | grep -iE 'vulkan|gpu|offload|device'

If only a CPU device appears, check that /dev/dri/renderD128 exists on the NUCBox and that the amdgpu module is loaded.

Adding Or Replacing A Model

Copy deployment-qwen38-27b.yaml to deployment-<new>.yaml and change the model repositories, alias, labels, and Service name.
Add the matching alias and Service URL to litellm/litellm.yaml.
No gen-apps.sh change is needed because the llamacpp ArgoCD Application syncs the directory recursively.
Check the model and KV-cache size against the NUCBox's available VRAM.