k3s-cluster/llamacpp/README.md

llama.cpp (llama-server)

In-cluster LLM inference via llama.cpp's llama-server, serving a local model on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB unified memory).

LiteLLM (litellm/) points at the in-cluster Service instead of the external Ollama endpoint.

Active Model

Alias	Model	Configuration	Service
qwen3.8-27b	Qwen3.8-27B with MTP	Q4_K_M primary, Q4_0 draft, 196k context, q8_0 K/V cache	llamacpp-qwen38-27b.llamacpp:80

An initContainer downloads both model files atomically before llama-server starts:

• Primary: ggml-org/Qwen3.8-27B-GGUF:Q4_K_M
• Draft: ggml-org/Qwen3.8-27B-GGUF:Q4_0

The model files are stored on the shared hostPath PVC at /data/llamacpp/models on the NUCBox. The server is configured with --spec-default --spec-type draft-mtp, --reasoning-preserve, --fit off, and --agent.

Retired Models

The following deployments are no longer active, but their argument references are retained for future redeployment:

• DeepSeek-V4-Flash-0731
• Qwen3.6-27B
• Qwen3.6-35B-A3B

GPU / Vulkan

The server-vulkan image (ghcr.io/ggml-org/llama.cpp:server-vulkan) bundles the Mesa/RADV Vulkan driver for the Radeon 8060S. The container mounts /dev/dri and runs privileged, which is the current way to provide Vulkan access on k3s without a device plugin.

Verify GPU use with:

kubectl logs -n llamacpp deploy/llamacpp-qwen38-27b | grep -iE 'vulkan|gpu|offload|device'

If only a CPU device appears, check that /dev/dri/renderD128 exists on the NUCBox and that the amdgpu module is loaded.

Adding Or Replacing A Model

1. Copy deployment-qwen38-27b.yaml to deployment-<new>.yaml and change the model repositories, alias, labels, and Service name.
2. Add the matching alias and Service URL to litellm/litellm.yaml.
3. No gen-apps.sh change is needed because the llamacpp ArgoCD Application syncs the directory recursively.
4. Check the model and KV-cache size against the NUCBox's available VRAM.