# llama.cpp (llama-server)

In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model
on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB
unified memory).

LiteLLM (`litellm/`) points at the in-cluster Service instead of the external
Ollama endpoint.

## Active Model

| Alias | Model | Configuration | Service |
|---|---|---|---|
| `qwen3.8-27b` | Qwen3.8-27B with MTP | `Q4_K_M` primary, `Q4_0` draft, 196k context, q8_0 K/V cache | `llamacpp-qwen38-27b.llamacpp:80` |

An initContainer downloads both model files atomically before llama-server
starts:

- Primary: `ggml-org/Qwen3.8-27B-GGUF:Q4_K_M`
- Draft: `ggml-org/Qwen3.8-27B-GGUF:Q4_0`

The model files are stored on the shared hostPath PVC at
`/data/llamacpp/models` on the NUCBox. The server is configured with
`--spec-default --spec-type draft-mtp`, `--reasoning-preserve`, `--fit off`,
and `--agent`.

## Retired Models

The following deployments are no longer active, but their argument references
are retained for future redeployment:

- [DeepSeek-V4-Flash-0731](args-deepseek-v4-flash-0731.md)
- [Qwen3.6-27B](args-qwen36-27b.md)
- [Qwen3.6-35B-A3B](args-qwen36-35b-a3b.md)

## GPU / Vulkan

The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
the Mesa/RADV Vulkan driver for the Radeon 8060S. The container mounts
`/dev/dri` and runs privileged, which is the current way to provide Vulkan
access on k3s without a device plugin.

Verify GPU use with:

```bash
kubectl logs -n llamacpp deploy/llamacpp-qwen38-27b | grep -iE 'vulkan|gpu|offload|device'
```

If only a CPU device appears, check that `/dev/dri/renderD128` exists on the
NUCBox and that the `amdgpu` module is loaded.

## Adding Or Replacing A Model

1. Copy `deployment-qwen38-27b.yaml` to `deployment-<new>.yaml` and change the
   model repositories, alias, labels, and Service name.
2. Add the matching alias and Service URL to `litellm/litellm.yaml`.
3. No `gen-apps.sh` change is needed because the `llamacpp` ArgoCD Application
   syncs the directory recursively.
4. Check the model and KV-cache size against the NUCBox's available VRAM.