60 lines
2.1 KiB
Markdown
60 lines
2.1 KiB
Markdown
# llama.cpp (llama-server)
|
|
|
|
In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model
|
|
on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB
|
|
unified memory).
|
|
|
|
LiteLLM (`litellm/`) points at the in-cluster Service instead of the external
|
|
Ollama endpoint.
|
|
|
|
## Active Model
|
|
|
|
| Alias | Model | Configuration | Service |
|
|
|---|---|---|---|
|
|
| `qwen3.8-27b` | Qwen3.8-27B with MTP | `Q4_K_M` primary, `Q4_0` draft, 196k context, q8_0 K/V cache | `llamacpp-qwen38-27b.llamacpp:80` |
|
|
|
|
An initContainer downloads both model files atomically before llama-server
|
|
starts:
|
|
|
|
- Primary: `ggml-org/Qwen3.8-27B-GGUF:Q4_K_M`
|
|
- Draft: `ggml-org/Qwen3.8-27B-GGUF:Q4_0`
|
|
|
|
The model files are stored on the shared hostPath PVC at
|
|
`/data/llamacpp/models` on the NUCBox. The server is configured with
|
|
`--spec-default --spec-type draft-mtp`, `--reasoning-preserve`, `--fit off`,
|
|
and `--agent`.
|
|
|
|
## Retired Models
|
|
|
|
The following deployments are no longer active, but their argument references
|
|
are retained for future redeployment:
|
|
|
|
- [DeepSeek-V4-Flash-0731](args-deepseek-v4-flash-0731.md)
|
|
- [Qwen3.6-27B](args-qwen36-27b.md)
|
|
- [Qwen3.6-35B-A3B](args-qwen36-35b-a3b.md)
|
|
|
|
## GPU / Vulkan
|
|
|
|
The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
|
|
the Mesa/RADV Vulkan driver for the Radeon 8060S. The container mounts
|
|
`/dev/dri` and runs privileged, which is the current way to provide Vulkan
|
|
access on k3s without a device plugin.
|
|
|
|
Verify GPU use with:
|
|
|
|
```bash
|
|
kubectl logs -n llamacpp deploy/llamacpp-qwen38-27b | grep -iE 'vulkan|gpu|offload|device'
|
|
```
|
|
|
|
If only a CPU device appears, check that `/dev/dri/renderD128` exists on the
|
|
NUCBox and that the `amdgpu` module is loaded.
|
|
|
|
## Adding Or Replacing A Model
|
|
|
|
1. Copy `deployment-qwen38-27b.yaml` to `deployment-<new>.yaml` and change the
|
|
model repositories, alias, labels, and Service name.
|
|
2. Add the matching alias and Service URL to `litellm/litellm.yaml`.
|
|
3. No `gen-apps.sh` change is needed because the `llamacpp` ArgoCD Application
|
|
syncs the directory recursively.
|
|
4. Check the model and KV-cache size against the NUCBox's available VRAM.
|