89 lines
3.9 KiB
Markdown
89 lines
3.9 KiB
Markdown
# llama.cpp (llama-server)
|
||
|
||
In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on
|
||
the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified
|
||
memory: 32 GiB RAM / 96 GiB VRAM).
|
||
|
||
This replaces the bare-metal Ollama setup for models that benefit from
|
||
always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these
|
||
in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
|
||
|
||
## Layout
|
||
|
||
One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
|
||
to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
|
||
|
||
| Model | GGUF | Service | litellm alias |
|
||
|----------|---------------------------------------------|----------------------------------|---------------|
|
||
| qwen3.6 | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL) | `llamacpp-qwen36.llamacpp:80` | `qwen3.6` |
|
||
|
||
Model files are downloaded idempotently by an initContainer into a shared
|
||
hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
|
||
without re-downloading.
|
||
|
||
## GPU / Vulkan
|
||
|
||
The `server-vulkan` image bundles the Mesa/RADV Vulkan driver, which supports
|
||
the Radeon 8060S (RDNA 3.5). Full layer offload (`-ngl 999`) puts the ~16 GiB
|
||
Q4 model entirely in the 96 GiB VRAM pool.
|
||
|
||
The container mounts `/dev/dri` and runs `privileged: true` — the simplest
|
||
reliable way to give Vulkan access to the DRM render node on k3s without a
|
||
device plugin. Tighten later with `supplementalGroups` (the host's `render`
|
||
group GID) if desired.
|
||
|
||
### Verify the GPU is actually used
|
||
|
||
```bash
|
||
kubectl exec -n llamacpp deploy/llamacpp-qwen36 -- \
|
||
llama-server --list-devices -m /models/Qwen3.6-27B-UD-Q4_K_XL.gguf
|
||
# or check the startup logs for a "vulkan" device line + ngl offload count
|
||
kubectl logs -n llamacpp deploy/llamacpp-qwen36 | grep -iE 'vulkan|gpu|offload|device'
|
||
```
|
||
|
||
If only a CPU device shows up, the container can't see the GPU — check that
|
||
`/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
|
||
|
||
## Tuning
|
||
|
||
The key knobs (in `deployment-qwen36.yaml`):
|
||
|
||
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight (it
|
||
isn't, with 96 GiB).
|
||
- `-c 32768` — total KV-cache context. With `-np 4` this is 8192 tokens per
|
||
concurrent request. For a 27B model the full 32k×4 KV cache is ~32 GiB of
|
||
VRAM; raise or lower `-c` to trade context length for VRAM headroom.
|
||
- `-np 4` — parallel slots (concurrent requests). Matches the requested
|
||
concurrency. Each extra slot multiplies KV-cache VRAM usage.
|
||
- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
|
||
full GPU offload; tune if CPU-bound.
|
||
|
||
## Memory accounting
|
||
|
||
k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is
|
||
reserved by firmware and managed by `amdgpu`). The model weights and KV cache
|
||
live in VRAM and are **not** counted against the container's cgroup memory
|
||
limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages
|
||
during load. If the pod is OOM-killed during model load, raise the memory limit.
|
||
|
||
## Adding a model
|
||
|
||
1. Create `deployment-<model>.yaml` + `service-<model>.yaml` (copy the qwen3.6
|
||
pair; change `model:` label, the GGUF URL/file, `--alias`, and Service name).
|
||
2. Point LiteLLM at it in `litellm/litellm.yaml`:
|
||
```yaml
|
||
- model_name: <alias> # keep the alias so consumers don't change
|
||
litellm_params:
|
||
model: openai/<alias>
|
||
api_base: http://<service>.llamacpp/v1
|
||
api_key: "sk-no-auth"
|
||
```
|
||
3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
|
||
directory recursively.)
|
||
|
||
## TODO
|
||
|
||
- `glm-4.7-flash`: still served by the external Ollama at `10.88.20.12:11434`
|
||
in `litellm/litellm.yaml`. Migrate once a GGUF source is confirmed (add a
|
||
`deployment-glm47-flash.yaml` + Service and flip the litellm entry).
|