forked from roger/k3s-cluster
110 lines
5.5 KiB
Markdown
110 lines
5.5 KiB
Markdown
# llama.cpp (llama-server)
|
||
|
||
In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on
|
||
the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified
|
||
memory: 32 GiB RAM / 96 GiB VRAM).
|
||
|
||
This replaces the bare-metal Ollama setup for models that benefit from
|
||
always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these
|
||
in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
|
||
|
||
## Layout
|
||
|
||
One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
|
||
to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
|
||
|
||
| Alias | Model | GGUF | Service |
|
||
|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
|
||
| `qwen3.6-27b` | Qwen3.6-27B (dense) | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB) | `llamacpp-qwen36-27b.llamacpp:80` |
|
||
| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-35b-a3b.llamacpp:80` |
|
||
|
||
The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
|
||
inference is significantly faster than the dense 27B despite more total weights —
|
||
hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for
|
||
deeper reasoning.
|
||
|
||
Model files are downloaded idempotently by an initContainer into a shared
|
||
hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
|
||
without re-downloading.
|
||
|
||
## GPU / Vulkan
|
||
|
||
The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
|
||
the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full
|
||
layer offload (`-ngl 999`) puts model weights entirely in the 96 GiB VRAM pool.
|
||
|
||
The container mounts `/dev/dri` and runs `privileged: true` — the simplest
|
||
reliable way to give Vulkan access to the DRM render node on k3s without a
|
||
device plugin. Tighten later with `supplementalGroups` (the host's `render`
|
||
group GID) if desired.
|
||
|
||
### Verify the GPU is actually used
|
||
|
||
```bash
|
||
kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b | grep -iE 'vulkan|gpu|offload|device'
|
||
kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b | grep -iE 'vulkan|gpu|offload|device'
|
||
```
|
||
|
||
If only a CPU device shows up, the container can't see the GPU — check that
|
||
`/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
|
||
|
||
## VRAM budget (both models co-resident)
|
||
|
||
Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
|
||
|
||
| Model | Weights | KV cache | Subtotal |
|
||
|-------------------|----------|-------------------------------------------|----------|
|
||
| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB |
|
||
| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB |
|
||
| **Total** | | | **~61 GiB** |
|
||
|
||
~35 GiB headroom — comfortable. Both models' KV caches are quantized
|
||
to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
|
||
large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
|
||
a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
|
||
|
||
## Tuning
|
||
|
||
The key knobs (in each `deployment-*.yaml`):
|
||
|
||
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
|
||
- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot
|
||
gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each).
|
||
The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense
|
||
27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B
|
||
uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
|
||
- `-np` — parallel slots (concurrent requests). Each extra slot multiplies
|
||
KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request,
|
||
dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV).
|
||
Bump higher only if you need more concurrent throughput.
|
||
- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
|
||
q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
|
||
27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.
|
||
Drop to q4_0 for even less VRAM if retrieval quality allows.
|
||
- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
|
||
full GPU offload; tune if CPU-bound.
|
||
|
||
## Memory accounting
|
||
|
||
k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is
|
||
reserved by firmware and managed by `amdgpu`). The model weights and KV cache
|
||
live in VRAM and are **not** counted against the container's cgroup memory
|
||
limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages
|
||
during load. If the pod is OOM-killed during model load, raise the memory limit.
|
||
|
||
## Adding a model
|
||
|
||
1. Copy `deployment-qwen36-27b.yaml` → `deployment-<new>.yaml`; change the
|
||
`model:` label, GGUF URL/file, `--alias`, and Service name.
|
||
2. Point LiteLLM at it in `litellm/litellm.yaml`:
|
||
```yaml
|
||
- model_name: <alias>
|
||
litellm_params:
|
||
model: openai/<alias>
|
||
api_base: http://<service>.llamacpp/v1
|
||
api_key: "sk-no-auth"
|
||
```
|
||
3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
|
||
directory recursively.)
|
||
4. Check the VRAM budget table above — two large models may not coexist.
|