forked from roger/k3s-cluster
138 lines
6.9 KiB
Markdown
138 lines
6.9 KiB
Markdown
# llama.cpp (llama-server)
|
||
|
||
In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model
|
||
on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB
|
||
unified memory: ~90 GiB VRAM / 30 GiB CPU RAM).
|
||
|
||
LiteLLM (`litellm/`) points at these in-cluster Services instead of the
|
||
external `10.88.20.12:11434` Ollama endpoint.
|
||
|
||
## Layout
|
||
|
||
One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
|
||
to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
|
||
|
||
| Alias | Model | GGUF | Service | Args ref |
|
||
|--------------------------|--------------------------------|-------------------------------------------------------|------------------------------------------------------|----------|
|
||
| `deepseek-v4-flash-0731` | DeepSeek-V4-Flash-0731 (MoE) | unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB) | `llamacpp-deepseek-v4-flash-0731.llamacpp:80` | [args-deepseek-v4-flash-0731.md](args-deepseek-v4-flash-0731.md) |
|
||
|
||
DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per
|
||
token) with MLA attention, so only a small fraction of the weights is computed
|
||
per token. The full ~87 GiB of IQ1_M weights is loaded into the unified memory
|
||
pool.
|
||
|
||
> **Previously** the NUCBox ran two co-resident Qwen3.6 models (a 27B dense and
|
||
> a 35B-A3B MoE "flash"). DeepSeek-V4-Flash-0731 (~87 GiB) nearly fills the
|
||
> whole 90 GiB VRAM pool on its own, so both Qwen models were removed to make
|
||
> room. Their GGUF files are deleted from the shared PVC by the new pod's
|
||
> `fetch-model` initContainer on first boot. Their deployment arguments are
|
||
> still documented for redeployment:
|
||
> - [args-qwen36-27b.md](args-qwen36-27b.md) — dense 27B (deeper reasoning)
|
||
> - [args-qwen36-35b-a3b.md](args-qwen36-35b-a3b.md) — MoE 35B-A3B "flash" (fast)
|
||
|
||
Model files are downloaded idempotently by an initContainer into a shared
|
||
hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
|
||
without re-downloading. The UD-IQ1_M GGUF is split across 3 shards
|
||
(`-00001-of-00003` … `-00003-of-00003`); llama.cpp auto-loads all shards when
|
||
pointed at the first one.
|
||
|
||
## GPU / Vulkan
|
||
|
||
The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
|
||
the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). The
|
||
Vulkan backend supports the `IQ1_M` matmul (including the MoE `matmul_id`
|
||
variant), so the whole model runs on the GPU. `deepseek4` is a brand-new arch
|
||
(2026-07), so the floating `server-vulkan` tag is used to pull a recent enough
|
||
build — pin to a specific `server-vulkan-bXXXX` tag once a known-good one is
|
||
verified.
|
||
|
||
The container mounts `/dev/dri` and runs `privileged: true` — the simplest
|
||
reliable way to give Vulkan access to the DRM render node on k3s without a
|
||
device plugin. Tighten later with `supplementalGroups` (the host's `render`
|
||
group GID) if desired.
|
||
|
||
### Verify the GPU is actually used
|
||
|
||
```bash
|
||
kubectl logs -n llamacpp deploy/llamacpp-deepseek-v4-flash-0731 | grep -iE 'vulkan|gpu|offload|device'
|
||
```
|
||
|
||
If only a CPU device shows up, the container can't see the GPU — check that
|
||
`/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
|
||
|
||
## VRAM budget (single model)
|
||
|
||
The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
|
||
it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
|
||
VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
|
||
`-ngl 40` offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on
|
||
CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and
|
||
co-resident pods. Approximate VRAM usage:
|
||
|
||
| Component | VRAM |
|
||
|---------------------------------|-------------|
|
||
| Weights (40 GPU layers) | ~81 GiB |
|
||
| KV cache (q8_0, 64k, 1 slot) | ~1.6 GiB |
|
||
| Vulkan compute buffers | ~2 GiB |
|
||
| **Total in VRAM** | **~85 GiB** |
|
||
| **Headroom (of 90 GiB)** | **~5–8 GiB**|
|
||
|
||
3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory
|
||
limit, not VRAM).
|
||
|
||
KV cache is tiny thanks to DeepSeek-V4's **MLA** attention
|
||
(`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
|
||
elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is
|
||
cheap — `-c` is capped at 65536 (the required minimum) to maximise VRAM
|
||
headroom, not because KV is the constraint.
|
||
|
||
## Tuning
|
||
|
||
The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`):
|
||
|
||
- `-ngl 40` — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the
|
||
whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute
|
||
buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom.
|
||
Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs
|
||
out of device memory.
|
||
- `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets
|
||
the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped
|
||
at the minimum to maximise VRAM headroom. Raise if headroom allows.
|
||
- `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request).
|
||
Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal.
|
||
- `--cache-type-k q8_0 --cache-type-v q8_0` — quantize the KV cache to q8_0,
|
||
halving KV VRAM with ~negligible quality loss. Essential to keep headroom.
|
||
- `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4
|
||
recommendation). These are server defaults; clients can override per request
|
||
via the OpenAI-compatible API.
|
||
- `--threads 8` — CPU threads for sampling + the 3 CPU-resident layers.
|
||
|
||
## Memory accounting
|
||
|
||
k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is
|
||
reserved by firmware and managed by `amdgpu`). The GPU-resident model weights
|
||
and KV cache live in VRAM and are **not** counted against the container's cgroup
|
||
memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages
|
||
for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load.
|
||
If the pod is OOM-killed during model load, raise the memory limit (and/or
|
||
lower `-ngl`).
|
||
|
||
## Adding / replacing a model
|
||
|
||
1. Copy `deployment-deepseek-v4-flash-0731.yaml` → `deployment-<new>.yaml`;
|
||
change the `model:` label, GGUF URL/file(s), `--alias`, and Service name.
|
||
For split GGUFs, point `-m` at the first shard and download all shards in
|
||
the `fetch-model` initContainer.
|
||
2. Point LiteLLM at it in `litellm/litellm.yaml`:
|
||
```yaml
|
||
- model_name: <alias>
|
||
litellm_params:
|
||
model: openai/<alias>
|
||
api_base: http://<service>.llamacpp/v1
|
||
api_key: "sk-no-auth"
|
||
```
|
||
3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
|
||
directory recursively.)
|
||
4. Check the VRAM budget table above — at ~87 GiB this model nearly fills the
|
||
90 GiB pool on its own, so co-locating another large model is not possible.
|