deploy deepseek v4

llamacpp/README.md

@@ -1,37 +1,50 @@
 # llama.cpp (llama-server)
 
-In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on
-the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified
-memory: 32 GiB RAM / 96 GiB VRAM).
+In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model
+on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB
+unified memory: ~90 GiB VRAM / 30 GiB CPU RAM).
 
-This replaces the bare-metal Ollama setup for models that benefit from
-always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these
-in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
+LiteLLM (`litellm/`) points at these in-cluster Services instead of the
+external `10.88.20.12:11434` Ollama endpoint.
 
 ## Layout
 
 One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
 to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
 
-| Alias             | Model                      | GGUF                                                  | Service                                   |
-|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
-| `qwen3.6-27b`     | Qwen3.6-27B (dense)        | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB)      | `llamacpp-qwen36-27b.llamacpp:80`         |
-| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB)  | `llamacpp-qwen36-35b-a3b.llamacpp:80`     |
+| Alias                    | Model                          | GGUF                                                  | Service                                              | Args ref |
+|--------------------------|--------------------------------|-------------------------------------------------------|------------------------------------------------------|----------|
+| `deepseek-v4-flash-0731` | DeepSeek-V4-Flash-0731 (MoE)   | unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB) | `llamacpp-deepseek-v4-flash-0731.llamacpp:80`        | [args-deepseek-v4-flash-0731.md](args-deepseek-v4-flash-0731.md) |
 
-The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
-inference is significantly faster than the dense 27B despite more total weights —
-hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for
-deeper reasoning.
+DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per
+token) with MLA attention, so only a small fraction of the weights is computed
+per token. The full ~87 GiB of IQ1_M weights is loaded into the unified memory
+pool.
+
+> **Previously** the NUCBox ran two co-resident Qwen3.6 models (a 27B dense and
+> a 35B-A3B MoE "flash"). DeepSeek-V4-Flash-0731 (~87 GiB) nearly fills the
+> whole 90 GiB VRAM pool on its own, so both Qwen models were removed to make
+> room. Their GGUF files are deleted from the shared PVC by the new pod's
+> `fetch-model` initContainer on first boot. Their deployment arguments are
+> still documented for redeployment:
+> - [args-qwen36-27b.md](args-qwen36-27b.md) — dense 27B (deeper reasoning)
+> - [args-qwen36-35b-a3b.md](args-qwen36-35b-a3b.md) — MoE 35B-A3B "flash" (fast)
 
 Model files are downloaded idempotently by an initContainer into a shared
 hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
-without re-downloading.
+without re-downloading. The UD-IQ1_M GGUF is split across 3 shards
+(`-00001-of-00003` … `-00003-of-00003`); llama.cpp auto-loads all shards when
+pointed at the first one.
 
 ## GPU / Vulkan
 
 The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
-the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full
-layer offload (`-ngl 999`) puts model weights entirely in the 96 GiB VRAM pool.
+the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). The
+Vulkan backend supports the `IQ1_M` matmul (including the MoE `matmul_id`
+variant), so the whole model runs on the GPU. `deepseek4` is a brand-new arch
+(2026-07), so the floating `server-vulkan` tag is used to pull a recent enough
+build — pin to a specific `server-vulkan-bXXXX` tag once a known-good one is
+verified.
 
 The container mounts `/dev/dri` and runs `privileged: true` — the simplest
 reliable way to give Vulkan access to the DRM render node on k3s without a
@@ -41,61 +54,75 @@ group GID) if desired.
 ### Verify the GPU is actually used
 
 ```bash
-kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b     | grep -iE 'vulkan|gpu|offload|device'
-kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b  | grep -iE 'vulkan|gpu|offload|device'
+kubectl logs -n llamacpp deploy/llamacpp-deepseek-v4-flash-0731 | grep -iE 'vulkan|gpu|offload|device'
 ```
 
 If only a CPU device shows up, the container can't see the GPU — check that
 `/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
 
-## VRAM budget (both models co-resident)
+## VRAM budget (single model)
 
-Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
+The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
+it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
+VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
+`-ngl 40` offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on
+CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and
+co-resident pods. Approximate VRAM usage:
 
-| Model             | Weights  | KV cache                                  | Subtotal |
-|-------------------|----------|-------------------------------------------|----------|
-| qwen3.6-27b       | ~16 GiB  | ~16 GiB  (q8_0, 131k total, 1 slot)      | ~32 GiB  |
-| qwen3.6-35b-a3b   | ~20 GiB  | ~9 GiB   (q8_0, 262k total, 131k/slot)   | ~29 GiB  |
-| **Total**         |          |                                           | **~61 GiB** |
+| Component                       | VRAM        |
+|---------------------------------|-------------|
+| Weights (40 GPU layers)         | ~81 GiB     |
+| KV cache (q8_0, 64k, 1 slot)    | ~1.6 GiB    |
+| Vulkan compute buffers          | ~2 GiB      |
+| **Total in VRAM**               | **~85 GiB** |
+| **Headroom (of 90 GiB)**        | **~5–8 GiB**|
 
-~35 GiB headroom — comfortable. Both models' KV caches are quantized
-to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
-large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
-a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
+3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory
+limit, not VRAM).
+
+KV cache is tiny thanks to DeepSeek-V4's **MLA** attention
+(`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
+elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is
+cheap — `-c` is capped at 65536 (the required minimum) to maximise VRAM
+headroom, not because KV is the constraint.
 
 ## Tuning
 
-The key knobs (in each `deployment-*.yaml`):
+The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`):
 
-- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
-- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot
-  gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each).
-  The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense
-  27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B
-  uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
-- `-np` — parallel slots (concurrent requests). Each extra slot multiplies
-  KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request,
-  dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV).
-  Bump higher only if you need more concurrent throughput.
-- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
-  q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
-  27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.
-  Drop to q4_0 for even less VRAM if retrieval quality allows.
-- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
-  full GPU offload; tune if CPU-bound.
+- `-ngl 40` — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the
+  whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute
+  buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom.
+  Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs
+  out of device memory.
+- `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets
+  the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped
+  at the minimum to maximise VRAM headroom. Raise if headroom allows.
+- `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request).
+  Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal.
+- `--cache-type-k q8_0 --cache-type-v q8_0` — quantize the KV cache to q8_0,
+  halving KV VRAM with ~negligible quality loss. Essential to keep headroom.
+- `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4
+  recommendation). These are server defaults; clients can override per request
+  via the OpenAI-compatible API.
+- `--threads 8` — CPU threads for sampling + the 3 CPU-resident layers.
 
 ## Memory accounting
 
-k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is
-reserved by firmware and managed by `amdgpu`). The model weights and KV cache
-live in VRAM and are **not** counted against the container's cgroup memory
-limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages
-during load. If the pod is OOM-killed during model load, raise the memory limit.
+k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is
+reserved by firmware and managed by `amdgpu`). The GPU-resident model weights
+and KV cache live in VRAM and are **not** counted against the container's cgroup
+memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages
+for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load.
+If the pod is OOM-killed during model load, raise the memory limit (and/or
+lower `-ngl`).
 
-## Adding a model
+## Adding / replacing a model
 
-1. Copy `deployment-qwen36-27b.yaml` → `deployment-<new>.yaml`; change the
-   `model:` label, GGUF URL/file, `--alias`, and Service name.
+1. Copy `deployment-deepseek-v4-flash-0731.yaml` → `deployment-<new>.yaml`;
+   change the `model:` label, GGUF URL/file(s), `--alias`, and Service name.
+   For split GGUFs, point `-m` at the first shard and download all shards in
+   the `fetch-model` initContainer.
 2. Point LiteLLM at it in `litellm/litellm.yaml`:
    ```yaml
    - model_name: <alias>
@@ -106,4 +133,5 @@ during load. If the pod is OOM-killed during model load, raise the memory limit.
    ```
 3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
    directory recursively.)
-4. Check the VRAM budget table above — two large models may not coexist.
+4. Check the VRAM budget table above — at ~87 GiB this model nearly fills the
+   90 GiB pool on its own, so co-locating another large model is not possible.