deploy deepseek v4

llamacpp/args-deepseek-v4-flash-0731.md

@@ -0,0 +1,109 @@
+# Deployment arguments — `deepseek-v4-flash-0731`
+
+Reference for the llama-server flags used in
+`deployment-deepseek-v4-flash-0731.yaml`. Keep this in sync if the
+Deployment is edited.
+
+## Model & source
+
+- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` (DeepSeek-V4-Flash, 0731 weights)
+- **Quantization:** `unsloth/DeepSeek-V4-Flash-0731-GGUF` → `UD-IQ1_M`
+  (Unsloth Dynamic IQ1_M), **split across 3 shards** (~87 GiB total):
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf`
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf`
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf`
+- **HuggingFace repo:** `https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (floating tag — `deepseek4`
+  is a brand-new arch; pin to a specific `server-vulkan-bXXXX` once verified).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5),
+~120 GiB unified memory split by firmware into **~90 GiB VRAM** and
+**~30 GiB CPU RAM**. Pinned via `nodeSelector: {kubernetes.io/arch: amd64,
+hardware: high-memory}`.
+
+## Why this model nearly fills the machine
+
+The UD-IQ1_M model is **~87 GiB**, almost the entire 90 GiB VRAM pool. Unlike
+the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one
+**cannot be fully offloaded** to the GPU: putting all 43 layers + the KV cache +
+Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all
+about fitting the model while leaving headroom for the KV cache, compute
+buffers, and co-resident pods.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf` | Model file. Pointed at the **first shard** only; llama.cpp auto-discovers and loads `-00002-…` / `-00003-…` from the same directory. |
+| `--alias` | `deepseek-v4-flash-0731` | Name reported by `GET /v1/models`. Must match the `model_name` in `litellm/litellm.yaml` so LiteLLM routes to this server. |
+| `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. |
+| `-ngl` | `40` | **GPU layer offload.** Offload 40 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM. |
+| `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. |
+| `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. |
+| `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). |
+| `--cache-type-k` | `q8_0` | Quantize the K cache to q8_0 (halves KV VRAM, ~negligible quality loss). |
+| `--cache-type-v` | `q8_0` | Quantize the V cache to q8_0 (same). |
+| `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. |
+| `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. |
+| `--threads` | `8` | CPU threads for sampling + the 3 CPU-resident layers. Mostly irrelevant under heavy GPU offload. |
+
+## VRAM budget (90 GiB pool)
+
+| Component                       | VRAM        |
+|---------------------------------|-------------|
+| Weights (40 GPU layers)         | ~81 GiB     |
+| KV cache (q8_0, 64k, 1 slot)    | ~1.6 GiB    |
+| Vulkan compute buffers          | ~2 GiB      |
+| **Total in VRAM**               | **~85 GiB** |
+| **Headroom (of 90 GiB)**        | **~5–8 GiB**|
+
+The remaining **3 layers (~6 GiB) live in CPU RAM** and are counted against the
+pod's cgroup memory limit (not VRAM). KV cache is tiny thanks to DeepSeek-V4's
+**MLA** attention (`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
+elements/token/layer); at 64k context q8_0 KV is only ~1.6 GiB, so context is
+**not** the constraint — `-c` is capped at the minimum purely to maximise VRAM
+headroom.
+
+## How to tune if it OOMs / has spare headroom
+
+- **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl`
+  (e.g. `38`) to keep more layers on CPU, or raise the container `memory`
+  limit.
+- **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full
+  offload) and/or raise `-c` for larger context.
+- **Need more concurrent requests:** raise `-np` (KV is cheap), but each extra
+  slot multiplies KV VRAM and the context per slot shrinks (`-c / -np`).
+
+## initContainer (`fetch-model`)
+
+- Idempotently downloads the 3 shards into the shared models PVC, skipping if
+  the first shard is already present (pod restart / recreate).
+- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim space for the
+  new 87 GiB model.
+
+## Resources
+
+```
+requests:  cpu 1000m, memory 4Gi
+limits:    cpu 4000m, memory 20Gi
+```
+
+VRAM (weights + KV + compute) is **not** counted against the cgroup memory
+limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the
+3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. k8s
+sees ~30 GiB as the node's allocatable system RAM.
+
+## Probes
+
+- `readinessProbe`: `GET /health` after 30s, every 10s, 6 failures.
+- `livenessProbe`: `GET /health` after **300s** (87 GiB load + Vulkan init
+  takes several minutes), every 30s, 5 failures.
+
+## Security / GPU access
+
+`privileged: true` + mounts `/dev/dri` (DRM render nodes) — simplest reliable
+way to give Vulkan access to the AMD APU on k3s without a device plugin.