deploy deepseek v4

llamacpp/args-qwen36-35b-a3b.md

@@ -0,0 +1,96 @@
+# Deployment arguments — `qwen3.6-35b-a3b` (REMOVED)
+
+> **Status:** This model was **removed** from the cluster to make VRAM room for
+> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
+> 90 GiB VRAM pool on its own. This file documents the flags used when the
+> model was live, so it can be redeployed later if the DeepSeek model is taken
+> down or moved to different hardware.
+>
+> To redeploy: restore `deployment-qwen36-35b-a3b.yaml` (the manifest is
+> preserved in git history) and re-add the LiteLLM entry. Re-check the VRAM
+> budget — co-locating with the 87 GiB DeepSeek model is **not** possible on
+> the current 90 GiB pool.
+
+## Model & source
+
+- **Model:** `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` — Qwen3.6-35B-A3B, the "flash"
+  **Mixture-of-Experts** variant (35B total params, **only 3B active** per
+  token), MTP variant.
+- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~20 GiB**,
+  single GGUF file (`Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf`).
+- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
+  driver, supports Radeon 8060S / RDNA 3.5).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
+memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
+{kubernetes.io/arch: amd64, hardware: high-memory}`.
+
+## Why it was the "flash" model
+
+Despite having **more total parameters** than the dense 27B, only **3B are
+active per token** (MoE), so inference is significantly faster. The full ~20 GiB
+of Q4 weights is still loaded into VRAM, but only a small fraction is computed
+per token. Its KV cache is also tiny (~72 KiB/token), so large context is nearly
+free — hence the much larger `-c` and the 2-slot split.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
+| `--alias` | `qwen3.6-35b-a3b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
+| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
+| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
+| `-c` | `262144` | Total KV-cache context (262k), **split across parallel slots**. |
+| `-np` | `2` | 2 parallel slots ⇒ 131k tokens per concurrent request (262k / 2). MoE KV is cheap, so splitting is affordable. |
+| `--cont-batching` | *(flag)* | Continuous batching across the 2 slots. |
+| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB at 262k); frees headroom for the large `-c`. Drop to `q4_0` for even less VRAM if retrieval quality allows. |
+| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
+| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
+
+## VRAM budget (when live)
+
+| Component             | VRAM       |
+|-----------------------|------------|
+| Weights (full offload) | ~20 GiB   |
+| KV cache (q8_0, 262k)  | ~9 GiB    |
+| **Subtotal**          | **~29 GiB** |
+
+Left ~67 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
+co-locating the dense 27B model (combined ~61 GiB).
+
+## Combined VRAM budget (both Qwen models, when live)
+
+| Model             | Weights  | KV cache                                  | Subtotal |
+|-------------------|----------|-------------------------------------------|----------|
+| qwen3.6-27b       | ~16 GiB  | ~16 GiB  (q8_0, 131k total, 1 slot)      | ~32 GiB  |
+| qwen3.6-35b-a3b   | ~20 GiB  | ~9 GiB   (q8_0, 262k total, 131k/slot)   | ~29 GiB  |
+| **Total**         |          |                                           | **~61 GiB** |
+
+~35 GiB headroom on the 96 GiB pool — comfortable. (DeepSeek-V4-Flash-0731 at
+~87 GiB cannot coexist with either of these; that's why both were removed.)
+
+## Notes for redeployment
+
+- This was the **fast/flash** model (MoE, 3B active). The `qwen3.6-27b` was the
+  **deeper-reasoning** model (dense, all params active). If redeploying only
+  one, decide based on latency-vs-quality needs.
+- LiteLLM entry that went with it:
+  ```yaml
+  - model_name: qwen3.6-35b-a3b
+    litellm_params:
+      model: openai/qwen3.6-35b-a3b
+      api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
+      api_key: "sk-no-auth"
+  ```
+- Consumers at removal time: `home-manager` (default + auxiliary),
+  `platform-engineer` (auxiliary compression + title generation). See git
+  history for exact config.
+- The `fetch-model` initContainer downloaded the single GGUF idempotently into
+  the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
+  GGUF on first boot to reclaim space, so a redeploy will re-download it.