deploy deepseek v4

llamacpp/args-qwen36-27b.md

@@ -0,0 +1,85 @@
+# Deployment arguments — `qwen3.6-27b` (REMOVED)
+
+> **Status:** This model was **removed** from the cluster to make VRAM room for
+> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
+> 90 GiB VRAM pool on its own. This file documents the flags used when the
+> model was live, so it can be redeployed later if the DeepSeek model is taken
+> down or moved to different hardware.
+>
+> To redeploy: restore `deployment-qwen36-27b.yaml` (the manifest is preserved
+> in git history) and re-add the LiteLLM entry. Re-check the VRAM budget —
+> co-locating with the 87 GiB DeepSeek model is **not** possible on the current
+> 90 GiB pool.
+
+## Model & source
+
+- **Model:** `unsloth/Qwen3.6-27B-MTP-GGUF` — Qwen3.6-27B, **dense** (all
+  params active per token), MTP variant.
+- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~16 GiB**,
+  single GGUF file (`Qwen3.6-27B-UD-Q4_K_XL.gguf`).
+- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
+  driver, supports Radeon 8060S / RDNA 3.5).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
+memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
+{kubernetes.io/arch: amd64, hardware: high-memory}`.
+
+## Why it fit (and co-existed with the 35B-A3B)
+
+At ~16 GiB, the Q4 model is small enough to **fully offload** to the GPU
+(`-ngl 999`) and still leave ~80 GiB of VRAM — which is why it ran alongside
+the Qwen3.6-35B-A3B model (see `args-qwen36-35b-a3b.md`). Combined the two
+models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was **KV
+cache**, not weights: dense attention KV is ~256 KiB/token (f16) / ~128
+KiB/token (q8_0), so large context is expensive.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/Qwen3.6-27B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
+| `--alias` | `qwen3.6-27b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
+| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
+| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
+| `-c` | `131072` | Total KV-cache context (131k). Single slot gets the full window. |
+| `-np` | `1` | 1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting). |
+| `--cont-batching` | *(flag)* | Continuous batching across slots. |
+| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model. |
+| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
+| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
+
+## VRAM budget (when live)
+
+| Component             | VRAM        |
+|-----------------------|-------------|
+| Weights (full offload) | ~16 GiB    |
+| KV cache (q8_0, 131k)  | ~16 GiB    |
+| **Subtotal**          | **~32 GiB** |
+
+Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
+co-locating the 35B-A3B model.
+
+## Notes for redeployment
+
+- This was the **deeper-reasoning** model (dense, all params active). The
+  `qwen3.6-35b-a3b` was the **fast/flash** variant. If redeploying only one,
+  decide based on latency-vs-quality needs.
+- LiteLLM entry that went with it:
+  ```yaml
+  - model_name: qwen3.6-27b
+    litellm_params:
+      model: openai/qwen3.6-27b
+      api_base: http://llamacpp-qwen36-27b.llamacpp/v1
+      api_key: "sk-no-auth"
+  ```
+- Consumers at removal time: `platform-engineer` (default model),
+  `home-manager` (used the 35B-A3B, not this one). See git history for exact
+  config.
+- The `fetch-model` initContainer downloaded the single GGUF idempotently into
+  the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
+  GGUF on first boot to reclaim space, so a redeploy will re-download it.