forked from roger/k3s-cluster
97 lines
4.9 KiB
Markdown
97 lines
4.9 KiB
Markdown
# Deployment arguments — `qwen3.6-35b-a3b` (REMOVED)
|
|
|
|
> **Status:** This model was **removed** from the cluster to make VRAM room for
|
|
> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
|
|
> 90 GiB VRAM pool on its own. This file documents the flags used when the
|
|
> model was live, so it can be redeployed later if the DeepSeek model is taken
|
|
> down or moved to different hardware.
|
|
>
|
|
> To redeploy: restore `deployment-qwen36-35b-a3b.yaml` (the manifest is
|
|
> preserved in git history) and re-add the LiteLLM entry. Re-check the VRAM
|
|
> budget — co-locating with the 87 GiB DeepSeek model is **not** possible on
|
|
> the current 90 GiB pool.
|
|
|
|
## Model & source
|
|
|
|
- **Model:** `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` — Qwen3.6-35B-A3B, the "flash"
|
|
**Mixture-of-Experts** variant (35B total params, **only 3B active** per
|
|
token), MTP variant.
|
|
- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~20 GiB**,
|
|
single GGUF file (`Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf`).
|
|
- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF`
|
|
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
|
|
driver, supports Radeon 8060S / RDNA 3.5).
|
|
|
|
## Hardware target
|
|
|
|
NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
|
|
memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
|
|
{kubernetes.io/arch: amd64, hardware: high-memory}`.
|
|
|
|
## Why it was the "flash" model
|
|
|
|
Despite having **more total parameters** than the dense 27B, only **3B are
|
|
active per token** (MoE), so inference is significantly faster. The full ~20 GiB
|
|
of Q4 weights is still loaded into VRAM, but only a small fraction is computed
|
|
per token. Its KV cache is also tiny (~72 KiB/token), so large context is nearly
|
|
free — hence the much larger `-c` and the 2-slot split.
|
|
|
|
## Argument-by-argument
|
|
|
|
| Flag | Value | Meaning |
|
|
|------|-------|---------|
|
|
| `-m` | `/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
|
|
| `--alias` | `qwen3.6-35b-a3b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
|
|
| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
|
|
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
|
|
| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
|
|
| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
|
|
| `-c` | `262144` | Total KV-cache context (262k), **split across parallel slots**. |
|
|
| `-np` | `2` | 2 parallel slots ⇒ 131k tokens per concurrent request (262k / 2). MoE KV is cheap, so splitting is affordable. |
|
|
| `--cont-batching` | *(flag)* | Continuous batching across the 2 slots. |
|
|
| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB at 262k); frees headroom for the large `-c`. Drop to `q4_0` for even less VRAM if retrieval quality allows. |
|
|
| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
|
|
| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
|
|
|
|
## VRAM budget (when live)
|
|
|
|
| Component | VRAM |
|
|
|-----------------------|------------|
|
|
| Weights (full offload) | ~20 GiB |
|
|
| KV cache (q8_0, 262k) | ~9 GiB |
|
|
| **Subtotal** | **~29 GiB** |
|
|
|
|
Left ~67 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
|
|
co-locating the dense 27B model (combined ~61 GiB).
|
|
|
|
## Combined VRAM budget (both Qwen models, when live)
|
|
|
|
| Model | Weights | KV cache | Subtotal |
|
|
|-------------------|----------|-------------------------------------------|----------|
|
|
| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB |
|
|
| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB |
|
|
| **Total** | | | **~61 GiB** |
|
|
|
|
~35 GiB headroom on the 96 GiB pool — comfortable. (DeepSeek-V4-Flash-0731 at
|
|
~87 GiB cannot coexist with either of these; that's why both were removed.)
|
|
|
|
## Notes for redeployment
|
|
|
|
- This was the **fast/flash** model (MoE, 3B active). The `qwen3.6-27b` was the
|
|
**deeper-reasoning** model (dense, all params active). If redeploying only
|
|
one, decide based on latency-vs-quality needs.
|
|
- LiteLLM entry that went with it:
|
|
```yaml
|
|
- model_name: qwen3.6-35b-a3b
|
|
litellm_params:
|
|
model: openai/qwen3.6-35b-a3b
|
|
api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
|
|
api_key: "sk-no-auth"
|
|
```
|
|
- Consumers at removal time: `home-manager` (default + auxiliary),
|
|
`platform-engineer` (auxiliary compression + title generation). See git
|
|
history for exact config.
|
|
- The `fetch-model` initContainer downloaded the single GGUF idempotently into
|
|
the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
|
|
GGUF on first boot to reclaim space, so a redeploy will re-download it.
|