forked from roger/k3s-cluster
86 lines
4.1 KiB
Markdown
86 lines
4.1 KiB
Markdown
# Deployment arguments — `qwen3.6-27b` (REMOVED)
|
|
|
|
> **Status:** This model was **removed** from the cluster to make VRAM room for
|
|
> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
|
|
> 90 GiB VRAM pool on its own. This file documents the flags used when the
|
|
> model was live, so it can be redeployed later if the DeepSeek model is taken
|
|
> down or moved to different hardware.
|
|
>
|
|
> To redeploy: restore `deployment-qwen36-27b.yaml` (the manifest is preserved
|
|
> in git history) and re-add the LiteLLM entry. Re-check the VRAM budget —
|
|
> co-locating with the 87 GiB DeepSeek model is **not** possible on the current
|
|
> 90 GiB pool.
|
|
|
|
## Model & source
|
|
|
|
- **Model:** `unsloth/Qwen3.6-27B-MTP-GGUF` — Qwen3.6-27B, **dense** (all
|
|
params active per token), MTP variant.
|
|
- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~16 GiB**,
|
|
single GGUF file (`Qwen3.6-27B-UD-Q4_K_XL.gguf`).
|
|
- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF`
|
|
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
|
|
driver, supports Radeon 8060S / RDNA 3.5).
|
|
|
|
## Hardware target
|
|
|
|
NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
|
|
memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
|
|
{kubernetes.io/arch: amd64, hardware: high-memory}`.
|
|
|
|
## Why it fit (and co-existed with the 35B-A3B)
|
|
|
|
At ~16 GiB, the Q4 model is small enough to **fully offload** to the GPU
|
|
(`-ngl 999`) and still leave ~80 GiB of VRAM — which is why it ran alongside
|
|
the Qwen3.6-35B-A3B model (see `args-qwen36-35b-a3b.md`). Combined the two
|
|
models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was **KV
|
|
cache**, not weights: dense attention KV is ~256 KiB/token (f16) / ~128
|
|
KiB/token (q8_0), so large context is expensive.
|
|
|
|
## Argument-by-argument
|
|
|
|
| Flag | Value | Meaning |
|
|
|------|-------|---------|
|
|
| `-m` | `/models/Qwen3.6-27B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
|
|
| `--alias` | `qwen3.6-27b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
|
|
| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
|
|
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
|
|
| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
|
|
| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
|
|
| `-c` | `131072` | Total KV-cache context (131k). Single slot gets the full window. |
|
|
| `-np` | `1` | 1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting). |
|
|
| `--cont-batching` | *(flag)* | Continuous batching across slots. |
|
|
| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model. |
|
|
| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
|
|
| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
|
|
|
|
## VRAM budget (when live)
|
|
|
|
| Component | VRAM |
|
|
|-----------------------|-------------|
|
|
| Weights (full offload) | ~16 GiB |
|
|
| KV cache (q8_0, 131k) | ~16 GiB |
|
|
| **Subtotal** | **~32 GiB** |
|
|
|
|
Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
|
|
co-locating the 35B-A3B model.
|
|
|
|
## Notes for redeployment
|
|
|
|
- This was the **deeper-reasoning** model (dense, all params active). The
|
|
`qwen3.6-35b-a3b` was the **fast/flash** variant. If redeploying only one,
|
|
decide based on latency-vs-quality needs.
|
|
- LiteLLM entry that went with it:
|
|
```yaml
|
|
- model_name: qwen3.6-27b
|
|
litellm_params:
|
|
model: openai/qwen3.6-27b
|
|
api_base: http://llamacpp-qwen36-27b.llamacpp/v1
|
|
api_key: "sk-no-auth"
|
|
```
|
|
- Consumers at removal time: `platform-engineer` (default model),
|
|
`home-manager` (used the 35B-A3B, not this one). See git history for exact
|
|
config.
|
|
- The `fetch-model` initContainer downloaded the single GGUF idempotently into
|
|
the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
|
|
GGUF on first boot to reclaim space, so a redeploy will re-download it.
|