# Deployment arguments — `qwen3.6-27b` (REMOVED)

> **Status:** This model was **removed** from the cluster to make VRAM room for
> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
> 90 GiB VRAM pool on its own. This file documents the flags used when the
> model was live, so it can be redeployed later if the DeepSeek model is taken
> down or moved to different hardware.
>
> To redeploy: restore `deployment-qwen36-27b.yaml` from git history and
> re-add the LiteLLM entry. Re-check the VRAM budget —
> co-locating with the 87 GiB DeepSeek model is **not** possible on the current
> 90 GiB pool.

## Model & source

- **Model:** `unsloth/Qwen3.6-27B-MTP-GGUF` — Qwen3.6-27B, **dense** (all
  params active per token), MTP variant.
- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~16 GiB**,
  single GGUF file (`Qwen3.6-27B-UD-Q4_K_XL.gguf`).
- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF`
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
  driver, supports Radeon 8060S / RDNA 3.5).

## Hardware target

NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
{kubernetes.io/arch: amd64, hardware: high-memory}`.

## Why it fit (and co-existed with the 35B-A3B)

At ~16 GiB, the Q4 model is small enough to **fully offload** to the GPU
(`-ngl 999`) and still leave ~80 GiB of VRAM — which is why it ran alongside
the Qwen3.6-35B-A3B model (see `args-qwen36-35b-a3b.md`). Combined the two
models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was **KV
cache**, not weights: dense attention KV is ~256 KiB/token (f16) / ~128
KiB/token (q8_0), so large context is expensive.

## Argument-by-argument

| Flag | Value | Meaning |
|------|-------|---------|
| `-m` | `/models/Qwen3.6-27B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
| `--alias` | `qwen3.6-27b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
| `-c` | `131072` | Total KV-cache context (131k). Single slot gets the full window. |
| `-np` | `1` | 1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting). |
| `--cont-batching` | *(flag)* | Continuous batching across slots. |
| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model. |
| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |

## VRAM budget (when live)

| Component             | VRAM        |
|-----------------------|-------------|
| Weights (full offload) | ~16 GiB    |
| KV cache (q8_0, 131k)  | ~16 GiB    |
| **Subtotal**          | **~32 GiB** |

Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
co-locating the 35B-A3B model.

## Notes for redeployment

- This was the **deeper-reasoning** model (dense, all params active). The
  `qwen3.6-35b-a3b` was the **fast/flash** variant. If redeploying only one,
  decide based on latency-vs-quality needs.
- LiteLLM entry that went with it:
  ```yaml
  - model_name: qwen3.6-27b
    litellm_params:
      model: openai/qwen3.6-27b
      api_base: http://llamacpp-qwen36-27b.llamacpp/v1
      api_key: "sk-no-auth"
  ```
- Consumers at removal time: `platform-engineer` (default model),
  `home-manager` (used the 35B-A3B, not this one). See git history for exact
  config.
- The `fetch-model` initContainer downloaded the single GGUF idempotently into
  the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
  GGUF on first boot to reclaim space, so a redeploy will re-download it.