k3s-cluster/llamacpp/args-qwen36-27b.md

Deployment arguments — qwen3.6-27b (REMOVED)

Status: This model was removed from the cluster to make VRAM room for deepseek-v4-flash-0731 (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's 90 GiB VRAM pool on its own. This file documents the flags used when the model was live, so it can be redeployed later if the DeepSeek model is taken down or moved to different hardware.

To redeploy: restore deployment-qwen36-27b.yaml from git history and re-add the LiteLLM entry. Re-check the VRAM budget — co-locating with the 87 GiB DeepSeek model is not possible on the current 90 GiB pool.

Model & source

• Model: unsloth/Qwen3.6-27B-MTP-GGUF — Qwen3.6-27B, dense (all params active per token), MTP variant.
• Quantization: UD-Q4_K_XL (Unsloth Dynamic Q4_K_XL), ~16 GiB, single GGUF file (Qwen3.6-27B-UD-Q4_K_XL.gguf).
• HuggingFace repo: https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
• Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (Mesa/RADV Vulkan driver, supports Radeon 8060S / RDNA 3.5).

Hardware target

NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}.

Why it fit (and co-existed with the 35B-A3B)

At ~16 GiB, the Q4 model is small enough to fully offload to the GPU (-ngl 999) and still leave ~80 GiB of VRAM — which is why it ran alongside the Qwen3.6-35B-A3B model (see args-qwen36-35b-a3b.md). Combined the two models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was KV cache, not weights: dense attention KV is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), so large context is expensive.

Argument-by-argument

Flag	Value	Meaning
-m	/models/Qwen3.6-27B-UD-Q4_K_XL.gguf	Model file (single GGUF).
--alias	qwen3.6-27b	Name reported by GET /v1/models; matched the LiteLLM model_name.
--host	0.0.0.0	Bind on all interfaces (k8s Service reach).
--port	8080	Listen port (matches containerPort + Service targetPort).
--jinja	(flag)	Use the GGUF's chat template (Qwen3 thinking format).
-ngl	999	Full GPU offload — all layers into VRAM (fits easily in 96 GiB).
-c	131072	Total KV-cache context (131k). Single slot gets the full window.
-np	1	1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting).
--cont-batching	(flag)	Continuous batching across slots.
--cache-type-k	q8_0	Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model.
--cache-type-v	q8_0	Quantize V cache to q8_0 (same rationale).
--threads	8	CPU threads for sampling/overhead (GPU does the heavy lifting under full offload).

VRAM budget (when live)

Component	VRAM
Weights (full offload)	~16 GiB
KV cache (q8_0, 131k)	~16 GiB
Subtotal	~32 GiB

Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for co-locating the 35B-A3B model.

Notes for redeployment

• This was the deeper-reasoning model (dense, all params active). The qwen3.6-35b-a3b was the fast/flash variant. If redeploying only one, decide based on latency-vs-quality needs.
• LiteLLM entry that went with it:

  - model_name: qwen3.6-27b
    litellm_params:
      model: openai/qwen3.6-27b
      api_base: http://llamacpp-qwen36-27b.llamacpp/v1
      api_key: "sk-no-auth"
  

• Consumers at removal time: platform-engineer (default model), home-manager (used the 35B-A3B, not this one). See git history for exact config.
• The fetch-model initContainer downloaded the single GGUF idempotently into the shared models PVC; the new DeepSeek pod's initContainer deletes this GGUF on first boot to reclaim space, so a redeploy will re-download it.