k3s-cluster/llamacpp/args-qwen36-35b-a3b.md

Deployment arguments — qwen3.6-35b-a3b (REMOVED)

Status: This model was removed from the cluster to make VRAM room for deepseek-v4-flash-0731 (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's 90 GiB VRAM pool on its own. This file documents the flags used when the model was live, so it can be redeployed later if the DeepSeek model is taken down or moved to different hardware.

To redeploy: restore deployment-qwen36-35b-a3b.yaml (the manifest is preserved in git history) and re-add the LiteLLM entry. Re-check the VRAM budget — co-locating with the 87 GiB DeepSeek model is not possible on the current 90 GiB pool.

Model & source

• Model: unsloth/Qwen3.6-35B-A3B-MTP-GGUF — Qwen3.6-35B-A3B, the "flash" Mixture-of-Experts variant (35B total params, only 3B active per token), MTP variant.
• Quantization: UD-Q4_K_XL (Unsloth Dynamic Q4_K_XL), ~20 GiB, single GGUF file (Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf).
• HuggingFace repo: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF
• Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (Mesa/RADV Vulkan driver, supports Radeon 8060S / RDNA 3.5).

Hardware target

NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}.

Why it was the "flash" model

Despite having more total parameters than the dense 27B, only 3B are active per token (MoE), so inference is significantly faster. The full ~20 GiB of Q4 weights is still loaded into VRAM, but only a small fraction is computed per token. Its KV cache is also tiny (~72 KiB/token), so large context is nearly free — hence the much larger -c and the 2-slot split.

Argument-by-argument

Flag	Value	Meaning
-m	/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf	Model file (single GGUF).
--alias	qwen3.6-35b-a3b	Name reported by GET /v1/models; matched the LiteLLM model_name.
--host	0.0.0.0	Bind on all interfaces (k8s Service reach).
--port	8080	Listen port (matches containerPort + Service targetPort).
--jinja	(flag)	Use the GGUF's chat template (Qwen3 thinking format).
-ngl	999	Full GPU offload — all layers into VRAM (fits easily in 96 GiB).
-c	262144	Total KV-cache context (262k), split across parallel slots.
-np	2	2 parallel slots ⇒ 131k tokens per concurrent request (262k / 2). MoE KV is cheap, so splitting is affordable.
--cont-batching	(flag)	Continuous batching across the 2 slots.
--cache-type-k	q8_0	Quantize K cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB at 262k); frees headroom for the large -c. Drop to q4_0 for even less VRAM if retrieval quality allows.
--cache-type-v	q8_0	Quantize V cache to q8_0 (same rationale).
--threads	8	CPU threads for sampling/overhead (GPU does the heavy lifting under full offload).

VRAM budget (when live)

Component	VRAM
Weights (full offload)	~20 GiB
KV cache (q8_0, 262k)	~9 GiB
Subtotal	~29 GiB

Left ~67 GiB of headroom on the 96 GiB pool — comfortable, and the basis for co-locating the dense 27B model (combined ~61 GiB).

Combined VRAM budget (both Qwen models, when live)

Model	Weights	KV cache	Subtotal
qwen3.6-27b	~16 GiB	~16 GiB (q8_0, 131k total, 1 slot)	~32 GiB
qwen3.6-35b-a3b	~20 GiB	~9 GiB (q8_0, 262k total, 131k/slot)	~29 GiB
Total			~61 GiB

~35 GiB headroom on the 96 GiB pool — comfortable. (DeepSeek-V4-Flash-0731 at ~87 GiB cannot coexist with either of these; that's why both were removed.)

Notes for redeployment

• This was the fast/flash model (MoE, 3B active). The qwen3.6-27b was the deeper-reasoning model (dense, all params active). If redeploying only one, decide based on latency-vs-quality needs.
• LiteLLM entry that went with it:

  - model_name: qwen3.6-35b-a3b
    litellm_params:
      model: openai/qwen3.6-35b-a3b
      api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
      api_key: "sk-no-auth"
  

• Consumers at removal time: home-manager (default + auxiliary), platform-engineer (auxiliary compression + title generation). See git history for exact config.
• The fetch-model initContainer downloaded the single GGUF idempotently into the shared models PVC; the new DeepSeek pod's initContainer deletes this GGUF on first boot to reclaim space, so a redeploy will re-download it.