k3s-cluster/llamacpp/args-deepseek-v4-flash-0731.md

Deployment arguments — deepseek-v4-flash-0731

Reference for the llama-server flags used in deployment-deepseek-v4-flash-0731.yaml. Keep this in sync if the Deployment is edited.

Model & source

• Model: deepseek-ai/DeepSeek-V4-Flash-0731 (DeepSeek-V4-Flash, 0731 weights)
• Quantization: unsloth/DeepSeek-V4-Flash-0731-GGUF → UD-IQ1_M (Unsloth Dynamic IQ1_M), split across 3 shards (~87 GiB total):
  • DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf
  • DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf
  • DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf
• HuggingFace repo: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
• Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (floating tag — deepseek4 is a brand-new arch; pin to a specific server-vulkan-bXXXX once verified).

Hardware target

NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5), ~120 GiB unified memory split by firmware into ~90 GiB VRAM and ~30 GiB CPU RAM. Pinned via nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}.

Why this model nearly fills the machine

The UD-IQ1_M model is ~87 GiB, almost the entire 90 GiB VRAM pool. Unlike the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one cannot be fully offloaded to the GPU: putting all 43 layers + the KV cache + Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all about fitting the model while leaving headroom for the KV cache, compute buffers, and co-resident pods.

Argument-by-argument

Flag	Value	Meaning
-m	/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf	Model file. Pointed at the first shard only; llama.cpp auto-discovers and loads -00002-… / -00003-… from the same directory.
--alias	deepseek-v4-flash-0731	Name reported by GET /v1/models. Must match the model_name in litellm/litellm.yaml so LiteLLM routes to this server.
--host	0.0.0.0	Bind on all interfaces so the k8s Service can reach the pod.
--port	8080	Listen port (matches containerPort + Service targetPort).
--jinja	(flag)	Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in reasoning_content.
-ngl	38	GPU layer offload. Offload 38 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM, and the f16 KV cache (no Flash Attention on Vulkan) is larger than q8_0 would be, so 38 (not 40) layers are offloaded to leave ~5 GiB VRAM headroom. 5 layers (~10 GiB) run on CPU RAM.
-c	65536	Total KV-cache context window (64k, the required minimum). Single slot gets the full window.
-np	1	Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request.
--cont-batching	(flag)	Continuous batching across slots (no-op with 1 slot, but harmless and correct if -np is raised).
--cache-type-k	f16	f16 K cache (NOT quantized). The Vulkan backend has no Flash Attention for the deepseek4 arch, and quantized V cache requires Flash Attention (llama.cpp hard-errors otherwise). Additionally, deepseek4/MLA models require K and V cache types to be identical, so K cannot be quantized either.
--cache-type-v	f16	f16 V cache. Same reason — quantized V cache needs Flash Attention, which Vulkan lacks for deepseek4.
--temp	1.0	Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API.
--top-p	0.95	Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request.
--threads	8	CPU threads for sampling + the 5 CPU-resident layers.

VRAM budget (90 GiB pool)

The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so it cannot be fully offloaded: -ngl 999 would try to put all 43 layers into VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead -ngl 38 offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and fragmentation.

KV cache is f16, not q8_0 — the Vulkan backend has no Flash Attention for deepseek4, and quantized V cache requires Flash Attention (llama.cpp hard-errors: "quantized V cache was requested, but this requires Flash Attention"). deepseek4/MLA models also require K and V cache types to be identical, so K cannot be quantized either. f16 KV at 64k is ~5.7 GiB (MLA KV: 576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes).

Approximate VRAM usage:

Component	VRAM
Weights (38 GPU layers)	~77 GiB
KV cache (f16, 64k, 1 slot)	~5.7 GiB
Vulkan compute buffers	~2 GiB
Total in VRAM	~85 GiB
Headroom (of 90 GiB)	~5 GiB

5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory limit, not VRAM). VRAM is exclusive to this model (no other pod uses it); the other NUCBox pods only compete for the 30 GiB CPU RAM.

Note: several deepseek4-specific fused ops (Lightning Indexer, HC pre/comb/post) are not yet implemented in the Vulkan backend and fall back to CPU (logged as warnings, not fatal). Inference still works; it is slower than it will be once those ops land in a future server-vulkan build.

How to tune if it OOMs / has spare headroom

• Pod OOM-killed or Vulkan out-of-device-memory during load: lower -ngl (e.g. 36) to keep more layers on CPU, or raise the container memory limit. Remember the KV cache is f16 (cannot be quantized — no Flash Attention on Vulkan), so the only ways to free VRAM are fewer GPU layers or a smaller context window.
• VRAM headroom looks generous in logs: raise -ngl toward 43 (full offload) and/or raise -c for larger context.
• Need more concurrent requests: raise -np (each extra slot multiplies the f16 KV cost, ~5.7 GiB/slot at 64k), and the context per slot shrinks (-c / -np).

initContainer (fetch-model)

• Idempotently downloads the 3 shards into the shared models PVC, skipping if all shards are already present and non-empty (pod restart / recreate).
• Atomic, resumable downloads: each shard is fetched to a .partial file (resumable via curl -C -) and only renamed to the final name on success, so an interrupted download never leaves a half-written final file that would wrongly skip re-download.
• Free-space check: requires ~95 GiB free on /models before downloading; fails loudly with a clear message if the hostPath disk is too small (a PVC capacity bump does not add physical space to a hostPath volume — the disk on the NUCBox must be expanded).
• Deletes the retired Qwen3.6 GGUFs from the PVC to reclaim ~36 GiB.

Resources

requests:  cpu 1000m, memory 6Gi
limits:    cpu 4000m, memory 24Gi

VRAM (weights + KV + compute) is not counted against the cgroup memory limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the 5 CPU-resident layers (~10 GiB, resident during inference), and reclaimable page cache during load. k8s sees ~30 GiB as the node's allocatable system RAM.

Probes

• readinessProbe: GET /health after 30s, every 10s, 6 failures.
• livenessProbe: GET /health after 300s (87 GiB load + Vulkan init takes several minutes), every 30s, 5 failures.

Security / GPU access

privileged: true + mounts /dev/dri (DRM render nodes) — simplest reliable way to give Vulkan access to the AMD APU on k3s without a device plugin.