k3s-cluster/llamacpp/args-deepseek-v4-flash-0731.md

Deployment arguments — deepseek-v4-flash-0731

Reference for the llama-server flags used in deployment-deepseek-v4-flash-0731.yaml. Keep this in sync if the Deployment is edited.

Model & source

• Model: deepseek-ai/DeepSeek-V4-Flash-0731 (DeepSeek-V4-Flash, 0731 weights)
• Quantization: unsloth/DeepSeek-V4-Flash-0731-GGUF → UD-IQ1_M (Unsloth Dynamic IQ1_M), split across 3 shards (~87 GiB total):
  • DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf
  • DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf
  • DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf
• HuggingFace repo: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
• Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (floating tag — deepseek4 is a brand-new arch; pin to a specific server-vulkan-bXXXX once verified).

Hardware target

NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5), ~120 GiB unified memory split by firmware into ~90 GiB VRAM and ~30 GiB CPU RAM. Pinned via nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}.

Why this model nearly fills the machine

The UD-IQ1_M model is ~87 GiB, almost the entire 90 GiB VRAM pool. Unlike the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one cannot be fully offloaded to the GPU: putting all 43 layers + the KV cache + Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all about fitting the model while leaving headroom for the KV cache, compute buffers, and co-resident pods.

Argument-by-argument

Flag	Value	Meaning
-m	/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf	Model file. Pointed at the first shard only; llama.cpp auto-discovers and loads -00002-… / -00003-… from the same directory.
--alias	deepseek-v4-flash-0731	Name reported by GET /v1/models. Must match the model_name in litellm/litellm.yaml so LiteLLM routes to this server.
--host	0.0.0.0	Bind on all interfaces so the k8s Service can reach the pod.
--port	8080	Listen port (matches containerPort + Service targetPort).
--jinja	(flag)	Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in reasoning_content.
-ngl	40	GPU layer offload. Offload 40 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM.
-c	65536	Total KV-cache context window (64k, the required minimum). Single slot gets the full window.
-np	1	Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request.
--cont-batching	(flag)	Continuous batching across slots (no-op with 1 slot, but harmless and correct if -np is raised).
--cache-type-k	q8_0	Quantize the K cache to q8_0 (halves KV VRAM, ~negligible quality loss).
--cache-type-v	q8_0	Quantize the V cache to q8_0 (same).
--temp	1.0	Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API.
--top-p	0.95	Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request.
--threads	8	CPU threads for sampling + the 3 CPU-resident layers. Mostly irrelevant under heavy GPU offload.

VRAM budget (90 GiB pool)

Component	VRAM
Weights (40 GPU layers)	~81 GiB
KV cache (q8_0, 64k, 1 slot)	~1.6 GiB
Vulkan compute buffers	~2 GiB
Total in VRAM	~85 GiB
Headroom (of 90 GiB)	~5–8 GiB

The remaining 3 layers (~6 GiB) live in CPU RAM and are counted against the pod's cgroup memory limit (not VRAM). KV cache is tiny thanks to DeepSeek-V4's MLA attention (num_kv_heads=1, head_dim=512 + 64 decoupled RoPE ⇒ ~576 elements/token/layer); at 64k context q8_0 KV is only ~1.6 GiB, so context is not the constraint — -c is capped at the minimum purely to maximise VRAM headroom.

How to tune if it OOMs / has spare headroom

• Pod OOM-killed or Vulkan out-of-device-memory during load: lower -ngl (e.g. 38) to keep more layers on CPU, or raise the container memory limit.
• VRAM headroom looks generous in logs: raise -ngl toward 43 (full offload) and/or raise -c for larger context.
• Need more concurrent requests: raise -np (KV is cheap), but each extra slot multiplies KV VRAM and the context per slot shrinks (-c / -np).

initContainer (fetch-model)

• Idempotently downloads the 3 shards into the shared models PVC, skipping if the first shard is already present (pod restart / recreate).
• Deletes the retired Qwen3.6 GGUFs from the PVC to reclaim space for the new 87 GiB model.

Resources

requests:  cpu 1000m, memory 4Gi
limits:    cpu 4000m, memory 20Gi

VRAM (weights + KV + compute) is not counted against the cgroup memory limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. k8s sees ~30 GiB as the node's allocatable system RAM.

Probes

• readinessProbe: GET /health after 30s, every 10s, 6 failures.
• livenessProbe: GET /health after 300s (87 GiB load + Vulkan init takes several minutes), every 30s, 5 failures.

Security / GPU access

privileged: true + mounts /dev/dri (DRM render nodes) — simplest reliable way to give Vulkan access to the AMD APU on k3s without a device plugin.