k3s-cluster/llamacpp/args-deepseek-v4-flash-0731.md

# Deployment arguments — `deepseek-v4-flash-0731` (RETIRED)

Historical reference for the llama-server flags used by the retired
`deployment-deepseek-v4-flash-0731.yaml`. The Deployment can be recovered from
git history if this model is needed again.

## Model & source

- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` (DeepSeek-V4-Flash, 0731 weights)
- **Quantization:** `unsloth/DeepSeek-V4-Flash-0731-GGUF` → `UD-IQ1_M`
  (Unsloth Dynamic IQ1_M), **split across 3 shards** (~87 GiB total):
  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf`
  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf`
  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf`
- **HuggingFace repo:** `https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF`
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (floating tag — `deepseek4`
  is a brand-new arch; pin to a specific `server-vulkan-bXXXX` once verified).

## Hardware target

NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5),
~120 GiB unified memory split by firmware into **~90 GiB VRAM** and
**~30 GiB CPU RAM**. Pinned via `nodeSelector: {kubernetes.io/arch: amd64,
hardware: high-memory}`.

## Why this model nearly fills the machine

The UD-IQ1_M model is **~87 GiB**, almost the entire 90 GiB VRAM pool. Unlike
the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one
**cannot be fully offloaded** to the GPU: putting all 43 layers + the KV cache +
Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all
about fitting the model while leaving headroom for the KV cache, compute
buffers, and co-resident pods.

## Argument-by-argument

| Flag | Value | Meaning |
|------|-------|---------|
| `-m` | `/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf` | Model file. Pointed at the **first shard** only; llama.cpp auto-discovers and loads `-00002-…` / `-00003-…` from the same directory. |
| `--alias` | `deepseek-v4-flash-0731` | Name reported by `GET /v1/models`. Must match the `model_name` in `litellm/litellm.yaml` so LiteLLM routes to this server. |
| `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. |
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
| `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. |
| `-ngl` | `38` | **GPU layer offload.** Offload 38 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM, and the f16 KV cache (no Flash Attention on Vulkan) is larger than q8_0 would be, so 38 (not 40) layers are offloaded to leave ~5 GiB VRAM headroom. 5 layers (~10 GiB) run on CPU RAM. |
| `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. |
| `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. |
| `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). |
| `--cache-type-k` | `f16` | **f16 K cache (NOT quantized).** The Vulkan backend has no Flash Attention for the `deepseek4` arch, and quantized V cache requires Flash Attention (llama.cpp hard-errors otherwise). Additionally, `deepseek4`/MLA models require K and V cache types to be *identical*, so K cannot be quantized either. |
| `--cache-type-v` | `f16` | **f16 V cache.** Same reason — quantized V cache needs Flash Attention, which Vulkan lacks for deepseek4. |
| `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. |
| `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. |
| `--threads` | `8` | CPU threads for sampling + the 5 CPU-resident layers. |

## VRAM budget (90 GiB pool)

The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
`-ngl 38` offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on
CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and
fragmentation.

**KV cache is f16, not q8_0** — the Vulkan backend has no Flash Attention for
`deepseek4`, and quantized V cache requires Flash Attention (llama.cpp
hard-errors: *"quantized V cache was requested, but this requires Flash
Attention"*). `deepseek4`/MLA models also require K and V cache types to be
*identical*, so K cannot be quantized either. f16 KV at 64k is ~5.7 GiB (MLA
KV: 576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes).

Approximate VRAM usage:

| Component                       | VRAM        |
|---------------------------------|-------------|
| Weights (38 GPU layers)         | ~77 GiB     |
| KV cache (f16, 64k, 1 slot)     | ~5.7 GiB    |
| Vulkan compute buffers          | ~2 GiB      |
| **Total in VRAM**               | **~85 GiB** |
| **Headroom (of 90 GiB)**        | **~5 GiB**  |

5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory
limit, not VRAM). VRAM is exclusive to this model (no other pod uses it); the
other NUCBox pods only compete for the 30 GiB CPU RAM.

Note: several `deepseek4`-specific fused ops (Lightning Indexer, HC pre/comb/post)
are not yet implemented in the Vulkan backend and fall back to CPU (logged as
warnings, not fatal). Inference still works; it is slower than it will be once
those ops land in a future `server-vulkan` build.

## How to tune if it OOMs / has spare headroom

- **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl`
  (e.g. `36`) to keep more layers on CPU, or raise the container `memory`
  limit. Remember the KV cache is f16 (cannot be quantized — no Flash Attention
  on Vulkan), so the only ways to free VRAM are fewer GPU layers or a smaller
  context window.
- **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full
  offload) and/or raise `-c` for larger context.
- **Need more concurrent requests:** raise `-np` (each extra slot multiplies the
  f16 KV cost, ~5.7 GiB/slot at 64k), and the context per slot shrinks
  (`-c / -np`).

## initContainer (`fetch-model`)

- Idempotently downloads the 3 shards into the shared models PVC, skipping if
  all shards are already present and non-empty (pod restart / recreate).
- **Atomic, resumable downloads:** each shard is fetched to a `.partial` file
  (resumable via `curl -C -`) and only renamed to the final name on success, so
  an interrupted download never leaves a half-written final file that would
  wrongly skip re-download.
- **Free-space check:** requires ~95 GiB free on `/models` before downloading;
  fails loudly with a clear message if the hostPath disk is too small (a PVC
  capacity bump does not add physical space to a hostPath volume — the disk on
  the NUCBox must be expanded).
- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim ~36 GiB.

## Resources

```
requests:  cpu 1000m, memory 6Gi
limits:    cpu 4000m, memory 24Gi
```

VRAM (weights + KV + compute) is **not** counted against the cgroup memory
limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the
5 CPU-resident layers (~10 GiB, resident during inference), and reclaimable
page cache during load. k8s sees ~30 GiB as the node's allocatable system RAM.

## Probes

- `readinessProbe`: `GET /health` after 30s, every 10s, 6 failures.
- `livenessProbe`: `GET /health` after **300s** (87 GiB load + Vulkan init
  takes several minutes), every 30s, 5 failures.

## Security / GPU access

`privileged: true` + mounts `/dev/dri` (DRM render nodes) — simplest reliable
way to give Vulkan access to the AMD APU on k3s without a device plugin.