fix deepseek deployment

llamacpp/README.md

@@ -65,47 +65,61 @@ If only a CPU device shows up, the container can't see the GPU — check that
 The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
 it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
 VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
-`-ngl 40` offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on
+`-ngl 38` offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on
-CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and
+CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and
-co-resident pods. Approximate VRAM usage:
+fragmentation.
+
+**KV cache is f16, not q8_0** — the Vulkan backend has no Flash Attention for
+the `deepseek4` arch, and quantized V cache requires Flash Attention (llama.cpp
+hard-errors: *"quantized V cache was requested, but this requires Flash
+Attention"*). `deepseek4`/MLA models also require K and V cache types to be
+*identical*, so K cannot be quantized either. f16 MLA KV at 64k is ~5.7 GiB
+(576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes) —
+larger than q8_0 would be, which is why `-ngl` is 38 rather than 40.
+
+Approximate VRAM usage:
 
 | Component                       | VRAM        |
 |---------------------------------|-------------|
-| Weights (40 GPU layers)         | ~81 GiB     |
+| Weights (38 GPU layers)         | ~77 GiB     |
-| KV cache (q8_0, 64k, 1 slot)    | ~1.6 GiB    |
+| KV cache (f16, 64k, 1 slot)     | ~5.7 GiB    |
 | Vulkan compute buffers          | ~2 GiB      |
 | **Total in VRAM**               | **~85 GiB** |
-| **Headroom (of 90 GiB)**        | **~5–8 GiB**|
+| **Headroom (of 90 GiB)**        | **~5 GiB**  |
 
-3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory
+5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory
-limit, not VRAM).
+limit, not VRAM). VRAM is exclusive to this model; the other NUCBox pods only
+compete for the 30 GiB CPU RAM.
 
-KV cache is tiny thanks to DeepSeek-V4's **MLA** attention
+Several `deepseek4`-specific fused ops (Lightning Indexer, HC pre/comb/post)
-(`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
+are not yet implemented in the Vulkan backend and fall back to CPU (logged as
-elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is
+warnings, not fatal). Inference still works; it will speed up once those ops
-cheap — `-c` is capped at 65536 (the required minimum) to maximise VRAM
+land in a future `server-vulkan` build.
-headroom, not because KV is the constraint.
 
 ## Tuning
 
 The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`):
 
-- `-ngl 40` — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the
+- `-ngl 38` — offload 38 of 43 layers to GPU. The model (~87 GiB) is nearly the
-  whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute
+  whole 90 GiB VRAM pool, so full offload would overflow once the f16 KV cache +
-  buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom.
+  compute buffers are added. 5 layers (~10 GiB) on CPU leaves ~5 GiB VRAM
-  Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs
+  headroom. Raise toward 43 if VRAM allows; lower (e.g. 36) if the pod OOMs /
-  out of device memory.
+  Vulkan runs out of device memory.
 - `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets
-  the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped
+  the full 64k. f16 MLA KV at 64k is ~5.7 GiB; capped at the minimum to maximise
-  at the minimum to maximise VRAM headroom. Raise if headroom allows.
+  VRAM headroom. Raise if headroom allows.
 - `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request).
-  Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal.
+  Extra slots multiply the f16 KV cost (~5.7 GiB/slot); 1 slot keeps headroom
-- `--cache-type-k q8_0 --cache-type-v q8_0` — quantize the KV cache to q8_0,
+  maximal.
-  halving KV VRAM with ~negligible quality loss. Essential to keep headroom.
+- `--cache-type-k f16 --cache-type-v f16` — **f16 KV cache (NOT quantized).**
+  The Vulkan backend has no Flash Attention for `deepseek4`, and quantized V
+  cache requires Flash Attention. `deepseek4`/MLA models also require K and V
+  cache types to be identical, so K cannot be quantized either. This is the
+  reason `-ngl` is 38 rather than 40.
 - `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4
   recommendation). These are server defaults; clients can override per request
   via the OpenAI-compatible API.
-- `--threads 8` — CPU threads for sampling + the 3 CPU-resident layers.
+- `--threads 8` — CPU threads for sampling + the 5 CPU-resident layers.
 
 ## Memory accounting
 
@@ -113,9 +127,10 @@ k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is
 reserved by firmware and managed by `amdgpu`). The GPU-resident model weights
 and KV cache live in VRAM and are **not** counted against the container's cgroup
 memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages
-for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load.
+for the 5 CPU-resident layers (~10 GiB, resident during inference), and
-If the pod is OOM-killed during model load, raise the memory limit (and/or
+reclaimable page cache during load. If the pod is OOM-killed during model load
-lower `-ngl`).
+or inference, raise the memory limit (and/or lower `-ngl` to push more layers
+to VRAM).
 
 ## Adding / replacing a model
 

llamacpp/args-deepseek-v4-flash-0731.md

@@ -41,61 +41,89 @@ buffers, and co-resident pods.
 | `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. |
 | `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
 | `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. |
-| `-ngl` | `40` | **GPU layer offload.** Offload 40 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM. |
+| `-ngl` | `38` | **GPU layer offload.** Offload 38 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM, and the f16 KV cache (no Flash Attention on Vulkan) is larger than q8_0 would be, so 38 (not 40) layers are offloaded to leave ~5 GiB VRAM headroom. 5 layers (~10 GiB) run on CPU RAM. |
 | `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. |
 | `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. |
 | `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). |
-| `--cache-type-k` | `q8_0` | Quantize the K cache to q8_0 (halves KV VRAM, ~negligible quality loss). |
+| `--cache-type-k` | `f16` | **f16 K cache (NOT quantized).** The Vulkan backend has no Flash Attention for the `deepseek4` arch, and quantized V cache requires Flash Attention (llama.cpp hard-errors otherwise). Additionally, `deepseek4`/MLA models require K and V cache types to be *identical*, so K cannot be quantized either. |
-| `--cache-type-v` | `q8_0` | Quantize the V cache to q8_0 (same). |
+| `--cache-type-v` | `f16` | **f16 V cache.** Same reason — quantized V cache needs Flash Attention, which Vulkan lacks for deepseek4. |
 | `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. |
 | `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. |
-| `--threads` | `8` | CPU threads for sampling + the 3 CPU-resident layers. Mostly irrelevant under heavy GPU offload. |
+| `--threads` | `8` | CPU threads for sampling + the 5 CPU-resident layers. |
 
 ## VRAM budget (90 GiB pool)
 
+The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
+it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
+VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
+`-ngl 38` offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on
+CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and
+fragmentation.
+
+**KV cache is f16, not q8_0** — the Vulkan backend has no Flash Attention for
+`deepseek4`, and quantized V cache requires Flash Attention (llama.cpp
+hard-errors: *"quantized V cache was requested, but this requires Flash
+Attention"*). `deepseek4`/MLA models also require K and V cache types to be
+*identical*, so K cannot be quantized either. f16 KV at 64k is ~5.7 GiB (MLA
+KV: 576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes).
+
+Approximate VRAM usage:
+
 | Component                       | VRAM        |
 |---------------------------------|-------------|
-| Weights (40 GPU layers)         | ~81 GiB     |
+| Weights (38 GPU layers)         | ~77 GiB     |
-| KV cache (q8_0, 64k, 1 slot)    | ~1.6 GiB    |
+| KV cache (f16, 64k, 1 slot)     | ~5.7 GiB    |
 | Vulkan compute buffers          | ~2 GiB      |
 | **Total in VRAM**               | **~85 GiB** |
-| **Headroom (of 90 GiB)**        | **~5–8 GiB**|
+| **Headroom (of 90 GiB)**        | **~5 GiB**  |
 
-The remaining **3 layers (~6 GiB) live in CPU RAM** and are counted against the
+5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory
-pod's cgroup memory limit (not VRAM). KV cache is tiny thanks to DeepSeek-V4's
+limit, not VRAM). VRAM is exclusive to this model (no other pod uses it); the
-**MLA** attention (`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
+other NUCBox pods only compete for the 30 GiB CPU RAM.
-elements/token/layer); at 64k context q8_0 KV is only ~1.6 GiB, so context is
+
-**not** the constraint — `-c` is capped at the minimum purely to maximise VRAM
+Note: several `deepseek4`-specific fused ops (Lightning Indexer, HC pre/comb/post)
-headroom.
+are not yet implemented in the Vulkan backend and fall back to CPU (logged as
+warnings, not fatal). Inference still works; it is slower than it will be once
+those ops land in a future `server-vulkan` build.
 
 ## How to tune if it OOMs / has spare headroom
 
 - **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl`
-  (e.g. `38`) to keep more layers on CPU, or raise the container `memory`
+  (e.g. `36`) to keep more layers on CPU, or raise the container `memory`
-  limit.
+  limit. Remember the KV cache is f16 (cannot be quantized — no Flash Attention
+  on Vulkan), so the only ways to free VRAM are fewer GPU layers or a smaller
+  context window.
 - **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full
   offload) and/or raise `-c` for larger context.
-- **Need more concurrent requests:** raise `-np` (KV is cheap), but each extra
+- **Need more concurrent requests:** raise `-np` (each extra slot multiplies the
-  slot multiplies KV VRAM and the context per slot shrinks (`-c / -np`).
+  f16 KV cost, ~5.7 GiB/slot at 64k), and the context per slot shrinks
+  (`-c / -np`).
 
 ## initContainer (`fetch-model`)
 
 - Idempotently downloads the 3 shards into the shared models PVC, skipping if
-  the first shard is already present (pod restart / recreate).
+  all shards are already present and non-empty (pod restart / recreate).
-- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim space for the
+- **Atomic, resumable downloads:** each shard is fetched to a `.partial` file
-  new 87 GiB model.
+  (resumable via `curl -C -`) and only renamed to the final name on success, so
+  an interrupted download never leaves a half-written final file that would
+  wrongly skip re-download.
+- **Free-space check:** requires ~95 GiB free on `/models` before downloading;
+  fails loudly with a clear message if the hostPath disk is too small (a PVC
+  capacity bump does not add physical space to a hostPath volume — the disk on
+  the NUCBox must be expanded).
+- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim ~36 GiB.
 
 ## Resources
 
 ```
-requests:  cpu 1000m, memory 4Gi
+requests:  cpu 1000m, memory 6Gi
-limits:    cpu 4000m, memory 20Gi
+limits:    cpu 4000m, memory 24Gi
 ```
 
 VRAM (weights + KV + compute) is **not** counted against the cgroup memory
 limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the
-3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. k8s
+5 CPU-resident layers (~10 GiB, resident during inference), and reclaimable
-sees ~30 GiB as the node's allocatable system RAM.
+page cache during load. k8s sees ~30 GiB as the node's allocatable system RAM.
 
 ## Probes
 

llamacpp/deployment-deepseek-v4-flash-0731.yaml

@@ -6,24 +6,31 @@
 # IQ1_M model (~87 GiB) is *almost* the size of the whole VRAM pool, so it
 # CANNOT be fully offloaded to the GPU: offloading all 43 layers + the KV
 # cache + Vulkan compute buffers would overflow 90 GiB. Instead we offload
-# 40 of 43 layers (-ngl 40) and keep the last 3 (~6 GiB) on CPU RAM, leaving
+# 38 of 43 layers (-ngl 38) and keep 5 layers (~10 GiB) on CPU RAM, leaving
-# ~8 GiB of VRAM headroom for the KV cache, compute buffers, and co-resident
+# ~5 GiB of VRAM headroom for the KV cache + Vulkan compute buffers.
-# pods. This is the only model served on the NUCBox — the two Qwen3.6 models
-# were removed to make room (their GGUF files should be deleted from the PVC,
-# which the initContainer below does on first boot).
 #
-# KV cache is tiny thanks to DeepSeek-V4's MLA attention (num_kv_heads=1,
+# VRAM is exclusive to this model (no other pod uses it); the other pods on
-# head_dim=512 + 64 decoupled RoPE ⇒ ~576 elements/token/layer). At 64k
+# the NUCBox only compete for the 30 GiB CPU RAM, so the headroom that
-# context, q8_0 KV is only ~1.6 GiB, so context is cheap — but we cap -c at
+# matters here is VRAM headroom for compute buffers / fragmentation.
-# 65536 (the required minimum) to maximise VRAM headroom, not because KV is
+#
-# the constraint.
+# KV CACHE MUST BE f16 (NOT quantized). The Vulkan backend has no Flash
+# Attention for the deepseek4 arch, and quantized V cache requires Flash
+# Attention (llama.cpp hard-errors otherwise: "quantized V cache was
+# requested, but this requires Flash Attention"). Additionally, deepseek4 /
+# MLA models require K and V cache types to be *identical*, so K cannot be
+# quantized either. f16 KV at 64k is ~5.7 GiB (MLA KV: 576 K + 512 V
+# elements/token/layer × 43 layers × 65536 tokens × 2 bytes). This is why
+# -ngl is 38 rather than 40 — the larger f16 KV cache needs the extra VRAM.
 #
 # Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan
 # driver, which supports the Radeon 8060S (RDNA 3.5). The Vulkan backend
 # supports the IQ1_M matmul (incl. the MoE matmul_id variant), so the whole
-# model runs on the GPU. deepseek4 is a brand-new arch (2026-07) so the
+# model runs on the GPU. deepseek4 is a brand-new arch (2026-07); several
-# floating `server-vulkan` tag is used to pull a recent enough build; pin to a
+# deepseek4-specific fused ops (Lightning Indexer, HC pre/comb/post) are not
-# specific server-vulkan-bXXXX tag once a known-good one is verified.
+# yet implemented in Vulkan and fall back to CPU (logged as warnings, not
+# fatal). The floating `server-vulkan` tag is used to pull a recent enough
+# build; pin to a specific server-vulkan-bXXXX tag once a known-good one is
+# verified.
 #
 # GPU access: the container mounts /dev/dri (the DRM render nodes) and runs
 # privileged — the simplest reliable option on k3s without a Vulkan device
@@ -57,8 +64,12 @@ spec:
       initContainers:
       # Idempotently download the (3-part, split) GGUF into the shared models
       # PVC on first boot. Also removes the retired Qwen3.6 GGUFs so the new
-      # 87 GiB model fits on the PVC alongside any other data. Exits
+      # 87 GiB model fits on the PVC. Downloads are atomic (→ .partial, then
-      # immediately if the first shard is already present (pod restart).
+      # rename) and resumable, so a failed/interrupted download is recovered
+      # on the next pod start without re-fetching from scratch. A free-space
+      # check fails loudly if the hostPath disk is genuinely too small (no
+      # manifest can create physical disk space — that needs the disk expanded
+      # on the NUCBox).
       - name: fetch-model
         image: alpine:3.20
         command: ["/bin/sh", "-c"]
@@ -73,18 +84,42 @@ spec:
               rm -f "/models/$old"
             fi
           done
-          # Download any missing shards of the split UD-IQ1_M GGUF.
+          # Skip entirely if every shard is already fully downloaded.
-          if [ -f "/models/$SHARD1" ]; then
+          if [ -s "/models/$SHARD1" ] && [ -s "/models/$SHARD2" ] && [ -s "/models/$SHARD3" ]; then
-            echo "First shard $SHARD1 already present — skipping download."
+            echo "All 3 shards already present — skipping download."
+            ls -lh /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-*.gguf
             exit 0
           fi
           echo "Installing curl..."
           apk add --no-cache curl
+          # Free-space check: the model is ~87 GiB; require ~95 GiB free as a
+          # safety buffer. df reports KiB.
+          FREE_KB=$(df -P /models | awk 'NR==2 {print $4}')
+          NEEDED_KB=$((95 * 1024 * 1024))
+          if [ "$FREE_KB" -lt "$NEEDED_KB" ]; then
+            avail_gb=$((FREE_KB / 1024 / 1024))
+            echo "ERROR: only ${avail_gb} GiB free on /models, need ~95 GiB to" >&2
+            echo "       download the 87 GiB DeepSeek-V4-Flash-0731 GGUF." >&2
+            echo "       Expand the hostPath disk at /data/llamacpp/models on" >&2
+            echo "       the NUCBox (a PVC capacity bump alone does not add" >&2
+            echo "       physical space to a hostPath volume)." >&2
+            exit 1
+          fi
+          # Download each missing shard to a .partial file (resumable via -C -),
+          # then atomically rename to the final name on success. A crash leaves
+          # only the .partial behind, which the next run resumes — never a
+          # half-written final file that would skip the download.
           for s in "$SHARD1" "$SHARD2" "$SHARD3"; do
+            if [ -s "/models/$s" ]; then
+              echo "Shard $s already complete — skipping."
+              continue
+            fi
             echo "Downloading $s from $HF_REPO ..."
-            curl -fL --retry 5 --retry-delay 5 -o "/models/$s" "$HF_REPO/$s"
+            curl -fL --retry 5 --retry-delay 5 -C - -o "/models/$s.partial" "$HF_REPO/$s"
+            mv "/models/$s.partial" "/models/$s"
+            echo "  done: $(ls -lh "/models/$s")"
           done
-          echo "Download complete:"
+          echo "All shards downloaded:"
           ls -lh /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-*.gguf
         env:
         - name: HF_REPO
@@ -114,50 +149,53 @@ spec:
         - --port
         - "8080"
         - --jinja                  # use the GGUF's DeepSeek-V4 chat template (DSML / thinking)
-        - -ngl                     # offload 40 of 43 layers to the GPU. The model (~87 GiB) is
+        - -ngl                     # offload 38 of 43 layers to the GPU. The model (~87 GiB) is
-        - "40"                     # nearly the whole 90 GiB VRAM pool, so full offload (-ngl 999)
+        - "38"                     # nearly the whole 90 GiB VRAM pool, so full offload (-ngl 999)
-                                   # would overflow once KV cache + Vulkan compute buffers are
+                                   # would overflow once the f16 KV cache + Vulkan compute buffers
-                                   # added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB of
+                                   # are added. 38 layers (~77 GiB) + f16 KV (~5.7 GiB) + compute
-                                   # VRAM headroom for the KV cache, compute buffers, and
+                                   # (~2 GiB) ≈ 85 GiB, leaving ~5 GiB VRAM headroom. KV cache is
-                                   # co-resident pods. Raise toward 43 if VRAM allows; lower
+                                   # f16 (not q8_0) because Vulkan has no Flash Attention for
-                                   # (e.g. 38) if the pod OOMs / Vulkan runs out of device mem.
+                                   # deepseek4, which makes the KV cache ~2× larger than q8_0 would
+                                   # be — hence 38 rather than 40 layers offloaded. 5 layers
+                                   # (~10 GiB) run on CPU RAM. Raise toward 43 if VRAM allows;
+                                   # lower (e.g. 36) if the pod OOMs / Vulkan runs out of device mem.
         - -c                       # total KV-cache context (single slot gets the full window).
-        - "65536"                  # 64k — the required minimum. MLA KV is tiny (~1.6 GiB at
+        - "65536"                  # 64k — the required minimum. f16 MLA KV at 64k is ~5.7 GiB,
-                                   # q8_0), so context is cheap; -c is capped at the minimum to
+                                   # so context is affordable but not negligible. -c is capped at
-                                   # maximise VRAM headroom, not because KV is the constraint.
+                                   # the minimum to maximise VRAM headroom; raise if headroom allows.
-                                   # Raise if VRAM headroom allows.
         - -np                      # 1 slot => the full 64k goes to a single concurrent request
-        - "1"                      # (extra slots would multiply KV VRAM, which is fine here, but
+        - "1"                      # (extra slots would multiply KV VRAM; 1 slot keeps headroom maximal).
-                                   # 1 slot keeps it simple and headroom maximal).
         - --cont-batching          # continuous batching across slots
-        - --cache-type-k            # quantize KV cache to q8_0 — MLA KV is already small (~576
+        - --cache-type-k            # f16 K cache. deepseek4 / MLA models require K and V cache
-        - q8_0                     # elem/token/layer); q8_0 halves it to ~1.6 GiB at 64k and
+        - f16                      # types to be IDENTICAL, and quantized V cache requires Flash
-        - --cache-type-v            # maximises VRAM headroom with ~negligible quality loss.
+        - --cache-type-v            # Attention, which the Vulkan backend does NOT support for
-        - q8_0
+        - f16                      # deepseek4 (llama.cpp hard-errors otherwise). So both K and V
+                                   # must stay f16. KV at 64k ≈ 5.7 GiB.
         - --temp                   # default sampling temperature (DeepSeek-V4 recommendation)
         - "1.0"
         - --top-p                  # default nucleus sampling threshold (DeepSeek-V4 recommendation)
         - "0.95"
-        - --threads                # CPU threads for sampling + the 3 CPU-resident layers
+        - --threads                # CPU threads for sampling + the 5 CPU-resident layers
         - "8"
         ports:
         - name: http
           containerPort: 8080
         resources:
-          # The model weights + KV cache live in GPU VRAM (~90 GiB pool) and
+          # The GPU-resident model weights + KV cache live in VRAM (~90 GiB pool)
-          # are NOT counted against the cgroup memory limit. This limit only
+          # and are NOT counted against the cgroup memory limit. This limit only
-          # covers CPU-side overhead + the mmap'd GGUF pages for the 3
+          # covers CPU-side overhead + the mmap'd GGUF pages for the 5 CPU-resident
-          # CPU-resident layers (~6 GiB) plus reclaimable page cache during
+          # layers (~10 GiB, resident during inference) plus reclaimable page cache
-          # load. k8s sees ~30 GiB as the node's allocatable system RAM, so the
+          # during load. k8s sees ~30 GiB as the node's allocatable system RAM, so
-          # limit is sized to cover the CPU layers + overhead while leaving
+          # the limit is sized to cover the CPU layers + overhead while leaving RAM
-          # RAM for co-resident pods (litellm, the agents, etc.). If the pod is
+          # for co-resident pods (litellm, the agents, etc.). If the pod is
-          # OOM-killed during model load, raise the limit.
+          # OOM-killed during model load or inference, raise the limit (and/or
+          # lower -ngl to push more layers to VRAM).
           requests:
             cpu: "1000m"
-            memory: "4Gi"
+            memory: "6Gi"
           limits:
             cpu: "4000m"
-            memory: "20Gi"
+            memory: "24Gi"
         readinessProbe:
           httpGet:
             path: /health

llamacpp/pv.yaml

@@ -7,6 +7,13 @@
 # idempotently on first boot instead.
 #
 # nodeAffinity keeps the PV bound to the NUCBox even if labels change later.
+#
+# IMPORTANT: capacity is only metadata for a hostPath volume — k8s does NOT
+# enforce it and bumping it does NOT add physical disk space. The
+# DeepSeek-V4-Flash-0731 UD-IQ1_M GGUF is ~87 GiB across 3 shards, so the
+# hostPath filesystem (/data on the NUCBox) must physically have ~95 GiB free.
+# The fetch-model initContainer checks free space and fails loudly if the disk
+# is too small; expanding the disk is a host operation, not a manifest change.
 apiVersion: v1
 kind: PersistentVolume
 metadata: