fix deepseek deployment

llamacpp/README.md

@@ -65,47 +65,61 @@ If only a CPU device shows up, the container can't see the GPU — check that
 The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
 it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
 VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
-`-ngl 40` offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on
-CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and
-co-resident pods. Approximate VRAM usage:
+`-ngl 38` offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on
+CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and
+fragmentation.
+
+**KV cache is f16, not q8_0** — the Vulkan backend has no Flash Attention for
+the `deepseek4` arch, and quantized V cache requires Flash Attention (llama.cpp
+hard-errors: *"quantized V cache was requested, but this requires Flash
+Attention"*). `deepseek4`/MLA models also require K and V cache types to be
+*identical*, so K cannot be quantized either. f16 MLA KV at 64k is ~5.7 GiB
+(576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes) —
+larger than q8_0 would be, which is why `-ngl` is 38 rather than 40.
+
+Approximate VRAM usage:
 
 | Component                       | VRAM        |
 |---------------------------------|-------------|
-| Weights (40 GPU layers)         | ~81 GiB     |
-| KV cache (q8_0, 64k, 1 slot)    | ~1.6 GiB    |
+| Weights (38 GPU layers)         | ~77 GiB     |
+| KV cache (f16, 64k, 1 slot)     | ~5.7 GiB    |
 | Vulkan compute buffers          | ~2 GiB      |
 | **Total in VRAM**               | **~85 GiB** |
-| **Headroom (of 90 GiB)**        | **~5–8 GiB**|
+| **Headroom (of 90 GiB)**        | **~5 GiB**  |
 
-3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory
-limit, not VRAM).
+5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory
+limit, not VRAM). VRAM is exclusive to this model; the other NUCBox pods only
+compete for the 30 GiB CPU RAM.
 
-KV cache is tiny thanks to DeepSeek-V4's **MLA** attention
-(`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
-elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is
-cheap — `-c` is capped at 65536 (the required minimum) to maximise VRAM
-headroom, not because KV is the constraint.
+Several `deepseek4`-specific fused ops (Lightning Indexer, HC pre/comb/post)
+are not yet implemented in the Vulkan backend and fall back to CPU (logged as
+warnings, not fatal). Inference still works; it will speed up once those ops
+land in a future `server-vulkan` build.
 
 ## Tuning
 
 The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`):
 
-- `-ngl 40` — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the
-  whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute
-  buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom.
-  Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs
-  out of device memory.
+- `-ngl 38` — offload 38 of 43 layers to GPU. The model (~87 GiB) is nearly the
+  whole 90 GiB VRAM pool, so full offload would overflow once the f16 KV cache +
+  compute buffers are added. 5 layers (~10 GiB) on CPU leaves ~5 GiB VRAM
+  headroom. Raise toward 43 if VRAM allows; lower (e.g. 36) if the pod OOMs /
+  Vulkan runs out of device memory.
 - `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets
-  the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped
-  at the minimum to maximise VRAM headroom. Raise if headroom allows.
+  the full 64k. f16 MLA KV at 64k is ~5.7 GiB; capped at the minimum to maximise
+  VRAM headroom. Raise if headroom allows.
 - `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request).
-  Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal.
-- `--cache-type-k q8_0 --cache-type-v q8_0` — quantize the KV cache to q8_0,
-  halving KV VRAM with ~negligible quality loss. Essential to keep headroom.
+  Extra slots multiply the f16 KV cost (~5.7 GiB/slot); 1 slot keeps headroom
+  maximal.
+- `--cache-type-k f16 --cache-type-v f16` — **f16 KV cache (NOT quantized).**
+  The Vulkan backend has no Flash Attention for `deepseek4`, and quantized V
+  cache requires Flash Attention. `deepseek4`/MLA models also require K and V
+  cache types to be identical, so K cannot be quantized either. This is the
+  reason `-ngl` is 38 rather than 40.
 - `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4
   recommendation). These are server defaults; clients can override per request
   via the OpenAI-compatible API.
-- `--threads 8` — CPU threads for sampling + the 3 CPU-resident layers.
+- `--threads 8` — CPU threads for sampling + the 5 CPU-resident layers.
 
 ## Memory accounting
 
@@ -113,9 +127,10 @@ k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is
 reserved by firmware and managed by `amdgpu`). The GPU-resident model weights
 and KV cache live in VRAM and are **not** counted against the container's cgroup
 memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages
-for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load.
-If the pod is OOM-killed during model load, raise the memory limit (and/or
-lower `-ngl`).
+for the 5 CPU-resident layers (~10 GiB, resident during inference), and
+reclaimable page cache during load. If the pod is OOM-killed during model load
+or inference, raise the memory limit (and/or lower `-ngl` to push more layers
+to VRAM).
 
 ## Adding / replacing a model