forked from roger/k3s-cluster
fix deepseek deployment
This commit is contained in:
Roger Oriol
@@ -41,61 +41,89 @@ buffers, and co-resident pods.
|
||||
| `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. |
|
||||
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
|
||||
| `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. |
|
||||
| `-ngl` | `40` | **GPU layer offload.** Offload 40 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM. |
|
||||
| `-ngl` | `38` | **GPU layer offload.** Offload 38 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM, and the f16 KV cache (no Flash Attention on Vulkan) is larger than q8_0 would be, so 38 (not 40) layers are offloaded to leave ~5 GiB VRAM headroom. 5 layers (~10 GiB) run on CPU RAM. |
|
||||
| `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. |
|
||||
| `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. |
|
||||
| `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). |
|
||||
| `--cache-type-k` | `q8_0` | Quantize the K cache to q8_0 (halves KV VRAM, ~negligible quality loss). |
|
||||
| `--cache-type-v` | `q8_0` | Quantize the V cache to q8_0 (same). |
|
||||
| `--cache-type-k` | `f16` | **f16 K cache (NOT quantized).** The Vulkan backend has no Flash Attention for the `deepseek4` arch, and quantized V cache requires Flash Attention (llama.cpp hard-errors otherwise). Additionally, `deepseek4`/MLA models require K and V cache types to be *identical*, so K cannot be quantized either. |
|
||||
| `--cache-type-v` | `f16` | **f16 V cache.** Same reason — quantized V cache needs Flash Attention, which Vulkan lacks for deepseek4. |
|
||||
| `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. |
|
||||
| `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. |
|
||||
| `--threads` | `8` | CPU threads for sampling + the 3 CPU-resident layers. Mostly irrelevant under heavy GPU offload. |
|
||||
| `--threads` | `8` | CPU threads for sampling + the 5 CPU-resident layers. |
|
||||
|
||||
## VRAM budget (90 GiB pool)
|
||||
|
||||
The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
|
||||
it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
|
||||
VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
|
||||
`-ngl 38` offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on
|
||||
CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and
|
||||
fragmentation.
|
||||
|
||||
**KV cache is f16, not q8_0** — the Vulkan backend has no Flash Attention for
|
||||
`deepseek4`, and quantized V cache requires Flash Attention (llama.cpp
|
||||
hard-errors: *"quantized V cache was requested, but this requires Flash
|
||||
Attention"*). `deepseek4`/MLA models also require K and V cache types to be
|
||||
*identical*, so K cannot be quantized either. f16 KV at 64k is ~5.7 GiB (MLA
|
||||
KV: 576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes).
|
||||
|
||||
Approximate VRAM usage:
|
||||
|
||||
| Component | VRAM |
|
||||
|---------------------------------|-------------|
|
||||
| Weights (40 GPU layers) | ~81 GiB |
|
||||
| KV cache (q8_0, 64k, 1 slot) | ~1.6 GiB |
|
||||
| Weights (38 GPU layers) | ~77 GiB |
|
||||
| KV cache (f16, 64k, 1 slot) | ~5.7 GiB |
|
||||
| Vulkan compute buffers | ~2 GiB |
|
||||
| **Total in VRAM** | **~85 GiB** |
|
||||
| **Headroom (of 90 GiB)** | **~5–8 GiB**|
|
||||
| **Headroom (of 90 GiB)** | **~5 GiB** |
|
||||
|
||||
The remaining **3 layers (~6 GiB) live in CPU RAM** and are counted against the
|
||||
pod's cgroup memory limit (not VRAM). KV cache is tiny thanks to DeepSeek-V4's
|
||||
**MLA** attention (`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
|
||||
elements/token/layer); at 64k context q8_0 KV is only ~1.6 GiB, so context is
|
||||
**not** the constraint — `-c` is capped at the minimum purely to maximise VRAM
|
||||
headroom.
|
||||
5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory
|
||||
limit, not VRAM). VRAM is exclusive to this model (no other pod uses it); the
|
||||
other NUCBox pods only compete for the 30 GiB CPU RAM.
|
||||
|
||||
Note: several `deepseek4`-specific fused ops (Lightning Indexer, HC pre/comb/post)
|
||||
are not yet implemented in the Vulkan backend and fall back to CPU (logged as
|
||||
warnings, not fatal). Inference still works; it is slower than it will be once
|
||||
those ops land in a future `server-vulkan` build.
|
||||
|
||||
## How to tune if it OOMs / has spare headroom
|
||||
|
||||
- **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl`
|
||||
(e.g. `38`) to keep more layers on CPU, or raise the container `memory`
|
||||
limit.
|
||||
(e.g. `36`) to keep more layers on CPU, or raise the container `memory`
|
||||
limit. Remember the KV cache is f16 (cannot be quantized — no Flash Attention
|
||||
on Vulkan), so the only ways to free VRAM are fewer GPU layers or a smaller
|
||||
context window.
|
||||
- **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full
|
||||
offload) and/or raise `-c` for larger context.
|
||||
- **Need more concurrent requests:** raise `-np` (KV is cheap), but each extra
|
||||
slot multiplies KV VRAM and the context per slot shrinks (`-c / -np`).
|
||||
- **Need more concurrent requests:** raise `-np` (each extra slot multiplies the
|
||||
f16 KV cost, ~5.7 GiB/slot at 64k), and the context per slot shrinks
|
||||
(`-c / -np`).
|
||||
|
||||
## initContainer (`fetch-model`)
|
||||
|
||||
- Idempotently downloads the 3 shards into the shared models PVC, skipping if
|
||||
the first shard is already present (pod restart / recreate).
|
||||
- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim space for the
|
||||
new 87 GiB model.
|
||||
all shards are already present and non-empty (pod restart / recreate).
|
||||
- **Atomic, resumable downloads:** each shard is fetched to a `.partial` file
|
||||
(resumable via `curl -C -`) and only renamed to the final name on success, so
|
||||
an interrupted download never leaves a half-written final file that would
|
||||
wrongly skip re-download.
|
||||
- **Free-space check:** requires ~95 GiB free on `/models` before downloading;
|
||||
fails loudly with a clear message if the hostPath disk is too small (a PVC
|
||||
capacity bump does not add physical space to a hostPath volume — the disk on
|
||||
the NUCBox must be expanded).
|
||||
- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim ~36 GiB.
|
||||
|
||||
## Resources
|
||||
|
||||
```
|
||||
requests: cpu 1000m, memory 4Gi
|
||||
limits: cpu 4000m, memory 20Gi
|
||||
requests: cpu 1000m, memory 6Gi
|
||||
limits: cpu 4000m, memory 24Gi
|
||||
```
|
||||
|
||||
VRAM (weights + KV + compute) is **not** counted against the cgroup memory
|
||||
limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the
|
||||
3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. k8s
|
||||
sees ~30 GiB as the node's allocatable system RAM.
|
||||
5 CPU-resident layers (~10 GiB, resident during inference), and reclaimable
|
||||
page cache during load. k8s sees ~30 GiB as the node's allocatable system RAM.
|
||||
|
||||
## Probes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user