diff --git a/llamacpp/README.md b/llamacpp/README.md index fcffee2..73a5af9 100644 --- a/llamacpp/README.md +++ b/llamacpp/README.md @@ -52,24 +52,25 @@ If only a CPU device shows up, the container can't see the GPU — check that Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage: -| Model | Weights | KV cache (32k×4) | Subtotal | -|-------------------|----------|-------------------|----------| -| qwen3.6-27b | ~16 GiB | ~34 GiB | ~50 GiB | -| qwen3.6-35b-a3b | ~20 GiB | ~10 GiB | ~30 GiB | -| **Total** | | | **~80 GiB** | +| Model | Weights | KV cache (131k×4 for a3b, 32k×4 for 27b) | Subtotal | +|-------------------|----------|-------------------------------------------|----------| +| qwen3.6-27b | ~16 GiB | ~8 GiB (32k total, 8k/slot) | ~24 GiB | +| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (131k total, 33k/slot) | ~29 GiB | +| **Total** | | | **~53 GiB** | -~16 GiB headroom — comfortable but not infinite. If VRAM is exhausted ( Vulkan -allocation failures in logs), reduce `-c` on the 27B (its KV cache dominates) or -drop `-np` to 2 on either model. +~43 GiB headroom — very comfortable. The MoE's KV cache is tiny (~72 KiB/token +vs the dense 27B's ~256 KiB/token), so large context is nearly free. ## Tuning The key knobs (in each `deployment-*.yaml`): - `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight. -- `-c 32768` — total KV-cache context. With `-np 4` this is 8192 tokens per - concurrent request. The 27B's dense KV cache is the larger consumer (~34 GiB - at 32k×4); the MoE's is much smaller (~10 GiB). +- `-c 131072` (35b-a3b) / `32768` (27b) — total KV-cache context. With `-np 4` + the 35b-a3b gets 32768 tokens per slot (enough for the full SOUL.md + prompt); + the 27b gets 8192 per slot. The MoE's KV cache is ~72 KiB/token so large + context is cheap; the dense 27B's is ~256 KiB/token. Raise the 27B's `-c` too + if interactive sessions hit the context limit. - `-np 4` — parallel slots (concurrent requests). Each extra slot multiplies KV-cache VRAM usage. Bump higher on the flash model if you need more throughput (it has VRAM headroom). diff --git a/llamacpp/deployment-qwen36-35b-a3b.yaml b/llamacpp/deployment-qwen36-35b-a3b.yaml index 28af4d8..77cd695 100644 --- a/llamacpp/deployment-qwen36-35b-a3b.yaml +++ b/llamacpp/deployment-qwen36-35b-a3b.yaml @@ -85,8 +85,8 @@ spec: - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - "999" - -c # total KV-cache context, split across parallel slots - - "32768" - - -np # 4 parallel slots => 8192 tokens per concurrent request + - "131072" + - -np # 4 parallel slots => 32768 tokens per concurrent request - "4" - --cont-batching # continuous batching across slots - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)