platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

increase llamacpp models context window!

Browse Source
This commit is contained in:
Roger Oriol
2026-07-24 20:11:30 +02:00
parent 496e46076c
commit c982b33015
2 changed files with 14 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
llamacpp/README.md
View File

@@ -52,24 +52,25 @@ If only a CPU device shows up, the container can't see the GPU — check that
Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage: Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
| Model | Weights | KV cache (32k×4) | Subtotal | | Model | Weights | KV cache (131k×4 for a3b, 32k×4 for 27b) | Subtotal |
|-------------------|----------|-------------------|----------| |-------------------|----------|-------------------------------------------|----------|
| qwen3.6-27b | ~16 GiB | ~34 GiB | ~50 GiB | | qwen3.6-27b | ~16 GiB | ~8 GiB (32k total, 8k/slot) | ~24 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~10 GiB | ~30 GiB | | qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (131k total, 33k/slot) | ~29 GiB |
| **Total** | | | **~80 GiB** | | **Total** | | | **~53 GiB** |
~16 GiB headroom — comfortable but not infinite. If VRAM is exhausted ( Vulkan ~43 GiB headroom — very comfortable. The MoE's KV cache is tiny (~72 KiB/token
allocation failures in logs), reduce `-c` on the 27B (its KV cache dominates) or vs the dense 27B's ~256 KiB/token), so large context is nearly free.
drop `-np` to 2 on either model.
## Tuning ## Tuning
The key knobs (in each `deployment-*.yaml`): The key knobs (in each `deployment-*.yaml`):
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight. - `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
- `-c 32768` — total KV-cache context. With `-np 4` this is 8192 tokens per - `-c 131072` (35b-a3b) / `32768` (27b) — total KV-cache context. With `-np 4`
concurrent request. The 27B's dense KV cache is the larger consumer (~34 GiB the 35b-a3b gets 32768 tokens per slot (enough for the full SOUL.md + prompt);
at 32k×4); the MoE's is much smaller (~10 GiB). the 27b gets 8192 per slot. The MoE's KV cache is ~72 KiB/token so large
context is cheap; the dense 27B's is ~256 KiB/token. Raise the 27B's `-c` too
if interactive sessions hit the context limit.
- `-np 4` — parallel slots (concurrent requests). Each extra slot multiplies - `-np 4` — parallel slots (concurrent requests). Each extra slot multiplies
KV-cache VRAM usage. Bump higher on the flash model if you need more KV-cache VRAM usage. Bump higher on the flash model if you need more
throughput (it has VRAM headroom). throughput (it has VRAM headroom).

4
llamacpp/deployment-qwen36-35b-a3b.yaml
View File

@@ -85,8 +85,8 @@ spec:
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999" - "999"
- -c # total KV-cache context, split across parallel slots - -c # total KV-cache context, split across parallel slots
- "32768" - "131072"
- -np # 4 parallel slots => 8192 tokens per concurrent request - -np # 4 parallel slots => 32768 tokens per concurrent request
- "4" - "4"
- --cont-batching # continuous batching across slots - --cont-batching # continuous batching across slots
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting) - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)