platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

increase qwen3.6 models context available to 131k

Browse Source
This commit is contained in:
Roger Oriol
2026-07-26 01:29:23 +02:00
parent 5eaea8c387
commit 7af7ae66de
3 changed files with 20 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
llamacpp/README.md
View File

@@ -54,11 +54,11 @@ Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
| Model | Weights | KV cache | Subtotal |
|-------------------|----------|-------------------------------------------|----------|
| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 65k/slot) | ~32 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~4.5 GiB (q8_0, 131k total, 65k/slot) | ~24.5 GiB|
| **Total** | | | **~56.5 GiB** |
| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB |
| **Total** | | | **~61 GiB** |
~39.5 GiB headroom — still comfortable. Both models' KV caches are quantized
~35 GiB headroom — comfortable. Both models' KV caches are quantized
to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
@@ -68,14 +68,15 @@ a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
The key knobs (in each `deployment-*.yaml`):
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
- `-c 131072` (both) — total KV-cache context. With `-np 2` each slot gets
65536 tokens. The MoE a3b's KV cache is ~72 KiB/token so large context is
cheap; the dense 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0),
which is why the 27B also uses q8_0 KV to keep 131k affordable (~16 GiB).
- `-np 2` (both) — parallel slots (concurrent requests). Each extra slot
multiplies KV-cache VRAM usage. 2 slots give each cron/request a large context
slice (65k); bump higher only if you need more concurrent throughput (both have
VRAM headroom, but the 27B's dense KV is the constraint).
- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot
gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each).
The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense
27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B
uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
- `-np` — parallel slots (concurrent requests). Each extra slot multiplies
KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request,
dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV).
Bump higher only if you need more concurrent throughput.
- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.