platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

increase qwen3.6 models context available to 131k

Browse Source
This commit is contained in:
Roger Oriol
2026-07-26 01:29:23 +02:00
parent 5eaea8c387
commit 7af7ae66de
3 changed files with 20 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
llamacpp/README.md
View File

@@ -54,11 +54,11 @@ Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
| Model | Weights | KV cache | Subtotal | | Model | Weights | KV cache | Subtotal |
|-------------------|----------|-------------------------------------------|----------| |-------------------|----------|-------------------------------------------|----------|
| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 65k/slot) | ~32 GiB | | qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~4.5 GiB (q8_0, 131k total, 65k/slot) | ~24.5 GiB| | qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB |
| **Total** | | | **~56.5 GiB** | | **Total** | | | **~61 GiB** |
~39.5 GiB headroom — still comfortable. Both models' KV caches are quantized ~35 GiB headroom — comfortable. Both models' KV caches are quantized
to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
a3b's KV is tiny (~72 KiB/token) so its large context is nearly free. a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
@@ -68,14 +68,15 @@ a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
The key knobs (in each `deployment-*.yaml`): The key knobs (in each `deployment-*.yaml`):
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight. - `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
- `-c 131072` (both) — total KV-cache context. With `-np 2` each slot gets - `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot
65536 tokens. The MoE a3b's KV cache is ~72 KiB/token so large context is gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each).
cheap; the dense 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense
which is why the 27B also uses q8_0 KV to keep 131k affordable (~16 GiB). 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B
- `-np 2` (both) — parallel slots (concurrent requests). Each extra slot uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
multiplies KV-cache VRAM usage. 2 slots give each cron/request a large context - `-np` — parallel slots (concurrent requests). Each extra slot multiplies
slice (65k); bump higher only if you need more concurrent throughput (both have KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request,
VRAM headroom, but the 27B's dense KV is the constraint). dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV).
Bump higher only if you need more concurrent throughput.
- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to - `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b. 27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.

6
llamacpp/deployment-qwen36-27b.yaml
View File

@@ -83,10 +83,10 @@ spec:
- --jinja # use the GGUF's chat template (Qwen3 thinking format) - --jinja # use the GGUF's chat template (Qwen3 thinking format)
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999" - "999"
- -c # total KV-cache context, split across parallel slots - -c # total KV-cache context (single slot gets the full window)
- "131072" - "131072"
- -np # 2 parallel slots => 65536 tokens per concurrent request - -np # 1 slot => 131072 tokens for the single request (no split)
- "2" - "1"
- --cont-batching # continuous batching across slots - --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB
- q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c - q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c

8
llamacpp/deployment-qwen36-35b-a3b.yaml
View File

@@ -85,12 +85,12 @@ spec:
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999" - "999"
- -c # total KV-cache context, split across parallel slots - -c # total KV-cache context, split across parallel slots
- "131072" - "262144"
- -np # 2 parallel slots => 65536 tokens per concurrent request - -np # 2 parallel slots => 131072 tokens per concurrent request
- "2" - "2"
- --cont-batching # continuous batching across slots - --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~9 GiB → ~4.5 GiB) - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB
- q8_0 # with ~negligible quality loss; frees headroom for larger -c later - q8_0 # at 262k ctx); ~negligible quality loss, frees headroom for large -c
- --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows) - --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows)
- q8_0 - q8_0
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting) - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)