diff --git a/llamacpp/README.md b/llamacpp/README.md index 9184177..93664dc 100644 --- a/llamacpp/README.md +++ b/llamacpp/README.md @@ -54,11 +54,11 @@ Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage: | Model | Weights | KV cache | Subtotal | |-------------------|----------|-------------------------------------------|----------| -| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 65k/slot) | ~32 GiB | -| qwen3.6-35b-a3b | ~20 GiB | ~4.5 GiB (q8_0, 131k total, 65k/slot) | ~24.5 GiB| -| **Total** | | | **~56.5 GiB** | +| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB | +| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB | +| **Total** | | | **~61 GiB** | -~39.5 GiB headroom — still comfortable. Both models' KV caches are quantized +~35 GiB headroom — comfortable. Both models' KV caches are quantized to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The a3b's KV is tiny (~72 KiB/token) so its large context is nearly free. @@ -68,14 +68,15 @@ a3b's KV is tiny (~72 KiB/token) so its large context is nearly free. The key knobs (in each `deployment-*.yaml`): - `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight. -- `-c 131072` (both) — total KV-cache context. With `-np 2` each slot gets - 65536 tokens. The MoE a3b's KV cache is ~72 KiB/token so large context is - cheap; the dense 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), - which is why the 27B also uses q8_0 KV to keep 131k affordable (~16 GiB). -- `-np 2` (both) — parallel slots (concurrent requests). Each extra slot - multiplies KV-cache VRAM usage. 2 slots give each cron/request a large context - slice (65k); bump higher only if you need more concurrent throughput (both have - VRAM headroom, but the 27B's dense KV is the constraint). +- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot + gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each). + The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense + 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B + uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot. +- `-np` — parallel slots (concurrent requests). Each extra slot multiplies + KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request, + dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV). + Bump higher only if you need more concurrent throughput. - `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense 27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b. diff --git a/llamacpp/deployment-qwen36-27b.yaml b/llamacpp/deployment-qwen36-27b.yaml index 334e5f9..b4d2caa 100644 --- a/llamacpp/deployment-qwen36-27b.yaml +++ b/llamacpp/deployment-qwen36-27b.yaml @@ -83,10 +83,10 @@ spec: - --jinja # use the GGUF's chat template (Qwen3 thinking format) - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - "999" - - -c # total KV-cache context, split across parallel slots + - -c # total KV-cache context (single slot gets the full window) - "131072" - - -np # 2 parallel slots => 65536 tokens per concurrent request - - "2" + - -np # 1 slot => 131072 tokens for the single request (no split) + - "1" - --cont-batching # continuous batching across slots - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB - q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c diff --git a/llamacpp/deployment-qwen36-35b-a3b.yaml b/llamacpp/deployment-qwen36-35b-a3b.yaml index 966e198..c2d50a0 100644 --- a/llamacpp/deployment-qwen36-35b-a3b.yaml +++ b/llamacpp/deployment-qwen36-35b-a3b.yaml @@ -85,12 +85,12 @@ spec: - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - "999" - -c # total KV-cache context, split across parallel slots - - "131072" - - -np # 2 parallel slots => 65536 tokens per concurrent request + - "262144" + - -np # 2 parallel slots => 131072 tokens per concurrent request - "2" - --cont-batching # continuous batching across slots - - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~9 GiB → ~4.5 GiB) - - q8_0 # with ~negligible quality loss; frees headroom for larger -c later + - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB + - q8_0 # at 262k ctx); ~negligible quality loss, frees headroom for large -c - --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows) - q8_0 - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)