set qwen models parameters to allow for a higher context window

llamacpp/README.md

@@ -52,28 +52,34 @@ If only a CPU device shows up, the container can't see the GPU — check that
 
 Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
 
-| Model             | Weights  | KV cache (131k×4 for a3b, 32k×4 for 27b) | Subtotal |
+| Model             | Weights  | KV cache                                  | Subtotal |
 |-------------------|----------|-------------------------------------------|----------|
-| qwen3.6-27b       | ~16 GiB  | ~8 GiB  (32k total, 8k/slot)              | ~24 GiB  |
-| qwen3.6-35b-a3b   | ~20 GiB  | ~9 GiB  (131k total, 33k/slot)            | ~29 GiB  |
-| **Total**         |          |                                           | **~53 GiB** |
+| qwen3.6-27b       | ~16 GiB  | ~16 GiB  (q8_0, 131k total, 65k/slot)    | ~32 GiB  |
+| qwen3.6-35b-a3b   | ~20 GiB  | ~4.5 GiB (q8_0, 131k total, 65k/slot)    | ~24.5 GiB|
+| **Total**         |          |                                           | **~56.5 GiB** |
 
-~43 GiB headroom — very comfortable. The MoE's KV cache is tiny (~72 KiB/token
-vs the dense 27B's ~256 KiB/token), so large context is nearly free.
+~39.5 GiB headroom — still comfortable. Both models' KV caches are quantized
+to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
+large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
+a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
 
 ## Tuning
 
 The key knobs (in each `deployment-*.yaml`):
 
 - `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
-- `-c 131072` (35b-a3b) / `32768` (27b) — total KV-cache context. With `-np 4`
-  the 35b-a3b gets 32768 tokens per slot (enough for the full SOUL.md + prompt);
-  the 27b gets 8192 per slot. The MoE's KV cache is ~72 KiB/token so large
-  context is cheap; the dense 27B's is ~256 KiB/token. Raise the 27B's `-c` too
-  if interactive sessions hit the context limit.
-- `-np 4` — parallel slots (concurrent requests). Each extra slot multiplies
-  KV-cache VRAM usage. Bump higher on the flash model if you need more
-  throughput (it has VRAM headroom).
+- `-c 131072` (both) — total KV-cache context. With `-np 2` each slot gets
+  65536 tokens. The MoE a3b's KV cache is ~72 KiB/token so large context is
+  cheap; the dense 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0),
+  which is why the 27B also uses q8_0 KV to keep 131k affordable (~16 GiB).
+- `-np 2` (both) — parallel slots (concurrent requests). Each extra slot
+  multiplies KV-cache VRAM usage. 2 slots give each cron/request a large context
+  slice (65k); bump higher only if you need more concurrent throughput (both have
+  VRAM headroom, but the 27B's dense KV is the constraint).
+- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
+  q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
+  27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.
+  Drop to q4_0 for even less VRAM if retrieval quality allows.
 - `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
   full GPU offload; tune if CPU-bound.