From 4694034a04624ad4e885ef25f198b12df5a918f4 Mon Sep 17 00:00:00 2001 From: Roger Oriol Date: Fri, 24 Jul 2026 21:30:01 +0200 Subject: [PATCH] set qwen models parameters to allow for a higher context window --- llamacpp/README.md | 34 +++++++++++++++---------- llamacpp/deployment-qwen36-27b.yaml | 10 +++++--- llamacpp/deployment-qwen36-35b-a3b.yaml | 8 ++++-- platform-engineer/cron-seed.yaml | 19 ++++++++++++-- 4 files changed, 50 insertions(+), 21 deletions(-) diff --git a/llamacpp/README.md b/llamacpp/README.md index 73a5af9..9184177 100644 --- a/llamacpp/README.md +++ b/llamacpp/README.md @@ -52,28 +52,34 @@ If only a CPU device shows up, the container can't see the GPU — check that Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage: -| Model | Weights | KV cache (131k×4 for a3b, 32k×4 for 27b) | Subtotal | +| Model | Weights | KV cache | Subtotal | |-------------------|----------|-------------------------------------------|----------| -| qwen3.6-27b | ~16 GiB | ~8 GiB (32k total, 8k/slot) | ~24 GiB | -| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (131k total, 33k/slot) | ~29 GiB | -| **Total** | | | **~53 GiB** | +| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 65k/slot) | ~32 GiB | +| qwen3.6-35b-a3b | ~20 GiB | ~4.5 GiB (q8_0, 131k total, 65k/slot) | ~24.5 GiB| +| **Total** | | | **~56.5 GiB** | -~43 GiB headroom — very comfortable. The MoE's KV cache is tiny (~72 KiB/token -vs the dense 27B's ~256 KiB/token), so large context is nearly free. +~39.5 GiB headroom — still comfortable. Both models' KV caches are quantized +to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's +large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The +a3b's KV is tiny (~72 KiB/token) so its large context is nearly free. ## Tuning The key knobs (in each `deployment-*.yaml`): - `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight. -- `-c 131072` (35b-a3b) / `32768` (27b) — total KV-cache context. With `-np 4` - the 35b-a3b gets 32768 tokens per slot (enough for the full SOUL.md + prompt); - the 27b gets 8192 per slot. The MoE's KV cache is ~72 KiB/token so large - context is cheap; the dense 27B's is ~256 KiB/token. Raise the 27B's `-c` too - if interactive sessions hit the context limit. -- `-np 4` — parallel slots (concurrent requests). Each extra slot multiplies - KV-cache VRAM usage. Bump higher on the flash model if you need more - throughput (it has VRAM headroom). +- `-c 131072` (both) — total KV-cache context. With `-np 2` each slot gets + 65536 tokens. The MoE a3b's KV cache is ~72 KiB/token so large context is + cheap; the dense 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), + which is why the 27B also uses q8_0 KV to keep 131k affordable (~16 GiB). +- `-np 2` (both) — parallel slots (concurrent requests). Each extra slot + multiplies KV-cache VRAM usage. 2 slots give each cron/request a large context + slice (65k); bump higher only if you need more concurrent throughput (both have + VRAM headroom, but the 27B's dense KV is the constraint). +- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to + q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense + 27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b. + Drop to q4_0 for even less VRAM if retrieval quality allows. - `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under full GPU offload; tune if CPU-bound. diff --git a/llamacpp/deployment-qwen36-27b.yaml b/llamacpp/deployment-qwen36-27b.yaml index b25fbd2..334e5f9 100644 --- a/llamacpp/deployment-qwen36-27b.yaml +++ b/llamacpp/deployment-qwen36-27b.yaml @@ -84,10 +84,14 @@ spec: - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - "999" - -c # total KV-cache context, split across parallel slots - - "32768" - - -np # 4 parallel slots => 8192 tokens per concurrent request - - "4" + - "131072" + - -np # 2 parallel slots => 65536 tokens per concurrent request + - "2" - --cont-batching # continuous batching across slots + - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB + - q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c + - --cache-type-v + - q8_0 - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting) - "8" ports: diff --git a/llamacpp/deployment-qwen36-35b-a3b.yaml b/llamacpp/deployment-qwen36-35b-a3b.yaml index 77cd695..966e198 100644 --- a/llamacpp/deployment-qwen36-35b-a3b.yaml +++ b/llamacpp/deployment-qwen36-35b-a3b.yaml @@ -86,9 +86,13 @@ spec: - "999" - -c # total KV-cache context, split across parallel slots - "131072" - - -np # 4 parallel slots => 32768 tokens per concurrent request - - "4" + - -np # 2 parallel slots => 65536 tokens per concurrent request + - "2" - --cont-batching # continuous batching across slots + - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~9 GiB → ~4.5 GiB) + - q8_0 # with ~negligible quality loss; frees headroom for larger -c later + - --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows) + - q8_0 - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting) - "8" ports: diff --git a/platform-engineer/cron-seed.yaml b/platform-engineer/cron-seed.yaml index cd7b8ae..5680a38 100644 --- a/platform-engineer/cron-seed.yaml +++ b/platform-engineer/cron-seed.yaml @@ -71,8 +71,23 @@ spec: } # ---- Watchdog checks (silent unless something is wrong) ---- - create "cluster-health-check" "every 6h" "discord" \ - "Check cluster health using the HTTP APIs documented in your SOUL.md. Check: (1) any node that is NotReady, (2) any pod not in Running phase, (3) any recent error/panic/crashloop/backoff log lines in Loki across all namespaces in the last 20 minutes, (4) any ArgoCD app that is not Synced plus Healthy. If everything is healthy, reply with exactly [SILENT]. Otherwise give a concise per-resource summary of what is wrong." + # NOTE: the old single "cluster-health-check" combined 4 heavy HTTP + # gathers (Prometheus + a cluster-wide raw Loki log dump) into one + # cron session and exceeded the flash model's per-request context. + # The 35b-a3b llama-server runs -c 131072 -np 4, so each request only + # gets ~32k tokens of KV cache (NOT 132k). It has been split into + # focused sub-checks so each run stays small, and the log check now + # uses LogQL aggregation instead of dumping raw lines cluster-wide. + create "node-health-check" "every 6h" "discord" \ + "Check node health using the Prometheus API documented in your SOUL.md. Query kube_node_status_condition{condition=\"Ready\",status!=\"true\"}. If all nodes are Ready, reply with exactly [SILENT]. Otherwise list each NotReady node by name. Keep it short; do not paste raw JSON — summarize with jq." + + create "pod-phase-check" "every 6h" "discord" \ + "Check pod health using the Prometheus API documented in your SOUL.md. Query kube_pod_status_phase{phase!=\"Running\"} and exclude Completed/Succeeded pods (jobs). If all Running, reply with exactly [SILENT]. Otherwise list each non-Running pod as namespace/pod:phase, grouped by namespace. Summarize with jq; do not dump raw JSON." + + create "pod-error-log-check" "every 6h" "discord" \ + "Check for recent crash-loop/error logs using the Loki API documented in your SOUL.md. IMPORTANT context discipline: do NOT fetch raw log lines across all namespaces — that overflows your context window. First run a LogQL aggregation that returns only counts: sum by (namespace, pod) (count_over_time({namespace=~\".+\"} |~ \"(?i)backoff|crashloop|panic\" [20m])). Parse the counts with jq. If every count is zero, reply with exactly [SILENT]. If any namespace/pod has a non-zero count, fetch at most 10 sample lines for THAT pod only (limit=10) to identify the cause. Summarize concisely; never paste more than a handful of lines." + + # (ArgoCD health is already covered by the argocd-sync-health cron below.) create "pod-restart-loop" "every 1h" "discord" \ "Find pods with high restart rates using the Prometheus API documented in your SOUL.md. If any pod has more than 3 restarts in the last 15 minutes, fetch its logs from Loki to diagnose the cause. If the cause is clearly fixable via a manifest change such as bumping a memory limit, fixing a config value, or bumping the restartedAt annotation, make the edit in /workspace/k3s-cluster, commit and push, then trigger an ArgoCD sync via the API. Report what you did in one line. If not clearly fixable, post the log excerpt and proposed fix, and wait for Roger. If no high-restart pods, reply [SILENT]."