fix lfm 2.5

home-manager/README.md

@@ -57,7 +57,7 @@ Docs: https://hermes-agent.nousresearch.com/docs/user-guide/docker
 | Discord bot | dedicated | dedicated (separate token + channel) |
 | `HERMES_WRITE_SAFE_ROOT` | `/` (needs to edit manifests) | unset → defaults to `/opt/data` (tighter) |
 | Image | `nousresearch/hermes-agent:latest` | same |
-| Model | `qwen3.6` via LiteLLM | same |
+| Model | `deepseek-v4-flash-0731` via LiteLLM | same |
 
 ---
 

home-manager/configmap.yaml

@@ -9,18 +9,18 @@ data:
   config.yaml: |
     model:
       provider: openai-api
-      default: qwen3.6-35b-a3b
+      default: lfm2.5-2.6b
       base_url: "http://litellm-service.litellm:80/v1"
       api_mode: chat_completions
 
     auxiliary:
       compression:
         provider: openai-api
-        model: qwen3.6-35b-a3b
+        model: lfm2.5-2.6b
         base_url: "http://litellm-service.litellm:80/v1"
       title_generation:
         provider: openai-api
-        model: qwen3.6-35b-a3b
+        model: lfm2.5-2.6b
         base_url: "http://litellm-service.litellm:80/v1"
 
     terminal:
@@ -82,6 +82,14 @@ data:
     the home comfortable, safe, and efficient by talking to Home Assistant, and
     to notify your owner (Roger) on Discord when something needs a human.
 
+    ## Response speed
+
+    Do not use extended thinking or produce a hidden reasoning process for this
+    agent. Answer directly and concisely, using only the reasoning needed to
+    safely complete the request. Never include a chain of thought or a
+    `reasoning_content` section in the response. This is a prompt-level
+    instruction; do not change the llama.cpp server configuration.
+
     ## The home you look after
 
     - **Home Assistant:** the brain of the home. Runs in namespace

homeassistant/homeassistant.yaml

@@ -120,6 +120,11 @@ spec:
                 values:
                 - raspberrypi
       hostNetwork: true
+      # Required for hostNetwork pods to resolve cluster DNS
+      # (*.svc.cluster.local) via kube-dns instead of the host's resolver.
+      # Without this, resolv.conf points at the host (1.1.1.1) and in-cluster
+      # services like wyoming-whisper.llamacpp.svc.cluster.local won't resolve.
+      dnsPolicy: ClusterFirstWithHostNet
       volumes:
         - name: config
           persistentVolumeClaim:

litellm/litellm.yaml

@@ -39,22 +39,18 @@ data:
         litellm_params:
           model: ollama/glm-4.7-flash
           api_base: http://10.88.20.12:11434
-      # Used by the platform-engineer Hermes agent (deployed in ns platform-engineer).
+      # In-cluster models served by llama.cpp (llama-server, Vulkan on the
-      # Served by llama.cpp (llama-server, Vulkan on the NUCBox APU) — see
+      # NUCBox APU). The chat template is applied via --jinja; reasoning (when
-      # llamacpp/deployment-qwen36-27b.yaml. Thinking is left ON; reasoning comes
+      # enabled) comes back in `reasoning_content`.
-      # back in `reasoning_content` (llama-server --jinja + default reasoning format).
+      - model_name: qwen3.8-27b
-      - model_name: qwen3.6-27b
         litellm_params:
-          model: openai/qwen3.6-27b
+          model: openai/qwen3.8-27b
-          api_base: http://llamacpp-qwen36-27b.llamacpp/v1
+          api_base: http://llamacpp-qwen38-27b.llamacpp/v1
           api_key: "sk-no-auth"
-      # Faster "flash" variant: Qwen3.6-35B-A3B is a MoE (3B active params), so
+      - model_name: lfm2.5-2.6b
-      # inference is much faster than the dense 27B despite more total weights.
-      # See llamacpp/deployment-qwen36-35b-a3b.yaml.
-      - model_name: qwen3.6-35b-a3b
         litellm_params:
-          model: openai/qwen3.6-35b-a3b
+          model: openai/lfm2.5-2.6b
-          api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
+          api_base: http://llamacpp-lfm25-26b.llamacpp/v1
           api_key: "sk-no-auth"
     litellm_settings:
       #set_verbose: True  # Uncomment this if you want to see verbose logs; not recommended in production
@@ -140,4 +136,3 @@ spec:
     - name: http
       port: 80
       targetPort: 4000
-

llamacpp/README.md

@@ -1,109 +1,60 @@
 # llama.cpp (llama-server)
 
-In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on
+In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model
-the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified
+on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB
-memory: 32 GiB RAM / 96 GiB VRAM).
+unified memory).
 
-This replaces the bare-metal Ollama setup for models that benefit from
+LiteLLM (`litellm/`) points at the in-cluster Service instead of the external
-always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these
+Ollama endpoint.
-in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
 
-## Layout
+## Active Model
 
-One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
+| Alias | Model | Configuration | Service |
-to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
+|---|---|---|---|
+| `qwen3.8-27b` | Qwen3.8-27B with MTP | `Q4_K_M` primary, `Q4_0` draft, 196k context, q8_0 K/V cache | `llamacpp-qwen38-27b.llamacpp:80` |
+| `lfm2.5-2.6b` | LiquidAI LFM2.5-2.6B with speculative decoding | `Q4_K_M` primary, `Q4_0` draft, 121k context, temperature 0.1 | `llamacpp-lfm25-26b.llamacpp:80` |
 
-| Alias             | Model                      | GGUF                                                  | Service                                   |
+An initContainer downloads both model files atomically before llama-server
-|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
+starts:
-| `qwen3.6-27b`     | Qwen3.6-27B (dense)        | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB)      | `llamacpp-qwen36-27b.llamacpp:80`         |
-| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB)  | `llamacpp-qwen36-35b-a3b.llamacpp:80`     |
 
-The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
+- Primary: `ggml-org/Qwen3.8-27B-GGUF:Q4_K_M`
-inference is significantly faster than the dense 27B despite more total weights —
+- Draft: `ggml-org/Qwen3.8-27B-GGUF:Q4_0`
-hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for
-deeper reasoning.
 
-Model files are downloaded idempotently by an initContainer into a shared
+The model files are stored on the shared hostPath PVC at
-hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
+`/data/llamacpp/models` on the NUCBox. The server is configured with
-without re-downloading.
+`--spec-default --spec-type draft-mtp`, `--reasoning-preserve`, `--fit off`,
+and `--agent`.
+
+## Retired Models
+
+The following deployments are no longer active, but their argument references
+are retained for future redeployment:
+
+- [DeepSeek-V4-Flash-0731](args-deepseek-v4-flash-0731.md)
+- [Qwen3.6-27B](args-qwen36-27b.md)
+- [Qwen3.6-35B-A3B](args-qwen36-35b-a3b.md)
 
 ## GPU / Vulkan
 
 The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
-the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full
+the Mesa/RADV Vulkan driver for the Radeon 8060S. The container mounts
-layer offload (`-ngl 999`) puts model weights entirely in the 96 GiB VRAM pool.
+`/dev/dri` and runs privileged, which is the current way to provide Vulkan
+access on k3s without a device plugin.
 
-The container mounts `/dev/dri` and runs `privileged: true` — the simplest
+Verify GPU use with:
-reliable way to give Vulkan access to the DRM render node on k3s without a
-device plugin. Tighten later with `supplementalGroups` (the host's `render`
-group GID) if desired.
-
-### Verify the GPU is actually used
 
 ```bash
-kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b     | grep -iE 'vulkan|gpu|offload|device'
+kubectl logs -n llamacpp deploy/llamacpp-qwen38-27b | grep -iE 'vulkan|gpu|offload|device'
-kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b  | grep -iE 'vulkan|gpu|offload|device'
 ```
 
-If only a CPU device shows up, the container can't see the GPU — check that
+If only a CPU device appears, check that `/dev/dri/renderD128` exists on the
-`/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
+NUCBox and that the `amdgpu` module is loaded.
 
-## VRAM budget (both models co-resident)
+## Adding Or Replacing A Model
 
-Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
+1. Copy `deployment-qwen38-27b.yaml` to `deployment-<new>.yaml` and change the
-
+   model repositories, alias, labels, and Service name.
-| Model             | Weights  | KV cache                                  | Subtotal |
+2. Add the matching alias and Service URL to `litellm/litellm.yaml`.
-|-------------------|----------|-------------------------------------------|----------|
+3. No `gen-apps.sh` change is needed because the `llamacpp` ArgoCD Application
-| qwen3.6-27b       | ~16 GiB  | ~16 GiB  (q8_0, 131k total, 1 slot)      | ~32 GiB  |
+   syncs the directory recursively.
-| qwen3.6-35b-a3b   | ~20 GiB  | ~9 GiB   (q8_0, 262k total, 131k/slot)   | ~29 GiB  |
+4. Check the model and KV-cache size against the NUCBox's available VRAM.
-| **Total**         |          |                                           | **~61 GiB** |
-
-~35 GiB headroom — comfortable. Both models' KV caches are quantized
-to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
-large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
-a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
-
-## Tuning
-
-The key knobs (in each `deployment-*.yaml`):
-
-- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
-- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot
-  gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each).
-  The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense
-  27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B
-  uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
-- `-np` — parallel slots (concurrent requests). Each extra slot multiplies
-  KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request,
-  dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV).
-  Bump higher only if you need more concurrent throughput.
-- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
-  q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
-  27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.
-  Drop to q4_0 for even less VRAM if retrieval quality allows.
-- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
-  full GPU offload; tune if CPU-bound.
-
-## Memory accounting
-
-k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is
-reserved by firmware and managed by `amdgpu`). The model weights and KV cache
-live in VRAM and are **not** counted against the container's cgroup memory
-limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages
-during load. If the pod is OOM-killed during model load, raise the memory limit.
-
-## Adding a model
-
-1. Copy `deployment-qwen36-27b.yaml` → `deployment-<new>.yaml`; change the
-   `model:` label, GGUF URL/file, `--alias`, and Service name.
-2. Point LiteLLM at it in `litellm/litellm.yaml`:
-   ```yaml
-   - model_name: <alias>
-     litellm_params:
-       model: openai/<alias>
-       api_base: http://<service>.llamacpp/v1
-       api_key: "sk-no-auth"
-   ```
-3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
-   directory recursively.)
-4. Check the VRAM budget table above — two large models may not coexist.

llamacpp/args-deepseek-v4-flash-0731.md

@@ -0,0 +1,137 @@
+# Deployment arguments — `deepseek-v4-flash-0731` (RETIRED)
+
+Historical reference for the llama-server flags used by the retired
+`deployment-deepseek-v4-flash-0731.yaml`. The Deployment can be recovered from
+git history if this model is needed again.
+
+## Model & source
+
+- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` (DeepSeek-V4-Flash, 0731 weights)
+- **Quantization:** `unsloth/DeepSeek-V4-Flash-0731-GGUF` → `UD-IQ1_M`
+  (Unsloth Dynamic IQ1_M), **split across 3 shards** (~87 GiB total):
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf`
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf`
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf`
+- **HuggingFace repo:** `https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (floating tag — `deepseek4`
+  is a brand-new arch; pin to a specific `server-vulkan-bXXXX` once verified).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5),
+~120 GiB unified memory split by firmware into **~90 GiB VRAM** and
+**~30 GiB CPU RAM**. Pinned via `nodeSelector: {kubernetes.io/arch: amd64,
+hardware: high-memory}`.
+
+## Why this model nearly fills the machine
+
+The UD-IQ1_M model is **~87 GiB**, almost the entire 90 GiB VRAM pool. Unlike
+the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one
+**cannot be fully offloaded** to the GPU: putting all 43 layers + the KV cache +
+Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all
+about fitting the model while leaving headroom for the KV cache, compute
+buffers, and co-resident pods.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf` | Model file. Pointed at the **first shard** only; llama.cpp auto-discovers and loads `-00002-…` / `-00003-…` from the same directory. |
+| `--alias` | `deepseek-v4-flash-0731` | Name reported by `GET /v1/models`. Must match the `model_name` in `litellm/litellm.yaml` so LiteLLM routes to this server. |
+| `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. |
+| `-ngl` | `38` | **GPU layer offload.** Offload 38 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM, and the f16 KV cache (no Flash Attention on Vulkan) is larger than q8_0 would be, so 38 (not 40) layers are offloaded to leave ~5 GiB VRAM headroom. 5 layers (~10 GiB) run on CPU RAM. |
+| `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. |
+| `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. |
+| `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). |
+| `--cache-type-k` | `f16` | **f16 K cache (NOT quantized).** The Vulkan backend has no Flash Attention for the `deepseek4` arch, and quantized V cache requires Flash Attention (llama.cpp hard-errors otherwise). Additionally, `deepseek4`/MLA models require K and V cache types to be *identical*, so K cannot be quantized either. |
+| `--cache-type-v` | `f16` | **f16 V cache.** Same reason — quantized V cache needs Flash Attention, which Vulkan lacks for deepseek4. |
+| `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. |
+| `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. |
+| `--threads` | `8` | CPU threads for sampling + the 5 CPU-resident layers. |
+
+## VRAM budget (90 GiB pool)
+
+The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
+it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
+VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
+`-ngl 38` offloads 38 of 43 layers to the GPU and keeps 5 layers (~10 GiB) on
+CPU RAM, leaving ~5 GiB of VRAM headroom for the KV cache, compute buffers, and
+fragmentation.
+
+**KV cache is f16, not q8_0** — the Vulkan backend has no Flash Attention for
+`deepseek4`, and quantized V cache requires Flash Attention (llama.cpp
+hard-errors: *"quantized V cache was requested, but this requires Flash
+Attention"*). `deepseek4`/MLA models also require K and V cache types to be
+*identical*, so K cannot be quantized either. f16 KV at 64k is ~5.7 GiB (MLA
+KV: 576 K + 512 V elements/token/layer × 43 layers × 65536 tokens × 2 bytes).
+
+Approximate VRAM usage:
+
+| Component                       | VRAM        |
+|---------------------------------|-------------|
+| Weights (38 GPU layers)         | ~77 GiB     |
+| KV cache (f16, 64k, 1 slot)     | ~5.7 GiB    |
+| Vulkan compute buffers          | ~2 GiB      |
+| **Total in VRAM**               | **~85 GiB** |
+| **Headroom (of 90 GiB)**        | **~5 GiB**  |
+
+5 layers (~10 GiB) live in CPU RAM (counted against the pod's cgroup memory
+limit, not VRAM). VRAM is exclusive to this model (no other pod uses it); the
+other NUCBox pods only compete for the 30 GiB CPU RAM.
+
+Note: several `deepseek4`-specific fused ops (Lightning Indexer, HC pre/comb/post)
+are not yet implemented in the Vulkan backend and fall back to CPU (logged as
+warnings, not fatal). Inference still works; it is slower than it will be once
+those ops land in a future `server-vulkan` build.
+
+## How to tune if it OOMs / has spare headroom
+
+- **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl`
+  (e.g. `36`) to keep more layers on CPU, or raise the container `memory`
+  limit. Remember the KV cache is f16 (cannot be quantized — no Flash Attention
+  on Vulkan), so the only ways to free VRAM are fewer GPU layers or a smaller
+  context window.
+- **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full
+  offload) and/or raise `-c` for larger context.
+- **Need more concurrent requests:** raise `-np` (each extra slot multiplies the
+  f16 KV cost, ~5.7 GiB/slot at 64k), and the context per slot shrinks
+  (`-c / -np`).
+
+## initContainer (`fetch-model`)
+
+- Idempotently downloads the 3 shards into the shared models PVC, skipping if
+  all shards are already present and non-empty (pod restart / recreate).
+- **Atomic, resumable downloads:** each shard is fetched to a `.partial` file
+  (resumable via `curl -C -`) and only renamed to the final name on success, so
+  an interrupted download never leaves a half-written final file that would
+  wrongly skip re-download.
+- **Free-space check:** requires ~95 GiB free on `/models` before downloading;
+  fails loudly with a clear message if the hostPath disk is too small (a PVC
+  capacity bump does not add physical space to a hostPath volume — the disk on
+  the NUCBox must be expanded).
+- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim ~36 GiB.
+
+## Resources
+
+```
+requests:  cpu 1000m, memory 6Gi
+limits:    cpu 4000m, memory 24Gi
+```
+
+VRAM (weights + KV + compute) is **not** counted against the cgroup memory
+limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the
+5 CPU-resident layers (~10 GiB, resident during inference), and reclaimable
+page cache during load. k8s sees ~30 GiB as the node's allocatable system RAM.
+
+## Probes
+
+- `readinessProbe`: `GET /health` after 30s, every 10s, 6 failures.
+- `livenessProbe`: `GET /health` after **300s** (87 GiB load + Vulkan init
+  takes several minutes), every 30s, 5 failures.
+
+## Security / GPU access
+
+`privileged: true` + mounts `/dev/dri` (DRM render nodes) — simplest reliable
+way to give Vulkan access to the AMD APU on k3s without a device plugin.

llamacpp/args-qwen36-27b.md

@@ -0,0 +1,85 @@
+# Deployment arguments — `qwen3.6-27b` (REMOVED)
+
+> **Status:** This model was **removed** from the cluster to make VRAM room for
+> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
+> 90 GiB VRAM pool on its own. This file documents the flags used when the
+> model was live, so it can be redeployed later if the DeepSeek model is taken
+> down or moved to different hardware.
+>
+> To redeploy: restore `deployment-qwen36-27b.yaml` from git history and
+> re-add the LiteLLM entry. Re-check the VRAM budget —
+> co-locating with the 87 GiB DeepSeek model is **not** possible on the current
+> 90 GiB pool.
+
+## Model & source
+
+- **Model:** `unsloth/Qwen3.6-27B-MTP-GGUF` — Qwen3.6-27B, **dense** (all
+  params active per token), MTP variant.
+- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~16 GiB**,
+  single GGUF file (`Qwen3.6-27B-UD-Q4_K_XL.gguf`).
+- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
+  driver, supports Radeon 8060S / RDNA 3.5).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
+memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
+{kubernetes.io/arch: amd64, hardware: high-memory}`.
+
+## Why it fit (and co-existed with the 35B-A3B)
+
+At ~16 GiB, the Q4 model is small enough to **fully offload** to the GPU
+(`-ngl 999`) and still leave ~80 GiB of VRAM — which is why it ran alongside
+the Qwen3.6-35B-A3B model (see `args-qwen36-35b-a3b.md`). Combined the two
+models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was **KV
+cache**, not weights: dense attention KV is ~256 KiB/token (f16) / ~128
+KiB/token (q8_0), so large context is expensive.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/Qwen3.6-27B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
+| `--alias` | `qwen3.6-27b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
+| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
+| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
+| `-c` | `131072` | Total KV-cache context (131k). Single slot gets the full window. |
+| `-np` | `1` | 1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting). |
+| `--cont-batching` | *(flag)* | Continuous batching across slots. |
+| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model. |
+| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
+| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
+
+## VRAM budget (when live)
+
+| Component             | VRAM        |
+|-----------------------|-------------|
+| Weights (full offload) | ~16 GiB    |
+| KV cache (q8_0, 131k)  | ~16 GiB    |
+| **Subtotal**          | **~32 GiB** |
+
+Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
+co-locating the 35B-A3B model.
+
+## Notes for redeployment
+
+- This was the **deeper-reasoning** model (dense, all params active). The
+  `qwen3.6-35b-a3b` was the **fast/flash** variant. If redeploying only one,
+  decide based on latency-vs-quality needs.
+- LiteLLM entry that went with it:
+  ```yaml
+  - model_name: qwen3.6-27b
+    litellm_params:
+      model: openai/qwen3.6-27b
+      api_base: http://llamacpp-qwen36-27b.llamacpp/v1
+      api_key: "sk-no-auth"
+  ```
+- Consumers at removal time: `platform-engineer` (default model),
+  `home-manager` (used the 35B-A3B, not this one). See git history for exact
+  config.
+- The `fetch-model` initContainer downloaded the single GGUF idempotently into
+  the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
+  GGUF on first boot to reclaim space, so a redeploy will re-download it.

llamacpp/args-qwen36-35b-a3b.md

@@ -0,0 +1,96 @@
+# Deployment arguments — `qwen3.6-35b-a3b` (REMOVED)
+
+> **Status:** This model was **removed** from the cluster to make VRAM room for
+> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
+> 90 GiB VRAM pool on its own. This file documents the flags used when the
+> model was live, so it can be redeployed later if the DeepSeek model is taken
+> down or moved to different hardware.
+>
+> To redeploy: restore `deployment-qwen36-35b-a3b.yaml` (the manifest is
+> preserved in git history) and re-add the LiteLLM entry. Re-check the VRAM
+> budget — co-locating with the 87 GiB DeepSeek model is **not** possible on
+> the current 90 GiB pool.
+
+## Model & source
+
+- **Model:** `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` — Qwen3.6-35B-A3B, the "flash"
+  **Mixture-of-Experts** variant (35B total params, **only 3B active** per
+  token), MTP variant.
+- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~20 GiB**,
+  single GGUF file (`Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf`).
+- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
+  driver, supports Radeon 8060S / RDNA 3.5).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
+memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
+{kubernetes.io/arch: amd64, hardware: high-memory}`.
+
+## Why it was the "flash" model
+
+Despite having **more total parameters** than the dense 27B, only **3B are
+active per token** (MoE), so inference is significantly faster. The full ~20 GiB
+of Q4 weights is still loaded into VRAM, but only a small fraction is computed
+per token. Its KV cache is also tiny (~72 KiB/token), so large context is nearly
+free — hence the much larger `-c` and the 2-slot split.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
+| `--alias` | `qwen3.6-35b-a3b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
+| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
+| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
+| `-c` | `262144` | Total KV-cache context (262k), **split across parallel slots**. |
+| `-np` | `2` | 2 parallel slots ⇒ 131k tokens per concurrent request (262k / 2). MoE KV is cheap, so splitting is affordable. |
+| `--cont-batching` | *(flag)* | Continuous batching across the 2 slots. |
+| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB at 262k); frees headroom for the large `-c`. Drop to `q4_0` for even less VRAM if retrieval quality allows. |
+| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
+| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
+
+## VRAM budget (when live)
+
+| Component             | VRAM       |
+|-----------------------|------------|
+| Weights (full offload) | ~20 GiB   |
+| KV cache (q8_0, 262k)  | ~9 GiB    |
+| **Subtotal**          | **~29 GiB** |
+
+Left ~67 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
+co-locating the dense 27B model (combined ~61 GiB).
+
+## Combined VRAM budget (both Qwen models, when live)
+
+| Model             | Weights  | KV cache                                  | Subtotal |
+|-------------------|----------|-------------------------------------------|----------|
+| qwen3.6-27b       | ~16 GiB  | ~16 GiB  (q8_0, 131k total, 1 slot)      | ~32 GiB  |
+| qwen3.6-35b-a3b   | ~20 GiB  | ~9 GiB   (q8_0, 262k total, 131k/slot)   | ~29 GiB  |
+| **Total**         |          |                                           | **~61 GiB** |
+
+~35 GiB headroom on the 96 GiB pool — comfortable. (DeepSeek-V4-Flash-0731 at
+~87 GiB cannot coexist with either of these; that's why both were removed.)
+
+## Notes for redeployment
+
+- This was the **fast/flash** model (MoE, 3B active). The `qwen3.6-27b` was the
+  **deeper-reasoning** model (dense, all params active). If redeploying only
+  one, decide based on latency-vs-quality needs.
+- LiteLLM entry that went with it:
+  ```yaml
+  - model_name: qwen3.6-35b-a3b
+    litellm_params:
+      model: openai/qwen3.6-35b-a3b
+      api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
+      api_key: "sk-no-auth"
+  ```
+- Consumers at removal time: `home-manager` (default + auxiliary),
+  `platform-engineer` (auxiliary compression + title generation). See git
+  history for exact config.
+- The `fetch-model` initContainer downloaded the single GGUF idempotently into
+  the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
+  GGUF on first boot to reclaim space, so a redeploy will re-download it.

llamacpp/deployment-lfm25-26b.yaml

@@ -0,0 +1,141 @@
+# LiquidAI LFM2.5-2.6B served by llama.cpp (Vulkan, no speculative decoding).
+# The primary GGUF is downloaded into the shared model PVC before
+# llama-server starts.
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llamacpp-lfm25-26b
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    model: lfm2.5-2.6b
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: llamacpp
+      model: lfm2.5-2.6b
+  template:
+    metadata:
+      labels:
+        app: llamacpp
+        model: lfm2.5-2.6b
+    spec:
+      nodeSelector:
+        kubernetes.io/arch: amd64
+        hardware: high-memory
+      initContainers:
+      - name: fetch-models
+        image: alpine:3.20
+        command: ["/bin/sh", "-c"]
+        args:
+        - |
+          set -eu
+          apk add --no-cache curl
+          for entry in \
+            "https://huggingface.co/LiquidAI/LFM2.5-2.6B-GGUF/resolve/main/LFM2.5-2.6B-Q4_K_M.gguf|LFM2.5-2.6B-Q4_K_M.gguf"; do
+            url=${entry%%|*}
+            file=${entry##*|}
+            if [ -s "/models/$file" ]; then
+              echo "$file already present - skipping download."
+              continue
+            fi
+            echo "Downloading $file ..."
+            curl -fL --retry 5 --retry-delay 5 -C - \
+              -o "/models/$file.partial" "$url"
+            mv "/models/$file.partial" "/models/$file"
+            echo "Download complete: $(ls -lh "/models/$file")"
+          done
+        volumeMounts:
+        - name: models
+          mountPath: /models
+      containers:
+      - name: llama-server
+        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
+        imagePullPolicy: IfNotPresent
+        args:
+        - -m
+        - /models/LFM2.5-2.6B-Q4_K_M.gguf
+        # Draft model + speculative decoding removed: the spec path crashes the
+        # server (GGML_ASSERT slot.spec_i_batch...) when the KV cache is under
+        # pressure, taking all in-flight requests (incl. home-manager crons) down.
+        - --ctx-size
+        - "80000"
+        - --parallel
+        - "2"
+        - --temp
+        - "0.1"
+        - --cache-type-k
+        - q8_0
+        - --cache-type-v
+        - q8_0
+        - --fit
+        - "off"
+        - --alias
+        - lfm2.5-2.6b
+        - --host
+        - 0.0.0.0
+        - --port
+        - "8080"
+        - --jinja
+        ports:
+        - name: http
+          containerPort: 8080
+        resources:
+          requests:
+            cpu: "500m"
+            memory: 2Gi
+          limits:
+            cpu: "4000m"
+            memory: 12Gi
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        livenessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 180
+          periodSeconds: 30
+          failureThreshold: 5
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: models
+          mountPath: /models
+          readOnly: true
+        - name: dri
+          mountPath: /dev/dri
+      volumes:
+      - name: models
+        persistentVolumeClaim:
+          claimName: llamacpp-models
+      - name: dri
+        hostPath:
+          path: /dev/dri
+          type: Directory
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llamacpp-lfm25-26b
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    model: lfm2.5-2.6b
+spec:
+  type: ClusterIP
+  selector:
+    app: llamacpp
+    model: lfm2.5-2.6b
+  ports:
+  - name: http
+    port: 80
+    targetPort: 8080

llamacpp/deployment-qwen36-27b.yaml

@@ -1,163 +0,0 @@
-# Qwen3.6-27B (dense, Q4_K_XL) served by llama.cpp's llama-server on the NUCBox APU.
-#
-# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
-# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). The Q4 model
-# (~16 GiB) is fully offloaded to the GPU via the Vulkan backend.
-#
-# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan
-# driver, which supports the Radeon 8060S (RDNA 3.5). The project moved from the
-# legacy `ggerganov/llama.cpp` namespace (which only has light/full tags) to
-# `ggml-org/llama.cpp` (server-vulkan + pinned build tags like server-vulkan-bXXXX).
-# Pin to a build tag (e.g. server-vulkan-b4738) for production reproducibility.
-#
-# GPU access: the container mounts /dev/dri (the DRM render nodes) and runs
-# privileged. This is the simplest reliable option on k3s without a Vulkan
-# device plugin; tighten later with supplementalGroups if desired.
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: llamacpp-qwen36-27b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-27b
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate   # never run two pods loading the same model into VRAM
-  selector:
-    matchLabels:
-      app: llamacpp
-      model: qwen3.6-27b
-  template:
-    metadata:
-      labels:
-        app: llamacpp
-        model: qwen3.6-27b
-    spec:
-      nodeSelector:
-        kubernetes.io/arch: amd64
-        hardware: high-memory
-      initContainers:
-      # Idempotently download the GGUF into the shared models PVC on first boot.
-      # Exits immediately if the file is already present (pod restart / recreate).
-      - name: fetch-model
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        args:
-        - |
-          set -e
-          if [ -f "/models/$MODEL_FILE" ]; then
-            echo "Model $MODEL_FILE already present — skipping download."
-            exit 0
-          fi
-          echo "Installing curl..."
-          apk add --no-cache curl
-          echo "Downloading $MODEL_FILE from $MODEL_URL ..."
-          curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL"
-          echo "Download complete: $(ls -lh /models/$MODEL_FILE)"
-        env:
-        - name: MODEL_URL
-          value: "https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-UD-Q4_K_XL.gguf"
-        - name: MODEL_FILE
-          value: "Qwen3.6-27B-UD-Q4_K_XL.gguf"
-        volumeMounts:
-        - name: models
-          mountPath: /models
-      containers:
-      - name: llama-server
-        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
-        imagePullPolicy: IfNotPresent
-        # llama.cpp's CLI parser does NOT split on '=' — every value flag must be a
-        # separate argv element (flag, then value). See common/arg.cpp in the repo.
-        args:
-        - -m                       # model file
-        - /models/Qwen3.6-27B-UD-Q4_K_XL.gguf
-        - --alias                 # /v1/models reports this name; matches the litellm alias
-        - qwen3.6-27b
-        - --host
-        - 0.0.0.0
-        - --port
-        - "8080"
-        - --jinja                  # use the GGUF's chat template (Qwen3 thinking format)
-        - -ngl                     # offload ALL layers to the GPU (fits in 96 GiB VRAM)
-        - "999"
-        - -c                       # total KV-cache context (single slot gets the full window)
-        - "131072"
-        - -np                      # 1 slot => 131072 tokens for the single request (no split)
-        - "1"
-        - --cont-batching          # continuous batching across slots
-        - --cache-type-k            # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB
-        - q8_0                     # at 131k ctx); ~negligible quality loss, frees headroom for large -c
-        - --cache-type-v
-        - q8_0
-        - --threads                # CPU threads for sampling/overhead (GPU does the heavy lifting)
-        - "8"
-        ports:
-        - name: http
-          containerPort: 8080
-        resources:
-          # The model weights + KV cache live in GPU VRAM (96 GiB pool) and are
-          # NOT counted against the cgroup memory limit. This limit only covers
-          # CPU-side overhead + the mmap'd GGUF file pages during load (~16 GiB,
-          # reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM,
-          # so the request is kept low to stay schedulable alongside other pods.
-          # If the pod OOM-kills during load, the amdgpu driver may be counting
-          # some VRAM against the cgroup — raise the limit.
-          requests:
-            cpu: "1000m"
-            memory: "2Gi"
-          limits:
-            cpu: "4000m"
-            memory: "24Gi"
-        readinessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 30
-          periodSeconds: 10
-          failureThreshold: 6
-        livenessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 180   # model load + Vulkan init can take a few minutes
-          periodSeconds: 30
-          failureThreshold: 5
-        securityContext:
-          # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the
-          # simplest reliable path on k3s without a device plugin.
-          privileged: true
-        volumeMounts:
-        - name: models
-          mountPath: /models
-          readOnly: true
-        - name: dri
-          mountPath: /dev/dri
-      volumes:
-      - name: models
-        persistentVolumeClaim:
-          claimName: llamacpp-models
-      - name: dri
-        hostPath:
-          path: /dev/dri
-          type: Directory
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: llamacpp-qwen36-27b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-27b
-spec:
-  type: ClusterIP
-  selector:
-    app: llamacpp
-    model: qwen3.6-27b
-  ports:
-  - name: http
-    port: 80
-    targetPort: 8080

llamacpp/deployment-qwen36-35b-a3b.yaml

@@ -1,162 +0,0 @@
-# Qwen3.6-35B-A3B (MoE: 35B total / 3B active, Q4_K_XL) — the "flash" variant.
-#
-# Despite having more total parameters than the 27B dense model, only 3B are
-# active per token (Mixture-of-Experts), so inference is much faster. The full
-# ~20 GiB of Q4 weights is still loaded into VRAM but only a small fraction is
-# computed per token.
-#
-# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
-# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). Full GPU
-# offload via the Vulkan backend. Shares the 96 GiB VRAM pool with the 27B
-# model — see llamacpp/README.md for the combined VRAM budget.
-#
-# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (Mesa/RADV Vulkan driver,
-# supports the Radeon 8060S / RDNA 3.5). Pin to a build tag for production.
-#
-# GPU access: mounts /dev/dri + privileged (simplest reliable path on k3s).
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: llamacpp-qwen36-35b-a3b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-35b-a3b
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate   # never run two pods loading the same model into VRAM
-  selector:
-    matchLabels:
-      app: llamacpp
-      model: qwen3.6-35b-a3b
-  template:
-    metadata:
-      labels:
-        app: llamacpp
-        model: qwen3.6-35b-a3b
-    spec:
-      nodeSelector:
-        kubernetes.io/arch: amd64
-        hardware: high-memory
-      initContainers:
-      # Idempotently download the GGUF into the shared models PVC on first boot.
-      # Exits immediately if the file is already present (pod restart / recreate).
-      - name: fetch-model
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        args:
-        - |
-          set -e
-          if [ -f "/models/$MODEL_FILE" ]; then
-            echo "Model $MODEL_FILE already present — skipping download."
-            exit 0
-          fi
-          echo "Installing curl..."
-          apk add --no-cache curl
-          echo "Downloading $MODEL_FILE from $MODEL_URL ..."
-          curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL"
-          echo "Download complete: $(ls -lh /models/$MODEL_FILE)"
-        env:
-        - name: MODEL_URL
-          value: "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
-        - name: MODEL_FILE
-          value: "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
-        volumeMounts:
-        - name: models
-          mountPath: /models
-      containers:
-      - name: llama-server
-        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
-        imagePullPolicy: IfNotPresent
-        # llama.cpp's CLI parser does NOT split on '=' — every value flag must be a
-        # separate argv element (flag, then value). See common/arg.cpp in the repo.
-        args:
-        - -m                       # model file
-        - /models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
-        - --alias                 # /v1/models reports this name; matches the litellm alias
-        - qwen3.6-35b-a3b
-        - --host
-        - 0.0.0.0
-        - --port
-        - "8080"
-        - --jinja                  # use the GGUF's chat template (Qwen3 thinking format)
-        - -ngl                     # offload ALL layers to the GPU (fits in 96 GiB VRAM)
-        - "999"
-        - -c                       # total KV-cache context, split across parallel slots
-        - "262144"
-        - -np                      # 2 parallel slots => 131072 tokens per concurrent request
-        - "2"
-        - --cont-batching          # continuous batching across slots
-        - --cache-type-k            # quantize KV cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB
-        - q8_0                     # at 262k ctx); ~negligible quality loss, frees headroom for large -c
-        - --cache-type-v            # (raise to q4_0 for even less VRAM if retrieval quality allows)
-        - q8_0
-        - --threads                # CPU threads for sampling/overhead (GPU does the heavy lifting)
-        - "8"
-        ports:
-        - name: http
-          containerPort: 8080
-        resources:
-          # The model weights + KV cache live in GPU VRAM (96 GiB pool) and are
-          # NOT counted against the cgroup memory limit. This limit only covers
-          # CPU-side overhead + the mmap'd GGUF file pages during load (~20 GiB,
-          # reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM.
-          # If the pod OOM-kills during load, raise the limit.
-          requests:
-            cpu: "1000m"
-            memory: "2Gi"
-          limits:
-            cpu: "4000m"
-            memory: "24Gi"
-        readinessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 30
-          periodSeconds: 10
-          failureThreshold: 6
-        livenessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 180   # model load + Vulkan init can take a few minutes
-          periodSeconds: 30
-          failureThreshold: 5
-        securityContext:
-          # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the
-          # simplest reliable path on k3s without a device plugin.
-          privileged: true
-        volumeMounts:
-        - name: models
-          mountPath: /models
-          readOnly: true
-        - name: dri
-          mountPath: /dev/dri
-      volumes:
-      - name: models
-        persistentVolumeClaim:
-          claimName: llamacpp-models
-      - name: dri
-        hostPath:
-          path: /dev/dri
-          type: Directory
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: llamacpp-qwen36-35b-a3b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-35b-a3b
-spec:
-  type: ClusterIP
-  selector:
-    app: llamacpp
-    model: qwen3.6-35b-a3b
-  ports:
-  - name: http
-    port: 80
-    targetPort: 8080

llamacpp/deployment-qwen38-27b.yaml

@@ -0,0 +1,144 @@
+# Qwen3.8-27B with MTP speculative decoding served by llama.cpp.
+# The primary and draft GGUFs are downloaded by an initContainer into the
+# shared model PVC before llama-server starts.
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llamacpp-qwen38-27b
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    model: qwen3.8-27b
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: llamacpp
+      model: qwen3.8-27b
+  template:
+    metadata:
+      labels:
+        app: llamacpp
+        model: qwen3.8-27b
+    spec:
+      nodeSelector:
+        kubernetes.io/arch: amd64
+        hardware: high-memory
+      initContainers:
+      - name: fetch-models
+        image: alpine:3.20
+        command: ["/bin/sh", "-c"]
+        args:
+        - |
+          set -eu
+          apk add --no-cache curl
+          for entry in \
+            "https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF/resolve/main/Qwen3.8-27B-Q4_K_M.gguf|Qwen3.8-27B-Q4_K_M.gguf" \
+            "https://huggingface.co/ggml-org/Qwen3.8-27B-GGUF/resolve/main/mtp-Qwen3.8-27B-Q4_0.gguf|mtp-Qwen3.8-27B-Q4_0.gguf"; do
+            url=${entry%%|*}
+            file=${entry##*|}
+            if [ -s "/models/$file" ]; then
+              echo "$file already present - skipping download."
+              continue
+            fi
+            echo "Downloading $file ..."
+            curl -fL --retry 5 --retry-delay 5 -C - \
+              -o "/models/$file.partial" "$url"
+            mv "/models/$file.partial" "/models/$file"
+            echo "Download complete: $(ls -lh "/models/$file")"
+          done
+        volumeMounts:
+        - name: models
+          mountPath: /models
+      containers:
+      - name: llama-server
+        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
+        imagePullPolicy: IfNotPresent
+        args:
+        - -m
+        - /models/Qwen3.8-27B-Q4_K_M.gguf
+        - --model-draft
+        - /models/mtp-Qwen3.8-27B-Q4_0.gguf
+        - --spec-default
+        - --spec-type
+        - draft-mtp
+        - --ctx-size
+        - "196608"
+        - --cache-type-k
+        - q8_0
+        - --cache-type-v
+        - q8_0
+        - --reasoning-preserve
+        - --fit
+        - "off"
+        - --agent
+        - --chat-template-kwargs
+        - '{"reasoning_effort":"medium"}'
+        - --alias
+        - qwen3.8-27b
+        - --host
+        - 0.0.0.0
+        - --port
+        - "8080"
+        - --jinja
+        ports:
+        - name: http
+          containerPort: 8080
+        resources:
+          requests:
+            cpu: "1000m"
+            memory: 2Gi
+          limits:
+            cpu: "4000m"
+            memory: 24Gi
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        livenessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 600
+          periodSeconds: 30
+          failureThreshold: 5
+        securityContext:
+          privileged: true
+        volumeMounts:
+        - name: models
+          mountPath: /models
+          readOnly: true
+        - name: dri
+          mountPath: /dev/dri
+      volumes:
+      - name: models
+        persistentVolumeClaim:
+          claimName: llamacpp-models
+      - name: dri
+        hostPath:
+          path: /dev/dri
+          type: Directory
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llamacpp-qwen38-27b
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    model: qwen3.8-27b
+spec:
+  type: ClusterIP
+  selector:
+    app: llamacpp
+    model: qwen3.8-27b
+  ports:
+  - name: http
+    port: 80
+    targetPort: 8080

llamacpp/deployment-stt.yaml

@@ -0,0 +1,100 @@
+# Wyoming Speech-to-Text (rhasspy/wyoming-whisper).
+#
+# Exposes the Wyoming protocol over TCP on port 10300 so Home Assistant's
+# Wyoming STT integration can stream audio and receive transcriptions. Uses
+# faster-whisper under the hood with the language pinned to Catalan (`ca`).
+#
+# These are lightweight CPU-only pods (no GPU), so they are NOT pinned to the
+# NUCBox via nodeSelector — k3s can schedule them on any amd64 node. Models are
+# cached in a local-path PVC (`/data`) so they survive pod restarts without
+# re-downloading.
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: wyoming-whisper-data
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: stt
+spec:
+  storageClassName: local-path
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 5Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: wyoming-whisper
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: stt
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: llamacpp
+      component: stt
+  template:
+    metadata:
+      labels:
+        app: llamacpp
+        component: stt
+    spec:
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      containers:
+      - name: wyoming-whisper
+        image: rhasspy/wyoming-whisper
+        imagePullPolicy: IfNotPresent
+        args:
+        - --uri
+        - tcp://0.0.0.0:10300
+        - --language      # pin transcription language to Catalan
+        - ca
+        - --model         # faster-whisper model (tune up to medium for accuracy)
+        - small
+        ports:
+        - name: wyoming
+          containerPort: 10300
+        resources:
+          # faster-whisper `small` is ~244M params (fp16 ~500 MiB weights)
+          # plus CTranslate2 runtime / workspace buffers during decode.
+          # 1 GiB OOM-kills during model load; 2 GiB gives headroom.
+          requests:
+            cpu: "100m"
+            memory: "512Mi"
+          limits:
+            cpu: "1000m"
+            memory: "2Gi"
+        volumeMounts:
+        - name: data
+          mountPath: /data
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: wyoming-whisper-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: wyoming-whisper
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: stt
+spec:
+  type: ClusterIP
+  selector:
+    app: llamacpp
+    component: stt
+  ports:
+  - name: wyoming
+    port: 10300
+    targetPort: 10300

llamacpp/deployment-tts.yaml

@@ -0,0 +1,93 @@
+# Wyoming Text-to-Speech (rhasspy/wyoming-piper).
+#
+# Exposes the Wyoming protocol over TCP on port 10200 so Home Assistant's
+# Wyoming TTS integration can send text and receive synthesized audio. Uses
+# Piper under the hood with the Catalan voice `ca_ES-upc_ona-medium`.
+#
+# Lightweight CPU-only pod (no GPU), not pinned to the NUCBox. The voice is
+# cached in a local-path PVC (`/data`) so it survives pod restarts.
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: wyoming-piper-data
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: tts
+spec:
+  storageClassName: local-path
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: wyoming-piper
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: tts
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: llamacpp
+      component: tts
+  template:
+    metadata:
+      labels:
+        app: llamacpp
+        component: tts
+    spec:
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      containers:
+      - name: wyoming-piper
+        image: rhasspy/wyoming-piper
+        imagePullPolicy: IfNotPresent
+        args:
+        - --uri
+        - tcp://0.0.0.0:10200
+        - --voice         # Catalan (Spain) medium-quality UPC Ona voice
+        - ca_ES-upc_ona-medium
+        ports:
+        - name: wyoming
+          containerPort: 10200
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "128Mi"
+          limits:
+            cpu: "500m"
+            memory: "512Mi"
+        volumeMounts:
+        - name: data
+          mountPath: /data
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: wyoming-piper-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: wyoming-piper
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: tts
+spec:
+  type: ClusterIP
+  selector:
+    app: llamacpp
+    component: tts
+  ports:
+  - name: wyoming
+    port: 10200
+    targetPort: 10200

llamacpp/deployment-wakeword.yaml

@@ -0,0 +1,91 @@
+# Wyoming Wake Word (rhasspy/wyoming-openwakeword).
+#
+# Exposes the Wyoming protocol over TCP on port 10400 so Home Assistant's
+# Wyoming wake word integration can stream microphone audio and receive wake
+# word detections. Uses openWakeWord under the hood with ONNX models.
+#
+# Lightweight CPU-only pod (no GPU), not pinned to the NUCBox. Models are
+# cached in a local-path PVC (`/data`) so they don't re-download on restart.
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: wyoming-openwakeword-data
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: wakeword
+spec:
+  storageClassName: local-path
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: wyoming-openwakeword
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: wakeword
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: llamacpp
+      component: wakeword
+  template:
+    metadata:
+      labels:
+        app: llamacpp
+        component: wakeword
+    spec:
+      nodeSelector:
+        kubernetes.io/arch: amd64
+      containers:
+      - name: wyoming-openwakeword
+        image: rhasspy/wyoming-openwakeword
+        imagePullPolicy: IfNotPresent
+        args:
+        - --uri
+        - tcp://0.0.0.0:10400
+        ports:
+        - name: wyoming
+          containerPort: 10400
+        resources:
+          requests:
+            cpu: "100m"
+            memory: "128Mi"
+          limits:
+            cpu: "500m"
+            memory: "512Mi"
+        volumeMounts:
+        - name: data
+          mountPath: /data
+      volumes:
+      - name: data
+        persistentVolumeClaim:
+          claimName: wyoming-openwakeword-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: wyoming-openwakeword
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    component: wakeword
+spec:
+  type: ClusterIP
+  selector:
+    app: llamacpp
+    component: wakeword
+  ports:
+  - name: wyoming
+    port: 10400
+    targetPort: 10400

llamacpp/pv.yaml

@@ -7,6 +7,11 @@
 # idempotently on first boot instead.
 #
 # nodeAffinity keeps the PV bound to the NUCBox even if labels change later.
+#
+# IMPORTANT: capacity is only metadata for a hostPath volume — k8s does NOT
+# enforce it and bumping it does NOT add physical disk space. The active Qwen
+# primary and draft GGUFs are downloaded into this directory, so the hostPath
+# filesystem must have enough free space for both models.
 apiVersion: v1
 kind: PersistentVolume
 metadata:
@@ -40,6 +45,12 @@ metadata:
     app: llamacpp
 spec:
   storageClassName: manual
+  volumeName: llamacpp-models   # pin to the static hostPath PV by name — forces
+                              # static binding and disables dynamic provisioning.
+                              # The "manual" StorageClass is never created as an
+                              # object; without volumeName a stuck PVC falls
+                              # through to the (nonexistent) provisioner and errors
+                              # with: storageclass.storage.k8s.io "manual" not found
   accessModes:
     - ReadWriteMany
   resources:

platform-engineer/README.md

@@ -208,21 +208,21 @@ HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=<strong password>
 
 ---
 
-## 7. Cron jobs to seed (Hermes-native)
+## 7. Single daily cron job (Hermes-native)
 
-These are written by an init script (one-shot Job `hermes-cron-seed`) that runs
+The one-shot Job `hermes-cron-seed` creates one daily check at 06:00 local cluster time (`0 4 * * *` UTC; Hermes has no timezone configured). It removes the known legacy jobs first, then runs
-`hermes cron create ...` against the gateway on first install, and is idempotent
+`hermes cron create ...` against the gateway and is idempotent
-(it checks existing job names). All deliver to Discord. Examples:
+(it removes old names and recreates the daily job). It delivers to Discord and combines node, resource, PVC, pod, ArgoCD, and certificate checks. It stays silent when healthy and sends one concise Catalan report when there is an issue.
 
 | Name | Schedule | Prompt (abbreviated) |
 |------|----------|------------------------|
-| `cluster-health-check` | `every 15m` | Run `kubectl get nodes,pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded` and `kubectl get events -A --field-selector type=Warning --since=20m`. If everything healthy, reply with only `[SILENT]`. Otherwise summarize failures and root-cause briefly. |
+| `daily-cluster-check` | `0 4 * * *` UTC (06:00 local) | Combined daily platform health check; `[SILENT]` when healthy. |
-| `pod-restart-loop` | `every 10m` | Find pods in `CrashLoopBackOff`/`ImagePullBackOff` across all namespaces. For `CrashLoopBackOff`, fetch logs and if a clear transient cause (OOM, config parse, missing secret) is visible, attempt `kubectl rollout restart <deploy>`; otherwise notify me with the log excerpt. Reply `[SILENT]` if none found. |
+
-| `pvc-pressure` | `every 30m` | `kubectl get pv` + node disk via `kubectl top nodes`. Alert if any PVC `Bound` to a near-full volume or node disk >85%. `[SILENT]` otherwise. |
+
-| `argocd-sync-health` | `every 1h` | `kubectl get applications -n argocd -o wide` (or `argocd app sync --dry-run` if CLI present). Report any `OutOfSync`/`Degraded` app. `[SILENT]` if all `Synced`+`Healthy`. |
+
-| `cert-expiry` | `every 1d at 09:00` | List cert-manager `Certificate` resources with expiry < 21 days. Notify only if any. `[SILENT]` otherwise. |
+
-| `node-resource-drift` | `every 30m` | `kubectl top nodes`. Alert if any node CPU>90% or mem>90% sustained, or any node `NotReady`. `[SILENT]` otherwise. |
+
-| `daily-cluster-report` | `0 8 * * *` | Summarize: node count/status, top 5 pods by CPU/mem, # pods not Running, # ArgoCD apps OutOfSync, cert warnings. Always deliver (no `[SILENT]`). |
+
 
 Design rules baked into SOUL.md:
 - **Read-only checks** run frequently (10–30m) and stay silent unless wrong.
@@ -274,7 +274,7 @@ Design rules baked into SOUL.md:
 4. Apply the Deployment; wait for the pod to go Running.
 5. `kubectl exec` in and run the one-shot cron seed:
    `hermes cron create ...` (or apply the `cron-seed` Job).
-6. Trigger the first `cluster-health-check` manually: `hermes cron run cluster-health-check`.
+6. Trigger the first `daily-cluster-check` manually: `hermes cron run daily-cluster-check`.
 7. Add the app to `argocd/gen-apps.sh`, regenerate, commit, push.
 
 ---
@@ -299,11 +299,12 @@ Design rules baked into SOUL.md:
 
 ## 11. Deployment checklist (do in this order)
 
-1. **Add the Ollama model to LiteLLM** (already done in `litellm/litellm.yaml`):
+1. **Add the in-cluster model to LiteLLM** (already done in `litellm/litellm.yaml`):
-   the `qwen-3.6:27b` entry points at `http://10.88.20.12:11434`. Make sure
+   the `deepseek-v4-flash-0731` entry points at the in-cluster llama.cpp server
-   `qwen3.6:27b` is actually pulled on that Ollama host
+   `http://llamacpp-deepseek-v4-flash-0731.llamacpp/v1` (served from the NUCBox
-   (`ollama pull qwen3.6:27b`). Apply: `kubectl apply -f litellm/` and restart
+   APU; see `llamacpp/deployment-deepseek-v4-flash-0731.yaml`). Apply:
-   the LiteLLM pod so the new config takes effect.
+   `kubectl apply -f litellm/` and restart the LiteLLM pod so the new config
+   takes effect.
 2. **Create the `gitea-registry` secret in the new namespace** (ArgoCD won't
    create it — it's not in the repo):
    ```
@@ -323,12 +324,12 @@ Design rules baked into SOUL.md:
    password + a 32-byte session secret.
 5. **Commit & push** the whole change. ArgoCD will create the namespace
    resources, deploy the pod, and bring up the ingress at `platform-engineer.rogi.casa`.
-6. **Seed the cron jobs:**
+6. **Seed the cron job:**
    `kubectl apply -f platform-engineer/cron-seed.yaml` (one-shot Job) — it waits
-   for the hermes pod, then runs `hermes cron create ...` for each watchdog.
+   for the Hermes pod, removes the legacy schedules, and creates the single
-   Re-run it any time you want to re-seed after a wipe.
+   daily check. Re-run it to migrate an existing PVC or after a wipe.
-7. **Smoke test:** trigger the first health check manually —
+7. **Smoke test:** trigger the daily health check manually —
-   `kubectl exec -n platform-engineer deploy/hermes -- hermes cron run cluster-health-check` —
+   `kubectl exec -n platform-engineer deploy/hermes -- hermes cron run daily-cluster-check` —
    and confirm the message lands in Discord.
 8. **ArgoCD:** the `Application` (`argocd/apps/platform-engineer.yaml`) is
    already generated. After commit, Argo will reconcile it like every other app.
@@ -338,10 +339,11 @@ Design rules baked into SOUL.md:
 - **ArgoCD owns** (in git): namespace, RBAC, Secret, ConfigMap (seed), PVC,
   Deployment, Service, Ingress, cron-seed Job.
 - **Runtime state (on the PVC, NOT reconciled):** `config.yaml`, `SOUL.md`,
-  `.env`, `cron/jobs.json`, `sessions/`, `memories/`, `skills/`. The ConfigMap
+  `.env`, `cron/jobs.json`, `sessions/`, `memories/`, `skills/`. The cron seed
-  only *seeds* these on first boot; after that, edits you make via the
+  migrates the existing `cron/jobs.json` to the single daily check, but the
-  dashboard or `hermes cron edit` persist on the PVC and Argo will not revert
+  file remains runtime state afterward. The ConfigMap only *seeds* these on
-  them. If you ever want a hard reset, delete the PVC and re-apply.
+  first boot; after that, edits you make via the dashboard or `hermes cron edit`
+  persist on the PVC and Argo will not revert them. If you ever want a hard reset, delete the PVC and re-apply.
 
 ---
 

platform-engineer/configmap.yaml

@@ -9,18 +9,18 @@ data:
   config.yaml: |
     model:
       provider: openai-api
-      default: qwen3.6-27b
+      default: qwen3.8-27b
       base_url: "http://litellm-service.litellm:80/v1"
       api_mode: chat_completions
 
     auxiliary:
       compression:
         provider: openai-api
-        model: qwen3.6-35b-a3b    # flash model for speed-sensitive auxiliary tasks
+        model: qwen3.8-27b    # in-cluster llama.cpp model
         base_url: "http://litellm-service.litellm:80/v1"
       title_generation:
         provider: openai-api
-        model: qwen3.6-35b-a3b    # flash model for speed-sensitive auxiliary tasks
+        model: qwen3.8-27b    # in-cluster llama.cpp model
         base_url: "http://litellm-service.litellm:80/v1"
 
     terminal:
@@ -85,6 +85,14 @@ data:
     to keep it healthy, fix small problems before they grow, and notify your
     owner (Roger) on Discord when something needs a human.
 
+    ## Response speed
+
+    Do not use extended thinking or produce a hidden reasoning process for this
+    agent. Answer directly and concisely, using only the reasoning needed to
+    safely complete the request. Never include a chain of thought or a
+    `reasoning_content` section in the response. This is a prompt-level
+    instruction; do not change the llama.cpp server configuration.
+
     ## The cluster you look after
 
     - **Nodes:** `raspberrypi` (control-plane, arm64, 4 GiB), `rpi2` (arm,

platform-engineer/cron-seed.yaml

@@ -1,10 +1,10 @@
-# One-shot Job that seeds Hermes' built-in cron schedule on first install.
+# One-shot Job that replaces Hermes' built-in cron schedule with one daily check.
-# Idempotent: skips job names that already exist.
+# The seed is intentionally destructive for the known legacy jobs so existing
+# PVC state is simplified on the next ArgoCD sync.
 #
-# Cron prompts are deliberately written as plain-English instructions (no inline
+# The prompt is written as plain-English instructions to avoid triggering
-# curl commands) to avoid tripping Hermes' threat-pattern scanner, which blocks
+# Hermes' threat-pattern scanner. API endpoints and query examples are in
-# cron prompts containing curl+auth-header patterns. The exact API endpoints and
+# SOUL.md.
-# query examples are documented in the agent's SOUL.md instead.
 ---
 apiVersion: batch/v1
 kind: Job
@@ -48,72 +48,25 @@ spec:
 
           echo "Waiting for hermes pod to be Ready..."
           kubectl -n platform-engineer wait --for=condition=Ready pod -l app=hermes --timeout=300s || true
-
           POD=$(kubectl -n platform-engineer get pod -l app=hermes -o jsonpath='{.items[0].metadata.name}')
           echo "Using pod: $POD"
 
-          exists() { kubectl -n platform-engineer exec "$POD" -- hermes cron list 2>/dev/null | grep -qi " $1 "; }
+          # Cron state lives on the PVC, so remove the old schedules as well
+          # as creating the new one. This makes the migration idempotent.
+          for name in \
+            cluster-health-check node-health-check pod-phase-check pod-error-log-check \
+            pod-restart-loop pvc-pressure argocd-sync-health cert-expiry \
+            node-resource-drift daily-cluster-report daily-cluster-check; do
+            echo "removing legacy cron '$name' (if present)"
+            kubectl -n platform-engineer exec "$POD" -- hermes cron delete "$name" 2>/dev/null || true
+          done
 
-          # NOTE: `hermes cron create` has no --model flag. New jobs inherit the
+          echo "creating daily cluster check"
-          # global default model from config.yaml at creation time. To pin a
+          # Hermes runs cron expressions in UTC. 04:00 UTC is 06:00 local time
-          # specific model per-job (e.g. the flash model qwen3.6-35b-a3b for all
+          # for this cluster (UTC+2).
-          # cron tasks), edit /opt/data/cron/jobs.json directly after seeding:
+          kubectl -n platform-engineer exec "$POD" -- hermes cron create "0 4 * * *" \
-          #   kubectl exec deploy/hermes -- python3 -c "...set model field..."
+            "Run the single daily platform health check for Roger using the HTTP APIs documented in your SOUL.md. Check: (1) every node is Ready and node CPU, memory, and root filesystem are below 90 percent; (2) PVCs have at least 15 percent free space; (3) pods not Running, excluding Completed/Succeeded jobs; (4) pod restart increases over the last 15 minutes and recent crash, error, panic, OOM, or backoff activity, using LogQL aggregation first and at most 10 sample lines for only an affected pod; (5) ArgoCD applications that are OutOfSync or Degraded; and (6) certificates expiring within 30 days. If everything is healthy, reply with exactly [SILENT]. Otherwise send one concise report in Catalan, grouped by issue, including namespace/resource names and relevant values. Keep it under 1800 characters. You may perform only clearly safe, idempotent GitOps remediations described in SOUL.md; otherwise report the proposed fix and wait for Roger." \
-          # See llamacpp/README.md and the deployment notes for details.
+            --name "daily-cluster-check" --deliver "discord"
-          create() {
-            name="$1"; schedule="$2"; deliver="$3"; prompt="$4"
-            if exists "$name"; then
-              echo "cron job '$name' already exists — skipping"
-            else
-              echo "creating cron job '$name' ..."
-              kubectl -n platform-engineer exec "$POD" -- hermes cron create "$schedule" "$prompt" --name "$name" --deliver "$deliver"
-            fi
-          }
-
-          # ---- Watchdog checks (silent unless something is wrong) ----
-          # NOTE: the old single "cluster-health-check" combined 4 heavy HTTP
-          # gathers (Prometheus + a cluster-wide raw Loki log dump) into one
-          # cron session and exceeded the flash model's per-request context.
-          # The 35b-a3b llama-server runs -c 131072 -np 4, so each request only
-          # gets ~32k tokens of KV cache (NOT 132k). It has been split into
-          # focused sub-checks so each run stays small, and the log check now
-          # uses LogQL aggregation instead of dumping raw lines cluster-wide.
-          create "node-health-check" "every 6h" "discord" \
-            "Check node health using the Prometheus API documented in your SOUL.md. Query kube_node_status_condition{condition=\"Ready\",status!=\"true\"}. If all nodes are Ready, reply with exactly [SILENT]. Otherwise list each NotReady node by name. Keep it short; do not paste raw JSON — summarize with jq."
-
-          create "pod-phase-check" "every 6h" "discord" \
-            "Check pod health using the Prometheus API documented in your SOUL.md. Query kube_pod_status_phase{phase!=\"Running\"} and exclude Completed/Succeeded pods (jobs). If all Running, reply with exactly [SILENT]. Otherwise list each non-Running pod as namespace/pod:phase, grouped by namespace. Summarize with jq; do not dump raw JSON."
-
-          create "pod-error-log-check" "every 6h" "discord" \
-            "Check for recent crash-loop/error logs using the Loki API documented in your SOUL.md. IMPORTANT context discipline: do NOT fetch raw log lines across all namespaces — that overflows your context window. First run a LogQL aggregation that returns only counts: sum by (namespace, pod) (count_over_time({namespace=~\".+\"} |~ \"(?i)backoff|crashloop|panic\" [20m])). Parse the counts with jq. If every count is zero, reply with exactly [SILENT]. If any namespace/pod has a non-zero count, fetch at most 10 sample lines for THAT pod only (limit=10) to identify the cause. Summarize concisely; never paste more than a handful of lines."
-
-          # (ArgoCD health is already covered by the argocd-sync-health cron below.)
-
-          create "pod-restart-loop" "every 1h" "discord" \
-            "Find pods with high restart rates using the Prometheus API documented in your SOUL.md. Run increase(kube_pod_container_status_restarts_total[15m]) and list only pods whose value is greater than 3. If none, reply [SILENT]. For any pod that qualifies, IMPORTANT context discipline: do NOT fetch raw logs across the cluster or over a wide time window — that overflows your context window (a raw cluster-wide Loki dump can exceed 60k tokens). Fetch at most 10 sample lines for THAT specific pod only, using Loki query_range with limit=10 over the last 15 minutes, filtered to error/crash patterns (e.g. {namespace=\"<ns>\",pod=\"<pod>\"} |~ \"(?i)error|panic|crash|oom|backoff\"). Summarize the root cause in one line; never paste more than a handful of log lines. If the cause is clearly fixable via a manifest change such as bumping a memory limit, fixing a config value, or bumping the restartedAt annotation, make the edit in /workspace/k3s-cluster, commit and push, then trigger an ArgoCD sync via the API. Report what you did in one line. If not clearly fixable, post the log excerpt and proposed fix, and wait for Roger."
-
-          create "pvc-pressure" "every 1d" "discord" \
-            "Check storage health using the Prometheus API documented in your SOUL.md. Alert if any PVC has less than 15 percent free space, or if any node filesystem is over 85 percent full. If all healthy, reply [SILENT]."
-
-          create "argocd-sync-health" "every 6h" "discord" \
-            "Check ArgoCD app health using the API documented in your SOUL.md. If every app is Synced and Healthy, reply [SILENT]. Otherwise list the OutOfSync or Degraded apps with their status. If an app is OutOfSync and you believe a recent git push caused it, you may trigger a sync via the API. Do NOT hand-edit resources to fix them — fix the source repo."
-
-          create "cert-expiry" "0 7 * * *" "discord" \
-            "Check certificate expiry using the Prometheus API documented in your SOUL.md. Alert on any certificate expiring in under 21 days, with its name and namespace. If none, reply [SILENT]."
-
-          create "node-resource-drift" "every 1d" "discord" \
-            "Check node resources using the Prometheus API documented in your SOUL.md. Alert if any node is NotReady, or if any node has CPU over 90 percent or memory over 90 percent. Otherwise reply [SILENT]."
-
-          # ---- Daily report (always delivered) ----
-          # NOTE: Hermes' cron runs in UTC (no TZ set on the container). The
-          # cluster is at UTC+2, so fixed daily times are expressed in UTC and
-          # shifted back 2h from the intended local wall-clock time:
-          #   08:00 local -> 0 6   (daily cluster report)
-          #   09:00 local -> 0 7   (cert expiry check)
-          # Relative schedules (every 6h / every 1h / every 1d) are
-          # timezone-independent.
-          create "daily-cluster-report" "0 6 * * *" "discord" \
-            "Produce a daily cluster report for Roger using the HTTP APIs documented in your SOUL.md. Include: (1) node count and Ready/NotReady status per node, (2) top 5 pods by CPU and by memory, (3) count of pods not Running grouped by namespace, (4) any ArgoCD apps that are OutOfSync or Degraded, (5) any certificates expiring within 30 days, (6) recent error/crash activity: IMPORTANT context discipline — do NOT fetch raw log lines across the cluster over 24h — that overflows your context window. First run a LogQL aggregation that returns only counts: sum by (namespace, pod) (count_over_time({namespace=~\".+\"} |~ \"(?i)error|panic|crash|oom|backoff\" [24h])). Parse the counts with jq and summarize the top few noisiest pods by namespace/pod:count. If one stands out, fetch at most 5 sample lines for THAT pod only (limit=5) to identify the cause. Never paste more than a handful of lines total. Keep the whole report under 1800 chars. Always deliver (no [SILENT])."
 
           echo "Done. Listing all cron jobs:"
           kubectl -n platform-engineer exec "$POD" -- hermes cron list