deploy deepseek v4

home-manager/README.md

@@ -57,7 +57,7 @@ Docs: https://hermes-agent.nousresearch.com/docs/user-guide/docker
 | Discord bot | dedicated | dedicated (separate token + channel) |
 | `HERMES_WRITE_SAFE_ROOT` | `/` (needs to edit manifests) | unset → defaults to `/opt/data` (tighter) |
 | Image | `nousresearch/hermes-agent:latest` | same |
-| Model | `qwen3.6` via LiteLLM | same |
+| Model | `deepseek-v4-flash-0731` via LiteLLM | same |
 
 ---
 

home-manager/configmap.yaml

@@ -9,18 +9,18 @@ data:
   config.yaml: |
     model:
       provider: openai-api
-      default: qwen3.6-35b-a3b
+      default: deepseek-v4-flash-0731
       base_url: "http://litellm-service.litellm:80/v1"
       api_mode: chat_completions
 
     auxiliary:
       compression:
         provider: openai-api
-        model: qwen3.6-35b-a3b
+        model: deepseek-v4-flash-0731
         base_url: "http://litellm-service.litellm:80/v1"
       title_generation:
         provider: openai-api
-        model: qwen3.6-35b-a3b
+        model: deepseek-v4-flash-0731
         base_url: "http://litellm-service.litellm:80/v1"
 
     terminal:

litellm/litellm.yaml

@@ -39,22 +39,17 @@ data:
         litellm_params:
           model: ollama/glm-4.7-flash
           api_base: http://10.88.20.12:11434
-      # Used by the platform-engineer Hermes agent (deployed in ns platform-engineer).
+      # In-cluster LLM served by llama.cpp (llama-server, Vulkan on the
-      # Served by llama.cpp (llama-server, Vulkan on the NUCBox APU) — see
+      # NUCBox APU) — see llamacpp/deployment-deepseek-v4-flash-0731.yaml.
-      # llamacpp/deployment-qwen36-27b.yaml. Thinking is left ON; reasoning comes
+      # DeepSeek-V4-Flash-0731 (UD-IQ1_M, ~87 GiB) is the sole model on the
-      # back in `reasoning_content` (llama-server --jinja + default reasoning format).
+      # NUCBox; the two Qwen3.6 models were removed to make VRAM room. The
-      - model_name: qwen3.6-27b
+      # chat template is applied via llama-server --jinja; reasoning (when
+      # thinking is enabled) comes back in `reasoning_content`. Default
+      # sampling (temp 1.0 / top-p 0.95) is set on the llama-server itself.
+      - model_name: deepseek-v4-flash-0731
         litellm_params:
-          model: openai/qwen3.6-27b
+          model: openai/deepseek-v4-flash-0731
-          api_base: http://llamacpp-qwen36-27b.llamacpp/v1
+          api_base: http://llamacpp-deepseek-v4-flash-0731.llamacpp/v1
-          api_key: "sk-no-auth"
-      # Faster "flash" variant: Qwen3.6-35B-A3B is a MoE (3B active params), so
-      # inference is much faster than the dense 27B despite more total weights.
-      # See llamacpp/deployment-qwen36-35b-a3b.yaml.
-      - model_name: qwen3.6-35b-a3b
-        litellm_params:
-          model: openai/qwen3.6-35b-a3b
-          api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
           api_key: "sk-no-auth"
     litellm_settings:
       #set_verbose: True  # Uncomment this if you want to see verbose logs; not recommended in production

llamacpp/README.md

@@ -1,37 +1,50 @@
 # llama.cpp (llama-server)
 
-In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on
+In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model
-the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified
+on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB
-memory: 32 GiB RAM / 96 GiB VRAM).
+unified memory: ~90 GiB VRAM / 30 GiB CPU RAM).
 
-This replaces the bare-metal Ollama setup for models that benefit from
+LiteLLM (`litellm/`) points at these in-cluster Services instead of the
-always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these
+external `10.88.20.12:11434` Ollama endpoint.
-in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
 
 ## Layout
 
 One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
 to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
 
-| Alias             | Model                      | GGUF                                                  | Service                                   |
+| Alias                    | Model                          | GGUF                                                  | Service                                              | Args ref |
-|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
+|--------------------------|--------------------------------|-------------------------------------------------------|------------------------------------------------------|----------|
-| `qwen3.6-27b`     | Qwen3.6-27B (dense)        | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB)      | `llamacpp-qwen36-27b.llamacpp:80`         |
+| `deepseek-v4-flash-0731` | DeepSeek-V4-Flash-0731 (MoE)   | unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB) | `llamacpp-deepseek-v4-flash-0731.llamacpp:80`        | [args-deepseek-v4-flash-0731.md](args-deepseek-v4-flash-0731.md) |
-| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB)  | `llamacpp-qwen36-35b-a3b.llamacpp:80`     |
 
-The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
+DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per
-inference is significantly faster than the dense 27B despite more total weights —
+token) with MLA attention, so only a small fraction of the weights is computed
-hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for
+per token. The full ~87 GiB of IQ1_M weights is loaded into the unified memory
-deeper reasoning.
+pool.
+
+> **Previously** the NUCBox ran two co-resident Qwen3.6 models (a 27B dense and
+> a 35B-A3B MoE "flash"). DeepSeek-V4-Flash-0731 (~87 GiB) nearly fills the
+> whole 90 GiB VRAM pool on its own, so both Qwen models were removed to make
+> room. Their GGUF files are deleted from the shared PVC by the new pod's
+> `fetch-model` initContainer on first boot. Their deployment arguments are
+> still documented for redeployment:
+> - [args-qwen36-27b.md](args-qwen36-27b.md) — dense 27B (deeper reasoning)
+> - [args-qwen36-35b-a3b.md](args-qwen36-35b-a3b.md) — MoE 35B-A3B "flash" (fast)
 
 Model files are downloaded idempotently by an initContainer into a shared
 hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
-without re-downloading.
+without re-downloading. The UD-IQ1_M GGUF is split across 3 shards
+(`-00001-of-00003` … `-00003-of-00003`); llama.cpp auto-loads all shards when
+pointed at the first one.
 
 ## GPU / Vulkan
 
 The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
-the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full
+the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). The
-layer offload (`-ngl 999`) puts model weights entirely in the 96 GiB VRAM pool.
+Vulkan backend supports the `IQ1_M` matmul (including the MoE `matmul_id`
+variant), so the whole model runs on the GPU. `deepseek4` is a brand-new arch
+(2026-07), so the floating `server-vulkan` tag is used to pull a recent enough
+build — pin to a specific `server-vulkan-bXXXX` tag once a known-good one is
+verified.
 
 The container mounts `/dev/dri` and runs `privileged: true` — the simplest
 reliable way to give Vulkan access to the DRM render node on k3s without a
@@ -41,61 +54,75 @@ group GID) if desired.
 ### Verify the GPU is actually used
 
 ```bash
-kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b     | grep -iE 'vulkan|gpu|offload|device'
+kubectl logs -n llamacpp deploy/llamacpp-deepseek-v4-flash-0731 | grep -iE 'vulkan|gpu|offload|device'
-kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b  | grep -iE 'vulkan|gpu|offload|device'
 ```
 
 If only a CPU device shows up, the container can't see the GPU — check that
 `/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
 
-## VRAM budget (both models co-resident)
+## VRAM budget (single model)
 
-Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
+The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
+it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
+VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
+`-ngl 40` offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on
+CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and
+co-resident pods. Approximate VRAM usage:
 
-| Model             | Weights  | KV cache                                  | Subtotal |
+| Component                       | VRAM        |
-|-------------------|----------|-------------------------------------------|----------|
+|---------------------------------|-------------|
-| qwen3.6-27b       | ~16 GiB  | ~16 GiB  (q8_0, 131k total, 1 slot)      | ~32 GiB  |
+| Weights (40 GPU layers)         | ~81 GiB     |
-| qwen3.6-35b-a3b   | ~20 GiB  | ~9 GiB   (q8_0, 262k total, 131k/slot)   | ~29 GiB  |
+| KV cache (q8_0, 64k, 1 slot)    | ~1.6 GiB    |
-| **Total**         |          |                                           | **~61 GiB** |
+| Vulkan compute buffers          | ~2 GiB      |
+| **Total in VRAM**               | **~85 GiB** |
+| **Headroom (of 90 GiB)**        | **~5–8 GiB**|
 
-~35 GiB headroom — comfortable. Both models' KV caches are quantized
+3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory
-to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
+limit, not VRAM).
-large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
+
-a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
+KV cache is tiny thanks to DeepSeek-V4's **MLA** attention
+(`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
+elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is
+cheap — `-c` is capped at 65536 (the required minimum) to maximise VRAM
+headroom, not because KV is the constraint.
 
 ## Tuning
 
-The key knobs (in each `deployment-*.yaml`):
+The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`):
 
-- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
+- `-ngl 40` — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the
-- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot
+  whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute
-  gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each).
+  buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom.
-  The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense
+  Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs
-  27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B
+  out of device memory.
-  uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
+- `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets
-- `-np` — parallel slots (concurrent requests). Each extra slot multiplies
+  the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped
-  KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request,
+  at the minimum to maximise VRAM headroom. Raise if headroom allows.
-  dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV).
+- `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request).
-  Bump higher only if you need more concurrent throughput.
+  Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal.
-- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
+- `--cache-type-k q8_0 --cache-type-v q8_0` — quantize the KV cache to q8_0,
-  q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
+  halving KV VRAM with ~negligible quality loss. Essential to keep headroom.
-  27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.
+- `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4
-  Drop to q4_0 for even less VRAM if retrieval quality allows.
+  recommendation). These are server defaults; clients can override per request
-- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
+  via the OpenAI-compatible API.
-  full GPU offload; tune if CPU-bound.
+- `--threads 8` — CPU threads for sampling + the 3 CPU-resident layers.
 
 ## Memory accounting
 
-k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is
+k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is
-reserved by firmware and managed by `amdgpu`). The model weights and KV cache
+reserved by firmware and managed by `amdgpu`). The GPU-resident model weights
-live in VRAM and are **not** counted against the container's cgroup memory
+and KV cache live in VRAM and are **not** counted against the container's cgroup
-limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages
+memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages
-during load. If the pod is OOM-killed during model load, raise the memory limit.
+for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load.
+If the pod is OOM-killed during model load, raise the memory limit (and/or
+lower `-ngl`).
 
-## Adding a model
+## Adding / replacing a model
 
-1. Copy `deployment-qwen36-27b.yaml` → `deployment-<new>.yaml`; change the
+1. Copy `deployment-deepseek-v4-flash-0731.yaml` → `deployment-<new>.yaml`;
-   `model:` label, GGUF URL/file, `--alias`, and Service name.
+   change the `model:` label, GGUF URL/file(s), `--alias`, and Service name.
+   For split GGUFs, point `-m` at the first shard and download all shards in
+   the `fetch-model` initContainer.
 2. Point LiteLLM at it in `litellm/litellm.yaml`:
    ```yaml
    - model_name: <alias>
@@ -106,4 +133,5 @@ during load. If the pod is OOM-killed during model load, raise the memory limit.
    ```
 3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
    directory recursively.)
-4. Check the VRAM budget table above — two large models may not coexist.
+4. Check the VRAM budget table above — at ~87 GiB this model nearly fills the
+   90 GiB pool on its own, so co-locating another large model is not possible.

llamacpp/args-deepseek-v4-flash-0731.md

@@ -0,0 +1,109 @@
+# Deployment arguments — `deepseek-v4-flash-0731`
+
+Reference for the llama-server flags used in
+`deployment-deepseek-v4-flash-0731.yaml`. Keep this in sync if the
+Deployment is edited.
+
+## Model & source
+
+- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` (DeepSeek-V4-Flash, 0731 weights)
+- **Quantization:** `unsloth/DeepSeek-V4-Flash-0731-GGUF` → `UD-IQ1_M`
+  (Unsloth Dynamic IQ1_M), **split across 3 shards** (~87 GiB total):
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf`
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf`
+  - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf`
+- **HuggingFace repo:** `https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (floating tag — `deepseek4`
+  is a brand-new arch; pin to a specific `server-vulkan-bXXXX` once verified).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5),
+~120 GiB unified memory split by firmware into **~90 GiB VRAM** and
+**~30 GiB CPU RAM**. Pinned via `nodeSelector: {kubernetes.io/arch: amd64,
+hardware: high-memory}`.
+
+## Why this model nearly fills the machine
+
+The UD-IQ1_M model is **~87 GiB**, almost the entire 90 GiB VRAM pool. Unlike
+the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one
+**cannot be fully offloaded** to the GPU: putting all 43 layers + the KV cache +
+Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all
+about fitting the model while leaving headroom for the KV cache, compute
+buffers, and co-resident pods.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf` | Model file. Pointed at the **first shard** only; llama.cpp auto-discovers and loads `-00002-…` / `-00003-…` from the same directory. |
+| `--alias` | `deepseek-v4-flash-0731` | Name reported by `GET /v1/models`. Must match the `model_name` in `litellm/litellm.yaml` so LiteLLM routes to this server. |
+| `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. |
+| `-ngl` | `40` | **GPU layer offload.** Offload 40 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM. |
+| `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. |
+| `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. |
+| `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). |
+| `--cache-type-k` | `q8_0` | Quantize the K cache to q8_0 (halves KV VRAM, ~negligible quality loss). |
+| `--cache-type-v` | `q8_0` | Quantize the V cache to q8_0 (same). |
+| `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. |
+| `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. |
+| `--threads` | `8` | CPU threads for sampling + the 3 CPU-resident layers. Mostly irrelevant under heavy GPU offload. |
+
+## VRAM budget (90 GiB pool)
+
+| Component                       | VRAM        |
+|---------------------------------|-------------|
+| Weights (40 GPU layers)         | ~81 GiB     |
+| KV cache (q8_0, 64k, 1 slot)    | ~1.6 GiB    |
+| Vulkan compute buffers          | ~2 GiB      |
+| **Total in VRAM**               | **~85 GiB** |
+| **Headroom (of 90 GiB)**        | **~5–8 GiB**|
+
+The remaining **3 layers (~6 GiB) live in CPU RAM** and are counted against the
+pod's cgroup memory limit (not VRAM). KV cache is tiny thanks to DeepSeek-V4's
+**MLA** attention (`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
+elements/token/layer); at 64k context q8_0 KV is only ~1.6 GiB, so context is
+**not** the constraint — `-c` is capped at the minimum purely to maximise VRAM
+headroom.
+
+## How to tune if it OOMs / has spare headroom
+
+- **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl`
+  (e.g. `38`) to keep more layers on CPU, or raise the container `memory`
+  limit.
+- **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full
+  offload) and/or raise `-c` for larger context.
+- **Need more concurrent requests:** raise `-np` (KV is cheap), but each extra
+  slot multiplies KV VRAM and the context per slot shrinks (`-c / -np`).
+
+## initContainer (`fetch-model`)
+
+- Idempotently downloads the 3 shards into the shared models PVC, skipping if
+  the first shard is already present (pod restart / recreate).
+- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim space for the
+  new 87 GiB model.
+
+## Resources
+
+```
+requests:  cpu 1000m, memory 4Gi
+limits:    cpu 4000m, memory 20Gi
+```
+
+VRAM (weights + KV + compute) is **not** counted against the cgroup memory
+limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the
+3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. k8s
+sees ~30 GiB as the node's allocatable system RAM.
+
+## Probes
+
+- `readinessProbe`: `GET /health` after 30s, every 10s, 6 failures.
+- `livenessProbe`: `GET /health` after **300s** (87 GiB load + Vulkan init
+  takes several minutes), every 30s, 5 failures.
+
+## Security / GPU access
+
+`privileged: true` + mounts `/dev/dri` (DRM render nodes) — simplest reliable
+way to give Vulkan access to the AMD APU on k3s without a device plugin.

llamacpp/args-qwen36-27b.md

@@ -0,0 +1,85 @@
+# Deployment arguments — `qwen3.6-27b` (REMOVED)
+
+> **Status:** This model was **removed** from the cluster to make VRAM room for
+> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
+> 90 GiB VRAM pool on its own. This file documents the flags used when the
+> model was live, so it can be redeployed later if the DeepSeek model is taken
+> down or moved to different hardware.
+>
+> To redeploy: restore `deployment-qwen36-27b.yaml` (the manifest is preserved
+> in git history) and re-add the LiteLLM entry. Re-check the VRAM budget —
+> co-locating with the 87 GiB DeepSeek model is **not** possible on the current
+> 90 GiB pool.
+
+## Model & source
+
+- **Model:** `unsloth/Qwen3.6-27B-MTP-GGUF` — Qwen3.6-27B, **dense** (all
+  params active per token), MTP variant.
+- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~16 GiB**,
+  single GGUF file (`Qwen3.6-27B-UD-Q4_K_XL.gguf`).
+- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
+  driver, supports Radeon 8060S / RDNA 3.5).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
+memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
+{kubernetes.io/arch: amd64, hardware: high-memory}`.
+
+## Why it fit (and co-existed with the 35B-A3B)
+
+At ~16 GiB, the Q4 model is small enough to **fully offload** to the GPU
+(`-ngl 999`) and still leave ~80 GiB of VRAM — which is why it ran alongside
+the Qwen3.6-35B-A3B model (see `args-qwen36-35b-a3b.md`). Combined the two
+models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was **KV
+cache**, not weights: dense attention KV is ~256 KiB/token (f16) / ~128
+KiB/token (q8_0), so large context is expensive.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/Qwen3.6-27B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
+| `--alias` | `qwen3.6-27b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
+| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
+| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
+| `-c` | `131072` | Total KV-cache context (131k). Single slot gets the full window. |
+| `-np` | `1` | 1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting). |
+| `--cont-batching` | *(flag)* | Continuous batching across slots. |
+| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model. |
+| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
+| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
+
+## VRAM budget (when live)
+
+| Component             | VRAM        |
+|-----------------------|-------------|
+| Weights (full offload) | ~16 GiB    |
+| KV cache (q8_0, 131k)  | ~16 GiB    |
+| **Subtotal**          | **~32 GiB** |
+
+Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
+co-locating the 35B-A3B model.
+
+## Notes for redeployment
+
+- This was the **deeper-reasoning** model (dense, all params active). The
+  `qwen3.6-35b-a3b` was the **fast/flash** variant. If redeploying only one,
+  decide based on latency-vs-quality needs.
+- LiteLLM entry that went with it:
+  ```yaml
+  - model_name: qwen3.6-27b
+    litellm_params:
+      model: openai/qwen3.6-27b
+      api_base: http://llamacpp-qwen36-27b.llamacpp/v1
+      api_key: "sk-no-auth"
+  ```
+- Consumers at removal time: `platform-engineer` (default model),
+  `home-manager` (used the 35B-A3B, not this one). See git history for exact
+  config.
+- The `fetch-model` initContainer downloaded the single GGUF idempotently into
+  the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
+  GGUF on first boot to reclaim space, so a redeploy will re-download it.

llamacpp/args-qwen36-35b-a3b.md

@@ -0,0 +1,96 @@
+# Deployment arguments — `qwen3.6-35b-a3b` (REMOVED)
+
+> **Status:** This model was **removed** from the cluster to make VRAM room for
+> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
+> 90 GiB VRAM pool on its own. This file documents the flags used when the
+> model was live, so it can be redeployed later if the DeepSeek model is taken
+> down or moved to different hardware.
+>
+> To redeploy: restore `deployment-qwen36-35b-a3b.yaml` (the manifest is
+> preserved in git history) and re-add the LiteLLM entry. Re-check the VRAM
+> budget — co-locating with the 87 GiB DeepSeek model is **not** possible on
+> the current 90 GiB pool.
+
+## Model & source
+
+- **Model:** `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` — Qwen3.6-35B-A3B, the "flash"
+  **Mixture-of-Experts** variant (35B total params, **only 3B active** per
+  token), MTP variant.
+- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~20 GiB**,
+  single GGUF file (`Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf`).
+- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF`
+- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
+  driver, supports Radeon 8060S / RDNA 3.5).
+
+## Hardware target
+
+NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
+memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
+{kubernetes.io/arch: amd64, hardware: high-memory}`.
+
+## Why it was the "flash" model
+
+Despite having **more total parameters** than the dense 27B, only **3B are
+active per token** (MoE), so inference is significantly faster. The full ~20 GiB
+of Q4 weights is still loaded into VRAM, but only a small fraction is computed
+per token. Its KV cache is also tiny (~72 KiB/token), so large context is nearly
+free — hence the much larger `-c` and the 2-slot split.
+
+## Argument-by-argument
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `-m` | `/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
+| `--alias` | `qwen3.6-35b-a3b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
+| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
+| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
+| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
+| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
+| `-c` | `262144` | Total KV-cache context (262k), **split across parallel slots**. |
+| `-np` | `2` | 2 parallel slots ⇒ 131k tokens per concurrent request (262k / 2). MoE KV is cheap, so splitting is affordable. |
+| `--cont-batching` | *(flag)* | Continuous batching across the 2 slots. |
+| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB at 262k); frees headroom for the large `-c`. Drop to `q4_0` for even less VRAM if retrieval quality allows. |
+| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
+| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
+
+## VRAM budget (when live)
+
+| Component             | VRAM       |
+|-----------------------|------------|
+| Weights (full offload) | ~20 GiB   |
+| KV cache (q8_0, 262k)  | ~9 GiB    |
+| **Subtotal**          | **~29 GiB** |
+
+Left ~67 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
+co-locating the dense 27B model (combined ~61 GiB).
+
+## Combined VRAM budget (both Qwen models, when live)
+
+| Model             | Weights  | KV cache                                  | Subtotal |
+|-------------------|----------|-------------------------------------------|----------|
+| qwen3.6-27b       | ~16 GiB  | ~16 GiB  (q8_0, 131k total, 1 slot)      | ~32 GiB  |
+| qwen3.6-35b-a3b   | ~20 GiB  | ~9 GiB   (q8_0, 262k total, 131k/slot)   | ~29 GiB  |
+| **Total**         |          |                                           | **~61 GiB** |
+
+~35 GiB headroom on the 96 GiB pool — comfortable. (DeepSeek-V4-Flash-0731 at
+~87 GiB cannot coexist with either of these; that's why both were removed.)
+
+## Notes for redeployment
+
+- This was the **fast/flash** model (MoE, 3B active). The `qwen3.6-27b` was the
+  **deeper-reasoning** model (dense, all params active). If redeploying only
+  one, decide based on latency-vs-quality needs.
+- LiteLLM entry that went with it:
+  ```yaml
+  - model_name: qwen3.6-35b-a3b
+    litellm_params:
+      model: openai/qwen3.6-35b-a3b
+      api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
+      api_key: "sk-no-auth"
+  ```
+- Consumers at removal time: `home-manager` (default + auxiliary),
+  `platform-engineer` (auxiliary compression + title generation). See git
+  history for exact config.
+- The `fetch-model` initContainer downloaded the single GGUF idempotently into
+  the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
+  GGUF on first boot to reclaim space, so a redeploy will re-download it.

llamacpp/deployment-deepseek-v4-flash-0731.yaml

@@ -0,0 +1,210 @@
+# DeepSeek-V4-Flash-0731 (MoE: 256 experts / 6 active, UD-IQ1_M ≈ 87 GiB)
+# served by llama.cpp's llama-server on the NUCBox APU.
+#
+# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
+# ~120 GiB unified memory (≈90 GiB VRAM / 30 GiB CPU RAM via firmware). The
+# IQ1_M model (~87 GiB) is *almost* the size of the whole VRAM pool, so it
+# CANNOT be fully offloaded to the GPU: offloading all 43 layers + the KV
+# cache + Vulkan compute buffers would overflow 90 GiB. Instead we offload
+# 40 of 43 layers (-ngl 40) and keep the last 3 (~6 GiB) on CPU RAM, leaving
+# ~8 GiB of VRAM headroom for the KV cache, compute buffers, and co-resident
+# pods. This is the only model served on the NUCBox — the two Qwen3.6 models
+# were removed to make room (their GGUF files should be deleted from the PVC,
+# which the initContainer below does on first boot).
+#
+# KV cache is tiny thanks to DeepSeek-V4's MLA attention (num_kv_heads=1,
+# head_dim=512 + 64 decoupled RoPE ⇒ ~576 elements/token/layer). At 64k
+# context, q8_0 KV is only ~1.6 GiB, so context is cheap — but we cap -c at
+# 65536 (the required minimum) to maximise VRAM headroom, not because KV is
+# the constraint.
+#
+# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan
+# driver, which supports the Radeon 8060S (RDNA 3.5). The Vulkan backend
+# supports the IQ1_M matmul (incl. the MoE matmul_id variant), so the whole
+# model runs on the GPU. deepseek4 is a brand-new arch (2026-07) so the
+# floating `server-vulkan` tag is used to pull a recent enough build; pin to a
+# specific server-vulkan-bXXXX tag once a known-good one is verified.
+#
+# GPU access: the container mounts /dev/dri (the DRM render nodes) and runs
+# privileged — the simplest reliable option on k3s without a Vulkan device
+# plugin.
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llamacpp-deepseek-v4-flash-0731
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    model: deepseek-v4-flash-0731
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate   # never run two pods loading the same model into VRAM
+  selector:
+    matchLabels:
+      app: llamacpp
+      model: deepseek-v4-flash-0731
+  template:
+    metadata:
+      labels:
+        app: llamacpp
+        model: deepseek-v4-flash-0731
+    spec:
+      nodeSelector:
+        kubernetes.io/arch: amd64
+        hardware: high-memory
+      initContainers:
+      # Idempotently download the (3-part, split) GGUF into the shared models
+      # PVC on first boot. Also removes the retired Qwen3.6 GGUFs so the new
+      # 87 GiB model fits on the PVC alongside any other data. Exits
+      # immediately if the first shard is already present (pod restart).
+      - name: fetch-model
+        image: alpine:3.20
+        command: ["/bin/sh", "-c"]
+        args:
+        - |
+          set -e
+          # Reclaim space from the retired Qwen3.6 models (their Deployments
+          # are gone; the GGUFs are dead weight on the shared PVC).
+          for old in Qwen3.6-27B-UD-Q4_K_XL.gguf Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf; do
+            if [ -f "/models/$old" ]; then
+              echo "Removing retired model $old ..."
+              rm -f "/models/$old"
+            fi
+          done
+          # Download any missing shards of the split UD-IQ1_M GGUF.
+          if [ -f "/models/$SHARD1" ]; then
+            echo "First shard $SHARD1 already present — skipping download."
+            exit 0
+          fi
+          echo "Installing curl..."
+          apk add --no-cache curl
+          for s in "$SHARD1" "$SHARD2" "$SHARD3"; do
+            echo "Downloading $s from $HF_REPO ..."
+            curl -fL --retry 5 --retry-delay 5 -o "/models/$s" "$HF_REPO/$s"
+          done
+          echo "Download complete:"
+          ls -lh /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-*.gguf
+        env:
+        - name: HF_REPO
+          value: "https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF/resolve/main/UD-IQ1_M"
+        - name: SHARD1
+          value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf"
+        - name: SHARD2
+          value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf"
+        - name: SHARD3
+          value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf"
+        volumeMounts:
+        - name: models
+          mountPath: /models
+      containers:
+      - name: llama-server
+        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
+        imagePullPolicy: IfNotPresent
+        # llama.cpp's CLI parser does NOT split on '=' — every value flag must
+        # be a separate argv element (flag, then value). See common/arg.cpp.
+        args:
+        - -m                       # model file (first shard; llama.cpp auto-loads the rest)
+        - /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf
+        - --alias                 # /v1/models reports this name; matches the litellm alias
+        - deepseek-v4-flash-0731
+        - --host
+        - 0.0.0.0
+        - --port
+        - "8080"
+        - --jinja                  # use the GGUF's DeepSeek-V4 chat template (DSML / thinking)
+        - -ngl                     # offload 40 of 43 layers to the GPU. The model (~87 GiB) is
+        - "40"                     # nearly the whole 90 GiB VRAM pool, so full offload (-ngl 999)
+                                   # would overflow once KV cache + Vulkan compute buffers are
+                                   # added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB of
+                                   # VRAM headroom for the KV cache, compute buffers, and
+                                   # co-resident pods. Raise toward 43 if VRAM allows; lower
+                                   # (e.g. 38) if the pod OOMs / Vulkan runs out of device mem.
+        - -c                       # total KV-cache context (single slot gets the full window).
+        - "65536"                  # 64k — the required minimum. MLA KV is tiny (~1.6 GiB at
+                                   # q8_0), so context is cheap; -c is capped at the minimum to
+                                   # maximise VRAM headroom, not because KV is the constraint.
+                                   # Raise if VRAM headroom allows.
+        - -np                      # 1 slot => the full 64k goes to a single concurrent request
+        - "1"                      # (extra slots would multiply KV VRAM, which is fine here, but
+                                   # 1 slot keeps it simple and headroom maximal).
+        - --cont-batching          # continuous batching across slots
+        - --cache-type-k            # quantize KV cache to q8_0 — MLA KV is already small (~576
+        - q8_0                     # elem/token/layer); q8_0 halves it to ~1.6 GiB at 64k and
+        - --cache-type-v            # maximises VRAM headroom with ~negligible quality loss.
+        - q8_0
+        - --temp                   # default sampling temperature (DeepSeek-V4 recommendation)
+        - "1.0"
+        - --top-p                  # default nucleus sampling threshold (DeepSeek-V4 recommendation)
+        - "0.95"
+        - --threads                # CPU threads for sampling + the 3 CPU-resident layers
+        - "8"
+        ports:
+        - name: http
+          containerPort: 8080
+        resources:
+          # The model weights + KV cache live in GPU VRAM (~90 GiB pool) and
+          # are NOT counted against the cgroup memory limit. This limit only
+          # covers CPU-side overhead + the mmap'd GGUF pages for the 3
+          # CPU-resident layers (~6 GiB) plus reclaimable page cache during
+          # load. k8s sees ~30 GiB as the node's allocatable system RAM, so the
+          # limit is sized to cover the CPU layers + overhead while leaving
+          # RAM for co-resident pods (litellm, the agents, etc.). If the pod is
+          # OOM-killed during model load, raise the limit.
+          requests:
+            cpu: "1000m"
+            memory: "4Gi"
+          limits:
+            cpu: "4000m"
+            memory: "20Gi"
+        readinessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 6
+        livenessProbe:
+          httpGet:
+            path: /health
+            port: 8080
+          initialDelaySeconds: 300   # 87 GiB load + Vulkan init takes several minutes
+          periodSeconds: 30
+          failureThreshold: 5
+        securityContext:
+          # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is
+          # the simplest reliable path on k3s without a device plugin.
+          privileged: true
+        volumeMounts:
+        - name: models
+          mountPath: /models
+          readOnly: true
+        - name: dri
+          mountPath: /dev/dri
+      volumes:
+      - name: models
+        persistentVolumeClaim:
+          claimName: llamacpp-models
+      - name: dri
+        hostPath:
+          path: /dev/dri
+          type: Directory
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llamacpp-deepseek-v4-flash-0731
+  namespace: llamacpp
+  labels:
+    app: llamacpp
+    model: deepseek-v4-flash-0731
+spec:
+  type: ClusterIP
+  selector:
+    app: llamacpp
+    model: deepseek-v4-flash-0731
+  ports:
+  - name: http
+    port: 80
+    targetPort: 8080

llamacpp/deployment-qwen36-27b.yaml

@@ -1,163 +0,0 @@
-# Qwen3.6-27B (dense, Q4_K_XL) served by llama.cpp's llama-server on the NUCBox APU.
-#
-# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
-# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). The Q4 model
-# (~16 GiB) is fully offloaded to the GPU via the Vulkan backend.
-#
-# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan
-# driver, which supports the Radeon 8060S (RDNA 3.5). The project moved from the
-# legacy `ggerganov/llama.cpp` namespace (which only has light/full tags) to
-# `ggml-org/llama.cpp` (server-vulkan + pinned build tags like server-vulkan-bXXXX).
-# Pin to a build tag (e.g. server-vulkan-b4738) for production reproducibility.
-#
-# GPU access: the container mounts /dev/dri (the DRM render nodes) and runs
-# privileged. This is the simplest reliable option on k3s without a Vulkan
-# device plugin; tighten later with supplementalGroups if desired.
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: llamacpp-qwen36-27b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-27b
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate   # never run two pods loading the same model into VRAM
-  selector:
-    matchLabels:
-      app: llamacpp
-      model: qwen3.6-27b
-  template:
-    metadata:
-      labels:
-        app: llamacpp
-        model: qwen3.6-27b
-    spec:
-      nodeSelector:
-        kubernetes.io/arch: amd64
-        hardware: high-memory
-      initContainers:
-      # Idempotently download the GGUF into the shared models PVC on first boot.
-      # Exits immediately if the file is already present (pod restart / recreate).
-      - name: fetch-model
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        args:
-        - |
-          set -e
-          if [ -f "/models/$MODEL_FILE" ]; then
-            echo "Model $MODEL_FILE already present — skipping download."
-            exit 0
-          fi
-          echo "Installing curl..."
-          apk add --no-cache curl
-          echo "Downloading $MODEL_FILE from $MODEL_URL ..."
-          curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL"
-          echo "Download complete: $(ls -lh /models/$MODEL_FILE)"
-        env:
-        - name: MODEL_URL
-          value: "https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-UD-Q4_K_XL.gguf"
-        - name: MODEL_FILE
-          value: "Qwen3.6-27B-UD-Q4_K_XL.gguf"
-        volumeMounts:
-        - name: models
-          mountPath: /models
-      containers:
-      - name: llama-server
-        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
-        imagePullPolicy: IfNotPresent
-        # llama.cpp's CLI parser does NOT split on '=' — every value flag must be a
-        # separate argv element (flag, then value). See common/arg.cpp in the repo.
-        args:
-        - -m                       # model file
-        - /models/Qwen3.6-27B-UD-Q4_K_XL.gguf
-        - --alias                 # /v1/models reports this name; matches the litellm alias
-        - qwen3.6-27b
-        - --host
-        - 0.0.0.0
-        - --port
-        - "8080"
-        - --jinja                  # use the GGUF's chat template (Qwen3 thinking format)
-        - -ngl                     # offload ALL layers to the GPU (fits in 96 GiB VRAM)
-        - "999"
-        - -c                       # total KV-cache context (single slot gets the full window)
-        - "131072"
-        - -np                      # 1 slot => 131072 tokens for the single request (no split)
-        - "1"
-        - --cont-batching          # continuous batching across slots
-        - --cache-type-k            # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB
-        - q8_0                     # at 131k ctx); ~negligible quality loss, frees headroom for large -c
-        - --cache-type-v
-        - q8_0
-        - --threads                # CPU threads for sampling/overhead (GPU does the heavy lifting)
-        - "8"
-        ports:
-        - name: http
-          containerPort: 8080
-        resources:
-          # The model weights + KV cache live in GPU VRAM (96 GiB pool) and are
-          # NOT counted against the cgroup memory limit. This limit only covers
-          # CPU-side overhead + the mmap'd GGUF file pages during load (~16 GiB,
-          # reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM,
-          # so the request is kept low to stay schedulable alongside other pods.
-          # If the pod OOM-kills during load, the amdgpu driver may be counting
-          # some VRAM against the cgroup — raise the limit.
-          requests:
-            cpu: "1000m"
-            memory: "2Gi"
-          limits:
-            cpu: "4000m"
-            memory: "24Gi"
-        readinessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 30
-          periodSeconds: 10
-          failureThreshold: 6
-        livenessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 180   # model load + Vulkan init can take a few minutes
-          periodSeconds: 30
-          failureThreshold: 5
-        securityContext:
-          # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the
-          # simplest reliable path on k3s without a device plugin.
-          privileged: true
-        volumeMounts:
-        - name: models
-          mountPath: /models
-          readOnly: true
-        - name: dri
-          mountPath: /dev/dri
-      volumes:
-      - name: models
-        persistentVolumeClaim:
-          claimName: llamacpp-models
-      - name: dri
-        hostPath:
-          path: /dev/dri
-          type: Directory
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: llamacpp-qwen36-27b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-27b
-spec:
-  type: ClusterIP
-  selector:
-    app: llamacpp
-    model: qwen3.6-27b
-  ports:
-  - name: http
-    port: 80
-    targetPort: 8080

llamacpp/deployment-qwen36-35b-a3b.yaml

@@ -1,162 +0,0 @@
-# Qwen3.6-35B-A3B (MoE: 35B total / 3B active, Q4_K_XL) — the "flash" variant.
-#
-# Despite having more total parameters than the 27B dense model, only 3B are
-# active per token (Mixture-of-Experts), so inference is much faster. The full
-# ~20 GiB of Q4 weights is still loaded into VRAM but only a small fraction is
-# computed per token.
-#
-# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
-# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). Full GPU
-# offload via the Vulkan backend. Shares the 96 GiB VRAM pool with the 27B
-# model — see llamacpp/README.md for the combined VRAM budget.
-#
-# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (Mesa/RADV Vulkan driver,
-# supports the Radeon 8060S / RDNA 3.5). Pin to a build tag for production.
-#
-# GPU access: mounts /dev/dri + privileged (simplest reliable path on k3s).
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: llamacpp-qwen36-35b-a3b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-35b-a3b
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate   # never run two pods loading the same model into VRAM
-  selector:
-    matchLabels:
-      app: llamacpp
-      model: qwen3.6-35b-a3b
-  template:
-    metadata:
-      labels:
-        app: llamacpp
-        model: qwen3.6-35b-a3b
-    spec:
-      nodeSelector:
-        kubernetes.io/arch: amd64
-        hardware: high-memory
-      initContainers:
-      # Idempotently download the GGUF into the shared models PVC on first boot.
-      # Exits immediately if the file is already present (pod restart / recreate).
-      - name: fetch-model
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        args:
-        - |
-          set -e
-          if [ -f "/models/$MODEL_FILE" ]; then
-            echo "Model $MODEL_FILE already present — skipping download."
-            exit 0
-          fi
-          echo "Installing curl..."
-          apk add --no-cache curl
-          echo "Downloading $MODEL_FILE from $MODEL_URL ..."
-          curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL"
-          echo "Download complete: $(ls -lh /models/$MODEL_FILE)"
-        env:
-        - name: MODEL_URL
-          value: "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
-        - name: MODEL_FILE
-          value: "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
-        volumeMounts:
-        - name: models
-          mountPath: /models
-      containers:
-      - name: llama-server
-        image: ghcr.io/ggml-org/llama.cpp:server-vulkan
-        imagePullPolicy: IfNotPresent
-        # llama.cpp's CLI parser does NOT split on '=' — every value flag must be a
-        # separate argv element (flag, then value). See common/arg.cpp in the repo.
-        args:
-        - -m                       # model file
-        - /models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
-        - --alias                 # /v1/models reports this name; matches the litellm alias
-        - qwen3.6-35b-a3b
-        - --host
-        - 0.0.0.0
-        - --port
-        - "8080"
-        - --jinja                  # use the GGUF's chat template (Qwen3 thinking format)
-        - -ngl                     # offload ALL layers to the GPU (fits in 96 GiB VRAM)
-        - "999"
-        - -c                       # total KV-cache context, split across parallel slots
-        - "262144"
-        - -np                      # 2 parallel slots => 131072 tokens per concurrent request
-        - "2"
-        - --cont-batching          # continuous batching across slots
-        - --cache-type-k            # quantize KV cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB
-        - q8_0                     # at 262k ctx); ~negligible quality loss, frees headroom for large -c
-        - --cache-type-v            # (raise to q4_0 for even less VRAM if retrieval quality allows)
-        - q8_0
-        - --threads                # CPU threads for sampling/overhead (GPU does the heavy lifting)
-        - "8"
-        ports:
-        - name: http
-          containerPort: 8080
-        resources:
-          # The model weights + KV cache live in GPU VRAM (96 GiB pool) and are
-          # NOT counted against the cgroup memory limit. This limit only covers
-          # CPU-side overhead + the mmap'd GGUF file pages during load (~20 GiB,
-          # reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM.
-          # If the pod OOM-kills during load, raise the limit.
-          requests:
-            cpu: "1000m"
-            memory: "2Gi"
-          limits:
-            cpu: "4000m"
-            memory: "24Gi"
-        readinessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 30
-          periodSeconds: 10
-          failureThreshold: 6
-        livenessProbe:
-          httpGet:
-            path: /health
-            port: 8080
-          initialDelaySeconds: 180   # model load + Vulkan init can take a few minutes
-          periodSeconds: 30
-          failureThreshold: 5
-        securityContext:
-          # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the
-          # simplest reliable path on k3s without a device plugin.
-          privileged: true
-        volumeMounts:
-        - name: models
-          mountPath: /models
-          readOnly: true
-        - name: dri
-          mountPath: /dev/dri
-      volumes:
-      - name: models
-        persistentVolumeClaim:
-          claimName: llamacpp-models
-      - name: dri
-        hostPath:
-          path: /dev/dri
-          type: Directory
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: llamacpp-qwen36-35b-a3b
-  namespace: llamacpp
-  labels:
-    app: llamacpp
-    model: qwen3.6-35b-a3b
-spec:
-  type: ClusterIP
-  selector:
-    app: llamacpp
-    model: qwen3.6-35b-a3b
-  ports:
-  - name: http
-    port: 80
-    targetPort: 8080

llamacpp/pv.yaml

@@ -17,7 +17,7 @@ metadata:
 spec:
   storageClassName: manual
   capacity:
-    storage: 100Gi
+    storage: 200Gi
   accessModes:
     - ReadWriteMany
   hostPath:
@@ -44,4 +44,4 @@ spec:
     - ReadWriteMany
   resources:
     requests:
-      storage: 100Gi
+      storage: 200Gi

platform-engineer/README.md

@@ -299,11 +299,12 @@ Design rules baked into SOUL.md:
 
 ## 11. Deployment checklist (do in this order)
 
-1. **Add the Ollama model to LiteLLM** (already done in `litellm/litellm.yaml`):
+1. **Add the in-cluster model to LiteLLM** (already done in `litellm/litellm.yaml`):
-   the `qwen-3.6:27b` entry points at `http://10.88.20.12:11434`. Make sure
+   the `deepseek-v4-flash-0731` entry points at the in-cluster llama.cpp server
-   `qwen3.6:27b` is actually pulled on that Ollama host
+   `http://llamacpp-deepseek-v4-flash-0731.llamacpp/v1` (served from the NUCBox
-   (`ollama pull qwen3.6:27b`). Apply: `kubectl apply -f litellm/` and restart
+   APU; see `llamacpp/deployment-deepseek-v4-flash-0731.yaml`). Apply:
-   the LiteLLM pod so the new config takes effect.
+   `kubectl apply -f litellm/` and restart the LiteLLM pod so the new config
+   takes effect.
 2. **Create the `gitea-registry` secret in the new namespace** (ArgoCD won't
    create it — it's not in the repo):
    ```

platform-engineer/configmap.yaml

@@ -9,18 +9,18 @@ data:
   config.yaml: |
     model:
       provider: openai-api
-      default: qwen3.6-27b
+      default: deepseek-v4-flash-0731
       base_url: "http://litellm-service.litellm:80/v1"
       api_mode: chat_completions
 
     auxiliary:
       compression:
         provider: openai-api
-        model: qwen3.6-35b-a3b    # flash model for speed-sensitive auxiliary tasks
+        model: deepseek-v4-flash-0731    # in-cluster llama.cpp model (replaces the Qwen flash/dense pair)
         base_url: "http://litellm-service.litellm:80/v1"
       title_generation:
         provider: openai-api
-        model: qwen3.6-35b-a3b    # flash model for speed-sensitive auxiliary tasks
+        model: deepseek-v4-flash-0731    # in-cluster llama.cpp model (replaces the Qwen flash/dense pair)
         base_url: "http://litellm-service.litellm:80/v1"
 
     terminal:

platform-engineer/cron-seed.yaml

@@ -56,7 +56,7 @@ spec:
 
           # NOTE: `hermes cron create` has no --model flag. New jobs inherit the
           # global default model from config.yaml at creation time. To pin a
-          # specific model per-job (e.g. the flash model qwen3.6-35b-a3b for all
+          # specific model per-job (e.g. deepseek-v4-flash-0731 for all
           # cron tasks), edit /opt/data/cron/jobs.json directly after seeding:
           #   kubectl exec deploy/hermes -- python3 -c "...set model field..."
           # See llamacpp/README.md and the deployment notes for details.