diff --git a/home-manager/README.md b/home-manager/README.md index f36a52d..a1eb608 100644 --- a/home-manager/README.md +++ b/home-manager/README.md @@ -57,7 +57,7 @@ Docs: https://hermes-agent.nousresearch.com/docs/user-guide/docker | Discord bot | dedicated | dedicated (separate token + channel) | | `HERMES_WRITE_SAFE_ROOT` | `/` (needs to edit manifests) | unset → defaults to `/opt/data` (tighter) | | Image | `nousresearch/hermes-agent:latest` | same | -| Model | `qwen3.6` via LiteLLM | same | +| Model | `deepseek-v4-flash-0731` via LiteLLM | same | --- diff --git a/home-manager/configmap.yaml b/home-manager/configmap.yaml index 47f1583..e79d0d8 100644 --- a/home-manager/configmap.yaml +++ b/home-manager/configmap.yaml @@ -9,18 +9,18 @@ data: config.yaml: | model: provider: openai-api - default: qwen3.6-35b-a3b + default: deepseek-v4-flash-0731 base_url: "http://litellm-service.litellm:80/v1" api_mode: chat_completions auxiliary: compression: provider: openai-api - model: qwen3.6-35b-a3b + model: deepseek-v4-flash-0731 base_url: "http://litellm-service.litellm:80/v1" title_generation: provider: openai-api - model: qwen3.6-35b-a3b + model: deepseek-v4-flash-0731 base_url: "http://litellm-service.litellm:80/v1" terminal: diff --git a/litellm/litellm.yaml b/litellm/litellm.yaml index 6d69615..a4189c6 100644 --- a/litellm/litellm.yaml +++ b/litellm/litellm.yaml @@ -39,22 +39,17 @@ data: litellm_params: model: ollama/glm-4.7-flash api_base: http://10.88.20.12:11434 - # Used by the platform-engineer Hermes agent (deployed in ns platform-engineer). - # Served by llama.cpp (llama-server, Vulkan on the NUCBox APU) — see - # llamacpp/deployment-qwen36-27b.yaml. Thinking is left ON; reasoning comes - # back in `reasoning_content` (llama-server --jinja + default reasoning format). - - model_name: qwen3.6-27b + # In-cluster LLM served by llama.cpp (llama-server, Vulkan on the + # NUCBox APU) — see llamacpp/deployment-deepseek-v4-flash-0731.yaml. + # DeepSeek-V4-Flash-0731 (UD-IQ1_M, ~87 GiB) is the sole model on the + # NUCBox; the two Qwen3.6 models were removed to make VRAM room. The + # chat template is applied via llama-server --jinja; reasoning (when + # thinking is enabled) comes back in `reasoning_content`. Default + # sampling (temp 1.0 / top-p 0.95) is set on the llama-server itself. + - model_name: deepseek-v4-flash-0731 litellm_params: - model: openai/qwen3.6-27b - api_base: http://llamacpp-qwen36-27b.llamacpp/v1 - api_key: "sk-no-auth" - # Faster "flash" variant: Qwen3.6-35B-A3B is a MoE (3B active params), so - # inference is much faster than the dense 27B despite more total weights. - # See llamacpp/deployment-qwen36-35b-a3b.yaml. - - model_name: qwen3.6-35b-a3b - litellm_params: - model: openai/qwen3.6-35b-a3b - api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1 + model: openai/deepseek-v4-flash-0731 + api_base: http://llamacpp-deepseek-v4-flash-0731.llamacpp/v1 api_key: "sk-no-auth" litellm_settings: #set_verbose: True # Uncomment this if you want to see verbose logs; not recommended in production diff --git a/llamacpp/README.md b/llamacpp/README.md index 93664dc..58aaf94 100644 --- a/llamacpp/README.md +++ b/llamacpp/README.md @@ -1,37 +1,50 @@ # llama.cpp (llama-server) -In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on -the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified -memory: 32 GiB RAM / 96 GiB VRAM). +In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model +on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB +unified memory: ~90 GiB VRAM / 30 GiB CPU RAM). -This replaces the bare-metal Ollama setup for models that benefit from -always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these -in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint. +LiteLLM (`litellm/`) points at these in-cluster Services instead of the +external `10.88.20.12:11434` Ollama endpoint. ## Layout One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`): -| Alias | Model | GGUF | Service | -|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------| -| `qwen3.6-27b` | Qwen3.6-27B (dense) | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB) | `llamacpp-qwen36-27b.llamacpp:80` | -| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-35b-a3b.llamacpp:80` | +| Alias | Model | GGUF | Service | Args ref | +|--------------------------|--------------------------------|-------------------------------------------------------|------------------------------------------------------|----------| +| `deepseek-v4-flash-0731` | DeepSeek-V4-Flash-0731 (MoE) | unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB) | `llamacpp-deepseek-v4-flash-0731.llamacpp:80` | [args-deepseek-v4-flash-0731.md](args-deepseek-v4-flash-0731.md) | -The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so -inference is significantly faster than the dense 27B despite more total weights — -hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for -deeper reasoning. +DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per +token) with MLA attention, so only a small fraction of the weights is computed +per token. The full ~87 GiB of IQ1_M weights is loaded into the unified memory +pool. + +> **Previously** the NUCBox ran two co-resident Qwen3.6 models (a 27B dense and +> a 35B-A3B MoE "flash"). DeepSeek-V4-Flash-0731 (~87 GiB) nearly fills the +> whole 90 GiB VRAM pool on its own, so both Qwen models were removed to make +> room. Their GGUF files are deleted from the shared PVC by the new pod's +> `fetch-model` initContainer on first boot. Their deployment arguments are +> still documented for redeployment: +> - [args-qwen36-27b.md](args-qwen36-27b.md) — dense 27B (deeper reasoning) +> - [args-qwen36-35b-a3b.md](args-qwen36-35b-a3b.md) — MoE 35B-A3B "flash" (fast) Model files are downloaded idempotently by an initContainer into a shared hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots -without re-downloading. +without re-downloading. The UD-IQ1_M GGUF is split across 3 shards +(`-00001-of-00003` … `-00003-of-00003`); llama.cpp auto-loads all shards when +pointed at the first one. ## GPU / Vulkan The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles -the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full -layer offload (`-ngl 999`) puts model weights entirely in the 96 GiB VRAM pool. +the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). The +Vulkan backend supports the `IQ1_M` matmul (including the MoE `matmul_id` +variant), so the whole model runs on the GPU. `deepseek4` is a brand-new arch +(2026-07), so the floating `server-vulkan` tag is used to pull a recent enough +build — pin to a specific `server-vulkan-bXXXX` tag once a known-good one is +verified. The container mounts `/dev/dri` and runs `privileged: true` — the simplest reliable way to give Vulkan access to the DRM render node on k3s without a @@ -41,61 +54,75 @@ group GID) if desired. ### Verify the GPU is actually used ```bash -kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b | grep -iE 'vulkan|gpu|offload|device' -kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b | grep -iE 'vulkan|gpu|offload|device' +kubectl logs -n llamacpp deploy/llamacpp-deepseek-v4-flash-0731 | grep -iE 'vulkan|gpu|offload|device' ``` If only a CPU device shows up, the container can't see the GPU — check that `/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded. -## VRAM budget (both models co-resident) +## VRAM budget (single model) -Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage: +The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so +it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into +VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead +`-ngl 40` offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on +CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and +co-resident pods. Approximate VRAM usage: -| Model | Weights | KV cache | Subtotal | -|-------------------|----------|-------------------------------------------|----------| -| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB | -| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB | -| **Total** | | | **~61 GiB** | +| Component | VRAM | +|---------------------------------|-------------| +| Weights (40 GPU layers) | ~81 GiB | +| KV cache (q8_0, 64k, 1 slot) | ~1.6 GiB | +| Vulkan compute buffers | ~2 GiB | +| **Total in VRAM** | **~85 GiB** | +| **Headroom (of 90 GiB)** | **~5–8 GiB**| -~35 GiB headroom — comfortable. Both models' KV caches are quantized -to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's -large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The -a3b's KV is tiny (~72 KiB/token) so its large context is nearly free. +3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory +limit, not VRAM). + +KV cache is tiny thanks to DeepSeek-V4's **MLA** attention +(`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576 +elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is +cheap — `-c` is capped at 65536 (the required minimum) to maximise VRAM +headroom, not because KV is the constraint. ## Tuning -The key knobs (in each `deployment-*.yaml`): +The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`): -- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight. -- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot - gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each). - The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense - 27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B - uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot. -- `-np` — parallel slots (concurrent requests). Each extra slot multiplies - KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request, - dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV). - Bump higher only if you need more concurrent throughput. -- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to - q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense - 27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b. - Drop to q4_0 for even less VRAM if retrieval quality allows. -- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under - full GPU offload; tune if CPU-bound. +- `-ngl 40` — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the + whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute + buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom. + Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs + out of device memory. +- `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets + the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped + at the minimum to maximise VRAM headroom. Raise if headroom allows. +- `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request). + Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal. +- `--cache-type-k q8_0 --cache-type-v q8_0` — quantize the KV cache to q8_0, + halving KV VRAM with ~negligible quality loss. Essential to keep headroom. +- `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4 + recommendation). These are server defaults; clients can override per request + via the OpenAI-compatible API. +- `--threads 8` — CPU threads for sampling + the 3 CPU-resident layers. ## Memory accounting -k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is -reserved by firmware and managed by `amdgpu`). The model weights and KV cache -live in VRAM and are **not** counted against the container's cgroup memory -limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages -during load. If the pod is OOM-killed during model load, raise the memory limit. +k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is +reserved by firmware and managed by `amdgpu`). The GPU-resident model weights +and KV cache live in VRAM and are **not** counted against the container's cgroup +memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages +for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. +If the pod is OOM-killed during model load, raise the memory limit (and/or +lower `-ngl`). -## Adding a model +## Adding / replacing a model -1. Copy `deployment-qwen36-27b.yaml` → `deployment-.yaml`; change the - `model:` label, GGUF URL/file, `--alias`, and Service name. +1. Copy `deployment-deepseek-v4-flash-0731.yaml` → `deployment-.yaml`; + change the `model:` label, GGUF URL/file(s), `--alias`, and Service name. + For split GGUFs, point `-m` at the first shard and download all shards in + the `fetch-model` initContainer. 2. Point LiteLLM at it in `litellm/litellm.yaml`: ```yaml - model_name: @@ -106,4 +133,5 @@ during load. If the pod is OOM-killed during model load, raise the memory limit. ``` 3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole directory recursively.) -4. Check the VRAM budget table above — two large models may not coexist. +4. Check the VRAM budget table above — at ~87 GiB this model nearly fills the + 90 GiB pool on its own, so co-locating another large model is not possible. diff --git a/llamacpp/args-deepseek-v4-flash-0731.md b/llamacpp/args-deepseek-v4-flash-0731.md new file mode 100644 index 0000000..38b44ba --- /dev/null +++ b/llamacpp/args-deepseek-v4-flash-0731.md @@ -0,0 +1,109 @@ +# Deployment arguments — `deepseek-v4-flash-0731` + +Reference for the llama-server flags used in +`deployment-deepseek-v4-flash-0731.yaml`. Keep this in sync if the +Deployment is edited. + +## Model & source + +- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` (DeepSeek-V4-Flash, 0731 weights) +- **Quantization:** `unsloth/DeepSeek-V4-Flash-0731-GGUF` → `UD-IQ1_M` + (Unsloth Dynamic IQ1_M), **split across 3 shards** (~87 GiB total): + - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf` + - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf` + - `DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf` +- **HuggingFace repo:** `https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF` +- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (floating tag — `deepseek4` + is a brand-new arch; pin to a specific `server-vulkan-bXXXX` once verified). + +## Hardware target + +NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5), +~120 GiB unified memory split by firmware into **~90 GiB VRAM** and +**~30 GiB CPU RAM**. Pinned via `nodeSelector: {kubernetes.io/arch: amd64, +hardware: high-memory}`. + +## Why this model nearly fills the machine + +The UD-IQ1_M model is **~87 GiB**, almost the entire 90 GiB VRAM pool. Unlike +the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one +**cannot be fully offloaded** to the GPU: putting all 43 layers + the KV cache + +Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all +about fitting the model while leaving headroom for the KV cache, compute +buffers, and co-resident pods. + +## Argument-by-argument + +| Flag | Value | Meaning | +|------|-------|---------| +| `-m` | `/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf` | Model file. Pointed at the **first shard** only; llama.cpp auto-discovers and loads `-00002-…` / `-00003-…` from the same directory. | +| `--alias` | `deepseek-v4-flash-0731` | Name reported by `GET /v1/models`. Must match the `model_name` in `litellm/litellm.yaml` so LiteLLM routes to this server. | +| `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. | +| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). | +| `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. | +| `-ngl` | `40` | **GPU layer offload.** Offload 40 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM. | +| `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. | +| `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. | +| `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). | +| `--cache-type-k` | `q8_0` | Quantize the K cache to q8_0 (halves KV VRAM, ~negligible quality loss). | +| `--cache-type-v` | `q8_0` | Quantize the V cache to q8_0 (same). | +| `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. | +| `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. | +| `--threads` | `8` | CPU threads for sampling + the 3 CPU-resident layers. Mostly irrelevant under heavy GPU offload. | + +## VRAM budget (90 GiB pool) + +| Component | VRAM | +|---------------------------------|-------------| +| Weights (40 GPU layers) | ~81 GiB | +| KV cache (q8_0, 64k, 1 slot) | ~1.6 GiB | +| Vulkan compute buffers | ~2 GiB | +| **Total in VRAM** | **~85 GiB** | +| **Headroom (of 90 GiB)** | **~5–8 GiB**| + +The remaining **3 layers (~6 GiB) live in CPU RAM** and are counted against the +pod's cgroup memory limit (not VRAM). KV cache is tiny thanks to DeepSeek-V4's +**MLA** attention (`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576 +elements/token/layer); at 64k context q8_0 KV is only ~1.6 GiB, so context is +**not** the constraint — `-c` is capped at the minimum purely to maximise VRAM +headroom. + +## How to tune if it OOMs / has spare headroom + +- **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl` + (e.g. `38`) to keep more layers on CPU, or raise the container `memory` + limit. +- **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full + offload) and/or raise `-c` for larger context. +- **Need more concurrent requests:** raise `-np` (KV is cheap), but each extra + slot multiplies KV VRAM and the context per slot shrinks (`-c / -np`). + +## initContainer (`fetch-model`) + +- Idempotently downloads the 3 shards into the shared models PVC, skipping if + the first shard is already present (pod restart / recreate). +- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim space for the + new 87 GiB model. + +## Resources + +``` +requests: cpu 1000m, memory 4Gi +limits: cpu 4000m, memory 20Gi +``` + +VRAM (weights + KV + compute) is **not** counted against the cgroup memory +limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the +3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. k8s +sees ~30 GiB as the node's allocatable system RAM. + +## Probes + +- `readinessProbe`: `GET /health` after 30s, every 10s, 6 failures. +- `livenessProbe`: `GET /health` after **300s** (87 GiB load + Vulkan init + takes several minutes), every 30s, 5 failures. + +## Security / GPU access + +`privileged: true` + mounts `/dev/dri` (DRM render nodes) — simplest reliable +way to give Vulkan access to the AMD APU on k3s without a device plugin. diff --git a/llamacpp/args-qwen36-27b.md b/llamacpp/args-qwen36-27b.md new file mode 100644 index 0000000..ba78c47 --- /dev/null +++ b/llamacpp/args-qwen36-27b.md @@ -0,0 +1,85 @@ +# Deployment arguments — `qwen3.6-27b` (REMOVED) + +> **Status:** This model was **removed** from the cluster to make VRAM room for +> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's +> 90 GiB VRAM pool on its own. This file documents the flags used when the +> model was live, so it can be redeployed later if the DeepSeek model is taken +> down or moved to different hardware. +> +> To redeploy: restore `deployment-qwen36-27b.yaml` (the manifest is preserved +> in git history) and re-add the LiteLLM entry. Re-check the VRAM budget — +> co-locating with the 87 GiB DeepSeek model is **not** possible on the current +> 90 GiB pool. + +## Model & source + +- **Model:** `unsloth/Qwen3.6-27B-MTP-GGUF` — Qwen3.6-27B, **dense** (all + params active per token), MTP variant. +- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~16 GiB**, + single GGUF file (`Qwen3.6-27B-UD-Q4_K_XL.gguf`). +- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF` +- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan + driver, supports Radeon 8060S / RDNA 3.5). + +## Hardware target + +NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified +memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector: +{kubernetes.io/arch: amd64, hardware: high-memory}`. + +## Why it fit (and co-existed with the 35B-A3B) + +At ~16 GiB, the Q4 model is small enough to **fully offload** to the GPU +(`-ngl 999`) and still leave ~80 GiB of VRAM — which is why it ran alongside +the Qwen3.6-35B-A3B model (see `args-qwen36-35b-a3b.md`). Combined the two +models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was **KV +cache**, not weights: dense attention KV is ~256 KiB/token (f16) / ~128 +KiB/token (q8_0), so large context is expensive. + +## Argument-by-argument + +| Flag | Value | Meaning | +|------|-------|---------| +| `-m` | `/models/Qwen3.6-27B-UD-Q4_K_XL.gguf` | Model file (single GGUF). | +| `--alias` | `qwen3.6-27b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. | +| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). | +| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). | +| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). | +| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). | +| `-c` | `131072` | Total KV-cache context (131k). Single slot gets the full window. | +| `-np` | `1` | 1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting). | +| `--cont-batching` | *(flag)* | Continuous batching across slots. | +| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model. | +| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). | +| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). | + +## VRAM budget (when live) + +| Component | VRAM | +|-----------------------|-------------| +| Weights (full offload) | ~16 GiB | +| KV cache (q8_0, 131k) | ~16 GiB | +| **Subtotal** | **~32 GiB** | + +Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for +co-locating the 35B-A3B model. + +## Notes for redeployment + +- This was the **deeper-reasoning** model (dense, all params active). The + `qwen3.6-35b-a3b` was the **fast/flash** variant. If redeploying only one, + decide based on latency-vs-quality needs. +- LiteLLM entry that went with it: + ```yaml + - model_name: qwen3.6-27b + litellm_params: + model: openai/qwen3.6-27b + api_base: http://llamacpp-qwen36-27b.llamacpp/v1 + api_key: "sk-no-auth" + ``` +- Consumers at removal time: `platform-engineer` (default model), + `home-manager` (used the 35B-A3B, not this one). See git history for exact + config. +- The `fetch-model` initContainer downloaded the single GGUF idempotently into + the shared models PVC; the new DeepSeek pod's initContainer **deletes** this + GGUF on first boot to reclaim space, so a redeploy will re-download it. diff --git a/llamacpp/args-qwen36-35b-a3b.md b/llamacpp/args-qwen36-35b-a3b.md new file mode 100644 index 0000000..ae18258 --- /dev/null +++ b/llamacpp/args-qwen36-35b-a3b.md @@ -0,0 +1,96 @@ +# Deployment arguments — `qwen3.6-35b-a3b` (REMOVED) + +> **Status:** This model was **removed** from the cluster to make VRAM room for +> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's +> 90 GiB VRAM pool on its own. This file documents the flags used when the +> model was live, so it can be redeployed later if the DeepSeek model is taken +> down or moved to different hardware. +> +> To redeploy: restore `deployment-qwen36-35b-a3b.yaml` (the manifest is +> preserved in git history) and re-add the LiteLLM entry. Re-check the VRAM +> budget — co-locating with the 87 GiB DeepSeek model is **not** possible on +> the current 90 GiB pool. + +## Model & source + +- **Model:** `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` — Qwen3.6-35B-A3B, the "flash" + **Mixture-of-Experts** variant (35B total params, **only 3B active** per + token), MTP variant. +- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~20 GiB**, + single GGUF file (`Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf`). +- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF` +- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan + driver, supports Radeon 8060S / RDNA 3.5). + +## Hardware target + +NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified +memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector: +{kubernetes.io/arch: amd64, hardware: high-memory}`. + +## Why it was the "flash" model + +Despite having **more total parameters** than the dense 27B, only **3B are +active per token** (MoE), so inference is significantly faster. The full ~20 GiB +of Q4 weights is still loaded into VRAM, but only a small fraction is computed +per token. Its KV cache is also tiny (~72 KiB/token), so large context is nearly +free — hence the much larger `-c` and the 2-slot split. + +## Argument-by-argument + +| Flag | Value | Meaning | +|------|-------|---------| +| `-m` | `/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` | Model file (single GGUF). | +| `--alias` | `qwen3.6-35b-a3b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. | +| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). | +| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). | +| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). | +| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). | +| `-c` | `262144` | Total KV-cache context (262k), **split across parallel slots**. | +| `-np` | `2` | 2 parallel slots ⇒ 131k tokens per concurrent request (262k / 2). MoE KV is cheap, so splitting is affordable. | +| `--cont-batching` | *(flag)* | Continuous batching across the 2 slots. | +| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB at 262k); frees headroom for the large `-c`. Drop to `q4_0` for even less VRAM if retrieval quality allows. | +| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). | +| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). | + +## VRAM budget (when live) + +| Component | VRAM | +|-----------------------|------------| +| Weights (full offload) | ~20 GiB | +| KV cache (q8_0, 262k) | ~9 GiB | +| **Subtotal** | **~29 GiB** | + +Left ~67 GiB of headroom on the 96 GiB pool — comfortable, and the basis for +co-locating the dense 27B model (combined ~61 GiB). + +## Combined VRAM budget (both Qwen models, when live) + +| Model | Weights | KV cache | Subtotal | +|-------------------|----------|-------------------------------------------|----------| +| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB | +| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB | +| **Total** | | | **~61 GiB** | + +~35 GiB headroom on the 96 GiB pool — comfortable. (DeepSeek-V4-Flash-0731 at +~87 GiB cannot coexist with either of these; that's why both were removed.) + +## Notes for redeployment + +- This was the **fast/flash** model (MoE, 3B active). The `qwen3.6-27b` was the + **deeper-reasoning** model (dense, all params active). If redeploying only + one, decide based on latency-vs-quality needs. +- LiteLLM entry that went with it: + ```yaml + - model_name: qwen3.6-35b-a3b + litellm_params: + model: openai/qwen3.6-35b-a3b + api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1 + api_key: "sk-no-auth" + ``` +- Consumers at removal time: `home-manager` (default + auxiliary), + `platform-engineer` (auxiliary compression + title generation). See git + history for exact config. +- The `fetch-model` initContainer downloaded the single GGUF idempotently into + the shared models PVC; the new DeepSeek pod's initContainer **deletes** this + GGUF on first boot to reclaim space, so a redeploy will re-download it. diff --git a/llamacpp/deployment-deepseek-v4-flash-0731.yaml b/llamacpp/deployment-deepseek-v4-flash-0731.yaml new file mode 100644 index 0000000..bf66749 --- /dev/null +++ b/llamacpp/deployment-deepseek-v4-flash-0731.yaml @@ -0,0 +1,210 @@ +# DeepSeek-V4-Flash-0731 (MoE: 256 experts / 6 active, UD-IQ1_M ≈ 87 GiB) +# served by llama.cpp's llama-server on the NUCBox APU. +# +# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S, +# ~120 GiB unified memory (≈90 GiB VRAM / 30 GiB CPU RAM via firmware). The +# IQ1_M model (~87 GiB) is *almost* the size of the whole VRAM pool, so it +# CANNOT be fully offloaded to the GPU: offloading all 43 layers + the KV +# cache + Vulkan compute buffers would overflow 90 GiB. Instead we offload +# 40 of 43 layers (-ngl 40) and keep the last 3 (~6 GiB) on CPU RAM, leaving +# ~8 GiB of VRAM headroom for the KV cache, compute buffers, and co-resident +# pods. This is the only model served on the NUCBox — the two Qwen3.6 models +# were removed to make room (their GGUF files should be deleted from the PVC, +# which the initContainer below does on first boot). +# +# KV cache is tiny thanks to DeepSeek-V4's MLA attention (num_kv_heads=1, +# head_dim=512 + 64 decoupled RoPE ⇒ ~576 elements/token/layer). At 64k +# context, q8_0 KV is only ~1.6 GiB, so context is cheap — but we cap -c at +# 65536 (the required minimum) to maximise VRAM headroom, not because KV is +# the constraint. +# +# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan +# driver, which supports the Radeon 8060S (RDNA 3.5). The Vulkan backend +# supports the IQ1_M matmul (incl. the MoE matmul_id variant), so the whole +# model runs on the GPU. deepseek4 is a brand-new arch (2026-07) so the +# floating `server-vulkan` tag is used to pull a recent enough build; pin to a +# specific server-vulkan-bXXXX tag once a known-good one is verified. +# +# GPU access: the container mounts /dev/dri (the DRM render nodes) and runs +# privileged — the simplest reliable option on k3s without a Vulkan device +# plugin. +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: llamacpp-deepseek-v4-flash-0731 + namespace: llamacpp + labels: + app: llamacpp + model: deepseek-v4-flash-0731 +spec: + replicas: 1 + strategy: + type: Recreate # never run two pods loading the same model into VRAM + selector: + matchLabels: + app: llamacpp + model: deepseek-v4-flash-0731 + template: + metadata: + labels: + app: llamacpp + model: deepseek-v4-flash-0731 + spec: + nodeSelector: + kubernetes.io/arch: amd64 + hardware: high-memory + initContainers: + # Idempotently download the (3-part, split) GGUF into the shared models + # PVC on first boot. Also removes the retired Qwen3.6 GGUFs so the new + # 87 GiB model fits on the PVC alongside any other data. Exits + # immediately if the first shard is already present (pod restart). + - name: fetch-model + image: alpine:3.20 + command: ["/bin/sh", "-c"] + args: + - | + set -e + # Reclaim space from the retired Qwen3.6 models (their Deployments + # are gone; the GGUFs are dead weight on the shared PVC). + for old in Qwen3.6-27B-UD-Q4_K_XL.gguf Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf; do + if [ -f "/models/$old" ]; then + echo "Removing retired model $old ..." + rm -f "/models/$old" + fi + done + # Download any missing shards of the split UD-IQ1_M GGUF. + if [ -f "/models/$SHARD1" ]; then + echo "First shard $SHARD1 already present — skipping download." + exit 0 + fi + echo "Installing curl..." + apk add --no-cache curl + for s in "$SHARD1" "$SHARD2" "$SHARD3"; do + echo "Downloading $s from $HF_REPO ..." + curl -fL --retry 5 --retry-delay 5 -o "/models/$s" "$HF_REPO/$s" + done + echo "Download complete:" + ls -lh /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-*.gguf + env: + - name: HF_REPO + value: "https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF/resolve/main/UD-IQ1_M" + - name: SHARD1 + value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf" + - name: SHARD2 + value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf" + - name: SHARD3 + value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf" + volumeMounts: + - name: models + mountPath: /models + containers: + - name: llama-server + image: ghcr.io/ggml-org/llama.cpp:server-vulkan + imagePullPolicy: IfNotPresent + # llama.cpp's CLI parser does NOT split on '=' — every value flag must + # be a separate argv element (flag, then value). See common/arg.cpp. + args: + - -m # model file (first shard; llama.cpp auto-loads the rest) + - /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf + - --alias # /v1/models reports this name; matches the litellm alias + - deepseek-v4-flash-0731 + - --host + - 0.0.0.0 + - --port + - "8080" + - --jinja # use the GGUF's DeepSeek-V4 chat template (DSML / thinking) + - -ngl # offload 40 of 43 layers to the GPU. The model (~87 GiB) is + - "40" # nearly the whole 90 GiB VRAM pool, so full offload (-ngl 999) + # would overflow once KV cache + Vulkan compute buffers are + # added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB of + # VRAM headroom for the KV cache, compute buffers, and + # co-resident pods. Raise toward 43 if VRAM allows; lower + # (e.g. 38) if the pod OOMs / Vulkan runs out of device mem. + - -c # total KV-cache context (single slot gets the full window). + - "65536" # 64k — the required minimum. MLA KV is tiny (~1.6 GiB at + # q8_0), so context is cheap; -c is capped at the minimum to + # maximise VRAM headroom, not because KV is the constraint. + # Raise if VRAM headroom allows. + - -np # 1 slot => the full 64k goes to a single concurrent request + - "1" # (extra slots would multiply KV VRAM, which is fine here, but + # 1 slot keeps it simple and headroom maximal). + - --cont-batching # continuous batching across slots + - --cache-type-k # quantize KV cache to q8_0 — MLA KV is already small (~576 + - q8_0 # elem/token/layer); q8_0 halves it to ~1.6 GiB at 64k and + - --cache-type-v # maximises VRAM headroom with ~negligible quality loss. + - q8_0 + - --temp # default sampling temperature (DeepSeek-V4 recommendation) + - "1.0" + - --top-p # default nucleus sampling threshold (DeepSeek-V4 recommendation) + - "0.95" + - --threads # CPU threads for sampling + the 3 CPU-resident layers + - "8" + ports: + - name: http + containerPort: 8080 + resources: + # The model weights + KV cache live in GPU VRAM (~90 GiB pool) and + # are NOT counted against the cgroup memory limit. This limit only + # covers CPU-side overhead + the mmap'd GGUF pages for the 3 + # CPU-resident layers (~6 GiB) plus reclaimable page cache during + # load. k8s sees ~30 GiB as the node's allocatable system RAM, so the + # limit is sized to cover the CPU layers + overhead while leaving + # RAM for co-resident pods (litellm, the agents, etc.). If the pod is + # OOM-killed during model load, raise the limit. + requests: + cpu: "1000m" + memory: "4Gi" + limits: + cpu: "4000m" + memory: "20Gi" + readinessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 30 + periodSeconds: 10 + failureThreshold: 6 + livenessProbe: + httpGet: + path: /health + port: 8080 + initialDelaySeconds: 300 # 87 GiB load + Vulkan init takes several minutes + periodSeconds: 30 + failureThreshold: 5 + securityContext: + # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is + # the simplest reliable path on k3s without a device plugin. + privileged: true + volumeMounts: + - name: models + mountPath: /models + readOnly: true + - name: dri + mountPath: /dev/dri + volumes: + - name: models + persistentVolumeClaim: + claimName: llamacpp-models + - name: dri + hostPath: + path: /dev/dri + type: Directory +--- +apiVersion: v1 +kind: Service +metadata: + name: llamacpp-deepseek-v4-flash-0731 + namespace: llamacpp + labels: + app: llamacpp + model: deepseek-v4-flash-0731 +spec: + type: ClusterIP + selector: + app: llamacpp + model: deepseek-v4-flash-0731 + ports: + - name: http + port: 80 + targetPort: 8080 diff --git a/llamacpp/deployment-qwen36-27b.yaml b/llamacpp/deployment-qwen36-27b.yaml deleted file mode 100644 index b4d2caa..0000000 --- a/llamacpp/deployment-qwen36-27b.yaml +++ /dev/null @@ -1,163 +0,0 @@ -# Qwen3.6-27B (dense, Q4_K_XL) served by llama.cpp's llama-server on the NUCBox APU. -# -# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S, -# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). The Q4 model -# (~16 GiB) is fully offloaded to the GPU via the Vulkan backend. -# -# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan -# driver, which supports the Radeon 8060S (RDNA 3.5). The project moved from the -# legacy `ggerganov/llama.cpp` namespace (which only has light/full tags) to -# `ggml-org/llama.cpp` (server-vulkan + pinned build tags like server-vulkan-bXXXX). -# Pin to a build tag (e.g. server-vulkan-b4738) for production reproducibility. -# -# GPU access: the container mounts /dev/dri (the DRM render nodes) and runs -# privileged. This is the simplest reliable option on k3s without a Vulkan -# device plugin; tighten later with supplementalGroups if desired. ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: llamacpp-qwen36-27b - namespace: llamacpp - labels: - app: llamacpp - model: qwen3.6-27b -spec: - replicas: 1 - strategy: - type: Recreate # never run two pods loading the same model into VRAM - selector: - matchLabels: - app: llamacpp - model: qwen3.6-27b - template: - metadata: - labels: - app: llamacpp - model: qwen3.6-27b - spec: - nodeSelector: - kubernetes.io/arch: amd64 - hardware: high-memory - initContainers: - # Idempotently download the GGUF into the shared models PVC on first boot. - # Exits immediately if the file is already present (pod restart / recreate). - - name: fetch-model - image: alpine:3.20 - command: ["/bin/sh", "-c"] - args: - - | - set -e - if [ -f "/models/$MODEL_FILE" ]; then - echo "Model $MODEL_FILE already present — skipping download." - exit 0 - fi - echo "Installing curl..." - apk add --no-cache curl - echo "Downloading $MODEL_FILE from $MODEL_URL ..." - curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL" - echo "Download complete: $(ls -lh /models/$MODEL_FILE)" - env: - - name: MODEL_URL - value: "https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-UD-Q4_K_XL.gguf" - - name: MODEL_FILE - value: "Qwen3.6-27B-UD-Q4_K_XL.gguf" - volumeMounts: - - name: models - mountPath: /models - containers: - - name: llama-server - image: ghcr.io/ggml-org/llama.cpp:server-vulkan - imagePullPolicy: IfNotPresent - # llama.cpp's CLI parser does NOT split on '=' — every value flag must be a - # separate argv element (flag, then value). See common/arg.cpp in the repo. - args: - - -m # model file - - /models/Qwen3.6-27B-UD-Q4_K_XL.gguf - - --alias # /v1/models reports this name; matches the litellm alias - - qwen3.6-27b - - --host - - 0.0.0.0 - - --port - - "8080" - - --jinja # use the GGUF's chat template (Qwen3 thinking format) - - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - - "999" - - -c # total KV-cache context (single slot gets the full window) - - "131072" - - -np # 1 slot => 131072 tokens for the single request (no split) - - "1" - - --cont-batching # continuous batching across slots - - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB - - q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c - - --cache-type-v - - q8_0 - - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting) - - "8" - ports: - - name: http - containerPort: 8080 - resources: - # The model weights + KV cache live in GPU VRAM (96 GiB pool) and are - # NOT counted against the cgroup memory limit. This limit only covers - # CPU-side overhead + the mmap'd GGUF file pages during load (~16 GiB, - # reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM, - # so the request is kept low to stay schedulable alongside other pods. - # If the pod OOM-kills during load, the amdgpu driver may be counting - # some VRAM against the cgroup — raise the limit. - requests: - cpu: "1000m" - memory: "2Gi" - limits: - cpu: "4000m" - memory: "24Gi" - readinessProbe: - httpGet: - path: /health - port: 8080 - initialDelaySeconds: 30 - periodSeconds: 10 - failureThreshold: 6 - livenessProbe: - httpGet: - path: /health - port: 8080 - initialDelaySeconds: 180 # model load + Vulkan init can take a few minutes - periodSeconds: 30 - failureThreshold: 5 - securityContext: - # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the - # simplest reliable path on k3s without a device plugin. - privileged: true - volumeMounts: - - name: models - mountPath: /models - readOnly: true - - name: dri - mountPath: /dev/dri - volumes: - - name: models - persistentVolumeClaim: - claimName: llamacpp-models - - name: dri - hostPath: - path: /dev/dri - type: Directory ---- -apiVersion: v1 -kind: Service -metadata: - name: llamacpp-qwen36-27b - namespace: llamacpp - labels: - app: llamacpp - model: qwen3.6-27b -spec: - type: ClusterIP - selector: - app: llamacpp - model: qwen3.6-27b - ports: - - name: http - port: 80 - targetPort: 8080 diff --git a/llamacpp/deployment-qwen36-35b-a3b.yaml b/llamacpp/deployment-qwen36-35b-a3b.yaml deleted file mode 100644 index c2d50a0..0000000 --- a/llamacpp/deployment-qwen36-35b-a3b.yaml +++ /dev/null @@ -1,162 +0,0 @@ -# Qwen3.6-35B-A3B (MoE: 35B total / 3B active, Q4_K_XL) — the "flash" variant. -# -# Despite having more total parameters than the 27B dense model, only 3B are -# active per token (Mixture-of-Experts), so inference is much faster. The full -# ~20 GiB of Q4 weights is still loaded into VRAM but only a small fraction is -# computed per token. -# -# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S, -# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). Full GPU -# offload via the Vulkan backend. Shares the 96 GiB VRAM pool with the 27B -# model — see llamacpp/README.md for the combined VRAM budget. -# -# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (Mesa/RADV Vulkan driver, -# supports the Radeon 8060S / RDNA 3.5). Pin to a build tag for production. -# -# GPU access: mounts /dev/dri + privileged (simplest reliable path on k3s). ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: llamacpp-qwen36-35b-a3b - namespace: llamacpp - labels: - app: llamacpp - model: qwen3.6-35b-a3b -spec: - replicas: 1 - strategy: - type: Recreate # never run two pods loading the same model into VRAM - selector: - matchLabels: - app: llamacpp - model: qwen3.6-35b-a3b - template: - metadata: - labels: - app: llamacpp - model: qwen3.6-35b-a3b - spec: - nodeSelector: - kubernetes.io/arch: amd64 - hardware: high-memory - initContainers: - # Idempotently download the GGUF into the shared models PVC on first boot. - # Exits immediately if the file is already present (pod restart / recreate). - - name: fetch-model - image: alpine:3.20 - command: ["/bin/sh", "-c"] - args: - - | - set -e - if [ -f "/models/$MODEL_FILE" ]; then - echo "Model $MODEL_FILE already present — skipping download." - exit 0 - fi - echo "Installing curl..." - apk add --no-cache curl - echo "Downloading $MODEL_FILE from $MODEL_URL ..." - curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL" - echo "Download complete: $(ls -lh /models/$MODEL_FILE)" - env: - - name: MODEL_URL - value: "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" - - name: MODEL_FILE - value: "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" - volumeMounts: - - name: models - mountPath: /models - containers: - - name: llama-server - image: ghcr.io/ggml-org/llama.cpp:server-vulkan - imagePullPolicy: IfNotPresent - # llama.cpp's CLI parser does NOT split on '=' — every value flag must be a - # separate argv element (flag, then value). See common/arg.cpp in the repo. - args: - - -m # model file - - /models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf - - --alias # /v1/models reports this name; matches the litellm alias - - qwen3.6-35b-a3b - - --host - - 0.0.0.0 - - --port - - "8080" - - --jinja # use the GGUF's chat template (Qwen3 thinking format) - - -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM) - - "999" - - -c # total KV-cache context, split across parallel slots - - "262144" - - -np # 2 parallel slots => 131072 tokens per concurrent request - - "2" - - --cont-batching # continuous batching across slots - - --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB - - q8_0 # at 262k ctx); ~negligible quality loss, frees headroom for large -c - - --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows) - - q8_0 - - --threads # CPU threads for sampling/overhead (GPU does the heavy lifting) - - "8" - ports: - - name: http - containerPort: 8080 - resources: - # The model weights + KV cache live in GPU VRAM (96 GiB pool) and are - # NOT counted against the cgroup memory limit. This limit only covers - # CPU-side overhead + the mmap'd GGUF file pages during load (~20 GiB, - # reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM. - # If the pod OOM-kills during load, raise the limit. - requests: - cpu: "1000m" - memory: "2Gi" - limits: - cpu: "4000m" - memory: "24Gi" - readinessProbe: - httpGet: - path: /health - port: 8080 - initialDelaySeconds: 30 - periodSeconds: 10 - failureThreshold: 6 - livenessProbe: - httpGet: - path: /health - port: 8080 - initialDelaySeconds: 180 # model load + Vulkan init can take a few minutes - periodSeconds: 30 - failureThreshold: 5 - securityContext: - # Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the - # simplest reliable path on k3s without a device plugin. - privileged: true - volumeMounts: - - name: models - mountPath: /models - readOnly: true - - name: dri - mountPath: /dev/dri - volumes: - - name: models - persistentVolumeClaim: - claimName: llamacpp-models - - name: dri - hostPath: - path: /dev/dri - type: Directory ---- -apiVersion: v1 -kind: Service -metadata: - name: llamacpp-qwen36-35b-a3b - namespace: llamacpp - labels: - app: llamacpp - model: qwen3.6-35b-a3b -spec: - type: ClusterIP - selector: - app: llamacpp - model: qwen3.6-35b-a3b - ports: - - name: http - port: 80 - targetPort: 8080 diff --git a/llamacpp/pv.yaml b/llamacpp/pv.yaml index a0a8063..2003690 100644 --- a/llamacpp/pv.yaml +++ b/llamacpp/pv.yaml @@ -17,7 +17,7 @@ metadata: spec: storageClassName: manual capacity: - storage: 100Gi + storage: 200Gi accessModes: - ReadWriteMany hostPath: @@ -44,4 +44,4 @@ spec: - ReadWriteMany resources: requests: - storage: 100Gi + storage: 200Gi diff --git a/platform-engineer/README.md b/platform-engineer/README.md index 459aea9..60e01fe 100644 --- a/platform-engineer/README.md +++ b/platform-engineer/README.md @@ -299,11 +299,12 @@ Design rules baked into SOUL.md: ## 11. Deployment checklist (do in this order) -1. **Add the Ollama model to LiteLLM** (already done in `litellm/litellm.yaml`): - the `qwen-3.6:27b` entry points at `http://10.88.20.12:11434`. Make sure - `qwen3.6:27b` is actually pulled on that Ollama host - (`ollama pull qwen3.6:27b`). Apply: `kubectl apply -f litellm/` and restart - the LiteLLM pod so the new config takes effect. +1. **Add the in-cluster model to LiteLLM** (already done in `litellm/litellm.yaml`): + the `deepseek-v4-flash-0731` entry points at the in-cluster llama.cpp server + `http://llamacpp-deepseek-v4-flash-0731.llamacpp/v1` (served from the NUCBox + APU; see `llamacpp/deployment-deepseek-v4-flash-0731.yaml`). Apply: + `kubectl apply -f litellm/` and restart the LiteLLM pod so the new config + takes effect. 2. **Create the `gitea-registry` secret in the new namespace** (ArgoCD won't create it — it's not in the repo): ``` diff --git a/platform-engineer/configmap.yaml b/platform-engineer/configmap.yaml index 08ec717..672d7f7 100644 --- a/platform-engineer/configmap.yaml +++ b/platform-engineer/configmap.yaml @@ -9,18 +9,18 @@ data: config.yaml: | model: provider: openai-api - default: qwen3.6-27b + default: deepseek-v4-flash-0731 base_url: "http://litellm-service.litellm:80/v1" api_mode: chat_completions auxiliary: compression: provider: openai-api - model: qwen3.6-35b-a3b # flash model for speed-sensitive auxiliary tasks + model: deepseek-v4-flash-0731 # in-cluster llama.cpp model (replaces the Qwen flash/dense pair) base_url: "http://litellm-service.litellm:80/v1" title_generation: provider: openai-api - model: qwen3.6-35b-a3b # flash model for speed-sensitive auxiliary tasks + model: deepseek-v4-flash-0731 # in-cluster llama.cpp model (replaces the Qwen flash/dense pair) base_url: "http://litellm-service.litellm:80/v1" terminal: diff --git a/platform-engineer/cron-seed.yaml b/platform-engineer/cron-seed.yaml index cc1e941..a7681f3 100644 --- a/platform-engineer/cron-seed.yaml +++ b/platform-engineer/cron-seed.yaml @@ -56,7 +56,7 @@ spec: # NOTE: `hermes cron create` has no --model flag. New jobs inherit the # global default model from config.yaml at creation time. To pin a - # specific model per-job (e.g. the flash model qwen3.6-35b-a3b for all + # specific model per-job (e.g. deepseek-v4-flash-0731 for all # cron tasks), edit /opt/data/cron/jobs.json directly after seeding: # kubectl exec deploy/hermes -- python3 -c "...set model field..." # See llamacpp/README.md and the deployment notes for details.