new llamacpp service

llamacpp/README.md

@@ -0,0 +1,88 @@
+# llama.cpp (llama-server)
+
+In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on
+the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified
+memory: 32 GiB RAM / 96 GiB VRAM).
+
+This replaces the bare-metal Ollama setup for models that benefit from
+always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these
+in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
+
+## Layout
+
+One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
+to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
+
+| Model    | GGUF                                        | Service                          | litellm alias |
+|----------|---------------------------------------------|----------------------------------|---------------|
+| qwen3.6  | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL)      | `llamacpp-qwen36.llamacpp:80`    | `qwen3.6`     |
+
+Model files are downloaded idempotently by an initContainer into a shared
+hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
+without re-downloading.
+
+## GPU / Vulkan
+
+The `server-vulkan` image bundles the Mesa/RADV Vulkan driver, which supports
+the Radeon 8060S (RDNA 3.5). Full layer offload (`-ngl 999`) puts the ~16 GiB
+Q4 model entirely in the 96 GiB VRAM pool.
+
+The container mounts `/dev/dri` and runs `privileged: true` — the simplest
+reliable way to give Vulkan access to the DRM render node on k3s without a
+device plugin. Tighten later with `supplementalGroups` (the host's `render`
+group GID) if desired.
+
+### Verify the GPU is actually used
+
+```bash
+kubectl exec -n llamacpp deploy/llamacpp-qwen36 -- \
+  llama-server --list-devices -m /models/Qwen3.6-27B-UD-Q4_K_XL.gguf
+# or check the startup logs for a "vulkan" device line + ngl offload count
+kubectl logs -n llamacpp deploy/llamacpp-qwen36 | grep -iE 'vulkan|gpu|offload|device'
+```
+
+If only a CPU device shows up, the container can't see the GPU — check that
+`/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
+
+## Tuning
+
+The key knobs (in `deployment-qwen36.yaml`):
+
+- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight (it
+  isn't, with 96 GiB).
+- `-c 32768` — total KV-cache context. With `-np 4` this is 8192 tokens per
+  concurrent request. For a 27B model the full 32k×4 KV cache is ~32 GiB of
+  VRAM; raise or lower `-c` to trade context length for VRAM headroom.
+- `-np 4` — parallel slots (concurrent requests). Matches the requested
+  concurrency. Each extra slot multiplies KV-cache VRAM usage.
+- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
+  full GPU offload; tune if CPU-bound.
+
+## Memory accounting
+
+k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is
+reserved by firmware and managed by `amdgpu`). The model weights and KV cache
+live in VRAM and are **not** counted against the container's cgroup memory
+limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages
+during load. If the pod is OOM-killed during model load, raise the memory limit.
+
+## Adding a model
+
+1. Create `deployment-<model>.yaml` + `service-<model>.yaml` (copy the qwen3.6
+   pair; change `model:` label, the GGUF URL/file, `--alias`, and Service name).
+2. Point LiteLLM at it in `litellm/litellm.yaml`:
+   ```yaml
+   - model_name: <alias>            # keep the alias so consumers don't change
+     litellm_params:
+       model: openai/<alias>
+       api_base: http://<service>.llamacpp/v1
+       api_key: "sk-no-auth"
+   ```
+3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
+   directory recursively.)
+
+## TODO
+
+- `glm-4.7-flash`: still served by the external Ollama at `10.88.20.12:11434`
+  in `litellm/litellm.yaml`. Migrate once a GGUF source is confirmed (add a
+  `deployment-glm47-flash.yaml` + Service and flip the litellm entry).