roger/k3s-cluster
1
0
Fork 1
Code Issues Pull Requests 6 Actions Packages Projects Releases Wiki Activity

deploy both qwen3.6 versions

Browse Source
This commit is contained in:
Roger Oriol
2026-07-23 22:10:44 +02:00
parent 8ef9db1e37
commit 04bd4d8e5e
16 changed files with 906 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
llamacpp/README.md
View File

@@ -13,9 +13,15 @@ in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
| Model | GGUF | Service | litellm alias |
|----------|---------------------------------------------|----------------------------------|---------------|
| qwen3.6 | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL) | `llamacpp-qwen36.llamacpp:80` | `qwen3.6` |
| Alias | Model | GGUF | Service |
|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
| `qwen3.6-27b` | Qwen3.6-27B (dense) | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB) | `llamacpp-qwen36-27b.llamacpp:80` |
| `qwen3.6-36b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-36b-a3b.llamacpp:80` |
The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
inference is significantly faster than the dense 27B despite more total weights —
hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for
deeper reasoning.
Model files are downloaded idempotently by an initContainer into a shared
hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
@@ -24,9 +30,8 @@ without re-downloading.
## GPU / Vulkan
The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
the Mesa/RADV Vulkan driver, which supports
the Radeon 8060S (RDNA 3.5). Full layer offload (`-ngl 999`) puts the ~16 GiB
Q4 model entirely in the 96 GiB VRAM pool.
the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full
layer offload (`-ngl 999`) puts model weights entirely in the 96 GiB VRAM pool.
The container mounts `/dev/dri` and runs `privileged: true` — the simplest
reliable way to give Vulkan access to the DRM render node on k3s without a
@@ -36,26 +41,38 @@ group GID) if desired.
### Verify the GPU is actually used
```bash
kubectl exec -n llamacpp deploy/llamacpp-qwen36 -- \
llama-server --list-devices -m /models/Qwen3.6-27B-UD-Q4_K_XL.gguf
# or check the startup logs for a "vulkan" device line + ngl offload count
kubectl logs -n llamacpp deploy/llamacpp-qwen36 | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-36b-a3b | grep -iE 'vulkan|gpu|offload|device'
```
If only a CPU device shows up, the container can't see the GPU — check that
`/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
## VRAM budget (both models co-resident)
Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
| Model | Weights | KV cache (32k×4) | Subtotal |
|-------------------|----------|-------------------|----------|
| qwen3.6-27b | ~16 GiB | ~34 GiB | ~50 GiB |
| qwen3.6-36b-a3b | ~20 GiB | ~10 GiB | ~30 GiB |
| **Total** | | | **~80 GiB** |
~16 GiB headroom — comfortable but not infinite. If VRAM is exhausted ( Vulkan
allocation failures in logs), reduce `-c` on the 27B (its KV cache dominates) or
drop `-np` to 2 on either model.
## Tuning
The key knobs (in `deployment-qwen36.yaml`):
The key knobs (in each `deployment-*.yaml`):
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight (it
isn't, with 96 GiB).
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
- `-c 32768` — total KV-cache context. With `-np 4` this is 8192 tokens per
concurrent request. For a 27B model the full 32k×4 KV cache is ~32 GiB of
VRAM; raise or lower `-c` to trade context length for VRAM headroom.
- `-np 4` — parallel slots (concurrent requests). Matches the requested
concurrency. Each extra slot multiplies KV-cache VRAM usage.
concurrent request. The 27B's dense KV cache is the larger consumer (~34 GiB
at 32k×4); the MoE's is much smaller (~10 GiB).
- `-np 4` — parallel slots (concurrent requests). Each extra slot multiplies
KV-cache VRAM usage. Bump higher on the flash model if you need more
throughput (it has VRAM headroom).
- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
full GPU offload; tune if CPU-bound.
@@ -69,11 +86,11 @@ during load. If the pod is OOM-killed during model load, raise the memory limit.
## Adding a model
1. Create `deployment-<model>.yaml` + `service-<model>.yaml` (copy the qwen3.6
pair; change `model:` label, the GGUF URL/file, `--alias`, and Service name).
1. Copy `deployment-qwen36-27b.yaml` → `deployment-<new>.yaml`; change the
`model:` label, GGUF URL/file, `--alias`, and Service name.
2. Point LiteLLM at it in `litellm/litellm.yaml`:
```yaml
- model_name: <alias> # keep the alias so consumers don't change
- model_name: <alias>
litellm_params:
model: openai/<alias>
api_base: http://<service>.llamacpp/v1
@@ -81,9 +98,4 @@ during load. If the pod is OOM-killed during model load, raise the memory limit.
```
3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
directory recursively.)
## TODO
- `glm-4.7-flash`: still served by the external Ollama at `10.88.20.12:11434`
in `litellm/litellm.yaml`. Migrate once a GGUF source is confirmed (add a
`deployment-glm47-flash.yaml` + Service and flip the litellm entry).
4. Check the VRAM budget table above — two large models may not coexist.