platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

set platform engineer url to platform-engineer.rogi.casa

Browse Source
This commit is contained in:
Roger Oriol
2026-07-23 23:57:41 +02:00
parent 04bd4d8e5e
commit 496e46076c
7 changed files with 31 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
llamacpp/README.md
View File

@@ -16,7 +16,7 @@ to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}
| Alias | Model | GGUF | Service |
|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
| `qwen3.6-27b` | Qwen3.6-27B (dense) | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB) | `llamacpp-qwen36-27b.llamacpp:80` |
| `qwen3.6-36b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-36b-a3b.llamacpp:80` |
| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-35b-a3b.llamacpp:80` |
The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
inference is significantly faster than the dense 27B despite more total weights —
@@ -42,7 +42,7 @@ group GID) if desired.
```bash
kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-36b-a3b | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b | grep -iE 'vulkan|gpu|offload|device'
```
If only a CPU device shows up, the container can't see the GPU — check that
@@ -55,7 +55,7 @@ Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
| Model | Weights | KV cache (32k×4) | Subtotal |
|-------------------|----------|-------------------|----------|
| qwen3.6-27b | ~16 GiB | ~34 GiB | ~50 GiB |
| qwen3.6-36b-a3b | ~20 GiB | ~10 GiB | ~30 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~10 GiB | ~30 GiB |
| **Total** | | | **~80 GiB** |
~16 GiB headroom — comfortable but not infinite. If VRAM is exhausted ( Vulkan