roger/k3s-cluster
1
0
Fork 1
Code Issues Pull Requests 6 Actions Packages Projects Releases Wiki Activity

set platform engineer url to platform-engineer.rogi.casa

Browse Source
This commit is contained in:
Roger Oriol
2026-07-23 23:57:41 +02:00
parent 04bd4d8e5e
commit 496e46076c
7 changed files with 31 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
llamacpp/README.md
View File

@@ -16,7 +16,7 @@ to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}
| Alias | Model | GGUF | Service |
|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
| `qwen3.6-27b` | Qwen3.6-27B (dense) | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB) | `llamacpp-qwen36-27b.llamacpp:80` |
| `qwen3.6-36b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-36b-a3b.llamacpp:80` |
| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-35b-a3b.llamacpp:80` |
The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
inference is significantly faster than the dense 27B despite more total weights —
@@ -42,7 +42,7 @@ group GID) if desired.
```bash
kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-36b-a3b | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b | grep -iE 'vulkan|gpu|offload|device'
```
If only a CPU device shows up, the container can't see the GPU — check that
@@ -55,7 +55,7 @@ Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
| Model | Weights | KV cache (32k×4) | Subtotal |
|-------------------|----------|-------------------|----------|
| qwen3.6-27b | ~16 GiB | ~34 GiB | ~50 GiB |
| qwen3.6-36b-a3b | ~20 GiB | ~10 GiB | ~30 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~10 GiB | ~30 GiB |
| **Total** | | | **~80 GiB** |
~16 GiB headroom — comfortable but not infinite. If VRAM is exhausted ( Vulkan

16
llamacpp/deployment-qwen36-36b-a3b.yaml → llamacpp/deployment-qwen36-35b-a3b.yaml
View File

@@ -18,11 +18,11 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: llamacpp-qwen36-36b-a3b
name: llamacpp-qwen36-35b-a3b
namespace: llamacpp
labels:
app: llamacpp
model: qwen3.6-36b-a3b
model: qwen3.6-35b-a3b
spec:
replicas: 1
strategy:
@@ -30,12 +30,12 @@ spec:
selector:
matchLabels:
app: llamacpp
model: qwen3.6-36b-a3b
model: qwen3.6-35b-a3b
template:
metadata:
labels:
app: llamacpp
model: qwen3.6-36b-a3b
model: qwen3.6-35b-a3b
spec:
nodeSelector:
kubernetes.io/arch: amd64
@@ -76,7 +76,7 @@ spec:
- -m # model file
- /models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
- --alias # /v1/models reports this name; matches the litellm alias
- qwen3.6-36b-a3b
- qwen3.6-35b-a3b
- --host
- 0.0.0.0
- --port
@@ -142,16 +142,16 @@ spec:
apiVersion: v1
kind: Service
metadata:
name: llamacpp-qwen36-36b-a3b
name: llamacpp-qwen36-35b-a3b
namespace: llamacpp
labels:
app: llamacpp
model: qwen3.6-36b-a3b
model: qwen3.6-35b-a3b
spec:
type: ClusterIP
selector:
app: llamacpp
model: qwen3.6-36b-a3b
model: qwen3.6-35b-a3b
ports:
- name: http
port: 80