platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

deploy deepseek v4

Browse Source
This commit is contained in:
Roger Oriol
2026-08-01 00:24:26 +02:00
parent 5fe0ee361a
commit b132aae09c
14 changed files with 610 additions and 411 deletions
Download Patch File Download Diff File Expand all files Collapse all files

140
llamacpp/README.md
View File

@@ -1,37 +1,50 @@
# llama.cpp (llama-server)
In-cluster LLM inference via llama.cpp's `llama-server`, serving local models on
the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, 128 GiB unified
memory: 32 GiB RAM / 96 GiB VRAM).
In-cluster LLM inference via llama.cpp's `llama-server`, serving a local model
on the NUCBox APU (AMD Ryzen AI Max 395 / Strix Halo, Radeon 8060S, ~120 GiB
unified memory: ~90 GiB VRAM / 30 GiB CPU RAM).
This replaces the bare-metal Ollama setup for models that benefit from
always-loaded weights + tuned batching. LiteLLM (`litellm/`) points at these
in-cluster Services instead of the external `10.88.20.12:11434` Ollama endpoint.
LiteLLM (`litellm/`) points at these in-cluster Services instead of the
external `10.88.20.12:11434` Ollama endpoint.
## Layout
One Deployment + Service **per model**, all in namespace `llamacpp`, all pinned
to the NUCBox (`nodeSelector: {kubernetes.io/arch: amd64, hardware: high-memory}`):
| Alias | Model | GGUF | Service |
|-------------------|----------------------------|-------------------------------------------------------|-------------------------------------------|
| `qwen3.6-27b` | Qwen3.6-27B (dense) | unsloth/Qwen3.6-27B-MTP-GGUF (Q4_K_XL, ~16 GiB) | `llamacpp-qwen36-27b.llamacpp:80` |
| `qwen3.6-35b-a3b` | Qwen3.6-35B-A3B (MoE/flash)| unsloth/Qwen3.6-35B-A3B-MTP-GGUF (Q4_K_XL, ~20 GiB) | `llamacpp-qwen36-35b-a3b.llamacpp:80` |
| Alias | Model | GGUF | Service | Args ref |
|--------------------------|--------------------------------|-------------------------------------------------------|------------------------------------------------------|----------|
| `deepseek-v4-flash-0731` | DeepSeek-V4-Flash-0731 (MoE) | unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-IQ1_M, ~87 GiB) | `llamacpp-deepseek-v4-flash-0731.llamacpp:80` | [args-deepseek-v4-flash-0731.md](args-deepseek-v4-flash-0731.md) |
The 35B-A3B is a Mixture-of-Experts model (3B active params per token), so
inference is significantly faster than the dense 27B despite more total weights —
hence the "flash" label. Use it for latency-sensitive workloads; use the 27B for
deeper reasoning.
DeepSeek-V4-Flash-0731 is a Mixture-of-Experts model (256 experts, 6 active per
token) with MLA attention, so only a small fraction of the weights is computed
per token. The full ~87 GiB of IQ1_M weights is loaded into the unified memory
pool.
> **Previously** the NUCBox ran two co-resident Qwen3.6 models (a 27B dense and
> a 35B-A3B MoE "flash"). DeepSeek-V4-Flash-0731 (~87 GiB) nearly fills the
> whole 90 GiB VRAM pool on its own, so both Qwen models were removed to make
> room. Their GGUF files are deleted from the shared PVC by the new pod's
> `fetch-model` initContainer on first boot. Their deployment arguments are
> still documented for redeployment:
> - [args-qwen36-27b.md](args-qwen36-27b.md) — dense 27B (deeper reasoning)
> - [args-qwen36-35b-a3b.md](args-qwen36-35b-a3b.md) — MoE 35B-A3B "flash" (fast)
Model files are downloaded idempotently by an initContainer into a shared
hostPath PVC (`/data/llamacpp/models` on the NUCBox), so pods survive reboots
without re-downloading.
without re-downloading. The UD-IQ1_M GGUF is split across 3 shards
(`-00001-of-00003` … `-00003-of-00003`); llama.cpp auto-loads all shards when
pointed at the first one.
## GPU / Vulkan
The `server-vulkan` image (`ghcr.io/ggml-org/llama.cpp:server-vulkan`) bundles
the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). Full
layer offload (`-ngl 999`) puts model weights entirely in the 96 GiB VRAM pool.
the Mesa/RADV Vulkan driver, which supports the Radeon 8060S (RDNA 3.5). The
Vulkan backend supports the `IQ1_M` matmul (including the MoE `matmul_id`
variant), so the whole model runs on the GPU. `deepseek4` is a brand-new arch
(2026-07), so the floating `server-vulkan` tag is used to pull a recent enough
build — pin to a specific `server-vulkan-bXXXX` tag once a known-good one is
verified.
The container mounts `/dev/dri` and runs `privileged: true` — the simplest
reliable way to give Vulkan access to the DRM render node on k3s without a
@@ -41,61 +54,75 @@ group GID) if desired.
### Verify the GPU is actually used
```bash
kubectl logs -n llamacpp deploy/llamacpp-qwen36-27b | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-qwen36-35b-a3b | grep -iE 'vulkan|gpu|offload|device'
kubectl logs -n llamacpp deploy/llamacpp-deepseek-v4-flash-0731 | grep -iE 'vulkan|gpu|offload|device'
```
If only a CPU device shows up, the container can't see the GPU — check that
`/dev/dri/renderD128` exists on the NUCBox and that the `amdgpu` module is loaded.
## VRAM budget (both models co-resident)
## VRAM budget (single model)
Both models run simultaneously on the same 96 GiB VRAM pool. Approximate usage:
The model (~87 GiB IQ1_M) is almost the size of the entire 90 GiB VRAM pool, so
it **cannot be fully offloaded**: `-ngl 999` would try to put all 43 layers into
VRAM and overflow once the KV cache + Vulkan compute buffers are added. Instead
`-ngl 40` offloads 40 of 43 layers to the GPU and keeps the last 3 (~6 GiB) on
CPU RAM, leaving ~8 GiB of VRAM headroom for the KV cache, compute buffers, and
co-resident pods. Approximate VRAM usage:
| Model | Weights | KV cache | Subtotal |
|-------------------|----------|-------------------------------------------|----------|
| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB |
| **Total** | | | **~61 GiB** |
| Component | VRAM |
|---------------------------------|-------------|
| Weights (40 GPU layers) | ~81 GiB |
| KV cache (q8_0, 64k, 1 slot) | ~1.6 GiB |
| Vulkan compute buffers | ~2 GiB |
| **Total in VRAM** | **~85 GiB** |
| **Headroom (of 90 GiB)** | **~5–8 GiB**|
~35 GiB headroom — comfortable. Both models' KV caches are quantized
to q8_0 (halved vs f16, ~negligible quality loss), which makes the dense 27B's
large-context KV affordable (~256 KiB/token f16 → ~128 KiB/token q8_0). The
a3b's KV is tiny (~72 KiB/token) so its large context is nearly free.
3 layers (~6 GiB) live in CPU RAM (counted against the pod's cgroup memory
limit, not VRAM).
KV cache is tiny thanks to DeepSeek-V4's **MLA** attention
(`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
elements/token/layer). At 64k context, q8_0 KV is only ~1.6 GiB, so context is
cheap — `-c` is capped at 65536 (the required minimum) to maximise VRAM
headroom, not because KV is the constraint.
## Tuning
The key knobs (in each `deployment-*.yaml`):
The key knobs (in `deployment-deepseek-v4-flash-0731.yaml`):
- `-ngl 999` — offload all layers to GPU. Reduce only if VRAM is tight.
- `-c` — total KV-cache context. The 27B runs `-c 131072 -np 1` (single slot
gets the full 131k); the a3b runs `-c 262144 -np 2` (2 slots × 131k each).
The MoE a3b's KV cache is ~72 KiB/token so large context is cheap; the dense
27B's is ~256 KiB/token (f16) / ~128 KiB/token (q8_0), which is why the 27B
uses q8_0 KV to keep 131k affordable (~16 GiB) and sticks to 1 slot.
- `-np` — parallel slots (concurrent requests). Each extra slot multiplies
KV-cache VRAM usage. The 27B uses 1 slot (full 131k to the single request,
dense KV is the constraint); the a3b uses 2 slots (131k each, cheap MoE KV).
Bump higher only if you need more concurrent throughput.
- `--cache-type-k q8_0 --cache-type-v q8_0` (both) — quantize the KV cache to
q8_0, halving KV VRAM with ~negligible quality loss. Essential for the dense
27B at 131k (f16 would be ~32 GiB KV alone); nearly free headroom on the a3b.
Drop to q4_0 for even less VRAM if retrieval quality allows.
- `--threads 8` — CPU threads for sampling/overhead. Mostly irrelevant under
full GPU offload; tune if CPU-bound.
- `-ngl 40` — offload 40 of 43 layers to GPU. The model (~87 GiB) is nearly the
whole 90 GiB VRAM pool, so full offload would overflow once KV cache + compute
buffers are added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB headroom.
Raise toward 43 if VRAM allows; lower (e.g. 38) if the pod OOMs / Vulkan runs
out of device memory.
- `-c 65536` — total KV-cache context (64k, the required minimum). 1 slot gets
the full 64k. MLA KV is tiny (~1.6 GiB at q8_0), so context is cheap; capped
at the minimum to maximise VRAM headroom. Raise if headroom allows.
- `-np 1` — 1 parallel slot (the full 64k goes to a single concurrent request).
Extra slots multiply KV VRAM (cheap here), but 1 slot keeps headroom maximal.
- `--cache-type-k q8_0 --cache-type-v q8_0` — quantize the KV cache to q8_0,
halving KV VRAM with ~negligible quality loss. Essential to keep headroom.
- `--temp 1.0 --top-p 0.95` — default sampling parameters (DeepSeek-V4
recommendation). These are server defaults; clients can override per request
via the OpenAI-compatible API.
- `--threads 8` — CPU threads for sampling + the 3 CPU-resident layers.
## Memory accounting
k8s sees only the ~32 GiB system RAM as allocatable (the 96 GiB VRAM is
reserved by firmware and managed by `amdgpu`). The model weights and KV cache
live in VRAM and are **not** counted against the container's cgroup memory
limit — that limit only covers CPU-side overhead and the mmap'd GGUF pages
during load. If the pod is OOM-killed during model load, raise the memory limit.
k8s sees only the ~30 GiB system RAM as allocatable (the ~90 GiB VRAM is
reserved by firmware and managed by `amdgpu`). The GPU-resident model weights
and KV cache live in VRAM and are **not** counted against the container's cgroup
memory limit — that limit only covers CPU-side overhead, the mmap'd GGUF pages
for the 3 CPU-resident layers (~6 GiB), and reclaimable page cache during load.
If the pod is OOM-killed during model load, raise the memory limit (and/or
lower `-ngl`).
## Adding a model
## Adding / replacing a model
1. Copy `deployment-qwen36-27b.yaml` → `deployment-<new>.yaml`; change the
`model:` label, GGUF URL/file, `--alias`, and Service name.
1. Copy `deployment-deepseek-v4-flash-0731.yaml` → `deployment-<new>.yaml`;
change the `model:` label, GGUF URL/file(s), `--alias`, and Service name.
For split GGUFs, point `-m` at the first shard and download all shards in
the `fetch-model` initContainer.
2. Point LiteLLM at it in `litellm/litellm.yaml`:
```yaml
- model_name: <alias>
@@ -106,4 +133,5 @@ during load. If the pod is OOM-killed during model load, raise the memory limit.
```
3. (No gen-apps.sh change needed — the `llamacpp` app already syncs the whole
directory recursively.)
4. Check the VRAM budget table above — two large models may not coexist.
4. Check the VRAM budget table above — at ~87 GiB this model nearly fills the
90 GiB pool on its own, so co-locating another large model is not possible.

109
llamacpp/args-deepseek-v4-flash-0731.md Normal file
View File

@@ -0,0 +1,109 @@
# Deployment arguments — `deepseek-v4-flash-0731`
Reference for the llama-server flags used in
`deployment-deepseek-v4-flash-0731.yaml`. Keep this in sync if the
Deployment is edited.
## Model & source
- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` (DeepSeek-V4-Flash, 0731 weights)
- **Quantization:** `unsloth/DeepSeek-V4-Flash-0731-GGUF` → `UD-IQ1_M`
(Unsloth Dynamic IQ1_M), **split across 3 shards** (~87 GiB total):
- `DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf`
- `DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf`
- `DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf`
- **HuggingFace repo:** `https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF`
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (floating tag — `deepseek4`
is a brand-new arch; pin to a specific `server-vulkan-bXXXX` once verified).
## Hardware target
NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S (RDNA 3.5),
~120 GiB unified memory split by firmware into **~90 GiB VRAM** and
**~30 GiB CPU RAM**. Pinned via `nodeSelector: {kubernetes.io/arch: amd64,
hardware: high-memory}`.
## Why this model nearly fills the machine
The UD-IQ1_M model is **~87 GiB**, almost the entire 90 GiB VRAM pool. Unlike
the old Qwen models (16–20 GiB, fully offloaded with room to spare), this one
**cannot be fully offloaded** to the GPU: putting all 43 layers + the KV cache +
Vulkan compute buffers in VRAM would exceed 90 GiB. The tuning below is all
about fitting the model while leaving headroom for the KV cache, compute
buffers, and co-resident pods.
## Argument-by-argument
| Flag | Value | Meaning |
|------|-------|---------|
| `-m` | `/models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf` | Model file. Pointed at the **first shard** only; llama.cpp auto-discovers and loads `-00002-…` / `-00003-…` from the same directory. |
| `--alias` | `deepseek-v4-flash-0731` | Name reported by `GET /v1/models`. Must match the `model_name` in `litellm/litellm.yaml` so LiteLLM routes to this server. |
| `--host` | `0.0.0.0` | Bind on all interfaces so the k8s Service can reach the pod. |
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
| `--jinja` | *(flag)* | Use the chat template baked into the GGUF (DeepSeek-V4 DSML format, with thinking/reasoning support). Reasoning comes back in `reasoning_content`. |
| `-ngl` | `40` | **GPU layer offload.** Offload 40 of 43 layers to the GPU. See “VRAM budget” below — full offload would OOM. |
| `-c` | `65536` | Total KV-cache context window (64k, the required minimum). Single slot gets the full window. |
| `-np` | `1` | Parallel slots. 1 slot ⇒ the full 64k goes to a single concurrent request. |
| `--cont-batching` | *(flag)* | Continuous batching across slots (no-op with 1 slot, but harmless and correct if `-np` is raised). |
| `--cache-type-k` | `q8_0` | Quantize the K cache to q8_0 (halves KV VRAM, ~negligible quality loss). |
| `--cache-type-v` | `q8_0` | Quantize the V cache to q8_0 (same). |
| `--temp` | `1.0` | Default sampling temperature (DeepSeek-V4 recommendation). Clients may override per request via the OpenAI API. |
| `--top-p` | `0.95` | Default nucleus-sampling threshold (DeepSeek-V4 recommendation). Overrideable per request. |
| `--threads` | `8` | CPU threads for sampling + the 3 CPU-resident layers. Mostly irrelevant under heavy GPU offload. |
## VRAM budget (90 GiB pool)
| Component | VRAM |
|---------------------------------|-------------|
| Weights (40 GPU layers) | ~81 GiB |
| KV cache (q8_0, 64k, 1 slot) | ~1.6 GiB |
| Vulkan compute buffers | ~2 GiB |
| **Total in VRAM** | **~85 GiB** |
| **Headroom (of 90 GiB)** | **~5–8 GiB**|
The remaining **3 layers (~6 GiB) live in CPU RAM** and are counted against the
pod's cgroup memory limit (not VRAM). KV cache is tiny thanks to DeepSeek-V4's
**MLA** attention (`num_kv_heads=1`, `head_dim=512` + 64 decoupled RoPE ⇒ ~576
elements/token/layer); at 64k context q8_0 KV is only ~1.6 GiB, so context is
**not** the constraint — `-c` is capped at the minimum purely to maximise VRAM
headroom.
## How to tune if it OOMs / has spare headroom
- **Pod OOM-killed or Vulkan out-of-device-memory during load:** lower `-ngl`
(e.g. `38`) to keep more layers on CPU, or raise the container `memory`
limit.
- **VRAM headroom looks generous in logs:** raise `-ngl` toward `43` (full
offload) and/or raise `-c` for larger context.
- **Need more concurrent requests:** raise `-np` (KV is cheap), but each extra
slot multiplies KV VRAM and the context per slot shrinks (`-c / -np`).
## initContainer (`fetch-model`)
- Idempotently downloads the 3 shards into the shared models PVC, skipping if
the first shard is already present (pod restart / recreate).
- **Deletes the retired Qwen3.6 GGUFs** from the PVC to reclaim space for the
new 87 GiB model.
## Resources
```
requests: cpu 1000m, memory 4Gi
limits: cpu 4000m, memory 20Gi
```
VRAM (weights + KV + compute) is **not** counted against the cgroup memory
limit — the limit only covers CPU-side overhead, the mmap'd GGUF pages for the
3 CPU-resident layers (~6 GiB), and reclaimable page cache during load. k8s
sees ~30 GiB as the node's allocatable system RAM.
## Probes
- `readinessProbe`: `GET /health` after 30s, every 10s, 6 failures.
- `livenessProbe`: `GET /health` after **300s** (87 GiB load + Vulkan init
takes several minutes), every 30s, 5 failures.
## Security / GPU access
`privileged: true` + mounts `/dev/dri` (DRM render nodes) — simplest reliable
way to give Vulkan access to the AMD APU on k3s without a device plugin.

85
llamacpp/args-qwen36-27b.md Normal file
View File

@@ -0,0 +1,85 @@
# Deployment arguments — `qwen3.6-27b` (REMOVED)
> **Status:** This model was **removed** from the cluster to make VRAM room for
> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
> 90 GiB VRAM pool on its own. This file documents the flags used when the
> model was live, so it can be redeployed later if the DeepSeek model is taken
> down or moved to different hardware.
>
> To redeploy: restore `deployment-qwen36-27b.yaml` (the manifest is preserved
> in git history) and re-add the LiteLLM entry. Re-check the VRAM budget —
> co-locating with the 87 GiB DeepSeek model is **not** possible on the current
> 90 GiB pool.
## Model & source
- **Model:** `unsloth/Qwen3.6-27B-MTP-GGUF` — Qwen3.6-27B, **dense** (all
params active per token), MTP variant.
- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~16 GiB**,
single GGUF file (`Qwen3.6-27B-UD-Q4_K_XL.gguf`).
- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF`
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
driver, supports Radeon 8060S / RDNA 3.5).
## Hardware target
NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
{kubernetes.io/arch: amd64, hardware: high-memory}`.
## Why it fit (and co-existed with the 35B-A3B)
At ~16 GiB, the Q4 model is small enough to **fully offload** to the GPU
(`-ngl 999`) and still leave ~80 GiB of VRAM — which is why it ran alongside
the Qwen3.6-35B-A3B model (see `args-qwen36-35b-a3b.md`). Combined the two
models used ~61 GiB of the 96 GiB pool. The dense 27B's constraint was **KV
cache**, not weights: dense attention KV is ~256 KiB/token (f16) / ~128
KiB/token (q8_0), so large context is expensive.
## Argument-by-argument
| Flag | Value | Meaning |
|------|-------|---------|
| `-m` | `/models/Qwen3.6-27B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
| `--alias` | `qwen3.6-27b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
| `-c` | `131072` | Total KV-cache context (131k). Single slot gets the full window. |
| `-np` | `1` | 1 parallel slot ⇒ the full 131k goes to a single request (dense KV is the constraint, so no slot splitting). |
| `--cont-batching` | *(flag)* | Continuous batching across slots. |
| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB at 131k); essential to make 131k affordable on a dense model. |
| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
## VRAM budget (when live)
| Component | VRAM |
|-----------------------|-------------|
| Weights (full offload) | ~16 GiB |
| KV cache (q8_0, 131k) | ~16 GiB |
| **Subtotal** | **~32 GiB** |
Left ~64 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
co-locating the 35B-A3B model.
## Notes for redeployment
- This was the **deeper-reasoning** model (dense, all params active). The
`qwen3.6-35b-a3b` was the **fast/flash** variant. If redeploying only one,
decide based on latency-vs-quality needs.
- LiteLLM entry that went with it:
```yaml
- model_name: qwen3.6-27b
litellm_params:
model: openai/qwen3.6-27b
api_base: http://llamacpp-qwen36-27b.llamacpp/v1
api_key: "sk-no-auth"
```
- Consumers at removal time: `platform-engineer` (default model),
`home-manager` (used the 35B-A3B, not this one). See git history for exact
config.
- The `fetch-model` initContainer downloaded the single GGUF idempotently into
the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
GGUF on first boot to reclaim space, so a redeploy will re-download it.

96
llamacpp/args-qwen36-35b-a3b.md Normal file
View File

@@ -0,0 +1,96 @@
# Deployment arguments — `qwen3.6-35b-a3b` (REMOVED)
> **Status:** This model was **removed** from the cluster to make VRAM room for
> `deepseek-v4-flash-0731` (UD-IQ1_M, ~87 GiB), which nearly fills the NUCBox's
> 90 GiB VRAM pool on its own. This file documents the flags used when the
> model was live, so it can be redeployed later if the DeepSeek model is taken
> down or moved to different hardware.
>
> To redeploy: restore `deployment-qwen36-35b-a3b.yaml` (the manifest is
> preserved in git history) and re-add the LiteLLM entry. Re-check the VRAM
> budget — co-locating with the 87 GiB DeepSeek model is **not** possible on
> the current 90 GiB pool.
## Model & source
- **Model:** `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` — Qwen3.6-35B-A3B, the "flash"
**Mixture-of-Experts** variant (35B total params, **only 3B active** per
token), MTP variant.
- **Quantization:** `UD-Q4_K_XL` (Unsloth Dynamic Q4_K_XL), **~20 GiB**,
single GGUF file (`Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf`).
- **HuggingFace repo:** `https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF`
- **Image:** `ghcr.io/ggml-org/llama.cpp:server-vulkan` (Mesa/RADV Vulkan
driver, supports Radeon 8060S / RDNA 3.5).
## Hardware target
NUCBox APU — AMD Ryzen AI Max 395 (Strix Halo), Radeon 8060S, 128 GiB unified
memory (32 GiB RAM / 96 GiB VRAM via firmware). Pinned via `nodeSelector:
{kubernetes.io/arch: amd64, hardware: high-memory}`.
## Why it was the "flash" model
Despite having **more total parameters** than the dense 27B, only **3B are
active per token** (MoE), so inference is significantly faster. The full ~20 GiB
of Q4 weights is still loaded into VRAM, but only a small fraction is computed
per token. Its KV cache is also tiny (~72 KiB/token), so large context is nearly
free — hence the much larger `-c` and the 2-slot split.
## Argument-by-argument
| Flag | Value | Meaning |
|------|-------|---------|
| `-m` | `/models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf` | Model file (single GGUF). |
| `--alias` | `qwen3.6-35b-a3b` | Name reported by `GET /v1/models`; matched the LiteLLM `model_name`. |
| `--host` | `0.0.0.0` | Bind on all interfaces (k8s Service reach). |
| `--port` | `8080` | Listen port (matches `containerPort` + Service `targetPort`). |
| `--jinja` | *(flag)* | Use the GGUF's chat template (Qwen3 thinking format). |
| `-ngl` | `999` | **Full GPU offload** — all layers into VRAM (fits easily in 96 GiB). |
| `-c` | `262144` | Total KV-cache context (262k), **split across parallel slots**. |
| `-np` | `2` | 2 parallel slots ⇒ 131k tokens per concurrent request (262k / 2). MoE KV is cheap, so splitting is affordable. |
| `--cont-batching` | *(flag)* | Continuous batching across the 2 slots. |
| `--cache-type-k` | `q8_0` | Quantize K cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB at 262k); frees headroom for the large `-c`. Drop to `q4_0` for even less VRAM if retrieval quality allows. |
| `--cache-type-v` | `q8_0` | Quantize V cache to q8_0 (same rationale). |
| `--threads` | `8` | CPU threads for sampling/overhead (GPU does the heavy lifting under full offload). |
## VRAM budget (when live)
| Component | VRAM |
|-----------------------|------------|
| Weights (full offload) | ~20 GiB |
| KV cache (q8_0, 262k) | ~9 GiB |
| **Subtotal** | **~29 GiB** |
Left ~67 GiB of headroom on the 96 GiB pool — comfortable, and the basis for
co-locating the dense 27B model (combined ~61 GiB).
## Combined VRAM budget (both Qwen models, when live)
| Model | Weights | KV cache | Subtotal |
|-------------------|----------|-------------------------------------------|----------|
| qwen3.6-27b | ~16 GiB | ~16 GiB (q8_0, 131k total, 1 slot) | ~32 GiB |
| qwen3.6-35b-a3b | ~20 GiB | ~9 GiB (q8_0, 262k total, 131k/slot) | ~29 GiB |
| **Total** | | | **~61 GiB** |
~35 GiB headroom on the 96 GiB pool — comfortable. (DeepSeek-V4-Flash-0731 at
~87 GiB cannot coexist with either of these; that's why both were removed.)
## Notes for redeployment
- This was the **fast/flash** model (MoE, 3B active). The `qwen3.6-27b` was the
**deeper-reasoning** model (dense, all params active). If redeploying only
one, decide based on latency-vs-quality needs.
- LiteLLM entry that went with it:
```yaml
- model_name: qwen3.6-35b-a3b
litellm_params:
model: openai/qwen3.6-35b-a3b
api_base: http://llamacpp-qwen36-35b-a3b.llamacpp/v1
api_key: "sk-no-auth"
```
- Consumers at removal time: `home-manager` (default + auxiliary),
`platform-engineer` (auxiliary compression + title generation). See git
history for exact config.
- The `fetch-model` initContainer downloaded the single GGUF idempotently into
the shared models PVC; the new DeepSeek pod's initContainer **deletes** this
GGUF on first boot to reclaim space, so a redeploy will re-download it.

210
llamacpp/deployment-deepseek-v4-flash-0731.yaml Normal file
View File

@@ -0,0 +1,210 @@
# DeepSeek-V4-Flash-0731 (MoE: 256 experts / 6 active, UD-IQ1_M ≈ 87 GiB)
# served by llama.cpp's llama-server on the NUCBox APU.
#
# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
# ~120 GiB unified memory (≈90 GiB VRAM / 30 GiB CPU RAM via firmware). The
# IQ1_M model (~87 GiB) is *almost* the size of the whole VRAM pool, so it
# CANNOT be fully offloaded to the GPU: offloading all 43 layers + the KV
# cache + Vulkan compute buffers would overflow 90 GiB. Instead we offload
# 40 of 43 layers (-ngl 40) and keep the last 3 (~6 GiB) on CPU RAM, leaving
# ~8 GiB of VRAM headroom for the KV cache, compute buffers, and co-resident
# pods. This is the only model served on the NUCBox — the two Qwen3.6 models
# were removed to make room (their GGUF files should be deleted from the PVC,
# which the initContainer below does on first boot).
#
# KV cache is tiny thanks to DeepSeek-V4's MLA attention (num_kv_heads=1,
# head_dim=512 + 64 decoupled RoPE ⇒ ~576 elements/token/layer). At 64k
# context, q8_0 KV is only ~1.6 GiB, so context is cheap — but we cap -c at
# 65536 (the required minimum) to maximise VRAM headroom, not because KV is
# the constraint.
#
# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan
# driver, which supports the Radeon 8060S (RDNA 3.5). The Vulkan backend
# supports the IQ1_M matmul (incl. the MoE matmul_id variant), so the whole
# model runs on the GPU. deepseek4 is a brand-new arch (2026-07) so the
# floating `server-vulkan` tag is used to pull a recent enough build; pin to a
# specific server-vulkan-bXXXX tag once a known-good one is verified.
#
# GPU access: the container mounts /dev/dri (the DRM render nodes) and runs
# privileged — the simplest reliable option on k3s without a Vulkan device
# plugin.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: llamacpp-deepseek-v4-flash-0731
namespace: llamacpp
labels:
app: llamacpp
model: deepseek-v4-flash-0731
spec:
replicas: 1
strategy:
type: Recreate # never run two pods loading the same model into VRAM
selector:
matchLabels:
app: llamacpp
model: deepseek-v4-flash-0731
template:
metadata:
labels:
app: llamacpp
model: deepseek-v4-flash-0731
spec:
nodeSelector:
kubernetes.io/arch: amd64
hardware: high-memory
initContainers:
# Idempotently download the (3-part, split) GGUF into the shared models
# PVC on first boot. Also removes the retired Qwen3.6 GGUFs so the new
# 87 GiB model fits on the PVC alongside any other data. Exits
# immediately if the first shard is already present (pod restart).
- name: fetch-model
image: alpine:3.20
command: ["/bin/sh", "-c"]
args:
- |
set -e
# Reclaim space from the retired Qwen3.6 models (their Deployments
# are gone; the GGUFs are dead weight on the shared PVC).
for old in Qwen3.6-27B-UD-Q4_K_XL.gguf Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf; do
if [ -f "/models/$old" ]; then
echo "Removing retired model $old ..."
rm -f "/models/$old"
fi
done
# Download any missing shards of the split UD-IQ1_M GGUF.
if [ -f "/models/$SHARD1" ]; then
echo "First shard $SHARD1 already present — skipping download."
exit 0
fi
echo "Installing curl..."
apk add --no-cache curl
for s in "$SHARD1" "$SHARD2" "$SHARD3"; do
echo "Downloading $s from $HF_REPO ..."
curl -fL --retry 5 --retry-delay 5 -o "/models/$s" "$HF_REPO/$s"
done
echo "Download complete:"
ls -lh /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-*.gguf
env:
- name: HF_REPO
value: "https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF/resolve/main/UD-IQ1_M"
- name: SHARD1
value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf"
- name: SHARD2
value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00002-of-00003.gguf"
- name: SHARD3
value: "DeepSeek-V4-Flash-0731-UD-IQ1_M-00003-of-00003.gguf"
volumeMounts:
- name: models
mountPath: /models
containers:
- name: llama-server
image: ghcr.io/ggml-org/llama.cpp:server-vulkan
imagePullPolicy: IfNotPresent
# llama.cpp's CLI parser does NOT split on '=' — every value flag must
# be a separate argv element (flag, then value). See common/arg.cpp.
args:
- -m # model file (first shard; llama.cpp auto-loads the rest)
- /models/DeepSeek-V4-Flash-0731-UD-IQ1_M-00001-of-00003.gguf
- --alias # /v1/models reports this name; matches the litellm alias
- deepseek-v4-flash-0731
- --host
- 0.0.0.0
- --port
- "8080"
- --jinja # use the GGUF's DeepSeek-V4 chat template (DSML / thinking)
- -ngl # offload 40 of 43 layers to the GPU. The model (~87 GiB) is
- "40" # nearly the whole 90 GiB VRAM pool, so full offload (-ngl 999)
# would overflow once KV cache + Vulkan compute buffers are
# added. Keeping 3 layers (~6 GiB) on CPU leaves ~8 GiB of
# VRAM headroom for the KV cache, compute buffers, and
# co-resident pods. Raise toward 43 if VRAM allows; lower
# (e.g. 38) if the pod OOMs / Vulkan runs out of device mem.
- -c # total KV-cache context (single slot gets the full window).
- "65536" # 64k — the required minimum. MLA KV is tiny (~1.6 GiB at
# q8_0), so context is cheap; -c is capped at the minimum to
# maximise VRAM headroom, not because KV is the constraint.
# Raise if VRAM headroom allows.
- -np # 1 slot => the full 64k goes to a single concurrent request
- "1" # (extra slots would multiply KV VRAM, which is fine here, but
# 1 slot keeps it simple and headroom maximal).
- --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — MLA KV is already small (~576
- q8_0 # elem/token/layer); q8_0 halves it to ~1.6 GiB at 64k and
- --cache-type-v # maximises VRAM headroom with ~negligible quality loss.
- q8_0
- --temp # default sampling temperature (DeepSeek-V4 recommendation)
- "1.0"
- --top-p # default nucleus sampling threshold (DeepSeek-V4 recommendation)
- "0.95"
- --threads # CPU threads for sampling + the 3 CPU-resident layers
- "8"
ports:
- name: http
containerPort: 8080
resources:
# The model weights + KV cache live in GPU VRAM (~90 GiB pool) and
# are NOT counted against the cgroup memory limit. This limit only
# covers CPU-side overhead + the mmap'd GGUF pages for the 3
# CPU-resident layers (~6 GiB) plus reclaimable page cache during
# load. k8s sees ~30 GiB as the node's allocatable system RAM, so the
# limit is sized to cover the CPU layers + overhead while leaving
# RAM for co-resident pods (litellm, the agents, etc.). If the pod is
# OOM-killed during model load, raise the limit.
requests:
cpu: "1000m"
memory: "4Gi"
limits:
cpu: "4000m"
memory: "20Gi"
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 6
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 300 # 87 GiB load + Vulkan init takes several minutes
periodSeconds: 30
failureThreshold: 5
securityContext:
# Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is
# the simplest reliable path on k3s without a device plugin.
privileged: true
volumeMounts:
- name: models
mountPath: /models
readOnly: true
- name: dri
mountPath: /dev/dri
volumes:
- name: models
persistentVolumeClaim:
claimName: llamacpp-models
- name: dri
hostPath:
path: /dev/dri
type: Directory
---
apiVersion: v1
kind: Service
metadata:
name: llamacpp-deepseek-v4-flash-0731
namespace: llamacpp
labels:
app: llamacpp
model: deepseek-v4-flash-0731
spec:
type: ClusterIP
selector:
app: llamacpp
model: deepseek-v4-flash-0731
ports:
- name: http
port: 80
targetPort: 8080

163
llamacpp/deployment-qwen36-27b.yaml
View File

@@ -1,163 +0,0 @@
# Qwen3.6-27B (dense, Q4_K_XL) served by llama.cpp's llama-server on the NUCBox APU.
#
# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). The Q4 model
# (~16 GiB) is fully offloaded to the GPU via the Vulkan backend.
#
# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan bundles the Mesa/RADV Vulkan
# driver, which supports the Radeon 8060S (RDNA 3.5). The project moved from the
# legacy `ggerganov/llama.cpp` namespace (which only has light/full tags) to
# `ggml-org/llama.cpp` (server-vulkan + pinned build tags like server-vulkan-bXXXX).
# Pin to a build tag (e.g. server-vulkan-b4738) for production reproducibility.
#
# GPU access: the container mounts /dev/dri (the DRM render nodes) and runs
# privileged. This is the simplest reliable option on k3s without a Vulkan
# device plugin; tighten later with supplementalGroups if desired.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: llamacpp-qwen36-27b
namespace: llamacpp
labels:
app: llamacpp
model: qwen3.6-27b
spec:
replicas: 1
strategy:
type: Recreate # never run two pods loading the same model into VRAM
selector:
matchLabels:
app: llamacpp
model: qwen3.6-27b
template:
metadata:
labels:
app: llamacpp
model: qwen3.6-27b
spec:
nodeSelector:
kubernetes.io/arch: amd64
hardware: high-memory
initContainers:
# Idempotently download the GGUF into the shared models PVC on first boot.
# Exits immediately if the file is already present (pod restart / recreate).
- name: fetch-model
image: alpine:3.20
command: ["/bin/sh", "-c"]
args:
- |
set -e
if [ -f "/models/$MODEL_FILE" ]; then
echo "Model $MODEL_FILE already present — skipping download."
exit 0
fi
echo "Installing curl..."
apk add --no-cache curl
echo "Downloading $MODEL_FILE from $MODEL_URL ..."
curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL"
echo "Download complete: $(ls -lh /models/$MODEL_FILE)"
env:
- name: MODEL_URL
value: "https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-UD-Q4_K_XL.gguf"
- name: MODEL_FILE
value: "Qwen3.6-27B-UD-Q4_K_XL.gguf"
volumeMounts:
- name: models
mountPath: /models
containers:
- name: llama-server
image: ghcr.io/ggml-org/llama.cpp:server-vulkan
imagePullPolicy: IfNotPresent
# llama.cpp's CLI parser does NOT split on '=' — every value flag must be a
# separate argv element (flag, then value). See common/arg.cpp in the repo.
args:
- -m # model file
- /models/Qwen3.6-27B-UD-Q4_K_XL.gguf
- --alias # /v1/models reports this name; matches the litellm alias
- qwen3.6-27b
- --host
- 0.0.0.0
- --port
- "8080"
- --jinja # use the GGUF's chat template (Qwen3 thinking format)
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999"
- -c # total KV-cache context (single slot gets the full window)
- "131072"
- -np # 1 slot => 131072 tokens for the single request (no split)
- "1"
- --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB
- q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c
- --cache-type-v
- q8_0
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)
- "8"
ports:
- name: http
containerPort: 8080
resources:
# The model weights + KV cache live in GPU VRAM (96 GiB pool) and are
# NOT counted against the cgroup memory limit. This limit only covers
# CPU-side overhead + the mmap'd GGUF file pages during load (~16 GiB,
# reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM,
# so the request is kept low to stay schedulable alongside other pods.
# If the pod OOM-kills during load, the amdgpu driver may be counting
# some VRAM against the cgroup — raise the limit.
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "4000m"
memory: "24Gi"
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 6
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 180 # model load + Vulkan init can take a few minutes
periodSeconds: 30
failureThreshold: 5
securityContext:
# Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the
# simplest reliable path on k3s without a device plugin.
privileged: true
volumeMounts:
- name: models
mountPath: /models
readOnly: true
- name: dri
mountPath: /dev/dri
volumes:
- name: models
persistentVolumeClaim:
claimName: llamacpp-models
- name: dri
hostPath:
path: /dev/dri
type: Directory
---
apiVersion: v1
kind: Service
metadata:
name: llamacpp-qwen36-27b
namespace: llamacpp
labels:
app: llamacpp
model: qwen3.6-27b
spec:
type: ClusterIP
selector:
app: llamacpp
model: qwen3.6-27b
ports:
- name: http
port: 80
targetPort: 8080

162
llamacpp/deployment-qwen36-35b-a3b.yaml
View File

@@ -1,162 +0,0 @@
# Qwen3.6-35B-A3B (MoE: 35B total / 3B active, Q4_K_XL) — the "flash" variant.
#
# Despite having more total parameters than the 27B dense model, only 3B are
# active per token (Mixture-of-Experts), so inference is much faster. The full
# ~20 GiB of Q4 weights is still loaded into VRAM but only a small fraction is
# computed per token.
#
# Hardware: AMD Ryzen AI Max 395 (Strix Halo) — integrated Radeon 8060S,
# 128 GiB unified memory (32 GiB RAM / 96 GiB VRAM via firmware). Full GPU
# offload via the Vulkan backend. Shares the 96 GiB VRAM pool with the 27B
# model — see llamacpp/README.md for the combined VRAM budget.
#
# Image: ghcr.io/ggml-org/llama.cpp:server-vulkan (Mesa/RADV Vulkan driver,
# supports the Radeon 8060S / RDNA 3.5). Pin to a build tag for production.
#
# GPU access: mounts /dev/dri + privileged (simplest reliable path on k3s).
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: llamacpp-qwen36-35b-a3b
namespace: llamacpp
labels:
app: llamacpp
model: qwen3.6-35b-a3b
spec:
replicas: 1
strategy:
type: Recreate # never run two pods loading the same model into VRAM
selector:
matchLabels:
app: llamacpp
model: qwen3.6-35b-a3b
template:
metadata:
labels:
app: llamacpp
model: qwen3.6-35b-a3b
spec:
nodeSelector:
kubernetes.io/arch: amd64
hardware: high-memory
initContainers:
# Idempotently download the GGUF into the shared models PVC on first boot.
# Exits immediately if the file is already present (pod restart / recreate).
- name: fetch-model
image: alpine:3.20
command: ["/bin/sh", "-c"]
args:
- |
set -e
if [ -f "/models/$MODEL_FILE" ]; then
echo "Model $MODEL_FILE already present — skipping download."
exit 0
fi
echo "Installing curl..."
apk add --no-cache curl
echo "Downloading $MODEL_FILE from $MODEL_URL ..."
curl -fL --retry 5 --retry-delay 5 -o "/models/$MODEL_FILE" "$MODEL_URL"
echo "Download complete: $(ls -lh /models/$MODEL_FILE)"
env:
- name: MODEL_URL
value: "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
- name: MODEL_FILE
value: "Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
volumeMounts:
- name: models
mountPath: /models
containers:
- name: llama-server
image: ghcr.io/ggml-org/llama.cpp:server-vulkan
imagePullPolicy: IfNotPresent
# llama.cpp's CLI parser does NOT split on '=' — every value flag must be a
# separate argv element (flag, then value). See common/arg.cpp in the repo.
args:
- -m # model file
- /models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
- --alias # /v1/models reports this name; matches the litellm alias
- qwen3.6-35b-a3b
- --host
- 0.0.0.0
- --port
- "8080"
- --jinja # use the GGUF's chat template (Qwen3 thinking format)
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999"
- -c # total KV-cache context, split across parallel slots
- "262144"
- -np # 2 parallel slots => 131072 tokens per concurrent request
- "2"
- --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB
- q8_0 # at 262k ctx); ~negligible quality loss, frees headroom for large -c
- --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows)
- q8_0
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)
- "8"
ports:
- name: http
containerPort: 8080
resources:
# The model weights + KV cache live in GPU VRAM (96 GiB pool) and are
# NOT counted against the cgroup memory limit. This limit only covers
# CPU-side overhead + the mmap'd GGUF file pages during load (~20 GiB,
# reclaimable). k8s sees ~32 GiB as the node's allocatable system RAM.
# If the pod OOM-kills during load, raise the limit.
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "4000m"
memory: "24Gi"
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 6
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 180 # model load + Vulkan init can take a few minutes
periodSeconds: 30
failureThreshold: 5
securityContext:
# Vulkan on the AMD APU needs /dev/dri + the driver. Privileged is the
# simplest reliable path on k3s without a device plugin.
privileged: true
volumeMounts:
- name: models
mountPath: /models
readOnly: true
- name: dri
mountPath: /dev/dri
volumes:
- name: models
persistentVolumeClaim:
claimName: llamacpp-models
- name: dri
hostPath:
path: /dev/dri
type: Directory
---
apiVersion: v1
kind: Service
metadata:
name: llamacpp-qwen36-35b-a3b
namespace: llamacpp
labels:
app: llamacpp
model: qwen3.6-35b-a3b
spec:
type: ClusterIP
selector:
app: llamacpp
model: qwen3.6-35b-a3b
ports:
- name: http
port: 80
targetPort: 8080

4
llamacpp/pv.yaml
View File

@@ -17,7 +17,7 @@ metadata:
spec:
storageClassName: manual
capacity:
storage: 100Gi
storage: 200Gi
accessModes:
- ReadWriteMany
hostPath:
@@ -44,4 +44,4 @@ spec:
- ReadWriteMany
resources:
requests:
storage: 100Gi
storage: 200Gi