platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

increase qwen3.6 models context available to 131k

Browse Source
This commit is contained in:
Roger Oriol
2026-07-26 01:29:23 +02:00
parent 5eaea8c387
commit 7af7ae66de
3 changed files with 20 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
llamacpp/deployment-qwen36-35b-a3b.yaml
View File

@@ -85,12 +85,12 @@ spec:
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999"
- -c # total KV-cache context, split across parallel slots
- "131072"
- -np # 2 parallel slots => 65536 tokens per concurrent request
- "262144"
- -np # 2 parallel slots => 131072 tokens per concurrent request
- "2"
- --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~9 GiB → ~4.5 GiB)
- q8_0 # with ~negligible quality loss; frees headroom for larger -c later
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~18 GiB → ~9 GiB
- q8_0 # at 262k ctx); ~negligible quality loss, frees headroom for large -c
- --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows)
- q8_0
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)