platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

increase qwen3.6 models context available to 131k

Browse Source
This commit is contained in:
Roger Oriol
2026-07-26 01:29:23 +02:00
parent 5eaea8c387
commit 7af7ae66de
3 changed files with 20 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
llamacpp/deployment-qwen36-27b.yaml
View File

@@ -83,10 +83,10 @@ spec:
- --jinja # use the GGUF's chat template (Qwen3 thinking format)
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999"
- -c # total KV-cache context, split across parallel slots
- -c # total KV-cache context (single slot gets the full window)
- "131072"
- -np # 2 parallel slots => 65536 tokens per concurrent request
- "2"
- -np # 1 slot => 131072 tokens for the single request (no split)
- "1"
- --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB
- q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c