roger/k3s-cluster
1
0
Fork 1
Code Issues Pull Requests 6 Actions Packages Projects Releases Wiki Activity

set qwen models parameters to allow for a higher context window

Browse Source
This commit is contained in:
Roger Oriol
2026-07-24 21:30:01 +02:00
parent bc165d3cf1
commit 4694034a04
4 changed files with 50 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
llamacpp/deployment-qwen36-35b-a3b.yaml
View File

@@ -86,9 +86,13 @@ spec:
- "999"
- -c # total KV-cache context, split across parallel slots
- "131072"
- -np # 4 parallel slots => 32768 tokens per concurrent request
- "4"
- -np # 2 parallel slots => 65536 tokens per concurrent request
- "2"
- --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~9 GiB → ~4.5 GiB)
- q8_0 # with ~negligible quality loss; frees headroom for larger -c later
- --cache-type-v # (raise to q4_0 for even less VRAM if retrieval quality allows)
- q8_0
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)
- "8"
ports: