roger/k3s-cluster
1
0
Fork 1
Code Issues Pull Requests 6 Actions Packages Projects Releases Wiki Activity

set qwen models parameters to allow for a higher context window

Browse Source
This commit is contained in:
Roger Oriol
2026-07-24 21:30:01 +02:00
parent bc165d3cf1
commit 4694034a04
4 changed files with 50 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
llamacpp/deployment-qwen36-27b.yaml
View File

@@ -84,10 +84,14 @@ spec:
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999"
- -c # total KV-cache context, split across parallel slots
- "32768"
- -np # 4 parallel slots => 8192 tokens per concurrent request
- "4"
- "131072"
- -np # 2 parallel slots => 65536 tokens per concurrent request
- "2"
- --cont-batching # continuous batching across slots
- --cache-type-k # quantize KV cache to q8_0 — halves KV VRAM (~32 GiB → ~16 GiB
- q8_0 # at 131k ctx); ~negligible quality loss, frees headroom for large -c
- --cache-type-v
- q8_0
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)
- "8"
ports: