roger/k3s-cluster
1
0
Fork 1
Code Issues Pull Requests 6 Actions Packages Projects Releases Wiki Activity

increase llamacpp models context window!

Browse Source
This commit is contained in:
Roger Oriol
2026-07-24 20:11:30 +02:00
parent 496e46076c
commit c982b33015
2 changed files with 14 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
llamacpp/deployment-qwen36-35b-a3b.yaml
View File

@@ -85,8 +85,8 @@ spec:
- -ngl # offload ALL layers to the GPU (fits in 96 GiB VRAM)
- "999"
- -c # total KV-cache context, split across parallel slots
- "32768"
- -np # 4 parallel slots => 8192 tokens per concurrent request
- "131072"
- -np # 4 parallel slots => 32768 tokens per concurrent request
- "4"
- --cont-batching # continuous batching across slots
- --threads # CPU threads for sampling/overhead (GPU does the heavy lifting)