4 Commits

Author SHA1 Message Date
platform-engineer
9c134d5dd0 fix(litellm): add memory and CPU resource limits to prevent OOMKilled
Litellm pod was being killed by the kernel OOM killer (7 restarts, last
termination reason: OOMKilled). Added explicit resource requests (512Mi/250m)
and limits (2Gi/1000m) to ensure the container gets bounded memory.
2026-07-21 21:59:15 +00:00
Platform Engineer
6df0be81c9 fix(platform): fix high-restart pods - prometheus nodeSelector, litellm resources, gitea-runner limits
- Prometheus: change nodeSelector from hardware=high-memory (nonexistent label) to kubernetes.io/arch: amd64
  Fixes OOMKilled cause - pod was being scheduled on wrong nodes.
- litellm-deployment: add resource limits (2Gi/500m requests, 4Gi/2000m limits) + pin image to v1.34.0
  Fixes OOMKilled cause - no resource limits were set.
- gitea-runner: pin image to v0.12.0 from :latest + add resource limits (512Mi/250m requests, 1Gi/500m limits)
  Addresses extreme restart count (281) caused by unbounded memory usage and rolling image updates.
2026-07-21 20:28:12 +00:00
a5291da0b2 fix(gitea): bump restartedAt annotation to clear stuck gitea-runner pods (281 restarts from DNS resolution failures) 2026-07-21 15:36:57 +02:00
platform-engineer
075bdd8ca3 fix(gitea-runner): pin image tag to v0.7.2, add resource limits (256Mi/100m req, 512Mi/500m lim) to prevent OOM restarts 2026-07-20 22:19:37 +00:00
3 changed files with 22 additions and 3 deletions

View File

@@ -25,6 +25,8 @@ metadata:
namespace: gitea namespace: gitea
labels: labels:
app: gitea app: gitea
annotations:
kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
spec: spec:
replicas: 1 replicas: 1
selector: selector:
@@ -101,6 +103,8 @@ metadata:
namespace: gitea namespace: gitea
labels: labels:
app: gitea-runner app: gitea-runner
annotations:
kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
spec: spec:
replicas: 1 replicas: 1
selector: selector:
@@ -115,7 +119,14 @@ spec:
kubernetes.io/arch: arm64 kubernetes.io/arch: arm64
containers: containers:
- name: gitea-runner - name: gitea-runner
image: vegardit/gitea-act-runner:latest image: vegardit/gitea-act-runner:v0.12.0
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
env: env:
- name: GITEA_INSTANCE_URL - name: GITEA_INSTANCE_URL
valueFrom: valueFrom:

View File

@@ -112,6 +112,13 @@ spec:
env: env:
- name: STORE_MODEL_IN_DB - name: STORE_MODEL_IN_DB
value: "True" value: "True"
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "2Gi"
cpu: "1000m"
volumes: volumes:
- name: config-volume - name: config-volume
configMap: configMap:

View File

@@ -15,9 +15,10 @@ spec:
labels: labels:
app: prometheus app: prometheus
spec: spec:
# Prevent scheduling on Raspberry Pi due to resource requirements (512Mi-1Gi memory, 500m-1000m CPU) # Target the nucbox (amd64, 24Gi RAM) which is the only node with enough memory for Prometheus.
nodeSelector: nodeSelector:
hardware: high-memory kubernetes.io/os: linux
kubernetes.io/arch: amd64
serviceAccountName: prometheus serviceAccountName: prometheus
containers: containers:
- name: prometheus - name: prometheus