4 Commits

Author SHA1 Message Date
platform-engineer
9c134d5dd0 fix(litellm): add memory and CPU resource limits to prevent OOMKilled
Litellm pod was being killed by the kernel OOM killer (7 restarts, last
termination reason: OOMKilled). Added explicit resource requests (512Mi/250m)
and limits (2Gi/1000m) to ensure the container gets bounded memory.
2026-07-21 21:59:15 +00:00
Platform Engineer
6df0be81c9 fix(platform): fix high-restart pods - prometheus nodeSelector, litellm resources, gitea-runner limits
- Prometheus: change nodeSelector from hardware=high-memory (nonexistent label) to kubernetes.io/arch: amd64
  Fixes OOMKilled cause - pod was being scheduled on wrong nodes.
- litellm-deployment: add resource limits (2Gi/500m requests, 4Gi/2000m limits) + pin image to v1.34.0
  Fixes OOMKilled cause - no resource limits were set.
- gitea-runner: pin image to v0.12.0 from :latest + add resource limits (512Mi/250m requests, 1Gi/500m limits)
  Addresses extreme restart count (281) caused by unbounded memory usage and rolling image updates.
2026-07-21 20:28:12 +00:00
a5291da0b2 fix(gitea): bump restartedAt annotation to clear stuck gitea-runner pods (281 restarts from DNS resolution failures) 2026-07-21 15:36:57 +02:00
platform-engineer
075bdd8ca3 fix(gitea-runner): pin image tag to v0.7.2, add resource limits (256Mi/100m req, 512Mi/500m lim) to prevent OOM restarts 2026-07-20 22:19:37 +00:00
4 changed files with 22 additions and 7 deletions

View File

@@ -25,6 +25,8 @@ metadata:
namespace: gitea
labels:
app: gitea
annotations:
kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
spec:
replicas: 1
selector:
@@ -101,6 +103,8 @@ metadata:
namespace: gitea
labels:
app: gitea-runner
annotations:
kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
spec:
replicas: 1
selector:
@@ -115,7 +119,14 @@ spec:
kubernetes.io/arch: arm64
containers:
- name: gitea-runner
image: vegardit/gitea-act-runner:latest
image: vegardit/gitea-act-runner:v0.12.0
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
env:
- name: GITEA_INSTANCE_URL
valueFrom:

View File

@@ -112,6 +112,13 @@ spec:
env:
- name: STORE_MODEL_IN_DB
value: "True"
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "2Gi"
cpu: "1000m"
volumes:
- name: config-volume
configMap:

View File

@@ -15,9 +15,10 @@ spec:
labels:
app: prometheus
spec:
# Prevent scheduling on Raspberry Pi due to resource requirements (512Mi-1Gi memory, 500m-1000m CPU)
# Target the nucbox (amd64, 24Gi RAM) which is the only node with enough memory for Prometheus.
nodeSelector:
hardware: high-memory
kubernetes.io/os: linux
kubernetes.io/arch: amd64
serviceAccountName: prometheus
containers:
- name: prometheus

View File

@@ -31,10 +31,6 @@ spec:
- sh
- -c
- |
# Normalize the repo URL host to the correct Gitea subdomain.
# The GIT_REPO_URL secret historically contained gitea.rogi.casa,
# which 526s through Cloudflare; the working subdomain is git.rogi.casa.
export GIT_REPO_URL="$(echo "${GIT_REPO_URL}" | sed -E 's#https?://[^/@]+@?gitea\.rogi\.casa#https://'"${GIT_USERNAME}"':'"${GIT_TOKEN}"'@git.rogi.casa#')"
if [ ! -d /data/myorg/.git ]; then
echo "Cloning repository..."
git clone ${GIT_REPO_URL} /data/myorg