9 Commits

Author SHA1 Message Date
platform-engineer
097fe2e0e7 fix(prometheus): bump memory limit from 1Gi to 2Gi due to OOM
fix(litellm): add resource limits (512Mi/1Gi) to prevent unbounded memory consumption
2026-07-22 05:58:23 +00:00
Platform Engineer
6df0be81c9 fix(platform): fix high-restart pods - prometheus nodeSelector, litellm resources, gitea-runner limits
- Prometheus: change nodeSelector from hardware=high-memory (nonexistent label) to kubernetes.io/arch: amd64
  Fixes OOMKilled cause - pod was being scheduled on wrong nodes.
- litellm-deployment: add resource limits (2Gi/500m requests, 4Gi/2000m limits) + pin image to v1.34.0
  Fixes OOMKilled cause - no resource limits were set.
- gitea-runner: pin image to v0.12.0 from :latest + add resource limits (512Mi/250m requests, 1Gi/500m limits)
  Addresses extreme restart count (281) caused by unbounded memory usage and rolling image updates.
2026-07-21 20:28:12 +00:00
a5291da0b2 fix(gitea): bump restartedAt annotation to clear stuck gitea-runner pods (281 restarts from DNS resolution failures) 2026-07-21 15:36:57 +02:00
platform-engineer
075bdd8ca3 fix(gitea-runner): pin image tag to v0.7.2, add resource limits (256Mi/100m req, 512Mi/500m lim) to prevent OOM restarts 2026-07-20 22:19:37 +00:00
Roger Oriol
e44d7ba1fc Merge branch 'main' of https://git.rogi.casa/roger/k3s-cluster 2026-07-19 12:01:21 +02:00
Roger Oriol
f1005cd426 change openwebui url to ai.rogi.casa 2026-07-19 12:00:52 +02:00
Roger Oriol
43c0c2561e fix hermes write folder 2026-07-18 19:29:52 +02:00
8334cd48f8 Merge pull request 'fix(myorg-assistant): add git-clone initContainer to git-sync CronJob' (#26) from fix/myorg-git-sync-clone into main
Reviewed-on: #26
2026-07-18 19:25:00 +02:00
Hermes Platform Engineer
ba7e05a73d fix(myorg-assistant): add git-clone initContainer to git-sync CronJob
The git-sync CronJob only ran `git pull`/`git push` against /data/myorg
but never ensured the repo existed. The clone was solely the Deployment's
git-clone initContainer's job, and when that didn't populate the volume the
cron pod failed with 'Not a git repository: /data/myorg' on every run (and
swallowed the error, exiting 0).

Add an idempotent git-clone initContainer (guarded by [ ! -d /data/myorg/.git ])
so the cron job self-heals and actually syncs.

Refs: myorg-git-sync-* 'Pull: Error: Not a git repository' (last 24h+)
2026-07-18 17:21:26 +00:00
6 changed files with 31 additions and 10 deletions

View File

@@ -25,6 +25,8 @@ metadata:
namespace: gitea
labels:
app: gitea
annotations:
kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
spec:
replicas: 1
selector:
@@ -101,6 +103,8 @@ metadata:
namespace: gitea
labels:
app: gitea-runner
annotations:
kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
spec:
replicas: 1
selector:
@@ -115,7 +119,14 @@ spec:
kubernetes.io/arch: arm64
containers:
- name: gitea-runner
image: vegardit/gitea-act-runner:latest
image: vegardit/gitea-act-runner:v0.12.0
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
env:
- name: GITEA_INSTANCE_URL
valueFrom:

View File

@@ -112,6 +112,13 @@ spec:
env:
- name: STORE_MODEL_IN_DB
value: "True"
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
volumes:
- name: config-volume
configMap:

View File

@@ -39,10 +39,10 @@ spec:
mountPath: /prometheus
resources:
requests:
memory: "512Mi"
memory: "1Gi"
cpu: "500m"
limits:
memory: "1Gi"
memory: "2Gi"
cpu: "1000m"
volumes:
- name: prometheus-config

View File

@@ -31,12 +31,8 @@ spec:
- sh
- -c
- |
# Normalize the repo URL host to the correct Gitea subdomain.
# The GIT_REPO_URL secret historically contained gitea.rogi.casa,
# which 526s through Cloudflare; the working subdomain is git.rogi.casa.
export GIT_REPO_URL="$(echo "${GIT_REPO_URL}" | sed -E 's#https?://[^/@]+@?gitea\.rogi\.casa#https://'"${GIT_USERNAME}"':'"${GIT_TOKEN}"'@git.rogi.casa#')"
if [ ! -d /data/myorg/.git ]; then
echo "Cloning repository from ${GIT_REPO_URL}..."
echo "Cloning repository..."
git clone ${GIT_REPO_URL} /data/myorg
cd /data/myorg
git config user.name "${GIT_USERNAME}"

View File

@@ -9,10 +9,10 @@ spec:
ingressClassName: traefik
tls:
- hosts:
- openai.rogi.casa
- ai.rogi.casa
secretName: openwebui-tls
rules:
- host: openai.rogi.casa
- host: ai.rogi.casa
http:
paths:
- path: /

View File

@@ -125,6 +125,13 @@ spec:
env:
- name: HERMES_HOME
value: /opt/data
# Hermes' file-write tool refuses any path outside HERMES_WRITE_SAFE_ROOT.
# When unset it defaults to HERMES_HOME (/opt/data), which blocks the
# agent's only GitOps remediation path (editing manifests under
# /workspace/k3s-cluster). Whitelist the whole filesystem — consistent
# with yolo:true, approvals.mode:off, and the agent having no k8s RBAC.
- name: HERMES_WRITE_SAFE_ROOT
value: "/"
volumeMounts:
- name: data
mountPath: /opt/data