- Prometheus: change nodeSelector from hardware=high-memory (nonexistent label) to kubernetes.io/arch: amd64
Fixes OOMKilled cause - pod was being scheduled on wrong nodes.
- litellm-deployment: add resource limits (2Gi/500m requests, 4Gi/2000m limits) + pin image to v1.34.0
Fixes OOMKilled cause - no resource limits were set.
- gitea-runner: pin image to v0.12.0 from :latest + add resource limits (512Mi/250m requests, 1Gi/500m limits)
Addresses extreme restart count (281) caused by unbounded memory usage and rolling image updates.
The git-sync CronJob only ran `git pull`/`git push` against /data/myorg
but never ensured the repo existed. The clone was solely the Deployment's
git-clone initContainer's job, and when that didn't populate the volume the
cron pod failed with 'Not a git repository: /data/myorg' on every run (and
swallowed the error, exiting 0).
Add an idempotent git-clone initContainer (guarded by [ ! -d /data/myorg/.git ])
so the cron job self-heals and actually syncs.
Refs: myorg-git-sync-* 'Pull: Error: Not a git repository' (last 24h+)
CronJob pods were stuck in ImagePullBackOff because they referenced
the local-only image 'myorg-assistant:latest' which is not present
on the node. Switch all 5 cronjobs to the Gitea registry image
git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf
(matching the Deployment), set imagePullPolicy: Always, and add
imagePullSecrets: gitea-registry so they can authenticate to the
private registry.
The k3s-cluster-root app-of-apps has been in a continuous Synced<->OutOfSync
oscillation (489 transitions in 24h). Each cycle, ArgoCD syncs the 'argocd'
Application CRD, succeeds, then immediately detects drift because ArgoCD
adds status/operation fields to Application resources at runtime.
Adding ignoreDifferences for /status and /operation on Application resources
stops the drift loop while keeping the app-of-apps functional.
Survey of dashboards that could be built from existing and not-yet-enabled
metrics across the cluster's services (traefik, coredns, metallb, cert-manager,
phoenix, litellm, gitea, postgres, etc.), with per-service enable steps and
a recommended priority order.
