platform-engineer-agent/k3s-cluster
1
0
Fork 0
forked from roger/k3s-cluster
Code Pull Requests Activity

simplify all of platform engineer's crons into one single daily cron

Browse Source
This commit is contained in:
Roger Oriol
2026-08-01 16:52:45 +02:00
parent 33b1b2dc8f
commit 58bba66f18
2 changed files with 43 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
platform-engineer/README.md
View File

@@ -208,21 +208,21 @@ HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=<strong password>
---
## 7. Cron jobs to seed (Hermes-native)
## 7. Single daily cron job (Hermes-native)
These are written by an init script (one-shot Job `hermes-cron-seed`) that runs
`hermes cron create ...` against the gateway on first install, and is idempotent
(it checks existing job names). All deliver to Discord. Examples:
The one-shot Job `hermes-cron-seed` creates one daily check at 06:00 local cluster time (`0 4 * * *` UTC; Hermes has no timezone configured). It removes the known legacy jobs first, then runs
`hermes cron create ...` against the gateway and is idempotent
(it removes old names and recreates the daily job). It delivers to Discord and combines node, resource, PVC, pod, ArgoCD, and certificate checks. It stays silent when healthy and sends one concise Catalan report when there is an issue.
| Name | Schedule | Prompt (abbreviated) |
|------|----------|------------------------|
| `cluster-health-check` | `every 15m` | Run `kubectl get nodes,pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded` and `kubectl get events -A --field-selector type=Warning --since=20m`. If everything healthy, reply with only `[SILENT]`. Otherwise summarize failures and root-cause briefly. |
| `pod-restart-loop` | `every 10m` | Find pods in `CrashLoopBackOff`/`ImagePullBackOff` across all namespaces. For `CrashLoopBackOff`, fetch logs and if a clear transient cause (OOM, config parse, missing secret) is visible, attempt `kubectl rollout restart <deploy>`; otherwise notify me with the log excerpt. Reply `[SILENT]` if none found. |
| `pvc-pressure` | `every 30m` | `kubectl get pv` + node disk via `kubectl top nodes`. Alert if any PVC `Bound` to a near-full volume or node disk >85%. `[SILENT]` otherwise. |
| `argocd-sync-health` | `every 1h` | `kubectl get applications -n argocd -o wide` (or `argocd app sync --dry-run` if CLI present). Report any `OutOfSync`/`Degraded` app. `[SILENT]` if all `Synced`+`Healthy`. |
| `cert-expiry` | `every 1d at 09:00` | List cert-manager `Certificate` resources with expiry < 21 days. Notify only if any. `[SILENT]` otherwise. |
| `node-resource-drift` | `every 30m` | `kubectl top nodes`. Alert if any node CPU>90% or mem>90% sustained, or any node `NotReady`. `[SILENT]` otherwise. |
| `daily-cluster-report` | `0 8 * * *` | Summarize: node count/status, top 5 pods by CPU/mem, # pods not Running, # ArgoCD apps OutOfSync, cert warnings. Always deliver (no `[SILENT]`). |
| `daily-cluster-check` | `0 4 * * *` UTC (06:00 local) | Combined daily platform health check; `[SILENT]` when healthy. |
Design rules baked into SOUL.md:
- **Read-only checks** run frequently (10–30m) and stay silent unless wrong.
@@ -274,7 +274,7 @@ Design rules baked into SOUL.md:
4. Apply the Deployment; wait for the pod to go Running.
5. `kubectl exec` in and run the one-shot cron seed:
`hermes cron create ...` (or apply the `cron-seed` Job).
6. Trigger the first `cluster-health-check` manually: `hermes cron run cluster-health-check`.
6. Trigger the first `daily-cluster-check` manually: `hermes cron run daily-cluster-check`.
7. Add the app to `argocd/gen-apps.sh`, regenerate, commit, push.
---
@@ -324,12 +324,12 @@ Design rules baked into SOUL.md:
password + a 32-byte session secret.
5. **Commit & push** the whole change. ArgoCD will create the namespace
resources, deploy the pod, and bring up the ingress at `platform-engineer.rogi.casa`.
6. **Seed the cron jobs:**
6. **Seed the cron job:**
`kubectl apply -f platform-engineer/cron-seed.yaml` (one-shot Job) — it waits
for the hermes pod, then runs `hermes cron create ...` for each watchdog.
Re-run it any time you want to re-seed after a wipe.
7. **Smoke test:** trigger the first health check manually —
`kubectl exec -n platform-engineer deploy/hermes -- hermes cron run cluster-health-check` —
for the Hermes pod, removes the legacy schedules, and creates the single
daily check. Re-run it to migrate an existing PVC or after a wipe.
7. **Smoke test:** trigger the daily health check manually —
`kubectl exec -n platform-engineer deploy/hermes -- hermes cron run daily-cluster-check` —
and confirm the message lands in Discord.
8. **ArgoCD:** the `Application` (`argocd/apps/platform-engineer.yaml`) is
already generated. After commit, Argo will reconcile it like every other app.
@@ -339,10 +339,11 @@ Design rules baked into SOUL.md:
- **ArgoCD owns** (in git): namespace, RBAC, Secret, ConfigMap (seed), PVC,
Deployment, Service, Ingress, cron-seed Job.
- **Runtime state (on the PVC, NOT reconciled):** `config.yaml`, `SOUL.md`,
`.env`, `cron/jobs.json`, `sessions/`, `memories/`, `skills/`. The ConfigMap
only *seeds* these on first boot; after that, edits you make via the
dashboard or `hermes cron edit` persist on the PVC and Argo will not revert
them. If you ever want a hard reset, delete the PVC and re-apply.
`.env`, `cron/jobs.json`, `sessions/`, `memories/`, `skills/`. The cron seed
migrates the existing `cron/jobs.json` to the single daily check, but the
file remains runtime state afterward. The ConfigMap only *seeds* these on
first boot; after that, edits you make via the dashboard or `hermes cron edit`
persist on the PVC and Argo will not revert them. If you ever want a hard reset, delete the PVC and re-apply.
---