roger/k3s-cluster
1
0
Fork 1
Code Issues Pull Requests 6 Actions Packages Projects Releases Wiki Activity

simplify all of platform engineer's crons into one single daily cron

Browse Source
This commit is contained in:
Roger Oriol
2026-08-01 16:52:45 +02:00
parent 33b1b2dc8f
commit 58bba66f18
2 changed files with 43 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
platform-engineer/README.md
View File

@@ -208,21 +208,21 @@ HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=<strong password>
--- ---
## 7. Cron jobs to seed (Hermes-native) ## 7. Single daily cron job (Hermes-native)
These are written by an init script (one-shot Job `hermes-cron-seed`) that runs The one-shot Job `hermes-cron-seed` creates one daily check at 06:00 local cluster time (`0 4 * * *` UTC; Hermes has no timezone configured). It removes the known legacy jobs first, then runs
`hermes cron create ...` against the gateway on first install, and is idempotent `hermes cron create ...` against the gateway and is idempotent
(it checks existing job names). All deliver to Discord. Examples: (it removes old names and recreates the daily job). It delivers to Discord and combines node, resource, PVC, pod, ArgoCD, and certificate checks. It stays silent when healthy and sends one concise Catalan report when there is an issue.
| Name | Schedule | Prompt (abbreviated) | | Name | Schedule | Prompt (abbreviated) |
|------|----------|------------------------| |------|----------|------------------------|
| `cluster-health-check` | `every 15m` | Run `kubectl get nodes,pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded` and `kubectl get events -A --field-selector type=Warning --since=20m`. If everything healthy, reply with only `[SILENT]`. Otherwise summarize failures and root-cause briefly. | | `daily-cluster-check` | `0 4 * * *` UTC (06:00 local) | Combined daily platform health check; `[SILENT]` when healthy. |
| `pod-restart-loop` | `every 10m` | Find pods in `CrashLoopBackOff`/`ImagePullBackOff` across all namespaces. For `CrashLoopBackOff`, fetch logs and if a clear transient cause (OOM, config parse, missing secret) is visible, attempt `kubectl rollout restart <deploy>`; otherwise notify me with the log excerpt. Reply `[SILENT]` if none found. |
| `pvc-pressure` | `every 30m` | `kubectl get pv` + node disk via `kubectl top nodes`. Alert if any PVC `Bound` to a near-full volume or node disk >85%. `[SILENT]` otherwise. |
| `argocd-sync-health` | `every 1h` | `kubectl get applications -n argocd -o wide` (or `argocd app sync --dry-run` if CLI present). Report any `OutOfSync`/`Degraded` app. `[SILENT]` if all `Synced`+`Healthy`. |
| `cert-expiry` | `every 1d at 09:00` | List cert-manager `Certificate` resources with expiry < 21 days. Notify only if any. `[SILENT]` otherwise. |
| `node-resource-drift` | `every 30m` | `kubectl top nodes`. Alert if any node CPU>90% or mem>90% sustained, or any node `NotReady`. `[SILENT]` otherwise. |
| `daily-cluster-report` | `0 8 * * *` | Summarize: node count/status, top 5 pods by CPU/mem, # pods not Running, # ArgoCD apps OutOfSync, cert warnings. Always deliver (no `[SILENT]`). |
Design rules baked into SOUL.md: Design rules baked into SOUL.md:
- **Read-only checks** run frequently (10–30m) and stay silent unless wrong. - **Read-only checks** run frequently (10–30m) and stay silent unless wrong.
@@ -274,7 +274,7 @@ Design rules baked into SOUL.md:
4. Apply the Deployment; wait for the pod to go Running. 4. Apply the Deployment; wait for the pod to go Running.
5. `kubectl exec` in and run the one-shot cron seed: 5. `kubectl exec` in and run the one-shot cron seed:
`hermes cron create ...` (or apply the `cron-seed` Job). `hermes cron create ...` (or apply the `cron-seed` Job).
6. Trigger the first `cluster-health-check` manually: `hermes cron run cluster-health-check`. 6. Trigger the first `daily-cluster-check` manually: `hermes cron run daily-cluster-check`.
7. Add the app to `argocd/gen-apps.sh`, regenerate, commit, push. 7. Add the app to `argocd/gen-apps.sh`, regenerate, commit, push.
--- ---
@@ -324,12 +324,12 @@ Design rules baked into SOUL.md:
password + a 32-byte session secret. password + a 32-byte session secret.
5. **Commit & push** the whole change. ArgoCD will create the namespace 5. **Commit & push** the whole change. ArgoCD will create the namespace
resources, deploy the pod, and bring up the ingress at `platform-engineer.rogi.casa`. resources, deploy the pod, and bring up the ingress at `platform-engineer.rogi.casa`.
6. **Seed the cron jobs:** 6. **Seed the cron job:**
`kubectl apply -f platform-engineer/cron-seed.yaml` (one-shot Job) — it waits `kubectl apply -f platform-engineer/cron-seed.yaml` (one-shot Job) — it waits
for the hermes pod, then runs `hermes cron create ...` for each watchdog. for the Hermes pod, removes the legacy schedules, and creates the single
Re-run it any time you want to re-seed after a wipe. daily check. Re-run it to migrate an existing PVC or after a wipe.
7. **Smoke test:** trigger the first health check manually — 7. **Smoke test:** trigger the daily health check manually —
`kubectl exec -n platform-engineer deploy/hermes -- hermes cron run cluster-health-check` — `kubectl exec -n platform-engineer deploy/hermes -- hermes cron run daily-cluster-check` —
and confirm the message lands in Discord. and confirm the message lands in Discord.
8. **ArgoCD:** the `Application` (`argocd/apps/platform-engineer.yaml`) is 8. **ArgoCD:** the `Application` (`argocd/apps/platform-engineer.yaml`) is
already generated. After commit, Argo will reconcile it like every other app. already generated. After commit, Argo will reconcile it like every other app.
@@ -339,10 +339,11 @@ Design rules baked into SOUL.md:
- **ArgoCD owns** (in git): namespace, RBAC, Secret, ConfigMap (seed), PVC, - **ArgoCD owns** (in git): namespace, RBAC, Secret, ConfigMap (seed), PVC,
Deployment, Service, Ingress, cron-seed Job. Deployment, Service, Ingress, cron-seed Job.
- **Runtime state (on the PVC, NOT reconciled):** `config.yaml`, `SOUL.md`, - **Runtime state (on the PVC, NOT reconciled):** `config.yaml`, `SOUL.md`,
`.env`, `cron/jobs.json`, `sessions/`, `memories/`, `skills/`. The ConfigMap `.env`, `cron/jobs.json`, `sessions/`, `memories/`, `skills/`. The cron seed
only *seeds* these on first boot; after that, edits you make via the migrates the existing `cron/jobs.json` to the single daily check, but the
dashboard or `hermes cron edit` persist on the PVC and Argo will not revert file remains runtime state afterward. The ConfigMap only *seeds* these on
them. If you ever want a hard reset, delete the PVC and re-apply. first boot; after that, edits you make via the dashboard or `hermes cron edit`
persist on the PVC and Argo will not revert them. If you ever want a hard reset, delete the PVC and re-apply.
--- ---

89
platform-engineer/cron-seed.yaml
View File

@@ -1,10 +1,10 @@
# One-shot Job that seeds Hermes' built-in cron schedule on first install. # One-shot Job that replaces Hermes' built-in cron schedule with one daily check.
# Idempotent: skips job names that already exist. # The seed is intentionally destructive for the known legacy jobs so existing
# PVC state is simplified on the next ArgoCD sync.
# #
# Cron prompts are deliberately written as plain-English instructions (no inline # The prompt is written as plain-English instructions to avoid triggering
# curl commands) to avoid tripping Hermes' threat-pattern scanner, which blocks # Hermes' threat-pattern scanner. API endpoints and query examples are in
# cron prompts containing curl+auth-header patterns. The exact API endpoints and # SOUL.md.
# query examples are documented in the agent's SOUL.md instead.
--- ---
apiVersion: batch/v1 apiVersion: batch/v1
kind: Job kind: Job
@@ -48,72 +48,25 @@ spec:
echo "Waiting for hermes pod to be Ready..." echo "Waiting for hermes pod to be Ready..."
kubectl -n platform-engineer wait --for=condition=Ready pod -l app=hermes --timeout=300s || true kubectl -n platform-engineer wait --for=condition=Ready pod -l app=hermes --timeout=300s || true
POD=$(kubectl -n platform-engineer get pod -l app=hermes -o jsonpath='{.items[0].metadata.name}') POD=$(kubectl -n platform-engineer get pod -l app=hermes -o jsonpath='{.items[0].metadata.name}')
echo "Using pod: $POD" echo "Using pod: $POD"
exists() { kubectl -n platform-engineer exec "$POD" -- hermes cron list 2>/dev/null | grep -qi " $1 "; } # Cron state lives on the PVC, so remove the old schedules as well
# as creating the new one. This makes the migration idempotent.
for name in \
cluster-health-check node-health-check pod-phase-check pod-error-log-check \
pod-restart-loop pvc-pressure argocd-sync-health cert-expiry \
node-resource-drift daily-cluster-report daily-cluster-check; do
echo "removing legacy cron '$name' (if present)"
kubectl -n platform-engineer exec "$POD" -- hermes cron delete "$name" 2>/dev/null || true
done
# NOTE: `hermes cron create` has no --model flag. New jobs inherit the echo "creating daily cluster check"
# global default model from config.yaml at creation time. To pin a # Hermes runs cron expressions in UTC. 04:00 UTC is 06:00 local time
# specific model per-job (e.g. deepseek-v4-flash-0731 for all # for this cluster (UTC+2).
# cron tasks), edit /opt/data/cron/jobs.json directly after seeding: kubectl -n platform-engineer exec "$POD" -- hermes cron create "0 4 * * *" \
# kubectl exec deploy/hermes -- python3 -c "...set model field..." "Run the single daily platform health check for Roger using the HTTP APIs documented in your SOUL.md. Check: (1) every node is Ready and node CPU, memory, and root filesystem are below 90 percent; (2) PVCs have at least 15 percent free space; (3) pods not Running, excluding Completed/Succeeded jobs; (4) pod restart increases over the last 15 minutes and recent crash, error, panic, OOM, or backoff activity, using LogQL aggregation first and at most 10 sample lines for only an affected pod; (5) ArgoCD applications that are OutOfSync or Degraded; and (6) certificates expiring within 30 days. If everything is healthy, reply with exactly [SILENT]. Otherwise send one concise report in Catalan, grouped by issue, including namespace/resource names and relevant values. Keep it under 1800 characters. You may perform only clearly safe, idempotent GitOps remediations described in SOUL.md; otherwise report the proposed fix and wait for Roger." \
# See llamacpp/README.md and the deployment notes for details. --name "daily-cluster-check" --deliver "discord"
create() {
name="$1"; schedule="$2"; deliver="$3"; prompt="$4"
if exists "$name"; then
echo "cron job '$name' already exists — skipping"
else
echo "creating cron job '$name' ..."
kubectl -n platform-engineer exec "$POD" -- hermes cron create "$schedule" "$prompt" --name "$name" --deliver "$deliver"
fi
}
# ---- Watchdog checks (silent unless something is wrong) ----
# NOTE: the old single "cluster-health-check" combined 4 heavy HTTP
# gathers (Prometheus + a cluster-wide raw Loki log dump) into one
# cron session and exceeded the flash model's per-request context.
# The 35b-a3b llama-server runs -c 131072 -np 4, so each request only
# gets ~32k tokens of KV cache (NOT 132k). It has been split into
# focused sub-checks so each run stays small, and the log check now
# uses LogQL aggregation instead of dumping raw lines cluster-wide.
create "node-health-check" "every 6h" "discord" \
"Check node health using the Prometheus API documented in your SOUL.md. Query kube_node_status_condition{condition=\"Ready\",status!=\"true\"}. If all nodes are Ready, reply with exactly [SILENT]. Otherwise list each NotReady node by name. Keep it short; do not paste raw JSON — summarize with jq."
create "pod-phase-check" "every 6h" "discord" \
"Check pod health using the Prometheus API documented in your SOUL.md. Query kube_pod_status_phase{phase!=\"Running\"} and exclude Completed/Succeeded pods (jobs). If all Running, reply with exactly [SILENT]. Otherwise list each non-Running pod as namespace/pod:phase, grouped by namespace. Summarize with jq; do not dump raw JSON."
create "pod-error-log-check" "every 6h" "discord" \
"Check for recent crash-loop/error logs using the Loki API documented in your SOUL.md. IMPORTANT context discipline: do NOT fetch raw log lines across all namespaces — that overflows your context window. First run a LogQL aggregation that returns only counts: sum by (namespace, pod) (count_over_time({namespace=~\".+\"} |~ \"(?i)backoff|crashloop|panic\" [20m])). Parse the counts with jq. If every count is zero, reply with exactly [SILENT]. If any namespace/pod has a non-zero count, fetch at most 10 sample lines for THAT pod only (limit=10) to identify the cause. Summarize concisely; never paste more than a handful of lines."
# (ArgoCD health is already covered by the argocd-sync-health cron below.)
create "pod-restart-loop" "every 1h" "discord" \
"Find pods with high restart rates using the Prometheus API documented in your SOUL.md. Run increase(kube_pod_container_status_restarts_total[15m]) and list only pods whose value is greater than 3. If none, reply [SILENT]. For any pod that qualifies, IMPORTANT context discipline: do NOT fetch raw logs across the cluster or over a wide time window — that overflows your context window (a raw cluster-wide Loki dump can exceed 60k tokens). Fetch at most 10 sample lines for THAT specific pod only, using Loki query_range with limit=10 over the last 15 minutes, filtered to error/crash patterns (e.g. {namespace=\"<ns>\",pod=\"<pod>\"} |~ \"(?i)error|panic|crash|oom|backoff\"). Summarize the root cause in one line; never paste more than a handful of log lines. If the cause is clearly fixable via a manifest change such as bumping a memory limit, fixing a config value, or bumping the restartedAt annotation, make the edit in /workspace/k3s-cluster, commit and push, then trigger an ArgoCD sync via the API. Report what you did in one line. If not clearly fixable, post the log excerpt and proposed fix, and wait for Roger."
create "pvc-pressure" "every 1d" "discord" \
"Check storage health using the Prometheus API documented in your SOUL.md. Alert if any PVC has less than 15 percent free space, or if any node filesystem is over 85 percent full. If all healthy, reply [SILENT]."
create "argocd-sync-health" "every 6h" "discord" \
"Check ArgoCD app health using the API documented in your SOUL.md. If every app is Synced and Healthy, reply [SILENT]. Otherwise list the OutOfSync or Degraded apps with their status. If an app is OutOfSync and you believe a recent git push caused it, you may trigger a sync via the API. Do NOT hand-edit resources to fix them — fix the source repo."
create "cert-expiry" "0 7 * * *" "discord" \
"Check certificate expiry using the Prometheus API documented in your SOUL.md. Alert on any certificate expiring in under 21 days, with its name and namespace. If none, reply [SILENT]."
create "node-resource-drift" "every 1d" "discord" \
"Check node resources using the Prometheus API documented in your SOUL.md. Alert if any node is NotReady, or if any node has CPU over 90 percent or memory over 90 percent. Otherwise reply [SILENT]."
# ---- Daily report (always delivered) ----
# NOTE: Hermes' cron runs in UTC (no TZ set on the container). The
# cluster is at UTC+2, so fixed daily times are expressed in UTC and
# shifted back 2h from the intended local wall-clock time:
# 08:00 local -> 0 6 (daily cluster report)
# 09:00 local -> 0 7 (cert expiry check)
# Relative schedules (every 6h / every 1h / every 1d) are
# timezone-independent.
create "daily-cluster-report" "0 6 * * *" "discord" \
"Produce a daily cluster report for Roger using the HTTP APIs documented in your SOUL.md. Include: (1) node count and Ready/NotReady status per node, (2) top 5 pods by CPU and by memory, (3) count of pods not Running grouped by namespace, (4) any ArgoCD apps that are OutOfSync or Degraded, (5) any certificates expiring within 30 days, (6) recent error/crash activity: IMPORTANT context discipline — do NOT fetch raw log lines across the cluster over 24h — that overflows your context window. First run a LogQL aggregation that returns only counts: sum by (namespace, pod) (count_over_time({namespace=~\".+\"} |~ \"(?i)error|panic|crash|oom|backoff\" [24h])). Parse the counts with jq and summarize the top few noisiest pods by namespace/pod:count. If one stands out, fetch at most 5 sample lines for THAT pod only (limit=5) to identify the cause. Never paste more than a handful of lines total. Keep the whole report under 1800 chars. Always deliver (no [SILENT])."
echo "Done. Listing all cron jobs:" echo "Done. Listing all cron jobs:"
kubectl -n platform-engineer exec "$POD" -- hermes cron list kubectl -n platform-engineer exec "$POD" -- hermes cron list