370 lines
16 KiB
Markdown
370 lines
16 KiB
Markdown
# Platform Engineer Agent — Deployment Plan
|
||
|
||
An autonomous **Hermes Agent** that runs inside the k3s cluster, watches its
|
||
health on a schedule, tries to fix simple problems, and notifies me (via
|
||
Discord) when something needs my attention or a fix failed.
|
||
|
||
Docs: https://hermes-agent.nousresearch.com/docs/user-guide/docker
|
||
|
||
---
|
||
|
||
## 1. Goal & operating model
|
||
|
||
- **One Hermes container** in a new namespace `platform-engineer`, scheduled on
|
||
the powerful amd64 node (`roger-nucbox-evo-x2`, 24 GiB RAM).
|
||
- Hermes runs in **gateway mode** under s6 supervision (`command: gateway run`),
|
||
so the built-in **cron scheduler** is active and survives restarts.
|
||
- The agent talks to the cluster with `kubectl` from *inside* the container
|
||
(terminal backend = `local`). We give the pod a **ServiceAccount + ClusterRole**
|
||
scoped to read-mostly + restart/scale/delete-pod permissions.
|
||
- LLM calls are routed through the in-cluster **LiteLLM** proxy
|
||
(`litellm.rogi.casa`) — no external API keys needed in the cluster.
|
||
- Notifications go to **Discord** (reuse the pattern from `myorg-assistant`).
|
||
- A set of **cron jobs** (Hermes-native, not Kubernetes CronJobs) make the agent
|
||
run periodic checks. Watchdog checks use `[SILENT]` so it only pings me when
|
||
something is wrong.
|
||
|
||
Why Hermes-native cron (not k8s CronJobs):
|
||
- Hermes cron ticks inside the gateway, runs in an isolated agent session,
|
||
supports `[SILENT]` suppression, `deliver="discord"`, `workdir`, and
|
||
`context_from` chaining — far less plumbing than spawning a fresh pod per run.
|
||
- Cron jobs live in `~/.hermes/cron/jobs.json` on the PVC, so they survive pod
|
||
restarts and can be edited live via `hermes cron edit` without redeploying.
|
||
|
||
---
|
||
|
||
## 2. Files to create (this directory)
|
||
|
||
```
|
||
platform-engineer/
|
||
├── namespace.yaml # namespace platform-engineer
|
||
├── rbac.yaml # ServiceAccount + ClusterRole (+binding)
|
||
├── configmap.yaml # hermes config.yaml + SOUL.md + cron seed script
|
||
├── secret.yaml # DISCORD bot token, LITELLM_API_KEY, kubeconfig-less SA token
|
||
├── pvc.yaml # persistent /opt/data (HERMES_HOME)
|
||
├── dockerfile # derived image: hermes-agent + kubectl + helm
|
||
├── deployment.yaml # Deployment, schedules on amd64, mounts kube SA token
|
||
├── ingress.yaml # platform-engineer.rogi.casa → dashboard (optional)
|
||
└── README.md # this file
|
||
```
|
||
|
||
Then add a line to `argocd/gen-apps.sh` `APPS=(...)`:
|
||
```
|
||
"platform-engineer|platform-engineer|platform-engineer|true|true"
|
||
```
|
||
and re-run `./argocd/gen-apps.sh` to generate `argocd/apps/platform-engineer.yaml`
|
||
so ArgoCD reconciles it like every other app in the repo.
|
||
|
||
---
|
||
|
||
## 3. RBAC — least privilege
|
||
|
||
ServiceAccount `platform-engineer` in ns `platform-engineer`, bound to a
|
||
**ClusterRole** scoped to *platform engineer* actions:
|
||
|
||
**Read (get/list/watch):** nodes, pods, services, deployments, statefulsets,
|
||
daemonsets, replicasets, jobs, cronjobs, events, configmaps, secrets, PVCs,
|
||
ingresses, namespaces.
|
||
|
||
**Act (patch/update on a allowlist):**
|
||
- `pods` → `delete` (force-restart a stuck pod), `patch` (`/evict`, annotations)
|
||
- `deployments`, `statefulsets`, `daemonsets`, `replicasets` → `patch` (restart
|
||
via `kubectl rollout restart` / scale), `update`
|
||
- `jobs`, `cronjobs` → `delete`, `patch`
|
||
- `pods/exec` (subresource) → `create` (only if we want the agent to `kubectl
|
||
exec` into pods for log-style debugging — optional; keep off initially)
|
||
- `events` → `get/list/watch` only
|
||
|
||
**No cluster-scoped writes** (no creating namespaces, no node taints, no RBAC
|
||
edits, no CRDs). The agent can *propose* those and tell me; it cannot do them
|
||
itself. All mutating calls are auditable via Kubernetes audit logs and
|
||
`kubectl auth can-i --as=system:serviceaccount:platform-engineer:platform-engineer`.
|
||
|
||
The pod uses the k3s in-cluster ServiceAccount token (`/var/run/secrets/...
|
||
/serviceaccount/token`) + the `KUBERNETES_SERVICE_HOST/PORT` env vars k3s already
|
||
injects — **no kubeconfig file, no long-lived token on disk**.
|
||
|
||
---
|
||
|
||
## 4. Image — thin derived Dockerfile
|
||
|
||
```dockerfile
|
||
FROM nousresearch/hermes-agent:latest
|
||
USER root
|
||
RUN apt-get update \
|
||
&& apt-get install -y --no-install-recommends curl gnupg \
|
||
&& curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key \
|
||
| gpg --dearmor -o /usr/share/keyrings/kubernetes-apt-keyring.gpg \
|
||
&& echo 'deb [signed-by=/usr/share/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /' \
|
||
> /etc/apt/sources.list.d/kubernetes.list \
|
||
&& apt-get update \
|
||
&& apt-get install -y --no-install-recommends kubectl \
|
||
&& curl -fsSL https://get.helm.sh/helm-v3.16.0-linux-amd64.tar.gz \
|
||
| tar -xz -C /usr/local/bin --strip-components=1 linux-amd64/helm \
|
||
&& rm -rf /var/lib/apt/lists/*
|
||
USER hermes
|
||
```
|
||
|
||
> Note: the cluster is mixed arch (arm64/amd64/arm). The agent pod is pinned to
|
||
> the amd64 node, so `linux-amd64` helm + `kubectl` packages are fine. If you
|
||
> later want it portable, switch to a multi-arch build with
|
||
> `TARGETARCH` and install matching helm arch.
|
||
|
||
Build & push to your Gitea registry (`git.rogi.casa/roger/...`) — same
|
||
`imagePullSecrets: gitea-registry` pattern as `gym-tracker`. Tag with the
|
||
hermes version + a short git sha.
|
||
|
||
---
|
||
|
||
## 5. Hermes configuration (mounted via ConfigMap → /opt/data/config.yaml)
|
||
|
||
```yaml
|
||
# config.yaml (seeded into the PVC on first boot)
|
||
model:
|
||
provider: openai-api
|
||
default: claude-4.5-haiku
|
||
base_url: "https://litellm.rogi.casa/v1"
|
||
api_mode: chat_completions
|
||
|
||
# Use a cheap, fast model for auxiliary tasks (titling, compression)
|
||
auxiliary:
|
||
compression:
|
||
provider: openai-api
|
||
model: gemini-3.6-flash
|
||
title_generation:
|
||
provider: openai-api
|
||
model: gemini-3.6-flash
|
||
|
||
terminal:
|
||
backend: local
|
||
cwd: /workspace # a working dir for any kubectl output / scratch
|
||
timeout: 180
|
||
home_mode: profile # isolate tool credentials under HERMES_HOME/home
|
||
|
||
# Unattended gateway → circuit-breaker on tool-call loops
|
||
tool_loop_guardrails:
|
||
hard_stop_enabled: true
|
||
hard_stop_after:
|
||
exact_failure: 5
|
||
idempotent_no_progress: 5
|
||
|
||
sessions:
|
||
auto_prune: true
|
||
retention_days: 90
|
||
|
||
cron:
|
||
wrap_response: false # cleaner Discord messages
|
||
|
||
memory:
|
||
memory_enabled: true
|
||
user_profile_enabled: true
|
||
```
|
||
|
||
`.env` (from Secret, mounted to `/opt/data/.env`):
|
||
```
|
||
OPENAI_API_KEY=<LITELLM_API_KEY value, i.e. sk-...>
|
||
OPENAI_BASE_URL=https://litellm.rogi.casa/v1
|
||
DISCORD_BOT_TOKEN=<new dedicated bot token>
|
||
DISCORD_HOME_CHANNEL=<your user/channel id for alerts>
|
||
# Dashboard auth (homelab, trusted LAN behind ingress)
|
||
HERMES_DASHBOARD_BASIC_AUTH_USERNAME=roger
|
||
HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=<strong password>
|
||
```
|
||
|
||
> Why `OPENAI_API_KEY` + `OPENAI_BASE_URL`: the `openai-api` provider honours
|
||
> `OPENAI_BASE_URL`, so this is the simplest way to point Hermes at the
|
||
> in-cluster LiteLLM. `claude-4.5-haiku` / `gemini-3.6-flash` are the model names
|
||
> already exposed by your `litellm/litellm.yaml` ConfigMap.
|
||
|
||
`SOUL.md` (personality + guardrails) — see `configmap.yaml`. Key points:
|
||
- Identity: "Platform Engineer for the rogi.casa k3s cluster."
|
||
- Knows the cluster layout (3 nodes, ArgoCD GitOps, Traefik+cert-manager,
|
||
LiteLLM, services list).
|
||
- Operating rules: read-first; only act on the allowlisted verbs; never edit
|
||
RBAC / taints / namespaces / CRDs; when in doubt, notify instead of acting;
|
||
always cite the resource and the command used.
|
||
- How to reach me: `deliver="discord"`.
|
||
|
||
---
|
||
|
||
## 6. Deployment
|
||
|
||
- `replicas: 1` (Hermes data dir is single-writer — never scale >1).
|
||
- `nodeSelector: kubernetes.io/arch: amd64` + preferred `hardware: high-memory`
|
||
affinity → lands on the NUC.
|
||
- `resources`: requests 512Mi/250m, limits 2Gi/1 core (Hermes recommends
|
||
2–4 GiB; 1 GiB is fine without browser tools, which we keep off).
|
||
- Volume: PVC mounted at `/opt/data` (HERMES_HOME), RWX not needed (single pod).
|
||
- Ports: 8642 (gateway API, internal only) and 9119 (dashboard) → exposed via
|
||
Ingress `platform-engineer.rogi.casa` with TLS + basic-auth (already enforced by the
|
||
`HERMES_DASHBOARD_BASIC_AUTH_*` env vars).
|
||
- `imagePullSecrets: gitea-registry`.
|
||
- env from Secret; `HERMES_DASHBOARD=1`.
|
||
- Init: on first boot the s6 `01-hermes-setup` hook seeds config/SOUL/.env from
|
||
the ConfigMap if the volume is empty. We mount the ConfigMap as a readonly
|
||
projection at `/opt/seed/` and run a tiny initContainer to copy it into
|
||
`/opt/data` only when `/opt/data/config.yaml` doesn't exist (so ArgoCD
|
||
self-heal never fights the agent's live-edited config).
|
||
|
||
---
|
||
|
||
## 7. Single daily cron job (Hermes-native)
|
||
|
||
The one-shot Job `hermes-cron-seed` creates one daily check at 06:00 local cluster time (`0 4 * * *` UTC; Hermes has no timezone configured). It removes the known legacy jobs first, then runs
|
||
`hermes cron create ...` against the gateway and is idempotent
|
||
(it removes old names and recreates the daily job). It delivers to Discord and combines node, resource, PVC, pod, ArgoCD, and certificate checks. It stays silent when healthy and sends one concise Catalan report when there is an issue.
|
||
|
||
| Name | Schedule | Prompt (abbreviated) |
|
||
|------|----------|------------------------|
|
||
| `daily-cluster-check` | `0 4 * * *` UTC (06:00 local) | Combined daily platform health check; `[SILENT]` when healthy. |
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
Design rules baked into SOUL.md:
|
||
- **Read-only checks** run frequently (10–30m) and stay silent unless wrong.
|
||
- **Mutating actions** are restricted to safe idempotent ones (rollout restart,
|
||
delete stuck pod so controller recreates). Anything riskier → notify me with
|
||
a proposed command and wait for me to run it (I can reply in Discord to the
|
||
continuable thread).
|
||
- Cron sessions are isolated and **cannot create new cron jobs** (Hermes
|
||
disables that inside cron runs) → no runaway loops.
|
||
|
||
---
|
||
|
||
## 8. Safety & guardrails
|
||
|
||
1. **RBAC is the real boundary.** Even if the agent goes rogue, the SA can't
|
||
touch other namespaces' secrets beyond read, can't change RBAC, can't taint
|
||
nodes, can't create namespaces.
|
||
2. **`tool_loop_guardrails.hard_stop_enabled: true`** — circuit-breaks a stuck
|
||
gateway (recommended in the Docker doc for unattended deployments).
|
||
3. **`skills.write_approval: false` but `memory.write_approval: true`** (so the
|
||
agent can build skills/memories but I review memory writes lazily — flip
|
||
this if it gets noisy).
|
||
4. **No `pods/exec` subresource** initially (keep the agent from shelling into
|
||
workloads). Enable later only if you want log-grep-style debugging.
|
||
5. **Dashboard behind ingress TLS + basic auth** (the June-2026 hardening makes
|
||
auth mandatory on non-loopback binds; we satisfy it with the bundled
|
||
basic-auth provider).
|
||
6. **Single replica / single-writer PVC** — the Docker doc is explicit that two
|
||
gateways on the same `/opt/data` corrupt session/memory stores. Use a
|
||
`podAntiAffinity` so an accidental scale-up doesn't co-run.
|
||
7. **ArgoCD interaction:** keep `syncPolicy.automated.prune+selfHeal` but
|
||
exclude the live-edited hermes state. Practically: Argo owns the *manifests*
|
||
(deployment, configmap, secret, pvc), while `/opt/data` (config.yaml,
|
||
cron/jobs.json, SOUL.md edits made via the dashboard) is runtime state on the
|
||
PVC and is *not* reconciled by Argo. The ConfigMap only *seeds* it on first
|
||
boot. Document this clearly in the README so future-you doesn't expect Argo
|
||
to reset the agent's personality.
|
||
|
||
---
|
||
|
||
## 9. Rollout plan
|
||
|
||
1. Build & push the derived image to `git.rogi.casa/roger/hermes-agent` (tag
|
||
`v1.35-<sha>`).
|
||
2. Create the namespace + RBAC + Secret + ConfigMap + PVC:
|
||
`kubectl apply -f platform-engineer/`.
|
||
3. Create the `platform-engineer` Discord bot, invite it, put its token + your
|
||
channel id in `secret.yaml` (base64).
|
||
4. Apply the Deployment; wait for the pod to go Running.
|
||
5. `kubectl exec` in and run the one-shot cron seed:
|
||
`hermes cron create ...` (or apply the `cron-seed` Job).
|
||
6. Trigger the first `daily-cluster-check` manually: `hermes cron run daily-cluster-check`.
|
||
7. Add the app to `argocd/gen-apps.sh`, regenerate, commit, push.
|
||
|
||
---
|
||
|
||
## 10. Decisions (locked in)
|
||
|
||
1. **Notifications:** dedicated `platform-engineer` Discord bot → its own token
|
||
in `secret.yaml` (`DISCORD_BOT_TOKEN`, `DISCORD_HOME_CHANNEL`).
|
||
2. **Dashboard:** public at `platform-engineer.rogi.casa` (Traefik TLS + cert-manager + the
|
||
bundled Hermes basic-auth provider). Reach the dashboard on port 9119; the
|
||
gateway API on 8642 is ClusterIP-only.
|
||
3. **Image:** derived image pushed to `git.rogi.casa/roger/hermes-agent`, pulled
|
||
via the existing `gitea-registry` imagePullSecret (must also exist in the
|
||
`platform-engineer` ns — see deploy steps).
|
||
4. **Model:** `qwen-3.6:27b` via the in-cluster Ollama box (`10.88.20.12:11434`),
|
||
exposed through LiteLLM as `qwen-3.6:27b`. Added to `litellm/litellm.yaml`.
|
||
Hermes reaches LiteLLM at `https://litellm.rogi.casa/v1` (never Ollama directly).
|
||
5. **pods/exec:** granted (`pods/exec` → `create` in the ClusterRole) so the
|
||
agent can `kubectl exec`/`kubectl logs` for debugging.
|
||
|
||
---
|
||
|
||
## 11. Deployment checklist (do in this order)
|
||
|
||
1. **Add the in-cluster model to LiteLLM** (already done in `litellm/litellm.yaml`):
|
||
the `deepseek-v4-flash-0731` entry points at the in-cluster llama.cpp server
|
||
`http://llamacpp-deepseek-v4-flash-0731.llamacpp/v1` (served from the NUCBox
|
||
APU; see `llamacpp/deployment-deepseek-v4-flash-0731.yaml`). Apply:
|
||
`kubectl apply -f litellm/` and restart the LiteLLM pod so the new config
|
||
takes effect.
|
||
2. **Create the `gitea-registry` secret in the new namespace** (ArgoCD won't
|
||
create it — it's not in the repo):
|
||
```
|
||
kubectl create namespace platform-engineer
|
||
kubectl create secret docker-registry gitea-registry \
|
||
--docker-server=git.rogi.casa \
|
||
--docker-username=<your-gitea-user> \
|
||
--docker-password=<gitea-access-token> \
|
||
--docker-email=<your-email> \
|
||
-n platform-engineer
|
||
```
|
||
3. **Build & push the image:** `./platform-engineer/build-and-push.sh`
|
||
(after `docker login git.rogi.casa`).
|
||
4. **Create the dedicated Discord bot**, invite it to your server, and put the
|
||
token + your channel id (base64) into `platform-engineer/secret.yaml`. Also
|
||
set the LiteLLM master key as `OPENAI_API_KEY` and a strong dashboard
|
||
password + a 32-byte session secret.
|
||
5. **Commit & push** the whole change. ArgoCD will create the namespace
|
||
resources, deploy the pod, and bring up the ingress at `platform-engineer.rogi.casa`.
|
||
6. **Seed the cron job:**
|
||
`kubectl apply -f platform-engineer/cron-seed.yaml` (one-shot Job) — it waits
|
||
for the Hermes pod, removes the legacy schedules, and creates the single
|
||
daily check. Re-run it to migrate an existing PVC or after a wipe.
|
||
7. **Smoke test:** trigger the daily health check manually —
|
||
`kubectl exec -n platform-engineer deploy/hermes -- hermes cron run daily-cluster-check` —
|
||
and confirm the message lands in Discord.
|
||
8. **ArgoCD:** the `Application` (`argocd/apps/platform-engineer.yaml`) is
|
||
already generated. After commit, Argo will reconcile it like every other app.
|
||
|
||
## 12. What ArgoCD owns vs. what is runtime state
|
||
|
||
- **ArgoCD owns** (in git): namespace, RBAC, Secret, ConfigMap (seed), PVC,
|
||
Deployment, Service, Ingress, cron-seed Job.
|
||
- **Runtime state (on the PVC, NOT reconciled):** `config.yaml`, `SOUL.md`,
|
||
`.env`, `cron/jobs.json`, `sessions/`, `memories/`, `skills/`. The cron seed
|
||
migrates the existing `cron/jobs.json` to the single daily check, but the
|
||
file remains runtime state afterward. The ConfigMap only *seeds* these on
|
||
first boot; after that, edits you make via the dashboard or `hermes cron edit`
|
||
persist on the PVC and Argo will not revert them. If you ever want a hard reset, delete the PVC and re-apply.
|
||
|
||
---
|
||
|
||
## Files in this directory
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `namespace.yaml` | namespace `platform-engineer` |
|
||
| `rbac.yaml` | ServiceAccount + ClusterRole (+binding), least-privilege |
|
||
| `configmap.yaml` | seed `config.yaml` + `SOUL.md` |
|
||
| `secret.yaml` | Discord token, LiteLLM key, dashboard auth (PLACEHOLDERS — fill in) |
|
||
| `pvc.yaml` | 5 Gi PVC for `/opt/data` |
|
||
| `dockerfile` | derived image: hermes-agent + kubectl + helm (linux/amd64) |
|
||
| `build-and-push.sh` | builds & pushes the image to the Gitea registry |
|
||
| `deployment.yaml` | Deployment (1 replica, Recreate, pinned to amd64 NUC) + Service |
|
||
| `ingress.yaml` | `platform-engineer.rogi.casa` → dashboard (TLS + basic auth) |
|
||
| `cron-seed.yaml` | one-shot Job that creates the Hermes cron schedule |
|
||
|
||
Also changed outside this directory:
|
||
- `litellm/litellm.yaml` — added `qwen-3.6:27b` model entry.
|
||
- `argocd/gen-apps.sh` + `argocd/apps/platform-engineer.yaml` — ArgoCD
|
||
Application for this folder.
|
||
```
|