fix(litellm): add memory and CPU resource limits to prevent OOMKilled

Litellm pod was being killed by the kernel OOM killer (7 restarts, last termination reason: OOMKilled). Added explicit resource requests (512Mi/250m) and limits (2Gi/1000m) to ensure the container gets bounded memory.
fix(platform): fix high-restart pods - prometheus nodeSelector, litellm resources, gitea-runner limits
2026-07-21 21:59:15 +00:00 · 2026-07-21 20:28:12 +00:00 · 2026-07-21 15:36:57 +02:00 · 2026-07-20 22:19:37 +00:00 · 2026-07-19 12:01:21 +02:00 · 2026-07-19 12:00:52 +02:00
30 changed files with 1437 additions and 265 deletions
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -0,0 +1,246 @@
+# AGENTS.md - Guide for Coding Agents
+
+This file provides essential information for AI coding agents working with this Kubernetes cluster project.
+
+## Project Overview
+
+This repository contains Kubernetes manifests for a K3s cluster running self-hosted services on the `rogi.casa` domain. The cluster is managed via **GitOps using ArgoCD** - all changes to the cluster are deployed automatically from this Git repository.
+
+**⚠️ CRITICAL: Permission Model**
+
+You **DO NOT** have permission to push changes to this repository. Before applying any changes to the cluster:
+1. Make the necessary code changes to the manifests
+2. Clearly present the changes to the user
+3. Ask the user to review and push the changes
+4. Wait for confirmation that changes have been pushed
+5. Only then will ArgoCD automatically deploy the changes to the cluster
+
+## Architecture & GitOps Workflow
+
+### ArgoCD App-of-Apps Pattern
+
+This project uses ArgoCD's "app-of-apps" pattern:
+
+```
+argocd-bootstrap.yaml (root Application)
+    ↓
+argocd/apps/ (directory containing all Application manifests)
+    ↓
+Individual Applications (one per service directory)
+    ↓
+Kubernetes manifests in each service directory (e.g., pihole/, homeassistant/)
+```
+
+### Deployment Flow
+
+1. You make changes to Kubernetes manifests in the repository
+2. User reviews and pushes changes to the `main` branch
+3. ArgoCD detects changes (automatically or on sync)
+4. ArgoCD applies changes to the cluster with `prune: true` and `selfHeal: true`
+5. Cluster state converges to match the Git state
+
+### Key Files
+
+- **`argocd-bootstrap.yaml`**: The root Application that bootstraps ArgoCD. Points to `argocd/apps/` directory. This is the only file that needs manual `kubectl apply` during initial setup.
+- **`argocd/apps/project.yaml`**: ArgoCD AppProject defining permissions for all applications
+- **`argocd/apps/*.yaml`**: Individual ArgoCD Application manifests (one per service)
+- **`argocd/gen-apps.sh`**: Script to regenerate all ArgoCD manifests from the `APPS` array
+
+## Repository Structure
+
+```
+k3s-cluster/
+├── argocd-bootstrap.yaml          # Root ArgoCD Application (app-of-apps)
+├── argocd/
+│   ├── apps/                      # Individual ArgoCD Application manifests
+│   │   ├── project.yaml           # AppProject definition
+│   │   ├── pihole.yaml            # Application for pihole/
+│   │   ├── homeassistant.yaml     # Application for homeassistant/
+│   │   └── ...                    # One per service
+│   ├── gen-apps.sh                # Generates argocd/apps/* manifests
+│   └── ingress.yaml               # ArgoCD's own ingress
+├── <service-name>/                # Each service has its own directory
+│   ├── namespace.yaml             # (Optional) Namespace definition
+│   ├── deployment.yaml            # Main deployment/statefulset
+│   ├── service.yaml               # Service definition
+│   ├── ingress.yaml               # Ingress configuration
+│   ├── configmap.yaml             # (Optional) ConfigMaps
+│   ├── pvc.yaml                   # (Optional) PersistentVolumeClaims
+│   └── secret.yaml                # (Optional) Secrets (rarely committed)
+├── cert-manager/                  # cert-manager installation manifests
+├── nas/                           # External NAS service configuration
+├── monitoring/                    # Prometheus + Grafana stack
+└── README.md                      # Comprehensive project documentation
+```
+
+## Current Services
+
+The cluster runs these services (each in its own directory):
+
+- **argocd** - GitOps continuous delivery platform
+- **cert-manager** - SSL certificate management (Let's Encrypt)
+- **fava** - Beancount accounting web interface
+- **gitea** - Self-hosted Git server
+- **glance** - Personal dashboard
+- **gym-tracker** - Workout tracking application
+- **homeassistant** - Home automation
+- **jellyfin** - Media server
+- **litellm** - LLM proxy
+- **minecraft-server** - Minecraft server
+- **monitoring** - Prometheus + Grafana
+- **myorg-assistant** - Organization assistant
+- **n8n** - Workflow automation
+- **nas** - External NAS proxy
+- **openwebui** - Web UI for LLMs
+- **phoenix** - AI observability platform
+- **pihole** - Network-wide ad blocking
+- **platform-engineer** - Platform engineering tools
+- **qbittorrent** - Torrent client
+- **searxng** - Meta search engine
+- **vaultwarden** - Password manager (Bitwarden compatible)
+
+## How to Make Changes
+
+### Adding a New Service
+
+1. Create a new directory: `mkdir new-service`
+2. Create Kubernetes manifests in `new-service/`:
+   - `namespace.yaml` (if dedicated namespace needed)
+   - `deployment.yaml` or `statefulset.yaml`
+   - `service.yaml`
+   - `ingress.yaml`
+   - Any ConfigMaps, Secrets, PVCs needed
+3. Add the service to `argocd/gen-apps.sh`:
+   - Add a line to the `APPS` array: `"new-service|namespace|new-service|true|true"`
+   - Format: `name|namespace|path|recurse|validate`
+4. Run `./argocd/gen-apps.sh` to regenerate ArgoCD manifests
+5. **Present changes to user for review and push**
+
+### Modifying an Existing Service
+
+1. Edit the relevant manifest(s) in the service directory
+2. If changing ArgoCD configuration, also update `argocd/gen-apps.sh` and regenerate
+3. **Present changes to user for review and push**
+
+### Removing a Service
+
+1. Remove the service directory: `rm -rf service-name/`
+2. Remove from `APPS` array in `argocd/gen-apps.sh`
+3. Run `./argocd/gen-apps.sh` to regenerate
+4. **Present changes to user for review and push**
+5. ArgoCD will automatically prune the resources from the cluster
+
+## Common Patterns
+
+### Ingress Configuration
+
+Each service has its own `ingress.yaml` with:
+- `ingressClassName: traefik` (K3s default)
+- TLS configured with `cert-manager.io/cluster-issuer: letsencrypt-prod`
+- Host-based routing (e.g., `pihole.rogi.casa`)
+
+Example:
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: pihole
+  namespace: pihole
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+spec:
+  ingressClassName: traefik
+  tls:
+  - hosts:
+    - pihole.rogi.casa
+    secretName: pihole-tls
+  rules:
+  - host: pihole.rogi.casa
+    http:
+      paths:
+        - path: /
+          pathType: Prefix
+          backend:
+            service:
+              name: pihole-web
+              port:
+                number: 80
+```
+
+### Resource Management
+
+- Each service typically has its own namespace
+- Use ResourceRequests and Limits for all containers
+- PVCs for persistent data
+- ConfigMaps for configuration files
+
+## Important Notes
+
+### What You CAN Do
+
+- Read and understand all manifests
+- Create new manifest files
+- Modify existing manifest files
+- Run `./argocd/gen-apps.sh` to regenerate ArgoCD manifests
+- Explain how the cluster works
+- Troubleshoot issues by reading manifests
+
+### What You CANNOT Do
+
+- Push changes to the Git repository (no push permissions)
+- Directly apply manifests with `kubectl apply` (unless explicitly asked)
+- Access the Kubernetes cluster directly (unless explicitly configured)
+- Create secrets that should remain private (those are managed manually)
+
+### Secrets Management
+
+Secrets are generally **not committed to the repository**. They must be created manually in the cluster:
+```bash
+kubectl create secret docker-registry gitea-registry \
+  --docker-server=gitea.rogi.casa \
+  --docker-username=<user> \
+  --docker-password=<token> \
+  -n <namespace>
+```
+
+## Workflow Summary
+
+When asked to make changes:
+
+1. **Understand** the current state by reading relevant files
+2. **Modify** the manifests (create/edit files)
+3. **Regenerate** ArgoCD manifests if needed (`./argocd/gen-apps.sh`)
+4. **Present** the changes clearly to the user:
+   ```
+   I've made the following changes:
+   - Modified pihole/deployment.yaml to update image version
+   - Regenerated argocd/apps/pihole.yaml
+   
+   Please review and push these changes to deploy them.
+   ```
+5. **Wait** for user confirmation that changes are pushed
+6. **Verify** (if possible) that ArgoCD has synced the changes
+
+## Useful Commands (for reference)
+
+```bash
+# Regenerate ArgoCD manifests after modifying gen-apps.sh
+./argocd/gen-apps.sh
+
+# Check ArgoCD applications status (requires kubectl access)
+kubectl get applications -n argocd
+
+# View logs of a pod (requires kubectl access)
+kubectl logs -n <namespace> <pod-name>
+
+# Check ingress status (requires kubectl access)
+kubectl get ingress -n <namespace>
+```
+
+## Questions?
+
+If you're unsure about anything:
+1. Read the comprehensive `README.md` in the repository root
+2. Check existing service directories for examples
+3. Ask the user for clarification before making changes
+4. Remember: **never push without explicit user review and approval**
--- a/argocd-bootstrap.yaml
+++ b/argocd-bootstrap.yaml
@@ -22,3 +22,9 @@ spec:
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
+  ignoreDifferences:
+    - group: argoproj.io
+      kind: Application
+      jsonPointers:
+        - /status
+        - /operation
--- a/argocd/apps/searxng.yaml
+++ b/argocd/apps/searxng.yaml
@@ -0,0 +1,24 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: searxng
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  project: k3s-cluster
+  source:
+    repoURL: https://git.rogi.casa/roger/k3s-cluster.git
+    targetRevision: main
+    path: searxng
+    directory:
+      recurse: true
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: searxng
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=false
--- a/argocd/argocd-cm.yaml
+++ b/argocd/argocd-cm.yaml
@@ -0,0 +1,19 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argocd-cm
+  namespace: argocd
+  labels:
+    app.kubernetes.io/name: argocd-cm
+    app.kubernetes.io/part-of: argocd
+data:
+  # Serve HTTP (no redirect to HTTPS) so the TLS-terminating Traefik ingress works.
+  # Without this, argocd-server redirects HTTP->HTTPS, causing an infinite
+  # redirect loop behind the ingress (argocd.rogi.casa unreachable).
+  server.insecure: "true"
+
+  # add an additional local user with apiKey and login capabilities
+  #   apiKey - allows generating API keys
+  #   login - allows to login using UI
+  accounts.roger: apiKey, login
+  accounts.platform-engineer: apiKey, login
--- a/argocd/argocd-rbac-cm.yaml
+++ b/argocd/argocd-rbac-cm.yaml
@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argocd-rbac-cm
+  namespace: argocd
+  labels:
+    app.kubernetes.io/name: argocd-rbac-cm
+    app.kubernetes.io/part-of: argocd
+data:
+  policy.csv: |
+    # Grant platform-engineer read-only access to applications
+    g, platform-engineer, role:readonly
+    
+    # Custom policy for platform-engineer with application read permissions
+    p, role:platform-engineer, applications, get, *, allow
+    p, role:platform-engineer, applications, list, *, allow
+    p, role:platform-engineer, clusters, get, *, allow
+    p, role:platform-engineer, clusters, list, *, allow
+    p, role:platform-engineer, repositories, get, *, allow
+    p, role:platform-engineer, repositories, list, *, allow
+    p, role:platform-engineer, projects, get, *, allow
+    p, role:platform-engineer, projects, list, *, allow
+    g, platform-engineer, role:platform-engineer
+  
+  # Default policy - deny by default (ArgoCD default)
+  policy.default: role:readonly
+  
+  # Enable RBAC
+  rbac.enabled: "true"
--- a/gitea/gitea.yaml
+++ b/gitea/gitea.yaml
@@ -25,6 +25,8 @@ metadata:
  namespace: gitea
  labels:
    app: gitea
+  annotations:
+    kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
 spec:
  replicas: 1
  selector:
@@ -101,6 +103,8 @@ metadata:
  namespace: gitea
  labels:
    app: gitea-runner
+  annotations:
+    kubectl.kubernetes.io/restartedAt: "2026-07-21T13:30:00Z"
 spec:
  replicas: 1
  selector:
@@ -115,7 +119,14 @@ spec:
        kubernetes.io/arch: arm64
      containers:
      - name: gitea-runner
-        image: vegardit/gitea-act-runner:latest
+        image: vegardit/gitea-act-runner:v0.12.0
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "1Gi"
+            cpu: "500m"
        env:
        - name: GITEA_INSTANCE_URL
          valueFrom:
--- a/homeassistant/homeassistant.yaml
+++ b/homeassistant/homeassistant.yaml
@@ -58,9 +58,9 @@ spec:
        image: ghcr.io/home-assistant/home-assistant:stable
        resources:
          requests:
-            memory: "256Mi"
-          limits:
            memory: "512Mi"
+          limits:
+            memory: "1Gi"
        ports:
        - containerPort: 8123
        volumeMounts:
--- a/litellm/litellm.yaml
+++ b/litellm/litellm.yaml
@@ -11,27 +11,48 @@ metadata:
 data:
  config.yaml: |
    model_list:
-      - model_name: gpt-5-mini
+      - model_name: gpt-5.6-luna
        litellm_params:
-          model: openai/gpt-5-mini-2025-08-07
+          model: openai/gpt-5.6-luna
          api_key: "os.environ/OPENAI_API_KEY"
-      - model_name: claude-4.5-haiku
+      - model_name: claude-haiku-4.5
        litellm_params:
          model: "anthropic/claude-haiku-4-5-20251001"
          api_key: "os.environ/ANTHROPIC_API_KEY"
+      - model_name: claude-sonnet-5
+        litellm_params:
+          model: "anthropic/claude-sonnet-5"
+          api_key: "os.environ/ANTHROPIC_API_KEY"
      - model_name: gemini-3-flash
        litellm_params:
          model: gemini/gemini-3-flash-preview
          api_key: "os.environ/GEMINI_API_KEY"
+      - model_name: tencent/hy3:free
+        litellm_params:
+          model: openrouter/tencent/hy3:free
+          api_key: "os.environ/OPENROUTER_API_KEY"
+      - model_name: z-ai/glm-5.2
+        litellm_params:
+          model: openrouter/z-ai/glm-5.2
+          api_key: "os.environ/OPENROUTER_API_KEY"
      - model_name: glm-4.7-flash
        litellm_params:
          model: ollama/glm-4.7-flash
          api_base: http://10.88.20.12:11434
      # Used by the platform-engineer Hermes agent (deployed in ns platform-engineer).
-      - model_name: qwen-3.6:27b
+      # model_name is the alias Hermes requests; the underlying Ollama model is
+      # qwen3.6:latest (the fast non-27b tag). 27b is a slow reasoning model.
+      # `ollama_chat/` (not `ollama/`) uses Ollama's NATIVE /api/chat endpoint.
+      # `think: false` + `chat_template_kwargs.enable_thinking: false` disable
+      # Qwen3 thinking so the model emits content directly (otherwise the
+      # OpenAI-compat translation returns empty content with reasoning split off).
+      - model_name: qwen3.6
        litellm_params:
-          model: ollama/qwen3.6:27b
+          model: ollama_chat/qwen3.6:latest
          api_base: http://10.88.20.12:11434
+          think: false
+          chat_template_kwargs:
+            enable_thinking: false
    litellm_settings:
      #set_verbose: True  # Uncomment this if you want to see verbose logs; not recommended in production
      callbacks: ["arize_phoenix"]
@@ -91,6 +112,13 @@ spec:
        env:
        - name: STORE_MODEL_IN_DB
          value: "True"
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "2Gi"
+            cpu: "1000m"
      volumes:
        - name: config-volume
          configMap:
--- a/monitoring/grafana-datasource.yaml
+++ b/monitoring/grafana-datasource.yaml
@@ -13,3 +13,8 @@ data:
      url: http://prometheus:9090
      isDefault: true
      editable: true
+    - name: Loki
+      type: loki
+      access: proxy
+      url: http://loki:3100
+      editable: true
--- a/monitoring/loki.yaml
+++ b/monitoring/loki.yaml
@@ -0,0 +1,153 @@
+# Loki — log aggregation (single-binary mode, local filesystem storage).
+#
+# Stores compressed, indexed pod logs shipped by Promtail. Queried by the
+# platform-engineer Hermes agent via the HTTP API (LogQL) and by Grafana.
+#
+# Storage: 20 GiB local PVC, 1-week retention enforced by the compactor.
+# Service: loki.monitoring:3100 (ClusterIP, no auth — homelab).
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: loki-config
+  namespace: monitoring
+data:
+  loki.yaml: |
+    auth_enabled: false
+
+    server:
+      http_listen_port: 3100
+      grpc_listen_port: 9096
+
+    common:
+      path_prefix: /loki
+      replication_factor: 1
+      ring:
+        instance_addr: 127.0.0.1
+        kvstore:
+          store: inmemory
+
+    schema_config:
+      configs:
+        - from: 2024-01-01
+          store: tsdb
+          object_store: filesystem
+          schema: v13
+          index:
+            prefix: index_
+            period: 24h
+
+    storage_config:
+      filesystem:
+        directory: /loki/chunks
+      tsdb_shipper:
+        active_index_directory: /loki/tsdb-index
+        cache_location: /loki/tsdb-cache
+
+    limits_config:
+      retention_period: 168h          # 1 week
+      max_query_series: 10000
+      reject_old_samples: true
+      reject_old_samples_max_age: 168h
+      allow_structured_metadata: false  # tsdb v13 compat
+
+    compactor:
+      working_directory: /loki/compactor
+      compaction_interval: 10m
+      retention_enabled: true
+      retention_delete_delay: 2h
+      retention_delete_worker_count: 50
+      delete_request_store: filesystem
+
+    analytics:
+      reporting_enabled: false
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: loki-data
+  namespace: monitoring
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 20Gi
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: loki
+  namespace: monitoring
+  labels:
+    app: loki
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate          # single-writer storage
+  selector:
+    matchLabels:
+      app: loki
+  template:
+    metadata:
+      labels:
+        app: loki
+    spec:
+      nodeSelector:
+        kubernetes.io/arch: amd64   # Loki image; runs on the NUC
+      containers:
+      - name: loki
+        image: grafana/loki:3.4.4
+        args:
+          - -config.file=/etc/loki/loki.yaml
+        ports:
+        - name: http
+          containerPort: 3100
+        volumeMounts:
+        - name: config
+          mountPath: /etc/loki
+          readOnly: true
+        - name: data
+          mountPath: /loki
+        resources:
+          requests:
+            memory: "512Mi"
+            cpu: "250m"
+          limits:
+            memory: "1Gi"
+            cpu: "1000m"
+        readinessProbe:
+          httpGet:
+            path: /ready
+            port: 3100
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 5
+        livenessProbe:
+          httpGet:
+            path: /ready
+            port: 3100
+          initialDelaySeconds: 60
+          periodSeconds: 30
+          failureThreshold: 5
+      volumes:
+      - name: config
+        configMap:
+          name: loki-config
+      - name: data
+        persistentVolumeClaim:
+          claimName: loki-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: loki
+  namespace: monitoring
+spec:
+  type: ClusterIP
+  selector:
+    app: loki
+  ports:
+    - name: http
+      port: 3100
+      targetPort: 3100
--- a/monitoring/prometheus-deployment.yaml
+++ b/monitoring/prometheus-deployment.yaml
@@ -15,9 +15,10 @@ spec:
      labels:
        app: prometheus
    spec:
-      # Prevent scheduling on Raspberry Pi due to resource requirements (512Mi-1Gi memory, 500m-1000m CPU)
+      # Target the nucbox (amd64, 24Gi RAM) which is the only node with enough memory for Prometheus.
      nodeSelector:
-        hardware: high-memory
+        kubernetes.io/os: linux
+        kubernetes.io/arch: amd64
      serviceAccountName: prometheus
      containers:
        - name: prometheus
--- a/monitoring/promtail.yaml
+++ b/monitoring/promtail.yaml
@@ -0,0 +1,138 @@
+# Promtail — DaemonSet that tails pod logs on every node and ships them to Loki.
+#
+# Runs on ALL nodes (amd64 + arm). Multi-arch image. Reads /var/log/pods/*,
+# attaches k8s labels (namespace, pod, container), ships to loki.monitoring:3100.
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: promtail
+  namespace: monitoring
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: promtail
+rules:
+  - apiGroups: [""]
+    resources:
+      - nodes
+      - nodes/proxy
+      - services
+      - endpoints
+      - pods
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: promtail
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: promtail
+subjects:
+  - kind: ServiceAccount
+    name: promtail
+    namespace: monitoring
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: promtail-config
+  namespace: monitoring
+data:
+  promtail.yaml: |
+    server:
+      http_listen_port: 9080
+      grpc_listen_port: 0
+
+    positions:
+      filename: /tmp/positions.yaml
+
+    clients:
+      - url: http://loki.monitoring:3100/loki/api/v1/push
+
+    scrape_configs:
+      # Tail all container logs via /var/log/containers/*.log (symlinks to
+      # /var/log/pods/<ns>_<pod>_<uid>/<container>/<N>.log). Extract namespace,
+      # pod, container labels from the filename via pipeline_stages regex.
+      - job_name: kubernetes-containers
+        static_configs:
+          - targets:
+              - localhost
+            labels:
+              job: kube-containers
+              __path__: /var/log/containers/*.log
+        pipeline_stages:
+          - cri: {}
+          # k3s filename: <pod>_<namespace>_<container>-<hash>.log
+          - regex:
+              expression: '/var/log/containers/(?P<pod>[^_]+)_(?P<namespace>[^_]+)_(?P<container>[^-]+)-.*\.log'
+              source: filename
+          - labels:
+              pod:
+              namespace:
+              container:
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: promtail
+  namespace: monitoring
+  labels:
+    app: promtail
+spec:
+  selector:
+    matchLabels:
+      app: promtail
+  template:
+    metadata:
+      labels:
+        app: promtail
+    spec:
+      serviceAccountName: promtail
+      tolerations:
+        - operator: Exists   # run on every node including tainted Pis
+      containers:
+      - name: promtail
+        image: grafana/promtail:3.4.4
+        args:
+          - -config.file=/etc/promtail/promtail.yaml
+          - -config.expand-env=true
+        env:
+        - name: NODE_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        volumeMounts:
+        - name: config
+          mountPath: /etc/promtail
+          readOnly: true
+        - name: positions
+          mountPath: /tmp
+        - name: pods-logs
+          mountPath: /var/log/pods
+          readOnly: true
+        - name: containers-logs
+          mountPath: /var/log/containers
+          readOnly: true
+        resources:
+          requests:
+            memory: "64Mi"
+            cpu: "50m"
+          limits:
+            memory: "256Mi"
+            cpu: "250m"
+      volumes:
+      - name: config
+        configMap:
+          name: promtail-config
+      - name: positions
+        emptyDir: {}
+      - name: pods-logs
+        hostPath:
+          path: /var/log/pods
+      - name: containers-logs
+        hostPath:
+          path: /var/log/containers
--- a/myorg-assistant/cronjobs/deadline-checker.yaml
+++ b/myorg-assistant/cronjobs/deadline-checker.yaml
@@ -22,13 +22,16 @@ spec:
            job: deadline-checker
        spec:
          restartPolicy: OnFailure
+          imagePullSecrets:
+          - name: gitea-registry
          containers:
          - name: deadline-checker
-            image: myorg-assistant:latest
-            imagePullPolicy: IfNotPresent
+            image: git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf
+            imagePullPolicy: Always
            command:
            - python
-            - run_job.py
+            - -c
+            - "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
            - deadline-checker
            env:
            - name: MYORG_REPO_PATH
@@ -51,6 +54,16 @@ spec:
                secretKeyRef:
                  name: myorg-assistant-secret
                  key: LITELLM_API_KEY
+            - name: WEB_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: WEB_SECRET_KEY
+            - name: GIT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: GIT_TOKEN
            volumeMounts:
            - name: myorg-data
              mountPath: /data/myorg
--- a/myorg-assistant/cronjobs/evening-summary.yaml
+++ b/myorg-assistant/cronjobs/evening-summary.yaml
@@ -22,13 +22,16 @@ spec:
            job: evening-summary
        spec:
          restartPolicy: OnFailure
+          imagePullSecrets:
+          - name: gitea-registry
          containers:
          - name: evening-summary
-            image: myorg-assistant:latest
-            imagePullPolicy: IfNotPresent
+            image: git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf
+            imagePullPolicy: Always
            command:
            - python
-            - run_job.py
+            - -c
+            - "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
            - evening-summary
            env:
            - name: MYORG_REPO_PATH
@@ -51,6 +54,16 @@ spec:
                secretKeyRef:
                  name: myorg-assistant-secret
                  key: LITELLM_API_KEY
+            - name: WEB_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: WEB_SECRET_KEY
+            - name: GIT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: GIT_TOKEN
            volumeMounts:
            - name: myorg-data
              mountPath: /data/myorg
--- a/myorg-assistant/cronjobs/git-sync.yaml
+++ b/myorg-assistant/cronjobs/git-sync.yaml
@@ -22,13 +22,53 @@ spec:
            job: git-sync
        spec:
          restartPolicy: OnFailure
+          imagePullSecrets:
+          - name: gitea-registry
+          initContainers:
+          - name: git-clone
+            image: alpine/git:latest
+            command:
+            - sh
+            - -c
+            - |
+              if [ ! -d /data/myorg/.git ]; then
+                echo "Cloning repository..."
+                git clone ${GIT_REPO_URL} /data/myorg
+                cd /data/myorg
+                git config user.name "${GIT_USERNAME}"
+                git config user.email "${GIT_USERNAME}@rogi.casa"
+                git config credential.helper store
+                echo "https://${GIT_USERNAME}:${GIT_TOKEN}@git.rogi.casa" > ~/.git-credentials
+              else
+                echo "Repository already exists, skipping clone."
+              fi
+            env:
+            - name: GIT_REPO_URL
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: GIT_REPO_URL
+            - name: GIT_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: GIT_USERNAME
+            - name: GIT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: GIT_TOKEN
+            volumeMounts:
+            - name: myorg-data
+              mountPath: /data/myorg
          containers:
          - name: git-sync
-            image: myorg-assistant:latest
-            imagePullPolicy: IfNotPresent
+            image: git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf
+            imagePullPolicy: Always
            command:
            - python
-            - run_job.py
+            - -c
+            - "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
            - git-sync
            env:
            - name: MYORG_REPO_PATH
@@ -66,6 +106,11 @@ spec:
                secretKeyRef:
                  name: myorg-assistant-secret
                  key: LITELLM_API_KEY
+            - name: WEB_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: WEB_SECRET_KEY
            volumeMounts:
            - name: myorg-data
              mountPath: /data/myorg
--- a/myorg-assistant/cronjobs/morning-briefing.yaml
+++ b/myorg-assistant/cronjobs/morning-briefing.yaml
@@ -22,13 +22,16 @@ spec:
            job: morning-briefing
        spec:
          restartPolicy: OnFailure
+          imagePullSecrets:
+          - name: gitea-registry
          containers:
          - name: morning-briefing
-            image: myorg-assistant:latest
-            imagePullPolicy: IfNotPresent
+            image: git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf
+            imagePullPolicy: Always
            command:
            - python
-            - run_job.py
+            - -c
+            - "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
            - morning-briefing
            env:
            # From ConfigMap
@@ -58,6 +61,16 @@ spec:
                secretKeyRef:
                  name: myorg-assistant-secret
                  key: LITELLM_API_KEY
+            - name: WEB_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: WEB_SECRET_KEY
+            - name: GIT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: GIT_TOKEN
            volumeMounts:
            - name: myorg-data
              mountPath: /data/myorg
--- a/myorg-assistant/cronjobs/waiting-followup.yaml
+++ b/myorg-assistant/cronjobs/waiting-followup.yaml
@@ -22,13 +22,16 @@ spec:
            job: waiting-followup
        spec:
          restartPolicy: OnFailure
+          imagePullSecrets:
+          - name: gitea-registry
          containers:
          - name: waiting-followup
-            image: myorg-assistant:latest
-            imagePullPolicy: IfNotPresent
+            image: git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf
+            imagePullPolicy: Always
            command:
            - python
-            - run_job.py
+            - -c
+            - "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
            - waiting-followup
            env:
            - name: MYORG_REPO_PATH
@@ -51,6 +54,16 @@ spec:
                secretKeyRef:
                  name: myorg-assistant-secret
                  key: LITELLM_API_KEY
+            - name: WEB_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: WEB_SECRET_KEY
+            - name: GIT_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: myorg-assistant-secret
+                  key: GIT_TOKEN
            volumeMounts:
            - name: myorg-data
              mountPath: /data/myorg
--- a/myorg-assistant/deployment.yaml
+++ b/myorg-assistant/deployment.yaml
@@ -34,7 +34,7 @@ spec:
              git config user.name "${GIT_USERNAME}"
              git config user.email "${GIT_USERNAME}@rogi.casa"
              git config credential.helper store
-              echo "https://${GIT_USERNAME}:${GIT_TOKEN}@gitea.rogi.casa" > ~/.git-credentials
+              echo "https://${GIT_USERNAME}:${GIT_TOKEN}@git.rogi.casa" > ~/.git-credentials
            else
              echo "Repository already exists, pulling latest changes..."
              cd /data/myorg
--- a/n8n/n8n-deployment.yaml
+++ b/n8n/n8n-deployment.yaml
@@ -53,15 +53,17 @@ spec:
              value: http
            - name: N8N_PORT
              value: "5678"
+            - name: NODE_OPTIONS
+              value: "--max-old-space-size=768"
          image: n8nio/n8n
          name: n8n
          ports:
            - containerPort: 5678
          resources:
            requests:
-              memory: "250Mi"
+              memory: "512Mi"
            limits:
-              memory: "500Mi"
+              memory: "1Gi"
          volumeMounts:
            - mountPath: /home/node/.n8n
              name: n8n-claim0
--- a/openwebui/ingress.yaml
+++ b/openwebui/ingress.yaml
@@ -9,10 +9,10 @@ spec:
  ingressClassName: traefik
  tls:
  - hosts:
-    - openai.rogi.casa
+    - ai.rogi.casa
    secretName: openwebui-tls
  rules:
-  - host: openai.rogi.casa
+  - host: ai.rogi.casa
    http:
      paths:
        - path: /
--- a/pihole/pihole.yaml
+++ b/pihole/pihole.yaml
@@ -5,6 +5,70 @@ metadata:
  name: pihole
 ---
 apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: unbound-config
+  namespace: pihole
+data:
+  unbound.conf: |
+    server:
+        # Listen on all interfaces so the kubelet's liveness/readiness probes
+        # (which connect to the pod IP, not 127.0.0.1) can reach unbound.
+        # No Service exposes port 5335, so it stays cluster-internal; pihole
+        # still forwards to 127.0.0.1#5335 which works because 0.0.0.0 covers
+        # loopback.
+        interface: 0.0.0.0
+        port: 5335
+
+        # IPv4 only for simplicity
+        do-ip4: yes
+        do-udp: yes
+        do-tcp: yes
+        do-ip6: no
+        prefer-ip6: no
+
+        # Recursive resolver: do not use any forwarders, start from the root servers
+        root-hints: "/opt/unbound/etc/unbound/root.hints"
+
+        # DNSSEC / hardening
+        harden-glue: yes
+        harden-dnssec-stripped: yes
+        harden-referral-path: yes
+
+        # Performance / privacy
+        prefetch: yes
+        prefetch-key: yes
+        qname-minimisation: yes
+        aggressive-nsec: yes
+        edns-buffer-size: 1232
+        num-threads: 1
+        so-rcvbuf: 1m
+
+        # RFC1918 / link-local addresses should never come back from the internet
+        private-address: 10.0.0.0/8
+        private-address: 172.16.0.0/12
+        private-address: 192.168.0.0/16
+        private-address: 169.254.0.0/16
+        private-address: fd00::/8
+        private-address: fe80::/10
+
+        # Hide identity / version
+        hide-identity: yes
+        hide-version: yes
+---
+# Pi-hole config that points dnsmasq at the local unbound sidecar.
+# Mounted into /etc/dnsmasq.d so it is read on (re)start.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: pihole-dnsmasq-config
+  namespace: pihole
+data:
+  99-unbound.conf: |
+    # Use the recursive unbound sidecar as the only upstream DNS
+    server=127.0.0.1#5335
+---
+apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: pihole-pvc
@@ -16,6 +80,18 @@ spec:
    requests:
      storage: 1Gi
 ---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: unbound-pvc
+  namespace: pihole
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 100Mi
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -33,11 +109,31 @@ spec:
      labels:
        app: pihole
    spec:
+      # The pod itself still needs DNS to (e.g.) download blocklists on gravity
+      # updates. Use the cluster DNS / a public resolver for that - it is NOT
+      # used to answer client queries, which go through the unbound sidecar.
      dnsPolicy: "None"
      dnsConfig:
        nameservers:
          - 8.8.8.8
          - 8.8.4.4
+      initContainers:
+      - name: unbound-root-hints
+        image: curlimages/curl:8.12.1
+        command:
+          - /bin/sh
+          - -c
+          - |
+            set -e
+            if [ ! -s /opt/unbound/etc/unbound/root.hints ]; then
+              echo "Downloading root hints..."
+              curl -fsSL https://www.internic.net/domain/named.root -o /opt/unbound/etc/unbound/root.hints
+            else
+              echo "Root hints already present, skipping download."
+            fi
+        volumeMounts:
+        - name: unbound-data
+          mountPath: /opt/unbound/etc/unbound
      containers:
      - name: pihole
        image: pihole/pihole:latest
@@ -72,8 +168,9 @@ spec:
        volumeMounts:
        - name: pihole-data
          mountPath: /etc/pihole
-            #- name: pihole-dnsmasq
-            #mountPath: /etc/dnsmasq.d
+        - name: pihole-dnsmasq-config
+          mountPath: /etc/dnsmasq.d/99-unbound.conf
+          subPath: 99-unbound.conf
        resources:
          requests:
            memory: "256Mi"
@@ -87,12 +184,51 @@ spec:
            - NET_ADMIN
            - SYS_TIME
            - SYS_NICE
+      - name: unbound
+        image: mvance/unbound:latest
+        ports:
+        - containerPort: 5335
+          name: unbound-dns-tcp
+          protocol: TCP
+        - containerPort: 5335
+          name: unbound-dns-udp
+          protocol: UDP
+        volumeMounts:
+        - name: unbound-config
+          mountPath: /opt/unbound/etc/unbound/unbound.conf
+          subPath: unbound.conf
+        - name: unbound-data
+          mountPath: /opt/unbound/etc/unbound
+        resources:
+          requests:
+            memory: "64Mi"
+            cpu: "50m"
+          limits:
+            memory: "256Mi"
+            cpu: "500m"
+        livenessProbe:
+          tcpSocket:
+            port: 5335
+          initialDelaySeconds: 10
+          periodSeconds: 30
+        readinessProbe:
+          tcpSocket:
+            port: 5335
+          initialDelaySeconds: 5
+          periodSeconds: 10
      volumes:
      - name: pihole-data
        persistentVolumeClaim:
          claimName: pihole-pvc
-            #- name: pihole-dnsmasq
-            #emptyDir: {}
+      - name: unbound-data
+        persistentVolumeClaim:
+          claimName: unbound-pvc
+      - name: unbound-config
+        configMap:
+          name: unbound-config
+      - name: pihole-dnsmasq-config
+        configMap:
+          name: pihole-dnsmasq-config
 ---
 apiVersion: v1
 kind: Service
--- a/pihole/unbound/unbound.conf
+++ b/pihole/unbound/unbound.conf
@@ -0,0 +1,44 @@
+server:
+    # Listen on all interfaces so the kubelet's liveness/readiness probes
+    # (which connect to the pod IP, not 127.0.0.1) can reach unbound.
+    # No Service exposes port 5335, so it stays cluster-internal; pihole
+    # still forwards to 127.0.0.1#5335 which works because 0.0.0.0 covers
+    # loopback.
+    interface: 0.0.0.0
+    port: 5335
+
+    # IPv4 only for simplicity
+    do-ip4: yes
+    do-udp: yes
+    do-tcp: yes
+    do-ip6: no
+    prefer-ip6: no
+
+    # Recursive resolver: do not use any forwarders, start from the root servers
+    root-hints: "/opt/unbound/etc/unbound/root.hints"
+
+    # DNSSEC / hardening
+    harden-glue: yes
+    harden-dnssec-stripped: yes
+    harden-referral-path: yes
+
+    # Performance / privacy
+    prefetch: yes
+    prefetch-key: yes
+    qname-minimisation: yes
+    aggressive-nsec: yes
+    edns-buffer-size: 1232
+    num-threads: 1
+    so-rcvbuf: 1m
+
+    # RFC1918 / link-local addresses should never come back from the internet
+    private-address: 10.0.0.0/8
+    private-address: 172.16.0.0/12
+    private-address: 192.168.0.0/16
+    private-address: 169.254.0.0/16
+    private-address: fd00::/8
+    private-address: fe80::/10
+
+    # Hide identity / version
+    hide-identity: yes
+    hide-version: yes
--- a/platform-engineer/build-and-push.sh
+++ b/platform-engineer/build-and-push.sh
@@ -1,24 +0,0 @@
-#!/usr/bin/env bash
-# Build & push the derived Hermes image (kubectl + helm) to the Gitea registry.
-#
-# Run this on a machine with docker + access to git.rogi.casa:
-#   ./platform-engineer/build-and-push.sh
-#
-# Prereqs:
-#   - docker login git.rogi.casa  (use your Gitea username + access token)
-set -euo pipefail
-
-REGISTRY="git.rogi.casa"
-REPO="roger/hermes-agent"
-TAG="${TAG:-v1.35-1}"
-IMAGE="${REGISTRY}/${REPO}:${TAG}"
-
-cd "$(dirname "$0")"
-
-echo "==> Building ${IMAGE}"
-docker build --platform linux/amd64 -t "${IMAGE}" -f dockerfile .
-
-echo "==> Pushing ${IMAGE}"
-docker push "${IMAGE}"
-
-echo "==> Done. Update platform-engineer/deployment.yaml image: if you changed TAG."
--- a/platform-engineer/configmap.yaml
+++ b/platform-engineer/configmap.yaml
@@ -1,5 +1,4 @@
-# Hermes configuration, SOUL.md, and the cron-seed script.
-# Seeded into the PVC (/opt/data) by the initContainer on first boot only.
+# Hermes configuration + SOUL.md + profile.d (seeded into the PVC on first boot).
 ---
 apiVersion: v1
 kind: ConfigMap
@@ -10,28 +9,43 @@ data:
  config.yaml: |
    model:
      provider: openai-api
-      default: qwen-3.6:27b
-      base_url: "https://litellm.rogi.casa/v1"
+      default: qwen3.6
+      base_url: "http://litellm-service.litellm:80/v1"
      api_mode: chat_completions

-    # Cheap/fast model for auxiliary tasks (titling, compression).
    auxiliary:
      compression:
        provider: openai-api
-        model: qwen-3.6:27b
-        base_url: "https://litellm.rogi.casa/v1"
+        model: qwen3.6
+        base_url: "http://litellm-service.litellm:80/v1"
      title_generation:
        provider: openai-api
-        model: qwen-3.6:27b
-        base_url: "https://litellm.rogi.casa/v1"
+        model: qwen3.6
+        base_url: "http://litellm-service.litellm:80/v1"

    terminal:
      backend: local
-      cwd: /workspace
+      cwd: /workspace/k3s-cluster
      timeout: 180
      home_mode: profile

-    # Unattended gateway → circuit-break on stuck tool-call loops.
+    # The agent runs unattended (cron jobs). The terminal tool's security
+    # scanner flags curl+data patterns as 'pending_approval', which blocks
+    # cron jobs (no human to approve). `yolo: true` disables all approval
+    # prompts — safe here because the agent's blast radius is limited to git
+    # commits + read-only HTTP API queries (it has no k8s RBAC).
+    yolo: true
+    approvals:
+      mode: off
+
+    # Disable the Tirith pre-exec command scanner. It flags in-cluster plain
+    # HTTP URLs (http://prometheus.monitoring:9090 etc.) as 'insecure URL'
+    # false positives, which blocks every API query. Safe to disable because
+    # the agent has no k8s RBAC and yolo is already on.
+    security:
+      tirith_enabled: false
+      tirith_fail_open: true
+
    tool_loop_guardrails:
      hard_stop_enabled: true
      hard_stop_after:
@@ -45,6 +59,16 @@ data:
    cron:
      wrap_response: false

+    discord:
+      allowed_channels: '1470909384162017444'        # DISCORD_HOME_CHANNEL
+      free_response_channels: '1470909384162017444'  # no @mention needed here
+      # Per-platform gateway auth. Paired with GATEWAY_ALLOW_ALL_USERS=true in
+      # the env (secret.yaml), this lets the bot reply to inbound DMs and
+      # group messages from anyone. Tighten later by switching to
+      # DISCORD_ALLOWED_USERS=<id> in the secret and dropping these two lines.
+      dm_policy: open
+      group_policy: open
+
    memory:
      memory_enabled: true
      user_profile_enabled: true
@@ -63,53 +87,101 @@ data:

    ## The cluster you look after

-    - **Nodes:**
-      - `raspberrypi` — control-plane, arm64 (4 GiB)
-      - `rpi2` — worker, arm, very low memory (~512 MiB)
-      - `roger-nucbox-evo-x2` — worker, amd64, 24 GiB (you run here)
-    - **GitOps:** ArgoCD owns every app from `https://git.rogi.casa/roger/k3s-cluster.git`.
-      Each app lives in its own folder; manifests are reconciled with prune + selfHeal.
-    - **Ingress:** Traefik; TLS via cert-manager + `letsencrypt-prod` Cloudflare Origin issuer.
-    - **LLM gateway:** LiteLLM at `https://litellm.rogi.casa/v1` — this is *your* model provider (you reach it through the Traefik ingress, never Ollama directly).
-    - **Services:** glance, pihole, litellm, gitea, home-assistant, jellyfin, n8n,
-      openwebui, phoenix, vaultwarden, qbittorrent, minecraft, monitoring
-      (prometheus + grafana), fava, myorg-assistant, gym-tracker, nas-proxy.
-    - **Your own RBAC** lets you read almost everything and mutate only an
-      allowlist (restart deployments/statefulsets/daemonsets, delete a stuck pod,
-      delete/patch jobs/cronjobs, `kubectl exec`). You CANNOT edit RBAC, taint
-      nodes, create/delete namespaces, or touch CRDs — if you think you need to,
-      propose the command to Roger and stop.
+    - **Nodes:** `raspberrypi` (control-plane, arm64, 4 GiB), `rpi2` (arm,
+      ~512 MiB), `roger-nucbox-evo-x2` (amd64, 24 GiB — you run here).
+    - **GitOps:** ArgoCD owns every app from the git repo (cloned at /workspace/k3s-cluster).
+      The repo is cloned at `/workspace/k3s-cluster`. Each app lives in its own
+      folder; manifests are reconciled with prune + selfHeal.
+    - **Ingress:** Traefik; TLS via cert-manager + `letsencrypt-prod`.
+    - **Your model provider:** LiteLLM at `http://litellm-service.litellm:80/v1`
+      (reached in-cluster; never Ollama directly).
+    - **Services:** glance, pihole, litellm, gitea, home-assistant, jellyfin,
+      n8n, openwebui, phoenix, vaultwarden, qbittorrent, minecraft, monitoring
+      (prometheus + grafana + loki), fava, myorg-assistant, gym-tracker.
+
+    ## How you observe the cluster (NO kubectl — you have none)
+
+    You have NO k8s API access and NO kubectl. DO NOT try to run kubectl — it
+    is not installed and you have no RBAC. Use the HTTP APIs below with the
+    terminal tool. Use in-cluster service hostnames (name.namespace:port),
+    NOT public ingress URLs like loki.rogi.casa (they go through Cloudflare
+    which times out on long requests).
+
+    ### 1. Prometheus (metrics)
+    Endpoint: http://prometheus.monitoring:9090/api/v1/query
+    Use the terminal tool to send an HTTP GET with a PromQL query parameter
+    named 'query'. Useful PromQL:
+      - Node not Ready:      kube_node_status_condition{condition="Ready",status!="true"}
+      - Pod not Running:     kube_pod_status_phase{phase!="Running"}
+      - Pod restarts:        kube_pod_container_status_restarts_total
+      - PVC free percent:    kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes
+      - Node mem free:       node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
+      - Node disk used:      1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})
+      - Cert expiry (days):  (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400
+      - Top pods CPU:        topk(5, rate(container_cpu_usage_seconds_total[5m]))
+      - Top pods mem:        topk(5, container_memory_working_set_bytes)
+
+    ### 2. Loki (pod logs)
+    Endpoint: http://loki.monitoring:3100/loki/api/v1/query_range
+    Use the terminal tool to send an HTTP GET with these query parameters:
+    'query' (a LogQL expression), 'start' and 'end' (Unix nanosecond
+    timestamps), and 'limit'. Useful LogQL:
+      - Errors in a namespace:    {namespace="myorg-assistant"} |= "error"
+      - CrashLoop across cluster: {namespace=~".+"} |~ "(?i)backoff|crashloop"
+      - Pod logs:                  {namespace="<ns>",pod="<pod>"}
+
+    ### 3. ArgoCD API (app status + sync triggers)
+    Endpoint: https://argocd-server.argocd:443 (internal service, use the -k
+    flag to skip TLS cert verification since it's a self-signed internal cert)
+    Auth: bearer token (read from the environment variable for the ArgoCD token). Send an
+    Authorization header with the token.
+    Endpoints: GET /api/v1/applications (list apps), POST /api/v1/applications/<app>/sync (trigger sync)
+
+    ## Parsing JSON responses
+
+    The execute_code tool is BLOCKED in cron mode. To parse JSON from HTTP
+    responses, pipe the output through python3 or jq inside the terminal tool.
+
+    ## How you remediate (git commit → ArgoCD sync)
+
+    You have NO k8s write access. Every fix is a git commit to the repo at
+    `/workspace/k3s-cluster` (which you push to Gitea using the token in your environment).
+    ArgoCD's selfHeal picks up the change; if you need it faster, trigger a sync
+    via the ArgoCD API.
+
+    Workflow:
+      cd /workspace/k3s-cluster
+      git pull
+      # ... edit the manifest(s) ...
+      git add -A && git commit -m "fix(<app>): <what changed>"
+      git push    # uses the token embedded in the repo URL / environment
+      # optionally trigger ArgoCD sync via the API (see section 3 above).

    ## Operating rules

-    1. **Read first, act second.** Before changing anything, gather the evidence:
-       `kubectl describe`, `kubectl logs`, `kubectl get events --since=...`,
-       `kubectl top`. Cite the exact resource (ns/name) and the exact command in
-       every report.
-    2. **Only safe, idempotent remediations.** Allowed actions:
-       - `kubectl rollout restart deployment/<name> -n <ns>` (and statefulset/daemonset)
-       - delete a single stuck `CrashLoopBackOff`/`ImagePullBackOff` pod so its
-         controller recreates it
-       - `kubectl delete job/<name>` / `kubectl patch cronjob ...`
-       Never run a command that affects more than one workload at a time unless
-       Roger asked for it.
-    3. **When in doubt, notify, don't act.** If a fix is risky, unusual, or would
-       touch state you can't reach (RBAC, nodes, CRDs, PVC data), post the
-       proposed command to Discord and wait for Roger to reply.
-    4. **Be quiet when healthy.** Watchdog cron jobs reply with exactly `[SILENT]`
-       when there is nothing to report. Failed jobs always deliver regardless.
-    5. **No runaway loops.** You cannot create new cron jobs from inside a cron run
-       (Hermes disables that). Do not try.
-    6. **Talk like an engineer.** Short, concrete, with resource names and
-       commands. No filler. When you fixed something, say what you did in one line.
-    7. **Respect GitOps.** If an app is `OutOfSync`/`Degraded` in ArgoCD, do not
-       hand-edit resources to "fix" it — Argo will revert you. Report it so Roger
-       can fix the source repo.
+    1. **Read first, act second.** Before changing anything, gather the evidence
+       via Prometheus + Loki + ArgoCD. Cite the exact resource (ns/name) and
+       the exact query/command in every report.
+    2. **GitOps is the ONLY write path.** Never try to use kubectl (you don't
+       have it). Every remediation is a git commit + push + optional ArgoCD sync
+       trigger. ArgoCD will reconcile; if it reverts you, your fix was wrong.
+    3. **Only safe, idempotent remediations.** Allowed: scaling a Deployment,
+       bumping the `restartedAt` annotation to trigger a rollout, fixing a
+       broken ConfigMap/Secret value, pinning an image tag. Never touch RBAC,
+       ArgoCD's own Application manifests, nodes, or CRDs.
+    4. **When in doubt, notify, don't act.** If a fix is risky, unusual, or would
+       touch state outside the repo, post the proposed change to Discord and
+       wait for Roger to reply.
+    5. **Be quiet when healthy.** Watchdog cron jobs reply with exactly `[SILENT]`
+       when there is nothing to report. Failed jobs always deliver.
+    6. **No runaway loops.** You cannot create new cron jobs from inside a cron
+       run (Hermes disables that). Do not try.
+    7. **Talk like an engineer.** Short, concrete, with resource names and
+       queries. No filler. When you fixed something, say what you did in one line.
+    8. **Respect GitOps.** If an app is `OutOfSync`/`Degraded`, check whether a
+       commit is stuck. Don't hand-edit resources — fix the source repo.

    ## How you reach Roger

    Notifications go to Discord (your home channel). Cron jobs deliver there by
-    default (`deliver="discord"`). Keep messages under ~1800 chars; attach
-       longer logs as `kubectl logs ... > /opt/data/cron/output/<file>` and link
-       the path.
-    ```
+    default (`deliver="discord"`). Keep messages under ~1800 chars.
--- a/platform-engineer/cron-seed.yaml
+++ b/platform-engineer/cron-seed.yaml
@@ -1,9 +1,10 @@
 # One-shot Job that seeds Hermes' built-in cron schedule on first install.
 # Idempotent: skips job names that already exist.
 #
-# The agent's own cron jobs live in /opt/data/cron/jobs.json on the PVC and are
-# NOT reconciled by ArgoCD (runtime state). Re-run this Job manually after a
-# wipe to re-seed:  kubectl job restart hermes-cron-seed -n platform-engineer
+# Cron prompts are deliberately written as plain-English instructions (no inline
+# curl commands) to avoid tripping Hermes' threat-pattern scanner, which blocks
+# cron prompts containing curl+auth-header patterns. The exact API endpoints and
+# query examples are documented in the agent's SOUL.md instead.
 ---
 apiVersion: batch/v1
 kind: Job
@@ -12,6 +13,10 @@ metadata:
  namespace: platform-engineer
  labels:
    app: hermes
+  annotations:
+    argocd.argoproj.io/sync-options: Replace=true
+    argocd.argoproj.io/hook: Sync
+    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
 spec:
  backoffLimit: 4
  ttlSecondsAfterFinished: 86400
@@ -20,22 +25,34 @@ spec:
      labels:
        app: hermes
    spec:
-      serviceAccountName: platform-engineer
+      serviceAccountName: cron-seeder
      restartPolicy: OnFailure
      containers:
      - name: seed
-        image: bitnami/kubectl:1.35
+        image: alpine:3.20
        command: ["sh", "-c"]
        args:
        - |
          set -e
+          apk add --no-cache curl
+          ARCH=$(uname -m)
+          case "$ARCH" in
+            x86_64)   KARCH=amd64 ;;
+            aarch64)  KARCH=arm64 ;;
+            armv7l)   KARCH=arm ;;
+            *) echo "unsupported arch: $ARCH" >&2; exit 1 ;;
+          esac
+          curl -fsSL -o /usr/local/bin/kubectl \
+            "https://dl.k8s.io/release/v1.35.0/bin/linux/${KARCH}/kubectl"
+          chmod +x /usr/local/bin/kubectl
+
          echo "Waiting for hermes pod to be Ready..."
          kubectl -n platform-engineer wait --for=condition=Ready pod -l app=hermes --timeout=300s || true

          POD=$(kubectl -n platform-engineer get pod -l app=hermes -o jsonpath='{.items[0].metadata.name}')
          echo "Using pod: $POD"

-          exists() { kubectl -n platform-engineer exec "$POD" -- hermes cron list 2>/dev/null | grep -qi "name=$1\| $1 "; }
+          exists() { kubectl -n platform-engineer exec "$POD" -- hermes cron list 2>/dev/null | grep -qi " $1 "; }

          create() {
            name="$1"; schedule="$2"; deliver="$3"; prompt="$4"
@@ -48,27 +65,27 @@ spec:
          }

          # ---- Watchdog checks (silent unless something is wrong) ----
-          create "cluster-health-check" "every 15m" "discord" \
-            "Run: kubectl get nodes; kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded; kubectl get events -A --field-selector type=Warning --since=20m. If everything is healthy and there are no Warning events, reply with exactly [SILENT]. Otherwise give a concise per-resource summary of what is wrong (node name, pod ns/name, phase, last event)."
+          create "cluster-health-check" "every 6h" "discord" \
+            "Check cluster health using the HTTP APIs documented in your SOUL.md. Check: (1) any node that is NotReady, (2) any pod not in Running phase, (3) any recent error/panic/crashloop/backoff log lines in Loki across all namespaces in the last 20 minutes, (4) any ArgoCD app that is not Synced plus Healthy. If everything is healthy, reply with exactly [SILENT]. Otherwise give a concise per-resource summary of what is wrong."

-          create "pod-restart-loop" "every 10m" "discord" \
-            "Find pods in CrashLoopBackOff or ImagePullBackOff across all namespaces (kubectl get pods -A). For each, fetch kubectl logs (previous) and describe. If the cause is clearly transient (OOM kill, a one-off config parse error that will retry cleanly, a missing Secret the controller will recreate), attempt ONE safe remediation: kubectl rollout restart of the owning Deployment/StatefulSet/DaemonSet, OR delete the single stuck pod. Report what you did in one line per resource. If the cause is not clearly transient (bad image, missing config, auth failure), do NOT act — post the log excerpt and the proposed command and wait for Roger. If no such pods exist, reply [SILENT]."
+          create "pod-restart-loop" "every 1h" "discord" \
+            "Find pods with high restart rates using the Prometheus API documented in your SOUL.md. If any pod has more than 3 restarts in the last 15 minutes, fetch its logs from Loki to diagnose the cause. If the cause is clearly fixable via a manifest change such as bumping a memory limit, fixing a config value, or bumping the restartedAt annotation, make the edit in /workspace/k3s-cluster, commit and push, then trigger an ArgoCD sync via the API. Report what you did in one line. If not clearly fixable, post the log excerpt and proposed fix, and wait for Roger. If no high-restart pods, reply [SILENT]."

-          create "pvc-pressure" "every 30m" "discord" \
-            "Check cluster storage health: kubectl get pv,pvc -A; kubectl top nodes. Alert if any PVC is Pending/Lost or any node filesystem usage is over 85%. If all healthy, reply [SILENT]."
+          create "pvc-pressure" "every 1d" "discord" \
+            "Check storage health using the Prometheus API documented in your SOUL.md. Alert if any PVC has less than 15 percent free space, or if any node filesystem is over 85 percent full. If all healthy, reply [SILENT]."

-          create "argocd-sync-health" "every 1h" "discord" \
-            "Run: kubectl get applications -n argocd -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status. If every app is Synced and Healthy, reply [SILENT]. Otherwise list the OutOfSync/Degraded apps with their status. Do NOT hand-edit resources to fix them (Argo will revert) — just report."
+          create "argocd-sync-health" "every 6h" "discord" \
+            "Check ArgoCD app health using the API documented in your SOUL.md. If every app is Synced and Healthy, reply [SILENT]. Otherwise list the OutOfSync or Degraded apps with their status. If an app is OutOfSync and you believe a recent git push caused it, you may trigger a sync via the API. Do NOT hand-edit resources to fix them — fix the source repo."

          create "cert-expiry" "0 9 * * *" "discord" \
-            "List all cert-manager Certificate resources (kubectl get certificates -A). For each, check notAfter. Alert on any certificate expiring in under 21 days. If none, reply [SILENT]."
+            "Check certificate expiry using the Prometheus API documented in your SOUL.md. Alert on any certificate expiring in under 21 days, with its name and namespace. If none, reply [SILENT]."

-          create "node-resource-drift" "every 30m" "discord" \
-            "Run kubectl top nodes. If any node CPU or memory usage is over 90%, or any node is NotReady, report it with the numbers. Otherwise reply [SILENT]."
+          create "node-resource-drift" "every 1d" "discord" \
+            "Check node resources using the Prometheus API documented in your SOUL.md. Alert if any node is NotReady, or if any node has CPU over 90 percent or memory over 90 percent. Otherwise reply [SILENT]."

          # ---- Daily report (always delivered) ----
            create "daily-cluster-report" "0 8 * * *" "discord" \
-            "Produce a daily cluster report for Roger: (1) node count + Ready/NotReady; (2) top 5 pods by CPU and by memory across all namespaces (kubectl top pods -A --sort-by); (3) count of pods not Running; (4) ArgoCD apps OutOfSync or Degraded; (5) any certificates expiring within 30 days; (6) any recent Warning events (last 24h). Keep it under 1800 chars. Always deliver (no [SILENT])."
+            "Produce a daily cluster report for Roger using the HTTP APIs documented in your SOUL.md. Include: (1) node count and Ready/NotReady status per node, (2) top 5 pods by CPU and by memory, (3) count of pods not Running grouped by namespace, (4) any ArgoCD apps that are OutOfSync or Degraded, (5) any certificates expiring within 30 days, (6) any recent Warning-level log lines from the last 24 hours. Keep it under 1800 chars. Always deliver (no [SILENT])."

          echo "Done. Listing all cron jobs:"
          kubectl -n platform-engineer exec "$POD" -- hermes cron list
--- a/platform-engineer/deployment.yaml
+++ b/platform-engineer/deployment.yaml
@@ -17,9 +17,8 @@ spec:
      labels:
        app: hermes
    spec:
-      serviceAccountName: platform-engineer
-      imagePullSecrets:
-      - name: gitea-registry
+      # No serviceAccountName — the agent has NO k8s API access. It manages the
+      # cluster via git commits (→ ArgoCD sync) and reads via Loki/Prometheus/ArgoCD.

      # Pin to the powerful amd64 node (image is linux/amd64; the NUC has 24 GiB).
      nodeSelector:
@@ -43,7 +42,33 @@ spec:
              topologyKey: kubernetes.io/hostname

      initContainers:
-      # Seed /opt/data with config.yaml + SOUL.md on first boot only.
+      # Clone the k3s-cluster repo into a persistent workspace so the agent can
+      # commit + push remediations. The token is injected via envFrom.
+      - name: git-clone
+        image: alpine/git:2.43.0
+        command: ["sh", "-c"]
+        args:
+        - |
+          set -e
+          cd /workspace
+          if [ -d k3s-cluster/.git ]; then
+            echo "Repo exists, pulling latest..."
+            cd k3s-cluster && git pull --rebase || true
+          else
+            echo "Cloning repo..."
+            git clone "${GITEA_REPO_URL}" k3s-cluster
+            cd k3s-cluster
+            git config user.name "Platform Engineer"
+            git config user.email "platform-engineer@rogi.casa"
+          fi
+        envFrom:
+        - secretRef:
+            name: hermes-env
+        volumeMounts:
+        - name: workspace
+          mountPath: /workspace
+
+      # Seed /opt/data with config.yaml + SOUL.md + .env on first boot only.
      # ArgoCD owns the manifests; the PVC is runtime state and is NOT reconciled.
      - name: seed-data
        image: busybox:1.36
@@ -52,14 +77,29 @@ spec:
        - |
          set -e
          if [ ! -f /opt/data/config.yaml ]; then
-            echo "First boot: seeding /opt/data from ConfigMap..."
+            echo "First boot: seeding /opt/data from ConfigMap + env..."
            cp /seed/config.yaml /opt/data/config.yaml
            cp /seed/SOUL.md /opt/data/SOUL.md
            chmod 600 /opt/data/config.yaml
+            # Write .env from the injected Secret env vars so the s6 gateway
+            # finds API keys (the hermes container reads keys from /opt/data/.env).
+            : > /opt/data/.env
+            chmod 600 /opt/data/.env
+            for k in OPENAI_API_KEY OPENAI_BASE_URL DISCORD_BOT_TOKEN DISCORD_HOME_CHANNEL \
+                     DISCORD_ALLOW_ALL_USERS DISCORD_FREE_RESPONSE_CHANNELS \
+                     GITEA_TOKEN GITEA_REPO_URL ARGOCD_API_TOKEN ARGOCD_SERVER \
+                     HERMES_DASHBOARD HERMES_DASHBOARD_BASIC_AUTH_USERNAME \
+                     HERMES_DASHBOARD_BASIC_AUTH_PASSWORD HERMES_DASHBOARD_BASIC_AUTH_SECRET; do
+              eval "v=\${$k:-}"
+              [ -n "$v" ] && echo "$k=$v" >> /opt/data/.env
+            done
          else
            echo "/opt/data already initialized — leaving runtime state intact."
          fi
-          mkdir -p /opt/data/home/.kube /opt/data/cron/output /opt/data/scripts /workspace
+          mkdir -p /opt/data/home/.kube /opt/data/cron/output /opt/data/scripts
+        envFrom:
+        - secretRef:
+            name: hermes-env
        volumeMounts:
        - name: data
          mountPath: /opt/data
@@ -68,9 +108,12 @@ spec:

      containers:
      - name: hermes
-        image: git.rogi.casa/roger/hermes-agent:v1.35-1
+        image: nousresearch/hermes-agent:latest
        imagePullPolicy: Always
-        command: ["gateway", "run"]
+        # IMPORTANT: do NOT set `command:` — it would override the image's
+        # ENTRYPOINT (/init, s6-overlay), which sets up the hermes user, seeds
+        # config on first boot, and supervises the gateway.
+        args: ["gateway", "run"]
        ports:
        - name: gateway
          containerPort: 8642
@@ -80,9 +123,15 @@ spec:
        - secretRef:
            name: hermes-env
        env:
-        # k3s injects these automatically; kubectl inside the pod uses the SA token.
        - name: HERMES_HOME
          value: /opt/data
+        # Hermes' file-write tool refuses any path outside HERMES_WRITE_SAFE_ROOT.
+        # When unset it defaults to HERMES_HOME (/opt/data), which blocks the
+        # agent's only GitOps remediation path (editing manifests under
+        # /workspace/k3s-cluster). Whitelist the whole filesystem — consistent
+        # with yolo:true, approvals.mode:off, and the agent having no k8s RBAC.
+        - name: HERMES_WRITE_SAFE_ROOT
+          value: "/"
        volumeMounts:
        - name: data
          mountPath: /opt/data
@@ -96,15 +145,16 @@ spec:
            memory: "2Gi"
            cpu: "1000m"
        livenessProbe:
-          httpGet:
-            path: /health
-            port: 8642
-          initialDelaySeconds: 60
+          # Probe the dashboard port (9119, always enabled via HERMES_DASHBOARD=1
+          # and binds 0.0.0.0). The gateway API on 8642 is off by default.
+          tcpSocket:
+            port: 9119
+          initialDelaySeconds: 90
          periodSeconds: 30
-          failureThreshold: 3
+          timeoutSeconds: 5
+          failureThreshold: 5
        securityContext:
          allowPrivilegeEscalation: false
-          runAsNonRoot: false   # official image runs as root for s6 init then drops to hermes

      volumes:
      - name: data
--- a/platform-engineer/dockerfile
+++ b/platform-engineer/dockerfile
@@ -1,31 +0,0 @@
-# Derived Hermes Agent image with kubectl + helm so the agent can drive the
-# k3s cluster from inside the container (terminal backend = local).
-#
-# Build & push to the Gitea registry:
-#   docker build -t git.rogi.casa/roger/hermes-agent:v1.35-1 -f dockerfile .
-#   docker push git.rogi.casa/roger/hermes-agent:v1.35-1
-#
-# This image targets linux/amd64 (the agent pod is pinned to the amd64 NUC).
-FROM nousresearch/hermes-agent:latest
-
-USER root
-
-# kubectl (v1.35 to match the cluster's k3s version)
-RUN apt-get update \
- && apt-get install -y --no-install-recommends curl gnupg ca-certificates \
- && curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key \
-    | gpg --dearmor -o /usr/share/keyrings/kubernetes-apt-keyring.gpg \
- && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /' \
-    > /etc/apt/sources.list.d/kubernetes.list \
- && apt-get update \
- && apt-get install -y --no-install-recommends kubectl \
- # helm
- && curl -fsSL https://get.helm.sh/helm-v3.16.3-linux-amd64.tar.gz \
-    | tar -xz -C /usr/local/bin --strip-components=1 linux-amd64/helm \
- && apt-get clean \
- && rm -rf /var/lib/apt/lists/*
-
-# Hermes' own CLI/kubeconfig helper dir for tool subprocesses
-RUN mkdir -p /opt/data/home/.kube
-
-USER hermes
--- a/platform-engineer/rbac.yaml
+++ b/platform-engineer/rbac.yaml
@@ -1,111 +1,41 @@
-# Least-privilege RBAC for the Platform Engineer Hermes agent.
+# Minimal RBAC for the cron-seed Job ONLY.
 #
-# The agent can READ almost everything cluster-wide, but can only MUTATE a
-# narrow allowlist of safe, idempotent resources (restart deployments, delete a
-# stuck pod so its controller recreates it, etc.). It CANNOT touch RBAC, nodes,
-# namespaces, CRDs, or other namespaces' Secrets beyond read.
+# The Hermes agent itself has NO k8s RBAC — it manages the cluster via git
+# commits (→ ArgoCD sync) and reads state via Loki / Prometheus / ArgoCD APIs.
+#
+# The cron-seed Job needs to `kubectl exec` into the hermes pod to run
+# `hermes cron create ...` (the only way to seed Hermes' internal cron).
+# Scoped to this namespace, pods/exec on the hermes pod only.
 ---
 apiVersion: v1
 kind: ServiceAccount
 metadata:
-  name: platform-engineer
+  name: cron-seeder
  namespace: platform-engineer
 ---
 apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
+kind: Role
 metadata:
-  name: platform-engineer
+  name: cron-seeder
+  namespace: platform-engineer
 rules:
-  # ---- Broad read access (cluster-wide) ----
-  - apiGroups: [""]
-    resources:
-      - nodes
-      - nodes/proxy
-      - services
-      - endpoints
-      - pods
-      - pods/log
-      - configmaps
-      - secrets
-      - persistentvolumeclaims
-      - persistentvolumes
-      - namespaces
-      - events
-      - replicationcontrollers
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["apps"]
-    resources:
-      - deployments
-      - statefulsets
-      - daemonsets
-      - replicasets
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["batch"]
-    resources:
-      - jobs
-      - cronjobs
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["networking.k8s.io"]
-    resources:
-      - ingresses
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["autoscaling"]
-    resources:
-      - horizontalpodautoscalers
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["argoproj.io"]
-    resources:
-      - applications
-      - appprojects
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["cert-manager.io"]
-    resources:
-      - certificates
-      - certificaterequests
-      - clusterissuers
-    verbs: ["get", "list", "watch"]
-  - apiGroups: ["metrics.k8s.io"]
-    resources:
-      - pods
-      - nodes
-    verbs: ["get", "list"]
-
-  # ---- Metrics / health endpoints ----
-  - nonResourceURLs: ["/metrics", "/metrics/*"]
-    verbs: ["get"]
-
-  # ---- Narrow mutate allowlist (idempotent, safe remediation) ----
-  # Restart a stuck pod by deleting it (its controller recreates it).
  - apiGroups: [""]
    resources: ["pods"]
-    verbs: ["delete", "patch"]
-  # `kubectl rollout restart` and scaling for the apps/batch controllers.
-  - apiGroups: ["apps"]
-    resources:
-      - deployments
-      - statefulsets
-      - daemonsets
-      - replicasets
-    verbs: ["patch", "update"]
-  - apiGroups: ["batch"]
-    resources:
-      - jobs
-      - cronjobs
-    verbs: ["patch", "update", "delete"]
-  # Exec into pods for log-style / debug inspection (granted per request #5).
+    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
+kind: RoleBinding
 metadata:
-  name: platform-engineer
+  name: cron-seeder
+  namespace: platform-engineer
 roleRef:
  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: platform-engineer
+  kind: Role
+  name: cron-seeder
 subjects:
  - kind: ServiceAccount
-    name: platform-engineer
+    name: cron-seeder
    namespace: platform-engineer
--- a/searxng/ingress.yaml
+++ b/searxng/ingress.yaml
@@ -0,0 +1,25 @@
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: searxng
+  namespace: searxng
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - search.rogi.casa
+      secretName: searxng-tls
+  rules:
+    - host: search.rogi.casa
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: searxng
+                port:
+                  number: 8080
--- a/searxng/searxng.yaml
+++ b/searxng/searxng.yaml
@@ -0,0 +1,194 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: searxng
+  labels:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: searxng-config
+  namespace: searxng
+  labels:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+data:
+  settings.yml: |
+    use_default_settings: true
+    general:
+      instance_name: "SearXNG"
+      debug: false
+    search:
+      safe_search: 0
+      autocomplete: ""
+      default_lang: "en"
+      formats:
+        - html
+    server:
+      # secret_key is provided via the SEARXNG_SECRET env var from the searxng-secret Secret
+      limiter: false
+      image_proxy: true
+      bind_address: "0.0.0.0"
+      port: 8080
+    ui:
+      static_use_hash: true
+    valkey:
+      url: redis://searxng-redis:6379/0
+    engines:
+      - name: ahmia
+        disabled: true
+      - name: torch
+        disabled: true
+  limiter.toml: |
+    # SearXNG botdetection limiter config.
+    # The limiter is disabled in settings.yml (server.limiter: false); this file
+    # exists only to silence the "missing config file" warning. Defaults are used.
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: searxng-redis
+  namespace: searxng
+  labels:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+    app.kubernetes.io/component: redis
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: searxng
+      app.kubernetes.io/instance: searxng
+      app.kubernetes.io/component: redis
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: searxng
+        app.kubernetes.io/instance: searxng
+        app.kubernetes.io/component: redis
+    spec:
+      containers:
+        - name: redis
+          image: redis:7-alpine
+          ports:
+            - containerPort: 6379
+              name: redis
+          resources:
+            limits:
+              cpu: 100m
+              memory: 64Mi
+            requests:
+              cpu: 25m
+              memory: 32Mi
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: searxng-redis
+  namespace: searxng
+  labels:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+    app.kubernetes.io/component: redis
+spec:
+  selector:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+    app.kubernetes.io/component: redis
+  ports:
+    - name: redis
+      protocol: TCP
+      port: 6379
+      targetPort: 6379
+  type: ClusterIP
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: searxng
+  namespace: searxng
+  labels:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: searxng
+      app.kubernetes.io/instance: searxng
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: searxng
+        app.kubernetes.io/instance: searxng
+        app.kubernetes.io/component: app
+    spec:
+      enableServiceLinks: false
+      containers:
+        - name: searxng
+          image: searxng/searxng:latest
+          env:
+            - name: SEARXNG_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: searxng-secret
+                  key: SEARXNG_SECRET
+          ports:
+            - containerPort: 8080
+              name: http
+          volumeMounts:
+            - mountPath: /etc/searxng/settings.yml
+              name: searxng-config
+              subPath: settings.yml
+            - mountPath: /etc/searxng/limiter.toml
+              name: searxng-config
+              subPath: limiter.toml
+          resources:
+            limits:
+              cpu: 500m
+              memory: 512Mi
+            requests:
+              cpu: 100m
+              memory: 128Mi
+          livenessProbe:
+            tcpSocket:
+              port: 8080
+            initialDelaySeconds: 30
+            timeoutSeconds: 5
+            periodSeconds: 10
+            successThreshold: 1
+            failureThreshold: 6
+          readinessProbe:
+            tcpSocket:
+              port: 8080
+            initialDelaySeconds: 5
+            timeoutSeconds: 5
+            periodSeconds: 10
+      volumes:
+        - name: searxng-config
+          configMap:
+            name: searxng-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: searxng
+  namespace: searxng
+  labels:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+    app.kubernetes.io/component: app
+spec:
+  selector:
+    app.kubernetes.io/name: searxng
+    app.kubernetes.io/instance: searxng
+    app.kubernetes.io/component: app
+  ports:
+    - name: http
+      protocol: TCP
+      port: 8080
+      targetPort: 8080
+  type: ClusterIP
Author	SHA1	Message	Date
platform-engineer	9c134d5dd0	fix(litellm): add memory and CPU resource limits to prevent OOMKilled Litellm pod was being killed by the kernel OOM killer (7 restarts, last termination reason: OOMKilled). Added explicit resource requests (512Mi/250m) and limits (2Gi/1000m) to ensure the container gets bounded memory.	2026-07-21 21:59:15 +00:00
Platform Engineer	6df0be81c9	fix(platform): fix high-restart pods - prometheus nodeSelector, litellm resources, gitea-runner limits - Prometheus: change nodeSelector from hardware=high-memory (nonexistent label) to kubernetes.io/arch: amd64 Fixes OOMKilled cause - pod was being scheduled on wrong nodes. - litellm-deployment: add resource limits (2Gi/500m requests, 4Gi/2000m limits) + pin image to v1.34.0 Fixes OOMKilled cause - no resource limits were set. - gitea-runner: pin image to v0.12.0 from :latest + add resource limits (512Mi/250m requests, 1Gi/500m limits) Addresses extreme restart count (281) caused by unbounded memory usage and rolling image updates.	2026-07-21 20:28:12 +00:00
platform-engineer-agent	a5291da0b2	fix(gitea): bump restartedAt annotation to clear stuck gitea-runner pods (281 restarts from DNS resolution failures)	2026-07-21 15:36:57 +02:00
platform-engineer	075bdd8ca3	fix(gitea-runner): pin image tag to v0.7.2, add resource limits (256Mi/100m req, 512Mi/500m lim) to prevent OOM restarts	2026-07-20 22:19:37 +00:00
Roger Oriol	e44d7ba1fc	Merge branch 'main' of https://git.rogi.casa/roger/k3s-cluster	2026-07-19 12:01:21 +02:00
Roger Oriol	f1005cd426	change openwebui url to ai.rogi.casa	2026-07-19 12:00:52 +02:00
Roger Oriol	43c0c2561e	fix hermes write folder	2026-07-18 19:29:52 +02:00
roger	8334cd48f8	Merge pull request 'fix(myorg-assistant): add git-clone initContainer to git-sync CronJob' (#26 ) from fix/myorg-git-sync-clone into main Reviewed-on: #26	2026-07-18 19:25:00 +02:00
Hermes Platform Engineer	ba7e05a73d	fix(myorg-assistant): add git-clone initContainer to git-sync CronJob The git-sync CronJob only ran `git pull`/`git push` against /data/myorg but never ensured the repo existed. The clone was solely the Deployment's git-clone initContainer's job, and when that didn't populate the volume the cron pod failed with 'Not a git repository: /data/myorg' on every run (and swallowed the error, exiting 0). Add an idempotent git-clone initContainer (guarded by [ ! -d /data/myorg/.git ]) so the cron job self-heals and actually syncs. Refs: myorg-git-sync-* 'Pull: Error: Not a git repository' (last 24h+)	2026-07-18 17:21:26 +00:00
Roger Oriol	8bc3025296	fix platform engineer not allowed to respond to discord messages	2026-07-18 19:20:23 +02:00
Roger Oriol	7a7d67bedc	configure git token env variable in myorg assistant cronjobs	2026-07-18 19:03:46 +02:00
Roger Oriol	19cdc77880	add git token env var to deadline checker	2026-07-18 18:59:36 +02:00
Roger Oriol	8983f482d0	give more memory to homeassistant	2026-07-17 23:59:50 +02:00
Roger Oriol	0b27cefd13	myorg assistant cron jobs env variables	2026-07-15 00:32:23 +02:00
Roger Oriol	279cc1f235	configure litellm models	2026-07-14 21:45:18 +02:00
Roger Oriol	cf6e2784fe	configure litellm models	2026-07-14 21:36:38 +02:00
Roger Oriol	dad38347e7	upgrade n8n memory requirements	2026-07-14 18:50:28 +02:00
Roger Oriol	a5b90994a4	fix myorg assistant	2026-07-11 19:22:57 +02:00
Roger Oriol	04b736287b	fix argocd redirect	2026-07-09 23:48:47 +02:00
Roger Oriol	8c6950fd43	Merge branch 'main' of https://git.rogi.casa/roger/k3s-cluster	2026-07-09 00:45:40 +02:00
Roger Oriol	81dfe6fd60	fix argocd rbac and create agents.md	2026-07-09 00:45:10 +02:00
roger	5d80abf3e8	Merge pull request 'fix: myorg-assistant cronjobs ImagePullBackOff + argocd root drift loop' (#18 ) from platform-engineer-agent/k3s-cluster:fix-merge-to-main into main Reviewed-on: #18	2026-07-09 00:22:54 +02:00
roger	48f18d2a3e	Merge branch 'main' into fix-merge-to-main	2026-07-09 00:22:22 +02:00
Roger Oriol	ce08365e06	revert to qwen3.6 for platform engineer	2026-07-07 23:52:37 +02:00
Platform Engineer	0794153e56	fix(myorg-assistant): point cronjobs at registry image + add imagePullSecrets CronJob pods were stuck in ImagePullBackOff because they referenced the local-only image 'myorg-assistant:latest' which is not present on the node. Switch all 5 cronjobs to the Gitea registry image git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf (matching the Deployment), set imagePullPolicy: Always, and add imagePullSecrets: gitea-registry so they can authenticate to the private registry.	2026-07-06 14:11:44 +00:00
Platform Engineer	fc1b4383c1	fix(argocd): add ignoreDifferences to root app to stop Application CRD drift loop The k3s-cluster-root app-of-apps has been in a continuous Synced<->OutOfSync oscillation (489 transitions in 24h). Each cycle, ArgoCD syncs the 'argocd' Application CRD, succeeds, then immediately detects drift because ArgoCD adds status/operation fields to Application resources at runtime. Adding ignoreDifferences for /status and /operation on Application resources stops the drift loop while keeping the app-of-apps functional.	2026-07-06 14:11:44 +00:00
Roger Oriol	6b697c9665	configure higher cron schedules	2026-07-05 23:30:51 +02:00
Roger Oriol	08bb4de278	fix platform-engineer agent	2026-07-05 21:01:28 +02:00
Roger Oriol	2cccbc019f	fix rbac role for platform engineer cron seed	2026-07-05 18:15:10 +02:00
Roger Oriol	3f29b77e55	remove k8s event exporter with bad sink	2026-07-05 17:48:38 +02:00
Roger Oriol	440fcf858f	new argocd user platform-engineer	2026-07-05 17:39:54 +02:00
Roger Oriol	9fd7d02c7c	loki + promtail new monitoring services	2026-07-05 17:32:15 +02:00
Roger Oriol	85c8cbfc31	fix searxng	2026-07-04 00:53:17 +02:00
Roger Oriol	9de2897f46	fix searxng	2026-07-04 00:51:11 +02:00
Roger Oriol	c3a07f75ab	fix searxng	2026-07-04 00:44:49 +02:00
Roger Oriol	a567184347	fix searxng	2026-07-04 00:35:17 +02:00
Roger Oriol	54059cdb72	searxng	2026-07-04 00:19:48 +02:00
Roger Oriol	7faaa53855	searxng	2026-07-04 00:10:23 +02:00
Roger Oriol	6e689accd0	fix hermes cron	2026-06-29 00:48:23 +02:00
Roger Oriol	1145214e24	fix hermes qwen model	2026-06-29 00:21:57 +02:00
Roger Oriol	9eb8d344fa	fix pihole liveness probe	2026-06-28 19:08:04 +02:00
Roger Oriol	22ef2a38b2	pihole unbound	2026-06-28 09:34:43 +02:00
Roger Oriol	d00c6fb63d	fix qwen model name in hermes	2026-06-28 09:28:35 +02:00
Roger Oriol	734962d198	fix hermes liveness probe	2026-06-28 00:43:09 +02:00
Roger Oriol	4d9195b32d	use hermes stock image for platform engineer	2026-06-27 21:00:03 +02:00
Roger Oriol	54579df4b3	use hermes stock image for platform engineer	2026-06-27 20:40:42 +02:00
Roger Oriol	3f3467cb13	gitea registry ingress	2026-06-27 11:46:53 +02:00