Compare commits

..

10 Commits

Author SHA1 Message Date
platform-engineer
e637550bb7 fix(myorg-assistant): inject WEB_SECRET_KEY into all cronjobs
CronJob containers import src.config.Settings() at startup, which requires
WEB_SECRET_KEY. Only the Deployment set it; the 5 CronJobs (git-sync,
morning-briefing, evening-summary, waiting-followup, deadline-checker) crashed
at import with a pydantic ValidationError, causing restart loops (e.g.
myorg-git-sync ~5 restarts/15m). Pull WEB_SECRET_KEY from the existing
myorg-assistant-secret so the cronjobs start cleanly.
2026-07-13 13:53:58 +00:00
Roger Oriol
a5b90994a4 fix myorg assistant 2026-07-11 19:22:57 +02:00
Roger Oriol
04b736287b fix argocd redirect 2026-07-09 23:48:47 +02:00
Roger Oriol
8c6950fd43 Merge branch 'main' of https://git.rogi.casa/roger/k3s-cluster 2026-07-09 00:45:40 +02:00
Roger Oriol
81dfe6fd60 fix argocd rbac and create agents.md 2026-07-09 00:45:10 +02:00
5d80abf3e8 Merge pull request 'fix: myorg-assistant cronjobs ImagePullBackOff + argocd root drift loop' (#18) from platform-engineer-agent/k3s-cluster:fix-merge-to-main into main
Reviewed-on: roger/k3s-cluster#18
2026-07-09 00:22:54 +02:00
48f18d2a3e Merge branch 'main' into fix-merge-to-main 2026-07-09 00:22:22 +02:00
Roger Oriol
ce08365e06 revert to qwen3.6 for platform engineer 2026-07-07 23:52:37 +02:00
Platform Engineer
0794153e56 fix(myorg-assistant): point cronjobs at registry image + add imagePullSecrets
CronJob pods were stuck in ImagePullBackOff because they referenced
the local-only image 'myorg-assistant:latest' which is not present
on the node. Switch all 5 cronjobs to the Gitea registry image
git.rogi.casa/roger/myorg-assistant/myorg-assistant:fcf79bf
(matching the Deployment), set imagePullPolicy: Always, and add
imagePullSecrets: gitea-registry so they can authenticate to the
private registry.
2026-07-06 14:11:44 +00:00
Platform Engineer
fc1b4383c1 fix(argocd): add ignoreDifferences to root app to stop Application CRD drift loop
The k3s-cluster-root app-of-apps has been in a continuous Synced<->OutOfSync
oscillation (489 transitions in 24h). Each cycle, ArgoCD syncs the 'argocd'
Application CRD, succeeds, then immediately detects drift because ArgoCD
adds status/operation fields to Application resources at runtime.

Adding ignoreDifferences for /status and /operation on Application resources
stops the drift loop while keeping the app-of-apps functional.
2026-07-06 14:11:44 +00:00
12 changed files with 312 additions and 15 deletions

246
AGENTS.md Normal file
View File

@@ -0,0 +1,246 @@
# AGENTS.md - Guide for Coding Agents
This file provides essential information for AI coding agents working with this Kubernetes cluster project.
## Project Overview
This repository contains Kubernetes manifests for a K3s cluster running self-hosted services on the `rogi.casa` domain. The cluster is managed via **GitOps using ArgoCD** - all changes to the cluster are deployed automatically from this Git repository.
**⚠️ CRITICAL: Permission Model**
You **DO NOT** have permission to push changes to this repository. Before applying any changes to the cluster:
1. Make the necessary code changes to the manifests
2. Clearly present the changes to the user
3. Ask the user to review and push the changes
4. Wait for confirmation that changes have been pushed
5. Only then will ArgoCD automatically deploy the changes to the cluster
## Architecture & GitOps Workflow
### ArgoCD App-of-Apps Pattern
This project uses ArgoCD's "app-of-apps" pattern:
```
argocd-bootstrap.yaml (root Application)
argocd/apps/ (directory containing all Application manifests)
Individual Applications (one per service directory)
Kubernetes manifests in each service directory (e.g., pihole/, homeassistant/)
```
### Deployment Flow
1. You make changes to Kubernetes manifests in the repository
2. User reviews and pushes changes to the `main` branch
3. ArgoCD detects changes (automatically or on sync)
4. ArgoCD applies changes to the cluster with `prune: true` and `selfHeal: true`
5. Cluster state converges to match the Git state
### Key Files
- **`argocd-bootstrap.yaml`**: The root Application that bootstraps ArgoCD. Points to `argocd/apps/` directory. This is the only file that needs manual `kubectl apply` during initial setup.
- **`argocd/apps/project.yaml`**: ArgoCD AppProject defining permissions for all applications
- **`argocd/apps/*.yaml`**: Individual ArgoCD Application manifests (one per service)
- **`argocd/gen-apps.sh`**: Script to regenerate all ArgoCD manifests from the `APPS` array
## Repository Structure
```
k3s-cluster/
├── argocd-bootstrap.yaml # Root ArgoCD Application (app-of-apps)
├── argocd/
│ ├── apps/ # Individual ArgoCD Application manifests
│ │ ├── project.yaml # AppProject definition
│ │ ├── pihole.yaml # Application for pihole/
│ │ ├── homeassistant.yaml # Application for homeassistant/
│ │ └── ... # One per service
│ ├── gen-apps.sh # Generates argocd/apps/* manifests
│ └── ingress.yaml # ArgoCD's own ingress
├── <service-name>/ # Each service has its own directory
│ ├── namespace.yaml # (Optional) Namespace definition
│ ├── deployment.yaml # Main deployment/statefulset
│ ├── service.yaml # Service definition
│ ├── ingress.yaml # Ingress configuration
│ ├── configmap.yaml # (Optional) ConfigMaps
│ ├── pvc.yaml # (Optional) PersistentVolumeClaims
│ └── secret.yaml # (Optional) Secrets (rarely committed)
├── cert-manager/ # cert-manager installation manifests
├── nas/ # External NAS service configuration
├── monitoring/ # Prometheus + Grafana stack
└── README.md # Comprehensive project documentation
```
## Current Services
The cluster runs these services (each in its own directory):
- **argocd** - GitOps continuous delivery platform
- **cert-manager** - SSL certificate management (Let's Encrypt)
- **fava** - Beancount accounting web interface
- **gitea** - Self-hosted Git server
- **glance** - Personal dashboard
- **gym-tracker** - Workout tracking application
- **homeassistant** - Home automation
- **jellyfin** - Media server
- **litellm** - LLM proxy
- **minecraft-server** - Minecraft server
- **monitoring** - Prometheus + Grafana
- **myorg-assistant** - Organization assistant
- **n8n** - Workflow automation
- **nas** - External NAS proxy
- **openwebui** - Web UI for LLMs
- **phoenix** - AI observability platform
- **pihole** - Network-wide ad blocking
- **platform-engineer** - Platform engineering tools
- **qbittorrent** - Torrent client
- **searxng** - Meta search engine
- **vaultwarden** - Password manager (Bitwarden compatible)
## How to Make Changes
### Adding a New Service
1. Create a new directory: `mkdir new-service`
2. Create Kubernetes manifests in `new-service/`:
- `namespace.yaml` (if dedicated namespace needed)
- `deployment.yaml` or `statefulset.yaml`
- `service.yaml`
- `ingress.yaml`
- Any ConfigMaps, Secrets, PVCs needed
3. Add the service to `argocd/gen-apps.sh`:
- Add a line to the `APPS` array: `"new-service|namespace|new-service|true|true"`
- Format: `name|namespace|path|recurse|validate`
4. Run `./argocd/gen-apps.sh` to regenerate ArgoCD manifests
5. **Present changes to user for review and push**
### Modifying an Existing Service
1. Edit the relevant manifest(s) in the service directory
2. If changing ArgoCD configuration, also update `argocd/gen-apps.sh` and regenerate
3. **Present changes to user for review and push**
### Removing a Service
1. Remove the service directory: `rm -rf service-name/`
2. Remove from `APPS` array in `argocd/gen-apps.sh`
3. Run `./argocd/gen-apps.sh` to regenerate
4. **Present changes to user for review and push**
5. ArgoCD will automatically prune the resources from the cluster
## Common Patterns
### Ingress Configuration
Each service has its own `ingress.yaml` with:
- `ingressClassName: traefik` (K3s default)
- TLS configured with `cert-manager.io/cluster-issuer: letsencrypt-prod`
- Host-based routing (e.g., `pihole.rogi.casa`)
Example:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: pihole
namespace: pihole
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: traefik
tls:
- hosts:
- pihole.rogi.casa
secretName: pihole-tls
rules:
- host: pihole.rogi.casa
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: pihole-web
port:
number: 80
```
### Resource Management
- Each service typically has its own namespace
- Use ResourceRequests and Limits for all containers
- PVCs for persistent data
- ConfigMaps for configuration files
## Important Notes
### What You CAN Do
- Read and understand all manifests
- Create new manifest files
- Modify existing manifest files
- Run `./argocd/gen-apps.sh` to regenerate ArgoCD manifests
- Explain how the cluster works
- Troubleshoot issues by reading manifests
### What You CANNOT Do
- Push changes to the Git repository (no push permissions)
- Directly apply manifests with `kubectl apply` (unless explicitly asked)
- Access the Kubernetes cluster directly (unless explicitly configured)
- Create secrets that should remain private (those are managed manually)
### Secrets Management
Secrets are generally **not committed to the repository**. They must be created manually in the cluster:
```bash
kubectl create secret docker-registry gitea-registry \
--docker-server=gitea.rogi.casa \
--docker-username=<user> \
--docker-password=<token> \
-n <namespace>
```
## Workflow Summary
When asked to make changes:
1. **Understand** the current state by reading relevant files
2. **Modify** the manifests (create/edit files)
3. **Regenerate** ArgoCD manifests if needed (`./argocd/gen-apps.sh`)
4. **Present** the changes clearly to the user:
```
I've made the following changes:
- Modified pihole/deployment.yaml to update image version
- Regenerated argocd/apps/pihole.yaml
Please review and push these changes to deploy them.
```
5. **Wait** for user confirmation that changes are pushed
6. **Verify** (if possible) that ArgoCD has synced the changes
## Useful Commands (for reference)
```bash
# Regenerate ArgoCD manifests after modifying gen-apps.sh
./argocd/gen-apps.sh
# Check ArgoCD applications status (requires kubectl access)
kubectl get applications -n argocd
# View logs of a pod (requires kubectl access)
kubectl logs -n <namespace> <pod-name>
# Check ingress status (requires kubectl access)
kubectl get ingress -n <namespace>
```
## Questions?
If you're unsure about anything:
1. Read the comprehensive `README.md` in the repository root
2. Check existing service directories for examples
3. Ask the user for clarification before making changes
4. Remember: **never push without explicit user review and approval**

View File

@@ -22,3 +22,9 @@ spec:
selfHeal: true selfHeal: true
syncOptions: syncOptions:
- CreateNamespace=false - CreateNamespace=false
ignoreDifferences:
- group: argoproj.io
kind: Application
jsonPointers:
- /status
- /operation

View File

@@ -7,6 +7,11 @@ metadata:
app.kubernetes.io/name: argocd-cm app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd app.kubernetes.io/part-of: argocd
data: data:
# Serve HTTP (no redirect to HTTPS) so the TLS-terminating Traefik ingress works.
# Without this, argocd-server redirects HTTP->HTTPS, causing an infinite
# redirect loop behind the ingress (argocd.rogi.casa unreachable).
server.insecure: "true"
# add an additional local user with apiKey and login capabilities # add an additional local user with apiKey and login capabilities
# apiKey - allows generating API keys # apiKey - allows generating API keys
# login - allows to login using UI # login - allows to login using UI

View File

@@ -7,14 +7,23 @@ metadata:
app.kubernetes.io/name: argocd-rbac-cm app.kubernetes.io/name: argocd-rbac-cm
app.kubernetes.io/part-of: argocd app.kubernetes.io/part-of: argocd
data: data:
# Default policy: readonly for all authenticated users policy.csv: |
# Grant platform-engineer read-only access to applications
g, platform-engineer, role:readonly
# Custom policy for platform-engineer with application read permissions
p, role:platform-engineer, applications, get, *, allow
p, role:platform-engineer, applications, list, *, allow
p, role:platform-engineer, clusters, get, *, allow
p, role:platform-engineer, clusters, list, *, allow
p, role:platform-engineer, repositories, get, *, allow
p, role:platform-engineer, repositories, list, *, allow
p, role:platform-engineer, projects, get, *, allow
p, role:platform-engineer, projects, list, *, allow
g, platform-engineer, role:platform-engineer
# Default policy - deny by default (ArgoCD default)
policy.default: role:readonly policy.default: role:readonly
# platform-engineer account: read all apps + trigger syncs # Enable RBAC
policy.csv: | rbac.enabled: "true"
p, role:platform-engineer, applications, get, */*, allow
p, role:platform-engineer, applications, sync, */*, allow
p, role:platform-engineer, clusters, get, *, allow
p, role:platform-engineer, repositories, get, *, allow
p, role:platform-engineer, projects, get, *, allow
g, platform-engineer, role:platform-engineer

View File

@@ -30,7 +30,8 @@ spec:
imagePullPolicy: Always imagePullPolicy: Always
command: command:
- python - python
- run_job.py - -c
- "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
- deadline-checker - deadline-checker
env: env:
- name: MYORG_REPO_PATH - name: MYORG_REPO_PATH
@@ -53,6 +54,11 @@ spec:
secretKeyRef: secretKeyRef:
name: myorg-assistant-secret name: myorg-assistant-secret
key: LITELLM_API_KEY key: LITELLM_API_KEY
- name: WEB_SECRET_KEY
valueFrom:
secretKeyRef:
name: myorg-assistant-secret
key: WEB_SECRET_KEY
volumeMounts: volumeMounts:
- name: myorg-data - name: myorg-data
mountPath: /data/myorg mountPath: /data/myorg

View File

@@ -30,7 +30,8 @@ spec:
imagePullPolicy: Always imagePullPolicy: Always
command: command:
- python - python
- run_job.py - -c
- "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
- evening-summary - evening-summary
env: env:
- name: MYORG_REPO_PATH - name: MYORG_REPO_PATH
@@ -53,6 +54,11 @@ spec:
secretKeyRef: secretKeyRef:
name: myorg-assistant-secret name: myorg-assistant-secret
key: LITELLM_API_KEY key: LITELLM_API_KEY
- name: WEB_SECRET_KEY
valueFrom:
secretKeyRef:
name: myorg-assistant-secret
key: WEB_SECRET_KEY
volumeMounts: volumeMounts:
- name: myorg-data - name: myorg-data
mountPath: /data/myorg mountPath: /data/myorg

View File

@@ -30,7 +30,8 @@ spec:
imagePullPolicy: Always imagePullPolicy: Always
command: command:
- python - python
- run_job.py - -c
- "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
- git-sync - git-sync
env: env:
- name: MYORG_REPO_PATH - name: MYORG_REPO_PATH
@@ -68,6 +69,11 @@ spec:
secretKeyRef: secretKeyRef:
name: myorg-assistant-secret name: myorg-assistant-secret
key: LITELLM_API_KEY key: LITELLM_API_KEY
- name: WEB_SECRET_KEY
valueFrom:
secretKeyRef:
name: myorg-assistant-secret
key: WEB_SECRET_KEY
volumeMounts: volumeMounts:
- name: myorg-data - name: myorg-data
mountPath: /data/myorg mountPath: /data/myorg

View File

@@ -30,7 +30,8 @@ spec:
imagePullPolicy: Always imagePullPolicy: Always
command: command:
- python - python
- run_job.py - -c
- "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
- morning-briefing - morning-briefing
env: env:
# From ConfigMap # From ConfigMap
@@ -60,6 +61,11 @@ spec:
secretKeyRef: secretKeyRef:
name: myorg-assistant-secret name: myorg-assistant-secret
key: LITELLM_API_KEY key: LITELLM_API_KEY
- name: WEB_SECRET_KEY
valueFrom:
secretKeyRef:
name: myorg-assistant-secret
key: WEB_SECRET_KEY
volumeMounts: volumeMounts:
- name: myorg-data - name: myorg-data
mountPath: /data/myorg mountPath: /data/myorg

View File

@@ -30,7 +30,8 @@ spec:
imagePullPolicy: Always imagePullPolicy: Always
command: command:
- python - python
- run_job.py - -c
- "from src.scheduler.jobs import run_job; import sys; run_job(sys.argv[1])"
- waiting-followup - waiting-followup
env: env:
- name: MYORG_REPO_PATH - name: MYORG_REPO_PATH
@@ -53,6 +54,11 @@ spec:
secretKeyRef: secretKeyRef:
name: myorg-assistant-secret name: myorg-assistant-secret
key: LITELLM_API_KEY key: LITELLM_API_KEY
- name: WEB_SECRET_KEY
valueFrom:
secretKeyRef:
name: myorg-assistant-secret
key: WEB_SECRET_KEY
volumeMounts: volumeMounts:
- name: myorg-data - name: myorg-data
mountPath: /data/myorg mountPath: /data/myorg

View File

@@ -34,7 +34,7 @@ spec:
git config user.name "${GIT_USERNAME}" git config user.name "${GIT_USERNAME}"
git config user.email "${GIT_USERNAME}@rogi.casa" git config user.email "${GIT_USERNAME}@rogi.casa"
git config credential.helper store git config credential.helper store
echo "https://${GIT_USERNAME}:${GIT_TOKEN}@gitea.rogi.casa" > ~/.git-credentials echo "https://${GIT_USERNAME}:${GIT_TOKEN}@git.rogi.casa" > ~/.git-credentials
else else
echo "Repository already exists, pulling latest changes..." echo "Repository already exists, pulling latest changes..."
cd /data/myorg cd /data/myorg

View File

@@ -9,18 +9,18 @@ data:
config.yaml: | config.yaml: |
model: model:
provider: openai-api provider: openai-api
default: z-ai/glm-5.2 default: qwen3.6
base_url: "http://litellm-service.litellm:80/v1" base_url: "http://litellm-service.litellm:80/v1"
api_mode: chat_completions api_mode: chat_completions
auxiliary: auxiliary:
compression: compression:
provider: openai-api provider: openai-api
model: z-ai/glm-5.2 model: qwen3.6
base_url: "http://litellm-service.litellm:80/v1" base_url: "http://litellm-service.litellm:80/v1"
title_generation: title_generation:
provider: openai-api provider: openai-api
model: z-ai/glm-5.2 model: qwen3.6
base_url: "http://litellm-service.litellm:80/v1" base_url: "http://litellm-service.litellm:80/v1"
terminal: terminal:

View File

@@ -86,6 +86,7 @@ spec:
: > /opt/data/.env : > /opt/data/.env
chmod 600 /opt/data/.env chmod 600 /opt/data/.env
for k in OPENAI_API_KEY OPENAI_BASE_URL DISCORD_BOT_TOKEN DISCORD_HOME_CHANNEL \ for k in OPENAI_API_KEY OPENAI_BASE_URL DISCORD_BOT_TOKEN DISCORD_HOME_CHANNEL \
DISCORD_ALLOW_ALL_USERS DISCORD_FREE_RESPONSE_CHANNELS \
GITEA_TOKEN GITEA_REPO_URL ARGOCD_API_TOKEN ARGOCD_SERVER \ GITEA_TOKEN GITEA_REPO_URL ARGOCD_API_TOKEN ARGOCD_SERVER \
HERMES_DASHBOARD HERMES_DASHBOARD_BASIC_AUTH_USERNAME \ HERMES_DASHBOARD HERMES_DASHBOARD_BASIC_AUTH_USERNAME \
HERMES_DASHBOARD_BASIC_AUTH_PASSWORD HERMES_DASHBOARD_BASIC_AUTH_SECRET; do HERMES_DASHBOARD_BASIC_AUTH_PASSWORD HERMES_DASHBOARD_BASIC_AUTH_SECRET; do