k3s-cluster/AGENTS.md

AGENTS.md - Guide for Coding Agents

This file provides essential information for AI coding agents working with this Kubernetes cluster project.

Project Overview

This repository contains Kubernetes manifests for a K3s cluster running self-hosted services on the rogi.casa domain. The cluster is managed via GitOps using ArgoCD - all changes to the cluster are deployed automatically from this Git repository.

⚠️ CRITICAL: Permission Model

You DO NOT have permission to push changes to this repository. Before applying any changes to the cluster:

1. Make the necessary code changes to the manifests
2. Clearly present the changes to the user
3. Ask the user to review and push the changes
4. Wait for confirmation that changes have been pushed
5. Only then will ArgoCD automatically deploy the changes to the cluster

Architecture & GitOps Workflow

ArgoCD App-of-Apps Pattern

This project uses ArgoCD's "app-of-apps" pattern:

argocd-bootstrap.yaml (root Application)
    ↓
argocd/apps/ (directory containing all Application manifests)
    ↓
Individual Applications (one per service directory)
    ↓
Kubernetes manifests in each service directory (e.g., pihole/, homeassistant/)

Deployment Flow

1. You make changes to Kubernetes manifests in the repository
2. User reviews and pushes changes to the main branch
3. ArgoCD detects changes (automatically or on sync)
4. ArgoCD applies changes to the cluster with prune: true and selfHeal: true
5. Cluster state converges to match the Git state

Key Files

• argocd-bootstrap.yaml: The root Application that bootstraps ArgoCD. Points to argocd/apps/ directory. This is the only file that needs manual kubectl apply during initial setup.
• argocd/apps/project.yaml: ArgoCD AppProject defining permissions for all applications
• argocd/apps/*.yaml: Individual ArgoCD Application manifests (one per service)
• argocd/gen-apps.sh: Script to regenerate all ArgoCD manifests from the APPS array

Repository Structure

k3s-cluster/
├── argocd-bootstrap.yaml          # Root ArgoCD Application (app-of-apps)
├── argocd/
│   ├── apps/                      # Individual ArgoCD Application manifests
│   │   ├── project.yaml           # AppProject definition
│   │   ├── pihole.yaml            # Application for pihole/
│   │   ├── homeassistant.yaml     # Application for homeassistant/
│   │   └── ...                    # One per service
│   ├── gen-apps.sh                # Generates argocd/apps/* manifests
│   └── ingress.yaml               # ArgoCD's own ingress
├── <service-name>/                # Each service has its own directory
│   ├── namespace.yaml             # (Optional) Namespace definition
│   ├── deployment.yaml            # Main deployment/statefulset
│   ├── service.yaml               # Service definition
│   ├── ingress.yaml               # Ingress configuration
│   ├── configmap.yaml             # (Optional) ConfigMaps
│   ├── pvc.yaml                   # (Optional) PersistentVolumeClaims
│   └── secret.yaml                # (Optional) Secrets (rarely committed)
├── cert-manager/                  # cert-manager installation manifests
├── nas/                           # External NAS service configuration
├── monitoring/                    # Prometheus + Grafana stack
└── README.md                      # Comprehensive project documentation

Current Services

The cluster runs these services (each in its own directory):

• argocd - GitOps continuous delivery platform
• cert-manager - SSL certificate management (Let's Encrypt)
• fava - Beancount accounting web interface
• gitea - Self-hosted Git server
• glance - Personal dashboard
• gym-tracker - Workout tracking application
• homeassistant - Home automation
• jellyfin - Media server
• litellm - LLM proxy
• minecraft-server - Minecraft server
• monitoring - Prometheus + Grafana
• myorg-assistant - Organization assistant
• n8n - Workflow automation
• nas - External NAS proxy
• openwebui - Web UI for LLMs
• phoenix - AI observability platform
• pihole - Network-wide ad blocking
• platform-engineer - Platform engineering tools
• qbittorrent - Torrent client
• searxng - Meta search engine
• vaultwarden - Password manager (Bitwarden compatible)

How to Make Changes

Adding a New Service

1. Create a new directory: mkdir new-service
2. Create Kubernetes manifests in new-service/:
   • namespace.yaml (if dedicated namespace needed)
   • deployment.yaml or statefulset.yaml
   • service.yaml
   • ingress.yaml
   • Any ConfigMaps, Secrets, PVCs needed
3. Add the service to argocd/gen-apps.sh:
   • Add a line to the APPS array: "new-service|namespace|new-service|true|true"
   • Format: name|namespace|path|recurse|validate
4. Run ./argocd/gen-apps.sh to regenerate ArgoCD manifests
5. Present changes to user for review and push

Modifying an Existing Service

1. Edit the relevant manifest(s) in the service directory
2. If changing ArgoCD configuration, also update argocd/gen-apps.sh and regenerate
3. Present changes to user for review and push

Removing a Service

1. Remove the service directory: rm -rf service-name/
2. Remove from APPS array in argocd/gen-apps.sh
3. Run ./argocd/gen-apps.sh to regenerate
4. Present changes to user for review and push
5. ArgoCD will automatically prune the resources from the cluster

Common Patterns

Ingress Configuration

Each service has its own ingress.yaml with:

• ingressClassName: traefik (K3s default)
• TLS configured with cert-manager.io/cluster-issuer: letsencrypt-prod
• Host-based routing (e.g., pihole.rogi.casa)

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pihole
  namespace: pihole
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - pihole.rogi.casa
    secretName: pihole-tls
  rules:
  - host: pihole.rogi.casa
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: pihole-web
              port:
                number: 80

Resource Management

• Each service typically has its own namespace
• Use ResourceRequests and Limits for all containers
• PVCs for persistent data
• ConfigMaps for configuration files

Important Notes

What You CAN Do

• Read and understand all manifests
• Create new manifest files
• Modify existing manifest files
• Run ./argocd/gen-apps.sh to regenerate ArgoCD manifests
• Explain how the cluster works
• Troubleshoot issues by reading manifests

What You CANNOT Do

• Push changes to the Git repository (no push permissions)
• Directly apply manifests with kubectl apply (unless explicitly asked)
• Access the Kubernetes cluster directly (unless explicitly configured)
• Create secrets that should remain private (those are managed manually)

Secrets Management

Secrets are generally not committed to the repository. They must be created manually in the cluster:

kubectl create secret docker-registry gitea-registry \
  --docker-server=gitea.rogi.casa \
  --docker-username=<user> \
  --docker-password=<token> \
  -n <namespace>

Workflow Summary

When asked to make changes:

1. Understand the current state by reading relevant files
2. Modify the manifests (create/edit files)
3. Regenerate ArgoCD manifests if needed (./argocd/gen-apps.sh)
4. Present the changes clearly to the user:

   I've made the following changes:
   - Modified pihole/deployment.yaml to update image version
   - Regenerated argocd/apps/pihole.yaml
   
   Please review and push these changes to deploy them.
   

5. Wait for user confirmation that changes are pushed
6. Verify (if possible) that ArgoCD has synced the changes

Useful Commands (for reference)

# Regenerate ArgoCD manifests after modifying gen-apps.sh
./argocd/gen-apps.sh

# Check ArgoCD applications status (requires kubectl access)
kubectl get applications -n argocd

# View logs of a pod (requires kubectl access)
kubectl logs -n <namespace> <pod-name>

# Check ingress status (requires kubectl access)
kubectl get ingress -n <namespace>

Questions?

If you're unsure about anything:

1. Read the comprehensive README.md in the repository root
2. Check existing service directories for examples
3. Ask the user for clarification before making changes
4. Remember: never push without explicit user review and approval