k3s-cluster/monitoring/dashboard-ideas.md

Dashboard Ideas

This file collects ideas for additional Grafana dashboards to build for the rogi.casa k3s cluster. Each idea notes the data source (metrics already available vs. metrics that need to be enabled) and a rough panel layout.

To actually add a dashboard, create a grafana-dashboard-<name>.yaml ConfigMap in this folder, mount it in grafana-deployment.yaml (add a volume + volumeMount under /var/lib/grafana/dashboards/<name>), commit and push.

---

Already-scraped services (ready to dashboard now)

These exporters/services are already being scraped by Prometheus — dashboards can be built immediately with no infra changes.

1. Traefik (Ingress) — traefik_*

Traefik is scraped via the kubernetes-pods job (pod annotation on traefik-9bcdbbd9-x8zq4 in kube-system). It exposes request counters, entry point latency, TLS handshakes, config reloads.

Panels:

• Requests/sec by entrypoint (web / websecure / traefik) — rate(traefik_entrypoint_requests_total[5m])
• Request latency p50/p95/p99 — histogram_quantile(0.95, sum(rate(traefik_entrypoint_request_duration_seconds_bucket[5m])) by (le, entrypoint))
• HTTP status code distribution (2xx/3xx/4xx/5xx) — rate(traefik_entrypoint_requests_total{code=~"2xx|3xx|4xx|5xx"}[5m])
• TLS handshakes/sec — rate(traefik_entrypoint_requests_tls_total[5m])
• Config reloads + last reload success — traefik_config_reloads_total, traefik_config_last_reload_success
• Top routes/services by request volume — topk(10, sum by (service) (rate(traefik_service_requests_total[5m])))
• Bytes transferred in/out — rate(traefik_entrypoint_requests_bytes_total[5m])

Why useful: This is your front door. Knowing which routes get hit most, latency per ingress, and 5xx spikes is the single most valuable app-level dashboard in the cluster.

---

2. CoreDNS (cluster DNS) — coredns_*

Scraped via kube-dns Service annotation. Exposes query rate, cache hits, error types, response duration.

Panels:

• DNS queries/sec by zone / type — rate(coredns_dns_requests_total[5m])
• Cache hit ratio — rate(coredns_cache_hits_total[5m]) / rate(coredns_cache_requests_total[5m])
• DNS query latency p95 — histogram_quantile(0.95, sum(rate(coredns_dns_request_duration_seconds_bucket[5m])) by (le))
• Queries by response code (NOERROR / NXDOMAIN / SERVFAIL) — rate(coredns_dns_responses_total[5m])
• Cache size — coredns_cache_entries
• Forward requests/sec (upstream DNS) — rate(coredns_forward_requests_total[5m])

Why useful: DNS issues cause cascading failures (ImagePullBackOff, cert challenges, etc.). A spike in NXDOMAIN/SERVFAIL is an early warning.

---

3. MetalLB (LoadBalancer) — metallb_*

Scraped via pod annotation on speaker-* and controller in metallb-system. Exposes IP allocation usage, BGP/session state.

Panels:

• IP addresses in use vs. total — metallb_allocator_addresses_in_use_total / metallb_allocator_addresses_total
• IP pool utilization % (gauge) — metallb_allocator_addresses_in_use_total / metallb_allocator_addresses_total * 100
• BGP session up per speaker — metallb_bgp_session_up
• Config loaded / stale status — metallb_k8s_client_config_loaded_bool, metallb_k8s_client_config_stale_bool
• Announcements per speaker — rate(metallb_bgp_announcements_total[5m])

Why useful: If MetalLB runs out of IPs, new LoadBalancer services will hang in <pending>. Knowing pool utilization lets you act before that happens.

---

4. cert-manager (TLS certificates) — certmanager_*

Scraped via pod annotations on cert-manager pods. Exposes certificate expiration, renewal, ready status, ACME challenges.

Panels:

• Certificate expiration (days remaining, sorted) — table of (certmanager_certificate_not_after_timestamp_seconds - time()) / 86400
• Certificates not Ready — certmanager_certificate_ready_status{condition="Ready",status!="True"}
• Upcoming renewals (next 14 days) — certmanager_certificate_renewal_timestamp_seconds
• ACME challenge status — certmanager_certificate_challenge_status
• Failed renewals counter — rate(certmanager_certificate_renewal_total{condition="Failed"}[1h])

Why useful: A cert about to expire (or silently failing to renew) is the kind of thing that takes down *.rogi.casa HTTPS with no warning. This is a must-have alert/dashboard.

---

5. Phoenix (trace store) — phoenix_*

Already scraped via the phoenix Service annotation. Exposes bulk loader ingestion rates, span insertion times, retention sweeper, exceptions.

Panels:

• Span ingestion rate — rate(phoenix_bulk_loader_span_insertion_time_seconds_count[5m])
• Span insertion latency p95 — histogram_quantile(0.95, sum(rate(phoenix_bulk_loader_span_insertion_time_seconds_bucket[5m])) by (le))
• Span exceptions/sec — rate(phoenix_bulk_loader_span_exceptions_total[5m])
• Retention sweeper last run — phoenix_retention_sweeper_last_run_seconds
• Last activity timestamp — phoenix_bulk_loader_last_activity_timestamp_seconds

Why useful: Phoenix is your observability backend's own backend. Tracking ingestion health tells you whether traces are landing.

---

Infrastructure dashboards (compose from existing metrics)

6. Storage & PVC Health (KSM + kubelet + node-exporter)

Cross-source dashboard combining kube_persistentvolumeclaim_* (KSM), kubelet_volume_stats_* (kubelet), and node_filesystem_* (node-exporter).

Panels:

• PVC usage % per claim — kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100
• PVC requested vs. capacity — kube_persistentvolumeclaim_resource_requests_storage_bytes vs actual
• Node disk usage % (all mounts) — (1 - node_filesystem_avail / node_filesystem_size) * 100
• Inode usage % per mount — (1 - node_filesystem_files_free / node_filesystem_files) * 100
• Volume binding status (Bound/Pending) — kube_persistentvolumeclaim_status_phase
• Top 10 PVCs by usage (table)

Why useful: The local-path provisioner fills up node disks. Catching a PVC at 95% before it errors is a lifesaver.

---

7. Workload Health (KSM)

Uses kube-state-metrics to show deployment/StatefulSet/CronJob health cluster-wide.

Panels:

• Deployments with unavailable replicas — kube_deployment_status_replicas_available < kube_deployment_status_replicas
• Pods not in Running phase by namespace — kube_pod_status_phase{phase!="Running"}
• Container restarts (last 1h) — increase(kube_pod_container_status_restarts_total[1h])
• Pods stuck in CrashLoopBackOff / ImagePullBackOff — kube_pod_container_status_waiting_reason{reason=~"CrashLoopBackOff|ImagePullBackOff"}
• Job failures — kube_job_failed
• CronJob schedule heatmap — kube_cronjob_status_active
• HPA status (if any autoscaled) — kube_horizontalpodautoscaler_status_current_replicas vs desired

Why useful: This is the "is anything broken" board. Notice you already have some pods in ImagePullBackOff (myorg-assistant) — this dashboard surfaces that.

---

8. etcd / Control Plane Health (if exposed)

k3s embeds etcd (or sqlite on single-node). etcd metrics require exposing the etcd /metrics endpoint (typically --listen-metrics-urls on the control plane node). Requires config change to enable.

Panels:

• Leader changes — etcd_server_leader_changes_seen_total
• Proposal commits/sec — rate(etcd_server_proposals_committed_total[5m])
• Proposal failures/sec — rate(etcd_server_proposals_failed_total[5m])
• DB size — etcd_mvcc_db_total_size_in_bytes
• RPC latency p99 — histogram_quantile(0.99, sum(rate(etcd_grpc Unary grpc latency bucket[5m])) by (le))
• Active watchers — etcd_debugging_mvcc_watcher_total

Why useful: etcd is the brain of the cluster. Slow commits or a flipping leader indicates control-plane trouble.

---

App-service dashboards (require enabling metrics first)

Most of your apps don't expose /metrics yet. Below is the per-service setup plus the dashboard idea once metrics are on. To enable scraping for any of these, annotate the Service with:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "<port>"

The existing kubernetes-service-endpoints scrape job will pick them up automatically — no Prometheus config edit needed.

9. LiteLLM (LLM gateway) — needs enabling

LiteLLM exposes Prometheus metrics on its API port (/metrics). Annotate the litellm Service.

Panels:

• Requests/sec by model — rate(litellm_requests_total[5m]) by model
• Token usage (prompt/completion/total) — rate(litellm_total_tokens_total[5m])
• Spend by model — litellm_spend_total (if cost tracking enabled)
• Latency p95 per model — histogram_quantile(0.95, ...)
• Error rate by model — rate(litellm_requests_total{status=~"5.."}[5m])
• Rate-limit / quota hits

Why useful: LiteLLM is the gateway for all your AI apps (open-webui, myorg-assistant, etc.). Token spend + per-model latency is the single best cost/quality lever in the cluster.

---

10. Gitea (git + CI) — needs enabling

Gitea exposes metrics at /metrics when ENABLE_METRICS=true in app.ini. Annotate gitea-http Service (port 3000 inside, 80 via svc).

Panels:

• Git push/clone/fetch rate — gitea_actions_total by action
• Active users / repos / orgs — gitea_users_total, gitea_repos_total
• Issues / PRs open — gitea_issues_total, gitea_pulls_total
• HTTP request rate + latency
• Gitea Actions runner job duration — if runner metrics exposed

Why useful: Gitea hosts the cluster's own GitOps repo + CI. Tracking push rate and runner throughput catches CI storms.

---

11. Home Assistant — needs enabling

HA exposes Prometheus metrics via the prometheus integration (add to configuration.yaml). Then annotate the Service.

Panels:

• Active entities / sensors by domain
• State change events/sec — homeassistant_entity_states_total
• Automation triggers/sec — homeassistant_automation_triggered_total
• Integrations loaded + errors
• Database size / recorder queue depth
• Zigbee/Z-Wave mesh health (if exposed)

Why useful: HA is a home-critical service. Event/sec spikes often indicate sensor flapping or runaway automations.

---

12. Jellyfin — limited

Jellyfin doesn't ship first-class Prometheus metrics, but you can scrape it via a sidecar (jellyfin-prometheus-exporter) or build a blackbox-style dashboard on the /health endpoint.

Panels:

• Active streams — from exporter
• Transcode sessions + hw accel usage
• Library size by media type
• Playback errors

---

13. Pi-hole — needs enabling

Pi-hole exposes metrics on its FTL web API; the pihole-exporter sidecar converts them to Prometheus format. Add as a sidecar container + annotate.

Panels:

• DNS queries/sec (total, blocked, cached, forwarded)
• Block list size
• Top blocked domains
• Top permitted domains
• Clients by query volume
• Cache hit ratio

Why useful: Pi-hole is your network-wide adblock. Block rate + cache ratio are the headline metrics, and query spikes reveal misbehaving clients.

---

14. PostgreSQL (litellm + phoenix + n8n) — needs enabling

You have two Postgres instances (postgres in litellm and phoenix). Add prometheus-postgres-exporter as a sidecar or Deployment per DB.

Panels (per DB):

• Connections (active / idle / max) — pg_stat_activity_count
• Transactions/sec — rate(pg_stat_database_xact_commit[5m])
• Cache hit ratio — pg_stat_database_blks_hit / (blks_hit + blks_read)
• Table + index bloat
• Replication lag (if replicas)
• Slow queries (if pg_stat_statements enabled)
• DB size growth — pg_database_size_bytes

Why useful: DB connection exhaustion and cache ratio collapse are the two most common causes of slow app performance.

---

15. Minecraft — limited

The Minecraft server exposes metrics via RCON + an exporter (minecraft-exporter). Add as sidecar using the existing RCON_PASSWORD.

Panels:

• Players online — minecraft_players_online
• TPS (ticks per second) — minecraft_tps (server health)
• Entities loaded — minecraft_entities_total
• Chunk count — minecraft_chunks_loaded
• Memory used by JVM

Why useful: TPS < 20 means lag. Player count vs. server load is the only real signal a Minecraft server needs.

---

16. qBittorrent — limited

No native metrics. Options: a qbittorrent-exporter sidecar (uses the WebUI API), or a blackbox probe on the WebUI.

Panels:

• Download/upload speed
• Active torrents
• Torrent count by state (downloading/seeding/paused)
• Disk usage in download dir

---

Cluster meta dashboards

17. Network Topology / Service Map

Composite view: for each namespace, list services, their pods, scrape status, and request volume (from Traefik logs + cAdvisor network). A "what talks to what" overview.

Panels:

• Service → pod → container resource table
• Cross-namespace network flows (if network policy logging enabled)
• Scrape health matrix (every target up/down)
• Ingress route → backend service map

---

18. Backup / Snapshot Status

If you take Velero snapshots or local-path snapshots, build a dashboard on velero_* or CRD status. Requires Velero.

Panels:

• Last successful backup per namespace
• Failed backups
• Backup size growth
• Restore test status

---

19. Cost / Capacity Planning

Composite: per-namespace CPU/memory requests vs. actual usage, projected growth, node saturation forecast.

Panels:

• Requests vs. limits vs. actual (per namespace) — KSM + cAdvisor
• Node capacity vs. allocatable
• PVC growth trend + 30-day forecast
• "What if I removed node X" simulation (capacity headroom)

Why useful: Tells you when you'll need another node before you hit the wall.

---

Recommended priority order

If you only build a few, do them in this order (highest value-to-effort first):

1. Traefik Ingress (#1) — already scraped, your front door
2. Storage & PVC Health (#6) — local-path fills disks; high blast radius
3. Workload Health (#7) — surfaces CrashLoopBackOff / ImagePullBackOff
4. cert-manager (#4) — prevents silent cert expiry outages
5. CoreDNS (#2) — early warning for DNS cascades
6. LiteLLM (#9) — needs prometheus.io/scrape annotation only; big insights
7. MetalLB (#3) — small but catches LoadBalancer IP exhaustion

Items 8–19 are nice-to-have or require additional exporters/config.