homelab-monitoring
Skill2 starsChanged 4 months agoMITView on GitHub
You MUST consult this skill when deploying or operating a monitoring stack (Loki, Prometheus, Grafana, Alertmanager) on a homelab — k3s or Docker Compose. Also trigger when choosing a log collection agent (Alloy, Fluent Bit, Docker log driver), configuring retention, setting up alerting contact points (Discord, email, webhook), or troubleshooting Loki not ingesting logs, Prometheus showing 0 targets, or containers hanging due to the Docker log driver. NOT for query syntax (see logql, promql), dashboard authoring (see grafana), or cloud/EKS deployments.
- Reads credentials
In this repository67 other files it gives its agents.
- AGENTS.mdAGENTS.md
- agent-architectureSkill
- agent-evaluationSkill
- agent-memorySkill
- agent-post-trainingSkill
- agent-self-evolutionSkill
- agent-tool-designSkill
- api-designSkill
- application-architectureSkill
- brainstormSkill
- code-reviewerSkill
- coding-agent-designSkill
- concurrency-designSkill
- context-engineeringSkill
- dockerSkill
- game-audioSkill
- game-designSkill
- game-narrativeSkill
- game-patternsSkill
- game-performanceSkill
- game-renderingSkill
- game-visualsSkill
- git-commitSkill
- git-prSkill
- godot-shaderSkill
- godotSkill
- gpu-rendering-architectureSkill
- grafanaSkill
- helmSkill
- k3sSkill
- k8s-networkingSkill
- k8s-operationsSkill
- k8s-storageSkill
- k8s-workloadsSkill
- knowledge-baseSkill
- level-designSkill
- llm-inference-servingSkill
- loggingSkill
- logqlSkill
- love2d-fennelSkill
- love2dSkill
- ml-platform-architectureSkill
- ml-post-trainingSkill
- multi-agent-collaborationSkill
- nix-dendriticSkill
- nix-packagingSkill
- nixSkill
- odin-designSkill
- odin-gamedevSkill
- orchestratorSkill
- physical-ai-systemsSkill
- plannerSkill
- prd-to-storiesSkill
- prd-writingSkill
- promqlSkill
- pr-reviewerSkill
- python-designSkill
- rag-designSkill
- ray-data-pipelinesSkill
- ray-kubernetes-infrastructureSkill
- The first 60 shown.
The file
Show its rules and text
We split this skill into the rules it gives an agent
16 rules. About 0 are common, 7 are distinctive, and 9 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
9
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Distinctive: line 49
Recommendation: Use Alloy. It's the official Grafana agent (Promtail reached EOL March 2026), handles both log and metric collection, and …
- Very specific: line 97
Keep labels static and low-cardinality: app, container, host, namespace.
- Very specific: line 161
Check Loki is ready: curl http://loki:3100/ready → should return ready
- Very specific: line 162
Check Alloy logs: kubectl logs -n monitoring -l app.kubernetes.io/name=alloy or docker logs alloy
- Distinctive: line 163
Verify the push URL in Alloy config matches Loki's service name/port
- Very specific: line 164
Check for label cardinality errors: look for too many streams in Loki logs
- Distinctive: line 165
Verify filesystem permissions on the Loki data directory
- Very specific: line 185
Check Alloy pod is running: kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy
- Very specific: line 187
Check RBAC: Alloy needs get/list/watch on pods and namespaces
- Very specific: line 188
Test Loki connectivity from Alloy pod: kubectl exec -it alloy-xxx -- curl http://loki:3100/ready
- Distinctive: line 197
Immediate fix: Remove the log driver from the affected container and restart it:
- Very specific: line 212
Never commit secrets: Use .env files (Compose) or Kubernetes Secrets (k3s).
- Very specific: line 216
If you ever expose Loki or Grafana beyond your LAN (reverse proxy, VPN exit), enable auth_enabled: true on Loki and put an auth proxy in …
- Distinctive: line 222
Tempo: Add when you build custom instrumented services that call each other (distributed tracing)
- Distinctive: line 223
Mimir: Add when you need >90 days metrics retention or multiple Prometheus instances
- Distinctive: line 224
OTEL Collector: Add when you need a vendor-neutral pipeline or multiple telemetry backends
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Homelab Monitoring
Loki + Prometheus + Grafana + Alertmanager on k3s or Docker Compose. No cloud, no GPU, no multi-tenancy — just a single-node or small-cluster homelab running services like Plex, *arr stack, Home Assistant, and AdGuard.
Architecture
┌─────────────────────────────────────────────────────────┐
│ Collection Layer │
│ Alloy (DaemonSet/container) ──► Loki (logs) │
│ node-exporter + app /metrics ──► Prometheus (metrics) │
└─────────────────────────────────────────────────────────┘
│ │
▼ ▼
┌─────────┐ ┌───────────┐
│ Loki │ │ Prometheus │
│ (fs) │ │ (local PV) │
└────┬────┘ └─────┬─────┘
└──────────┬─────────┘
▼
┌─────────┐ ┌──────────────┐
│ Grafana │────►│ Alertmanager │
└─────────┘ └──────────────┘Both Loki and Prometheus use local filesystem storage — no MinIO, no S3, no object store needed at homelab scale.
Platform Decision
Running k3s? → See references/k3s-deploy.md for Helm-based deployment with kube-prometheus-stack, Grafana Operator CRDs, and Alloy DaemonSet.
Docker Compose only? → See references/compose-deploy.md for a full docker-compose.yml with file-based Grafana provisioning.
Log Collection Agent Decision
| Agent | Best For | Caveat |
| Alloy | Grafana-native, handles logs + metrics, resilient to Loki downtime | Slightly more config than the Docker driver |
| Docker Loki log driver | Simplest possible setup, zero extra containers | ⚠️ Blocks container stdout if Loki is unreachable — containers hang, stop logging, or fail to start. Never use in production. |
| Fluent Bit | Multiple output destinations, complex parsing, non-Grafana stacks | More moving parts; overkill for a single-destination homelab |
Recommendation: Use Alloy. It's the official Grafana agent (Promtail reached EOL March 2026), handles both log and metric collection, and buffers locally when Loki is down.
Storage and Retention
Loki (filesystem)
# In Helm values (grafana/loki chart)
loki:
auth_enabled: false
storage:
type: filesystem
commonConfig:
replication_factor: 1
limits_config:
retention_period: 720h # 30 days
compactor:
retention_enabled: true
compaction_interval: 10m
retention_delete_delay: 2hFilesystem storage path: /var/loki (mount a PV or bind-mount here).
Prometheus (local PV)
prometheus:
prometheusSpec:
retention: 30d
retentionSize: 10GB # whichever hits first
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: local-path # k3s default
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20GiSizing guide: ~1–2 bytes/sample at 30s scrape interval.
- 10 containers × 200 series each × 2 bytes × 2880 samples/day ≈ ~11 MB/day
- 10 containers with node-exporter + kube-state-metrics ≈ ~150 MB/day total
- 20 GB covers ~4 months at typical homelab scale
Label cardinality discipline (Loki)
Keep labels static and low-cardinality: app, container, host, namespace. Never use request IDs, trace IDs, or user IDs as labels — use filter expressions instead (|= "requestId=abc123"). High-cardinality labels fragment streams and degrade query performance.
Alerting Infrastructure
Alertmanager handles routing, deduplication, and delivery. It receives alerts from Prometheus (and optionally Loki ruler) and routes them to contact points.
Alertmanager config
# alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'job']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'discord'
routes:
- match:
severity: critical
receiver: 'discord'
- match:
severity: warning
receiver: 'email'
receivers:
- name: 'discord'
discord_configs:
- webhook_url: 'https://discord.com/api/webhooks/YOUR_ID/YOUR_TOKEN'
title: '{{ .GroupLabels.alertname }}'
message: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
- name: 'email'
email_configs:
- to: 'you@example.com'
from: 'alertmanager@example.com'
smarthost: 'smtp.example.com:587'
auth_username: 'you@example.com'
auth_password: 'your-app-password'
- name: 'webhook'
webhook_configs:
- url: 'http://your-service:8080/alerts'Alert rule expressions (PromQL/LogQL) belong in the promql and logql skills. This section covers only the routing infrastructure.
Troubleshooting
Quick-check table
| Symptom | First check | Fix |
| Loki not ingesting | curl http://loki:3100/ready | See below |
| Prometheus 0 targets | serviceMonitorSelectorNilUsesHelmValues | Set to false |
| Alloy not collecting | alloy status or pod logs | Check Loki URL, auth |
| Container hung/not logging | Docker log driver in use | Switch to Alloy |
| Grafana "No data" | Datasource URL wrong | Check service name |
| Alerts not firing | Alertmanager not reachable | Check alertmanagerUrl |
Loki not ingesting logs
- Check Loki is ready:
curl http://loki:3100/ready→ should returnready - Check Alloy logs:
kubectl logs -n monitoring -l app.kubernetes.io/name=alloyordocker logs alloy - Verify the push URL in Alloy config matches Loki's service name/port
- Check for label cardinality errors: look for
too many streamsin Loki logs - Verify filesystem permissions on the Loki data directory
Prometheus showing 0 targets
This is the #1 gotcha with kube-prometheus-stack. By default, Prometheus only discovers ServiceMonitors that have the same Helm release labels. ServiceMonitors you create manually are silently ignored.
Fix — set these in your Helm values:
prometheus:
prometheusSpec:
serviceMonitorSelectorNilUsesHelmValues: false
podMonitorSelectorNilUsesHelmValues: false
ruleSelectorNilUsesHelmValues: falseWithout this, kubectl get servicemonitor shows your monitors exist, but Prometheus never scrapes them.
Alloy not collecting
- Check Alloy pod is running:
kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy - Inspect Alloy config: verify
loki.writeendpoint URL - Check RBAC: Alloy needs
get/list/watchon pods and namespaces - Test Loki connectivity from Alloy pod:
kubectl exec -it alloy-xxx -- curl http://loki:3100/ready
Docker log driver blocking containers
The Docker Loki log driver sends logs synchronously to Loki. If Loki is unreachable (restarting, OOM, misconfigured), Docker blocks the container's stdout write, which can:
- Cause containers to hang indefinitely
- Prevent new containers from starting
- Cause cascading failures across your stack
Immediate fix: Remove the log driver from the affected container and restart it:
# Remove this from docker-compose.yml:
logging:
driver: loki
options:
loki-url: "http://loki:3100/loki/api/v1/push"Long-term fix: Switch to Alloy for log collection. Alloy runs as a separate process, buffers locally, and never blocks your application containers.
Security
Homelab doesn't mean insecure. Three rules:
- Never commit secrets. Use
.envfiles (Compose) or Kubernetes Secrets (k3s). Add.envto.gitignore. - Bind internal services to localhost. Loki, Prometheus, and Alertmanager don't need network exposure — only Grafana needs to be reachable. In Compose:
127.0.0.1:3100:3100. - Change defaults before first boot. Generate passwords:
openssl rand -base64 24. The reference files show.envpatterns for this.
If you ever expose Loki or Grafana beyond your LAN (reverse proxy, VPN exit), enable auth_enabled: true on Loki and put an auth proxy in front.
Upgrade Paths
These are one-line decisions — don't add them until you hit the specific need:
- Tempo: Add when you build custom instrumented services that call each other (distributed tracing)
- Mimir: Add when you need >90 days metrics retention or multiple Prometheus instances
- OTEL Collector: Add when you need a vendor-neutral pipeline or multiple telemetry backends
Related Skills
- logql — LogQL query syntax for Loki log queries
- promql — PromQL query syntax for Prometheus metrics
- k3s — k3s cluster setup and management
- docker — Docker Compose and container configuration
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

