agentleFS
mattstrublemattstruble/skills/homelab-monitoring/SKILL.md

homelab-monitoring

Skill2 starsChanged 4 months agoMITView on GitHub

You MUST consult this skill when deploying or operating a monitoring stack (Loki, Prometheus, Grafana, Alertmanager) on a homelab — k3s or Docker Compose. Also trigger when choosing a log collection agent (Alloy, Fluent Bit, Docker log driver), configuring retention, setting up alerting contact points (Discord, email, webhook), or troubleshooting Loki not ingesting logs, Prometheus showing 0 targets, or containers hanging due to the Docker log driver. NOT for query syntax (see logql, promql), dashboard authoring (see grafana), or cloud/EKS deployments.

  • Reads credentials

The file

Show its rules and text

We split this skill into the rules it gives an agent

16 rules. About 0 are common, 7 are distinctive, and 9 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 9

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  • could be automatic: a setting could enforce it
  1. Distinctive:

    Recommendation: Use Alloy. It's the official Grafana agent (Promtail reached EOL March 2026), handles both log and metric collection, and …

    line 49
  2. Very specific:

    Keep labels static and low-cardinality: app, container, host, namespace.

    line 97
  3. Very specific:

    Check Loki is ready: curl http://loki:3100/ready → should return ready

    line 161
  4. Very specific:

    Check Alloy logs: kubectl logs -n monitoring -l app.kubernetes.io/name=alloy or docker logs alloy

    line 162
  5. Distinctive:

    Verify the push URL in Alloy config matches Loki's service name/port

    line 163
  6. Very specific:

    Check for label cardinality errors: look for too many streams in Loki logs

    line 164
  7. Distinctive:

    Verify filesystem permissions on the Loki data directory

    line 165
  8. Very specific:

    Check Alloy pod is running: kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy

    line 185
  9. Very specific:

    Check RBAC: Alloy needs get/list/watch on pods and namespaces

    line 187
  10. Very specific:

    Test Loki connectivity from Alloy pod: kubectl exec -it alloy-xxx -- curl http://loki:3100/ready

    line 188
  11. Distinctive:

    Immediate fix: Remove the log driver from the affected container and restart it:

    line 197
  12. Very specific:

    Never commit secrets: Use .env files (Compose) or Kubernetes Secrets (k3s).

    line 212
  13. Very specific:

    If you ever expose Loki or Grafana beyond your LAN (reverse proxy, VPN exit), enable auth_enabled: true on Loki and put an auth proxy in …

    line 216
  14. Distinctive:

    Tempo: Add when you build custom instrumented services that call each other (distributed tracing)

    line 222
  15. Distinctive:

    Mimir: Add when you need >90 days metrics retention or multiple Prometheus instances

    line 223
  16. Distinctive:

    OTEL Collector: Add when you need a vendor-neutral pipeline or multiple telemetry backends

    line 224

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Homelab Monitoring

Loki + Prometheus + Grafana + Alertmanager on k3s or Docker Compose. No cloud, no GPU, no multi-tenancy — just a single-node or small-cluster homelab running services like Plex, *arr stack, Home Assistant, and AdGuard.

Architecture

┌─────────────────────────────────────────────────────────┐
│  Collection Layer                                        │
│  Alloy (DaemonSet/container) ──► Loki  (logs)           │
│  node-exporter + app /metrics ──► Prometheus (metrics)  │
└─────────────────────────────────────────────────────────┘
                    │                    │
                    ▼                    ▼
              ┌─────────┐         ┌───────────┐
              │  Loki   │         │ Prometheus │
              │ (fs)    │         │ (local PV) │
              └────┬────┘         └─────┬─────┘
                   └──────────┬─────────┘
                              ▼
                         ┌─────────┐     ┌──────────────┐
                         │ Grafana │────►│ Alertmanager │
                         └─────────┘     └──────────────┘

Both Loki and Prometheus use local filesystem storage — no MinIO, no S3, no object store needed at homelab scale.

Platform Decision

Running k3s? → See references/k3s-deploy.md for Helm-based deployment with kube-prometheus-stack, Grafana Operator CRDs, and Alloy DaemonSet.

Docker Compose only? → See references/compose-deploy.md for a full docker-compose.yml with file-based Grafana provisioning.

Log Collection Agent Decision

AgentBest ForCaveat
AlloyGrafana-native, handles logs + metrics, resilient to Loki downtimeSlightly more config than the Docker driver
Docker Loki log driverSimplest possible setup, zero extra containers⚠️ Blocks container stdout if Loki is unreachable — containers hang, stop logging, or fail to start. Never use in production.
Fluent BitMultiple output destinations, complex parsing, non-Grafana stacksMore moving parts; overkill for a single-destination homelab

Recommendation: Use Alloy. It's the official Grafana agent (Promtail reached EOL March 2026), handles both log and metric collection, and buffers locally when Loki is down.

Storage and Retention

Loki (filesystem)

# In Helm values (grafana/loki chart)
loki:
  auth_enabled: false
  storage:
    type: filesystem
  commonConfig:
    replication_factor: 1
  limits_config:
    retention_period: 720h   # 30 days
  compactor:
    retention_enabled: true
    compaction_interval: 10m
    retention_delete_delay: 2h

Filesystem storage path: /var/loki (mount a PV or bind-mount here).

Prometheus (local PV)

prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: 10GB   # whichever hits first
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path  # k3s default
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

Sizing guide: ~1–2 bytes/sample at 30s scrape interval.

  • 10 containers × 200 series each × 2 bytes × 2880 samples/day ≈ ~11 MB/day
  • 10 containers with node-exporter + kube-state-metrics ≈ ~150 MB/day total
  • 20 GB covers ~4 months at typical homelab scale

Label cardinality discipline (Loki)

Keep labels static and low-cardinality: app, container, host, namespace. Never use request IDs, trace IDs, or user IDs as labels — use filter expressions instead (|= "requestId=abc123"). High-cardinality labels fragment streams and degrade query performance.

Alerting Infrastructure

Alertmanager handles routing, deduplication, and delivery. It receives alerts from Prometheus (and optionally Loki ruler) and routes them to contact points.

Alertmanager config

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'discord'
  routes:
    - match:
        severity: critical
      receiver: 'discord'
    - match:
        severity: warning
      receiver: 'email'

receivers:
  - name: 'discord'
    discord_configs:
      - webhook_url: 'https://discord.com/api/webhooks/YOUR_ID/YOUR_TOKEN'
        title: '{{ .GroupLabels.alertname }}'
        message: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

  - name: 'email'
    email_configs:
      - to: 'you@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'
        auth_username: 'you@example.com'
        auth_password: 'your-app-password'

  - name: 'webhook'
    webhook_configs:
      - url: 'http://your-service:8080/alerts'

Alert rule expressions (PromQL/LogQL) belong in the promql and logql skills. This section covers only the routing infrastructure.

Troubleshooting

Quick-check table

SymptomFirst checkFix
Loki not ingestingcurl http://loki:3100/readySee below
Prometheus 0 targetsserviceMonitorSelectorNilUsesHelmValuesSet to false
Alloy not collectingalloy status or pod logsCheck Loki URL, auth
Container hung/not loggingDocker log driver in useSwitch to Alloy
Grafana "No data"Datasource URL wrongCheck service name
Alerts not firingAlertmanager not reachableCheck alertmanagerUrl

Loki not ingesting logs

  1. Check Loki is ready: curl http://loki:3100/ready → should return ready
  2. Check Alloy logs: kubectl logs -n monitoring -l app.kubernetes.io/name=alloy or docker logs alloy
  3. Verify the push URL in Alloy config matches Loki's service name/port
  4. Check for label cardinality errors: look for too many streams in Loki logs
  5. Verify filesystem permissions on the Loki data directory

Prometheus showing 0 targets

This is the #1 gotcha with kube-prometheus-stack. By default, Prometheus only discovers ServiceMonitors that have the same Helm release labels. ServiceMonitors you create manually are silently ignored.

Fix — set these in your Helm values:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false

Without this, kubectl get servicemonitor shows your monitors exist, but Prometheus never scrapes them.

Alloy not collecting

  1. Check Alloy pod is running: kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy
  2. Inspect Alloy config: verify loki.write endpoint URL
  3. Check RBAC: Alloy needs get/list/watch on pods and namespaces
  4. Test Loki connectivity from Alloy pod: kubectl exec -it alloy-xxx -- curl http://loki:3100/ready

Docker log driver blocking containers

The Docker Loki log driver sends logs synchronously to Loki. If Loki is unreachable (restarting, OOM, misconfigured), Docker blocks the container's stdout write, which can:

  • Cause containers to hang indefinitely
  • Prevent new containers from starting
  • Cause cascading failures across your stack

Immediate fix: Remove the log driver from the affected container and restart it:

# Remove this from docker-compose.yml:
logging:
  driver: loki
  options:
    loki-url: "http://loki:3100/loki/api/v1/push"

Long-term fix: Switch to Alloy for log collection. Alloy runs as a separate process, buffers locally, and never blocks your application containers.

Security

Homelab doesn't mean insecure. Three rules:

  1. Never commit secrets. Use .env files (Compose) or Kubernetes Secrets (k3s). Add .env to .gitignore.
  2. Bind internal services to localhost. Loki, Prometheus, and Alertmanager don't need network exposure — only Grafana needs to be reachable. In Compose: 127.0.0.1:3100:3100.
  3. Change defaults before first boot. Generate passwords: openssl rand -base64 24. The reference files show .env patterns for this.

If you ever expose Loki or Grafana beyond your LAN (reverse proxy, VPN exit), enable auth_enabled: true on Loki and put an auth proxy in front.

Upgrade Paths

These are one-line decisions — don't add them until you hit the specific need:

  • Tempo: Add when you build custom instrumented services that call each other (distributed tracing)
  • Mimir: Add when you need >90 days metrics retention or multiple Prometheus instances
  • OTEL Collector: Add when you need a vendor-neutral pipeline or multiple telemetry backends

Related Skills

  • logql — LogQL query syntax for Loki log queries
  • promql — PromQL query syntax for Prometheus metrics
  • k3s — k3s cluster setup and management
  • docker — Docker Compose and container configuration

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.