Monitoring

You are wiring up the outside view of a service that already shipped: is it alive, is it fast, and when it breaks, does exactly one human get exactly one actionable page. This skill emits a concrete setup — a checker config, a health-endpoint contract, symptom-based alert rules, and an on-call rotation. Not telemetry instrumentation (that is monitoring-observability), not the release-gating healthcheck.

The one rule

A page is justified only when there is real user impact AND a human action the system can't take itself. Everything below descends from this. Internalize the three tiers:

• Page (wake someone): users are hurting now and a human must intervene. Checkout returns 5xx. Site unreachable. Error budget burning fast.
• Ticket (look during business hours): degraded but not bleeding. Slow-burn budget use, cert expiring in 14 days.
• Dashboard-only (don't notify): CPU at 80%, a single retry, a transient blip the system already healed.

If an alert doesn't map to an immediate human action, it is not a page — it's noise, and noise trains people to ignore the one page that matters.

The 4-layer stack

Don't skip layers and don't collapse them — each answers a different question.

Layer 1 catches "the whole thing is gone." Layer 2 catches "a dependency died" before users do. Layer 3 catches "we're degrading faster than we can afford." Layer 4 makes sure a human shows up.

Pick a tool

Decide by budget, team size, and self-host appetite. Pricing as of 2026-06.

Default: Uptime Kuma if you already run a VPS (1 GB box is comfortable; needs ~400 MB RAM), UptimeRobot free tier if you don't want infra. Kuma 2.1 added Globalping worldwide probe locations (so you test from regions, not just your one box) and built-in domain-expiry monitors. Concrete docker-compose and notification wiring live in references/tool-setup.md.

Health endpoints done right

Split liveness from readiness — they trigger different machine actions.

• /livez (liveness): is the process healthy? If it fails, the orchestrator restarts the container. Keep it dumb: just "am I running and not deadlocked." Never check the database here.
• /readyz (readiness): are dependencies reachable so I can serve traffic? If it fails, the orchestrator pulls this instance from the load-balancer rotation but does not kill it.

The cheap-probe rule: a probe runs constantly, so it must be <100ms and must not cascade-check every downstream. Why: if /livez pings the DB and the DB is briefly slow, liveness fails, the container restarts, the restart hammers the recovering DB — a restart loop that turns a 30-second blip into an outage.

# Bad — one /health that cascades and returns 500 on any hiccup.
# A slow Redis takes the whole service down and triggers restart loops.
@app.get("/health")
def health():
    db.execute("SELECT 1")          # blocks
    redis.ping()                    # blocks
    requests.get(PAYMENTS_URL)      # blocks on a third party!
    return {"status": "ok"}         # 500 if ANY of these throws

# Good — split, cheap, correct status codes, small JSON.
@app.get("/livez")                  # liveness: process only. Restart if this fails.
def livez():
    return {"status": "alive"}      # 200, ~1ms, touches nothing downstream

@app.get("/readyz")                 # readiness: deps with short timeouts. Pull from LB if this fails.
def readyz():
    checks = {"db": ping(db, timeout=0.2), "cache": ping(redis, timeout=0.1)}
    ok = all(checks.values())
    return JSONResponse(
        {"status": "ready" if ok else "degraded", "checks": checks},
        status_code=200 if ok else 503,   # 503 so the LB pulls this instance
    )

Do not put a third-party API call in readiness — a payment provider's outage shouldn't pull all your instances from rotation and take you fully down. Degrade that path in code, don't fail-closed on it here. Go and FastAPI handler examples are in references/tool-setup.md.

What to actually monitor

The golden checklist. Monitor the symptom users feel, not just that one URL returns 200.

• Availability — the critical endpoint(s) reachable, from multiple regions.
• Latency p95/p99 — averages hide the tail; alert on p95/p99 against a budget, not the mean.
• Error rate — the 5xx-to-total ratio, because raw 5xx count says nothing without traffic volume.
• SSL cert expiry — alert at 14 days; an expired cert is a full outage that no app metric catches.
• Domain expiry — alert at 30 days; a lapsed domain is the most embarrassing avoidable outage.
• The critical user journey — a synthetic check that does login → core action → result, NOT just GET /. The homepage can be 200 while checkout is broken.

The homepage being up tells you almost nothing. Probe the path that makes you money.

Alerts that don't cry wolf

Alert on symptoms, not causes. Page on "checkout error rate >2% for 5 min" (user impact), not on "CPU >80%" (a cause that may be harmless and self-resolving). High CPU with happy users is a dashboard line, not a 3am page.

Use multi-window, multi-burn-rate for SLO alerts (Google SRE workbook). Burn rate = how fast you're spending the monthly error budget. Require a long and a short window to both fire — the long window says "this is real," the short window says "this is still happening," and together they kill false positives from a single spike.

Routing: critical → page, warning → ticket, info → dashboard/review. Dedupe and group related alerts into one incident (10 hosts failing the same check = one page, not ten). Set maintenance windows so planned deploys don't page anyone.

Scale by criticality: for a small or single-user service, the on-call is just the owner — no rotation, no escalation chain, and the SLO can be a single availability target (for example 99% uptime over 30 days). Add a rotation and a secondary only when more than one person is on call, and add burn-rate alerts only when the service has a real error budget to spend. Do not build a full on-call rotation for a service with one user. The full burn-rate math and a copy-paste Prometheus-style rule with a runbook_url annotation are in references/burn-rate-and-oncall.md.

On-call basics

• Rotation: weekly, with a primary and a secondary. One person can't be the single point of failure for the system that catches single points of failure.
• Escalation policy: page primary → if no ack in 5–10 min, page secondary → then the manager. No-ack must always hop; an unacked page is a dropped page.
• Runbook per alert: every alert links to a runbook with five fields — Symptom, Impact, First 3 checks, Mitigation, How to escalate. The person paged at 3am should not have to think from scratch. Template in references/burn-rate-and-oncall.md.
• Status page + comms: a public status page (Kuma and Better Stack include one) so customers self-serve "is it you or me," cutting inbound during an incident.
• Do NOT start a new on-call on Opsgenie — Atlassian is retiring it (EOL 2027-04-05; new sales ended 2025-06-04). Grafana OnCall OSS was also deprecated (folded into Grafana Cloud IRM). For a new setup use PagerDuty, incident.io, Better Stack, or Grafana Cloud IRM.

Verify it works

An untested alert is not an alert. Before you call monitoring "done," prove the wire end-to-end:

1. Trigger a real synthetic failure (stop the service, or point a monitor at a forced-503 route).
2. Confirm the page actually lands on a phone — not just "the rule exists in the UI."
3. Let the ack timeout lapse and confirm escalation hops to the secondary.
4. Restore, and confirm the resolve/all-clear notification fires too.

scripts/verify.sh enforces the structural half of this on your config: liveness split from readiness, at least one symptom/burn-rate alert with two windows, a runbook reference somewhere in the alert rules, and a banlist for Opsgenie-as-new-setup and homepage-only monitors. The runbook check is haystack-wide — one reference anywhere passes; it does not prove every alert has one (see the script's header). Run it in CI so config drift can't silently re-introduce a noisy or untested setup.

Anti-patterns

References

• references/tool-setup.md — Uptime Kuma docker-compose (rootless image, volume, port), HTTP + push-heartbeat + SSL/domain-expiry monitors, a synthetic multi-step journey, notification wiring (ntfy / Slack webhook / PagerDuty integration key), a UptimeRobot/Better Stack monitor JSON shape, and Go + FastAPI health-endpoint handlers.
• references/burn-rate-and-oncall.md — full multi-window multi-burn-rate math, a copy-paste Prometheus-style alert rule with runbook_url, severity mapping, a sample escalation-policy YAML, and a fill-in runbook template.

Related in this repo: monitoring-observability — the telemetry side (what the app emits: logs, metrics, traces). The other siblings this skill originally linked to (observability, deployment, error-handling, domains-dns, scaling) are not part of ORCINUS; the links were removed because they pointed nowhere after installation.