Monitoring & Alerting Design Review

Quick Reference

§1 Scope

In scope — monitoring and alerting for production backend services:

• SLI (Service Level Indicator) definition and measurement
• SLO (Service Level Objective) target setting and error budget
• Prometheus alerting rules (PromQL, for duration, severity labels)
• Burn-rate alerting (multi-window, multi-burn-rate SLO alerts)
• Grafana dashboard design (layout, variable templating, panel types)
• Alert routing configuration (PagerDuty/OpsGenie/Slack, severity-based routing)
• Alert fatigue audit (noise ratio, actionability, deduplication)
• Label cardinality management (high-cardinality label detection)
• On-call runbook integration (alert → runbook → action mapping)

Out of scope — delegate to dedicated skills:

• Metrics/tracing instrumentation in code → go-observability-review
• Application performance profiling → go-benchmark
• Infrastructure provisioning (Prometheus/Grafana setup) → ops tooling
• Log aggregation pipeline design → separate skill

§2 Mandatory Gates

Execute gates sequentially. Each gate has a STOP condition.

Gate 1: Context Collection

STOP: The request needs service context that is unavailable and unobtainable — no SLI candidates identifiable and the user cannot describe the service. Clarify before proceeding.

PROCEED, at the depth the context supports (this gate selects a §4 mode, it does not require full context):

Minimal mode is a legitimate entry point, not a gate failure — a pasted alert rule can be checked for a missing runbook or an absolute-count expression without knowing the service. What Minimal must not do is emit a threshold or an SLO verdict; those go to §8.9 as "requires service context". See §4 for the full matrix.

Gate 2: Scope Classification

STOP: Request is about code instrumentation (not alert/dashboard design). Redirect to go-observability-review.

PROCEED: Monitoring/alerting design intent confirmed.

Gate 3: Risk Classification

STOP: Any UNSAFE change without impact assessment.

PROCEED: Every change has risk level and mitigation.

Gate 4: Output Completeness

Before delivering output, verify all §8 Output Contract sections present. §8.9 Uncovered Risks must never be empty.

§3 Depth Selection

Force Standard or higher when any signal appears: SLO definition, burn-rate alerting, PagerDuty/OpsGenie routing, multi-service dashboard, alert fatigue investigation, label cardinality concern.

§4 Degradation Modes

When context is incomplete, degrade gracefully — never guess traffic patterns.

Hard rule: Never set alert thresholds without knowing the traffic pattern and the evaluation window — a ratio threshold is meaningless without both. At 10 QPS a [1m] window sees ~600 requests, so 1% needs 6 errors; a [10s] window sees ~100, so a single error crosses 1%. At 10K QPS the same 1% means ~100 errors/sec. The rule of thumb: min_errors_to_trip = threshold x QPS x window_seconds; when that number is below ~5, the alert is measuring noise. In Degraded mode, flag all thresholds as "requires traffic data to validate" in §8.9.

§5 Design Checklist

Execute every item. Mark PASS / WARN / FAIL with evidence.

5.1 SLI/SLO Foundation

1. SLIs defined for the service — every production service needs measurable SLIs, and each ratio SLI needs both halves defined: which events are valid (denominator — exclude health checks, synthetic probes, client cancellations) and which are good (numerator — decide explicitly how 429/499/4xx are treated). Undocumented exclusions make an SLI unauditable. Standard SLIs by service type:
   • API service: availability (good/valid ratio), latency as the proportion of requests under a threshold (e.g. 99% < 400ms — not a raw p99 value, which has no countable bad event to burn a budget against), error rate
   • Worker/consumer: processing rate, lag, error rate, processing latency
   • Batch job: completion rate, duration, data quality
   • Data pipeline: throughput, freshness, correctness
2. SLOs set with error budget, on a stated basis — each SLI has a target (e.g. "99.9% availability over 30 days") and error budget = 1 - SLO. Say which basis the budget is in: a time-based SLO converts directly to minutes (0.1% of 30 days = 43.2 min); a request-based SLO's budget is 0.1% of valid requests, which equals those minutes only under uniform traffic. Quoting minutes for a request-based SLO without that caveat promises a downtime guarantee the SLO does not make. SLOs must be agreed with stakeholders, not invented by engineers.
3. Burn-rate alerting for SLOs — instead of raw threshold alerts, use multi-window burn-rate alerts that fire when error budget is being consumed too fast. This dramatically reduces false positives. Load references/sli-slo-patterns.md for patterns.

5.2 Alert Rule Quality

4. Every alert is actionable — when this alert fires, is there a concrete action the on-call can take? If the answer is "look at it and hope it resolves," the alert should be a dashboard graph, not a page. Non-actionable alerts cause alert fatigue.

5. for duration prevents flapping — for is optional in Prometheus (omitted means fire on the first evaluation where the expression is true). Default to setting it, because absorbing transient spikes is what stops 3AM flapping: typical for: 5m for warning, for: 2m for critical. Too short on a noisy metric → flapping; too long (e.g. for: 30m) → delayed notification.

   Whether for is needed depends on whether the expression already expresses duration — "it looks like a liveness check" is not the same thing. up == 0 is true after a single failed scrape, so it needs for (Prometheus's own canonical example is up == 0 + for: 5m); a vector(1) watchdog needs none, because it is always firing and is alerted on by its absence. Full decision table, including absent() vs absent_over_time(): alert-anti-patterns.md §0.

   Flag a missing for as WARN, not FAIL, and name which row above you applied.

5b. Page-worthiness comes from user impact and redundancy, not from the signal — the gap the forward eval exposed: an agent explained that up is telemetry rather than customer impact, then set a single-replica up == 0 to severity: critical on the platform on-call in the same answer. Two questions decide it: how many replicas remain, and how much deadline headroom is left before the SLO is threatened. One of six replicas down is a ticket; all replicas down, a single-instance service, or a failing external probe is a page. State which you used — "paged because 3/3 replicas down" is reviewable, "paged because up == 0" is not. Full table: alert-anti-patterns.md §0.1.

6. Severity labels match routing — every alert carries a severity label from a closed, documented set, and every value in that set maps to exactly one routing policy. A policy is not a single destination: it has one primary action receiver (the thing that gets a human to act) plus zero or more secondary receivers for visibility, which is exactly what continue: true fan-out expresses (alertmanager-config-patterns.md §4). What must be unambiguous is who acts, not how many places see it. The common mapping is critical → page, warning → chat, info → dashboard, and the common tooling is PagerDuty/OpsGenie + Slack — but that is this skill's default example, not a requirement. Some orgs page from Slack with a bot, split critical into business hours vs out of hours, or have no info tier at all. What is actually reviewable: (a) the mapping is written down, (b) no severity value is unrouted, (c) nothing that pages a human is only sent somewhere nobody watches at 3AM. When the user states their own mapping, review against theirs.
7. Alert includes runbook link — every alerting rule should include an annotations.runbook_url pointing to a runbook with: what the alert means, how to diagnose, how to mitigate. Without runbooks, on-call responders waste time Googling their own alerts.

5.3 Dashboard Design

8. Dashboard follows USE/RED method — organize dashboards by signal type:
   • USE (infrastructure): Utilization, Saturation, Errors (CPU, memory, disk, network)
   • RED (services): Rate, Errors, Duration (request rate, error rate, latency)
   • Top row: golden signals overview. Detail rows: drill-down by endpoint/consumer/partition.
9. Dashboard uses variables for templating — Grafana variables ($service, $namespace, $instance) allow one dashboard to serve multiple instances. Avoid hardcoded label values in queries.
10. No unbounded labels in dashboard queries — get the causality right, because it decides where the fix goes. Series count is created at instrumentation time: an unbounded label (user_id, trace_id, full URL path) attached in application code multiplies series whether or not anything queries it. A query cannot create series — it can only load a huge number at once (slow panels, query-path memory, plausible OOM). So a query over an unbounded label is a symptom to flag, the real fix belongs in instrumentation (go-observability-review), and the dashboard-side fix is to aggregate away (sum by (status_code)) or filter to a bounded set. Use method, status_code, route template (/users/:id, not /users/12345).

5.4 Operations & Routing

11. Alert routing matches severity — no orphan alerts: every severity value in use resolves to a receiver, and the root route has a catch-all. Verify against the org's own mapping (§5.2 item 6) rather than assuming PagerDuty/Slack. The one non-negotiable: anything defined as page-worthy must reach a channel with an escalation path, not only a chat room.
12. Deduplication and grouping configured — group_by does not stop alert instances from firing; Prometheus still evaluates and fires one alert per label set, and they all appear in the Alertmanager UI and API. What grouping changes is notification volume: instances sharing the group_by labels are batched into one notification instead of N. group_wait sets how long to wait for more members of a new group, group_interval how long before sending an update for a group that already notified. Without grouping, one incident across N instances becomes N pages.
13. Inhibition rules prevent alert cascade — if the database is down, suppress all "elevated error rate" alerts from services that depend on it. Without inhibition, one root cause generates dozens of symptomatic alerts.
14. Alert fatigue metrics tracked — measure: total alerts/week, alerts-per-on-call-shift, % of alerts that required action, MTTA (mean time to acknowledge). Target: <5 pages/week per on-call, >80% actionability rate.

5.5 Rule Validation (Eat Your Own Dog Food)

A skill that preaches "no false positives, no silent gaps" must validate its rules mechanically, not by eyeball:

15. Rules pass promtool check rules alerts.yml — syntax + PromQL validation before any rule ships.

16. SLO-critical alerts have promtool test rules unit tests — Prometheus natively asserts "given this input series, this alert fires (or stays silent)". That is the executable form of the sensitivity/specificity tradeoff, and it is the only thing that catches a wrong window pairing or a wrong for, both of which parse perfectly. Write at least two cases per SLO-critical alert: one where it MUST fire, one where it MUST stay silent.

    A third case matters more than either: the alert must stop firing once the burn stops. A brief spike is silent whether the rule is single- or multi-window, so only "burn ended, must stop paging" proves the short-window gate is doing anything. Working example with all five cases (fire / silent / spike absorbed / health-checks excluded / clears after recovery), runnable as-is: tests/promtool/rules_test.yml.

17. Routing config passes amtool check-config alertmanager.yml — validate routing/inhibition changes before deploy.

If promtool/amtool are unavailable in the environment, state Not run — <tool> unavailable in §8.4 and list the exact commands for the user; never claim rules are validated without running them.

A skip is not a pass — for this skill's own suite too: run_regression.sh prints how many external validators ran and says PASS (text layer only) when that is zero. Report the same way.

18. Burn-rate arithmetic is derived, never quoted. budget_consumed = burn_rate x window / slo_window; time_to_exhaust = slo_window / burn_rate. These get confused constantly: for a 30-day SLO a 14.4x burn spends 2% of budget per hour and takes ~50 hours to exhaust it — not 2 hours. Compute both before writing either into a summary (scripts/lint_monitoring_docs.py checks every such claim here against the formulas).

§6 Anti-Examples

AE-1: Alert on absolute count instead of rate

# WRONG: fires when 10 errors exist (even over 24 hours = normal)
- alert: HighErrorCount
  expr: http_errors_total > 10
# RIGHT: rate relative to traffic — and complete, because a RIGHT example is copied
- alert: HighErrorRate
  expr: rate(http_errors_total[5m]) / rate(http_requests_total[5m]) > 0.01
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Error ratio above 1% on {{ $labels.job }}"
    runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

Check the threshold against traffic before shipping: at 10 QPS a [5m] window sees ~3000 requests, so 1% is ~30 errors — meaningful. At 1 QPS it is 3, which is noise (§4).

AE-2: No for duration — flapping on transient spike

# WRONG: fires immediately on any spike, resolves in seconds, pages at 3AM
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket[5m])) > 1
# RIGHT: require a sustained condition, and route it
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket[5m])) > 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "p99 latency above 1s on {{ $labels.job }}"
    runbook_url: "https://wiki.example.com/runbooks/high-latency"

This is a threshold alert on a percentile — fine as a symptom page. It is not an SLI: to burn an error budget you need the proportion of requests under the threshold instead (sli-slo-patterns.md §1.1).

AE-3: Alert without runbook — on-call doesn't know what to do

# WRONG: no annotations, no runbook
- alert: DatabaseConnectionPoolExhausted
  expr: db_pool_active >= db_pool_max
# RIGHT: include runbook and summary
- alert: DatabaseConnectionPoolExhausted
  expr: db_pool_active >= db_pool_max
  for: 2m
  annotations:
    summary: "DB connection pool exhausted on {{ $labels.instance }}"
    runbook_url: "https://wiki.example.com/runbooks/db-pool-exhausted"

AE-4: user_id in PromQL label — cardinality explosion

# WRONG: unique label per user → millions of time series
- record: user_request_duration
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket{user_id!=""}[5m]))
# RIGHT: use bounded labels only (method, status_code, endpoint)

AE-5: Page-worthy alert routed to chat only — no escalation

# WRONG: two problems. (a) page-worthy severity lands in a chat channel with no
# escalation, where it drowns in messages; (b) `match` is the deprecated form.
route:
  receiver: slack-team
  routes:
    - match: {severity: critical}
      receiver: slack-team
# RIGHT: modern `matchers`, and page-worthy severity reaches an escalation path
route:
  receiver: slack-team
  routes:
    - matchers: ['severity = "critical"']
      receiver: pagerduty-oncall

The receiver names are this skill's default example — review against the org's own severity → receiver mapping. What is never acceptable is a page-worthy alert whose only destination has no escalation.

AE-6: Monitoring gap reported as "system is stable"

WRONG: "no alerts fired this month, so the system is healthy". RIGHT: "no alerts fired — verify coverage first: are the SLIs measured, are the thresholds right? Absence of alerts ≠ absence of problems."

Extended anti-examples (AE-7 through AE-13) in references/alert-anti-patterns.md.

§7 Monitoring Scorecard

Critical — any FAIL means overall FAIL

• SLIs defined and measured, matching the service type (§5.1) — API: availability + latency + error rate; worker/consumer: lag + processing success rate; batch: on-time completion + correctness; pipeline: freshness + coverage + correctness. Judging a batch job against availability, latency, error rate fails a well-designed batch job for lacking SLIs it should not have
• Every alert is actionable (clear action when it fires, not just "investigate")
• Alert severity labels match routing — every severity value maps to a receiver under the org's own documented mapping, and page-worthy alerts reach a channel with escalation

Standard — 4 of 5 must pass

• SLOs set with error budget and stakeholder agreement
• for duration set on every alert that needs damping, or a stated reason it is omitted (deadman, discrete safety event, expression already integrates over time — §5.2 item 5)
• Runbook link included in alert annotations
• Dashboard follows USE/RED method with variable templating
• No unbounded labels in alert rules or dashboard queries (and the fix is routed to instrumentation, not just to the query)

Hygiene — 3 of 4 must pass

• Alert grouping and deduplication configured (Alertmanager group_by)
• Inhibition rules prevent alert cascade from single root cause
• Alert fatigue metrics tracked (alerts/week, actionability rate)
• Burn-rate alerting for SLO-critical services

Verdict: X/12; Critical: Y/3; Standard: Z/5; Hygiene: W/4. PASS requires: Critical 3/3 AND Standard ≥4/5 AND Hygiene ≥3/4.

§8 Output Contract

Every monitoring review MUST produce these sections. Write "N/A — [reason]" if inapplicable.

### 8.1 Context Gate
| Item | Value | Source |

### 8.2 Depth & Mode
[Lite/Standard/Deep] × [review/design/audit] — [rationale]

### 8.3 SLI/SLO Definition (Standard/Deep)
| SLI | Measurement | SLO Target | Error Budget |

### 8.4 Alert Rules
- Per alert: name, expr, for, severity, summary, runbook_url
- Validation evidence: `promtool check rules` / `promtool test rules` output,
  or `Not run — <tool> unavailable` + exact commands for the user

### 8.5 Dashboard Spec (Standard/Deep)
- Panel layout, queries, variables, drill-down structure

### 8.6 Routing Configuration
- Severity → destination mapping
- Grouping, deduplication, inhibition rules

### 8.7 Alert Fatigue Assessment (audit mode)
- Total alerts/week, actionability %, recommendations

### 8.8 Runbook Mapping
| Alert | Runbook URL | Last Updated |

### 8.9 Uncovered Risks (MANDATORY — never empty)
| Area | Reason | Impact | Follow-up |

Volume rules: FAIL always fully detailed; WARN up to 10 with overflow to §8.9; PASS summary only; §8.9 must document every assumption (especially an unknown traffic pattern).

Scorecard summary (append after §8.9):

Scorecard: X/12 — Critical Y/3, Standard Z/5, Hygiene W/4 — PASS/FAIL
Data basis: [full context | degraded | minimal | planning]

§9 Reference Loading Guide

§3's depth table sets the baseline. Load one reference early, regardless of depth, when its signal appears: sli-slo-patterns.md for SLO or burn-rate work, alert-anti-patterns.md for alert-fatigue investigation, alertmanager-config-patterns.md for routing, grouping or inhibition.