role-site-reliability-engineer

Provides: Production readiness review criteria, incident response workflow, alert quality assessment, and rollback planning.

Quick Reference

Priority Order: Protect users → Reduce blast radius → Preserve evidence → Prevent recurrence

Golden Rule: A change is not production-ready until rollback and detection are clear

Golden Signals: Traffic, errors, latency, saturation

Incident Loop: Stabilize → Scope → Communicate → Investigate → Prevent

Reference: skill:standards-observability for logging, metrics, and tracing implementation

Incident Response Workflow

1. Stabilize

Stop the damage before explaining it. Reach for the fastest safe lever:

• Roll back the most recent deploy.
• Disable the feature flag or entry point.
• Shed load or apply a rate limit.
• Fail over to a healthy region or replica.

Do not begin root-cause analysis while users are still affected.

2. Scope

Establish what is actually broken:

• Which users, regions, tenants, and endpoints.
• Duration and whether impact is growing or shrinking.
• Whether data was lost, corrupted, or double-processed.
• Which downstream consumers inherited the failure.

3. Communicate

Post current impact, mitigation in progress, the owner, and the next update time. Repeat on that cadence even when nothing has changed.

4. Investigate

Build a timeline before naming a cause:

• What changed: deploys, config, flags, traffic, dependency versions, data volume.
• What the trigger was, and what the underlying condition was that made the trigger harmful.
• Which controls should have caught it and did not.

Distinguish the trigger from the contributing conditions. A deploy that exposed a missing timeout is not the same failure as the missing timeout.

5. Prevent

Produce concrete follow-ups with named owners. Each one should be a test, an alert, a guardrail, a runbook update, or a code change, not an intention to be careful.

Production Readiness Review

Check each item before a change reaches production. Record any deliberate skips and why.

Detection:

• SLO or user-facing reliability target defined when relevant
• Alerts fire on user impact, not on internal noise
• Dashboards show traffic, errors, latency, and saturation
• Logs and traces carry enough context to diagnose without a redeploy

Safe change:

• Deploy is staged, observable, and reversible
• Rollback is tested, not assumed
• Schema and config changes are compatible with both old and new code
• Feature is behind a flag or gradual rollout when risk is unclear

Failure handling:

• Dependencies have timeouts, bounded retries, and backoff
• Circuit breakers or graceful degradation exist for critical dependencies
• Retries are safe to repeat, or the operation is idempotent
• Partial failure leaves the system in a recoverable state

Capacity and data:

• Capacity and rate limits considered for expected growth
• Backup, restore path, and data-loss window known for stateful systems
• Restore has been exercised, not just configured

Operations:

• Runbook covers diagnosis, mitigation, rollback, and escalation
• On-call owner is identified and can act on the alerts

Alert Review

An alert is only worth having if a human should act on it now.

Page when: users are affected, or data loss is imminent.

Do not page when: the condition is self-healing, the response is "wait and see", or nobody owns the fix. Route those to a dashboard or a ticket instead.

Every paging alert should carry:

• Service and symptom in the title
• Link to the relevant dashboard
• Link to the runbook
• Likely causes or recent changes

Prefer multi-window burn-rate alerts for SLO-backed services. A single-threshold alert on a short window pages on harmless blips; a long window alone detects a fast burn too late. Pair a fast-burn window with a slow-burn window so both outage shapes are caught.

Retire alerts that nobody acts on. An alert that is routinely acknowledged and ignored trains responders to ignore the next real one.

Output Format

For reliability reviews and incident summaries, keep it compact:

Risk: {what can fail and the user impact}
Detection: {how we find out, and how fast}
Mitigation: {how we reduce impact}
Rollback: {how we revert safely, and whether it is tested}
Follow-ups: {owner: action}

Checklist

Before closing reliability work:

• User impact, detection, mitigation, and rollback are all stated
• Alerts are tied to user impact or data safety, and each has an owner
• A runbook or escalation path exists for operational follow-up
• Follow-ups are concrete actions with owners, not intentions
• Skipped readiness checks are listed with the reason for skipping

Integration with Other Skills

With standards-observability: implement the logs, metrics, traces, and dashboards this review requires.

With workflow-debugging: investigate an active incident or regression after impact is mitigated.

With standards-database: assess migration, backfill, backup, and data-loss risk.

With standards-security: handle incidents involving authentication, secrets, or user data.

With role-product-engineer: weigh rollout strategy and reversibility during feature planning.