Nexus Production Incident Investigator

Time-boxed, evidence-driven workflow for investigating production incidents and evaluating release readiness.

Compatibility

• Sub-skills: release-readiness.md | Materials: checklists/incident-checklist.md, anti-patterns/common-mistakes.md, validation/output-validation.md
• Required: Read, Bash, Grep | Optional: WebSearch | Hands off to: nexus:debugging (code RCA), the roadmap-planner agent (post-mortem actions)

Core Principle

Stabilize first. Investigate second. Never skip the timeline.

Incident Classification

If you cannot classify immediately, default to P1 and downgrade after Phase 1.

Investigation Workflow

Phase 1 — Triage (0–5 min)

1. Confirm the signal — cross-check alert against at least one independent source (logs, dashboard, manual repro). Never declare from a single data point.
2. Classify severity using the table above.
3. Identify blast radius — which services, user segments, and whether data integrity is at risk.
4. Open incident channel and post: [P0/P1/P2] <service> — <symptom> — investigating
5. Assign roles (P0/P1): Incident Commander (comms only), Tech Lead (investigation), Scribe (timeline).

Phase 2 — Stabilize (5–15 min)

Apply mitigations before investigating. Stop at first "yes":

After mitigation, post: [Update] Mitigation: <what>. Error rate: <before> → <after>. Investigating root cause.

Phase 3 — Root Cause (15–60 min)

3.1 — Build the Timeline

Start here. Every other step depends on this. Format: T-Xh: [event] through T+Xm: [resolution]. Sources: deployment history, config audit logs, monitoring dashboards, application logs, infrastructure events (CloudTrail, K8s events).

3.2 — Signal Analysis

Check in priority order:

USE Method: Utilization → Saturation → Errors. Apply to CPUs, memory, disk I/O, network, connection pools.

3.3 — Correlation (priority order)

1. Deploys (within 2h) 2. Config/flag changes 3. Infrastructure changes 4. Traffic spikes 5. Dependency changes 6. Time-based patterns (cron/batch) 7. Data patterns (user/tenant/shape)

Rule: 80% of incidents are caused by the most recent deploy or config change.

git log --oneline --after="2 hours ago"
kubectl rollout history deployment/<service>
aws cloudtrail lookup-events --start-time ...

3.4 — Hypothesis Testing

For each hypothesis: what evidence supports it, what would disprove it, does it explain the full timeline? Root cause is confirmed when: (1) it explains all symptoms, (2) it explains why the mitigation worked, (3) you can state it in one sentence with a mechanism. If you cannot — you have a symptom, not a root cause.

3.5 — Reproduce in Staging (P1/P2)

Reproduce in staging, confirm symptoms match production, confirm fix resolves it before pushing.

Phase 4 — Post-Mortem (within 48h of resolution)

See checklists/incident-checklist.md for the full checklist. Required sections: Timeline, Root Cause (one sentence), Why It Happened, Why We Didn't Catch It Earlier, Mitigation/Resolution, Action Items (owner + due date + type), What Went Well, What Went Poorly.

Release Readiness

Route "is this safe to deploy" questions to release-readiness.md (pre-deploy checklist, risk scoring, Go/No-Go).

Output Contract

Every investigation must produce this report:

## Incident Report

Severity:             [P0/P1/P2/P3]
Status:               [Active / Resolved / Monitoring]
Duration:             [HH:MM from first symptom to resolution]
Impact:               [# users, % traffic, SLA breach Y/N, revenue impact]

Timeline:
  [T-Xh]: [event]
  [T+0]:  [incident start]
  [T+Xm]: [resolution]

Root Cause:           [one sentence — mechanism, not symptom]
Contributing Factors: [bullet list]
Blast Radius:         [services, features, user segments]

Mitigation:           [what stopped the bleeding and when]
Resolution:           [what fully fixed it]

Detection Gap:        [why not caught earlier]
Prevention:           [action items to prevent recurrence]

Next Step:            [one clear action, with owner]

Approval Gates

Ask before taking any of these actions — once per action:

Anti-Patterns

See anti-patterns/common-mistakes.md for the full list. Critical rules:

• Never investigate before stabilizing — apply available mitigations first.
• Never skip building the timeline — investigations without timelines produce wrong root causes.
• Never declare root cause before it explains all symptoms and the resolution.
• Never skip the post-mortem for "minor" incidents — they carry the most preventable learnings.
• Never roll back without confirming the rollback actually fixed the issue (verify metrics drop).

Heuristics Quick Reference

• Correlates with deploy → deployment is prime suspect
• Gradual degradation over hours → resource leak (memory, connections, disk)
• Sudden spike at consistent interval → cron job or batch process
• Subset of traffic affected → feature flag, canary, or data-pattern issue
• Single region only → regional config or network issue
• Recent change window → 80% of incidents caused by last deploy or config change