production-incident-diagnostic
Skill9 starsChanged 25 days agoMITView on GitHub
Use when diagnosing a production incident - "prod is down", "la prod est down", an API responding 502/500/slow, abnormal behavior in prod, a user reporting something broken in production, alerting gone silent. Triggers on localizing the faulty layer of a live system.
In this repository59 other files it gives its agents.
- AGENTS.mdAGENTS.md
- adversarial-feature-challengeSkill
- api-contract-versioningSkill
- bug-ticket-root-causeSkill
- bug-triage-structuredSkill
- ci-pipeline-orchestrationSkill
- commit-readiness-reviewSkill
- comprehensive-test-strategySkill
- database-schema-evolutionSkill
- ddd-backend-implementationSkill
- design-system-component-lifecycleSkill
- detection-sweepSkill
- domain-glossarySkill
- domain-modeling-designSkill
- frontend-fullstack-implementationSkill
- frontend-spec-call-site-auditSkill
- merge-request-reviewSkill
- performance-profilingSkill
- recording-decisionsSkill
- recurring-bug-root-causeSkill
- refactoring-shared-component-apiSkill
- validating-features-end-to-endSkill
- visual-regression-checkSkill
- writing-robust-testsSkill
- adversarial-feature-challengeSkill
- api-contract-versioningSkill
- audit-architectureSkill
- audit-ci-cdSkill
- audit-complianceSkill
- audit-industrialisationSkill
- audit-observabilitySkill
- audit-performance-frontendSkill
- audit-qualitySkill
- audit-reportSkill
- audit-securitySkill
- audit-testingSkill
- bug-ticket-root-causeSkill
- bug-triage-structuredSkill
- ci-pipeline-orchestrationSkill
- commit-readiness-reviewSkill
- comprehensive-test-strategySkill
- daily-workflow-optimizationSkill
- database-schema-evolutionSkill
- ddd-backend-implementationSkill
- design-system-component-lifecycleSkill
- detection-sweepSkill
- domain-glossarySkill
- domain-modeling-designSkill
- frontend-fullstack-implementationSkill
- frontend-spec-call-site-auditSkill
- merge-request-reviewSkill
- performance-profilingSkill
- qa-session-intakeSkill
- recording-decisionsSkill
- recurring-bug-root-causeSkill
- refactoring-shared-component-apiSkill
- validating-features-end-to-endSkill
- visual-regression-checkSkill
- writing-robust-testsSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 2 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 16
This skill: when: a production incident - prod down, API misbehaving (502/503/500/slow), abnormal behavior in prod, user report, silent …
- Very specific: line 17
ci-pipeline-orchestration: instead if: the CI pipeline is red but prod runs normally (build/release problem, not a live incident) - Very specific: line 18
recurring-bug-root-cause: instead if: a bug that recurs (2+ similar fixes) - class hunting, not a live incident - Very specific: line 19
detection-sweep: instead if: hunting static code problems across the whole project, outside an incident - Very specific: line 58
Look for: crash/panic/exception, OOM, "too many connections", exhausted descriptors, external dependency errors
- Distinctive: line 62
Should an alert have fired?
- Distinctive: line 65
If distributed tracing available: follow a faulty request end-to-end to locate the slow/failed hop
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Production incident triage - from symptom to faulty layer
If the current repo has a specific prod-diagnostic skill, it wins - it knows the hosting, the observability tool, the endpoints and the project tracker.
This skill vs. others
- This skill when: a production incident - prod down, API misbehaving (502/503/500/slow), abnormal behavior in prod, user report, silent alerting
ci-pipeline-orchestrationinstead if: the CI pipeline is red but prod runs normally (build/release problem, not a live incident)recurring-bug-root-causeinstead if: a bug that recurs (2+ similar fixes) - class hunting, not a live incidentdetection-sweepinstead if: hunting static code problems across the whole project, outside an incident
Quick rule: prod reachable but misbehaving → this skill; pipeline red, prod OK → ci-pipeline-orchestration.
Principle
Isolate the layer before fixing. An incident is traversed layer by layer - hosting/deployment → backend/API → alerting → frontend - until the faulty one is located. Evidence before conclusion: never "it's probably X" without a log, metric, trace or reproduction. The output = faulty layer + supported root cause + mitigation, not a blind restart.
Context to gather (before acting)
- Scope: is everything affected, or one endpoint / one feature / a subset of users?
- Since when: timestamp of the first symptom; does it correlate with a deployment, a migration, a traffic spike?
- Suspected layer from the symptom (see triage table)
- Access: health endpoint URLs, access to logs / metrics / traces (observability tool, dashboard), status of the last deployment, read-only DB access. If none of it is reachable (no dashboard, no shell, no endpoint), say so explicitly and run the diagnosis on the artefacts at hand - pasted logs, run history, the repository and its git history. A diagnosis without live access is a normal outcome, not a reason to stop; what it is not allowed to do is claim observations it could not make.
- Project conventions: deploy/rollback commands, tracker for the incident ticket
Protocol
1. Quick triage - categorize the symptom
- Reproduce the symptom (curl the endpoint, user journey) and note the exact code/behavior
- Hit the health endpoint (liveness/readiness) if exposed
- Without live access: reproduce locally if the failing thing can be run here; otherwise state that no reproduction was possible and delimit the scope from the artefacts (which runs failed, since when, what is still serving)
| Symptom | Likely layer |
| 502/503 | Hosting / process down / reverse proxy |
| Health 200 but 500 on endpoints | Partial backend (DB, external dependency) |
| Abnormal latency, timeouts | Saturation (DB pool, CPU, queue) |
| Blocked job/worker | Crashed worker or deadlock |
| Silent alerting | Channel config / rate limiting |
| Blank page, asset 404s | Corrupted frontend build/deploy, cache/CDN |
| Errors specific to one user | Auth / permissions / data |
2. Check deployment & version
- What version is actually running in prod? Compare to the expected commit (a "deploy success" serving the old image is a classic)
- Last deployment / migration: timing vs incident start → rollback candidate
3. Logs
- Filter the observability tool on the incident window and the suspected layer
- Look for: crash/panic/exception, OOM, "too many connections", exhausted descriptors, external dependency errors
4. Metrics & alerts
- Dashboard: CPU/memory, error rate, p95/p99 latency, connection pool saturation, queue depth
- Should an alert have fired? If not, an observability gap to track too
5. Traces
- If distributed tracing available: follow a faulty request end-to-end to locate the slow/failed hop
6. Layer bisection
- Confirm/rule out each layer with evidence (health responds → process up; direct DB query responds → DB up). Descend to the layer that faults.
7. Root cause + mitigation
- State the supported root cause (log/trace/repro). Decide mitigation: rollback (fast, reversible) vs forward-fix - when in doubt, ask the user
- Any destructive action in prod (UPDATE/DELETE, restart, purge): explicit user confirmation, read-only access by default
Templates
templates/incident-report.md- incident report (timeline, layer, evidence, root cause, mitigation, follow-up)
Traps & rationalizations
| Excuse | Reality |
| "I'll restart, it'll come back" | A restart without diagnosis erases the evidence and hides a cause that will return. Capture logs/metrics first. |
| "The symptom is gone, it's fixed" | Fixing a symptom without a root cause = an incident that replays. No closure without a supported cause. |
| "It's probably the DB / the network" | Unproven hypothesis. One piece of evidence per layer, otherwise you treat the wrong one. |
| "The deploy is green, prod is up to date" | Verify the version actually served - a green deploy can serve the old image. |
| "I'll write up the incident later" | "Later" never comes. Report + ticket before closure. |
Exit condition
- Symptom reproduced and scope delimited (no live access → reproduced locally, or the absence of a reproduction stated and the scope established from the artefacts)
- Faulty layer isolated with at least one piece of evidence (log, metric, trace or repro)
- Root cause documented - never an unproven hypothesis
- Mitigation applied or planned; destructive actions confirmed by the user
- Prod re-checked: health endpoint OK and symptom gone (output pasted) - no live access → say so, and name the check whoever has access must run before the incident is closed
- Incident report written + ticket created if prod was impacted
Tooling
templates/incident-report.md- report template to fill in- Generic tools:
curl -v/-Ion health endpoints, query the observability dashboard over the window, read-only DB query, trace following
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

