agentleFS
lalulalilalulali/vespyr/.agents/skills/incident/SKILL.md

incident

Skill11 starsChanged 11 days agoMITView on GitHub

Production incident management — triage, mitigate, root-cause analysis, and post-incident review. Use when the user says "the site is down", "error rates are spiking", "we have a production incident", "do a postmortem", or invokes /incident.

23 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use db-review, runbook and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • tranhieuttSkill

    db-review

    Also in 3 projects that use this

  • AnthropicSkill

    runbook

    Create or update an operational runbook for a recurring task or procedure. Use when documenting a task that on-call or ops needs to run repeatably, turning tribal knowledge into exact step-by-step commands, adding troubleshooting and rollback steps to an existing procedure, or writing escalation paths for when things go wrong.

    Also in 4 projects that use this

  • MicrosoftSkill

    perf

    Speed and memory performance rules for Rust crates, the Node addon, webui-framework, and webui-delivery.

    Also in 4 projects that use this

  • drewdrewthisSkill

    sweep

    Search for similar patterns, anti-patterns, or issues elsewhere in the codebase after a fix or feature. Use after implementing a fix to catch the same bug in other places, or proactively to audit for a known pattern.

    Also in 3 projects that use this

  • OpenAISkill

    observability

    Vercel Observability expert guidance — Drains (logs, traces, speed insights, web analytics), Web Analytics, Speed Insights, runtime logs, custom events, OpenTelemetry integration, and monitoring dashboards. Use when instrumenting, debugging, or optimizing application performance and user experience on Vercel.

    Also in 5 projects that use this

  • mijunriSkill

    database

    数据库脚本与操作规范。脚本放在 db_scripts 目录,遵循查询优化、事务、安全等实践。

    Also in 3 projects that use this

The file

Show its rules and text

We split this skill into the rules it gives an agent

13 rules. 0 are common, 10 are distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 21 edited versions we read keep these.

  • 10

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    When something breaks in production, this skill provides the structured process to triage, mitigate, analyze, and prevent recurrence.

    line 13
  2. Distinctive:

    Stop the bleeding: rollback, feature flag off, scale up, redirect traffic

    line 42
  3. Distinctive:

    Document what was done and timeline

    line 45
  4. Distinctive:

    Add regression tests that would have caught this

    line 69
  5. Distinctive:

    Update runbooks with the new failure mode

    line 81
  6. Distinctive:

    Close the loop: Every incident must produce at least one prevention action item.

    line 152
  7. Distinctive:

    No silent fixes: Every remediation goes through review and QA, even in an incident.

    line 153
  8. Very specific:

    After incident is resolved → load iterate to address follow-up improvements

    line 165
  9. Very specific:

    If incident reveals architectural issues → load develop for broader redesign

    line 166
  10. Very specific:

    If incident response process needs review → load retro

    line 167
  11. Distinctive:

    Run this for every incident directory, before it is treated as closed:

    line 178
  12. Distinctive:

    If pipeline is uninitialized (e.g., the project has no pipeline state yet but has a live incident), initialize with the current project …

    line 212
  13. Distinctive:

    File a Change Request for cross-team fixes: If remediation requires a change in another team's domain (e.g., a UX change requested by the …

    line 228

“Common” is measured: kept by at least 6 in 10 of the 21 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Incident — Production Response Workflow

What this skill does

When something breaks in production, this skill provides the structured process to triage, mitigate, analyze, and prevent recurrence. It is not for feature development — it is for when the system is down, errors are spiking, or users are reporting critical problems.

This skill is reactive. It is invoked when a production issue is detected, not during normal development.

Workflow steps

Step 1: Triage

Invoke @product-manager to lead triage:

  • Severity classification:
    • SEV1 (Critical): Service down, data loss, security breach → all hands, immediate response
    • SEV2 (High): Major feature broken, significant user impact → urgent response, 4-hour target
    • SEV3 (Medium): Feature degraded, workaround exists → next business day
    • SEV4 (Low): Minor issue, cosmetic, low impact → backlog
  • Assign incident commander (default: @product-manager)
  • Assemble responders based on issue type:
    • Infrastructure issue → @devops-engineer + @architect
    • Application bug → @developer + @tech-lead
    • Security incident → @security-engineer + @devops-engineer
    • Performance degradation → @performance-engineer + @architect
  • Open incident channel — create artifacts/output/08-incidents/INC-NNN/ directory

Output: artifacts/output/08-incidents/INC-NNN/triage.md (Use template: .agents/templates/incident/incident-triage-template.md)

Step 2: Mitigate

The first priority is user impact reduction, not root cause.

Invoke appropriate responders to:

  • Stop the bleeding: rollback, feature flag off, scale up, redirect traffic
  • Communicate status to stakeholders (internal and external if applicable)
  • Preserve evidence: logs, traces, screenshots, user reports
  • Document what was done and timeline

Key rule: Mitigate first, analyze second. Never debug in production when a rollback fixes the user-facing issue.

Output: artifacts/output/08-incidents/INC-NNN/mitigation.md

Step 3: Root-Cause Analysis

Once users are unblocked, invoke @architect or @tech-lead to lead RCA:

  • 5 Whys analysis — drill to root cause
  • Timeline reconstruction — from first signal to mitigation
  • Contributing factors — what conditions allowed this to happen
  • Detection gaps — why wasn't this caught before reaching users
  • Prevention mechanisms — what would stop this class of issue in the future

Output: artifacts/output/08-incidents/INC-NNN/rca.md (Use template: .agents/templates/incident/rca-template.md)

Step 4: Remediation (parallelizable)

Invoke appropriate agents for the fix. Code fix and monitoring improvements can run in parallel.

Step 4a: Code Fix ⟨parallel⟩

Invoke @developer (or @devops-engineer for infra issues) to:

  • Fix the root cause
  • Add regression tests that would have caught this

Then (sequential within this path):

  • @code-reviewer reviews the fix
  • @qa-engineer validates the fix and regression tests
  • @devops-engineer deploys the fix

Loop limit: Max 2 review cycles on the incident fix. If the fix is still contested, the incident commander (@product-manager) makes the ship/rollback call.

Step 4b: Monitoring Improvements ⟨parallel⟩

Invoke @devops-engineer to:

  • Improve monitoring/alerting to detect similar issues earlier
  • Update runbooks with the new failure mode

Output: artifacts/output/08-incidents/INC-NNN/remediation.md

Step 5: Post-Incident Review

Invoke @product-manager to conduct a blameless post-incident review:

  • What happened — timeline of events
  • What was the impact — users affected, duration, data impact
  • What was the root cause — from RCA
  • What was the mitigation — what stopped the bleeding
  • What was the remediation — what prevents recurrence
  • What went well — things that worked (monitoring, response time, etc.)
  • What could be improved — detection, response, communication, prevention
  • Action items with owners and deadlines

Output: artifacts/output/08-incidents/INC-NNN/post-incident-review.md (Use template: .agents/templates/incident/post-incident-review-template.md)

Step 6: Update Knowledge Base

Invoke @product-manager to write incident learnings to memory via @memory-controller:

@memory-controller write blockers-and-risks.md
### [RISK] {incident title} [date: YYYY-MM-DD] [agent: @product-manager] [RESOLVED: YYYY-MM-DD]
{new risks identified from this incident}
**Status:** resolved

@memory-controller write lessons-learned.md
### [LESSON] {title} [date: YYYY-MM-DD] [agent: @product-manager]
{incident lessons — what happened, root cause, prevention}
**Status:** active

@memory-controller write patterns-and-conventions.md
### [PROCESS] {title} [date: YYYY-MM-DD] [agent: @product-manager]
{new guardrail or operational pattern established}
**Status:** active

Write session summary:

@memory-controller session-write
Worked on: Incident response — INC-NNN ({brief description})
Decisions made:
- Severity: {SEV level}
- Mitigation: {what was done to stop user impact}
- Root cause: {one sentence}
- Prevention: {key action item}
Next step: {follow-up improvements via iterate, or "monitoring for recurrence"}
New blockers: {any new risks identified, or "none"}

Output artifacts

  • artifacts/output/08-incidents/INC-NNN/triage.md
  • artifacts/output/08-incidents/INC-NNN/mitigation.md
  • artifacts/output/08-incidents/INC-NNN/rca.md
  • artifacts/output/08-incidents/INC-NNN/remediation.md
  • artifacts/output/08-incidents/INC-NNN/post-incident-review.md

When to use

Use this when:

  1. A production service is down or critically degraded
  2. Error rates spike unexpectedly
  3. Security vulnerability is discovered in production
  4. Data integrity issue is detected
  5. Performance degrades below SLA thresholds

Key principles

  • Mitigate first, analyze second. Reduce user impact before finding root cause.
  • Blameless. Focus on systems and processes, not people.
  • Document everything. Future incident responders need the timeline.
  • Close the loop. Every incident must produce at least one prevention action item.
  • No silent fixes. Every remediation goes through review and QA, even in an incident.

Severity-specific SLAs

SeverityResponse TimeMitigation TargetUpdate Cadence
SEV115 minutes1 hourEvery 30 minutes
SEV230 minutes4 hoursEvery hour
SEV34 hoursNext business dayDaily
SEV4Next business dayNext sprintWeekly

Handoff

  • After incident is resolved → load iterate to address follow-up improvements
  • If incident reveals architectural issues → load develop for broader redesign
  • If incident response process needs review → load retro

Done when — checked mechanically

/incident shipped for its whole life with no gate: an incident directory could hold a triage record and nothing else, and nothing noticed. The skill's own principle — "close the loop, every incident must produce at least one prevention action item" — was a sentence nobody could falsify, and the SEV1 SLA promised a 15-minute response with nothing that reads a clock.

Run this for every incident directory, before it is treated as closed:

node .agents/scripts/incident_record_check.js check \
  --dir artifacts/output/08-incidents/INC-NNN --json

It requires a recorded SEV1-SEV4 severity, all five named artifacts present and substantial (a stub satisfies "the file exists" while recording nothing), a post-incident review — mandatory at SEV1 and SEV2 — at least one prevention action item carrying both an owner and a date, and blameless phrasing.

What it does not check: whether mitigation met the SLA, whether the RCA is correct, or whether the severity was honestly classified. Those need someone who was there. A green result means the record is complete and the loop was closed — not that the response was good.

Exit 0 conformant · 1 failures · 2 usage/IO error, fail-closed.

State Machine Integration

The pipeline state machine (node .agents/scripts/orchestrator_state.js) is the canonical record of project state. This skill must wire its work into it so other skills, the dashboard, and the code-graph see what happened.

At Start

Run:

node .agents/scripts/orchestrator_state.js status

Load incident context before triage:

@memory-controller load product-manager [incident — {brief description of symptoms}]

If pipeline is uninitialized (e.g., the project has no pipeline state yet but has a live incident), initialize with the current project name and type before recording. If the pipeline is mid-workflow, do NOT advance the phase — incident work happens in parallel.

At End — Record Completion

For each incident directory artifacts/output/08-incidents/INC-NNN/, record the artifacts produced:

node .agents/scripts/orchestrator_state.js complete --agent devops-engineer --artifact 08-incidents/INC-NNN/triage.md
node .agents/scripts/orchestrator_state.js complete --agent devops-engineer --artifact 08-incidents/INC-NNN/mitigation.md
node .agents/scripts/orchestrator_state.js complete --agent architect --artifact 08-incidents/INC-NNN/rca.md
node .agents/scripts/orchestrator_state.js complete --agent developer --artifact 08-incidents/INC-NNN/remediation.md
node .agents/scripts/orchestrator_state.js complete --agent devops-engineer --artifact 08-incidents/INC-NNN/post-incident-review.md

Replace INC-NNN with the actual incident ID. The developer complete call will trigger a code-graph refresh, ensuring the next agent (e.g., a follow-up architect reviewing the architectural impact) sees the new code state.

File a Change Request for cross-team fixes. If remediation requires a change in another team's domain (e.g., a UX change requested by the architect but owned by the product manager), file a CR:

node .agents/scripts/orchestrator_state.js file-cr --from developer --to product-manager --target <file> --issue "<issue>"

This creates an open CR in pipeline-state.json that blocks phase advancement until resolved.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.