Chaos Engineering Skill

Setup Check

Before loading context files, check if context/CONTEXT.md exists in the current directory.

If context/CONTEXT.md exists — read it and proceed normally.

If context/CONTEXT.md does not exist — this skill was installed standalone (e.g. via npx skills add). Ask the user these questions before proceeding:

1. Role — junior-sre / senior-sre / sre-manager (shapes output depth and tone)
2. Cloud provider — aws / gcp / azure / on-prem / hybrid
3. Observability stack — e.g. Datadog, Prometheus+Grafana, New Relic
4. Company name and primary services affected (if relevant to this task)

Use the answers inline for this session. For persistent setup across all skills, suggest:

pipx install sre-agent
sre-agent init

Instructions

Step 0: Safety Gate (MANDATORY)

Before any other step, verify:

1. Ask: "Is there currently an active P0 or P1 incident?" If yes: stop immediately. "Chaos experiments must not run during active incidents. Resolve the incident first."
2. Ask: "Has this experiment been approved by the on-call engineer and engineering manager?"
3. Confirm: "What is the rollback/abort procedure if this experiment goes wrong?"

Only proceed if all three are satisfied.

Step 1: Load Context

Read context/CONTEXT.md and context/company/tech-stack.md.

Step 2: Blast Radius Assessment

Before designing the experiment, assess the maximum possible impact:

Scope questions:

• Which services are in scope for the experiment?
• Which services have a dependency on in-scope services (blast radius)?
• What percentage of users could be affected if the experiment escapes its scope?
• Is there a way to limit the experiment to a subset of users or a single region?
• What is the recovery time if something goes wrong (RTO)?

Blast radius classification:

Rule: Only run Class "Contained" or "Service" experiments in production without explicit VP Engineering approval.

Step 3: Define the Hypothesis

Every experiment must have a clear, falsifiable hypothesis:

Hypothesis template:
"We believe that [service] will [behavior] when [failure condition],
because [reason], and we will verify this by [measurement]."

Example:
"We believe that the checkout-api will continue processing orders
within its normal SLO when the inventory-service is unavailable,
because checkout-api has a circuit breaker configured with a 30-second
timeout and a cached fallback, and we will verify this by measuring
the checkout error rate and latency during the experiment."

Step 4: Define Steady State

Establish measurable baseline before the experiment:

Steady state metrics (measure 10 minutes before starting):
- checkout error rate: <0.1%
- checkout P99 latency: <500ms
- order creation success rate: >99.9%

Step 5: Design the Experiment

Choose the failure injection type:

Design output:

Experiment: checkout-api circuit breaker validation

Failure injection:
  Tool: Chaos Mesh
  Target: inventory-service, all pods, production namespace
  Failure type: network partition (100% packet drop)
  Duration: 5 minutes

Monitoring:
  - Watch: checkout error rate (expect: <1%)
  - Watch: checkout P99 latency (expect: <1s with cache fallback)
  - Watch: circuit breaker open state in Datadog

Abort conditions:
  - checkout error rate > 5% → abort immediately
  - any other service shows unexpected degradation → abort
  - experiment duration > 10 minutes without resolution → abort

Rollback:
  - Remove Chaos Mesh network policy
  - Verify inventory-service connectivity restored
  - Confirm checkout metrics return to steady state

Step 6: Generate the Gameday Runbook

Produce a gameday document with:

• Experiment hypothesis
• Steady state definition
• Timeline (when each step happens, who does what)
• Monitoring checklist
• Abort conditions and rollback procedure
• Success/failure criteria
• Observation notes section

Step 7: Post-Experiment Analysis

After the experiment:

1. Did steady state hold? (Hypothesis confirmed or refuted?)
2. What surprises occurred?
3. What did the experiment reveal about the system?
4. Generate action items for gaps discovered.

Guidelines

• Never run chaos experiments during business peak hours without a compelling reason.
• Every experiment needs an abort condition — define it before starting.
• Start small: a single pod kill is more valuable than a full region failure as a first experiment.
• "It worked" is not a result — document what metrics confirmed it worked.
• Persona (junior-sre): read-only participation only; do not design or run experiments without senior supervision.