chaos-engineering

chaos-engineering

Break things on purpose, in a controlled way, to find weaknesses before they find you.

1. The experiment structure — never skip this

A chaos run without a hypothesis is just vandalism. Every experiment has five parts:

1. Steady state   — what does "healthy" look like right now? (metric + threshold)
2. Hypothesis     — "if we inject X, the system will continue to Y"
3. Blast radius   — scope the experiment so failures stay contained
4. Inject fault   — introduce the failure
5. Observe        — did the system behave as hypothesised? if not, why?

Write it down before touching anything:

## Experiment: DB primary failover

**Steady state:** order API p99 < 300 ms, error rate < 0.1 % over the last 5 min

**Hypothesis:** if the primary database node fails, the app reconnects to the
replica within 10 s and resumes serving requests with < 1 % errors

**Blast radius:** staging environment only; 10 % of synthetic traffic

**Fault:** kill the primary PostgreSQL container

**Abort condition:** error rate > 5 % for > 60 s → stop the experiment immediately

2. Blast radius control — non-negotiable

Never run a chaos experiment without limiting the blast radius. Start as small as possible and expand only when confidence is high.

Abort conditions must be defined before the experiment starts, not decided in the moment. If the abort metric is breached, stop the fault injection immediately and let the system recover before investigating.

3. Tools

4. Fault recipes

Latency injection — Toxiproxy (local / CI)

# Install
brew install toxiproxy   # or download binary from releases

# Start proxy
toxiproxy-server &

# Create a proxy in front of your PostgreSQL
toxiproxy-cli create postgres_proxy --listen 0.0.0.0:5433 --upstream localhost:5432

# Add 300 ms latency with 50 ms jitter (simulates a flaky network)
toxiproxy-cli toxic add postgres_proxy --type latency --attribute latency=300 --attribute jitter=50

# Point your app at port 5433 instead of 5432
DATABASE_URL=postgresql://user:pass@localhost:5433/mydb

# Observe — does your connection pool handle it? do queries time out gracefully?

# Remove the toxic
toxiproxy-cli toxic remove postgres_proxy --toxicName latency_downstream

# Nuclear option — cut the connection entirely (timeout simulation)
toxiproxy-cli toxic add postgres_proxy --type timeout --attribute timeout=0

Network latency — Linux tc/netem (no extra install)

# Add 200 ms delay to all outbound traffic on eth0
tc qdisc add dev eth0 root netem delay 200ms

# Add 200 ms delay with 20 ms jitter and 10 % packet loss
tc qdisc add dev eth0 root netem delay 200ms 20ms loss 10%

# Remove
tc qdisc del dev eth0 root

# Scope to a specific destination IP only (e.g., your DB server)
tc qdisc add dev eth0 root handle 1: prio
tc qdisc add dev eth0 parent 1:3 handle 30: netem delay 200ms
tc filter add dev eth0 protocol ip parent 1:0 prio 3 u32 \
  match ip dst <DB_IP>/32 flowid 1:3

Pod kill — Kubernetes / Litmus

# litmus-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: order-service-pod-delete
  namespace: staging
spec:
  appinfo:
    appns: staging
    applabel: "app=order-service"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"          # inject for 60 s
            - name: CHAOS_INTERVAL
              value: "10"          # kill a pod every 10 s
            - name: FORCE
              value: "false"       # graceful kill first

kubectl apply -f litmus-pod-delete.yaml
# Watch your monitoring dashboard — does the deployment recover?
# Does the load balancer stop routing to the killed pod fast enough?
kubectl delete -f litmus-pod-delete.yaml   # stop the experiment

CPU saturation — stress-ng

# Saturate 2 CPU cores for 60 seconds
stress-ng --cpu 2 --timeout 60s --metrics-brief

# Does your service degrade gracefully? Does it shed load or just get slower?
# Watch: request queue depth, p99 latency, error rate

Disk fill — simulate storage pressure

# Fill disk to 90 % capacity temporarily
fallocate -l $(df / | awk 'NR==2 {printf "%.0f", $4 * 0.8}')K /tmp/chaos_fill

# Observe: does the app handle ENOSPC errors? do logs rotate? does the DB crash?

# Clean up
rm /tmp/chaos_fill

Dependency kill — simulate a downstream service going down

# Block all traffic to an external dependency IP/port with iptables
iptables -A OUTPUT -p tcp --dport 443 -d api.stripe.com -j DROP

# Does your circuit breaker trip? Does the error surface cleanly to the user?
# Does the retry logic back off or hammer the blocked endpoint?

# Remove the rule
iptables -D OUTPUT -p tcp --dport 443 -d api.stripe.com -j DROP

5. What to observe during an experiment

Run the experiment and watch these simultaneously:

✓ Error rate          — is it within the abort threshold?
✓ p99 latency         — is it degrading or holding?
✓ Retry counters      — are retries firing? are they backing off?
✓ Circuit breaker     — did it open? did it half-open and recover?
✓ Queue depth         — is work piling up?
✓ Downstream errors   — is the fault spreading to other services?
✓ Recovery time       — how long from fault injection to steady state restored?

Record the actual numbers, not just "it worked" or "it failed."

6. GameDay — structured team exercise

A GameDay is a scheduled, team-wide chaos experiment run like a fire drill.

## GameDay runbook template

**Date:** YYYY-MM-DD  **Duration:** 2 hours  **Environment:** staging

### Scenario
Primary database AZ goes down. Replica promoted. S3 returns 503 for 5 min.

### Roles
- Experiment lead:   runs the fault injection, watches abort conditions
- Observability:     monitors dashboards, calls abort if threshold breached
- On-call engineer:  responds as if this were a real incident
- Scribe:            records timeline, findings, and action items

### Timeline
| Time  | Action |
|-------|--------|
| T+0   | Baseline captured, steady state confirmed |
| T+5   | Kill primary DB container |
| T+15  | Restore primary DB |
| T+20  | Inject S3 503 responses via Toxiproxy |
| T+30  | Remove S3 fault |
| T+35  | Debrief begins |

### Findings → action items
| Finding | Severity | Owner | Due |
|---------|----------|-------|-----|
| Replica failover took 45 s, SLA is 10 s | High | DB team | 2 weeks |
| S3 errors surfaced as 500 to users, not 503 | Medium | API team | 1 month |

7. Turn findings into hardening

Every experiment that reveals a gap produces a ticket:

Rules

• Define the hypothesis and abort conditions before starting. Never improvise mid-experiment.
• Start in staging. Graduate to production only after multiple successful staging runs.
• One fault at a time. Injecting multiple faults simultaneously makes root cause impossible to determine.
• Always have a kill switch ready — a single command that removes all fault injection immediately. Test the kill switch before running the experiment.
• Never experiment on payment flows, PII pipelines, or data that cannot be rolled back without explicit sign-off from the team.
• Document every run: hypothesis, what actually happened, recovery time, action items. An undocumented chaos run teaches the team nothing.