sre-remediation-actions
Skill4 starsChanged 25 days agoMITView on GitHub
Reference of gated recovery procedures (rollback, scale, restart, failover) with risk/reversibility labels, evidence capture and dry-run first. Loaded by sre-engineer before any remediation.
In this repository84 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- .cursor/rules/sre-safety-gates.mdcCursor rule
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dr-backup-testSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- sre-incident-communicationSkill
- sre-k8s-safe-remediationSkill
- sre-k8s-triageSkill
- sre-k8s-upgrade-autopilotSkill
- sre-message-queue-triageSkill
- sre-migration-safetySkill
- sre-observability-cost-cutterSkill
- sre-observability-query-cookbookSkill
- sre-oncall-auditSkill
- sre-oncall-handoffSkill
- sre-pagerduty-opsgenieSkill
- sre-php2shell-checkSkill
- sre-postmortem-templatesSkill
- sre-production-readinessSkill
- sre-prometheus-grafanaSkill
- sre-runbook-templatesSkill
- sre-safety-gatesSkill
- sre-security-incident-triageSkill
- sre-service-mesh-triageSkill
- sre-slo-alertingSkill
- sre-slo-burn-rate-rulesSkill
- sre-terraformSkill
- sre-toil-reviewSkill
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dr-backup-testSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- The first 60 shown.
The file
Show its rules and text
We split this skill into the rules it gives an agent
15 rules. About 0 are common, 10 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈10
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 9
Every kubectl/helm command below carries --context <ctx> -n <ns> (--kube-context <ctx> for helm) literally.
- Distinctive: line 15
Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Very specific: line 130
Verify after rollout status succeeds: kubectl --context <ctx> -n <ns> get <kind>/<name> shows ready = desired, kubectl --context <ctx> -n …
- Distinctive: line 231
Check recent deployments -> propose rollback if correlated
- Distinctive: line 232
Verify dependencies (DB, cache, upstream services)
- Distinctive: line 233
Check for config changes
- Distinctive: line 240
Check for resource saturation (CPU, memory, I/O)
- Distinctive: line 242
Check for cache misses
- Very specific: line 284
If idle in transaction sessions hold the slots, fix the app first, then clear them.
- Very specific: line 327
Check SHOW POOLS; until the database shows no active server connections, do the work, then resume:
- Distinctive: line 342
Only if those fail, restart, one pooler instance at a time when there are several, waiting for each to pass verification before the next.
- Very specific: line 364
Verify: the app error rate falls back toward baseline, SHOW POOLS.
- Distinctive: line 374
Document actions in incident log
- Distinctive: line 389
Verify latency within SLO
- Distinctive: line 390
Check synthetic monitors passing
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
SRE Remediation Actions
Every kubectl/helm command below carries --context <ctx> -n <ns> (--kube-context <ctx> for helm) literally; never rely on the current context. Never delete --force --grace-period=0, delete --all, or patch away finalizers without SRE approval.
Safety gates (REQUIRED)
- Read-only diagnosis first.
- Propose each change with risk (low/med/high) and reversibility.
- Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Act only as approved, verify, and stop and escalate if anything gets worse.
Full rules: sre-safety-gates. Read-only MCP setup: sre-safety-gates/references/mcp.md.
Legend: risk = low / med / high blast radius. reversible = yes / no (can be cleanly undone without data loss).
For Kubernetes writes, tell the user to run /sre-k8s-safe-remediation (Claude plugin install: /sre:sre-k8s-safe-remediation; Codex: $sre-k8s-safe-remediation) (manual-only gated workflow).
Remediation hierarchy
Priority order
- Capture evidence (logs, heap dumps, rollout history) — always first, read-only
- Rollback recent changes (if correlated with onset)
- Scale resources
- Disable non-critical features / degraded mode
- Restart (only after evidence captured; restarts destroy state)
- Failover to backup (high risk — last resort)
Evidence capture
read-only; do BEFORE any restart, delete, or rollback
Kubernetes
kubectl --context <ctx> rollout history deployment/<name> -n <ns>
kubectl --context <ctx> describe deployment/<name> -n <ns>
kubectl --context <ctx> events -n <ns> --types=Warning | tail -n 50
kubectl --context <ctx> logs <pod> -n <ns> --all-containers --timestamps > <pod>.log # may contain secrets/PII: keep out of repos, redact
kubectl --context <ctx> logs <pod> -n <ns> --all-containers --previous > <pod>.prev.log # errors if no previous instance; harmlessJVM or runtime
- Heap dump: jcmd
<pid>GC.heap_dump /tmp/<svc>-heap.hprof (pauses process; med risk — needs approval in prod) Thread dump:
jcmd <pid> Thread.print
Systemd
journalctl -u <service> --since '-1h' > <service>.logGeneral
- Snapshot dashboards / screenshot graphs with timestamps
- Record current config and image versions
Dry run first
kubectl --context <ctx> diff -f <file>; kubectl --context <ctx> apply --dry-run=server -f <file> # diff exits 1 when there are differences, so do not chain with &&; server dry-run still calls admission webhooks
terraform plan -input=false -out=<plan-file> && terraform show -no-color <plan-file> # an unsaved plan is speculative
helm diff upgrade --kube-context <ctx> <release> <chart> -n <ns> --suppress-secrets # helm-diff plugin
helm upgrade --kube-context <ctx> <release> <chart> -n <ns> --dry-run=server --hide-secret (Helm >= 3.13; without --hide-secret it prints rendered Secrets)
# helm diff also shows Secret changes unless the plugin is configured to suppress them (HELM_DIFF / --suppress-secrets)
helm history --kube-context <ctx> <release> -n <ns>Common actions
Restart
Kubernetes
Before proposing any restart, answer "Is it safe to restart or scale it?" in sre-k8s-triage (workload kind, ready vs desired, PDB, HPA, probes, strategy, blast radius) and obey its stop rule. A restart can't be undone: the old pods and their in-memory state are gone. Capture evidence first. Use the command for the workload's kind, and run the verification right after it.
| Kind | Risk | Reversible | How it restarts |
| Deployment | med | no | Rolling, paced by maxUnavailable/maxSurge. Recreate strategy takes every pod down at once (high) |
| StatefulSet | high | no | One pod at a time, highest ordinal first; partition > 0 restarts only ordinals >= partition; OnDelete does nothing until pods are deleted. Never on a primary/leader without the owner |
| DaemonSet | med-high | no | One node at a time per maxUnavailable; ingress controllers, CNI and log agents hit every node |
| Argo Rollout | med | no | Needs the Argo Rollouts kubectl plugin; kubectl rollout restart doesn't apply to a Rollout |
Deployment:
Gate: approval required.
kubectl --context <ctx> -n <ns> rollout restart deployment/<name>
kubectl --context <ctx> -n <ns> rollout status deployment/<name> --timeout=300sStatefulSet:
Gate: approval required.
kubectl --context <ctx> -n <ns> rollout restart statefulset/<name>
kubectl --context <ctx> -n <ns> rollout status statefulset/<name> --timeout=600sDaemonSet:
Gate: approval required.
kubectl --context <ctx> -n <ns> rollout restart daemonset/<name>
kubectl --context <ctx> -n <ns> rollout status daemonset/<name> --timeout=600sArgo Rollout:
Gate: approval required.
kubectl argo rollouts restart <name> --context <ctx> -n <ns>
kubectl argo rollouts status <name> --context <ctx> -n <ns> --timeout 300sVerify after rollout status succeeds: kubectl --context <ctx> -n <ns> get <kind>/<name> shows ready = desired, kubectl --context <ctx> -n <ns> events --for <kind>/<name> | tail -n 50 has no new warnings, and error rate and latency are back at baseline. If rollout status times out or pods don't become Ready, stop: don't re-run the restart. Report and escalate.
| Action | Risk | Reversible | Note |
kubectl --context <ctx> delete pod <pod-name> -n <ns> | med | no | Capture logs/--previous first; single pod only, never by label selector without approval |
Systemd
| Action | Risk | Reversible | Note |
systemctl reload <service> | low | yes | Config reload; prefer over restart |
systemctl restart <service> | med | no | Causes brief downtime on single hosts |
Docker
| Action | Risk | Reversible |
docker restart <container-id> | med | no |
docker compose restart <service> | med | no |
Rollback
Kubernetes
| Action | Risk | Reversible | Note |
kubectl --context <ctx> rollout history deployment/<name> -n <ns> | low | yes | Read-only; confirm target revision first |
kubectl --context <ctx> rollout undo deployment/<name> -n <ns> | med | yes | Check history; schema/data migrations may make this unsafe; uses ReplicaSet history (revisionHistoryLimit); GitOps (Argo/Flux) reverts it unless sync is paused or Git is reverted |
helm rollback --kube-context <ctx> <release> <revision> -n <ns> | med | yes | Check helm history first; GitOps caveat applies |
kubectl --context <ctx> rollout undo deployment/<name> -n <ns> --to-revision=<n> | med | yes |
Cloud
| Action | Risk | Reversible |
| Cloud console/CLI: redeploy previous version | med | yes |
Terraform
| Action | Risk | Reversible | Note |
git revert <commit> -> terraform plan -input=false -out=<plan-file> -> terraform show -no-color <plan-file> -> explicit approval -> terraform apply <plan-file> (plain apply re-plans instead of applying the reviewed plan; see sre-terraform) | high | partial | There is no 'terraform rollback'. Destroy/replace actions in the plan may be irreversible. |
Database
| Action | Risk | Reversible | Note |
| PITR (Point-in-Time Recovery) to new instance, then cut over | high | no | Data written after the restore point is lost; restore to a NEW instance first |
| Flashback to restore point | high | no | Oracle-only (Oracle Flashback Database/Table) |
Scale
Kubernetes
| Action | Risk | Reversible | Note |
kubectl --context <ctx> scale deployment/<name> --replicas=<n> -n <ns> (scale up) | low | yes | Note current replica count first; HPA overrides; GitOps reverts |
kubectl --context <ctx> scale deployment/<name> --replicas=<n> -n <ns> (scale down) | med | yes | Can overload remaining pods; check load and PDBs first |
kubectl --context <ctx> autoscale deployment/<name> --min=<min> --max=<max> -n <ns> | low | yes | Creates an HPA; conflicts with an existing HPA or GitOps. Revert: kubectl --context <ctx> delete hpa <name> -n <ns> |
Cloud
| Action | Risk | Reversible |
| AWS: Modify auto-scaling group desired/max (up / down) | low / med | yes |
| GCP: Update instance group size (up / down) | low / med | yes |
| Azure: Scale VM scale set (up / down) | low / med | yes |
Failover
Database
For managed databases (RDS/Aurora, Cloud SQL, Azure, ElastiCache/MemoryDB), use the per-platform gated failover blocks in sre-database-triage (each reference's "Managed services" section).
| Action | Risk | Reversible | Note |
| Promote read replica to primary | high | no | Split-brain/data-loss risk; check replication lag first |
| Update connection strings | med | yes | |
| DNS failover | high | yes | TTL delays both failover and failback |
Load balancer
| Action | Risk | Reversible |
| Drain unhealthy backend | low | yes |
| Enable backup region | high | yes |
| Update health check thresholds | med | yes |
Feature flags
| Action | Risk | Reversible |
| Disable non-critical feature flag | low | yes |
Incident specific actions
each step still passes the safety gates
High error rate
- Capture evidence (logs, rollout history)
- Check recent deployments -> propose rollback if correlated
- Verify dependencies (DB, cache, upstream services)
- Check for config changes
- Scale up if underprovisioned
- Enable degraded mode if needed
- Restart affected services only after evidence captured
High latency
- Check for resource saturation (CPU, memory, I/O)
- Review database queries (slow query log)
- Check for cache misses
- Verify network issues
- Propose scale-up or bottleneck fix
Memory exhaustion
- Capture heap dump BEFORE restart (approval needed in prod)
- Propose restart of affected instances
- Increase memory limits
- Add more instances
- Implement memory limits (cgroups, limits)
Database issues
Follow sre-database-triage (read-only triage, then its Fix order). Capture evidence first with bash ../sre-database-triage/scripts/db-snapshot.sh <engine> <outdir>. Diagnose read-only: connection pool exhaustion, slow queries (plain EXPLAIN; EXPLAIN ANALYZE executes the statement), and locks or blocking queries. Capture the pooler's own stats (PgBouncer SHOW POOLS; and SHOW STATS; as a stats_users member) before acting. Then work down this table, most preferred first:
| Action | Risk | Reversible | Note |
| App rollback, or lower the app's pool size | med | yes | First choice when a client is the cause |
Cancel or terminate one session (pg_cancel_backend, KILL QUERY) | med/high | no | Gated blocks in sre-database-triage; cancel before terminate, except idle-in-transaction sessions (see below) |
Reload pooler config (PgBouncer RELOAD) | low | yes | Prefer this to a restart |
| Restart connection pooler | high | no (drops every client; in-flight transactions roll back; reconnect storm) | Only after the app-side fix and evidence, with IC approval; see "Restart a connection pooler" below |
| Scale database resources | med | usually | Managed services may restart or fail over to resize |
| Managed failover to a replica | high | no | Last resort; check replica lag; IC and database owner approve |
Restart a connection pooler
Risk: high. Reversible: no (it drops every client connection through that pooler; in-flight transactions roll back, and every client reconnects at once). Do it after a rollback of the change that caused the problem, not before.
Evidence first (read-only). PgBouncer admin console, with the password in PGPASSFILE. The SHOW commands need only a user listed in stats_users; admin_users is needed only for RELOAD, PAUSE, RESUME and RECONNECT below.
psql -h <pooler-host> -p 6432 -U <pgbouncer-stats-user> pgbouncer -c 'SHOW POOLS;'
psql -h <pooler-host> -p 6432 -U <pgbouncer-stats-user> pgbouncer -c 'SHOW STATS;'
psql -h <db-host> -U <readonly-user> -d <db> -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state ORDER BY 2 DESC;"
psql -h <db-host> -U <readonly-user> -d <db> -c "SHOW max_connections;"Watch cl_waiting (clients queued for a server connection) and maxwait (seconds the oldest has waited). Record the output in the incident timeline.
Try these first, in order:
If
idle in transactionsessions hold the slots, fix the app first, then clear them.pg_cancel_backenddoes nothing to these sessions (they have no running query), so the choices are terminating them or setting a timeout. Preview who would be hit:SELECT pid, usename, application_name, now() - state_change AS idle_for FROM pg_stat_activity WHERE state = 'idle in transaction' AND usename = '<app_role>' AND application_name = '<app>' ORDER BY idle_for DESC LIMIT 20;Terminate them (waits up to 5 seconds per backend; on PostgreSQL 13 use the one-argument
pg_terminate_backend(pid)):Gate: approval required.
SELECT pg_terminate_backend(pid, 5000) FROM pg_stat_activity WHERE state = 'idle in transaction' AND usename = '<app_role>' AND application_name = '<app>' AND now() - state_change > interval '5 minutes';Or, as a separate config change for that role (applies to new sessions only):
Gate: approval required.
ALTER ROLE <app_role> SET idle_in_transaction_session_timeout = '60s';If the pool is simply too small for current load, check headroom across the whole fleet first: (
default_pool_size+reserve_pool_size) × each database/user pair × the number of PgBouncer instances must stay belowmax_connectionsminus the reserved slots (superuser_reserved_connections, andreserved_connectionson PostgreSQL 16+). A raise that looks safe on one pod is N times that across N poolers, and can recreate the exhaustion you are fixing. Only then raise the pool size inpgbouncer.iniand reload it.RELOADapplies the new config without dropping clients.Gate: approval required.
psql -h <pooler-host> -p 6432 -U <pgbouncer-admin-user> pgbouncer -c 'RELOAD;'To drain one database on a pooler without dropping clients, use
PAUSE <db>;: it waits for every server connection to be released and holds new queries;RESUME <db>;lets them through again. A single idle-in-transaction session makesPAUSEhang with every client queued behind it, so clear those first (step 1). Never run a barePAUSE;: it holds every database on that pooler. Before pausing, open a second admin session with theRESUMEline ready to paste. Keep the pause short.PAUSEis only practical withpool_mode = transaction. In session pooling it waits until every client disconnects and can hang indefinitely; useRECONNECT <db>below instead.Gate: approval required.
psql -h <pooler-host> -p 6432 -U <pgbouncer-admin-user> pgbouncer -c 'PAUSE <db>;'Check
SHOW POOLS;until the database shows no active server connections, do the work, then resume:Gate: approval required.
psql -h <pooler-host> -p 6432 -U <pgbouncer-admin-user> pgbouncer -c 'RESUME <db>;'
Session pooling only: RECONNECT <db>; asks PgBouncer to replace that database's server connections. It doesn't block or stop traffic, but each server connection is closed only once it's released, which in session mode means when its client disconnects. It takes effect gradually as clients reconnect, not straight away, so no change in the first minutes is expected and isn't a reason to escalate. KILL <db> drops every client at once and is in destructive-commands.md, not here.
Gate: approval required.
psql -h <pooler-host> -p 6432 -U <pgbouncer-admin-user> pgbouncer -c 'RECONNECT <db>;'Only if those fail, restart, one pooler instance at a time when there are several, waiting for each to pass verification before the next. PgBouncer 1.23 and later can do a rolling restart with so_reuseport across several processes, which avoids a full outage (PgBouncer usage).
On a VM or host, run it on that one pooler host:
Gate: approval required.
ssh <pooler-host> 'sudo systemctl restart pgbouncer && systemctl is-active pgbouncer'On Kubernetes:
Gate: approval required.
kubectl --context <ctx> -n <ns> rollout restart deployment/<pgbouncer>Then watch it (read-only):
kubectl --context <ctx> -n <ns> rollout status deployment/<pgbouncer> --timeout=300sVerify: the app error rate falls back toward baseline, SHOW POOLS; shows cl_waiting at or near 0, and the database connection count from pg_stat_activity stays below max_connections. If errors don't fall, stop and escalate; don't restart again.
Communication
drafts only; see sre-incident-communication gates
During incident
- Draft status page update for approval
- Draft stakeholder notification for approval
- Document actions in incident log
- Estimate resolution time if possible
Post incident
- Draft resolution announcement for approval
- Send postmortem notification (after approval)
- Schedule postmortem meeting
- Create follow-up tickets
Verification
After remediation
- Confirm error rates returned to baseline
- Verify latency within SLO
- Check synthetic monitors passing
- Confirm user-facing issues resolved
- Monitor for stability (5-15 min)
Stop and escalate
- Metrics worsen after an action
- Action fails or output is unexpected
- Required approval not given
- Any action labeled reversible: no that was not explicitly approved
- On stop: halt, report what was done, propose rollback, escalate to IC/on-call
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

