sre-dr-backup-test
Skill4 starsChanged 23 days agoMITView on GitHub
Use when testing backups or DR readiness: backup-restore and failover drills against RTO/RPO, restoring only to new isolated targets after approval, with an evidence record.
In this repository84 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- .cursor/rules/sre-safety-gates.mdcCursor rule
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dr-backup-testSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- sre-incident-communicationSkill
- sre-k8s-safe-remediationSkill
- sre-k8s-triageSkill
- sre-k8s-upgrade-autopilotSkill
- sre-message-queue-triageSkill
- sre-migration-safetySkill
- sre-observability-cost-cutterSkill
- sre-observability-query-cookbookSkill
- sre-oncall-auditSkill
- sre-oncall-handoffSkill
- sre-pagerduty-opsgenieSkill
- sre-php2shell-checkSkill
- sre-postmortem-templatesSkill
- sre-production-readinessSkill
- sre-prometheus-grafanaSkill
- sre-remediation-actionsSkill
- sre-runbook-templatesSkill
- sre-safety-gatesSkill
- sre-security-incident-triageSkill
- sre-service-mesh-triageSkill
- sre-slo-alertingSkill
- sre-slo-burn-rate-rulesSkill
- sre-terraformSkill
- sre-toil-reviewSkill
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Very specific: line 17
Never restore over a live resource.
- Distinctive: line 21
No written RTO/RPO target for the system.
- Distinctive: line 23
No owner or timebox; no approval.
- Distinctive: line 32
Run: start timer; restore to isolated target.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Backup and DR Testing
Safety gates (REQUIRED)
- Read-only diagnosis first.
- Propose each change with risk (low/med/high) and reversibility.
- Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Act only as approved, verify, and stop and escalate if anything gets worse.
Full rules: sre-safety-gates. Read-only MCP setup: sre-safety-gates/references/mcp.md.
Never restore over a live resource. Always restore to a new, isolated target (new resource group / account / project, isolated network). On Azure, restore --target-resource-group must be a new drill RG; ASR uses test failover into an isolated VNet only.
No-go rules
- No written RTO/RPO target for the system.
- Restore target is not isolated (could receive prod traffic or overwrite prod data).
- No owner or timebox; no approval.
- Active incident on the system.
Workflow
- Scope: system, data stores, RTO (time to restore) and RPO (acceptable data loss).
- Inventory backups (read-only): schedule, retention, last success, encryption, cross-region/account copy.
- Plan the drill: restore type (single table/file, full DB, cluster, region failover), target, validation queries, timebox, rollback/cleanup.
- Get approval.
- Run: start timer; restore to isolated target.
- Validate: row counts / checksums vs. source, app smoke test, and the recovery point (for RPO).
- Take the recovery point from the backup itself, not from the data:
pg_dumpuses one snapshot taken when it starts (pg_restore -lprints "Archive created at", which can show in a different time zone); a physical backup hasSTART TIMEandSTART WAL LOCATIONin itsbackup_label; a point-in-time restore has the target you gave it, and on the restored serverpg_last_xact_replay_timestamp()/pg_last_wal_replay_lsn()show what was actually replayed; RDS hasSnapshotCreateTimeor--restore-time. Compare that point (time, or LSN againstpg_current_wal_lsn()on the source) with when the failure would have happened. That gap is the achieved RPO. - The newest
updated_at/created_atin the restore is only a lower bound on the recovery point: updates that don't set the column, and quiet periods, make it older than the backup (lab: 20 minutes older). Use it only as a cross-check, and only with a column every write maintains (for example set by a trigger). Rows written after the recovery point must be absent; checking a few known ones proves it.
- Take the recovery point from the backup itself, not from the data:
- Record achieved RTO/RPO vs. target; gaps become tickets.
- Clean up restored resources (after approval) to avoid cost and data exposure.
Read-only inventory commands
# AWS
aws backup list-backup-jobs --profile <profile> --region <region> --by-state COMPLETED --by-created-after <ISO8601> --max-items 20
aws rds describe-db-snapshots --profile <profile> --region <region> --db-instance-identifier <db> --snapshot-type automated --max-items 20
# Azure
az backup job list --resource-group <rg> --vault-name <vault> --subscription <subscription-id> --status Completed --start-date <dd-mm-yyyy> -o table
# GCP
gcloud sql backups list --instance=<instance> --project=<PROJECT_ID> --limit=20
# Kubernetes (Velero)
velero backup get | head -30
velero backup describe <backup> --detailsRestore commands (WRITE — approval required, new target only)
Full AWS RDS drill (inventory, isolated restore, validation, RTO/RPO, tag-checked cleanup): references/aws-rds.md.
Each command below is a separate write and a separate approval. Restores go to a new, isolated target only.
RDS restores are billable and NOT isolated by default (default SG, default subnet group). Always override:
Gate: approval required.
aws rds restore-db-instance-from-db-snapshot --profile <profile> --region <region> --db-instance-identifier drtest-<db>-<date> --db-snapshot-identifier <snap> \
--db-subnet-group-name <isolated-subnet-group> --vpc-security-group-ids <isolated-sg> --no-publicly-accessible --no-multi-az \
--db-instance-class <small-class> --tags Key=purpose,Value=dr-drillMySQL/PostgreSQL restores may default to RDS Extended Support (extra charges); set --engine-lifecycle-support if needed.
Gate: approval required.
aws rds restore-db-instance-to-point-in-time --profile <profile> --region <region> --source-db-instance-identifier <db> --target-db-instance-identifier drtest-<db>-<date> \
--restore-time <ISO8601> --db-subnet-group-name <isolated-subnet-group> --vpc-security-group-ids <isolated-sg> --no-publicly-accessible --no-multi-az \
--db-instance-class <small-class> --tags Key=purpose,Value=dr-drill # or --use-latest-restorable-time instead of --restore-timeAurora: restore a cluster, then add an instance.
Gate: approval required.
aws rds restore-db-cluster-from-snapshot --profile <profile> --region <region> --db-cluster-identifier drtest-<cluster>-<date> --snapshot-identifier <snap> --engine <aurora-mysql|aurora-postgresql> \
--db-subnet-group-name <isolated-subnet-group> --vpc-security-group-ids <isolated-sg> --tags Key=purpose,Value=dr-drillCleanup: only if the identifier starts with drtest-.
Gate: approval required.
aws rds delete-db-instance --profile <profile> --region <region> --db-instance-identifier drtest-<db>-<date> --skip-final-snapshotKubernetes (Velero): restore into a mapped -drtest namespace.
Gate: approval required.
velero restore create --from-backup <backup> --namespace-mappings <ns>:<ns>-drtestGCP Cloud SQL: restore to a NEW isolated instance, never the source instance (Google: "The restore process overwrites all of the current data on the instance, including previous point-in-time recovery (PITR) logs." — https://cloud.google.com/sql/docs/mysql/backup-recovery/restoring)
Gate: approval required.
gcloud sql backups restore <BACKUP_ID> --restore-instance=<NEW_ISOLATED_INSTANCE> --backup-instance=<SOURCE_INSTANCE> --project=<PROJECT_ID>Azure VM: restore disks to a NEW resource group / storage account, never the original location.
Gate: approval required.
az backup restore restore-disks --resource-group <vault-rg> --vault-name <vault> --container-name <container> --item-name <vm> --rp-name <recovery-point> --storage-account <new-staging-sa> --target-resource-group <new-drtest-rg> --subscription <subscription-id>Azure SQL / SAP HANA workloads in VMs: restore as a new database to an alternate target.
Gate: approval required.
az backup restore restore-azurewl --resource-group <vault-rg> --vault-name <vault> --recovery-config <recovery-config.json> --subscription <subscription-id>Azure Site Recovery: run a TEST failover into an isolated network (not a planned or unplanned failover).
Gate: approval required.
az site-recovery protected-item test-failover --resource-group <rg> --vault-name <asr-vault> --fabric-name <fabric> --protection-container <container> --name <protected-item> --subscription <subscription-id> # params vary by version; check `az site-recovery protected-item test-failover -h`Drill record template
# DR drill: [system] — [date]
- **Scenario:** [restore / failover]
- **Targets:** RTO [x] / RPO [y]
- **Achieved:** RTO [x'] / RPO [y'] — [pass|fail]
- **Validation:** [checks + results]
- **Issues found:** [list → tickets]
- **Cleanup done:** [yes/no, by whom]
- **Approved by:** [name, time]Output
- Drill record with achieved vs. target RTO/RPO, validation evidence, gaps as tickets
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

