agentleFS
kryptophonikkryptophonik/SRE-Skills/skills/sre-dr-backup-test/SKILL.md

sre-dr-backup-test

Skill4 starsChanged 23 days agoMITView on GitHub

Use when testing backups or DR readiness: backup-restore and failover drills against RTO/RPO, restoring only to new isolated targets after approval, with an evidence record.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.

    line 12
  2. Very specific:

    Never restore over a live resource.

    line 17
  3. Distinctive:

    No written RTO/RPO target for the system.

    line 21
  4. Distinctive:

    No owner or timebox; no approval.

    line 23
  5. Distinctive:

    Run: start timer; restore to isolated target.

    line 32

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Backup and DR Testing

Safety gates (REQUIRED)

  1. Read-only diagnosis first.
  2. Propose each change with risk (low/med/high) and reversibility.
  3. Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
  4. Act only as approved, verify, and stop and escalate if anything gets worse.

Full rules: sre-safety-gates. Read-only MCP setup: sre-safety-gates/references/mcp.md.

Never restore over a live resource. Always restore to a new, isolated target (new resource group / account / project, isolated network). On Azure, restore --target-resource-group must be a new drill RG; ASR uses test failover into an isolated VNet only.

No-go rules

  • No written RTO/RPO target for the system.
  • Restore target is not isolated (could receive prod traffic or overwrite prod data).
  • No owner or timebox; no approval.
  • Active incident on the system.

Workflow

  1. Scope: system, data stores, RTO (time to restore) and RPO (acceptable data loss).
  2. Inventory backups (read-only): schedule, retention, last success, encryption, cross-region/account copy.
  3. Plan the drill: restore type (single table/file, full DB, cluster, region failover), target, validation queries, timebox, rollback/cleanup.
  4. Get approval.
  5. Run: start timer; restore to isolated target.
  6. Validate: row counts / checksums vs. source, app smoke test, and the recovery point (for RPO).
    • Take the recovery point from the backup itself, not from the data: pg_dump uses one snapshot taken when it starts (pg_restore -l prints "Archive created at", which can show in a different time zone); a physical backup has START TIME and START WAL LOCATION in its backup_label; a point-in-time restore has the target you gave it, and on the restored server pg_last_xact_replay_timestamp() / pg_last_wal_replay_lsn() show what was actually replayed; RDS has SnapshotCreateTime or --restore-time. Compare that point (time, or LSN against pg_current_wal_lsn() on the source) with when the failure would have happened. That gap is the achieved RPO.
    • The newest updated_at/created_at in the restore is only a lower bound on the recovery point: updates that don't set the column, and quiet periods, make it older than the backup (lab: 20 minutes older). Use it only as a cross-check, and only with a column every write maintains (for example set by a trigger). Rows written after the recovery point must be absent; checking a few known ones proves it.
  7. Record achieved RTO/RPO vs. target; gaps become tickets.
  8. Clean up restored resources (after approval) to avoid cost and data exposure.

Read-only inventory commands

# AWS
aws backup list-backup-jobs --profile <profile> --region <region> --by-state COMPLETED --by-created-after <ISO8601> --max-items 20
aws rds describe-db-snapshots --profile <profile> --region <region> --db-instance-identifier <db> --snapshot-type automated --max-items 20
# Azure
az backup job list --resource-group <rg> --vault-name <vault> --subscription <subscription-id> --status Completed --start-date <dd-mm-yyyy> -o table
# GCP
gcloud sql backups list --instance=<instance> --project=<PROJECT_ID> --limit=20
# Kubernetes (Velero)
velero backup get | head -30
velero backup describe <backup> --details

Restore commands (WRITE — approval required, new target only)

Full AWS RDS drill (inventory, isolated restore, validation, RTO/RPO, tag-checked cleanup): references/aws-rds.md.

Each command below is a separate write and a separate approval. Restores go to a new, isolated target only.

RDS restores are billable and NOT isolated by default (default SG, default subnet group). Always override:

Gate: approval required.

aws rds restore-db-instance-from-db-snapshot --profile <profile> --region <region> --db-instance-identifier drtest-<db>-<date> --db-snapshot-identifier <snap> \
  --db-subnet-group-name <isolated-subnet-group> --vpc-security-group-ids <isolated-sg> --no-publicly-accessible --no-multi-az \
  --db-instance-class <small-class> --tags Key=purpose,Value=dr-drill

MySQL/PostgreSQL restores may default to RDS Extended Support (extra charges); set --engine-lifecycle-support if needed.

Gate: approval required.

aws rds restore-db-instance-to-point-in-time --profile <profile> --region <region> --source-db-instance-identifier <db> --target-db-instance-identifier drtest-<db>-<date> \
  --restore-time <ISO8601> --db-subnet-group-name <isolated-subnet-group> --vpc-security-group-ids <isolated-sg> --no-publicly-accessible --no-multi-az \
  --db-instance-class <small-class> --tags Key=purpose,Value=dr-drill      # or --use-latest-restorable-time instead of --restore-time

Aurora: restore a cluster, then add an instance.

Gate: approval required.

aws rds restore-db-cluster-from-snapshot --profile <profile> --region <region> --db-cluster-identifier drtest-<cluster>-<date> --snapshot-identifier <snap> --engine <aurora-mysql|aurora-postgresql> \
  --db-subnet-group-name <isolated-subnet-group> --vpc-security-group-ids <isolated-sg> --tags Key=purpose,Value=dr-drill

Cleanup: only if the identifier starts with drtest-.

Gate: approval required.

aws rds delete-db-instance --profile <profile> --region <region> --db-instance-identifier drtest-<db>-<date> --skip-final-snapshot

Kubernetes (Velero): restore into a mapped -drtest namespace.

Gate: approval required.

velero restore create --from-backup <backup> --namespace-mappings <ns>:<ns>-drtest

GCP Cloud SQL: restore to a NEW isolated instance, never the source instance (Google: "The restore process overwrites all of the current data on the instance, including previous point-in-time recovery (PITR) logs." — https://cloud.google.com/sql/docs/mysql/backup-recovery/restoring)

Gate: approval required.

gcloud sql backups restore <BACKUP_ID> --restore-instance=<NEW_ISOLATED_INSTANCE> --backup-instance=<SOURCE_INSTANCE> --project=<PROJECT_ID>

Azure VM: restore disks to a NEW resource group / storage account, never the original location.

Gate: approval required.

az backup restore restore-disks --resource-group <vault-rg> --vault-name <vault> --container-name <container> --item-name <vm> --rp-name <recovery-point> --storage-account <new-staging-sa> --target-resource-group <new-drtest-rg> --subscription <subscription-id>

Azure SQL / SAP HANA workloads in VMs: restore as a new database to an alternate target.

Gate: approval required.

az backup restore restore-azurewl --resource-group <vault-rg> --vault-name <vault> --recovery-config <recovery-config.json> --subscription <subscription-id>

Azure Site Recovery: run a TEST failover into an isolated network (not a planned or unplanned failover).

Gate: approval required.

az site-recovery protected-item test-failover --resource-group <rg> --vault-name <asr-vault> --fabric-name <fabric> --protection-container <container> --name <protected-item> --subscription <subscription-id> # params vary by version; check `az site-recovery protected-item test-failover -h`

Drill record template

# DR drill: [system] — [date]
- **Scenario:** [restore / failover]
- **Targets:** RTO [x] / RPO [y]
- **Achieved:** RTO [x'] / RPO [y'] — [pass|fail]
- **Validation:** [checks + results]
- **Issues found:** [list → tickets]
- **Cleanup done:** [yes/no, by whom]
- **Approved by:** [name, time]

Output

  • Drill record with achieved vs. target RTO/RPO, validation evidence, gaps as tickets

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.