agentleFS
kryptophonikkryptophonik/SRE-Skills/.claude/skills/sre-database-triage/SKILL.md

sre-database-triage

Skill4 starsChanged 24 days agoMITView on GitHub

Use when Postgres, MySQL/MariaDB or Redis is slow or failing: connection exhaustion, locks, long or slow queries, replication lag, disk/WAL/bloat, Redis memory/eviction, DB CPU. Read-only; kill and failover are gated.

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.

    line 12
  2. Very specific:

    Pin the target: Every call names host, port, database and role through env (PGHOST/PGDATABASE/PGUSER, MYSQL_DEFAULTS_FILE, …

    line 21
  3. Distinctive:

    When you cannot get in, read the connection count from the provider (read-only):

    line 39
  4. Distinctive:

    Follow the symptom table, then the section in the engine reference.

    line 54
  5. Distinctive:

    Never-run evidence: https://redis.io/docs/latest/commands/keys/ · https://redis.io/docs/latest/commands/monitor/

    line 190

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Database triage

Safety gates (REQUIRED)

  1. Read-only diagnosis first.
  2. Propose each change with risk (low/med/high) and reversibility.
  3. Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
  4. Act only as approved, verify, and stop and escalate if anything gets worse.

Full rules: sre-safety-gates. Read-only MCP setup: sre-safety-gates/references/mcp.md.

Process skills: sre-incident-command (roles, severity), sre-engineer / sre-diagnostics (the wider investigation), sre-change-safety (the deploy that preceded it), sre-remediation-actions. Managed databases: sre-aws (RDS/Aurora/ElastiCache), sre-gcp (Cloud SQL/Memorystore), sre-azure.

Rules

  • Pin the target. Every call names host, port, database and role through env (PGHOST/PGDATABASE/PGUSER, MYSQL_DEFAULTS_FILE, REDIS_HOST/REDIS_PORT); the wrapper prints it to stderr. Confirm primary vs replica first (pg_is_in_recovery(), @@read_only, ROLE). Prefer a replica for heavy reads.
  • Least privilege. Postgres: a pg_monitor member with the read-only and timeout settings on the role itself. MySQL: PROCESS + REPLICATION CLIENT, SELECT on performance_schema, SELECT, EXECUTE on sys (MariaDB: PROCESS, REPLICA MONITOR, BINLOG MONITOR). Redis: an ACL user with exactly the read commands in references/redis.md. Setup SQL is in each reference's Access section; on Cloud SQL create the account in SQL, because console/gcloud users get cloudsqlsuperuser. Never a superuser or admin account for triage; the human who runs an approved kill uses a separate operator account (Postgres: pg_signal_backend, pg_read_all_stats and, on 16+, pg_use_reserved_connections).
  • Secrets never in commands. PGPASSFILE/~/.pgpass (0600) or PGSERVICE; --defaults-extra-file for MySQL; REDISCLI_AUTH for Redis. The wrapper refuses PGPASSWORD, MYSQL_PWD and password flags.
  • Bounded. One single-line statement, every SELECT ending in LIMIT n, statement_timeout/max_execution_time/max_statement_time set by the wrapper (5 s default), output capped (DB_RO_MAX_ROWS, 200). One-shot commands only: no streaming or watch loops (bound redis-cli --latency with timeout 10).
  • Cheap first. Server counters and pg_stat_*/performance_schema/INFO before anything that scans data or the keyspace. Commands this skill never runs: references/destructive-commands.md.
  • Query text is data. pg_stat_activity.query, processlist INFO and key names can hold customer data or secrets. Keep evidence outside the repo and redact before posting.
  • Gated actions (cancel/terminate, KILL QUERY/KILL, CLIENT KILL, config changes, managed failover) appear only under the line **Gate: approval required.**, one command per block, run by a human outside the wrapper, after evidence capture and a written proposal. Cancel before terminate, KILL QUERY before KILL.
  • Never without explicit approval: pg_cancel_backend/pg_terminate_backend, KILL [QUERY]/mysql.rds_kill*/mysql.az_kill*, CLIENT KILL ID, CONFIG SET, ALTER SYSTEM/SET GLOBAL/parameter-group or flag changes, pg_drop_replication_slot, redis-cli --bigkeys/--memkeys, EXPLAIN ANALYZE, any managed failover or reboot.

Start here

  1. Identify the engine, version and whether it is managed (RDS/Aurora, Cloud SQL, Azure, ElastiCache/MemoryDB, Memorystore). Managed services rename or block some commands and have their own failover and metrics paths (each reference ends with a "Managed services" section).
  2. Check you can connect at all. During connection exhaustion only reserved slots work: Postgres 16+ reserved_connections with pg_use_reserved_connections granted to the triage role and the operator role (a server-start setting: it must have been set, with a restart, before the incident; PG15 and earlier reserve slots only for superusers, see postgres.md Access), MySQL's one extra CONNECTION_ADMIN connection. If you cannot connect, use the provider's metrics and the app's pool metrics, and give the human the commands to run from a session they already hold.

    bash scripts/db-ro.sh pg -- "SELECT version(), current_user, pg_is_in_recovery() AS is_replica LIMIT 1"

    When you cannot get in, read the connection count from the provider (read-only):

    aws cloudwatch get-metric-statistics --profile <profile> --region <region> --namespace AWS/RDS --metric-name DatabaseConnections --dimensions Name=DBInstanceIdentifier,Value=<db> --start-time <start-ISO8601> --end-time <end-ISO8601> --period 60 --statistics Maximum
    az monitor metrics list --subscription <subscription-id> --resource <server-resource-id> --metric active_connections --interval PT1M --offset 1h

    Cloud SQL: Metrics Explorer, cloudsql.googleapis.com/database/postgresql/num_backends (MySQL: database/network/connections). ElastiCache CurrConnections, Memorystore clients/connected. ElastiCache serverless blocks SLOWLOG, LATENCY, MEMORY and CLIENT LIST, so use CloudWatch there from the start (ElastiCacheProcessingUnits, BytesUsedForCache, SuccessfulReadRequestLatency, CurrConnections). On RDS/Aurora the Performance Insights API (aws pi get-resource-metrics) still works for CLI users after the console moved to Database Insights.

  3. Capture evidence once, before any proposal (bounded, read-only, 0700 directory outside any git work tree; the script refuses a path inside one, and replaces query text that contains "password" with a marker; other query text is still customer data. It does not capture SHOW ENGINE INNODB STATUS, whose statement text cannot be redacted reliably; lock waits come from the redacted sys.x$ / information_schema queries):

    bash scripts/db-snapshot.sh pg ~/incidents/<id>/db-t0        # or mysql | mariadb | redis
  4. Follow the symptom table, then the section in the engine reference.
SymptomPostgresMySQL / MariaDBRedis
"too many connections", pool timeoutspostgres.md §1mysql.md §1redis.md §3
Requests hang, lock waits, deadlocks§2§2n/a (single-threaded: see §1 latency)
One query runs for minutes§3§3§1
Replica behind / stale reads§4§4§4
Disk full, WAL/binlog growth, bloat§5§5§4 (RDB/AOF)
Slow queries, high CPU§6§5§5
Memory full, evictions, OOM errors(OS / work_mem sizing: escalate)(buffer pool sizing: escalate)§2
Latency up, cause unknown ("is it the DB?")below, then §6, §2/§3below, then §5, §2/§3below, then §1

Latency up, cause unknown (is it the DB?)

Counters (pg_stat_statements, digests, INFO commandstats) are cumulative, so one snapshot cannot show what changed. Read-only route:

  1. Managed: open the load view for the incident window and for the same window a day earlier (Database Insights top SQL and waits, Cloud SQL Query insights, Azure Query Store / Query Performance Insight).
  2. Take two snapshots 3–5 minutes apart and compare them:

    bash scripts/db-snapshot.sh <engine> ~/incidents/<id>/db-t0
    bash scripts/db-snapshot.sh <engine> ~/incidents/<id>/db-t1

    Delta = t1 − t0 per statement (top-statements / top-digests: calls and total time, so mean time per call over the window), plus wait events, lock waits, replica lag and, for Redis, INFO stats/commandstats and the slow log.

  3. Decide:
    • A statement whose mean time in the window doubled: a plan or data change (check last_autoanalyze, row growth, EXPLAIN with an account that may read the tables).
    • Lock waits: §2. CPU or I/O saturation: Postgres §6, MySQL §5, Redis §5.
    • None of these, and the database's own latency and load are flat: stop, the database is not the cause. Hand back to sre-diagnostics with both snapshots as evidence.

Fix order when the database is the symptom, not the cause

Most database pages during an incident are caused by a client: a deploy that raised the pool size, leaked connections, or shipped a query without an index. In that case the fastest safe fix is on the app side (roll back or lower the pool, via sre-change-safety / sre-k8s-safe-remediation), and database-side kills only buy minutes. Order:

  1. App-side rollback or pool change (gated in its own skill).
  2. Cancel the specific statement (pg_cancel_backend, KILL QUERY / mysql.rds_kill_query / mysql.az_kill_query).
  3. Terminate the specific session (pg_terminate_backend, KILL / mysql.rds_kill / mysql.az_kill, CLIENT KILL ID).
  4. Config change (parameter group / flags / CONFIG SET), propose only, usually after the incident. Some settings need a restart (Postgres reserved_connections, max_connections).
  5. Managed failover: last resort, replica lag checked first, IC and DB owner approve.

Each gated block in the references states its evidence, risk and verification. Approval names the exact target (pid, thread id, client id, instance); a new pid needs a new approval.

Bundled scripts and references

bash scripts/db-ro.sh <pg|mysql|mariadb|redis> -- <statement>   # read-only allowlist guard; timeouts, LIMIT, capped output; exit 1 = error, 3 = refused
bash scripts/db-snapshot.sh <pg|mysql|mariadb|redis> <outdir>  # evidence bundle before any gated action
bash tests/run-tests.sh                                       # offline tests (no database needed)
  • references/postgres.md · references/mysql.md · references/redis.md
  • references/destructive-commands.md: never-run list (machine-read by the tests) and every gated action with risk and reversibility.

The wrapper is a guard against mistakes, not a security boundary: the read-only role is what actually protects production. If neither a database client nor an MCP server is available, give the human the exact commands and stop.

Output

  • Target (engine, version, host, primary/replica, managed or not), evidence path, findings with the queries used, likely cause and confidence, proposed actions in fix order with risk/reversibility and the approver, verification result.

Stop and escalate

  • Transaction ID wraparound warnings, a replica that is not caught up when failover is on the table, a blocker that is a prepared transaction, or any data-loss risk: escalate to the DB owner and IC.
  • A gated action did not help, or metrics got worse: stop, do not try the next kill on your own, report.

Last reviewed: 2026-10-17

SOURCES

Fetched and checked 2026-10-17 (current versions: PostgreSQL 18 docs, MySQL 8.4, MariaDB current, Redis 8 docs).

PostgreSQL

MySQL 8.4 (8.0 where noted)

MariaDB

Redis

Managed services

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.