sre-message-queue-triage
Skill4 starsChanged 23 days agoMITView on GitHub
Use when Kafka, RabbitMQ or SQS/SNS is backing up or dropping messages: consumer lag, DLQ growth, broker disk/under-replication, redelivery storms, poison-pill messages. Read-only; purge, rebalance and resize are gated.
In this repository84 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- .cursor/rules/sre-safety-gates.mdcCursor rule
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dr-backup-testSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- sre-incident-communicationSkill
- sre-k8s-safe-remediationSkill
- sre-k8s-triageSkill
- sre-k8s-upgrade-autopilotSkill
- sre-migration-safetySkill
- sre-observability-cost-cutterSkill
- sre-observability-query-cookbookSkill
- sre-oncall-auditSkill
- sre-oncall-handoffSkill
- sre-pagerduty-opsgenieSkill
- sre-php2shell-checkSkill
- sre-postmortem-templatesSkill
- sre-production-readinessSkill
- sre-prometheus-grafanaSkill
- sre-remediation-actionsSkill
- sre-runbook-templatesSkill
- sre-safety-gatesSkill
- sre-security-incident-triageSkill
- sre-service-mesh-triageSkill
- sre-slo-alertingSkill
- sre-slo-burn-rate-rulesSkill
- sre-terraformSkill
- sre-toil-reviewSkill
- sre-awsSkill
- sre-azureSkill
- sre-capacity-planningSkill
- sre-change-safetySkill
- sre-chaos-engineeringSkill
- sre-cloud-cost-reviewSkill
- sre-database-triageSkill
- sre-datadogSkill
- sre-dependency-mappingSkill
- sre-diagnosticsSkill
- sre-dr-backup-testSkill
- sre-dynatraceSkill
- sre-emudhra-ca-checkSkill
- sre-engineerSkill
- sre-expiry-auditSkill
- sre-gcpSkill
- sre-incident-commandSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Very specific: line 21
Pin the target: Every command names the cluster/vhost/queue/topic explicitly — never rely on a default bootstrap server, vhost, or region.
- Distinctive: line 23
A DLQ is evidence, not noise: Read its contents (bounded, non-destructive peek) before proposing a purge — it's often the only record of …
- Distinctive: line 24
Message bodies may hold customer data or secrets: Treat any peeked message body the way you'd treat a database query result: redact before …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Message queue triage
Safety gates (REQUIRED)
- Read-only diagnosis first.
- Propose each change with risk (low/med/high) and reversibility.
- Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
- Act only as approved, verify, and stop and escalate if anything gets worse.
Full rules: sre-safety-gates. Read-only MCP setup: sre-safety-gates/references/mcp.md.
Process skills: sre-incident-command (roles, severity), sre-engineer / sre-diagnostics (the wider investigation), sre-change-safety (the deploy that preceded it), sre-remediation-actions. Managed queues: sre-aws (SQS/SNS/MSK), sre-gcp (Pub/Sub), sre-azure (Service Bus/Event Hubs) for provider-side metrics and health.
Rules
- Pin the target. Every command names the cluster/vhost/queue/topic explicitly — never rely on a default bootstrap server, vhost, or region.
- Read-only first. Lag, depth, and consumer-group state are all queryable without touching data. Never consume-and-ack, purge, or republish to "test" something during triage.
- A DLQ is evidence, not noise. Read its contents (bounded, non-destructive peek) before proposing a purge — it's often the only record of what actually failed and why.
- Message bodies may hold customer data or secrets. Treat any peeked message body the way you'd treat a database query result: redact before pasting into tickets or chat, keep captures out of the repo.
- Gated actions (purge a queue/DLQ, pause/resume a consumer, trigger partition reassignment, resize a broker, change retention) appear only under
**Gate: approval required.**, one command per block, run after evidence capture and a written proposal. - Never without explicit approval: purging any queue or DLQ, force-deleting a topic/queue, triggering partition reassignment or leader election, changing retention/replication settings, scaling a broker cluster, redriving a DLQ back to the source queue in bulk.
Decision tree by symptom
| Symptom | Likely cause | Check |
| Consumer lag growing, processing not keeping up | Slow/stuck consumer, undersized consumer group, downstream dependency slow | consumer-group lag per partition; is the consumer process actually running and healthy (hand off to sre-k8s-triage/sre-diagnostics if it's app-side) |
| DLQ filling | Poison-pill messages, a bad deploy, schema change, downstream 4xx/5xx | peek DLQ messages (bounded), correlate with recent deploys via sre-change-safety |
| Redelivery storm (same message reprocessed repeatedly) | Consumer acking too late / visibility timeout too short, retry loop with no backoff, broker thinks the consumer died | visibility timeout vs. actual processing time (SQS), prefetch/ack settings (RabbitMQ), max.poll.interval.ms (Kafka) |
| Broker disk pressure / under-replicated partitions | Retention too long for the ingest rate, a slow/down replica, log segments not compacting | per-broker disk usage, under-replicated partition count (Kafka), queue memory/disk alarms (RabbitMQ) |
| Producers blocking or erroring | Queue/topic at capacity, broker down, network partition | producer error rate and type, broker/cluster health |
| Messages disappearing with no trace | TTL/expiration shorter than expected, a consumer silently acking before processing (SQS ReceiveMessage + crash before ack extension) | retention/TTL config, consumer ack pattern |
Read-only commands
Kafka
kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group> # lag per partition
kafka-topics.sh --bootstrap-server <broker> --describe --topic <topic> # replicas, ISR, under-replicated
kafka-log-dirs.sh --bootstrap-server <broker> --describe --topic-list <topic> # per-broker disk usage by topic
kafka-broker-api-versions.sh --bootstrap-server <broker> # broker reachable at all--describe --groupwith a lag of 0 but a consumer not making progress on wall-clock time usually means the consumer died and the group hasn't rebalanced yet (checkmax.poll.interval.msand session timeout).- Under-replicated partitions (
Isrshorter thanReplicasin topic describe) point at a slow or down broker, not the consumer.
RabbitMQ
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers memory
rabbitmqctl list_consumers
rabbitmqctl list_unacked
rabbitmqctl cluster_status- High
messages_unacknowledgedwith fewconsumersmeans consumers are holding messages too long (slow processing, or a prefetch count set too high) — not a queue problem. cluster_statusfirst if a queue's node is unreachable; node-down behaves very differently from queue backlog.
SQS / SNS (AWS)
aws sqs get-queue-attributes --queue-url <url> --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessagesDelayed
aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage --dimensions Name=QueueName,Value=<queue> --start-time <start-ISO8601> --end-time <end-ISO8601> --period 60 --statistics Maximum
aws sqs get-queue-attributes --queue-url <dlq-url> --attribute-names ApproximateNumberOfMessages # DLQ depthPeek DLQ contents without removing them (bounded, non-destructive — ReceiveMessage with a short VisibilityTimeout and no delete):
aws sqs receive-message --queue-url <dlq-url> --max-number-of-messages 5 --visibility-timeout 5ApproximateAgeOfOldestMessagerising with stableNumberOfMessagesusually means consumers stopped making progress, not an ingest spike.- A redrive policy's
maxReceiveCountreached is why a message landed in the DLQ — check the source queue's redrive policy before assuming it's an app bug.
Fix order when the queue is the symptom, not the cause
Most queue pages during an incident are caused by a consumer-side change: a deploy that slowed processing, a downstream dependency that got slow or started erroring, or a config change to visibility timeout/prefetch. Order:
- App-side rollback or config fix (gated in
sre-change-safety/sre-k8s-safe-remediation). - Scale consumers (more workers/pods) if the backlog is a throughput problem, not a stuck-consumer problem.
- Pause the producer at the source, if messages are being generated faster than they can ever be drained (propose only — usually a bigger decision than on-call should make alone).
- DLQ redrive, one message or a small bounded batch at a time, only after the root cause is fixed — redriving into a queue that will just re-fail recreates the storm.
- Broker/cluster resize or retention change: propose only, usually after the incident, with the queue/platform owner.
Gate: approval required.
aws sqs purge-queue --queue-url <url>Gate: approval required.
rabbitmqctl purge_queue <queue>Gate: approval required.
kafka-reassign-partitions.sh --bootstrap-server <broker> --reassignment-json-file <file>.json --executeOutput
- Target (broker/cluster, vhost/queue/topic, managed or self-hosted), evidence (lag/depth/DLQ counts with timestamps), likely cause and confidence, proposed actions in fix order with risk/reversibility and the approver, verification result (lag trending down, DLQ depth stable or empty).
Stop and escalate
- A broker/cluster is down or partitioned, not just backed up: escalate to the platform/infra owner immediately rather than triaging queue depth.
- A DLQ redrive would re-trigger the same failure at scale (root cause not yet fixed): don't redrive, report and wait.
- Producers are blocked and there's no safe way to shed load without dropping data: escalate for a product-level decision (which messages can be dropped, if any).
Last reviewed: 2026-10-23
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

