agentleFS
kryptophonikkryptophonik/SRE-Skills/.claude/skills/sre-message-queue-triage/SKILL.md

sre-message-queue-triage

Skill4 starsChanged 23 days agoMITView on GitHub

Use when Kafka, RabbitMQ or SQS/SNS is backing up or dropping messages: consumer lag, DLQ growth, broker disk/under-replication, redelivery storms, poison-pill messages. Read-only; purge, rebalance and resize are gated.

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.

    line 12
  2. Very specific:

    Pin the target: Every command names the cluster/vhost/queue/topic explicitly — never rely on a default bootstrap server, vhost, or region.

    line 21
  3. Distinctive:

    A DLQ is evidence, not noise: Read its contents (bounded, non-destructive peek) before proposing a purge — it's often the only record of …

    line 23
  4. Distinctive:

    Message bodies may hold customer data or secrets: Treat any peeked message body the way you'd treat a database query result: redact before …

    line 24

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Message queue triage

Safety gates (REQUIRED)

  1. Read-only diagnosis first.
  2. Propose each change with risk (low/med/high) and reversibility.
  3. Wait for explicit human approval before any prod-changing command, MCP write tool, or external post.
  4. Act only as approved, verify, and stop and escalate if anything gets worse.

Full rules: sre-safety-gates. Read-only MCP setup: sre-safety-gates/references/mcp.md.

Process skills: sre-incident-command (roles, severity), sre-engineer / sre-diagnostics (the wider investigation), sre-change-safety (the deploy that preceded it), sre-remediation-actions. Managed queues: sre-aws (SQS/SNS/MSK), sre-gcp (Pub/Sub), sre-azure (Service Bus/Event Hubs) for provider-side metrics and health.

Rules

  • Pin the target. Every command names the cluster/vhost/queue/topic explicitly — never rely on a default bootstrap server, vhost, or region.
  • Read-only first. Lag, depth, and consumer-group state are all queryable without touching data. Never consume-and-ack, purge, or republish to "test" something during triage.
  • A DLQ is evidence, not noise. Read its contents (bounded, non-destructive peek) before proposing a purge — it's often the only record of what actually failed and why.
  • Message bodies may hold customer data or secrets. Treat any peeked message body the way you'd treat a database query result: redact before pasting into tickets or chat, keep captures out of the repo.
  • Gated actions (purge a queue/DLQ, pause/resume a consumer, trigger partition reassignment, resize a broker, change retention) appear only under **Gate: approval required.**, one command per block, run after evidence capture and a written proposal.
  • Never without explicit approval: purging any queue or DLQ, force-deleting a topic/queue, triggering partition reassignment or leader election, changing retention/replication settings, scaling a broker cluster, redriving a DLQ back to the source queue in bulk.

Decision tree by symptom

SymptomLikely causeCheck
Consumer lag growing, processing not keeping upSlow/stuck consumer, undersized consumer group, downstream dependency slowconsumer-group lag per partition; is the consumer process actually running and healthy (hand off to sre-k8s-triage/sre-diagnostics if it's app-side)
DLQ fillingPoison-pill messages, a bad deploy, schema change, downstream 4xx/5xxpeek DLQ messages (bounded), correlate with recent deploys via sre-change-safety
Redelivery storm (same message reprocessed repeatedly)Consumer acking too late / visibility timeout too short, retry loop with no backoff, broker thinks the consumer diedvisibility timeout vs. actual processing time (SQS), prefetch/ack settings (RabbitMQ), max.poll.interval.ms (Kafka)
Broker disk pressure / under-replicated partitionsRetention too long for the ingest rate, a slow/down replica, log segments not compactingper-broker disk usage, under-replicated partition count (Kafka), queue memory/disk alarms (RabbitMQ)
Producers blocking or erroringQueue/topic at capacity, broker down, network partitionproducer error rate and type, broker/cluster health
Messages disappearing with no traceTTL/expiration shorter than expected, a consumer silently acking before processing (SQS ReceiveMessage + crash before ack extension)retention/TTL config, consumer ack pattern

Read-only commands

Kafka

kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group>   # lag per partition
kafka-topics.sh --bootstrap-server <broker> --describe --topic <topic>            # replicas, ISR, under-replicated
kafka-log-dirs.sh --bootstrap-server <broker> --describe --topic-list <topic>     # per-broker disk usage by topic
kafka-broker-api-versions.sh --bootstrap-server <broker>                          # broker reachable at all
  • --describe --group with a lag of 0 but a consumer not making progress on wall-clock time usually means the consumer died and the group hasn't rebalanced yet (check max.poll.interval.ms and session timeout).
  • Under-replicated partitions (Isr shorter than Replicas in topic describe) point at a slow or down broker, not the consumer.

RabbitMQ

rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers memory
rabbitmqctl list_consumers
rabbitmqctl list_unacked
rabbitmqctl cluster_status
  • High messages_unacknowledged with few consumers means consumers are holding messages too long (slow processing, or a prefetch count set too high) — not a queue problem.
  • cluster_status first if a queue's node is unreachable; node-down behaves very differently from queue backlog.

SQS / SNS (AWS)

aws sqs get-queue-attributes --queue-url <url> --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessagesDelayed
aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage --dimensions Name=QueueName,Value=<queue> --start-time <start-ISO8601> --end-time <end-ISO8601> --period 60 --statistics Maximum
aws sqs get-queue-attributes --queue-url <dlq-url> --attribute-names ApproximateNumberOfMessages   # DLQ depth

Peek DLQ contents without removing them (bounded, non-destructive — ReceiveMessage with a short VisibilityTimeout and no delete):

aws sqs receive-message --queue-url <dlq-url> --max-number-of-messages 5 --visibility-timeout 5
  • ApproximateAgeOfOldestMessage rising with stable NumberOfMessages usually means consumers stopped making progress, not an ingest spike.
  • A redrive policy's maxReceiveCount reached is why a message landed in the DLQ — check the source queue's redrive policy before assuming it's an app bug.

Fix order when the queue is the symptom, not the cause

Most queue pages during an incident are caused by a consumer-side change: a deploy that slowed processing, a downstream dependency that got slow or started erroring, or a config change to visibility timeout/prefetch. Order:

  1. App-side rollback or config fix (gated in sre-change-safety / sre-k8s-safe-remediation).
  2. Scale consumers (more workers/pods) if the backlog is a throughput problem, not a stuck-consumer problem.
  3. Pause the producer at the source, if messages are being generated faster than they can ever be drained (propose only — usually a bigger decision than on-call should make alone).
  4. DLQ redrive, one message or a small bounded batch at a time, only after the root cause is fixed — redriving into a queue that will just re-fail recreates the storm.
  5. Broker/cluster resize or retention change: propose only, usually after the incident, with the queue/platform owner.

Gate: approval required.

aws sqs purge-queue --queue-url <url>

Gate: approval required.

rabbitmqctl purge_queue <queue>

Gate: approval required.

kafka-reassign-partitions.sh --bootstrap-server <broker> --reassignment-json-file <file>.json --execute

Output

  • Target (broker/cluster, vhost/queue/topic, managed or self-hosted), evidence (lag/depth/DLQ counts with timestamps), likely cause and confidence, proposed actions in fix order with risk/reversibility and the approver, verification result (lag trending down, DLQ depth stable or empty).

Stop and escalate

  • A broker/cluster is down or partitioned, not just backed up: escalate to the platform/infra owner immediately rather than triaging queue depth.
  • A DLQ redrive would re-trigger the same failure at scale (root cause not yet fixed): don't redrive, report and wait.
  • Producers are blocked and there's no safe way to shed load without dropping data: escalate for a product-level decision (which messages can be dropped, if any).

Last reviewed: 2026-10-23

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.