storm-guard

A single client reconnecting is a non-event. The same reconnect logic run by every connected client within the same few hundred milliseconds is a different problem entirely, and it's one teams routinely don't see coming because nothing in local testing or a staging environment with a dozen clients ever synchronizes them. The trigger isn't exotic: a rolling deploy closes every socket served by the pods being replaced at once; AWS's ALB and GCP's global external Application Load Balancer both let you set a backend/idle timeout that becomes a de facto maximum WebSocket lifetime — GCP documents an effective ceiling around 24 hours (86400 seconds) on that backend service timeout and recommends against going higher, since Google periodically restarts the GFEs serving the connection for maintenance regardless of what you configure — and that ceiling batches disconnects by connection start time — every socket opened during yesterday's deploy can get force-closed together, a day later, whether or not anyone remembers that correlation; an upstream auth or session-store blip disconnects the whole fleet in one shot. "We have exponential backoff" is not the same claim as "we have backoff that actually decorrelates the fleet" — a fixed delay, or backoff computed identically from the same formula on every client with no randomization, still re-synchronizes everyone to the same retry instant.

This is a documented cause of real, multi-hour outages, not a theoretical edge case. Discord's own engineering postmortem for the March 25, 2026 voice outage ("You've Got (Too Much) Mail") describes a Kubernetes config change killing a slice of session servers, the survivors' clients all reconnecting at once, and the resulting thundering herd overwhelming a single Erlang supervisor process with a mailbox queue that grew faster than it could be drained — turning a partial node loss into a cascading, hours-long recovery. Slack built an entire dedicated service, Flannel, specifically because a Gateway Server dying and 100K clients reconnecting simultaneously would otherwise hit the backend databases directly; Flannel serves reconnecting clients from an edge cache instead, which is infrastructure investment that exists only because "mass reconnect hits the database" was a real, costly failure mode first. (Primary blog access was unavailable when this skill was written; both incidents are corroborated across multiple independent secondary write-ups describing consistent, specific technical detail — not a single unverified source.)

Step 1: find the reconnect trigger and whether disconnects are already synchronized

Find the client's reconnect scheduling (onclose/onerror handlers, a reconnect-manager class, an SDK's built-in retry). Separately, find what can close many connections in the same narrow window on the backend side: a rolling deploy or pod-replacement strategy for the service holding the sockets, any load balancer or proxy in front of it with a max connection lifetime, or a single shared dependency (auth service, session store, the realtime server itself) whose outage would disconnect everyone who depends on it at once. A deploy or an LB-enforced lifetime limit means disconnects aren't a rare random event you can treat as independent per-client — they're a scheduled, periodic batch.

Step 2: check whether the reconnect delay actually decorrelates the fleet

A delay existing is not the bar. Check specifically: is the wait time randomized per client (full jitter — a random value between zero and a capped maximum, recomputed on every attempt) or is it a fixed constant, or a backoff formula with no randomization term at all? Both of the latter mean every client that disconnected together reconnects together, at the same instant, no matter how long the delay is. Also check whether backoff actually resets after a successful connection — a client that fails once early and keeps compounding backoff off that first failure forever behaves very differently under a mass event than one whose backoff state tracks its own recent history correctly.

Step 3: check what one reconnect costs the backend, then multiply by the fleet

Trace what happens server-side when a connection is accepted: a session or token lookup against a database, a call out to an auth service, a query that hydrates the client's full current state before the first message is sent, a call to a subscribe/topic-registration endpoint. Each of these is normal, even cheap, for one connection. None of them are cheap for every currently-active connection arriving inside the same jitter window. Check for anything that shields the backend at that multiplied scale: an edge/read cache in front of the expensive lookup (Flannel's role at Slack), rate limiting or admission control on the connection-accept or auth path, or a staged/batched rollout of the backend change itself (bleeding pods out of rotation over minutes instead of cycling all of them at once) that spreads the client-side disconnect-and-reconnect across a longer window in the first place.

Step 4: check whether this has ever actually been tested, not just coded

"The code has backoff with jitter" and "we know what happens when this service's whole client fleet reconnects at once" are different claims verified by different evidence. Look for a load test, a chaos/game-day exercise, or a deploy runbook that specifically watches auth/session/DB load during a rollout — versus jitter sitting in the client code, unverified at the connection counts and lookup costs steps 2–3 actually found.

Report

Lead with the verdict. BLOCK when backoff is unjittered (fixed delay or a deterministic formula with no randomization) and a mass-disconnect trigger is real for this system (a deploy pattern, an LB-enforced connection lifetime, a shared dependency with its own failure history) — this is a known, citable path to a multi-hour outage, not a hypothetical. Also BLOCK when jitter exists but the backend has zero shielding and the per-connection work includes a DB/auth-service round trip, since the jitter window just becomes the outage's duration instead of its prevention. CAUTION when jitter exists and bounds look reasonable but haven't been verified at this system's actual connection count, or backend shielding exists but has never been exercised under a simulated mass-reconnect. SAFE only with evidence: true per-client randomized jitter with a sane cap, backend work that's either genuinely cheap per connection or explicitly shielded (cache, rate limit, or staged rollout that spreads the disconnect window), and at least one load test or real incident that exercised the mass-reconnect path and didn't fall over.

Boundaries

• Not job-warden's territory — that skill's thundering-herd note is about cron jobs sharing a schedule trigger (0 0 * * *), not clients reconnecting to a live service.
• Not stale-guard's stampede case — a hot cache key expiring under concurrent request load is a different mechanism (singleflight/lock-and-wait) than many independent client connections arriving at once.
• Not backpressure-guard's territory — that skill audits the ongoing health of connections that are already established and subscribed (a slow consumer stalling delivery, unbounded per-connection state); this skill audits the arrival event itself, before a connection is even accepted, and the client-side behavior that determines when it arrives.
• Not resync-guard's territory — that skill audits whether a reconnecting client recovers the data it missed; this skill audits whether the backend survives the reconnect attempt in the first place. A system can resync perfectly and still fall over if every client resyncs at the same millisecond.
• Can't measure real fleet size, actual reconnect-timing distribution, or true per-lookup cost from source alone — say so plainly when the verdict genuinely depends on production scale (how many clients, how expensive the auth/DB round trip really is under load) rather than guessing a number.
• Doesn't audit load-balancer or infrastructure configuration outside the application's own control — can flag that an LB-enforced max connection lifetime is a mass-disconnect trigger worth knowing about, but can't review the LB's actual config unless it's checked into the repo as code (Terraform, CDK, etc.).