failover-guard

Failover gets tested for the easy case: kill the primary cleanly, confirm a replica gets promoted, confirm the app reconnects. That proves failover happens — it says nothing about what it costs. The two things that actually determine the cost are invisible in a clean test: how far behind the promoted replica was at the instant of promotion, and whether the old primary was fully silenced before anyone started trusting the new one as the single source of truth. A clean kill answers neither, because a clean kill doesn't produce replication lag or a lingering half-dead node — it's the one scenario where both questions happen to not matter. GitHub's October 21, 2018 incident is the canonical case of what happens when they do: a 43-second network partition during routine maintenance caused their orchestrator to promote a West Coast MySQL replica while the original East Coast primary — unreachable to the orchestrator, not actually dead — kept accepting and committing writes locally for several more seconds before anything stopped it. Two primaries, two diverging histories, for a window measured in seconds; the outage that followed lasted over a day, because reconciling which writes were "real" wasn't a reconnect, it was forensics.

Step 1: find what triggers promotion and what "the write succeeded" actually promised

Identify the failover mechanism — an orchestrator (Patroni, Orchestrator, Vitess, a cloud-managed one like RDS/Aurora Multi-AZ or Cloud SQL HA), a manual runbook, or nothing automated at all. Separately, find what the application told the client when a write committed: does "200 OK" / "transaction committed" mean "fsynced on the primary" only, or does it mean "replicated to at least one other node" (synchronous or semi-sync commit)? These are different promises with different failure surfaces, and most teams have only ever examined the first one — the write path's happy path — never the second, which only shows up when promotion actually happens.

Step 2: replication mode — what's actually in flight when promotion fires

• Fully async replication (the default for MySQL/Postgres streaming replication with no synchronous_commit/semi-sync wait): the primary acknowledges a write the instant it's durable locally, before any replica has it. Every write in whatever the current replication lag is at the moment of promotion is a candidate for loss — not a hypothetical "if lag spikes," but the normal, steady-state gap this mode always runs. Ask what that lag actually measures as, in this system, under real load, not "usually under a second."
• Semi-sync with a timeout fallback (MySQL semi-sync, Postgres synchronous_commit = remote_write with a timeout): looks synchronous until the one replica it's waiting on is slow or unreachable, at which point most configurations silently fall back to async rather than blocking forever — and nothing about that fallback is visible to the caller who just got told their write was durable under the stronger guarantee. Check whether a fallback event is logged or alerted on anywhere, or whether the only trace it ever happened is a lost write discovered later.
• True synchronous replication (wait for a quorum of replicas, no fallback) has no lag window by construction, but then step 3's fencing question is the whole remaining risk — sync replication protects data that reached a quorum; it says nothing about which node the quorum, or the application, is still listening to.

Step 3: fencing — can the old primary still accept writes after a new one exists

The dangerous case isn't "the primary died," it's "the primary is unreachable from where the decision gets made, but still alive and still accepting connections from somewhere." Check what actually prevents that node from committing further writes once failover starts: a STONITH-style fence that kills the process or cuts its network, a VIP/DNS cutover the old primary has no way to reclaim, a lease/fencing-token scheme the storage layer itself enforces (so even a live old primary can't write to shared storage anymore) — versus nothing stronger than "the orchestrator tells everyone else there's a new primary now" and trusts every client to reconnect promptly. If the same partition that triggered promotion is also what makes the old primary unreachable to the fencing mechanism, ask explicitly what the orchestrator does next: proceed with promotion anyway (GitHub's exact shape — the thing it couldn't reach to fence was never actually stopped), or refuse to promote without confirmed fencing (safer, at the cost of staying down longer). Codebases that have only load-tested the "primary responds to fence request" path have never exercised the one branch this incident shape depends on.

Step 4: does anyone ever learn a loss/divergence window happened

After a real failover (not a drill against a clean kill), is there any mechanism that surfaces "writes between timestamp X and promotion may be affected" — a reconciliation job that diffs the old primary's unreplicated writes against what the new primary has, a replay queue, an audit log flagging the affected window, or even a coarse alert that fires on every promotion so a human looks? Or does the app simply start serving the new primary as though continuity was never in question, with the only evidence of loss being whatever a customer notices missing on their own? The second shape is the common one, and it's indistinguishable from "nothing was lost" until something downstream breaks — which is exactly why this needs to be checked in the code and the runbook, not inferred from the absence of a complaint.

Report

Lead with the combination that produced an actual incident, not either half in isolation: BLOCK for async/semi-sync-with-silent-fallback replication where the realistic lag window has no corresponding "writes in this window are unverified" mechanism — name the replication mode and what's known about real lag. BLOCK equally for a failover path that promotes without confirmed fencing of the old primary, especially when the same failure (a partition) that triggers promotion is also what would block fencing — name the exact gap (STONITH absent, VIP reclaimable, no storage-level fencing). CAUTION when the mechanism is right in shape but unverified — synchronous replication is configured but the fallback-to-async path has never been confirmed to alert, or fencing exists but has only ever been exercised against a cooperative/reachable old primary, never a partitioned one. SAFE only with evidence for both: a replication mode with a bounded, monitored lag (or true sync) and a demonstrated fencing path that holds even when the old primary is unreachable rather than cleanly shut down, plus some signal path (even a coarse one) for the window if loss ever does occur.

Boundaries

• Not migration-guard's territory — that skill reviews a planned schema change for lock/deploy-order/data-loss hazards before it ships. This skill audits runtime primary failover on an already-stable schema, independent of any migration.
• Not skew-check's territory — that skill is about old and new application code running concurrently during a rolling deploy. This skill is about old and new database primaries both existing briefly during a failover; the hazard and the fix are both at the data-replication layer, not the app-version layer.
• Not residency-guard's territory — that skill checks whether the failover target satisfies a geographic/compliance boundary (is the promoted node in the right region). This skill checks whether failover itself preserves acknowledged writes and prevents split-brain, a durability question that applies identically whether the promoted node is next door or across the planet.
• Doesn't design or stand up an HA topology from scratch (which orchestrator, sync vs. async settings, how many replicas) — it audits whether an existing or planned failover mechanism's replication mode and fencing actually hold under a realistic partition, not which HA product to adopt.
• Can't run a live failover drill itself or measure real production replication lag — it can say what evidence (a chaos-engineering partition test, a prior incident's postmortem, lag metrics captured at a past promotion) would prove the gap closed or open, not generate that evidence directly. Say plainly when a claim ("fencing holds under partition") has no supporting test or incident evidence in front of you, rather than assuming a clean-kill drill covers it.