Distributed Consensus Design

Consensus is expensive; most systems pay for it without needing it. The skill is identifying the minimum coordination required, because every quorum round-trip is latency you cannot remove.

Start from the impossibility results, not the frameworks

• FLP: no deterministic consensus with even one faulty node in a fully async system. Real systems escape via timeouts/randomization — meaning your liveness depends on timing assumptions that will be violated.
• CAP is a false trilemma at runtime: during a partition you choose C or A; with no partition you have both. The real axis is PACELC — Else, Latency vs Consistency — which governs the 99.9% of time with no partition.
• Coordination avoidance: if operations commute or can be made conflict-free (CRDTs), you may need no consensus at all. Check this first.

Pick the weakest sufficient primitive

Ordered from cheap to expensive — stop at the first that meets the requirement:

1. No coordination — immutable data, idempotent writes, CRDTs, last-writer-wins where safe.
2. Causal consistency — preserves cause→effect ordering; available under partition.
3. Single-leader replication — total order via one writer; cheap reads, failover cost on leader loss.
4. Quorum consensus (Raft/Paxos) — linearizable; survives minority failure; pays 1+ round-trips per decision.
5. Atomic commit (2PC/3PC) — cross-shard atomicity; blocks on coordinator failure (2PC) — usually wrong for high availability.

Adversarial validation

• Network is unreliable: messages drop, duplicate, reorder, and arrive arbitrarily late. Design for "delayed", not "lost" — a late message can arrive after you assumed it dead.
• Split brain: two nodes both believe they're leader. Does a stale leader's write get accepted? Fencing tokens / epoch numbers are mandatory, not optional.
• Clock skew: never order events by wall clock across nodes. Use logical/hybrid logical clocks; "synchronized NTP" drifts and jumps.
• The exactly-once myth: networks give at-most-once or at-least-once. "Exactly-once" only exists as at-least-once delivery + idempotent processing. Design idempotency keys, not magic.

Non-obvious risks

• A distributed lock without a fencing token does not provide mutual exclusion — the lock holder can pause (GC, VM migration) past the lease and corrupt state.
• Quorum reads/writes (W+R>N) give freshness but not linearizability without read-repair/leases.
• Adding nodes increases availability of reads but the coordination cost and failure surface of writes.

Output structure

Deliver: the exact consistency requirement (per operation, not per system), the weakest primitive that satisfies it with justification, the partition behavior (what's sacrificed and when), the fencing/idempotency mechanism, and the failure scenario (leader loss, partition, slow node) walked through to a safe outcome.