Error Handling Architecture

Errors are control flow you forgot to design. A system's behavior under failure is its real behavior; the happy path is the rare case at scale.

Classify before you handle

Every failure falls into one of three buckets — they demand opposite responses:

1. Bugs (programmer errors): broken invariants, nulls that "can't happen". Fail fast and loud; do not catch-and-continue. Recovering from a bug corrupts state.
2. Expected operational faults: network blips, contention, rate limits, transient unavailability. Anticipate, retry with bounds, degrade gracefully.
3. Domain outcomes: "insufficient funds", "not found". These are not errors — model them as ordinary return values, not exceptions.

Conflating these is the root cause of most bad error handling: catching bugs as if they were faults, or modeling business outcomes as exceptions.

Propagation strategy

• Decide where errors are handled vs propagated. Handle at the layer with enough context to act; propagate (with added context) everywhere else. Don't catch what you can't meaningfully handle.
• Add context on the way up (what operation, which inputs) without leaking internals to untrusted callers.
• Pick one mechanism per layer (exceptions vs result types) and convert deliberately at boundaries; mixing silently drops errors.

Recovery primitives — and their failure modes

• Timeout: every remote call needs one; an unbounded wait is a liveness bug. Tune to the dependency's p99, not its mean.
• Retry: only for idempotent ops; use exponential backoff + jitter. Naive retries cause retry storms that amplify the outage.
• Circuit breaker: stop hammering a dead dependency; trade freshness for system survival.
• Bulkhead: isolate resource pools so one failing dependency can't exhaust threads for all.
• Fallback / degrade: define the reduced-but-correct mode (stale cache, default, queue-for-later).

Adversarial validation

• Trace one failure end to end: does it surface, get swallowed, or cascade? Swallowed errors are the worst — they corrupt silently.
• Does a retry policy multiplied across N callers DDoS your own dependency?
• On partial failure of a multi-step operation, is the system left consistent or half-written? (Needs compensation/saga or atomicity.)
• Does the error path allocate, lock, or call the failing dependency again? Error handling must be cheaper and more robust than the happy path.

Non-obvious risks

• Logging-and-rethrow at every layer produces duplicate noise and hides the real handler.
• Catch-all handlers turn crash-worthy bugs into silent data corruption that surfaces days later.
• Fallbacks can mask outages until the fallback itself fails under load with no headroom left.

Output structure

Deliver: the three-way classification applied to the actual failures, the propagation boundaries, the recovery primitives with their tuning and idempotency constraints, and a traced example proving a representative failure ends in a defined, consistent state.