Error Handling

The failure mode: error handling grows call-site by call-site — a try/catch here because a test flaked, a retry there because prod paged — until the module has five inconsistent policies and no strategy. Errors are a design surface, same as the API. The gate: you may not write a catch/rescue/match-on-Err block until every failure source in the diff is classified, and each class has exactly one designated handling pattern.

Phase 1 — Inventory and classify (gate)

Walk the diff (or module). List every operation that can fail: I/O, network calls, parsing, arithmetic on external data, lock acquisition, subprocess exits. Classify each into exactly one of four classes and write the table down before any handler code:

FAILURE INVENTORY — payment_worker.py
| Source                          | Class                | Policy (from taxonomy) |
| json.loads(webhook_body)        | invalid-input        | reject-with-identity   |
| stripe.Charge.create(...)       | retryable-transient  | retry-then-surface     |
| stripe: card_declined           | permanent-external   | surface, no retry      |
| db.save(receipt)                | retryable-transient  | retry-then-crash       |
| KeyError on our own config dict | bug                  | crash loudly           |

If two sources in the same class get different treatment, either the classification is wrong or you're inventing policy at the call site — both are gate failures.

Taxonomy — per-class policy

The classes are about what the correct reaction is, not where the error came from. The same Stripe SDK exception type may map to two classes (429 → transient, card_declined → permanent); classify by semantics, not by exception type alone.

Per-ecosystem idiom table

Phase 2 — Boundary placement

Handlers live at boundaries; interior code propagates. Deciding where each class's designated pattern executes is half the design:

Two rules fall out of this table. First, log at the edge, wrap in the interior — if every layer logs, one failure produces five log entries and the on-call engineer can't tell one incident from five. A catch block that logs and rethrows without adding new context is noise; either add the identity this layer knows (which invoice, which attempt, which shard) or don't touch the error. Second, retry in exactly one layer — the adapter that owns the idempotency mechanism — because stacked retry loops multiply (see cookbook).

Timeouts are part of placement: every network call gets an explicit deadline (no library default is trustworthy — many are infinite), and the timeout value must be shorter at inner layers than outer ones (inner 2s, edge 10s), otherwise the edge gives up while the interior is still retrying and the work is wasted or, worse, completes unobserved. A timeout is the canonical producer of the retryable-transient class — but remember it means "unknown outcome," not "didn't happen."

Retry cookbook (retryable-transient class only)

• Formula: delay = min(cap, base * 2^attempt) * random(0.5, 1.0) — full-ish jitter. Typical: base=200ms, cap=30s, attempts=4. Without jitter, every client that failed together retries together (thundering herd).
• Idempotency precondition — non-negotiable: retry only operations that are idempotent by nature (GET, PUT-with-full-body) or made so (POST with an Idempotency-Key, INSERT with a unique constraint + ON CONFLICT). A timeout does NOT mean the operation didn't happen; retrying a non-idempotent charge after a timeout is how customers get billed twice.
• Retry budget: cap retries as a fraction of total traffic (e.g., retries ≤ 10% of requests, circuit-break past that). Per-request retries multiply across layers: client×3 → gateway×3 → service×3 = 27 attempts hitting a struggling database.
• When retrying makes it worse: the dependency is overloaded (your retries are the load); the error is permanent-external misclassified as transient; you're inside a caller that also retries (pick ONE layer to own retries — usually the outermost that has the idempotency key).

Reference shape (the parameters above, in ~10 lines — adapt, don't import a framework for this):

def with_retry(op, retry_on, no_retry=(), attempts=4, base=0.2, cap=30.0):
    for i in range(attempts):
        try:
            return op()
        except no_retry:
            raise                                   # permanent-external: out immediately
        except retry_on as e:
            if i == attempts - 1:
                raise RetryExhausted(attempts=attempts) from e
            time.sleep(min(cap, base * 2**i) * random.uniform(0.5, 1.0))

Note what is deliberately absent: no retry on bare Exception (bug-class must escape on attempt one, not after 4 sleeps), and no logging inside the loop — the edge logs once, with attempts attached via the exception chain.

Edge translation table — the taxonomy class fixes the external representation; write it once per service and stop deciding per-endpoint:

The queue column encodes the least-obvious rule: invalid-input must not be retried by redelivery — a poison message nacked in a loop burns the consumer forever. Classification is what tells the consumer to DLQ it on the first attempt.

Error message content checklist

Every surfaced error and every log line at the edge carries, checkably:

• Identity of the failing input: the id, key, row number, or a safe prefix of the payload — never just the operation name
• The expectation that broke: "expected ISO-8601 date, got '2026-13-01'" — rule + observed value
• The taxonomy class's dispatch info: retryable or not; for user-facing errors, what the user can do about it
• Correlation: request/trace id so the edge log line joins to the interior wrap chain
• Nothing secret: no tokens, no full card numbers, no raw payload dumps of PII — identity means reference, not contents

Phase 3 — Worked example

Before — one function, three failure modes, zero strategy:

def process_webhook(body: bytes):
    try:
        event = json.loads(body)                     # invalid-input
        charge = stripe.Charge.create(               # transient OR permanent
            amount=event["amount"], idempotency_key=event["id"])
        db.save_receipt(charge)                      # transient
    except Exception as e:
        logger.error(f"webhook failed: {e}")         # which webhook? retriable?
        return {"ok": False}                         # caller can't tell garbage from outage

Every real problem is here: catch-log-continue means a DB blip silently drops a paid charge's receipt (money moved, no record — corrupted state); the caller gets the same {"ok": False} for a malformed body (never retry) and a Stripe timeout (must retry); the log line has no event id.

After — one pattern per class, boundary translation at the top:

def process_webhook(body: bytes) -> WebhookResult:
    try:
        event = parse_event(body)          # raises InvalidWebhook(payload_prefix=body[:64])
    except InvalidWebhook as e:
        return WebhookResult.reject(e)     # invalid-input: 400, no retry, identity attached

    charge = with_retry(                   # transient: backoff+jitter, budget=4
        lambda: stripe.Charge.create(amount=event.amount,
                                     idempotency_key=event.id),  # retry precondition
        retry_on=(RateLimitError, APIConnectionError),
        no_retry=(CardError,))             # permanent-external falls through
    # CardError propagates to the boundary handler → domain error, no retry, not paged

    with_retry(lambda: db.save_receipt(charge), retry_on=(OperationalError,),
               on_exhaust=raise_)          # transient→bug promotion: charge exists,
                                           # receipt doesn't → crash; the queue redelivers
                                           # and idempotency_key makes redelivery safe
    return WebhookResult.ok(event.id)

Note the last line's design move: "retries exhausted after money moved" is promoted to crash because continuing would run with corrupted state, and crash+redeliver is safe only because the idempotency key exists. That coupling is the strategy.

Output contract

Deliver: (1) the failure-inventory table, (2) the per-class pattern chosen (with ecosystem idiom named), (3) the retry parameters and their idempotency justification, (4) the code, (5) one sentence per catch block saying which class it handles. A catch block you can't assign to a class gets deleted.

Anti-patterns (hard no)

• Catch-log-continue past a failure that leaves state half-written. Log-and-continue is only legal when the operation is provably complete-or-untouched.
• Retrying any operation without stating why the retry is safe (idempotent by nature, or key/constraint shown).
• catch (e) { log("operation failed"); rethrow e; } — log-and-rethrow without adding context (which invoice? which attempt?) guarantees the same error is logged N times with no new information. Add context or don't touch it.
• Error messages/log lines without the failing input's identity. "Invalid date format" is useless; "invalid date '2026-13-01' in row 4823 of upload_7Kx.csv" is actionable.
• One generic AppError/CustomException for everything — callers are forced to string-match messages, which is the same as no error types.
• Designing handlers before the inventory table exists.