Concurrency Design

Concurrency is an architectural decision, not a library choice. The model you pick determines your fault boundaries, your communication patterns, your latency profile, and how hard the system is to reason about under load. Choose wrong and no amount of tuning recovers you.

Work through these five questions in order. Each answer constrains the next.

1. Topology: How Do Threads Map to Cores?

Topology is the first decision because it determines whether threads share state at all, and what the cost of cross-thread communication is.

Thread-per-core eliminates lock contention by construction — each shard owns its data exclusively. Cross-shard operations require explicit messages, which is a feature: the communication cost is visible. The weakness is load imbalance: a hot shard can't borrow capacity from an idle one without coordination.

Work-stealing gives you automatic load balancing at the cost of unpredictable latency. A task migrated to another thread loses its cache warmth. Under contention, the steal mechanism itself becomes a bottleneck.

Event loop is simple for I/O-heavy code but doesn't scale to multiple cores without clustering (multiple processes or worker threads). Any CPU-intensive work blocks the loop.

See references/topology.md for decision matrix, trade-off analysis, and worked examples.

2. Unit of Concurrency: What's Your Fault Boundary?

Once you've chosen a topology, decide what the unit of concurrent execution is. This is also your fault boundary — what crashes independently.

Actors / isolates: independent units with private state, communicating only via messages. Process messages sequentially from a mailbox. No shared memory. A crash in one actor doesn't corrupt another's state. Erlang processes, Tina isolates, Pony actors.

OS threads: preemptively scheduled, share address space. Fault boundary is the process, not the thread — a thread crash (SIGSEGV) typically kills the process unless you trap it explicitly.

Green threads / goroutines: M:N scheduling, cooperative or preemptive depending on runtime. Go goroutines are preemptive since Go 1.14. Cheap to create; fault boundary depends on whether the runtime provides supervision (Go does not — a panicking goroutine kills the program unless recovered).

Coroutines: cooperative suspension at explicit yield points. No hidden preemption. The "colored function" problem (async/await propagation) is a symptom of coroutines leaking into the type system.

State machine encoding: instead of suspending mid-function, encode progress as an explicit state enum. The scheduler calls back with the next event; the state machine transitions and returns. No hidden stack frames, no colored functions, no implicit suspension points. This is the preferred pattern when you want coroutine-like behavior without the runtime complexity.

See references/actors-and-state-machines.md for the actor model, state machine encoding, cooperative scheduling via effects, and comparisons with CSP channels and async/await.

3. Communication: How Do Units Exchange Data?

Default to message passing. It makes data ownership explicit and eliminates data races by construction. Shared memory is appropriate when the shared data is small, access is infrequent, and the contention is understood.

Lock-free structures are not simpler than locks — they're harder to get right and harder to reason about. Use them on hot paths where profiling shows lock contention is the bottleneck, and only with well-tested implementations (LMAX Disruptor, Rigtorp SPSC, Vyukov MPMC). Rolling your own is a bug.

Channels provide backpressure when bounded. A full channel blocks the producer, which propagates pressure upstream. This is a feature. Unbounded queues hide backpressure and cause unbounded memory growth under load.

See references/lock-free-structures.md for SPSC ring buffer implementation details (Rigtorp cached-index optimization, Snellman unmasked indices), MPMC patterns (LMAX Disruptor, Vyukov), and when to use each.

4. Fault Isolation: What Crashes Independently?

Design for failure before designing for success. The question is not "how do I prevent crashes?" but "when this unit crashes, what else goes down with it?"

Architectural constraint: every unit of concurrency should be cheap to restart, own isolated state, and sit under a supervisor that decides whether to restart it, escalate, or shut down.

Supervision strategies (Erlang/OTP model):

• One-for-one: restart only the crashed child. Use when children are independent.
• One-for-all: restart all children when one crashes. Use when children share invariants.
• Rest-for-one: restart the crashed child and all children started after it. Use for ordered pipelines.

Recovery budgets: rate-limit restarts. After N crashes in T seconds, escalate to the next supervision level. This prevents infinite crash loops from consuming all resources.

"Let it crash" is a design philosophy, not an excuse for sloppy code. It works only when: (1) restart is cheap and fast, (2) state is isolated and recoverable, (3) a supervisor is watching. Without all three, "let it crash" is just "let it fail silently."

See references/fault-isolation.md for trap boundaries, supervision trees, recovery budgets, guard pages, and designing for simulation testability.

5. Message Patterns: How Do Units Coordinate?

Within your chosen communication mechanism, pick the right interaction pattern for each operation.

Every request-reply must have a mandatory timeout. A missing timeout converts a transient failure into a permanent hang. The timeout is not an optimization — it is a correctness requirement.

Separate control plane from data plane. Control messages (configuration changes, shutdown signals, health checks) should travel on a separate channel from data messages. A congested data plane must not block control messages. This is how you drain a system gracefully under load.

Backpressure propagates upstream. When a downstream consumer is slow, the pressure should flow back to the producer, not accumulate in an unbounded buffer. Bounded channels, bounded mailboxes, and explicit flow control are the mechanisms. Design for this from the start — retrofitting backpressure is painful.

Worked Example: Go Pipeline with Fault Isolation

A Kafka→Postgres pipeline — four stages, buffered channels, worker pool. This example shows how the five decisions interact.

Topology: work-stealing (Go's M:N scheduler). Correct for mixed I/O + CPU with variable task sizes. No topology change needed.

Fault boundary: goroutines have no fault boundary by default — an unrecovered panic kills the process. Fix: defer/recover in every worker goroutine, supervised by a goroutine that restarts crashed workers.

// Worker with trap boundary
func runWorker(id int, in <-chan Event, out chan<- Result, faults chan<- int) {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("worker %d panic: %v\n%s", id, r, debug.Stack())
            faults <- id  // signal supervisor to restart
        }
    }()
    for event := range in {
        out <- process(event)
    }
}

// Supervisor: one-for-one restart with recovery budget
// one-for-one because workers are independent — a crash in one doesn't
// affect the others. Budget: 5 restarts in 30s; exceed → escalate.
func supervise(ctx context.Context, count int, in <-chan Event, out chan<- Result) {
    faults := make(chan int, count)
    history := make([][]time.Time, count)
    for i := range count { go runWorker(i, in, out, faults) }
    for {
        select {
        case <-ctx.Done(): return
        case id := <-faults:
            if exceedsBudget(history, id, 5, 30*time.Second) {
                cancelPipeline()  // escalate: budget exceeded
                return
            }
            time.AfterFunc(backoff(history[id]), func() {
                go runWorker(id, in, out, faults)
            })
        }
    }
}

Communication: CSP channels. Bounded (capacity 1000) — backpressure propagates upstream automatically: slow Postgres → aggregator blocks → workers block → Kafka consumer stops reading. This is correct; don't fight it.

Supervision strategy: one-for-one — workers are independent. If the aggregator and writer shared invariants, use one-for-all.

Control vs data plane: shutdown signal (ctx.Done()) travels on a separate path from data channels. Every stage has a select watching both. A congested data channel must not block a shutdown signal.

// Each stage: ctx.Done() is a separate select arm, not in the data path
for {
    select {
    case <-ctx.Done(): flush(); return   // control plane
    case event := <-in: process(event)  // data plane
    }
}

Graceful drain: close channels in pipeline order. Consumer closes its output when done → workers drain → aggregator closes its output → writer drains. sync.WaitGroup lets main wait for complete drain.

Decision Checklist

1. Topology first: thread-per-core, work-stealing, event loop, or thread pool?
2. Fault boundary: what is the unit that crashes independently?
3. Communication: message passing, channels, shared memory, or lock-free?
4. Supervision: who restarts what, under what conditions?
   • One-for-one: independent workers. One-for-all: shared invariants. Rest-for-one: ordered pipeline.
5. Message patterns: fire-and-forget, request-reply (with timeout!), pub/sub, scatter-gather?
6. Control vs data plane: are they separated? Control signals must not be blocked by data congestion.
7. Backpressure: does pressure propagate upstream, or does it accumulate?