Capacity & Scalability Planning

Scalability is not a property you add; it's the absence of a bottleneck you haven't hit yet. Find the next limit before traffic does, and put a number on it.

Build a load model, not a vibe

1. Characterize demand: peak vs average (the ratio is what matters), arrival pattern (steady, bursty, diurnal, thundering-herd at the top of the hour), and growth rate.
2. Map demand to resources: requests/sec → CPU, memory, IOPS, connections, bandwidth, and downstream calls (fan-out multiplies). Find the resource that saturates first.
3. Apply the laws:
   • Little's Law (L = λW): concurrency = arrival rate × latency. If latency rises under load, required concurrency explodes non-linearly.
   • Universal Scalability Law: throughput gains decay from contention (serialization) and reverse from coherence (cross-node coordination). More nodes can mean less throughput.
   • Queueing theory: utilization above ~70-80% makes latency rise hyperbolically. Plan headroom; a system run at 95% util is already in latency collapse.

Scale-up vs scale-out

• Up (vertical): simpler, preserves locality and strong consistency, hits a hard ceiling, single failure domain.
• Out (horizontal): higher ceiling, but introduces coordination cost, partial failure, and data partitioning problems. Only scales if the workload shards cleanly (stateless or partitionable).
• The real question: what is the shard key? If there isn't a clean one, scale-out adds coordination cost without throughput.

Adversarial validation

• The headroom lie: average utilization is 40% — but peak is 95% for the 15 minutes that matter. Plan for peak-of-peak, not the daily mean.
• Correlated load: autoscaling assumes independent demand; a retry storm or cache flush spikes every instance simultaneously, and scale-up takes minutes you don't have.
• The shared bottleneck: you scaled the app tier, but every instance hits the same database/lock/license/quota. Scaling the wrong tier wastes money and moves nothing.
• Cold scale-out: new instances start cold (empty cache, cold JIT, connection warmup) and can be slower than the load they were added to relieve.

Non-obvious risks

• Autoscaling reacts to load it already failed to serve; for sharp spikes you need pre-provisioning or admission control, not reactive scaling.
• Connection pools, file descriptors, ephemeral ports, and thread pools are silent ceilings that cap throughput far below CPU/memory.
• Scaling out increases the probability that something is always failing (more nodes = more partial failure); the system must tolerate it.

Output structure

Deliver: the load model (peak/avg, growth, pattern), the resource that saturates first with the math, the scale-up vs out decision with the shard key, the headroom target and the utilization at which latency collapses, and the specific next bottleneck that appears after this scaling step.