Rate Limiting Design

A rate limiter is an admission-control system that trades rejected requests for system survival. Its job is to fail a few requests cheaply so it doesn't fail all requests expensively.

Decide what you are actually protecting

1. The protected resource: CPU, a downstream dependency, a database connection pool, money. The limit must map to the resource's true capacity, not a round number.
2. The unit of fairness: per-user, per-IP, per-API-key, per-tenant, global. The wrong unit lets one tenant starve the rest (noisy neighbor) or punishes shared NATs.
3. The dimension: requests/sec, concurrent requests, bytes, or cost-weighted (a search query ≠ a health check).

Algorithm choice and its behavior

• Token bucket: allows bursts up to bucket size, smooths to refill rate. Best general default.
• Leaky bucket: constant output rate, no bursts. Good for protecting a fixed-throughput downstream.
• Fixed window: simple but has a 2x burst at window boundaries — a real exploit.
• Sliding window log/counter: accurate, more memory. Use when boundary bursts matter.
• Concurrency limit (semaphore): often better than rate for protecting against slow requests — caps in-flight work, which is what actually exhausts resources.

Distributed enforcement

• Per-node limits don't sum to a global limit; N nodes each allowing R gives N×R. Use a shared store (Redis) with atomic ops, or allocate quota slices per node with periodic rebalancing.
• The limiter store itself must not become the bottleneck or SPOF — local fallback (fail-open vs fail-closed) is a deliberate choice with opposite risks.

Adversarial validation

• Thundering retry: rejected clients retry immediately and in sync, turning a limit into a self-amplifying storm. Return Retry-After and require backoff + jitter; without it the limiter increases load.
• Limiter as SPOF: if the rate-limit store is down, do you fail open (lose protection) or closed (lose availability)? Decide explicitly per endpoint.
• Cost asymmetry: an attacker sends the most expensive requests under the limit. Cost-weight the limit, don't count requests.
• Boundary burst: fixed window lets 2x through at the edge — can that 2x kill the downstream?

Non-obvious risks

• Rate limiting upstream of a queue just moves the overload into the queue (bufferbloat) — limit at admission, shed early.
• Returning 429 still costs work (auth, parsing); for true overload you need load shedding that drops cheaply before expensive work.
• Graceful degradation (serve stale/partial) often beats hard rejection for user experience under load.

Output structure

Deliver: the protected resource and its real capacity, the fairness unit and cost dimension, the algorithm with burst behavior, the distributed-enforcement mechanism with fail-open/closed decision, the client-backoff contract (Retry-After), and the load-shedding path for true overload.