Networking & Transport - Architecture Patterns

The transport-and-wire-format layer underneath API design. The API design skill (${HANDBOOK_ROOT}/skills/api-design/SKILL.md) and rule (${HANDBOOK_ROOT}/rules/320-api-design.mdc) cover the API surface. This skill covers TCP, HTTP/n, QUIC, serialization, connection lifecycle, head-of-line blocking, and TTFB budgets.

Get this layer wrong and your P99 latency, mobile UX, edge cold-start cost, and east-west microservice throughput all suffer in ways that profiling your application code can't fix.

When to invoke

• Designing or reviewing a service-to-service API (REST vs gRPC choice)
• Debugging tail latency / TTFB spikes / "fast in isolation, slow in production"
• Configuring load balancer idle timeouts (NLB, ALB, API Gateway, Cloudflare)
• Designing long-lived connections (websockets, SSE, gRPC streaming)
• Evaluating HTTP/3 / QUIC for mobile / cross-region / lossy-network workloads
• Choosing a wire format (JSON vs Protobuf vs MessagePack vs Avro)
• Reviewing a .proto file or a protobuf schema change

Five golden rules

1. TCP keepalive is mandatory on long-lived clients. Linux defaults (tcp_keepalive_time = 7200s = 2 hours) are wrong for cloud. Set explicit per-language values.
2. Reuse HTTP clients. A fresh client per request defeats keepalive and burns ~50-150ms on TCP+TLS handshake. The single most common production performance bug.
3. Choose the wire by callers, not by hype. External / browser / human-debugged -> REST/JSON. Internal east-west / polyglot / streaming -> gRPC/Protobuf. Don't gRPC your public API just because it's faster.
4. HTTP/2 multiplexes streams; HTTP/3 multiplexes connections. HTTP/2 still has TCP-level head-of-line. HTTP/3 fixes it. Matters when packet loss is non-zero (mobile, satellite, cross-region).
5. Idle timeouts kill connections silently. AWS NLB drops at 350s, ALB at 60s, API Gateway REST at 29s. Set keepalive < min(idle timeout) or watch your "random" disconnects.

1. Mental model: four layers, four latency budgets

+----------------------------------------+
|  L7 wire format    JSON / Protobuf     |  Serialization CPU + size on wire
+----------------------------------------+
|  L7 protocol       HTTP/1.1 /2 /3      |  Multiplexing + HoL behavior
+----------------------------------------+
|  L6 transport sec  TLS 1.2 /1.3        |  Handshake RTTs (1-RTT, 0-RTT, session resume)
+----------------------------------------+
|  L4 transport      TCP / QUIC (UDP)    |  Handshake, congestion control, HoL
+----------------------------------------+
|  L3 IP             routing / DNS       |  DNS lookup, geo distance, MTU
+----------------------------------------+

Latency tuning at each layer:

2. TCP keepalive

What it is

OS-level periodic probes on idle TCP connections to detect dead peers. Without keepalive, a connection in a half-open state (peer died, NAT timed out, LB recycled) appears alive until the next read/write attempt fails.

Why defaults are wrong

What to set per environment

Per-language settings

// Go: customize the Transport's DialContext
tr := &http.Transport{
    DialContext: (&net.Dialer{
        Timeout:   5 * time.Second,
        KeepAlive: 30 * time.Second,    // probe interval
    }).DialContext,
    IdleConnTimeout:       90 * time.Second,
    MaxIdleConnsPerHost:   100,
    DisableKeepAlives:     false,
}
client := &http.Client{Transport: tr}

# Python (requests/httpx) - OS-level keepalive only; tune via Session
import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
session.mount('https://', HTTPAdapter(pool_connections=20, pool_maxsize=100))
# OS-level: sysctl net.ipv4.tcp_keepalive_time=60 (container or host)

// Node: enable keepalive on the Agent (NOT default in older Node versions)
const https = require('https');
const agent = new https.Agent({
  keepAlive: true,
  keepAliveMsecs: 30_000,           // probe interval
  maxSockets: 100,
  timeout: 30_000,
});

Anti-patterns

• Default OS keepalive in cloud (2-hour idle). Connection looks alive; first send fails.
• Creating a new http.Client / requests object per request. No pooled connection, no keepalive, ~50-150ms handshake tax per call.
• Setting keepalive longer than the load balancer's idle timeout. Backwards; LB still wins.

3. TCP head-of-line blocking

The problem at three layers

When this matters

• Mobile networks (packet loss 1-5%): HTTP/2 single connection performs worse than HTTP/1.1 with parallel connections because one lost packet stalls everything.
• Cross-region (especially via lossy intermediate networks): same.
• High-fan-out backends: one slow upstream response on an HTTP/2 connection delays unrelated responses sharing the connection.

When it doesn't matter

• LAN / data-center east-west with <0.001% loss: HTTP/2 multiplexing wins; HoL is theoretical.
• HTTP/1.1 with one request at a time (most internal microservices today): never see it.

4. TTFB (time to first byte) budget

The streaming insight: if your handler takes 800ms to compute a 4KB response, TTFB is 800ms + 1 RTT. If you stream incrementally, TTFB drops to whatever the first chunk takes (often <100ms). Big TTFB win, no algorithmic change required.

5. HTTP/1.1 vs HTTP/2 vs HTTP/3

HTTP/3 prerequisites

• UDP port 443 must be reachable end-to-end. Some corporate proxies, old firewalls, and certain ISP middleboxes drop or rate-limit UDP/443.
• Server advertises via alt-svc header in an HTTP/2 or 1.1 response: alt-svc: h3=":443"; ma=86400.
• Client must support and choose to attempt H3. Browsers do this transparently; SDKs / curl need explicit flags.
• TLS 1.3 mandatory (HTTP/3 only runs over TLS 1.3+).

When HTTP/3 is the right choice

• Public-facing edge traffic for mobile users
• Cross-region / cross-cloud / high-RTT paths
• Edge POPs to mobile clients (Cloudflare, Fastly, CloudFront all support)
• Anywhere packet loss meaningfully affects current HTTP/2 P95/P99

When HTTP/2 is still right

• Internal east-west (low loss, low RTT, no UDP-path concerns)
• gRPC (which requires HTTP/2)
• Anywhere your egress path doesn't reliably pass UDP/443

6. REST/JSON vs gRPC/Protobuf vs GraphQL vs Connect

Decision matrix

See references/grpc-vs-rest-decision.md for the long version with worked examples.

gRPC's real wins

• Schema-first contract. .proto is the single source of truth; client + server generated from it. No "is this field optional?" debates.
• Streaming. Server, client, and bidirectional streams as first-class concepts. Long-lived RPC over a single connection.
• Polyglot code-gen. Same .proto -> Go, Python, Java, TS, Swift, Kotlin, Ruby, C++, Rust. Type-safe in each.
• Smaller wire, faster parse. Protobuf vs JSON: ~3-10x smaller, ~3-5x faster to (de)serialize.
• HTTP/2 native. Multiplexed streams, no per-request connection.

gRPC's real losses

• Browsers don't speak gRPC. Need gRPC-Web + Envoy translation, or use Connect.
• Debuggable from a CLI? Need grpcurl (not curl) and a reflection server or the .proto file.
• Load balancing. gRPC's persistent HTTP/2 connection breaks naive L4 load balancers; need L7 (Envoy, Linkerd, AWS Application LB with gRPC support).
• No browser-friendly caching. REST Cache-Control and CDN integration is a solved problem; gRPC isn't.
• Tail-latency observability. Per-call metrics need extra instrumentation; less out-of-box than HTTP middleware.

Connect (and Twirp): the middle ground

• Protobuf schemas + code-gen (like gRPC)
• HTTP/1.1-friendly (like REST)
• Browser-callable directly with fetch (unlike gRPC)
• Streaming supported (Connect: yes; Twirp: no)
• Less feature-complete than gRPC but covers the common 80%

Anti-pattern: gRPC for the public API

If your callers are external (third-party developers, mobile teams you don't control, browsers): use REST or Connect. gRPC's operational burden on external callers is real.

7. Protobuf design essentials

syntax = "proto3";

package myorg.users.v1;     // semantic-version the package; never break clients of v1

message User {
  string id = 1;            // field numbers are wire identity - never re-number
  string email = 2;
  string display_name = 3;
  reserved 4, 5;            // numbers of removed fields; prevents accidental reuse
  reserved "deprecated_field_name";

  // Presence semantics: in proto3, scalar zero values don't serialize.
  // Use `optional` (proto3 since 2020) or wrapper types for true "absent" semantics.
  optional string phone = 6;  // absent vs empty distinguishable
}

message UpdateUserRequest {
  string id = 1;
  // oneof for mutually-exclusive update modes
  oneof update {
    string new_email = 2;
    string new_display_name = 3;
  }
}

Schema-evolution rules (memorize)

See references/protobuf-schema-evolution.md for the long form with examples.

Wire-format gotchas

• Default values don't appear on the wire (proto3 scalars). Receiver can't distinguish "set to zero" from "absent". Use optional (since 2020) or wrapper types (google.protobuf.StringValue).
• Field numbers 1-15 use 1 byte tag; 16+ use 2. Reserve 1-15 for hot fields.
• Unknown fields are preserved by default in most languages. Don't strip them.
• bytes and string are similar on the wire; bytes skips UTF-8 validation. Use bytes for binary; string for text.

8. Connection pooling per language

The #1 production-performance bug: a fresh HTTP client per request.

Go

// BAD: new Client per request -> no keepalive
func bad(url string) {
    resp, _ := http.Get(url)  // uses http.DefaultClient with default Transport
    // but if you do `http.Client{Transport: &http.Transport{...}}` per call, you lose pooling
}

// GOOD: package-level (or DI'd) client, reused
var client = &http.Client{
    Timeout: 10 * time.Second,
    Transport: &http.Transport{
        MaxIdleConnsPerHost: 100,
        IdleConnTimeout:     90 * time.Second,
        DialContext: (&net.Dialer{
            Timeout:   5 * time.Second,
            KeepAlive: 30 * time.Second,
        }).DialContext,
    },
}

Python

# BAD: requests.get() per call - no session reuse
def bad(url):
    return requests.get(url)

# GOOD: Session pools connections, sends keepalive headers
session = requests.Session()
adapter = HTTPAdapter(pool_connections=20, pool_maxsize=100, max_retries=3)
session.mount('https://', adapter)
session.mount('http://', adapter)

Node

// BAD: implicit Agent per call (older Node)
const res = await fetch(url);

// GOOD: shared keepalive agent (Node 19+ defaults to keepAlive: true; older versions don't)
const https = require('https');
const agent = new https.Agent({ keepAlive: true, maxSockets: 100, timeout: 30_000 });
const res = await fetch(url, { agent });

gRPC clients

# Single channel reused across calls (NOT a channel per call)
channel = grpc.insecure_channel('service:50051', options=[
    ('grpc.keepalive_time_ms', 30_000),
    ('grpc.keepalive_timeout_ms', 5_000),
    ('grpc.keepalive_permit_without_calls', True),
    ('grpc.http2.max_pings_without_data', 0),
])
stub = MyServiceStub(channel)

A gRPC channel is itself a connection pool; create one per (service, credentials) pair and reuse for the lifetime of the process.

9. Long-lived connections

Idle-timeout cheat sheet

Application-level heartbeats (every 30-60s) are the only reliable way to defeat idle-timeout disconnects across this matrix.

10. Edge / CDN patterns

• Cold POPs are real: first request to a POP that hasn't seen your origin pays a full handshake. Subsequent requests reuse the warm connection.
• CDN-to-origin keepalive is on by default in Cloudflare, Fastly, CloudFront. Verify and tune origin-side keepalive timeouts to match (origin must not drop before CDN reuses).
• Anycast routing brings the connection to the nearest POP; routes between POPs and origin use the CDN's own backbone (faster than the public internet for cross-region).
• HTTP/3 from edge to client is supported by all major CDNs; origin-to-edge usually remains HTTP/2.
• Regional fall-through: if a POP can't reach origin, CDN may try other regions. Configure origin allow-lists or auth that work from any POP.

11. Observability for the transport layer

Metrics to capture per outbound HTTP/gRPC client:

For deeper observability patterns see rules/330-observability.mdc.

12. Anti-patterns

1. New HTTP client per request. Defeats connection reuse, burns ~50-150ms / call on handshake. #1 production performance bug.
2. Default OS TCP keepalive in cloud. 2-hour idle means connections look alive long after LB cut them; first write fails.
3. HTTP/2 single connection across mobile / lossy networks. TCP HoL stalls all streams; HTTP/1.1 parallel or HTTP/3 wins.
4. gRPC for a browser-facing API without gRPC-Web/Connect. Doesn't work; teams discover too late.
5. Re-numbering a protobuf field. Wire-incompatible; existing data and clients corrupt.
6. proto3 scalar field for true "optional" semantics. Default-zero-doesn't-serialize means receiver can't distinguish; use optional (since 2020) or wrappers.
7. Buffering then sending. TTFB blows up; stream incrementally where possible.
8. Keepalive longer than LB idle timeout. LB wins; "random" disconnects appear in P99.
9. One TCP connection per RPC in a high-call-rate service. Channel-per-call instead of channel-reuse is gRPC's equivalent of #1.
10. Assuming HTTP/3 works without verifying UDP/443. Corporate proxies and some ISPs filter; fall back gracefully.

13. Reviewer checklist

For any PR introducing or modifying an HTTP / gRPC client:

• Client is created once (package-level, DI, singleton) and reused
• Explicit timeout set (overall request, not just connect)
• TCP keepalive interval set < load balancer idle timeout
• MaxIdleConnsPerHost / pool size sized to the workload (default is often 2)
• TLS minimum is 1.2; prefer 1.3
• Retry policy bounded (max attempts, max total time, backoff with jitter)
• Metrics: latency histogram, status counter, connection-reuse ratio
• If gRPC: channel reused; keepalive options set; permit_without_calls if appropriate

For any PR adding or changing a .proto:

• No field re-numbered
• Removed fields have reserved for both number and name
• New fields use unused numbers (never reused)
• Optional semantics: optional keyword (proto3 since 2020) or wrapper types
• Package versioned in the namespace (myorg.users.v1)
• Generated code committed or generated in CI from the .proto

References

• references/grpc-vs-rest-decision.md - Decision matrix with worked examples
• references/protobuf-schema-evolution.md - Schema evolution rules, presence semantics, the field-renumber hazard

Related

• API design: ${HANDBOOK_ROOT}/skills/api-design/SKILL.md and ${HANDBOOK_ROOT}/rules/320-api-design.mdc - the contract layer above this skill
• Rule: 316-zero-trust.mdc - Network section (mTLS, default-deny egress) - the security model on top of this transport
• Rule: 330-observability.mdc - Logging, metrics, tracing - including the transport-layer metrics above
• Rule: 400-cloudflare.mdc - CDN patterns, Workers, edge-to-origin connection lifecycle
• Rule: 410-aws.mdc - AWS ALB/NLB/API Gateway idle-timeout specifics
• Rule: 483-kafka.mdc - Kafka transport + Avro/Protobuf schema registry
• Rule: 510-mcp-servers.mdc - MCP protocol patterns
• Skill: zero-trust - Network-tier security pairing
• Skill: cloud-platforms - Per-cloud LB and edge specifics