Rust Networking

Apply this policy to HTTP, gRPC, WebSocket, and custom TCP clients and servers. Use the timeout, retry, pool, body-limit, and shutdown controls of the stack that the workspace already uses. Do not replace the framework to get one middleware type.

Use these skills, when they are installed, for adjacent work: rust-async-internals for select!, task ownership, cancel safety, and runtime configuration; rust-observability for tracing, metrics, and exporters; rust-security for dependency advisories and untrusted-input parser hardening.

Start from one operation contract

Record these decisions next to the client or server configuration before you tune individual settings:

Do not copy timeout values from another service. Derive them from the caller's deadline and the service latency objective. Keep all values configurable. Give each value a finite production default when the caller does not supply one.

Set the limits the stack leaves open

Most Rust network stacks ship without the limits in the contract. An unset limit compiles and passes tests, then fails under a slow or hostile peer. Set each one explicitly:

Read references/stack-defaults.md when the workspace uses one of these crates: it has the reqwest 0.13 upgrade changes, working client and server setups, and the rustls and Android TLS setup.

Verify the configuration

Run these checks when you add or review network configuration. Review each hit; a hit is not always a defect.

rg -n --type rust 'danger_accept_invalid|tls_danger_|\.dangerous\(\)|tls_sslkeylogfile\(true\)|KeyLogFile|DefaultBodyLimit::disable|\bTimeoutLayer::new|axum::serve\(|with_graceful_shutdown\(|\breqwest::get\(|\bClient(Builder)?::(new|builder)\(\)|\.(bytes|text|json)\(\)\.await|\.json::<'
cargo tree --locked -e features -i rustls --prefix none | grep -oE '^rustls feature "(ring|aws_lc_rs|custom-provider)"' | sort -u
cargo deny --config deny.toml --locked check advisories

A green grep does not prove the timing contract; the tests in "Test deterministic seams" do. The work is complete when each contract field and each stack-defaults row has a finite, configurable, tested value, and the TLS root source, the pool key, the outcome-unknown error state, and the diagnostics field list match their sections. In a review, each gap is a finding.

Failure triage

Keep TLS verification on

Use the TLS backend that the workspace already selected. Apply these rules:

• Verify the certificate chain and the server name on every production connection.
• Load trust roots from one explicit source. Choose the platform store when the product must honor managed enterprise roots. Choose a bundled store when the product requires the same roots on every target. A reqwest 0.12 rustls-tls build (bundled webpki roots) silently moves to the platform verifier on 0.13; call tls_certs_only(roots) to keep a bundled or private set.
• Fail startup or client construction when the required root store is empty.
• Keep certificate and hostname bypass APIs (the rg check above lists them) out of production code paths. A test verifier must not be reachable from runtime configuration.
• Send SNI for a DNS name. Verify the original service name, not the resolved IP address.
• Keep private keys out of logs and error chains. Never enable key logging in production: reqwest tls_sslkeylogfile(true) and rustls KeyLogFile write session secrets to the file that SSLKEYLOGFILE names.
• Rebuild the client and drain old pooled connections after a trust-root, client-certificate, or private-key rotation.
• Use certificate pinning only when the product has a rotation and recovery plan. Ship at least one backup identity before the active identity changes.

Do not force a protocol-version policy that conflicts with the platform or the service contract. Use the secure defaults of the maintained TLS backend. Raise the minimum only when the deployment matrix proves that every peer supports it.

Use one absolute deadline

Create one monotonic deadline at the operation boundary. Carry it through DNS, connect, proxy negotiation, TLS, every request attempt, response streaming, and retry delay. Do not restart the full timeout after a retry.

Before each phase, calculate this duration:

phase allowance = min(configured phase cap, operation deadline - monotonic now)

Fail before the phase starts when no time remains. Report which phase consumed the budget. Do not report every deadline failure as connect timeout. Read references/deadlines-and-retries.md when you set the phase caps: it gives the start, the end, and the failure mode of each cap.

On a server, start the header-read deadline when the connection is accepted. A slow client must not hold an accepted-connection slot without completing a bounded header.

An idle timeout is not a total transfer timeout. Reset it only after useful progress. A peer that sends one byte before every idle timeout can still consume the operation deadline, so enforce both. With tower-http, pair RequestBodyTimeoutLayer (idle) with RequestBodyDeadlineLayer (total, tower-http 0.7+). With reqwest, pass the remaining time to RequestBuilder::timeout on every attempt.

Use tokio::time::timeout_at or the runtime equivalent when the stack does not accept a deadline directly. Dropping the timed future cancels it, so check its cancel safety.

Retry only a safe operation

Retry only when all three conditions hold:

1. The operation is idempotent by protocol semantics or by an application idempotency key that the server stores atomically.
2. The complete request can be replayed byte for byte. A consumed stream is not replayable unless the caller can open a fresh stream.
3. The failure is transient and the operation deadline can hold another attempt plus its delay.

HTTP GET, HEAD, PUT, DELETE, OPTIONS, and TRACE have idempotent method semantics. Application behavior can still make an operation unsafe to repeat. Do not infer replay safety from the method name alone. Do not retry POST unless the application contract provides idempotency or proves that the server did not apply the request.

Use this default classification, then narrow it for the target service:

Do not retry every 5xx. A retry can multiply overload and repeat an application failure. Keep the retry status set explicit.

Count the first request as attempt one. Use a small finite attempt limit. Cap retries again with the operation deadline. Never retry forever in a background task.

Also bound retries per upstream across the process. Require a retry permit or token before a second attempt. Limit retry concurrency, and suppress retries when the local service is overloaded or the upstream failure rate crosses the configured threshold. If no retry budget remains, return the original failure instead of adding outage load. tower::retry::budget::TpsBudget implements such a budget for a tower stack.

Use exponential backoff with full jitter. Let base be the first cap and maximum be the largest cap. For retry number n, starting at zero, select a uniform delay in 0 ..= min(maximum, base * 2^n).

Parse both Retry-After forms: delay seconds and HTTP date. Reject invalid or negative values. Use the larger of the server delay and the local jitter delay. Do not sleep past the operation deadline. Clamp arithmetic to prevent overflow.

Keep delay calculation pure. Pass the random sample and current time into it. This seam makes retry tests deterministic without a mock HTTP framework. Read references/deadlines-and-retries.md when you write the delay function: it has a tested implementation.

Stream with hard limits and backpressure

Reject a declared body length that exceeds the route limit before allocation. Do not trust the declaration as the only limit. Count actual decoded bytes as chunks arrive, and stop when the count exceeds the limit.

Apply limits to the representation that the application consumes. A small compressed body can expand into a large decoded body. If the stack exposes both wire and decoded sizes, limit and record both.

Process large bodies as streams. Do not call a collect-to-bytes helper unless the route limit is small enough to allocate safely. On a server, use http_body_util::Limited, axum::body::to_bytes(body, limit), or a per-route DefaultBodyLimit::max(n). Put a bounded channel between network reads and a slower consumer. Await capacity or cancel the request when the channel is full. Never add an unbounded queue to hide backpressure.

For uploads, make replayability explicit. A byte buffer is replayable. A file is replayable only when each attempt opens a new handle and starts at the same offset. A live channel, socket, decoder, or one-shot generator is not replayable.

Stop reading after cancellation, a body-limit error, or an expired deadline. Release or reset the protocol stream as the library requires. Do not return a connection to the pool when its protocol state is uncertain.

Define cancellation at the network boundary

Read references/cancellation.md when a caller can cancel a state-changing request, or the application must recover from an Outcome unknown result.

Treat proxy and DNS as connection identity

Read references/proxy-and-dns.md when you key or reuse a connection pool, or configure a proxy, NO_PROXY, a resolver, or a DNS cache.

Bound server overload

Read references/overload-and-shutdown.md when you set server admission, concurrency, queue, or handler-timeout limits.

Shut down in a fixed order

Read references/overload-and-shutdown.md when you add or review graceful shutdown for a server or a client.

Emit safe network diagnostics

Read references/diagnostics.md when you add a log field, metric label, or trace attribute to network code.

Test deterministic seams

Test policy without a real public network. Read references/test-seams.md when you write a network policy test: it maps each control to the smallest seam that exposes it, and gives the paused-clock and test-run rules.