HTTP Debugging

Work out which hop generated the response, then what it was reacting to.

Purpose

Use this when:

• A request returns 502, 503, 504, 499, 413, 431, or an unexpected redirect.
• Requests work directly against the backend but fail through the proxy.
• Requests fail intermittently at a regular interval.
• Latency is high with no errors, or a response is truncated.

Do not use this when:

• The name does not resolve — use dns-troubleshooting.
• The TLS handshake fails — use tls-debugging. A handshake failure never produces an HTTP status.
• The problem is Kubernetes Service or endpoint routing — use kubernetes-networking.

What this skill assumes: curl, and ideally access to the proxy's access log. The access log is the authority on what the proxy did; client-side evidence alone is often ambiguous.

Operating Procedure

1. Identify which hop answered. A 502 from your CDN and a 502 from your ingress are different incidents. Response headers usually name the responder.
2. Reproduce with curl, with timings. The timing breakdown localises the phase immediately.
3. Bypass hops one at a time. Client → CDN → load balancer → ingress → service → pod. Test directly against each and find where behaviour changes.
4. Read the status code for what it actually means, not for what it is commonly assumed to mean.
5. Check both ends of any timeout or keepalive pair. Most intermittent HTTP failures are a mismatch between two configured values.
6. Fix the mismatch and verify with a loop, not a single request.

The status code and headers you saw are Observed. Which component set them is an Assumption until a header or an access log confirms it.

Investigation

Step 1 - curl with timings

curl -sS -o /dev/null -D - -m 30 -w '
dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect}
ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code} size=%{size_download}
' https://<host>/<path>

The phase boundaries map directly onto layers:

A fast connect with a slow ttfb is an application problem and not a network one. That single distinction resolves a large fraction of "the network is slow" reports.

-D - dumps the response headers, which usually identify the responder: Server: nginx, via: 1.1 google, x-envoy-upstream-service-time, cf-ray, x-amzn-requestid.

Step 2 - what the status codes actually mean

The 502/503/504 distinction is the most useful thing in this skill:

• 503 means the proxy had nowhere to send it. Look at the upstream health check and the endpoint list — usually not an application bug at all.
• 502 means it sent it and got something wrong back. Look at the backend process: it crashed, closed the connection, or spoke the wrong protocol.
• 504 means the backend took it and went quiet. Look at the application's own latency.

499 is worth calling out because it is routinely misfiled as a server error. It means the client disconnected before the response. A wave of 499s usually means an upstream timeout shorter than this service's processing time — and often that the client is now retrying, adding load.

Step 3 - bypass hops

# Directly at the pod, bypassing Service, ingress and LB
kubectl port-forward pod/<pod> 8080:8080 &
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/<path>

# At the Service from inside the cluster
kubectl run -it --rm curl --image=curlimages/curl --restart=Never -- \
  curl -sS -o /dev/null -w '%{http_code}\n' http://<svc>.<ns>:8080/<path>

# At the ingress, skipping DNS and the CDN
curl -sS -o /dev/null -w '%{http_code}\n' --resolve <host>:443:<ingress-ip> https://<host>/<path>

--resolve is the most useful flag here: it forces curl to a specific IP while keeping the correct Host header and TLS SNI, letting you test one hop without changing DNS or /etc/hosts.

The hop at which behaviour changes is the hop that owns the problem.

Step 4 - keepalive and timeout mismatches

This is the cause of most intermittent HTTP failures, and the signature is distinctive: a small percentage of requests fail, at random, with no pattern in the payload.

The race: the proxy keeps an idle connection in its pool. The backend's idle timeout is shorter, so the backend closes it. The proxy, not yet aware, sends a request onto the closing connection. The result is a 502 or a reset.

The rule: the backend's idle timeout must be longer than the proxy's.

# nginx / ingress-nginx
grep -rE 'keepalive_timeout|proxy_read_timeout|proxy_connect_timeout' /etc/nginx/
kubectl get configmap -n ingress-nginx ingress-nginx-controller -o yaml

# What the backend advertises
curl -sSI https://<host>/ | grep -i 'keep-alive\|connection'

Typical defaults worth checking against each other:

The Node.js default of 5 seconds behind a 60-second ALB is a well-known source of intermittent 502s, and the fix is to raise the application's value above the proxy's.

The same ordering logic applies to request timeouts, from the outside in: client > CDN > LB > ingress > application. When an inner timeout is longer than an outer one, the outer hop gives up first and the work is wasted — and the client typically retries, multiplying the load.

Step 5 - the proxy's own log

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=200 | grep -E ' (50[234]|499) '

The access log tells you what the proxy chose and what happened, which no client-side observation can. Look for the upstream address (which backend), the upstream status (what it said), and the upstream response time (how long it took). A 502 with an empty upstream status means the connection failed before any response — a completely different problem from a 502 with upstream status 500.

Decision Tree

HTTP request fails
├── 503 → the proxy has no healthy upstream
│   ├── Endpoints empty     → kubernetes-networking
│   └── Health checks failing → the health endpoint or the app
├── 502 → invalid or closed upstream response
│   ├── Intermittent, low rate  → keepalive race: raise the backend idle timeout
│   ├── Constant                → backend crashed, or wrong protocol (HTTPS backend as HTTP)
│   └── On large responses only → buffer size, or header limits
├── 504 → upstream too slow
│   ├── App p99 exceeds the proxy timeout → fix the app; do not just raise the timeout
│   └── Upstream never accepted            → connection saturation
├── 499 → the client gave up first
│   └── The caller's timeout is shorter than this service's work; check for retries
├── 413 / 431 → size limits on body or headers
├── Works direct, fails via proxy
│   ├── Host header rewritten by the proxy
│   ├── Path rewritten or stripped
│   └── Protocol mismatch (h2 vs h1, TLS to a plaintext backend)
└── Intermittent with no pattern
    ├── Keepalive race
    ├── One unhealthy backend out of N → loop the request to find it
    └── Connection pool exhaustion at the proxy

Common Causes

Safety

Read-only

curl with GET or HEAD, reading proxy configuration, reading access logs. Note that a GET is only read-only if the endpoint is — do not assume, and never use curl to exercise an endpoint you have not read the handler for.

curl --resolve changes only your own request's routing and is safe.

Potentially disruptive

• Changing proxy timeouts or keepalive settings — affects every request through that proxy. Applying a config reload drops nothing on nginx (nginx -s reload is graceful), but a bad configuration fails the reload. Validate with nginx -t or the equivalent first.
• Reloading or restarting an ingress controller — a reload is graceful; a restart drops every in-flight connection. With a single replica, a restart is an outage.
• Enabling or disabling compression, buffering, or HTTP/2 — changes behaviour for all traffic, and can break streaming or long-polling endpoints that worked by accident.
• Load testing through the proxy — you are adding load to the component you suspect. Bound it, and do not do it during an incident.
• Raising a body-size limit — accepts larger uploads, with memory and disk implications at the proxy.

Destructive

This skill proposes no destructive operations. Be aware that using curl to reproduce a failing request against an endpoint that mutates state — a POST, PUT, PATCH or DELETE — performs that operation for real. Reproduce against a non-production environment, or against an endpoint you have confirmed is idempotent and safe.

Verification

Before: 2.1% of requests return 502, no pattern; nginx log shows upstream closed connection
After:  0 of 5000 requests return 502; Node keepAliveTimeout 65s > ALB idle timeout 60s

for i in $(seq 1 500); do
  curl -sS -o /dev/null -w '%{http_code}\n' https://<host>/<path>
done | sort | uniq -c

curl -sS -o /dev/null -w '%{http_code} %{time_starttransfer}\n' --keepalive-time 120 \
  https://<host>/<path> https://<host>/<path> https://<host>/<path>

Two requirements specific to HTTP:

1. Loop it. A 2% failure rate passes a single check 98% of the time. Run at least 500 requests for anything intermittent.
2. Reuse connections. Passing multiple URLs to one curl invocation reuses the connection, which is what exercises the keepalive path. A fresh connection per request will never reproduce a keepalive race — which is exactly why these bugs survive testing.

Also confirm at the hop that reported the problem: the proxy access log should show the error rate falling, not just your client.

Anti-Patterns

Bad: Raise the timeout to fix 504s. Better: A 504 says the backend is slow. Raising the timeout converts fast failures into slow ones, holds connections open longer, and hides the regression while the user waits longer.

Bad: Treat 499 as a server error and investigate the server. Better: 499 means the client disconnected first. Look at the caller's timeout — and at whether it is now retrying and doubling your load.

Bad: Test with a single curl and declare it fixed. Better: Intermittent HTTP failures are the majority of real ones. One request proves nothing; loop it, and reuse the connection.

Bad: Restart the ingress controller when requests fail. Better: That drops every in-flight connection cluster-wide. Read the access log first — it usually contains the answer.

Bad: Add a retry to work around intermittent 502s. Better: Retries hide the cause and multiply load during partial failures. Fix the keepalive mismatch; if you do add retries, they need backoff, jitter and a cap.

Bad: Assume 502 means the application crashed. Better: 502 covers "the connection closed", "the response was malformed" and "the protocol was wrong" as well. The proxy's upstream status field distinguishes them in one line.

Composition

• tls-debugging — handshake failures, which never reach HTTP.
• dns-troubleshooting — when time_namelookup is the slow phase.
• kubernetes-networking — 503s with empty endpoints.
• linux-network-debugging — resets, timeouts, and conntrack beneath HTTP.

References

• MDN: HTTP response status codes
• curl: --write-out variables — the full timing variable list.