Linux Network Debugging

Work down the stack — socket, host, path — and let the failure mode tell you which layer to look at.

Purpose

Use this when:

• Connections from a host fail with connection refused, connection timed out, or connection reset by peer.
• Throughput is poor or transfers stall at a consistent size.
• nf_conntrack: table full appears in dmesg.
• You need to interpret ss, netstat, tcpdump or conntrack output.

Do not use this when:

• The failure is name resolution — use dns-troubleshooting. no such host is never a TCP problem.
• The failure is HTTP-level (status codes, keepalive, proxies) — use http-debugging.
• The failure is inside Kubernetes Services or NetworkPolicies — use kubernetes-networking.

What this skill assumes: shell access on at least one end of the connection, and sudo for packet capture and conntrack. Capturing on both ends is worth the effort: it is the only way to tell "we never sent it" from "they never received it".

Operating Procedure

1. Read the error precisely. Refused, timeout and reset are three different layers.
2. Confirm the target. Resolve the name, and note the IP you are actually connecting to. Half of "network" problems are the client talking to the wrong address.
3. Test the socket layer with ss locally: is anything listening, is the backlog overflowing.
4. Test the path with curl, nc, or mtr. One hop at a time.
5. Check host-level limits: conntrack, ephemeral ports, backlog, fd limits.
6. Capture packets only when the layers above have not answered it. tcpdump is precise and slow to read; it is a third step, not a first one.
7. Fix and verify with the same test that failed.

tcpdump output is the strongest Observed evidence available here. Everything inferred from counters is weaker; say which you are using.

Investigation

Step 0 - what does the failure mean

Refused versus timeout is the single most valuable distinction: refused proves packets reach the host and come back, which eliminates the entire firewall and routing layer in one observation.

Step 1 - the socket layer

ss -lntp                      # listening TCP sockets with the owning process
ss -s                         # summary by state
ss -tn state established '( dport = :5432 )' | wc -l
ss -lnt '( sport = :8080 )'   # Send-Q on a listening socket = accept backlog size

On a listening socket, Recv-Q is the number of connections waiting to be accepted and Send-Q is the configured backlog. Recv-Q approaching Send-Q means the application is not calling accept() fast enough, and new connections are being dropped — which the client sees as a timeout, not as a refusal.

nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPSynRetrans

ListenOverflows incrementing is unambiguous proof of backlog overflow. This is a common production failure with a confusing signature: the server is healthy, the port is open, and clients time out.

Step 2 - host-level limits

# conntrack
sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
sudo dmesg -T | grep -i conntrack

# ephemeral ports
sysctl net.ipv4.ip_local_port_range
ss -tan state time-wait | wc -l

# per-process sockets
sudo ls /proc/<pid>/fd | wc -l

Conntrack tracks every connection through a NAT or stateful firewall. When nf_conntrack_count approaches nf_conntrack_max, new connections are dropped silently and dmesg logs table full, dropping packet. This presents as random timeouts under load and is frequently misattributed to the application. It is especially common on Kubernetes nodes and NAT gateways.

Ephemeral port exhaustion has a similar signature. A host making many short-lived outbound connections to the same destination can exhaust the ~28k default range; TIME_WAIT sockets hold ports for 60 seconds after close. A large time-wait count is normal and usually harmless — it only matters when the port range is genuinely exhausted, so count before concluding.

Step 3 - the path

curl -sS -o /dev/null -m 10 -w \
  'dns=%{time_namelookup}s connect=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s code=%{http_code}\n' \
  https://<host>/

nc -vz <host> <port>
mtr -rwzc 50 <host>
ip route get <destination-ip>

The curl timing breakdown localises the delay immediately: a slow time_namelookup is DNS, a slow time_connect is the TCP handshake (network path), a slow time_appconnect is TLS, and a slow time_starttransfer with fast connect is the server thinking.

mtr sends 50 probes and shows per-hop loss. Read it carefully: loss at an intermediate hop that does not persist to the final hop is not real. Routers deprioritise ICMP to their own control plane. Only loss that continues to the destination row counts.

ip route get shows which interface and gateway the kernel will actually use, including policy-based routing — the fastest way to catch a host that has two interfaces and is sending traffic out the wrong one.

Step 4 - MTU and path MTU discovery

ip link show                                   # MTU per interface
ping -M do -s 1472 <destination>               # 1472 + 28 = 1500
ping -M do -s 1400 <destination>

-M do sets Don't Fragment. If 1400 succeeds and 1472 fails with Message too long locally, your own MTU is smaller. If 1472 simply times out while small packets work, you have a path MTU black hole: something on the path is dropping large packets and the ICMP "fragmentation needed" message is being filtered.

The signature is distinctive and very commonly misdiagnosed: TCP connections establish fine, small requests work, and anything large hangs forever. It appears constantly with VPNs, overlay networks and tunnels, where encapsulation reduces the usable MTU below 1500.

Step 5 - packet capture

sudo tcpdump -ni any -c 200 "host <ip> and port <port>"
sudo tcpdump -ni any -w /tmp/cap.pcap "host <ip> and port <port>"
sudo tcpdump -nr /tmp/cap.pcap | head -50

Always use -n to skip reverse DNS, which otherwise adds latency and generates its own traffic. Always bound the capture with -c or a filter; an unfiltered capture on a busy host fills the disk and adds load.

What to look for:

Capture on both ends where possible. Packets leaving one host and not arriving at the other locates the drop to the path; packets arriving and not being answered locates it to the far host.

Decision Tree

Connection problem
├── "connection refused" → packets arrive; nothing listening
│   ├── ss -lntp shows no listener → the service is down or bound elsewhere
│   └── Bound to 127.0.0.1 → binding, not networking
├── "connection timed out" → no answer
│   ├── tcpdump shows SYN out, nothing back → dropped on the path (firewall/SG/route)
│   ├── tcpdump shows nothing out at all    → local routing or a local filter rule
│   ├── conntrack table full in dmesg       → conntrack exhaustion
│   └── ListenOverflows rising              → backlog overflow; the app is not accepting
├── "connection reset" mid-stream
│   ├── Regular interval (~minutes)  → idle timeout on a middlebox or conntrack expiry
│   └── Under load                   → server closing; check its limits and logs
├── Small requests fine, large hang → path MTU black hole
├── Fails only past N connections   → ephemeral ports, conntrack, or fd limits
└── Slow but working → mtr for loss; curl -w to localise the phase

Common Causes

Safety

Read-only

ss, ip route get, ip link show, nstat, sysctl reads, conntrack -C, curl, nc -z, ping, and mtr. These generate a little traffic but change nothing.

tcpdump is read-only with two real costs: it puts the interface into promiscuous mode, and an unbounded capture can fill the disk on a host that may already be struggling. Always filter, always bound with -c or a size limit, and write captures outside the filesystem you are protecting.

Capture files contain payloads. On an unencrypted protocol that includes credentials and personal data. Treat a .pcap as sensitive: restrict it, and delete it when the investigation ends.

Potentially disruptive

• sysctl -w net.netfilter.nf_conntrack_max=... — takes effect immediately host-wide. Raising it consumes kernel memory (roughly 300 bytes per entry); a large increase on a small node is a memory problem in a new place. Record the old value first.
• sysctl -w net.core.somaxconn=... — only helps if the application also raises its own listen() backlog. Changing one without the other achieves nothing.
• ip link set <dev> mtu <n> — takes effect immediately and can break connectivity to the host you are changing it on. On a remote host this can lock you out.
• iptables/nft rule changes — a mistaken rule can cut off your own session. Where the tooling supports it, use a scheduled rollback; otherwise have out-of-band console access first.
• conntrack -D to clear entries — kills the affected established connections immediately.
• ss -K to kill sockets — terminates live connections.

Destructive

This skill proposes no destructive operations. Flushing all firewall rules (iptables -F) to "test whether the firewall is the problem" is not on this list because it should not be done: on a cloud host it can remove the rules that protect the instance and leave it exposed, and on a Kubernetes node it can break the CNI's own rules and take every pod on the node offline. Add a narrow, temporary allow rule instead, and remove it afterwards.

Verification

Before: 8% of outbound connections time out; nf_conntrack_count 65530/65536
After:  0 timeouts over 30 min; nf_conntrack_count 41200/262144; no dmesg conntrack lines

for i in $(seq 1 100); do
  curl -sS -o /dev/null -m 5 -w '%{http_code} %{time_connect}\n' https://<host>/ 2>&1
done | awk '{print $1}' | sort | uniq -c

nstat -az TcpExtListenOverflows TcpExtListenDrops
sudo sysctl net.netfilter.nf_conntrack_count

Loop the test at least 100 times. Intermittent network faults — the only kind that reach this skill, since constant ones are found in minutes — will pass a single check by chance. Verify under the load that produced the failure, and check that the counter which was climbing is now flat.

Anti-Patterns

Bad: ping works, so the network is fine. Better: ping is ICMP. It proves nothing about a TCP port, a firewall rule scoped to that port, MTU for large packets, or an application that is not accepting connections.

Bad: mtr shows 40% loss at hop 4, that is the problem. Better: Only loss that persists to the final hop is real. Intermediate routers rate-limit ICMP to their own control plane as a matter of policy.

Bad: Lots of TIME_WAIT sockets, so enable tcp_tw_recycle. Better: That option was removed from the kernel in 4.12 because it broke connections from clients behind NAT. TIME_WAIT is normal; only act if ports are genuinely exhausted, and then use connection pooling.

Bad: Raise somaxconn to fix backlog overflow. Better: The application's listen() backlog is capped by somaxconn but set independently. Raising only the kernel limit changes nothing if the application asks for 128.

Bad: tcpdump first. Better: ss and curl -w answer most questions in seconds. Packet capture is precise, slow to read, and generates load on a host you may already be worried about.

Bad: Blame the network because the application logs a timeout. Better: A timeout is where the symptom appeared, not where the cause is. Check whether the server accepted the connection at all — the answer is frequently that it did, and then took 30 seconds to respond.

Composition

• linux-troubleshooting — host resource exhaustion showing up as network failure.
• dns-troubleshooting — resolution failures, which are not TCP problems.
• kubernetes-networking — the same stack inside a cluster, with Services and policies added.
• http-debugging — once packets flow and the problem is at the protocol layer.

References

• iproute2 ss manual — socket states and filters.
• Linux kernel: TCP sysctl reference — every tunable mentioned here, with its real default.