Kubernetes Networking

Follow a request through the five hops between a client and a container, and find the one that drops it.

Purpose

Use this when:

• kubectl get endpoints <svc> is empty or short.
• An Ingress or load balancer returns 502, 503 or 504 while pods are Ready.
• Service A cannot reach Service B: connection refused, timeout, or no such host.
• A NetworkPolicy was added and something stopped working.
• An external LoadBalancer Service is stuck <pending>.

Do not use this when:

• Pods are not Ready — fix that first with kubernetes-troubleshooting. An empty endpoint list is a symptom of unready pods far more often than a networking fault.
• The failure is TLS handshake or certificate validation — use tls-debugging.
• The failure is outside the cluster entirely — use dns-troubleshooting or http-debugging.

What this skill assumes: read access to the namespace, and the ability to run a debug pod or kubectl exec into an existing one. Without in-cluster execution you can only inspect objects, which is rarely enough.

Operating Procedure

The request path has five hops. Test them in order, from the pod outwards — starting at the ingress means you retest four working hops before finding the broken one.

1. Pod — does the process listen on the port, inside the pod?
2. Endpoints — does the Service select the pod, and is the pod Ready?
3. Service — does the ClusterIP route, and does targetPort match the container port?
4. Policy — does a NetworkPolicy allow this connection, in both directions?
5. Ingress / LoadBalancer — does the external layer route to the Service and pass health checks?

At each hop state what you Observed, and do not skip a hop because it "should" be fine. Record the exact client-side error text: connection refused, i/o timeout and no such host point at three completely different hops.

Investigation

Step 0 - what does the failure actually say?

Timeout and refused are the most valuable distinction in this whole skill: refused means packets arrive, timeout means they do not.

Step 1 - is the process listening, inside the pod?

kubectl exec -it <pod> -n <namespace> -- sh -c 'command -v ss >/dev/null && ss -lntp || netstat -lntp'

Look at the bind address. 127.0.0.1:8080 is the most common self-inflicted Kubernetes networking bug: the process is healthy, the probe on localhost passes, and nothing outside the pod can ever connect. It must bind 0.0.0.0 (or ::) to be reachable.

If the image has no shell or no tools, use an ephemeral debug container instead:

kubectl debug -it <pod> -n <namespace> --image=nicolaka/netshoot --target=<container>

kubectl debug shares the target pod's network namespace, so ss, curl and dig there see exactly what the application sees. Requires Kubernetes 1.25+ for the stable ephemeral-container API; verify with kubectl debug --help if you are unsure of the cluster version.

Step 2 - endpoints

kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<svc>
kubectl get endpoints <svc> -n <namespace> -o wide
kubectl get pods -n <namespace> --show-labels
kubectl get svc <svc> -n <namespace> -o jsonpath='{.spec.selector}{"\n"}'

An empty endpoint list has exactly three causes, in order of frequency:

1. No pod is Ready — the Service only publishes ready pods. This is a workload problem.
2. The Service selector does not match the pod labels. Compare the two outputs character by character; app: api and app.kubernetes.io/name: api are different labels.
3. The pods are in a different namespace. Services do not select across namespaces.

EndpointSlice is the source of truth on modern clusters; endpoints is the compatibility view and truncates above 1000 addresses.

Step 3 - Service and port mapping

kubectl get svc <svc> -n <namespace> -o yaml | grep -A6 '^\s*ports:'
kubectl exec -it <client-pod> -n <namespace> -- \
  curl -sS -m 5 -o /dev/null -w 'code=%{http_code} connect=%{time_connect}s total=%{time_total}s\n' \
  http://<svc>.<namespace>.svc.cluster.local:<port>/

port is what clients dial; targetPort is the container port. A mismatch produces connection refused with perfectly healthy endpoints. If targetPort is a name, that name must exist in the container's ports: list — a named port that does not resolve fails silently.

Interpret the curl timings: a fast connect with a slow total means the backend is slow, not the network. A connect that hangs to the timeout means hop 4.

Step 4 - NetworkPolicy

kubectl get networkpolicy -A
kubectl describe networkpolicy -n <namespace>

Three rules that account for most NetworkPolicy incidents:

• Policies are additive allow-lists. The moment any policy selects a pod, everything not explicitly allowed for that direction is denied.
• Ingress and egress are separate. Allowing ingress on the server does nothing if the client has an egress policy that omits the destination.
• DNS is egress too. A default-deny egress policy without a rule allowing UDP/TCP 53 to kube-system breaks name resolution for every pod it selects, which presents as no such host rather than as a policy problem.

Also confirm your CNI enforces policy at all. Flannel does not implement NetworkPolicy; the objects apply cleanly and do nothing, which is worse than an error.

Step 5 - Ingress and LoadBalancer

kubectl describe ingress <name> -n <namespace>
kubectl logs -n <ingress-namespace> deploy/<ingress-controller> --tail=100 | grep -E ' 50[234] '
kubectl get svc -n <ingress-namespace> <controller-svc> -o wide

The controller log is the authority on which backend it chose and what happened. A 503 with "no active endpoints" in the log confirms hop 2, not an ingress bug.

An external IP stuck at <pending> means the cloud controller manager did not provision a load balancer: missing IAM permissions, no eligible subnets, or a missing subnet tag. Check the Events on the Service — the cloud provider writes the real reason there.

Decision Tree

Client cannot reach the service
├── "no such host" → DNS
│   ├── Other names also fail → CoreDNS pods / kube-dns Service → dns-troubleshooting
│   ├── Only this name fails  → wrong Service name or namespace in the URL
│   └── Egress policy present → DNS blocked by NetworkPolicy
├── "connection refused" → something rejected the packet
│   ├── Endpoints empty        → pods unready or selector mismatch
│   ├── targetPort != container port → Service definition
│   └── Process bound to 127.0.0.1   → application config
├── "i/o timeout" → nothing answered
│   ├── NetworkPolicy selects either end → policy
│   ├── Works pod→pod but not via ClusterIP → kube-proxy / CNI on the client node
│   └── Only across nodes/zones → overlay, MTU, or security-group rules
├── 503 from ingress → no healthy backend → hop 2
├── 502 from ingress → backend closed the connection
│   ├── Backend protocol mismatch (HTTPS backend declared as HTTP)
│   └── Backend keepalive shorter than the proxy's → race on connection reuse
└── 504 from ingress → backend too slow → the application, not the network

Common Causes

Safety

Read-only

get, describe, endpoint and policy inspection, controller logs, and curl/dig from inside a debug pod. None of these change cluster state.

kubectl debug creates an ephemeral container on a running pod. It cannot be removed without restarting the pod, so it leaves a trace on the object — harmless, but note it in the incident log.

Potentially disruptive

• kubectl apply on a Service or NetworkPolicy — takes effect immediately across the whole cluster for every selected pod. Capture kubectl get -o yaml first so you can restore it.
• kubectl rollout restart deployment/<ingress-controller> — drops every in-flight connection through that controller. This is a production outage if you have one replica.
• Editing kube-proxy or CNI configuration — affects every pod on every node. Treat it as a cluster change with a maintenance window, not as a debugging step.
• Removing a NetworkPolicy "to test" — this opens traffic paths in production. Prefer adding a narrow, explicitly-scoped allow rule that you can remove again, and set a reminder to remove it.

Destructive

This skill proposes no destructive operations.

Verification

Before: 503 on 30% of requests; endpoints 2/8; p99 12s
After:  0 errors over 5 min; endpoints 8/8; p99 180ms

kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<svc> \
  -o jsonpath='{range .items[*]}{range .endpoints[*]}{.addresses[0]}{" ready="}{.conditions.ready}{"\n"}{end}{end}'
for i in $(seq 1 30); do
  kubectl exec -n <namespace> <client-pod> -- \
    curl -sS -o /dev/null -m 5 -w '%{http_code}\n' http://<svc>:<port>/ 2>&1
done | sort | uniq -c

Verify from the client's position, not from yours. A curl from your laptop through a different ingress path proves nothing about pod-to-pod traffic. Loop the request at least 30 times: an intermittent fault caused by one bad endpoint out of eight will pass a single check.

Anti-Patterns

Bad: The Service is broken, recreate it. Better: Check endpoints first. In most cases the Service object is correct and the pods are not Ready — deleting and recreating the Service changes its ClusterIP and breaks every client that cached it.

Bad: Test connectivity by curling the pod IP directly and declaring the network fine. Better: Pod IP works while ClusterIP fails is a specific and useful signal — it points at kube-proxy or the CNI on the client node. Do not stop at the first success.

Bad: Delete the NetworkPolicy to unblock the incident. Better: Add a scoped allow rule for the specific flow. Deleting the policy removes the isolation from every pod it selected, which is a security incident layered on a availability one.

Bad: Increase the ingress timeout to fix 504s. Better: A 504 says the backend is slow. Raising the timeout converts fast failures into slow failures and hides the regression while making the outage last longer.

Bad: Assume DNS is fine because nslookup works from your laptop. Better: Resolve from inside the pod, with the pod's own /etc/resolv.conf and ndots setting. Cluster DNS behaviour is not the same as host DNS behaviour.

Composition

• kubernetes-troubleshooting — pods are not Ready, which is hop 2's usual cause.
• dns-troubleshooting — CoreDNS, ndots, search domains, resolution failures.
• http-debugging — status codes, keepalive, proxy behaviour above the transport.
• tls-debugging — handshake failures, SNI, certificate chains at the ingress.
• linux-network-debugging — conntrack, MTU, packet capture on the node.

References

• examples/503-from-ingress.md — a worked end-to-end example of the 30%-503 scenario.
• Kubernetes: Debug Services — upstream walkthrough of the same hop chain.
• Kubernetes: Network Policies — semantics of additive allow-lists and the isolation model.