agentleFS
foxj77foxj77/claude-code-skills/k8s-network-troubleshooting/SKILL.md

k8s-network-troubleshooting

Skill19 starsChanged 10 months agoMITView on GitHub

Use when pods cannot reach other pods or services, connections are timing out or refused, network policies may be blocking traffic, ingress is not routing, egress to external services fails, or Istio service mesh is causing connectivity issues

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Note: If connections fail with "could not resolve host", the problem is DNS, not network.

    line 32
  2. Distinctive:

    Run this checklist first. Complete all checks before drawing conclusions.

    line 61
  3. Distinctive:

    Use this decision tree after the health check identifies failures.

    line 156
  4. Distinctive:

    No policies = all traffic allowed (default allow)

    line 461
  5. Distinctive:

    When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

    line 740

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Kubernetes Network Troubleshooting

Diagnose network connectivity issues within and outside a Kubernetes cluster. Covers pod-to-pod, pod-to-service, ingress, egress, network policies, and Istio service mesh.

Keywords

network, connectivity, connection refused, timeout, unreachable, curl, wget, pod-to-pod, pod-to-service, service mesh, istio, envoy, sidecar, mTLS, network policy, ingress, egress, ClusterIP, NodePort, LoadBalancer, CNI, kube-proxy, iptables, calico, cilium, VirtualService, DestinationRule, Gateway

When to Use This Skill

  • Pods cannot reach other pods or services (connection refused / timeout)
  • Connection timeouts between services that were previously working
  • Ingress controller is not routing external traffic to services
  • Egress to external APIs or the internet is blocked
  • Intermittent connectivity failures between services
  • Istio sidecar injection issues or mTLS failures
  • Traffic routing behaves differently than VirtualService rules specify
  • Services are reachable by ClusterIP but not by name (check DNS first)
  • NodePort or LoadBalancer services are not externally accessible

When NOT to Use

  • DNS resolution failures (pods can't resolve names) → use k8s-dns-troubleshooting
  • Pod crashes, scheduling failures, or OOMKills → use k8s-namespace-troubleshooting
  • TLS certificate issuance or renewal failures → use cert-manager-troubleshooting

Note: If connections fail with "could not resolve host", the problem is DNS, not network. Start with k8s-dns-troubleshooting. If DNS resolves but the connection still fails, return here.

Related Skills

  • k8s-dns-troubleshooting - DNS resolution and CoreDNS issues
  • k8s-namespace-troubleshooting - General namespace diagnosis
  • k8s-platform-operations - Cluster-wide health checks
  • k8s-security-hardening - Network policies and security controls
  • Shared: Network Policies

Quick Reference

TaskCommand
Test connectivity from a podkubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${TARGET}:${PORT} 2>&1
Check service endpointskubectl get endpoints ${SVC} -n ${NS}
List network policieskubectl get networkpolicies -n ${NS}
Describe a servicekubectl describe svc ${SVC} -n ${NS}
Check pod IPkubectl get pod ${POD} -n ${NS} -o jsonpath='{.status.podIP}'
Check Istio sidecar statuskubectl get pod ${POD} -n ${NS} -o jsonpath='{.spec.containers[*].name}'
Envoy proxy logskubectl logs ${POD} -n ${NS} -c istio-proxy --tail=100
Ingress resourceskubectl get ingress -A
kube-proxy podskubectl get pods -n kube-system -l k8s-app=kube-proxy
CNI podskubectl get pods -A -l k8s-app=calico-node or kubectl get pods -A -l k8s-app=cilium

Quick Network Health Check

Run this checklist first. Complete all checks before drawing conclusions.

Step 1: Pod-to-Pod (Same Namespace)

# Find two running pods in the same namespace
kubectl get pods -n ${NS} --field-selector=status.phase=Running -o wide

# Get the IP of the target pod
TARGET_IP=$(kubectl get pod ${TARGET_POD} -n ${NS} -o jsonpath='{.status.podIP}')

# Test connectivity from source pod to target pod IP
kubectl exec -n ${NS} ${SOURCE_POD} -- wget -qO- --timeout=5 http://${TARGET_IP}:${PORT} 2>&1

Step 2: Pod-to-Service (Same Namespace)

# Verify the service exists and has a ClusterIP
kubectl get svc ${SVC} -n ${NS}

# Verify endpoints are populated
kubectl get endpoints ${SVC} -n ${NS}

# Test connectivity via service name
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${SVC}:${SVC_PORT} 2>&1

Step 3: Pod-to-Service (Cross Namespace)

# Test connectivity to a service in a different namespace
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${SVC}.${TARGET_NS}.svc.cluster.local:${SVC_PORT} 2>&1

Step 4: Pod-to-External

Requires internet access from the cluster. Substitute any reachable external endpoint in air-gapped or restricted environments.

# Test internet connectivity
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://httpbin.org/get 2>&1 | head -5

# Test specific external endpoint
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 https://${EXTERNAL_HOST} 2>&1 | head -3

Step 5: Network Policies

# Check for network policies in the namespace
kubectl get networkpolicies -n ${NS}

# Check for policies that restrict egress or ingress
kubectl get networkpolicies -n ${NS} -o json | \
  jq -r '.items[] | "\(.metadata.name)\tTypes: \(.spec.policyTypes // ["Ingress"] | join(", "))"'

Step 6: Istio Sidecar (If Applicable)

# Check if the pod has an istio-proxy sidecar container
kubectl get pod ${POD} -n ${NS} -o jsonpath='{.spec.containers[*].name}' | tr ' ' '\n' | grep istio-proxy

# Check if the namespace has sidecar injection enabled
kubectl get namespace ${NS} -o jsonpath='{.metadata.labels.istio-injection}'

Health Summary Template

Present results using this format:

## Network Health Summary

| Check | Status | Detail |
|-------|--------|--------|
| Pod-to-pod (same NS) | PASS/FAIL/SKIP | Connected / Timeout / Refused / Not tested |
| Pod-to-service (same NS) | PASS/FAIL/SKIP | Endpoints: X, Response: OK / Timeout / Refused |
| Pod-to-service (cross NS) | PASS/FAIL/SKIP | Connected / Blocked / Not tested |
| Pod-to-external | PASS/FAIL/SKIP | Connected / Timeout / Blocked |
| Network policies | NONE/PRESENT | X policies found (Y with egress rules) |
| Istio sidecar injected | YES/NO/N/A | istio-proxy present / absent / Istio not installed |

**Overall: HEALTHY / DEGRADED / UNHEALTHY**
  • HEALTHY — All connectivity checks pass
  • DEGRADED — Some paths work, others fail (targeted issue)
  • UNHEALTHY — Broad connectivity failure

Diagnostic Workflow

Use this decision tree after the health check identifies failures.

Connection failing?
├─ Same namespace pod-to-pod fails?
│   ├─ Both pods on same node → CNI issue (Section 7)
│   ├─ Pods on different nodes → CNI cross-node or overlay issue (Section 7)
│   └─ Network policy blocking → Ingress policy on target (Section 5)
├─ Service connection fails but pod IP works?
│   ├─ Endpoints empty → No ready pods backing the service (Section 2)
│   ├─ Port mismatch → Service port vs targetPort vs containerPort (Section 2)
│   ├─ kube-proxy not running → No iptables/IPVS rules for ClusterIP (Section 7)
│   └─ Istio routing override → VirtualService or DestinationRule (Section 6)
├─ Cross-namespace fails but same-namespace works?
│   ├─ Network policy restricting cross-NS traffic → Egress or ingress rules (Section 5)
│   ├─ Istio mTLS mode mismatch → STRICT on one side, no sidecar on other (Section 6)
│   └─ DNS issue → Resolve ${SVC}.${NS}.svc.cluster.local (→ k8s-dns-troubleshooting)
├─ External/egress fails?
│   ├─ All pods affected → Cluster-wide egress blocked (Section 3)
│   ├─ Only some pods → Egress network policy (Section 5)
│   ├─ Istio sidecar blocking → Outbound traffic policy (Section 6)
│   └─ NAT/SNAT issue → Node cannot route to internet (Section 3)
├─ Ingress fails?
│   ├─ Ingress controller not running → Controller pods down (Section 4)
│   ├─ Ingress resource misconfigured → Backend service or path (Section 4)
│   ├─ TLS termination failing → Certificate or secret issue (Section 4)
│   └─ Istio Gateway misconfigured → Gateway or VirtualService (Section 6)
└─ Intermittent failures?
    ├─ Some requests succeed → Load balancing to unhealthy pod (Section 2)
    ├─ Timeouts then recovery → Conntrack table full or CNI flap (Section 7)
    └─ Istio circuit breaking → DestinationRule outlier detection (Section 6)

Section 1: Pod-to-Pod Connectivity

Direct pod-to-pod communication uses pod IPs, bypassing services and kube-proxy.

Verifying Pod IPs and Placement

# Get pod IPs and node placement
kubectl get pods -n ${NS} -o wide

# Confirm pods have IPs assigned (no IP = CNI problem)
kubectl get pod ${POD} -n ${NS} -o jsonpath='{.status.podIP}'

# Check if pods are on the same or different nodes
kubectl get pod ${SOURCE_POD} -n ${NS} -o jsonpath='{.spec.nodeName}'
kubectl get pod ${TARGET_POD} -n ${NS} -o jsonpath='{.spec.nodeName}'

Testing Pod-to-Pod

# Direct connectivity test using pod IP
TARGET_IP=$(kubectl get pod ${TARGET_POD} -n ${NS} -o jsonpath='{.status.podIP}')
kubectl exec -n ${NS} ${SOURCE_POD} -- wget -qO- --timeout=5 http://${TARGET_IP}:${PORT} 2>&1

# If wget is unavailable, test with /dev/tcp (bash-based images)
kubectl exec -n ${NS} ${SOURCE_POD} -- bash -c "echo > /dev/tcp/${TARGET_IP}/${PORT} && echo OPEN || echo CLOSED" 2>&1

# Check if the target pod is listening on the expected port
kubectl exec -n ${NS} ${TARGET_POD} -- sh -c 'netstat -tlnp 2>/dev/null || ss -tlnp 2>/dev/null' | head -20

Same Node vs Cross Node

# If same-node pod-to-pod fails: CNI plugin local routing is broken
# If cross-node works: unlikely to be CNI, check network policy on target

# If cross-node pod-to-pod fails: overlay network issue
# Check CNI pods are running on both nodes (use the label for the installed CNI)
kubectl get pods -n kube-system -l k8s-app=calico-node -o wide --field-selector spec.nodeName=${NODE1} 2>/dev/null
kubectl get pods -n kube-system -l k8s-app=cilium -o wide --field-selector spec.nodeName=${NODE1} 2>/dev/null

Interpreting Failures

SymptomLikely Cause
Connection refusedTarget pod is not listening on that port
Connection timed outNetwork path blocked (policy, CNI, or routing)
No pod IP assignedCNI plugin failed to allocate an IP
Works same-node, fails cross-nodeOverlay/tunnel issue between nodes

Section 2: Pod-to-Service Connectivity

Services route traffic through ClusterIP (kube-proxy rules) to pod endpoints.

Checking Service Configuration

# Service details: ClusterIP, ports, selector
kubectl describe svc ${SVC} -n ${NS}

# Verify selector matches pod labels
SVC_SELECTOR=$(kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.spec.selector}')
echo "Service selector: ${SVC_SELECTOR}"
kubectl get pods -n ${NS} -l "${LABEL_KEY}=${LABEL_VALUE}" -o wide

# Port mapping — the three ports that must align
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{range .spec.ports[*]}port:{.port} targetPort:{.targetPort} protocol:{.protocol}{"\n"}{end}'

The Three-Port Model

Client → Service:port → Pod:targetPort → Container listens on containerPort

- service.spec.ports[].port      = port clients connect to
- service.spec.ports[].targetPort = port on the pod (defaults to .port if omitted)
- container.ports[].containerPort = port the app listens on (must match targetPort)

A mismatch between targetPort and the actual port the container listens on is a common cause of "connection refused" after DNS resolves.

Checking Endpoints

# Endpoints show which pod IPs back the service
kubectl get endpoints ${SVC} -n ${NS}

# Detailed endpoint view
kubectl get endpoints ${SVC} -n ${NS} -o yaml

# If using EndpointSlices (Kubernetes 1.21+)
kubectl get endpointslices -n ${NS} -l kubernetes.io/service-name=${SVC}

Empty Endpoints Causes

CauseDiagnostic
No pods match selectorCompare svc.spec.selector with pod.metadata.labels
Pods not readyCheck kubectl get pods -n ${NS} — pods must pass readiness probes
Wrong namespaceService and pods must be in the same namespace (for selector matching)
Readiness probe failingkubectl describe pod ${POD} -n ${NS} — check probe events

Service Types

# ClusterIP (default) — internal only
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.spec.type}'

# NodePort — verify port allocation
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{range .spec.ports[*]}nodePort:{.nodePort}{"\n"}{end}'

# LoadBalancer — check external IP assignment
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# If LoadBalancer shows <pending>, the cloud controller hasn't provisioned the LB
kubectl describe svc ${SVC} -n ${NS} | grep -A5 Events

Testing Service Connectivity

# Via service name (tests DNS + kube-proxy + pod)
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${SVC}:${SVC_PORT} 2>&1

# Via ClusterIP directly (bypasses DNS, tests kube-proxy + pod)
CLUSTER_IP=$(kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.spec.clusterIP}')
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${CLUSTER_IP}:${SVC_PORT} 2>&1

# If service works by ClusterIP but not by name → DNS problem
# If service works by pod IP but not by ClusterIP → kube-proxy problem
# If neither works → pod/network problem

Section 3: Egress / External Connectivity

Pods reaching endpoints outside the cluster.

Testing External Connectivity

These examples use httpbin.org as a public test endpoint. Substitute any reachable external host in air-gapped or restricted environments.

# Test general internet access
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://httpbin.org/get 2>&1 | head -5

# Test specific external API
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 https://${EXTERNAL_HOST}${PATH} 2>&1 | head -5

# If wget/curl unavailable, test TCP connectivity
kubectl exec -n ${NS} ${POD} -- sh -c "echo | nc -w5 ${EXTERNAL_HOST} ${PORT}" 2>&1

Common Egress Failures

SymptomLikely CauseDiagnostic
All pods can't reach externalCluster-wide egress issueCheck node internet access, NAT gateway
Only some pods blockedEgress network policykubectl get networkpolicies -n ${NS}
HTTPS fails, HTTP worksTLS interception or proxyCheck for corporate proxy env vars
Timeout to specific hostFirewall or security groupCheck cloud provider firewall rules
Connection refused to externalTarget service rejectingVerify external service is up

NAT and SNAT

# Pods typically use the node's IP for outbound traffic (SNAT)
# Verify the node has internet access

# Check if a pod sees the expected source IP
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://httpbin.org/ip 2>&1

# If pods use an egress gateway (Istio or CNI-specific), check its health
kubectl get pods -n istio-system -l istio=egressgateway 2>/dev/null

Proxy Configuration

# Check if pods expect a proxy
kubectl exec -n ${NS} ${POD} -- sh -c 'echo "HTTP_PROXY=$HTTP_PROXY HTTPS_PROXY=$HTTPS_PROXY NO_PROXY=$NO_PROXY"'

# Check if cluster-internal traffic is excluded from proxy
# NO_PROXY should include: .svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

Section 4: Ingress Connectivity

External traffic reaching services through an ingress controller.

Ingress Controller Health

# Find the ingress controller pods (common labels)
kubectl get pods -A -l app.kubernetes.io/name=ingress-nginx -o wide
kubectl get pods -A -l app=istio-ingressgateway -o wide
kubectl get pods -A -l app.kubernetes.io/name=traefik -o wide

# Check controller service (must have an external IP or be NodePort)
kubectl get svc -A | grep -E 'ingress|gateway'

# Controller logs
kubectl logs -n ${INGRESS_NS} -l app.kubernetes.io/name=ingress-nginx --tail=100

Ingress Resource Configuration

# List all ingress resources
kubectl get ingress -A

# Describe a specific ingress
kubectl describe ingress ${INGRESS_NAME} -n ${NS}

# Check the backend service references
kubectl get ingress ${INGRESS_NAME} -n ${NS} -o jsonpath='{range .spec.rules[*]}{.host}{"\t"}{range .http.paths[*]}{.path}\t{.backend.service.name}:{.backend.service.port.number}{"\n"}{end}{end}'

Common Ingress Failures

SymptomLikely CauseDiagnostic
502 Bad GatewayBackend service has no ready endpointsCheck kubectl get endpoints ${BACKEND_SVC} -n ${NS}
503 Service UnavailableBackend service doesn't existVerify service name and namespace in ingress spec
404 Not FoundPath routing mismatchCheck pathType (Prefix vs Exact) and path values
Connection refused on port 80/443Ingress controller not running or no external IPCheck controller pods and service
TLS handshake failureWrong or missing TLS secretkubectl get secret ${TLS_SECRET} -n ${NS} — verify it exists
Works by IP, not by hostnameDNS for the hostname not pointing to ingressCheck external DNS records

TLS Termination

# Verify the TLS secret exists and has data
kubectl get secret ${TLS_SECRET} -n ${NS} -o jsonpath='{.data}' | jq 'keys'

# Check ingress TLS configuration
kubectl get ingress ${INGRESS_NAME} -n ${NS} -o jsonpath='{.spec.tls}'

Section 5: Network Policies

Network policies control traffic flow between pods and to/from external endpoints. Policies are enforced by the CNI plugin.

Listing and Inspecting Policies

# All policies in a namespace
kubectl get networkpolicies -n ${NS}

# Policy details
kubectl describe networkpolicy ${POLICY_NAME} -n ${NS}

# Summary of all policies: name, pod selector, and policy types
kubectl get networkpolicies -n ${NS} -o json | \
  jq -r '.items[] | "\(.metadata.name)\tSelector: \(.spec.podSelector.matchLabels // "all")\tTypes: \(.spec.policyTypes // ["Ingress"] | join(", "))"'

How Network Policies Work

  • No policies = all traffic allowed (default allow)
  • Any policy selecting a pod = all traffic of that type (ingress/egress) is denied by default, except what the policy explicitly allows
  • Multiple policies are additive (union of allowed traffic)
  • Policies are namespace-scoped

Identifying Blocking Policies

# Find policies that affect a specific pod
POD_LABELS=$(kubectl get pod ${POD} -n ${NS} -o json | jq -r '.metadata.labels | to_entries[] | "\(.key)=\(.value)"' | tr '\n' ',')
echo "Pod labels: ${POD_LABELS}"

# Check each policy's podSelector to see if it matches
kubectl get networkpolicies -n ${NS} -o json | \
  jq -r '.items[] | "\(.metadata.name): \(.spec.podSelector)"'

# Check for egress restrictions (blocks outbound traffic)
kubectl get networkpolicies -n ${NS} -o json | \
  jq -r '.items[] | select(.spec.policyTypes[]? == "Egress") | "\(.metadata.name): egress rules: \(.spec.egress // "DENY ALL")"'

Common Policy Issues

ProblemSymptomDiagnostic
Egress policy without DNS exceptionAll DNS lookups timeoutCheck if any egress policy lacks a rule for UDP/TCP 53
Ingress policy missing a source namespaceCross-NS traffic blockedCheck if ingress rules include a namespaceSelector for the source
Default-deny with no allow rulesAll traffic blockedList policies and verify allow rules exist for expected traffic paths
Policy selects wrong podsWrong pods are affectedCompare podSelector.matchLabels against actual pod labels
CNI doesn't support policiesPolicies exist but aren't enforcedVerify CNI supports NetworkPolicy (e.g., Calico, Cilium — Flannel does not)

Verifying CNI Supports Network Policies

# Not all CNIs enforce policies — Flannel does not, Calico and Cilium do
# Check which CNI is installed
kubectl get pods -A | grep -E 'calico|cilium|weave|flannel|canal'

# If flannel: network policies exist but are NOT enforced
# If calico/cilium: policies are enforced

Section 6: Istio Service Mesh

Istio adds a sidecar proxy (Envoy) to each pod, intercepting all traffic. This enables mTLS, traffic routing, and observability — but adds failure modes.

Sidecar Injection Status

# Check if the pod has an istio-proxy container
kubectl get pod ${POD} -n ${NS} -o jsonpath='{.spec.containers[*].name}' | tr ' ' '\n'
# Look for "istio-proxy" in the output

# Check if the namespace has automatic injection enabled
kubectl get namespace ${NS} -o jsonpath='{.metadata.labels.istio-injection}'
# Should return "enabled" if auto-injection is on

# Check for per-pod injection annotation
kubectl get pod ${POD} -n ${NS} -o jsonpath='{.metadata.annotations.sidecar\.istio\.io/inject}'

# List all pods missing sidecars in an injection-enabled namespace
kubectl get pods -n ${NS} -o json | \
  jq -r '.items[] | select(.spec.containers[].name != "istio-proxy") | .metadata.name' | sort -u

mTLS Mode (PeerAuthentication)

# Check mesh-wide mTLS policy
kubectl get peerauthentication -A

# Check namespace-level mTLS policy
kubectl get peerauthentication -n ${NS}

# Describe the policy to see the mode
kubectl describe peerauthentication ${POLICY_NAME} -n ${NS}
ModeBehaviorImpact
PERMISSIVEAccepts both plaintext and mTLSSafe default, allows gradual migration
STRICTRequires mTLS for all trafficNon-mesh pods (no sidecar) cannot communicate
DISABLENo mTLSDisables Istio mTLS for selected workloads

Common mTLS failures:

  • Pod without sidecar → service with STRICT mTLS = connection reset
  • Service with sidecar → service without sidecar but STRICT on source = outbound failure
  • Mixed mode across namespaces with strict policies = cross-namespace failures
# Quick check: does the destination pod have a sidecar?
kubectl get pod ${DEST_POD} -n ${DEST_NS} -o jsonpath='{.spec.containers[*].name}' | grep istio-proxy

# If STRICT mode and destination has no sidecar, that's the problem

VirtualService and DestinationRule

# List VirtualServices in the namespace
kubectl get virtualservice -n ${NS}

# Describe a VirtualService — check hosts, routes, match rules
kubectl describe virtualservice ${VS_NAME} -n ${NS}

# List DestinationRules
kubectl get destinationrule -n ${NS}

# Check for traffic policies (circuit breaking, load balancing, mTLS settings)
kubectl describe destinationrule ${DR_NAME} -n ${NS}

Common misconfigurations:

ProblemSymptomDiagnostic
VirtualService host doesn't match serviceTraffic not routedCompare spec.hosts[] with actual service FQDN
Wrong port in VirtualService route503 or connection refusedVerify route[].destination.port.number matches service port
DestinationRule subset label mismatchNo healthy endpointsCompare subsets[].labels with pod labels
Circuit breaker too aggressiveIntermittent 503sCheck outlierDetection settings in DestinationRule
Retry policy causing amplificationHigh latency, cascading failuresCheck retries in VirtualService

Envoy Proxy Logs and Configuration

# View istio-proxy (Envoy) logs for a pod
kubectl logs ${POD} -n ${NS} -c istio-proxy --tail=100

# Filter for errors and connection failures
kubectl logs ${POD} -n ${NS} -c istio-proxy --tail=500 | grep -iE 'error|refused|timeout|reset|UH|UF|UC|NR'

# Envoy response flags in access logs:
# UH = no healthy upstream (no endpoints)
# UF = upstream connection failure
# UC = upstream connection termination
# NR = no route configured
# RL = rate limited
# DC = downstream connection termination

# Dump Envoy configuration (large output — filter as needed)
kubectl exec ${POD} -n ${NS} -c istio-proxy -- pilot-agent request GET /config_dump 2>/dev/null | head -100

# Check Envoy clusters (upstream endpoints as Envoy sees them)
kubectl exec ${POD} -n ${NS} -c istio-proxy -- pilot-agent request GET /clusters 2>/dev/null | grep "${TARGET_SVC}" | head -20

# Check Envoy listeners
kubectl exec ${POD} -n ${NS} -c istio-proxy -- pilot-agent request GET /listeners 2>/dev/null | head -50

Istio Proxy Sync Status

# Check if istiod is running
kubectl get pods -n istio-system -l app=istiod

# Verify proxy config is synced (equivalent to istioctl proxy-status)
# Check istiod logs for push errors
kubectl logs -n istio-system -l app=istiod --tail=200 | grep -iE 'error|push|timeout|reject'

# Check proxy connection to istiod from a sidecar
kubectl exec ${POD} -n ${NS} -c istio-proxy -- pilot-agent request GET /ready 2>&1

Gateway Configuration (Istio Ingress)

# List Istio Gateways
kubectl get gateway -A

# Describe the gateway
kubectl describe gateway ${GW_NAME} -n ${NS}

# Check the istio-ingressgateway pods
kubectl get pods -n istio-system -l istio=ingressgateway -o wide

# Check the ingressgateway service (external IP)
kubectl get svc istio-ingressgateway -n istio-system

# Verify VirtualService is bound to the gateway
kubectl get virtualservice -n ${NS} -o json | \
  jq -r '.items[] | "\(.metadata.name): gateways: \(.spec.gateways // "none")"'

Common Istio Connectivity Patterns

ScenarioRoot CauseDiagnostic
Pod-to-service works without sidecar, fails with sidecarmTLS STRICT on destination, no sidecar on sourceCheck PeerAuthentication mode and sidecar presence
503 errors after deploying VirtualServiceRoute destination doesn't match a real service/subsetVerify destination host and subset labels
Intermittent 503 with UH flagOutlier detection ejecting endpointsCheck DestinationRule outlierDetection settings
External service calls fail after mesh enrollmentIstio blocks unknown external traffic by defaultCheck meshConfig.outboundTrafficPolicy.mode (ALLOW_ANY vs REGISTRY_ONLY)
Traffic ignores VirtualService rulesVirtualService host doesn't match or not bound to gatewayVerify spec.hosts and spec.gateways
Connection reset between namespacesPeerAuthentication mismatch across namespacesCheck per-namespace PeerAuthentication policies

Section 7: CNI and kube-proxy

The CNI plugin manages pod networking (IP allocation, routing). kube-proxy manages Service ClusterIP routing via iptables or IPVS rules.

CNI Health

# Identify the CNI plugin
kubectl get pods -A | grep -E 'calico-node|cilium|weave-net|flannel|canal|kube-router'

# Check CNI pod status
kubectl get pods -n kube-system -l k8s-app=calico-node -o wide 2>/dev/null
kubectl get pods -n kube-system -l k8s-app=cilium -o wide 2>/dev/null

# CNI pod logs (pick the relevant one)
kubectl logs -n kube-system -l k8s-app=calico-node --tail=100 2>/dev/null | grep -iE 'error|fail|warn'
kubectl logs -n kube-system -l k8s-app=cilium --tail=100 2>/dev/null | grep -iE 'error|fail|warn'

# Check node status — NotReady often indicates CNI failure
kubectl get nodes -o wide

kube-proxy

# kube-proxy pod status
kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

# kube-proxy mode (iptables or IPVS)
kubectl get configmap kube-proxy -n kube-system -o jsonpath='{.data.config\.conf}' | grep mode

# kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=100 | grep -iE 'error|fail|warn'

iptables Service Rules

These commands require direct node access (SSH or a node-shell pod). They cannot be run via kubectl exec into application pods.

# Check if iptables rules exist for a service ClusterIP
iptables-save | grep ${CLUSTER_IP}

# Count KUBE-SERVICES rules (should have one per service port)
iptables-save | grep KUBE-SERVICES | wc -l

# Check conntrack table size (full table causes connection drops)
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

CNI and kube-proxy Issues

SymptomLikely CauseDiagnostic
Pod stuck in ContainerCreatingCNI failed to assign IPkubectl describe pod — check events for CNI errors
Node NotReadyCNI plugin not running on nodeCheck CNI pods on that node
ClusterIP unreachablekube-proxy not running or iptables rules missingCheck kube-proxy pods and logs
Intermittent connection dropsConntrack table fullCompare nf_conntrack_count with nf_conntrack_max
Cross-node pod traffic failsOverlay network (VXLAN/WireGuard/IPIP) misconfiguredCheck CNI tunnel interface and node-to-node connectivity

Common Mistakes

MistakeWhy It FailsInstead
Assuming DNS failure when service has no endpointsDNS resolves the ClusterIP correctly, but no pods back the service — connection refusedCheck kubectl get endpoints before blaming DNS or network
Testing with ping instead of wget/curlICMP is often blocked by network policies; services don't respond to pingUse wget -qO- --timeout=5 or curl --connect-timeout 5 to test TCP
Deploying test pods to diagnoseMay lack permissions, creates resources, adds cleanupUse kubectl exec into existing running pods
Ignoring the service port vs targetPort distinctionClient connects to service port, traffic goes to targetPort on the pod — a mismatch causes refused connectionsVerify all three ports: service port, targetPort, and containerPort
Applying egress network policy without DNS exceptionAll DNS lookups timeout because UDP/TCP 53 is blockedCheck egress policies for missing UDP/TCP 53 rules
Blaming Istio before checking basicsPod-to-pod and service fundamentals are broken regardless of meshRun the quick health check (non-Istio) first, then check Istio-specific issues
Checking PeerAuthentication without verifying sidecar presenceSTRICT mTLS requires both sides to have sidecarsConfirm istio-proxy container exists in both source and destination pods
Restarting kube-proxy as first stepHides the evidence (logs, metrics) and rarely fixes the real issueCollect logs, check iptables rules, then restart if needed

MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

  • mcp__flux-operator-mcp__get_kubernetes_resources - Query services, endpoints, pods, ingress, network policies, and Istio resources
  • mcp__flux-operator-mcp__get_kubernetes_logs - Retrieve logs from application pods, istio-proxy sidecars, CNI pods, and kube-proxy
  • mcp__flux-operator-mcp__get_kubernetes_metrics - Check resource consumption for CNI and kube-proxy pods

Behavioural Guidelines

  1. Confirm DNS works first — Before diagnosing network issues, verify DNS resolution is healthy. If pods can't resolve names, start with k8s-dns-troubleshooting.
  2. Run the health check first — Complete the Quick Network Health Check before deep-diving. Present the summary so the user sees the overall picture.
  3. Never deploy test workloads — Use kubectl exec into existing running pods. If no pods have network tools, use indirect methods (logs, endpoint checks, pod describe).
  4. Work from layer 3 up — Verify pod IPs assigned → pod-to-pod works → service endpoints populated → service routing works → ingress configured → mesh rules correct.
  5. Distinguish service failure from network failure — If endpoints are empty, the problem is the application (not ready) or label mismatch, not the network.
  6. Check network policies early — Missing DNS egress rules and overly restrictive ingress rules are among the most common connectivity issues.
  7. Check Istio only after basics pass — If pod-to-pod and service fundamentals are broken, Istio is not the cause. Diagnose the mesh layer only when non-mesh connectivity works.
  8. Never expose secrets — List Secret names and ConfigMap keys. Never decode or print secret values.
  9. Collect logs before restarting — Envoy, kube-proxy, and CNI logs are ephemeral. Capture them before any restart.
  10. Report what you find, not what to change — This skill is diagnostic. Present findings and let the user decide on remediation.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.