Kubernetes Troubleshooting

Take a workload from "it is not working" to a named root cause, using the pod lifecycle as the search order.

Purpose

Use this when:

• A pod is in CrashLoopBackOff, Pending, ImagePullBackOff, ErrImagePull, Init:Error, CreateContainerConfigError, Evicted, or stuck Terminating.
• A kubectl rollout status never completes.
• A workload reports Running but READY 0/1.

Do not use this when:

• Pods are healthy and the problem is traffic not reaching them — use kubernetes-networking.
• The question is "is this workload production-ready?" — use kubernetes-production-readiness.
• The cluster itself is degraded (API server unreachable, all nodes NotReady) — that is a cluster incident; start with incident-triage.

What this skill assumes: a working kubeconfig with at least read access to the namespace, and kubectl on the path.

Operating Procedure

1. Establish context. Confirm which cluster and namespace you are in before running anything else. Getting this wrong is the single most common cause of a wasted investigation — and of changing the wrong environment.
2. Classify the phase. Pod status maps to a layer: Pending is the scheduler, ImagePull* is the kubelet or registry, CrashLoopBackOff is the container, Running 0/1 is the probe or the application. Search only that layer first.
3. Read events before logs. Events explain why Kubernetes did or did not do something. Logs explain what the application did. Most non-crash problems are answered by events alone.
4. Correlate with change. Check the rollout history and recent deploys. Most workload failures follow a change to the image, the config, the resource requests, or the cluster.
5. Form one hypothesis and test it. Prefer a test that eliminates a class of causes: exec into a healthy pod, describe a node, or pull the image locally.
6. Apply the smallest safe change. Fix the manifest or the config, not the running pod. Changes made with kubectl edit on a live object are erased by the next deploy.
7. Verify. See Verification.
8. Record the symptom, the root cause, the change and the rollback command.

Label every statement as Observed (from command output), Assumed, Recommended, or Unverified. If the namespace, image tag or cluster is unknown, run a command or ask — do not infer it from the repository.

Investigation

Step 1 - context and blast radius

kubectl config current-context
kubectl version --output=json | jq -r '.serverVersion.gitVersion'
kubectl get pods -n <namespace> -o wide

current-context tells you which cluster you are about to affect. The server version matters because API deprecations and probe fields differ across minor releases. -o wide gives you the node name and pod IP in one shot — if every failing pod is on one node, the problem is the node, not the workload.

Step 2 - the whole-cluster sweep

kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | tail -40

The first command shows whether this is one workload or many. Many workloads failing at once means a shared dependency: a node, the registry, DNS, a webhook, or a storage backend.

Events are cluster-wide and time-ordered. FailedScheduling points at the scheduler, FailedMount at storage or the CSI driver, Failed to pull image at the registry, FailedCreatePodSandBox at the CNI. Note that events are garbage collected after one hour by default — an empty event list on an old problem is not evidence of health.

Step 3 - describe the pod

kubectl describe pod <pod> -n <namespace>

Read from the bottom. In order of signal:

Step 4 - logs, including the previous container

kubectl logs <pod> -n <namespace> --tail=200
kubectl logs <pod> -n <namespace> --previous --tail=200
kubectl logs <pod> -n <namespace> -c <init-container>

For a crash loop the live container is usually too young to have logged anything useful. --previous returns the logs of the instance that actually died; it is the highest-value command in this entire skill. If it errors with previous terminated container not found, the container has not restarted yet — the pod is failing before the container starts, so go back to events.

Step 5 - correlate with recent change

kubectl rollout history deployment/<name> -n <namespace>
kubectl get deployment <name> -n <namespace> -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'
kubectl get replicaset -n <namespace> --sort-by=.metadata.creationTimestamp | tail -5

A new ReplicaSet created minutes before the symptom began is your prime suspect. The image path tells you whether the tag is mutable (:latest, :main) — a mutable tag means the running code may differ from the last deploy even though nothing in Git changed.

Step 6 - probe and dependency checks

kubectl get pod <pod> -n <namespace> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.readinessProbe}{"\n"}{end}'
kubectl exec -it <healthy-pod> -n <namespace> -- curl -sS -m 5 -o /dev/null -w '%{http_code}\n' \
  http://<pod-ip>:<port>/healthz

Run the probe yourself from inside the cluster. A readiness endpoint that takes longer than timeoutSeconds, or that checks a downstream dependency, will flap the pod out of the Service endpoints without the application ever crashing. This is the most commonly misdiagnosed Kubernetes failure: the application is fine and the probe is wrong, or the probe is honest and the dependency is down.

Decision Tree

Pod is not healthy
├── Pending
│   ├── "Insufficient cpu/memory"      → capacity or oversized requests → kubernetes-resource-debugging
│   ├── "untolerated taint" / nodeSelector → placement constraints
│   ├── "pod has unbound PersistentVolumeClaim" → storage class / PV provisioner
│   └── no events at all               → scheduler down, or the pod is gated
├── ImagePullBackOff / ErrImagePull
│   ├── "unauthorized"                 → imagePullSecrets
│   ├── "manifest unknown"             → tag does not exist
│   └── i/o timeout                    → node egress or registry reachability
├── Init:Error / Init:CrashLoopBackOff → logs of the init container, not the main one
├── CreateContainerConfigError         → missing ConfigMap/Secret key
├── CrashLoopBackOff
│   ├── exit 137 / OOMKilled           → memory limit → kubernetes-resource-debugging
│   ├── exit 1 / 2                     → application error → --previous logs
│   ├── exit 127                       → command or entrypoint not found → container-image-review
│   └── exit 0 in a Deployment         → the process is not long-running; wrong entrypoint
├── Running but 0/1 Ready
│   ├── probe times out                → probe config, slow start, or saturated container
│   └── probe returns 5xx              → the application's own dependency check is failing
└── Terminating and stuck
    ├── finalizers present             → a controller is not releasing the object
    └── node NotReady                  → the kubelet cannot confirm deletion

Common Causes

Safety

Read-only

Everything in Investigation except kubectl exec. get, describe, logs, events, and rollout history cannot change cluster state. Run them freely.

kubectl exec is read-only only if the command you run is. curl against a health endpoint is fine; a shell in a production container is an audited action in most organisations and can change container state.

Potentially disruptive

Each of these needs a stated reason and a rollback before you run it.

• kubectl rollout restart deployment/<name> — recreates every pod in the deployment under the configured maxUnavailable. Blast radius: all replicas. Rollback: none needed, but in-flight requests are dropped if the app does not handle SIGTERM.
• kubectl rollout undo deployment/<name> — returns to the previous ReplicaSet. This is the correct first action when a deploy caused the symptom. Rollback: kubectl rollout undo again.
• kubectl scale deployment/<name> --replicas=N — changes capacity. Rollback: scale back. Note that an HPA will fight a manual scale.
• kubectl delete pod <pod> — the controller recreates it. Destroys the evidence you need. Do this only after capturing describe and --previous logs, never as a first step.
• kubectl cordon <node> / kubectl drain <node> — moves every pod off a node. Drain respects PodDisruptionBudgets and will hang if one cannot be satisfied; that hang is a correct signal, not a problem to force past.

Destructive

This skill proposes no destructive operations. Deleting PersistentVolumeClaims, namespaces or StatefulSets is out of scope here on purpose — those operations destroy data and belong to a deliberate, separately-reviewed change.

Verification

A pod reaching Running is not verification. Verify the workload, at two layers:

Before: 3/8 pods Ready, restart count climbing, 12% 5xx at the ingress
After:  8/8 pods Ready for >10 min, restart count flat, <0.1% 5xx at the ingress

kubectl rollout status deployment/<name> -n <namespace> --timeout=5m
kubectl get pods -n <namespace> -l app=<label> \
  -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount

Then wait. A CrashLoopBackOff fix that has been up for 30 seconds has not been verified — the backoff interval alone reaches 5 minutes. Confirm the restart count is still flat several minutes later, and confirm the user-visible symptom (error rate, latency, queue depth) has recovered in your monitoring, not just in kubectl.

Anti-Patterns

Bad: Restart everything and see if it helps. Better: Identify the failing layer first. A restart destroys the evidence and, if the cause is a bad image or a missing secret, changes nothing except your ability to diagnose it.

Bad: Raise the CPU limit because the pod looks slow. Better: Check container_cpu_cfs_throttled_seconds_total before assuming throttling. CPU limits do not cause OOMKills, and raising a limit that was never being hit only reduces how many pods fit on a node.

Bad: Delete the pod to make the error go away. Better: Understand why it is unhealthy. A pod recreated into the same broken state has cost you the --previous logs and taught you nothing.

Bad: kubectl edit the live Deployment to fix it. Better: Change the manifest in Git and deploy. A live edit is silently reverted by the next sync and produces a failure that nobody can reproduce.

Bad: Conclude "the app is broken" because the container logs an error. Better: Check whether the error is a symptom of a dependency — a database connection refused in the logs usually means the database, DNS, or a NetworkPolicy is the actual fault.

Composition

• kubernetes-networking — pods are healthy but traffic is not reaching them.
• kubernetes-resource-debugging — OOMKills, throttling, requests and limits, capacity.
• container-image-review — exit 127, missing entrypoints, image bloat, wrong base image.
• linux-troubleshooting — node-level disk, memory or process problems.
• production-debugging — the workload is a symptom of a wider outage.

References

• references/pod-status-matrix.md — exhaustive status/reason/exit-code lookup table.
• Kubernetes: Debug Pods — upstream reference for pod phases and conditions.
• Kubernetes: Pod Lifecycle — authoritative definition of phases, conditions and restart policy.