Kubernetes workload troubleshooting

Find out why a pod, Deployment, StatefulSet, DaemonSet, Job or CronJob is unhealthy, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

The commands use placeholders: my-namespace, my-app (the workload) and my-pod (one of its pods). Replace them with the real names. Never read Secrets. If logs or output contain passwords, tokens or keys, do not repeat them.

When to use

• A pod is Pending, ContainerCreating, ErrImagePull, ImagePullBackOff, CrashLoopBackOff, OOMKilled, Error, or Running but not ready (0/1).
• A container keeps restarting, or a probe keeps failing (Unhealthy events).
• A rollout never finishes: kubectl rollout status waits or reports that the Deployment "exceeded its progress deadline".
• A Job fails or never completes, or a CronJob's runs fail.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

• The node is the cause (NotReady, memory, disk or PID pressure, evictions, taints no pod tolerates): k8s-node-troubleshooting.
• A PersistentVolumeClaim is the cause (unbound claim, attach or mount errors): k8s-storage-troubleshooting.
• The pods are healthy and Ready, but traffic still does not arrive (Service, DNS, NetworkPolicy, Ingress): k8s-network-troubleshooting.
• A controller cannot create its pods at all (FailedCreate with failed calling webhook or denied the request), or a namespace stays Terminating: k8s-api-troubleshooting.
• The pods belong to a CloudNativePG Cluster (label cnpg.io/cluster, or an -initdb or -join job pod): k8s-cloudnative-pg-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See the pods and their status. The STATUS, READY and RESTARTS columns say which reference applies.

kubectl get pods -n my-namespace -o wide
kubectl get pods -n my-namespace -l app=my-app

app=my-app is an example label. Take the real selector from the workload, with kubectl get deployment my-app -n my-namespace -o jsonpath='{.spec.selector.matchLabels}' or the Selector: line of kubectl describe deployment my-app -n my-namespace, and use it wherever a command here says -l app=my-app.

2. Describe the failing pod. Read the container State and Last State (reason and exit code), Restart Count, Limits and Requests, the probes, the pod Conditions and the Events at the bottom.

kubectl describe pod my-pod -n my-namespace
kubectl events -n my-namespace --for pod/my-pod

3. Read the logs, of the current container and of the one that ran before the last restart. A crash usually explains itself in the previous container's last lines.

kubectl logs my-pod -n my-namespace --tail=100
kubectl logs my-pod -n my-namespace --previous --tail=100
kubectl logs my-pod -n my-namespace --all-containers --tail=100

4. Look at the owning controller. Find what changed and whether a rollout is stuck.

kubectl get deployment my-app -n my-namespace
kubectl describe deployment my-app -n my-namespace
kubectl rollout status deployment/my-app -n my-namespace --timeout=10s
kubectl rollout history deployment/my-app -n my-namespace
kubectl get replicasets -n my-namespace -l app=my-app
kubectl get deployment my-app -n my-namespace -o jsonpath='{.spec.progressDeadlineSeconds}'

describe deployment does not print the progress deadline; the last command reads it.

For a StatefulSet or DaemonSet use the same describe, rollout status and rollout history commands with statefulset/my-app or daemonset/my-app. For a Job:

kubectl get jobs -n my-namespace
kubectl describe job my-app -n my-namespace
kubectl get pods -n my-namespace -l job-name=my-app

5. Check the namespace's recent warnings, which also show problems of objects that never got a pod (quota, admission, ReplicaSet creation).

kubectl events -n my-namespace --types=Warning

6. Open the matching reference and follow its steps:

Read the results

What the common findings mean (Kubernetes 1.37):

• Pod phase Pending: the pod is accepted but not all containers are set up and ready to run. This covers both waiting to be scheduled and pulling images. If the PodScheduled condition is False with a FailedScheduling event, the scheduler found no node; see pending.md.
• Container state Waiting with reason ErrImagePull or ImagePullBackOff: the image could not be pulled. The pull event's message says why (tag not found, unauthorized, cannot reach the registry). The kubelet keeps retrying with a growing delay, capped at 5 minutes.
• CrashLoopBackOff: the container starts, exits, and the kubelet waits longer before each restart (10s, 20s, 40s, ... capped at 5 minutes by default; reset after 10 minutes of running). The cause is in Last State and in kubectl logs --previous, not in the word CrashLoopBackOff.
• Last State: Terminated, Reason: OOMKilled, Exit Code: 137: the container used more memory than its memory limit and the kernel killed it. Raise the limit or reduce the app's memory use. A node under memory pressure evicting pods is a different cause (an Evicted event and pod status); that one belongs to k8s-node-troubleshooting.
• Reason: Error with another exit code: the app exited by itself. The logs of the previous container say why (missing config, bad argument, failed dependency).
• Running but READY 0/1, no restarts, Unhealthy "Readiness probe failed" events: the readiness probe fails, so the pod is not a ready endpoint of its Services and gets no traffic (unless a Service sets publishNotReadyAddresses: true). The container is not restarted.
• Restarts with Unhealthy "Liveness probe failed" (or startup probe) events: the kubelet kills the container after failureThreshold failures in a row and restarts it.
• Deployment condition Progressing=False, reason ProgressDeadlineExceeded: the new ReplicaSet made no progress for the Deployment's progressDeadlineSeconds (read it from the spec; 600 is only the default when unset). Kubernetes only reports this; it does not roll back. The new ReplicaSet's pods show the real cause (image pull, crash, readiness, quota).
• Job condition Failed, reason BackoffLimitExceeded: pods failed more times than backoffLimit (default 6). Reason DeadlineExceeded: the Job ran longer than activeDeadlineSeconds.

State the cause in one sentence, then the evidence you saw (the event reason and message, the exit code, the condition), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. The fixes here are reversible, so they go under "Ask first" and run only after the user says yes. Anything destructive or hard to undo (deleting a pod or a Job, replace --force) is never run by you: hand it to the user as a proposed fix and say why. Prefer fixing the source the workload is deployed from (manifest, Helm values, GitOps repository) when there is one; a direct command is for when there is not, or for an urgent mitigation. Typical fixes:

Correct an image tag that does not exist (the container name is the one in the pod spec):

Ask first: the agent runs this only after the user approves it.

kubectl set image deployment/my-app app=registry.example.com/my-app:1.4.2 -n my-namespace
# Undo: kubectl set image deployment/my-app app=registry.example.com/my-app:1.4.3 -n my-namespace

Roll a stuck Deployment back to the previous revision. First note the current revision with kubectl rollout history deployment/my-app -n my-namespace; the undo returns to it by number (here revision 3):

Ask first: the agent runs this only after the user approves it.

kubectl rollout undo deployment/my-app -n my-namespace
# Undo: kubectl rollout undo deployment/my-app -n my-namespace --to-revision=3

Raise a memory limit that the app outgrows (choose the value from the app's real use):

Ask first: the agent runs this only after the user approves it.

kubectl set resources deployment/my-app -c app --limits=memory=256Mi --requests=memory=256Mi -n my-namespace
# Undo: kubectl set resources deployment/my-app -c app --limits=memory=32Mi --requests=memory=32Mi -n my-namespace

Point a readiness probe at a path the app serves:

Ask first: the agent runs this only after the user approves it.

kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/"}]'
# Undo: kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/httpGet/path", "value": "/healthz"}]'

Write the undo with the values you actually found, so it restores the current state.

Sources

Checked against the Kubernetes 1.37 documentation:

• Pod Lifecycle: phases, container states, restart backoff, CrashLoopBackOff, pod conditions, probes.
• Liveness, Readiness, and Startup Probes and Configure Liveness, Readiness and Startup Probes: probe fields, defaults, HTTP success range.
• Resource Management for Pods and Containers: OOM kills, CPU throttling, FailedScheduling, OOMKilled with exit code 137.
• EndpointSlices: endpoint ready condition, publishNotReadyAddresses.
• Images: ImagePullBackOff, pull back-off, image pull policy.
• Deployments: progressDeadlineSeconds, ProgressDeadlineExceeded, rollout history and undo.
• Jobs and CronJob: backoffLimit, pod failure back-off, activeDeadlineSeconds, CronJob policies.
• Debug Pods and Debug Running Pods: Pending and Waiting pods, kubectl logs --previous.