Kubernetes node troubleshooting

Find out why a node is unhealthy, or why pods will not be scheduled onto it, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

You see the node only through the Kubernetes API. Checks on the node itself (the kubelet, the container runtime, the disk) need someone with access to the machine, so they are always proposed fixes for a human.

The commands use placeholders: my-node, my-namespace, my-app and my-pod. Replace them with the real names. Never read Secrets. If output contains passwords, tokens or keys, do not repeat them.

When to use

• kubectl get nodes shows a node NotReady, Unknown or SchedulingDisabled.
• A node reports MemoryPressure, DiskPressure or PIDPressure, or pods were Evicted.
• Pods stay Pending and the FailedScheduling event names taints, Insufficient cpu or Insufficient memory, or unschedulable nodes.
• The cluster autoscaler does not add a node for Pending pods, or does not remove an empty one.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

• The nodes are healthy and have room, and the pod itself fails (crash, image, probe, quota): k8s-workload-troubleshooting.
• A volume cannot attach or mount on the node: k8s-storage-troubleshooting.
• NetworkUnavailable is True, or the CNI pods on the node fail: k8s-network-troubleshooting.
• On AKS, a node pool or the cluster is Failed, or an upgrade or scale fails in Azure (quota, VM capacity, a drain blocked by a PodDisruptionBudget): k8s-aks-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. List the nodes. STATUS shows Ready, NotReady or Unknown, plus SchedulingDisabled when a node is cordoned.

kubectl get nodes -o wide

2. Describe the node. Read Conditions (status, reason, LastHeartbeatTime), Taints, Unschedulable, Lease (RenewTime), Capacity and Allocatable, Non-terminated Pods with their requests, and Allocated resources.

kubectl describe node my-node

3. See the pods on the node, in every namespace you can read.

kubectl get pods -A -o wide --field-selector spec.nodeName=my-node
kubectl get pods -A --field-selector spec.nodeName=my-node,status.phase=Failed

The second command lists failed pods there, which include Evicted ones.

4. Read the node's events and its heartbeat. Node events are kept in the default namespace; the heartbeat is a Lease named after the node. The built-in view role cannot read Leases; if the second command is forbidden, use LastHeartbeatTime of the conditions in describe node instead.

kubectl get events -A --field-selector involvedObject.kind=Node,involvedObject.name=my-node
kubectl get lease my-node -n kube-node-lease

5. For a Pending pod, read why the scheduler refused it, and what it asks for.

kubectl describe pod my-pod -n my-namespace
kubectl get pod my-pod -n my-namespace -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.tolerations}{"\n"}{.spec.containers[*].resources.requests}'

6. Look at usage, if the cluster has metrics-server. top shows usage, not requests; the scheduler uses requests. Without metrics-server it fails with "Metrics API not available"; say so and go on.

kubectl top node my-node

7. Open the matching reference and follow its steps:

Read the results

What the common findings mean (Kubernetes 1.37):

• Ready Unknown: the node controller has not heard from the kubelet within node-monitor-grace-period (50 seconds by default). The kubelet is down, or the node cannot reach the API server. The Lease in kube-node-lease, normally renewed every 10 seconds, stops changing. The node gets the node.kubernetes.io/unreachable taints.
• Ready False: the kubelet is running and reports the node is not healthy (for example the container runtime or the network plugin is not ready). The reason and message say which. The node gets the node.kubernetes.io/not-ready taint.
• Pods on a NotReady node keep their status for a while. Pods tolerate the not-ready and unreachable NoExecute taints for 300 seconds by default (added by the DefaultTolerationSeconds admission controller), then they are evicted. DaemonSet pods tolerate them without a time limit and are not evicted.
• MemoryPressure, DiskPressure or PIDPressure True: the kubelet crossed an eviction threshold. It adds a NoSchedule taint for that pressure, frees resources, then evicts pods. Node-pressure eviction does not respect PodDisruptionBudgets.
• FailedScheduling with 0/N nodes are available: ...: each part counts the nodes refused for one reason, for example 1 node(s) had untolerated taint(s), 2 Insufficient cpu or 1 node(s) were unschedulable. The text after preemption: says whether evicting lower-priority pods could help.
• Insufficient cpu on an idle node: the scheduler adds up the requests of the pods on the node and compares them with Allocatable, not with actual use. Allocated resources in describe node shows the requests as a percentage of Allocatable.
• SchedulingDisabled, Unschedulable: true: the node is cordoned. New pods do not go there; pods already there keep running.

State the cause in one sentence, then the evidence you saw (the condition and its message, the taint, the event, the numbers), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Cordoning, uncordoning, adding or removing a NoSchedule taint and labelling a node are reversible, so they go under "Ask first" and run only after the user says yes. Draining or deleting a node, force-deleting pods and anything on the node host are never run by you: hand them to the user as proposed fixes and say why. Prefer fixing the source the workload or the node pool is managed from when there is one. Write each undo with the values you found.

Uncordon a node whose maintenance is over and that is Ready:

Ask first: the agent runs this only after the user approves it.

kubectl uncordon my-node
# Undo: kubectl cordon my-node

Remove a taint left from a maintenance that is over (the key, value and effect exactly as describe node shows them):

Ask first: the agent runs this only after the user approves it.

kubectl taint nodes my-node example.com/maintenance=true:NoSchedule-
# Undo: kubectl taint nodes my-node example.com/maintenance=true:NoSchedule

Check the kubelet on a node that stopped reporting. This runs on the node, which you cannot reach; a person with access to the machine runs it:

Proposed fix: review before running. The agent does not run this.

systemctl status kubelet
journalctl -u kubelet --since "30 min ago" --no-pager | tail -n 50
sudo systemctl restart kubelet
# Undo: nothing to undo for the first two lines, they only read; a restart cannot be undone, the kubelet just starts again

Move the pods off a node that will not come back soon. Draining evicts every pod that is not a DaemonSet pod and deletes emptyDir data, so never run it yourself:

Proposed fix: review before running. The agent does not run this.

kubectl drain my-node --ignore-daemonsets --delete-emptydir-data
# Undo: kubectl uncordon my-node (the evicted pods are recreated by their controllers, not moved back)

Sources

Checked against the Kubernetes 1.37 documentation:

• Node Status: conditions, the 50-second node-monitor-grace-period, Leases in kube-node-lease and their 10-second renewal.
• Nodes: cordon and spec.unschedulable.
• Force Delete StatefulSet Pods: StatefulSet pods on an unreachable node are not replaced on their own.
• Taints and Tolerations and Well-Known Labels, Annotations and Taints: taint effects, the node.kubernetes.io/* taints, taint removal.
• Admission Control: DefaultTolerationSeconds and the 300-second default.
• DaemonSet: tolerations DaemonSet pods get.
• Node-pressure Eviction: signals, default thresholds, reclaim order, no PDBs.
• Reserve Compute Resources for System Daemons and Resource Management for Pods and Containers: Allocatable, scheduling by requests.
• kubectl drain, kubectl cordon, kubectl taint and kubectl top node.
• Resource metrics pipeline and Troubleshooting Clusters: metrics-server, describe node output.
• Cluster Autoscaler FAQ: scale-up and scale-down rules, events, the status ConfigMap, annotations.
• The FailedScheduling reason texts are from the Kubernetes 1.37 scheduler source (tainttoleration plugin), because the docs show only an older example.