Kubernetes triage

Find out where a cluster is unhealthy when nobody knows yet, rank the problems, and hand each one to the right place. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Triage is a first look, not a root-cause analysis. For each problem, report it, show the evidence, propose a first fix, and name the skill that investigates it fully. Do not dig deeper than the short first look this skill describes.

The commands use placeholders: my-namespace, my-app, my-pod and my-node. Replace them with the real names. Never read Secrets. If output contains passwords, tokens or keys, do not repeat them.

When to use

• The user reports a problem without naming a workload: "the cluster is broken", "everything is slow", "lots of pods are failing", "users say the app is down".
• Something changed cluster-wide (an upgrade, a new node pool, a network change) and "something is off".
• You were asked for a general health check of a cluster.

When the user names one workload that is unhealthy, start with k8s-workload-troubleshooting instead.

Hand-offs, by skill name. For each issue you find, name the skill that goes deeper. If it is not installed, give the issue, its evidence, a first fix and the short first look below, then tell the user which skill would help, and stop there:

• Pods crash, restart, stay Pending, fail probes, or a rollout or Job is stuck: k8s-workload-troubleshooting.
• A node is NotReady, under memory, disk or PID pressure, cordoned, or pods are being evicted: k8s-node-troubleshooting.
• Pods are healthy but traffic does not arrive (Service with no endpoints, DNS, NetworkPolicy, Ingress): k8s-network-troubleshooting.
• A PersistentVolumeClaim is Pending, or volumes fail to attach or mount: k8s-storage-troubleshooting.
• The API server refuses requests (failed calling webhook, an APIService that is not True, a namespace stuck Terminating): k8s-api-troubleshooting.
• On AKS, the error comes from Azure (SyncLoadBalancerFailed, a ProvisioningFailed with an Azure error code, AADSTS errors, a cluster or node pool Failed): k8s-aks-troubleshooting.

Investigate

Work through the steps in order. They are cheap and cluster-wide; do all of them before you rank anything.

1. Check what you can see. A user with access to one namespace cannot list nodes or other namespaces. Know this before you read any result.

kubectl auth can-i list nodes
kubectl auth can-i list pods --all-namespaces
kubectl auth can-i list events --all-namespaces

If a check says no, or a command below fails with Forbidden, say exactly what you could not see, and narrow the sweep to the namespaces you can read. Start with the namespace your kubeconfig points at: commands without -n use it, and users with access to one namespace usually have it set.

kubectl auth can-i list pods
kubectl get pods
kubectl auth can-i list pods -n my-namespace

If none of these works and you know no namespace, ask the user which one to look at. Never report something you could not see as healthy.

2. Nodes. Look at STATUS (Ready, NotReady, and SchedulingDisabled for a cordoned node), then describe every node that is not plainly Ready, and read its Conditions, Taints and Allocated resources.

kubectl get nodes -o wide
kubectl describe node my-node

3. Pods that are not running. Across every namespace you can read:

kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get pods -A | grep -v -E "Running|Completed"
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'

The first command finds Pending, Failed and Unknown pods. The second also catches pods whose phase is Running but whose STATUS column says CrashLoopBackOff, ImagePullBackOff, Error or Terminating. The third puts the most restarted pods last. Also look for Running pods that are not ready, such as 0/1 in the READY column.

When a node is not Ready, list the pods on it:

kubectl get pods -A --field-selector=spec.nodeName=my-node -o wide

4. Warning events. For each one, check whether the object it is about is still unhealthy now. Only those count as issues; a Warning about an object that has since recovered is history, even if it is recent (a new cluster has several from start-up).

kubectl events -A --types=Warning

5. CPU and memory. kubectl top works only when metrics-server is installed and running. If it fails, write "metrics not available" in the overall status and go on; Allocated resources from step 2 still shows requests and limits against capacity.

kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl top pods -A --sort-by=cpu

6. Controllers short of replicas. A Deployment, StatefulSet or DaemonSet whose ready count is below the desired count is affected even when its pods look fine one by one.

kubectl get deployments -A
kubectl get statefulsets -A
kubectl get daemonsets -A

7. PersistentVolumeClaims and Services, only if steps 3 and 4 point there (pods Pending with volume events, or an app that is up but unreachable):

kubectl get pvc -A
kubectl get endpointslices -n my-namespace

8. Pick the area of each issue and its next skill, from this table:

A short first look inside a container can confirm a network issue. Ask first: show the command, and run it only after the user agrees (pick a running pod in the namespace that has nslookup; many minimal images do not):

Ask first: the agent runs this only after the user approves it.

kubectl exec my-pod -n my-namespace -- nslookup my-app.my-namespace.svc.cluster.local

Read the results

Rank by severity. Report issues in this order, most urgent first:

1. Nodes. NotReady or Unknown, MemoryPressure, DiskPressure or PIDPressure True, NetworkUnavailable, cordoned (SchedulingDisabled).
2. Crashing pods. CrashLoopBackOff, OOMKilled, Error, ErrImagePull or ImagePullBackOff, CreateContainerConfigError.
3. Pods that do not become ready. Pending, Running but not ready, controllers with fewer ready replicas than desired, Services without ready endpoints.
4. Early warnings. Warning events about objects that are still unhealthy, and resource pressure that has not broken anything yet (a node above about 90% of its memory, pods at their CPU or memory limit).

Within a tier, more affected nodes or pods rank higher. One cause, one issue: when a node problem explains pod symptoms (the pods on a NotReady node show Unknown, Terminating, or are being recreated elsewhere), report the node issue once and list the pods as its evidence. Do not report those pods as separate application failures.

Read nodes. Ready is Unknown when the node controller has not heard from the node within its grace period (50 seconds by default): the kubelet is down or the node is unreachable. Ready is False when the kubelet reports the node is not healthy. A pressure condition True means the kubelet may evict pods from that node. The taints node.kubernetes.io/not-ready, node.kubernetes.io/unreachable and node.kubernetes.io/unschedulable (a cordoned node) show the same state to the scheduler. By default, pods on a NotReady or unreachable node are evicted after five minutes; until then they look stuck, and pods of a StatefulSet can stay Terminating.

Read events. Typical reasons: FailedScheduling, BackOff, Failed (image pull), Unhealthy (probes), FailedMount, Evicted, NodeNotReady, OOMKilling. An event is evidence only when its object is unhealthy now: a Running, ready pod that once failed a readiness probe is not an issue.

Write the report in this shape:

1. Overall status. Nodes (how many, how many Ready), pods (how many are not healthy, in how many namespaces), metrics (available or not), and what you could not see because of access.
2. Issues, ranked. For each: one sentence for the issue, the evidence you saw (the status, the condition, the event reason and message, the number), the next skill, and a first fix in the format under Proposed fix.
3. When the sweep is clean, do not invent a problem. Say what you checked and that it looked healthy from where you could see, then ask the user to narrow it down: which app, namespace or URL, what exactly they saw (an error message, a timeout, slowness), and since when.

Proposed fix

Triage proposes first fixes only. Show each one to the user first: what it changes and how to undo it. A reversible fix goes under "Ask first" and runs only after the user says yes. Anything destructive or hard to undo is never run by you; say why and hand it over. Prefer fixing the source the workload is deployed from (manifest, Helm values, GitOps repository) when there is one. Write each undo with the values you found.

Uncordon a node that was left cordoned after maintenance and is otherwise healthy:

Ask first: the agent runs this only after the user approves it.

kubectl uncordon my-node
# Undo: kubectl cordon my-node

Roll back a Deployment whose pods started failing right after a rollout. First note the current revision with kubectl rollout history deployment/my-app -n my-namespace; the undo below returns to it by number (here revision 3), so it holds even if another rollout happens in between:

Ask first: the agent runs this only after the user approves it.

kubectl rollout undo deployment/my-app -n my-namespace
# Undo: kubectl rollout undo deployment/my-app -n my-namespace --to-revision=3

Force-delete a pod stuck Terminating on a node that is gone for good. Never run this yourself: for a StatefulSet pod it can start a second copy while the first still runs on a node that is only unreachable. A human must first confirm the node is really gone.

Proposed fix: review before running. The agent does not run this.

kubectl delete pod my-pod -n my-namespace --force --grace-period=0
# Undo: none; the pod's controller recreates it on another node

For a node that stays NotReady, a workload that keeps crashing, a Service without endpoints or a stuck volume, the fix needs the deeper skill named in the issue.

Sources

Checked against the Kubernetes 1.37 documentation:

• Nodes and Node Status: conditions, the 50-second grace period before Unknown, cordoned nodes shown as SchedulingDisabled, eviction five minutes after Unknown.
• Node-pressure Eviction: pressure conditions and eviction signals.
• Taints and Tolerations: the node.kubernetes.io/* taints, the default 300-second tolerations, pods still running on a partitioned node.
• Pod Lifecycle: pod phases (Unknown when the node cannot be reached), Terminating, CrashLoopBackOff.
• Field Selectors: status.phase and spec.nodeName for pods.
• kubectl events: --types, -A.
• Resource metrics pipeline and kubectl top: kubectl top needs metrics-server.
• Troubleshooting Clusters: listing and describing nodes.
• Using RBAC Authorization: the view role and kubectl auth can-i.