agentleFS
k8s-servantk8s-servant/k8s-skills/skills/k8s-upgrade-readiness/SKILL.md

k8s-upgrade-readiness

Skill38 starsChanged 16 days agoApache-2.0View on GitHub

Use when a Kubernetes cluster is about to be upgraded to its next minor version, or someone asks "is the cluster ready to upgrade?", "what will break when we upgrade?" or why a node drain hangs during an upgrade: PodDisruptionBudgets with ALLOWED DISRUPTIONS 0, pods with no controller ("cannot delete Pods that declare no controller"), admission webhooks with failurePolicy Fail and no endpoints, deprecated or removed APIs still in use (apiserver_requested_deprecated_apis, CRD versions marked deprecated), kubelet and control plane version skew, nodes on cgroup v1 or containerd 1.x, and add-ons that do not support the target version. Reports what to fix first. Read-only: never upgrades, drains or changes anything itself.

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check a cluster before a Kubernetes minor version upgrade, and report what has to be fixed first.

    line 21
  2. Distinctive:

    Before a minor version upgrade, or when planning one.

    line 41
  3. Very specific:

    After an upgrade, clients fail with no matches for kind or the server could not find the requested resource.

    line 44
  4. Distinctive:

    Run every step: the goal is a complete list, not the first problem.

    line 63

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Upgrade readiness

Check a cluster before a Kubernetes minor version upgrade, and report what has to be fixed first. Read only. Run the investigation commands freely. Never upgrade, drain, cordon or evict anything, and never change a workload, a PodDisruptionBudget or a webhook yourself: everything here is a proposed fix for the user and the owners of each object. Never read Secrets.

An upgrade replaces the control plane first, one minor version at a time, and then every node: each node is cordoned and drained (its pods are evicted and recreated elsewhere), then replaced or upgraded. Managed clusters (AKS, EKS, GKE) do both for you, but they drain nodes the same way. So most of what breaks an upgrade is in the cluster: pods that cannot be evicted, pods that cannot be recreated, and clients or add-ons that use an API or a node setup the new version no longer supports.

The commands use placeholders: my-namespace, my-app, my-node. Replace them with the real names. Some checks need more than the built-in view role (nodes, CRDs, webhook configurations, the API server's metrics); if a command is forbidden, say which check it was and give the user the command to run.

When to use

  • Before a minor version upgrade, or when planning one.
  • A node drain during an upgrade hangs, or a managed upgrade fails or stalls on evictions.
  • After an upgrade, clients fail with no matches for kind or the server could not find the requested resource.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • On AKS: available versions, node pool upgrades and upgrades that failed: k8s-aks-troubleshooting.
  • A webhook or an APIService is already failing requests today: k8s-api-troubleshooting.
  • A node is NotReady or under pressure today: k8s-node-troubleshooting.
  • Pods are already failing today: k8s-workload-troubleshooting, or k8s-triage for a first look.
  • The Gateway API CRDs and their versions: k8s-gateway-api-troubleshooting.
  • Standing risks that are not about the upgrade (privileged pods, broad RBAC bindings, leftovers), for a periodic review: k8s-cluster-review.

Investigate

Run every step: the goal is a complete list, not the first problem. Note the current version and the target version (the next minor) first.

1. Versions and skew: the API server, every kubelet, the container runtime, the kernel and the OS image.

kubectl version
kubectl get nodes -o wide
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" kubelet="}{.status.nodeInfo.kubeletVersion}{" runtime="}{.status.nodeInfo.containerRuntimeVersion}{" kernel="}{.status.nodeInfo.kernelVersion}{" os="}{.status.nodeInfo.osImage}{"\n"}{end}'
kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].image}{"\n"}{end}'

2. Nodes that are not ready, cordoned or under pressure.

kubectl get nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" unschedulable="}{.spec.unschedulable}{" "}{range .status.conditions[?(@.status=="True")]}{.type}{" "}{end}{"\n"}{end}'

3. What a drain cannot evict: PodDisruptionBudgets that allow no disruption, and pods with no controller.

kubectl get pdb -A
kubectl get pdb -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" allowed="}{.status.disruptionsAllowed}{" healthy="}{.status.currentHealthy}/{.status.desiredHealthy}{" selector="}{.spec.selector}{" unhealthyPodEvictionPolicy="}{.spec.unhealthyPodEvictionPolicy}{"\n"}{end}'
kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,NODE:.spec.nodeName
kubectl get deployments,statefulsets -A

4. What would refuse the recreated pods: admission webhooks that fail closed, and whether their Services have endpoints.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations -o jsonpath='{range .items[*]}{range .webhooks[*]}{.name}{" failurePolicy="}{.failurePolicy}{" service="}{.clientConfig.service.namespace}/{.clientConfig.service.name}{" rules="}{.rules}{" namespaceSelector="}{.namespaceSelector}{"\n"}{end}{end}'
kubectl get mutatingwebhookconfigurations -o jsonpath='{range .items[*]}{range .webhooks[*]}{.name}{" failurePolicy="}{.failurePolicy}{" service="}{.clientConfig.service.namespace}/{.clientConfig.service.name}{" rules="}{.rules}{" namespaceSelector="}{.namespaceSelector}{"\n"}{end}{end}'
kubectl get endpointslices -n my-namespace

5. Deprecated APIs still in use: requests the API server has served for deprecated versions since it started, and custom resource versions marked deprecated or still stored.

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
kubectl get crds -o jsonpath='{range .items[*]}{.metadata.name}{" stored="}{.status.storedVersions}{" deprecated="}{.spec.versions[?(@.deprecated==true)].name}{"\n"}{end}'
kubectl api-versions

6. Add-ons and their versions: what runs as part of the cluster, to check against each project's supported Kubernetes versions.

kubectl get deployments,daemonsets,statefulsets -A -o jsonpath='{range .items[*]}{.kind}{" "}{.metadata.namespace}/{.metadata.name}{" "}{.spec.template.spec.containers[*].image}{"\n"}{end}'
kubectl get apiservices

7. Open the matching reference for the details:

What you checkReference
Version skew, the upgrade order, cgroup v1, containerd 1.x, deprecated and removed APIs, CRD stored versions, add-on compatibilityreferences/versions-and-apis.md
PodDisruptionBudgets, unmanaged pods, local storage, single replicas, webhooks during a drainreferences/drains.md

Read the results

What the common findings mean (Kubernetes 1.37):

  • A PDB with ALLOWED DISRUPTIONS 0 while its pods are healthy (for example minAvailable equal to the replicas, or maxUnavailable: 0): no pod it selects can be evicted, so a drain of a node running one of them waits forever, and managed upgrades stall or fail on it. A PDB with 0 because pods are unhealthy blocks the same way unless its unhealthyPodEvictionPolicy is AlwaysAllow.
  • A pod with no owner (OWNER <none>): kubectl drain refuses it with cannot delete Pods that declare no controller (use --force to override); with --force it is deleted and nothing recreates it. Pods owned by a Node are static (mirror) pods, and DaemonSet pods are skipped by drains; both are fine.
  • A pod with an emptyDir volume: drain refuses it with cannot delete Pods with local storage (use --delete-emptydir-data to override); the data is lost on eviction. Usually expected; say so only if the data matters.
  • A webhook with failurePolicy: Fail whose Service has no endpoints: requests it matches are refused. If it matches pods, every pod a drain evicts in those namespaces cannot be recreated, although the running ones look fine today.
  • A kubelet newer than the API server, or more than three minor versions older: outside the version skew policy. Its upgrade order is stricter: before the API server moves to the target, every kubelet should be at most two minor versions older than the target. Upgrade those nodes first. The API server itself moves one minor version at a time.
  • A containerRuntimeVersion of containerd://1.x with a target of 1.36 or later: the kubelet no longer supports containerd 1.x; upgrade containerd to 2.0 or later first. A kernel older than 5.8, or an OS image that still uses cgroup v1: since 1.35 the kubelet does not start on cgroup v1 by default.
  • apiserver_requested_deprecated_apis lines: a client used that group, version and resource since the API server started. A removed_release at or below the target version means that client breaks after the upgrade. version="v1" resource="endpoints" is the core Endpoints API, deprecated in favour of EndpointSlices with no removal planned.
  • A CRD whose storedVersions contains a version marked deprecated, or a version the next release of its add-on drops: objects are still stored in that version. They must be migrated, and storedVersions trimmed, before the add-on removes it.
  • Add-on versions: compare each against the project's compatibility table for the target version; an add-on that does not support it is upgraded first.

Report the result as a list, most important first: blockers (the upgrade will fail or cause an outage), then risks (downtime, data loss, a client that will break), then what was checked and found fine. For each, the object, the evidence and the proposed fix.

Proposed fix

Show each fix to the user; the agent runs none of them. Every finding here belongs to the owner of the object (an application team, a platform team), and changing a PDB, a webhook or a workload's replicas during an upgrade window affects availability. Write each undo with the values you found.

Let a single-replica app keep a pod running while another starts (two or more replicas; the PDB then allows one disruption):

Proposed fix: review before running. The agent does not run this.

kubectl scale deployment my-app -n my-namespace --replicas=2
# Undo: kubectl scale deployment my-app -n my-namespace --replicas=1

Allow one disruption instead of keeping every pod (for the app's owners; prefer the change in the app's source):

Proposed fix: review before running. The agent does not run this.

kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/minAvailable"}, {"op": "add", "path": "/spec/maxUnavailable", "value": 1}]'
# Undo: kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/maxUnavailable"}, {"op": "add", "path": "/spec/minAvailable", "value": 1}]'

A pod with no controller is recreated under a Deployment (or a Job) by its owner, or deleted if nobody needs it, before the upgrade. A webhook without endpoints gets its backend back before the upgrade (see k8s-api-troubleshooting). Clients and manifests that use a removed API are moved to its replacement version first. The references give the commands for each.

Sources

Checked against the Kubernetes 1.37 documentation, and on a Kubernetes 1.37 kind cluster:

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.