k8s-aks-troubleshooting
Skill38 starsChanged 16 days agoApache-2.0View on GitHub
Use when a problem is specific to Azure Kubernetes Service (AKS): a cluster or node pool in "Failed" state, an upgrade or scale that fails ("UpgradeFailed", "violate the pod's disruption budget", "QuotaExceeded", "AllocationFailed"), a LoadBalancer Service stuck at <pending> ("SyncLoadBalancerFailed", "AuthorizationFailed"), an Azure Disk or Files PVC that fails ("ProvisioningFailed", "volume node affinity conflict", "Multi-Attach error", "Mount error(13)"), workload identity errors ("AADSTS700016", "AADSTS70021"), CoreDNS edits that AKS reverts, "error dialing backend", Azure Policy denials, or API server access (authorized IP ranges, private clusters, kubelogin, "User does not have access to the resource in Azure"). Investigates with kubectl and read-only az commands, and proposes fixes. Reads first and asks before changing anything.
In this repository32 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- k8s-api-troubleshootingSkill
- k8s-argocd-troubleshootingSkill
- k8s-autoscaling-troubleshootingSkill
- k8s-cert-manager-troubleshootingSkill
- k8s-cilium-troubleshootingSkill
- k8s-cloudnative-pg-troubleshootingSkill
- k8s-cluster-reviewSkill
- k8s-coredns-troubleshootingSkill
- k8s-envoy-gateway-troubleshootingSkill
- k8s-envoy-troubleshootingSkill
- k8s-external-secrets-troubleshootingSkill
- k8s-fluent-bit-troubleshootingSkill
- k8s-flux-troubleshootingSkill
- k8s-gatekeeper-troubleshootingSkill
- k8s-gateway-api-troubleshootingSkill
- k8s-grafana-troubleshootingSkill
- k8s-helm-troubleshootingSkill
- k8s-istio-troubleshootingSkill
- k8s-kyverno-troubleshootingSkill
- k8s-loki-troubleshootingSkill
- k8s-network-troubleshootingSkill
- k8s-node-troubleshootingSkill
- k8s-opentelemetry-collector-troubleshootingSkill
- k8s-prometheus-troubleshootingSkill
- k8s-rbac-troubleshootingSkill
- k8s-storage-troubleshootingSkill
- k8s-triageSkill
- k8s-upgrade-readinessSkill
- k8s-velero-troubleshootingSkill
- k8s-workload-troubleshootingSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 1 is distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 23
Read first, ask before changing: Run the investigation commands freely: kubectl reads and the read-only az commands below (show, list, get).
- Very specific: line 27
Every az change (update, upgrade, role assignments, anything that writes) is a proposed fix.
- Very specific: line 69
Before an upgrade, a check of everything in the cluster that would block or break it (PDBs, unmanaged pods, webhooks, deprecated APIs, …
- Distinctive: line 74
1. Find the cluster and its state: If the user did not name it, list the clusters the login can see, then read the one in question:
- Very specific: line 143
No matching federated identity record found for presented assertion: the identity exists, but no federated credential matches the token's …
- Very specific: line 163
After the cause of a Failed state is fixed, a reconcile re-applies the cluster's current configuration without changing it:
- Very specific: line 182
Never start another operation while one is still running (az aks operation show-latest shows InProgress), and never edit the scale sets, …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AKS troubleshooting
Find out why something that depends on Azure fails in an AKS cluster, and propose a fix.
Read first, ask before changing. Run the investigation commands freely: kubectl reads
and the read-only az commands below (show, list, get). Run a command under an "Ask
first" line only after the user approves that exact command; if your credentials do not
allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
Every az change (update, upgrade, role assignments, anything that writes) is a proposed
fix.
An AKS cluster lives in two places. Kubernetes objects are in the cluster; the cluster
itself, its node pools (virtual machine scale sets), load balancers, public IPs, disks
and identities are Azure resources. The cluster resource is in the user's resource group;
the resources AKS manages for it are in a separate node resource group (by default
MC_<resource group>_<cluster>_<region>). Many AKS failures show in Kubernetes only as a
short event, while the real error is in Azure: in the cluster's provisioningState, in
the latest operation, or in the activity log.
The commands use placeholders: my-resource-group and my-cluster (the cluster),
my-node-resource-group, my-nodepool, my-namespace, my-app, my-pvc, my-pod,
my-node and my-identity. Replace them with the real names. The az commands need an
Azure CLI login (az account show tells which subscription it uses). Never read Secrets,
and do not print tokens, keys or client secrets from any source.
When to use
- The cluster or a node pool shows
Failed, or an upgrade, scale or update failed. - A LoadBalancer Service has no
EXTERNAL-IP, or traffic through it fails. - A PVC on an Azure Disk or Azure Files class stays
Pending, or a pod cannot mount it. - A pod using workload identity cannot get an Azure token.
- A change to the
corednsConfigMap disappears,kubectl logsorexecfail witherror dialing backend, or Azure Policy denies a pod. - kubectl cannot reach or log in to the cluster.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- A pod that crashes for its own reasons (its logs, probes, OOMKilled):
k8s-workload-troubleshooting. A node that is NotReady or full, or pods that do not fit:k8s-node-troubleshooting. - A PVC problem that is not about Azure (no StorageClass, access modes, a full volume):
k8s-storage-troubleshooting. - A Service with no endpoints, or a NetworkPolicy:
k8s-network-troubleshooting. The Corefile itself:k8s-coredns-troubleshooting. - A
Forbiddenthat comes from Kubernetes RBAC (no Azure suffix):k8s-rbac-troubleshooting. Webhook or admission errors that are not Azure Policy:k8s-api-troubleshooting; Gatekeeper constraints themselves:k8s-gatekeeper-troubleshooting. - Before an upgrade, a check of everything in the cluster that would block or break it
(PDBs, unmanaged pods, webhooks, deprecated APIs, add-ons):
k8s-upgrade-readiness.
Investigate
1. Find the cluster and its state. If the user did not name it, list the clusters the login can see, then read the one in question:
az aks list -o table
az aks show -g my-resource-group -n my-cluster --query "{state:provisioningState, power:powerState.code, version:kubernetesVersion, nodeResourceGroup:nodeResourceGroup, network:networkProfile.networkPlugin, networkMode:networkProfile.networkPluginMode, outbound:networkProfile.outboundType, private:apiServerAccessProfile.enablePrivateCluster, authorizedIpRanges:apiServerAccessProfile.authorizedIpRanges, azureRbac:aadProfile.enableAzureRbac, localAccountsDisabled:disableLocalAccounts, oidcIssuer:oidcIssuerProfile.issuerUrl, workloadIdentity:securityProfile.workloadIdentity.enabled}" -o json2. Node pools and recent operations. A Failed state is about the last operation, not
the running workloads:
az aks nodepool list -g my-resource-group --cluster-name my-cluster --query "[].{name:name, mode:mode, state:provisioningState, power:powerState.code, count:count, min:minCount, max:maxCount, autoscale:enableAutoScaling, vmSize:vmSize, zones:availabilityZones, version:currentOrchestratorVersion, maxSurge:upgradeSettings.maxSurge}" -o table
az monitor activity-log list -g my-resource-group --offset 3d --status Failed --query "[].{time:eventTimestamp, operation:operationName.value, status:status.value, message:properties.statusMessage}" -o json
az aks operation show-latest -g my-resource-group -n my-clusteraz aks operation show-latest needs the aks-preview extension; if it is missing, the
activity log has the same error.
3. The Kubernetes side.
kubectl get nodes -L kubernetes.azure.com/agentpool,topology.kubernetes.io/zone,node.kubernetes.io/instance-type
kubectl events -A --types=Warning
kubectl get pods -n kube-system -o wide4. Follow the symptom to its reference, which has the commands for that area:
| Symptom | Read |
Cluster or node pool Failed, upgrade or scale errors, quota and VM capacity, autoscaler, node auto-repair, versions | references/nodepools-upgrades.md |
LoadBalancer Service <pending> or unreachable, SyncLoadBalancerFailed, static IPs, internal load balancers, health probes, SNAT | references/load-balancers.md |
Azure Disk or Azure Files PVC Pending or not mounting, zones, attach limits; workload identity and AADSTS errors | references/storage-identity.md |
CoreDNS customization, error dialing backend, Azure Policy, other managed add-ons; API server reachability, Entra ID login, Azure RBAC | references/addons-access.md |
Read the results
provisioningStateisFailedwhile nodes and pods run normally: the last Azure operation on the cluster failed; the cluster itself usually keeps working. The activity log'sstatusMessagenames the cause, often nested: an outer code such asResourceOperationFailure,UpgradeVMSSAgentPoolFailedorCreateOrUpdateExtensionFailedwraps the real one indetails.Update failed for this resource, as there is a conflicting operation in progressmeans two operations overlapped (for example an add-on update during a scale); nothing else needs fixing. Fix the real cause first, then reconcile the cluster (see Proposed fix). A reconcile does not fix an unresolved quota, policy, network or capacity error.(UpgradeFailed) Drain node ... failed when evicting pod ... Cannot evict pod as it would violate the pod's disruption budget: a PodDisruptionBudget allows 0 disruptions; see references/nodepools-upgrades.md.QuotaExceeded,AllocationFailed,ZonalAllocationFailed,SkuNotAvailable: Azure could not create the virtual machines, the subscription's quota or the region's capacity, not Kubernetes.- A LoadBalancer Service stays
<pending>withSyncLoadBalancerFailed: the event's message is the Azure error from the cloud provider.AuthorizationFailed ... does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '.../subnets/my-subnet' or the scope is invalidon an internal load balancer usually means that subnet does not exist (a typo inazure-load-balancer-internal-subnet): the cluster identity can read only the subnets it was granted, so a missing one looks like missing permission. Check the subnet exists before granting any role. - A PVC
PendingwithProvisioningFailed ... RESPONSE 400 ... InvalidParameter: Azure refused the disk itself, for exampleThe value 40960 of parameter 'disk.diskSizeGb' is out of range. The value must be between '1' and '32767', inclusive.(an Azure Disk is at most 32 TiB). The message afterERROR CODEis Azure's; the rest is the CSI driver. - A workload identity pod logs
AADSTS700016: Application with identifier '<client ID>' was not found in the directory: the client ID in the ServiceAccount annotationazure.workload.identity/client-idbelongs to no identity in that tenant.AADSTS70021: No matching federated identity record found for presented assertion: the identity exists, but no federated credential matches the token's issuer and subject (system:serviceaccount:my-namespace:my-app). See references/storage-identity.md. Error from server: error dialing backend: dial tcp <node ip>:10250: i/o timeoutonkubectl logsorexec, whilegetworks: the API server cannot reach the kubelet through the konnectivity tunnel; see references/addons-access.md.... User does not have access to the resource in Azure. Update role assignment to allow access.: the cluster uses Azure RBAC for Kubernetes, and the caller's Azure role assignments, not Kubernetes RoleBindings, decide.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it. Every change to an Azure resource is a proposed fix the user runs; make it in the cluster's source of truth (Bicep, Terraform, ARM templates, a pipeline) when there is one, since a later deployment would undo a change made by hand. Kubernetes objects such as Services, PVCs and ServiceAccounts belong in the workload's manifests or chart.
After the cause of a Failed state is fixed, a reconcile re-applies the cluster's current
configuration without changing it:
Proposed fix: review before running. The agent does not run this.
az aks update -g my-resource-group -n my-cluster --yes
# Undo: none needed; it re-applies the existing configurationFor a node pool in Failed:
Proposed fix: review before running. The agent does not run this.
az aks nodepool update -g my-resource-group --cluster-name my-cluster -n my-nodepool
# Undo: none needed; it re-applies the existing node pool configurationNever start another operation while one is still running (az aks operation show-latest
shows InProgress), and never edit the scale sets, load balancers or disks in the node
resource group by hand: AKS owns them, reverts such changes, and they are not supported.
The fixes for each area, with their undo, are in the references.
Sources
Checked against AKS with Kubernetes 1.33 (a private cluster with Azure CNI, Entra ID and Azure RBAC, workload identity and the Azure Policy add-on) and the Microsoft Learn pages below, as of October 2026:
- Troubleshoot AKS upgrade, scaling and failed-state errors, PodDrainFailure, QuotaExceeded and ZonalAllocationFailed and AllocationFailed.
- Supported Kubernetes versions, upgrade options, upgrade rules and version skew, node auto-repair and cluster autoscaler.
- Standard load balancer, internal load balancer, static public IP and SNAT.
- Storage concepts, Azure Disk mount failures and Azure Files mount failures.
- Workload identity.
- CoreDNS customization, tunnel connectivity, API server connection issues, kubelogin and Azure RBAC for Kubernetes.
- Messages quoted from the test cluster: the internal load balancer
AuthorizationFailedfor a missing subnet, the Azure DiskInvalidParametersize error,AADSTS700016from a workload identity pod, and aFailedcluster afterCreateOrUpdateExtensionFailedwithconflicting operation in progress.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

