k8s-cluster-review
Skill38 starsChanged 16 days agoApache-2.0View on GitHub
Use when nothing is broken but someone wants to know whether a Kubernetes cluster, or one namespace, is still in good shape: "review the cluster", "check it against best practices", "what drifted since we deployed it?", after taking over a cluster, or before a busy period. Reviews the live objects, not manifests in a repository, in three groups. Reliability: workloads with one replica, no readiness probe or no requests, PodDisruptionBudgets that allow 0 disruptions or match no pod, all replicas on one node or zone. Security: privileged or root containers, hostPath and hostNetwork, namespaces without Pod Security labels, cluster-admin bound to users or ServiceAccounts, namespaces without NetworkPolicies, :latest images. Hygiene and cost: PVCs no pod mounts, Released PVs, Services without endpoints, LoadBalancers, workloads at 0 replicas, failed Jobs, stale CronJobs. Reports each finding once per workload, with why it matters and when it is fine. Read-only: never changes anything itself.
In this repository32 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- k8s-aks-troubleshootingSkill
- k8s-api-troubleshootingSkill
- k8s-argocd-troubleshootingSkill
- k8s-autoscaling-troubleshootingSkill
- k8s-cert-manager-troubleshootingSkill
- k8s-cilium-troubleshootingSkill
- k8s-cloudnative-pg-troubleshootingSkill
- k8s-coredns-troubleshootingSkill
- k8s-envoy-gateway-troubleshootingSkill
- k8s-envoy-troubleshootingSkill
- k8s-external-secrets-troubleshootingSkill
- k8s-fluent-bit-troubleshootingSkill
- k8s-flux-troubleshootingSkill
- k8s-gatekeeper-troubleshootingSkill
- k8s-gateway-api-troubleshootingSkill
- k8s-grafana-troubleshootingSkill
- k8s-helm-troubleshootingSkill
- k8s-istio-troubleshootingSkill
- k8s-kyverno-troubleshootingSkill
- k8s-loki-troubleshootingSkill
- k8s-network-troubleshootingSkill
- k8s-node-troubleshootingSkill
- k8s-opentelemetry-collector-troubleshootingSkill
- k8s-prometheus-troubleshootingSkill
- k8s-rbac-troubleshootingSkill
- k8s-storage-troubleshootingSkill
- k8s-triageSkill
- k8s-upgrade-readinessSkill
- k8s-velero-troubleshootingSkill
- k8s-workload-troubleshootingSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 3 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 63
Run every step: the goal is a complete, short list, not the first finding.
- Very specific: line 139
A PDB with
allowed=0while every pod is healthy: no pod it selects can be evicted, so a drain waits on it forever. - Distinctive: line 167
A namespace with workloads and no NetworkPolicy: every pod can reach every other pod.
- Very specific: line 227
Add a readiness probe and requests (use the app's real health path, port and usage).
- Very specific: line 270
Remove a cluster-admin binding that is not needed (save it first.
- Distinctive: line 281
Delete leftovers once their owner confirms nobody needs them:
- Very specific: line 300
Add a default-deny NetworkPolicy, and the allow rules the apps need, through their owners: see references/security.md.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Cluster review
Review a running cluster for what drifted in after it was deployed: settings that do
not break anything today, but hurt during the next drain, load spike or incident, or
cost money for nothing. Read only. Run the investigation commands freely. Never
change, scale, label or delete anything yourself: every fix here is a proposed fix for
the user or the owner of the object. Never read Secrets, and never exec into a pod;
the review needs neither.
Manifest linters (kube-linter, Polaris, kube-score) check what is in a repository. This review checks what ended up in the cluster. If the user already has output from one of those tools, use it as extra evidence; never install them.
The commands use placeholders: my-namespace, my-app, my-node. Replace them with
the real names. Cluster-wide checks need read access to nodes, PersistentVolumes and
the RBAC objects, which the built-in view role leaves out. If a command is forbidden,
do not treat it as "nothing found": put the check in the report's Skipped section with
the command for the user to run.
When to use
- A periodic review: monthly, before a busy period or a launch, after taking over a cluster or a namespace from another team.
- Someone asks whether the cluster follows good practice, or what is risky or wasteful in it, and nothing is broken.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- Something is broken today (pods in CrashLoopBackOff or Pending, a node NotReady):
the matching troubleshooting skill, or
k8s-triagefor a first look. A review is not the place to debug it. - Deprecated or removed APIs, version skew, anything about the next upgrade:
k8s-upgrade-readiness. A PDB that allows 0 disruptions is reported here in one line, with a pointer there for what it does to a drain. - Certificates that expire or fail to renew:
k8s-cert-manager-troubleshooting. - Who can do what in detail, and why a request is forbidden:
k8s-rbac-troubleshooting.
Investigate
Run every step: the goal is a complete, short list, not the first finding.
0. Scope. Review the whole cluster unless the user names a namespace; then add
-n my-namespace instead of -A to every namespaced command, and skip the cluster-wide
checks that do not concern it (nodes are still read for the spread check). Always skip
kube-system, kube-public and kube-node-lease. If other namespaces look like
add-ons installed with the cluster (for example local-path-storage, cert-manager,
monitoring), name them, say you treat them as system, and let the user say otherwise.
kubectl get namespaces --show-labels
kubectl get nodes -L topology.kubernetes.io/zone
kubectl get daemonsets -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" "}{.spec.template.spec.containers[*].image}{"\n"}{end}'1. Reliability: replicas, probes, requests, PodDisruptionBudgets and spread.
kubectl get deployments,statefulsets -A -o custom-columns='KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas'
kubectl get deployments,statefulsets,daemonsets -A -o jsonpath='{range .items[*]}{.kind}{" "}{.metadata.namespace}/{.metadata.name}{"\n"}{range .spec.template.spec.containers[*]}{" "}{.name}{" readiness="}{.readinessProbe}{" requests="}{.resources.requests}{"\n"}{end}{end}'
kubectl get pdb -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" allowed="}{.status.disruptionsAllowed}{" healthy="}{.status.currentHealthy}/{.status.desiredHealthy}{" expected="}{.status.expectedPods}{" minAvailable="}{.spec.minAvailable}{" maxUnavailable="}{.spec.maxUnavailable}{"\n"}{end}'
kubectl get pods -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,OWNER_NAME:.metadata.ownerReferences[0].name,NODE:.spec.nodeName,PHASE:.status.phase'2. Security: what pods may do, Pod Security labels, cluster-admin, NetworkPolicies and image tags.
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" owner="}{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{" pod.runAsNonRoot="}{.spec.securityContext.runAsNonRoot}{" pod.runAsUser="}{.spec.securityContext.runAsUser}{" hostNetwork="}{.spec.hostNetwork}{" hostPID="}{.spec.hostPID}{" hostIPC="}{.spec.hostIPC}{" hostPath="}{.spec.volumes[*].hostPath.path}{"\n"}{range .spec.containers[*]}{" "}{.name}{" image="}{.image}{" privileged="}{.securityContext.privileged}{" allowPrivilegeEscalation="}{.securityContext.allowPrivilegeEscalation}{" runAsNonRoot="}{.securityContext.runAsNonRoot}{" runAsUser="}{.securityContext.runAsUser}{"\n"}{end}{end}'
kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{" enforce="}{.metadata.labels.pod-security\.kubernetes\.io/enforce}{" warn="}{.metadata.labels.pod-security\.kubernetes\.io/warn}{"\n"}{end}'
kubectl get clusterrolebindings -o jsonpath='{range .items[*]}{.metadata.name}{" role="}{.roleRef.name}{" subjects="}{.subjects}{"\n"}{end}'
kubectl get rolebindings -A -o jsonpath='{range .items[?(@.roleRef.name=="cluster-admin")]}{.metadata.namespace}/{.metadata.name}{" subjects="}{.subjects}{"\n"}{end}'
kubectl get clusterroles -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.rules}{"\n"}{end}'
kubectl get networkpolicies -A3. Hygiene and cost: volumes, Services, idle workloads, Jobs and CronJobs.
kubectl get pvc -A
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}{end}'
kubectl get pv -o custom-columns='NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.namespace,CLAIM_NAME:.spec.claimRef.name,RECLAIM:.spec.persistentVolumeReclaimPolicy,SIZE:.spec.capacity.storage'
kubectl get services -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,SELECTOR:.spec.selector'
kubectl get endpointslices -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.labels.kubernetes\.io/service-name}{" ready="}{.endpoints[?(@.conditions.ready==true)].addresses}{"\n"}{end}'
kubectl get jobs -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" owner="}{.metadata.ownerReferences[0].kind}{" failed="}{.status.failed}{" conditions="}{.status.conditions[*].type}{"\n"}{end}'
kubectl get cronjobs -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST_SCHEDULE:.status.lastScheduleTime,LAST_SUCCESS:.status.lastSuccessfulTime'4. Open the matching reference for the field details and edge cases of a check:
| Group | Reference |
| Replicas, probes, requests, PDBs, spread over nodes and zones | references/reliability.md |
| securityContext, host access, Pod Security labels, RBAC, NetworkPolicies, image tags | references/security.md |
| PVCs, PVs, Services, LoadBalancers, idle workloads, Jobs, CronJobs | references/hygiene.md |
Read the results
Each check, what it means, and when it is fine (Kubernetes 1.37). Report a finding once per workload (Deployment, StatefulSet, DaemonSet, CronJob), not once per pod: group pods by their owner, and for a ReplicaSet name the Deployment it belongs to. A pod with no owner (a debug pod someone started by hand, for example) is reported by its own name, as a bare pod: nothing recreates it, which is a finding of its own.
Reliability
- One replica (Deployment or StatefulSet,
replicas: 1): the app is down while its pod is replaced by a drain, an upgrade or a crash. Fine for batch tools, dev namespaces and singletons by design (a controller with leader election, a database run as one instance on purpose). DaemonSets and Jobs are not counted. - No readiness probe: the Service sends traffic to a pod as soon as it starts, and keeps sending it while the app cannot answer. A missing liveness probe is not a finding. Fine for workers that take no traffic through a Service.
- No CPU or memory requests: the scheduler places the pod as if it needs nothing, and it is the first to be evicted under node pressure (QoS BestEffort). Fine for rare one-off pods.
- A PDB with
allowed=0while every pod is healthy: no pod it selects can be evicted, so a drain waits on it forever. Report it in one line and point tok8s-upgrade-readiness. A PDB withexpected=0: it matches no pod; stale. - All replicas of a workload with 2 or more replicas on one node (or in one zone,
when nodes carry
topology.kubernetes.io/zone): one node or zone failure takes the whole app down. On a one-node cluster this cannot be checked: put it in Skipped, not in the findings or the counts.
Security
privileged=true,hostNetwork,hostPID,hostIPCor ahostPathvolume: the container can reach the node and everything on it. Fine for node agents that need it (CNI, CSI, log and metrics collectors), usually DaemonSets in an add-on namespace.allowPrivilegeEscalationnotfalse, or running as root (runAsNonRootnottrueat pod or container level and no non-zerorunAsUser): a process that breaks out of the app gets more rights than it needs. Name it in the evidence even when the same container is privileged. The image may still set a non-root user; say that the field does not show it.- A namespace without
pod-security.kubernetes.io/enforce: nothing stops a privileged pod from being created there. Fine for system namespaces; a labelwarnorauditonly reports. cluster-admin, or a ClusterRole with*verbs on*resources, bound to a user or a group whose name does not start withsystem:, or to a ServiceAccount outside the system and add-on namespaces: that subject can do anything, including read every Secret. Installer defaults (kubeadm'skubeadm:cluster-adminsgroup, a managed cloud's admin group, a ServiceAccount ofkube-systemor of an add-on namespace) are fine if intended: name them as such, not as a problem.- A namespace with workloads and no NetworkPolicy: every pod can reach every other
pod. A finding only when the CNI enforces NetworkPolicy. Decide from the CNI's
DaemonSet (step 0; an operator-installed Calico runs in
calico-system, Flannel inkube-flannel): Cilium, Calico and Antrea enforce, and so does kindnet on current kind releases (seen on the 1.37 node image; on an old kind cluster, treat it as unknown); Flannel alone does not. On managed clusters it depends on an option (EKSaws-nodeonly with its network policy agent, AKS only with a policy engine): unless you see that option, treat it as unknown. If you cannot tell, report "no NetworkPolicy; matters only if the CNI enforces them" in Skipped, not as a finding. - An image with
:latestor no tag (a@sha256:digest is fine): which version runs depends on when the node pulled it, and a rollback cannot go back to it.
Hygiene and cost
- A PVC that no pod mounts: its volume is paid for and holds data nobody reads. Fine if a Job or a CronJob mounts it from time to time; check their templates.
- A PV in phase
Released: its claim was deleted andRetainkept the volume. It is never bound again by itself. - A Service (not headless, not
ExternalName) with no ready endpoints: its selector matches no ready pod. Traffic to it fails. Fine if its app is scaled to 0 on purpose. - LoadBalancer Services: list them for a second look (each one is a cloud load balancer and usually a public address); do not call them wrong.
- A Deployment or StatefulSet at 0 replicas, a failed Job (condition
Failed, not owned by a CronJob that has since succeeded), a CronJob that is suspended or whoseLAST_SUCCESSis older than several schedule periods: leftovers, or a task that stopped working without anyone noticing.
Report the result in this shape, findings ordered by impact within each group. Stay quiet about what is fine: fewer findings with reasons beat a long list.
Summary: Reliability 2, Security 3, Hygiene and cost 1 (scope: whole cluster)
Security
- Deployment my-namespace/my-app: privileged container and a hostPath volume (/var/log).
Evidence: privileged=true, hostPath=/var/log. Why it matters: ... Fine when: ...
Proposed fix: ... (below)
Skipped
- kube-system, kube-public, kube-node-lease; local-path-storage (treated as an add-on).
- ClusterRoleBindings: forbidden for this user (run: kubectl get clusterrolebindings).Proposed fix
Show each fix to the user; the agent runs none of them. Every finding belongs to the owner of the object, and a fix made in the cluster is undone by the next deploy unless the source (the manifest, the Helm values, the Git repository) changes too: prefer the change in the source. Write each undo with the values you found.
Give a single-replica app a second pod:
Proposed fix: review before running. The agent does not run this.
kubectl scale deployment my-app -n my-namespace --replicas=2
# Undo: kubectl scale deployment my-app -n my-namespace --replicas=1Add a readiness probe and requests (use the app's real health path, port and usage).
A JSON patch add replaces the whole field: if the container already has some requests,
put them in the value too:
Proposed fix: review before running. The agent does not run this.
kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/readinessProbe", "value": {"httpGet": {"path": "/healthz", "port": 8080}}}, {"op": "add", "path": "/spec/template/spec/containers/0/resources/requests", "value": {"cpu": "100m", "memory": "128Mi"}}]'
# Undo: kubectl rollout undo deployment my-app -n my-namespaceLet one pod of a PDB be evicted:
Proposed fix: review before running. The agent does not run this.
kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/minAvailable"}, {"op": "add", "path": "/spec/maxUnavailable", "value": 1}]'
# Undo: kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/maxUnavailable"}, {"op": "add", "path": "/spec/minAvailable", "value": 1}]'Drop privileges from a container (only once its owner confirms the app runs without
them; a container that needs root fails to start). The patch replaces the container's
whole securityContext: copy the fields it already has into the value. A hostPath
volume found with it is removed in the source, with its volume mount:
Proposed fix: review before running. The agent does not run this.
kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": false, "allowPrivilegeEscalation": false, "runAsNonRoot": true, "capabilities": {"drop": ["ALL"]}}}]'
# Undo: kubectl rollout undo deployment my-app -n my-namespaceEnforce a Pod Security level on a namespace. Check first what it would refuse: the server-side dry run below only prints warnings and changes nothing.
Proposed fix: review before running. The agent does not run this.
kubectl label --dry-run=server --overwrite namespace my-namespace pod-security.kubernetes.io/enforce=baseline
kubectl label namespace my-namespace pod-security.kubernetes.io/enforce=baseline
# Undo: kubectl label namespace my-namespace pod-security.kubernetes.io/enforce-Remove a cluster-admin binding that is not needed (save it first; give the subject a
narrower role before, if it still needs some access):
Proposed fix: review before running. The agent does not run this.
kubectl get clusterrolebinding my-app-admin -o yaml > my-app-admin.yaml
kubectl delete clusterrolebinding my-app-admin
# Undo: kubectl create -f my-app-admin.yamlDelete leftovers once their owner confirms nobody needs them:
Proposed fix: review before running. The agent does not run this.
kubectl delete job my-job -n my-namespace
kubectl get service my-app -n my-namespace -o yaml > my-app-service.yaml
kubectl delete service my-app -n my-namespace
# Undo: re-run the Job from its source; kubectl create -f my-app-service.yaml (it asks for the same ClusterIP again, and fails if another Service took it: remove spec.clusterIP and spec.clusterIPs from the file to get a new one)Proposed fix: review before running. The agent does not run this.
kubectl delete pvc my-data -n my-namespace
# Undo: not reversible. With reclaim policy Delete the volume and its data are deleted;
# with Retain the PV stays Released and needs an admin to bind it again.Add a default-deny NetworkPolicy, and the allow rules the apps need, through their
owners: see references/security.md. A Released PV is
deleted or bound again by an admin: see references/hygiene.md.
Sources
Checked against the Kubernetes 1.37 documentation, and on a Kubernetes 1.37 kind cluster:
- Configuration Good Practices.
- Configure Liveness, Readiness and Startup Probes.
- Resource Management for Pods and Containers and Pod Quality of Service Classes.
- Specifying a Disruption Budget.
- Pod Topology Spread Constraints.
- Pod Security Standards and Pod Security Admission.
- Configure a Security Context.
- Role Based Access Control Good Practices.
- Network Policies.
- Images: tags, digests and
imagePullPolicy. - Persistent Volumes:
reclaim policies and the
Releasedphase. - Jobs and CronJob.
- That kindnet enforces NetworkPolicy was seen on the kind cluster: a default-deny policy stopped traffic that reached the pod before it.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

