agentleFS
k8s-servantk8s-servant/k8s-skills/skills/k8s-cluster-review/SKILL.md

k8s-cluster-review

Skill38 starsChanged 16 days agoApache-2.0View on GitHub

Use when nothing is broken but someone wants to know whether a Kubernetes cluster, or one namespace, is still in good shape: "review the cluster", "check it against best practices", "what drifted since we deployed it?", after taking over a cluster, or before a busy period. Reviews the live objects, not manifests in a repository, in three groups. Reliability: workloads with one replica, no readiness probe or no requests, PodDisruptionBudgets that allow 0 disruptions or match no pod, all replicas on one node or zone. Security: privileged or root containers, hostPath and hostNetwork, namespaces without Pod Security labels, cluster-admin bound to users or ServiceAccounts, namespaces without NetworkPolicies, :latest images. Hygiene and cost: PVCs no pod mounts, Released PVs, Services without endpoints, LoadBalancers, workloads at 0 replicas, failed Jobs, stale CronJobs. Reports each finding once per workload, with why it matters and when it is fine. Read-only: never changes anything itself.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 3 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run every step: the goal is a complete, short list, not the first finding.

    line 63
  2. Very specific:

    A PDB with allowed=0 while every pod is healthy: no pod it selects can be evicted, so a drain waits on it forever.

    line 139
  3. Distinctive:

    A namespace with workloads and no NetworkPolicy: every pod can reach every other pod.

    line 167
  4. Very specific:

    Add a readiness probe and requests (use the app's real health path, port and usage).

    line 227
  5. Very specific:

    Remove a cluster-admin binding that is not needed (save it first.

    line 270
  6. Distinctive:

    Delete leftovers once their owner confirms nobody needs them:

    line 281
  7. Very specific:

    Add a default-deny NetworkPolicy, and the allow rules the apps need, through their owners: see references/security.md.

    line 300

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Cluster review

Review a running cluster for what drifted in after it was deployed: settings that do not break anything today, but hurt during the next drain, load spike or incident, or cost money for nothing. Read only. Run the investigation commands freely. Never change, scale, label or delete anything yourself: every fix here is a proposed fix for the user or the owner of the object. Never read Secrets, and never exec into a pod; the review needs neither.

Manifest linters (kube-linter, Polaris, kube-score) check what is in a repository. This review checks what ended up in the cluster. If the user already has output from one of those tools, use it as extra evidence; never install them.

The commands use placeholders: my-namespace, my-app, my-node. Replace them with the real names. Cluster-wide checks need read access to nodes, PersistentVolumes and the RBAC objects, which the built-in view role leaves out. If a command is forbidden, do not treat it as "nothing found": put the check in the report's Skipped section with the command for the user to run.

When to use

  • A periodic review: monthly, before a busy period or a launch, after taking over a cluster or a namespace from another team.
  • Someone asks whether the cluster follows good practice, or what is risky or wasteful in it, and nothing is broken.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • Something is broken today (pods in CrashLoopBackOff or Pending, a node NotReady): the matching troubleshooting skill, or k8s-triage for a first look. A review is not the place to debug it.
  • Deprecated or removed APIs, version skew, anything about the next upgrade: k8s-upgrade-readiness. A PDB that allows 0 disruptions is reported here in one line, with a pointer there for what it does to a drain.
  • Certificates that expire or fail to renew: k8s-cert-manager-troubleshooting.
  • Who can do what in detail, and why a request is forbidden: k8s-rbac-troubleshooting.

Investigate

Run every step: the goal is a complete, short list, not the first finding.

0. Scope. Review the whole cluster unless the user names a namespace; then add -n my-namespace instead of -A to every namespaced command, and skip the cluster-wide checks that do not concern it (nodes are still read for the spread check). Always skip kube-system, kube-public and kube-node-lease. If other namespaces look like add-ons installed with the cluster (for example local-path-storage, cert-manager, monitoring), name them, say you treat them as system, and let the user say otherwise.

kubectl get namespaces --show-labels
kubectl get nodes -L topology.kubernetes.io/zone
kubectl get daemonsets -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" "}{.spec.template.spec.containers[*].image}{"\n"}{end}'

1. Reliability: replicas, probes, requests, PodDisruptionBudgets and spread.

kubectl get deployments,statefulsets -A -o custom-columns='KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,REPLICAS:.spec.replicas'
kubectl get deployments,statefulsets,daemonsets -A -o jsonpath='{range .items[*]}{.kind}{" "}{.metadata.namespace}/{.metadata.name}{"\n"}{range .spec.template.spec.containers[*]}{"  "}{.name}{" readiness="}{.readinessProbe}{" requests="}{.resources.requests}{"\n"}{end}{end}'
kubectl get pdb -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" allowed="}{.status.disruptionsAllowed}{" healthy="}{.status.currentHealthy}/{.status.desiredHealthy}{" expected="}{.status.expectedPods}{" minAvailable="}{.spec.minAvailable}{" maxUnavailable="}{.spec.maxUnavailable}{"\n"}{end}'
kubectl get pods -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,OWNER_NAME:.metadata.ownerReferences[0].name,NODE:.spec.nodeName,PHASE:.status.phase'

2. Security: what pods may do, Pod Security labels, cluster-admin, NetworkPolicies and image tags.

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" owner="}{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{" pod.runAsNonRoot="}{.spec.securityContext.runAsNonRoot}{" pod.runAsUser="}{.spec.securityContext.runAsUser}{" hostNetwork="}{.spec.hostNetwork}{" hostPID="}{.spec.hostPID}{" hostIPC="}{.spec.hostIPC}{" hostPath="}{.spec.volumes[*].hostPath.path}{"\n"}{range .spec.containers[*]}{"  "}{.name}{" image="}{.image}{" privileged="}{.securityContext.privileged}{" allowPrivilegeEscalation="}{.securityContext.allowPrivilegeEscalation}{" runAsNonRoot="}{.securityContext.runAsNonRoot}{" runAsUser="}{.securityContext.runAsUser}{"\n"}{end}{end}'
kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{" enforce="}{.metadata.labels.pod-security\.kubernetes\.io/enforce}{" warn="}{.metadata.labels.pod-security\.kubernetes\.io/warn}{"\n"}{end}'
kubectl get clusterrolebindings -o jsonpath='{range .items[*]}{.metadata.name}{" role="}{.roleRef.name}{" subjects="}{.subjects}{"\n"}{end}'
kubectl get rolebindings -A -o jsonpath='{range .items[?(@.roleRef.name=="cluster-admin")]}{.metadata.namespace}/{.metadata.name}{" subjects="}{.subjects}{"\n"}{end}'
kubectl get clusterroles -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.rules}{"\n"}{end}'
kubectl get networkpolicies -A

3. Hygiene and cost: volumes, Services, idle workloads, Jobs and CronJobs.

kubectl get pvc -A
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}{end}'
kubectl get pv -o custom-columns='NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.namespace,CLAIM_NAME:.spec.claimRef.name,RECLAIM:.spec.persistentVolumeReclaimPolicy,SIZE:.spec.capacity.storage'
kubectl get services -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,SELECTOR:.spec.selector'
kubectl get endpointslices -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.labels.kubernetes\.io/service-name}{" ready="}{.endpoints[?(@.conditions.ready==true)].addresses}{"\n"}{end}'
kubectl get jobs -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{" owner="}{.metadata.ownerReferences[0].kind}{" failed="}{.status.failed}{" conditions="}{.status.conditions[*].type}{"\n"}{end}'
kubectl get cronjobs -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,SCHEDULE:.spec.schedule,SUSPEND:.spec.suspend,LAST_SCHEDULE:.status.lastScheduleTime,LAST_SUCCESS:.status.lastSuccessfulTime'

4. Open the matching reference for the field details and edge cases of a check:

GroupReference
Replicas, probes, requests, PDBs, spread over nodes and zonesreferences/reliability.md
securityContext, host access, Pod Security labels, RBAC, NetworkPolicies, image tagsreferences/security.md
PVCs, PVs, Services, LoadBalancers, idle workloads, Jobs, CronJobsreferences/hygiene.md

Read the results

Each check, what it means, and when it is fine (Kubernetes 1.37). Report a finding once per workload (Deployment, StatefulSet, DaemonSet, CronJob), not once per pod: group pods by their owner, and for a ReplicaSet name the Deployment it belongs to. A pod with no owner (a debug pod someone started by hand, for example) is reported by its own name, as a bare pod: nothing recreates it, which is a finding of its own.

Reliability

  • One replica (Deployment or StatefulSet, replicas: 1): the app is down while its pod is replaced by a drain, an upgrade or a crash. Fine for batch tools, dev namespaces and singletons by design (a controller with leader election, a database run as one instance on purpose). DaemonSets and Jobs are not counted.
  • No readiness probe: the Service sends traffic to a pod as soon as it starts, and keeps sending it while the app cannot answer. A missing liveness probe is not a finding. Fine for workers that take no traffic through a Service.
  • No CPU or memory requests: the scheduler places the pod as if it needs nothing, and it is the first to be evicted under node pressure (QoS BestEffort). Fine for rare one-off pods.
  • A PDB with allowed=0 while every pod is healthy: no pod it selects can be evicted, so a drain waits on it forever. Report it in one line and point to k8s-upgrade-readiness. A PDB with expected=0: it matches no pod; stale.
  • All replicas of a workload with 2 or more replicas on one node (or in one zone, when nodes carry topology.kubernetes.io/zone): one node or zone failure takes the whole app down. On a one-node cluster this cannot be checked: put it in Skipped, not in the findings or the counts.

Security

  • privileged=true, hostNetwork, hostPID, hostIPC or a hostPath volume: the container can reach the node and everything on it. Fine for node agents that need it (CNI, CSI, log and metrics collectors), usually DaemonSets in an add-on namespace.
  • allowPrivilegeEscalation not false, or running as root (runAsNonRoot not true at pod or container level and no non-zero runAsUser): a process that breaks out of the app gets more rights than it needs. Name it in the evidence even when the same container is privileged. The image may still set a non-root user; say that the field does not show it.
  • A namespace without pod-security.kubernetes.io/enforce: nothing stops a privileged pod from being created there. Fine for system namespaces; a label warn or audit only reports.
  • cluster-admin, or a ClusterRole with * verbs on * resources, bound to a user or a group whose name does not start with system:, or to a ServiceAccount outside the system and add-on namespaces: that subject can do anything, including read every Secret. Installer defaults (kubeadm's kubeadm:cluster-admins group, a managed cloud's admin group, a ServiceAccount of kube-system or of an add-on namespace) are fine if intended: name them as such, not as a problem.
  • A namespace with workloads and no NetworkPolicy: every pod can reach every other pod. A finding only when the CNI enforces NetworkPolicy. Decide from the CNI's DaemonSet (step 0; an operator-installed Calico runs in calico-system, Flannel in kube-flannel): Cilium, Calico and Antrea enforce, and so does kindnet on current kind releases (seen on the 1.37 node image; on an old kind cluster, treat it as unknown); Flannel alone does not. On managed clusters it depends on an option (EKS aws-node only with its network policy agent, AKS only with a policy engine): unless you see that option, treat it as unknown. If you cannot tell, report "no NetworkPolicy; matters only if the CNI enforces them" in Skipped, not as a finding.
  • An image with :latest or no tag (a @sha256: digest is fine): which version runs depends on when the node pulled it, and a rollback cannot go back to it.

Hygiene and cost

  • A PVC that no pod mounts: its volume is paid for and holds data nobody reads. Fine if a Job or a CronJob mounts it from time to time; check their templates.
  • A PV in phase Released: its claim was deleted and Retain kept the volume. It is never bound again by itself.
  • A Service (not headless, not ExternalName) with no ready endpoints: its selector matches no ready pod. Traffic to it fails. Fine if its app is scaled to 0 on purpose.
  • LoadBalancer Services: list them for a second look (each one is a cloud load balancer and usually a public address); do not call them wrong.
  • A Deployment or StatefulSet at 0 replicas, a failed Job (condition Failed, not owned by a CronJob that has since succeeded), a CronJob that is suspended or whose LAST_SUCCESS is older than several schedule periods: leftovers, or a task that stopped working without anyone noticing.

Report the result in this shape, findings ordered by impact within each group. Stay quiet about what is fine: fewer findings with reasons beat a long list.

Summary: Reliability 2, Security 3, Hygiene and cost 1 (scope: whole cluster)

Security
- Deployment my-namespace/my-app: privileged container and a hostPath volume (/var/log).
  Evidence: privileged=true, hostPath=/var/log. Why it matters: ... Fine when: ...
  Proposed fix: ... (below)

Skipped
- kube-system, kube-public, kube-node-lease; local-path-storage (treated as an add-on).
- ClusterRoleBindings: forbidden for this user (run: kubectl get clusterrolebindings).

Proposed fix

Show each fix to the user; the agent runs none of them. Every finding belongs to the owner of the object, and a fix made in the cluster is undone by the next deploy unless the source (the manifest, the Helm values, the Git repository) changes too: prefer the change in the source. Write each undo with the values you found.

Give a single-replica app a second pod:

Proposed fix: review before running. The agent does not run this.

kubectl scale deployment my-app -n my-namespace --replicas=2
# Undo: kubectl scale deployment my-app -n my-namespace --replicas=1

Add a readiness probe and requests (use the app's real health path, port and usage). A JSON patch add replaces the whole field: if the container already has some requests, put them in the value too:

Proposed fix: review before running. The agent does not run this.

kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/readinessProbe", "value": {"httpGet": {"path": "/healthz", "port": 8080}}}, {"op": "add", "path": "/spec/template/spec/containers/0/resources/requests", "value": {"cpu": "100m", "memory": "128Mi"}}]'
# Undo: kubectl rollout undo deployment my-app -n my-namespace

Let one pod of a PDB be evicted:

Proposed fix: review before running. The agent does not run this.

kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/minAvailable"}, {"op": "add", "path": "/spec/maxUnavailable", "value": 1}]'
# Undo: kubectl patch pdb my-app -n my-namespace --type=json -p '[{"op": "remove", "path": "/spec/maxUnavailable"}, {"op": "add", "path": "/spec/minAvailable", "value": 1}]'

Drop privileges from a container (only once its owner confirms the app runs without them; a container that needs root fails to start). The patch replaces the container's whole securityContext: copy the fields it already has into the value. A hostPath volume found with it is removed in the source, with its volume mount:

Proposed fix: review before running. The agent does not run this.

kubectl patch deployment my-app -n my-namespace --type=json -p '[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": false, "allowPrivilegeEscalation": false, "runAsNonRoot": true, "capabilities": {"drop": ["ALL"]}}}]'
# Undo: kubectl rollout undo deployment my-app -n my-namespace

Enforce a Pod Security level on a namespace. Check first what it would refuse: the server-side dry run below only prints warnings and changes nothing.

Proposed fix: review before running. The agent does not run this.

kubectl label --dry-run=server --overwrite namespace my-namespace pod-security.kubernetes.io/enforce=baseline
kubectl label namespace my-namespace pod-security.kubernetes.io/enforce=baseline
# Undo: kubectl label namespace my-namespace pod-security.kubernetes.io/enforce-

Remove a cluster-admin binding that is not needed (save it first; give the subject a narrower role before, if it still needs some access):

Proposed fix: review before running. The agent does not run this.

kubectl get clusterrolebinding my-app-admin -o yaml > my-app-admin.yaml
kubectl delete clusterrolebinding my-app-admin
# Undo: kubectl create -f my-app-admin.yaml

Delete leftovers once their owner confirms nobody needs them:

Proposed fix: review before running. The agent does not run this.

kubectl delete job my-job -n my-namespace
kubectl get service my-app -n my-namespace -o yaml > my-app-service.yaml
kubectl delete service my-app -n my-namespace
# Undo: re-run the Job from its source; kubectl create -f my-app-service.yaml (it asks for the same ClusterIP again, and fails if another Service took it: remove spec.clusterIP and spec.clusterIPs from the file to get a new one)

Proposed fix: review before running. The agent does not run this.

kubectl delete pvc my-data -n my-namespace
# Undo: not reversible. With reclaim policy Delete the volume and its data are deleted;
# with Retain the PV stays Released and needs an admin to bind it again.

Add a default-deny NetworkPolicy, and the allow rules the apps need, through their owners: see references/security.md. A Released PV is deleted or bound again by an admin: see references/hygiene.md.

Sources

Checked against the Kubernetes 1.37 documentation, and on a Kubernetes 1.37 kind cluster:

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.