Kubernetes Production Readiness

Review a workload against the failures that actually take services down: node loss, rollouts, restarts, and saturation.

Purpose

Use this when:

• Reviewing a Deployment, StatefulSet or Helm chart before it takes production traffic.
• A service exists in production and nobody has ever checked it against a baseline.
• A post-incident action item is "make this service more reliable" and you need a concrete list.

Do not use this when:

• The workload is currently broken — fix it first with kubernetes-troubleshooting. Readiness review is for healthy services.
• The question is about cluster or platform readiness (control plane HA, upgrade strategy, backup of etcd) rather than about a workload. That is a different review.

What this skill assumes: access to the manifests (Git, Helm values, or the live objects), and a stated environment. A review of kubectl get -o yaml output misses defaults applied by admission controllers, so prefer reviewing both the source manifest and the live object.

Operating Procedure

1. Get the real manifest. Render Helm (helm template) or fetch the live object. Review what is deployed, not what someone believes is deployed.
2. Classify the workload. Stateless request-serving, stateful, batch, and singleton workloads have different correct answers. Applying stateless advice to a StatefulSet causes data loss.
3. Walk the checklist below in order. Record each item as pass, fail, or not-applicable with a reason — "not applicable" without a reason is a fail.
4. Rank findings by blast radius, not by how easy they are to fix. A missing PodDisruptionBudget on a two-replica service is a real outage during the next node upgrade; a missing label is not.
5. Propose changes as manifest diffs, with the rollout risk of each stated.
6. Verify after the change by exercising the failure the change was meant to survive.

Everything not visible in the manifest — the actual startup time, the real memory peak, whether SIGTERM is handled — is an Assumption until measured. Say so.

Investigation

Step 1 - render what is actually deployed

helm template <release> <chart> -f <values.yaml> > /tmp/rendered.yaml
kubectl get deployment <name> -n <namespace> -o yaml > /tmp/live.yaml
diff <(yq -P 'sort_keys(..)' /tmp/rendered.yaml) <(yq -P 'sort_keys(..)' /tmp/live.yaml) | head -50

Differences between rendered and live are the most interesting output of this entire review: they are either a mutating admission webhook (a sidecar injector, a policy engine) or drift from a manual change that the next deploy will silently erase.

Step 2 - the checklist

Availability

A PDB with minAvailable equal to replicas is worse than none: it makes the node undrainable and blocks cluster upgrades forever. That is a common and expensive mistake.

Health

Resources

Configuration and supply chain

Observability

Step 3 - test the failure modes, not the happy path

kubectl get pdb -n <namespace>
kubectl drain <node> --dry-run=server --ignore-daemonsets --delete-emptydir-data

--dry-run=server asks the API server whether the eviction would be allowed without performing it. If it reports that a PDB blocks eviction, you have found a workload that will stall your next cluster upgrade — before the upgrade, rather than during it.

kubectl get deployment <name> -n <namespace> \
  -o jsonpath='{.spec.strategy}{"\n"}{.spec.template.spec.terminationGracePeriodSeconds}{"\n"}'

Compare the grace period against your p99 request duration plus any shutdown work. A 30-second default with a 45-second batch handler means every rollout kills work in flight.

Decision Tree

Is this workload safe for production traffic?
├── Stateless request-serving
│   ├── replicas < 2 → not ready; every restart is downtime
│   ├── no readinessProbe → not ready; rollouts will drop traffic
│   ├── no PDB → not ready for a cluster upgrade
│   └── no requests set → BestEffort; will be evicted first under pressure
├── Stateful (databases, queues, anything with a PVC)
│   ├── Does a restore actually work? → untested backup is not a backup
│   ├── PDB minAvailable respects quorum → otherwise a drain can break quorum
│   └── OnDelete vs RollingUpdate chosen deliberately → surprise rollouts corrupt state
├── Batch / Job
│   ├── backoffLimit and activeDeadlineSeconds set → otherwise it retries forever
│   └── Idempotent? → retries will re-run it
└── Singleton (leader-elected, cron, migration)
    ├── Leader election implemented → otherwise two replicas both act
    └── PDB and grace period aligned with the work unit

Common Causes

Safety

Read-only

Rendering manifests, kubectl get, describe, and --dry-run=server on any command. Server-side dry run is genuinely read-only: it runs admission and validation and discards the result.

Potentially disruptive

• Applying probe, resource or strategy changes — triggers a rolling restart of the whole workload. Roll out during a change window; one workload at a time. Rollback: kubectl rollout undo.
• Adding a PodDisruptionBudget — takes effect immediately and can block an in-progress node drain or cluster autoscaler scale-down. Check minAvailable against the current replica count before applying, not after.
• Tightening a securityContext (readOnlyRootFilesystem, runAsNonRoot) — will crash-loop any application that writes to the filesystem or needs root. Test in a non-production environment first; this is the most common cause of a "hardening" change becoming an outage.
• kubectl drain for a real spread test — evicts live workloads. Use --dry-run=server instead unless you are in a maintenance window.

Destructive

This skill proposes no destructive operations. Note that deleting and recreating a StatefulSet or its volume claims — sometimes suggested to "apply changes cleanly" — destroys data and is out of scope here.

Verification

Verify by exercising the failure the change was meant to survive, not by re-reading the manifest.

Before: rollout produced 340 x 502 over 90s; drain of node-3 evicted all 2 replicas at once
After:  rollout produced 0 x 502; drain evicts 1 replica at a time and waits for Ready

kubectl rollout restart deployment/<name> -n <namespace>
kubectl rollout status deployment/<name> -n <namespace> --timeout=10m
kubectl drain <node> --dry-run=server --ignore-daemonsets

Run a client-side request loop through the ingress during the rollout restart. A rollout that kubectl reports as successful can still have dropped every in-flight request; only the client sees that.

Anti-Patterns

Bad: Add a liveness probe pointing at the same /healthz as readiness. Better: Liveness answers "would a restart fix this?" and must only test the process itself. Readiness answers "should traffic come here now?". Pointing liveness at a dependency check means a slow database restarts every replica in the fleet at once — converting a degradation into an outage.

Bad: Three replicas, so it is highly available. Better: Check where they are. Three replicas on one node survive nothing. Availability is a property of the spread, not of the count.

Bad: Set minAvailable: 100% to protect the service. Better: That makes the workload permanently undrainable and blocks every cluster upgrade. Use minAvailable: N-1, or a percentage below 100.

Bad: Ship with :latest and a note to pin it later. Better: Pin at the start. An unpinned tag means that during the next incident you cannot answer "what changed?" or roll back to a known state.

Bad: Treat this checklist as a gate to pass once. Better: Re-run it when replica counts, dependencies or traffic change. A configuration that was right at 2 replicas and 100 rps is often wrong at 20 replicas and 10k rps.

Composition

• kubernetes-resource-debugging — to derive requests and limits from measurement.
• kubernetes-networking — to validate Service, Ingress and policy configuration.
• container-image-review — image size, base image, non-root user, pinning.
• slo-design — to define what "healthy" means for this service in the first place.
• secrets-management-review — for how credentials reach the container.

References

• references/readiness-checklist.md — the checklist in copy-pasteable form for a PR review.
• Kubernetes: Configure Liveness, Readiness and Startup Probes
• Kubernetes: Pod Disruption Budgets