Kubernetes Resource Debugging

Decide what a container actually needs, using measurements, before changing a request or a limit.

Purpose

Use this when:

• A container is OOMKilled (exit 137).
• A pod is Pending with Insufficient cpu or Insufficient memory.
• A workload is slow and CPU throttling is suspected.
• Nodes report MemoryPressure or DiskPressure, or pods are being Evicted.
• Someone is about to change requests or limits and has no measurement to justify it.

Do not use this when:

• The container is crashing with a non-137 exit code — that is an application failure; use kubernetes-troubleshooting.
• The slowness is in a downstream dependency — use production-debugging.

What this skill assumes: metrics-server for kubectl top, and ideally Prometheus with kube-state-metrics and cAdvisor for historical data. Instantaneous top output is not enough to size a limit; peak usage is what kills containers.

Operating Procedure

1. Establish what the workload is configured to get. Read requests and limits from the spec, not from memory or from a wiki.
2. Measure what it actually uses. Current usage via kubectl top, and the peak over a meaningful window via Prometheus. One number at one moment is not a measurement.
3. Separate the two independent failures. Memory limits kill. CPU limits throttle. They have different symptoms, different metrics, and different fixes — never treat them together.
4. Check node-level pressure before concluding the workload is at fault. An evicted pod is usually a victim, not a cause.
5. Compute the change from the measurement. Peak plus headroom, not a doubling.
6. Apply to the manifest and roll out. One workload at a time.
7. Verify against the metric that showed the problem.

Record requests and limits as Observed from the spec; record usage as Observed with the window it was measured over. "The pod needs 2Gi" without a window is an Assumption.

Investigation

Step 1 - what is configured

kubectl get pod <pod> -n <namespace> -o jsonpath='{range .spec.containers[*]}{.name}{"\treq="}{.resources.requests}{"\tlim="}{.resources.limits}{"\n"}{end}'
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.qosClass}{"\n"}'

The QoS class is derived and determines eviction order: Guaranteed (requests equal limits on every container) is evicted last, BestEffort (nothing set) first, Burstable in between and in order of how far over its requests it is. A production workload with no requests set is BestEffort and will be the first thing killed when a node is under pressure — regardless of how important it is.

An empty limits on memory is not automatically wrong: it means the container can use whatever the node has, which is safer for a workload with spiky memory and more dangerous for a noisy neighbour.

Step 2 - what it uses now

kubectl top pod -n <namespace> --containers
kubectl top node

kubectl top is a point-in-time sample from metrics-server with roughly a 15-60 second resolution. Use it to catch an order-of-magnitude mismatch, never to size a memory limit — a container that OOMs during a 3-second startup spike will look comfortable in every sample you take.

If kubectl top errors with Metrics API not available, metrics-server is not installed or its --kubelet-insecure-tls configuration is wrong. Note that as Observed; it also means the HorizontalPodAutoscaler cannot work.

Step 3 - the peak, from Prometheus

max_over_time(container_memory_working_set_bytes{namespace="<ns>",container="<c>"}[7d])

rate(container_cpu_cfs_throttled_seconds_total{namespace="<ns>",container="<c>"}[5m])
  /
rate(container_cpu_cfs_periods_total{namespace="<ns>",container="<c>"}[5m])

The first query is what you size a memory limit against. Use container_memory_working_set_bytes — it is what the kernel's OOM killer acts on. Do not use container_memory_usage_bytes, which includes reclaimable page cache and will lead you to over-provision.

The second query is the throttling ratio, and it is the only honest answer to "is the CPU limit hurting us?". Interpretation:

A raw container_cpu_cfs_throttled_seconds_total counter without the ratio is meaningless: every busy container accumulates some throttling.

Step 4 - node pressure and capacity

kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'
kubectl describe node <node> | sed -n '/Allocated resources:/,$p'
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -20

Allocated resources shows requests as a percentage of allocatable, not usage. These are two different numbers and conflating them causes both of the classic capacity mistakes:

• Requests at 95% while usage is 20% — the cluster is full of reservations nobody uses. You are paying for nodes to hold air. The fix is to lower requests, not to add nodes.
• Requests at 40% while usage is 90% — the cluster is oversubscribed and one traffic spike away from cascading evictions. The fix is to raise requests so the scheduler stops overpacking.

Allocatable is always less than capacity: the kubelet reserves memory and CPU for the system and for the eviction threshold. Sizing against capacity produces pods that can never be scheduled.

Step 5 - confirm an OOM kill

kubectl describe pod <pod> -n <namespace> | grep -A5 'Last State'
kubectl get events -n <namespace> --field-selector reason=OOMKilling

Reason: OOMKilled with exit 137 in Last State is the container exceeding its own memory limit — a cgroup-level kill. That is different from a node-level OOM, which shows as an Evicted pod with a MemoryPressure node condition and means the node ran out. The first is fixed in the workload; the second is fixed in capacity or in whichever neighbour is misbehaving.

Decision Tree

Resource symptom
├── exit 137 + Reason: OOMKilled
│   ├── working set near limit at the kill → limit too low: raise to peak + 25-30%
│   ├── working set climbing steadily      → memory leak: raising the limit delays the crash
│   └── spike only at startup              → startup allocation; size for the spike, not the steady state
├── Pod Evicted + node MemoryPressure
│   ├── this pod is BestEffort/Burstable   → it is a victim; set requests
│   └── a neighbour exceeded its request   → find the neighbour, not this pod
├── Pending: Insufficient cpu/memory
│   ├── request > allocatable on every node → the request cannot ever be satisfied
│   ├── cluster genuinely full              → scale the node pool
│   └── fragmentation (sum fits, no single node) → smaller requests or a topology change
├── Slow, CPU suspected
│   ├── throttle ratio > 0.10 → the limit is the cause; raise or remove it
│   ├── ratio < 0.01          → not the CPU limit; the bottleneck is elsewhere
│   └── no limit set          → contention with neighbours → check node load
└── Cost or density concern
    ├── requests >> usage → lower requests (reclaims schedulable capacity)
    └── usage >> requests → raise requests (prevents eviction roulette)

Common Causes

Safety

Read-only

kubectl get, describe, top, and all PromQL queries. Reading metrics cannot change anything.

Potentially disruptive

• Changing requests or limits — every pod in the workload is recreated. This is a rolling restart with the same blast radius as a deploy. Do it through the manifest and the normal rollout, during a normal change window, one workload at a time. Rollback: kubectl rollout undo deployment/<name>.
• Raising requests significantly — may leave pods Pending if the cluster lacks capacity. Check Allocated resources on the nodes before applying, or you turn a slow service into a down one.
• kubectl drain <node> to rebalance — evicts everything on the node subject to PodDisruptionBudgets.
• Changing a LimitRange or ResourceQuota — affects every workload in the namespace, including ones you are not looking at. New pods can start failing admission immediately.

Destructive

This skill proposes no destructive operations.

Verification

Before: OOMKills 14/day; memory limit 512Mi; 7d peak working set 690Mi
After:  OOMKills 0 over 72h; memory limit 900Mi; 7d peak working set 700Mi

kubectl get pods -n <namespace> -l app=<label> \
  -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount
kubectl rollout status deployment/<name> -n <namespace>

The restart count is the primary signal: it must be flat, not merely low. A pod recreated by the rollout starts at 0, so a fresh 0 proves nothing — compare the same pod against itself over time. rollout status confirms the new resource values actually reached every replica rather than stalling partway.

max_over_time(container_memory_working_set_bytes{namespace="<ns>",container="<c>"}[1h])
  / on() group_left kube_pod_container_resource_limits{resource="memory",container="<c>"}

That last query is the headroom ratio, and it is the number to keep an eye on afterwards: below 0.8 is comfortable, above 0.95 means you have moved the cliff rather than removed it. Verify over at least one full traffic cycle — a limit raised on a Saturday and verified on a Sunday has not been tested against the Monday peak.

Anti-Patterns

Bad: It was OOMKilled, so double the memory limit. Better: Look at the working-set curve first. If it climbs steadily until the kill, it is a leak, and doubling the limit converts a 6-hour crash cycle into a 12-hour one while doubling the cost. If it is flat with a spike, size for the spike.

Bad: Set limits equal to requests everywhere for predictability. Better: That gives Guaranteed QoS and best eviction protection, which is right for memory-sensitive workloads — but a hard CPU limit equal to the request throttles any burst. Requests equal to limits for memory, request-only for CPU, is a defensible default for most latency-sensitive services.

Bad: Raise CPU limits because the service is slow. Better: Measure the throttle ratio. Below 0.01 there is no throttling and the limit is not the problem; you have spent a rollout and learned nothing.

Bad: Add nodes because Allocated resources shows 95%. Better: That is 95% of requests, not usage. Check actual usage. Adding nodes to a cluster full of inflated requests is expensive and fixes nothing.

Bad: Set requests from a load test on a laptop. Better: Measure the workload in production over at least a week, including its peak.

Composition

• kubernetes-troubleshooting — the pod is failing for a non-resource reason.
• kubernetes-production-readiness — setting requests and limits correctly for a new service.
• prometheus-troubleshooting — the metrics you need are missing or wrong.
• linux-performance — the node itself is saturated below Kubernetes.

References

• Kubernetes: Resource Management — requests, limits and QoS semantics.
• Kubernetes: Node-pressure Eviction — thresholds, signals and eviction ordering.