K8s rightsizing

A recommendation is the peak of real usage over a representative window plus headroom: never the average, never a number read off someone else's dashboard. A requests cut is only worth an MR if it removes nodes: the bill is nodes, not requests.

1. Find the series

Federated or global stores usually drop the raw container_* and kube_pod_container_* series and keep recording rules instead (pod-level CPU rate, memory usage, OOM counters). If the project CLAUDE.md names them, use those first. Otherwise list what exists for the target cluster before any real query: an empty result means wrong series or datasource, not zero usage.

gcx metrics query -d <uid> 'count by (__name__) ({__name__=~".*(cpu_usage|memory_usage|working_set|oom).*", <cluster_label>="<cluster>"})' -o json --jq '[.data.result[].metric.__name__]'

Check the oldest point before claiming a 30-day window: retention differs per store.

The window starts when the workload took its real traffic. After a migration, a cutover or a ramp-up, find that date first (traffic series of the app, or the old platform going to zero) and measure from it: idle days before it dilute every quantile.

2. Measure

Always filter on cluster and namespace (and container on raw series): an unfiltered [7d] range over a multi-cluster store can OOM the queriers. Window [7d] by default, [14d] when the load is weekly-cyclic. Raw series shown; swap in the recording rule names where they exist.

Live spec for comparison: kubectl klens reqlim --context <ctx> -n <ns> --by-owner.

3. Recommend

Also check the replicas and the spread rules: a required podAntiAffinity on kubernetes.io/hostname makes the node count >= replicas, whatever the requests. It is often the biggest lever and no requests cut can beat it.

4. Node impact: the gate

Nodes come from requests, through the placement constraints. Before proposing an MR, simulate the node count with the constraints that decide where each pod may land:

Run the simulator on a live snapshot. It groups nodes by class, capacity type, arch and taints, packs each group into its most common machine type, and prices with the GCE catalog (list price, no CUD: a rough order of magnitude):

S=<scratchpad>
kubectl --context <ctx> get nodes -o json > "$S/nodes.json"
kubectl --context <ctx> get pods -A -o json > "$S/pods.json"
SHAPES=c4d-highmem-2,c4d-highmem-8,c4d-standard-8   # candidates: same arch as the group, allowed by its compute class
python3 scripts/fetch_prices.py "$S/nodes.json" --shapes "$SHAPES" > "$S/prices.json"   # GKE only, see its docstring for the format
python3 scripts/node_impact.py --nodes "$S/nodes.json" --pods "$S/pods.json" \
  --changes "$S/changes.json" --prices "$S/prices.json" --fill 0.75 --max-share 0.25 --zones auto --shapes "$SHAPES"

Machine types. --shapes repacks each group after all changes into other machine types of the same arch. Pick candidates from the workload profile: memory-bound pods want a high GiB/vCPU ratio, a pod bigger than half a node wastes the rest. A cheaper shape only counts if the compute class (or node pool) can provision it: changing its priorities is a separate, platform-wide change. Moving to arm64 needs every image of the group to be multi-arch.

Blast radius. --max-share (default 0.25) caps the share of one workload's replicas on a single node, at least 1 pod, and a required hostname anti-affinity means 1. Never raise it to make bigger nodes win: a few big nodes holding every replica of an app is a single point of failure, whatever it saves. Read three columns: the worst share of one workload on a node, in a zone, and the share of the group one node loss takes. Calibration ignores the cap; a BLAST RADIUS line says how many nodes today's state lacks to respect it, which the after includes.

Zones. --zones auto (default) spreads each workload evenly over the zones of today's nodes, as the default zone spread does, keeps a pod with a PVC in its zone (zonal PV), and packs each zone apart: small workloads then cost one node per zone. Zone spread only matters on a multi-zone cluster: a single-zone cluster is a design choice (no inter-zone egress), so its 100% zone share is expected, not a finding, and adding zones is not a rightsizing proposal. --zones a,b,c simulates other node locations only when the user asks for it.

changes.json is keyed <namespace>/<owner> (Deployment name, StatefulSet name): {"ns/app": {"cpu": "250m", "memory": "2Gi", "replicas": 10, "spread": "soft"}}.

• Calibration first. "Model today" must equal "Today" per group. A gap means a constraint the model misses (mixed shapes, pending pods, scale-down lag): explain it before trusting the after.
• Range, not a point. The ideal packing is a floor; real schedulers and the cluster autoscaler pack worse (spread preferences, 50% scale-down threshold). --fill defaults to 0.75; calibrate it on a cluster already in the target state when one exists.
• Structural changes first. Simulate the spread and replicas changes alone, then add the requests cuts on top: under a hard one-node-per-replica floor every requests cut shows 0, which hides what it saves once the floor is gone.
• A change with 0 in "Nodes saved by this change alone" gets no MR. Report it in the table as no gain and stop there, unless the user asks for it anyway.
• Scale-down is progressive. Soft spread and lower requests do not move running pods: the autoscaler drains underused nodes over hours.

5. Output

The answer is, in this order:

1. One line: environment, window (start date and why), pod- or container-level.
2. The workload table from node_impact.py, with the measured p95/p99/peak added per row: replicas, CPU req/limit, memory req/limit, before → after, and the nodes each change saves alone.
3. The node table per group: today, after (fill to ideal), machine type, list price per month today and after.
4. The machine-type table: nodes, USD/month, worst share of one workload per node and per zone, share of the group lost with one node. Recommend another shape only if it is cheaper and keeps both blast-radius columns at or below the current shape's.
5. The gain per change (one MR each), from the "Gain per change" table: nodes and USD/month each MR adds on top of the previous ones, structural changes first, so the rows sum to the total. A 1-node row is within bin-packing noise: say so.
6. The total saving in USD/month (list price), and the changes rejected for zero node gain.

6. Ship

Values change in the owning GitOps repo (mr-open), one MR per workload with a node gain; its description carries the node and cost lines. Then check the rollout. Offer a reminder at +7 days (tickler) to re-measure, count the nodes and look for OOMKills.