Linux Performance

Find which resource is saturated before changing anything, using utilisation, saturation and errors for each one.

Purpose

Use this when:

• A host is slow but not failing: high latency, low throughput, no errors.
• Latency regressed and you need to know whether the cause is CPU, memory, disk or the application.
• Someone proposes resizing an instance or adding replicas without evidence.
• A container is slow and the container's own metrics look fine.

Do not use this when:

• The host is out of a resource entirely (disk full, OOM) — use linux-troubleshooting.
• The latency is in network round trips — use linux-network-debugging.
• The workload is in Kubernetes and the suspicion is cgroup limits — use kubernetes-resource-debugging, which measures throttling directly.

What this skill assumes: shell access, and ideally sysstat (sar, iostat, pidstat). If those are unavailable, /proc gives you most of the same numbers; the skill notes where.

Operating Procedure

Work the USE method: for each resource, measure Utilisation, Saturation and Errors. Utilisation alone is misleading — a disk at 100% utilisation with 0.2ms latency is fine, and a CPU at 40% with a run queue of 30 is not.

1. Define slow. Which operation, measured where, how much worse than what baseline? "The site is slow" is not a starting point.
2. Take a 60-second system snapshot covering all four resources.
3. Identify the saturated resource. One usually stands out.
4. Attribute it to a process with pidstat — a saturated host tells you nothing about which workload is responsible.
5. Confirm the mechanism before changing anything: throttling, swapping, I/O waiting, lock contention.
6. Change one thing.
7. Verify against the original latency measurement, not against the resource metric.

Resource metrics are Observed. The link from a resource to the user-visible symptom is an Assumption until you have shown they move together.

Investigation

Step 1 - the 60-second snapshot

uptime
vmstat 1 10
mpstat -P ALL 1 5      # or: cat /proc/stat
iostat -xz 1 5         # or: cat /proc/diskstats
pidstat 1 5
free -h

This is the standard first-minute sweep. What to read from vmstat:

Ignore the first row of vmstat: it is an average since boot, not a current sample. Reading it as current is a common mistake.

st (steal) is the one people miss. On a burstable or oversubscribed cloud instance, sustained steal means your neighbours are taking the CPU; no amount of application tuning fixes that, and the answer is a different instance type.

Step 2 - CPU: utilisation, saturation, and who

mpstat -P ALL 1 5
pidstat -u 1 5
cat /proc/pressure/cpu        # kernel >= 4.20

mpstat -P ALL splits per core. One core at 100% while others idle means single-threaded work or interrupt affinity pinning, and adding cores will not help.

Pressure Stall Information is the best saturation signal available and is widely under-used:

some avg10=42.31 avg60=38.90 avg300=30.11 total=...

some is the share of time at least one task was stalled waiting for CPU. Above ~20% sustained means real contention regardless of what utilisation says. /proc/pressure/memory and /proc/pressure/io give the same for their resources and are the fastest way to identify which of the three is hurting.

Step 3 - memory: pressure, not free bytes

free -h
vmstat 1 5     # si/so columns
cat /proc/pressure/memory
sar -B 1 5     # pgscank/pgscand/pgsteal: reclaim activity

Memory is a problem when the kernel is working for it, not when free is low:

• si/so sustained non-zero: actively swapping. Latency will be terrible and inconsistent.
• pgscan/pgsteal high: the kernel is reclaiming page cache under pressure. This shows up as slow disk reads because the cache keeps getting dropped.
• /proc/pressure/memory some above ~10%: real memory pressure.

A host with 200MB free and no swap activity and no reclaim is healthy. A host with 4GB free that is swapping is not.

Step 4 - disk: latency is the number that matters

iostat -xz 1 5

%util is the most misread field in Linux performance work. On a spinning disk it approximated saturation; on an SSD or NVMe device that services many requests in parallel, 100% %util with 0.3ms await is a healthy, well-fed device. Judge disks by await and aqu-sz.

Rough expectations: NVMe under 1ms, SSD 1-5ms, network-attached volumes 2-20ms, spinning disk 10-20ms. Read latency an order of magnitude above these means the device, the volume's IOPS credit balance, or the storage backend.

pidstat -d 1 5

Attributes I/O to processes. Without this you know the disk is busy but not who is making it busy.

Step 5 - attribute to the application

pidstat -u -r -d -p <pid> 1 5
cat /proc/<pid>/status | grep -E 'Threads|voluntary'
sudo cat /proc/<pid>/stack 2>/dev/null

A high nonvoluntary_ctxt_switches count means the scheduler is preempting the process — CPU contention. A high voluntary_ctxt_switches count means it is blocking on something itself — I/O, a lock, or a downstream call. That single ratio separates "starved of CPU" from "waiting on someone else" and points at completely different fixes.

If deeper attribution is needed and the tools are present:

sudo perf top -p <pid>            # where CPU time is spent, by symbol
sudo biolatency 10 1              # bcc: disk latency histogram

perf and bcc tools add overhead. On a production host under load, sample briefly rather than running them continuously.

Decision Tree

Host is slow
├── vmstat st > 2%          → hypervisor steal; the host is oversubscribed; change instance type
├── vmstat r >> cores       → CPU saturation → pidstat to find the process
│   ├── One core pinned     → single-threaded or IRQ affinity
│   └── All cores busy      → genuine CPU demand; scale or profile
├── vmstat si/so non-zero   → swapping; reduce memory use or add memory; disable swap for latency
├── iostat await high
│   ├── aqu-sz high         → device saturated → pidstat -d to attribute
│   ├── aqu-sz low          → device itself is slow → volume type, IOPS credits, backend
│   └── Cloud volume        → check burst-balance / provisioned IOPS before blaming the app
├── PSI io "some" high, CPU low → I/O bound regardless of what %util says
├── Nothing saturated
│   ├── voluntary ctx switches high → blocked on a downstream dependency, not on this host
│   └── Application-level → lock contention, GC, connection pool → production-debugging
└── Container context → check cgroup throttling → kubernetes-resource-debugging

Common Causes

Safety

Read-only

uptime, vmstat, mpstat, iostat, pidstat, sar, free, and everything under /proc. These read counters and cannot change behaviour.

perf and bcc/eBPF tools are read-only in effect but add measurable overhead — a few percent CPU, occasionally more on a busy host. Sample for seconds, not hours, and say you are running them.

Potentially disruptive

• sysctl -w on vm.swappiness, vm.dirty_ratio, scheduler or network parameters — takes effect host-wide immediately. Record the current value first (sysctl <key>) so you can restore it, and put the permanent version in /etc/sysctl.d/ only after it is proven.
• renice / ionice on a running process — changes scheduling priority. Deprioritising the wrong process makes the incident worse.
• swapoff -a — forces everything back into RAM. On a host that is already under memory pressure this triggers the OOM killer immediately.
• Changing IRQ affinity or RPS/RFS settings — affects all traffic on the host.
• Resizing an instance — requires a stop/start on most clouds. That is downtime; drain first.

Destructive

This skill proposes no destructive operations. Performance work is measurement; if a fix requires destroying and recreating an instance or a volume, that is a change to plan under terraform-safe-changes or the equivalent, with the data path considered separately.

Verification

Verify against the symptom, not the resource.

Before: p99 API latency 2.4s; disk w_await 45ms; PSI io some avg60 38%
After:  p99 API latency 180ms; disk w_await 1.2ms; PSI io some avg60 2%

iostat -xz 1 10
cat /proc/pressure/io /proc/pressure/cpu /proc/pressure/memory
vmstat 1 10

Two rules:

1. Measure over a full traffic cycle. A fix verified at 03:00 has not met the peak.
2. Confirm the resource metric and the user metric moved together. If await improved and p99 did not, the disk was never the cause — which is a genuinely useful result, not a failure.

Anti-Patterns

Bad: %util is 100%, the disk is the bottleneck. Better: Check await and aqu-sz. On SSD and NVMe, %util saturates at trivial load and says almost nothing.

Bad: Add CPU because load average is high. Better: Linux load includes uninterruptible-sleep tasks. High load with low CPU utilisation is an I/O problem, and more cores will not touch it.

Bad: Tune sysctl parameters found in a blog post. Better: Measure first, change one parameter, verify. Most published tuning is for a specific workload on a specific kernel, and some of it is actively harmful on modern defaults.

Bad: Read the first line of vmstat output. Better: That row is the average since boot. Always take several samples and read from the second onwards.

Bad: Conclude the application is slow because the host looks idle. Better: An idle host with high voluntary_ctxt_switches means the process is waiting on something else — a database, a lock, a remote call. That is a real, findable cause, not a dead end.

Bad: Scale out to fix latency. Better: If the bottleneck is a shared dependency or a single-threaded section, more replicas add load to the same bottleneck and make it worse.

Composition

• linux-troubleshooting — a resource has run out rather than being merely contended.
• linux-network-debugging — the latency is in the network path.
• kubernetes-resource-debugging — cgroup limits and CPU throttling in containers.
• production-debugging — the bottleneck is a downstream service.

References

• Brendan Gregg: the USE method — the framework this skill follows.
• Linux kernel: Pressure Stall Information — what some and full mean.