system-performance

Measure first, and say what you measured

"It's slow" is four different problems and they need opposite fixes. Do not tune anything until a reading points at one of them. The order below is by how often each turns out to be the answer, not by how interesting it is.

Every reading is run_command, which is visible and returns the exit code. Some of the sharper tools (iostat, mpstat, smartctl) are not installed by default — install_packages sysstat smartmontools when you need them, and say you are installing something before you do.

The four readings

Load and who is eating the CPU. run_command "uptime" gives the three load averages; compare them against run_command "nproc". A load of 8 on 8 cores is busy; a load of 40 is a queue. Then list_processes — it is the dedicated tool and it returns structure a ps line does not.

Memory and swap. run_command "free -h". The number that matters is available, not free: Linux uses spare RAM for cache and giving it back is instant. Swap in active use (si/so moving in vmstat 1 5) is the real signal, and it makes everything slow at once, which is why a memory problem often presents as a disk problem.

Disk. run_command "df -h" and — the one people forget — run_command "df -i" for inodes, which fill silently and break writes on a partition with gigabytes free. For I/O wait, the wa column of run_command "vmstat 1 5"; high wa with low CPU means the machine is waiting on a disk, not computing.

Network. run_command "ss -s" for socket totals, run_command "ip -s link" for errors and drops. A slow browser on a fast machine is usually DNS: run_command "dig +short example.com" and time it.

Name the bottleneck out loud

Before any change, say which of the four the readings point at and which reading says so. A tuning applied to the wrong one leaves the machine slow AND changed, which is worse than slow.

Changing things — the same rule as every other skill here

kill_process on something a person is using, a sysctl change, disabling a service: state what you are about to do and why, and use ask_human when the call is theirs rather than yours. vm.swappiness and /etc/security/limits.conf survive reboots and are the kind of change somebody should agree to.

Prefer the smallest change that tests the hypothesis. Then re-run the same reading and show before and after. A tuning that returned ok and did not move the number is the failure this project ranks worst — it looks fixed and is not.

The long ones are jobs

smartctl -t long, a full find /, a big dd benchmark: job_start, then job_wait. A wait that times out has not cancelled the work — wait again rather than starting a second copy, which on an I/O benchmark makes the very number you are measuring worse.