incident-triage

The loop, and why it is a loop

Guessing is how a dead service becomes two dead services. Diagnose by hypothesis: form one, make one reading that would confirm or kill it, and only then act. check_errors exists for exactly this first pass — use it before you start reading raw logs by hand.

Every reading below is run_command (visible, exit code returned) or a dedicated tool. Nothing here changes the machine until the "act" section, which announces itself first.

1. What is actually wrong

• The symptom, precisely. "It is down" is three different problems: the process is not running, it is running and not listening, or it is listening and refusing. service_control <unit> status, then list_processes, then run_command "ss -tulpn".
• What it is saying. run_command "journalctl -u <unit> -n 100 --no-pager" and journalctl -p 3 -b for this-boot errors across everything. The last lines before it stopped are the ones that matter — read the tail, not the head.
• What changed. run_command "journalctl -b -1 -p warning" for the previous boot; recent package changes via run_command "grep -E 'install|upgrade' /var/log/dpkg.log | tail -40". Most incidents are a change nobody connected to the symptom.

2. The four resources that cause most of it

• Disk full — the commonest and most disguised. run_command "df -h" and run_command "df -i" (inodes fill silently). A full / breaks things that look unrelated: a service that cannot write its socket, a database that cannot extend, a log that stopped mid-line.
• Memory / OOM — run_command "free -h", then run_command "dmesg -T | grep -i oom". A process that "crashed" may have been killed by the kernel.
• A dependency — the database, the socket, the mount, DNS. If the unit needs another and that one is down, fix the cause, not the symptom.
• Permissions / SELinux/AppArmor — a service that starts and immediately dies is often a file it can no longer read. The journal says so; read it.

3. Act — say it first, then do it

State the hypothesis, the fix, and how you will confirm it, in one line, before you run it. Restarting a service a person is mid-request on is itself an incident: desktop_state says who is here; a service_control restart on a shared box is worth a word first, or an ask_human if the call is theirs.

Prefer the smallest fix that tests the hypothesis: clear the specific full partition, restart the one unit, fix the one permission. A reboot "to be sure" throws away the state that would have told you the cause, so it is the last move, not the first.

4. Confirm, and leave the trail

After the fix, re-run the reading that showed the symptom. ok from service_control is not recovery — the port answering is. Then say, in one paragraph: what was wrong, the evidence, what you changed, and how you confirmed it. The action log already has every command; your summary is the why that the log cannot hold, and it is what makes the next person's incident shorter.