Linux Server Ops

A full disk or a starved server usually has one boring cause: a log nobody rotates, a cache nobody prunes, a process that leaks. The real danger is the cleanup, like docker system prune -a --volumes on a live box deleting a database. This skill finds the cause with measurements, frees space in the safest order, and stops it coming back.

Commands target Ubuntu and Debian. Notes for RHEL-family, Alpine and Arch are in references/commands.md, which holds every copy-paste command this file refers to.

What it does

1. Inventory the server read-only: disks, inodes, memory, swap, top processes, services, containers.
2. Diagnose what is actually growing and why, with numbers.
3. Fix in a fixed order, safest first, measuring each step. Anything destructive waits for a yes.
4. Prevent recurrence: rotation, size limits, alerts.
5. Verify services are healthy and write SERVER-REPORT.md from templates/SERVER-REPORT.md.

Safety rules

• Read-only first. Inventory and diagnose change nothing. Some passive-sounding commands mutate: npm cache verify garbage-collects, docker buildx du can write config, and any clean, prune, purge, gc, vacuum or autoremove deletes. Keep those for the fix step.
• Measure before and after every change. The final df difference is the real result, not the sum of what you deleted (hard links, sparse files and other writers make them differ).
• Never delete data you cannot identify. If you cannot say what wrote a file, what reads it and how it would be rebuilt, it stays. Report it as "unknown, kept".
• Always keep, unless the owner names the exact item: databases and their WAL or journal files, Docker volumes, backups and snapshots, git repositories, user uploads and home directories, credentials and keys, anything under /var/lib that belongs to a running service, files under /tmp that are recent or held open.
• Ask before destructive steps. Show the exact command, paths, measured size and what is lost. A yes covers that bundle only.
• Prefer reversible moves. Truncate instead of delete, move instead of remove, disable a service before removing its unit, back up a config before editing it.
• Do not kill what you do not understand. Find the process's service and owner, and stop it through systemd rather than kill -9.
• Lock-out safety. Never change SSH, firewall or network config without a second way in (provider console or a second SSH path). Keep the current session open and test from a new one. Validate first (sshd -t, nft -c). No second path, no change: write it up instead.
• Do not make it worse. On a nearly full disk, install and download nothing during the audit.
• Secrets stay secret. Never print .env files, keys or credentialed unit files. Redact.

Steps

1. Inventory (read-only)

Run the inventory block in references/commands.md: df in bytes and inodes (separate problems), memory and swap, top processes, failed units, docker system df, journalctl --disk-usage, and the timestamp. Write the numbers into the report's "Before" section straight away.

2. Diagnose

Work out which of these it is. More than one is common.

Rules for diagnosis:

• Use du -x so mounts are not counted twice. du -d1 / hides regular files directly under /; a swap file there is often the "missing" gigabytes.
• The largest file is not automatically the cause. A large database can be correct; an unbounded log or cache is the defect.
• Measure growth twice, do not guess it. Check whether a noisy log is the cause or a symptom of a disk that was already full.
• For memory, page cache is fine. "Available is low and swap is busy" is the problem, not "free is low".

3. Fix (safest first)

Go in this order and stop once there is comfortable headroom (at least 15% or a few GB free). Record bytes freed per step.

1. Stop the bleeding. Truncate a fast-growing file in place (truncate -s 0 FILE) once you know its producer. Deleting it frees nothing while the process holds it open.
2. Deleted-but-open files. Restart or reload the owning service. Ask first on production.
3. Package caches. apt-get clean; language caches only if nothing is mid-install.
4. Journald. journalctl --vacuum-size=500M.
5. Rotated logs, core dumps, crash reports past retention, after noting any crash in the report.
6. Old kernels. apt-get autoremove --purge, never the one from uname -r.
7. Docker: truncate oversized container logs, then dangling images, build cache, identified stopped containers. Ask first before docker image prune -a. Never remove volumes unless the owner names each one: that is where databases live.
8. /tmp: only files past a stated age, not open, from a producer you recognise.
9. Memory: restart a leaking process through systemd, cap it with MemoryMax=, or add swap if there is none. Do not disable the OOM killer.

If the disk is still full after the safe steps, more deletion is not the answer. Give the owner options with measured sizes: bigger disk, move data to another volume, or retire a workload.

4. Prevent

Add only what addresses the cause you found, and validate each config with its own checker. Snippets are in references/commands.md.

• logrotate for app logs without it. Test with logrotate -d.
• journald limits via a drop-in (SystemMaxUse=, SystemKeepFree=).
• Docker log limits in daemon.json (max-size, max-file). They only apply to new containers; existing ones need recreating (ask first) or a logrotate rule for their exact log paths.
• Bounded scheduled cleanup, e.g. weekly build-cache prune older than 7 days. No -a, no volumes in cron.
• A disk alert that is silent when healthy, checks block %, free bytes and inode %, and has a cooldown. Wire it to the notification channel you already use.
• Core dump cap with MaxUse=.

5. Verify

• Rerun df -B1, df -i, free -h. Report used-space reduction and available-space gain separately (on ext4 freed space refills reserved blocks first).
• systemctl --failed unchanged; healthy containers still healthy, same IDs unless recreated on purpose.
• No new "No space left on device" since the fix; the growth source measured again and stable.
• Every new config passed its validator; the alert is silent when healthy and fires on a forced test.

Output

Fill templates/SERVER-REPORT.md (off the full disk if possible): a three-line summary (what was wrong, what was done, current headroom), before and after numbers, actions with bytes freed, what was kept and why, prevention added, and decisions waiting on the owner. In the chat, give only the summary, the before and after line, and pending yes or no questions.

Honest limits

• Triage and hygiene, not capacity planning. If data is legitimately growing, the fix is more disk.
• Numbers are point-in-time; on a busy server they move while you measure.
• It cannot know your retention rules (compliance logs, audit trails). State them before cleanup.
• It does not tune databases (VACUUM, compaction). That needs the database's own tools and a maintenance window.
• Tested on Ubuntu and Debian; other distros get notes, not guarantees.
• With no backup and no second way in, it will tell you to fix that first and will refuse SSH or firewall changes.