Linux Troubleshooting

Establish whether the host is the cause, using the four resources that actually run out: disk, memory, PIDs, and file descriptors.

Purpose

Use this when:

• A host or Kubernetes node is degraded, unresponsive, or NotReady.
• A systemd unit fails to start or restarts repeatedly.
• An application fails with "no space left on device", "cannot allocate memory", "too many open files", or is killed with no log entry of its own.
• A container symptom has no explanation inside the container.

Do not use this when:

• The host is healthy and the question is throughput or latency — use linux-performance.
• The problem is connectivity — use linux-network-debugging.

What this skill assumes: shell access to the host, and sudo for the parts that read other users' processes and the journal. State which you have; half of these commands silently show less without privileges.

Operating Procedure

1. Take a baseline in one pass. Uptime, load, memory, disk. Four commands before any theory.
2. Check the kernel's own record — dmesg and the journal. The kernel logs OOM kills, disk errors and filesystem remounts that no application log will ever contain.
3. Find what actually ran out. Disk bytes, disk inodes, memory, PIDs, file descriptors. These fail differently and are confused constantly.
4. Correlate with time. When did it start? What changed then — a deploy, a cron job, a log rotation, a certificate expiry?
5. Confirm the mechanism before fixing. "Disk is full" is a symptom; "the journal grew to 40GB because rate limiting is off" is a cause.
6. Fix the cause, mitigate the symptom. Often in that order reversed during an incident — free space now, fix the growth after.
7. Verify that the resource is recovered and stays recovered.

Distinguish Observed (command output on this host) from Assumed (what is usually true). Never assume a path, a unit name, or a distribution — check.

Investigation

Step 1 - the four-command baseline

uptime
free -h
df -h
df -i

Read them together:

• uptime load averages are runnable plus uninterruptible-sleep tasks on Linux. A load of 40 on an 8-core box with low CPU usage means processes are blocked on I/O or a dead mount, not that the CPU is busy. This distinction is the most commonly missed signal in Linux triage.
• free -h: the number that matters is available, not free. Linux uses spare memory for page cache by design; low free with high available is a healthy system.
• df -h is bytes. df -i is inodes. A filesystem can be at 12% bytes and 100% inodes, and the error is the same No space left on device. Running df -h alone and concluding there is space is a classic wrong turn.

Step 2 - the kernel's record

sudo dmesg -T | tail -50
sudo journalctl -p err -b --no-pager | tail -50
sudo dmesg -T | grep -iE 'oom|killed process|I/O error|read-only|ext4|xfs'

-T gives human timestamps. What to look for:

An application that vanished with no error in its own logs is almost always in this output.

Step 3 - what ran out

# Bytes: largest directories under the full filesystem
sudo du -x -h -d1 /var 2>/dev/null | sort -rh | head -10

# Deleted-but-open files: space not visible to du
sudo lsof -nP +L1 2>/dev/null | head -20

# Inodes: many small files
sudo find /var -xdev -type f 2>/dev/null | wc -l

# PIDs
ps -eo pid | wc -l ; cat /proc/sys/kernel/pid_max

# File descriptors, system-wide and per process
cat /proc/sys/fs/file-nr
sudo ls /proc/<pid>/fd | wc -l ; cat /proc/<pid>/limits | grep 'open files'

lsof +L1 is the answer to the most confusing disk-full case: df shows 100%, du accounts for far less, and nothing is deleting. A process holding a deleted file keeps its blocks allocated until the file descriptor closes. Restarting the holding process reclaims the space instantly; deleting more files does nothing.

-x on du keeps it on one filesystem, so you do not spend ten minutes walking a network mount.

Step 4 - systemd units

systemctl --failed
systemctl status <unit> --no-pager -l
sudo journalctl -u <unit> -n 100 --no-pager
sudo journalctl -u <unit> -b -1 --no-pager | tail -50
systemctl cat <unit>

systemctl status gives the last few lines and the exit code; journalctl -u gives the whole story. -b -1 reads the previous boot, which is the only way to see why a machine died.

systemctl cat shows the unit plus every drop-in override. A unit that behaves differently from the file you are reading almost always has a drop-in in /etc/systemd/system/<unit>.d/.

Step 5 - processes and time

ps aux --sort=-%mem | head -10
ps aux --sort=-%cpu | head -10
ps -eo pid,ppid,stat,etime,wchan:25,comm --sort=-etime | head -20
timedatectl

STAT of D is uninterruptible sleep — blocked in the kernel, usually on I/O or an unresponsive NFS mount. D-state processes cannot be killed, contribute to load average, and mean the fault is below the process.

Z is a zombie: harmless individually, but thousands mean a parent is not reaping children and will eventually exhaust the PID table.

timedatectl matters more than it looks: clock skew breaks TLS validation, token expiry, Kerberos, and distributed consensus, and produces errors that look like nothing to do with time.

Decision Tree

Host is unhealthy
├── df -h shows 100%
│   ├── du accounts for it       → find and rotate the large files
│   ├── du does not account for it → lsof +L1: deleted open files → restart the holder
│   └── df -i shows 100% inodes  → many small files (sessions, cache, mail spool)
├── Application killed with no log
│   ├── dmesg "Out of memory: Killed process" → host OOM
│   └── dmesg "Memory cgroup out of memory"   → container/unit limit
├── Load high, CPU idle
│   ├── Many D-state processes → I/O or a hung mount
│   └── iowait high            → storage → linux-performance
├── systemd unit fails
│   ├── 203/EXEC → path or permissions
│   ├── repeated too quickly → crash loop; read the app logs
│   └── Dependency failed → fix the upstream unit
├── "Too many open files"
│   ├── Process fd count near its limit → LimitNOFILE in the unit
│   └── System-wide file-nr near max    → fs.file-max
└── Filesystem read-only
    └── dmesg I/O errors → hardware or storage backend; do not just remount rw

Common Causes

Safety

Read-only

Every inspection command above: df, du, free, ps, lsof, dmesg, journalctl, systemctl status, systemctl cat, timedatectl. Safe on production.

du on a very large or network filesystem is read-only but can be slow and generate real I/O load on an already-struggling host. Use -x and target a subdirectory.

Potentially disruptive

• systemctl restart <unit> — drops connections and in-flight work. Check what depends on the unit first. Rollback: none; a restart cannot be undone.
• Truncating a log file with : > /var/log/<file> — frees space immediately without breaking the writer's file descriptor. Safer than deleting, but the contents are gone; copy what you need for the investigation first.
• sudo journalctl --vacuum-size=1G — deletes historical journal entries, including the ones that explain the current incident. Extract what you need before running it.
• kill -TERM <pid> — asks a process to stop. kill -9 does not ask. Use -9 only after -TERM has failed, and never on a database.
• Editing sysctl values live — takes effect immediately host-wide. Note the old value first: sysctl <key>.

Destructive

• rm -rf on any path under /var, /data, or a mount point — deletes data with no recovery. Required first: confirm the exact path with ls -la (never rely on a variable in the command), confirm the data is reproducible or backed up, and prefer truncation for logs.
• docker system prune -a / removing container storage — deletes images, stopped containers, build cache, and with --volumes any unused volume, which may hold data. Required first: an explicit confirmation and a check of what --dry-run-equivalent listing shows. On a Kubernetes node the kubelet handles image garbage collection itself; manual pruning can remove images a running workload will need to restart.
• Anything that writes to a block device (dd if=, mkfs) — this is not part of troubleshooting. If a disk needs repartitioning or reformatting, that is a planned maintenance change with the node drained.

On a Kubernetes node, cordon and drain before disruptive work rather than operating on a node that is still receiving traffic.

Verification

Before: / at 100% (0 bytes free), nginx failing with "No space left on device"
After:  / at 62% (34GB free), nginx serving, logrotate running daily, 24h growth 400MB

df -h / ; df -i /
systemctl is-active <unit> ; systemctl is-failed <unit>
sudo journalctl -u <unit> --since '5 minutes ago' -p err --no-pager

Then check it holds. A disk freed by deleting logs refills on the same schedule that filled it; verification is confirming the growth rate is now sustainable, not that there is space right now. Re-check after one full cycle of whatever caused it — usually 24 hours.

Anti-Patterns

Bad: Disk full, so rm -rf the biggest directory. Better: Find out what is growing and why. Deleting /var/log removes the evidence and the space comes back tomorrow. Truncate, then fix rotation.

Bad: free -h shows almost no free memory, so the host is out of memory. Better: Read available. Page cache is reclaimable and counting it as used makes every healthy Linux host look like it is about to die.

Bad: Load average is 30, so add CPU. Better: Check whether the tasks are runnable or blocked in D state. I/O-blocked load is not fixed by more cores.

Bad: kill -9 anything that will not stop. Better: -9 gives the process no chance to flush or release locks. For a database or a queue consumer that means corruption or duplicate work. Try -TERM, and if it is in D state, -9 will not work either — the fault is below the process.

Bad: Remount a read-only filesystem read-write to get the service back. Better: The kernel made it read-only because of an I/O error. Read dmesg first; remounting over a failing disk turns a contained failure into silent corruption.

Composition

• linux-performance — the host is healthy but slow.
• linux-network-debugging — the symptom is timeouts or connection failures.
• kubernetes-troubleshooting — the host is a node and the symptom appears in pods.
• production-debugging — the host is one signal in a wider outage.

References

• systemd: journalctl
• Linux kernel: OOM killer — how the kernel chooses a victim.