Troubleshooting Methodology

Follow this sequence. Do not skip steps.

1. Gather symptoms: What exactly is broken? When did it start? What changed recently?
2. Check the obvious: Is the service running? Is there disk space? Is DNS resolving? Is the port open?
3. Read the logs: Start with the application logs, then system logs, then infrastructure logs.
4. Isolate the layer: Network, DNS, OS, runtime, application, database, external dependency?
5. Form a hypothesis: "I think X is causing Y because the logs show Z."
6. Test the hypothesis: Make one change at a time. Verify before and after.
7. Document: Record the root cause and the fix for future incidents.

Never make changes in production without understanding what you are changing and how to reverse it.

Log Analysis

When analyzing logs:

• Start with the error/fatal level. Work backward from the first error.
• Check timestamps. Correlate events across services using a shared timeline.
• Look for patterns: repeated errors, error rate changes, periodic spikes.
• Search for request IDs or correlation IDs to trace a request across services.
• Count occurrences. A single error is different from 10,000 errors per minute.

Key log locations:

• systemd services: journalctl -u service-name --since "1 hour ago"
• Nginx/Apache: /var/log/nginx/error.log, access log for 5xx responses
• Application: Check the app's configured log path or stdout/stderr
• Kernel: dmesg -T for OOM kills, disk errors, network issues

Monitoring

The four golden signals (monitor all four):

1. Latency: How long requests take. Track p50, p95, p99. Alert on p99 breaches.
2. Traffic: Requests per second. Understand normal baselines to detect anomalies.
3. Errors: Error rate as a percentage of total requests. Alert on sustained increases.
4. Saturation: CPU, memory, disk, network utilization. Alert before hitting limits.

Dashboard layout: Overview (up/down, error rate) at the top. Golden signals in the middle. Infrastructure metrics at the bottom.

Incident Response

Severity levels:

• SEV1: Service down, data loss, security breach. All hands. Communicate every 15 minutes.
• SEV2: Degraded performance, partial outage. Assigned responder. Communicate every 30 minutes.
• SEV3: Minor issue, workaround exists. Fix in next business day.

During an incident:

1. Mitigate first, investigate second. Restore service, then find root cause.
2. Communicate early and often. "We are investigating" is better than silence.
3. Keep a timeline of actions taken and observations.
4. Do not deploy new code during an incident unless it is the fix.

After an incident: Write a blameless postmortem within 48 hours. Include: timeline, root cause, impact, what went well, what to improve, and action items with owners.

Common Checks

Network: ping, traceroute, dig, curl -v, ss -tlnp, netstat -an Disk: df -h, du -sh *, lsof +D /path, iostat Memory: free -h, vmstat 1, check OOM killer in dmesg CPU: top, htop, mpstat, load average context (1/5/15 min) Process: ps aux, systemctl status, strace -p PID for stuck processes

Security Basics

• Keep systems updated. Unpatched vulnerabilities are the most common attack vector.
• Use SSH keys, not passwords. Disable password auth in sshd_config.
• Firewall: Default deny. Open only what is needed. Document every rule.
• Least privilege: Services run as non-root. Users get minimum necessary permissions.
• Audit: Enable auditd or equivalent. Log authentication events and privilege escalation.
• Backups: 3-2-1 rule. Test restores regularly. Untested backups are not backups.