Server Security Hardening — Ubuntu VPS

When to use

• A new server was bootstrapped and needs the security layer.
• The user asks to secure/harden a server.
• After a large apt upgrade on any of these servers (verify everything survived).

Ground rules (apply before touching anything)

• This is an existing-server recipe, not blanket permission to upgrade, reboot, remove users or replace firewall policy. Inspect OS/Docker/firewall versions, current access and backups first. Keep an established SSH session and a recovery path; use sudo-handoff if root access is genuinely needed and unavailable.
• The usernames, ports and historical exclusions below belong to the original deployment. Infer the current host's authorized policy rather than copying them or treating an old backup exclusion as a universal preference.

1. Run everything inside server-side tmux (tmux new -d -s harden "bash x.sh 2>&1 | tee x.log"), fetch results with scp. Inline SSH stdout garbles when the user's connection drops; a half-applied sshd change is the worst outcome of a dropped packet.
2. Order is load-bearing: admin user(s) FIRST, sshd lockdown LAST. Never close root login before proving ssh <admin>@<host> + sudo -n whoami works from your workstation. When REPLACING an admin user: create the new one, widen AllowUsers to include old+new, verify the new one from the workstation, then tighten AllowUsers and userdel -r the old one (and delete its sudoers file) — never a big-bang rename.
3. If the target repo ships a hardening script (e.g. bootstrap/harden-server.sh, this procedure parameterized via its CONFIG block), that script is the canonical sequence; this skill explains the why and the verification.

The layers (in execution order)

1. Admin user(s) — one or more; the original deployment used two (<admin1>, <admin2>). Per user: useradd -m -s /bin/bash -G sudo <admin>; copy root's authorized_keys with correct ownership (install -m 600 -o <admin> -g <admin>); passwd -l <admin>; passwordless sudo via /etc/sudoers.d/90-<admin> (chmod 440, visudo -c to validate). Reason: keys are per-host, and reusing root for daily work is the thing we're eliminating.
2. Sysctl — /etc/sysctl.d/99-hardening.conf: rp_filter on, syncookies, accept/send redirects off, source-route off, log_martians, kptr/dmesg restrict, unprivileged BPF disabled. Apply with sysctl --system.
3. nginx — server_tokens off; (uncomment the existing commented line in nginx.conf — beware the commented line matches greps; verify with curl -sI, header must show nginx with no version).
4. DOCKER-USER guard — Docker's NAT rules BYPASS ufw INPUT for published container ports. Publishing 127.0.0.1:... is the primary control; the safety net is a DOCKER-USER chain rule dropping NEW/UNTRACKED traffic from external interfaces (v4 iptables + v6 ip6tables, with ESTABLISHED,RELATED accept first). Installed as oneshot docker-ufw-guard.service (After/Requires docker). Without this, a compose file with ports: ["3101:3101"] would be internet-exposed despite UFW.
5. fail2ban — backend = systemd (no auth.log on these images), jails: sshd (maxretry 4, ban 2h) and nginx-http-auth (ports http,https plus the app's public entry range, e.g. 3000:3099, logpath /var/log/nginx/error.log) for the basic-auth public entries.
6. unattended-upgrades — enable; local config /etc/apt/apt.conf.d/52unattended-upgrades-local: security updates daily, Automatic-Reboot "false" (uptime is a hard requirement for the user — when /var/run/reboot-required appears, surface it to the user and let them choose the moment; never reboot on your own initiative), remove unused deps.
7. sshd lockdown (last) — ONE drop-in /etc/ssh/sshd_config.d/00-hardening.conf: PermitRootLogin no, AllowUsers <admin1> <admin2> ..., PubkeyAuthentication yes, PasswordAuthentication no, KbdInteractiveAuthentication no, PermitEmptyPasswords no, X11Forwarding no, MaxAuthTries 3. Validate sshd -t, then systemctl reload ssh (reload, not restart — keeps existing sessions).
8. Upgrade + reboot — separately from the hardening script, only when authorized. Preview package/removal changes and confirm a maintenance/recovery plan before a full upgrade. Check /var/run/reboot-required; do not reboot until the user authorizes its timing, then run the post-reboot verification below.

sshd gotcha (bit us once)

sshd applies first-wins per directive. Include order means the drop-in must sort FIRST (00-), and older drop-ins (00-key-only.conf with prohibit-password) silently defeat a later 10-hardening.conf with no. Always end with a single merged 00-hardening.conf; verify with sshd -T | grep -E "permitrootlogin|allowusers" (that shows the EFFECTIVE values, not the files).

Post-reboot verification (every layer)

uname -r                                   # new kernel actually booted
ip route show default                      # has `src <primary-ip>` (dual-IP egress fix)
sudo docker ps                             # containers restarted
sudo systemctl is-active docker-ufw-guard fail2ban unattended-upgrades
sudo iptables -S DOCKER-USER | head        # guard rules present
sudo ufw status verbose                    # ACTIVE — see gotcha below
sudo sshd -T | grep -E "^(permitrootlogin|allowusers)"
curl -sk -o /dev/null -w "%{http_code}\n" https://127.0.0.1:<public-entry-port>/   # public entry 401
curl -s -o /dev/null -m 10 -w "%{http_code}\n" https://api.snapcraft.io  # egress 200
nslookup google.com                        # DNS works

From your workstation, in this order: ssh <admin>@<host> works → sudo -n whoami → only then confirm root login is refused (Permission denied is the expected result).

UFW gotcha (bit us once)

A large apt upgrade can reinstall/reconfigure ufw and leave it inactive (ENABLED=no in /etc/ufw/ufw.conf) even though rules persist. After any major upgrade: sudo ufw status. Before enabling an inactive firewall, verify the actual SSH port/source allow rule and recovery access; enabling is not automatically safe just because old rules exist.

Rules that must never regress

• Confirm Docker's firewall backend. The DOCKER-USER recipe above applies to the iptables backend; nftables uses different hook/chain handling. Consult the matching official backend documentation and verify IPv4/IPv6 exposure; do not create a fictitious DOCKER-USER chain for native nftables or flush existing policy.
• Never run the sample userdel -r during an ordinary hardening pass. Account removal needs explicit authorization and preservation of owned files/keys after replacement access is proven.
• No application port on 0.0.0.0 — 127.0.0.1:<host>:<container> only; databases get no ports: at all.
• Public surface = sshd (22) + nginx (80/443) + the deployment's public entry range (in the original deployment 3000–3099; TLS + basic auth + app auth; /api/websocket locations need auth_basic off — browsers drop the header on WS handshakes).
• Compose review checklist before any deploy (privileged/host modes/docker.sock/host mounts/cap_add/user: root) — see project AGENTS.md.
• Compose/infra changes go through tmux + logs + scp, never long inline SSH.

Non-goals / options not deployed

• WireGuard-only management (no public basic-auth entry) — stronger, but the original deployment chose a public IP:<port> basic-auth entry for now; kept there as a pending option.
• Off-server backups — out of scope in the original deployment; the compose/data layer should eventually add them.

Validation

• Confirm current admin SSH and required privilege access before tightening old access; effective sshd/firewall values match the intended host policy.
• Run applicable post-change checks above after reload and again after any authorized reboot. Mark unavailable checks as not run; never infer successful reboot or protection from written config alone.
• Verify external exposure from an approved client and preserve rollback/recovery access; no unauthorized account removal, upgrade or reboot occurred.