server-security

---
name: server-security
description: Audit and harden basic security on a Debian/Ubuntu server over SSH — system updates, non-root sudo user, SSH key authentication, sshd hardening, UFW firewall, unattended-upgrades, fail2ban, sudo and home directory permissions. Use this whenever the user mentions auditing a server, hardening SSH, locking down a VPS, checking server security, configuring ufw, setting up fail2ban, securing a new server, or asks "is my server secure" / "what should I do to my new VPS" — even when they don't use the word "audit". Always produces a read-only audit report first, then applies fixes only after explicit confirmation, with a fail-safe automatic rollback for SSH configuration changes so the user cannot get locked out.
---

# Server Security

A disciplined audit-then-harden workflow for a fresh or unsupervised Debian/Ubuntu server, reached over SSH. The skill covers the eight areas that block roughly 80% of automated attacks: system packages, a non-root sudo user, ed25519 SSH keys, sshd hardening, UFW, unattended-upgrades, fail2ban, and sudo + home permissions.

The skill is split into six phases. Do not skip phases. The audit phase **must not mutate anything**; mutations only happen in Phase 5 after the user explicitly approves the plan.

When parsing or fixing a specific check, consult `references/checks.md` for the precise commands and target values. When touching `sshd_config`, consult `references/ssh-safety.md` for the fail-safe rollback protocol — never edit sshd without it.

## Phase 1 — Connect and detect environment

Ask the user for whatever connection details are missing. Required:

- **Host** (IP or DNS name)
- **User** to connect as (often `root` initially on a fresh server, or an existing `deploy`-style user)
- **Port** (default 22)
- **Identity file** (optional; SSH agent or default keys are fine)

Then verify reachability and capabilities. Use one `ssh` invocation that runs a small bundle of read-only commands and parse the output:

```bash
ssh -o ConnectTimeout=10 -o BatchMode=yes -p <port> <user>@<host> \
    'set -e; \
     . /etc/os-release && echo "OS=$ID VERSION=$VERSION_ID"; \
     id; \
     command -v sudo >/dev/null && echo "sudo=present" || echo "sudo=missing"; \
     sudo -n true 2>/dev/null && echo "sudo=passwordless" || echo "sudo=needs-password"; \
     uname -r'
```

From the output decide:

- **OS check.** If `ID` is not `debian` or `ubuntu` (or a derivative like `linuxmint`, `pop`), stop. Tell the user the skill currently targets Debian/Ubuntu and ask whether to continue at their own risk (in which case some commands may fail).
- **Privilege check.** Many audit commands require root reads (`/etc/ssh/sshd_config.d/`, `sshd -T`, `ufw status`, `fail2ban-client`). If the user has `sudo` but it requires a password, the skill must run with sudo and the user must be prepared to type their password — or grant passwordless sudo for the duration. If the connecting user has no `sudo` at all (and is not root), abort with a clear message.
- **Connecting-as-root note.** Connecting as root is normal on a brand-new VPS. The skill should plan to create a non-root user as part of the fix, then advise switching the SSH session to that user before disabling root login.

**Speed tip.** If you will run more than a few SSH commands, enable a ControlMaster session so each subsequent command does not pay the handshake cost:

```bash
mkdir -p ~/.ssh/cm && \
ssh -o ControlMaster=auto \
    -o ControlPath=~/.ssh/cm/%r@%h:%p \
    -o ControlPersist=10m \
    -p <port> <user>@<host> true
```

Subsequent `ssh` calls with the same `ControlPath` reuse the connection. Tear it down at the end with `ssh -O exit <user>@<host>`.

## Phase 2 — Audit (read-only)

Run all eight checks. **Read only — no mutations, no service restarts, no package installs.** If a command requires root, prepend `sudo` (it will use the cached or passwordless sudo).

Collect the raw output and parse into a structured result per check. `references/checks.md` contains the exact detection command and the expected pass condition for each. The eight checks are:

1. **System updates** — `apt list --upgradable` count, age of `/var/cache/apt/pkgcache.bin`.
2. **Non-root sudo user** — at least one non-root user in `sudo` (or `wheel` on RHEL-derivatives, but we only support Debian/Ubuntu) with a populated `~/.ssh/authorized_keys`.
3. **SSH key strength** — the keys in `authorized_keys` are ed25519 or rsa ≥ 3072 bits.
4. **sshd hardening** — effective values via `sudo sshd -T` match the target table in `references/checks.md`. Also enumerate `/etc/ssh/sshd_config.d/*.conf` in lexical order and flag any earlier-loading file that conflicts.
5. **UFW firewall** — active, default-deny incoming, default-allow outgoing, SSH port allowed.
6. **Unattended-upgrades** — package installed, enabled in `20auto-upgrades`, security origin enabled in `50unattended-upgrades`, automatic-reboot configured.
7. **Fail2ban** — service active, `sshd` jail enabled, ban policy reasonable.
8. **Sudo and home permissions** — `sudo visudo -c` clean, target user's home is mode `700`.

For each check, record:

- `status`: `pass` / `warn` / `fail` / `skipped`
- `current`: what was observed (short string, e.g. `PermitRootLogin=yes`)
- `target`: what the article expects
- `severity`: `info` / `low` / `medium` / `high`
- `notes`: anything the user needs to know (e.g. "drop-in `00-cloud-init.conf` sets `PasswordAuthentication yes` before our hardening file would load")

## Phase 3 — Present report

Produce a single markdown report in this exact shape so the user can scan it in one pass:

```markdown
# Server security audit — <host>

Connected as `<user>` to `<host>:<port>` on <ID> <VERSION>. Kernel `<uname -r>`.

## Summary
- ✅ Passing: N
- ⚠️ Warnings: N
- ❌ Failing: N

## Findings

| # | Area | Status | Current | Target | Severity |
|---|------|--------|---------|--------|----------|
| 1 | System updates | ⚠️ | 23 packages upgradable | up to date | low |
| 2 | Non-root sudo user | ✅ | `deploy` with ed25519 key | non-root + key | – |
| 3 | SSH PermitRootLogin | ❌ | `yes` | `no` | high |
| … |

## Details

### 3. SSH PermitRootLogin
Currently `PermitRootLogin yes` (set in `/etc/ssh/sshd_config` line 32). Allowing direct root logins exposes the most-targeted account on the server to credential attacks. Fix: drop `PermitRootLogin no` into `/etc/ssh/sshd_config.d/01-hardening.conf`. **Before disabling, verify a non-root sudo user with a working key — Phase 2 check #2.**

### …
```

For each failing or warning check, include a short `Details` paragraph explaining what was found and why it matters. Do not propose commands here — that comes in Phase 4.

## Phase 4 — Confirm fix plan

Group fixes into two batches and present them separately.

**Batch A — Safe fixes** (low risk of lockout):
1. `apt update && apt upgrade -y` and install missing packages.
2. Create the non-root sudo user and install the SSH public key (if missing).
2b. Configure sudo authentication for that user — NOPASSWD drop-in or `passwd`. Ask the operator to pick one during the Phase 4 prompt; this is mandatory, not optional. See `references/checks.md` § 2 for the trade-offs.
3. UFW configuration.
4. Unattended-upgrades configuration.
5. Fail2ban configuration.
6. Home directory permissions, sudoers validation.

**Batch B — SSH hardening** (lockout risk if misconfigured):
7. Drop-in `/etc/ssh/sshd_config.d/01-hardening.conf` with the target values.
8. Restart `ssh` — but only via the rollback-guarded protocol in `references/ssh-safety.md`.

Show each batch as a numbered list. For each item, show the exact command(s) that will run. Ask the user, in one message, "Apply Batch A?" — yes/no/select-subset — and separately "Apply Batch B?" with an explicit note that this is the lockout-risk batch and the rollback protocol will run automatically.

If the user wants to cherry-pick, accept a list like `A: 1,3,5` or natural language ("skip the firewall, do the rest"). Re-state what will run and ask once more for confirmation before doing anything.

## Phase 5 — Apply fixes

**Interactive commands.** Anything that prompts the operator for input — `passwd`, an editor opened by `visudo` without `-c`, `dpkg-reconfigure` at its default priority, `apt` configfile diffs — cannot run through the agent's shell. The agent's stdin is not wired to the operator. When the skill needs one of these, tell the operator to run it in a separate terminal (or in their existing one) and wait for confirmation in chat before continuing. Never try to script around an interactive prompt with heredocs, `expect`, or by accepting the secret in chat to "automate" it — these either fail outright or leak the secret into the transcript.

Order matters. Run Batch A first, in this order, stopping the whole batch on the first failure:

1. **System updates and packages**
   ```bash
   sudo apt update
   sudo apt upgrade -y
   sudo apt install -y sudo curl wget gnupg ca-certificates ufw fail2ban unattended-upgrades
   ```
   `apt upgrade -y` can prompt about modified config files (`/etc/...`). Pass `DEBIAN_FRONTEND=noninteractive` plus `-o Dpkg::Options::="--force-confold"` to keep existing config files. If a kernel upgrade installed, note that a reboot is recommended (do not reboot inside the skill — surface it as a final action item).

2. **Non-root user — account + key.** Skip if a suitable user already exists with a working key.
   ```bash
   sudo adduser --gecos "" --disabled-password <name>
   sudo usermod -aG sudo <name>
   sudo install -d -m 700 -o <name> -g <name> /home/<name>/.ssh
   echo "<public-key>" | sudo install -m 600 -o <name> -g <name> /dev/stdin /home/<name>/.ssh/authorized_keys
   ```
   The public key must come from the user. If they do not have one, point them at `ssh-keygen -t ed25519 -a 100 -C "<name>@<hostname>"` on **their local machine** — never generate the private key on the server.

2b. **Non-root user — sudo authentication.** `adduser --disabled-password` puts `!` in `/etc/shadow` — the account cannot use sudo until you give it a way to authenticate. This is a mandatory step before Batch B; do not skip it. The operator picked Option A or Option B during Phase 4 — apply the matching commands now (full rationale in `references/checks.md` § 2):

   - **Option A — NOPASSWD sudo:**
     ```bash
     printf '%s ALL=(ALL) NOPASSWD:ALL\n' "<name>" \
         | sudo install -m 0440 -o root -g root /dev/stdin /etc/sudoers.d/90-<name>
     sudo visudo -c
     ```
   - **Option B — password-protected sudo:** `passwd` is interactive and **cannot run through the agent's shell**. Do not try — no heredoc, no piping, no accepting the password in chat. Tell the operator to open a separate terminal, log in as the existing sudo user (or root), and run there:
     ```bash
     sudo passwd <name>
     ```
     Wait for the operator to confirm "done" in chat before continuing.

   **Verify before continuing.** Ask the operator to open a second terminal: `ssh -p <port> <name>@<host>`, then run a sudo command **in that session** — `sudo -n true` for Option A, `sudo true` for Option B. Wait for the operator to confirm "got a shell + sudo works" in chat. Without this confirmation, Batch B is unsafe: the SSH rollback can only be cancelled from a session that can run sudo, and the existing root session may not survive the reload.

3. **UFW.**
   ```bash
   sudo ufw default deny incoming
   sudo ufw default allow outgoing
   sudo ufw allow <ssh-port>/tcp comment 'ssh'
   sudo ufw logging low
   yes | sudo ufw enable
   ```
   If the user already has other rules (web ports, etc.), preserve them — list current rules first with `sudo ufw status numbered` and confirm.

4. **Unattended-upgrades.** Configure non-interactively rather than running `dpkg-reconfigure`:
   ```bash
   sudo tee /etc/apt/apt.conf.d/20auto-upgrades >/dev/null <<'EOF'
   APT::Periodic::Update-Package-Lists "1";
   APT::Periodic::Unattended-Upgrade "1";
   APT::Periodic::AutocleanInterval "7";
   EOF
   ```
   Then patch `/etc/apt/apt.conf.d/50unattended-upgrades` to set `Unattended-Upgrade::Automatic-Reboot "true";`, `Unattended-Upgrade::Automatic-Reboot-Time "04:00";`, and `Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";`. See `references/checks.md` for the sed/awk patterns.

5. **Fail2ban.**
   ```bash
   sudo cp -n /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
   ```
   Write `/etc/fail2ban/jail.d/sshd.local` with the values from `references/checks.md` (bantime, findtime, maxretry, mode=aggressive). Then:
   ```bash
   sudo systemctl enable --now fail2ban
   sudo fail2ban-client status sshd
   ```

6. **Home permissions and sudoers.**
   ```bash
   sudo chmod 700 /home/<name>
   sudo visudo -c
   ```

Then Batch B — **only proceed if Batch A items 2 and 2b both succeeded and the operator has confirmed, from a second terminal logged in as the deploy user, that both `ssh` and `sudo` work in that session.** Without both confirmations, refuse to touch sshd; explain that disabling root login or password auth before key + sudo are verified in the deploy session will leave the operator unable to cancel the rollback if the root session dies during reload.

7. **sshd hardening with rollback.** Use the full protocol in `references/ssh-safety.md`. The short version:
   1. Write the proposed config to `/tmp/01-hardening.conf.new` and validate with `sudo sshd -t -f /tmp/01-hardening.conf.new` — abort on error.
   2. Back up any existing `/etc/ssh/sshd_config.d/01-hardening.conf` to `/tmp/01-hardening.conf.bak`.
   3. Schedule an auto-rollback in 5 minutes using a transient `systemd-run` unit that restores the backup (or deletes the new file if there was no backup) and restarts ssh.
   4. Move the new file into place and `sudo systemctl reload ssh` (reload, not restart — existing sessions stay alive either way, but reload is the documented hot-swap path).
   5. Tell the user, verbatim: "I have applied the SSH hardening. **Open a new terminal and run `ssh -p <port> <name>@<host>`** to confirm you can still log in. Reply once you see a shell prompt — I have 5 minutes before the config rolls back automatically."
   6. On confirmation, cancel the rollback unit.
   7. If the user reports they cannot log in, do not try to fix it — wait. The rollback will fire and restore working ssh.

## Phase 6 — Verify and report

After Batch A and (optionally) Batch B, re-run the audit phase. Show a tiny diff: "Before: 3 failing, 2 warnings. After: 0 failing, 1 warning." Surface any item that did not improve and explain why.

Finish with a short action list of things the user must do off-server (or on-server, but outside this skill's scope):

- Confirm their local SSH config has `Host <alias>\n  HostName <host>\n  User <name>\n  IdentityFile ~/.ssh/<key>` so future connections use the new user automatically.
- **Don't leave the deploy user as "NOPASSWD sudo + no password set" forever.** If Option A was chosen during step 2b, anyone who acquires the SSH private key gets instant root with no second factor. Run `sudo passwd <name>` to set a sudo password. Then either remove `/etc/sudoers.d/90-<name>` to require the password on every sudo (full second factor restored), or leave NOPASSWD for ergonomics — that is a trade-off the operator should make consciously. The one state to avoid is "non-root user with NOPASSWD *and* no password set": no second factor, and `su -` / any password-gated tool will not work either.
- **Ubuntu locks the root account by default.** With `PermitRootLogin no` and a locked root, the only way in is via the deploy user — which is exactly the desired state. If the operator ever genuinely needs `su - root` from deploy (rare), `sudo passwd root` sets a root password; otherwise leave root locked, because a locked root account combined with `PermitRootLogin no` is strictly better than any password-protected root.
- If a kernel was upgraded, schedule a reboot.
- Rotate any API keys or secrets that may have been pasted to the server during setup, and create per-project keys with spend limits in Anthropic / OpenAI / etc. consoles if the server runs LLM agents.

## Non-goals

- **Application-layer hardening** (nginx, docker, postgres, app configs) — out of scope for v1; see `references/checks.md` for what is *not* covered.
- **OS reinstall / image rebuild.** The skill assumes a running server you can SSH into.
- **IDS, SIEM, audit logging beyond fail2ban**, and full CIS-benchmark compliance — also out of scope.
- **Generating SSH keypairs on the server** — keys must be generated on the user's local machine; only the public half ever touches the server.