agentleFS
VorElzeVorElze/hermes-office-skills/skills/hermes-service-persistence/SKILL.md

hermes-service-persistence

Skill0 starsChanged 20 days agoMITView on GitHub

Use when Hermes services must survive a server reboot.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 2 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Before declaring a server production-ready.

    line 25
  2. Very specific:

    Check: list-dependencies default.target lists both units, is-enabled prints enabled, Linger=yes.

    line 60
  3. Distinctive:

    When the user authorizes a reboot test (often remote, wants proof):

    line 71
  4. Very specific:

    Before firing: post an interim comment to every running card (workers are gateway children — they die too), and tell the user a short …

    line 108
  5. Very specific:

    Remove empty hermes-worker-kanban-*.scope leftovers of finished cards (confirm zero processes first - see Pitfalls).

    line 144
  6. Very specific:

    Check: adapters log a fresh reconnect, the crashing attribute has zero new hits in <profile>/logs/errors.log, and at least one real turn …

    line 166
  7. Very specific:

    A worker scope may read active with empty MainPID: after its card is done (leftover cgroup, no process).

    line 192

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Hermes service persistence across reboot

Make every surface the user depends on - gateway (Kanban dispatcher, cron ticker, messaging), dashboard/webbase, and hand-launched companion processes - come back automatically after power loss or reboot, with no SSH login and nothing left behind.

When to Use

  • User asks for auto-start / services that must survive restart.
  • A fleet machine has services started manually (shell/nohup/tmux).
  • Before declaring a server production-ready.

Procedure (each step has a check)

1. Inventory: what runs, and how it was started

ps -eo pid,ppid,user,cmd | grep -E 'hermes' | grep -v grep
ss -tlnp | grep <port>
ls ~/.config/systemd/user/
systemctl --user is-enabled <unit>   # exists != enabled
loginctl show-user <user> -p Linger

Classify every component: systemd-managed / manual orphan / child of the gateway. Cron ticker, Kanban dispatcher, and worker scopes run inside the gateway unit - they need no unit of their own, only a healthy gateway. Units are needed only for gateway + dashboard.

2. Units

  • Gateway: a unit usually already exists (hermes-gateway.service) but may be disabled.
  • Dashboard: often has NO unit - copy templates/hermes-dashboard.service into ~/.config/systemd/user/, set ExecStart to hermes dashboard --no-open, HERMES_HOME, PATH, Restart=always, After=network-online.target, then systemctl --user daemon-reload.

3. Enable + linger (both required)

systemctl --user enable <unit>    # symlink into default.target.wants
loginctl enable-linger <user>     # user manager survives logout

Check: list-dependencies default.target lists both units, is-enabled prints enabled, Linger=yes.

4. Adopt live manual processes (zero-downtime handoff)

SIGTERM the manual PID, systemctl --user start <unit>, then prove the handoff: health probe returns 200 and the port-owner PID equals systemctl --user show <unit> -p MainPID.

5. Prove recovery with a DELAYED reboot (user authorizes link drop)

When the user authorizes a reboot test (often remote, wants proof):

hermes kanban list                       # FIRST: zero running/blocked cards
systemd-run --unit=autoreboot-test --on-active=120 /usr/sbin/reboot

The 120s delay lets your status message arrive before the link drops; a transient timer fires even though the gateway will die with the host. Then verify post-boot:

uptime -s                                             # confirms a real reboot
systemctl --user is-enabled hermes-gateway.service hermes-dashboard.service
systemctl --user is-active  hermes-gateway.service hermes-dashboard.service
systemctl --user show <unit> -p MainPID -p NRestarts  # NRestarts=0 = clean start
curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/
stat -c '%y' <profile>/cron/ticker_heartbeat          # fresh = cron scheduler alive

The user's next message arriving IS proof the gateway + messaging path works; report checklist results item-by-item, never a blanket "sudah normal". Empty-scope leftovers (hermes-worker-...scope) count as stale during recovery checks — see Pitfalls.

7b. Reload the credential pool from inside a session — delayed external restart

Pool state (auth.json) is held in the gateway's memory; editing it after start does not reach the live process. To close that gap when a restart is required AND the caller sits inside a session:

systemd-run --user --on-active=150s --unit=gw-reload-pool hermes -p default gateway restart

The delay lets the announcing message reach the user first; the transient timer fires outside the blocked in-session path (the gateway dies 150s later, not the caller's shell). Then verify with MainPID + ActiveEnterTimestamp newer than the change, and treat the first cron tick after the restart as the end-to-end proof — note the scheduler reschedules from the new start, so one cron slot gets swallowed and self-heals at the next slot.

Before firing: post an interim comment to every running card (workers are gateway children — they die too), and tell the user a short (≈15s) message gap is expected. See kanban-rate-limit-resilience for the comment-first recipe.

Kalau gateway dijalankan manual (Running manually, not as a system service), gateway restart adalah no-op — perintah kembali tanpa mengganti PID, dan berulang kali meninggalkan gateway berstatus draining yang berhenti menerima run baru. Jangan ulangi gateway restart sebagai obat; ganti prosesnya dengan skrip yang dijadwalkan:

# pids dikumpulkan di luar skrip, bukan dengan pgrep di dalamnya
PIDS=$(ps -eo pid,cmd | awk '/gateway run/ && !/awk/ {print $1}' | tr '\n' ' ')
systemd-run --user --unit=gw-restart --on-active=90 \
  --setenv=GATEWAY_PIDS="$PIDS" /bin/bash <profile>/scripts/restart_gateway.sh

Isi skrip: kill PID eksplisit dari $GATEWAY_PIDS → tunggu mati (poll kill -0) → HERMES_HOME=... setsid nohup hermes gateway run >>log 2>&1 & → ulangi maks 3× → verifikasi dengan hermes gateway status dan tulis hasil (PID baru / GAGAL) ke file state supaya bisa dibaca setelah sesi pemanggil mati. Salinan siap pakai ada di scripts/restart_gateway.sh pada skill ini — copy, lalu sesuaikan HERMES_BIN dan HERMES_HOME_DIR.

  • Jangan pakai pola pgrep -f ... | kill di dalam skrip: guard Hermes memblokir kill berbasis nama proses (dianggap bisa membunuh proses pemanggil). Kumpulkan PID eksplisit dari ps dan kirim lewat env.
  • Pastikan sesi ini tidak berjalan DI DALAM gateway sebelum kill: telusuri ps -o ppid=,cmd= -p $$ berulang — kalau ujungnya PID gateway, kill langsung memutus turn ini (itulah alasan penjadwalan delayed, bukan kehati-hatian).
  • systemd-run exit 0 bukan bukti restart: timer yang terdaftar hanya berarti jadwal dibuat. Baca file state + gateway status setelah timer menyala, dan anggap gateway sehat hanya kalau PID berubah.

6. Sweep leftovers, then verify

  • Remove empty hermes-worker-kanban-*.scope leftovers of finished cards (confirm zero processes first - see Pitfalls).
  • is-active both units; health HTTP 200; cron ticker_heartbeat mtime ~ now across profiles; Kanban shows no stuck running/blocked cards.
  • Report proof level honestly: enabled + linger + active + healthy = configuration-verified; actual reboot recovery is only proven by a real reboot or an outside-shell restart.

7. After hermes update — restart before the next turn

Updates pull new code but leave the running gateway serving pre-update modules (mixed sys.modules), so every newly added attribute crashes the turn path until the process is replaced — AttributeError: '<Class>' object has no attribute '<new_field>', surfaced to users as a generic "Something went wrong" reply. The update command warns about this; treat the warning as a mandatory step, not noise:

hermes -p <owner-profile> gateway restart   # the profile that owns the gateway
hermes -p <owner-profile> gateway status     # new MainPID, start time after the update

Check: adapters log a fresh reconnect, the crashing attribute has zero new hits in <profile>/logs/errors.log, and at least one real turn completes (Turn ended in agent.log) before you report success.

Pitfalls

  • Restart from the profile that owns the gateway, and trust only a changed PID — hermes gateway restart issued from a worker profile stops only that profile's lane while the multiplexer keeps serving stale code, and still prints a plausible-looking result; the owning profile (here hermes -p default gateway restart) is the one that actually replaces the process. Confirm a new MainPID whose start time postdates the change — never the command's exit status — and expect the in-flight turn to die, so approve only when nothing critical is mid-run.
  • Never restart/stop the gateway from a session running inside it — systemd blocks it because SIGTERM propagation would kill the caller mid-run; use the delayed systemd-run recipe in step 7b instead, or test restarts from an outside shell.
  • A unit file that exists is not a unit that starts - disabled units are skipped at boot; verify via the default.target.wants symlink and is-enabled, not file presence.
  • Unbench cadence lives in provider-endpoint-verification (hourly no_agent auth reset watchdog keyed to the provider's quota-reset interval) — see it before building one.
  • Units die at logout without linger - Linger=yes is what lets user units run headless across boot; check it, never assume it.
  • HTTP 200 alone does not prove the unit serves the port - the old manual process can keep holding it; compare port-owner PID with MainPID.
  • A worker scope may read active with empty MainPID after its card is done (leftover cgroup, no process); verify no process remains before stop <scope> - stopping a scope with live children kills a worker.
  • To start a long-lived helper process, use the terminal tool directly with background=true (tracked by Hermes) — hermes_tools.terminal() inside execute_code has no background mode, and shell wrappers (nohup/setsid) are hard-blocked, so the only working path is the terminal tool. Then run readiness checks (port bind, HTTP probe) in a separate call. Write the systemd unit only after the process is proven alive, and hand it off (pkill manual PID → systemctl --user start) so exactly one owner holds the port.
  • Office 2D app units (hermes3d.service, hermes3d-backend.service) are USER units too — bare systemctl restart hermes3d fails with Unit hermes3d.service not found while the app keeps serving stale code from the pre-deploy build (a green HTTP 200 then misreads as successful deploy). Fix: XDG_RUNTIME_DIR=/run/user/0 systemctl --user restart hermes3d hermes3d-backend, then prove the new build served (BUILD_ID + 200), never the exit status alone. Identify ownership first: ss -ltnp | grep ':3000 ' → owner PID → readlink /proc/<pid>/cwd + cat /proc/<pid>/cmdline distinguishes app server from gateway-managed processes.
  • Dashboard binds 127.0.0.1 by default; public webbase access keeps depending on the reverse proxy. Autostart is never a reason to open ports.
  • Expose a loopback-only dashboard remotely with a private overlay, not a public bind: install Tailscale from its signed OS-repository instructions, enable/start tailscaled, join the server to the owner's existing tailnet, then use Tailscale Serve to forward the private HTTPS endpoint to http://127.0.0.1:<port>. Never enable Funnel or open the app port in a public firewall for a private-access request. The full decision table and command recipe belongs in the hermes-gateway-web-ui skill (reference file private-remote-access-mesh-vpn.md).
  • Treat Tailscale device authentication as a user approval gate. Start tailscale up --hostname=<name> in a persistent process, give the owner the login URL, and wait for them to approve the server on their own account before configuring Serve. Do not try to bypass login with an auth key unless the owner explicitly provides/authorizes that method; verify tailscale status and tailscale ip -4 after approval.
  • Prove remote access layer-by-layer: confirm local backend response, confirm listener is loopback-only, confirm server and client show online in the same tailnet, configure Serve using the installed CLI's help, inspect tailscale serve status, then have the user test the generated private URL from the client. Report access as pending until the user approval and peer test both complete.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.