hermes-service-persistence
Skill0 starsChanged 20 days agoMITView on GitHub
Use when Hermes services must survive a server reboot.
In this repository86 other files it gives its agents.
- ai-assistant-editor-setupSkill
- api-and-interface-designSkill
- banner-designSkill
- brainstormingSkill
- caveman-exploreSkill
- caveman-reviewSkill
- chat-silence-diagnosisSkill
- ci-cd-and-automationSkill
- code-auditSkill
- codebase-designSkill
- koordinator-team-dispatchSkill
- constraint-driven-developmentSkill
- context-engineeringSkill
- cpanel-shared-hosting-deploySkill
- cross-profile-cron-opsSkill
- deprecation-and-migrationSkill
- documentation-and-adrsSkill
- domain-modelingSkill
- doubt-driven-developmentSkill
- finishing-a-development-branchSkill
- frontend-designSkill
- git-workflow-and-versioningSkill
- graphic-design-laneSkill
- hermes-gateway-web-uiSkill
- hermes-profile-hygieneSkill
- hermes-profile-provisioningSkill
- hermes-stream-stale-diagnosisSkill
- hermes-team-dashboardSkill
- idea-refineSkill
- incremental-implementationSkill
- indonesian-official-lettersSkill
- internal-commsSkill
- interview-meSkill
- investigate-firstSkill
- kanban-rate-limit-resilienceSkill
- karpathy-guidelinesSkill
- large-project-gated-deliverySkill
- lean-buildSkill
- live-agent-office-operationsSkill
- live-react-runtime-qaSkill
- live-web-release-verificationSkill
- markdown-pdf-pipelineSkill
- marketing-strategySkill
- meta-social-publishingSkill
- meta-threads-oauth-lifecycleSkill
- migrationSkill
- multi-profile-project-orchestrationSkill
- network-operationsSkill
- observability-and-instrumentationSkill
- office-skill-distributionSkill
- organization-site-menu-researchSkill
- performance-optimizationSkill
- php-upload-size-limitsSkill
- planning-and-task-breakdownSkill
- ponytail-auditSkill
- ponytail-debtSkill
- ponytail-reviewSkill
- ponytailSkill
- portable-app-handoffSkill
- receiving-code-reviewSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 2 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 25
Before declaring a server production-ready.
- Very specific: line 60
Check: list-dependencies default.target lists both units, is-enabled prints enabled, Linger=yes.
- Distinctive: line 71
When the user authorizes a reboot test (often remote, wants proof):
- Very specific: line 108
Before firing: post an interim comment to every running card (workers are gateway children — they die too), and tell the user a short …
- Very specific: line 144
Remove empty hermes-worker-kanban-*.scope leftovers of finished cards (confirm zero processes first - see Pitfalls).
- Very specific: line 166
Check: adapters log a fresh reconnect, the crashing attribute has zero new hits in <profile>/logs/errors.log, and at least one real turn …
- Very specific: line 192
A worker scope may read
activewith empty MainPID: after its card is done (leftover cgroup, no process).
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Hermes service persistence across reboot
Make every surface the user depends on - gateway (Kanban dispatcher, cron ticker, messaging), dashboard/webbase, and hand-launched companion processes - come back automatically after power loss or reboot, with no SSH login and nothing left behind.
When to Use
- User asks for auto-start / services that must survive restart.
- A fleet machine has services started manually (shell/nohup/tmux).
- Before declaring a server production-ready.
Procedure (each step has a check)
1. Inventory: what runs, and how it was started
ps -eo pid,ppid,user,cmd | grep -E 'hermes' | grep -v grep
ss -tlnp | grep <port>
ls ~/.config/systemd/user/
systemctl --user is-enabled <unit> # exists != enabled
loginctl show-user <user> -p LingerClassify every component: systemd-managed / manual orphan / child of the gateway. Cron ticker, Kanban dispatcher, and worker scopes run inside the gateway unit - they need no unit of their own, only a healthy gateway. Units are needed only for gateway + dashboard.
2. Units
- Gateway: a unit usually already exists (
hermes-gateway.service) but may bedisabled. - Dashboard: often has NO unit - copy
templates/hermes-dashboard.serviceinto~/.config/systemd/user/, setExecStarttohermes dashboard --no-open,HERMES_HOME, PATH,Restart=always,After=network-online.target, thensystemctl --user daemon-reload.
3. Enable + linger (both required)
systemctl --user enable <unit> # symlink into default.target.wants
loginctl enable-linger <user> # user manager survives logoutCheck: list-dependencies default.target lists both units, is-enabled
prints enabled, Linger=yes.
4. Adopt live manual processes (zero-downtime handoff)
SIGTERM the manual PID, systemctl --user start <unit>, then prove the
handoff: health probe returns 200 and the port-owner PID equals
systemctl --user show <unit> -p MainPID.
5. Prove recovery with a DELAYED reboot (user authorizes link drop)
When the user authorizes a reboot test (often remote, wants proof):
hermes kanban list # FIRST: zero running/blocked cards
systemd-run --unit=autoreboot-test --on-active=120 /usr/sbin/rebootThe 120s delay lets your status message arrive before the link drops; a transient timer fires even though the gateway will die with the host. Then verify post-boot:
uptime -s # confirms a real reboot
systemctl --user is-enabled hermes-gateway.service hermes-dashboard.service
systemctl --user is-active hermes-gateway.service hermes-dashboard.service
systemctl --user show <unit> -p MainPID -p NRestarts # NRestarts=0 = clean start
curl -sS -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/
stat -c '%y' <profile>/cron/ticker_heartbeat # fresh = cron scheduler aliveThe user's next message arriving IS proof the gateway + messaging path works; report checklist results item-by-item, never a blanket "sudah normal". Empty-scope leftovers (hermes-worker-...scope) count as stale during recovery checks — see Pitfalls.
7b. Reload the credential pool from inside a session — delayed external restart
Pool state (auth.json) is held in the gateway's memory; editing it after
start does not reach the live process. To close that gap when a restart is
required AND the caller sits inside a session:
systemd-run --user --on-active=150s --unit=gw-reload-pool hermes -p default gateway restartThe delay lets the announcing message reach the user first; the transient
timer fires outside the blocked in-session path (the gateway dies 150s later,
not the caller's shell). Then verify with MainPID + ActiveEnterTimestamp
newer than the change, and treat the first cron tick after the restart as the
end-to-end proof — note the scheduler reschedules from the new start, so one
cron slot gets swallowed and self-heals at the next slot.
Before firing: post an interim comment to every running card (workers are
gateway children — they die too), and tell the user a short (≈15s) message gap
is expected. See kanban-rate-limit-resilience for the comment-first recipe.
Kalau gateway dijalankan manual (Running manually, not as a system
service), gateway restart adalah no-op — perintah kembali tanpa mengganti
PID, dan berulang kali meninggalkan gateway berstatus draining yang berhenti
menerima run baru. Jangan ulangi gateway restart sebagai obat; ganti prosesnya
dengan skrip yang dijadwalkan:
# pids dikumpulkan di luar skrip, bukan dengan pgrep di dalamnya
PIDS=$(ps -eo pid,cmd | awk '/gateway run/ && !/awk/ {print $1}' | tr '\n' ' ')
systemd-run --user --unit=gw-restart --on-active=90 \
--setenv=GATEWAY_PIDS="$PIDS" /bin/bash <profile>/scripts/restart_gateway.shIsi skrip: kill PID eksplisit dari $GATEWAY_PIDS → tunggu mati (poll kill -0)
→ HERMES_HOME=... setsid nohup hermes gateway run >>log 2>&1 & → ulangi maks 3×
→ verifikasi dengan hermes gateway status dan tulis hasil (PID baru / GAGAL) ke
file state supaya bisa dibaca setelah sesi pemanggil mati. Salinan siap pakai ada
di scripts/restart_gateway.sh pada skill ini — copy, lalu sesuaikan HERMES_BIN
dan HERMES_HOME_DIR.
- Jangan pakai pola
pgrep -f ... | killdi dalam skrip: guard Hermes memblokir kill berbasis nama proses (dianggap bisa membunuh proses pemanggil). Kumpulkan PID eksplisit daripsdan kirim lewat env. - Pastikan sesi ini tidak berjalan DI DALAM gateway sebelum kill: telusuri
ps -o ppid=,cmd= -p $$berulang — kalau ujungnya PID gateway, kill langsung memutus turn ini (itulah alasan penjadwalan delayed, bukan kehati-hatian). systemd-runexit 0 bukan bukti restart: timer yang terdaftar hanya berarti jadwal dibuat. Baca file state +gateway statussetelah timer menyala, dan anggap gateway sehat hanya kalau PID berubah.
6. Sweep leftovers, then verify
- Remove empty
hermes-worker-kanban-*.scopeleftovers of finished cards (confirm zero processes first - see Pitfalls). is-activeboth units; health HTTP 200; cronticker_heartbeatmtime ~ now across profiles; Kanban shows no stuck running/blocked cards.- Report proof level honestly: enabled + linger + active + healthy = configuration-verified; actual reboot recovery is only proven by a real reboot or an outside-shell restart.
7. After hermes update — restart before the next turn
Updates pull new code but leave the running gateway serving pre-update
modules (mixed sys.modules), so every newly added attribute crashes the
turn path until the process is replaced — AttributeError: '<Class>' object
has no attribute '<new_field>', surfaced to users as a generic "Something
went wrong" reply. The update command warns about this; treat the warning
as a mandatory step, not noise:
hermes -p <owner-profile> gateway restart # the profile that owns the gateway
hermes -p <owner-profile> gateway status # new MainPID, start time after the updateCheck: adapters log a fresh reconnect, the crashing attribute has zero new
hits in <profile>/logs/errors.log, and at least one real turn completes
(Turn ended in agent.log) before you report success.
Pitfalls
- Restart from the profile that owns the gateway, and trust only a changed
PID —
hermes gateway restartissued from a worker profile stops only that profile's lane while the multiplexer keeps serving stale code, and still prints a plausible-looking result; the owning profile (herehermes -p default gateway restart) is the one that actually replaces the process. Confirm a newMainPIDwhose start time postdates the change — never the command's exit status — and expect the in-flight turn to die, so approve only when nothing critical is mid-run. - Never restart/stop the gateway from a session running inside it —
systemd blocks it because SIGTERM propagation would kill the caller
mid-run; use the delayed
systemd-runrecipe in step 7b instead, or test restarts from an outside shell. - A unit file that exists is not a unit that starts -
disabledunits are skipped at boot; verify via thedefault.target.wantssymlink andis-enabled, not file presence. - Unbench cadence lives in
provider-endpoint-verification(hourlyno_agentauth resetwatchdog keyed to the provider's quota-reset interval) — see it before building one. - Units die at logout without linger -
Linger=yesis what lets user units run headless across boot; check it, never assume it. - HTTP 200 alone does not prove the unit serves the port - the old
manual process can keep holding it; compare port-owner PID with
MainPID. - A worker scope may read
activewith empty MainPID after its card isdone(leftover cgroup, no process); verify no process remains beforestop <scope>- stopping a scope with live children kills a worker. - To start a long-lived helper process, use the
terminaltool directly withbackground=true(tracked by Hermes) —hermes_tools.terminal()insideexecute_codehas no background mode, and shell wrappers (nohup/setsid) are hard-blocked, so the only working path is the terminal tool. Then run readiness checks (port bind, HTTP probe) in a separate call. Write the systemd unit only after the process is proven alive, and hand it off (pkillmanual PID →systemctl --user start) so exactly one owner holds the port. - Office 2D app units (
hermes3d.service,hermes3d-backend.service) are USER units too — baresystemctl restart hermes3dfails withUnit hermes3d.service not foundwhile the app keeps serving stale code from the pre-deploy build (a green HTTP 200 then misreads as successful deploy). Fix:XDG_RUNTIME_DIR=/run/user/0 systemctl --user restart hermes3d hermes3d-backend, then prove the new build served (BUILD_ID+ 200), never the exit status alone. Identify ownership first:ss -ltnp | grep ':3000 '→ owner PID →readlink /proc/<pid>/cwd+cat /proc/<pid>/cmdlinedistinguishes app server from gateway-managed processes. - Dashboard binds 127.0.0.1 by default; public webbase access keeps depending on the reverse proxy. Autostart is never a reason to open ports.
- Expose a loopback-only dashboard remotely with a private overlay, not a public bind: install Tailscale from its signed OS-repository instructions, enable/start
tailscaled, join the server to the owner's existing tailnet, then use Tailscale Serve to forward the private HTTPS endpoint tohttp://127.0.0.1:<port>. Never enable Funnel or open the app port in a public firewall for a private-access request. The full decision table and command recipe belongs in thehermes-gateway-web-uiskill (reference fileprivate-remote-access-mesh-vpn.md). - Treat Tailscale device authentication as a user approval gate. Start
tailscale up --hostname=<name>in a persistent process, give the owner the login URL, and wait for them to approve the server on their own account before configuring Serve. Do not try to bypass login with an auth key unless the owner explicitly provides/authorizes that method; verifytailscale statusandtailscale ip -4after approval. - Prove remote access layer-by-layer: confirm local backend response, confirm listener is loopback-only, confirm server and client show online in the same tailnet, configure Serve using the installed CLI's help, inspect
tailscale serve status, then have the user test the generated private URL from the client. Report access as pending until the user approval and peer test both complete.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

