Hermes bots in NemoClaw sandboxes

You are operating the nemoclaw-hermes-swarm example. It runs several Hermes bots on one machine, each in its own OpenShell (NemoClaw) sandbox, with NeMo Relay tracing into an OpenTelemetry collector. Everything goes through one command, ./swarm, in the example's root directory.

The machine needs Docker, openshell, nemoclaw, and Hermes 0.21+. It can be a Linux host the user reaches over SSH, or the user's own Mac (Colima) or Linux box, where the bots appear in Desktop under "This device". An OpenAI-compatible inference endpoint must already exist; deploying a model is out of scope.

Two habits first

• Never openshell policy set. It replaces a sandbox's whole policy and silently drops its inference and teammate rules. Only ever add, and only through ./swarm (which uses nemoclaw <sandbox> policy-add).
• Login shell for every remote command. openshell, nemoclaw, and hermes live in ~/.local/bin, which non-login shells do not put on PATH. ssh host 'openshell ...' fails with command not found. Use ssh host 'bash -lc "cd ~/nemoclaw-hermes-swarm && ./swarm status"'.

From nothing to a working swarm

cp swarm.env.example swarm.env      # INFERENCE_BASE_URL, INFERENCE_MODEL
umask 077; printf '%s' '<key>' > ~/.secrets/inference.key
./swarm up

swarm up runs preflight, builds the image with Hermes baked in, starts the collector, creates every bot in BOTS, restores bots recorded by earlier swarm add commands, wires the mesh, and prints a status table. First run is 8 to 12 minutes, dominated by the image build and two sandbox creations. Re-running is idempotent and is also how you recover after a reboot: existing bots are restored, missing configured bots are created.

Then have the user restart Hermes Desktop; the roster is read at launch. For a remote host they first add it under Settings, Connections, Add connection, SSH. For their own machine there is nothing to add.

One bot at a time

./swarm add nemoclaw-analyst --soul souls/nemoclaw-researcher.md   # meshed to all others
./swarm add nemoclaw-analyst --soul ./my-role.md
./swarm rm nemoclaw-analyst --yes                 # sandbox, profile, key, peers
./swarm ls
./swarm status                                    # health ladder per bot
./swarm test                                      # configuration-aware live suite
./swarm traces nemoclaw-analyst                   # relay state + collector counters

add takes 3 to 4 minutes, so give the terminal tool a timeout of at least 600 seconds. New bots share the inference endpoint; there is no model load. If a tool call still times out mid-add, do not conclude failure: run ./swarm ls and ./swarm status and read the result. add refuses a name that is already tracked; use ./swarm up to restore or reconcile it, or remove it explicitly before changing its role.

./swarm fixes its own environment (real HOME from the passwd database, HERMES_HOME/HERMES_PROFILE cleared, ~/.local/bin on PATH), so it works from an agent's terminal tool, where HOME is rewritten to the profile's private directory. Do not wrap it in hermes -p. If a bot creation ever stops right after api port NNNN with exit 1 and no error, that is the symptom of an older copy without this fix: the key file was looked up under the wrong home.

Writing a role (SOUL)

The soul file is the bot's system prompt and matters more than any config. Give it a method rather than an identity. Two paragraphs that fixed observed failures:

Deliver a usable answer in THE SAME MESSAGE. Never ask the user to scope the
task and never promise to report back later. Nothing re-prompts you.

If a source is unreachable, report what you have and name the blocker in one
line. Three failed attempts with the same tool means the path is closed.

Never let a bot claim a fact a tool did not return this session. Without a reachable source, models produce plausible issue numbers and versions from training data.

swarm add appends a short Runtime section (you are a NemoClaw bot in sandbox X, here is what you can reach) and a Teammates section to whatever soul you give it.

Name bots nemoclaw-<role>. The name is the @handle in Desktop, the host profile, and the sandbox, so the roster reads as a fleet of NemoClaw bots.

To give one bot more network reach than the others, add policies/<bot>.yaml (a preset; see policies/nemoclaw-researcher.yaml). swarm add applies it automatically when the file exists. Bots without a file get only the model endpoint, the collector, and their teammates.

For a local video that an owned, Ready nemoclaw-vss bot should analyze, the host operator must add it explicitly:

./swarm video-add /absolute/path/to/clip.mp4

Use the sanitized filename the command prints in chat. Chat text and Desktop attachments never select or copy host files. The VSS tools accept only a file already under /sandbox/videos, never a URL or host path. The operator command accepts one nonempty, regular, non-symlink .mp4, .webm, .mov, .mkv, or .avi no larger than 40 MiB.

Diagnosing "a bot is down"

In this order. Stop at the first failure.

1. Is the Desktop app running on the user's machine? pgrep -f "Hermes.app/Contents/MacOS/Hermes". Several bots erroring at once with no error text is nearly always the client having exited. Relaunch.
2. Does the bot answer directly? hermes -p <bot> chat -q "Reply with OK" on the host. A reply means the bot is fine and the fault is Desktop-side.
3. Is the model endpoint up? ./swarm doctor checks auth and that the model is listed. Since 0.21 a dead endpoint surfaces in Desktop as [reason: model_unavailable] rather than a generic error.
4. ./swarm status. Each rung is a real probe: sandbox phase, api_server 200, a chat turn through the sandbox, relay active, host profile running.
5. Restart the Desktop app. Every in-sandbox gateway restart invalidates the Desktop's backend, so swarm up after a reboot means the user restarts the app too. Never restart gateways right before a demonstration.

Failures that look like something else

• Bot works over HTTP but is missing from the roster. Two gateways per bot: one inside the sandbox (serves the api_server), one on the host (hermes -p <bot> gateway run) that makes the profile report running. The roster lists only the second. swarm up starts both; swarm status checks both.
• 403 vs 502 vs 000 from inside a sandbox. 403: policy denied it (host not allowed, or the calling binary is not listed; policies bind to both). 502: allowed, but nothing listening, usually a service on host loopback. 000 with CONNECT tunnel failed, response 403 in stderr: HTTPS to an unlisted host, also a denial. 127.0.0.1 inside a sandbox is the sandbox; cross to the host with host.openshell.internal.
• policy-add says Preset must declare preset.name. Files given to policy-add are presets: top-level preset: {name, description}, no version:. policies/otlp-export.yaml is the reference shape.
• message_teammate says "No API key available … HERMES_PEER_NEMOCLAW-X_KEY". An old copy of the teammates plugin; Hermes turns dashes into underscores in that variable name and the plugin now does too. swarm up reinstalls it.
• Relay "not active" but spans arrive. The activation line is logged at INFO to /sandbox/.hermes/logs/agent.log inside the sandbox, not gateway.log and not the stderr captured on the host. Trust the collector counters (./swarm traces <bot>); they are the only delivery signal.
• A repeated identical prompt returns instantly and no new session appears. The api_server dedupes identical requests through its response store. Use a unique token when probing.
• swarm rm printed nothing. It asked for confirmation on a stdin that was closed. Pass --yes when scripting.
• Removed bots reappear as stopped profiles a minute later. The Desktop's host backend (hermes serve --isolated) cached the profile list at start and its cron ticker recreates <profile>/cron/ every 60 s. Kill that process, remove its ~/.hermes/desktop-ssh/<hash>/, delete the ghosts, and have the user restart Desktop.
• A bot replies (pass). Correct. The room prompt says reply only with something new.
• A chained task stalls. @b research, @c summarise, @a brief deadlocks when @b never got the message. Check whether the first bot in the chain received anything.

Traps in your own diagnostics

• gateway.pid holds JSON, not a bare PID. Ask hermes profile list.
• Nested shell quoting mangles keys and yields 401s from healthy endpoints. Read keys from files inside a script; never interpolate them through ssh '… "… \"…\" …" …'. Same for python -c through sandbox exec: two shells eat the quotes. Use a heredoc on stdin.
• pkill -f <pattern> over SSH can match your own session and kill it. Patterns that include the sandbox name (sandbox exec -n nemoclaw-x) match the ssh command that contains them. Kill by PID or by a pattern the outer command cannot contain (--timeout 0).
• docker ps showing Up proves nothing. Probe the port.

Before claiming success

Run ./swarm test and quote its SUMMARY: N passed, 0 failed line along with the enabled sections. The total varies with the tracked bots and optional services. If something failed, name the check and the error rather than reporting a clean run. Do not trust a bot's self-description; it can recite its role while every tool is broken.

For a public change, ./swarm presubmit gates secrets, internal hostnames, SPDX headers, and shell syntax.

Deeper reading