remote-cluster-agent
Skill43 starsChanged 7 months agoMITView on GitHub
Remote GPU cluster ops via `rca` CLI — run commands, transfer files, GPU/node inspection, mutagen sync. Trigger for any remote execution, training, GPU, cluster status, sync, or tail-log task even when 'cluster' isn't said. Triggers: cluster, GPU, rca, training, node, sync code, tail log, 集群, 节点, 远程执行, 同步代码, 跑训练, 看日志, GPU 占用.
- Installs packages
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- bibtex-fetchSkill
- context-auditSkill
- daily-summarySkill
- experiment-registrySkill
- handoffSkill
- init-projectSkill
- notion-lifeosSkill
- paper-analyzerSkill
- paper-self-reviewSkill
- project-reviewSkill
- research-leadSkill
- session-teamSkill
- sync-docsSkill
- team-digestSkill
- web-fetcherSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
16 rules. About 0 are common, 5 are distinctive, and 11 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
11
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Very specific: line 12
Check if
rcais installed: run which rca. - Very specific: line 13
If not found → run the first-time install flow (below), then continue to step 2.
- Very specific: line 14
Check daemon status: run rca daemon status.
- Very specific: line 15
If daemon request failed → run rca daemon start.
- Very specific: line 16
Check node connectivity: run rca nodes.
- Very specific: line 17
If the target node is dead → rca connect <node>.
- Distinctive: line 19
Run the user's command.
- Very specific: line 25
When which rca returns not found, execute in order:
- Very specific: line 126
If mutagen real-time sync is configured, the cluster's outputs/ appears locally automatically.
- Very specific: line 137
When rca nodes --check shows dead, or rca exec reports connect failed:
- Distinctive: line 147
When the user mentions "skill updated" or "redeploy", check mutagen session config:
- Very specific: line 154
Sync mode: should be one-way-replica.
- Distinctive: line 227
Read shared storage restrictions from your config and follow them strictly — files on shared paths often belong to other teams, there's …
- Distinctive: line 228
If the cluster has no public internet, use internal PyPI mirrors — don't attempt external URLs (GitHub, PyPI.org, etc.).
- Distinctive: line 229
Don't auto-push to master/main — avoid shipping unreviewed code to teammates.
- Very specific: line 231
Long-running processes must be backgrounded: if a process doesn't exit (inference server, training loop), use nohup ...
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
remote-cluster-agent
Local control plane for remote GPU clusters: a Go daemon + rca CLI. All commands go through the daemon's Unix socket to a cluster-side Python agent.py over persistent SSH, ~0.1s/command latency.
What to do when triggered
- Check if
rcais installed: runwhich rca.- If
not found→ run the first-time install flow (below), then continue to step 2.
- If
- Check daemon status: run
rca daemon status.- If
daemon request failed→ runrca daemon start.
- If
- Check node connectivity: run
rca nodes.- If the target node is dead →
rca connect <node>. Still dead → re-establish whatever SSH tunneling / VPN / jump host you depend on, then retry. - Second round still dead → read
rca daemon logs -ffor error detail.
- If the target node is dead →
- Run the user's command.
Don't write raw ssh commands, always use rca exec / rca batch.
First-time install (auto, no user action needed beyond SSH info)
When which rca returns not found, execute in order:
cd <skill_dir>
make install # builds rca into ~/go/bin (requires Go 1.21+)
rca config init # migrates legacy ~/.config/remote-cluster-agent/*.md, or generates blank templateThen ask the user for each node's SSH command, and write them into ~/.config/rca/config.toml under [nodes.*]. Example:
[nodes.train]
ssh = "ssh gpu-train"
[nodes.eval]
ssh = "ssh -p 2222 gpu-eval"
dir = "/home/user/project"
agent_path = "/shared/.agent/agent.py"Optional per-node overrides: dir (default working directory) and agent_path (where agent.py lives on that node). Both fall back to globals if unset.
Finally start and verify:
rca daemon start
rca daemon status # confirm running
rca nodes # confirm nodes connected
rca agent check # if agent missing → rca agent deployCommon operations
Single-node exec / heredoc
rca exec -n train "nvidia-smi"
rca exec -n train -d /home/user/project "git pull"
rca exec -t 600 -n train "python train.py"
# heredoc (bypasses shell escaping, recommended for multi-line or special chars)
rca exec --stdin -n train <<'EOF'
cd /home/user/project
python -c "import json; print(json.dumps({'k':'v \"q\"'}))"
EOFMulti-node parallel (batch)
rca batch "nvidia-smi | head -20"
rca batch -n train,eval "df -h /home"
rca batch --json "hostname" | jq -r '.results[] | "\(.node): \(.output)"'Node status / connection management
rca nodes # current connection state
rca nodes --check # deep ping (with latency)
rca nodes --health # latency history (monitor-tracked)
rca connect train # manually reconnect a dead node
rca disconnect train # actively close a node's connectionFile transfer (cp)
rca cp moves files through the agent's JSON-Lines channel (base64-encoded, 50 MB/file limit). Works on any SSH setup — no separate SCP/rsync path needed.
rca cp train:/home/user/logs/train.log ./
rca cp ./config.yaml train:/home/user/project/config.yaml
rca cp -r train:/home/user/checkpoints/exp07 ./local-copy/For very large files (multi-GB checkpoints), use shared filesystem or object storage — rca cp peaks around 2–3 MB/s.
Streaming / long tasks
rca exec -s -n train "tail -f /var/log/train.log"
rca exec -s -n train "python train.py"Cluster health inspection
Triggered when the user says "cluster status", "which node is free", "cluster health", "GPU usage". Use rca batch for parallel sampling:
rca nodes --check
rca batch "nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv"
rca batch "df -h /home | tail -1"
rca batch "uptime && tmux ls 2>/dev/null | wc -l"Summarize into a table and recommend the most idle node. See reference/cluster-health.md for detailed probe commands, parse rules, and report format.
Reading files vs running commands
rca exec is for executing commands (starting training, killing processes, checking process state, installing deps) — not reading file contents.
If mutagen real-time sync is configured, the cluster's outputs/ appears locally automatically. For logs, JSON results, CSV — use the native Read tool on the local path. That's ~20x faster than piping through the SSH connection.
Decision rule:
- Want to read file content (logs, JSON, CSV, config files) →
Read("outputs/...")locally - Want to run an operation (ps, nvidia-smi, start/stop, pip install) →
rca exec - Unsure if the file is synced locally →
ls outputs/orGlob("outputs/**/<pattern>")first
Typical mistake: rca exec -n train "cat /home/user/outputs/.../log.txt" to read a log. That file is already synced — just Read it.
SSH connection failures
When rca nodes --check shows dead, or rca exec reports connect failed:
- Try
rca connect <node>once — most transient issues resolve here. - Still dead? The SSH transport underneath is broken.
rcaitself doesn't manage tunnels / VPN / jump hosts — re-establish those through your usual workflow (ssh config, tunnel script, corporate VPN, etc.), then retry step 1. - Inspect daemon errors with
rca daemon logs -f.
Report failures grouped by cause — don't just say "all nodes failed". Which nodes recovered? Which need tunnel re-establishment? Which look like SSH config issues?
Mutagen session health
When the user mentions "skill updated" or "redeploy", check mutagen session config:
mutagen sync list 2>/dev/nullFor each session:
- Sync mode: should be
one-way-replica. Ifone-way-safeortwo-way-resolved, prompt to rebuild. - Ignore list:
.gitshould NOT be ignored (v0.4.0+ syncs.gitto keep cluster-sidegit statusclean).
Rebuild (only after user confirms):
mutagen sync list <name> --long # record current params first
mutagen sync terminate <name>
bash <skill_dir>/mutagen-setup.sh <ssh_host> <local_dir> <remote_dir> <session_name>Container restart: mutagen fails to reconnect
Symptom: mutagen sync list shows Beta Connected: No; mutagen sync resume reports server magic number incorrect.
Root cause: the cluster container restarted and lost ~/.mutagen/ (container home is ephemeral). Mutagen's agent binary and staging data live in a persistent path (typically mounted storage like /shared/.mutagen/).
Fix:
# Recreate the symlink on the cluster node
rca exec -n <node> "ln -sf /shared/.mutagen ~/.mutagen"
# Then resume locally
mutagen sync pause <session-name>
mutagen sync resume <session-name>Prevention: have the container init script create this symlink on startup.
See reference/mutagen-troubleshooting.md for more failure modes.
Config
Main config: ~/.config/rca/config.toml (edit via rca config edit).
First-time install flow is in "What to do when triggered → First-time install". Day-to-day, just keep the daemon running (rca daemon status).
Agent deployment / updates
rca agent check # show agent version per node
rca agent deploy # copy local agent.py (only where missing)
rca agent deploy --force # force overwriteNote:
rca agent deployuses direct SSH (bypasses the daemon) and iterates all nodes inconfig.tomlserially. Disconnected nodes will block for SSH's default connect timeout (minutes). Workaround — use a temp config with only live nodes:cp ~/.config/rca/config.toml /tmp/rca_deploy.toml # manually delete [nodes.*] sections for dead nodes rca --config /tmp/rca_deploy.toml agent deploy --force
Auto-start (optional, macOS)
rca daemon register # register launchd agent (requires App Management permission)Without this, start the daemon manually with rca daemon start whenever you reboot.
Troubleshooting
| Symptom | Fix |
daemon request failed | rca daemon start; if still failing rca daemon logs -f |
| Node status=dead | rca connect <node>; if still dead, re-establish SSH tunneling |
| tunnel down | Reconnect through your usual workflow (ssh config, VPN, tunnel script) |
| agent missing | rca agent deploy |
| Command special chars | Use rca exec --stdin <<'EOF' ... EOF |
| daemon crash | rca daemon start (launchd auto-restarts if registered) |
Safety boundaries
- Read shared storage restrictions from your config and follow them strictly — files on shared paths often belong to other teams, there's typically no recycle bin, and deletion can destroy days of someone else's work.
- If the cluster has no public internet, use internal PyPI mirrors — don't attempt external URLs (GitHub, PyPI.org, etc.).
- Don't auto-push to
master/main— avoid shipping unreviewed code to teammates. pkill -fmust use the bracket trick:pkill -f "[s]glang.launch_server"notpkill -f "sglang.launch_server"— because the SSH process command line contains the kill pattern,pkill -fwould match SSH itself and tear the connection down. Same applies topgrep -f,grepover process lists, etc.- Long-running processes must be backgrounded: if a process doesn't exit (inference server, training loop), use
nohup ... &ortmux new-session -d, withechoplaced after the background command using;:# Correct: nohup background + echo outside nohup python -m sglang.launch_server ... > /tmp/log 2>&1 & echo "PID=$!" # Correct: tmux detach + echo outside tmux new-session -d -s sglang "python -m sglang.launch_server ..."; echo "started" # Wrong: echo inside tmux's && chain, never executes tmux new-session -d -s sglang "python ... && echo started"
Architecture
The daemon runs independently of Claude Code (Unix socket), so every session shares one connection pool. The CLI is standalone too — usable in shells, scripts, or cron. Cluster-side agent.py speaks JSON-Lines v2.1.0 (streaming, cancel, batch, file transfer).
Upgrading from MCP v0.3.x:
remote_bash(node, cmd)→rca exec -n node "cmd",remote_bash_batch→rca batch. Legacy~/.config/remote-cluster-agent/*.mdmarkdown config auto-migrates onrca config init.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

