agentleFS
jiahao-shao1jiahao-shao1/sjh-skills/skills/remote-cluster-agent/SKILL.md

remote-cluster-agent

Skill43 starsChanged 7 months agoMITView on GitHub

Remote GPU cluster ops via `rca` CLI — run commands, transfer files, GPU/node inspection, mutagen sync. Trigger for any remote execution, training, GPU, cluster status, sync, or tail-log task even when 'cluster' isn't said. Triggers: cluster, GPU, rca, training, node, sync code, tail log, 集群, 节点, 远程执行, 同步代码, 跑训练, 看日志, GPU 占用.

  • Installs packages

The file

Show its rules and text

We split this skill into the rules it gives an agent

16 rules. About 0 are common, 5 are distinctive, and 11 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 11

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  • could be automatic: a setting could enforce it
  1. Very specific:

    Check if rca is installed: run which rca.

    line 12
  2. Very specific:

    If not found → run the first-time install flow (below), then continue to step 2.

    line 13
  3. Very specific:

    Check daemon status: run rca daemon status.

    line 14
  4. Very specific:

    If daemon request failed → run rca daemon start.

    line 15
  5. Very specific:

    Check node connectivity: run rca nodes.

    line 16
  6. Very specific:

    If the target node is dead → rca connect <node>.

    line 17
  7. Distinctive:

    Run the user's command.

    line 19
  8. Very specific:

    When which rca returns not found, execute in order:

    line 25
  9. Very specific:

    If mutagen real-time sync is configured, the cluster's outputs/ appears locally automatically.

    line 126
  10. Very specific:

    When rca nodes --check shows dead, or rca exec reports connect failed:

    line 137
  11. Distinctive:

    When the user mentions "skill updated" or "redeploy", check mutagen session config:

    line 147
  12. Very specific:

    Sync mode: should be one-way-replica.

    line 154
  13. Distinctive:

    Read shared storage restrictions from your config and follow them strictly — files on shared paths often belong to other teams, there's …

    line 227
  14. Distinctive:

    If the cluster has no public internet, use internal PyPI mirrors — don't attempt external URLs (GitHub, PyPI.org, etc.).

    line 228
  15. Distinctive:

    Don't auto-push to master/main — avoid shipping unreviewed code to teammates.

    line 229
  16. Very specific:

    Long-running processes must be backgrounded: if a process doesn't exit (inference server, training loop), use nohup ...

    line 231

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

remote-cluster-agent

Local control plane for remote GPU clusters: a Go daemon + rca CLI. All commands go through the daemon's Unix socket to a cluster-side Python agent.py over persistent SSH, ~0.1s/command latency.

What to do when triggered

  1. Check if rca is installed: run which rca.
    • If not found → run the first-time install flow (below), then continue to step 2.
  2. Check daemon status: run rca daemon status.
    • If daemon request failed → run rca daemon start.
  3. Check node connectivity: run rca nodes.
    • If the target node is dead → rca connect <node>. Still dead → re-establish whatever SSH tunneling / VPN / jump host you depend on, then retry.
    • Second round still dead → read rca daemon logs -f for error detail.
  4. Run the user's command.

Don't write raw ssh commands, always use rca exec / rca batch.

First-time install (auto, no user action needed beyond SSH info)

When which rca returns not found, execute in order:

cd <skill_dir>
make install              # builds rca into ~/go/bin (requires Go 1.21+)
rca config init           # migrates legacy ~/.config/remote-cluster-agent/*.md, or generates blank template

Then ask the user for each node's SSH command, and write them into ~/.config/rca/config.toml under [nodes.*]. Example:

[nodes.train]
ssh = "ssh gpu-train"

[nodes.eval]
ssh = "ssh -p 2222 gpu-eval"
dir = "/home/user/project"
agent_path = "/shared/.agent/agent.py"

Optional per-node overrides: dir (default working directory) and agent_path (where agent.py lives on that node). Both fall back to globals if unset.

Finally start and verify:

rca daemon start
rca daemon status         # confirm running
rca nodes                 # confirm nodes connected
rca agent check           # if agent missing → rca agent deploy

Common operations

Single-node exec / heredoc

rca exec -n train "nvidia-smi"
rca exec -n train -d /home/user/project "git pull"
rca exec -t 600 -n train "python train.py"

# heredoc (bypasses shell escaping, recommended for multi-line or special chars)
rca exec --stdin -n train <<'EOF'
cd /home/user/project
python -c "import json; print(json.dumps({'k':'v \"q\"'}))"
EOF

Multi-node parallel (batch)

rca batch "nvidia-smi | head -20"
rca batch -n train,eval "df -h /home"
rca batch --json "hostname" | jq -r '.results[] | "\(.node): \(.output)"'

Node status / connection management

rca nodes                   # current connection state
rca nodes --check           # deep ping (with latency)
rca nodes --health          # latency history (monitor-tracked)
rca connect train           # manually reconnect a dead node
rca disconnect train        # actively close a node's connection

File transfer (cp)

rca cp moves files through the agent's JSON-Lines channel (base64-encoded, 50 MB/file limit). Works on any SSH setup — no separate SCP/rsync path needed.

rca cp train:/home/user/logs/train.log ./
rca cp ./config.yaml train:/home/user/project/config.yaml
rca cp -r train:/home/user/checkpoints/exp07 ./local-copy/

For very large files (multi-GB checkpoints), use shared filesystem or object storage — rca cp peaks around 2–3 MB/s.

Streaming / long tasks

rca exec -s -n train "tail -f /var/log/train.log"
rca exec -s -n train "python train.py"

Cluster health inspection

Triggered when the user says "cluster status", "which node is free", "cluster health", "GPU usage". Use rca batch for parallel sampling:

rca nodes --check
rca batch "nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv"
rca batch "df -h /home | tail -1"
rca batch "uptime && tmux ls 2>/dev/null | wc -l"

Summarize into a table and recommend the most idle node. See reference/cluster-health.md for detailed probe commands, parse rules, and report format.

Reading files vs running commands

rca exec is for executing commands (starting training, killing processes, checking process state, installing deps) — not reading file contents.

If mutagen real-time sync is configured, the cluster's outputs/ appears locally automatically. For logs, JSON results, CSV — use the native Read tool on the local path. That's ~20x faster than piping through the SSH connection.

Decision rule:

  • Want to read file content (logs, JSON, CSV, config files) → Read("outputs/...") locally
  • Want to run an operation (ps, nvidia-smi, start/stop, pip install) → rca exec
  • Unsure if the file is synced locally → ls outputs/ or Glob("outputs/**/<pattern>") first

Typical mistake: rca exec -n train "cat /home/user/outputs/.../log.txt" to read a log. That file is already synced — just Read it.

SSH connection failures

When rca nodes --check shows dead, or rca exec reports connect failed:

  1. Try rca connect <node> once — most transient issues resolve here.
  2. Still dead? The SSH transport underneath is broken. rca itself doesn't manage tunnels / VPN / jump hosts — re-establish those through your usual workflow (ssh config, tunnel script, corporate VPN, etc.), then retry step 1.
  3. Inspect daemon errors with rca daemon logs -f.

Report failures grouped by cause — don't just say "all nodes failed". Which nodes recovered? Which need tunnel re-establishment? Which look like SSH config issues?

Mutagen session health

When the user mentions "skill updated" or "redeploy", check mutagen session config:

mutagen sync list 2>/dev/null

For each session:

  1. Sync mode: should be one-way-replica. If one-way-safe or two-way-resolved, prompt to rebuild.
  2. Ignore list: .git should NOT be ignored (v0.4.0+ syncs .git to keep cluster-side git status clean).

Rebuild (only after user confirms):

mutagen sync list <name> --long    # record current params first
mutagen sync terminate <name>
bash <skill_dir>/mutagen-setup.sh <ssh_host> <local_dir> <remote_dir> <session_name>

Container restart: mutagen fails to reconnect

Symptom: mutagen sync list shows Beta Connected: No; mutagen sync resume reports server magic number incorrect.

Root cause: the cluster container restarted and lost ~/.mutagen/ (container home is ephemeral). Mutagen's agent binary and staging data live in a persistent path (typically mounted storage like /shared/.mutagen/).

Fix:

# Recreate the symlink on the cluster node
rca exec -n <node> "ln -sf /shared/.mutagen ~/.mutagen"

# Then resume locally
mutagen sync pause <session-name>
mutagen sync resume <session-name>

Prevention: have the container init script create this symlink on startup.

See reference/mutagen-troubleshooting.md for more failure modes.

Config

Main config: ~/.config/rca/config.toml (edit via rca config edit).

First-time install flow is in "What to do when triggered → First-time install". Day-to-day, just keep the daemon running (rca daemon status).

Agent deployment / updates

rca agent check               # show agent version per node
rca agent deploy              # copy local agent.py (only where missing)
rca agent deploy --force      # force overwrite

Note: rca agent deploy uses direct SSH (bypasses the daemon) and iterates all nodes in config.toml serially. Disconnected nodes will block for SSH's default connect timeout (minutes). Workaround — use a temp config with only live nodes:

cp ~/.config/rca/config.toml /tmp/rca_deploy.toml
# manually delete [nodes.*] sections for dead nodes
rca --config /tmp/rca_deploy.toml agent deploy --force

Auto-start (optional, macOS)

rca daemon register           # register launchd agent (requires App Management permission)

Without this, start the daemon manually with rca daemon start whenever you reboot.

Troubleshooting

SymptomFix
daemon request failedrca daemon start; if still failing rca daemon logs -f
Node status=deadrca connect <node>; if still dead, re-establish SSH tunneling
tunnel downReconnect through your usual workflow (ssh config, VPN, tunnel script)
agent missingrca agent deploy
Command special charsUse rca exec --stdin <<'EOF' ... EOF
daemon crashrca daemon start (launchd auto-restarts if registered)

Safety boundaries

  • Read shared storage restrictions from your config and follow them strictly — files on shared paths often belong to other teams, there's typically no recycle bin, and deletion can destroy days of someone else's work.
  • If the cluster has no public internet, use internal PyPI mirrors — don't attempt external URLs (GitHub, PyPI.org, etc.).
  • Don't auto-push to master/main — avoid shipping unreviewed code to teammates.
  • pkill -f must use the bracket trick: pkill -f "[s]glang.launch_server" not pkill -f "sglang.launch_server" — because the SSH process command line contains the kill pattern, pkill -f would match SSH itself and tear the connection down. Same applies to pgrep -f, grep over process lists, etc.
  • Long-running processes must be backgrounded: if a process doesn't exit (inference server, training loop), use nohup ... & or tmux new-session -d, with echo placed after the background command using ;:
    # Correct: nohup background + echo outside
    nohup python -m sglang.launch_server ... > /tmp/log 2>&1 & echo "PID=$!"
    # Correct: tmux detach + echo outside
    tmux new-session -d -s sglang "python -m sglang.launch_server ..."; echo "started"
    # Wrong: echo inside tmux's && chain, never executes
    tmux new-session -d -s sglang "python ... && echo started"

Architecture

The daemon runs independently of Claude Code (Unix socket), so every session shares one connection pool. The CLI is standalone too — usable in shells, scripts, or cron. Cluster-side agent.py speaks JSON-Lines v2.1.0 (streaming, cancel, batch, file transfer).

Upgrading from MCP v0.3.x: remote_bash(node, cmd) → rca exec -n node "cmd", remote_bash_batch → rca batch. Legacy ~/.config/remote-cluster-agent/*.md markdown config auto-migrates on rca config init.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.