Lab HPC clusters & remote-compute workflow

Per-cluster detail is in references/: *-hpc.md for hosts, network and storage; *-slurm.md for partitions and submitting.

Before designing a large-data pipeline or starting a long data job, read and apply cluster-run, including on a held node.

Step 0 — before the first ssh

1. Run bash scripts/check-hpc-config.sh (--live also tests ssh). If the cluster the task needs is NOT CONFIGURED, refuse: name the missing aliases, point at the README's "Per-user config", never improvise with raw addresses or keys, and resolve hosts from ~/.ssh/config only. config dir: OLD LOCATION → refuse too, and relay the printed fix for the user to run.
2. Read ~/.claude/lab/personal.md — usernames, code dirs, the standing job, sbatch defaults. It overrides this skill. Say so if it is absent.
3. Ask which cluster unless context pins it down. Then develop locally, push to GitHub, and git pull on the remote — code reaches a cluster through git, not scp.

Shell and environment

• The local shell on lab Macs is zsh. Quote every URL (? and * are globs). An unquoted $var is not split into words, so use an array. End a probe that may find nothing with || true, so an empty result doesn't hide real errors.
• Run project code on a cluster through pixi run, not the system python3.

Work on a compute node, not the login node

Get onto a compute node and stay there. A login node is one shared machine per cluster, and a fan-out of "small" agent commands takes it down. ssh arc '<cmd>' is the wrong reflex even for a one-liner: the whole Slurm client (squeue, sbatch, scancel) runs on the compute node. Only two reasons justify a login node:

1. You hold no job, so no compute node is yours yet. Only a login node can submit that first one.
2. The compute node lacks what you need: on ircbc, internet and git. On arc, nothing: its compute nodes have internet and a full toolchain, so once you hold one, never leave.

SOP — take a foothold, then keep it

1. Find and reuse the node you already hold. The lab parks a long-lived idle GPU job on arc for this. Check personal.md, then probe aliases with ssh -o BatchMode=yes -o ConnectTimeout=10 <alias> true. Judge by exit status, not stderr.
2. Land and stay. On arc, wrap commands in a login shell — ssh <node> 'bash -lc "…"' — or pixi and other user-installed tools are invisible. Put each probe's commands into one ssh call: every call is a connection through the login node. If ssh hangs after a network drop, clear the stale shared connection with ssh -O exit <alias>, then retry once.
3. No node anywhere? Only then use the login node, and only to fix that: show the exact sbatch/salloc, get confirmation, submit, poll to RUNNING, ssh in, and leave. If a foothold is preempted, repeat this step.

Watching a job

Poll a job only with bash scripts/watch-job.sh <alias> <jobid> [file] in the background, never a sleep loop; -h lists its exit codes. Every job-watching subagent's brief names it.

Hard rules

• Work on a compute node; the login node has exactly the two uses above.
• No sbatch/srun outside an approved run plan. Approved before the pilot, sized for the full run, scripts shown: work folder, partitions, most CPU, memory and wall time per job, most jobs at once, target measure and range. Inside it, submit and cancel its own jobs unasked. A job or pilot estimate past a limit needs a revised plan. Cancelling a job the plan did not start, and writes to shared data or the shared package store, need a yes. Post it on the run's issue (or work folder README); quote it verbatim in subagent briefs. It ends with the run. No plan: a yes per script. Probes like squeue need none.
• Reachable never means permitted. ircbc lets you ssh to a node you hold no job on; working there steals from the scheduler. Hold an allocation.
• If ssh to ircbc hangs, stop and tell the user to check the VPN (atrust, managed by hand). arc is not behind it. Never retry in a loop.