run-skills
AGENTS.md5 starsChanged 13 days agoMITView on GitHub
Guidance for Codex and other coding agents — and, via the CLAUDE.md symlink, Claude Code — working in this repo. One agent skill, remote-run (Slurm clusters + GPU workstations; it replaced the hpc-run / ws-run pair), plus the bash harness it drives, shipped as the uv tool remote-run whose command rsk (src/remote_run/, standard library only, Python = 3.11) installs and updates it and is the agent's path for every read, launch and cancel; the README has the user-facing story, INSTALL.md…
Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.
Of the AGENTS.md files we read, the 33,315 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.
In this repositoryOne other file it gives its agents.
- remote-runSkill
The file
We split this AGENTS.md into the rules it gives an agent
4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 82
Every file copy goes to rs_xfer_host (the profile's T_XFER_HOST, a site fact with no conf key, else the ssh host), every ssh to rs_host: a …
- Very specific: line 283
Always in $PROJ: HF/torch caches, wandb dirs, logs/, ckpts/.
- Very specific: line 364
Never in this repo. Personal: ~/.config/run-skills/targets/<name>/{profile.sh,policy.sh,reference.md,tests.sh} (start from …
- Distinctive: line 384
Every Slurm / wandb / bash fact written into a doc or script comes with a reproduction or a test — never from memory.
≈ “Common” is an estimate: at least 1 in 5 of the 42,283 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AGENTS.md
Guidance for Codex and other coding agents — and, via the CLAUDE.md symlink, Claude Code — working in this repo.
What this repo is
One agent skill, remote-run (Slurm clusters + GPU workstations; it replaced the hpc-run / ws-run pair), plus the bash
harness it drives, shipped as the uv tool remote-run whose command rsk (src/remote_run/, standard library only, Python
= 3.11) installs and updates it and is the agent's path for every read, launch and cancel; the README has the user-facing story,
INSTALL.mdthe runbook an agent follows to install it for a user (shipped to the harness dir; tested by70-docs). MIT, public. Edit the clone only — the tool env,~/.agents/run-skillsand~/.agents/skills/remote-runare generated (uv tool install+rsk setup, which runsinstall.shfrom the wheel's copy of the harness). The reasons behind the constraints below:docs/DESIGN.md; target packs:docs/packs.md.
Commands
bash selftest.sh # the only test suite, offline (stubbed sbatch/sinfo/scontrol/squeue/ssh/uv/rsync): phase A =
# tests/NN-*.sh on fixture targets in a temp RS_USER_DIR; phase B = targets/*/tests.sh (+ the
# packs' and the overlay's tests{,.local}.sh). Prints "===== RESULT: N pass, M fail =====", exit M. Run after ANY edit.
bash install.sh # copy to ~/.agents/run-skills, ONE skill copy in ~/.agents/skills (+ ~/.claude/skills symlink), INDEX, hpc-run/ws-run uninstalled, forwarders, selftest -q (full log: ~/.agents/run-skills/selftest.log)
uv tool install --editable . && rsk setup # the same through rsk, from this clone (edits are live; re-run setup after them)
bash tests/pkg-wheel.sh # NOT in selftest (uv + network once): wheel contents, uv tool install from git, setup, update, rollbackrsk (src/remote_run/): cli.py; install.py (how it was installed: PEP 610 direct_url.json + uv-receipt.toml
→ update commands); packs.py; harness.py calls lib/common.sh / install.sh (targets, conf → remote dir) — never a
second implementation in Python; observe.py + status.py (rsk status / rsk logs, --grep = status.grep_body;
read-only, one ssh per target, discovery by WorkDir / logs/*.pid; a failed read is UNOBSERVABLE, never "no jobs" (a hung mount: within observe.FS_WAIT s); also the ledger's runs, unledgered jobs,
SUBMIT_UNKNOWN reconciled from the remote journal, and per Slurm target this user's jobs in other dirs, counted (other_dirs)); ledger.py + grants.py (the per-project ledger and the one authorisation
record type, below); launch.py (rsk launch), cancel.py (rsk cancel, --wait S = follow an accepted scancel to its observed end) — both end with ONE stdout
verdict line rsk launch|cancel: exit N (…)[ — the error] (cli.verdict_line), and their plan / submit / cancel ssh probe the project dir first (observe._DIRPROBE: a hung mount = exit 5, nothing run); watch.py (rsk watch = the lease around
bin/run-watch), adopt.py (rsk adopt), wrappers.py (rsk sync|bootstrap|pull|avail exec bin/run-*, no logic; sync_modes
= run-sync's grammar: --also / --root take the next word, --root alone is no deletion), legacy.py
(rsk doctor --legacy-refs); errors.py = the exit codes (0 ok 1 failed 2 usage 3 refused 4 needs user 5 unknown 6 conflict 7 lease
ended 8 placement). The wheel force-includes the harness under remote_run/_harness/ (pyproject.toml): a new top-level dir the
harness needs at run time goes there too, and tests/pkg-wheel.sh lists what must be in it (install.sh copies every dir the docs name under ~/.agents/run-skills: 70-docs). Unit tests: tests/py/
(run by tests/90-py.sh with an isolated interpreter, scratch HOME and RS_* dirs; python < 3.11 = SKIP in a clone, FAIL only packaged);
tests/62-rsk-launch.sh runs the real CLI end to end (plan → card → approve / grant / deny, drift, sweep, lost reply, cancel identity,
busy GPUs, the lease) against the fixture targets through an executing ssh stub, so the deployed submit.sh / launch.sh do the work.
Hard rules: never commit generated files (they exist only in installed copies); this repo is public and ships no
site at all — no cluster/host/site name, user id, account, group path or personal policy gate, not even in a
test's word list or a comment (sites live in target packs or the user's overlay; tests use fixtures). The selftest
greps the repo for the overlay's private forbidden-words patterns (the list itself stays out too); never
hardcode the check count.
Architecture
Three layers
skills/remote-run/SKILL.md(thin, both kinds; everything throughrsk: reads first,rsk launch/rsk cancelcheck the user's authorisation; a brief bash fallback for installs without rsk) +references/:INDEX.md(line 1root:= harness dir, one row per target with akindcolumn, Routing rules; a stub thatinstall.shregenerates withrs_gen_index),procedure-slurm.md/procedure-nohup.md,conf-keys.md(both confs),experiment-conventions.md(run classes, naming, wandb mapping, checkpoint/resume),project-setup.md(onboarding) — ONE copy each. The agent reads INDEX, then exactly onetargets/<t>/reference.md.- Local:
bin/run-sync,bin/run-pull,bin/run-bootstrap,bin/run-launch+lib/common.sh(sourced; never deployed);bin/run-avail(free GPUs on every target before an open placement: one parallel read-only ssh each, one deadline);bin/run-probe(INSTALL.md's discovery: ssh hosts, one read-only BatchMode probe, wandb key state;tests/85-probe.sh);bin/run-watch(batch watch of a run table: one read-only ssh per target per round, events, state inlogs/run-watch/— each line names its target: status/jobs carry over only on the same target, unknown submits and (every round) every fail-stop from any state file naming the row (a table keeps all of them;--releaseclears only its trigger's); a stop that holds the table later in the watch withdraws an unused--release(reason=release-withdrawn); row targets: exactrs_targetsnames, else exit 2; a table holding a NUL or a control character other than tab (CR too) is exit 2, as rsk refuses it (R7-7); acts only with--acton#@ ok-approved rows within#@ allow; one per project, lock → exit 6;--act/--approvewithoutRS_VIA_RSKwhile rsk is installed →use: rsk watch …, exit 2; exit 7 in every mode onceRS_WATCH_PARENT(rsk watch's pid) is gone or no longer its parent;tests/47-watch.sh). Prologue:RS_HOME=$(cd "$(dirname "$0")/.." && pwd);TARGET(aliasesCLUSTER/STATION) →rs_load_target. - Remote:
remote/env.sh,remote/bootstrap.sh,remote/status.sh(both kinds; status.sh = the agent's one bounded monitoring read),remote/submit.sh,remote/job.sbatch(slurm),remote/launch.sh(nohup), deployed byrun-syncwith the target'sprofile.sh(+T_NAME='<t>'andT_REF=rs_ref, the local reference path(s) that submit.sh/status.sh re-print, appended),policy.sh, the user'spolicy.local.shand, written last,MANIFEST(<sha256> <file>per deployed file). Remote scripts never sourcelib/;env.shandbootstrap.shcarry the same resolver (TARGET→$PROJ/.<t>if it exists else$PROJ/.ws→profile.sh→ conf →profile.shagain: the site'sT_*win).
Where each script runs
| local (cwd = project root) | on the target | |
| slurm | run-sync, run-pull, run-bootstrap (ssh wrapper) | <REMOTE_DIR>/.<t>/{API,env.sh,bootstrap.sh,status.sh,submit.sh,job.sbatch,profile.sh,MANIFEST[,policy.sh,policy.local.sh]}; submit = ssh host 'bash …/.<t>/submit.sh …' |
| nohup | run-sync, run-pull, run-bootstrap, run-launch (one ssh) | ~/<WS_REMOTE_DIR>/.ws/{API,env.sh,bootstrap.sh,status.sh,launch.sh,profile.sh,MANIFEST} |
Every file copy goes to rs_xfer_host (the profile's T_XFER_HOST, a site fact with no conf key, else the ssh
host), every ssh to rs_host: a transfer node's restricted shell takes rsync only, so run-pull never sshes.
An edit to remote/* reaches a project only after install.sh and a re-sync. Logs: slurm logs/<name>_<jobid>.out (line 1 rs-attempt=
before env.sh, header target= job= restart=N node=… attempt=, footer EXIT rc=N; submit.sh prints the path), nohup logs/<name>-<timestamp>.log + .pid.
status.sh <jobid|log> [ckpt-dir] (squeue/sacct or the PID, last 256 KiB → hints + last lines, last 64 MiB → harness start/exit markers,
~2 KB output) is the monitoring read. ckpt_entry is directory activity, never proof of resume; log reads are byte-bounded.
Targets: discovery, packs and overlay (lib/common.sh)
rs_roots (highest first) = $RS_USER_DIR/targets/ (default ~/.config/run-skills/targets) → each
$RS_PACKS_DIR/<pack>/targets/ (default $RS_USER_DIR/packs, glob order) → $RS_HOME/targets/ (templates only).
rs_targets = dirs with a profile.sh under any root, _* skipped, sorted unique; rs_target_dir = the first root
that has one. A pack (and $RS_USER_DIR) may add routing/remote-run.md, appended verbatim to the INDEX Routing section — else its
legacy routing/hpc-run.md + ws-run.md (never required to rename), each rule under a kind lead-in (rs_legacy_routing). A user dir with profile.sh replaces a pack's target; one
without extends it (policy.local.sh sourced by submit.sh after policy.sh, reference.local.md appended to
the INDEX row, tests.local.sh in phase B). rs_load_target unsets every T_*, sources the profile, sets
T_NAME T_TAG T_DIR T_LOCAL_DIR, validates T_KIND/T_HOST, refuses the name rs (.rs/ is the journal). Tag = rs_tag NAME (alnum only, uppercased:
ws-box → WSBOX) — no knob. profile.sh is POSIX data: single-quoted paths, only $HOME/$USER expand, on the
remote (rs_path). Consumers: sync excludes (.<name>/ + literal .ws/), INDEX, the linter, install, phase B.
Config precedence
.hpc.conf / .ws.conf are sourced as shell. Resolution, highest first: CLI flag → <TAG>_KEY → KEY →
default (ovr KEY [DEFAULT], eval-based). Prefixable: RS_PREFIXABLE_SLURM / RS_PREFIXABLE_NOHUP (+ the
profile's T_CONF_KEYS); DEFAULT_PARTITION/DEFAULT_GPU_TYPE are prefixed-only (generic ignored, profile
T_DEFAULT_* is the fallback); ENV_KIND ENV_HOOK BOOTSTRAP_HOOK UV_SYNC_ARGS WANDB_MODE EXTRA_EXCLUDES SYNC_ALSO
REMOTE_HOST DEFAULT_CLUSTER DEFAULT_STATION WS_REMOTE_DIR are generic-only; rs_lint_conf warns (not honoured)
on any other <TAG>_KEY line. Remote dir: slurm ovr REMOTE_DIR (tagged form mandatory when T_REMOTE_DIR_REQUIRED=1);
nohup <TAG>_REMOTE_DIR else WS_REMOTE_DIR — never guessed. Jobs go out with --export=ALL, so every local /
login-node reader unsets the keys it reads (generic + <TAG>_) before sourcing the conf (rs_load_conf KEY... in
bin/*, submit.sh's list); env.sh trusts that cleared environment. Extend those lists when adding a key. submit.sh sources the
conf (after profile.sh) in a child bash --noprofile --norc and imports only that list as NUL-separated data, never eval'd: a conf's function/alias/PATH never reaches the policy.
OUTPUT_ROOT (prefixable for both kinds; run-pull / run-sync unset it) = the project's own root OUTSIDE its dir, opt-in, never guessed:
rs_output_root (absolute or $HOME-relative, plain, not / / home / above it, never the project dir or inside / around it, compared
by rs_rpath: a path under the local $HOME = ~/…, an unquoted $HOME in the sourced conf) and rs_root_rel (env/harness state or a
credential dir in ANY component refused); read only by run-pull --root (paths relative to it) and run-sync --root PATH (only those
paths: the root listed first, -n too, then -azR -u -K to <root>/./: a missing root is never created; no ssh, no deploy, stamp or
.wandb_env; with -D / --also / --take-over exit 2); tests/48-output-root.sh.
The fail-closed sentinel — preserve this shape
remote/env.sh is sourced (bash or zsh, never set -e); its body is one function whose every failure is
_rs_fail MSG; return 1 (env.sh: … — refusing to continue) and whose last lines export RUN_ENV_READY=$PROJ plus
the kind's legacy sentinel (HPC_ENV_READY | WS_ENV_READY). Every consumer does unset RUN_ENV_READY HPC_ENV_READY
WS_ENV_READY CONDA_DEFAULT_ENV → source → require RUN_ENV_READY == $PROJ, then verifies the env materialised:
job.sbatch needs $UV_PROJECT_ENVIRONMENT/bin/python (uv venv missing; waived only for the payload bash $PROJ/.<t>/bootstrap.sh
= run-bootstrap --job) or CONDA_DEFAULT_ENV == ENV_NAME (ENV_KIND=none: nothing); launch.sh's env subshell needs python -ef $RS_VENV/bin/python (uv venv not active) or the conda env active;
bootstrap.sh (uv and none source env.sh, conda never does; uv additionally) needs UV_PROJECT_ENVIRONMENT under a root snapshotted
before sourcing ($PROJ, T_ENV_ROOT, VENV_DIR), so an ENV_HOOK (sourced last, may override anything) cannot
move the guard; a derived venv's owner file .run-skills-proj must match unless VENV_DIR is explicit. job.sbatch sources
the attempt's env.sh snapshot if present; env.sh/bootstrap.sh in a job with a valid RS_ATTEMPT and its dir read profile/conf/hook
from it (a missing snapshot or another job's receipt fails closed, never the live file; else RS_ATTEMPT is unset, NOTE). New early
exits in env.sh and new consumers keep this pattern and get a test.
The attempt journal (<PROJ>/.rs/attempts/<NONCE>/; submit.sh, launch.sh)
Each call mkdirs its dir (the claim: --rs-attempt NONCE, a separate word an old parser rejects with exit 2, else generated;
NONCE = [A-Za-z0-9][A-Za-z0-9._-]{0,63}, the set spelled out; taken = exit 75), keeps the caller's fds as 3/4, exec >>out
2>>err </dev/null (submit.sh then runs the rest, policy included, in a subshell with 3/4 closed), writes stage, the receipt (once the job exists, before a byte about it is printed) and, in the EXIT trap,
result (stage= rc= verdict=submitted|launched|refused|failed|unknown), THEN replays out/err (trap '' PIPE): a dead ssh
(SIGPIPE on the first write) loses nothing, the exit status is unchanged. Slurm adds the snapshots conf hook profile.sh env.sh;
/.rs/ is an anchored sync exclude, refused by run-pull; nothing deletes attempts. 6 = a live same-name job of the project.
api=3 = the line # rs-api=3 inside the deployed submit.sh / launch.sh: rsk's plan ssh reads it in the script it is
about to run, run-launch checks .ws/launch.sh before passing any --rs- word (else exit 2 does not speak rs-api=3);
remote/API (api=3, in MANIFEST) is still deployed but proves nothing (an older run-sync leaves it while rewriting the
script). Both pre-scan --rs-mode plan, --rs-expect <64 hex> and --rs-warnings <warnings_id: 64 hex> (with --rs-expect
only; submit.sh also --rs-chain) as separate words before -- — an older parser exits 2 (tested against git show
6df6e56: / 8c8129e: / b746431: submit.sh and launch.sh; rsk maps its unknown arg to needs: sync, exit 6); an unknown
--rs-*, and an --rs- word as an option's value, is exit 2 (submit.sh, launch.sh, run-launch, rsk launch). Plan mode runs
every check (FORCE=1: a -y gate prints its WARNING: and goes on; submit.sh probes the policy first, below; launch.sh
checks the env gate read-only — the venv's python executable, a conda env that activates in a throwaway shell, the ENV_HOOK
file, a valid ENV_KIND: else exit 1, no card — and creates nothing outside .rs/) and prints ONE @@plan v1 … @@end
block: api= kind= target=, the bound fields (rs_bound, one function; slurm nodes=K right after gres=, always; also env_hook=<path>:<sha256>, venv_dir,
uv_sync_args; slurm conf_extra = the conf's own SBATCH_EXTRA words, then flags = T_FLAGS (the policy flags, a %q
list), right before chain, the last; nohup launch_cfg=<path>:<sha256>,… — every launcher config / hostfile
rs_launchers read, sorted, <path>:unreadable, empty when none — then busy_ok=<id>:<uuid>:<user>,… sorted by id, or
unknown), info_* (envelope; nohup info_run_gpu=checked and, after info_prior, info_launchers=checked (the
config / container WARNINGs and the env -i / exec -c / APPTAINERENV_ refusals: an older card lacks them, and a grant
would cover it), which rsk requires — a block without either, or launch_cfg=, or a slurm one without flags=, is
needs: sync, a stored nohup plan without them fails --plan; info_prior=<n|unknown>, slurm after info_source_dirty, nohup after info_run_gpu: rs_prior, the run's earlier attempts
= journal receipts named the run id or <run id>--aNN, slurm + this WorkDir's squeue rows no receipt names, unknown when
squeue failed or a receipt is unreadable — rsk derives relaunch from n > 0 or no number, never needs: sync; slurm then
info_wandb_env=none|no|names|unreadable: does $PROJ/.wandb_env name WANDB_MODE — a smoke's conf disabled is sure only at none / no),
warnings= (the distinct WARNING: lines in out+err, not WARNING (lint)), warnings_avail= (the availability gate's
share), nohup warnings_gpu= (rs_launchers' share, LAUNCH_WARN; NF8), warnings_id= (sha256 of those lines, sorted, digit runs → #, + a silent policy gate's probe output, below),
bound_hash= (sha256 of the lines kind= … the last bound field, each + newline); it stops before the snapshots / the
wrapper (stage=plan rc=0 verdict=planned). --rs-expect H recomputes that hash and the
warnings_id right before the snapshots and sbatch (nohup: before the env gate's subshell): another hash, or a warnings_id
other than --rs-warnings (none given: no warning at all), prints the block, changed: <keys> (against the approved plan's
block in its own journal entry) and ERROR: … differs from the approved plan, exit 6 (stage=expect, refused) — -y never
passes a new warning, nor one swapped for another; under it a failed squeue is exit 1. --rs-chain adds
--dependency=singleton (another dependency: exit 2) and passes the duplicate guard. Every time limit sbatch could read
(-t, DEFAULT_TIME, each --time/-t of the policy's or SBATCH_EXTRA's words, a value word included) must be explicit (0
/ UNLIMITED / INFINITE / -1 / unreadable: exit 2) and within MaxTime; an SBATCH_EXTRA word that is no option and no option's
value, or a trailing value option, is exit 2 (sbatch's + getopt runs it as the script: tests/vgate bare_word=); the
lint reads the words as that scan does (--comment -t1: -t1 is a value, tests/vgate comment_value=). launch.sh: a GPU
with a compute process (--query-compute-apps, parsed without a temp file; unreadable = unknown) is never auto-picked (too
few free: exit 1), -d onto one is exit 6, an unknown state exit 1 — unless --gpu-busy-ok. RUN never picks the GPUs:
leading words (rs_gpuenv) — before its command, also after a leading env — that set or clear CUDA_VISIBLE_DEVICES
CUDA_DEVICE_ORDER NVIDIA_/HIP_/ROCR_VISIBLE_DEVICES, or an env option but -v --debug -C DIR -u OTHER (GNU env's rules:
tests/61, tests/vgate/local.sh env_facts=), and RUN's TEXT anywhere (rs_gputext: NAME=/NAME+= after a blank,
; | & ( ' " = or a backquote, -u NAME/--unset[= ]NAME, unset … NAME, export -n, declare/typeset +x,
APPTAINERENV_NAME=/SINGULARITYENV_NAME=; also with quotes and backslashes removed — a mention as data counts; and
emptying the whole environment: env -i/-vi/-0i/-/-- -/--i…, after a wrapper, env's other options or inside
-S, and bash's exec -c where a command starts — never exec -ac, exec -a -c or a subcommand's exec -c: tests/61
R6-6 and 'round-6 integration', tests/vgate/local.sh env_anywhere=) are exit 2 before the claim; seen only once
expanded (first command), the wrapper refuses to start the payload (rc 2). The conf is read (profile, then .ws.conf,
child bash --noprofile --norc, fds 3/4/9 closed) in every mode before rs_launchers (a plan / --rs-expect refuses one
that does not source, exit 2). rs_launchers: deepspeed / accelerate launch (and their aliases) with their own GPU
flags; accelerate's config gpu_ids (read-only: --config_file, else the default config under
$PROJ/.cache/huggingface; unreadable, > 64 KiB, relative after a cd, or a RUN naming HF_HOME warns too; no
--config_file beside an ENV_HOOK, ENV_KIND=conda (R8-4: conda activate applies the env's own variables) or a
conf / RUN / .wandb_env (NF7; read-only, never printed) naming HF_HOME / ACCELERATE_…: the config accelerate
reads is unknown); deepspeed with /job/hostfile
and no --hostfile; container runtimes (docker / podman / nerdctl --gpus --runtime --device nvidia… --privileged, a
bare -e NVIDIA_VISIBLE_DEVICES, nvidia-docker run; apptainer / singularity --cleanenv --containall); a tool that
starts the command without this launch's environment or elsewhere (container exec, compose, kubectl exec, tmux — any
alias or unique prefix tmux runs as a warned command (R8-6: a threshold table that tests/88 checks against this
machine's tmux) —, screen -dm, setpriv --reset-env, sudo without -E, doas, systemd-run, ssh, srun) → a
counted WARNING: naming it, what and landing, also counted as warnings_gpu= (NF8: no grant covers one, whatever its
kind; a block without it: unknown, uncovered — grants.gpu_warn_miss; spellings: tests/61 'R5-2' / 'R7-3'; facts:
tests/vgate/launchers.sh + containers.sh + envdrop.sh, plumbing tests/88); the configs / hostfiles it read are
bound (launch_cfg; R8-5: each file deepspeed's -H / --hostfile names too, resolved as --config_file). What the
payload's own code does is beyond any check. P7: run-launch without RS_VIA_RSK while rsk is installed (on PATH or
$(uv tool dir --bin)/rsk) prints use: rsk launch … and exits 2 (else it launches, with a NOTE; a Slurm target points
at rsk launch) — a nudge, not a guard; tests/lib.sh exports RS_VIA_RSK.
Authorisation (grants.py, ledger.py; used by launch.py, cancel.py; docs/DESIGN.md "Authorization")
The user's policy: never ask again for what they already allowed, but check every action against it. Every launch, relaunch,
chain and cancel calls grants.check() first — a deny → 3, a covering record → act and record the use, else
needs user: → 4. rsk stores no user words: --approve is a flag (an optional value is accepted and ignored, as is
cancel's old question flag), rsk grant … --quote is optional, and audit events carry no words. One record type in the ledger's grants.json (<state>/projects/<sha256(realpath)[:16]>/, flock, atomic
writes with .prev, single-use plans/, append-only audit.jsonl): approval (the plan ids + bound hashes of the cards
the user approved, claimed and recorded in one lock hold, expiring with the plans (24 h) — a sweep is one record),
grant (a standing predicate: scope + actions + max_live / max_launches / expires, all three stated, none = the
user set none; resubmit is refused in a new one; allow_warnings = availability (warnings == warnings_avail) | any,
a stored true = any; launch covers relaunches too, as scope.relaunch_after: any did — still accepted, now
implied; cancel_own stays valid but is not needed and counts nothing), deny (always wins;
one of launch stops relaunch/chain too; its name globs also match the run id,
and a glob ending in an attempt's --aNN (2+ digits) or in a glob that can match one (x--a*) stands for its run —
every attempt, cancels and rsk watch rows included (grants.run_glob(s), R7-1; a NOTE at rsk grant deny|import);
a fact it scopes that is unknown matches: a cancel of a job without a plan, an rsk watch row — no bound hash, its GPU type
and partition never certain —, a run with an attempt recorded without a class; targets compared as grants.target_key
names — fx/, ./fx, FX are fx, a spelling that is not one plain name matches every target deny; scope.jobs only
with cancel actions, else invalid; rsk grant add|deny|import refuse a target rs_targets does not list, exit 2
unknown target, and a partition no scoped Slurm target's T_PARTITIONS lists, unless --unlisted-partition (import:
"unlisted_ok": true, not stored) — then a NOTE (R8-3, grants.unlisted_partitions, grants.check_scope(…,
unlisted_ok); T_PARTITIONS is a display list, harness.split_partitions: commas and whitespace, policy.sh decides;
no list or only workstations: the NOTE only, grants.partition_note; a GPU type: a NOTE only); every list argument
split on commas and stripped, an empty or spaced item exit 2, --job ids / pids only (grants._split, R8-1), a stored
spaced item invalid (grants._spaced); rsk grant list, rsk status (R7-6) and rsk doctor (== grants ==, every
ledger, exit code unchanged) flag an active record naming an unknown TARGET (grants.scope_problems, --json
scope_problems: never matches), list / status NOTE an unlisted partition (grants.scope_notes, --json
scope_notes), and show a record validate() rejects as INVALID — --json {record, invalid, fail_closed}, never
described (R8-7); a deny naming a run
(scope.names) with relaunch/resubmit matches every launch-type action of it, launch included, on any target in its
scope (grants.run_deny; a launch grant right after it: FLAG). A deny validate() rejects, an id d-… or a missing /
damaged kind is a deny matching everything until fixed or revoked (a non-object: until removed by hand); any
other invalid record covers nothing; an unreadable grants.json (no good .prev) is exit 6 grants unreadable for launch,
cancel and watch --act/--approve, and while it is recovered or missing every deny in audit.jsonl still counts
(rsk grant recount restores those denies and the revocations; a lost grant or approval needs the user again). The action
is derived (launch.plan_actions): a run with attempts that may have run (not only NOT_SUBMITTED, not the plan's own) in
the ledger, or info_prior > 0, or no readable info_prior (launch.info_prior) → relaunch, chain=1 → chain; every one applies (a deny of any matches, a record must allow all). The class is the name's,
set only at the run's first attempt (none there: a later one is exit 2). A NEW run (no ledger attempt that may have run, info_prior
not > 0) named outside experiment-conventions §2's shape, or with an opaque word, gets a NOTE and a card line, launched as named
(launch.name_problems / name_notes; --free-name: the user's own name, unchecked); a smoke is exit 2 only when its command's env
words (launch.env_scan, GNU env's reading: a box line from its first word) or a Slurm conf (bound wandb_mode) leave the tracker on
for certain, else a card WARNING where rsk cannot tell (launch.smoke_problem: a box's conf, a compound line, an expansion, an env
option, a conf value an ENV_HOOK / conda / info_wandb_env may change); a formal launch with it on prints §4's wandb fields (launch.tracker_note). A covering grant also covers policy flags (flags=),
--same-name-ok and conf_extra words outside grants.EXTRA_ALLOW (each a card NOTE; policy words never count) — but never a conf_extra word that changes the
allocation the caps check (grants.alloc_words: --exclusive, --gres, -G/--gpus*, --cpus-per-*, -c, --mem*, --mincpus, -s/--oversubscribe, -O/--overcommit, -S/--core-spec, --thread-spec, --hint, --sockets-per-node/--cores-per-socket/--threads-per-core, -B/--extra-node-info, --tmp, --tres-per-task, in any spelling or abbreviation sbatch accepts, read with
getopt_long's option / value boundaries from grants.SBATCH_LONG / SBATCH_SHORT (Slurm's slurm_opt.c); licenses stay coverable; the card's
NO GRANT … changes the allocation the caps check), and a
relaunch under launch is judged as a launch whatever the end and the configuration — never while an earlier attempt
may be live (grants._any_misses: is_live on any target, or info_prior_here above the ledger's count there /
unreadable). Never grant-covered: warnings the grant's allow_warnings kind does not cover (-y is sent only for a card
that showed warnings, under a record that covers them), a workstation's warnings_gpu, a time over max_time_s in any
limit sbatch could read (0 / UNLIMITED = unlimited), under a relaunch-only grant a relaunch whose latest attempt that
ran did not end in INFRA_END (PREEMPTED NODE_FAIL BOOT_FAIL REBOOT) or that does not repeat that attempt's
configuration (a plan named after an older attempt is the next aNN), a workstation plan on a busy GPU;
rsk launch first reads every target holding a ledger-live attempt of the run (launch.refresh); --check-after S
judges each submission S s later (launch.check_after: ended otherwise = exit 1 + the digest);
rsk sync -D (run-sync's grammar: --also -n, --root -n are paths) / rsk bootstrap --job (any --job word) /
rsk adopt --not-submitted need --approve (audited); rsk cancel of a live attempt in this project's ledger (launched or
adopted) needs no grant or approval, anything else of this user's needs --approve for that command (else 4; nothing
stored), a deny of cancel still refuses; never at all: someone else's job or process. A provably unsent submit (refused / failed with no receipt, no attempt dir, run-launch
before its ssh; nohup: the journal decides first — a receipt with gate go is launched whatever the exit) is NOT_SUBMITTED:
the use is refunded and the plan's claim released (rsk launch --plan P retries it under the same record); a slurm
submit's ssh 255 with no receipt is SUBMIT_UNKNOWN whatever ssh's stderr says (a ProxyCommand's fallback route or a login
shell prints Connection refused on a session that ran: use kept, plan used). A cancel with
no @@start from an ssh that did not end with the remote's own status is exit 5 whatever ssh's stderr says (test_cancel / test_launch test_r6_4_…, tests/62 ssh.fallback); a
non-zero scancel is exit 5 CANCELLING?, its own line quoted (cancel.scancel_line: the one naming that job id,
else one naming none), unless scancel's own line for that id says Invalid job id / already completing or completed (scancel_moot:
exit 0, COMPLETING; the words: tests/vgate/slurm-remote.sh scancel_ended= / scancel_invalid=). run-watch keeps its own approvals (#@ ok, bash; rsk watch --approve hands it a fixed note as its words) — rsk watch adds the lease, checks every row against the
denies for every action run-watch could take (--act / --approve; a deny scoped by job ids only for a table whose #@ allow has fail-stop; and at each lease refresh: TERM, exit 3), refuses a table that is not plain text (a NUL or a control character but tab, CR included: exit 2 in every mode, before the lease, in run-watch's words; R7-7), sets
RS_VIA_RSK=1 RS_WATCH_PARENT=<its pid>, and refuses (6) beside an orphaned run-watch lock. A record is bookkeeping, not
proof of consent: an agent can pass any flag; rsk constrains only what goes through it.
Storage policy (env.sh, driven by T_*)
Always in $PROJ: HF/torch caches, wandb dirs, logs/, ckpts/. ENV_ROOT = T_ENV_ROOT or $PROJ holds the
venv ($PROJ/.venv$SUF | $ENV_ROOT/envs/<proj>$SUF | conf VENV_DIR), .cache/uv*$SUF, bin/ (uv binary) and,
when ENV_ROOT != $PROJ, small-file state under state/<proj>$SUF. T_TMPDIR: project → $PROJ/tmp; inherit
→ the preset $TMPDIR if writable else /tmp/$USER/run-skills$SUF. T_COMPILE_CACHE: project | tmp | path;
compile dirs are $SUF-suffixed except nv, not mkdir'd when in $TMPDIR. RAY_TMPDIR stays under /tmp.
PATH="$VENV/bin:$ENV_ROOT/bin:$PATH" for both kinds, UV_NO_SYNC=1. Exports TARGET RS_TARGET RS_KIND RS_CUDA_ARCH
RS_SUFFIX RS_SOURCE_COMMIT RS_ENV_ROOT RS_VENV (RS_SOURCE_COMMIT from the .run-source stamp each run-sync writes, which
also makes run-sync refuse a second local checkout of the same remote dir without --take-over) and CLUSTER (slurm) xor WS_TAG (nohup, lowercase). T_SUFFIX is empty or hyphen-prefixed
(sync protects the literal glob /.venv-*). The protected-exclude list (rs_sync_excludes; run-pull's push-back
check reuses it) is static on purpose (hand-built .venv-foo dirs must survive -D); any dir env.sh creates under
$PROJ must be an anchored entry there.
policy.sh API (slurm; bash; sourced only by submit.sh)
t_partition NAME sets P_GPUS=min-max P_CPU_PER_GPU P_MEM_PER_GPU P_GTYPE P_GTYPE_DEFAULT P_PREEMPT P_MAXTIME P_CPU_ONLY
P_MEM_PER_CPU P_NOTE (submit.sh resets all to '' first), may SB_ARGS+=(…), returns 1 for unknown. t_policy (optional;
every target should define it so an overlay can wrap it as t_policy_shipped) runs right after arg parsing with PART
PART_FROM_CLI NGPU GTYPE GTYPE_FROM_CLI TIME FORCE ACK T_FLAGS SB_ARGS SB_EXTRA NAME PROJ CLUSTER TAG PLAN NODES visible
(PLAN=1: plan mode, with FORCE=1); its exit code propagates verbatim (3 = refusal; -y never unlocks it). In plan mode
and under --rs-expect submit.sh probes ( FORCE=0; t_policy ) (rs_probe: a subshell, output journaled and counted), then
runs the real one, and probes t_partition in its own FORCE=0 subshell right before its real call, with the PART that
is submitted (return 1 = an unknown partition, not a gate) — so both may run twice and must have no side effects. In the
probe every exit 1 is a passed -y gate and the function goes on, as in plan mode: a function's first gate hides none after
it, and code after a gate must not assume it stopped the run; any other exit is the verdict. A gate with no WARNING: line
gets the generic one, and then the probe's whole output, digit-masked, joins warnings_id. Gate idiom (the template,
docs/packs.md): echo 'WARNING: …' >&2; [ "$FORCE" = 1 ] || exit 1 — print, then test FORCE, so a plan's real call and a
-y submit show it too; never the P2 form [ "$FORCE" = 1 ] || { echo …; exit 1; }, silent under -y (tests/52 'R5-10'). A bash fact (70-docs, and
tests/vgate/slurm-remote.sh bash_exit_fn=): a function named exit outranks the builtin outside POSIX mode; under
set -o posix bash refuses it and the probe fails closed, exit 2. Unknown --word flags land in T_FLAGS (bound
flags=: a card NOTE, grant-coverable), --*-user-asked sets ACK=1. Then: (probe) partition → -g range → cpu/mem heuristics →
MaxTime (P_MAXTIME else T_MAXTIME; explicit limits only) → payload guard → SBATCH_EXTRA scan (-p, T_FORBIDDEN_QOS, a node / task layout word,
every word an option or its option's value, no trailing value option) → the policy's own SB_ARGS words: no node / task layout
word but one equal to the harness's (--nodes=K, --ntasks-per-node=1 for K > 1; getopt's clusters: -vN2 = -v -N2, as grants.layout_words), else exit 2 policy.sh sets the node/task layout; the harness's leading words rewritten: exit 2 → every --time/-t in SB_ARGS vs MaxTime → sinfo
availability (WARNING: only … Re-run -y; the nodes of SB_ARGS' last --exclude/-x not counted, a hostlist range: a NOTE) → duplicate guard (--same-name-ok, singleton chains and --rs-chain skip it) →
SBATCH_EXTRA lint on the scan's words (WARN: not in the allowlist + T_SBATCH_EXTRA_ALLOW, or a harness option) → plan
block / --rs-expect check → snapshots → sbatch (env RS_ATTEMPT, last --comment + rs=<NONCE>). Nothing
cluster-named lives in shared code; a policy's own EXIT trap, exit or fds stay in submit.sh's subshell, away from the journal.
Multi-node jobs (slurm; DESIGN "Multi-node jobs")
rsk launch --nodes K (1..64; a workstation: exit 2) sends submit.sh --nodes K only when K > 1 (old submit.sh: needs: sync);
-g -m -c per node; K > 1 adds --nodes=K --ntasks-per-node=1, exports RS_NODES / RS_SRUN_GRES; bound nodes= (absent = 1).
job.sbatch: MASTER_ADDR = line 1 of scontrol show hostnames (none: EXIT rc=2), MASTER_PORT, NNODES, then one srun
--nodes=K --ntasks-per-node=1 --kill-on-bad-exit=1 --gres=… --export=ALL bash -c '…NODE_RANK=$SLURM_NODEID…' rs-node <RUN> & in
the usual wait loop, USR1 to srun (relayed), TERM once as scancel --signal=TERM $SLURM_JOB_ID (srun answers TERM with SIGKILL;
failed: TERM to srun, term=srun); K = 1 unchanged. SBATCH_EXTRA node / task layout words are exit 2 (never in grants.EXTRA_ALLOW), so are the policy's own (SB_ARGS+=; one equal
to the harness's passes); rsk: a layout word in a block's extra / conf_extra beside nodes= = needs: sync, nodes unknown (grants.layout_words); grants:
max_nodes (absent = 1) ≥ K, caps on totals; run-watch rows single-node. Facts: tests/vgate/multinode.sh.
Tests contract (tests/lib.sh)
Caller sets RS_HOME; $T is a scratch dir; helpers ok bad has hasnt nosub mk_proj sub with_path stub_sinfo
stub_fail stub_rsync last_attempt fulltmp fulltmp_ok (the sbatch stub prints ARG:<arg> lines and ENV:RS_ATTEMPT=;
fulltmp CMD… runs CMD with TMPDIR on a full tmpfs and BASH_COMPAT=42 in an unshare -rm namespace, fulltmp_ok
says whether this machine reproduces that — SKIP when not). Stubs live in $T/bin, on PATH only via sub/with_path/an explicit
PATH="$T/bin:$PATH", and dispatch on literal strings (ssh answers nvidia-smi --query-gpu=…, logs anything
else as a launch; sinfo returns lines registered with stub_sinfo PATTERN LINE) — a reworded remote command or
a new partition needs a stub case. mk_proj TARGET CONF fakes a deploy from rs_deploy_files (absolute paths);
sub drops stderr: assert rejections by exit status. Phase A = fixture targets only; per-target checks go in tests.sh.
One skill copy for Claude Code, Codex and pi
install.sh installs the one skill into $RS_SKILLS_ROOT (default ~/.agents/skills/remote-run/, scanned natively by Codex 0.149+
and pi), symlinks ~/.claude/skills/remote-run to it, and moves a copy under ~/.codex/skills / ~/.pi/agent/skills to <root>.pre-v2/
(listed twice otherwise). The replaced hpc-run / ws-run leave all four roots: a link into the shared root, a dangling link or a
copy install.sh generated in the shared root (references/INDEX.md line 1 root:) is removed, anything else moved to <root>.pre-v2/ (tests/75-install.sh); rsk doctor FAILs while one is discoverable. One SKILL.md: allowed-tools: (Claude; accepted-and-ignored elsewhere) + codex/remote-run.openai.yaml
as agents/openai.yaml (Codex UI metadata). No sed over scripts. The skill reaches the harness through INDEX's root: line; its description is < 1024 chars with every target's T_TRIGGERS prepended (install prepends missing ones; ship it < 870), no <>.
Portability
Sourced files (remote/env.sh, every profile.sh) are zsh-safe: no arrays, [[ ]], ${var^^}, ${!var}, BASH_SOURCE.
Local scripts run on macOS bash 3.2 / BSD sed: no ${var^^} (use tr), declare -A, readarray, sed -i. Remote scripts
run under a bare non-login ssh host cmd: no conda/uv on PATH, $USER possibly unset (RS_USER=${USER:-$(id -un)}).
job.sbatch runs in tests without Slurm: ${SLURM_*:-} reads, "$@" & … wait shape. Remote parsing uses no here-string or
heredoc (up to bash 5.0 they are temp files in $TMPDIR, and a full one fails silently): split by parameter expansion, fail
closed — status.sh, submit.sh, launch.sh (its run wrapper is a single-quoted literal written with printf: never a
single quote inside it) and run-watch's remote bodies; fulltmp tests it. Local parsing (run-watch's, run-probe) may still
use them.
Adding a target
Never in this repo. Personal: ~/.config/run-skills/targets/<name>/{profile.sh,policy.sh,reference.md,tests.sh}
(start from targets/_template-*). Shared by a group: the same files under a target pack's targets/<name>/
(skeleton: templates/pack/, installer tested by tests/80-pack.sh). The name is the tag, the deploy dir, the
sync exclude and the INDEX row — nothing else needs editing. Extending a pack's target: policy.local.sh /
reference.local.md / tests.local.sh in an overlay dir without profile.sh. References are self-contained, <user>-neutral, never
mention another target (cross-target routing lives only in INDEX's Routing section).
Editing the docs
The rails in SKILL.md / procedure-*.md / experiment-conventions.md (check the existing authorisation before every
submit/launch/cancel and ask only outside it — the user's table in docs/DESIGN.md; cancel only after the identity check, never
someone else's job or process; a grant records what the user allowed, never invented or widened, and is no proof of consent; never
create a conda env, never guess the remote dir, own wandb key only, smokes run WANDB_MODE=disabled, formal runs checkpoint +
auto-resume, monitoring = bounded reviews, adaptive cadence, explicit wake-up capability, detect-only — its exceptions are
run-watch --act on user-approved rows within their budget, rsk actions a user's grant covers and rsk cancel of the
project's own ledger runs) are the product — do not trim
them (70-docs pins them). Facts live
once: values in profile.sh, limits in policy.sh, prose in reference.md, keys in conf-keys.md. The INDEX stub
must equal rs_gen_index OUT ROOT output with no targets (test 70-docs): regenerate under bash into a temp file
(RS_USER_DIR=/nonexistent RS_PACKS_DIR=/nonexistent, root ~/.agents/run-skills; the generator copies Routing
from the stub) and move it over.
Every Slurm / wandb / bash fact written into a doc or script comes with a reproduction or a test — never from memory.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

