agentleFS
Gilles86Gilles86/gilles-claude-skills/skills/sciencecluster/SKILL.md

sciencecluster

Skill3 starsChanged 15 days agoMITView on GitHub

SLURM cluster (UZH sciencecluster) operational knowledge — including the cause and fix of tasks stuck in "user env retrieval failed requeued held" (any `--export` LIST, also `--export=ALL,VAR=x`, triggers it; pass variables as `VAR=x sbatch ...` instead) — submitting jobs, conda activation in SLURM context, GPU constraints, log conventions, common failure modes (cuInit race, partition contention, quota/TMPDIR), and cluster-only vs local code paths. Use this skill whenever the user is working with sciencecluster, SLURM (sbatch, squeue, scontrol, sacct), conda envs on the cluster, GPU jobs with `--gres`, the user mentions "the cluster", or the working directory contains a `slurm_jobs/` folder. Also use for diagnosing held / failed / zombie SLURM jobs, and for copying data from a departmental SMB share (NAS) straight onto the cluster.

  • Deletes or force-pushes
  • Installs packages
  • Commits and pushes

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 0 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Fix: do not pass an --export list: Put variables in the environment of the sbatch call.

    line 37
  2. Very specific:

    Put the per-job export TMPDIR=… line in the script.

    line 142
  3. Very specific:

    If you must keep conda run (matching an existing repo convention), it must be conda run --no-capture-output -n <env> python -u … — but know …

    line 222
  4. Very specific:

    When editing or adding any slurm_jobs/*.sh, check this — many older repo scripts still use bare conda run and will silently buffer.

    line 225
  5. Very specific:

    If you need module load …, switch the shebang to #!/bin/bash -l — see references/script_shell_setup.md for why plain #!/bin/bash silently …

    line 234

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

UZH sciencecluster — SLURM operational knowledge

Operational knowledge for the UZH sciencecluster. Cluster-wide constants (partition names, QoS values, login host, GPU types) are baked in verbatim. Per-user values (SLURM account, conda paths, project env names) are written as <angle-bracket> placeholders that the user/agent fills in from their private global config or the project's CLAUDE.md.

The body of this skill is short on purpose. Constants, golden rules that apply to every job, and a reference index — that's it. Topical playbooks (apptainer, GPU jobs, cuInit race, exit codes, partition swap, …) live in references/ and load on demand. If you hit a symptom that matches a reference title, read that file.

user env retrieval failed requeued held → you passed an --export list

Cause (verified on sciencecluster 2026-11-02, slurm 25.05.4, from the slurm source and test jobs). Any --export value other than exactly ALL (so --export=VAR=x, --export=ALL,VAR=x, --export=NONE) makes sbatch set SLURM_GET_USER_ENV=1 (src/sbatch/sbatch.c: only the literal string ALL skips it). Before launch, slurmd then runs su - $USER -c env on the node; if that fails the task is requeued held with this reason. On this cluster it fails intermittently on a few nodes and within 1–5 s, so it is a node-side error, not a timeout from slow rc files or launch load. Over 2026-10-15…11-02, all ~2 400 held allocations came from submits with an --export list and none from plain submits, including plain jobs that ran on the same bad nodes minutes apart. (An earlier note here said --export=ALL,VAR=x avoids the retrieval. It does not: the job's own variables override the retrieved ones, so PATH looks like the submit shell's even though retrieval ran.)

Fix: do not pass an --export list. Put variables in the environment of the sbatch call. The default (--export=ALL) passes them on and no retrieval happens:

# held-prone:  sbatch --export=ALL,MODEL=m1,KIND=bar job.sh
MODEL=m1 KIND=bar sbatch job.sh                  # same variables, no retrieval
ssh sciencecluster 'cd ~/git/<project> && MODEL=m1 sbatch job.sh'   # remote
MODEL=$m sb --array=1-30 job.sh                  # also through a bash function sb()

(Positional script arguments, sbatch job.sh m1 bar, also work.) To check, print ${SLURM_GET_USER_ENV:-unset} in the job script: 1 means retrieval ran.

One side effect. The retrieved login environment used to supply MODULEPATH and the module function. Without it, a job submitted from a non-interactive ssh 'sbatch …' has no module (verified). A script that calls module load needs #!/bin/bash -l (references/script_shell_setup.md).

Already held? scripts/heal_env_holds.sh [--dry-run] (login node; only squeue/sacct/scontrol) excludes the nodes the launches failed on and releases in waves of 25. By hand: scontrol update JobId=<id> ExcNodeList=<existing>,<node> (keep existing exclusions), then scontrol release <id>. The healer is a safety net for old submit lines and can run from scrontab (scripts/heal_env_holds.scrontab); note that every normal scrontab cycle shows up as REQUEUED in sacct. It is not the fix; the submit line is. Throttling (%N), an in-script sleep stagger, or pacing sbatch calls do not help: retrieval happens in slurmd before the script starts, and the failures are fast node-side errors, not a timeout under load.

Constants

ThingValue
SSH alias (typical)sciencecluster
Login hostcluster.s3it.uzh.ch
Partitionslowprio (fast dispatch, preemptable), standard (fairshare)
QoSesdebug (4 min, MaxJobsPU=1, +prio), normal (1 day, default), medium (2 day, standard only), long (7 day, MaxJobsPU=24)
GPU typesL4 (1/node, no cuInit race), A100 (8/node), H100/H200 (CUDA 12, exclude when env is on CUDA 11). No V100 any more (checked 2026-11-02 with sinfo -o "%f"): a constraint naming V100 fails with "Invalid feature specification".
Shared data root/shares/<account>/ (per-account; group-owned share)

Partition convention (not enforced by policy): reserve lowprio for jobs that are GPU-bound, or short and easily resumable. Multi-hour CPU jobs (fmriprep, GLMsingle, encoding fits) belong on standard + medium (≤2 d) or long (≤7 d). lowprio is preemptable, so long non-resumable CPU work invites rework when a higher-priority job claims the node mid-flight.

Per-user placeholders (substitute from the project's CLAUDE.md):

  • <account> — your SLURM account (e.g. zne.uzh)
  • <conda-base> — conda install root (e.g. ~/data/miniforge3)
  • <conda-envs> — envs dir (e.g. ~/data/conda/envs)
  • <env> — project conda env name (e.g. retsupp_cuda)
  • PyMC / pytensor jobs: point the compile cache at /scratch, per task. pytensor C-compiles the model graph into $HOME/.pytensor by default. An array of concurrent fits therefore (a) hammers one lock directory and (b) bursts thousands of small files into $HOME — observed to die with OSError: [Errno 122] Disk quota exceeded inside cmodule.py … compile_str when ~12 tasks compiled simultaneously (tms_risk lfxgrid, 2026-08-22), even though steady-state home usage was under quota. In every SLURM script that runs PyMC (also when sampling via numpyro/JAX — the logp graph and compute_log_likelihood still compile through pytensor):

    export PYTENSOR_FLAGS="base_compiledir=/scratch/$USER/pytensor/${SLURM_JOB_ID:-manual}_${SLURM_ARRAY_TASK_ID:-0}"
    export TMPDIR="/scratch/$USER/tmp/${SLURM_JOB_ID:-manual}_${SLURM_ARRAY_TASK_ID:-0}"
    mkdir -p "$TMPDIR"

    Both lines (the TMPDIR one is the general rule above — g++ writes its intermediates there). One deception specific to pytensor: it logs rewrite-stage compile failures as scary ERROR tracebacks but continues on a python fallback, so a job can look RUNNING while compiling nothing — treat any /tmp/cc* or compile error in a PyMC log as this bug even if the job is alive. Per-task compiledirs also eliminate compile-lock contention between array elements. Sweep occasionally (rm -rf /scratch/$USER/pytensor).

  • /tmp is node-local, not shared — and can be full. Two distinct failure modes. (1) A script written to /tmp on the login node is invisible to srun/sbatch on a compute node — the job dies with can't open file. Stage scripts/code under $HOME (git repos under ~/git/<project>); stage scratch data under /scratch/$USER. (2) $TMPDIR defaults to /tmp, and any node's /tmp can be full or quota'd — and everything honors $TMPDIR: compilers spawned by pytensor/JAX/numba (fatal error: error writing to /tmp/cc*.s: Disk quota exceeded, observed 2026-08-22), Python's tempfile, sort, ffmpeg, … So in every job script:

    export TMPDIR="/scratch/$USER/tmp/${SLURM_JOB_ID:-manual}_${SLURM_ARRAY_TASK_ID:-0}"
    mkdir -p "$TMPDIR"

    and as a backstop put export TMPDIR=/scratch/$USER/tmp in the cluster ~/.bash_profile — sbatch propagates the submission environment, so even scripts that forget the line inherit a safe temp dir. Sweep old dirs occasionally (rm -rf /scratch/$USER/tmp). Caveat (verified 2026-10-19): the backstop only works if the export sits ABOVE any "if not interactive, return" guard, and only for jobs submitted from an interactive login shell. sbatch run via ssh host 'sbatch …' (how agents submit) inherits a non-interactive env, so test jobs saw TMPDIR=/tmp. Put the per-job export TMPDIR=… line in the script; don't rely on the backstop.

Templates

Copy-pasteable SLURM scaffolding lives in references/:

  • array_cpu_template.sh — CPU array job (logging + conda activation)
  • array_gpu_template.sh — same, plus --gres=gpu:1, GPU constraint, cuInit-race defense
  • submit_chain.sh — orchestrator that wires per-subject afterok chains

Fill the placeholders before submitting.

Golden rules (apply to every job)

1. Never run compute on the login node. The login node is shared across all users; ulimit -u is 256, and even a "diagnostic" script that imports nilearn/TF or loads a large NIfTI is heavy enough to slow other people's ls and squeue and risk an admin kill. For diagnostics: pull data local and run on your laptop, or srun --pty for an interactive compute-node shell, or wrap in sbatch --time=00:10:00. Only pure stat queries (squeue, sacct, ls, head, wc -l, git pull) belong on the login node.

2. Always git pull on the cluster before submitting. Sync code via git — commit local, push, git pull on the cluster. Never rsync individual files into the cluster's repo working tree (skips commit history, breaks the always-pull invariant, silently misplaces files when paths don't match). If the cluster has uncommitted job-script tweaks, git stash → git pull → git stash pop.

# local
git add path/to/new_script.py path/to/new_job.sh
git commit -m "Add ..."
git push

# cluster
ssh sciencecluster 'cd ~/git/<project> && git stash && git pull && git stash pop'

3. Inside the SLURM script — shell + conda setup:

#!/bin/bash             # use #!/bin/bash -l if you need `module load …`
set -eo pipefail        # NOT set -u (conda activation aborts under -u)

# PREFERRED: env binary directly, under `exec` (no activation, streams
# live, and SLURM's SIGTERM on scancel/timeout hits python directly).
exec <conda-envs>/<env>/bin/python -u script.py

# Only if the env needs activate.d hooks (conda-CUDA/cuDNN, GDAL/PROJ, R):
source "<conda-base>/etc/profile.d/conda.sh"
conda activate <env>
exec python -u script.py     # still exec, so signals reach python

The non-obvious bits:

  • PYTHONUNBUFFERED=1 + python -u — no downside in batch contexts; without them logs only flush when buffers fill, which hides a stuck job for hours.
  • conda run -n <env> python BUFFERS ALL STDOUT until the process exits — python -u and PYTHONUNBUFFERED=1 do NOT fix it, because conda run itself captures the child's pipe. Failure signature: the .txt log sits frozen at the first few lines for the entire run, so you cannot tell a working job from a hung one (sstat -j <id> --format=AveCPU will show CPU accruing — that's the only live signal). conda run also does not forward SIGTERM to the child — even with --no-capture-output — so on scancel/wall-time the python child is orphaned and any atexit/signal flush handler never runs (conda#10351, still open). Fixes, in order of preference:
    1. exec <conda-envs>/<env>/bin/python -u script.py — no activation, no wrapper, streams live, AND exec makes python the job-step process so SLURM's SIGTERM reaches it directly. The default for one-shot job scripts. Safe for pure-Python/numpy/TF/JAX envs; only envs with activate.d hooks (conda-CUDA/cuDNN, GDAL/PROJ, R) need source conda.sh && conda activate <env> && exec python -u … (check ls <env>/etc/conda/activate.d/ — empty ⇒ direct binary is equivalent).
    2. If you must keep conda run (matching an existing repo convention), it must be conda run --no-capture-output -n <env> python -u … — but know it still won't forward signals. When editing or adding any slurm_jobs/*.sh, check this — many older repo scripts still use bare conda run and will silently buffer. (micromamba/pixi run avoid the activation dance but their buffering / signal behavior is unverified — test before relying on them in batch.)
  • No set -u — conda activation scripts reference unset vars ($ADDR2LINE, $AR, $CC) and abort. Job dies in <5 s with FAILED 1:0, Elapsed=00:00:02.
  • The init_conda.sh indirection has occasionally failed inside SLURM jobs (unclear cause). Source conda.sh directly.
  • If you need module load …, switch the shebang to #!/bin/bash -l — see references/script_shell_setup.md for why plain #!/bin/bash silently breaks module and MODULEPATH.
  • Same trap applies to srun bash -c "...". That spawns a non-login non-interactive bash where module isn't defined, so the module load apptainer line fails silently (apptainer: command not found) and downstream commands abort. Use srun bash -lc "..." (login shell) instead. Verified on UZH sciencecluster 2026-05.

4. Keep --time tight. Fair-share favors tight walltime requests. A 24 h request waits longer than a 30-min one even if both run for 10 min. Set --time to ~2× the actual expected runtime; calibrate with sacct --format=JobID,JobName,Elapsed | grep <jobname>.

5. Always emit a visible progress signal on anything that runs

1 minute — a tail-able log line, a tqdm bar, pm.sample(progressbar=True), etc. — with PYTHONUNBUFFERED=1 so updates flush promptly. For long sweeps/loops, prefer writing each result row to the output file and flush()-ing as you go rather than collecting everything and writing once at the end: the growing file is a progress meter that survives stdout buffering AND a wall-time kill (an end-only write means a timeout loses the entire job's output). The row-count (wc -l) tells you exactly how far it got.

6. Orchestrate multi-stage pipelines with Snakemake, not hand-written afterok chains. See the snakemake skill for the SLURM-aware specifics (driver placement, plugin quirks, recovery after a crashed driver). This skill and that one are designed to be read together.

7. Never pass an --export list (not even --export=ALL,VAR=x): use VAR=x sbatch job.sh. Any list triggers --get-user-env on the node, and that is what lands tasks in user env retrieval failed requeued held (top of this file).

Building conda envs: sbatch the build, but no GPU node needed

For the modern pip-wheel CUDA stack (tensorflow[and-cuda], jax[cuda12], torch+cu*) you do not need --gres=gpu:1 to build the env — even for full CUDA stacks. Those wheels are precompiled and ship their own CUDA runtime via nvidia-cuda-runtime-cu* / nvidia-cudnn-cu* packages; the install is just wheel extraction, no compilation, no driver probe. CUDA initializes lazily at first device-use inside the running job.

Verified end-to-end on sciencecluster 2026-05-27: env built on a standard CPU node (no GPU visible, no nvidia-smi), then activated on a --gres=gpu:1 job — TF 2.20, JAX 0.10, and torch 2.12+cu130 all detected the L4 and ran matmul without any post-install fixup.

So: sbatch the build (login-node ulimit rule applies — conda + pip fork too many processes) to lowprio or standard with no --gres. Faster dispatch, and you don't burn a GPU slot just to unpack wheels. Legacy create_gpu_env.sh scripts on older projects (tms_risk etc.) that still ask for --gres=gpu:1 are carryover from the pre-bundled-wheels era — drop the --gres next time you touch them.

Cases where you would still need a GPU at build: pip install --no-binary :all: <pkg>, or custom CUDA-kernel compilation. Full operational details (paths convention, build sbatch template, solver notes) in references/conda_envs.md.

Reference index

Load these on demand — each is a focused playbook, not required reading.

Conda envs

  • references/conda_envs.md — paths convention, build sbatch template, why a GPU node isn't needed for the build, when it still would be, solver speed notes

Inside the SLURM script

  • references/script_shell_setup.md — #!/bin/bash -l, module, why .bashrc doesn't fix it
  • references/containers_apptainer.md — apptainer (not singularity); reusing FreeSurfer / etc. from sandbox containers via PATH
  • references/gpu_jobs.md — --gres=gpu:1, GPU constraints, GPU→CPU memory pitfall, cuInit race + flock warm-up
  • references/logs_and_naming.md — ~/logs/<jobname>_<jobid>.txt redirect, --job-name convention, scontrol update name= for arrays

Submission lifecycle

  • references/debug_qos.md — --qos=debug fast-lane smoke tests when fair-share has buried you
  • references/array_throttling.md — when NOT to throttle (usually); Reason=JobArrayTaskLimit is your own %N, not the cluster
  • references/idempotent_resubmit.md — per-(unit, stage) done() predicates; stale-chunk pitfall with chunk-NNNN-of-MMMM.npz
  • references/partition_swap.md — scontrol update Partition= on PD jobs (preserves JobIDs, keeps afterok graph intact)
  • references/cancel_and_zombies.md — cancel by JobID range, sweep DependencyNeverSatisfied zombies

Diagnosis

  • references/exit_codes.md — 0:125 (host RAM) vs 1:0 (GPU VRAM / Python raise) vs TIMEOUT

Getting data onto the cluster

  • references/smb_share_pull.md — pull straight from a departmental SMB/NAS share with smbclient + a user-made Kerberos ticket (kinit), instead of share → laptop → cluster; ticket lives on one login node only; read-only on append-only archives

Workstation hygiene

  • references/ssh_multiplexing.md — ControlMaster auto to dodge the login-host connection rate limit

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.