sciencecluster
Skill3 starsChanged 15 days agoMITView on GitHub
SLURM cluster (UZH sciencecluster) operational knowledge — including the cause and fix of tasks stuck in "user env retrieval failed requeued held" (any `--export` LIST, also `--export=ALL,VAR=x`, triggers it; pass variables as `VAR=x sbatch ...` instead) — submitting jobs, conda activation in SLURM context, GPU constraints, log conventions, common failure modes (cuInit race, partition contention, quota/TMPDIR), and cluster-only vs local code paths. Use this skill whenever the user is working with sciencecluster, SLURM (sbatch, squeue, scontrol, sacct), conda envs on the cluster, GPU jobs with `--gres`, the user mentions "the cluster", or the working directory contains a `slurm_jobs/` folder. Also use for diagnosing held / failed / zombie SLURM jobs, and for copying data from a departmental SMB share (NAS) straight onto the cluster.
- Deletes or force-pushes
- Installs packages
- Commits and pushes
In this repository14 other files it gives its agents.
- bayesian-workflowSkill
- braincoderSkill
- cogneuro-projectSkill
- data-archivalSkill
- eyetracking-edfSkill
- finalize-projectSkill
- fmriprepSkill
- pycortexSkill
- sciencecloud-openstackSkill
- scientific-figuresSkill
- scientific-presentationsSkill
- scientific-writingSkill
- snakemakeSkill
- spm-glmSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 0 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 37
Fix: do not pass an
--exportlist: Put variables in the environment of the sbatch call. - Very specific: line 142
Put the per-job export TMPDIR=… line in the script.
- Very specific: line 222
If you must keep conda run (matching an existing repo convention), it must be conda run --no-capture-output -n <env> python -u … — but know …
- Very specific: line 225
When editing or adding any slurm_jobs/*.sh, check this — many older repo scripts still use bare conda run and will silently buffer.
- Very specific: line 234
If you need module load …, switch the shebang to #!/bin/bash -l — see references/script_shell_setup.md for why plain #!/bin/bash silently …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
UZH sciencecluster — SLURM operational knowledge
Operational knowledge for the UZH sciencecluster. Cluster-wide
constants (partition names, QoS values, login host, GPU types) are
baked in verbatim. Per-user values (SLURM account, conda paths,
project env names) are written as <angle-bracket> placeholders that
the user/agent fills in from their private global config or the
project's CLAUDE.md.
The body of this skill is short on purpose. Constants, golden
rules that apply to every job, and a reference index — that's it.
Topical playbooks (apptainer, GPU jobs, cuInit race, exit codes,
partition swap, …) live in references/ and load on demand. If you
hit a symptom that matches a reference title, read that file.
user env retrieval failed requeued held → you passed an --export list
Cause (verified on sciencecluster 2026-11-02, slurm 25.05.4, from the slurm
source and test jobs). Any --export value other than exactly ALL (so
--export=VAR=x, --export=ALL,VAR=x, --export=NONE) makes sbatch set
SLURM_GET_USER_ENV=1 (src/sbatch/sbatch.c: only the literal string ALL
skips it). Before launch, slurmd then runs su - $USER -c env on the node; if
that fails the task is requeued held with this reason. On this cluster it fails
intermittently on a few nodes and within 1–5 s, so it is a node-side error,
not a timeout from slow rc files or launch load. Over 2026-10-15…11-02,
all ~2 400 held allocations came from submits with an --export list and
none from plain submits, including plain jobs that ran on the same bad nodes
minutes apart. (An earlier note here said --export=ALL,VAR=x avoids the
retrieval. It does not: the job's own variables override the retrieved ones,
so PATH looks like the submit shell's even though retrieval ran.)
Fix: do not pass an --export list. Put variables in the environment of
the sbatch call. The default (--export=ALL) passes them on and no
retrieval happens:
# held-prone: sbatch --export=ALL,MODEL=m1,KIND=bar job.sh
MODEL=m1 KIND=bar sbatch job.sh # same variables, no retrieval
ssh sciencecluster 'cd ~/git/<project> && MODEL=m1 sbatch job.sh' # remote
MODEL=$m sb --array=1-30 job.sh # also through a bash function sb()(Positional script arguments, sbatch job.sh m1 bar, also work.) To check,
print ${SLURM_GET_USER_ENV:-unset} in the job script: 1 means retrieval ran.
One side effect. The retrieved login environment used to supply
MODULEPATH and the module function. Without it, a job submitted from a
non-interactive ssh 'sbatch …' has no module (verified). A script that
calls module load needs #!/bin/bash -l (references/script_shell_setup.md).
Already held? scripts/heal_env_holds.sh [--dry-run] (login node; only
squeue/sacct/scontrol) excludes the nodes the launches failed on and releases
in waves of 25. By hand: scontrol update JobId=<id>
ExcNodeList=<existing>,<node> (keep existing exclusions), then scontrol
release <id>. The healer is a safety net for old submit lines and can run
from scrontab (scripts/heal_env_holds.scrontab); note that every normal
scrontab cycle shows up as REQUEUED in sacct. It is not the fix; the
submit line is. Throttling (%N), an in-script sleep stagger, or pacing
sbatch calls do not help: retrieval happens in slurmd before the script
starts, and the failures are fast node-side errors, not a timeout under load.
Constants
| Thing | Value |
| SSH alias (typical) | sciencecluster |
| Login host | cluster.s3it.uzh.ch |
| Partitions | lowprio (fast dispatch, preemptable), standard (fairshare) |
| QoSes | debug (4 min, MaxJobsPU=1, +prio), normal (1 day, default), medium (2 day, standard only), long (7 day, MaxJobsPU=24) |
| GPU types | L4 (1/node, no cuInit race), A100 (8/node), H100/H200 (CUDA 12, exclude when env is on CUDA 11). No V100 any more (checked 2026-11-02 with sinfo -o "%f"): a constraint naming V100 fails with "Invalid feature specification". |
| Shared data root | /shares/<account>/ (per-account; group-owned share) |
Partition convention (not enforced by policy): reserve
lowpriofor jobs that are GPU-bound, or short and easily resumable. Multi-hour CPU jobs (fmriprep, GLMsingle, encoding fits) belong onstandard+medium(≤2 d) orlong(≤7 d).lowpriois preemptable, so long non-resumable CPU work invites rework when a higher-priority job claims the node mid-flight.
Per-user placeholders (substitute from the project's CLAUDE.md):
<account>— your SLURM account (e.g.zne.uzh)<conda-base>— conda install root (e.g.~/data/miniforge3)<conda-envs>— envs dir (e.g.~/data/conda/envs)<env>— project conda env name (e.g.retsupp_cuda)PyMC / pytensor jobs: point the compile cache at
/scratch, per task. pytensor C-compiles the model graph into$HOME/.pytensorby default. An array of concurrent fits therefore (a) hammers one lock directory and (b) bursts thousands of small files into$HOME— observed to die withOSError: [Errno 122] Disk quota exceededinsidecmodule.py … compile_strwhen ~12 tasks compiled simultaneously (tms_risk lfxgrid, 2026-08-22), even though steady-state home usage was under quota. In every SLURM script that runs PyMC (also when sampling via numpyro/JAX — the logp graph andcompute_log_likelihoodstill compile through pytensor):export PYTENSOR_FLAGS="base_compiledir=/scratch/$USER/pytensor/${SLURM_JOB_ID:-manual}_${SLURM_ARRAY_TASK_ID:-0}" export TMPDIR="/scratch/$USER/tmp/${SLURM_JOB_ID:-manual}_${SLURM_ARRAY_TASK_ID:-0}" mkdir -p "$TMPDIR"Both lines (the
TMPDIRone is the general rule above — g++ writes its intermediates there). One deception specific to pytensor: it logs rewrite-stage compile failures as scary ERROR tracebacks but continues on a python fallback, so a job can look RUNNING while compiling nothing — treat any/tmp/cc*or compile error in a PyMC log as this bug even if the job is alive. Per-task compiledirs also eliminate compile-lock contention between array elements. Sweep occasionally (rm -rf /scratch/$USER/pytensor)./tmpis node-local, not shared — and can be full. Two distinct failure modes. (1) A script written to/tmpon the login node is invisible tosrun/sbatchon a compute node — the job dies withcan't open file. Stage scripts/code under$HOME(git repos under~/git/<project>); stage scratch data under/scratch/$USER. (2)$TMPDIRdefaults to/tmp, and any node's/tmpcan be full or quota'd — and everything honors$TMPDIR: compilers spawned by pytensor/JAX/numba (fatal error: error writing to /tmp/cc*.s: Disk quota exceeded, observed 2026-08-22), Python'stempfile,sort, ffmpeg, … So in every job script:export TMPDIR="/scratch/$USER/tmp/${SLURM_JOB_ID:-manual}_${SLURM_ARRAY_TASK_ID:-0}" mkdir -p "$TMPDIR"and as a backstop put
export TMPDIR=/scratch/$USER/tmpin the cluster~/.bash_profile—sbatchpropagates the submission environment, so even scripts that forget the line inherit a safe temp dir. Sweep old dirs occasionally (rm -rf /scratch/$USER/tmp). Caveat (verified 2026-10-19): the backstop only works if theexportsits ABOVE any "if not interactive, return" guard, and only for jobs submitted from an interactive login shell.sbatchrun viassh host 'sbatch …'(how agents submit) inherits a non-interactive env, so test jobs sawTMPDIR=/tmp. Put the per-jobexport TMPDIR=…line in the script; don't rely on the backstop.
Templates
Copy-pasteable SLURM scaffolding lives in references/:
array_cpu_template.sh— CPU array job (logging + conda activation)array_gpu_template.sh— same, plus--gres=gpu:1, GPU constraint, cuInit-race defensesubmit_chain.sh— orchestrator that wires per-subjectafterokchains
Fill the placeholders before submitting.
Golden rules (apply to every job)
1. Never run compute on the login node. The login node is shared
across all users; ulimit -u is 256, and even a "diagnostic" script
that imports nilearn/TF or loads a large NIfTI is heavy enough to
slow other people's ls and squeue and risk an admin kill. For
diagnostics: pull data local and run on your laptop, or srun --pty
for an interactive compute-node shell, or wrap in
sbatch --time=00:10:00. Only pure stat queries (squeue, sacct,
ls, head, wc -l, git pull) belong on the login node.
2. Always git pull on the cluster before submitting. Sync code
via git — commit local, push, git pull on the cluster. Never
rsync individual files into the cluster's repo working tree (skips
commit history, breaks the always-pull invariant, silently misplaces
files when paths don't match). If the cluster has uncommitted
job-script tweaks, git stash → git pull → git stash pop.
# local
git add path/to/new_script.py path/to/new_job.sh
git commit -m "Add ..."
git push
# cluster
ssh sciencecluster 'cd ~/git/<project> && git stash && git pull && git stash pop'3. Inside the SLURM script — shell + conda setup:
#!/bin/bash # use #!/bin/bash -l if you need `module load …`
set -eo pipefail # NOT set -u (conda activation aborts under -u)
# PREFERRED: env binary directly, under `exec` (no activation, streams
# live, and SLURM's SIGTERM on scancel/timeout hits python directly).
exec <conda-envs>/<env>/bin/python -u script.py
# Only if the env needs activate.d hooks (conda-CUDA/cuDNN, GDAL/PROJ, R):
source "<conda-base>/etc/profile.d/conda.sh"
conda activate <env>
exec python -u script.py # still exec, so signals reach pythonThe non-obvious bits:
PYTHONUNBUFFERED=1+python -u— no downside in batch contexts; without them logs only flush when buffers fill, which hides a stuck job for hours.conda run -n <env> pythonBUFFERS ALL STDOUT until the process exits —python -uandPYTHONUNBUFFERED=1do NOT fix it, becauseconda runitself captures the child's pipe. Failure signature: the.txtlog sits frozen at the first few lines for the entire run, so you cannot tell a working job from a hung one (sstat -j <id> --format=AveCPUwill show CPU accruing — that's the only live signal).conda runalso does not forward SIGTERM to the child — even with--no-capture-output— so onscancel/wall-time the python child is orphaned and anyatexit/signal flush handler never runs (conda#10351, still open). Fixes, in order of preference:exec <conda-envs>/<env>/bin/python -u script.py— no activation, no wrapper, streams live, ANDexecmakes python the job-step process so SLURM's SIGTERM reaches it directly. The default for one-shot job scripts. Safe for pure-Python/numpy/TF/JAX envs; only envs withactivate.dhooks (conda-CUDA/cuDNN, GDAL/PROJ, R) needsource conda.sh && conda activate <env> && exec python -u …(checkls <env>/etc/conda/activate.d/— empty ⇒ direct binary is equivalent).- If you must keep
conda run(matching an existing repo convention), it must beconda run --no-capture-output -n <env> python -u …— but know it still won't forward signals. When editing or adding anyslurm_jobs/*.sh, check this — many older repo scripts still use bareconda runand will silently buffer. (micromamba/pixirunavoid the activation dance but their buffering / signal behavior is unverified — test before relying on them in batch.)
- No
set -u— conda activation scripts reference unset vars ($ADDR2LINE,$AR,$CC) and abort. Job dies in <5 s withFAILED 1:0,Elapsed=00:00:02. - The
init_conda.shindirection has occasionally failed inside SLURM jobs (unclear cause). Sourceconda.shdirectly. - If you need
module load …, switch the shebang to#!/bin/bash -l— seereferences/script_shell_setup.mdfor why plain#!/bin/bashsilently breaksmoduleandMODULEPATH. - Same trap applies to
srun bash -c "...". That spawns a non-login non-interactive bash wheremoduleisn't defined, so themodule load apptainerline fails silently (apptainer: command not found) and downstream commands abort. Usesrun bash -lc "..."(login shell) instead. Verified on UZH sciencecluster 2026-05.
4. Keep --time tight. Fair-share favors tight walltime requests.
A 24 h request waits longer than a 30-min one even if both run for
10 min. Set --time to ~2× the actual expected runtime; calibrate
with sacct --format=JobID,JobName,Elapsed | grep <jobname>.
5. Always emit a visible progress signal on anything that runs
1 minute — a tail-able log line, a
tqdmbar,pm.sample(progressbar=True), etc. — withPYTHONUNBUFFERED=1so updates flush promptly. For long sweeps/loops, prefer writing each result row to the output file andflush()-ing as you go rather than collecting everything and writing once at the end: the growing file is a progress meter that survives stdout buffering AND a wall-time kill (an end-only write means a timeout loses the entire job's output). The row-count (wc -l) tells you exactly how far it got.
6. Orchestrate multi-stage pipelines with Snakemake, not
hand-written afterok chains. See the snakemake skill for the
SLURM-aware specifics (driver placement, plugin quirks, recovery
after a crashed driver). This skill and that one are designed to be
read together.
7. Never pass an --export list (not even --export=ALL,VAR=x): use
VAR=x sbatch job.sh. Any list triggers --get-user-env on the node, and that
is what lands tasks in user env retrieval failed requeued held (top of this
file).
Building conda envs: sbatch the build, but no GPU node needed
For the modern pip-wheel CUDA stack
(tensorflow[and-cuda], jax[cuda12], torch+cu*) you do not
need --gres=gpu:1 to build the env — even for full CUDA stacks.
Those wheels are precompiled and ship their own CUDA runtime via
nvidia-cuda-runtime-cu* / nvidia-cudnn-cu* packages; the install
is just wheel extraction, no compilation, no driver probe. CUDA
initializes lazily at first device-use inside the running job.
Verified end-to-end on sciencecluster 2026-05-27: env built on a
standard CPU node (no GPU visible, no nvidia-smi), then activated
on a --gres=gpu:1 job — TF 2.20, JAX 0.10, and torch 2.12+cu130 all
detected the L4 and ran matmul without any post-install fixup.
So: sbatch the build (login-node ulimit rule applies — conda +
pip fork too many processes) to lowprio or standard with no
--gres. Faster dispatch, and you don't burn a GPU slot just to
unpack wheels. Legacy create_gpu_env.sh scripts on older projects
(tms_risk etc.) that still ask for --gres=gpu:1 are carryover
from the pre-bundled-wheels era — drop the --gres next time you
touch them.
Cases where you would still need a GPU at build: pip install
--no-binary :all: <pkg>, or custom CUDA-kernel compilation. Full
operational details (paths convention, build sbatch template, solver
notes) in references/conda_envs.md.
Reference index
Load these on demand — each is a focused playbook, not required reading.
Conda envs
references/conda_envs.md— paths convention, build sbatch template, why a GPU node isn't needed for the build, when it still would be, solver speed notes
Inside the SLURM script
references/script_shell_setup.md—#!/bin/bash -l,module, why.bashrcdoesn't fix itreferences/containers_apptainer.md— apptainer (not singularity); reusing FreeSurfer / etc. from sandbox containers via PATHreferences/gpu_jobs.md—--gres=gpu:1, GPU constraints, GPU→CPU memory pitfall, cuInit race +flockwarm-upreferences/logs_and_naming.md—~/logs/<jobname>_<jobid>.txtredirect,--job-nameconvention,scontrol update name=for arrays
Submission lifecycle
references/debug_qos.md—--qos=debugfast-lane smoke tests when fair-share has buried youreferences/array_throttling.md— when NOT to throttle (usually);Reason=JobArrayTaskLimitis your own%N, not the clusterreferences/idempotent_resubmit.md— per-(unit, stage)done()predicates; stale-chunk pitfall withchunk-NNNN-of-MMMM.npzreferences/partition_swap.md—scontrol update Partition=on PD jobs (preserves JobIDs, keepsafterokgraph intact)references/cancel_and_zombies.md— cancel by JobID range, sweepDependencyNeverSatisfiedzombies
Diagnosis
references/exit_codes.md—0:125(host RAM) vs1:0(GPU VRAM / Python raise) vsTIMEOUT
Getting data onto the cluster
references/smb_share_pull.md— pull straight from a departmental SMB/NAS share withsmbclient+ a user-made Kerberos ticket (kinit), instead of share → laptop → cluster; ticket lives on one login node only; read-only on append-only archives
Workstation hygiene
references/ssh_multiplexing.md—ControlMaster autoto dodge the login-host connection rate limit
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

