Slurm Workload Manager

Slurm is the dominant HPC batch scheduler. Jobs are submitted to queues (partitions), allocated nodes, and run with full or partial node access.

Quick Command Reference

Job State Codes

Writing Job Scripts

Directives go at the top of the script as #SBATCH comments before any executable line.

Minimal working script:

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=32
#SBATCH --time=01:00:00
#SBATCH --account=myproject
#SBATCH --output=myjob_%j.out
#SBATCH --error=myjob_%j.err

module load gcc openmpi
srun ./my_application

Key #SBATCH Directives

Output Filename Tokens

Submitting Jobs

# Batch submission
sbatch script.sh

# Override directives at submission
sbatch --nodes=8 --time=4:00:00 script.sh

# Interactive allocation (drops into a shell on allocated node)
salloc --nodes=2 --time=1:00:00 --partition=debug
srun ./my_app          # run within the allocation
exit                   # release allocation

# One-off parallel command (creates its own allocation)
srun --nodes=1 --ntasks=4 ./my_app

Monitoring Jobs

# Your jobs
squeue -u $USER

# All jobs, long format
squeue -l

# Specific job
squeue -j 12345

# Estimated start times for pending jobs
squeue --start -u $USER

# Full detail (reason for pending, TRES, etc.)
scontrol show job 12345

# Live updates every 5 seconds
watch -n5 squeue -u $USER

Reading output files: Slurm writes stdout/stderr when the job finishes (or continuously if --output is to a path). Watch a running job's output:

tail -f myjob_12345.out

After job completes:

sacct -j 12345 --format=JobID,State,Elapsed,MaxRSS,ExitCode

Cancelling and Holding Jobs

scancel 12345                      # cancel by job ID
scancel 12345_5                    # cancel one array element
scancel -u $USER --state=PENDING   # cancel all your pending jobs
scancel -n myjob                   # cancel by name

scontrol hold 12345                # hold a pending job
scontrol release 12345             # release held job

Job Arrays

Run many similar jobs from one submission. Each element gets a unique SLURM_ARRAY_TASK_ID.

#SBATCH --array=0-99             # indices 0..99
#SBATCH --array=1-50:2           # odd indices 1,3,5,...,49
#SBATCH --array=0-999%20         # up to 20 running at once
#SBATCH --output=job_%A_%a.out   # %A=parent ID, %a=task index

Inside the script, use $SLURM_ARRAY_TASK_ID to select input:

INPUT="data/input_${SLURM_ARRAY_TASK_ID}.txt"
srun ./process "$INPUT"

Array management:

squeue -r -j 36           # show all elements
scancel 36                # cancel entire array
scancel 36_[1-5]          # cancel elements 1–5

Job Dependencies

Chain jobs so they start only when a predecessor finishes:

JOB1=$(sbatch stage1.sh | awk '{print $4}')
sbatch --dependency=afterok:$JOB1 stage2.sh

# Common dependency types:
# afterok:<id>    — start only if <id> succeeds
# afterany:<id>   — start regardless of <id> outcome
# afternotok:<id> — start only if <id> fails
# singleton       — only one job with this name runs at a time

Common Environment Variables (Inside a Job)

Module System

module avail              # list available modules
module spider <pkg>       # search for a package
module load gcc/12 openmpi/4.1
module list               # show loaded modules
module purge              # unload everything

Always load modules inside the job script, not just in your login shell.

Additional Resources

• Annotated job script examples: references/job-scripts.md
• Monitoring, troubleshooting, scontrol recipes: references/job-control.md
• Accounting, fairshare, allocation reporting: references/accounting.md
• Copy-paste templates: assets/templates/