slurm
Skill8 starsChanged 8 months agoMITView on GitHub
Provides Slurm Workload Manager commands for HPC cluster job scheduling, GPU allocation with GRES, job arrays, MPI integration, resource management, QOS configuration, and accounting. Applies to tasks involving batch job submission (sbatch), interactive jobs (srun, salloc), job monitoring (squeue, sacct), GPU/CUDA workloads, parallel computing, cluster resource allocation, or HPC workflow management.
5 other projects use a version of this.
Adopted by
Most recently taken up by
- 14790897/my-skillstheir own version, first seen Nov 15, 2026
- surajinacademia/Computational_modeling_skillstheir own version, first seen Oct 20, 2026
- NVIDIA/AI-Factory-Operations-Agenttheir own version, first seen Oct 13, 2026
- AI-ModCon/genesis-skillstheir own version, first seen Sep 24, 2026
- jaaxk/claude-skillstheir own version, first seen Mar 22, 2026
Other versions5 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 5
- NVIDIA/AI-Factory-Operations-Agent★ 41same name
- AI-ModCon/genesis-skills★ 30same name
- 14790897/my-skills★ 2same name
- surajinacademia/Computational_modeling_skills★ 1same name
- jaaxk/claude-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository23 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- validate-skillsSkill
- alchemi-toolkit-opsSkill
- autoresearch-setupSkill
- bionemoSkill
- biosimspaceSkill
- cuda-quantumSkill
- cutileSkill
- daskSkill
- flashinferSkill
- materials-projectSkill
- mpiSkill
- nemotronSkill
- omniverse-simreadySkill
- physicsnemoSkill
- polarsSkill
- prime-verifiersSkill
- qiskitSkill
- raySkill
- rdkitSkill
- tinkerSkill
- tritonSkill
The file
We split this skill into the rules it gives an agent
7 rules. 0 are common, 2 are distinctive, and 5 are very specific to this project.
0
Common
Most of the 5 edited versions we read keep these.
2
Distinctive
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 400
Limit concurrent array tasks: Use % to avoid overwhelming scheduler
- Very specific: line 403
Check CUDA_VISIBLE_DEVICES: Verify GPU allocation
- Distinctive: line 404
Monitor job efficiency: Use sacct to optimize resource requests
- Distinctive: line 406
Set memory limits explicitly: Avoid node oversubscription
- Very specific: line 413
Forgetting --gpus flag: No GPUs allocated despite CUDA code
- Very specific: line 415
Not using $SLURM_ARRAY_TASK_ID: All array tasks do same work
- Very specific: line 427
Check sinfo --version for cluster version
“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Slurm Workload Manager
Overview
Open-source cluster management and job scheduling system for Linux clusters. Slurm allocates compute resources, provides a framework for executing parallel jobs, and manages resource contention through job queues.
Quick Pattern
Incorrect — missing resource specifications:
# No resource specification, will fail
sbatch my_script.sh
# Implicit assumptions about GPUs
srun python train.pyCorrect — explicit resource requests:
# Batch job with resources
sbatch --nodes=2 --ntasks-per-node=4 --gpus=2 --time=02:00:00 my_script.sh
# Interactive with GPU
srun --gpus=1 --mem=16G --pty bash
# Job array with concurrency limit
sbatch --array=0-99%10 --gpus-per-task=1 --mem=8G array_script.shQuick Command
# Job submission
sbatch script.sh # Submit batch job
srun command # Interactive execution
salloc -N 2 bash # Allocate resources, spawn shell
# Job monitoring
squeue # View job queue
squeue -u $USER # Your jobs only
sinfo # Partition/node status
sacct -j JOBID # Job accounting info
sstat -j JOBID # Running job stats
# Job control
scancel JOBID # Cancel job
scancel -u $USER # Cancel all your jobs
scontrol hold JOBID # Hold job
scontrol release JOBID # Release held jobQuick Reference
Core Commands
| Command | Purpose |
sbatch | Submit batch script for later execution |
srun | Submit job for immediate execution or initiate job step |
salloc | Allocate resources and spawn shell |
squeue | View job queue status |
scancel | Cancel jobs or job steps |
sinfo | Display partition and node info |
scontrol | View/modify Slurm state (admin tool) |
sacct | Job accounting information from database |
sstat | Resource utilization for running jobs |
Resource Allocation
| Option | Purpose |
-N, --nodes | Number of nodes |
-n, --ntasks | Number of tasks |
-c, --cpus-per-task | CPUs per task |
--mem | Memory per node (MB) |
--mem-per-cpu | Memory per CPU |
-t, --time | Time limit (hh:mm:ss) |
-p, --partition | Partition name |
-G, --gpus | Total GPUs required |
--gpus-per-node | GPUs per node |
--gres | Generic resources (e.g., gpu:2) |
GPU Allocation
| Option | Purpose |
--gpus=N | Total GPUs for job |
--gpus-per-node=N | GPUs per node |
--gpus-per-task=N | GPUs per task |
--gres=gpu:type:N | GPUs with specific type |
--mem-per-gpu=SIZE | Memory per GPU |
--gpu-bind | GPU binding strategy |
Job Arrays
| Syntax | Meaning |
--array=0-15 | Tasks 0 through 15 |
--array=0,5,10 | Specific task IDs |
--array=0-15:4 | Step of 4 (0,4,8,12) |
--array=0-99%10 | Max 10 concurrent tasks |
Environment Variables:
$SLURM_ARRAY_JOB_ID: Array job ID$SLURM_ARRAY_TASK_ID: Current task ID$SLURM_ARRAY_TASK_COUNT: Total tasks
Output/Error Files
| Format | Meaning |
%j | Job ID |
%A | Array job ID |
%a | Array task ID |
%N | Node name |
%u | Username |
Core Patterns
1. Basic Batch Job
#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=output_%j.txt
#SBATCH --error=error_%j.txt
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=2
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --partition=compute
# Your commands
srun hostname
srun my_application2. GPU Job
#!/bin/bash
#SBATCH --job-name=gpu_job
#SBATCH --gpus=2
#SBATCH --cpus-per-gpu=4
#SBATCH --mem-per-gpu=16G
#SBATCH --time=04:00:00
# CUDA environment set automatically
nvidia-smi
python train_model.py3. Job Array
#!/bin/bash
#SBATCH --job-name=array_job
#SBATCH --output=logs/job_%A_%a.out
#SBATCH --array=0-99%20
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
# Use $SLURM_ARRAY_TASK_ID
python process_data.py --index $SLURM_ARRAY_TASK_ID4. MPI Job
#!/bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --time=02:00:00
# Method 1: Direct launch (PMI2/PMIx)
srun ./my_mpi_app
# Method 2: mpirun with Slurm
mpirun -n 32 ./my_mpi_app5. Multi-Step Job
#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --gpus=4
# Step 1: Preprocessing
srun --ntasks=4 --gpus=0 preprocess.sh
# Step 2: Training (use GPUs)
srun --ntasks=4 --gpus=4 python train.py
# Step 3: Evaluation
srun --ntasks=2 --gpus=2 python eval.py6. Interactive Job
# Allocate resources and get shell
salloc --nodes=1 --ntasks=4 --gpus=1 --time=02:00:00
# Inside allocation:
srun python interactive_work.py
exit # Release resources
# Or one-liner
srun --pty --gpus=1 --mem=16G bash7. Job Dependencies
# Submit first job
JOBID1=$(sbatch --parsable job1.sh)
# Submit dependent job (runs after job1 succeeds)
sbatch --dependency=afterok:$JOBID1 job2.sh
# Multiple dependencies
sbatch --dependency=afterok:$JOBID1:$JOBID2 job3.sh
# Array dependency (corresponding tasks)
sbatch --dependency=aftercorr:$JOBID1 --array=0-9 job2.sh8. Heterogeneous Jobs
# Different resource requirements per component
sbatch -n4 --cpus-per-task=2 : -n8 --cpus-per-task=1 hetero_script.shGRES and GPU Patterns
Request Specific GPU Type
sbatch --gres=gpu:tesla:2 script.sh
sbatch --gres=gpu:a100:4 script.shGPU Binding
# Closest GPU to task
sbatch --gpus-per-task=1 --gpu-bind=closest script.sh
# Single GPU binding
sbatch --gpus-per-task=1 --gpu-bind=single:1 script.shMPS (Multi-Process Service)
# Request MPS percentage instead of full GPU
sbatch --gres=mps:50 script.sh # 50% of GPUCUDA_VISIBLE_DEVICES
Automatically set by Slurm for GPU jobs.
#!/bin/bash
#SBATCH --gpus=2
echo "GPUs available: $CUDA_VISIBLE_DEVICES"
# Example output: 0,1Monitoring and Accounting
Queue Status
# All jobs
squeue
# Your jobs
squeue -u $USER
# Specific job
squeue -j JOBID
# Detailed format
squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"
# Array job tasks
squeue --array -j JOBIDJob Accounting
# Job summary
sacct -j JOBID
# Detailed format
sacct -j JOBID --format=JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode
# Time range
sacct -S 2026-01-01 -E 2026-01-31
# GPU usage
sacct -j JOBID --format=JobID,AllocGRES,TRESUsageInAveRunning Job Stats
# Resource utilization
sstat -j JOBID
# Specific fields
sstat -j JOBID --format=JobID,AveCPU,AveRSS,AveVMSizeNode Information
# Partition status
sinfo
# Available nodes
sinfo -N -l
# GPU nodes
sinfo --Format=NodeList,Gres:30,GresUsed:30
# Specific partition
sinfo -p gpuJob Control
Cancel Jobs
scancel JOBID # Single job
scancel JOBID_[1-5] # Array tasks 1-5
scancel JOBID_[1,3,5] # Specific array tasks
scancel -u $USER # All your jobs
scancel -n job_name # By name
scancel -p partition_name # All in partitionModify Jobs
# Hold/release
scontrol hold JOBID
scontrol release JOBID
# Update time limit
scontrol update JobId=JOBID TimeLimit=05:00:00
# Update QOS
scontrol update JobId=JOBID QOS=high
# Show job details
scontrol show job JOBIDEnvironment Variables
| Variable | Description |
SLURM_JOB_ID | Job ID |
SLURM_JOB_NAME | Job name |
SLURM_SUBMIT_DIR | Submission directory |
SLURM_JOB_NODELIST | Nodes allocated |
SLURM_NTASKS | Number of tasks |
SLURM_CPUS_PER_TASK | CPUs per task |
SLURM_LOCALID | Task rank on node |
SLURM_PROCID | Global task rank |
CUDA_VISIBLE_DEVICES | GPU devices (auto-set) |
Best Practices
- Always specify time limits: Prevents jobs running indefinitely
- Use job arrays for parameter sweeps: More efficient than individual jobs
- Limit concurrent array tasks: Use
%to avoid overwhelming scheduler - Request only needed resources: Improves scheduling and fairness
- Use dependencies for workflows: Automate multi-stage pipelines
- Check CUDA_VISIBLE_DEVICES: Verify GPU allocation
- Monitor job efficiency: Use sacct to optimize resource requests
- Use specific partitions: Target appropriate hardware
- Set memory limits explicitly: Avoid node oversubscription
- Consolidate into job steps: Better than many small jobs
Common Pitfalls
- Not specifying time limit: May use partition default or fail
- Requesting more GPUs than available: Job never schedules
- Forgetting --gpus flag: No GPUs allocated despite CUDA code
- Array without concurrency limit: Overwhelming the scheduler
- Not using $SLURM_ARRAY_TASK_ID: All array tasks do same work
- Hardcoding node names: Reduces portability
- Excessive memory requests: Delays scheduling
- Not checking job output: Missing errors until later
- Using srun in sbatch without resources: Inherits full job allocation
- Mixing salloc and sbatch: Confusing resource accounting
Version Notes
- Slurm 23.x+: Improved GPU scheduling, MPS support
- Slurm 22.x: Enhanced job arrays (up to 4M tasks)
- Slurm 21.x: Heterogeneous job support
- Check
sinfo --versionfor cluster version
References
Official Slurm documentation at slurm.schedmd.com:
- Quick Start: quickstart.html
- sbatch: sbatch.html
- GRES/GPU: gres.html
- Job Arrays: job_array.html
- QOS: qos.html
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

