agentleFS
dtunaidtunai/agent-skills-for-compute/skills/slurm/SKILL.md

slurm

Skill8 starsChanged 8 months agoMITView on GitHub

Provides Slurm Workload Manager commands for HPC cluster job scheduling, GPU allocation with GRES, job arrays, MPI integration, resource management, QOS configuration, and accounting. Applies to tasks involving batch job submission (sbatch), interactive jobs (srun, salloc), job monitoring (squeue, sacct), GPU/CUDA workloads, parallel computing, cluster resource allocation, or HPC workflow management.

5 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

7 rules. 0 are common, 2 are distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 5 edited versions we read keep these.

  • 2

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Limit concurrent array tasks: Use % to avoid overwhelming scheduler

    line 400
  2. Very specific:

    Check CUDA_VISIBLE_DEVICES: Verify GPU allocation

    line 403
  3. Distinctive:

    Monitor job efficiency: Use sacct to optimize resource requests

    line 404
  4. Distinctive:

    Set memory limits explicitly: Avoid node oversubscription

    line 406
  5. Very specific:

    Forgetting --gpus flag: No GPUs allocated despite CUDA code

    line 413
  6. Very specific:

    Not using $SLURM_ARRAY_TASK_ID: All array tasks do same work

    line 415
  7. Very specific:

    Check sinfo --version for cluster version

    line 427

“Common” is measured: kept by at least 6 in 10 of the 5 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Slurm Workload Manager

Overview

Open-source cluster management and job scheduling system for Linux clusters. Slurm allocates compute resources, provides a framework for executing parallel jobs, and manages resource contention through job queues.

Quick Pattern

Incorrect — missing resource specifications:

# No resource specification, will fail
sbatch my_script.sh

# Implicit assumptions about GPUs
srun python train.py

Correct — explicit resource requests:

# Batch job with resources
sbatch --nodes=2 --ntasks-per-node=4 --gpus=2 --time=02:00:00 my_script.sh

# Interactive with GPU
srun --gpus=1 --mem=16G --pty bash

# Job array with concurrency limit
sbatch --array=0-99%10 --gpus-per-task=1 --mem=8G array_script.sh

Quick Command

# Job submission
sbatch script.sh                    # Submit batch job
srun command                        # Interactive execution
salloc -N 2 bash                    # Allocate resources, spawn shell

# Job monitoring
squeue                              # View job queue
squeue -u $USER                     # Your jobs only
sinfo                               # Partition/node status
sacct -j JOBID                      # Job accounting info
sstat -j JOBID                      # Running job stats

# Job control
scancel JOBID                       # Cancel job
scancel -u $USER                    # Cancel all your jobs
scontrol hold JOBID                 # Hold job
scontrol release JOBID              # Release held job

Quick Reference

Core Commands

CommandPurpose
sbatchSubmit batch script for later execution
srunSubmit job for immediate execution or initiate job step
sallocAllocate resources and spawn shell
squeueView job queue status
scancelCancel jobs or job steps
sinfoDisplay partition and node info
scontrolView/modify Slurm state (admin tool)
sacctJob accounting information from database
sstatResource utilization for running jobs

Resource Allocation

OptionPurpose
-N, --nodesNumber of nodes
-n, --ntasksNumber of tasks
-c, --cpus-per-taskCPUs per task
--memMemory per node (MB)
--mem-per-cpuMemory per CPU
-t, --timeTime limit (hh:mm:ss)
-p, --partitionPartition name
-G, --gpusTotal GPUs required
--gpus-per-nodeGPUs per node
--gresGeneric resources (e.g., gpu:2)

GPU Allocation

OptionPurpose
--gpus=NTotal GPUs for job
--gpus-per-node=NGPUs per node
--gpus-per-task=NGPUs per task
--gres=gpu:type:NGPUs with specific type
--mem-per-gpu=SIZEMemory per GPU
--gpu-bindGPU binding strategy

Job Arrays

SyntaxMeaning
--array=0-15Tasks 0 through 15
--array=0,5,10Specific task IDs
--array=0-15:4Step of 4 (0,4,8,12)
--array=0-99%10Max 10 concurrent tasks

Environment Variables:

  • $SLURM_ARRAY_JOB_ID: Array job ID
  • $SLURM_ARRAY_TASK_ID: Current task ID
  • $SLURM_ARRAY_TASK_COUNT: Total tasks

Output/Error Files

FormatMeaning
%jJob ID
%AArray job ID
%aArray task ID
%NNode name
%uUsername

Core Patterns

1. Basic Batch Job

#!/bin/bash
#SBATCH --job-name=my_job
#SBATCH --output=output_%j.txt
#SBATCH --error=error_%j.txt
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=2
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --partition=compute

# Your commands
srun hostname
srun my_application

2. GPU Job

#!/bin/bash
#SBATCH --job-name=gpu_job
#SBATCH --gpus=2
#SBATCH --cpus-per-gpu=4
#SBATCH --mem-per-gpu=16G
#SBATCH --time=04:00:00

# CUDA environment set automatically
nvidia-smi
python train_model.py

3. Job Array

#!/bin/bash
#SBATCH --job-name=array_job
#SBATCH --output=logs/job_%A_%a.out
#SBATCH --array=0-99%20
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G

# Use $SLURM_ARRAY_TASK_ID
python process_data.py --index $SLURM_ARRAY_TASK_ID

4. MPI Job

#!/bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --time=02:00:00

# Method 1: Direct launch (PMI2/PMIx)
srun ./my_mpi_app

# Method 2: mpirun with Slurm
mpirun -n 32 ./my_mpi_app

5. Multi-Step Job

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --gpus=4

# Step 1: Preprocessing
srun --ntasks=4 --gpus=0 preprocess.sh

# Step 2: Training (use GPUs)
srun --ntasks=4 --gpus=4 python train.py

# Step 3: Evaluation
srun --ntasks=2 --gpus=2 python eval.py

6. Interactive Job

# Allocate resources and get shell
salloc --nodes=1 --ntasks=4 --gpus=1 --time=02:00:00

# Inside allocation:
srun python interactive_work.py
exit  # Release resources

# Or one-liner
srun --pty --gpus=1 --mem=16G bash

7. Job Dependencies

# Submit first job
JOBID1=$(sbatch --parsable job1.sh)

# Submit dependent job (runs after job1 succeeds)
sbatch --dependency=afterok:$JOBID1 job2.sh

# Multiple dependencies
sbatch --dependency=afterok:$JOBID1:$JOBID2 job3.sh

# Array dependency (corresponding tasks)
sbatch --dependency=aftercorr:$JOBID1 --array=0-9 job2.sh

8. Heterogeneous Jobs

# Different resource requirements per component
sbatch -n4 --cpus-per-task=2 : -n8 --cpus-per-task=1 hetero_script.sh

GRES and GPU Patterns

Request Specific GPU Type

sbatch --gres=gpu:tesla:2 script.sh
sbatch --gres=gpu:a100:4 script.sh

GPU Binding

# Closest GPU to task
sbatch --gpus-per-task=1 --gpu-bind=closest script.sh

# Single GPU binding
sbatch --gpus-per-task=1 --gpu-bind=single:1 script.sh

MPS (Multi-Process Service)

# Request MPS percentage instead of full GPU
sbatch --gres=mps:50 script.sh  # 50% of GPU

CUDA_VISIBLE_DEVICES

Automatically set by Slurm for GPU jobs.

#!/bin/bash
#SBATCH --gpus=2

echo "GPUs available: $CUDA_VISIBLE_DEVICES"
# Example output: 0,1

Monitoring and Accounting

Queue Status

# All jobs
squeue

# Your jobs
squeue -u $USER

# Specific job
squeue -j JOBID

# Detailed format
squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"

# Array job tasks
squeue --array -j JOBID

Job Accounting

# Job summary
sacct -j JOBID

# Detailed format
sacct -j JOBID --format=JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode

# Time range
sacct -S 2026-01-01 -E 2026-01-31

# GPU usage
sacct -j JOBID --format=JobID,AllocGRES,TRESUsageInAve

Running Job Stats

# Resource utilization
sstat -j JOBID

# Specific fields
sstat -j JOBID --format=JobID,AveCPU,AveRSS,AveVMSize

Node Information

# Partition status
sinfo

# Available nodes
sinfo -N -l

# GPU nodes
sinfo --Format=NodeList,Gres:30,GresUsed:30

# Specific partition
sinfo -p gpu

Job Control

Cancel Jobs

scancel JOBID                    # Single job
scancel JOBID_[1-5]              # Array tasks 1-5
scancel JOBID_[1,3,5]            # Specific array tasks
scancel -u $USER                 # All your jobs
scancel -n job_name              # By name
scancel -p partition_name        # All in partition

Modify Jobs

# Hold/release
scontrol hold JOBID
scontrol release JOBID

# Update time limit
scontrol update JobId=JOBID TimeLimit=05:00:00

# Update QOS
scontrol update JobId=JOBID QOS=high

# Show job details
scontrol show job JOBID

Environment Variables

VariableDescription
SLURM_JOB_IDJob ID
SLURM_JOB_NAMEJob name
SLURM_SUBMIT_DIRSubmission directory
SLURM_JOB_NODELISTNodes allocated
SLURM_NTASKSNumber of tasks
SLURM_CPUS_PER_TASKCPUs per task
SLURM_LOCALIDTask rank on node
SLURM_PROCIDGlobal task rank
CUDA_VISIBLE_DEVICESGPU devices (auto-set)

Best Practices

  1. Always specify time limits: Prevents jobs running indefinitely
  2. Use job arrays for parameter sweeps: More efficient than individual jobs
  3. Limit concurrent array tasks: Use % to avoid overwhelming scheduler
  4. Request only needed resources: Improves scheduling and fairness
  5. Use dependencies for workflows: Automate multi-stage pipelines
  6. Check CUDA_VISIBLE_DEVICES: Verify GPU allocation
  7. Monitor job efficiency: Use sacct to optimize resource requests
  8. Use specific partitions: Target appropriate hardware
  9. Set memory limits explicitly: Avoid node oversubscription
  10. Consolidate into job steps: Better than many small jobs

Common Pitfalls

  • Not specifying time limit: May use partition default or fail
  • Requesting more GPUs than available: Job never schedules
  • Forgetting --gpus flag: No GPUs allocated despite CUDA code
  • Array without concurrency limit: Overwhelming the scheduler
  • Not using $SLURM_ARRAY_TASK_ID: All array tasks do same work
  • Hardcoding node names: Reduces portability
  • Excessive memory requests: Delays scheduling
  • Not checking job output: Missing errors until later
  • Using srun in sbatch without resources: Inherits full job allocation
  • Mixing salloc and sbatch: Confusing resource accounting

Version Notes

  • Slurm 23.x+: Improved GPU scheduling, MPS support
  • Slurm 22.x: Enhanced job arrays (up to 4M tasks)
  • Slurm 21.x: Heterogeneous job support
  • Check sinfo --version for cluster version

References

Official Slurm documentation at slurm.schedmd.com:

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.