Perlmutter — NERSC Supercomputer

Perlmutter is NERSC's GPU-accelerated Cray EX system at Lawrence Berkeley National Laboratory. It uses Slurm as its workload manager.

System Overview

Critical Requirements

Every Perlmutter job MUST include -A <account> and -C cpu or -C gpu. Jobs without these will fail or target the wrong node type.

#SBATCH -A myproject       # your NERSC project name (required)
#SBATCH -C gpu             # or -C cpu (required)

Partitions and QOS

Big job discount: jobs using ≥128 GPU nodes or ≥256 CPU nodes are charged at 50% SU rate.

Specify with --qos=<name>, e.g. #SBATCH --qos=debug.

Quick Slurm Reference

sbatch script.sh                   # submit
squeue -u $USER                    # your jobs
squeue -j 12345 --start            # estimated start time
scontrol show job 12345            # full detail
scancel 12345                      # cancel
sacct -j 12345 --format=JobID,State,Elapsed,MaxRSS,ExitCode
salloc -N1 -C gpu --qos=debug -t 30:00 -A myproject   # interactive

Job state codes: PD=Pending, R=Running, CD=Completed, CA=Cancelled, F=Failed, TO=Timeout

Minimal Job Script — GPU

#!/bin/bash
#SBATCH -A myproject
#SBATCH -C gpu
#SBATCH -q regular
#SBATCH -N 1
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-task=1
#SBATCH -c 32                    # CPU cores per task (128 cores / 4 tasks)
#SBATCH -t 2:00:00
#SBATCH -o job_%j.out
#SBATCH -e job_%j.err

module load cudatoolkit
export SLURM_CPU_BIND="cores"
srun ./my_gpu_app

Minimal Job Script — CPU

#!/bin/bash
#SBATCH -A myproject
#SBATCH -C cpu
#SBATCH -q regular
#SBATCH -N 4
#SBATCH --ntasks-per-node=128
#SBATCH -t 4:00:00
#SBATCH -o job_%j.out

module load PrgEnv-gnu cray-mpich
srun ./my_mpi_app

Checking Allocations

# CLI (run on any Perlmutter login node)
iris                          # summary of your projects
iris hours                    # CPU/GPU hours used vs. allocated
iris jobs                     # recent job history with charges
iris jobs --project myproject # filter by project

# Web
# https://iris.nersc.gov  — "My Account" tab shows remaining balance

Storage

Warning: Files in $SCRATCH not accessed for 8 weeks are automatically deleted without notice. Always check ages with lfs find $SCRATCH --atime +56 -maxdepth 2.

Always write job output to $SCRATCH, not $HOME.

Modules

module avail                      # list available modules
module spider <pkg>               # search for a package
module load PrgEnv-gnu            # GCC-based Cray PE
module load cudatoolkit           # NVIDIA CUDA for GPU nodes
module load cray-mpich            # Cray MPI library
module list                       # show what's loaded
module purge                      # unload everything

Cray compiler wrappers (cc, CC, ftn) link the correct MPI and libraries automatically when the appropriate PrgEnv module is loaded.

Interactive Jobs

# GPU interactive session (debug QOS, 30 min)
salloc -N 1 -C gpu --qos=debug -t 30:00 -A myproject \
  --ntasks-per-node=4 --gpus-per-task=1 -c 32

# CPU interactive session
salloc -N 2 -C cpu --qos=debug -t 30:00 -A myproject \
  --ntasks-per-node=128

# Once inside, run steps with srun
srun -n 4 --gpus-per-task=1 ./my_app

Common Gotchas

• Forget -C cpu/-C gpu → job may queue indefinitely or hit wrong partition
• Write to $HOME instead of $SCRATCH → quota exceeded, job fails
• $SCRATCH files older than 8 weeks → silently deleted
• Not setting SLURM_CPU_BIND=cores on GPU nodes → suboptimal CPU affinity
• Using wrong account name → charges wrong allocation or job rejected

Additional Resources

• Annotated job script examples: references/job-scripts.md
• Storage details, Lustre striping, HPSS archiving: references/storage.md
• Allocation system, iris, ERCAP, sacct: references/allocations.md
• NERSC docs: https://docs.nersc.gov/