PBS / OpenPBS Job Scheduler

PBS (Portable Batch System) is widely used at DOE labs (ALCF, etc.) and many academic HPC centers. Jobs are described with #PBS directives and submitted via qsub.

Quick Command Reference

Job State Codes

Writing Job Scripts

#PBS directives must appear before any non-comment executable line.

Minimal working script:

#!/bin/bash
#PBS -N myjob
#PBS -A myproject
#PBS -l select=1:ncpus=32:mem=64gb
#PBS -l walltime=01:00:00
#PBS -o myjob.out
#PBS -e myjob.err
#PBS -q workq

cd $PBS_O_WORKDIR
module load gcc openmpi
mpiexec -n 32 ./my_application

Key #PBS Directives

Resource Select Syntax

-l select=<N>:<property>=<value>:... where N is the number of "chunks" (typically nodes).

# 1 node, 16 CPUs, 32 GB RAM
#PBS -l select=1:ncpus=16:mem=32gb

# 4 nodes, 128 CPUs each, 2 GPUs each
#PBS -l select=4:ncpus=128:mem=256gb:ngpus=2

# MPI + OpenMP hybrid (mpiprocs = MPI ranks per chunk)
#PBS -l select=2:ncpus=64:mpiprocs=4:ompthreads=16:mem=128gb

# Request specific node features
#PBS -l select=1:ncpus=32:arch=broadwell

place directive controls layout:

#PBS -l place=scatter          # one chunk per node (default)
#PBS -l place=pack             # pack chunks onto fewest nodes
#PBS -l place=excl             # exclusive node access
#PBS -l place=scatter:excl     # one per node, exclusive

Submitting Jobs

# Batch submission
qsub script.sh

# Override directives at submission
qsub -l select=4 -l walltime=2:00:00 script.sh

# Interactive job (drops into shell on allocated node)
qsub -I -l select=1:ncpus=8 -l walltime=1:00:00 -q debug

# Interactive with environment exported
qsub -I -V -l select=1:ncpus=16:mem=32gb -l walltime=2:00:00

Monitoring Jobs

# All your jobs
qstat -u $USER

# All running jobs
qstat -r

# All queued/held jobs
qstat -i

# Full detail for one job
qstat -f 12345

# Finished job details (past ~7 days, if server configured)
qstat -xf 12345

# Estimated start times
qstat -T

# List queues and their limits
qstat -Q

# All node states
pbsnodes -a

Default output file locations (when -o/-e not specified):

• Stdout: <jobname>.o<jobid> in $PBS_O_WORKDIR
• Stderr: <jobname>.e<jobid> in $PBS_O_WORKDIR

Watch a running job's output:

tail -f myjob.o12345

Cancelling, Holding, and Modifying Jobs

qdel 12345                    # cancel job
qdel 12345[5]                 # cancel array element 5
qdel -W force 12345           # force delete (if qdel hangs)

qhold 12345                   # place user hold (state → H)
qrls 12345                    # release hold

qalter -l walltime=4:00:00 12345   # extend walltime (if pending)
qalter -l select=8 12345           # change node count (if pending)

Job Arrays

Submit many similar jobs from one qsub. Resources apply to each array element independently.

# In the script:
#PBS -t 1-100            # indices 1..100
#PBS -t 1-50:2           # odd indices 1,3,5,...,49
#PBS -t 1-100%10         # max 10 running simultaneously

Or at submission: qsub -t 1-100 script.sh

Inside the script, use $PBS_ARRAY_INDEX:

INPUT="data/input_${PBS_ARRAY_INDEX}.txt"
mpiexec ./process "$INPUT"

Array management:

qstat -t 12345            # show all elements
qdel 12345[]              # delete entire array
qdel 12345[5]             # delete element 5

Job Dependencies

# Submit job1, capture ID
JOB1=$(qsub stage1.sh)

# Submit job2 to run only if job1 succeeds
qsub -W depend=afterok:$JOB1 stage2.sh

# Common dependency types:
# afterok:<id>     — start if <id> exits with code 0
# afternotok:<id>  — start if <id> exits with non-zero code
# afterany:<id>    — start regardless of <id> outcome
# before:<id>      — <id> starts only after this job begins
# beforeok:<id>    — <id> starts only if this job succeeds

Common Environment Variables (Inside a Job)

Using $PBS_NODEFILE with MPI:

mpiexec -hostfile $PBS_NODEFILE -n $PBS_NP ./my_app
# or simply:
mpiexec -n $PBS_NP ./my_app   # if MPI is PBS-aware

Module System

module avail              # list available modules
module spider <pkg>       # search for a package
module load gcc/12 openmpi/4.1
module list               # show loaded modules
module purge              # unload everything

Always load modules inside the job script, not just in your login shell.

Additional Resources

• Annotated job script examples: references/job-scripts.md
• Monitoring, qstat field glossary, troubleshooting: references/job-control.md
• Full resource select and place reference: references/resource-spec.md
• Copy-paste templates: assets/templates/