SLURM Commands

SLURM (Simple Linux Utility for Resource Management) is the job scheduler used on the Caltech HPC cluster.

Job Submission

sbatch

Submit batch (non-interactive) jobs. Output is sent to slurm-$SLURM_JOB_ID.out by default.

sbatch myscript.sh

salloc

Obtain a job allocation for interactive sessions:

salloc -n 4 -t 1:00:00

srun

Execute an application, obtaining allocation if needed. Also used to distribute MPI processes:

srun --pty -t 1:00:00 -n 4 -N 1 /bin/bash -l

For GUI applications, add --x11 to forward the display back to your workstation. You must have connected to the login node with ssh -X (or ssh -Y) first:

srun --x11 -t 1:00:00 -N 1 xterm

Resource Request Parameters

Parameter	Description
`--nodes` or `-N`	Number of nodes
`-n`	Total tasks required
`--mem`	Memory per node (e.g., `64G`)
`--mem-per-cpu`	Memory per CPU
`--gres gpu:#`	Number of GPUs
`--qos`	Quality of Service (`normal` or `debug`)
`--exclusive`	Exclusive node access
`--constraint`	Processor type (`skylake`, `cascadelake`, `broadwell`)
`--partition`	Target partition (`expansion` or `gpu`)

Environment Variables

Variable	Description
`SLURM_JOB_ID`	Job ID
`SLURM_SUBMIT_DIR`	Submission directory
`SLURM_CPUS_ON_NODE`	CPUs on the node
`SLURM_JOB_NAME`	Job name
`SLURM_JOB_NODELIST`	Nodes allocated
`SLURM_JOB_NUMNODES`	Number of nodes
`SLURM_NPROCS`	Number of processes

Queue Management

squeue

View the job queue:

# All jobs
squeue

# Long format
squeue -l

# Your jobs only
squeue -u $USER

# By account
squeue -A mygroup

# By state
squeue --state=pending
squeue --state=running

scancel

Cancel jobs:

# By job ID
scancel 12345

# All your jobs
scancel -u $USER

# By state
scancel --state=pending -u $USER

scontrol

View and manage jobs:

# Detailed job info
scontrol show job 12345

# Hold a job
scontrol hold 12345

# Release a job
scontrol release 12345

Usage Reporting

sreport

Historical usage by user or group:

sreport cluster AccountUtilizationByUser

sacct

Detailed job information:

# Recent jobs
sacct -u $USER

# Specific job
sacct -j 12345

# With specific fields
sacct -o JobID,JobName,Elapsed,MaxRSS -j 12345

Account Management

If you belong to more than one group, jobs run against your default account unless you say otherwise.

# Show the accounts you're allowed to submit against
sacctmgr show user $USER withassoc

# Change your default account
sacctmgr modify user $USER set defaultaccount=<account>

To charge a single job to a specific account without changing your default, pass --account:

sbatch --account=<account> myscript.sh

Task Launching

MPI Jobs

Two common ways to launch MPI tasks. Either works; pick based on what your application or MPI build expects.

With mpirun (OpenMPI or Intel MPI):

mpirun -np 32 ./myprogram

With srun:

srun ./myprogram

Tip

srun integrates directly with SLURM’s task management and tends to be the smaller-friction option when it works. Some MPI builds and application-level launchers expect mpirun specifically — when in doubt, check the application’s documentation.

Example Batch Script

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --nodes=1
#SBATCH --ntasks=32
#SBATCH --mem=64G
#SBATCH --time=24:00:00
#SBATCH --partition=expansion

module load openmpi/4.1.0
srun ./myprogram