Running Jobs with Slurm¶

Slurm (Simple Linux Utility for Resource Management) is the workload manager. You describe what resources your job needs (CPUs, memory, time, GPUs), and Slurm schedules your job to run when those resources are available. You never SSH directly to a compute node — Slurm handles that for you.

Key Slurm Concepts¶

Partition: A logical group of nodes, often differentiated by hardware (e.g., general, astro, teraram,teslap100). Choose the right one for your job.
Job: A unit of work submitted to Slurm. Can be a single task or a complex multi-node parallel job.
Allocation: The set of resources Slurm reserves for your job.

Useful Slurm Commands¶

# View the job queue
squeue

# View only your jobs
squeue -u your_username

# View available partitions and their status
sinfo

# Cancel a job
scancel JOB_ID

# Detailed accounting for a completed job
sacct -j JOB_ID --format=JobID,JobName,Elapsed,CPUTime,MaxRSS,State

The Partition System¶

A partition is a named group of nodes with its own access rules and limits. Select one with #SBATCH --partition=<name> (or -p <name>). Most users only ever need general.

Partition	Purpose	Who can use it	Nodes	Max walltime
`general`	Default, general-purpose CPU compute	`generalusers` group (all standard accounts)	36 nodes (3,872 CPUs), Broadwell → Sapphire Rapids + AMD	30 days
`astro`	Astrophysics group	`astro` group	`meitner` (144 CPUs)	30 days
`teraram`	Big-memory jobs	`teo` group	`feynman` (96 CPUs, 1.25 TB RAM)	30 days
`teslap100`	GPU jobs	`teo` group	`cabibbo` (NVIDIA Tesla P100, 32 CPUs)	30 days

Access is controlled by Unix group membership — if you are not in the group, sbatch -p <that partition> is rejected. Standard accounts are in generalusers and so can use general. Check your groups with id; see live partition state and exact node counts with sinfo.

Choosing a partition

Start with general. Reach for a specialised partition only when your job genuinely needs it — a terabyte of RAM (teraram) or a GPU (teslap100) — and you have access. general holds the overwhelming majority of the capacity.

How Slurm Counts CPUs Here¶

The farm schedules with CR_Core and SMT (hyper-threading) enabled, which has two consequences worth internalising:

A Slurm "CPU" is a hardware thread, and jobs are allocated whole physical cores. So --cpus-per-task=8 reserves 8 threads = 4 physical cores (both hyper-threads of each). The CPU counts shown by sinfo (%C) are threads.
Always request memory explicitly. If you omit --mem, you do not get "as much as the node has" — you get the cluster default of DefMemPerCPU = 1000 MB per allocated CPU. A single-CPU job therefore gets 1 GB, and is killed the moment it exceeds that. This is a common and confusing first failure: the code works fine interactively but the batch job dies with an out-of-memory kill. Use --mem=<total> or --mem-per-cpu=<per thread>.

⚠️ Some third-party job submitters do not pass --mem.

Tools that generate their own Slurm submissions (MadGraph's cluster.py, for example) may not forward a memory request at all, leaving their jobs on the 1000 MB-per-CPU default and dying at scale for no visible reason. If you are driving Slurm through a framework rather than writing #SBATCH lines yourself, check what it actually submits before blaming your physics.

Free CPUs are not enough — you also need free memory

A node can show many idle CPUs yet have little free RAM (most of it committed to running jobs). Your job starts only when a slot has both enough free CPUs and enough free memory. Check both at once:

sinfo -p general -o "%.14N %.6t %.15C %.10e %.10m"   # %e = free MB, %m = total MB

Smaller, shorter jobs also backfill into schedule gaps sooner than large, long ones — right-size your request.

Batch Jobs¶

A batch job is the standard workflow: you write a shell script describing your job and its resource requirements, submit it, and retrieve results when it finishes. This is ideal for long runs, parameter sweeps, and any job that does not need interactive supervision.

Anatomy of a Slurm Batch Script¶

Lines beginning with #SBATCH are directives to the Slurm scheduler — they are not executed as shell commands, but parsed before your job starts.

#!/bin/bash
# ─────────────────────────────────────────────────────
#  Slurm directives
# ─────────────────────────────────────────────────────
#SBATCH --job-name=thrust_nnlo       # A human-readable name for your job
#SBATCH --partition=general          # Which partition to run on
#SBATCH --nodes=1                    # Number of compute nodes
#SBATCH --ntasks=1                   # Number of parallel tasks (MPI ranks)
#SBATCH --cpus-per-task=8            # CPU cores per task (for OpenMP/threads)
#SBATCH --mem=16G                    # Total memory for the job
#SBATCH --time=04:00:00              # Maximum wall time (days-HH:MM:SS)
#SBATCH --output=logs/job_%j.out     # Standard output (%j = job ID)
#SBATCH --error=logs/job_%j.err      # Standard error
#SBATCH --mail-type=END,FAIL         # Email on completion or failure
#SBATCH --mail-user=user@lcm.mi.infn.it

# ─────────────────────────────────────────────────────
#  Environment setup
# ─────────────────────────────────────────────────────
# REQUIRED: batch scripts are non-login shells, where `module`
# does not exist until this file is sourced. Without it, every
# `module load` below fails silently and you get system GCC 11.5.
source /etc/profile.d/modules.sh

module purge
module load gcc/13
module load mpi/mpich-x86_64

# Cheap insurance — confirm you got the compiler you asked for
echo "Compiler: $(gcc --version | head -1)"

# ─────────────────────────────────────────────────────
#  Job execution
# ─────────────────────────────────────────────────────
# Set OpenMP thread count to match requested CPUs
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

echo "Job started on $(hostname) at $(date)"
echo "Running with $SLURM_CPUS_PER_TASK threads"

# Change to temporary for faster I/O
cd $SLURM_TMPDIR

# Run your compiled executable
srun thrust_calculator --order NLO --output results_nnlo.dat

echo "Job finished at $(date)"

Submit the script with:

sbatch my_job.sh
# Output: Submitted batch job 482910

A Fortran/C++ Compilation + Run Example¶

Because you cannot compile on the login node, a common pattern is to use a short "build job" first, then a "run job":

#!/bin/bash
#SBATCH --job-name=compile_mc
#SBATCH --partition=general
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=4G
#SBATCH --time=00:30:00
#SBATCH --output=logs/compile_%j.out
#SBATCH --mail-type=END,FAIL         # Email on completion or failure
#SBATCH --mail-user=user@lcm.mi.infn.it


source /etc/profile.d/modules.sh
module purge
module load gcc/15
echo "Compiler: $(gcc --version | head -1)"   # expect 15.2.1

cd /home/your_username/myproject

mkdir -p build && cd build
# Use -march=x86-64-v3 (not -march=native) so the binary runs on every node.
# See System Architecture → Instruction Sets for why.
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-O3 -march=x86-64-v3"
make -j $SLURM_CPUS_PER_TASK

echo "Build complete."

Once the build job finishes (check with squeue or wait for the email), your executable is ready and you can submit your production run job.

Job Arrays — Running Parameter Sweeps¶

If you need to run the same code with many different inputs (e.g., different values of a coupling constant, different scales μ), use a job array instead of submitting dozens of identical scripts:

#!/bin/bash
#SBATCH --job-name=scale_scan
#SBATCH --array=1-50           # Launches 50 jobs, SLURM_ARRAY_TASK_ID = 1..50
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=02:00:00
#SBATCH --output=logs/scan_%A_%a.out   # %A = array job ID, %a = task ID

source /etc/profile.d/modules.sh
module purge
module load gcc/14

# Use the task ID to select a parameter from a list
SCALE_FILE=/home/your_username/scales.txt
SCALE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" $SCALE_FILE)

echo "Running with scale mu = ${SCALE} GeV"
./my_program --mu $SCALE --output results/output_${SLURM_ARRAY_TASK_ID}.dat

💡 Limit concurrent array tasks to be a good citizen.

A job array of 500 tasks could flood the queue. Use the %N suffix to cap concurrent running tasks: #SBATCH --array=1-500%20 runs at most 20 simultaneously.

Interactive Sessions¶

Sometimes you need to work interactively on a compute node — for example, to test a compilation, debug a script, or run Mathematica interactively. Use salloc or srun for this.

⚠️ Interactive sessions still consume cluster resources.

An interactive session holds a reservation on compute nodes for its entire duration, even if you are idle. Request only what you need, and exit your session as soon as you are done.

`salloc` — Request an Allocation, Then Work¶

salloc requests resources and drops you into a new shell. From there, you can run commands directly on the allocated node(s).

# Request 1 node, 4 CPUs, 8 GB RAM, for up to 2 hours.
# Use `--pty bash -l` — the -l (login shell) is what makes `module` available.
salloc --nodes=1 --ntasks=1 --cpus-per-task=4 --mem=8G --time=02:00:00 \
       --pty bash -l

# Once the allocation is granted, your prompt changes.
# You are now running on a compute node.
# Load modules and work interactively:
module load gcc/15
make -j 4
./my_program --test

# When done, exit the allocation
exit

⚠️ Without -l, module will not exist.

A bare salloc ... --pty bash gives you a non-login shell, in which module load fails with command not found — the same trap as in batch scripts. Either use --pty bash -l as above, or run source /etc/profile.d/modules.sh once you are in.

`srun` — Run Commands on Compute Nodes¶

srun is used to launch tasks directly on compute nodes under Slurm allocation. It’s ideal for:

Running a single command non-interactively
Launching parallel jobs
Opening an interactive session on a compute node

Unlike sbatch, srun runs jobs immediately (subject to scheduling), and unlike ssh, it ensures proper resource accounting and isolation.

Run a Single Command¶

Use srun when you need to execute one command with allocated resources:

# Compile using 8 CPU cores on a compute node
srun --ntasks=1 --cpus-per-task=8 --mem=4G --time=00:15:00 \
     make -j8

Interactive Shell on a Compute Node¶

To debug, test code, or run commands manually:

# Open an interactive bash shell on a compute node.
# `-l` makes it a login shell so that `module` is available.
srun --ntasks=1 --cpus-per-task=4 --mem=8G --time=01:00:00 --pty bash -l

This gives you a shell inside a compute node, not the login node.