Troubleshooting & FAQ¶

My job is stuck in PENDING — why?¶

Run squeue -j JOB_ID and inspect the REASON column. Common reasons:

My job failed immediately — where do I look?¶

# Check the job's exit state
sacct -j JOB_ID --format=JobID,State,ExitCode

# Read the error log (you specified this with --error in your script)
cat logs/job_JOB_ID.err

Common causes: missing module load in the script, incorrect file paths, insufficient memory (OOM Killed).

module: command not found in my job — but it works when I type it¶

This is the most common surprise on this cluster. Batch scripts run as non-login shells, which never read /etc/profile.d/, so the module shell function does not exist there. Add this line before your first module command:

source /etc/profile.d/modules.sh

The reason it bites people is that it is not a fatal error. The script carries on with the system GCC 11.5 instead of the compiler you asked for, so the job "succeeds" with the wrong toolchain. If a build behaves differently in a job than it did interactively, check this first:

grep -i "module" logs/job_JOBID.err     # look for "command not found"

The same applies to interactive sessions — use --pty bash -l so you get a login shell. See The Module System.

My job ran out of memory but I never asked for much¶

If you do not pass --mem or --mem-per-cpu, you do not get the node's memory — you get the default of 1000 MB per allocated CPU. A one-CPU job therefore has a 1 GB ceiling and is killed the moment it exceeds it. Always request memory explicitly, and check what you actually used:

sacct -j JOB_ID --format=JobID,ReqMem,MaxRSS,State

My Mathematica job died within the first minute¶

Almost always licensing, not your code. Mathematica here uses the INFN network licence (lm-mathematica.infn.it), so every kernel checks out a seat from a pool shared across INFN. If the pool is exhausted — commonly by someone's own uncapped job array — kernels fail at startup.

Cap concurrency on Mathematica arrays (#SBATCH --array=1-200%8) and retry. If it persists, the licence server may be unreachable; report it to the admins. See Software → Mathematica.

math: command not found¶

You are on the login node. Mathematica is installed on the compute nodes only — request one with srun, salloc, or sbatch.

How do I know how much memory my job actually used?¶

sacct -j JOB_ID --format=JobID,MaxRSS,AveRSS,Elapsed

MaxRSS is the peak resident memory used. Use this to tune future jobs.

I need to run a Jupyter notebook — how?¶

Launch Jupyter on a compute node via srun, forward the port via SSH:

# Step 1: On the cluster login node, start an interactive session
srun --ntasks=1 --cpus-per-task=2 --mem=8G --time=04:00:00 --pty bash -l

# Step 2: Note which compute node you landed on — you need it in step 4
hostname     # e.g. bardeen.ipa.internal

# Step 3: Activate your environment and start Jupyter there
source /home/your_username/myproject/.venv/bin/activate
jupyter lab --no-browser --port=8888 --ip=0.0.0.0

# Step 4: On your LOCAL machine, tunnel to that node via the login node.
# Substitute the hostname from step 2. Compute nodes are not reachable
# directly, which is why the tunnel is forwarded through galileo.
ssh -N -L 8888:bardeen.ipa.internal:8888 your_username@galileo.mi.infn.it

# Step 5: Open the http://127.0.0.1:8888/?token=... URL that Jupyter
# printed in step 3 — the token is required to log in.

I accidentally ran something heavy on the login node — what do I do?¶

Kill the process immediately:

# Find the process
ps aux | grep your_username

# Kill it by PID
kill -9 PID

Then submit the work as a proper Slurm job. If you are unsure whether a task is "too heavy" for the login node, the answer is almost always: use salloc to be safe.