System Architecture¶

Before issuing a single command, it helps to have a mental map of what the cluster actually is. It is not one machine — it is a network of specialised nodes, each playing a distinct role.

flowchart TD
    WS([Your Workstation]) -- "SSH (port 22)" --> LN["Login Node<br>• Shell, editing, sbatch<br>• Hard limit: 1 CPU, 1 GB<br>• Shared by all users"]

    LN --> CN{Slurm workload manager}

    CN --> INFN["INFN<br>General, Teraram"]
    CN --> ASTRO["Astrophysics<br>Astro"]
    CN --> GPU["GPU Nodes<br>teslap100"]

    INFN --> NET{"High-speed (10Gb)<br>shared network filesystem"}
    ASTRO --> NET
    GPU --> NET

    NET --> SS[("Shared Storage <br>/home<br>/farmstorage<br>/projects<br><br>Visible and consistent on all nodes")]

    style LN fill:#ffe6e6,stroke:#cc0000,stroke-width:2px,color:#333

Key insight: because the filesystem is shared, any file you create or edit on the login node is instantly visible on compute nodes, and vice versa. You write your scripts on the login node; your jobs execute them on compute nodes. The scheduler — Slurm — is the only path to the compute nodes.

Compute Nodes¶

The farm is a heterogeneous fleet of roughly 40 compute nodes (~4,100 Slurm CPUs in total), spanning several generations of Intel Xeon plus one AMD EPYC family. Every node has Simultaneous Multi-Threading (hyper-threading) enabled, so what Slurm calls a CPU is a hardware thread — there are two threads per physical core.

CPU model	Microarchitecture	Example nodes	Cores / threads per node	RAM	AVX-512
Xeon E5-2640 v4	Broadwell (2016)	`noether`, `rutherford`	20c / 40t	64 GB	No
Xeon Silver 4110	Skylake-SP	`cabibbo` (GPU node)	16c / 32t	192 GB	Yes
Xeon Silver 4214 / 4216	Cascade Lake	`dittrich`, `oppenheimer`, `goeppert`, `hack`, …	24–32c / 48–64t	64 GB	Yes
Xeon Gold 5218	Cascade Lake	`newton`, `curie`, `glauber`, `kosterlitz`, …	64c / 128t	128 GB	Yes
Xeon Gold 6238R	Cascade Lake	`einstein`, `friedmann`, `landau`, `schwarzschild`	56c / 112t	128 GB	Yes
Xeon Gold 5318H	Cooper Lake	`dirac`, `heaviside`, `lorentz`, `turing`, `meitner`, …	72c / 144t	128–256 GB	Yes
Xeon Gold 6418H	Sapphire Rapids	`pauli`	96c / 192t	512 GB	Yes
AMD EPYC 7413	Zen 3 (Milan)	`fourier`, `kolmogorov`, `lagrange`, `feynman`	48c / 96t	256 GB – 1.25 TB	No

Memory per node ranges from ~58 GB on the oldest nodes to ~1.25 TB on the big-memory node feynman (in the teraram partition). The GPU node cabibbo carries an NVIDIA Tesla P100.

Query the live hardware yourself

Node inventories change — don't hard-code node names into scripts. Ask Slurm:

# Sockets × cores × threads, CPUs and memory for every node
sinfo -N -o "%.14N %.10P %.4X %.4Y %.4Z %.5c %.9m %.6t"
# Free memory and idle CPUs on one partition right now
sinfo -p general -o "%.14N %.6t %.15C %.9e"   # %C = Alloc/Idle/Other/Total

Instruction Sets & Compiler Flags¶

Because the fleet spans Broadwell (2016) through Sapphire Rapids (2023) plus AMD Zen 3, not every node understands the same CPU instructions. Getting this wrong produces an Illegal instruction (core dumped): a binary compiled for a newer instruction set crashes the moment it lands on an older node.

The safe common denominator across every compute node is the x86-64-v3 microarchitecture level — AVX, AVX2, FMA, BMI½, F16C. It is supported farm-wide (verified on all nodes), so a -march=x86-64-v3 build runs everywhere while still using AVX2/FMA.

AVX-512 (x86-64-v4) is not portable

Six nodes have no AVX-512, so a v4 binary will SIGILL on them:

Broadwell: noether, rutherford
AMD EPYC (Milan): feynman, fourier, kolmogorov, lagrange

Only target x86-64-v4 if you also --exclude those nodes (or restrict to a node set you know has AVX-512).

Do not ship -march=native binaries to Slurm

-march=native bakes in the instruction set of whatever node did the compile. Build on a modern node (e.g. pauli) and the binary will crash on the Broadwell and AMD nodes. For a build that is both portable and optimised, use:

-O3 -march=x86-64-v3      # AVX2/FMA, runs on every farm node

The login node cannot run compute binaries

The login node (galileo) is a small virtual machine without AVX2 — meant for editing, sbatch, and light work only. Even a portable x86-64-v3 binary will SIGILL there. Always compile and run your code on a compute node (through Slurm), never on the login node.