How do I write a SLURM batch script for jobs that use both MPI and OpenMP?

Q&A
, , , , ,
1

I have a job that has 6 MPI processes with 4 OpenMP threads per process. How do I write a SLURM script to submit this job?

18.04.19:
Notes:

  1. Assumptions:
    processor == logical CPU
    node hierarchy: node/socket/core/thread
    threads per task = cpus per task (see below)
    MPI process == MPI task == task
    default: 1 cpu per task
    => “OpenMP threads per process” == cpus-per-task == OMP_NUM_THREADS
  2. Node specs not given: we will assume 16 core/node.
  3. Hyperthreading not turned on (generally not useful for HPC environs).
    *For SLURM, this makes one thread per core, and equivalently, one cpu per core
  4. SelectType=select/cons_res
    => CPU is consumable resource

So, “6 MPI processes with 4 OpenMP threads per process” specifies a total of
6 MPI tasks, with 4 threads per task, the latter expressed as OMP_NUM_THREADS = 4
and/or cpus-per-task = 4. These variables correspond to -n and -c, which are the minimum specifications needed to submit a job.

The number of nodes is not specified, so we leave that to SLURM.

#!/bin/bash
#SBATCH   --ntasks=6
#SBATCH   --cpus-per-task=4

OMP_NUM_THREADS = 4

CURATOR: Torey

2

Hi Jack,
I’d set up the script in terms of number of nodes and number of tasks per node, then specify number of threads; something along these lines (assume each node has 16 core, or cpu):

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=2
#SBATCH --ntasks=32

OMP_NUM_THREADS=2

In the srun line, you can specify number of nodes and number of processes per node, but that shouldn’t be necessary if you’ve defined those as above. The MPI part is the number of tasks, the OpenMP is the number of processes per task.

I hope this helps; questions are welcome!