As mentioned in the Europa kickoff meeting, Europa is an High Throughput Computing Environment, or HTC for short. This is in contrast to Ganymede which is an HPC system. The major difference between HTC and HPC are the networking (and sometime storage).
Ganymede has a high performance interconnect and parallel scratch storage system while Europa (at this time) does not. So, while Ganymede is great for running large, multi-node, single simulations, Europa is not. Europa is for many smaller jobs to run concurrently; this is high throughput computing.
In future posts, we will look at other tools we can use for managing our HTC workloads, but the first tool we’re going to look out is a SLURM feature. SLURM is the scheduling / queue system we use and it has support for the notion of job arrays.
Let’s start with a simple SLURM job array example
#!/bin/bash
#SBATCH -J test-arrays # Job name
#SBATCH -o job.%j.out # Name of stdout output file (%j expands to jobId)
#SBATCH -N 1 # Total number of nodes requested
#SBATCH -n 1 # Total number of tasks requested
#SBATCH --array=1-5 # array ranks to run
#SBATCH -t 01:30:00 # Run time (hh:mm:ss) - 1.5 hours
echo My SLURM Job Array Task ID is: "$SLURM_ARRAY_TASK_ID"
echo I ran on host: `hostname`
sleep 10
Here, we have a simply example job array script that will submit 5 jobs with this one sbatch command one 1 node and 1 core each. Each node gets a different value of $SLURM_ARRAY_TASK_ID
[css180002@europa arrays]$ sbatch job-array.sh
Submitted batch job 225
[css180002@europa arrays]$ squeue -u$USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
225_1 normal test-arr css18000 R 0:07 1 compute-1-1-8
225_2 normal test-arr css18000 R 0:07 1 compute-1-1-9
225_3 normal test-arr css18000 R 0:07 1 compute-1-1-11
225_4 normal test-arr css18000 R 0:07 1 compute-1-1-18
225_5 normal test-arr css18000 R 0:07 1 compute-1-1-19
[css180002@europa arrays]$ cat job.22*.out
My SLURM Job Array Task ID is: 5
I ran on host: compute-1-1-19
My SLURM Job Array Task ID is: 1
I ran on host: compute-1-1-8
My SLURM Job Array Task ID is: 2
I ran on host: compute-1-1-9
My SLURM Job Array Task ID is: 3
I ran on host: compute-1-1-11
My SLURM Job Array Task ID is: 4
I ran on host: compute-1-1-18
If you are abstract your workflow such that “all” it needs is to take an input task ID “rank”, you now have a very easy to use infrastructure for generating HTC workflows.
Note, a slurm job is run for each individual array rank. If you’d like to run many small jobs as part of one larger job, that requires a different tool such as TACC launcher.
By default, SLURM will try to run all of your array jobs at the same time (if the nodes are available).
If you’d like to change this behavior, you can include a percent sign to set a max number of concurrent jobs that will run, with --array=1-100%4 for instance.
You can also request custom task IDs with --array=1,3,5,7 and a custom step size with --array=1-7:2.
In your job scripts, several environment variables are available. The $SLURM_JOBID is sequential, and includes an underscore to denote both the initial submitted job and the task. The $SLURM_ARRAY_JOB_ID is the same for all jobs in the array while the $SLURM_ARRAY_TASK_ID is equal to the index supplied by the array option.