Hi y’all, I spent a bunch of time (with @csim 's help) getting my singularity containers to build and run on Europa. Here’s a post about what I learned / a small guide on how to build singularity containers that will run.
I attempted to run some singularity containers I built on my desktop computer which contained compiled code. These ended up not working because of architecture incompatibility with the Sandy Bridge nodes that Europa is made of. I also found that I couldn’t build the singularity sif with my default settings on the login nodes either since they’re newer than the compute nodes. So, step 1 was compile my code with -march=sandybridge as a compiler option. I simply added this as a cmake option (and then as a pip option since my code is python and C++ interfaced).
In order to run a singularity sif on Europa, you also need a base image with a compatible version of MPI. For my def file, I used TACC’s Ubuntu18-mvapich2.3-ib image. They also provide a Centos7-mvapich2.3-ib image. IB is important. psm2 will not work. So, the header of my def file looks like this:
Bootstrap: docker
From: tacc/tacc-ubuntu18-mvapich2.3-ib
Stage: build
@csim also recommends these ohpc singularity bases, but I have not tried them yet.
Next, Europa loads openmpi4 by default. To use mvapich2, run the command module swap openmpi4 mvapich2. You can put this in your.bashrc file if you want it to run every time, but I just put it in my SLURM launch scripts after the module load singularity.
Singularity looks at the environment variables SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR. At minimum, you must set SINGULARITY_TMPDIR to something besides /tmp. I made a $SCRATCH/tmpdir directory and set it in my .bashrc file so it was always set correctly for building.
To build a Singularity sif on Europa, you need to use the --fakeroot option:
singularity build --fakeroot <sif file> <def file>
This will work on the login nodes, but did not work on the compute nodes last I tried it.
Finally, once your sif is built, if you’re running your job out of $SCRATCH you need to bind $SCRATCH to $SCRATCH to make relative pathing work correctly:
singularity exec --bind $SCRATCH:$SCRATCH <command>
Let me know if you have questions and I’ll try to help!
Georgia