I am developing a containerized application that can run on end user’s workstations/personal computers and on HPC clusters. I am thinking about usability as it seems Docker is pretty easy to install and manage while also considering that our HPC environment only employs Singularity on its clusters plus Singularity seems better suited to the HPC environment and the work we are doing. It seems that Singularity may be a bit more difficult to install on Windows and MacOS.
So, I am wondering if it’s better to develop the application using Docker containers from a usability standpoint so that users might run the application as Docker containers on their local systems. And then for those who want to run the application in an HPC environment, run the application as Singularity containers using the Docker images? Are there any performance losses to this type of workflow? Are there any roadblocks or other considerations that might crop up? I am super interested in others opinions, perspectives, and experiences!
The e4s team have had good experience getting close to “bare metal” performance with containers (Docker, Singularity, and Charliecloud). Using a container within a container will probably add some overhead, but the only real way to know is to measure. If you have some standardized performance tests on your application, then I would compare bare metal against Singularity and against Singularity+Docker. Definitely let us know how it goes!
On our HPC, we have ‘experimental’ support for Singularity, meaning it is installed, but we don’t have the expertise to help users install or debug their own containers.
You don’t run Docker inside Singularity, rather you can convert a Docker image to a Singularity image. We have had many folks successfully do that. To run on HPC, you would do something like:
module load singularity
singularity build myimage.simg docker://pathto/mydockerimage
The times that it didn’t work were with homegrown (rather than curated and published) Docker containers…so I think if you install something with Docker and do the testing to confirm that it actually works on an HPC after doing “singularity build”, then - hopefully(!) - you’re good.