I am using a script that calls a Singularity container. When it does, the script appears to freeze and eventually times out. Why is this happening and what can be done about it?
If the amount of memory used by the Singularity container exceeds the memory available for the job, the script will hang. Expected SLURM behavior is to terminate the job when its memory is exceeded, however the use of a Singularity container defies this expectation.
To more definitively diagnose the issue, try using the seff command after the script has timed out or otherwise ended. The seff command will display memory usage of the job. If the job had 100% usage, or very nearly 100% usage, it is likely the container is using too much memory.
To resolve the issue, please find ways to reduce memory usage or increase memory allocated to the job.