I am building some singularity images from docker images and this used to take just a few minutes if I remember correctly.
I usually run this build on the login node directly on the scratch drive since it would only take a few minutes to run, but now it seems to take two hours for the same images. I’m really not sure what’s causing the slow down. I mean, there’s only one job on one node in the queue running, so it’s probably not that.
Is there something I can do to speed this up? Maybe the login node needs a restart?
We have identified the cause of the I/O issues on Europa. We know which user is causing it but he has a dissertation deadline to finish a large batch of computations. We’re going to let it slide for another few days and then we will crack down.
I have a few ideas on how to remedy the situation but it’s going to take some time to get in place.
Please bear with us and if the problem is still happening middle of next week, let us know.
Thanks for getting back to me. No worries. It’s an inconvenience, but I can work around it until they are done.
I really just wanted to make sure it was a known issue or if there was a better way for me to run my stuff.
I tried building the image on a node in the tmp directory, but I didn’t get quite the speedup I was expecting. Do you have some ideas on how I can build faster?
Never mind. I created a script and it’s MUCH faster if I just batch the build on a node. I guess it was the interactive part that was slowed.
Now it builds in 4 minutes instead of 2+ hours.