Here is the scenario:
In order to have my jobs run properly, I need to load a specific module.
Say, I load the module, then run a script that will submit several jobs at once, and within those jobs, I have code that depends on that specific module. Some of the jobs start running right away and some become pending. Therefore, not all the jobs have begun running during this session (when I have loaded the module).
So:
If I close the terminal - and therefore terminate the session that has the loaded module - while some jobs are still pending, will SLURM remember the environment when those jobs were submitted? i.e., will it know that when those jobs submit and that code runs, it was intended to be run with that module loaded?
I ask because I did not want to load the module in each script within each job, I wanted to do it once, and so now, I am sitting waiting for jobs to submit, but Iām wondering if I need to keep my machine on with that terminal open until all jobs are submitted.
Aaron,
When you submit jobs to compute nodes via SLURM, you should ensure that the executables will run on those nodes with the complete environment needed to work correctly, independent of one another and your login session. This means that you should load all the necessary modules, and specify any other environment variables in your SLURM jobscript accordingly. Doing so will ensure that you can logout from your login node after you have submitted your jobs and know that they will run with the environment they need.
Once a job is submitted, the environment is set and will only be altered by whatever is included in the submit script when the job starts on the node. Once the sbatch command completes and returns a job number then there is no longer any dependency on the environment of the parent shell. This is sensible if you think about it as otherwise, you could screw up pending jobs just by changing your environment in your login session. Derek is correct that best practice is probably to completely define the environment in the slurm script and use #SBATCH --export=NONE to minimize the environment that gets inherited from the parent session.