I’d like to be able to extract from the slurm accounting database (or through another means) information about applications run by users, and/or application modules loaded by users, over a given time period. I’ve investigated the usual report options provided with slurm but am not finding any way to do this. Does anyone have experience with this; tools that might help or otherwise?
Thank you!
Sincerely,
torey
As far as I know, applications run by users are not recorded in slurm accounting database. What we do is we log/track all module loads that happen on the system. When it logs it also records if the module was loaded on a console or within a job, and if it was within a job, a jobid is also recorded in module load logs. This helps us tie applications to jobid and users. I have not found any out of the box way to do this. I will be happy to hear what others have done here. Hope this helps.
Torey,
I agree with Amit that this is not something Slurm would typically be aware of. We do what Amit does and our Lmod configuration is very similar to the one at
https://lmod.readthedocs.io/en/latest/300_tracking_module_usage.html
The example shows how get an environment variable to use, so setting the jobID would certainly be possible as would grabbing other values. We use pretty much what is shown for the documentation, but now that Amit mentions it, including the jobID might be a really good idea (setting jobID to ‘login’ and overwriting if the module is loaded within a job).
We also then run a script on the first of the month that generates text reports for the previous month of each module’s load counts, the first report sorted by module name and the second by count. Those are accessible on a web page and are what most people want. The data is also available for special queries if people want something different.
We also record the user, in case we find a bug in software or otherwise need to contact users of a particular package. We use this very sparingly.
It might be interesting to see more of what other people save about module loading and what they do with it as far as regular reporting.
Now you’ve got me thinking about it, one other thing that would be useful would be to record in the message sent to syslog is whether the module is being loaded directly or as an auto-load because it is a dependency of another module – that is, being loaded by depends_on() in Lmod-speak.
Thank you, Amit and Bennet! I very much appreciate your responses, shares and suggestions.
Sincerely,
torey