Hi experts,
Over the last 48 hours, my jobs (which were previously running successfully) have failed to run. I submitted three jobs using Slurm and Launcher, where I aimed to distribute a compiled MATLAB executable to 9 different nodes. However, each job timed out.
When looking at the job output files, I saw none of the usual signs of my code running–whenever the compiled MATLAB file begins, it pastes “Starting process on data XYZ” into the terminal. This line was conspicuously absent from all job files. I did see the usual Launcher boilerplate (Working directory, Processes per host, etc), as well as “starting job on compute-X-Y-Z”, suggesting that Launcher was able to connect to the nodes.
Thinking that something might be wrong with my compiled files, I used command line to load MATLAB Compile Runtime and then ran the same executable file. I had no issues with this task and saw the usual output from MATLAB in moments.
I then re-submitted today, skipping launcher entirely and only requesting one node. The job still timed out, but I did notice one line of the usual MATLAB output–the very first one. I do not know when that line output, unfortunately.
I would appreciate any insight on how to proceed!
Matthew