I’m a PhD student at UT Dallas working on a combination of first principle quantum (Density functional theory using VASP or GAMESS (US)) and Monte Carlo particle transport simulations.
Experience with HPC:
I still consider myself as a novice in using UNIX-like systems and HPC systems. I submit a combination of parallel and many serial jobs. I mostly used TACC stampede2 (Intel Xeon Platinum 8160) and lonestar5 (Xeon E5-2690 v3) systems. I have experience in using other clusters resources such as Cray XC40/50 systems.
My Workflow:
I use my personal computer as a server that connects/manages (multiple ssh’s) multiple remote accesses via visual studio code (yes I use IDEs I’m a little embarrassed by that). From there I use a combination of python3 and shell scripts to automate various parallel and multiple serial jobs simulations. I use mostly use python3 to visualize/post-process results from simulations I run.
Challenge:
My biggest challenge for me right now is educating my self in performing better data analysis/visualizations and just learn better techniques in using computational clusters. I know I have a lot to learn in using computational clusters.
Can I quickly jump in and ask what’s wrong about IDEs? They are pretty much essential when creating complicated code. VS code/Atom/Jetbrains (Pycharm, IntelliJ)/Eclipse are all tools that anyone writing code can benefit from. I feel sorry for any student that joins a computational project and walks away/finishes without having learned about IDEs. Nothing to be embarrassed about.
Of course your graphical IDE doesn’t run on the HPC system, but your HPC system is not the place to develop code. That should IMHO happen before you upload the code.
All the best, Claude (worked as professional software engineer besides science).
Historically, in the HPC space there have been a lot of snobs who tell you you should only use the terminal and only use their text editor of choice (where that is usually vim or emacs). I’m one of them! (or used to be; I’m working on that. )
I from the generation that doesn’t use an IDE per se but in reality most of us who still “don’t use an IDE” just use simple, console-based tools to “create” our own remote / terminal-based IDE. Once you set it up, there’s very little difference between a modern vim + assorted tools setup and a GUI IDE.
If you do use an IDE on your desktop/laptop, I encourage you to still learn a unix text editor.
Doing the majority of your development via an IDE is fine (and probably optimal) but having to copy files back and forth to edit any text file is not. (I see this quite often).
Here’s some other things that make remote work easier and in some cases transparent.
If any of these tools / technologies are of interest (to anyone), let us know and we will develop trainings and documentation.
Finally, once you get up and running on europa, I’m happy to sit down and take a look at your workloads and workflow and suggest ways on making it better.
It def has its place in a good science workflow that requires coding a lot.