I’m a doctoral student working for Prof. KJ Cho in the Materials Science and Engineering Department. Our group studies materials for energy storage, electronic devices, and quantum computing. I focus mainly on Li-ion and related battery technologies.
Our primary workhorse is density functional theory. We almost exclusively use VASP, but occasionally other packages like Quantum Espresso, SIESTA, Octopus are needed. For modelling problems outside the scope of DFT, we may use other methods such as classical molecular dynamics or kinetic monte carlo (KMC). Our workflow entails intensive DFT calculations on Ganymede or TACC, followed by post processing using (usually python) scripts.
Our calculations depend heavily on total memory and memory bandwidth, so we often find ourselves using many nodes per calculation. This has us competing with ourselves and other users for space, and stretching the limits of the hardware.
Yes, I’ve tried the new nodes. I actually have a detailed writeup that I’ve been meaning to send. Is there a way to attach documents here (.pptx), or should I just email it to you?
I expected I would need to rebuild VASP before I could run it on the AMD nodes due to Intel-specific hardware optimizations built into the Intel fortran compiler, MKL, and IMPI. My jobs do fail if I try to use more than one node due to incompatibilities between the AMD hardware and IMPI, but otherwise the Ganymede VASP module runs just fine on the AMD nodes. Moreover, performance is comparable to the XAP nodes. This is unexpected, and means (among other things) that VASP isn’t making use of any AVX instruction sets. Which linear algebra dependencies are baked into the module?
Ganymede’s VASP was compiled on the login node so it has AVX2 but doesn’t have AVX512. I didn’t realize you were using the vasp module binaries for the testing. I guess I should have; my bad.
We should definitely recompile the VASP binaries specifically for the XAP nodes (and the AMD nodes) and then test them both. I should be able to find some time to do the multiple vasp compiles early next week.