We have recently got the new HPC cluster. I’m exploring to standarize the application stack for end-users using Lmod, Spack, EasyBuild etc., Can you suggest best pratices and what are the recommended hierarchy of tools/compilers/libraries/runtimes to provide in user environments.
One place to start is to look at the Linux Foundation OpenHPC project - OpenHPC maintains and community-supported repository of common HPC cluster software and tools. See https://openhpc.community/ - OpenHPC recently released version 2.0 of their distribution, which supports CentOS 8.2 or OpenSUSE Leap 15 as the base OS on x86_64 or aarch64 hardware. Take a look at the installation recipe at https://openhpc.community/downloads/
We have started down the path of using Spack for most of our packages. For some of our binary (usually commercial tools) we are still using Modules, but I am pretty convinced that Spack is the way to go long term.