I would like to create some instructions and documentation for using Julia on our HPC at the same level of detail as we have for Conda, specifically with regard to creating environments and installing distributed memory packages like mpi4py or Rmpi.
Does anyone know the Julia equivalent of:
conda env create --prefix /path/to/env -f environment.yml
And do you know how to install Julia’s “Distributed” package with the system’s scheduler aware MPI?
Q1. A Conda-like environment for Julia?
A1. There does not seem to be a Conda-like package- and environment-management system for Julia. Julia has Pkg, which provides environments within the Julia REPL (as opposed to Conda which provides environments in the shell). There is also Conda.jl which provides Conda-like semantics and management of channels containing packages.
Q2. Integration with cluster job- and resource-manager?
A2. Here is an example of Slurm integration. One could make it “nicer” by replacing the hardcoded numbers with environment variables like $SLURM_JOB_NUM_NODES, $SLURM_TASKS_PER_NODE, etc.
I found a simple way to deal with the ‘different Julia’ versions. It is not brilliant, but it works.
By default, Julia will install the packages in the home directory in ~/.julia.
As to not fill the home directory, before installing packages, you set the location for the installs by setting an environment variable JULIA_DEPOT_PATH.
So if someone wants to have different ‘environments’ with different versions of Julia and the respective packages, they can just set that depo path differently before doing whatever Julia thing they need to do.