We would like to install some most popular (or hard to install) Julia packages into a central location to avoid having multiple instances of the same package in user home directories. I could not find any official Julia guidance for this. Do you have any recommendations on how this could be done in a multi-user cluster environment?
After much searching I did find a way to install Julia in a non-standard location that can then be used to install other packages. I did not do this for a multi-user cluster environment but hopefully what I found could be helpful.
The key for me was to add to the default paths that Julia looks for to install packages. More specifically I added to the JULIA_LOAD_PATH and JULIA_DEPOT_PATH environment variables. I think the documentation does a pretty good job at describing them: https://docs.julialang.org/en/v1/manual/environment-variables/
Once I appended the directory I chose as my central package location to both of those environment variables, I changed the permissions so that other accounts could access that directory. After that I was able to install and run packages from that central location.
I imagine that as long as folks can access the directory and set the Julia environment variables correctly then they should be able to reference the packages in their scripts and programs.
One limitation that might affect this on a larger scale is managing the permissions for the different packages. Since my use case was small, I didn’t need worry too much about best practices or adding additional packages.
We use spack for the base install of julia Package List — Spack 0.16.1 documentation as it generates a default module file that manages environment variables such as LD_LIBRARY_PATH, PATH, MANPATH, CMAKE_PREFIX_PATH so users can load, unload various versions.
We have not yet investigated the issue of building additional modules either in the central location or in project, home directory space.
We’ve had a lot of trouble setting things so that users can access a central package location but also install packages into their own user-owned directories. When we set JULIA_DEPOT_PATH to be a colon-separated list (i.e., the standard format for PATH, LD_LIBRARY_PATH, etc.), Julia seems to ignore all but the first element.
If you are the person who installed Julia on the cluster (or have a friendly relationship) you should be able to add packages to the installation that will be available to all users in the same way generally that you can use pip to install packages for Python users.
$ julia
julia> ]
(v1.0) pkg> add IJulia
… installs Conda, IJulia, and 11 dependencies …
^C
julia>