@csim, can we get FSL, SPM, AFNI, and FreeSurfer as modules on Europa? I think they are on ganymede already.
@ylwang, @lhan and @epongpipat (as far as i know, there might be more) are likely going to make use of them to do preprocessing/analysis as well (guys, feel free to chime in). The pipeline they implement may differ, but it will draw from similar tools.
You mentioned using a container system, for it, let me know how you think we could proceed with that 
Sure we can start working on this.
I was just talking to @solj about this and I think what we’re going to do is first get some modules set up and make it like ganymede so you guys can continue the same workflows for now.
Longer term, I’d like to move you guys to containers. That way, you can use the same tools and workflows on any system and longer term I suspect we’re going to move large portions of your workflows to TACC. Out at TACC you wont have quite the same level of software installation support and you’ll probably have to maintain your own stacks, so, might as well containerize right?
So, @ylwang, @lhan and @epongpipat out of the software @mychan mentioned, which ones are the most important? Are there any others you need?
I’m giving an SC20 tutorial on Tuesday and I’m way behind as usual. I dont have much bandwidth until middle / end of next week but I might be able to find some time this weekend or Monday. Additionally, @solj might have a bit of time here and there so just let us know what all you need and the priority and we will get to as we can.
Hi, the pipeline we are using has some software requirements as follows. @mychan already mentioned most of them. Just copy the list from their tutorial and post here.
A 64-bit Linux Operating System
FSL (http://fsl.fmrib.ox.ac.uk/) and configuration file properly sourced.
FreeSurfer (http://surfer.nmr.mgh.harvard.edu/fswiki/DownloadAndInstall/)
Connectome Workbench (http://www.humanconnectome.org/software/connectome-workbench.html)
Gradunwarp (latest release): https://github.com/Washington-University/gradunwarp/releases
MSM_HOCR (latest release): https://github.com/ecr05/MSM_HOCR/releases
Thanks!
Hi,
Yes please allow me to list the dependencies we need for our pipeline, which is pretty similar with @lhan has mentioned above:
Dependencies of UKBioBank Pipeline:
1 A 64-bit Linux Operating System
2 FSL (http://fsl.fmrib.ox.ac.uk/)
3 AFNI (https://afni.nimh.nih.gov/ )
4 Freesurfer (http://surfer.nmr.mgh.harvard.edu/fswiki/DownloadAndInstall/)
5 Anaconda/Miniconda (perhaps the system already has it)
6 git (>=2)
7 gfortran (installation only)
Thanks so much!
Hey Guys,
On Yile’s post (linked below) I gave some info on FSL, git, miniconda, and gfortran.
For the rest of the packages, we will get to them in the next week or so. @solj will be working on the module / from source compilations and I’m going to work on the containerization.
Ok let’s try freesurfer next.@solj is going to work on the binary compiles to provide traditional modules over time.
I’m going to provide containers and document how they are built. In a separate thread (and eventually via documentation and training) we will provide more background info on containers. But for now, just know that Containers are an OS-level virtualization paradigm that allow multiple, isolated user environments to run concurrently.
Using containers, we will be able to provide software faster and also provide it in a way that you can run the same containers on your laptop, desktop, on Europa, or out at TACC. It’s one big step towards creating reproducible workflows, reducing our support burden, and giving you the tools to support yourself. That’s one of the main themes of TRECIS.
Europa currently supports charliecloud and singularity containers. We also plan to support podman longer term.
For now, let’s focus on using Singularity. We are going to start with the Docker images available on the freesurfer dockerhub page. Going to tags, we see there are two versions available: 7.1.1 and 6.0
Let’s convert both of them to singularity images. To do this, we’re going to write singularity definition files.. Also, since freesurfer is licensed software, I’m also going to bake the license file into the singularity image. Finally, we also need to create a mount point inside our container for mounting $SCRATCH. $HOME is available by default when running singularity, but we have to pass extra flags to get access to $SCRATCH inside the container.
[csim@europa freesurfer]$ cat freesurfer-7.1.1.def
Bootstrap: docker
From: freesurfer/freesurfer:7.1.1
IncludeCmd: yes # Use the CMD as runscript instead of ENTRYPOINT
%setup
mkdir -p ${SINGULARITY_ROOTFS}/opt/freesurfer
mkdir -p ${SINGULARITY_ROOTFS}/scratch && chmod 777 ${SINGULARITY_ROOTFS}/scratch
chmod 644 license.txt
%files
license.txt /opt/freesurfer/.license
[csim@europa freesurfer]$ cat freesurfer-6.0.def
Bootstrap: docker
From: freesurfer/freesurfer:6.0
IncludeCmd: yes # Use the CMD as runscript instead of ENTRYPOINT
%setup
mkdir -p ${SINGULARITY_ROOTFS}/opt/freesurfer
mkdir -p ${SINGULARITY_ROOTFS}/scratch && chmod 777 ${SINGULARITY_ROOTFS}/scratch
chmod 644 license.txt
%files
license.txt /opt/freesurfer/.license
Now that we have our definition files, we can build our sif, our singularity image format file.
$ singularity build --fakeroot freesurfer-7.1.1.sif freesurfer-7.1.1.def
$ singularity build --fakeroot freesurfer-6.0.sif freesurfer-6.0.def
We now have two Singularity SIFs that can be used to invoke different versions of freesurfer.
Note: You can only run Singularity containers on compute nodes; if you need to run a container on the login nodes, you need to use podman or charliecloud
The containers that I publish will all be in /opt/ohpc/pub/containers. Now, let’s see how to run it.
First, let’s start an interactive slurm job and load the singularity module
[css180002@europa ~]$ srun -N1 --pty /bin/bash
[css180002@compute-1-1-1 ~]$ ml load singularity
[css180002@compute-1-1-1 ~]$
After that, you can call the singularity containers and either work with the container interactively, or have the container invoke a command and work directly on data on the host system.
[css180002@compute-1-1-1 ~]$ /opt/ohpc/pub/containers/freesurfer/sifs/freesurfer-7.1.1.sif freesurfer
FreeSurfer is a set of tools for the analysis and visualization
of structural and functional brain imaging data. FreeSurfer
also refers to the structural imaging stream within the
FreeSurfer suite.
Users should consult the online documentation available at:
http://surfer.nmr.mgh.harvard.edu
Alternatively, the 'recon-all' command help-text provides
adequate information to begin processing subject data, such
as the sample subject 'bert' found in the 'freesurfer/subjects'
directory. Type 'recon-all --help' to view this help-text.
Direct comments and questions to:
freesurfer@nmr.mgh.harvard.edu
You are running this version of FreeSurfer:
freesurfer-linux-centos7_x86_64-7.1.1-20200723-8b40551
[css180002@compute-1-1-1 ~]$ /opt/ohpc/pub/containers/freesurfer/sifs/freesurfer-7.1.1.sif recon-all --help
If you just invoke the container directly without passing arguments, it will put you inside the container but with access to the host filesystem.
Before running container directly.
[css180002@europa ~]$ srun -N1 --pty /bin/bash
[css180002@compute-1-1-1 ~]$ pwd
/home/css180002
[css180002@compute-1-1-1 ~]$ ls
containers hello.c local matlab miniconda src tmp
[css180002@compute-1-1-1 ~]$ ls /scratch/css180002/
git this_is_scratch tmp
[css180002@compute-1-1-1 ~]$ freesurfer
bash: freesurfer: command not found
After running container with no arguments.
[css180002@compute-1-1-1 ~]$ /opt/ohpc/pub/containers/freesurfer/sifs/freesurfer-7.1.1.sif
[css180002@compute-1-1-1 ~]$ pwd
/home/css180002
[css180002@compute-1-1-1 ~]$ ls
containers hello.c local matlab miniconda src tmp
[css180002@compute-1-1-1 ~]$ ls /scratch/
[css180002@compute-1-1-1 ~]$ freesurfer
FreeSurfer is a set of tools for the analysis and visualization
of structural and functional brain imaging data. FreeSurfer
also refers to the structural imaging stream within the
FreeSurfer suite.
Notice how $HOME is available inside the container but $SCRATCH isn’t? We can fix that by passing a -B and using the singularity shell command.
[css180002@compute-1-1-1 ~]$ singularity shell -B $SCRATCH:/scratch containers/freesurfer/sifs/freesurfer-7.1.1.sif
Singularity freesurfer-7.1.1.sif:~> ls /scratch/
git this_is_scratch tmp
Singularity freesurfer-7.1.1.sif:~> exit
[css180002@compute-1-1-1 ~]$
Alternatively, you can just set the $SINGULARITY_BINDPATH environment variable and never worry about it again.
[css180002@compute-1-1-1 ~]$ export SINGULARITY_BINDPATH="$SCRATCH:$SCRATCH"
[css180002@compute-1-1-1 ~]$ /opt/ohpc/pub/containers/freesurfer/sifs/freesurfer-7.1.1.sif
[css180002@compute-1-1-1 ~]$ ls $SCRATCH
git this_is_scratch tmpAFNI is available now as well
[css180002@europa ~]$ srun -N1 --pty /bin/bash
[css180002@compute-1-1-1 ~]$ ml load singularity
[css180002@compute-1-1-1 ~]$ /opt/ohpc/pub/containers/afni/sifs/afni-latest.sif
css180002@compute-1-1-1:~$ which afni
/opt/afni-latest/afni
css180002@compute-1-1-1:~$ exit
[css180002@compute-1-1-1 ~]$ /opt/ohpc/pub/containers/afni/sifs/afni-latest.sif afni_vcheck
This program was compiled with the following settings:
Version ID = AFNI_20.3.01
++ now fetching https://afni.nimh.nih.gov/pub/dist/AFNI.version
** Error fetching https://afni.nimh.nih.gov/pub/dist/AFNI.version!
Note, the error fetching the version number is expected as compute nodes cannot access the internet at this time
FSL, Freesurfer and AFNI are now all available as modules as well.
FreeSurfer
$ ml load freesurfer
$ freesurfer --version
FreeSurfer is a set of tools for the analysis and visualization
of structural and functional brain imaging data. FreeSurfer
also refers to the structural imaging stream within the
FreeSurfer suite.
Users should consult the online documentation available at:
http://surfer.nmr.mgh.harvard.edu
Alternatively, the 'recon-all' command help-text provides
adequate information to begin processing subject data, such
as the sample subject 'bert' found in the 'freesurfer/subjects'
directory. Type 'recon-all --help' to view this help-text.
Direct comments and questions to:
freesurfer@nmr.mgh.harvard.edu
You are running this version of FreeSurfer:
freesurfer-linux-centos7_x86_64-7.1.1-20200723-8b40551
FSL
(base) [css180002@europa ~]$ ml load fsl
(base) [css180002@europa ~]$ which fsl
/opt/ohpc/pub/unpackaged/apps/fslbinary/6.0.4/bin/fsl
AFNI
(base) [css180002@europa ~]$ ml load afni
(base) [css180002@europa ~]$ which afni
/opt/ohpc/pub/unpackaged/apps/afnibinary/21.0.06/afni