How can I access a jupyter notebook that I start on an interactive job? Sometimes I want to prototype some analysis using a notebook before I translate it into a python script, but I need HPC resources (like a fancy GPU for example) to do the prototyping.
Mike, my advice for something like this is to encourage your site admins to install OpenOnDemand - which has built-in support for running Jupyter notebooks on compute nodes, without having to deal with setting up manual tunnels.
However, the tunneling method does work. It’s what we used to recommend at our site for connecting to desktops before we created OOD.
This can be accomplished with some ssh tunneling. First, start an interactive job asking for the resources you need then load the necessary modules for your code. Then start a notebook like this jupyter notebook --no-browser --port=5556. This will start a notebook without launching a browser and use port 5556. Then open another terminal on your local computer and run this command ssh -L 9999:localhost:9999 username@hpc_cluster ssh -L 9999:localhost:5556 -N username@hpc_node. This will open a tunnel from your local computer to the node, mapping the nodes port 5556 to your 9999 port. Then on your local computer, open your favorite web browser and got to localhost:9999 and you should see your notebook!
This method isn’t bullet proof since if these ports are in use, it won’t work and you will have to switch to an unused port. This method also won’t work if the cluster won’t let you ssh onto a node directly (I’ve found most places let you as long as you have a job running on the node).
Here at Berkeley, we have a visualization node from which one can run a browser and access the notebook (which is running on the compute node) without the user having to set up the tunnel. But I’m not advocating for this compared to Open OnDemand.