Running Jupyter Notebooks on Slurm Nodes
So we want to crunch some big numbers, but we also want the flexibility and quick iterations that Jupyter gives us.
Setup the Conda Environment
Using VS Code, login to sysbio.
- Create a conda environment with jupyter-lab
note
You don't need to recreate the environment everytime, you can just activate it in step 1
login $ ml load anaconda3
login $ conda create -n ag-jupyter -c conda-forge jupyterlab
- Login to an interactive job
login $ srun --ntasks-per-node 2 --pty bash
compute-1 $ ml load anaconda3
compute-1 $ source activate ag-jupyter
- Launch Jupyterlab
compute-1 $ jupyter-lab --no-browser --port 8888
- Open another terminal and login to the compute node with port forwarding.
login $ ssh sysbio-1 -L 8888:localhost:8888
- Go back to the first terminal and copy the jupyterlab link into your local browser. It should look something like
http://localhost:8888/lab?token=4d08d209468541b333109fc4f957c8b3d1c97bed59072148
info
Alternatively you can paste the url in VS Code to connect to it as well.
What just happened?
compute node port 8888 -> login node port 8888 -> local computer port 8888