Running Jupyter Notebook on the compute node using Slurm
The step to run Jupyter Notebook on the compute node of the Cluster using Slurm:
step1: SSH to cluster ssh <<usrname>>@<hostname>> step2: Create a Slurm sbatch file vi jupyter_cn_run.sub (ref. sample slurm batch file) step3: Submit this sbatch to Slurm sbatch jupyter_cn_run.sub step4: Check that your job is running squeue -u $USER step5: Once it is running you can continue. Wait for a few minutes for Jupyter to start. Check the log output file to find out the IP we need to use to create an SSH tunnel cat ~/jupyter.log step6: Then on your laptop, open a new Terminal and create an SSH tunnel ssh -L8888:<<IP>>:8888 <<username>>@<<hostname>> Important: You must replace the IP address in the command below with your IP address from step5 step7: On your laptop open a browser window and you can then browse to the link from the log file of step5 to open the Jupyter Notebook http://127.0.0.1:8888/<<token>>
Sample Slurm batch file (jupyter_cn_run.sub)
#!/bin/bash # #SBATCH --job-name=jupyter_run #SBATCH --partition=short #SBATCH --ntasks=1 #SBATCH --time=00:05:00 #SBATCH --mem=10GB #SBATCH --output=jupyter.log ##SBATCH --error=jupyter.err # source /share/anaconda3/bin/activate export XDG_RUNTIME_DIR="" # echo "node ip: " `hostname -i` # date jupyter lab --no-browser --ip=0.0.0.0 --port=8888 date