JupyterHub

Aus PC2 Doc
Wechseln zu: Navigation, Suche

PC² JupyterHub

The JupyterHub is only available for the Noctua system.

Access

The JupyterHub server can be reached at the following address:
https://jh.noctua.pc2.uni-paderborn.de/

Note:
The JupyterHub is only available for the HPC system Noctua and can be accessed via VPN or on-site at the University of Paderborn.

JH login.png


Server Options


Jh serveroptions.png

Spawner

LocalSpawner

The LocalSpawner spawns a notebook server on the jupyterhub host as a simple process. From the notebooks (or also from the terminal) slurm jobs can then be started. Magic Commands are used for this.

For more information see How-To -> Run slurm jobs via a notebook cell (LocalSpawner only)

NoctuaSpawner

The NoctuaSpawner starts a notebook server within a slurm batch job. If you then start a terminal via the interface, you will get a shell on the compute node.

Once a Slurm job is queued and an estimated start time for that job is assigned, the Jupyter notebook server will also start at the specified time (even after a logout):

JupyterHub Estimated start time.png

Remote Desktop (noVNC)

The RemoteDesktop Spawner creates a noVNC desktop environment in the web browser.
Included are the Slurm tools, /scratch/ and the Noctua software modules.

Jh desktop spawner.png










JupyterLab

The JupyterLab is a modern web-based Jupyter notebook interface with many features:

  • Open more than one notebooks in one window
  • Integrated text editor
  • Terminals
  • Extension Manager
  • ...

See here for more information: https://jupyterlab.readthedocs.io/en/stable/getting_started/overview.html


PC2 JupyterLab.png









Jupyter Kernel

Jupyter kernels can be loaded and used via Lmod (module). From the JupyterLab interface the kernels can be loaded via the graphical Lmod tool.

Another way to use Jupyter kernels is Singularity container. See "Singularity Container" which containers are installed with which Jupyter kernels.

Jupyterhub lmod kernel.png



















Singularity Container

Singularity Container Purpose Installed kernel Packages
jupyter_scientifc_python Scientific Python Ecosystem IPython dask, pandas, numexpr, matplotlib, scipy, seaborn, scikit-learn, scikit-image, sympy, cython, patsy, statsmodel, cloudpickle, dill, numba, bokeh, sqlalchemy, hdf5, vincent, beautifulsoup, protobuf, xlrd, bottleneck, and pytables packages, ipywidgets, ipympl, Facets
jupyter_datascience data analysis from the Julia, Python, and R communities. IPython, IJulia, R The Julia compiler and base environment, IJulia to support Julia code in Jupyter notebooks, HDF5, Gadfly, RDatasets, everything in jupyter_scientific_python

See here for more information:

https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html

If you want to use your own Singularity Container, see here:

Create my own Singularity Container

How To

Default values on page "Spawner Options"

It is possible to enter default values on the "Server Options" page, which will be applied after each page refresh.

For this purpose a predefined XML document can be placed under $HOME/.jupyter/pc2-jupyterhub/.

The XML document (pc2-jupyterhub.xml) looks like this:

 1 <!-- 
 2 JupyterHub - Paderborn Center for Parallel Computing (PC²)
 3 
 4 Description:
 5 Set default options on https://jh.noctua.pc2.uni-paderborn.de to faster start a jupyter notebook server
 6 
 7 If some value does not exists or is invalid, it will be ignored :-)
 8 
 9 Just put this file in your $HOME-directory: $HOME/.jupyter/pc2-jupyterhub/pc2-jupyterhub.xml
10 -->
11 
12 <JupyterHub_PC2>
13     <!-- absolute path of your notebook directory -->
14     <notebook_directory></notebook_directory>
15     <!-- absolute path of a singularity container (This container should exists in $HOME/.jupyter/pc2-jupyterhub/)  -->
16     <singularity_container></singularity_container>
17 
18     <!-- Default values to start a slurm job with -->
19     <!-- The endtime will be automatically calculated (FORMAT: %H:%M) - Example: 1:00 -->
20     <runtime></runtime>
21     <partition></partition>
22     <account></account>
23     <reservation></reservation>
24     <prologue></prologue>
25 </JupyterHub_PC2>

An example would be:

 1 <JupyterHub_PC2>
 2     <!-- absolute path of your notebook directory -->
 3     <notebook_directory>/scratch/pc2-mitarbeiter/mawi/</notebook_directory>
 4     <!-- absolute path of a singularity container (This container should exists in $HOME/.jupyter/pc2-jupyterhub/)  -->
 5     <singularity_container>/upb/departments/pc2/users/m/mawi/.jupyter/pc2-jupyterhub/jupyter_julia.sif</singularity_container>
 6 
 7     <!-- Default values to start a slurm job with -->
 8     <!-- The endtime will be automatically calculated (FORMAT: %H:%M) - Example: 1:00 -->
 9     <runtime>01:30</runtime>
10     <partition>batch</partition>
11     <account>hpc-lco-jupyter</account>
12     <reservation></reservation>
13     <prologue>
14 export SINGULARITY_BIND="/scratch/pc2-mitarbeiter/mawi/:/mawi/:rw"
15 export CUSTOM_VAR="Hello JupyterHub friend!"
16     </prologue>
17 </JupyterHub_PC2>

Note:
If you don't want to store a fixed value for an attribute, just leave it blank.

Magic Commands: Run slurm jobs via a notebook cell (LocalSpawner only)

  1. Load the extension:
1 %load_ext slurm_magic

Available commands:

 %sacct
 %sacctmgr
 %salloc
 %sattach
 %%sbatch
 %sbcast
 %scancel
 %scontrol
 %sdiag
 %sinfo
 %snmap
 %sprio
 %squeue
 %sreport
 %srun
 %sshare
 %sstat
 %strigger
 %sview

Example running a batch job

Jupyterhub cell sbatch.png

Use of a conda environment (NoctuaSpawner)

  1. Load module anaconda using the Prologue input block
1 module load anaconda/3-5.0.0.1
2 conda activate /scratch/pc2-mitarbeiter/mawi/conda_env/

Create multiple notebook servers

  1. Go to the Home-Section and create a named server.
    1. All created servers can be joined seperately.

Run my own Jupyter Notebook configuration

  1. Start a notebook server with the LocalSpawner and open a terminal
  2. Run following commands to create a default config file:
1 mkdir $HOME/.jupyter/
2 cd $HOME/.jupyter/
3 jupyter notebook --generate-config

Configuration file options:
https://jupyter-notebook.readthedocs.io/en/stable/config.html

Note:
Note that the configuration file will not be loaded if you are using a Singularity Container and do not mount your $HOME or manually set the $JUPYTER_CONFIG_DIR environment variable.

Create my own singularity container

Skeleton Singularity Recipe file

You can use following definition file to build your own Singularity container:

 1 Bootstrap: docker
 2 From: debian
 3 
 4 %post
 5 
 6 # base setup
 7 apt update
 8 apt install -y wget build-essential python3 python3-pip git procps nodejs npm vim
 9 
10 # install lua
11 apt install -y lua5.3 lua-bit32 lua-posix liblua5.3-0 liblua5.3-dev tcl tcl-dev tcl8.6 tcl8.6-dev libtcl8.6
12 
13 # install Lmod
14 wget https://github.com/TACC/Lmod/archive/refs/tags/8.4.tar.gz -P /opt/lmod/
15 tar -xf /opt/lmod/8.4.tar.gz -C /opt/lmod/
16 cd /opt/lmod/Lmod-8.4/
17 ./configure --prefix=/opt/apps/
18 make install
19 
20 echo "module () \n{\n    eval \$(\$LMOD_CMD bash \"\$@\") && eval \$(\${LMOD_SETTARG_CMD:-:} -s sh)\n}" >> /etc/profile
21 
22 python3 -m pip install --upgrade pip
23 python3 -m pip install batchspawner notebook
24 
25 # using version 2.2.9 for extension jupyterlab-lmod
26 python3 -m pip install jupyterlab==2.2.9
27 
28 python3 -m pip install jupyterlmod
29 jupyter labextension install jupyterlab-lmod
30 
31 %environment
32 export LMOD_CMD=/opt/apps/lmod/lmod/libexec/lmod

Following software is included:

  • IPython Jupyter Kernel
  • JupyterLab
  • module command
  • JupyterLab-module GUI

Just add your commands under section %post

If you want to build your Singularity container from scratch, use following recipe:

 1 Bootstrap: docker
 2 From: debian
 3  
 4 %post
 5         ##### BASE JUPYTER INSTALLATION #####
 6         apt update
 7         apt install -y python3 python3-pip git
 8         python3 -m pip install --upgrade pip
 9         python3 -m pip install notebook batchspawner jupyterlab
10 
11         ##### INSTALL ADDITIONAL PACKAGES #####
12         # python3 -m pip install numpy

It is also possible to build singularity containers from the official jupyter docker stacks:

https://jupyter-docker-stacks.readthedocs.io/en/latest/

Here are more information on how to build a singularity container from DockerHub:

https://sylabs.io/guides/3.7/user-guide/build_a_container.html

Build container

You can build you container on your host by executing following command:
sudo singularity build <container_name>.sif <recipe_file>

If you want to build the container on Noctua, you can build your container using the --remote option:
singularity build --remote <container_name>.sif <recipe_file>

Note: You need an account at https://sylabs.io/ to use the remote build feature.

Container Location

Your new created container should be placed in your $HOME directory:

  • $HOME/.jupyter/pc2-jupyterhub/
    • If the directoy/ies does not exists, just create it with a simple mkdir.
  • Alternatively you can create a link from your $PC2PFS to your $HOME directory:
1 [mawi@jupyterhub ~]$ ls -l /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif
2 -rw-r--r--. 1 mawi pc2-mitarbeiter 0 Dec 17 07:53 /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif
3 [mawi@jupyterhub ~]$ ln -s /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif $HOME/.jupyter/pc2-jupyterhub/

Note: All containers with type .sif will be automatically detected in $HOME/.jupyter/pc2-jupyterhub/

Share my jupyter notebook with other users

  1. Go to: https://jh.noctua.pc2.uni-paderborn.de/services/nbviewer/
  2. Paste your URL, render it and share the URL with other users

Embed WebVNC as IFrame in a jupyter notebook

1 from IPython.display import IFrame
2 IFrame(src="/user/<your-username>/desktop/", width='100%', height=700)

Creating presentations using the RISE extension

  • The RISE extension is installed in the default singularity containers and is available if using the LocalSpawner.

Note:
You cannot use the RISE extension with JupyterLab. You need the classic notebook view.

  • From JupyterLab: Help -> Launch Classic Notebook

More information: https://rise.readthedocs.io/en/stable/

Mount additional paths into a singularity container

  • You can use the input block "Prologue" to do this.
    • Just export following environment variable:
1 export SINGULARITY_BIND="SOURCE:DEST:OPTS,SOURCE:DEST:OPTS,..."
2 # Example:
3 # export SINGULARITY_BIND="/scratch/pc2-mitarbeiter/mawi/:/mawi/:rw"
4 # Then /scratch/pc2-mitarbeiter/mawi/ would be mount to /mawi/ (read & write) into the container.

See here for more information: https://sylabs.io/guides/3.7/user-guide/bind_paths_and_mounts.html

Troubleshooting

"Terminals unavailable"

  • If you have terminado installed in your $HOME directory (pip3 install --user), please make sure that the version of terminado is at least 0.8.3.
    • How to upgrade: pip3 install --user --upgrade terminado
  • User specific config file:
    • The following option may be responsible for this: terminals_enabled

JupyterLab + HTML: CSS and JavaScript files are not trusted

PC² Support

If you have any other problems that won't be solved, please contact the pc2-support.