JupyterHub

Aus PC2 Doc
Dies ist die bestätigte sowie die neueste Version dieser Seite.
Zur Navigation springen Zur Suche springen

PC² JupyterHub[Bearbeiten]

The JupyterHub is only available for the Noctua1 system.

Access[Bearbeiten]

The JupyterHub server can be reached at the following address:
https://jh.noctua.pc2.uni-paderborn.de/

Note:
The JupyterHub is only available for the HPC system Noctua and can be accessed via VPN or on-site at the University of Paderborn.

JH login.png


Server Options[Bearbeiten]


Jh serveroptions.png

Spawner[Bearbeiten]

LocalSpawner[Bearbeiten]

The LocalSpawner spawns a notebook server on the jupyterhub host as a simple process. From the notebooks (or also from the terminal) slurm jobs can then be started. Magic Commands are used for this.

For more information see How-To -> Run slurm jobs via a notebook cell (LocalSpawner only)

NoctuaSpawner[Bearbeiten]

The NoctuaSpawner starts a notebook server within a slurm batch job. If you then start a terminal via the interface, you will get a shell on the compute node.

Once a Slurm job is queued and an estimated start time for that job is assigned, the Jupyter notebook server will also start at the specified time (even after a logout):

JupyterHub Estimated start time.png

Remote Desktop (noVNC)[Bearbeiten]

The RemoteDesktop Spawner creates a noVNC desktop environment in the web browser.
Included are the Slurm tools, /scratch/ and the Noctua software modules.

Jh desktop spawner.png










JupyterLab[Bearbeiten]

The JupyterLab is a modern web-based Jupyter notebook interface with many features:

  • Open more than one notebooks in one window
  • Integrated text editor
  • Terminals
  • Extension Manager
  • ...

See here for more information: https://jupyterlab.readthedocs.io/en/stable/getting_started/overview.html


PC2 JupyterLab.png









Jupyter Kernel[Bearbeiten]

Jupyter kernels can be loaded and used via Lmod (module). From the JupyterLab interface the kernels can be loaded via the graphical Lmod tool.

Another way to use Jupyter kernels is Singularity container. See "Singularity Container" which containers are installed with which Jupyter kernels.

Jupyterhub lmod kernel.png



















Singularity Container[Bearbeiten]

Singularity Container Purpose Installed kernel Packages
jupyter_scientifc_python Scientific Python Ecosystem IPython dask, pandas, numexpr, matplotlib, scipy, seaborn, scikit-learn, scikit-image, sympy, cython, patsy, statsmodel, cloudpickle, dill, numba, bokeh, sqlalchemy, hdf5, vincent, beautifulsoup, protobuf, xlrd, bottleneck, and pytables packages, ipywidgets, ipympl, Facets
jupyter_datascience data analysis from the Julia, Python, and R communities. IPython, IJulia, R The Julia compiler and base environment, IJulia to support Julia code in Jupyter notebooks, HDF5, Gadfly, RDatasets, everything in jupyter_scientific_python

See here for more information:

https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html

If you want to use your own Singularity Container, see here:

Create my own Singularity Container

How To[Bearbeiten]

Loading software modules in Jupyter[Bearbeiten]

To load software modules inside JupyterLab, click on the Lmod extension (Jupyterhub lmod logo.png) tab. Then you have the possibility to search, load and unload modules.

If you are using the Classic Notebook View, click on tab "Softwares" to load software modules.

Jupyterhub classic lmod.png









Add a path to the module environment in Jupyter[Bearbeiten]

To add a new path to search for modules goto Help -> Launch Classic Notebook inside JupyterLab. Then click on "Softwares" and open the settings with the tool icon.

Jupyterhub edit modulepath.png
















Default values on page "Spawner Options"[Bearbeiten]

It is possible to enter default values on the "Server Options" page, which will be applied after each page refresh.

For this purpose a predefined XML document can be placed under $HOME/.jupyter/pc2-jupyterhub/.

The XML document (pc2-jupyterhub.xml) looks like this:

 1<!-- 
 2JupyterHub - Paderborn Center for Parallel Computing (PC²)
 3
 4Description:
 5Set default options on https://jh.noctua.pc2.uni-paderborn.de to faster start a jupyter notebook server
 6
 7If some value does not exists or is invalid, it will be ignored :-)
 8
 9Just put this file in your $HOME-directory: $HOME/.jupyter/pc2-jupyterhub/pc2-jupyterhub.xml
10-->
11
12<JupyterHub_PC2>
13    <!-- absolute path of your notebook directory -->
14    <notebook_directory></notebook_directory>
15    <!-- absolute path of a singularity container (This container should exists in $HOME/.jupyter/pc2-jupyterhub/)  -->
16    <singularity_container></singularity_container>
17
18    <!-- Default values to start a slurm job with -->
19    <!-- The endtime will be automatically calculated (FORMAT: %H:%M) - Example: 1:00 -->
20    <runtime></runtime>
21    <partition></partition>
22    <account></account>
23    <reservation></reservation>
24    <prologue></prologue>
25</JupyterHub_PC2>

An example would be:

 1<JupyterHub_PC2>
 2    <!-- absolute path of your notebook directory -->
 3    <notebook_directory>/scratch/pc2-mitarbeiter/mawi/</notebook_directory>
 4    <!-- absolute path of a singularity container (This container should exists in $HOME/.jupyter/pc2-jupyterhub/)  -->
 5    <singularity_container>/upb/departments/pc2/users/m/mawi/.jupyter/pc2-jupyterhub/jupyter_julia.sif</singularity_container>
 6
 7    <!-- Default values to start a slurm job with -->
 8    <!-- The endtime will be automatically calculated (FORMAT: %H:%M) - Example: 1:00 -->
 9    <runtime>01:30</runtime>
10    <partition>batch</partition>
11    <account>hpc-lco-jupyter</account>
12    <reservation></reservation>
13    <prologue>
14export SINGULARITY_BIND="/scratch/pc2-mitarbeiter/mawi/:/mawi/:rw"
15export CUSTOM_VAR="Hello JupyterHub friend!"
16    </prologue>
17</JupyterHub_PC2>

Note:
If you don't want to store a fixed value for an attribute, just leave it blank.

Magic Commands: Run slurm jobs via a notebook cell (LocalSpawner only)[Bearbeiten]

  1. Load the extension:
1%load_ext slurm_magic

Available commands:

 %sacct
 %sacctmgr
 %salloc
 %sattach
 %%sbatch
 %sbcast
 %scancel
 %scontrol
 %sdiag
 %sinfo
 %snmap
 %sprio
 %squeue
 %sreport
 %srun
 %sshare
 %sstat
 %strigger
 %sview

Example running a batch job[Bearbeiten]

Jupyterhub cell sbatch.png

Use of a conda environment (NoctuaSpawner)[Bearbeiten]

  1. Load module anaconda using the Prologue input block
1module load anaconda/3-5.0.0.1
2conda activate /scratch/pc2-mitarbeiter/mawi/conda_env/

Create multiple notebook servers[Bearbeiten]

  1. Go to the Home-Section and create a named server.
    1. All created servers can be joined seperately.

Run my own Jupyter Notebook configuration[Bearbeiten]

  1. Start a notebook server with the LocalSpawner and open a terminal
  2. Run following commands to create a default config file:
1mkdir $HOME/.jupyter/
2cd $HOME/.jupyter/
3jupyter notebook --generate-config

Configuration file options:
https://jupyter-notebook.readthedocs.io/en/stable/config.html

Note:
Note that the configuration file will not be loaded if you are using a Singularity Container and do not mount your $HOME or manually set the $JUPYTER_CONFIG_DIR environment variable.

Create my own singularity container[Bearbeiten]

Skeleton Singularity Recipe file[Bearbeiten]

You can use following definition file to build your own Singularity container:

 1Bootstrap: docker
 2From: debian
 3
 4%post
 5
 6# base setup
 7apt update
 8apt install -y wget build-essential python3 python3-pip git procps nodejs npm vim
 9
10# install lua
11apt install -y lua5.3 lua-bit32 lua-posix liblua5.3-0 liblua5.3-dev tcl tcl-dev tcl8.6 tcl8.6-dev libtcl8.6
12
13# install Lmod
14wget https://github.com/TACC/Lmod/archive/refs/tags/8.4.tar.gz -P /opt/lmod/
15tar -xf /opt/lmod/8.4.tar.gz -C /opt/lmod/
16cd /opt/lmod/Lmod-8.4/
17./configure --prefix=/opt/apps/
18make install
19
20echo "module () \n{\n    eval \$(\$LMOD_CMD bash \"\$@\") && eval \$(\${LMOD_SETTARG_CMD:-:} -s sh)\n}" >> /etc/profile
21
22python3 -m pip install --upgrade pip
23python3 -m pip install batchspawner notebook
24
25# using version 2.2.9 for extension jupyterlab-lmod
26python3 -m pip install jupyterlab==2.2.9
27
28python3 -m pip install jupyterlmod
29jupyter labextension install jupyterlab-lmod
30
31%environment
32export LMOD_CMD=/opt/apps/lmod/lmod/libexec/lmod

Following software is included:

  • IPython Jupyter Kernel
  • JupyterLab
  • module command
  • JupyterLab-module GUI

Just add your commands under section %post

If you want to build your Singularity container from scratch, use following recipe:

 1Bootstrap: docker
 2From: debian
 3 
 4%post
 5        ##### BASE JUPYTER INSTALLATION #####
 6        apt update
 7        apt install -y python3 python3-pip git
 8        python3 -m pip install --upgrade pip
 9        python3 -m pip install notebook batchspawner jupyterlab
10
11        ##### INSTALL ADDITIONAL PACKAGES #####
12        # python3 -m pip install numpy

It is also possible to build singularity containers from the official jupyter docker stacks:

https://jupyter-docker-stacks.readthedocs.io/en/latest/

Here are more information on how to build a singularity container from DockerHub:

https://sylabs.io/guides/3.7/user-guide/build_a_container.html

Build container[Bearbeiten]

You can build you container on your host by executing following command:
sudo singularity build <container_name>.sif <recipe_file>

If you want to build the container on Noctua, you can build your container using the --remote option:
singularity build --remote <container_name>.sif <recipe_file>

Note: You need an account at https://sylabs.io/ to use the remote build feature.

Container Location[Bearbeiten]

Your new created container should be placed in your $HOME directory:

  • $HOME/.jupyter/pc2-jupyterhub/
    • If the directoy/ies does not exists, just create it with a simple mkdir.
  • Alternatively you can create a link from your $PC2PFS to your $HOME directory:
1[mawi@jupyterhub ~]$ ls -l /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif
2-rw-r--r--. 1 mawi pc2-mitarbeiter 0 Dec 17 07:53 /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif
3[mawi@jupyterhub ~]$ ln -s /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif $HOME/.jupyter/pc2-jupyterhub/

Note: All containers with type .sif will be automatically detected in $HOME/.jupyter/pc2-jupyterhub/

Share my jupyter notebook with other users[Bearbeiten]

  1. Go to: https://jh.noctua.pc2.uni-paderborn.de/services/nbviewer/
  2. Paste your URL, render it and share the URL with other users

Embed WebVNC as IFrame in a jupyter notebook[Bearbeiten]

1from IPython.display import IFrame
2IFrame(src="/user/<your-username>/desktop/", width='100%', height=700)

Creating presentations using the RISE extension[Bearbeiten]

  • The RISE extension is installed in the default singularity containers and is available if using the LocalSpawner.

Note:
You cannot use the RISE extension with JupyterLab. You need the classic notebook view.

  • From JupyterLab: Help -> Launch Classic Notebook

More information: https://rise.readthedocs.io/en/stable/

Mount additional paths into a singularity container[Bearbeiten]

  • You can use the input block "Prologue" to do this.
    • Just export following environment variable:
1export SINGULARITY_BIND="SOURCE:DEST:OPTS,SOURCE:DEST:OPTS,..."
2# Example:
3# export SINGULARITY_BIND="/scratch/pc2-mitarbeiter/mawi/:/mawi/:rw"
4# Then /scratch/pc2-mitarbeiter/mawi/ would be mount to /mawi/ (read & write) into the container.

See here for more information: https://sylabs.io/guides/3.7/user-guide/bind_paths_and_mounts.html

Troubleshooting[Bearbeiten]

"Terminals unavailable"[Bearbeiten]

  • If you have terminado installed in your $HOME directory (pip3 install --user), please make sure that the version of terminado is at least 0.8.3.
    • How to upgrade: pip3 install --user --upgrade terminado
  • User specific config file:
    • The following option may be responsible for this: terminals_enabled

JupyterLab + HTML: CSS and JavaScript files are not trusted[Bearbeiten]

PC² Support[Bearbeiten]

If you have any other problems that won't be solved, please contact the pc2-support.