JupyterHub

Aus PC2 Doc
Wechseln zu: Navigation, Suche

PC² JupyterHub

The JupyterHub is only available for the Noctua system.

Access

The JupyterHub server can be reached at the following address:
https://jh.noctua.pc2.uni-paderborn.de/

JH login.png


Server Options


Jh serveroptions.png

Spawner

LocalSpawner

The LocalSpawner spawns a notebook server on the jupyterhub host as a simple process. From the notebooks (or also from the terminal) slurm jobs can then be started. Magic Commands are used for this.

For more information see How-To -> Run slurm jobs via a notebook cell (LocalSpawner only)

NoctuaSpawner

The NoctuaSpawner starts a notebook server within a slurm batch job. If you then start a terminal via the interface, you will get a shell on the compute node.

Once a Slurm job is queued and an estimated start time for that job is assigned, the Jupyter notebook server will also start at the specified time (even after a logout):

JupyterHub Estimated start time.png


JupyterLab

The JupyterLab is a modern web-based Jupyter notebook interface with many features:

  • Open more than one notebooks in one window
  • Integrated text editor
  • Terminals
  • Extension Manager
  • ...

See here for more information: https://jupyterlab.readthedocs.io/en/stable/getting_started/overview.html


PC2 JupyterLab.png









Singularity Container

Singularity Container Purpose Installed kernel Packages
jupyter_buster_py3.7.3 Default Singularity Container IPython /
jupyter_scientifc_python Scientific Python Ecosystem IPython dask, pandas, numexpr, matplotlib, scipy, seaborn, scikit-learn, scikit-image, sympy, cython, patsy, statsmodel, cloudpickle, dill, numba, bokeh, sqlalchemy, hdf5, vincent, beautifulsoup, protobuf, xlrd, bottleneck, and pytables packages, ipywidgets, ipympl, Facets
jupyter_datascience data analysis from the Julia, Python, and R communities. IPython, IJulia, R The Julia compiler and base environment, IJulia to support Julia code in Jupyter notebooks, HDF5, Gadfly, RDatasets, everything in jupyter_scientific_python

See here for more information:

https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html

If you want to use your own Singularity Container, see here:

Create my own Singularity Container

How To

Default values on page "Spawner Options"

It is possible to enter default values on the "Server Options" page, which will be applied after each page refresh.

For this purpose a predefined XML document can be placed under $HOME/.jupyter/pc2-jupyterhub/.

The XML document (pc2-jupyterhub.xml) looks like this:

 1 <!-- 
 2 JupyterHub - Paderborn Center for Parallel Computing (PC²)
 3 
 4 Description:
 5 Set default options on https://jh.noctua.pc2.uni-paderborn.de to faster start a jupyter notebook server
 6 
 7 If some value does not exists or is invalid, it will be ignored :-)
 8 
 9 Just put this file in your $HOME-directory: $HOME/.jupyter/pc2-jupyterhub/pc2-jupyterhub.xml
10 -->
11 
12 <JupyterHub_PC2>
13     <!-- absolute path of your notebook directory -->
14     <notebook_directory></notebook_directory>
15     <!-- absolute path of a singularity container (This container should exists in $HOME/.jupyter/pc2-jupyterhub/)  -->
16     <singularity_container></singularity_container>
17 
18     <!-- Default values to start a slurm job with -->
19     <!-- The endtime will be automatically calculated (FORMAT: %H:%M) - Example: 1:00 -->
20     <runtime></runtime>
21     <partition></partition>
22     <account></account>
23     <reservation></reservation>
24     <prologue></prologue>
25 </JupyterHub_PC2>

An example would be:

 1 <JupyterHub_PC2>
 2     <!-- absolute path of your notebook directory -->
 3     <notebook_directory>/scratch/pc2-mitarbeiter/mawi/</notebook_directory>
 4     <!-- absolute path of a singularity container (This container should exists in $HOME/.jupyter/pc2-jupyterhub/)  -->
 5     <singularity_container>/upb/departments/pc2/users/m/mawi/.jupyter/pc2-jupyterhub/jupyter_julia.sif</singularity_container>
 6 
 7     <!-- Default values to start a slurm job with -->
 8     <!-- The endtime will be automatically calculated (FORMAT: %H:%M) - Example: 1:00 -->
 9     <runtime>01:30</runtime>
10     <partition>batch</partition>
11     <account>hpc-lco-jupyter</account>
12     <reservation></reservation>
13     <prologue>
14 export SINGULARITY_BIND="/scratch/pc2-mitarbeiter/mawi/:/mawi/:rw"
15 export CUSTOM_VAR="Hello JupyterHub friend!"
16     </prologue>
17 </JupyterHub_PC2>

Note:
If you don't want to store a fixed value for an attribute, just leave it blank.

Magic Commands: Run slurm jobs via a notebook cell (LocalSpawner only)

  1. Load the extension:
1 %load_ext slurm_magic

Available commands:

 %sacct
 %sacctmgr
 %salloc
 %sattach
 %%sbatch
 %sbcast
 %scancel
 %scontrol
 %sdiag
 %sinfo
 %snmap
 %sprio
 %squeue
 %sreport
 %srun
 %sshare
 %sstat
 %strigger
 %sview

Example running a batch job

Jupyterhub cell sbatch.png

Use of a conda environment (NoctuaSpawner)

  1. Load module anaconda using the Prologue input block
1 module load anaconda/3-5.0.0.1
2 conda activate /scratch/pc2-mitarbeiter/mawi/conda_env/

Create multiple notebook servers

  1. Go to the Home-Section and create a named server.
    1. All created servers can be joined seperately.

Run my own Jupyter Notebook configuration

  1. Start a notebook server with the LocalSpawner and open a terminal
  2. Run following commands to create a default config file:
1 mkdir $HOME/.jupyter/
2 cd $HOME/.jupyter/
3 jupyter notebook --generate-config

Configuration file options:
https://jupyter-notebook.readthedocs.io/en/stable/config.html

Note:
Note that the configuration file will not be loaded if you are using a Singularity Container and do not mount your $HOME or manually set the $JUPYTER_CONFIG_DIR environment variable.

Create my own singularity container

Skeleton Singularity Recipe file

 1 Bootstrap: docker
 2 From: debian
 3  
 4 %post
 5         ##### BASE JUPYTER INSTALLATION #####
 6         apt update
 7         apt install -y python3 python3-pip git
 8         python3 -m pip install --upgrade pip
 9         python3 -m pip install notebook batchspawner
10 
11         ##### INSTALL ADDITIONAL PACKAGES #####
12         # python3 -m pip install numpy

It is also possible to build singularity containers from the official jupyter docker stacks:

https://jupyter-docker-stacks.readthedocs.io/en/latest/

Here are more information on how to build a singularity container from DockerHub:

https://sylabs.io/guides/3.7/user-guide/build_a_container.html

Build container

You can build you container on your host by executing following command:
sudo singularity build <container_name>.sif <recipe_file>

If you want to build the container on Noctua, you can build your container using the --remote option:
singularity build --remote <container_name>.sif <recipe_file>

Note: You need an account at https://sylabs.io/ to use the remote build feature.

Container Location

Your new created container should be placed in your $HOME directory:

  • $HOME/.jupyter/pc2-jupyterhub/
    • If the directoy/ies does not exists, just create it with a simple mkdir.
  • Alternatively you can create a link from your $PC2PFS to your $HOME directory:
1 [mawi@jupyterhub ~]$ ls -l /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif
2 -rw-r--r--. 1 mawi pc2-mitarbeiter 0 Dec 17 07:53 /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif
3 [mawi@jupyterhub ~]$ ln -s /scratch/pc2-mitarbeiter/mawi/jupyter_container.sif $HOME/.jupyter/pc2-jupyterhub/

Note: All containers with type .sif will be automatically detected in $HOME/.jupyter/pc2-jupyterhub/

Share my jupyter notebook with other users

  1. Go to: https://jh.noctua.pc2.uni-paderborn.de/services/nbviewer/
  2. Paste your URL, render it and share the URL with other users

Embed WebVNC as IFrame in a jupyter notebook

1 from IPython.display import IFrame
2 IFrame(src="/user/<your-username>/desktop/", width='100%', height=700)

Creating presentations using the RISE extension

  • The RISE extension is installed in the default singularity containers and is available if using the LocalSpawner.

Note:
You cannot use the RISE extension with JupyterLab. You need the classic notebook view.

  • From JupyterLab: Help -> Launch Classic Notebook

More information: https://rise.readthedocs.io/en/stable/

Mount additional paths into a singularity container

  • You can use the input block "Prologue" to do this.
    • Just export following environment variable:
1 export SINGULARITY_BIND="SOURCE:DEST:OPTS,SOURCE:DEST:OPTS,..."
2 # Example:
3 # export SINGULARITY_BIND="/scratch/pc2-mitarbeiter/mawi/:/mawi/:rw"
4 # Then /scratch/pc2-mitarbeiter/mawi/ would be mount to /mawi/ (read & write) into the container.

See here for more information: https://sylabs.io/guides/3.7/user-guide/bind_paths_and_mounts.html

Troubleshooting

"Terminals unavailable"

  • If you have terminado installed in your $HOME directory (pip3 install --user), please make sure that the version of terminado is at least 0.8.3.
    • How to upgrade: pip3 install --user --upgrade terminado
  • User specific config file:
    • The following option may be responsible for this: terminals_enabled

JupyterLab + HTML: CSS and JavaScript files are not trusted

PC² Support

If you have any other problems that won't be solved, please contact the pc2-support.