Skip to content
Snippets Groups Projects
Commit 1faf39c7 authored by Martin Schroschk's avatar Martin Schroschk
Browse files

Review: fix wording, typos, swap figures and text

parent d6c14b38
No related branches found
No related tags found
3 merge requests!322Merge preview into main,!319Merge preview into main,!286Resolve "docs/access/jupyterhub.md"
# JupyterHub # JupyterHub
With our JupyterHub service we offer you a quick and easy way to With our JupyterHub service we offer you a quick and easy way to work with Jupyter notebooks on ZIH
work with Jupyter notebooks on ZIH systems. systems. This page covers starting and stopping JuperterHub sessions, error handling and customizing
the environment.
Subpages: We also provide a comprehensive documentation on how to use
[JupyterHub for Teaching (git-pull feature, quickstart links, direct links to notebook files)](jupyterhub_for_teaching.md).
- [JupyterHub for Teaching (git-pull feature, quickstart links, direct
links to notebook files)](jupyterhub_for_teaching.md)
## Disclaimer ## Disclaimer
This service is provided "as-is", use at your own discretion. !!! warning
Please understand that JupyterHub is a complex software system of which we are
not the developers and don't have any downstream support contracts for, The JupyterHub service is provided *as-is*, use at your own discretion.
so we merely offer an installation of it but cannot give extensive support
in every case. Please understand that JupyterHub is a complex software system of which we are not the developers
and don't have any downstream support contracts for, so we merely offer an installation of it but
cannot give extensive support in every case.
## Access ## Access
...@@ -22,13 +23,12 @@ in every case. ...@@ -22,13 +23,12 @@ in every case.
This service is only available for users with an active HPC project. This service is only available for users with an active HPC project.
See [here](../access/overview.md) how to apply for an HPC project. See [here](../access/overview.md) how to apply for an HPC project.
JupyterHub is available here: [https://taurus.hrsk.tu-dresden.de/jupyter](https://taurus.hrsk.tu-dresden.de/jupyter) JupyterHub is available at
[https://taurus.hrsk.tu-dresden.de/jupyter](https://taurus.hrsk.tu-dresden.de/jupyter).
## Start a Session ## Start a Session
Start a new session by clicking on the Start a new session by clicking on the `Start my server` button.
![Start my server](misc/start_my_server.png)
button.
A form opens up where you can customize your session. Our simple form A form opens up where you can customize your session. Our simple form
offers you the most important settings to start quickly. offers you the most important settings to start quickly.
...@@ -39,13 +39,11 @@ offers you the most important settings to start quickly. ...@@ -39,13 +39,11 @@ offers you the most important settings to start quickly.
For advanced users we have an extended form where you can change many For advanced users we have an extended form where you can change many
settings. You can: settings. You can:
- modify Slurm parameters to your needs ( [more about - modify batch system parameters to your needs ([more about batch system Slurm](../jobs_and_resources/slurm.md))
Slurm](../jobs_and_resources/slurm.md)) - assign your session to a project or reservation
- assign your session to a project or reservation - load modules from the [module system](../software/runtime_environment.md)
- load modules from the [LMOD module - choose a different standard environment (in preparation for future
system](../software/runtime_environment.md) software updates or testing additional features)
- choose a different standard environment (in preparation for future
software updates or testing additional features)
![Advanced form](misc/advanced_form.png) ![Advanced form](misc/advanced_form.png)
{: align="center"} {: align="center"}
...@@ -56,12 +54,12 @@ the import/export feature (available through the button) to save your ...@@ -56,12 +54,12 @@ the import/export feature (available through the button) to save your
presets in text files. presets in text files.
!!! info !!! info
The [alpha](https://doc.zih.tu-dresden.de/hpc-wiki/bin/view/Compendium/AlphaCentauri) The partition [alpha](https://doc.zih.tu-dresden.de/hpc-wiki/bin/view/Compendium/AlphaCentauri)
partition is available only in the extended form. is available only in the extended form.
## Applications ## Applications
You can choose between JupyterLab or classic Jupyter notebooks. You can choose between JupyterLab or classic Jupyter notebooks as outlined in the following.
### JupyterLab ### JupyterLab
...@@ -73,42 +71,38 @@ terminals. Those documents are organized with tabs and a very versatile ...@@ -73,42 +71,38 @@ terminals. Those documents are organized with tabs and a very versatile
split screen feature. On the left side of the screen you can open split screen feature. On the left side of the screen you can open
several views: several views:
- file manager - file manager
- controller for running kernels and terminals - controller for running kernels and terminals
- overview of commands and settings - overview of commands and settings
- details about selected notebook cell - details about selected notebook cell
- list of open tabs - list of open tabs
### Classic Jupyter Notebook ### Classic Jupyter Notebook
Initially your `home` directory is listed. You can open existing notebooks or files by navigating to
the corresponding path and clicking on them.
![Jupyter notebook file browser](misc/jupyter_notebook_file_browser.png) ![Jupyter notebook file browser](misc/jupyter_notebook_file_browser.png)
{: align="center"} {: align="center"}
Above the table on the right side is the button `New` which lets you create new notebooks, files,
directories or terminals.
![Jupyter notebook example matplotlib](misc/jupyter_notebook_example_matplotlib.png) ![Jupyter notebook example matplotlib](misc/jupyter_notebook_example_matplotlib.png)
{: align="center"} {: align="center"}
Initially you will get a list of your home directory. You can open
existing notebooks or files by clicking on them.
Above the table on the right side is the button `New` which lets you
create new notebooks, files, directories or terminals.
## Jupyter Notebooks in General ## Jupyter Notebooks in General
In JupyterHub you can create scripts in notebooks. In JupyterHub you can create scripts in notebooks. Notebooks are programs which are split into
Notebooks are programs which are split in multiple logical code blocks. multiple logical code blocks. In between those code blocks you can insert text blocks for
In between those code blocks you can insert text blocks for documentation and each block can be executed individually. Each notebook is paired with a kernel
documentation and each block can be executed individually. Each notebook running the code. We currently offer one for Python, C++, MATLAB and R.
is paired with a kernel which runs the code. We currently offer one for
Python, C++, MATLAB and R.
## Stop a Session ## Stop a Session
It's good practise to stop your session once your work is done. This It is good practise to stop your session once your work is done. This releases resources for other
releases resources for other users and your quota is less charged. If users and your quota is less charged. If you just log out or close the window, your server continues
you just log out or close the window your server continues running and running and **will not stop** until the Slurm job runtime hits the limit (usually 8 hours).
will not stop until the Slurm job runtime hits the limit (usually 8
hours).
At first you have to open the JupyterHub control panel. At first you have to open the JupyterHub control panel.
...@@ -129,68 +123,61 @@ Now you are back on the JupyterHub page and you can stop your server by clicking ...@@ -129,68 +123,61 @@ Now you are back on the JupyterHub page and you can stop your server by clicking
![Stop my server](misc/stop_my_server.png) ![Stop my server](misc/stop_my_server.png)
{: align="center"} {: align="center"}
## Error handling ## Error Handling
We want to explain some errors that you might face sooner or later. We want to explain some errors that you might face sooner or later.
If you need help open a ticket at [HPC support](mailto:hpcsupport@zih.tu-dresden.de). If you need help open a ticket at [HPC support](mailto:hpcsupport@zih.tu-dresden.de).
### Error while starting a session ### Error at Session Start
![Error batch job submission failed](misc/error_batch_job_submission_failed.png) ![Error batch job submission failed](misc/error_batch_job_submission_failed.png)
{: align="center"} {: align="center"}
This message often appears instantly if your Slurm parameters are not valid. This message appears instantly if your batch system parameters are not valid.
Please check those settings against the available hardware. Please check those settings against the available hardware.
Useful pages for valid Slurm parameters: Useful pages for valid batch system parameters:
- [Slurm batch system (Taurus)](../jobs_and_resources/system_taurus.md#batch-system) - [Slurm batch system (Taurus)](../jobs_and_resources/system_taurus.md#batch-system)
- [General information how to use Slurm](../jobs_and_resources/slurm.md) - [General information how to use Slurm](../jobs_and_resources/slurm.md)
### Error message in JupyterLab ### Error Message in JupyterLab
![JupyterLab error directory not found](misc/jupyterlab_error_directory_not_found.png) ![JupyterLab error directory not found](misc/jupyterlab_error_directory_not_found.png)
{: align="center"} {: align="center"}
If the connection to your notebook server unexpectedly breaks you maybe If the connection to your notebook server unexpectedly breaks, you will get this error message.
will get this error message. Sometimes your notebook server might hit a batch system or hardware limit and gets killed. Then
Sometimes your notebook server might hit a Slurm or hardware limit and usually the logfile of the corresponding batch job might contain useful information. These logfiles
gets killed. Then usually the logfile of the corresponding Slurm job are located in your `home` directory and have the name `jupyter-session-<jobid>.log`.
might contain useful information. These logfiles are located in your
home directory and have the name `jupyter-session-<jobid>.log`.
------------------------------------------------------------------------ ## Advanced Tips
## Advanced tips ### Standard Environments
### Standard environments The default Python kernel uses conda environments based on the
[Watson Machine Learning Community Edition (formerly PowerAI)](https://developer.ibm.com/linuxonpower/deep-learning-powerai/)
The default python kernel uses conda environments based on the [Watson
Machine Learning Community Edition (formerly
PowerAI)](https://developer.ibm.com/linuxonpower/deep-learning-powerai/)
package suite. You can open a list with all included packages of the package suite. You can open a list with all included packages of the
exact standard environment through the spawner form: exact standard environment through the spawner form:
![Environment package list](misc/environment_package_list.png) ![Environment package list](misc/environment_package_list.png)
{: align="center"} {: align="center"}
This list shows all packages of the currently selected conda This list shows all packages of the currently selected conda environment. This depends on your
environment. This depends on your settings for partition (cpu settings for partition (CPU architecture) and standard environment.
architecture) and standard environment.
There are three standard environments: There are three standard environments:
- production - production
- test - test
- python-env-python3.8.6 - python-env-python3.8.6
**Python-env-python3.8.6** virtual environment can be used for all x86 **Python-env-python3.8.6** virtual environment can be used for all x86 partitions(`gpu2`, `alpha`,
partitions(gpu2, alpha, etc). It gives the opportunity to create a user etc). It gives the opportunity to create a user kernel with the help of a Python environment.
kernel with the help of a python environment.
Here's a short list of some included software: Here is a short list of some included software:
| | | |
|------------|-----------|--------|
| | generic\* | ml | | | generic\* | ml |
|------------|-----------|--------|
| Python | 3.6.10 | 3.6.10 | | Python | 3.6.10 | 3.6.10 |
| R\*\* | 3.6.2 | 3.6.0 | | R\*\* | 3.6.2 | 3.6.0 |
| WML CE | 1.7.0 | 1.7.0 | | WML CE | 1.7.0 | 1.7.0 |
...@@ -204,12 +191,13 @@ Here's a short list of some included software: ...@@ -204,12 +191,13 @@ Here's a short list of some included software:
\*\* R is loaded from the [module system](../software/runtime_environment.md) \*\* R is loaded from the [module system](../software/runtime_environment.md)
### Creating and using your own environment ### Creating and Using a Custom Environment
Interactive code interpreters which are used by Jupyter Notebooks are !!! info
called kernels.
Creating and using your own kernel has the benefit that you can install Interactive code interpreters which are used by Jupyter notebooks are called *kernels*. Creating
your own preferred python packages and use them in your notebooks. and using your own kernel has the benefit that you can install your own preferred Python
packages and use them in your notebooks.
We currently have two different architectures at ZIH systems. We currently have two different architectures at ZIH systems.
Build your kernel environment on the **same architecture** that you want to use Build your kernel environment on the **same architecture** that you want to use
...@@ -218,25 +206,25 @@ later on with the kernel. In the examples below we use the name ...@@ -218,25 +206,25 @@ later on with the kernel. In the examples below we use the name
with keywords like `haswell`, `ml`, `romeo`, `venv`, `conda`. This way you with keywords like `haswell`, `ml`, `romeo`, `venv`, `conda`. This way you
can later recognize easier how you built the kernel and on which hardware it will work. can later recognize easier how you built the kernel and on which hardware it will work.
**Intel nodes** (e.g. haswell, gpu2): **Intel nodes** (e.g. partition `haswell`, `gpu2`):
```console ```console
maria@login$ srun --pty --ntasks=1 --cpus-per-task=2 --mem-per-cpu=2541 --time=08:00:00 bash -l maria@login$ srun --pty --ntasks=1 --cpus-per-task=2 --mem-per-cpu=2541 --time=08:00:00 bash -l
``` ```
**Power nodes** (ml partition): **Power nodes** (partition `ml`):
```console ```console
maria@login$ srun --pty --partition=ml --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1443 --time=08:00:00 bash -l maria@login$ srun --pty --partition=ml --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1443 --time=08:00:00 bash -l
```` ```
Create a virtual environment in your home directory. You can decide Create a virtual environment in your `home` directory. You can decide between Python virtualenvs or
between python virtualenvs or conda environments. conda environments.
!!! note !!! note
Please take in mind that Python venv is the preferred way to create a Python virtual environment. Please take in mind that Python venv is the preferred way to create a Python virtual environment.
#### Python virtualenv #### Python Virtualenv
```console ```console
marie@compute$ module load Python/3.8.6-GCCcore-10.2.0 marie@compute$ module load Python/3.8.6-GCCcore-10.2.0
...@@ -258,7 +246,7 @@ marie@compute$ pip install [...] # now install additional packages for your note ...@@ -258,7 +246,7 @@ marie@compute$ pip install [...] # now install additional packages for your note
marie@compute$ deactivate marie@compute$ deactivate
``` ```
#### Conda environment #### Conda Environment
Load the needed module for Intel nodes Load the needed module for Intel nodes
...@@ -266,14 +254,13 @@ Load the needed module for Intel nodes ...@@ -266,14 +254,13 @@ Load the needed module for Intel nodes
marie@compute$ module load Anaconda3 marie@compute$ module load Anaconda3
``` ```
... or for IBM nodes (ml partition): ... or for IBM nodes (partition `ml`):
```console ```console
marie@ml$ module load PythonAnaconda marie@ml$ module load PythonAnaconda
``` ```
Continue with environment creation, package installation and kernel Continue with environment creation, package installation and kernel registration:
registration:
```console ```console
marie@compute$ mkdir user-kernel # please use workspaces! marie@compute$ mkdir user-kernel # please use workspaces!
...@@ -317,11 +304,9 @@ You can switch kernels of existing notebooks in the kernel menu: ...@@ -317,11 +304,9 @@ You can switch kernels of existing notebooks in the kernel menu:
!!! note !!! note
Both python venv and conda virtual environments will be mention in the same list. Both python venv and conda virtual environments will be mention in the same list.
### Loading modules ### Loading Modules
You have now the option to preload modules from the LMOD module You have now the option to preload modules from the [module system](../software/modules.md).
system. Select multiple modules that will be preloaded before your notebook server starts. The list of
Select multiple modules that will be preloaded before your notebook available modules depends on the module environment you want to start the session in (`scs5` or
server starts. The list of available modules depends on the module `ml`). The right module environment will be chosen by your selected partition.
environment you want to start the session in (scs5 or ml). The right
module environment will be chosen by your selected partition.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment