diff --git a/doc.zih.tu-dresden.de/docs/access/jupyterhub.md b/doc.zih.tu-dresden.de/docs/access/jupyterhub.md
index 3457cd669e53fa4b1ca9aaa1e087d6c469bca895..130e8ce4014b8b1c81499dffe404208699048d8f 100644
--- a/doc.zih.tu-dresden.de/docs/access/jupyterhub.md
+++ b/doc.zih.tu-dresden.de/docs/access/jupyterhub.md
@@ -62,35 +62,35 @@ presets in text files.
You can choose between JupyterLab or classic Jupyter notebooks as outlined in the following.
-### JupyterLab
+=== "JupyterLab"
-
-{: align="center"}
+ 
+ {: align="center"}
-The main workspace is used for multiple notebooks, consoles or
-terminals. Those documents are organized with tabs and a very versatile
-split screen feature. On the left side of the screen you can open
-several views:
+ The main workspace is used for multiple notebooks, consoles or
+ terminals. Those documents are organized with tabs and a very versatile
+ split screen feature. On the left side of the screen you can open
+ several views:
-- file manager
-- controller for running kernels and terminals
-- overview of commands and settings
-- details about selected notebook cell
-- list of open tabs
+ - file manager
+ - controller for running kernels and terminals
+ - overview of commands and settings
+ - details about selected notebook cell
+ - list of open tabs
-### Classic Jupyter Notebook
+=== "Classic Jupyter Notebook"
-Initially, your home directory is listed. You can open existing notebooks or
-files by navigating to the corresponding path and clicking on them.
+ Initially, your home directory is listed. You can open existing notebooks or
+ files by navigating to the corresponding path and clicking on them.
-
-{: align="center"}
+ 
+ {: align="center"}
-Above the table on the right side, there is the button `New` which lets you create new
-notebooks, files, directories or terminals.
+ Above the table on the right side, there is the button `New` which lets you create new
+ notebooks, files, directories or terminals.
-
-{: align="center"}
+ 
+ {: align="center"}
## Jupyter Notebooks in General
@@ -126,20 +126,22 @@ Slurm job runtime hits the limit (usually 8 hours).
At first, you have to open the JupyterHub control panel.
-**JupyterLab**: Open the file menu and then click on `Logout`. You can
-also click on `Hub Control Panel`, which opens the control panel in a new tab instead.
+=== "JupyterLab"
+ Open the file menu and then click on `Logout`. You can
+ also click on `Hub Control Panel`, which opens the control panel in a new tab instead.
-
-{: align="center"}
+ 
+ {: align="center"}
-**Classic Jupyter notebook**: Click on the control panel button on the top right of your screen.
+=== "Classic Jupyter notebook"
+ Click on the control panel button on the top right of your screen.
-
-{: align="center"}
+ 
+ {: align="center"}
-Now, you are back on the JupyterHub page and you can stop your server by clicking on
-
-{: align="center"}
+ Now, you are back on the JupyterHub page and you can stop your server by clicking on
+ 
+ {: align="center"}
## Error Handling
@@ -182,8 +184,8 @@ home directory and have the name `jupyter-session-<jobid>.log`.
The default Python kernel uses conda environments based on the
[Watson Machine Learning Community Edition (formerly PowerAI)](https://developer.ibm.com/linuxonpower/deep-learning-powerai/)
-package suite. You can open a list with all included packages of the
-exact standard environment through the spawner form:
+package suite. You can open a list with all included python-packages that
+are available within the standard environment through the spawner form:
{: align="center"}
@@ -192,169 +194,69 @@ This list shows all packages of the currently selected conda environment. This
depends on your settings for partition (CPU architecture) and standard
environment.
-There are three standard environments:
-
-- production
-- test
-- python-env-python3.8.6
+We have a number of standard environments, namely:
-**Python-env-python3.8.6** virtual environment can be used for all x86
-partitions (`gpu2`, `alpha`, etc). It gives the opportunity to create a user
-kernel with the help of a Python environment.
+| name | optimized for | specially recommended for | Modenv | Partitions |
+|------------------------------------------------------------|-----------------|---------------------------|-------------------------|----------------------------------------------------------------------------|
+| foss-2020b | x86_64, ppc64le | | modenv/hiera, modenv/ml | `romeo`, `alpha`, `ml` |
+| fosscuda-2019b | GPUs | | modenv/hiera, modenv/ml | `romeo`, `alpha`, `ml` |
+| fosscuda-2020b | GPUs | | modenv/hiera, modenv/ml | `romeo`, `alpha`, `ml` |
+| gcccore-10.3.0_python-3.9.5_matlab-2021b | x86_64 | MATLAB | modenv/hiera | `romeo`, `alpha` |
+| hiera_gcccore-10.2.0_python-3.8.6 | x86_64 (AMD) | | modenv/hiera | `romeo`, `alpha` |
+| hiera_gcccore-10.3.0_python-3.9.5_matlab-2021b | x86_64 (AMD) | MATLAB | modenv/hiera | `romeo`, `alpha` |
+| hiera_gcccore-10.3.0_python-3.9.5_r-4.1.0_rstudio-1.4.1717 | x86_64 (AMD) | RStudio | modenv/hiera | `romeo`, `alpha` |
+| **production** | x86_64, ppc64le | | modenv/scs5, modenv/ml | `default`, `haswell`, `interactive`, `gpu2`, `hpdlf`, `dcv`, `julia`, `ml` |
+| scs5_gcccore-10.2.0_python-3.8.6 | x86_64 (Intel) | | modenv/scs5 | `default`, `haswell`, `interactive`, `gpu2`, `hpdlf`, `dcv`, `julia` |
+| scs5_gcccore-10.3.0_python-3.9.5_matlab-2021b | x86_64 (Intel) | MATLAB | modenv/scs5 | `default`, `haswell`, `interactive`, `gpu2`, `hpdlf`, `dcv`, `julia` |
+| scs5_gcccore-8.3.0_python-3.7.4 | x86_64 (Intel) | | modenv/scs5 | `default`, `haswell`, `interactive`, `gpu2`, `hpdlf`, `dcv`, `julia` |
-Here is a short list of some included software:
+<p style="font-size: 80%;">Actually 'romeo', 'alpha' and 'ml' refer to their respective interactive
+partitions 'romeo-interactive', 'alpha-interactive', 'ml-interactive' to reduce job queue waiting time.</p>
-| | generic\* | ml |
-|------------|-----------|--------|
-| Python | 3.6.10 | 3.6.10 |
-| R\*\* | 3.6.2 | 3.6.0 |
-| WML CE | 1.7.0 | 1.7.0 |
-| PyTorch | 1.3.1 | 1.3.1 |
-| TensorFlow | 2.1.1 | 2.1.1 |
-| Keras | 2.3.1 | 2.3.1 |
-| NumPy | 1.17.5 | 1.17.4 |
-| Matplotlib | 3.3.1 | 3.0.3 |
+The standard environments in Jupyter are also closely related to the available
+[Module Environments](../software/modules.md#module-environments).
-\* generic = all partitions except ml
+With these **standard environments**, we have tried to integrate a set of compatible software:
-\*\* R is loaded from the [module system](../software/modules.md)
+=== "production"
-### Creating and Using a Custom Environment
+ In this environment, we provide DCV and WebVNC access.
-!!! info
-
- Interactive code interpreters which are used by Jupyter notebooks are called
- *kernels*. Creating and using your own kernel has the benefit that you can
- install your own preferred Python packages and use them in your notebooks.
-
-We currently have two different architectures at ZIH systems.
-Build your kernel environment on the **same architecture** that you want to use
-later on with the kernel. In the examples below, we use the name
-"my-kernel" for our user kernel. We recommend to prefix your kernels
-with keywords like `haswell`, `ml`, `romeo`, `venv`, `conda`. This way, you
-can later recognize easier how you built the kernel and on which hardware it
-will work. Depending on that hardware, allocate resources:
-
-=== "x86 nodes (e.g. partition `haswell`, `gpu2`)"
- ```console
- maria@login$ srun --pty --ntasks=1 --cpus-per-task=2 --mem-per-cpu=2541 --time=08:00:00 bash -l
- ```
-=== "PowerPC nodes (partition `ml`)"
- ```console
- maria@login$ srun --pty --partition=ml --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1443 \
- --time=08:00:00 bash -l
- ```
-
-Create a virtual environment in your home directory. You can decide between
-Python virtualenv or conda environment.
+ You can load more software using our module system,
+ by selecting these through the option ['Preload modules (module load)'](#loading-modules).
-!!! note
- Please keep in mind that Python virtualenv is the preferred way to create a Python
- virtual environment.
- For working with conda virtual environments, it may be necessary to configure your shell
- as described in [Python virtual environments](../software/python_virtual_environments.md#conda-virtual-environment)
-
-#### Python Virtualenv
-
-```console
-marie@compute$ module load Python/3.8.6-GCCcore-10.2.0
-Module Python/3.8.6-GCCcore-10.2.0 and 11 dependencies loaded.
-marie@compute$ mkdir user-kernel # please use workspaces!
-marie@compute$ cd user-kernel
-marie@compute$ virtualenv --system-site-packages my-kernel
-created virtual environment CPython3.8.6.final.0-64 in 5985ms
- creator CPython3Posix(dest=[...]/my-kernel, clear=False, global=True)
- seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=[...])
- added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
- activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
-marie@compute$ source my-kernel/bin/activate
-(my-kernel) marie@compute$ pip install ipykernel
-Collecting ipykernel
-[...]
-Successfully installed [...] ipykernel-6.9.1 ipython-8.0.1 [...]
-(my-kernel) marie@compute$ pip install --upgrade pip
-(my-kernel) marie@compute$ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
-Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/my-kernel
-(my-kernel) marie@compute$ pip install [...] # now install additional packages for your notebooks
-(my-kernel) marie@compute$ deactivate
-```
+=== "...matlab"
-!!! warning
+ We integrated MATLAB into it, such that you can use it in your browser.
- Depending on the Python module you have loaded for creating your virtual environment, you should
- select the apropriate [Standard environment](#standard-environments). For example, you could
- select `scs5_gcccore-10.2.0_python-3.8.6`, when you want to use `my-kernel`. Furthermore,
- ensure, that you pre-load the same modules via [Spawner Options](#start-a-session) that you used
- for creating your kernel.
+ You can load more software using our module system,
+ by selecting these through the option ['Preload modules (module load)'](#loading-modules).
-#### Conda Environment
+=== "...rstudio"
-Load the needed module depending on partition architecture:
+ We integrated R & RStudio into it, such that you can use it in your browser.
+ Please click on RStudio's Icon for it, when you are in the JupyterHub overview.
-=== "x86 nodes (e.g. partition `haswell`, `gpu2`)"
- ```console
- marie@compute$ module load Anaconda3
- ```
-=== "PowerPC nodes (partition `ml`)"
- ```console
- marie@ml$ module load PythonAnaconda
- ```
-
-!!! hint
- For working with conda virtual environments, it may be necessary to configure your shell as
- described in
- [Python virtual environments](../software/python_virtual_environments.md#conda-virtual-environment).
-
-Continue with environment creation, package installation and kernel
-registration:
-
-```console
-marie@compute$ mkdir user-kernel # please use workspaces!
-marie@compute$ conda create --prefix $HOME/user-kernel/my-kernel python=3.8.6
-Collecting package metadata: done
-Solving environment: done
-[...]
-marie@compute$ conda activate $HOME/user-kernel/my-kernel
-marie@compute$ conda install ipykernel
-Collecting package metadata: done
-Solving environment: done
-[...]
-marie@compute$ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
-Installed kernelspec my-kernel in [...]
-marie@compute$ conda install [..] # now install additional packages for your notebooks
-marie@compute$ conda deactivate
-```
-
-Now you can start a new session and your kernel should be available.
-
-**JupyterLab**: Your kernels are listed on the launcher page:
-
-
-{: align="center"}
+=== "fosscuda-2020b"
-You can switch kernels of existing notebooks in the menu:
-
-
-{: align="center"}
-
-**Classic Jupyter notebook**: Your kernel is listed in the New menu:
-
-
-{: align="center"}
-
-You can switch kernels of existing notebooks in the kernel menu:
-
-
-{: align="center"}
-
-!!! note
- Both python venv and conda virtual environments will be mention in the same
- list.
+ Can be utilized with the partitions `gpu2`, `alpha` and `ml`.
+ It's specially geared towards GPU support.
### Loading Modules
You have now the option to preload modules from the [module system](../software/modules.md).
Select multiple modules that will be preloaded before your notebook server
starts. The list of available modules depends on the module environment you want
-to start the session in (`scs5` or `ml`). The right module environment will be
+to start the session in (`scs5`, `hiera` or `ml`). The right module environment will be
chosen by your selected partition.
+
+### Custom Kernels
+
+As you might have noticed, after launching Jupyter**Lab**,
+there are several boxes with icons therein visible in the `Launcher`.
+Each box therein represents a so called 'Kernel'.
+(note that these are not to be confused with operating system kernel,
+but similarly provide basic functionality for running your use cases,
+e.g. Python or R)
+
+You can also [create your own Kernels](jupyterhub_custom_environments.md).
diff --git a/doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md b/doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md
new file mode 100644
index 0000000000000000000000000000000000000000..9fe57b98a82ec0b1cc5a68ed42c6a115ed68b4e5
--- /dev/null
+++ b/doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md
@@ -0,0 +1,233 @@
+# Creating and Using a Custom Environment for JupyterHub
+
+!!! info
+
+ Interactive code interpreters which are used by Jupyter notebooks are called
+ *kernels*. Creating and using your own kernel has the benefit, that you can
+ install your own preferred Python packages and use them in your notebooks.
+
+We currently have two different architectures at ZIH systems.
+Build your kernel environment on the **same architecture** that you want to use
+later on with the kernel. In the examples below, we use the name
+"my-kernel" for our user kernel. We recommend to prefix your kernels
+with keywords like `haswell`, `ml`, `romeo`, `venv`, `conda`. This way, you
+can later recognize easier how you built the kernel and on which hardware it
+will work. Depending on that hardware, allocate resources as follows.
+
+## Preliminary Steps
+
+=== "Nodes with x86_64 (Intel) CPU"
+
+ Use **one srun command** of these:
+
+ ```console
+ maria@login$ srun --partition=haswell64 --pty --ntasks=1 --cpus-per-task=2 \
+ --mem-per-cpu=2541 --time=08:00:00 bash -l
+ maria@login$ srun --partition=gpu2 --pty --ntasks=1 --cpus-per-task=2 \
+ --mem-per-cpu=2541 --time=08:00:00 bash -l
+ ```
+
+=== "Nodes with x86_64 (AMD) CPU"
+
+ Use **one srun command** of these:
+
+ ```console
+ maria@login$ srun --partition=romeo --pty --ntasks=1 --cpus-per-task=3 \
+ --mem-per-cpu=1972 --time=08:00:00 bash -l
+ maria@login$ srun --partition=alpha --gres=gpu:1 --pty --ntasks=1 \
+ --cpus-per-task=6 --mem-per-cpu=10312 --time=08:00:00 bash -l
+ ```
+
+=== "Nodes with ppc64le CPU"
+
+ ```console
+ maria@ml$ srun --pty --partition=ml --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1443 \
+ --time=08:00:00 bash -l
+ ```
+
+When creating a virtual environment in your home directory, you got to decide
+to either use "Python virtualenv" or "conda environment".
+
+!!! note
+ Please keep in mind that Python virtualenv is the preferred way to create a Python
+ virtual environment.
+ For working with conda virtual environments, it may be necessary to configure your shell
+ as described in [Python virtual environments](../software/python_virtual_environments.md#conda-virtual-environment)
+
+## Python Virtualenv
+
+While we have a general description on
+[Python Virtual Environments](../software/python_virtual_environments.md), here we have a more detailed
+description on using them with JupyterHub:
+
+Depending on the CPU architecture that you are targeting, please choose a `modenv`:
+
+=== "scs5"
+
+ For use with Standard Environment `scs5_gcccore-10.2.0_python-3.8.6`,
+ please try to initialize your Python Virtual Environment like this:
+
+ ```console
+ marie@haswell$ module load Python/3.8.6-GCCcore-10.2.0
+ Module Python/3.8.6-GCCcore-10.2.0 and 11 dependencies loaded.
+ marie@haswell$ mkdir user-kernel # please use workspaces!
+ marie@haswell$ cd user-kernel
+ marie@haswell$ virtualenv --system-site-packages my-kernel
+ created virtual environment CPython3.8.6.final.0-64 in 5985ms
+ creator CPython3Posix(dest=[...]/my-kernel, clear=False, global=True)
+ seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=[...])
+ added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
+ activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
+ marie@haswell$ source my-kernel/bin/activate
+ (my-kernel) marie@haswell$ pip install ipykernel
+ Collecting ipykernel
+ [...]
+ Successfully installed [...] ipykernel-6.9.1 ipython-8.0.1 [...]
+ ```
+
+ Then continue with the steps below.
+
+=== "hiera"
+
+ For use with Standard Environment `hiera_gcccore-10.2.0_python-3.8.6`,
+ please try to initialize your Python Virtual Environment like this:
+
+ ```console
+ marie@romeo$ module load GCC/10.2.0 Python/3.8.6
+ Module GCC/10.2.0Python/3.8.6 and 11 dependencies loaded.
+ marie@romeo$ mkdir user-kernel # please use workspaces!
+ marie@romeo$ cd user-kernel
+ marie@romeo$ virtualenv --system-site-packages my-kernel
+ created virtual environment CPython3.8.6.final.0-64 in 5985ms
+ creator CPython3Posix(dest=[...]/my-kernel, clear=False, global=True)
+ seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=[...])
+ added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
+ activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
+ marie@romeo$ source my-kernel/bin/activate
+ (my-kernel) marie@romeo$ pip install ipykernel
+ Collecting ipykernel
+ [...]
+ Successfully installed [...] ipykernel-6.9.1 ipython-8.0.1 [...]
+ ```
+
+ Then continue with the steps below.
+
+=== "ml"
+
+ For use with the Standard Environment `fosscuda/2020b`,
+ please try to initialize your Python Virtual Environment like this:
+
+ ```console
+ marie@ml$ module load fosscuda/2020b ZeroMQ/4.3.3-GCCcore-10.2.0 Python/3.8.6-GCCcore-10.2.0
+ Module fosscuda/2020b and 23 dependencies loaded.
+ marie@ml$ mkdir user-kernel # please use workspaces!
+ marie@ml$ cd user-kernel
+ marie@ml$ python3 -m venv --system-site-packages my-kernel
+ marie@ml$ sourcde my-kernel/bin/activate
+ (my-kernel) marie@compute$ pip install ipykernel
+ Collecting ipykernel
+ [...]
+ Successfully installed asttokens-2.0.8 backcall-0.2.0 debugpy-1.6.3 entrypoints-0.4 executing-1.0.0 ipykernel-6.15.2 ipython-8.4.0 jedi-0.18.1 jupyter-client-7.3.5 jupyter-core-4.11.1 matplotlib-inline-0.1.6 nest-asyncio-1.5.5 parso-0.8.3 pickleshare-0.7.5 prompt-toolkit-3.0.30 pure-eval-0.2.2 python-dateutil-2.8.2 pyzmq-23.2.1 stack-data-0.5.0 tornado-6.2 traitlets-5.3.0
+
+ Then continue with the steps below.
+
+=== "default ('production')"
+
+ For use with Standard Environment `production`,
+ please try to initialize your Python Virtual Environment like this:
+
+ ```console
+ marie@compute$ module load Anaconda3/2022.05
+ Module Anaconda3/2022.05 loaded.
+ marie@compute$ mkdir user-kernel # please use workspaces!
+ marie@compute$ cd user-kernel
+ marie@compute$ python3 -m venv --system-site-packages my-kernel
+ (my-kernel) marie@compute$ pip install ipykernel
+ ```
+
+ Then continue with the steps below.
+
+After following the initialization of the environment (above),
+the usage of Python's Package manager `pip` is the same:
+
+```console
+(my-kernel) marie@compute$ pip install --upgrade pip
+(my-kernel) marie@compute$ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
+Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/my-kernel
+(my-kernel) marie@compute$ pip install [...] # now install additional packages for your notebooks
+(my-kernel) marie@compute$ deactivate
+```
+
+!!! warning
+
+ Take care to select the appropriate standard environment (as mentioned above)
+ when [spawning a new session](jupyterhub.md#start-a-session).
+
+## Conda Environment
+
+Load the needed module depending on partition architecture:
+
+=== "Nodes with x86_64 CPU"
+ ```console
+ marie@compute$ module load Anaconda3
+ ```
+
+=== "Nodes with ppc64le CPU"
+ ```console
+ marie@ml$ module load PythonAnaconda
+ ```
+
+!!! hint
+ For working with conda virtual environments, it may be necessary to configure your shell as
+ described in
+ [Python virtual environments](../software/python_virtual_environments.md#conda-virtual-environment).
+
+Continue with environment creation, package installation and kernel
+registration:
+
+```console
+marie@compute$ mkdir user-kernel # please use workspaces!
+marie@compute$ conda create --prefix $HOME/user-kernel/my-kernel python=3.8.6
+Collecting package metadata: done
+Solving environment: done
+[...]
+marie@compute$ conda activate $HOME/user-kernel/my-kernel
+marie@compute$ conda install ipykernel
+Collecting package metadata: done
+Solving environment: done
+[...]
+marie@compute$ python -m ipykernel install --user --name my-kernel --display-name="my kernel"
+Installed kernelspec my-kernel in [...]
+marie@compute$ conda install [..] # now install additional packages for your notebooks
+marie@compute$ conda deactivate
+```
+
+## Using your custom environment
+
+Now you can start a new session and your kernel should be available.
+
+=== "JupyterLab"
+ Your kernels are listed on the launcher page:
+
+ 
+ {: align="center"}
+
+ You can switch kernels of existing notebooks in the menu:
+
+ 
+ {: align="center"}
+
+=== "Classic Jupyter notebook"
+ Your kernel is listed in the New menu:
+
+ 
+ {: align="center"}
+
+ You can switch kernels of existing notebooks in the kernel menu:
+
+ 
+ {: align="center"}
+
+!!! note
+ Both python venv and conda virtual environments will be mentioned in the same
+ list.
diff --git a/doc.zih.tu-dresden.de/mkdocs.yml b/doc.zih.tu-dresden.de/mkdocs.yml
index 263e81f662b0f6b7aa737c8a5b5903ec90340203..573ddba029c7f441a595dd69342280e646b4bb7f 100644
--- a/doc.zih.tu-dresden.de/mkdocs.yml
+++ b/doc.zih.tu-dresden.de/mkdocs.yml
@@ -18,6 +18,7 @@ nav:
- Graphical Applications with WebVNC: access/graphical_applications_with_webvnc.md
- JupyterHub:
- JupyterHub: access/jupyterhub.md
+ - JupyterHub Custom Environments: access/jupyterhub_custom_environments.md
- JupyterHub for Teaching: access/jupyterhub_for_teaching.md
- JupyterHub Teaching Example: access/jupyterhub_teaching_example.md
- Key Fingerprints: access/key_fingerprints.md