From e2e75137955e4214eb6475b4059d0b74c73add12 Mon Sep 17 00:00:00 2001
From: Guilherme Calandrini <gupa977e@tu-dresden.de>
Date: Tue, 13 Feb 2024 10:25:24 +0100
Subject: [PATCH] fixes for creating jupyterlab custom kernels
---
.../access/jupyterhub_custom_environments.md | 133 ++++--------------
1 file changed, 30 insertions(+), 103 deletions(-)
diff --git a/doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md b/doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md
index 86b8486e2..44e162a0d 100644
--- a/doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md
+++ b/doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md
@@ -10,39 +10,24 @@ We currently have two different architectures at ZIH systems.
Build your kernel environment on the **same architecture** that you want to use
later on with the kernel. In the examples below, we use the name
"my-kernel" for our user kernel. We recommend to prefix your kernels
-with keywords like `haswell`, `ml`, `romeo`, `venv`, `conda`. This way, you
-can later recognize easier how you built the kernel and on which hardware it
+with keywords like `alpha`, `barnard`, `romeo`, `power9`, `venv`, `conda`.
+This way, you can later recognize easier how you built the kernel and on which hardware it
will work. Depending on that hardware, allocate resources as follows.
-## Preliminary Steps
-
-=== "Nodes with x86_64 (Intel) CPU"
-
- Use **one srun command** of these:
-
- ```console
- maria@login$ srun --partition=haswell64 --pty --ntasks=1 --cpus-per-task=2 \
- --mem-per-cpu=2541 --time=08:00:00 bash -l
- maria@login$ srun --partition=gpu2 --pty --ntasks=1 --cpus-per-task=2 \
- --mem-per-cpu=2541 --time=08:00:00 bash -l
- ```
-
-=== "Nodes with x86_64 (AMD) CPU"
-
- Use **one srun command** of these:
+| Cluster | Architecture name |
+|------------|-------------------------|
+| Alpha | x86_64 (AMD) |
+| Barnard | x86_64 (Intel) |
+| Romeo | x86_64 (AMD) |
+| Power9 | ppc64le (IBM) |
- ```console
- maria@login$ srun --partition=romeo --pty --ntasks=1 --cpus-per-task=3 \
- --mem-per-cpu=1972 --time=08:00:00 bash -l
- maria@login$ srun --partition=alpha --gres=gpu:1 --pty --ntasks=1 \
- --cpus-per-task=6 --mem-per-cpu=10312 --time=08:00:00 bash -l
- ```
+## Preliminary Steps
-=== "Nodes with ppc64le CPU"
+ Start an interactive job
```console
- maria@ml$ srun --pty --partition=ml --ntasks=1 --cpus-per-task=2 --mem-per-cpu=1443 \
- --time=08:00:00 bash -l
+ maria@login.<cluster>$ srun --pty --ntasks=1 --cpus-per-task=2 \
+ --mem-per-cpu=2541 --time=02:00:00 bash -l
```
When creating a virtual environment in your home directory, you got to decide
@@ -60,93 +45,40 @@ While we have a general description on
[Python Virtual Environments](../software/python_virtual_environments.md), here we have a more detailed
description on using them with JupyterHub:
-Depending on the CPU architecture that you are targeting, please choose a `modenv`:
+Depending on the Cluster that you are targeting, please choose the right modules:
-=== "scs5"
+=== "release/23.10"
- For use with Standard Environment `scs5_gcccore-10.2.0_python-3.8.6`,
+ For use with Python version 3.10.4,
please try to initialize your Python Virtual Environment like this:
```console
- marie@haswell$ module load Python/3.8.6-GCCcore-10.2.0
- Module Python/3.8.6-GCCcore-10.2.0 and 11 dependencies loaded.
- marie@haswell$ mkdir user-kernel # please use workspaces!
- marie@haswell$ cd user-kernel
- marie@haswell$ virtualenv --system-site-packages my-kernel
- created virtual environment CPython3.8.6.final.0-64 in 5985ms
- creator CPython3Posix(dest=[...]/my-kernel, clear=False, global=True)
- seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=[...])
- added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
- activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
- marie@haswell$ source my-kernel/bin/activate
- (my-kernel) marie@haswell$ pip install ipykernel
- Collecting ipykernel
- [...]
- Successfully installed [...] ipykernel-6.9.1 ipython-8.0.1 [...]
+ [marie@barnard ~]$ module load release/23.10 GCC/11.3.0 Python/3.10.4
+ Module GCC/11.3.0, Python/3.10.4 and 12 dependencies loaded.
```
- Then continue with the steps below.
-
-=== "hiera"
-
- For use with Standard Environment `hiera_gcccore-10.2.0_python-3.8.6`,
- please try to initialize your Python Virtual Environment like this:
+=== "release/23.04"
```console
- marie@romeo$ module load GCC/10.2.0 Python/3.8.6
- Module GCC/10.2.0Python/3.8.6 and 11 dependencies loaded.
- marie@romeo$ mkdir user-kernel # please use workspaces!
- marie@romeo$ cd user-kernel
- marie@romeo$ virtualenv --system-site-packages my-kernel
- created virtual environment CPython3.8.6.final.0-64 in 5985ms
- creator CPython3Posix(dest=[...]/my-kernel, clear=False, global=True)
- seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=[...])
- added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
- activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
- marie@romeo$ source my-kernel/bin/activate
- (my-kernel) marie@romeo$ pip install ipykernel
- Collecting ipykernel
- [...]
- Successfully installed [...] ipykernel-6.9.1 ipython-8.0.1 [...]
+ [marie@barnard ~]$ module load release/23.04 GCC/11.3.0 Python/3.10.4
+ Module GCC/11.3.0, Python/3.10.4 and 12 dependencies loaded.
```
- Then continue with the steps below.
-
-=== "ml"
-
- For use with the Standard Environment `fosscuda/2020b`,
- please try to initialize your Python Virtual Environment like this:
+Then continue with the steps below.
```console
- marie@ml$ module load fosscuda/2020b ZeroMQ/4.3.3-GCCcore-10.2.0 Python/3.8.6-GCCcore-10.2.0
- Module fosscuda/2020b and 23 dependencies loaded.
- marie@ml$ mkdir user-kernel # please use workspaces!
- marie@ml$ cd user-kernel
- marie@ml$ python3 -m venv --system-site-packages my-kernel
- marie@ml$ sourcde my-kernel/bin/activate
- (my-kernel) marie@compute$ pip install ipykernel
+ [marie@barnard ~]$ mkdir -p ~/usr/jlab-kernels # please use workspaces!
+ [marie@barnard ~]$ cd ~/usr/jlab-kernels
+ [marie@barnard jlab-kernels]$ python3 -m venv --system-site-packages my-kernel
+ [marie@barnard jlab-kernels]$
+ [marie@barnard jlab-kernels]$ source my-kernel/bin/activate
+ (my-kernel) [marie@barnard jlab-kernels]$
+ (my-kernel) [marie@barnard jlab-kernels]$ pip install ipykernel
Collecting ipykernel
[...]
- Successfully installed asttokens-2.0.8 backcall-0.2.0 debugpy-1.6.3 entrypoints-0.4 executing-1.0.0 ipykernel-6.15.2 ipython-8.4.0 jedi-0.18.1 jupyter-client-7.3.5 jupyter-core-4.11.1 matplotlib-inline-0.1.6 nest-asyncio-1.5.5 parso-0.8.3 pickleshare-0.7.5 prompt-toolkit-3.0.30 pure-eval-0.2.2 python-dateutil-2.8.2 pyzmq-23.2.1 stack-data-0.5.0 tornado-6.2 traitlets-5.3.0
-
- Then continue with the steps below.
-
-=== "default ('production')"
-
- For use with Standard Environment `production`,
- please try to initialize your Python Virtual Environment like this:
-
- ```console
- marie@compute$ module load Anaconda3/2022.05
- Module Anaconda3/2022.05 loaded.
- marie@compute$ mkdir user-kernel # please use workspaces!
- marie@compute$ cd user-kernel
- marie@compute$ python3 -m venv --system-site-packages my-kernel
- (my-kernel) marie@compute$ pip install ipykernel
+ Successfully installed [...] ipykernel-x.x.x ipython-x.x.x [...]
```
- Then continue with the steps below.
-
After following the initialization of the environment (above),
the usage of Python's Package manager `pip` is the same:
@@ -158,14 +90,9 @@ Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/my-kernel
(my-kernel) marie@compute$ deactivate
```
-!!! warning
-
- Take care to select the appropriate standard environment (as mentioned above)
- when [spawning a new session](jupyterhub.md#start-a-session).
-
## Conda Environment
-Load the needed module depending on partition architecture:
+Load the needed module depending on Cluster architecture:
=== "Nodes with x86_64 CPU"
```console
--
GitLab