Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
e2e75137
Commit
e2e75137
authored
1 year ago
by
Guilherme Calandrini
Browse files
Options
Downloads
Patches
Plain Diff
fixes for creating jupyterlab custom kernels
parent
be615023
No related branches found
Branches containing commit
No related tags found
2 merge requests
!990
Automated merge from preview to main
,
!977
gupa977e jhub venv
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md
+30
-103
30 additions, 103 deletions
...-dresden.de/docs/access/jupyterhub_custom_environments.md
with
30 additions
and
103 deletions
doc.zih.tu-dresden.de/docs/access/jupyterhub_custom_environments.md
+
30
−
103
View file @
e2e75137
...
...
@@ -10,39 +10,24 @@ We currently have two different architectures at ZIH systems.
Build your kernel environment on the
**same architecture**
that you want to use
later on with the kernel. In the examples below, we use the name
"my-kernel" for our user kernel. We recommend to prefix your kernels
with keywords like
`
haswell`
,
`ml
`
,
`romeo`
,
`venv`
,
`conda`
.
This way, you
can later recognize easier how you built the kernel and on which hardware it
with keywords like
`
alpha`
,
`barnard
`
,
`romeo`
,
`power9`
,
`venv`
,
`conda`
.
This way, you
can later recognize easier how you built the kernel and on which hardware it
will work. Depending on that hardware, allocate resources as follows.
## Preliminary Steps
=== "Nodes with x86_64 (Intel) CPU"
Use **one srun command** of these:
```console
maria@login$ srun --partition=haswell64 --pty --ntasks=1 --cpus-per-task=2 \
--mem-per-cpu=2541 --time=08:00:00 bash -l
maria@login$ srun --partition=gpu2 --pty --ntasks=1 --cpus-per-task=2 \
--mem-per-cpu=2541 --time=08:00:00 bash -l
```
=== "Nodes with x86_64 (AMD) CPU"
Use **one srun command** of these:
| Cluster | Architecture name |
|------------|-------------------------|
| Alpha | x86_64 (AMD) |
| Barnard | x86_64 (Intel) |
| Romeo | x86_64 (AMD) |
| Power9 | ppc64le (IBM) |
```console
maria@login$ srun --partition=romeo --pty --ntasks=1 --cpus-per-task=3 \
--mem-per-cpu=1972 --time=08:00:00 bash -l
maria@login$ srun --partition=alpha --gres=gpu:1 --pty --ntasks=1 \
--cpus-per-task=6 --mem-per-cpu=10312 --time=08:00:00 bash -l
```
## Preliminary Steps
=== "Nodes with ppc64le CPU"
Start an interactive job
```console
maria@
ml$ srun --pty --partition=ml
--ntasks=1 --cpus-per-task=2
--mem-per-cpu=1443
\
--time=0
8
:00:00 bash -l
maria@
login.<cluster>$ srun --pty
--ntasks=1 --cpus-per-task=2 \
--mem-per-cpu=2541
--time=0
2
:00:00 bash -l
```
When creating a virtual environment in your home directory, you got to decide
...
...
@@ -60,93 +45,40 @@ While we have a general description on
[
Python Virtual Environments
](
../software/python_virtual_environments.md
)
, here we have a more detailed
description on using them with JupyterHub:
Depending on the C
PU architecture
that you are targeting, please choose
a
`modenv`
:
Depending on the C
luster
that you are targeting, please choose
the right modules
:
=== "
scs5
"
=== "
release/23.10
"
For use with
Standard Environment `scs5_gcccore-10.2.0_pyth
on
-
3.
8.6`
,
For use with
Python versi
on
3.
10.4
,
please try to initialize your Python Virtual Environment like this:
```console
marie@haswell$ module load Python/3.8.6-GCCcore-10.2.0
Module Python/3.8.6-GCCcore-10.2.0 and 11 dependencies loaded.
marie@haswell$ mkdir user-kernel # please use workspaces!
marie@haswell$ cd user-kernel
marie@haswell$ virtualenv --system-site-packages my-kernel
created virtual environment CPython3.8.6.final.0-64 in 5985ms
creator CPython3Posix(dest=[...]/my-kernel, clear=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=[...])
added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
marie@haswell$ source my-kernel/bin/activate
(my-kernel) marie@haswell$ pip install ipykernel
Collecting ipykernel
[...]
Successfully installed [...] ipykernel-6.9.1 ipython-8.0.1 [...]
[marie@barnard ~]$ module load release/23.10 GCC/11.3.0 Python/3.10.4
Module GCC/11.3.0, Python/3.10.4 and 12 dependencies loaded.
```
Then continue with the steps below.
=== "hiera"
For use with Standard Environment `hiera_gcccore-10.2.0_python-3.8.6`,
please try to initialize your Python Virtual Environment like this:
=== "release/23.04"
```console
marie@romeo$ module load GCC/10.2.0 Python/3.8.6
Module GCC/10.2.0Python/3.8.6 and 11 dependencies loaded.
marie@romeo$ mkdir user-kernel # please use workspaces!
marie@romeo$ cd user-kernel
marie@romeo$ virtualenv --system-site-packages my-kernel
created virtual environment CPython3.8.6.final.0-64 in 5985ms
creator CPython3Posix(dest=[...]/my-kernel, clear=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=[...])
added seed packages: pip==20.2.3, setuptools==50.3.0, wheel==0.35.1
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
marie@romeo$ source my-kernel/bin/activate
(my-kernel) marie@romeo$ pip install ipykernel
Collecting ipykernel
[...]
Successfully installed [...] ipykernel-6.9.1 ipython-8.0.1 [...]
[marie@barnard ~]$ module load release/23.04 GCC/11.3.0 Python/3.10.4
Module GCC/11.3.0, Python/3.10.4 and 12 dependencies loaded.
```
Then continue with the steps below.
=== "ml"
For use with the Standard Environment `fosscuda/2020b`,
please try to initialize your Python Virtual Environment like this:
Then continue with the steps below.
```console
marie@
ml$ module load fosscuda/2020b ZeroMQ/4.3.3-GCCcore-10.2.0 Python/3.8.6-GCCcore-10.2.0
Module fosscuda/2020b and 23 dependencies loaded.
marie@
ml$ mkdir user-kernel # please use workspaces!
marie@
ml$ cd user
-kernel
marie@
ml$ python3 -m venv --system-site-packages my-kernel
marie@ml$ sourcde my-kernel/bin/activate
(my-kernel) marie@
compute
$ pip install ipykernel
[
marie@
barnard ~]$ mkdir -p ~/usr/jlab-kernels # please use workspaces!
[marie@barnard ~]$ cd ~/usr/jlab-kernels
[
marie@
barnard jlab-kernels]$ python3 -m venv --system-site-packages my-kernel
[
marie@
barnard jlab
-kernel
s]$
[
marie@
barnard jlab-kernels]$ source my-kernel/bin/activate
(my-kernel) [marie@barnard jlab-kernels]$
(my-kernel)
[
marie@
barnard jlab-kernels]
$ pip install ipykernel
Collecting ipykernel
[...]
Successfully installed asttokens-2.0.8 backcall-0.2.0 debugpy-1.6.3 entrypoints-0.4 executing-1.0.0 ipykernel-6.15.2 ipython-8.4.0 jedi-0.18.1 jupyter-client-7.3.5 jupyter-core-4.11.1 matplotlib-inline-0.1.6 nest-asyncio-1.5.5 parso-0.8.3 pickleshare-0.7.5 prompt-toolkit-3.0.30 pure-eval-0.2.2 python-dateutil-2.8.2 pyzmq-23.2.1 stack-data-0.5.0 tornado-6.2 traitlets-5.3.0
Then continue with the steps below.
=== "default ('production')"
For use with Standard Environment `production`,
please try to initialize your Python Virtual Environment like this:
```console
marie@compute$ module load Anaconda3/2022.05
Module Anaconda3/2022.05 loaded.
marie@compute$ mkdir user-kernel # please use workspaces!
marie@compute$ cd user-kernel
marie@compute$ python3 -m venv --system-site-packages my-kernel
(my-kernel) marie@compute$ pip install ipykernel
Successfully installed [...] ipykernel-x.x.x ipython-x.x.x [...]
```
Then continue with the steps below.
After following the initialization of the environment (above),
the usage of Python's Package manager
`pip`
is the same:
...
...
@@ -158,14 +90,9 @@ Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/my-kernel
(my-kernel) marie@compute$
deactivate
```
!!! warning
Take care to select the appropriate standard environment (as mentioned above)
when [spawning a new session](jupyterhub.md#start-a-session).
## Conda Environment
Load the needed module depending on
partition
architecture:
Load the needed module depending on
Cluster
architecture:
=== "Nodes with x86_64 CPU"
```
console
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment