diff --git a/doc.zih.tu-dresden.de/docs/access/jupyterhub_for_teaching.md b/doc.zih.tu-dresden.de/docs/access/jupyterhub_for_teaching.md index d3ab18892984458d53b3b55bbcf5ce70d6592a51..84367bda4f78961c8c4675bb0b883a0c0f9b0e74 100644 --- a/doc.zih.tu-dresden.de/docs/access/jupyterhub_for_teaching.md +++ b/doc.zih.tu-dresden.de/docs/access/jupyterhub_for_teaching.md @@ -73,7 +73,7 @@ The spawn form now offers a quick start mode by passing URL parameters. !!! example The following link would create a jupyter notebook session on the - `interactive` partition with the `test` environment being loaded: + partition `interactive` with the `test` environment being loaded: ``` https://taurus.hrsk.tu-dresden.de/jupyter/hub/spawn#/~(partition~'interactive~environment~'test) diff --git a/doc.zih.tu-dresden.de/docs/contrib/content_rules.md b/doc.zih.tu-dresden.de/docs/contrib/content_rules.md index 5afcf96350ddf28981dc651cadd8381f06b4bc6c..1b1d5b460d78b65f5f8516b827e06e7782480fe8 100644 --- a/doc.zih.tu-dresden.de/docs/contrib/content_rules.md +++ b/doc.zih.tu-dresden.de/docs/contrib/content_rules.md @@ -394,12 +394,12 @@ This should help to avoid errors. | Localhost | `marie@local$` | | Login nodes | `marie@login$` | | Arbitrary compute node | `marie@compute$` | -| `haswell` partition | `marie@haswell$` | -| `ml` partition | `marie@ml$` | -| `alpha` partition | `marie@alpha$` | -| `romeo` partition | `marie@romeo$` | -| `julia` partition | `marie@julia$` | -| `dcv` partition | `marie@dcv$` | +| Partition `haswell` | `marie@haswell$` | +| Partition `ml` | `marie@ml$` | +| Partition `alpha` | `marie@alpha$` | +| Partition `romeo` | `marie@romeo$` | +| Partition `julia` | `marie@julia$` | +| Partition `dcv` | `marie@dcv$` | * **Always use a prompt**, even if there is no output provided for the shown command. * All code blocks which specify some general command templates, e.g. containing `<` and `>` diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md index dadc94855ecc71a229e0ab19b15b6837f2bbf872..561d0ed622ae6b514866404a38ee5bc7d2f0c4ba 100644 --- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md @@ -20,7 +20,7 @@ It has 34 nodes, each with: ### Modules The easiest way is using the [module system](../software/modules.md). -The software for the partition alpha is available in `modenv/hiera` module environment. +The software for the partition `alpha` is available in module environment `modenv/hiera`. To check the available modules for `modenv/hiera`, use the command diff --git a/doc.zih.tu-dresden.de/docs/software/big_data_frameworks.md b/doc.zih.tu-dresden.de/docs/software/big_data_frameworks.md index 4bd9634db24b8ba81a02368a4f51c0b46004885f..47c7567b1a063a4b67cca2982d53bf729b288295 100644 --- a/doc.zih.tu-dresden.de/docs/software/big_data_frameworks.md +++ b/doc.zih.tu-dresden.de/docs/software/big_data_frameworks.md @@ -41,7 +41,7 @@ The Spark and Flink modules are available in both `scs5` and `ml` environments. Thus, Spark and Flink can be executed using different CPU architectures, e.g., Haswell and Power9. Let us assume that two nodes should be used for the computation. Use a `srun` command similar to -the following to start an interactive session using the partition haswell. The following code +the following to start an interactive session using the partition `haswell`. The following code snippet shows a job submission to haswell nodes with an allocation of two nodes with 60000 MB main memory exclusively for one hour: diff --git a/doc.zih.tu-dresden.de/docs/software/custom_easy_build_environment.md b/doc.zih.tu-dresden.de/docs/software/custom_easy_build_environment.md index e9283d6d8063bbc9dc6d4c2bd520d9dc96f341b1..9232e7472e8acc0254f876352310be0355d9aa4e 100644 --- a/doc.zih.tu-dresden.de/docs/software/custom_easy_build_environment.md +++ b/doc.zih.tu-dresden.de/docs/software/custom_easy_build_environment.md @@ -61,7 +61,7 @@ marie@login$ ws_list | grep 'directory.*EasyBuild' put commands in a batch file and source it. The latter is recommended for non-interactive jobs, using the command `sbatch` instead of `srun`. For the sake of illustration, we use an interactive job as an example. Depending on the partitions that you want the module to be usable on -later, you need to select nodes with the same architecture. Thus, use nodes from partition ml for +later, you need to select nodes with the same architecture. Thus, use nodes from partition `ml` for building, if you want to use the module on nodes of that partition. In this example, we assume that we want to use the module on nodes with x86 architecture and thus, we use Haswell nodes. @@ -80,14 +80,14 @@ environment variable called `WORKSPACE` with the path to your workspace: marie@compute$ export WORKSPACE=/scratch/ws/1/marie-EasyBuild #see output of ws_list above ``` -**Step 4:** Load the correct module environment `modenv` according to your current or target +**Step 4:** Load the correct module environment `modenv` according to your current or target architecture: -=== "x86 (default, e. g. partition haswell)" +=== "x86 (default, e. g. partition `haswell`)" ```console marie@compute$ module load modenv/scs5 ``` -=== "Power9 (partition ml)" +=== "Power9 (partition `ml`)" ```console marie@ml$ module load modenv/ml ``` diff --git a/doc.zih.tu-dresden.de/docs/software/hyperparameter_optimization.md b/doc.zih.tu-dresden.de/docs/software/hyperparameter_optimization.md index 688ada0e2aabf973f545d54b1c15168de98aa912..885e617f3f47797acd8858e18c363807e77bde67 100644 --- a/doc.zih.tu-dresden.de/docs/software/hyperparameter_optimization.md +++ b/doc.zih.tu-dresden.de/docs/software/hyperparameter_optimization.md @@ -190,8 +190,8 @@ There are the following script preparation steps for OmniOpt: ``` 1. Testing script functionality and determine software requirements for the chosen - [partition](../jobs_and_resources/partitions_and_limits.md). In the following, the alpha - partition is used. Please note the parameters `--out-layer1`, `--batchsize`, `--epochs` when + [partition](../jobs_and_resources/partitions_and_limits.md). In the following, the + partition `alpha` is used. Please note the parameters `--out-layer1`, `--batchsize`, `--epochs` when calling the Python script. Additionally, note the `RESULT` string with the output for OmniOpt. ??? hint "Hint for installing Python modules" diff --git a/doc.zih.tu-dresden.de/docs/software/machine_learning.md b/doc.zih.tu-dresden.de/docs/software/machine_learning.md index e293b007a9c07fbaf41ba3ec7ce25f29024f44d7..825c83ba2a4fd0993a9b771d4d82758e9b0de20b 100644 --- a/doc.zih.tu-dresden.de/docs/software/machine_learning.md +++ b/doc.zih.tu-dresden.de/docs/software/machine_learning.md @@ -5,25 +5,25 @@ For machine learning purposes, we recommend to use the partitions `alpha` and/or ## Partition `ml` -The compute nodes of the partition ML are built on the base of +The compute nodes of the partition `ml` are built on the base of [Power9 architecture](https://www.ibm.com/it-infrastructure/power/power9) from IBM. The system was created for AI challenges, analytics and working with data-intensive workloads and accelerated databases. The main feature of the nodes is the ability to work with the [NVIDIA Tesla V100](https://www.nvidia.com/en-gb/data-center/tesla-v100/) GPU with **NV-Link** support that allows a total bandwidth with up to 300 GB/s. Each node on the -partition ML has 6x Tesla V-100 GPUs. You can find a detailed specification of the partition in our +partition `ml` has 6x Tesla V-100 GPUs. You can find a detailed specification of the partition in our [Power9 documentation](../jobs_and_resources/hardware_overview.md). !!! note - The partition ML is based on the Power9 architecture, which means that the software built + The partition `ml` is based on the Power9 architecture, which means that the software built for x86_64 will not work on this partition. Also, users need to use the modules which are specially build for this architecture (from `modenv/ml`). ### Modules -On the partition ML load the module environment: +On the partition `ml` load the module environment: ```console marie@ml$ module load modenv/ml @@ -32,19 +32,20 @@ The following have been reloaded with a version change: 1) modenv/scs5 => moden ### Power AI -There are tools provided by IBM, that work on partition ML and are related to AI tasks. +There are tools provided by IBM, that work on partition `ml` and are related to AI tasks. For more information see our [Power AI documentation](power_ai.md). ## Partition: Alpha -Another partition for machine learning tasks is Alpha. It is mainly dedicated to -[ScaDS.AI](https://scads.ai/) topics. Each node on Alpha has 2x AMD EPYC CPUs, 8x NVIDIA A100-SXM4 -GPUs, 1 TB RAM and 3.5 TB local space (`/tmp`) on an NVMe device. You can find more details of the -partition in our [Alpha Centauri](../jobs_and_resources/alpha_centauri.md) documentation. +Another partition for machine learning tasks is `alpha`. It is mainly dedicated to +[ScaDS.AI](https://scads.ai/) topics. Each node on partition `alpha` has 2x AMD EPYC CPUs, 8x NVIDIA +A100-SXM4 GPUs, 1 TB RAM and 3.5 TB local space (`/tmp`) on an NVMe device. You can find more +details of the partition in our [Alpha Centauri](../jobs_and_resources/alpha_centauri.md) +documentation. ### Modules -On the partition alpha load the module environment: +On the partition `alpha` load the module environment: ```console marie@alpha$ module load modenv/hiera @@ -53,7 +54,7 @@ The following have been reloaded with a version change: 1) modenv/ml => modenv/ !!! note - On partition Alpha, the most recent modules are build in `hiera`. Alternative modules might be + On partition `alpha`, the most recent modules are build in `hiera`. Alternative modules might be build in `scs5`. ## Machine Learning via Console @@ -82,7 +83,7 @@ create documents containing live code, equations, visualizations, and narrative TensorFlow or PyTorch) on ZIH systems and to run your Jupyter notebooks on HPC nodes. After accessing JupyterHub, you can start a new session and configure it. For machine learning -purposes, select either partition **Alpha** or **ML** and the resources, your application requires. +purposes, select either partition `alpha` or `ml` and the resources, your application requires. In your session you can use [Python](data_analytics_with_python.md#jupyter-notebooks), [R](data_analytics_with_r.md#r-in-jupyterhub) or [RStudio](data_analytics_with_rstudio.md) for your diff --git a/doc.zih.tu-dresden.de/docs/software/modules.md b/doc.zih.tu-dresden.de/docs/software/modules.md index 9cf35854e953b4d9c2f9380d16217dec91dfea2a..74f67821cac0c8b030b06079f86e2514030fa5d6 100644 --- a/doc.zih.tu-dresden.de/docs/software/modules.md +++ b/doc.zih.tu-dresden.de/docs/software/modules.md @@ -133,8 +133,8 @@ marie@compute$ module load modenv/ml ### modenv/ml -* data analytics software (for use on the partition ml) -* necessary to run most software on the partition ml +* data analytics software (for use on the partition `ml`) +* necessary to run most software on the partition `ml` (The instruction set [Power ISA](https://en.wikipedia.org/wiki/Power_ISA#Power_ISA_v.3.0) is different from the usual x86 instruction set. Thus the 'machine code' of other modenvs breaks). diff --git a/doc.zih.tu-dresden.de/docs/software/power_ai.md b/doc.zih.tu-dresden.de/docs/software/power_ai.md index 5d1c397ab00d66fe61fc41fb4cee1efaeb25801b..1488d85f9f6b4a749b5535dea59474a7c2cf36a4 100644 --- a/doc.zih.tu-dresden.de/docs/software/power_ai.md +++ b/doc.zih.tu-dresden.de/docs/software/power_ai.md @@ -5,7 +5,7 @@ the PowerAI Framework for Machine Learning. In the following the links are valid for PowerAI version 1.5.4. !!! warning - The information provided here is available from IBM and can be used on partition ml only! + The information provided here is available from IBM and can be used on partition `ml` only! ## General Overview diff --git a/doc.zih.tu-dresden.de/docs/software/python_virtual_environments.md b/doc.zih.tu-dresden.de/docs/software/python_virtual_environments.md index 69e34f20e572d994ed2c1abc05fc1deb39091d75..026e194ee8e2f28be8e24eae4862cd358427792e 100644 --- a/doc.zih.tu-dresden.de/docs/software/python_virtual_environments.md +++ b/doc.zih.tu-dresden.de/docs/software/python_virtual_environments.md @@ -68,7 +68,7 @@ the environment as follows: ??? example - This is an example on partition Alpha. The example creates a conda virtual environment, and + This is an example on partition `alpha`. The example creates a conda virtual environment, and installs the package `torchvision` with conda. ```console marie@login$ srun --partition=alpha-interactive --nodes=1 --gres=gpu:1 --time=01:00:00 --pty bash @@ -179,7 +179,7 @@ can deactivate the conda environment as follows: ??? example - This is an example on partition Alpha. The example creates a conda virtual environment, and + This is an example on partition `alpha`. The example creates a conda virtual environment, and installs the package `torchvision` with conda. ```console marie@login$ srun --partition=alpha-interactive --nodes=1 --gres=gpu:1 --time=01:00:00 --pty bash diff --git a/doc.zih.tu-dresden.de/docs/software/tensorflow.md b/doc.zih.tu-dresden.de/docs/software/tensorflow.md index 746c78a39b21845b5164217390dcc141467345fa..df655fdb2dea0b2c8ab39f95b4544a261bb1c534 100644 --- a/doc.zih.tu-dresden.de/docs/software/tensorflow.md +++ b/doc.zih.tu-dresden.de/docs/software/tensorflow.md @@ -17,13 +17,13 @@ to find out, which TensorFlow modules are available on your partition. On ZIH systems, TensorFlow 2 is the default module version. For compatibility hints between TensorFlow 2 and TensorFlow 1, see the corresponding [section below](#compatibility-tf2-and-tf1). -We recommend using partitions **Alpha** and/or **ML** when working with machine learning workflows +We recommend using partitions `alpha` and/or `ml` when working with machine learning workflows and the TensorFlow library. You can find detailed hardware specification in our [Hardware](../jobs_and_resources/hardware_overview.md) documentation. ## TensorFlow Console -On the partition Alpha, load the module environment: +On the partition `alpha`, load the module environment: ```console marie@alpha$ module load modenv/scs5 @@ -47,7 +47,7 @@ marie@alpha$ module avail TensorFlow [...] ``` -On the partition ML load the module environment: +On the partition `ml` load the module environment: ```console marie@ml$ module load modenv/ml