From 74378c008777e11e6d58884bf77f27d06f02917c Mon Sep 17 00:00:00 2001
From: Ulf Markwardt <ulf.markwardt@tu-dresden.de>
Date: Thu, 1 Jun 2023 13:21:02 +0200
Subject: [PATCH] =?UTF-8?q?schritt=20f=C3=BCr=20schritt?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
---
.../docs/archive/hardware_overview_2022.md | 109 +++++++++++++++++
.../slurm_profiling.md | 0
.../jobs_and_resources/hardware_overview.md | 115 +++++++-----------
.../hardware_overview_2023.md | 84 -------------
.../docs/jobs_and_resources/overview.md | 12 +-
doc.zih.tu-dresden.de/mkdocs.yml | 4 +-
6 files changed, 163 insertions(+), 161 deletions(-)
create mode 100644 doc.zih.tu-dresden.de/docs/archive/hardware_overview_2022.md
rename doc.zih.tu-dresden.de/docs/{jobs_and_resources => archive}/slurm_profiling.md (100%)
delete mode 100644 doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview_2023.md
diff --git a/doc.zih.tu-dresden.de/docs/archive/hardware_overview_2022.md b/doc.zih.tu-dresden.de/docs/archive/hardware_overview_2022.md
new file mode 100644
index 000000000..538296b4e
--- /dev/null
+++ b/doc.zih.tu-dresden.de/docs/archive/hardware_overview_2022.md
@@ -0,0 +1,109 @@
+# HPC Resources
+
+HPC resources in ZIH systems comprise the *High Performance Computing and Storage Complex* and its
+extension *High Performance Computing – Data Analytics*. In total it offers scientists
+about 60,000 CPU cores and a peak performance of more than 1.5 quadrillion floating point
+operations per second. The architecture specifically tailored to data-intensive computing, Big Data
+analytics, and artificial intelligence methods with extensive capabilities for energy measurement
+and performance monitoring provides ideal conditions to achieve the ambitious research goals of the
+users and the ZIH.
+
+## Login and Export Nodes
+
+- 4 Login-Nodes `tauruslogin[3-6].hrsk.tu-dresden.de`
+ - Each login node is equipped with 2x Intel(R) Xeon(R) CPU E5-2680 v3 with 24 cores in total @
+ 2.50 GHz, Multithreading disabled, 64 GB RAM, 128 GB SSD local disk
+ - IPs: 141.30.73.\[102-105\]
+- 2 Data-Transfer-Nodes `taurusexport[3-4].hrsk.tu-dresden.de`
+ - DNS Alias `taurusexport.hrsk.tu-dresden.de`
+ - 2 Servers without interactive login, only available via file transfer protocols
+ (`rsync`, `ftp`)
+ - IPs: 141.30.73.\[82,83\]
+ - Further information on the usage is documented on the site
+ [Export Nodes](../data_transfer/export_nodes.md)
+
+## AMD Rome CPUs + NVIDIA A100
+
+- 34 nodes, each with
+ - 8 x NVIDIA A100-SXM4 Tensor Core-GPUs
+ - 2 x AMD EPYC CPU 7352 (24 cores) @ 2.3 GHz, Multithreading available
+ - 1 TB RAM
+ - 3.5 TB local memory on NVMe device at `/tmp`
+- Hostnames: `taurusi[8001-8034]`
+- Slurm partition: `alpha`
+- Further information on the usage is documented on the site [Alpha Centauri Nodes](alpha_centauri.md)
+
+## Island 7 - AMD Rome CPUs
+
+- 192 nodes, each with
+ - 2 x AMD EPYC CPU 7702 (64 cores) @ 2.0 GHz, Multithreading available
+ - 512 GB RAM
+ - 200 GB local memory on SSD at `/tmp`
+- Hostnames: `taurusi[7001-7192]`
+- Slurm partition: `romeo`
+- Further information on the usage is documented on the site [AMD Rome Nodes](rome_nodes.md)
+
+## Large SMP System HPE Superdome Flex
+
+- 1 node, with
+ - 32 x Intel(R) Xeon(R) Platinum 8276M CPU @ 2.20 GHz (28 cores)
+ - 47 TB RAM
+- Configured as one single node
+- 48 TB RAM (usable: 47 TB - one TB is used for cache coherence protocols)
+- 370 TB of fast NVME storage available at `/nvme/<projectname>`
+- Hostname: `taurussmp8`
+- Slurm partition: `julia`
+- Further information on the usage is documented on the site [HPE Superdome Flex](sd_flex.md)
+
+## IBM Power9 Nodes for Machine Learning
+
+For machine learning, we have IBM AC922 nodes installed with this configuration:
+
+- 32 nodes, each with
+ - 2 x IBM Power9 CPU (2.80 GHz, 3.10 GHz boost, 22 cores)
+ - 256 GB RAM DDR4 2666 MHz
+ - 6 x NVIDIA VOLTA V100 with 32 GB HBM2
+ - NVLINK bandwidth 150 GB/s between GPUs and host
+- Hostnames: `taurusml[1-32]`
+- Slurm partition: `ml`
+
+## Island 6 - Intel Haswell CPUs
+
+- 612 nodes, each with
+ - 2 x Intel(R) Xeon(R) CPU E5-2680 v3 (12 cores) @ 2.50 GHz, Multithreading disabled
+ - 128 GB local memory on SSD
+- Varying amounts of main memory (selected automatically by the batch system for you according to
+ your job requirements)
+ * 594 nodes with 2.67 GB RAM per core (64 GB in total): `taurusi[6001-6540,6559-6612]`
+ - 18 nodes with 10.67 GB RAM per core (256 GB in total): `taurusi[6541-6558]`
+- Hostnames: `taurusi[6001-6612]`
+- Slurm Partition: `haswell`
+
+??? hint "Node topology"
+
+ 
+ {: align=center}
+
+## Island 2 Phase 2 - Intel Haswell CPUs + NVIDIA K80 GPUs
+
+- 64 nodes, each with
+ - 2 x Intel(R) Xeon(R) CPU E5-E5-2680 v3 (12 cores) @ 2.50 GHz, Multithreading disabled
+ - 64 GB RAM (2.67 GB per core)
+ - 128 GB local memory on SSD
+ - 4 x NVIDIA Tesla K80 (12 GB GDDR RAM) GPUs
+- Hostnames: `taurusi[2045-2108]`
+- Slurm Partition: `gpu2`
+- Node topology, same as [island 4 - 6](#island-6-intel-haswell-cpus)
+
+## SMP Nodes - up to 2 TB RAM
+
+- 5 Nodes, each with
+ - 4 x Intel(R) Xeon(R) CPU E7-4850 v3 (14 cores) @ 2.20 GHz, Multithreading disabled
+ - 2 TB RAM
+- Hostnames: `taurussmp[3-7]`
+- Slurm partition: `smp2`
+
+??? hint "Node topology"
+
+ 
+ {: align=center}
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_profiling.md b/doc.zih.tu-dresden.de/docs/archive/slurm_profiling.md
similarity index 100%
rename from doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_profiling.md
rename to doc.zih.tu-dresden.de/docs/archive/slurm_profiling.md
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md
index 538296b4e..22d4d9517 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md
@@ -1,26 +1,43 @@
# HPC Resources
-HPC resources in ZIH systems comprise the *High Performance Computing and Storage Complex* and its
-extension *High Performance Computing – Data Analytics*. In total it offers scientists
-about 60,000 CPU cores and a peak performance of more than 1.5 quadrillion floating point
-operations per second. The architecture specifically tailored to data-intensive computing, Big Data
-analytics, and artificial intelligence methods with extensive capabilities for energy measurement
-and performance monitoring provides ideal conditions to achieve the ambitious research goals of the
-users and the ZIH.
-
-## Login and Export Nodes
-
-- 4 Login-Nodes `tauruslogin[3-6].hrsk.tu-dresden.de`
- - Each login node is equipped with 2x Intel(R) Xeon(R) CPU E5-2680 v3 with 24 cores in total @
- 2.50 GHz, Multithreading disabled, 64 GB RAM, 128 GB SSD local disk
- - IPs: 141.30.73.\[102-105\]
-- 2 Data-Transfer-Nodes `taurusexport[3-4].hrsk.tu-dresden.de`
- - DNS Alias `taurusexport.hrsk.tu-dresden.de`
- - 2 Servers without interactive login, only available via file transfer protocols
- (`rsync`, `ftp`)
- - IPs: 141.30.73.\[82,83\]
- - Further information on the usage is documented on the site
- [Export Nodes](../data_transfer/export_nodes.md)
+The architecture specifically tailored to data-intensive computing, Big Data
+analytics, and artificial intelligence methods with extensive capabilities
+for performance monitoring provides ideal conditions to achieve the ambitious
+research goals of the users and the ZIH.
+
+## Overview
+
+From the users' pespective, there are seperate clusters, all of them with their subdomains:
+
+| Name | Description | Year| DNS |
+| --- | --- | --- | --- |
+| **Barnard** | CPU cluster |2023| n[1001-1630].barnard.hpc.tu-dresden.de |
+| **Romeo** | CPU cluster |2020|r[001-192].romeo.hpc.tu-dresden.de |
+| **Alpha Centauri** | GPU cluster |2021|a[001-039].alpha.hpc.tu-dresden.de |
+| **Julia** | single SMP system |2021|julia.hpc.tu-dresden.de |
+| **Power** | IBM Power/GPU system |2018|m[001-032].power.hpc.tu-dresden.de |
+
+They run with their own Slurm batch system. Job submission is possible only from
+their respective login nodes.
+
+All clusters have access to these shared parallel file systems:
+
+| File system | Usable directory | Type | Capacity | Purpose |
+| --- | --- | --- | --- | --- |
+| `Home` | `/home` | Lustre | 40 TB |
+| `Project` | `/projects` | NFS | 40 TB |
+| `Scratch for large data / streams` | `/data/horse` | Lustre | 20 PB |
+| `Scratch for random access` | `/data/rabbit` | Lustre | 2 PB |
+| `Scratch for random access` | `/data/weasel` | WEKA | 232 TB |
+| `Scratch for random access` | `/data/weasel` | WEKA | 232 TB |
+
+## Barnard - Intel Sapphire Rapids CPUs
+
+- 630 diskless nodes, each with
+ - 2 x Intel(R) Xeon(R) CPU E5-2680 v3 (52 cores) @ 2.50 GHz, Multithreading enabled
+ - 512 GB RAM
+- Hostnames: `n1[001-630].barnard.hpc.tu-dresden.de`
+- Login nodes: `login[1-4].barnard.hpc.tu-dresden.de`
## AMD Rome CPUs + NVIDIA A100
@@ -29,8 +46,8 @@ users and the ZIH.
- 2 x AMD EPYC CPU 7352 (24 cores) @ 2.3 GHz, Multithreading available
- 1 TB RAM
- 3.5 TB local memory on NVMe device at `/tmp`
-- Hostnames: `taurusi[8001-8034]`
-- Slurm partition: `alpha`
+- Hostnames: `taurusi[8001-8034]` -> `a[1-37].alpha.hpc.tu-dresden.de`
+- Login nodes: `login[1-2].alpha.hpc.tu-dresden.de`
- Further information on the usage is documented on the site [Alpha Centauri Nodes](alpha_centauri.md)
## Island 7 - AMD Rome CPUs
@@ -39,8 +56,8 @@ users and the ZIH.
- 2 x AMD EPYC CPU 7702 (64 cores) @ 2.0 GHz, Multithreading available
- 512 GB RAM
- 200 GB local memory on SSD at `/tmp`
-- Hostnames: `taurusi[7001-7192]`
-- Slurm partition: `romeo`
+- Hostnames: `taurusi[7001-7192]` -> `r[1-190].romeo.hpc.tu-dresden.de`
+- Login nodes: `login[1-2].romeo.hpc.tu-dresden.de`
- Further information on the usage is documented on the site [AMD Rome Nodes](rome_nodes.md)
## Large SMP System HPE Superdome Flex
@@ -51,8 +68,7 @@ users and the ZIH.
- Configured as one single node
- 48 TB RAM (usable: 47 TB - one TB is used for cache coherence protocols)
- 370 TB of fast NVME storage available at `/nvme/<projectname>`
-- Hostname: `taurussmp8`
-- Slurm partition: `julia`
+- Hostname: `taurussmp8` -> `julia.hpc.tu-dresden.de`
- Further information on the usage is documented on the site [HPE Superdome Flex](sd_flex.md)
## IBM Power9 Nodes for Machine Learning
@@ -64,46 +80,5 @@ For machine learning, we have IBM AC922 nodes installed with this configuration:
- 256 GB RAM DDR4 2666 MHz
- 6 x NVIDIA VOLTA V100 with 32 GB HBM2
- NVLINK bandwidth 150 GB/s between GPUs and host
-- Hostnames: `taurusml[1-32]`
-- Slurm partition: `ml`
-
-## Island 6 - Intel Haswell CPUs
-
-- 612 nodes, each with
- - 2 x Intel(R) Xeon(R) CPU E5-2680 v3 (12 cores) @ 2.50 GHz, Multithreading disabled
- - 128 GB local memory on SSD
-- Varying amounts of main memory (selected automatically by the batch system for you according to
- your job requirements)
- * 594 nodes with 2.67 GB RAM per core (64 GB in total): `taurusi[6001-6540,6559-6612]`
- - 18 nodes with 10.67 GB RAM per core (256 GB in total): `taurusi[6541-6558]`
-- Hostnames: `taurusi[6001-6612]`
-- Slurm Partition: `haswell`
-
-??? hint "Node topology"
-
- 
- {: align=center}
-
-## Island 2 Phase 2 - Intel Haswell CPUs + NVIDIA K80 GPUs
-
-- 64 nodes, each with
- - 2 x Intel(R) Xeon(R) CPU E5-E5-2680 v3 (12 cores) @ 2.50 GHz, Multithreading disabled
- - 64 GB RAM (2.67 GB per core)
- - 128 GB local memory on SSD
- - 4 x NVIDIA Tesla K80 (12 GB GDDR RAM) GPUs
-- Hostnames: `taurusi[2045-2108]`
-- Slurm Partition: `gpu2`
-- Node topology, same as [island 4 - 6](#island-6-intel-haswell-cpus)
-
-## SMP Nodes - up to 2 TB RAM
-
-- 5 Nodes, each with
- - 4 x Intel(R) Xeon(R) CPU E7-4850 v3 (14 cores) @ 2.20 GHz, Multithreading disabled
- - 2 TB RAM
-- Hostnames: `taurussmp[3-7]`
-- Slurm partition: `smp2`
-
-??? hint "Node topology"
-
- 
- {: align=center}
+- Hostnames: `taurusml[1-32]` -> `p[1-29].ml.hpc.tu-dresden.de`
+- Login nodes: `login[1-2].ml.hpc.tu-dresden.de``
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview_2023.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview_2023.md
deleted file mode 100644
index 0013a115d..000000000
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview_2023.md
+++ /dev/null
@@ -1,84 +0,0 @@
-# HPC Resources
-
-The architecture specifically tailored to data-intensive computing, Big Data
-analytics, and artificial intelligence methods with extensive capabilities for performance monitoring provides ideal conditions to achieve the ambitious research goals of the users and the ZIH.
-
-## Overview
-
-From the users' pespective, there are seperate clusters, all of them with their subdomains:
-
-| Name | Description | Year| DNS |
-| --- | --- | --- | --- |
-| **Barnard** | CPU cluster |2023| *.barnard.hpc.tu-dresden.de |
-| **Romeo** | CPU cluster |2020|*.romeo.hpc.tu-dresden.de |
-| **Alpha Centauri** | GPU cluster |2021|*.alpha.hpc.tu-dresden.de |
-| **Julia** | single SMP system |2021|julia.hpc.tu-dresden.de |
-| **Power** | IBM Power/GPU system |2018|*.power.hpc.tu-dresden.de |
-
-
-They run with their own Slurm batch system. Job submission is possible only from their login nodes.
-
-All clusters have access to these shared parallel file systems:
-
-| File system | Usable directory | Capacity | Purpose |
-| --- | --- | --- | --- |
-| `Lustre` | `/lustre/bulk` | 20 PB |
-| `Lustre` | `/lustre/fast` | 2 PB |
-| `Weka` | `/weka` | 232 TB |
-| `Home` | `/home` | 40 TB |
-
-## Barnard - Intel Sapphire Rapids CPUs
-
-- 630 diskless nodes, each with
- - 2 x Intel(R) Xeon(R) CPU E5-2680 v3 (52 cores) @ 2.50 GHz, Multithreading enabled
- - 512 GB RAM
-- Hostnames: `n1[001-630].barnard.hpc.tu-dresden.de`
-- Login nodes: `login[1-4].barnard.hpc.tu-dresden.de`
-
-
-## AMD Rome CPUs + NVIDIA A100
-
-- 34 nodes, each with
- - 8 x NVIDIA A100-SXM4 Tensor Core-GPUs
- - 2 x AMD EPYC CPU 7352 (24 cores) @ 2.3 GHz, Multithreading available
- - 1 TB RAM
- - 3.5 TB local memory on NVMe device at `/tmp`
-- Hostnames: `taurusi[8001-8034]` -> `n[1-37].alpha.hpc.tu-dresden.de`
-- Login nodes: `login[1-2].alpha.hpc.tu-dresden.de`
-- Further information on the usage is documented on the site [Alpha Centauri Nodes](alpha_centauri.md)
-
-## Island 7 - AMD Rome CPUs
-
-- 192 nodes, each with
- - 2 x AMD EPYC CPU 7702 (64 cores) @ 2.0 GHz, Multithreading available
- - 512 GB RAM
- - 200 GB local memory on SSD at `/tmp`
-- Hostnames: `taurusi[7001-7192]` -> `n[1-190].romeo.hpc.tu-dresden.de`
-- Login nodes: `login[1-2].romeo.hpc.tu-dresden.de`
-- Further information on the usage is documented on the site [AMD Rome Nodes](rome_nodes.md)
-
-## Large SMP System HPE Superdome Flex
-
-- 1 node, with
- - 32 x Intel(R) Xeon(R) Platinum 8276M CPU @ 2.20 GHz (28 cores)
- - 47 TB RAM
-- Configured as one single node
-- 48 TB RAM (usable: 47 TB - one TB is used for cache coherence protocols)
-- 370 TB of fast NVME storage available at `/nvme/<projectname>`
-- Hostname: `taurussmp8`
-- Slurm partition: `julia`
-- Further information on the usage is documented on the site [HPE Superdome Flex](sd_flex.md)
-
-## IBM Power9 Nodes for Machine Learning
-
-For machine learning, we have IBM AC922 nodes installed with this configuration:
-
-- 32 nodes, each with
- - 2 x IBM Power9 CPU (2.80 GHz, 3.10 GHz boost, 22 cores)
- - 256 GB RAM DDR4 2666 MHz
- - 6 x NVIDIA VOLTA V100 with 32 GB HBM2
- - NVLINK bandwidth 150 GB/s between GPUs and host
-- Hostnames: `taurusml[1-32]`
-- Slurm partition: `ml`
-
-
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md
index d3e9674ce..7ff66f559 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/overview.md
@@ -1,7 +1,7 @@
# HPC Resources and Jobs
-ZIH operates a high performance computing (HPC) system with more than 60.000 cores, 720 GPUs, and a
-flexible storage hierarchy with about 16 PB total capacity. The HPC system provides an optimal
+ZIH operates a high performance computing (HPC) system with more than 90.000 cores, 500 GPUs, and
+a flexible storage hierarchy with about 20 PB total capacity. The HPC system provides an optimal
research environment especially in the area of data analytics and machine learning as well as for
processing extremely large data sets. Moreover it is also a perfect platform for highly scalable,
data-intensive and compute-intensive applications.
@@ -58,12 +58,12 @@ automatically select a suitable partition depending on your memory and GPU requi
**MPI jobs:** For MPI jobs typically allocates one core per task. Several nodes could be allocated
if it is necessary. The batch system [Slurm](slurm.md) will automatically find suitable hardware.
-Normal compute nodes are perfect for this task.
**OpenMP jobs:** SMP-parallel applications can only run **within a node**, so it is necessary to
include the [batch system](slurm.md) options `-N 1` and `-n 1`. Using `--cpus-per-task N` Slurm will
start one task and you will have `N` CPUs. The maximum number of processors for an SMP-parallel
-program is 896 on partition `julia`, see [partitions](partitions_and_limits.md).
+program is 896 on partition `julia`, see [partitions](partitions_and_limits.md) (be aware that
+the application has to be developed with that large number of threads in mind).
Partitions with GPUs are best suited for **repetitive** and **highly-parallel** computing tasks. If
you have a task with potential [data parallelism](../software/gpu_programming.md) most likely that
@@ -71,7 +71,9 @@ you need the GPUs. Beyond video rendering, GPUs excel in tasks such as machine
simulations and risk modeling. Use the partitions `gpu2` and `ml` only if you need GPUs! Otherwise
using the x86-based partitions most likely would be more beneficial.
-**Interactive jobs:** Slurm can forward your X11 credentials to the first node (or even all) for a job
+**Interactive jobs:** An interactive job is the best choice for testing and development. See
+ [interactive-jobs](slurm.md).
+Slurm can forward your X11 credentials to the first node (or even all) for a job
with the `--x11` option. To use an interactive job you have to specify `-X` flag for the ssh login.
## Interactive vs. Batch Mode
diff --git a/doc.zih.tu-dresden.de/mkdocs.yml b/doc.zih.tu-dresden.de/mkdocs.yml
index d05959ffa..c76398406 100644
--- a/doc.zih.tu-dresden.de/mkdocs.yml
+++ b/doc.zih.tu-dresden.de/mkdocs.yml
@@ -99,7 +99,6 @@ nav:
- HPC Resources and Jobs:
- Overview: jobs_and_resources/overview.md
- HPC Resources:
- - Overview: jobs_and_resources/hardware_overview.md
- AMD Rome Nodes: jobs_and_resources/rome_nodes.md
- NVMe Storage: jobs_and_resources/nvme_storage.md
- Alpha Centauri: jobs_and_resources/alpha_centauri.md
@@ -111,7 +110,6 @@ nav:
- Partitions and Limits: jobs_and_resources/partitions_and_limits.md
- Slurm Job File Generator: jobs_and_resources/slurm_generator.md
- Checkpoint/Restart: jobs_and_resources/checkpoint_restart.md
- - Job Profiling: jobs_and_resources/slurm_profiling.md
- Binding and Distribution of Tasks: jobs_and_resources/binding_and_distribution_of_tasks.md
- User Support: support/support.md
- Archive:
@@ -124,7 +122,9 @@ nav:
- Platform LSF: archive/platform_lsf.md
- BeeGFS Filesystem on Demand: archive/beegfs_on_demand.md
- Jupyter Installation: archive/install_jupyter.md
+ - Job Profiling: jobs_and_resources/slurm_profiling.md
- Switched-Off Systems:
+ - Overview 2022: archive/hardware_overview_2022.md
- Overview: archive/systems_switched_off.md
- Migration From Deimos to Atlas: archive/migrate_to_atlas.md
- System Altix: archive/system_altix.md
--
GitLab