From 9654885093e0df3db7324f2e87dbd532f3fb4b54 Mon Sep 17 00:00:00 2001
From: Noah Trumpik <noah.trumpik@tu-dresden.de>
Date: Mon, 27 Nov 2023 17:32:23 +0100
Subject: [PATCH] barnard update: dataports
---
.../docs/access/security_restrictions.md | 2 +-
.../docs/access/ssh_login.md | 10 +--
.../data_lifecycle/intermediate_archive.md | 2 +-
.../data_lifecycle/longterm_preservation.md | 2 +-
.../{export_nodes.md => dataport_nodes.md} | 80 ++++++++++++-------
.../docs/data_transfer/overview.md | 6 +-
.../jobs_and_resources/hardware_overview.md | 18 ++---
.../docs/quickstart/getting_started.md | 14 ++--
.../docs/software/gpu_programming.md | 6 +-
9 files changed, 79 insertions(+), 61 deletions(-)
rename doc.zih.tu-dresden.de/docs/data_transfer/{export_nodes.md => dataport_nodes.md} (62%)
diff --git a/doc.zih.tu-dresden.de/docs/access/security_restrictions.md b/doc.zih.tu-dresden.de/docs/access/security_restrictions.md
index b43d631c0..21767df64 100644
--- a/doc.zih.tu-dresden.de/docs/access/security_restrictions.md
+++ b/doc.zih.tu-dresden.de/docs/access/security_restrictions.md
@@ -20,7 +20,7 @@ The most important items for ZIH systems are:
* **ssh from ZIH system** is only possible inside TU Dresden campus.
(Direct SSH access to other computing centers was the spreading vector of the recent incident.)
-Data transfer is possible via the [export nodes](../data_transfer/export_nodes.md). We are working
+Data transfer is possible via the [dataport nodes](../data_transfer/dataport_nodes.md). We are working
on a bandwidth-friendly solution.
We understand that all this will change convenient workflows. If the measurements would render your
diff --git a/doc.zih.tu-dresden.de/docs/access/ssh_login.md b/doc.zih.tu-dresden.de/docs/access/ssh_login.md
index 059d487c4..bb4495b72 100644
--- a/doc.zih.tu-dresden.de/docs/access/ssh_login.md
+++ b/doc.zih.tu-dresden.de/docs/access/ssh_login.md
@@ -77,7 +77,7 @@ in it (you can omit lines starting with `#`):
```bash
Host barnard
#For login (shell access)
- HostName barnard.hpc.tu-dresden.de
+ HostName login1.barnard.hpc.tu-dresden.de
#Put your ZIH-Login after keyword "User":
User marie
#Path to private key:
@@ -87,9 +87,9 @@ Host barnard
#Enable X11 forwarding for graphical applications and compression. You don't need parameter -X and -C when invoking ssh then.
ForwardX11 yes
Compression yes
-Host taurusexport
+Host dataport
#For copying data without shell access
- HostName taurusexport.hrsk.tu-dresden.de
+ HostName dataport1.hpc.tu-dresden.de
#Put your ZIH-Login after keyword "User":
User marie
#Path to private key:
@@ -104,8 +104,8 @@ Afterwards, you can connect to the ZIH system using:
marie@local$ ssh barnard
```
-If you want to copy data from/to ZIH systems, please refer to [Export Nodes: Transfer Data to/from
-ZIH's Filesystems](../data_transfer/export_nodes.md) for more information on export nodes.
+If you want to copy data from/to ZIH systems, please refer to [Dataport Nodes: Transfer Data to/from
+ZIH's Filesystems](../data_transfer/dataport_nodes.md) for more information on dataport nodes.
## X11-Forwarding
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md
index 894626208..952b4f691 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md
@@ -44,7 +44,7 @@ marie@login$ dtrsync -av /archiv/marie/results /scratch/marie/results
!!! note "Listing files in archive"
- The intermediate archive is not mounted on the login nodes, but only on the [export nodes](../data_transfer/export_nodes.md).
+ The intermediate archive is not mounted on the login nodes, but only on the [dataport nodes](../data_transfer/dataport_nodes.md).
In order to list the user's files in the archive use the `dtls` command
```console
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md
index 0b78f1cda..558b24d3c 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/longterm_preservation.md
@@ -92,7 +92,7 @@ require some data preparation (e.g. packaging), but serve different use cases:
The intermediate archive is a tape storage easily accessible as a directory
(`/archiv/<HRSK-project>/` or `/archiv/<login>/`) using the
-[export nodes](../data_transfer/export_nodes.md)
+[dataport nodes](../data_transfer/dataport_nodes.md)
and
[Datamover tools](https://doc.zih.tu-dresden.de/data_transfer/datamover/) to move your data to.
For detailed information please visit the
diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md b/doc.zih.tu-dresden.de/docs/data_transfer/dataport_nodes.md
similarity index 62%
rename from doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md
rename to doc.zih.tu-dresden.de/docs/data_transfer/dataport_nodes.md
index 2b3a3da9e..3b3b845fa 100644
--- a/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md
+++ b/doc.zih.tu-dresden.de/docs/data_transfer/dataport_nodes.md
@@ -1,18 +1,30 @@
-# Transfer Data to/from ZIH Systems via Export Nodes
+# Transfer Data to/from ZIH Systems via Dataport Nodes
-To copy large data to/from ZIH systems, the so-called **export nodes** should be used. While it is
+To copy large data to/from ZIH systems, the so-called **dataport nodes** should be used. While it is
possible to transfer small files directly via the login nodes, they are not intended to be used that
way. Furthermore, longer transfers will hit the CPU time limit on the login nodes, i.e. the process
-get killed. The **export nodes** have a better uplink (10 GBit/s) allowing for higher bandwidth. Note
-that you cannot log in via SSH to the export nodes, but only use `scp`, `rsync` or `sftp`
+get killed. The **dataport nodes** have a better uplink (10 GBit/s) allowing for higher bandwidth. Note
+that you cannot log in via SSH to the dataport nodes, but only use <!-- [NT] currently not available:`scp`, -->`rsync` or `sftp`
(incl. FTP-clients like e.g. [FileZilla](https://filezilla-project.org/)) on them.
-The export nodes are reachable under the hostname `taurusexport.hrsk.tu-dresden.de` (or
-`taurusexport3.hrsk.tu-dresden.de` and `taurusexport4.hrsk.tu-dresden.de`).
+The dataport nodes are reachable under the hostnames
-Please keep in mind that there are different
-[filesystems](../data_lifecycle/file_systems.md#recommendations-for-filesystem-usage). Choose the
-one that matches your needs.
+- `dataport1.hpc.tu-dresden.de` (IP: 141.30.73.4)
+- `dataport2.hpc.tu-dresden.de` (IP: 141.30.73.5)
+
+Through the usage of these dataport nodes, you can bring your data to ZIH HPC systems or get data
+from there - they have access to the different HPC
+[filesystems](../data_lifecycle/file_systems.md#recommendations-for-filesystem-usage).
+As of 11/2023, the following directories are accessible:
+
+- `/home`
+- `/projects`
+- `/data/horse`
+- `/data/walrus`
+- `/data/archiv`
+- (`/data/software`)
+- (`/data/old/home`)
+- (`/data/old/software`)
## Access From Linux
@@ -22,7 +34,7 @@ are explained in the following section in more detail.
!!! important
The following explanations require that you have already set up your
[SSH configuration](../access/ssh_login.md#configuring-default-parameters-for-ssh).
-
+<!-- [NT] scp currently not available
### SCP
The tool [`scp`](https://www.man7.org/linux/man-pages/man1/scp.1.html)
@@ -32,35 +44,35 @@ in a directory, the option `-r` has to be specified.
??? example "Example: Copy a file from your workstation to ZIH systems"
```bash
- marie@local$ scp <file> taurusexport:<target-location>
+ marie@local$ scp <file> dataport:<target-location>
# Add -r to copy whole directory
- marie@local$ scp -r <directory> taurusexport:<target-location>
+ marie@local$ scp -r <directory> dataport:<target-location>
```
For example, if you want to copy your data file `mydata.csv` to the directory `input` in your
home directory, you would use the following:
```console
- marie@local$ scp mydata.csv taurusexport:input/
+ marie@local$ scp mydata.csv dataport:input/
```
??? example "Example: Copy a file from ZIH systems to your workstation"
```bash
- marie@local$ scp taurusexport:<file> <target-location>
+ marie@local$ scp dataport:<file> <target-location>
# Add -r to copy whole directory
- marie@local$ scp -r taurusexport:<directory> <target-location>
+ marie@local$ scp -r dataport:<directory> <target-location>
```
For example, if you have a directory named `output` in your home directory on ZIH systems and
you want to copy it to the directory `/tmp` on your workstation, you would use the following:
```console
- marie@local$ scp -r taurusexport:output /tmp
+ marie@local$ scp -r dataport:output /tmp
```
-
+-->
### SFTP
The tool [`sftp`](https://man7.org/linux/man-pages/man1/sftp.1.html) (OpenSSH secure file transfer)
@@ -69,24 +81,29 @@ use compression to increase performance.
`sftp` is basically a virtual command line, which you could access and exit as follows.
+!!! warning "Note"
+ It is important from which point in your directory tree you 'enter' `sftp`!
+ The current working directory (double ckeck with `pwd`) will be the target folder on your local
+ machine from/to which remote files from the ZIH system will be put/get by `sftp`.
+ The local folder might also be changed during a session with special commands.
+ During the `sftp` session, you can use regular commands like `ls`, `cd`, `pwd` etc.
+ But if you wish to access your local workstation, these must be prefixed with the letter `l`
+ (`l`ocal), e.g., `lls`, `lcd` or `lpwd`.
+
```console
# Enter virtual command line
-marie@local$ sftp taurusexport
+marie@local$ sftp dataport
# Exit virtual command line
sftp> exit
# or
sftp> <Ctrl+D>
```
-After that you have access to the filesystem on ZIH systems, you can use the same commands as on
-your local workstation, e.g., `ls`, `cd`, `pwd` etc. If you would access to your local workstation
-from this virtual command line, then you have to prefix the command with the letter `l`
-(`l`ocal),e.g., `lls`, `lcd` or `lpwd`.
-
??? example "Example: Copy a file from your workstation to ZIH systems"
```console
- marie@local$ sftp taurusexport
+ marie@local$ cd my/local/work
+ marie@local$ sftp dataport
# Copy file
sftp> put <file>
# Copy directory
@@ -96,9 +113,11 @@ from this virtual command line, then you have to prefix the command with the let
??? example "Example: Copy a file from ZIH systems to your local workstation"
```console
- marie@local$ sftp taurusexport
+ marie@local$ sftp dataport
# Copy file
sftp> get <file>
+ # change local (target) directory
+ sftp> lcd /my/local/work
# Copy directory
sftp> get -r <directory>
```
@@ -118,18 +137,18 @@ the local machine.
```console
# Copy file
- marie@local$ rsync <file> taurusexport:<target-location>
+ marie@local$ rsync <file> dataport:<target-location>
# Copy directory
- marie@local$ rsync -r <directory> taurusexport:<target-location>
+ marie@local$ rsync -r <directory> dataport:<target-location>
```
??? example "Example: Copy a file from ZIH systems to your local workstation"
```console
# Copy file
- marie@local$ rsync taurusexport:<file> <target-location>
+ marie@local$ rsync dataport:<file> <target-location>
# Copy directory
- marie@local$ rsync -r taurusexport:<directory> <target-location>
+ marie@local$ rsync -r dataport:<directory> <target-location>
```
## Access From Windows
@@ -138,7 +157,7 @@ the local machine.
Windows 10 (1809 and higher) comes with a
[built-in OpenSSH support](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview)
-including the above described [SCP](#scp) and [SFTP](#sftp).
+including the above described <!--[SCP](#scp) and -->[SFTP](#sftp).
### GUI - Using WinSCP
@@ -147,6 +166,7 @@ First you have to install [WinSCP](http://winscp.net/eng/download.php).
Then you have to execute the WinSCP application and configure some
option as described below.
+<!-- screenshots will have to be updated-->
{: align="center"}
diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/overview.md b/doc.zih.tu-dresden.de/docs/data_transfer/overview.md
index 6e8a1bf1c..3ba882cca 100644
--- a/doc.zih.tu-dresden.de/docs/data_transfer/overview.md
+++ b/doc.zih.tu-dresden.de/docs/data_transfer/overview.md
@@ -1,6 +1,6 @@
# Data Transfer
-## Data Transfer to/from ZIH Systems: Export Nodes
+## Data Transfer to/from ZIH Systems: Dataport Nodes
There are at least three tools for exchanging data between your local workstation and ZIH systems:
`scp`, `rsync`, and `sftp`. Please refer to the offline or online man pages of
@@ -8,9 +8,9 @@ There are at least three tools for exchanging data between your local workstatio
[rsync](https://man7.org/linux/man-pages/man1/rsync.1.html), and
[sftp](https://man7.org/linux/man-pages/man1/sftp.1.html) for detailed information.
-No matter what tool you prefer, it is crucial that the **export nodes** are used as preferred way to
+No matter what tool you prefer, it is crucial that the **dataport nodes** are used as preferred way to
copy data to/from ZIH systems. Please follow the link to the documentation on
-[export nodes](export_nodes.md) for further reference and examples.
+[dataport nodes](dataport_nodes.md) for further reference and examples.
## Data Transfer Inside ZIH Systems: Datamover
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md
index 3a400076e..e5963be3f 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/hardware_overview.md
@@ -113,23 +113,21 @@ only from their respective login nodes.
-## Login and Export Nodes
+## Login and Dataport Nodes
!!! Note " **On December 11 2023 Taurus will be decommissioned for good**."
Do not use Taurus for production anymore.
-- 4 Login-Nodes `tauruslogin[3-6].hrsk.tu-dresden.de`
- - Each login node is equipped with 2x Intel(R) Xeon(R) CPU E5-2680 v3 with 24 cores in total @
- 2.50 GHz, Multithreading disabled, 64 GB RAM, 128 GB SSD local disk
- - IPs: 141.30.73.\[102-105\]
-- 2 Data-Transfer-Nodes `taurusexport[3-4].hrsk.tu-dresden.de`
- - DNS Alias `taurusexport.hrsk.tu-dresden.de`
- - 2 Servers without interactive login, only available via file transfer protocols
+- Login-Nodes
+ - Individual for each cluster. See sections below.
+- 2 Data-Transfer-Nodes
+ - 2 servers without interactive login, only available via file transfer protocols
(`rsync`, `ftp`)
- - IPs: 141.30.73.\[82,83\]
+ - `dataport[3|4].hpc.tu-dresden.de`
+ - IPs: 141.30.73.\[4,5\]
- Further information on the usage is documented on the site
- [Export Nodes](../data_transfer/export_nodes.md)
+ [dataport Nodes](../data_transfer/dataport_nodes.md)
## Barnard
diff --git a/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md b/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md
index fe871b3d9..c9495588d 100644
--- a/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md
+++ b/doc.zih.tu-dresden.de/docs/quickstart/getting_started.md
@@ -195,31 +195,31 @@ The approach depends on the data volume: up to 100 MB or above.
transfer section.
### Transferring Data *To/From* ZIH HPC Systems
-
+<!-- [NT] currently not available
???+ example "`scp` for transferring data to ZIH HPC systems"
Copy the file `example.R` from your local machine to a workspace on the ZIH systems:
```console
- marie@local$ scp /home/marie/Documents/example.R marie@export.hpc.tu-dresden.de:/data/horse/ws/your_workspace/
+ marie@local$ scp /home/marie/Documents/example.R marie@dataport1.hpc.tu-dresden.de:/data/horse/ws/your_workspace/
Password:
example.R 100% 312 32.2KB/s 00:00``
```
- Note, the target path contains `export.hpc.tu-dresden.de`, which is one of the
- so called [export nodes](../data_transfer/export_nodes.md) that allows for data transfer from/to the outside.
+ Note, the target path contains `dataport1.hpc.tu-dresden.de`, which is one of the
+ so called [dataport nodes](../data_transfer/dataport_nodes.md) that allows for data transfer from/to the outside.
???+ example "`scp` to transfer data from ZIH HPC systems to local machine"
Copy the file `results.csv` from a workspace on the ZIH HPC systems to your local machine:
```console
- marie@local$ scp marie@export.hpc.tu-dresden.de:/data/horse/ws/marie-test-workspace/results.csv /home/marie/Documents/
+ marie@local$ scp marie@dataport1.hpc.tu-dresden.de:/data/horse/ws/marie-test-workspace/results.csv /home/marie/Documents/
```
Feel free to explore further [examples](http://bropages.org/scp) of the `scp` command
- and possibilities of the [export nodes](../data_transfer/export_nodes.md).
-
+ and possibilities of the [dataport nodes](../data_transfer/dataport_nodes.md).
+-->
!!! caution "Terabytes of data"
If you are planning to move terabytes or even more from an outside machine into ZIH systems,
diff --git a/doc.zih.tu-dresden.de/docs/software/gpu_programming.md b/doc.zih.tu-dresden.de/docs/software/gpu_programming.md
index 9c25859d5..84eb94c66 100644
--- a/doc.zih.tu-dresden.de/docs/software/gpu_programming.md
+++ b/doc.zih.tu-dresden.de/docs/software/gpu_programming.md
@@ -260,7 +260,7 @@ metrics and `--export-profile` to generate a report file, like this:
marie@compute$ nvprof --analysis-metrics --export-profile <output>.nvvp ./application [options]
```
-[Transfer the report file to your local system](../data_transfer/export_nodes.md) and analyze it in
+[Transfer the report file to your local system](../data_transfer/dataport_nodes.md) and analyze it in
the Visual Profiler (`nvvp`) locally. This will give the smoothest user experience. Alternatively,
you can use [X11-forwarding](../access/ssh_login.md). Refer to the documentation for details about
the individual
@@ -317,7 +317,7 @@ needs, this analysis may be sufficient to identify optimizations targets.
The graphical user interface version can be used for a thorough analysis of your previously
generated report file. For an optimal user experience, we recommend a local installation of NVIDIA
Nsight Systems. In this case, you can
-[transfer the report file to your local system](../data_transfer/export_nodes.md).
+[transfer the report file to your local system](../data_transfer/dataport_nodes.md).
Alternatively, you can use [X11-forwarding](../access/ssh_login.md). The graphical user interface is
usually available as `nsys-ui`.
@@ -361,7 +361,7 @@ manually.
This report file can be analyzed in the graphical user interface profiler. Again, we recommend you
generate a report file on a compute node and
-[transfer the report file to your local system](../data_transfer/export_nodes.md).
+[transfer the report file to your local system](../data_transfer/dataport_nodes.md).
Alternatively, you can use [X11-forwarding](../access/ssh_login.md). The graphical user interface is
usually available as `ncu-ui` or `nv-nsight-cu`.
--
GitLab