From 56f77652e566f6e0b6537e820190c4406d937469 Mon Sep 17 00:00:00 2001
From: Martin Schroschk <martin.schroschk@tu-dresden.de>
Date: Thu, 16 Sep 2021 12:15:43 +0200
Subject: [PATCH] Review data transfer pages and adjust links
---
.../data_lifecycle/intermediate_archive.md | 4 +-
.../docs/data_transfer/datamover.md | 2 +-
.../docs/data_transfer/export_nodes.md | 183 ++++++++----------
.../data_transfer/misc}/WinSCP_001_new.PNG | Bin
.../data_transfer/misc}/WinSCP_002_new.PNG | Bin
.../data_transfer/misc}/WinSCP_003_new.PNG | Bin
.../data_transfer/misc}/WinSCP_004_new.PNG | Bin
.../data_transfer/misc}/WinSCP_005_new.PNG | Bin
.../docs/data_transfer/overview.md | 33 +---
.../docs/software/deep_learning.md | 2 +-
.../docs/software/get_started_with_hpcda.md | 2 +-
11 files changed, 98 insertions(+), 128 deletions(-)
rename {Compendium_attachments/ExportNodes => doc.zih.tu-dresden.de/docs/data_transfer/misc}/WinSCP_001_new.PNG (100%)
rename {Compendium_attachments/ExportNodes => doc.zih.tu-dresden.de/docs/data_transfer/misc}/WinSCP_002_new.PNG (100%)
rename {Compendium_attachments/ExportNodes => doc.zih.tu-dresden.de/docs/data_transfer/misc}/WinSCP_003_new.PNG (100%)
rename {Compendium_attachments/ExportNodes => doc.zih.tu-dresden.de/docs/data_transfer/misc}/WinSCP_004_new.PNG (100%)
rename {Compendium_attachments/ExportNodes => doc.zih.tu-dresden.de/docs/data_transfer/misc}/WinSCP_005_new.PNG (100%)
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md
index e63f3f287..6aee19dd8 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/intermediate_archive.md
@@ -12,7 +12,7 @@ files.
Some more information:
- Maximum file size in the archive is 500 GB (split up your files, see
- [Datamover](../data_transfer/data_mover.md))
+ [Datamover](../data_transfer/datamover.md))
- Data will be stored in two copies on tape.
- The bandwidth to this data is very limited. Hence, this filesystem
must not be used directly as input or output for HPC jobs.
@@ -20,7 +20,7 @@ Some more information:
## Access the Intermediate Archive
For storing and restoring your data in/from the "Intermediate Archive" you can use the tool
-[Datamover](../data_transfer/data_mover.md). To use the DataMover you have to login to ZIH systems.
+[Datamover](../data_transfer/datamover.md). To use the DataMover you have to login to ZIH systems.
### Store Data
diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
index 4272d7ade..468c18238 100644
--- a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
+++ b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
@@ -7,7 +7,7 @@ another filesystem, you have to use the following commands:
- `dtcp`, `dtls`, `dtmv`, `dtrm`, `dtrsync`, `dttar`, and `dtwget`
-These commands submit a [batch job](jobs_and_resources/slurm.md) to the data transfer machines
+These commands submit a [batch job](../jobs_and_resources/slurm.md) to the data transfer machines
performing the selected command. Except the following options their syntax is the very same as the
wellknown shell commands without the prefix *dt*.
diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md b/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md
index ccd1e87e7..874d3dd27 100644
--- a/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md
+++ b/doc.zih.tu-dresden.de/docs/data_transfer/export_nodes.md
@@ -1,146 +1,135 @@
-# Move data to/from ZIH's File Systems
+# Export Nodes: Transfer data to/from ZIH's File Systems
-## Export Nodes
-
-To copy large data to/from the HPC machines, the Taurus export nodes should be used. While it is
+To copy large data to/from ZIH systems, the so-called **export nodes** should be used. While it is
possible to transfer small files directly via the login nodes, they are not intended to be used that
-way and there exists a CPU time limit on the login nodes, killing each process that takes up too
-much CPU time, which also affects file-copy processes if the copied files are very large. The export
-nodes have a better uplink (10GBit/s) and are generally the preferred way to transfer your data.
-Note that you cannot log in via ssh to the export nodes, but only use scp, rsync or sftp on them.
+way. Furthermore, longer transfers will hit the CPU time limit on the login nodes, i.e. the process
+get killed. The export nodes have a better uplink (10 GBit/s) allowing higher bandwith. Note that
+you cannot log in via SSH to the export nodes, but only use `scp`, `rsync` or `sftp` on them.
-They are reachable under the hostname: **taurusexport.hrsk.tu-dresden.de** (or
-taurusexport3.hrsk.tu-dresden.de, taurusexport4.hrsk.tu-dresden.de).
+The export nodes are reachable under the hostname `taurusexport.hrsk.tu-dresden.de` (or
+`taurusexport3.hrsk.tu-dresden.de` and `taurusexport4.hrsk.tu-dresden.de`).
-## Access from Linux Machine
+## Access From Linux
-There are three possibilities to exchange data between your local machine (lm) and the hpc machines
-(hm), which are explained in the following abstract in more detail.
+There are at least three tool to exchange data between your local workstation and ZIH systems. All
+are explained in the following abstract in more detail.
### SCP
-Type following commands in the terminal when you are in the directory of
-the local machine.
+The tool [`scp`](https://www.man7.org/linux/man-pages/man1/scp.1.html)
+(OpenSSH secure file copy) copies files between hosts on a network. To copy all files
+in a directory, the option `-r` has to be specified.
-#### Copy data from lm to hm
+??? example "Example: Copy a file from your workstation to ZIH systems"
-```Bash
-# Copy file
-scp <file> <zih-user>@<machine>:<target-location>
-# Copy directory
-scp -r <directory> <zih-user>@<machine>:<target-location>
-```
+ ```console
+ marie@local$ scp <file> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location>
-#### Copy data from hm to lm
+ # Add -r to copy whole directory
+ marie@local$ scp -r <directory> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location>
+ ```
-```Bash
-# Copy file
-scp <zih-user>@<machine>:<file> <target-location>
-# Copy directory
-scp -r <zih-user>@<machine>:<directory> <target-location>
-```
+??? example "Example: Copy a file from ZIH systems to your workstation"
-Example:
+ ```console
+ marie@login$ scp <zih-user>@taurusexport.hrsk.tu-dresden.de:<file> <target-location>
-```Bash
-scp helloworld.txt mustermann@taurusexport.hrsk.tu-dresden.de:~/.
-```
-
-Additional information: <http://www.computerhope.com/unix/scp.htm>
+ # Add -r to copy whole directory
+ marie@login$ scp -r <zih-user>@taurusexport.hrsk.tu-dresden.de:<directory> <target-location>
+ ```
### SFTP
-Is a virtual command line, which you could access with the following
-line:
+The tool [`sftp`](https://man7.org/linux/man-pages/man1/sftp.1.html) (OpenSSH secure file transfer)
+is a file transfer program, which performs all operations over an encrypted SSH transport. It may
+use compression to increase performance.
+
+`sftp` is basically a virtual command line, which you could access and exit as follows.
-```Bash
+```console
# Enter virtual command line
-sftp <zih-user>@<machine>
+marie@local$ sftp <zih-user>@taurusexport.hrsk.tu-dresden.de
# Exit virtual command line
-sftp> exit
+sftp> exit
# or
sftp> <Ctrl+D>
```
-After that you have access to the filesystem on the hpc machine and you
-can use the same commands as on your local machine, e.g. ls, cd, pwd and
-many more. If you would access to your local machine from this virtual
-command line, then you have to put the letter l (local machine) before
-the command, e.g. lls, lcd or lpwd.
+After that you have access to the filesystem on ZIH systems, you can use the same commands as on
+your local workstation, e.g., `ls`, `cd`, `pwd` etc. If you would access to your local workstation
+from this virtual command line, then you have to prefix the command with the letter `l`
+(`l`ocal),e.g., `lls`, `lcd` or `lpwd`.
-#### Copy data from lm to hm
+??? example "Example: Copy a file from your workstation to ZIH systems"
-```Bash
-# Copy file
-sftp> put <file>
-# Copy directory
-sftp> put -r <directory>
-```
+ ```console
+ marie@local$ sftp <zih-user>@taurusexport.hrsk.tu-dresden.de
+ # Copy file
+ sftp> put <file>
+ # Copy directory
+ sftp> put -r <directory>
+ ```
-#### Copy data from hm to lm
-
-```Bash
-# Copy file
-sftp> get <file>
-# Copy directory
-sftp> get -r <directory>
-```
+??? example "Example: Copy a file from ZIH systems to your local workstation"
-Example:
-
-```Bash
-sftp> get helloworld.txt
-```
-
-Additional information: http://www.computerhope.com/unix/sftp.htm
+ ```console
+ marie@local$ sftp <zih-user>@taurusexport.hrsk.tu-dresden.de
+ # Copy file
+ sftp> get <file>
+ # Copy directory
+ sftp> get -r <directory>
+ ```
### RSYNC
+[`Rsync`](https://man7.org/linux/man-pages/man1/rsync.1.html), is a fast and extraordinarily
+versatile file copying tool. It can copy locally, to/from another host over any remote shell, or
+to/from a remote `rsync` daemon. It is famous for its delta-transfer algorithm, which reduces the
+amount of data sent over the network by sending only the differences between the source files and
+the existing files in the destination.
+
Type following commands in the terminal when you are in the directory of
the local machine.
-#### Copy data from lm to hm
-
-```Bash
-# Copy file
-rsync <file> <zih-user>@<machine>:<target-location>
-# Copy directory
-rsync -r <directory> <zih-user>@<machine>:<target-location>
-```
-
-#### Copy data from hm to lm
+??? example "Example: Copy a file from your workstation to ZIH systems"
-```Bash
-# Copy file
-rsync <zih-user>@<machine>:<file> <target-location>
-# Copy directory
-rsync -r <zih-user>@<machine>:<directory> <target-location>
-```
+ ```console
+ # Copy file
+ marie@local$ rsync <file> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location>
+ # Copy directory
+ marie@local$ rsync -r <directory> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location>
+ ```
-Example:
-
-```Bash
-rsync helloworld.txt mustermann@taurusexport.hrsk.tu-dresden.de:~/.
-```
+??? example "Example: Copy a file from ZIH systems to your local workstation"
-Additional information: http://www.computerhope.com/unix/rsync.htm
+ ```console
+ # Copy file
+ marie@local$ rsync <zih-user>@taurusexport.hrsk.tu-dresden.de:<file> <target-location>
+ # Copy directory
+ marie@local$ rsync -r <zih-user>@taurusexport.hrsk.tu-dresden.de:<directory> <target-location>
+ ```
-## Access from Windows machine
+## Access From Windows
First you have to install [WinSCP](http://winscp.net/eng/download.php).
Then you have to execute the WinSCP application and configure some
option as described below.
-<span class="twiki-macro IMAGE" size="600">WinSCP_001_new.PNG</span>
+
+{: align="center"}
-<span class="twiki-macro IMAGE" size="600">WinSCP_002_new.PNG</span>
+
+{: align="center"}
-<span class="twiki-macro IMAGE" size="600">WinSCP_003_new.PNG</span>
+
+{: align="center"}
-<span class="twiki-macro IMAGE" size="600">WinSCP_004_new.PNG</span>
+
+{: align="center"}
-After your connection succeeded, you can copy files from your local
-machine to the hpc machine and the other way around.
+After your connection succeeded, you can copy files from your local workstation to ZIH systems and
+the other way around.
-<span class="twiki-macro IMAGE" size="600">WinSCP_005_new.PNG</span>
+
+{: align="center"}
diff --git a/Compendium_attachments/ExportNodes/WinSCP_001_new.PNG b/doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_001_new.PNG
similarity index 100%
rename from Compendium_attachments/ExportNodes/WinSCP_001_new.PNG
rename to doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_001_new.PNG
diff --git a/Compendium_attachments/ExportNodes/WinSCP_002_new.PNG b/doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_002_new.PNG
similarity index 100%
rename from Compendium_attachments/ExportNodes/WinSCP_002_new.PNG
rename to doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_002_new.PNG
diff --git a/Compendium_attachments/ExportNodes/WinSCP_003_new.PNG b/doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_003_new.PNG
similarity index 100%
rename from Compendium_attachments/ExportNodes/WinSCP_003_new.PNG
rename to doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_003_new.PNG
diff --git a/Compendium_attachments/ExportNodes/WinSCP_004_new.PNG b/doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_004_new.PNG
similarity index 100%
rename from Compendium_attachments/ExportNodes/WinSCP_004_new.PNG
rename to doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_004_new.PNG
diff --git a/Compendium_attachments/ExportNodes/WinSCP_005_new.PNG b/doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_005_new.PNG
similarity index 100%
rename from Compendium_attachments/ExportNodes/WinSCP_005_new.PNG
rename to doc.zih.tu-dresden.de/docs/data_transfer/misc/WinSCP_005_new.PNG
diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/overview.md b/doc.zih.tu-dresden.de/docs/data_transfer/overview.md
index 0270effeb..1cc64dfd9 100644
--- a/doc.zih.tu-dresden.de/docs/data_transfer/overview.md
+++ b/doc.zih.tu-dresden.de/docs/data_transfer/overview.md
@@ -2,40 +2,21 @@
## Moving Data to/from ZIH Systems
-There are at least three tools to exchange data between your local machine (lm) and ZIH systems:
+There are at least three tools to exchange data between your local workstation and ZIH systems:
`scp`, `rsync`, and `sftp`. Please refer to the offline or online man pages of
[scp](https://www.man7.org/linux/man-pages/man1/scp.1.html),
[rsync](https://man7.org/linux/man-pages/man1/rsync.1.html), and
[sftp](https://man7.org/linux/man-pages/man1/sftp.1.html) for detailed information.
-!!! hint
-
- No matter what tool you prefer, it is crucial that the **export nodes** are used prefered way to
- copy data to/from ZIH systems.
-
-!!! example "Example using `scp` to copy a file from your workstation to ZIH systems"
-
- ```console
- marie@local$ scp <file> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location>
-
- # Add -r to copy whole directory
- marie@local$ scp -r <directory> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location>
- ```
-
-!!! example "Example using `scp` to copy a file from ZIH systems to your workstation"
-
-```console
-marie@login$ scp <zih-user>@taurusexport.hrsk.tu-dresden.de:<file> <target-location>
-
-# Add -r to copy whole directory
-marie@login$ scp -r <zih-user>@taurusexport.hrsk.tu-dresden.de:<directory> <target-location>
-```
+No matter what tool you prefer, it is crucial that the **export nodes** are used prefered way to
+copy data to/from ZIH systems. Please follow the linkt to documentation on [export
+nodes](export_nodes.md) for further reference and examples.
## Moving Data Inside ZIH Systems: Datamover
The recommended way for data transfer inside ZIH Systems is the **datamover**. It is a special
data transfer machine that provides the best transfer speed. To load, move, copy etc. files from one
filesystem to another filesystem, you have to use commands prefixed with `dt`: `dtcp`, `dtwget`,
-`dtmv`, `dtrm`, `dtrsync`, `dttar`, `dtls`. These commands submit a job to the data transfer machines that
-execute the selected command.
-Plese refer to the detailed documentation regarding the [datamover](datamover.md).
+`dtmv`, `dtrm`, `dtrsync`, `dttar`, `dtls`. These commands submit a job to the data transfer
+machines that execute the selected command. Plese refer to the detailed documentation regarding the
+[datamover](datamover.md).
diff --git a/doc.zih.tu-dresden.de/docs/software/deep_learning.md b/doc.zih.tu-dresden.de/docs/software/deep_learning.md
index da8c9c461..00cbdad4b 100644
--- a/doc.zih.tu-dresden.de/docs/software/deep_learning.md
+++ b/doc.zih.tu-dresden.de/docs/software/deep_learning.md
@@ -117,7 +117,7 @@ public datasets without downloading it (for example
If you still need to download some datasets, first of all, be careful with the size of the datasets
which you would like to download (some of them have a size of few Terabytes). Don't download what
you really not need to use! Use login nodes only for downloading small files (hundreds of the
-megabytes). For downloading huge files use [DataMover](../data_transfer/data_mover.md).
+megabytes). For downloading huge files use [DataMover](../data_transfer/datamover.md).
For example, you can use command `dtwget` (it is an analogue of the general wget
command). This command submits a job to the data transfer machines. If you need to download or
allocate massive files (more than one terabyte) please contact the support before.
diff --git a/doc.zih.tu-dresden.de/docs/software/get_started_with_hpcda.md b/doc.zih.tu-dresden.de/docs/software/get_started_with_hpcda.md
index 850493f6d..8bf5d1278 100644
--- a/doc.zih.tu-dresden.de/docs/software/get_started_with_hpcda.md
+++ b/doc.zih.tu-dresden.de/docs/software/get_started_with_hpcda.md
@@ -122,7 +122,7 @@ scp -r <zih-user>@taurusexport.hrsk.tu-dresden.de:<directory> <target-location>
#### Moving data inside the HPC machines. Datamover
-The best way to transfer data inside the Taurus is the [data mover](../data_transfer/data_mover.md).
+The best way to transfer data inside the Taurus is the [data mover](../data_transfer/datamover.md).
It is the special data transfer machine providing the global file systems of each ZIH HPC system.
Datamover provides the best data speed. To load, move, copy etc. files from one file system to
another file system, you have to use commands with **dt** prefix, such as:
--
GitLab