From 8599902b9ed63c0f5ec19ed78350127f66a28e3f Mon Sep 17 00:00:00 2001
From: Noah Trumpik <noah.trumpik@tu-dresden.de>
Date: Tue, 28 Nov 2023 13:34:27 +0100
Subject: [PATCH] updated filesystem mountpoints, examples, put disclaimer
---
.../docs/data_transfer/datamover.md | 88 +++++++++++--------
1 file changed, 52 insertions(+), 36 deletions(-)
diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
index 80631a569..71e02c760 100644
--- a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
+++ b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
@@ -1,15 +1,15 @@
# Transfer Data Inside ZIH Systems with Datamover
-With the **Datamover**, we provide a special data transfer machine for transferring data with best
-transfer speed between the filesystems of ZIH systems. The Datamover machine is not accessible
+With the **Datamover**, we provide a special data transfer machine for transferring data between
+the ZIH filesystems with best transfer speed. The Datamover machine is not accessible
through SSH as it is dedicated to data transfers. To move or copy files from one filesystem to
-another filesystem, you have to use the following commands:
+another, you have to use the following commands after logging in to any of the ZIH HPC systems:
- `dtcp`, `dtls`, `dtmv`, `dtrm`, `dtrsync`, `dttar`, and `dtwget`
-These commands submit a [batch job](../jobs_and_resources/slurm.md) to the data transfer machines
-performing the selected command. Except the following options their syntax is the very same as the
-well-known shell commands without the prefix *dt*.
+These special commands submit a [batch job](../jobs_and_resources/slurm.md) to the data transfer machines
+performing the selected command. Their syntax and behavior is the very same as the
+well-known shell commands without the prefix *`dt`*, except for the following options.
| Additional Option | Description |
|---------------------|-------------------------------------------------------------------------------|
@@ -31,33 +31,56 @@ To identify the mount points of the different filesystems on the data transfer m
| ZIH system | Local directory | Directory on data transfer machine |
|:-------------------|:---------------------|:-----------------------------------|
-| Taurus | `/scratch/ws` | `/scratch/ws` |
-| | `/ssd/ws` | `/ssd/ws` |
-| | `/beegfs/global0/ws` | `/beegfs/global0/ws` |
-| | `/warm_archive/ws` | `/warm_archive/ws` |
-| | `/home` | `/home` |
+| **Barnard** | `/data/horse` | `/data/horse` |
+| | `/data/walrus` | `/data/walrus` |
+| *outdated: Taurus* | `/home` | `/data/old/home` |
+| | `/scratch/ws` | `/data/old/lustre/scratch2/ws` |
+| | `/ssd/ws` | `/data/old/lustre/ssd/ws` |
+| | `/beegfs/global0/ws` | `/data/old/beegfs/global0/ws` |
+| | `/warm_archive/ws` | `/data/old/warm_archive/ws` |
| | `/projects` | `/projects` |
-| **Archive** | | `/archiv` |
+| **Archive** | | `/archiv` |
| **Group storage** | | `/grp/<group storage>` |
## Usage of Datamover
-!!! example "Copying data from `/beegfs/global0` to `/projects` filesystem."
+<!--TODO: remove when released in May 2024-->
+??? "Data on outdated filesystems"
- ``` console
- marie@login$ dtcp -r /beegfs/global0/ws/marie-workdata/results /projects/p_number_crunch/.
- ```
+ !!! example "Copying data from `/beegfs/global0` to `/projects` filesystem."
+
+ ``` console
+ marie@login$ dtcp -r /data/old/beegfs/global0/ws/marie-workdata/results /projects/p_number_crunch/.
+ ```
+
+ !!! example "Moving data from `/beegfs/global0` to `/warm_archive` filesystem."
+
+ ``` console
+ marie@login$ dtmv /data/old/beegfs/global0/ws/marie-workdata/results /data/old/warm_archive/ws/marie-archive/.
+ ```
+
+ !!! example "Archive data from `/beegfs/global0` to `/archiv` filesystem."
+
+ ``` console
+ marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /data/old/beegfs/global0/ws/marie-workdata/results
+ ```
+
+!!! example "Copy data from `/data/horse` to `/projects` filesystem."
-!!! example "Moving data from `/beegfs/global0` to `/warm_archive` filesystem."
+ ``` console
+ marie@login$ dtcp -r /data/horse/ws/marie-workdata/results /projects/p_number_crunch/.
+ ```
+
+!!! example "Move data from `/data/horse` to `/data/walrus` filesystem."
``` console
- marie@login$ dtmv /beegfs/global0/ws/marie-workdata/results /warm_archive/ws/marie-archive/.
+ marie@login$ dtmv /data/horse/ws/marie-workdata/results /data/walrus/ws/marie-archive/.
```
-!!! example "Archive data from `/beegfs/global0` to `/archiv` filesystem."
+!!! example "Archive data from `/data/walrus` to `/archiv` filesystem."
``` console
- marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /beegfs/global0/ws/marie-workdata/results
+ marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /data/walrus/ws/marie-workdata/results
```
!!! warning
@@ -66,34 +89,27 @@ To identify the mount points of the different filesystems on the data transfer m
!!! note
The [warm archive](../data_lifecycle/warm_archive.md) and the `projects` filesystem are not
writable from within batch jobs.
- However, you can store the data in the `warm_archive` using the Datamover.
+ However, you can store the data in the `warm_archive` using the Datamover nodes via `dt*` commands.
## Transferring Files Between ZIH Systems and Group Drive
-1. Copy your public SSH key from ZIH system to `login1.zih.tu-dresden.de`.
+In order to let the datamover have access to your group drive, copy your public SSH key from ZIH
+system to `login1.zih.tu-dresden.de`, first.
``` console
marie@login$ ssh-copy-id -i ~/.ssh/id_rsa.pub login1.zih.tu-dresden.de
+ # Export the name of your group drive for reuse of example commands
+ marie@login$ export GROUP_DRIVE_NAME=<my-drive-name>
```
-1. Now you can access your group drive with the Datamover commands.
-!!! example "Export the name of your group drive."
-
- ``` console
- marie@login$ export GROUP_DRIVE_NAME=???
- ```
-
-!!! note
- Please replace `???` with the name of your group drive.
-
-!!! example "Copying data from your group drive to `/beegfs/global0` filesystem."
+!!! example "Copy data from your group drive to `/data/horse` filesystem."
``` console
- marie@login$ dtrsync -av dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/inputfile /beegfs/global0/ws/marie-workdata/.
+ marie@login$ dtrsync -av dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/inputfile /data/horse/ws/marie-workdata/.
```
-!!! example "Copying data from `/beegfs/global0` filesystem to your group drive."
+!!! example "Copy data from `/data/horse` filesystem to your group drive."
``` console
- marie@login$ dtrsync -av /beegfs/global0/ws/marie-workdata/resultfile dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/.
+ marie@login$ dtrsync -av /data/horse/ws/marie-workdata/resultfile dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/.
```
--
GitLab