diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md index 80631a56987f8b5f67fca331d65d558740ec80e2..71e02c760861f34439769606c2c014a33d6c9f13 100644 --- a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md +++ b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md @@ -1,15 +1,15 @@ # Transfer Data Inside ZIH Systems with Datamover -With the **Datamover**, we provide a special data transfer machine for transferring data with best -transfer speed between the filesystems of ZIH systems. The Datamover machine is not accessible +With the **Datamover**, we provide a special data transfer machine for transferring data between +the ZIH filesystems with best transfer speed. The Datamover machine is not accessible through SSH as it is dedicated to data transfers. To move or copy files from one filesystem to -another filesystem, you have to use the following commands: +another, you have to use the following commands after logging in to any of the ZIH HPC systems: - `dtcp`, `dtls`, `dtmv`, `dtrm`, `dtrsync`, `dttar`, and `dtwget` -These commands submit a [batch job](../jobs_and_resources/slurm.md) to the data transfer machines -performing the selected command. Except the following options their syntax is the very same as the -well-known shell commands without the prefix *dt*. +These special commands submit a [batch job](../jobs_and_resources/slurm.md) to the data transfer machines +performing the selected command. Their syntax and behavior is the very same as the +well-known shell commands without the prefix *`dt`*, except for the following options. | Additional Option | Description | |---------------------|-------------------------------------------------------------------------------| @@ -31,33 +31,56 @@ To identify the mount points of the different filesystems on the data transfer m | ZIH system | Local directory | Directory on data transfer machine | |:-------------------|:---------------------|:-----------------------------------| -| Taurus | `/scratch/ws` | `/scratch/ws` | -| | `/ssd/ws` | `/ssd/ws` | -| | `/beegfs/global0/ws` | `/beegfs/global0/ws` | -| | `/warm_archive/ws` | `/warm_archive/ws` | -| | `/home` | `/home` | +| **Barnard** | `/data/horse` | `/data/horse` | +| | `/data/walrus` | `/data/walrus` | +| *outdated: Taurus* | `/home` | `/data/old/home` | +| | `/scratch/ws` | `/data/old/lustre/scratch2/ws` | +| | `/ssd/ws` | `/data/old/lustre/ssd/ws` | +| | `/beegfs/global0/ws` | `/data/old/beegfs/global0/ws` | +| | `/warm_archive/ws` | `/data/old/warm_archive/ws` | | | `/projects` | `/projects` | -| **Archive** | | `/archiv` | +| **Archive** | | `/archiv` | | **Group storage** | | `/grp/<group storage>` | ## Usage of Datamover -!!! example "Copying data from `/beegfs/global0` to `/projects` filesystem." +<!--TODO: remove when released in May 2024--> +??? "Data on outdated filesystems" - ``` console - marie@login$ dtcp -r /beegfs/global0/ws/marie-workdata/results /projects/p_number_crunch/. - ``` + !!! example "Copying data from `/beegfs/global0` to `/projects` filesystem." + + ``` console + marie@login$ dtcp -r /data/old/beegfs/global0/ws/marie-workdata/results /projects/p_number_crunch/. + ``` + + !!! example "Moving data from `/beegfs/global0` to `/warm_archive` filesystem." + + ``` console + marie@login$ dtmv /data/old/beegfs/global0/ws/marie-workdata/results /data/old/warm_archive/ws/marie-archive/. + ``` + + !!! example "Archive data from `/beegfs/global0` to `/archiv` filesystem." + + ``` console + marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /data/old/beegfs/global0/ws/marie-workdata/results + ``` + +!!! example "Copy data from `/data/horse` to `/projects` filesystem." -!!! example "Moving data from `/beegfs/global0` to `/warm_archive` filesystem." + ``` console + marie@login$ dtcp -r /data/horse/ws/marie-workdata/results /projects/p_number_crunch/. + ``` + +!!! example "Move data from `/data/horse` to `/data/walrus` filesystem." ``` console - marie@login$ dtmv /beegfs/global0/ws/marie-workdata/results /warm_archive/ws/marie-archive/. + marie@login$ dtmv /data/horse/ws/marie-workdata/results /data/walrus/ws/marie-archive/. ``` -!!! example "Archive data from `/beegfs/global0` to `/archiv` filesystem." +!!! example "Archive data from `/data/walrus` to `/archiv` filesystem." ``` console - marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /beegfs/global0/ws/marie-workdata/results + marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /data/walrus/ws/marie-workdata/results ``` !!! warning @@ -66,34 +89,27 @@ To identify the mount points of the different filesystems on the data transfer m !!! note The [warm archive](../data_lifecycle/warm_archive.md) and the `projects` filesystem are not writable from within batch jobs. - However, you can store the data in the `warm_archive` using the Datamover. + However, you can store the data in the `warm_archive` using the Datamover nodes via `dt*` commands. ## Transferring Files Between ZIH Systems and Group Drive -1. Copy your public SSH key from ZIH system to `login1.zih.tu-dresden.de`. +In order to let the datamover have access to your group drive, copy your public SSH key from ZIH +system to `login1.zih.tu-dresden.de`, first. ``` console marie@login$ ssh-copy-id -i ~/.ssh/id_rsa.pub login1.zih.tu-dresden.de + # Export the name of your group drive for reuse of example commands + marie@login$ export GROUP_DRIVE_NAME=<my-drive-name> ``` -1. Now you can access your group drive with the Datamover commands. -!!! example "Export the name of your group drive." - - ``` console - marie@login$ export GROUP_DRIVE_NAME=??? - ``` - -!!! note - Please replace `???` with the name of your group drive. - -!!! example "Copying data from your group drive to `/beegfs/global0` filesystem." +!!! example "Copy data from your group drive to `/data/horse` filesystem." ``` console - marie@login$ dtrsync -av dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/inputfile /beegfs/global0/ws/marie-workdata/. + marie@login$ dtrsync -av dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/inputfile /data/horse/ws/marie-workdata/. ``` -!!! example "Copying data from `/beegfs/global0` filesystem to your group drive." +!!! example "Copy data from `/data/horse` filesystem to your group drive." ``` console - marie@login$ dtrsync -av /beegfs/global0/ws/marie-workdata/resultfile dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/. + marie@login$ dtrsync -av /data/horse/ws/marie-workdata/resultfile dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/. ```