Skip to content
Snippets Groups Projects
Commit 8599902b authored by Noah Löwer's avatar Noah Löwer
Browse files

updated filesystem mountpoints, examples, put disclaimer

parent bde9d645
No related branches found
No related tags found
2 merge requests!938Automated merge from preview to main,!936Update to Five-Cluster-Operation
Hide whitespace changes
Inline Side-by-side
doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
+ 52
− 36
View file @ 8599902b
# Transfer Data Inside ZIH Systems with Datamover # Transfer Data Inside ZIH Systems with Datamover
With the **Datamover**, we provide a special data transfer machine for transferring data with best With the **Datamover**, we provide a special data transfer machine for transferring data between
transfer speed between the filesystems of ZIH systems. The Datamover machine is not accessible the ZIH filesystems with best transfer speed. The Datamover machine is not accessible
through SSH as it is dedicated to data transfers. To move or copy files from one filesystem to through SSH as it is dedicated to data transfers. To move or copy files from one filesystem to
another filesystem, you have to use the following commands: another, you have to use the following commands after logging in to any of the ZIH HPC systems:
- `dtcp`, `dtls`, `dtmv`, `dtrm`, `dtrsync`, `dttar`, and `dtwget` - `dtcp`, `dtls`, `dtmv`, `dtrm`, `dtrsync`, `dttar`, and `dtwget`
These commands submit a [batch job](../jobs_and_resources/slurm.md) to the data transfer machines These special commands submit a [batch job](../jobs_and_resources/slurm.md) to the data transfer machines
performing the selected command. Except the following options their syntax is the very same as the performing the selected command. Their syntax and behavior is the very same as the
well-known shell commands without the prefix *dt*. well-known shell commands without the prefix *`dt`*, except for the following options.
| Additional Option | Description | | Additional Option | Description |
|---------------------|-------------------------------------------------------------------------------| |---------------------|-------------------------------------------------------------------------------|
...@@ -31,33 +31,56 @@ To identify the mount points of the different filesystems on the data transfer m ...@@ -31,33 +31,56 @@ To identify the mount points of the different filesystems on the data transfer m
| ZIH system | Local directory | Directory on data transfer machine | | ZIH system | Local directory | Directory on data transfer machine |
|:-------------------|:---------------------|:-----------------------------------| |:-------------------|:---------------------|:-----------------------------------|
| Taurus | `/scratch/ws` | `/scratch/ws` | | **Barnard** | `/data/horse` | `/data/horse` |
| | `/ssd/ws` | `/ssd/ws` | | | `/data/walrus` | `/data/walrus` |
| | `/beegfs/global0/ws` | `/beegfs/global0/ws` | | *outdated: Taurus* | `/home` | `/data/old/home` |
| | `/warm_archive/ws` | `/warm_archive/ws` | | | `/scratch/ws` | `/data/old/lustre/scratch2/ws` |
| | `/home` | `/home` | | | `/ssd/ws` | `/data/old/lustre/ssd/ws` |
| | `/beegfs/global0/ws` | `/data/old/beegfs/global0/ws` |
| | `/warm_archive/ws` | `/data/old/warm_archive/ws` |
| | `/projects` | `/projects` | | | `/projects` | `/projects` |
| **Archive** | | `/archiv` | | **Archive** | | `/archiv` |
| **Group storage** | | `/grp/<group storage>` | | **Group storage** | | `/grp/<group storage>` |
## Usage of Datamover ## Usage of Datamover
!!! example "Copying data from `/beegfs/global0` to `/projects` filesystem." <!--TODO: remove when released in May 2024-->
??? "Data on outdated filesystems"
``` console !!! example "Copying data from `/beegfs/global0` to `/projects` filesystem."
marie@login$ dtcp -r /beegfs/global0/ws/marie-workdata/results /projects/p_number_crunch/.
``` ``` console
marie@login$ dtcp -r /data/old/beegfs/global0/ws/marie-workdata/results /projects/p_number_crunch/.
```
!!! example "Moving data from `/beegfs/global0` to `/warm_archive` filesystem."
``` console
marie@login$ dtmv /data/old/beegfs/global0/ws/marie-workdata/results /data/old/warm_archive/ws/marie-archive/.
```
!!! example "Archive data from `/beegfs/global0` to `/archiv` filesystem."
``` console
marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /data/old/beegfs/global0/ws/marie-workdata/results
```
!!! example "Copy data from `/data/horse` to `/projects` filesystem."
!!! example "Moving data from `/beegfs/global0` to `/warm_archive` filesystem." ``` console
marie@login$ dtcp -r /data/horse/ws/marie-workdata/results /projects/p_number_crunch/.
```
!!! example "Move data from `/data/horse` to `/data/walrus` filesystem."
``` console ``` console
marie@login$ dtmv /beegfs/global0/ws/marie-workdata/results /warm_archive/ws/marie-archive/. marie@login$ dtmv /data/horse/ws/marie-workdata/results /data/walrus/ws/marie-archive/.
``` ```
!!! example "Archive data from `/beegfs/global0` to `/archiv` filesystem." !!! example "Archive data from `/data/walrus` to `/archiv` filesystem."
``` console ``` console
marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /beegfs/global0/ws/marie-workdata/results marie@login$ dttar -czf /archiv/p_number_crunch/results.tgz /data/walrus/ws/marie-workdata/results
``` ```
!!! warning !!! warning
...@@ -66,34 +89,27 @@ To identify the mount points of the different filesystems on the data transfer m ...@@ -66,34 +89,27 @@ To identify the mount points of the different filesystems on the data transfer m
!!! note !!! note
The [warm archive](../data_lifecycle/warm_archive.md) and the `projects` filesystem are not The [warm archive](../data_lifecycle/warm_archive.md) and the `projects` filesystem are not
writable from within batch jobs. writable from within batch jobs.
However, you can store the data in the `warm_archive` using the Datamover. However, you can store the data in the `warm_archive` using the Datamover nodes via `dt*` commands.
## Transferring Files Between ZIH Systems and Group Drive ## Transferring Files Between ZIH Systems and Group Drive
1. Copy your public SSH key from ZIH system to `login1.zih.tu-dresden.de`. In order to let the datamover have access to your group drive, copy your public SSH key from ZIH
system to `login1.zih.tu-dresden.de`, first.
``` console ``` console
marie@login$ ssh-copy-id -i ~/.ssh/id_rsa.pub login1.zih.tu-dresden.de marie@login$ ssh-copy-id -i ~/.ssh/id_rsa.pub login1.zih.tu-dresden.de
# Export the name of your group drive for reuse of example commands
marie@login$ export GROUP_DRIVE_NAME=<my-drive-name>
``` ```
1. Now you can access your group drive with the Datamover commands. !!! example "Copy data from your group drive to `/data/horse` filesystem."
!!! example "Export the name of your group drive."
``` console
marie@login$ export GROUP_DRIVE_NAME=???
```
!!! note
Please replace `???` with the name of your group drive.
!!! example "Copying data from your group drive to `/beegfs/global0` filesystem."
``` console ``` console
marie@login$ dtrsync -av dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/inputfile /beegfs/global0/ws/marie-workdata/. marie@login$ dtrsync -av dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/inputfile /data/horse/ws/marie-workdata/.
``` ```
!!! example "Copying data from `/beegfs/global0` filesystem to your group drive." !!! example "Copy data from `/data/horse` filesystem to your group drive."
``` console ``` console
marie@login$ dtrsync -av /beegfs/global0/ws/marie-workdata/resultfile dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/. marie@login$ dtrsync -av /data/horse/ws/marie-workdata/resultfile dgw.zih.tu-dresden.de:/glw/${GROUP_DRIVE_NAME}/.
``` ```
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Nutzungsbedingungen | Datenschutzerklärung | Barrierefreiheitserklärung