Skip to content
Snippets Groups Projects
Commit 62e5acab authored by Jan Frenzel's avatar Jan Frenzel
Browse files

Merge branch 'datamover' into 'preview'

Resolve #301 by naming it Datamover

Closes #301

See merge request !536
parents 70f051ee 39746197
No related branches found
No related tags found
2 merge requests!542Automated merge from preview to main,!536Resolve #301 by naming it Datamover
...@@ -20,7 +20,7 @@ Some more information: ...@@ -20,7 +20,7 @@ Some more information:
## Access the Intermediate Archive ## Access the Intermediate Archive
For storing and restoring your data in/from the "Intermediate Archive" you can use the tool For storing and restoring your data in/from the "Intermediate Archive" you can use the tool
[Datamover](../data_transfer/datamover.md). To use the DataMover you have to login to ZIH systems. [Datamover](../data_transfer/datamover.md). To use the Datamover you have to login to ZIH systems.
### Store Data ### Store Data
......
# Datamover - Data Transfer Inside ZIH Systems # Datamover - Data Transfer Inside ZIH Systems
With the **datamover**, we provide a special data transfer machine for transferring data with best With the **Datamover**, we provide a special data transfer machine for transferring data with best
transfer speed between the filesystems of ZIH systems. The datamover machine is not accessible transfer speed between the filesystems of ZIH systems. The Datamover machine is not accessible
through SSH as it is dedicated to data transfers. To move or copy files from one filesystem to through SSH as it is dedicated to data transfers. To move or copy files from one filesystem to
another filesystem, you have to use the following commands: another filesystem, you have to use the following commands:
...@@ -66,7 +66,7 @@ To identify the mount points of the different filesystems on the data transfer m ...@@ -66,7 +66,7 @@ To identify the mount points of the different filesystems on the data transfer m
!!! note !!! note
The [warm archive](../data_lifecycle/warm_archive.md) and the `projects` filesystem are not The [warm archive](../data_lifecycle/warm_archive.md) and the `projects` filesystem are not
writable from within batch jobs. writable from within batch jobs.
However, you can store the data in the `warm_archive` using the datamover. However, you can store the data in the `warm_archive` using the Datamover.
## Transferring Files Between ZIH Systems and Group Drive ## Transferring Files Between ZIH Systems and Group Drive
......
...@@ -14,9 +14,9 @@ copy data to/from ZIH systems. Please follow the link to the documentation on ...@@ -14,9 +14,9 @@ copy data to/from ZIH systems. Please follow the link to the documentation on
## Data Transfer Inside ZIH Systems: Datamover ## Data Transfer Inside ZIH Systems: Datamover
The recommended way for data transfer inside ZIH Systems is the **datamover**. It is a special The recommended way for data transfer inside ZIH Systems is the **Datamover**. It is a special
data transfer machine that provides the best transfer speed. To load, move, copy etc. files from one data transfer machine that provides the best transfer speed. To load, move, copy etc. files from one
filesystem to another filesystem, you have to use commands prefixed with `dt`: `dtcp`, `dtwget`, filesystem to another filesystem, you have to use commands prefixed with `dt`: `dtcp`, `dtwget`,
`dtmv`, `dtrm`, `dtrsync`, `dttar`, `dtls`. These commands submit a job to the data transfer `dtmv`, `dtrm`, `dtrsync`, `dttar`, `dtls`. These commands submit a job to the data transfer
machines that execute the selected command. Please refer to the detailed documentation regarding the machines that execute the selected command. Please refer to the detailed documentation regarding the
[datamover](datamover.md). [Datamover](datamover.md).
...@@ -29,7 +29,7 @@ can be installed individually by each user. If possible, the use of ...@@ -29,7 +29,7 @@ can be installed individually by each user. If possible, the use of
recommended (e.g. for Python). Likewise, software can be used within [containers](containers.md). recommended (e.g. for Python). Likewise, software can be used within [containers](containers.md).
For the transfer of larger amounts of data into and within the system, the For the transfer of larger amounts of data into and within the system, the
[export nodes and datamover](../data_transfer/overview.md) should be used. [export nodes and Datamover](../data_transfer/overview.md) should be used.
Data is stored in the [workspaces](../data_lifecycle/workspaces.md). Data is stored in the [workspaces](../data_lifecycle/workspaces.md).
Software modules or virtual environments can also be installed in workspaces to enable Software modules or virtual environments can also be installed in workspaces to enable
collaborative work even within larger groups. collaborative work even within larger groups.
......
...@@ -155,7 +155,7 @@ The following HPC related software is installed on all nodes: ...@@ -155,7 +155,7 @@ The following HPC related software is installed on all nodes:
There are many different datasets designed for research purposes. If you would like to download some There are many different datasets designed for research purposes. If you would like to download some
of them, keep in mind that many machine learning libraries have direct access to public datasets of them, keep in mind that many machine learning libraries have direct access to public datasets
without downloading it, e.g. [TensorFlow Datasets](https://www.tensorflow.org/datasets). If you without downloading it, e.g. [TensorFlow Datasets](https://www.tensorflow.org/datasets). If you
still need to download some datasets use [datamover](../data_transfer/datamover.md) machine. still need to download some datasets use [Datamover](../data_transfer/datamover.md) machine.
### The ImageNet Dataset ### The ImageNet Dataset
......
...@@ -51,7 +51,7 @@ Dask ...@@ -51,7 +51,7 @@ Dask
dataframes dataframes
DataFrames DataFrames
Dataheap Dataheap
datamover Datamover
DataParallel DataParallel
dataset dataset
Dataset Dataset
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment