Skip to content
Snippets Groups Projects
Commit 01afb36b authored by Martin Schroschk's avatar Martin Schroschk
Browse files

WIP: sketch target oriented data transfer instructions

parent fd4d9aa3
No related branches found
No related tags found
2 merge requests!920Automated merge from preview to main,!918Barnard cleanup
...@@ -22,14 +22,17 @@ Here, you can find few hints which might help you with the first steps. ...@@ -22,14 +22,17 @@ Here, you can find few hints which might help you with the first steps.
## Login to Barnard ## Login to Barnard
All users and projects from Taurus now can work on Barnard. !!! hint
All users and projects from Taurus now can work on Barnard.
They can use `login[1-4].barnard.hpc.tu-dresden.de` to access the system They can use `login[1-4].barnard.hpc.tu-dresden.de` to access the system
from campus (or VPN). [Fingerprints](/access/key_fingerprints/#barnard) from campus (or VPN). In order to verify the SSH fingerprints of the login nodes, please refer to
the page [Fingerprints](/access/key_fingerprints/#barnard).
All users have **new empty HOME** file systems, this means you have first to... All users have **new empty HOME** file systems, this means you have first to ...
??? "... install your public SSH key on the system" ??? "... install your public SSH key on Barnard"
- Please create a new SSH keypair with ed25519 encryption, secured with - Please create a new SSH keypair with ed25519 encryption, secured with
a passphrase. Please refer to this a passphrase. Please refer to this
...@@ -65,7 +68,6 @@ These `/data/horse` and `/data/walrus` can be accesed via workspaces. Please ref ...@@ -65,7 +68,6 @@ These `/data/horse` and `/data/walrus` can be accesed via workspaces. Please ref
!!! Warning !!! Warning
All old filesystems fill be shutdown by the end of 2023. All old filesystems fill be shutdown by the end of 2023.
To work with your data from Taurus you might have to move/copy them to the new storages. To work with your data from Taurus you might have to move/copy them to the new storages.
For this, we have four new [datamover nodes](/data_transfer/datamover) that have mounted all storages For this, we have four new [datamover nodes](/data_transfer/datamover) that have mounted all storages
...@@ -126,6 +128,80 @@ of the old and new system. (Do not use the datamovers from Taurus!) ...@@ -126,6 +128,80 @@ of the old and new system. (Do not use the datamovers from Taurus!)
We recommand to **take some minutes for planing the transfer process**. Do not act with We recommand to **take some minutes for planing the transfer process**. Do not act with
precipitation. precipitation.
#### Migration of Working Data from Old Filesystems `ssd`, `beegfs`, and `scratch`
In the following, we will provide instructions for the data transfer of your old working data to the
new filesystems `/data/horse` and `/data/walrus` with comprehensive examples.
Please **do not copy your entire work data** form the old to the new filesystems, but consider this
opportunity for cleaning up you data. E.g., it might make sense to delete outdated scripts, old log
files, etc., and move other files, e.g., results, to the `/data/walrus` filesystem. Thus, please
take your time and make a plan which directories and files need to be transfered to `/data/horse`
and which to `/data/walrus`. Therefor, the following instructions and examples are divided by the
target not the source filesystem.
!!! Hint
The following examples make use of the generic user name `marie` und the workspace
`marie-numbercrunch`. You need to replace this with you login and your workspace names!
Furthermore, your workspaced might be place on `/data/horse/lustre/scratch2/ws/0` or
/data/horse/lustre/scratch2/ws/1`. Use `ws_list` to verify the correct path!
!!! Hint "First Step: Allocate a workspace"
Before you invoke any data transer from the old working filesystems to the new ones, you need to
allocate a workspace. Both `/data/horse/` and `/data/walrus` can only be used with
[workspaces](../data_lifecycle/workspaces.md)
??? "Migration to `/data/horse`"
=== "Source: old `/scratch`"
If you transfer data from the old scratch to `/data/horse` it is sufficient to use `dtmv`
instead of `dtcp` since this data is already copied to the new filesystem and you just need to
move it to the right place.
```console
marie@barnard$ dtmove /data/horse/lustre/scratch2/0/marie-numbercrunch /data/horse/ws/marie-numbercrunch
```
=== "Source: old `/ssd`"
```console
marie@barnard$ dtcopy --recursive /data/old/lustre/ssd/ /data/horse/ws/marie-numbercrunch
```
=== "Source: old `/beegfs`"
The old `beegfs` filesystem is mounted at `/data/old/beegfs` the datamover nodes. You need to
invoke a `dtcopy` command to transfer data to a workspace in `/data/horse`, e.g.,
```console
marie@barnard$ dtcopy --recursive /data/old/beegfs/ /data/horse/ws/marie-numbercrunch
```
??? "Migration to `/data/walrus`"
=== "Source: old `/scratch`"
```console
marie@barnard$ dtcopy --recursive /data/horse/lustre/scratch2/0/marie-numbercrunch /data/walrus/ws/marie-numbercrunch
```
=== "Source: old `/ssd`"
```console
marie@barnard$ dtcopy --recursive /data/old/lustre/ssd/ /data/walrus/ws/marie-numbercrunch
```
=== "Source: old `/beegfs`"
The old `beegfs` filesystem is mounted at `/data/old/beegfs` the datamover nodes. You need to
invoke a `dtcopy` command to transfer data to a workspace in `/data/walrus`, e.g.,
```console
marie@barnard$ dtcopy --recursive /data/old/beegfs/ /data/walrus/ws/marie-numbercrunch
```
??? "Migration from `/lustre/ssd` or `/beegfs`" ??? "Migration from `/lustre/ssd` or `/beegfs`"
**You** are entirely responsible for the transfer of these data to the new location. **You** are entirely responsible for the transfer of these data to the new location.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment