Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
01afb36b
Commit
01afb36b
authored
1 year ago
by
Martin Schroschk
Browse files
Options
Downloads
Patches
Plain Diff
WIP: sketch target oriented data transfer instructions
parent
fd4d9aa3
No related branches found
No related tags found
2 merge requests
!920
Automated merge from preview to main
,
!918
Barnard cleanup
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard_test.md
+81
-5
81 additions, 5 deletions
...zih.tu-dresden.de/docs/jobs_and_resources/barnard_test.md
with
81 additions
and
5 deletions
doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard_test.md
+
81
−
5
View file @
01afb36b
...
@@ -22,14 +22,17 @@ Here, you can find few hints which might help you with the first steps.
...
@@ -22,14 +22,17 @@ Here, you can find few hints which might help you with the first steps.
## Login to Barnard
## Login to Barnard
All users and projects from Taurus now can work on Barnard.
!!! hint
All users and projects from Taurus now can work on Barnard.
They can use
`login[1-4].barnard.hpc.tu-dresden.de`
to access the system
They can use
`login[1-4].barnard.hpc.tu-dresden.de`
to access the system
from campus (or VPN).
[
Fingerprints
](
/access/key_fingerprints/#barnard
)
from campus (or VPN). In order to verify the SSH fingerprints of the login nodes, please refer to
the page
[
Fingerprints
](
/access/key_fingerprints/#barnard
)
.
All users have
**new empty HOME**
file systems, this means you have first to...
All users have
**new empty HOME**
file systems, this means you have first to
...
??? "... install your public SSH key on
the system
"
??? "... install your public SSH key on
Barnard
"
- Please create a new SSH keypair with ed25519 encryption, secured with
- Please create a new SSH keypair with ed25519 encryption, secured with
a passphrase. Please refer to this
a passphrase. Please refer to this
...
@@ -65,7 +68,6 @@ These `/data/horse` and `/data/walrus` can be accesed via workspaces. Please ref
...
@@ -65,7 +68,6 @@ These `/data/horse` and `/data/walrus` can be accesed via workspaces. Please ref
!!! Warning
!!! Warning
All old filesystems fill be shutdown by the end of 2023.
All old filesystems fill be shutdown by the end of 2023.
To work with your data from Taurus you might have to move/copy them to the new storages.
To work with your data from Taurus you might have to move/copy them to the new storages.
For this, we have four new
[
datamover nodes
](
/data_transfer/datamover
)
that have mounted all storages
For this, we have four new
[
datamover nodes
](
/data_transfer/datamover
)
that have mounted all storages
...
@@ -126,6 +128,80 @@ of the old and new system. (Do not use the datamovers from Taurus!)
...
@@ -126,6 +128,80 @@ of the old and new system. (Do not use the datamovers from Taurus!)
We recommand to **take some minutes for planing the transfer process**. Do not act with
We recommand to **take some minutes for planing the transfer process**. Do not act with
precipitation.
precipitation.
#### Migration of Working Data from Old Filesystems `ssd`, `beegfs`, and `scratch`
In the following, we will provide instructions for the data transfer of your old working data to the
new filesystems
`/data/horse`
and
`/data/walrus`
with comprehensive examples.
Please
**do not copy your entire work data**
form the old to the new filesystems, but consider this
opportunity for cleaning up you data. E.g., it might make sense to delete outdated scripts, old log
files, etc., and move other files, e.g., results, to the
`/data/walrus`
filesystem. Thus, please
take your time and make a plan which directories and files need to be transfered to
`/data/horse`
and which to
`/data/walrus`
. Therefor, the following instructions and examples are divided by the
target not the source filesystem.
!!! Hint
The following examples make use of the generic user name `marie` und the workspace
`marie-numbercrunch`. You need to replace this with you login and your workspace names!
Furthermore, your workspaced might be place on `/data/horse/lustre/scratch2/ws/0` or
/data/horse/lustre/scratch2/ws/1`. Use `ws_list` to verify the correct path!
!!! Hint "First Step: Allocate a workspace"
Before you invoke any data transer from the old working filesystems to the new ones, you need to
allocate a workspace. Both `/data/horse/` and `/data/walrus` can only be used with
[workspaces](../data_lifecycle/workspaces.md)
??? "Migration to
`/data/horse`
"
=== "Source: old
`/scratch`
"
If you transfer data from the old scratch to `/data/horse` it is sufficient to use `dtmv`
instead of `dtcp` since this data is already copied to the new filesystem and you just need to
move it to the right place.
```console
marie@barnard$ dtmove /data/horse/lustre/scratch2/0/marie-numbercrunch /data/horse/ws/marie-numbercrunch
```
=== "Source: old
`/ssd`
"
```console
marie@barnard$ dtcopy --recursive /data/old/lustre/ssd/ /data/horse/ws/marie-numbercrunch
```
=== "Source: old
`/beegfs`
"
The old `beegfs` filesystem is mounted at `/data/old/beegfs` the datamover nodes. You need to
invoke a `dtcopy` command to transfer data to a workspace in `/data/horse`, e.g.,
```console
marie@barnard$ dtcopy --recursive /data/old/beegfs/ /data/horse/ws/marie-numbercrunch
```
??? "Migration to
`/data/walrus`
"
=== "Source: old
`/scratch`
"
```console
marie@barnard$ dtcopy --recursive /data/horse/lustre/scratch2/0/marie-numbercrunch /data/walrus/ws/marie-numbercrunch
```
=== "Source: old
`/ssd`
"
```console
marie@barnard$ dtcopy --recursive /data/old/lustre/ssd/ /data/walrus/ws/marie-numbercrunch
```
=== "Source: old
`/beegfs`
"
The old `beegfs` filesystem is mounted at `/data/old/beegfs` the datamover nodes. You need to
invoke a `dtcopy` command to transfer data to a workspace in `/data/walrus`, e.g.,
```console
marie@barnard$ dtcopy --recursive /data/old/beegfs/ /data/walrus/ws/marie-numbercrunch
```
??? "Migration from
`/lustre/ssd`
or
`/beegfs`
"
??? "Migration from
`/lustre/ssd`
or
`/beegfs`
"
**You** are entirely responsible for the transfer of these data to the new location.
**You** are entirely responsible for the transfer of these data to the new location.
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment