Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
1796aafe
Commit
1796aafe
authored
3 years ago
by
Ulf Markwardt
Browse files
Options
Downloads
Patches
Plain Diff
hpc_storage_concept2019.md deleted
parent
7df63414
No related branches found
No related tags found
4 merge requests
!322
Merge preview into main
,
!319
Merge preview into main
,
!235
Quota
,
!234
Aufgeteilt
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
doc.zih.tu-dresden.de/docs/data_lifecycle/hpc_storage_concept2019.md
+0
-71
0 additions, 71 deletions
...dresden.de/docs/data_lifecycle/hpc_storage_concept2019.md
doc.zih.tu-dresden.de/mkdocs.yml
+0
-1
0 additions, 1 deletion
doc.zih.tu-dresden.de/mkdocs.yml
with
0 additions
and
72 deletions
doc.zih.tu-dresden.de/docs/data_lifecycle/hpc_storage_concept2019.md
deleted
100644 → 0
+
0
−
71
View file @
7df63414
# HPC Storage Changes 2019
## Hardware changes require new approach**
\<
font face="Open Sans, sans-serif">At the moment we are preparing to
remove our old hardware from 2013. This comes with a shrinking of our
/scratch from 5 to 4 PB. At the same time we have now our "warm archive"
operational for HPC with a capacity of 5 PB for now.
\<
/font>
\<
font face="Open Sans, sans-serif">The tool concept of "workspaces" is
common in a large number of HPC centers. The idea is to allocate a
workspace directory in a certain storage system - connected with an
expiry date. After a grace period the data is deleted automatically. The
validity of a workspace can be extended twice.
\<
/font>
## \<font face="Open Sans, sans-serif"> **How to use workspaces?** \</font>
\<
font face="Open Sans, sans-serif">We have prepared a few examples at
<https://doc.zih.tu-dresden.de/hpc-wiki/bin/view/Compendium/WorkSpaces>
\<
/font>
-
\<
p>
\<
font face="Open Sans, sans-serif">For transient data, allocate
a workspace, run your job, remove data, and release the workspace
from with
\<
/font>
\<
font face="Open Sans, sans-serif">i
\<
/font>
\<
font
face="Open Sans, sans-serif">n your job file.
\<
/font>
\<
/p>
-
\<
p>
\<
font face="Open Sans, sans-serif">If you are working on a set
of data for weeks you might use workspaces in scratch and share them
with your groups by setting the file access attributes.
\<
/font>
\<
/p>
-
\<
p>
\<
font face="Open Sans, sans-serif">For
\<
/font>
\<
font
face="Open Sans, sans-serif">mid-term storage (max 3 years), use our
"warm archive" which is large but slow. It is available read-only on
the compute hosts and read-write an login and export nodes. To move
in your data, you might want to use the
[
datamover nodes
](
../data_transfer/data_mover.md
)
.
\<
/font>
\<
/p>
## \<font face="Open Sans, sans-serif">Moving Data from /scratch and /lustre/ssd to your workspaces\</font>
We are now mounting /lustre/ssd and /scratch read-only on the compute
nodes. As soon as the non-workspace /scratch directories are mounted
read-only on the login nodes as well, you won't be able to remove your
old data from there in the usual way. So you will have to use the
DataMover commands and ideally just move your data to your prepared
workspace:
```
Shell Session
dtmv /scratch/p_myproject/some_data /scratch/ws/myuser-mynewworkspace
#or:
dtmv /scratch/p_myproject/some_data /warm_archive/ws/myuser-mynewworkspace
```
Obsolete data can also be deleted like this:
```
Shell Session
dtrm -rf /scratch/p_myproject/some_old_data
```
**
%RED%At the end of the year we will delete all data on /scratch and
/lsuter/ssd outside the workspaces.%ENDCOLOR%
**
## Data life cycle management
\<
font face="Open Sans, sans-serif">Please be aware:
\<
/font>
\<
font
face="Open Sans, sans-serif">Data in workspaces will be deleted
automatically after the grace period.
\<
/font>
\<
font face="Open Sans,
sans-serif"> This is especially true for the warm archive. If you want
to keep your data for a longer time please use our options for
[
long-term storage
](
preservation_research_data.md
)
.
\<
/font>
\<
font face="Open Sans, sans-serif">To
\<
/font>
\<
font face="Open Sans,
sans-serif">help you with that, you can attach your email address for
notification or simply create an ICAL entry for your calendar
(tu-dresden.de mailboxes only).
\<
/font>
This diff is collapsed.
Click to expand it.
doc.zih.tu-dresden.de/mkdocs.yml
+
0
−
1
View file @
1796aafe
...
@@ -82,7 +82,6 @@ nav:
...
@@ -82,7 +82,6 @@ nav:
-
Intermediate Archive
:
data_lifecycle/intermediate_archive.md
-
Intermediate Archive
:
data_lifecycle/intermediate_archive.md
-
Quotas
:
data_lifecycle/quotas.md
-
Quotas
:
data_lifecycle/quotas.md
-
Workspaces
:
data_lifecycle/workspaces.md
-
Workspaces
:
data_lifecycle/workspaces.md
-
HPC Storage Concept 2019
:
data_lifecycle/hpc_storage_concept2019.md
-
Preservation of Research Data
:
data_lifecycle/preservation_research_data.md
-
Preservation of Research Data
:
data_lifecycle/preservation_research_data.md
-
Structuring Experiments
:
data_lifecycle/experiments.md
-
Structuring Experiments
:
data_lifecycle/experiments.md
-
Jobs and Resources
:
-
Jobs and Resources
:
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment