Skip to content
Snippets Groups Projects
Commit a38d8832 authored by Martin Schroschk's avatar Martin Schroschk
Browse files

Review: prompts, typos, etc.

parent 4cb0f6f3
No related branches found
No related tags found
3 merge requests!392Merge preview into contrib guide for browser users,!366Merge preview into main,!359fix number of extension
Hide whitespace changes
Inline Side-by-side
doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md
+ 39
− 33
View file @ a38d8832
...@@ -4,13 +4,13 @@ Storage systems differ in terms of capacity, streaming bandwidth, IOPS rate, etc ...@@ -4,13 +4,13 @@ Storage systems differ in terms of capacity, streaming bandwidth, IOPS rate, etc
efficiency don't allow to have it all in one. That is why fast parallel filesystems at ZIH have efficiency don't allow to have it all in one. That is why fast parallel filesystems at ZIH have
restrictions with regards to **age of files** and [quota](quotas.md). The mechanism of workspaces restrictions with regards to **age of files** and [quota](quotas.md). The mechanism of workspaces
enables users to better manage their HPC data. enables users to better manage their HPC data.
<!--Workspaces are primarily login-related.-->
The concept of "workspaces" is common and used at a large number of HPC centers. The concept of workspaces is common and used at a large number of HPC centers.
!!! note !!! note
A workspace is a directory, with an associated expiration date, created on behalf of a user in a A **workspace** is a directory, with an associated expiration date, created on behalf of a user
certain storage system. in a certain filesystem.
Once the workspace has reached its expiration date, it gets moved to a hidden directory and enters a Once the workspace has reached its expiration date, it gets moved to a hidden directory and enters a
grace period. Once the grace period ends, the workspace is deleted permanently. The maximum lifetime grace period. Once the grace period ends, the workspace is deleted permanently. The maximum lifetime
...@@ -30,8 +30,8 @@ times. ...@@ -30,8 +30,8 @@ times.
To list all available filesystems for using workspaces use: To list all available filesystems for using workspaces use:
```bash ```console
zih$ ws_find -l marie@login$ ws_find -l
Available filesystems: Available filesystems:
scratch scratch
warm_archive warm_archive
...@@ -43,8 +43,8 @@ beegfs_global0 ...@@ -43,8 +43,8 @@ beegfs_global0
To list all workspaces you currently own, use: To list all workspaces you currently own, use:
```bash ```console
zih$ ws_list marie@login$ ws_list
id: test-workspace id: test-workspace
workspace directory : /scratch/ws/0/marie-test-workspace workspace directory : /scratch/ws/0/marie-test-workspace
remaining time : 89 days 23 hours remaining time : 89 days 23 hours
...@@ -59,8 +59,8 @@ id: test-workspace ...@@ -59,8 +59,8 @@ id: test-workspace
To create a workspace in one of the listed filesystems use `ws_allocate`. It is necessary to specify To create a workspace in one of the listed filesystems use `ws_allocate`. It is necessary to specify
a unique name and the duration of the workspace. a unique name and the duration of the workspace.
```bash ```console
ws_allocate: [options] workspace_name duration marie@login$ ws_allocate: [options] workspace_name duration
Options: Options:
-h [ --help] produce help message -h [ --help] produce help message
...@@ -74,13 +74,12 @@ Options: ...@@ -74,13 +74,12 @@ Options:
-u [ --username ] arg username -u [ --username ] arg username
-g [ --group ] group workspace -g [ --group ] group workspace
-c [ --comment ] arg comment -c [ --comment ] arg comment
``` ```
!!! example !!! example
```bash ```console
zih$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90 marie@login$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90
Info: creating workspace. Info: creating workspace.
/scratch/ws/marie-test-workspace /scratch/ws/marie-test-workspace
remaining extensions : 10 remaining extensions : 10
...@@ -95,22 +94,22 @@ days with an email reminder for 7 days before the expiration. ...@@ -95,22 +94,22 @@ days with an email reminder for 7 days before the expiration.
Setting the reminder to `7` means you will get a reminder email on every day starting `7` prior Setting the reminder to `7` means you will get a reminder email on every day starting `7` prior
to expiration date. to expiration date.
### Extention of a Workspace ### Extension of a Workspace
The lifetime of a workspace is finite. Different filesystems (storage systems) have different The lifetime of a workspace is finite. Different filesystems (storage systems) have different
maximum durations. A workspace can be extended multiple times, depending on the filesystem. maximum durations. A workspace can be extended multiple times, depending on the filesystem.
| Storage system (use with parameter -F ) | Duration, days | Extensions | Remarks | | Filesystem (use with parameter `-F`) | Duration, days | Extensions | Remarks |
|:------------------------------------------:|:----------:|:-------:|:---------------------------------------------------------------------------------------:| |:------------------------------------:|:----------:|:-------:|:-----------------------------------:|
| `ssd` | 30 | 2 | High-IOPS filesystem (`/lustre/ssd`) on SSDs. | | `ssd` | 30 | 2 | High-IOPS filesystem (`/lustre/ssd`) on SSDs. |
| `beegfs` | 30 | 2 | High-IOPS filesystem (`/lustre/ssd`) onNVMes. | | `beegfs` | 30 | 2 | High-IOPS filesystem (`/lustre/ssd`) on NVMes. |
| `scratch` | 100 | 10 | Scratch filesystem (`/scratch`) with high streaming bandwidth, based on spinning disks | | `scratch` | 100 | 10 | Scratch filesystem (`/scratch`) with high streaming bandwidth, based on spinning disks |
| `warm_archive` | 365 | 2 | Capacity filesystem based on spinning disks | | `warm_archive` | 365 | 2 | Capacity filesystem based on spinning disks |
To extend your workspace use the following command: To extent your workspace use the following command:
``` ```console
marie@login$ ws_extend -F scratch test-workspace 100 #extend the workspace for 100 days marie@login$ ws_extend -F scratch test-workspace 100
Info: extending workspace. Info: extending workspace.
/scratch/ws/marie-test-workspace /scratch/ws/marie-test-workspace
remaining extensions : 1 remaining extensions : 1
...@@ -122,29 +121,35 @@ remaining time in days: 100 ...@@ -122,29 +121,35 @@ remaining time in days: 100
With the `ws_extend` command, a new duration for the workspace is set. The new duration is not With the `ws_extend` command, a new duration for the workspace is set. The new duration is not
added! added!
This means when you extend a workspace that expires in 90 days with the `ws_extend -F scratch This means when you extend a workspace that expires in 90 days with the command
my-workspace 40`, it will now expire in 40 days **not** 130 days.
```console
marie@login$ ws_extend -F scratch my-workspace 40
```
it will now expire in 40 days **not** 130 days.
### Deletion of a Workspace ### Deletion of a Workspace
To delete a workspace use the `ws_release` command. It is mandatory to specify the name of the To delete a workspace use the `ws_release` command. It is mandatory to specify the name of the
workspace and the filesystem in which it is located: workspace and the filesystem in which it is located:
`ws_release -F <filesystem> <workspace name>` ```console
marie@login$ ws_release -F <filesystem> <workspace name>
```
### Restoring Expired Workspaces ### Restoring Expired Workspaces
At expiration time your workspace will be moved to a special, hidden directory. For a month (in At expiration time your workspace will be moved to a special, hidden directory. For a month (in
warm_archive: 2 months), you can still restore your data into an existing workspace. warm_archive: 2 months), you can still restore your data into an existing workspace.
!!!Warning !!! warning
When you release a workspace **by hand**, it will not receive a grace period and be When you release a workspace **by hand**, it will not receive a grace period and be
**permanently deleted** the **next day**. The advantage of this design is that you can create **permanently deleted** the **next day**. The advantage of this design is that you can create
and release workspaces inside jobs and not swamp the filesystem with data no one needs anymore and release workspaces inside jobs and not swamp the filesystem with data no one needs anymore
in the hidden directories (when workspaces are in the grace period). in the hidden directories (when workspaces are in the grace period).
Use: Use
```console ```console
marie@login$ ws_restore -l -F scratch marie@login$ ws_restore -l -F scratch
...@@ -174,8 +179,8 @@ workspaces within in the directory `DIR`. Calling this command will do the follo ...@@ -174,8 +179,8 @@ workspaces within in the directory `DIR`. Calling this command will do the follo
- The directory `DIR` will be created if necessary. - The directory `DIR` will be created if necessary.
- Links to all personal workspaces will be managed: - Links to all personal workspaces will be managed:
- Create links to all available workspaces if not already present. - Create links to all available workspaces if not already present.
- Remove links to released workspaces. - Remove links to released workspaces.
**Remark**: An automatic update of the workspace links can be invoked by putting the command **Remark**: An automatic update of the workspace links can be invoked by putting the command
`ws_register DIR` in your personal `shell` configuration file (e.g., `.bashrc`). `ws_register DIR` in your personal `shell` configuration file (e.g., `.bashrc`).
...@@ -198,8 +203,9 @@ A batch job needs a directory for temporary data. This can be deleted afterwards ...@@ -198,8 +203,9 @@ A batch job needs a directory for temporary data. This can be deleted afterwards
#SBATCH --ntasks=1 #SBATCH --ntasks=1
#SBATCH --cpus-per-task=24 #SBATCH --cpus-per-task=24
module load modenv/classic module purge
module load gaussian module load modenv/hiera
module load Gaussian
COMPUTE_DIR=gaussian_$SLURM_JOB_ID COMPUTE_DIR=gaussian_$SLURM_JOB_ID
export GAUSS_SCRDIR=$(ws_allocate -F ssd $COMPUTE_DIR 7) export GAUSS_SCRDIR=$(ws_allocate -F ssd $COMPUTE_DIR 7)
...@@ -259,7 +265,7 @@ remaining time in days: 365 ...@@ -259,7 +265,7 @@ remaining time in days: 365
The warm archive is not built for billions of files. There The warm archive is not built for billions of files. There
is a quota for 100.000 files per group. Please archive data. is a quota for 100.000 files per group. Please archive data.
To see your active quota use: To see your active quota use
```console ```console
marie@login$ qinfo quota /warm_archive/ws/ marie@login$ qinfo quota /warm_archive/ws/
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Nutzungsbedingungen | Datenschutzerklärung | Barrierefreiheitserklärung