Skip to content
Snippets Groups Projects
Commit 135b7d51 authored by Martin Schroschk's avatar Martin Schroschk
Browse files

Merge branch 'data_lifecycle_mf' into 'preview'

Data lifecycle mf

See merge request !225
parents 0f8e26fe 817b1707
No related branches found
No related tags found
3 merge requests!322Merge preview into main,!319Merge preview into main,!225Data lifecycle mf
...@@ -8,10 +8,10 @@ performance and permanence. ...@@ -8,10 +8,10 @@ performance and permanence.
| File system | Usable directory | Capacity | Availability | Backup | Remarks | | File system | Usable directory | Capacity | Availability | Backup | Remarks |
|:------------|:------------------|:---------|:-------------|:-------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------| |:------------|:------------------|:---------|:-------------|:-------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Lustre` | `/scratch/` | 4 PB | global | No | Only accessible via **todo link: workspaces - WorkSpaces**. Not made for billions of files! | | `Lustre` | `/scratch/` | 4 PB | global | No | Only accessible via [Workspaces](workspaces.md). Not made for billions of files! |
| `Lustre` | `/lustre/ssd` | 40 TB | global | No | Only accessible via **todo link: workspaces - WorkSpaces**. For small I/O operations | | `Lustre` | `/lustre/ssd` | 40 TB | global | No | Only accessible via [Workspaces](workspaces.md). For small I/O operations |
| `BeeGFS` | `/beegfs/global0` | 232 TB | global | No | Only accessible via **todo link: workspaces - WorkSpaces**. Fastest available file system, only for large parallel applications running with millions of small I/O operations | | `BeeGFS` | `/beegfs/global0` | 232 TB | global | No | Only accessible via [Workspaces](workspaces.md). Fastest available file system, only for large parallel applications running with millions of small I/O operations |
| `ext4` | `/tmp` | 95.0 GB | local | No | is cleaned up after the job automatically | | `ext4` | `/tmp` | 95 GB | local | No | is cleaned up after the job automatically |
## Warm Archive ## Warm Archive
......
...@@ -2,21 +2,24 @@ ...@@ -2,21 +2,24 @@
## Large Files in /scratch ## Large Files in /scratch
The data containers in Lustre are called object storage targets (OST). The capacity of one OST is The data containers in [Lustre](https://www.lustre.org) are called object storage targets (OST). The
about 21 TB. All files are striped over a certain number of these OSTs. For small and medium files, capacity of one OST is about 21 TB. All files are striped over a certain number of these OSTs. For
the default number is 2. As soon as a file grows above \~1 TB it makes sense to spread it over a small and medium files, the default number is 2. As soon as a file grows above ~1 TB it makes sense
higher number of OSTs, eg. 16. Once the file system is used \> 75%, the average space per OST is to spread it over a higher number of OSTs, e.g. 16. Once the file system is used >75%, the average
only 5 GB. So, it is essential to split your larger files so that the chunks can be saved! space per OST is only 5 GB. So, it is essential to split your larger files so that the chunks can be
saved!
Lets assume you have a dierctory where you tar your results, e.g. `/scratch/mark/tar`. Now, simply
Lets assume you have a directory where you tar your results, e.g. `/scratch/marie/tar`. Now, simply
set the stripe count to a higher number in this directory with: set the stripe count to a higher number in this directory with:
```Bash ```console
lfs setstripe -c 20 /scratch/ws/mark-stripe20/tar marie@login$ lfs setstripe -c 20 /scratch/ws/marie-stripe20/tar
``` ```
**Note:** This does not affect existing files. But all files that **will be created** in this !!! note
directory will be distributed over 20 OSTs.
This does not affect existing files. But all files that **will be created** in this
directory will be distributed over 20 OSTs.
## Useful Commands for Lustre ## Useful Commands for Lustre
...@@ -24,35 +27,35 @@ These commands work for `/scratch` and `/ssd`. ...@@ -24,35 +27,35 @@ These commands work for `/scratch` and `/ssd`.
### Listing Disk Usages per OST and MDT ### Listing Disk Usages per OST and MDT
```Bash ```console
lfs quota -h -u username /path/to/my/data marie@login$ lfs quota -h -u username /path/to/my/data
``` ```
It is possible to display the usage on each OST by adding the "-v"-parameter. It is possible to display the usage on each OST by adding the argument `-v`.
### Listing space usage per OST and MDT ### Listing Space Usage per OST and MDT
```Bash ```console
lfs df -h /path/to/my/data marie@login$ lfs df -h /path/to/my/data
``` ```
### Listing inode usage for an specific path ### Listing inode usage for an specific path
```Bash ```console
lfs df -i /path/to/my/data marie@login$ lfs df -i /path/to/my/data
``` ```
### Listing OSTs ### Listing OSTs
```Bash ```console
lfs osts /path/to/my/data marie@login$ lfs osts /path/to/my/data
``` ```
### View striping information ### View Striping Information
```Bash ```console
lfs getstripe myfile marie@login$ lfs getstripe myfile
lfs getstripe -d mydirectory marie@login$ lfs getstripe -d mydirectory
``` ```
The `-d`-parameter will also display striping for all files in the directory. The argument `-d` will also display striping for all files in the directory.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment