@@ -98,98 +82,106 @@ In case a project is above it's limits please...
| `BeeGFS` | `/beegfs/global0` | 232 TB | global | No | Only accessible via **todo link: workspaces - WorkSpaces**. Fastest available file system, only for large parallel applications running with millions of small I/O operations |
| `ext4` | `/tmp` | 95.0 GB | local | No | is cleaned up after the job automatically |
### Large files in /scratch
### Large Files in /scratch
The data containers in Lustre are called object storage targets (OST).
The capacity of one OST is about 21 TB. All files are striped over a
certain number of these OSTs. For small and medium files, the default
number is 2. As soon as a file grows above \~1 TB it makes sense to
spread it over a higher number of OSTs, eg. 16. Once the file system is
used \> 75%, the average space per OST is only 5 GB. So, it is essential
to split your larger files so that the chunks can be saved!
The data containers in Lustre are called object storage targets (OST). The capacity of one OST is
about 21 TB. All files are striped over a certain number of these OSTs. For small and medium files,
the default number is 2. As soon as a file grows above \~1 TB it makes sense to spread it over a
higher number of OSTs, eg. 16. Once the file system is used \> 75%, the average space per OST is
only 5 GB. So, it is essential to split your larger files so that the chunks can be saved!
Lets assume you have a dierctory where you tar your results, eg.
`/scratch/mark/tar` . Now, simply set the stripe count to a higher
number in this directory with:
Lets assume you have a dierctory where you tar your results, e.g. `/scratch/mark/tar`. Now, simply
set the stripe count to a higher number in this directory with:
lfs setstripe -c 20 /scratch/ws/mark-stripe20/tar
```Bash
lfs setstripe -c 20 /scratch/ws/mark-stripe20/tar
```
%RED%Note:<spanclass="twiki-macro ENDCOLOR"></span> This does not
affect existing files. But all files that **will be created** in this
**Note:** This does not affect existing files. But all files that **will be created** in this
directory will be distributed over 20 OSTs.
## Warm archive
## Warm Archive
TODO
## Recommendations for file system usage
## Recommendations for File System Usage
To work as efficient as possible, consider the following points
-Save source code etc. in `/home` or /projects/...
-Store checkpoints and other temporary data in `/scratch/ws/...`
-Compilation in `/dev/shm` or `/tmp`
- Save source code etc. in `/home` or /projects/...
- Store checkpoints and other temporary data in `/scratch/ws/...`
- Compilation in `/dev/shm` or `/tmp`
Getting high I/O-bandwitdh
- Use many clients
- Use many processes (writing in the same file at the same time is
possible)
- Use large I/O transfer blocks
- Use many clients
- Use many processes (writing in the same file at the same time is possible)
- Use large I/O transfer blocks
## Cheat Sheet for debugging file system issues
## Cheat Sheet for Debugging File System Issues
Every Taurus-User should normaly be able to perform the following
commands to get some intel about theire data.
Every Taurus-User should normaly be able to perform the following commands to get some intel about
their data.
### General
For the first view, you can easily use the "df-command".
df
```Bash
df
```
Alternativly you can use the "findmnt"-command, which is also able to
perform an "df" by adding the "-D"-parameter.
Alternativly you can use the "findmnt"-command, which is also able to perform an `df` by adding the
"-D"-parameter.
findmnt -D
```Bash
findmnt -D
```
Optional you can use the "-t"-parameter to specify the fs-type or the
"-o"-parameter to alter the output.
Optional you can use the `-t`-parameter to specify the fs-type or the`-o`-parameter to alter the
output.
We do **not recommend** the usage of the "du"-command for this purpose.
It is able to cause issues for other users, while reading data from the
filesystem.
We do **not recommend** the usage of the "du"-command for this purpose. It is able to cause issues
for other users, while reading data from the filesystem.
### Lustre file system
### Lustre File System
These commands work for /scratch and /ssd.
These commands work for `/scratch` and `/ssd`.
#### Listing disk usages per OST and MDT
#### Listing Disk Usages per OST and MDT
lfs quota -h -u username /path/to/my/data
```Bash
lfs quota -h -u username /path/to/my/data
```
It is possible to display the usage on each OST by adding the
"-v"-parameter.
It is possible to display the usage on each OST by adding the "-v"-parameter.
#### Listing space usage per OST and MDT
lfs df -h /path/to/my/data
```Bash
lfs df -h /path/to/my/data
```
#### Listing inode usage for an specific path
lfs df -i /path/to/my/data
```Bash
lfs df -i /path/to/my/data
```
#### Listing OSTs
lfs osts /path/to/my/data
```Bash
lfs osts /path/to/my/data
```
#### View striping information
lfs getstripe myfile
lfs getstripe -d mydirectory
```Bash
lfs getstripe myfile
lfs getstripe -d mydirectory
```
The "-d"-parameter will also display striping for all files in the
directory
The `-d`-parameter will also display striping for all files in the directory
### BeeGFS
...
...
@@ -197,57 +189,70 @@ Commands to work with the BeeGFS file system.
#### Capacity and file system health
View storage and inode capacity and utilization for metadata and storage
targets.
View storage and inode capacity and utilization for metadata and storage targets.
beegfs-df -p /beegfs/global0
```Bash
beegfs-df -p /beegfs/global0
```
The "-p" parameter needs to be the mountpoint of the file system and is
mandatory.
The `-p` parameter needs to be the mountpoint of the file system and is mandatory.
List storage and inode capacity, reachability and consistency
information of each storage target.
List storage and inode capacity, reachability and consistency information of each storage target.
