Skip to content
Snippets Groups Projects
Commit 70ad11f4 authored by Ulf Markwardt's avatar Ulf Markwardt
Browse files

erster sinnvoller wurf

parent 294e4159
No related branches found
No related tags found
3 merge requests!322Merge preview into main,!319Merge preview into main,!223Filesystems.md
......@@ -4,7 +4,6 @@ As soon as you have access to ZIH systems you have to manage your data. Several
available. Each file system serves for special purpose according to their respective capacity,
performance and permanence.
## Work Directories
| File system | Usable directory | Capacity | Availability | Backup | Remarks |
......@@ -12,8 +11,7 @@ performance and permanence.
| `Lustre` | `/scratch/` | 4 PB | global | No | Only accessible via **todo link: workspaces - WorkSpaces**. Not made for billions of files! |
| `Lustre` | `/lustre/ssd` | 40 TB | global | No | Only accessible via **todo link: workspaces - WorkSpaces**. For small I/O operations |
| `BeeGFS` | `/beegfs/global0` | 232 TB | global | No | Only accessible via **todo link: workspaces - WorkSpaces**. Fastest available file system, only for large parallel applications running with millions of small I/O operations |
| `ext4` | `/tmp` | 95.0 GB | local | No | is cleaned up after the job automatically |
| `ext4` | `/tmp` | 95.0 GB | local | No | is cleaned up after the job automatically |
## Warm Archive
......@@ -78,8 +76,6 @@ output.
We do **not recommend** the usage of the "du"-command for this purpose. It is able to cause issues
for other users, while reading data from the filesystem.
### BeeGFS
Commands to work with the BeeGFS file system.
......
# Lustre File System(s)
### Large Files in /scratch
## Large Files in /scratch
The data containers in Lustre are called object storage targets (OST). The capacity of one OST is
about 21 TB. All files are striped over a certain number of these OSTs. For small and medium files,
......@@ -20,8 +18,8 @@ lfs setstripe -c 20 /scratch/ws/mark-stripe20/tar
**Note:** This does not affect existing files. But all files that **will be created** in this
directory will be distributed over 20 OSTs.
## Useful Commands for Lustre
These commands work for `/scratch` and `/ssd`.
### Listing Disk Usages per OST and MDT
......@@ -57,5 +55,4 @@ lfs getstripe myfile
lfs getstripe -d mydirectory
```
The `-d`-parameter will also display striping for all files in the directory
The `-d`-parameter will also display striping for all files in the directory.
......@@ -5,12 +5,10 @@
Each user has 50 GB in a `/home` directory independent of the granted capacity for the project.
Hints for the usage of the global home directory:
- Do not use your `/home` as work directory: Frequent changes (like temporary output from a
running job) would fill snapshots and backups (see below).
- If you need distinct `.bashrc` files for each machine, you should
create separate files for them, named `.bashrc_<machine_name>`
- If you use various machines frequently, it might be useful to set
the environment variable HISTFILE in `.bashrc_deimos` and
`.bashrc_mars` to `$HOME/.bash_history_<machine_name>`. Setting
HISTSIZE and HISTFILESIZE to 10000 helps as well.
- Further, you may use private module files to simplify the process of
loading the right installation directories, see
**todo link: private modules - AnchorPrivateModule**.
......@@ -19,8 +17,7 @@ Hints for the usage of the global home directory:
For project data, we have a global project directory, that allows better collaboration between the
members of an HPC project. However, for compute nodes /projects is mounted as read-only, because it
is not a filesystem for parallel I/O. See below and also check the
**todo link: HPC introduction - %PUBURL%/Compendium/WebHome/HPC-Introduction.pdf** for more details.
is not a filesystem for parallel I/O.
## Backup and Snapshots of the File System
......@@ -70,4 +67,3 @@ In case a project is above it's limits please ...
archives with meaningful names or IDs and store e.g. them in an
[archive](intermediate_archive.md).
- Refer to the hints for [long term preservation for research data](preservation_research_data.md)
\ No newline at end of file
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment