Skip to content
Snippets Groups Projects
Commit 4d1c22c1 authored by Martin Schroschk's avatar Martin Schroschk
Browse files

Split documentation for permament and working filesystems into separate 

pages
parent f89b55a4
No related branches found
No related tags found
2 merge requests!938Automated merge from preview to main,!936Update to Five-Cluster-Operation
Hide whitespace changes
Inline Side-by-side
doc.zih.tu-dresden.de/docs/data_lifecycle/file_systems.md
+ 7
− 41
View file @ 4d1c22c1
...@@ -4,47 +4,13 @@ As soon as you have access to ZIH systems, you have to manage your data. Several ...@@ -4,47 +4,13 @@ As soon as you have access to ZIH systems, you have to manage your data. Several
available. Each filesystem serves for special purpose according to their respective capacity, available. Each filesystem serves for special purpose according to their respective capacity,
performance and permanence. performance and permanence.
## Work Directories We differentiate **between filesystems** and **working filesystems**:
!!! note "End of life of `scratch` and `ssd`" * the [permanent filesystems](permanent.md), i.e. `/home` and `/projects`, are meant to hold your
source code, configuration files, and other permanent data.
The filesystem `/lustre/scratch` and `/lustre/ssd` will be turned off on January 3 2024 for good * The [working filesystems](working.md), i.e, `horse`, `walrus`, etc., are designed as scratch
(no data access afterwards!). filesystems holding your working and temporary data, e.g., input and output of your compute
jobs.
The `/beegfs` filesystem will remain available to
[Alpha Centauri](../jobs_and_resources/hardware_overview_2023.md#alpha-centauri-amd-rome-cpus-nvidia-a100)
and
[Power9](../jobs_and_resources/hardware_overview_2023.md#ibm-power9-nodes-for-machine-learning)
users only.
All others need to migrate your data to Barnard’s new file system `/horse`. Please follow these
detailed instruction on how to [migrate to Barnard](../jobs_and_resources/migration_to_barnard.md).
TODO Where to add this information:
All clusters will have access to these shared parallel filesystems:
| Filesystem | Usable directory | Type | Capacity | Purpose |
| --- | --- | --- | --- | --- |
| Home | `/home` | Lustre | quota per user: 20 GB | permanent user data |
| Project | `/projects` | Lustre | quota per project | permanent project data |
| Scratch for large data / streaming | `/data/horse` | Lustre | 20 PB | |
<!--end-->
| Filesystem | Usable directory | Capacity | Availability | Backup | Remarks |
|:------------|:------------------|:---------|:-------------|:-------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Lustre` | `/data/horse` | 20 PB | global | No | Only accessible via [Workspaces](workspaces.md).
| `Lustre` | `/data/weasel` | 1 PB | global | No | Coming 2024. For high IOPS |
| `ext4` | `/tmp` | 95 GB | local | No | Systems: tbd. Is cleaned up after the job automatically. |
??? "Outdated filesystems"
| Filesystem | Usable directory | Capacity | Availability | Backup | Remarks |
|:------------|:------------------|:---------|:-------------|:-------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Lustre` | `/scratch/` | 4 PB | global | No | Only accessible via [Workspaces](workspaces.md). Not made for billions of files! |
| `Lustre` | `/lustre/ssd` | 40 TB | global | No | Only accessible via [Workspaces](workspaces.md). For small I/O operations |
| `BeeGFS` | `/beegfs/global0` | 280 TB | global | No | Only accessible via [Workspaces](workspaces.md). Fastest available filesystem, only for large parallel applications running with millions of small I/O operations |
| `BeeGFS` | `/beegfs/global1` | 232 TB | global | No | Only accessible via [Workspaces](workspaces.md). Fastest available filesystem, only for large parallel applications running with millions of small I/O operations |
| `ext4` | `/tmp` | 95 GB | local | No | is cleaned up after the job automatically |
## Recommendations for Filesystem Usage ## Recommendations for Filesystem Usage
......
doc.zih.tu-dresden.de/docs/data_lifecycle/permanent.md
+ 7
− 1
View file @ 4d1c22c1
...@@ -8,9 +8,15 @@ ...@@ -8,9 +8,15 @@
senselessly filling the disks, senselessly filling the disks,
- By the sheer number and volume of work files, they may keep the backup from working efficiently. - By the sheer number and volume of work files, they may keep the backup from working efficiently.
| Filesystem Name | Usable Directory | Availability | Type | Quota |
|:------------------|:------------------|:-------------|:---------|:-------------------|
| Home | `/home` | global (w/o Power9) | Lustre | per user: 20 GB |
| Projects | `/projects` | global /w/o Power9) | Lustre | per project |
| (Taurus/old) Home | `/home` | [Power9](../job_and_resources/power.md) | NFS | per user: 20 GB |
## Global /home Filesystem ## Global /home Filesystem
Each user has 50 GiB in a `/home` directory independent of the granted capacity for the project. Each user has 20 GiB in a `/home` directory independent of the granted capacity for the project.
The home directory is mounted with read-write permissions on all nodes of the ZIH system. The home directory is mounted with read-write permissions on all nodes of the ZIH system.
Hints for the usage of the global home directory: Hints for the usage of the global home directory:
......
doc.zih.tu-dresden.de/docs/data_lifecycle/working.md 0 → 100644
+ 79
− 0
View file @ 4d1c22c1
# Working Filesystems
As soon as you have access to ZIH systems, you have to manage your data. Several filesystems are
available. Each filesystem serves for special purpose according to their respective capacity,
performance and permanence.
!!! danger "End of life of `scratch` and `ssd`"
The filesystem `/lustre/scratch` and `/lustre/ssd` will be turned off on January 3 2024 for good
(no data access afterwards!).
The `/beegfs` filesystem will remain available to
[Alpha Centauri](../jobs_and_resources/hardware_overview_2023.md#alpha-centauri)
and
[Power](../jobs_and_resources/hardware_overview.md#power9)
users only.
All others need to migrate your data to Barnard’s new file system `/horse`. Please follow these
detailed instruction on how to [migrate to Barnard](../jobs_and_resources/barnard.md).
| Filesystem Type | Usable Directory | Capacity | Availability | Remarks |
|:----------------|:------------------|:---------|:-------------------|:----------------------------------------------------------|
| `Lustre` | `/data/horse` | 20 PB | global | Only accessible via [Workspaces](workspaces.md). **The(!)** working directory to meet almost all demands |
| `Lustre` | `/data/walrus` | 20 PB | global | Only accessible via [Workspaces](workspaces.md). For moderately low bandwidth, low IOPS. Mounted read-only on compute nodes. |
| `WEKAio` | `/data/weasel` | 1 PB | global (w/o Power) | *Coming 2024!* For high IOPS |
| `BeeGFS` | `/beegfs/global0` | 280 TB | [Alpha](../jobs_and_resources/alpha_centauri.md) and [Power9](../jobs_and_resources/power9.md) | Only accessible via [Workspaces](workspaces.md). Fastest available filesystem, only for large parallel applications running with millions of small I/O operations |
| `BeeGFS` | `/beegfs/global1` | 232 TB | [Alpha](../jobs_and_resources/alpha_centauri.md) and [Power9](../jobs_and_resources/power9.md) | Only accessible via [Workspaces](workspaces.md). Fastest available filesystem, only for large parallel applications running with millions of small I/O operations |
| `ext4` | `/tmp` | 95 GB | node local | Systems: tbd. Is cleaned up after the job automatically. |
| Scratch for large data / streaming | `/data/horse` | Lustre | 20 PB | |
??? "Outdated filesystems `/lustre/scratch` and `/lustre/ssd`"
| Filesystem | Usable directory | Capacity | Availability | Backup | Remarks |
|:------------|:------------------|:---------|:-------------|:-------|:---------------------------------------------------------------------------------|
| `Lustre` | `/scratch/` | 4 PB | global | No | Only accessible via [Workspaces](workspaces.md). Not made for billions of files! |
| `Lustre` | `/lustre/ssd` | 40 TB | global | No | Only accessible via [Workspaces](workspaces.md). For small I/O operations |
## Recommendations for Filesystem Usage
To work as efficient as possible, consider the following points
- Save source code etc. in `/home` or `/projects/...`
- Store checkpoints and other temporary data in `/scratch/ws/...`
- Compilation in `/dev/shm` or `/tmp`
Getting high I/O-bandwidth
- Use many clients
- Use many processes (writing in the same file at the same time is possible)
- Use large I/O transfer blocks
## Cheat Sheet for Debugging Filesystem Issues
Users can select from the following commands to get some idea about
their data.
### General
For the first view, you can use the command `df`.
```console
marie@login$ df
```
Alternatively, you can use the command `findmnt`, which is also able to report space usage
by adding the parameter `-D`:
```console
marie@login$ findmnt -D
```
Optionally, you can use the parameter `-t` to specify the filesystem type or the parameter `-o` to
alter the output.
!!! important
Do **not** use the `du`-command for this purpose. It is able to cause issues
for other users, while reading data from the filesystem.
doc.zih.tu-dresden.de/mkdocs.yml
+ 2
− 0
View file @ 4d1c22c1
...@@ -34,6 +34,7 @@ nav: ...@@ -34,6 +34,7 @@ nav:
- Filesystems: - Filesystems:
- Overview: data_lifecycle/file_systems.md - Overview: data_lifecycle/file_systems.md
- Permanent Filesystems: data_lifecycle/permanent.md - Permanent Filesystems: data_lifecycle/permanent.md
- Working Filesystems: data_lifecycle/working.md
- Lustre: data_lifecycle/lustre.md - Lustre: data_lifecycle/lustre.md
- BeeGFS: data_lifecycle/beegfs.md - BeeGFS: data_lifecycle/beegfs.md
- Warm Archive: data_lifecycle/warm_archive.md - Warm Archive: data_lifecycle/warm_archive.md
...@@ -235,6 +236,7 @@ plugins: ...@@ -235,6 +236,7 @@ plugins:
validate_external_urls: False validate_external_urls: False
# Validate the entire rendered template, slow! # Validate the entire rendered template, slow!
validate_rendered_template: True validate_rendered_template: True
- mkdocs-video:
is_video: True is_video: True
# Enable cross-page anchor validation # Enable cross-page anchor validation
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Nutzungsbedingungen | Datenschutzerklärung | Barrierefreiheitserklärung