diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/beegfs.md b/doc.zih.tu-dresden.de/docs/archive/beegfs.md
similarity index 100%
rename from doc.zih.tu-dresden.de/docs/data_lifecycle/beegfs.md
rename to doc.zih.tu-dresden.de/docs/archive/beegfs.md
diff --git a/doc.zih.tu-dresden.de/docs/archive/filesystems.md b/doc.zih.tu-dresden.de/docs/archive/filesystems.md
index 089e1fa75b10fea0f438bee8ab27925b926246ff..5f6bf4a5c1eb550f63b9b7ee775a157b9f073df8 100644
--- a/doc.zih.tu-dresden.de/docs/archive/filesystems.md
+++ b/doc.zih.tu-dresden.de/docs/archive/filesystems.md
@@ -66,7 +66,7 @@ set accordingly.
system (end of 2023). Thus, please **do not use** `warm_archive` any longer and **migrate you
data from** `warm_archive` to the new filesystems. We provide a quite comprehensive
documentation on the
- [data migration process to the new filesystems](../jobs_and_resources/barnard.md#data-migration-to-new-filesystems).
+ [data migration process to the new filesystems](migration_to_barnard.md#data-migration-to-new-filesystems).
You should consider the new `walrus` storage as an substitue for jobs with moderately low
bandwidth, low IOPS.
diff --git a/doc.zih.tu-dresden.de/docs/archive/migration_to_barnard.md b/doc.zih.tu-dresden.de/docs/archive/migration_to_barnard.md
new file mode 100644
index 0000000000000000000000000000000000000000..188fdb5fa5e09a94959fd99ad663a10eb3854818
--- /dev/null
+++ b/doc.zih.tu-dresden.de/docs/archive/migration_to_barnard.md
@@ -0,0 +1,323 @@
+# CPU Cluster Barnard
+
+* [Prepare login to Barnard](#login-to-barnard)
+* [Data management and data transfer to new filesystems](#data-management-and-data-transfer)
+* [Update job scripts and workflow to new software](#software)
+* [Update job scripts and workflow w.r.t. Slurm](#slurm)
+
+!!! note
+
+ We highly recommand to first read the entire page carefully, and then execute the steps.
+
+The migration can only be successful as a joint effort of HPC team and users.
+We value your feedback. Please provide it directly via our ticket system. For better processing,
+please add "Barnard:" as a prefix to the subject of the [support ticket](../support/support.md).
+
+## Login to Barnard
+
+You use `login[1-4].barnard.hpc.tu-dresden.de` to access the system
+from campus (or VPN). In order to verify the SSH fingerprints of the login nodes, please refer to
+the page [Fingerprints](../access/key_fingerprints.md#barnard).
+
+All users have **new empty HOME** file systems, this means you have first to ...
+
+??? "... install your public SSH key on Barnard"
+
+ - Please create a new SSH keypair with ed25519 encryption, secured with
+ a passphrase. Please refer to this
+ [page for instructions](../access/ssh_login.md#before-your-first-connection).
+ - After login, add the public key to your `.ssh/authorized_keys` file on Barnard.
+
+## Data Management and Data Transfer
+
+### Filesystems on Barnard
+
+Our new HPC system Barnard also comes with **two new Lustre filesystems**, namely `/data/horse` and
+`/data/walrus`. Both have a capacity of 20 PB, but differ in performance and intended usage, see
+below. In order to support the data life cycle management, the well-known
+[workspace concept](#workspaces-on-barnard) is applied.
+
+* The `/project` filesystem is the same on Taurus and Barnard
+(mounted read-only on the compute nodes).
+* The new work filesystem is `/data/horse`.
+* The slower `/data/walrus` can be considered as a substitute for the old
+ `/warm_archive`- mounted **read-only** on the compute nodes.
+ It can be used to store e.g. results.
+
+### Workspaces on Barnard
+
+The filesystems `/data/horse` and `/data/walrus` can only be accessed via workspaces. Please refer
+to the [workspace page](../data_lifecycle/workspaces.md), if you are not familiar with the
+workspace concept and the corresponding commands. You can find the settings for
+workspaces on these two filesystems in the
+[section Settings for Workspaces](../data_lifecycle/workspaces.md#workspace-lifetimes).
+
+### Data Migration to New Filesystems
+
+Since all old filesystems of Taurus will be shutdown by the end of 2023, your data needs to be
+migrated to the new filesystems on Barnard. This migration comprises
+
+* your personal `/home` directory,
+* your workspaces on `/ssd`, `/beegfs` and `/scratch`.
+
+!!! note "It's your turn"
+
+ **You are responsible for the migration of your data**. With the shutdown of the old
+ filesystems, all data will be deleted.
+
+!!! note "Make a plan"
+
+ We highly recommand to **take some minutes for planing the transfer process**. Do not act with
+ precipitation.
+
+ Please **do not copy your entire data** from the old to the new filesystems, but consider this
+ opportunity for **cleaning up your data**. E.g., it might make sense to delete outdated scripts,
+ old log files, etc., and move other files, e.g., results, to the `/data/walrus` filesystem.
+
+!!! hint "Generic login"
+
+ In the following we will use the generic login `marie` and workspace `numbercrunch`
+ ([cf. content rules on generic names](../contrib/content_rules.md#data-privacy-and-generic-names)).
+ **Please make sure to replace it with your personal login.**
+
+We have four new [Datamover nodes](../data_transfer/datamover.md) that have mounted all filesystems
+of the old Taurus and new Barnard system. Do not use the Datamover from Taurus, i.e., all data
+transfer need to be invoked from Barnard! Thus, the very first step is to
+[login to Barnard](#login-to-barnard).
+
+The command `dtinfo` will provide you the mount points of the old filesystems
+
+```console
+marie@barnard$ dtinfo
+[...]
+directory on datamover mounting clusters directory on cluster
+
+/data/old/home Taurus /home
+/data/old/lustre/scratch2 Taurus /scratch
+/data/old/lustre/ssd Taurus /lustre/ssd
+[...]
+```
+
+In the following, we will provide instructions with comprehensive examples for the data transfer of
+your data to the new `/home` filesystem, as well as the working filesystems `/data/horse` and
+`/data/walrus`.
+
+??? "Migration of Your Home Directory"
+
+ Your personal (old) home directory at Taurus will not be automatically transferred to the new
+ Barnard system. Please do not copy your entire home, but clean up your data. E.g., it might
+ make sense to delete outdated scripts, old log files, etc., and move other files to an archive
+ filesystem. Thus, please transfer only selected directories and files that you need on the new
+ system.
+
+ The steps are as follows:
+
+ 1. Login to Barnard, i.e.,
+
+ ```
+ ssh login[1-4].barnard.tu-dresden.de
+ ```
+
+ 1. The command `dtinfo` will provide you the mountpoint
+
+ ```console
+ marie@barnard$ dtinfo
+ [...]
+ directory on datamover mounting clusters directory on cluster
+
+ /data/old/home Taurus /home
+ [...]
+ ```
+
+ 1. Use the `dtls` command to list your files on the old home directory
+
+ ```
+ marie@barnard$ dtls /data/old/home/marie
+ [...]
+ ```
+
+ 1. Use the `dtcp` command to invoke a transfer job, e.g.,
+
+ ```console
+ marie@barnard$ dtcp --recursive /data/old/home/marie/<useful data> /home/marie/
+ ```
+
+ **Note**, please adopt the source and target paths to your needs. All available options can be
+ queried via `dtinfo --help`.
+
+ !!! warning
+
+ Please be aware that there is **no synchronisation process** between your home directories
+ at Taurus and Barnard. Thus, after the very first transfer, they will become divergent.
+
+Please follow these instructions for transferring you data from `ssd`, `beegfs` and `scratch` to the
+new filesystems. The instructions and examples are divided by the target not the source filesystem.
+
+This migration task requires a preliminary step: You need to allocate workspaces on the
+target filesystems.
+
+??? Note "Preliminary Step: Allocate a workspace"
+
+ Both `/data/horse/` and `/data/walrus` can only be used with
+ [workspaces](../data_lifecycle/workspaces.md). Before you invoke any data transer from the old
+ working filesystems to the new ones, you need to allocate a workspace first.
+
+ The command `ws_list --list` lists the available and the default filesystem for workspaces.
+
+ ```
+ marie@barnard$ ws_list --list
+ available filesystems:
+ horse (default)
+ walrus
+ ```
+
+ As you can see, `/data/horse` is the default workspace filesystem at Barnard. I.e., if you
+ want to allocate, extend or release a workspace on `/data/walrus`, you need to pass the
+ option `--filesystem=walrus` explicitly to the corresponding workspace commands. Please
+ refer to our [workspace documentation](../data_lifecycle/workspaces.md), if you need refresh
+ your knowledge.
+
+ The most simple command to allocate a workspace is as follows
+
+ ```
+ marie@barnard$ ws_allocate numbercrunch 90
+ ```
+
+ Please refer to the table holding the settings
+ (cf. [subsection workspaces on Barnard](#workspaces-on-barnard)) for the max. duration and
+ `ws_allocate --help` for all available options.
+
+??? "Migration to work filesystem `/data/horse`"
+
+ === "Source: old `/scratch`"
+
+ We are synchronizing the old `/scratch` to `/data/horse/lustre/scratch2/` (**last: October
+ 18**).
+ If you transfer data from the old `/scratch` to `/data/horse`, it is sufficient to use
+ `dtmv` instead of `dtcp` since this data has already been copied to a special directory on
+ the new `horse` filesystem. Thus, you just need to move it to the right place (the Lustre
+ metadata system will update the correspoding entries).
+
+ The workspaces within the subdirectories `ws/0` and `ws/1`, respectively. A corresponding
+ data transfer using `dtmv` looks like
+
+ ```console
+ marie@barnard$ dtmv /data/horse/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
+ ```
+
+ Please do **NOT** copy those data yourself. Instead check if it is already sychronized
+ to `/data/horse/lustre/scratch2/ws/0/marie-numbercrunch`.
+
+ In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in
+
+ ```
+ marie@barnard$ dtrsync -a /data/old/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
+ ```
+
+ === "Source: old `/ssd`"
+
+ The old `ssd` filesystem is mounted at `/data/old/lustre/ssd` on the datamover nodes and the
+ workspaces are within the subdirectory `ws/`. A corresponding data transfer using `dtcp`
+ looks like
+
+ ```console
+ marie@barnard$ dtcp --recursive /data/old/lustre/ssd/ws/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
+ ```
+
+ === "Source: old `/beegfs`"
+
+ The old `beegfs` filesystem is mounted at `/data/old/beegfs` on the datamover nodes and the
+ workspaces are within the subdirectories `ws/0` and `ws/1`, respectively. A corresponding
+ data transfer using `dtcp` looks like
+
+ ```console
+ marie@barnard$ dtcp --recursive /data/old/beegfs/ws/0/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
+ ```
+
+??? "Migration to `/data/walrus`"
+
+ === "Source: old `/scratch`"
+
+ We are synchronizing the old `/scratch` to `/data/horse/lustre/scratch2/` (**last: October
+ 18**). The old `scratch` filesystem has been already synchronized to
+ `/data/horse/lustre/scratch2` nodes and the workspaces are within the subdirectories `ws/0`
+ and `ws/1`, respectively. A corresponding data transfer using `dtcp` looks like
+
+ ```console
+ marie@barnard$ dtcp --recursive /data/horse/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
+ ```
+
+ Please do **NOT** copy those data yourself. Instead check if it is already sychronized
+ to `/data/horse/lustre/scratch2/ws/0/marie-numbercrunch`.
+
+ In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in
+
+ ```
+ marie@barnard$ dtrsync -a /data/old/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
+ ```
+
+ === "Source: old `/ssd`"
+
+ The old `ssd` filesystem is mounted at `/data/old/lustre/ssd` on the datamover nodes and the
+ workspaces are within the subdirectory `ws/`. A corresponding data transfer using `dtcp`
+ looks like
+
+ ```console
+ marie@barnard$ dtcp --recursive /data/old/lustre/ssd/<useful data> /data/walrus/ws/marie-numbercrunch/
+ ```
+
+ === "Source: old `/beegfs`"
+
+ The old `beegfs` filesystem is mounted at `/data/old/beegfs` on the datamover nodes and the
+ workspaces are within the subdirectories `ws/0` and `ws/1`, respectively. A corresponding
+ data transfer using `dtcp` looks like
+
+ ```console
+ marie@barnard$ dtcp --recursive /data/old/beegfs/ws/0/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
+ ```
+
+??? "Migration from `/warm_archive`"
+
+ We are synchronizing the old `/warm_archive` to `/data/walrus/warm_archive/`. Therefor, it can
+ be sufficient to use `dtmv` instead of `dtcp` (No data will be copied, but the Lustre system
+ will update the correspoding metadata entries). A corresponding data transfer using `dtmv` looks
+ like
+
+ ```console
+ marie@barnard$ dtmv /data/walrus/warm_archive/ws/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
+ ```
+
+ Please do **NOT** copy those data yourself. Instead check if it is already sychronized
+ to `/data/walrus/warm_archive/ws`.
+
+ In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in
+
+ ```
+ marie@barnard$ dtrsync -a /data/old/warm_archive/ws/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
+ ```
+
+When the last compute system will have been migrated the old file systems will be
+set write-protected and we start a final synchronization (scratch+walrus).
+The target directories for synchronization `/data/horse/lustre/scratch2/ws` and
+`/data/walrus/warm_archive/ws/` will not be deleted automatically in the meantime.
+
+## Software
+
+Barnard is running on Linux RHEL 8.7. All application software was re-built consequently using Git
+and CI/CD pipelines for handling the multitude of versions.
+
+We start with `release/23.10` which is based on software requests from user feedback of our
+HPC users. Most major software versions exist on all hardware platforms.
+
+Please use `module spider` to identify the software modules you need to load.
+
+## Slurm
+
+* We are running the most recent Slurm version.
+* You must not use the old partition names.
+* Not all things are tested.
+
+Note that most nodes on Barnard don't have a local disk and space in `/tmp` is **very** limited.
+If you need a local disk request this with the
+[Slurm feature](../jobs_and_resources/slurm.md#node-local-storage-in-jobs)
+`--constraint=local_disk` to `sbatch`, `salloc`, and `srun`.
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/working.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/working.md
index d5a11bdfa5b0bb8d2ac648506734af38e1f9a590..6c514c7737e7fe70e30e68c39a73942c2ed0706b 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/working.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/working.md
@@ -9,8 +9,6 @@ performance and permanence.
| `Lustre` | `/data/horse` | 20 PB | global | Only accessible via [Workspaces](workspaces.md). **The(!)** working directory to meet almost all demands |
| `Lustre` | `/data/walrus` | 20 PB | global | Only accessible via [Workspaces](workspaces.md). For moderately low bandwidth, low IOPS. Mounted read-only on compute nodes. |
| `WEKAio` | `/data/weasel` | 1 PB | global (w/o Power) | *Coming 2024!* For high IOPS |
-| `BeeGFS` | `/beegfs/.global0` | 280 TB | [Alpha](../jobs_and_resources/alpha_centauri.md) and [Power9](../jobs_and_resources/power9.md) | Only accessible via [Workspaces](workspaces.md). Fastest available filesystem, only for large parallel applications running with millions of small I/O operations |
-| `BeeGFS` | `/beegfs/.global1` | 232 TB | [Alpha](../jobs_and_resources/alpha_centauri.md) and [Power9](../jobs_and_resources/power9.md) | Only accessible via [Workspaces](workspaces.md). Fastest available filesystem, only for large parallel applications running with millions of small I/O operations |
| `ext4` | `/tmp` | 95 GB | node local | Systems: tbd. Is cleaned up after the job automatically. |
## Recommendations for Filesystem Usage
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md
index da470eda5178c27776ab91f9f0d0aa5eee98a98e..9b2c13b48c485d327cd4817fdc301b293c98c70b 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md
@@ -35,7 +35,6 @@ renewals are provided in the following table.
|:------------------------------------------------------------|---------------:|-----------:|---------:|
| `horse` | 100 | 10 | 30 |
| `walrus` | 100 | 10 | 60 |
-| `beegfs` | 30 | 2 | 30 |
{: summary="Settings for Workspace Filesystems."}
!!! note
@@ -64,9 +63,8 @@ provides information which filesystem is available on which cluster.
```console
marie@login.alpha$ ws_list -l
available filesystems:
- ssd
- beegfs_global0
- beegfs (default)
+ horse (default)
+ walrus
```
=== "Romeo"
@@ -74,7 +72,8 @@ provides information which filesystem is available on which cluster.
```console
marie@login.romeo$ ws_list -l
available filesystems:
- horse
+ horse (default)
+ walrus
```
!!! note "Default filesystem"
diff --git a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
index 71955aebc44f6f674a4270e69539c4903a562802..26388b51b16fdd6781eeff79a1051f7544d1be60 100644
--- a/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
+++ b/doc.zih.tu-dresden.de/docs/data_transfer/datamover.md
@@ -40,21 +40,6 @@ To identify the mount points of the different filesystems on the data transfer m
## Usage of Datamover
-<!--TODO: remove when released in May 2024-->
-??? "Data on outdated filesystems"
-
- !!! example "Copying data from `/beegfs/.global0` to `/projects` filesystem."
-
- ```console
- marie@login$ dtcp -r /data/old/beegfs/.global0/ws/marie-workdata/results /projects/p_number_crunch/.
- ```
-
- !!! example "Archive data from `/beegfs/.global0` to `/archiv` filesystem."
-
- ```console
- marie@login$ dttar -czf /data/archiv/p_number_crunch/results.tgz /data/old/beegfs/global0/ws/marie-workdata/results
- ```
-
!!! example "Copy data from `/data/horse` to `/projects` filesystem."
```console
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md
index 1c52090ab204005fe76de4ad6986e9fd4cdaf843..562ee88b8e1b506428de450f21cd7ed2a9b7d4da 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/alpha_centauri.md
@@ -5,17 +5,7 @@ The multi-GPU cluster `Alpha Centauri` has been installed for AI-related computa
The hardware specification is documented on the page
[HPC Resources](hardware_overview.md#alpha-centauri).
-## Becoming a Stand-Alone Cluster
-
-The former HPC system Taurus is partly switched-off and partly split up into separate clusters
-until the end of 2023. One such upcoming separate cluster is what you have known as partition
-`alpha` so far. With the end of the maintenance at November 30 2023, `Alpha Centauri` is now a
-stand-alone cluster with
-
-* homogeneous hardware resources incl. two login nodes `login[1-2].alpha.hpc.tu-dresden.de`,
-* and own [Slurm batch system](slurm.md).
-
-### Filesystems
+## Filesystems
Your new `/home` directory (from `Barnard`) is also your `/home` on `Alpha Centauri`. Since 5th July
2024, `Alpha Centauri` is fully integrated in the InfiniBand infrastructure of `Barnard`. With that,
@@ -69,9 +59,9 @@ marie@login.alpha$ module spider <module_name>
[...]
```
- Not all modules can be loaded directly. Most modules are build with a certain compiler or toolchain
- that need to be loaded beforehand.
- Luckely, the module system can tell us, what we need to do for a specific module or software version
+ Not all modules can be loaded directly. Most modules are build with a certain compiler or
+ toolchain that need to be loaded beforehand. Luckely, the module system can tell us, what we
+ need to do for a specific module or software version
```console
marie@login.alpha$ module spider PyTorch/1.12.1-CUDA-11.7.0
@@ -97,6 +87,8 @@ marie@login.alpha$ module spider <module_name>
Module GCC/11.3.0, OpenMPI/4.1.4, PyTorch/1.12.1-CUDA-11.7.0 and 64 dependencies loaded.
```
+ Now, you can verify with the following command that the pytorch module is available
+
```console
marie@login.alpha$ python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
1.12.1
@@ -115,13 +107,21 @@ additional Python packages and create an isolated runtime environment. We recomm
??? example "Example: Creating a virtual environment and installing `torchvision` package"
+ As first step, you should create a workspace
+
```console
marie@login.alpha$ srun --nodes=1 --cpus-per-task=1 --gres=gpu:1 --time=01:00:00 --pty bash -l
- marie@alpha$ ws_allocate -n python_virtual_environment -d 1
+ marie@alpha$ ws_allocate --name python_virtual_environment --duration 1
Info: creating workspace.
- /beegfs/ws/1/marie-python_virtual_environment
+ /horse/ws/marie-python_virtual_environment
remaining extensions : 2
remaining time in days: 1
+ ```
+
+ Now, you can load the desired modules and create a virtual environment within the allocated
+ workspace.
+
+ ```
marie@alpha$ module load release/23.04 GCCcore/11.3.0 GCC/11.3.0 OpenMPI/4.1.4 Python/3.10.4
Module GCC/11.3.0, OpenMPI/4.1.4, Python/3.10.4 and 21 dependencies loaded.
marie@alpha$ module load PyTorch/1.12.1-CUDA-11.7.0
@@ -130,13 +130,13 @@ additional Python packages and create an isolated runtime environment. We recomm
/software/rome/r23.04/Python/3.10.4-GCCcore-11.3.0/bin/python
marie@alpha$ pip list
[...]
- marie@alpha$ virtualenv --system-site-packages /beegfs/ws/1/marie-python_virtual_environment/my-torch-env
+ marie@alpha$ virtualenv --system-site-packages /data/horse/ws/marie-python_virtual_environment/my-torch-env
created virtual environment CPython3.8.6.final.0-64 in 42960ms
- creator CPython3Posix(dest=/beegfs/.global1/ws/marie-python_virtual_environment/my-torch-env, clear=False, global=True)
+ creator CPython3Posix(dest=/horse/.global1/ws/marie-python_virtual_environment/my-torch-env, clear=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=~/.local/share/virtualenv)
added seed packages: pip==21.1.3, setuptools==57.2.0, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
- marie@alpha$ source /beegfs/ws/1/marie-python_virtual_environment/my-torch-env/bin/activate
+ marie@alpha$ source /data/horse/ws/marie-python_virtual_environment/my-torch-env/bin/activate
(my-torch-env) marie@alpha$ pip install torchvision==0.13.1
[...]
Installing collected packages: torchvision
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard.md
index 95976c5f7f337f4f97ee97541bef4a8d91285598..c4fa64eb45a551bd98fb4f7d155f63df3d3e1686 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/barnard.md
@@ -9,7 +9,6 @@
We highly recommand to first read the entire page carefully, and then execute the steps.
-The migration can only be successful as a joint effort of HPC team and users.
We value your feedback. Please provide it directly via our ticket system. For better processing,
please add "Barnard:" as a prefix to the subject of the [support ticket](../support/support.md).
@@ -19,7 +18,7 @@ You use `login[1-4].barnard.hpc.tu-dresden.de` to access the system
from campus (or VPN). In order to verify the SSH fingerprints of the login nodes, please refer to
the page [Fingerprints](../access/key_fingerprints.md#barnard).
-All users have **new empty HOME** file systems, this means you have first to ...
+All users have **new empty HOME** filesystems, this means you have first to ...
??? "... install your public SSH key on Barnard"
@@ -37,20 +36,12 @@ Our new HPC system Barnard also comes with **two new Lustre filesystems**, namel
below. In order to support the data life cycle management, the well-known
[workspace concept](#workspaces-on-barnard) is applied.
-* The `/project` filesystem is the same on Taurus and Barnard
-(mounted read-only on the compute nodes).
+* The `/project` filesystem is mounted read-only on the compute nodes.
* The new work filesystem is `/data/horse`.
* The slower `/data/walrus` can be considered as a substitute for the old
`/warm_archive`- mounted **read-only** on the compute nodes.
It can be used to store e.g. results.
-!!! Warning
-
- All old filesystems, i.e., `ssd`, `beegfs`, and `scratch`, will be shutdown by the end of 2023.
- To work with your data from Taurus you might have to move/copy them to the new storages.
-
- Please, carefully read the following documentation and instructions.
-
### Workspaces on Barnard
The filesystems `/data/horse` and `/data/walrus` can only be accessed via workspaces. Please refer
@@ -59,255 +50,6 @@ workspace concept and the corresponding commands. You can find the settings for
workspaces on these two filesystems in the
[section Settings for Workspaces](../data_lifecycle/workspaces.md#workspace-lifetimes).
-### Data Migration to New Filesystems
-
-Since all old filesystems of Taurus will be shutdown by the end of 2023, your data needs to be
-migrated to the new filesystems on Barnard. This migration comprises
-
-* your personal `/home` directory,
-* your workspaces on `/ssd`, `/beegfs` and `/scratch`.
-
-!!! note "It's your turn"
-
- **You are responsible for the migration of your data**. With the shutdown of the old
- filesystems, all data will be deleted.
-
-!!! note "Make a plan"
-
- We highly recommand to **take some minutes for planing the transfer process**. Do not act with
- precipitation.
-
- Please **do not copy your entire data** from the old to the new filesystems, but consider this
- opportunity for **cleaning up your data**. E.g., it might make sense to delete outdated scripts,
- old log files, etc., and move other files, e.g., results, to the `/data/walrus` filesystem.
-
-!!! hint "Generic login"
-
- In the following we will use the generic login `marie` and workspace `numbercrunch`
- ([cf. content rules on generic names](../contrib/content_rules.md#data-privacy-and-generic-names)).
- **Please make sure to replace it with your personal login.**
-
-We have four new [datamover nodes](../data_transfer/datamover.md) that have mounted all storages
-of the old Taurus and new Barnard system. Do not use the datamovers from Taurus, i.e., all data
-transfer need to be invoked from Barnard! Thus, the very first step is to
-[login to Barnard](#login-to-barnard).
-
-The command `dtinfo` will provide you the mount points of the old filesystems
-
-```console
-marie@barnard$ dtinfo
-[...]
-directory on datamover mounting clusters directory on cluster
-
-/data/old/home Taurus /home
-/data/old/lustre/scratch2 Taurus /scratch
-/data/old/lustre/ssd Taurus /lustre/ssd
-[...]
-```
-
-In the following, we will provide instructions with comprehensive examples for the data transfer of
-your data to the new `/home` filesystem, as well as the working filesystems `/data/horse` and
-`/data/walrus`.
-
-??? "Migration of Your Home Directory"
-
- Your personal (old) home directory at Taurus will not be automatically transferred to the new
- Barnard system. Please do not copy your entire home, but clean up your data. E.g., it might
- make sense to delete outdated scripts, old log files, etc., and move other files to an archive
- filesystem. Thus, please transfer only selected directories and files that you need on the new
- system.
-
- The steps are as follows:
-
- 1. Login to Barnard, i.e.,
-
- ```
- ssh login[1-4].barnard.tu-dresden.de
- ```
-
- 1. The command `dtinfo` will provide you the mountpoint
-
- ```console
- marie@barnard$ dtinfo
- [...]
- directory on datamover mounting clusters directory on cluster
-
- /data/old/home Taurus /home
- [...]
- ```
-
- 1. Use the `dtls` command to list your files on the old home directory
-
- ```
- marie@barnard$ dtls /data/old/home/marie
- [...]
- ```
-
- 1. Use the `dtcp` command to invoke a transfer job, e.g.,
-
- ```console
- marie@barnard$ dtcp --recursive /data/old/home/marie/<useful data> /home/marie/
- ```
-
- **Note**, please adopt the source and target paths to your needs. All available options can be
- queried via `dtinfo --help`.
-
- !!! warning
-
- Please be aware that there is **no synchronisation process** between your home directories
- at Taurus and Barnard. Thus, after the very first transfer, they will become divergent.
-
-Please follow these instructions for transferring you data from `ssd`, `beegfs` and `scratch` to the
-new filesystems. The instructions and examples are divided by the target not the source filesystem.
-
-This migration task requires a preliminary step: You need to allocate workspaces on the
-target filesystems.
-
-??? Note "Preliminary Step: Allocate a workspace"
-
- Both `/data/horse/` and `/data/walrus` can only be used with
- [workspaces](../data_lifecycle/workspaces.md). Before you invoke any data transer from the old
- working filesystems to the new ones, you need to allocate a workspace first.
-
- The command `ws_list --list` lists the available and the default filesystem for workspaces.
-
- ```
- marie@barnard$ ws_list --list
- available filesystems:
- horse (default)
- walrus
- ```
-
- As you can see, `/data/horse` is the default workspace filesystem at Barnard. I.e., if you
- want to allocate, extend or release a workspace on `/data/walrus`, you need to pass the
- option `--filesystem=walrus` explicitly to the corresponding workspace commands. Please
- refer to our [workspace documentation](../data_lifecycle/workspaces.md), if you need refresh
- your knowledge.
-
- The most simple command to allocate a workspace is as follows
-
- ```
- marie@barnard$ ws_allocate numbercrunch 90
- ```
-
- Please refer to the table holding the settings
- (cf. [subsection workspaces on Barnard](#workspaces-on-barnard)) for the max. duration and
- `ws_allocate --help` for all available options.
-
-??? "Migration to work filesystem `/data/horse`"
-
- === "Source: old `/scratch`"
-
- We are synchronizing the old `/scratch` to `/data/horse/lustre/scratch2/` (**last: October
- 18**).
- If you transfer data from the old `/scratch` to `/data/horse`, it is sufficient to use
- `dtmv` instead of `dtcp` since this data has already been copied to a special directory on
- the new `horse` filesystem. Thus, you just need to move it to the right place (the Lustre
- metadata system will update the correspoding entries).
-
- The workspaces within the subdirectories `ws/0` and `ws/1`, respectively. A corresponding
- data transfer using `dtmv` looks like
-
- ```console
- marie@barnard$ dtmv /data/horse/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
- ```
-
- Please do **NOT** copy those data yourself. Instead check if it is already sychronized
- to `/data/horse/lustre/scratch2/ws/0/marie-numbercrunch`.
-
- In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in
-
- ```
- marie@barnard$ dtrsync -a /data/old/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
- ```
-
- === "Source: old `/ssd`"
-
- The old `ssd` filesystem is mounted at `/data/old/lustre/ssd` on the datamover nodes and the
- workspaces are within the subdirectory `ws/`. A corresponding data transfer using `dtcp`
- looks like
-
- ```console
- marie@barnard$ dtcp --recursive /data/old/lustre/ssd/ws/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
- ```
-
- === "Source: old `/beegfs`"
-
- The old `beegfs` filesystem is mounted at `/data/old/beegfs` on the datamover nodes and the
- workspaces are within the subdirectories `ws/0` and `ws/1`, respectively. A corresponding
- data transfer using `dtcp` looks like
-
- ```console
- marie@barnard$ dtcp --recursive /data/old/beegfs/ws/0/marie-numbercrunch/<useful data> /data/horse/ws/marie-numbercrunch/
- ```
-
-??? "Migration to `/data/walrus`"
-
- === "Source: old `/scratch`"
-
- We are synchronizing the old `/scratch` to `/data/horse/lustre/scratch2/` (**last: October
- 18**). The old `scratch` filesystem has been already synchronized to
- `/data/horse/lustre/scratch2` nodes and the workspaces are within the subdirectories `ws/0`
- and `ws/1`, respectively. A corresponding data transfer using `dtcp` looks like
-
- ```console
- marie@barnard$ dtcp --recursive /data/horse/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
- ```
-
- Please do **NOT** copy those data yourself. Instead check if it is already sychronized
- to `/data/horse/lustre/scratch2/ws/0/marie-numbercrunch`.
-
- In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in
-
- ```
- marie@barnard$ dtrsync -a /data/old/lustre/scratch2/ws/0/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
- ```
-
- === "Source: old `/ssd`"
-
- The old `ssd` filesystem is mounted at `/data/old/lustre/ssd` on the datamover nodes and the
- workspaces are within the subdirectory `ws/`. A corresponding data transfer using `dtcp`
- looks like
-
- ```console
- marie@barnard$ dtcp --recursive /data/old/lustre/ssd/<useful data> /data/walrus/ws/marie-numbercrunch/
- ```
-
- === "Source: old `/beegfs`"
-
- The old `beegfs` filesystem is mounted at `/data/old/beegfs` on the datamover nodes and the
- workspaces are within the subdirectories `ws/0` and `ws/1`, respectively. A corresponding
- data transfer using `dtcp` looks like
-
- ```console
- marie@barnard$ dtcp --recursive /data/old/beegfs/ws/0/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
- ```
-
-??? "Migration from `/warm_archive`"
-
- We are synchronizing the old `/warm_archive` to `/data/walrus/warm_archive/`. Therefor, it can
- be sufficient to use `dtmv` instead of `dtcp` (No data will be copied, but the Lustre system
- will update the correspoding metadata entries). A corresponding data transfer using `dtmv` looks
- like
-
- ```console
- marie@barnard$ dtmv /data/walrus/warm_archive/ws/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
- ```
-
- Please do **NOT** copy those data yourself. Instead check if it is already sychronized
- to `/data/walrus/warm_archive/ws`.
-
- In case you need to update this (Gigabytes, not Terabytes!) please run `dtrsync` like in
-
- ```
- marie@barnard$ dtrsync -a /data/old/warm_archive/ws/marie-numbercrunch/<useful data> /data/walrus/ws/marie-numbercrunch/
- ```
-
-When the last compute system will have been migrated the old file systems will be
-set write-protected and we start a final synchronization (scratch+walrus).
-The target directories for synchronization `/data/horse/lustre/scratch2/ws` and
-`/data/walrus/warm_archive/ws/` will not be deleted automatically in the meantime.
-
## Software
Barnard is running on Linux RHEL 8.7. All application software was re-built consequently using Git
@@ -324,6 +66,8 @@ Please use `module spider` to identify the software modules you need to load.
* You must not use the old partition names.
* Not all things are tested.
+### Node-Local Storage
+
Note that most nodes on Barnard don't have a local disk and space in `/tmp` is **very** limited.
If you need a local disk request this with the
[Slurm feature](slurm.md#node-local-storage-in-jobs)
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/julia.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/julia.md
index fee65e5638a56503f4ab489067585a6dec614285..e193e54aaad3da39147e379474ecd094486393b7 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/julia.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/julia.md
@@ -4,15 +4,6 @@ The HPE Superdome Flex is a large shared memory node. It is especially well suit
intensive application scenarios, for example to process extremely large data sets completely in main
memory or in very fast NVMe memory.
-## Becoming a Stand-Alone Cluster
-
-The former HPC system Taurus is partly switched-off and partly split up into separate clusters
-until the end of 2023. One such upcoming separate cluster is what you have known as partition
-`julia` so far. Since February 2024, `Julia` is now a stand-alone cluster with
-
-* homogenous hardware resources available at `julia.hpc.tu-dresden.de`,
-* and own Slurm batch system.
-
## Hardware Resources
The hardware specification is documented on the page
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_examples.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_examples.md
index 7d2453a7197fae446ab390000d099645d348048f..66d06f557678db40abfad7a39cac328b62d12e71 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_examples.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm_examples.md
@@ -36,10 +36,9 @@ For MPI-parallel jobs one typically allocates one core per task that has to be s
!!! warning "MPI libraries"
There are different MPI libraries on ZIH systems for the different micro architectures. Thus,
- you have to compile the binaries specifically for the target architecture of the cluster of interest. Please
- refer to the sections [building software](../software/building_software.md) and
- [module environments](../software/modules.md#module-environments) for detailed
- information.
+ you have to compile the binaries specifically for the target architecture of the cluster of
+ interest. Please refer to the sections [building software](../software/building_software.md) and
+ [module environments](../software/modules.md#module-environments) for detailed information.
!!! example "Job file for MPI application"
@@ -57,8 +56,8 @@ For MPI-parallel jobs one typically allocates one core per task that has to be s
### Multiple Programs Running Simultaneously in a Job
In this short example, our goal is to run four instances of a program concurrently in a **single**
-batch script. Of course, we could also start a batch script four times with `sbatch`, but this is not
-what we want to do here. However, you can also find an example about
+batch script. Of course, we could also start a batch script four times with `sbatch`, but this is
+not what we want to do here. However, you can also find an example about
[how to run GPU programs simultaneously in a single job](#running-multiple-gpu-applications-simultaneously-in-a-batch-job)
below.
@@ -169,7 +168,8 @@ srun: error: Unable to allocate resources: Job violates accounting/QOS policy (j
### Running Multiple GPU Applications Simultaneously in a Batch Job
Our starting point is a (serial) program that needs a single GPU and four CPU cores to perform its
-task (e.g. TensorFlow). The following batch script shows how to run such a job on the cluster `Power9`.
+task (e.g. TensorFlow). The following batch script shows how to run such a job on the cluster
+`Power9`.
!!! example
@@ -194,8 +194,8 @@ srun --ntasks=1 --cpus-per-task=4 [...] some-gpu-application
```
Now, our goal is to run four instances of this program concurrently in a single batch script. Of
-course we could also start the above script multiple times with `sbatch`, but this is not what we want
-to do here.
+course we could also start the above script multiple times with `sbatch`, but this is not what we
+want to do here.
#### Solution
@@ -420,25 +420,3 @@ In the following we provide two examples for scripts that submit chain jobs.
Job 3/3: jobfile_c.sh
Dependency: after job 2963709
```
-
-## Array-Job with Afterok-Dependency and Datamover Usage
-
-In this example scenario, imagine you need to move data, before starting the main job.
-For this you may use a data transfer job and tell Slurm to start the main job immediately after
-data transfer job successfully finish.
-
-First you have to start your data transfer job, which for example transfers your input data from one
-workspace to another.
-
-```console
-marie@login$ export DATAMOVER_JOB=$(dtcp /scratch/ws/1/marie-source/input.txt /beegfs/ws/1/marie-target/. | awk '{print $4}')
-```
-
-Now you can refer to the job id of the Datamover jobs from your work load jobs.
-
-```console
-marie@login$ srun --dependency afterok:${DATAMOVER_JOB} ls /beegfs/ws/1/marie-target
-srun: job 23872871 queued and waiting for resources
-srun: job 23872871 has been allocated resources
-input.txt
-```
diff --git a/doc.zih.tu-dresden.de/mkdocs.yml b/doc.zih.tu-dresden.de/mkdocs.yml
index 9e7e34bbd99eaf528b0a2f5f574802d2d916f6d5..ce8a4f792a3fef68868f8077592d1c567c0bb48e 100644
--- a/doc.zih.tu-dresden.de/mkdocs.yml
+++ b/doc.zih.tu-dresden.de/mkdocs.yml
@@ -36,7 +36,6 @@ nav:
- Permanent Filesystems: data_lifecycle/permanent.md
- Working Filesystems: data_lifecycle/working.md
- Lustre: data_lifecycle/lustre.md
- - BeeGFS: data_lifecycle/beegfs.md
- Intermediate Archive: data_lifecycle/intermediate_archive.md
- Workspaces: data_lifecycle/workspaces.md
- Long-Term Preservation of Research Data: data_lifecycle/longterm_preservation.md
@@ -124,6 +123,7 @@ nav:
- Load Leveler: archive/load_leveler.md
- Jobs without InfiniBand: archive/no_ib_jobs.md
- Migration towards Phase 2: archive/phase2_migration.md
+ - Migration towards Barnard: archive/migration_to_barnard.md
- Platform LSF: archive/platform_lsf.md
- Jupyter Installation: archive/install_jupyter.md
- Profile Jobs with Slurm: archive/slurm_profiling.md
@@ -131,6 +131,7 @@ nav:
- Overview: archive/systems_switched_off.md
- System Taurus: archive/system_taurus.md
- Filesystems: archive/filesystems.md
+ - BeeGFS: archive/beegfs.md
- Migration From Deimos to Atlas: archive/migrate_to_atlas.md
- System Altix: archive/system_altix.md
- System Atlas: archive/system_atlas.md