diff --git a/Dockerfile b/Dockerfile
index 731e831c9b2fc1ff1068ae2b2a80c04bbf0039c7..57490c2509a22302ba13ed4bd05d32f0d7b0fb51 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -4,18 +4,16 @@ FROM python:3.8-bullseye
# Base #
########
-COPY ./ /src/
-
-RUN pip install -r /src/doc.zih.tu-dresden.de/requirements.txt
+RUN pip install mkdocs>=1.1.2 mkdocs-material>=7.1.0
##########
# Linter #
##########
-RUN apt update && apt install -y nodejs npm aspell
+RUN apt update && apt install -y nodejs npm aspell git
RUN npm install -g markdownlint-cli markdown-link-check
-WORKDIR /src/doc.zih.tu-dresden.de
+WORKDIR /docs
CMD ["mkdocs", "build", "--verbose", "--strict"]
diff --git a/doc.zih.tu-dresden.de/README.md b/doc.zih.tu-dresden.de/README.md
index 1829a5bc54c26ce37f61f27410e45e8901488183..57cb9a23f94bc38d004049816873cd3105d618d6 100644
--- a/doc.zih.tu-dresden.de/README.md
+++ b/doc.zih.tu-dresden.de/README.md
@@ -40,13 +40,7 @@ Now, create a local clone of your fork
#### Install Dependencies
-**TODO:** Description
-
-```Shell Session
-~ cd hpc-compendium/doc.zih.tu-dresden.de
-~ pip install -r requirements.txt
-```
-
+See [Installation with Docker](#preview-using-mkdocs-with-dockerfile).
**TODO:** virtual environment
**TODO:** What we need for markdownlinter and checks?
@@ -393,278 +387,3 @@ BigDataFrameworksApacheSparkApacheFlinkApacheHadoop.md is not included in nav
pika.md is not included in nav
specific_software.md is not included in nav
```
-
-### Pre-commit Git Hook
-
-You can automatically run checks whenever you try to commit a change. In this case, failing checks
-prevent commits (unless you use option `--no-verify`). This can be accomplished by adding a
-pre-commit hook to your local clone of the repository. The following code snippet shows how to do
-that:
-
-```bash
-cp doc.zih.tu-dresden.de/util/pre-commit .git/hooks/
-```
-
-!!! note
- The pre-commit hook only works, if you can use docker without using `sudo`. If this is not
- already the case, use the command `adduser $USER docker` to enable docker commands without
- `sudo` for the current user. Restart the docker daemons afterwards.
-
-## Content Rules
-
-**Remark:** Avoid using tabs both in markdown files and in `mkdocs.yaml`. Type spaces instead.
-
-### New Page and Pages Structure
-
-The pages structure is defined in the configuration file [mkdocs.yaml](mkdocs.yml).
-
-```Shell Session
-docs/
- - Home: index.md
- - Application for HPC Login: application.md
- - Request for Resources: req_resources.md
- - Access to the Cluster: access.md
- - Available Software and Usage:
- - Overview: software/overview.md
- ...
-```
-
-To add a new page to the documentation follow these two steps:
-
-1. Create a new markdown file under `docs/subdir/file_name.md` and put the documentation inside. The
- sub directory and file name should follow the pattern `fancy_title_and_more.md`.
-1. Add `subdir/file_name.md` to the configuration file `mkdocs.yml` by updating the navigation
- section.
-
-Make sure that the new page **is not floating**, i.e., it can be reached directly from the documentation
-structure.
-
-### Markdown
-
-1. Please keep things simple, i.e., avoid using fancy markdown dialects.
- * [Cheat Sheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)
- * [Style Guide](https://github.com/google/styleguide/blob/gh-pages/docguide/style.md)
-
-1. Do not add large binary files or high resolution images to the repository. See this valuable
- document for [image optimization](https://web.dev/fast/#optimize-your-images).
-
-1. [Admonitions](https://squidfunk.github.io/mkdocs-material/reference/admonitions/) may be
-actively used, especially for longer code examples, warnings, tips, important information that
-should be highlighted, etc. Code examples, longer than half screen height should collapsed
-(and indented):
-
-??? example
- ```Bash
- [...]
- # very long example here
- [...]
- ```
-
-### Writing Style
-
-**TODO** Guide [Issue #14](#14)
-
-* Capitalize headings, e.g. *Exclusive Reservation of Hardware*
-
-### Spelling and Technical Wording
-
-To provide a consistent and high quality documentation, and help users to find the right pages,
-there is a list of conventions w.r.t. spelling and technical wording.
-
-* Language settings: en_us
-* `I/O` not `IO`
-* `Slurm` not `SLURM`
-* `Filesystem` not `file system`
-* `ZIH system` and `ZIH systems` not `Taurus`, `HRSKII`, `our HPC systems`, etc.
-* `Workspace` not `work space`
-* avoid term `HPC-DA`
-* Partition names after the keyword *partition*: *partition `ml`* not *ML partition*, *ml
- partition*, *`ml` partition*, *"ml" partition*, etc.
-
-### Code Blocks and Command Prompts
-
-Showing commands and sample output is an important part of all technical documentation. To make
-things as clear for readers as possible and provide a consistent documentation, some rules have to
-be followed.
-
-1. Use ticks to mark code blocks and commands, not italic font.
-1. Specify language for code blocks ([see below](#code-blocks-and-syntax-highlighting)).
-1. All code blocks and commands should be runnable from a login node or a node within a specific
- partition (e.g., `ml`).
-1. It should be clear from the prompt, where the command is run (e.g. local machine, login node or
- specific partition).
-
-#### Prompts
-
-We follow this rules regarding prompts:
-
-| Host/Partition | Prompt |
-|------------------------|------------------|
-| Login nodes | `marie@login$` |
-| Arbitrary compute node | `marie@compute$` |
-| `haswell` partition | `marie@haswell$` |
-| `ml` partition | `marie@ml$` |
-| `alpha` partition | `marie@alpha$` |
-| `alpha` partition | `marie@alpha$` |
-| `romeo` partition | `marie@romeo$` |
-| `julia` partition | `marie@julia$` |
-| Localhost | `marie@local$` |
-
-*Remarks:*
-
-* **Always use a prompt**, even there is no output provided for the shown command.
-* All code blocks should use long parameter names (e.g. Slurm parameters), if available.
-* All code blocks which specify some general command templates, e.g. containing `<` and `>`
- (see [Placeholders](#mark-placeholders)), should use `bash` for the code block. Additionally,
- an example invocation, perhaps with output, should be given with the normal `console` code block.
- See also [Code Block description below](#code-blocks-and-syntax-highlighting).
-* Using some magic, the prompt as well as the output is identified and will not be copied!
-* Stick to the [generic user name](#data-privacy-and-generic-user-name) `marie`.
-
-#### Code Blocks and Syntax Highlighting
-
-This project makes use of the extension
-[pymdownx.highlight](https://squidfunk.github.io/mkdocs-material/reference/code-blocks/) for syntax
-highlighting. There is a complete list of supported
-[language short codes](https://pygments.org/docs/lexers/).
-
-For consistency, use the following short codes within this project:
-
-With the exception of command templates, use `console` for shell session and console:
-
-```` markdown
-```console
-marie@login$ ls
-foo
-bar
-```
-````
-
-Make sure that shell session and console code blocks are executable on the login nodes of HPC system.
-
-Command templates use [Placeholders](#mark-placeholders) to mark replaceable code parts. Command
-templates should give a general idea of invocation and thus, do not contain any output. Use a
-`bash` code block followed by an invocation example (with `console`):
-
-```` markdown
-```bash
-marie@local$ ssh -NL <local port>:<compute node>:<remote port> <zih login>@tauruslogin.hrsk.tu-dresden.de
-```
-
-```console
-marie@local$ ssh -NL 5901:172.24.146.46:5901 marie@tauruslogin.hrsk.tu-dresden.de
-```
-````
-
-Also use `bash` for shell scripts such as jobfiles:
-
-```` markdown
-```bash
-#!/bin/bash
-#SBATCH --nodes=1
-#SBATCH --time=01:00:00
-#SBATCH --output=slurm-%j.out
-
-module load foss
-
-srun a.out
-```
-````
-
-!!! important
-
- Use long parameter names where possible to ease understanding.
-
-`python` for Python source code:
-
-```` markdown
-```python
-from time import gmtime, strftime
-print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
-```
-````
-
-`pycon` for Python console:
-
-```` markdown
-```pycon
->>> from time import gmtime, strftime
->>> print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
-2021-08-03 07:20:33
-```
-````
-
-Line numbers can be added via
-
-```` markdown
-```bash linenums="1"
-#!/bin/bash
-
-#SBATCH -N 1
-#SBATCH -n 23
-#SBATCH -t 02:10:00
-
-srun a.out
-```
-````
-
-_Result_:
-
-
-
-Specific Lines can be highlighted by using
-
-```` markdown
-```bash hl_lines="2 3"
-#!/bin/bash
-
-#SBATCH -N 1
-#SBATCH -n 23
-#SBATCH -t 02:10:00
-
-srun a.out
-```
-````
-
-_Result_:
-
-
-
-### Data Privacy and Generic User Name
-
-Where possible, replace login, project name and other private data with clearly arbitrary placeholders.
-E.g., use the generic login `marie` and the corresponding project name `p_marie`.
-
-```console
-marie@login$ ls -l
-drwxr-xr-x 3 marie p_marie 4096 Jan 24 2020 code
-drwxr-xr-x 3 marie p_marie 4096 Feb 12 2020 data
--rw-rw---- 1 marie p_marie 4096 Jan 24 2020 readme.md
-```
-
-### Mark Omissions
-
-If showing only a snippet of a long output, omissions are marked with `[...]`.
-
-### Mark Placeholders
-
-Stick to the Unix rules on optional and required arguments, and selection of item sets:
-
-* `<required argument or value>`
-* `[optional argument or value]`
-* `{choice1|choice2|choice3}`
-
-## Graphics and Attachments
-
-All graphics and attachments are saved within `misc` directory of the respective sub directory in
-`docs`.
-
-The syntax to insert a graphic or attachment into a page is
-
-```Bash
-
-{: align="center"}
-```
-
-The attribute `align` is optional. By default, graphics are left aligned. **Note:** It is crucial to
-have `{: align="center"}` on a new line.
diff --git a/doc.zih.tu-dresden.de/docs/access/ssh_login.md b/doc.zih.tu-dresden.de/docs/access/ssh_login.md
index 5e67c5279f701405224078d7234517c642d3e726..59e304f3a337f0b30a804ba21e4d4396452cd4c8 100644
--- a/doc.zih.tu-dresden.de/docs/access/ssh_login.md
+++ b/doc.zih.tu-dresden.de/docs/access/ssh_login.md
@@ -13,7 +13,8 @@ For more information on our VPN and how to set it up, please visit the correspon
## Connecting from Linux
-Please use an up-to-date SSH client. The login nodes accept the following encryption algorithms:
+SSH establishes secure connections using authentication and encryption. Thus, please use an
+up-to-date SSH client. The login nodes accept the following encryption algorithms:
* `aes128-ctr`
* `aes192-ctr`
@@ -23,58 +24,111 @@ Please use an up-to-date SSH client. The login nodes accept the following encryp
* `chacha20-poly1305@openssh.com`
* `chacha20-poly1305@openssh.com`
-### SSH Session
+### Before Your First Connection
-If your workstation is within the campus network, you can connect to the HPC login nodes directly.
+We suggest to create an SSH key pair before you work with the ZIH systems. This ensures high
+connection security.
```console
-marie@local$ ssh <zih-login>@taurus.hrsk.tu-dresden.de
+marie@local$ mkdir -p ~/.ssh
+marie@local$ ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
+Generating public/private ed25519 key pair.
+Enter passphrase (empty for no passphrase):
+Enter same passphrase again:
+[...]
```
-If you connect for the fist time, the client will ask you to verify the host by its fingerprint:
+Type in a passphrase for the protection of your key. The passphrase should be **non-empty**.
+Copy the public key to the ZIH system (Replace placeholder `marie` with your ZIH login):
```console
-marie@local$: ssh <zih-login>@taurus.hrsk.tu-dresden.de
+marie@local$ ssh-copy-id -i ~/.ssh/id_ed25519.pub marie@taurus.hrsk.tu-dresden.de
The authenticity of host 'taurus.hrsk.tu-dresden.de (141.30.73.104)' can't be established.
RSA key fingerprint is SHA256:HjpVeymTpk0rqoc8Yvyc8d9KXQ/p2K0R8TJ27aFnIL8.
Are you sure you want to continue connecting (yes/no)?
```
Compare the shown fingerprint with the [documented fingerprints](key_fingerprints.md). Make sure
-they match. Than you can accept by typing `y` or `yes`.
+they match. Then you can accept by typing `yes`.
-### X11-Forwarding
+!!! info
+ If `ssh-copy-id` is not available, you need to do additional steps:
-If you plan to use an application with graphical user interface (GUI), you need to enable
-X11-forwarding for the connection. Add the option `-X` or `-XC` to your SSH command. The `-C` enables
-compression which usually improves usability in this case).
+ ```console
+ marie@local$ scp ~/.ssh/id_ed25519.pub marie@taurus.hrsk.tu-dresden.de:
+ The authenticity of host 'taurus.hrsk.tu-dresden.de (141.30.73.104)' can't be established.
+ RSA key fingerprint is SHA256:HjpVeymTpk0rqoc8Yvyc8d9KXQ/p2K0R8TJ27aFnIL8.
+ Are you sure you want to continue connecting (yes/no)?
+ ```
+
+ After that, you need to manually copy the key to the right place:
+
+ ```console
+ marie@local$ ssh marie@taurus.hrsk.tu-dresden.de
+ [...]
+ marie@login$ mkdir -p ~/.ssh
+ marie@login$ touch ~/.ssh/authorized_keys
+ marie@login$ cat id_ed25519.pub >> ~/.ssh/authorized_keys
+ ```
+
+#### Configuring Default Parameters for SSH
+
+After you have copied your key to the ZIH system, you should be able to connect using:
```console
-marie@local$ ssh -XC <zih-login>@taurus.hrsk.tu-dresden.de
+marie@local$ ssh marie@taurus.hrsk.tu-dresden.de
+[...]
+marie@login$ exit
```
-!!! info
+However, you can make this more comfortable if you prepare an SSH configuration on your local
+workstation. Navigate to the subdirectory `.ssh` in your home directory and open the file `config`
+(`~/.ssh/config`) in your favorite editor. If it does not exist, create it. Put the following lines
+in it (you can omit lines starting with `#`):
+
+```bash
+Host taurus
+ HostName taurus.hrsk.tu-dresden.de
+ #Put your ZIH-Login after keyword "User":
+ User marie
+ #Path to private key:
+ IdentityFile ~/.ssh/id_ed25519
+ #Don't try other keys if you have more:
+ IdentitiesOnly yes
+ #Enable X11 forwarding for graphical applications and compression. You don't need parameter -X and -C when invoking ssh then.
+ ForwardX11 yes
+ Compression yes
+```
- Also consider to use a [DCV session](desktop_cloud_visualization.md) for remote desktop
- visualization at ZIH systems.
+Afterwards, you can connect to the ZIH system using:
-### Password-Less SSH
+```console
+marie@local$ ssh taurus
+```
+
+### X11-Forwarding
-Of course, password-less SSH connecting is supported at ZIH. All public SSH keys for ZIH systems
-have to be generated following these rules:
+If you plan to use an application with graphical user interface (GUI), you need to enable
+X11-forwarding for the connection. If you use the SSH configuration described above, everything is
+already prepared and you can simply use:
- * The **ED25519** algorithm has to be used, e.g., `ssh-keygen -t ed25519`
- * A **non-empty** passphrase for the private key must be set.
+```console
+marie@local$ ssh taurus
+```
-The generated public key is usually saved at `~/.ssh/id_ed25519` at your local system. To allow for
-password-less SSH connection to ZIH systems, it has to be added to the file `.ssh/authorized_keys` within
-your home directory `/home/<zih-login>/` at ZIH systems.
+If you have omitted the last two lines in the default configuration above, you need to add the
+option `-X` or `-XC` to your SSH command. The `-C` enables compression which usually improves
+usability in this case:
```console
-marie@local$ ssh -i id-ed25519 <zih-login>@taurus.hrsk.tu-dresden.de
-Enter passphrase for key 'id-ed25519':
+marie@local$ ssh -XC taurus
```
+!!! info
+
+ Also consider to use a [DCV session](desktop_cloud_visualization.md) for remote desktop
+ visualization at ZIH systems.
+
## Connecting from Windows
We recommend one of the following applications:
diff --git a/doc.zih.tu-dresden.de/docs/contrib/content_rules.md b/doc.zih.tu-dresden.de/docs/contrib/content_rules.md
index f5492e7f35ff26e425bff9c7b246f7c0d4a29fb0..2be83c1f78668abb764586741a7de764b5baa112 100644
--- a/doc.zih.tu-dresden.de/docs/contrib/content_rules.md
+++ b/doc.zih.tu-dresden.de/docs/contrib/content_rules.md
@@ -51,6 +51,12 @@ should be highlighted, etc. Code examples, longer than half screen height should
## Writing Style
* Capitalize headings, e.g. *Exclusive Reservation of Hardware*
+* Give keywords in link texts, e.g. [Code Blocks](#code-blocks-and-syntax-highlighting) is more
+ descriptive than [this subsection](#code-blocks-and-syntax-highlighting)
+* Use active over passive voice
+ * Write with confidence. This confidence should be reflected in the documentation, so that
+ the readers trust and follow it.
+ * Example: `We recommend something` instead of `Something is recommended.`
## Spelling and Technical Wording
@@ -61,8 +67,17 @@ there is a list of conventions w.r.t. spelling and technical wording.
* `I/O` not `IO`
* `Slurm` not `SLURM`
* `Filesystem` not `file system`
-* `ZIH system` and `ZIH systems` not `Taurus` etc. if possible
+* `ZIH system` and `ZIH systems` not `Taurus`, `HRSKII`, `our HPC systems`, etc.
* `Workspace` not `work space`
+* avoid term `HPC-DA`
+* Partition names after the keyword *partition*: *partition `ml`* not *ML partition*, *ml
+ partition*, *`ml` partition*, *"ml" partition*, etc.
+
+### Long Options
+
+* Use long over short options, e.g. `srun --nodes=2 --ntasks-per-node=4 ...` is preferred over
+ `srun -N 2 -n 4 ...`
+* Use `module` over the short front-end `ml` in documentation and examples
## Code Blocks and Command Prompts
@@ -114,7 +129,7 @@ For consistency, use the following short codes within this project:
With the exception of command templates, use `console` for shell session and console:
-```` markdown
+````markdown
```console
marie@login$ ls
foo
@@ -128,7 +143,7 @@ Command templates use [Placeholders](#mark-placeholders) to mark replaceable cod
templates should give a general idea of invocation and thus, do not contain any output. Use a
`bash` code block followed by an invocation example (with `console`):
-```` markdown
+````markdown
```bash
marie@local$ ssh -NL <local port>:<compute node>:<remote port> <zih login>@tauruslogin.hrsk.tu-dresden.de
```
@@ -140,7 +155,7 @@ marie@local$ ssh -NL 5901:172.24.146.46:5901 marie@tauruslogin.hrsk.tu-dresden.d
Also use `bash` for shell scripts such as job files:
-```` markdown
+````markdown
```bash
#!/bin/bash
#SBATCH --nodes=1
@@ -159,7 +174,7 @@ srun a.out
`python` for Python source code:
-```` markdown
+````markdown
```python
from time import gmtime, strftime
print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
@@ -168,7 +183,7 @@ print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
`pycon` for Python console:
-```` markdown
+````markdown
```pycon
>>> from time import gmtime, strftime
>>> print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
@@ -178,7 +193,7 @@ print(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
Line numbers can be added via
-```` markdown
+````markdown
```bash linenums="1"
#!/bin/bash
@@ -190,6 +205,10 @@ srun a.out
```
````
+_Result_:
+
+
+
Specific Lines can be highlighted by using
```` markdown
@@ -204,6 +223,10 @@ srun a.out
```
````
+_Result_:
+
+
+
### Data Privacy and Generic User Name
Where possible, replace login, project name and other private data with clearly arbitrary placeholders.
diff --git a/doc.zih.tu-dresden.de/docs/contrib/contribute_container.md b/doc.zih.tu-dresden.de/docs/contrib/contribute_container.md
index c0d04ffcd04a2655b352c64c7442403e46df18d8..dd44fafa136d63ae80267226f70dc00563507ba3 100644
--- a/doc.zih.tu-dresden.de/docs/contrib/contribute_container.md
+++ b/doc.zih.tu-dresden.de/docs/contrib/contribute_container.md
@@ -55,7 +55,7 @@ The command(s) to start the dockerized web server is this:
docker run --name=hpc-compendium -p 8000:8000 --rm -w /docs --mount src="$(pwd)"/doc.zih.tu-dresden.de,target=/docs,type=bind hpc-compendium bash -c "mkdocs build && mkdocs serve -a 0.0.0.0:8000"
```
-You can view the documentation via [http://localhost:8000](http://localhost:8000) in your browser, now.
+You can view the documentation via `http://localhost:8000` in your browser, now.
!!! note
@@ -86,7 +86,25 @@ To avoid a lot of retyping, use the following in your shell:
alias wiki="docker run --name=hpc-compendium --rm -it -w /docs --mount src=$PWD/doc.zih.tu-dresden.de,target=/docs,type=bind hpc-compendium bash -c"
```
-You are now ready to use the different checks
+You are now ready to use the different checks, however we suggest to try the pre-commit hook.
+
+#### Pre-commit Git Hook
+
+We recommend to automatically run checks whenever you try to commit a change. In this case, failing
+checks prevent commits (unless you use option `--no-verify`). This can be accomplished by adding a
+pre-commit hook to your local clone of the repository. The following code snippet shows how to do
+that:
+
+```bash
+cp doc.zih.tu-dresden.de/util/pre-commit .git/hooks/
+```
+
+!!! note
+ The pre-commit hook only works, if you can use docker without using `sudo`. If this is not
+ already the case, use the command `adduser $USER docker` to enable docker commands without
+ `sudo` for the current user. Restart the docker daemons afterwards.
+
+Read on if you want to run a specific check.
#### Linter
diff --git a/doc.zih.tu-dresden.de/docs/contrib/howto_contribute.md b/doc.zih.tu-dresden.de/docs/contrib/howto_contribute.md
index 31105a5208932ff49ee86d939ed8faa744dad854..e0d91cccc3f534e0d7057b72f1d6479f8932b6aa 100644
--- a/doc.zih.tu-dresden.de/docs/contrib/howto_contribute.md
+++ b/doc.zih.tu-dresden.de/docs/contrib/howto_contribute.md
@@ -4,14 +4,21 @@
Ink is better than the best memory.
+In principle, there are three possible ways how to contribute to this documentation.
+
## Contribute via Issue
Users can contribute to the documentation via the
[GitLab issue tracking system](https://gitlab.hrz.tu-chemnitz.de/zih/hpcsupport/hpc-compendium/-/issues).
For that, open an issue to report typos and missing documentation or request for more precise
-wording etc. ZIH staff will get in touch with you to resolve the issue and improve the
+wording etc. ZIH staff will get in touch with you to resolve the issue and improve the
documentation.
+??? tip "Create an issue in GitLab"
+
+ 
+ {: align=center}
+
!!! warning "HPC support"
Non-documentation issues and requests need to be send as ticket to
@@ -20,8 +27,15 @@ documentation.
## Contribute via Web IDE
GitLab offers a rich and versatile web interface to work with repositories. To fix typos and edit
-source files, just select the file of interest and click the `Edit` button. A text and commit
-editor are invoked: Do your changes, add a meaningful commit message and commit the changes.
+source files, follow these steps:
+
+1. Navigate to the repository at
+[https://gitlab.hrz.tu-chemnitz.de/zih/hpcsupport/hpc-compendium](https://gitlab.hrz.tu-chemnitz.de/zih/hpcsupport/hpc-compendium)
+and log in.
+1. Select the right branch.
+1. Select the file of interest in `doc.zih.tu-dresden.de/docs/...` and click the `Edit` button.
+1. A text and commit editor are invoked: Do your changes, add a meaningful commit message and commit
+ the changes.
The more sophisticated integrated Web IDE is reached from the top level menu of the repository or
by selecting any source file.
@@ -29,12 +43,9 @@ by selecting any source file.
Other git services might have an equivalent web interface to interact with the repository. Please
refer to the corresponding documentation for further information.
-<!--This option of contributing is only available for users of-->
-<!--[gitlab.hrz.tu-chemnitz.de](https://gitlab.hrz.tu-chemnitz.de). Furthermore, -->
-
## Contribute Using Git Locally
For experienced Git users, we provide a Docker container that includes all checks of the CI engine
used in the back-end. Using them should ensure that merge requests will not be blocked
due to automatic checking.
-For details, see [Work Locally Using Containers](contribute_container.md).
+For details, refer to the page [Work Locally Using Containers](contribute_container.md).
diff --git a/doc.zih.tu-dresden.de/docs/contrib/misc/create_gitlab_issue.gif b/doc.zih.tu-dresden.de/docs/contrib/misc/create_gitlab_issue.gif
new file mode 100644
index 0000000000000000000000000000000000000000..cb4910897903283e43b21feadcdb2acf2f42c15e
Binary files /dev/null and b/doc.zih.tu-dresden.de/docs/contrib/misc/create_gitlab_issue.gif differ
diff --git a/doc.zih.tu-dresden.de/misc/highlight_lines.png b/doc.zih.tu-dresden.de/docs/contrib/misc/highlight_lines.png
similarity index 100%
rename from doc.zih.tu-dresden.de/misc/highlight_lines.png
rename to doc.zih.tu-dresden.de/docs/contrib/misc/highlight_lines.png
diff --git a/doc.zih.tu-dresden.de/misc/lines.png b/doc.zih.tu-dresden.de/docs/contrib/misc/lines.png
similarity index 100%
rename from doc.zih.tu-dresden.de/misc/lines.png
rename to doc.zih.tu-dresden.de/docs/contrib/misc/lines.png
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/overview.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/overview.md
index bdbaa5a1523ec2fc06150195e18764cf14b618ef..a0152832601f63ef68ecb2191414265896237434 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/overview.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/overview.md
@@ -18,11 +18,11 @@ In the following, a brief overview on relevant topics w.r.t. data life cycle man
The main concept of working with data on ZIH systems bases on [Workspaces](workspaces.md). Use it
properly:
- * use a `/home` directory for the limited amount of personal data, simple examples and the results
- of calculations. The home directory is not a working directory! However, `/home` filesystem is
- [backed up](#backup) using snapshots;
- * use `workspaces` as a place for working data (i.e. data sets); Recommendations of choosing the
- correct storage system for workspace presented below.
+* use a `/home` directory for the limited amount of personal data, simple examples and the results
+ of calculations. The home directory is not a working directory! However, `/home` filesystem is
+ [backed up](#backup) using snapshots;
+* use `workspaces` as a place for working data (i.e. data sets); Recommendations of choosing the
+ correct storage system for workspace presented below.
### Taxonomy of Filesystems
@@ -31,38 +31,28 @@ It is important to design your data workflow according to characteristics, like
storage to efficiently use the provided storage and filesystems.
The page [filesystems](file_systems.md) holds a comprehensive documentation on the different
filesystems.
-<!--In general, the mechanisms of
-so-called--> <!--[Workspaces](workspaces.md) are compulsory for all HPC users to store data for a
-defined duration ---> <!--depending on the requirements and the storage system this time span might
-range from days to a few--> <!--years.-->
-<!--- [HPC filesystems](file_systems.md)-->
-<!--- [Intermediate Archive](intermediate_archive.md)-->
-<!--- [Special data containers] **todo** Special data containers (was no valid link in old compendium)-->
-<!--- [Move data between filesystems](../data_transfer/data_mover.md)-->
-<!--- [Move data to/from ZIH's filesystems](../data_transfer/export_nodes.md)-->
-<!--- [Longterm Preservation for ResearchData](preservation_research_data.md)-->
!!! hint "Recommendations to choose of storage system"
* For data that seldom changes but consumes a lot of space, the
[warm_archive](file_systems.md#warm_archive) can be used.
(Note that this is mounted **read-only** on the compute nodes).
- * For a series of calculations that works on the same data please use a `scratch` based [workspace](workspaces.md).
+ * For a series of calculations that works on the same data please use a `scratch` based
+ [workspace](workspaces.md).
* **SSD**, in its turn, is the fastest available filesystem made only for large parallel
applications running with millions of small I/O (input, output operations).
* If the batch job needs a directory for temporary data then **SSD** is a good choice as well.
The data can be deleted afterwards.
Keep in mind that every workspace has a storage duration. Thus, be careful with the expire date
-otherwise it could vanish. The core data of your project should be [backed up](#backup) and
-[archived]**todo link** (for the most [important]**todo link** data).
+otherwise it could vanish. The core data of your project should be [backed up](#backup) and the most
+important data should be [archived](preservation_research_data.md).
### Backup
The backup is a crucial part of any project. Organize it at the beginning of the project. The
backup mechanism on ZIH systems covers **only** the `/home` and `/projects` filesystems. Backed up
-files can be restored directly by the users. Details can be found
-[here](file_systems.md#backup-and-snapshots-of-the-file-system).
+files can be restored directly by users, see [Snapshots](permanent.md#snapshots).
!!! warning
@@ -73,13 +63,13 @@ files can be restored directly by the users. Details can be found
Organizing of living data using the filesystem helps for consistency of the
project. We recommend following the rules for your work regarding:
- * Organizing the data: Never change the original data; Automatize the organizing the data; Clearly
- separate intermediate and final output in the filenames; Carry identifier and original name
- along in your analysis pipeline; Make outputs clearly identifiable; Document your analysis
- steps.
- * Naming Data: Keep short, but meaningful names; Keep standard file endings; File names
- don’t replace documentation and metadata; Use standards of your discipline; Make rules for your
- project, document and keep them (See the [README recommendations]**todo link** below)
+* Organizing the data: Never change the original data; Automatize the organizing the data; Clearly
+ separate intermediate and final output in the filenames; Carry identifier and original name
+ along in your analysis pipeline; Make outputs clearly identifiable; Document your analysis
+ steps.
+* Naming Data: Keep short, but meaningful names; Keep standard file endings; File names
+ don’t replace documentation and metadata; Use standards of your discipline; Make rules for your
+ project, document and keep them (See the [README recommendations](#readme-recommendation) below)
This is the example of an organization (hierarchical) for the folder structure. Use it as a visual
illustration of the above:
@@ -128,49 +118,10 @@ Don't forget about data hygiene: Classify your current data into critical (need
its life cycle (from creation, storage and use to sharing, archiving and destruction); Erase the data
you don’t need throughout its life cycle.
-<!--## Software Packages-->
-
-<!--As was written before the module concept is the basic concept for using software on ZIH systems.-->
-<!--Uniformity of the project has to be achieved by using the same set of software on different levels.-->
-<!--It could be done by using environments. There are two types of environments should be distinguished:-->
-<!--runtime environment (the project level, use scripts to load [modules]**todo link**), Python virtual-->
-<!--environment. The concept of the environment will give an opportunity to use the same version of the-->
-<!--software on every level of the project for every project member.-->
-
-<!--### Private Individual and Project Modules Files-->
-
-<!--[Private individual and project module files]**todo link** will be discussed in [chapter 7]**todo-->
-<!--link**. Project modules list is a powerful instrument for effective teamwork.-->
-
-<!--### Python Virtual Environment-->
-
-<!--If you are working with the Python then it is crucial to use the virtual environment on ZIH systems. The-->
-<!--main purpose of Python virtual environments (don't mess with the software environment for modules)-->
-<!--is to create an isolated environment for Python projects (self-contained directory tree that-->
-<!--contains a Python installation for a particular version of Python, plus a number of additional-->
-<!--packages).-->
-
-<!--**Vitualenv (venv)** is a standard Python tool to create isolated Python environments. We-->
-<!--recommend using venv to work with Tensorflow and Pytorch on ZIH systems. It has been integrated into the-->
-<!--standard library under the [venv module]**todo link**. **Conda** is the second way to use a virtual-->
-<!--environment on the ZIH systems. Conda is an open-source package management system and environment-->
-<!--management system from the Anaconda.-->
-
-<!--[Detailed information]**todo link** about using the virtual environment.-->
-
-<!--## Application Software Availability-->
-
-<!--Software created for the purpose of the project should be available for all members of the group.-->
-<!--The instruction of how to use the software: installation of packages, compilation etc should be-->
-<!--documented and gives the opportunity to comfort efficient and safe work.-->
-
-## Access rights
+## Access Rights
The concept of **permissions** and **ownership** is crucial in Linux. See the
-[HPC-introduction]**todo link** slides for the understanding of the main concept. Standard Linux
-changing permission command (i.e `chmod`) valid for ZIH systems as well. The **group** access level
-contains members of your project group. Be careful with 'write' permission and never allow to change
-the original data.
-
-Useful links: [Data Management]**todo link**, [Filesystems]**todo link**,
-[Project Management]**todo link**, [Preservation research data[**todo link**
+[slides of HPC introduction](../misc/HPC-Introduction.pdf) for understanding of the main concept.
+Standard Linux changing permission command (i.e `chmod`) valid for ZIH systems as well. The
+**group** access level contains members of your project group. Be careful with 'write' permission
+and never allow to change the original data.
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/preservation_research_data.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/preservation_research_data.md
index 29399cf9f323337bacb34e76a5da8412d599119d..5c035e56d8a3fa647f9d847a08ed5be9ef903f93 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/preservation_research_data.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/preservation_research_data.md
@@ -47,42 +47,38 @@ stored in XML-format but free text is also possible. There are some meta-data st
Below are some examples:
- possible meta-data for a book would be:
- - Title
- - Author
- - Publisher
- - Publication
- - year
- - ISBN
+ - Title
+ - Author
+ - Publisher
+ - Publication
+ - year
+ - ISBN
- possible meta-data for an electronically saved image would be:
- - resolution of the image
- - information about the colour depth of the picture
- - file format (jpg or tiff or ...)
- - file size how was this image created (digital camera, scanner, ...)
- - description of what the image shows
- - creation date of the picture
- - name of the person who made the picture
+ - resolution of the image
+ - information about the colour depth of the picture
+ - file format (jpg or tiff or ...)
+ - file size how was this image created (digital camera, scanner, ...)
+ - description of what the image shows
+ - creation date of the picture
+ - name of the person who made the picture
- meta-data for the result of a calculation/simulation could be:
- - file format
- - file size
- - input data
- - which software in which version was used to calculate the result/to do the simulation
- - configuration of the software
- - date of the calculation/simulation (start/end or start/duration)
- - computer on which the calculation/simulation was done
- - name of the person who submitted the calculation/simulation
- - description of what was calculated/simulated
+ - file format
+ - file size
+ - input data
+ - which software in which version was used to calculate the result/to do the simulation
+ - configuration of the software
+ - date of the calculation/simulation (start/end or start/duration)
+ - computer on which the calculation/simulation was done
+ - name of the person who submitted the calculation/simulation
+ - description of what was calculated/simulated
## Where can I get more information about management of research data?
-Got to
-
-- <http://www.forschungsdaten.org/> (german version) or <http://www.forschungsdaten.org/en/>
-- (english version)
-
-to find more information about managing research data.
+Go to [http://www.forschungsdaten.org/en/](http://www.forschungsdaten.org/en/) to find more
+information about managing research data.
## I want to store my research data at ZIH. How can I do that?
Longterm preservation of research data is under construction at ZIH and in a testing phase.
Nevertheless you can already use the archiving service. If you would like to become a test
-user, please write an E-Mail to Dr. Klaus Köhler (klaus.koehler \[at\] tu-dresden.de).
+user, please write an E-Mail to [Dr. Klaus Köhler](mailto:klaus.koehler@tu-dresden.de).
diff --git a/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md b/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md
index f5e217de6b34e861004b54de3fb4d6cb5004a2ce..d970f38b9abb883fca7864a7af01a6f7037623a8 100644
--- a/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md
+++ b/doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md
@@ -4,13 +4,13 @@ Storage systems differ in terms of capacity, streaming bandwidth, IOPS rate, etc
efficiency don't allow to have it all in one. That is why fast parallel filesystems at ZIH have
restrictions with regards to **age of files** and [quota](quotas.md). The mechanism of workspaces
enables users to better manage their HPC data.
-<!--Workspaces are primarily login-related.-->
-The concept of "workspaces" is common and used at a large number of HPC centers.
+
+The concept of workspaces is common and used at a large number of HPC centers.
!!! note
- A workspace is a directory, with an associated expiration date, created on behalf of a user in a
- certain storage system.
+ A **workspace** is a directory, with an associated expiration date, created on behalf of a user
+ in a certain filesystem.
Once the workspace has reached its expiration date, it gets moved to a hidden directory and enters a
grace period. Once the grace period ends, the workspace is deleted permanently. The maximum lifetime
@@ -30,8 +30,8 @@ times.
To list all available filesystems for using workspaces use:
-```bash
-zih$ ws_find -l
+```console
+marie@login$ ws_find -l
Available filesystems:
scratch
warm_archive
@@ -43,8 +43,8 @@ beegfs_global0
To list all workspaces you currently own, use:
-```bash
-zih$ ws_list
+```console
+marie@login$ ws_list
id: test-workspace
workspace directory : /scratch/ws/0/marie-test-workspace
remaining time : 89 days 23 hours
@@ -59,8 +59,8 @@ id: test-workspace
To create a workspace in one of the listed filesystems use `ws_allocate`. It is necessary to specify
a unique name and the duration of the workspace.
-```bash
-ws_allocate: [options] workspace_name duration
+```console
+marie@login$ ws_allocate: [options] workspace_name duration
Options:
-h [ --help] produce help message
@@ -74,13 +74,12 @@ Options:
-u [ --username ] arg username
-g [ --group ] group workspace
-c [ --comment ] arg comment
-
```
!!! example
- ```bash
- zih$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90
+ ```console
+ marie@login$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90
Info: creating workspace.
/scratch/ws/marie-test-workspace
remaining extensions : 10
@@ -95,22 +94,23 @@ days with an email reminder for 7 days before the expiration.
Setting the reminder to `7` means you will get a reminder email on every day starting `7` prior
to expiration date.
-### Extention of a Workspace
+### Extension of a Workspace
The lifetime of a workspace is finite. Different filesystems (storage systems) have different
maximum durations. A workspace can be extended multiple times, depending on the filesystem.
-| Storage system (use with parameter -F ) | Duration, days | Extensions | Remarks |
-|:------------------------------------------:|:----------:|:-------:|:---------------------------------------------------------------------------------------:|
-| `ssd` | 30 | 10 | High-IOPS filesystem (`/lustre/ssd`) on SSDs. |
-| `beegfs` | 30 | 2 | High-IOPS filesystem (`/lustre/ssd`) onNVMes. |
-| `scratch` | 100 | 2 | Scratch filesystem (/scratch) with high streaming bandwidth, based on spinning disks |
-| `warm_archive` | 365 | 2 | Capacity filesystem based on spinning disks |
+| Filesystem (use with parameter `-F`) | Duration, days | Extensions | Remarks |
+|:------------------------------------:|:----------:|:-------:|:-----------------------------------:|
+| `ssd` | 30 | 2 | High-IOPS filesystem (`/lustre/ssd`, symbolic link: `/ssd`) on SSDs. |
+| `beegfs_global0` (deprecated) | 30 | 2 | High-IOPS filesystem (`/beegfs/global0`) on NVMes. |
+| `beegfs` | 30 | 2 | High-IOPS filesystem (`/beegfs`) on NVMes. |
+| `scratch` | 100 | 10 | Scratch filesystem (`/lustre/ssd`, symbolic link: `/scratch`) with high streaming bandwidth, based on spinning disks |
+| `warm_archive` | 365 | 2 | Capacity filesystem based on spinning disks |
-To extend your workspace use the following command:
+To extent your workspace use the following command:
-```
-zih$ ws_extend -F scratch test-workspace 100 #extend the workspace for 100 days
+```console
+marie@login$ ws_extend -F scratch test-workspace 100
Info: extending workspace.
/scratch/ws/marie-test-workspace
remaining extensions : 1
@@ -122,39 +122,46 @@ remaining time in days: 100
With the `ws_extend` command, a new duration for the workspace is set. The new duration is not
added!
-This means when you extend a workspace that expires in 90 days with the `ws_extend -F scratch
-my-workspace 40`, it will now expire in 40 days **not** 130 days.
+This means when you extend a workspace that expires in 90 days with the command
+
+```console
+marie@login$ ws_extend -F scratch my-workspace 40
+```
+
+it will now expire in 40 days **not** 130 days.
### Deletion of a Workspace
To delete a workspace use the `ws_release` command. It is mandatory to specify the name of the
workspace and the filesystem in which it is located:
-`ws_release -F <filesystem> <workspace name>`
+```console
+marie@login$ ws_release -F <filesystem> <workspace name>
+```
### Restoring Expired Workspaces
At expiration time your workspace will be moved to a special, hidden directory. For a month (in
warm_archive: 2 months), you can still restore your data into an existing workspace.
-!!!Warning
+!!! warning
When you release a workspace **by hand**, it will not receive a grace period and be
**permanently deleted** the **next day**. The advantage of this design is that you can create
and release workspaces inside jobs and not swamp the filesystem with data no one needs anymore
in the hidden directories (when workspaces are in the grace period).
-Use:
+Use
-```
-ws_restore -l -F scratch
+```console
+marie@login$ ws_restore -l -F scratch
```
to get a list of your expired workspaces, and then restore them like that into an existing, active
workspace 'new_ws':
-```
-ws_restore -F scratch marie-test-workspace-1234567 new_ws
+```console
+marie@login$ ws_restore -F scratch marie-test-workspace-1234567 new_ws
```
The expired workspace has to be specified by its full name as listed by `ws_restore -l`, including
@@ -174,8 +181,8 @@ workspaces within in the directory `DIR`. Calling this command will do the follo
- The directory `DIR` will be created if necessary.
- Links to all personal workspaces will be managed:
- - Create links to all available workspaces if not already present.
- - Remove links to released workspaces.
+ - Create links to all available workspaces if not already present.
+ - Remove links to released workspaces.
**Remark**: An automatic update of the workspace links can be invoked by putting the command
`ws_register DIR` in your personal `shell` configuration file (e.g., `.bashrc`).
@@ -198,8 +205,9 @@ A batch job needs a directory for temporary data. This can be deleted afterwards
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24
- module load modenv/classic
- module load gaussian
+ module purge
+ module load modenv/hiera
+ module load Gaussian
COMPUTE_DIR=gaussian_$SLURM_JOB_ID
export GAUSS_SCRDIR=$(ws_allocate -F ssd $COMPUTE_DIR 7)
@@ -218,8 +226,8 @@ Likewise, other jobs can use temporary workspaces.
For a series of jobs or calculations that work on the same data, you should allocate a workspace
once, e.g., in `scratch` for 100 days:
-```
-zih$ ws_allocate -F scratch my_scratchdata 100
+```console
+marie@login$ ws_allocate -F scratch my_scratchdata 100
Info: creating workspace.
/scratch/ws/marie-my_scratchdata
remaining extensions : 2
@@ -234,8 +242,8 @@ chmod g+wrx /scratch/ws/marie-my_scratchdata
And verify it with:
-```
-zih $ ls -la /scratch/ws/marie-my_scratchdata
+```console
+marie@login$ ls -la /scratch/ws/marie-my_scratchdata
total 8
drwxrwx--- 2 marie hpcsupport 4096 Jul 10 09:03 .
drwxr-xr-x 5 operator adm 4096 Jul 10 09:01 ..
@@ -247,8 +255,8 @@ For data that seldom changes but consumes a lot of space, the warm archive can b
this is mounted read-only on the compute nodes, so you cannot use it as a work directory for your
jobs!
-```
-zih$ ws_allocate -F warm_archive my_inputdata 365
+```console
+marie@login$ ws_allocate -F warm_archive my_inputdata 365
/warm_archive/ws/marie-my_inputdata
remaining extensions : 2
remaining time in days: 365
@@ -259,10 +267,10 @@ remaining time in days: 365
The warm archive is not built for billions of files. There
is a quota for 100.000 files per group. Please archive data.
-To see your active quota use:
+To see your active quota use
-```
-qinfo quota /warm_archive/ws/
+```console
+marie@login$ qinfo quota /warm_archive/ws/
```
Note that the workspaces reside under the mountpoint `/warm_archive/ws/` and not `/warm_archive`
diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md
index d7c3530fad85643c4f814a02c6e3250df427af38..a5bb1980e342b8f1c19ecb6b610a5d481cd98268 100644
--- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md
+++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md
@@ -1,9 +1,9 @@
# Batch System Slurm
-When log in to ZIH systems, you are placed on a login node. There you can manage your
+When logging in to ZIH systems, you are placed on a login node. There, you can manage your
[data life cycle](../data_lifecycle/overview.md),
[setup experiments](../data_lifecycle/experiments.md), and
-edit and prepare jobs. The login nodes are not suited for computational work! From the login nodes,
+edit and prepare jobs. The login nodes are not suited for computational work! From the login nodes,
you can interact with the batch system, e.g., submit and monitor your jobs.
??? note "Batch System"
@@ -32,7 +32,7 @@ ZIH uses the batch system Slurm for resource management and job scheduling.
Just specify the resources you need in terms
of cores, memory, and time and your Slurm will place your job on the system.
-This pages provides a brief overview on
+This page provides a brief overview on
* [Slurm options](#options) to specify resource requirements,
* how to submit [interactive](#interactive-jobs) and [batch jobs](#batch-jobs),
@@ -60,39 +60,39 @@ There are three basic Slurm commands for job submission and execution:
Using `srun` directly on the shell will be blocking and launch an
[interactive job](#interactive-jobs). Apart from short test runs, it is recommended to submit your
jobs to Slurm for later execution by using [batch jobs](#batch-jobs). For that, you can conveniently
-put the parameters directly in a [job file](#job-files) which you can submit using `sbatch [options]
-<job file>`.
+put the parameters directly in a [job file](#job-files), which you can submit using `sbatch
+[options] <job file>`.
-During runtime, the environment variable `SLURM_JOB_ID` will be set to the id of your job. The job
+At runtime, the environment variable `SLURM_JOB_ID` is set to the id of your job. The job
id is unique. The id allows you to [manage and control](#manage-and-control-jobs) your jobs.
## Options
-The following table holds the most important options for `srun/sbatch/salloc` to specify resource
+The following table contains the most important options for `srun/sbatch/salloc` to specify resource
requirements and control communication.
??? tip "Options Table"
| Slurm Option | Description |
|:---------------------------|:------------|
- | `-n, --ntasks=<N>` | number of (MPI) tasks (default: 1) |
- | `-N, --nodes=<N>` | number of nodes; there will be `--ntasks-per-node` processes started on each node |
- | `--ntasks-per-node=<N>` | number of tasks per allocated node to start (default: 1) |
- | `-c, --cpus-per-task=<N>` | number of CPUs per task; needed for multithreaded (e.g. OpenMP) jobs; typically `N` should be equal to `OMP_NUM_THREADS` |
- | `-p, --partition=<name>` | type of nodes where you want to execute your job (refer to [partitions](partitions_and_limits.md)) |
- | `--mem-per-cpu=<size>` | memory need per allocated CPU in MB |
- | `-t, --time=<HH:MM:SS>` | maximum runtime of the job |
- | `--mail-user=<your email>` | get updates about the status of the jobs |
- | `--mail-type=ALL` | for what type of events you want to get a mail; valid options: `ALL`, `BEGIN`, `END`, `FAIL`, `REQUEUE` |
- | `-J, --job-name=<name>` | name of the job shown in the queue and in mails (cut after 24 chars) |
- | `--no-requeue` | disable requeueing of the job in case of node failure (default: enabled) |
- | `--exclusive` | exclusive usage of compute nodes; you will be charged for all CPUs/cores on the node |
- | `-A, --account=<project>` | charge resources used by this job to the specified project |
- | `-o, --output=<filename>` | file to save all normal output (stdout) (default: `slurm-%j.out`) |
- | `-e, --error=<filename>` | file to save all error output (stderr) (default: `slurm-%j.out`) |
- | `-a, --array=<arg>` | submit an array job ([examples](slurm_examples.md#array-jobs)) |
- | `-w <node1>,<node2>,...` | restrict job to run on specific nodes only |
- | `-x <node1>,<node2>,...` | exclude specific nodes from job |
+ | `-n, --ntasks=<N>` | Number of (MPI) tasks (default: 1) |
+ | `-N, --nodes=<N>` | Number of nodes; there will be `--ntasks-per-node` processes started on each node |
+ | `--ntasks-per-node=<N>` | Number of tasks per allocated node to start (default: 1) |
+ | `-c, --cpus-per-task=<N>` | Number of CPUs per task; needed for multithreaded (e.g. OpenMP) jobs; typically `N` should be equal to `OMP_NUM_THREADS` |
+ | `-p, --partition=<name>` | Type of nodes where you want to execute your job (refer to [partitions](partitions_and_limits.md)) |
+ | `--mem-per-cpu=<size>` | Memory need per allocated CPU in MB |
+ | `-t, --time=<HH:MM:SS>` | Maximum runtime of the job |
+ | `--mail-user=<your email>` | Get updates about the status of the jobs |
+ | `--mail-type=ALL` | For what type of events you want to get a mail; valid options: `ALL`, `BEGIN`, `END`, `FAIL`, `REQUEUE` |
+ | `-J, --job-name=<name>` | Name of the job shown in the queue and in mails (cut after 24 chars) |
+ | `--no-requeue` | Disable requeueing of the job in case of node failure (default: enabled) |
+ | `--exclusive` | Exclusive usage of compute nodes; you will be charged for all CPUs/cores on the node |
+ | `-A, --account=<project>` | Charge resources used by this job to the specified project |
+ | `-o, --output=<filename>` | File to save all normal output (stdout) (default: `slurm-%j.out`) |
+ | `-e, --error=<filename>` | File to save all error output (stderr) (default: `slurm-%j.out`) |
+ | `-a, --array=<arg>` | Submit an array job ([examples](slurm_examples.md#array-jobs)) |
+ | `-w <node1>,<node2>,...` | Restrict job to run on specific nodes only |
+ | `-x <node1>,<node2>,...` | Exclude specific nodes from job |
!!! note "Output and Error Files"
@@ -109,19 +109,19 @@ requirements and control communication.
### Host List
If you want to place your job onto specific nodes, there are two options for doing this. Either use
-`-p, --partion=<name>` to specify a host group aka. [partition](partitions_and_limits.md) that fits
-your needs. Or, use `-w, --nodelist=<host1,host2,..>`) with a list of hosts that will work for you.
+`-p, --partition=<name>` to specify a host group aka. [partition](partitions_and_limits.md) that fits
+your needs. Or, use `-w, --nodelist=<host1,host2,..>` with a list of hosts that will work for you.
## Interactive Jobs
Interactive activities like editing, compiling, preparing experiments etc. are normally limited to
-the login nodes. For longer interactive sessions you can allocate cores on the compute node with the
-command `salloc`. It takes the same options like `sbatch` to specify the required resources.
+the login nodes. For longer interactive sessions, you can allocate cores on the compute node with
+the command `salloc`. It takes the same options as `sbatch` to specify the required resources.
`salloc` returns a new shell on the node, where you submitted the job. You need to use the command
`srun` in front of the following commands to have these commands executed on the allocated
resources. If you allocate more than one task, please be aware that `srun` will run the command on
-each allocated task!
+each allocated task by default!
The syntax for submitting a job is
@@ -132,16 +132,23 @@ marie@login$ srun [options] <command>
An example of an interactive session looks like:
```console
-marie@login$ srun --pty -n 1 -c 4 --time=1:00:00 --mem-per-cpu=1700 bash
-marie@login$ srun: job 13598400 queued and waiting for resources
-marie@login$ srun: job 13598400 has been allocated resources
+marie@login$ srun --pty --ntasks=1 --cpus-per-task=4 --time=1:00:00 --mem-per-cpu=1700 bash -l
+srun: job 13598400 queued and waiting for resources
+srun: job 13598400 has been allocated resources
marie@compute$ # Now, you can start interactive work with e.g. 4 cores
```
+!!! note "Using `module` commands"
+
+ The [module commands](../software/modules.md) are made available by sourcing the files
+ `/etc/profile` and `~/.bashrc`. This is done automatically by passing the parameter `-l` to your
+ shell, as shown in the example above. If you missed adding `-l` at submitting the interactive
+ session, no worry, you can source this files also later on manually.
+
!!! note "Partition `interactive`"
A dedicated partition `interactive` is reserved for short jobs (< 8h) with not more than one job
- per user. Please check the availability of nodes there with `sinfo -p interactive`.
+ per user. Please check the availability of nodes there with `sinfo --partition=interactive`.
### Interactive X11/GUI Jobs
@@ -176,10 +183,10 @@ Batch jobs are encapsulated within [job files](#job-files) and submitted to the
environment settings and the commands for executing the application. Using batch jobs and job files
has multiple advantages:
-* You can reproduce your experiments and work, because it's all steps are saved in a file.
+* You can reproduce your experiments and work, because all steps are saved in a file.
* You can easily share your settings and experimental setup with colleagues.
-* Submit your job file to the scheduling system for later execution. In the meanwhile, you can grab
- a coffee and proceed with other work (,e.g., start writing a paper).
+* You can submit your job file to the scheduling system for later execution. In the meanwhile, you can
+ grab a coffee and proceed with other work (e.g., start writing a paper).
!!! hint "The syntax for submitting a job file to Slurm is"
@@ -208,7 +215,7 @@ srun ./application [options] # Execute parallel application with srun
```
The following two examples show the basic resource specifications for a pure OpenMP application and
-a pure MPI application, respectively. Within the section [Job Examples](slurm_examples.md) we
+a pure MPI application, respectively. Within the section [Job Examples](slurm_examples.md), we
provide a comprehensive collection of job examples.
??? example "Job file OpenMP"
@@ -230,7 +237,7 @@ provide a comprehensive collection of job examples.
```
* Submisson: `marie@login$ sbatch batch_script.sh`
- * Run with fewer CPUs: `marie@login$ sbatch -c 14 batch_script.sh`
+ * Run with fewer CPUs: `marie@login$ sbatch --cpus-per-task=14 batch_script.sh`
??? example "Job file MPI"
@@ -248,7 +255,7 @@ provide a comprehensive collection of job examples.
```
* Submisson: `marie@login$ sbatch batch_script.sh`
- * Run with fewer MPI tasks: `marie@login$ sbatch --ntasks 14 batch_script.sh`
+ * Run with fewer MPI tasks: `marie@login$ sbatch --ntasks=14 batch_script.sh`
## Manage and Control Jobs
@@ -289,14 +296,14 @@ marie@login$ whypending <jobid>
### Editing Jobs
Jobs that have not yet started can be altered. Using `scontrol update timelimit=4:00:00
-jobid=<jobid>` it is for example possible to modify the maximum runtime. `scontrol` understands many
-different options, please take a look at the [man page](https://slurm.schedmd.com/scontrol.html) for
-more details.
+jobid=<jobid>`, it is for example possible to modify the maximum runtime. `scontrol` understands
+many different options, please take a look at the
+[scontrol documentation](https://slurm.schedmd.com/scontrol.html) for more details.
### Canceling Jobs
The command `scancel <jobid>` kills a single job and removes it from the queue. By using `scancel -u
-<username>` you can send a canceling signal to all of your jobs at once.
+<username>`, you can send a canceling signal to all of your jobs at once.
### Accounting
@@ -317,34 +324,34 @@ marie@login$ sacct
[...]
```
-We'd like to point your attention to the following options gain insight in your jobs.
+We'd like to point your attention to the following options to gain insight in your jobs.
??? example "Show specific job"
```console
- marie@login$ sacct -j <JOBID>
+ marie@login$ sacct --jobs=<JOBID>
```
??? example "Show all fields for a specific job"
```console
- marie@login$ sacct -j <JOBID> -o All
+ marie@login$ sacct --jobs=<JOBID> --format=All
```
??? example "Show specific fields"
```console
- marie@login$ sacct -j <JOBID> -o JobName,MaxRSS,MaxVMSize,CPUTime,ConsumedEnergy
+ marie@login$ sacct --jobs=<JOBID> --format=JobName,MaxRSS,MaxVMSize,CPUTime,ConsumedEnergy
```
-The manual page (`man sacct`) and the [online reference](https://slurm.schedmd.com/sacct.html)
+The manual page (`man sacct`) and the [sacct online reference](https://slurm.schedmd.com/sacct.html)
provide a comprehensive documentation regarding available fields and formats.
!!! hint "Time span"
By default, `sacct` only shows data of the last day. If you want to look further into the past
- without specifying an explicit job id, you need to provide a start date via the `-S` option.
- A certain end date is also possible via `-E`.
+ without specifying an explicit job id, you need to provide a start date via the option
+ `--starttime` (or short: `-S`). A certain end date is also possible via `--endtime` (or `-E`).
??? example "Show all jobs since the beginning of year 2021"
@@ -356,7 +363,7 @@ provide a comprehensive documentation regarding available fields and formats.
How to ask for a reservation is described in the section
[reservations](overview.md#exclusive-reservation-of-hardware).
-After we agreed with your requirements, we will send you an e-mail with your reservation name. Then
+After we agreed with your requirements, we will send you an e-mail with your reservation name. Then,
you could see more information about your reservation with the following command:
```console
@@ -387,7 +394,7 @@ constraints, please refer to the [Slurm documentation](https://slurm.schedmd.com
| Feature | Description |
|:--------|:-------------------------------------------------------------------------|
-| DA | subset of Haswell nodes with a high bandwidth to NVMe storage (island 6) |
+| DA | Subset of Haswell nodes with a high bandwidth to NVMe storage (island 6) |
#### Filesystem Features
@@ -396,10 +403,11 @@ these filesystems are tested every few minutes on each node and the Slurm featur
| Feature | Description |
|:-------------------|:---------------------------------------------------------------------|
-| `fs_lustre_scratch2` | `/scratch` mounted read-write (mount point is `/lustre/scratch2)` |
-| `fs_lustre_ssd` | `/lustre/ssd` mounted read-write |
+| `fs_lustre_scratch2` | `/scratch` mounted read-write (mount point is `/lustre/scratch2`) |
+| `fs_lustre_ssd` | `/ssd` mounted read-write (mount point is `/lustre/ssd`) |
| `fs_warm_archive_ws` | `/warm_archive/ws` mounted read-only |
| `fs_beegfs_global0` | `/beegfs/global0` mounted read-write |
+| `fs_beegfs` | `/beegfs` mounted read-write |
For certain projects, specific filesystems are provided. For those,
additional features are available, like `fs_beegfs_<projectname>`.
diff --git a/doc.zih.tu-dresden.de/docs/software/cfd.md b/doc.zih.tu-dresden.de/docs/software/cfd.md
index 492cb96d24f3761e2820fdba34eaa6b0a35db320..186d7b3a5a97a2daf06d8618c7c91dc91d7ab971 100644
--- a/doc.zih.tu-dresden.de/docs/software/cfd.md
+++ b/doc.zih.tu-dresden.de/docs/software/cfd.md
@@ -42,7 +42,7 @@ marie@login$ # source $FOAM_CSH
module load OpenFOAM
source $FOAM_BASH
cd /scratch/ws/1/marie-example-workspace # work directory using workspace
- srun pimpleFoam -parallel > "$OUTFILE"
+ srun pimpleFoam -parallel > "$OUTFILE"
```
## Ansys CFX
@@ -62,7 +62,7 @@ geometry and mesh generator cfx5pre, and the post-processor cfx5post.
module load ANSYS
cd /scratch/ws/1/marie-example-workspace # work directory using workspace
- cfx-parallel.sh -double -def StaticMixer.def
+ cfx-parallel.sh -double -def StaticMixer.def
```
## Ansys Fluent
diff --git a/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md b/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md
index a1974c5d288b275a33f621044209ec0e90ce201d..00ce0c5c4c3ddbd3654161bab69ee0a493cb4350 100644
--- a/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md
+++ b/doc.zih.tu-dresden.de/docs/software/data_analytics_with_python.md
@@ -85,15 +85,20 @@ For more examples of using pandarallel check out
### Dask
-[Dask](https://dask.org/) is a flexible and open-source library for parallel computing in Python.
-It replaces some Python data structures with parallel versions in order to provide advanced
-parallelism for analytics, enabling performance at scale for some of the popular tools. For
-instance: Dask arrays replace NumPy arrays, Dask dataframes replace Pandas dataframes.
+[Dask](https://dask.org/) is a flexible and open-source library
+for parallel computing in Python.
+It replaces some Python data structures with parallel versions
+in order to provide advanced
+parallelism for analytics, enabling performance at scale
+for some of the popular tools.
+For instance: Dask arrays replace NumPy arrays,
+Dask dataframes replace Pandas dataframes.
Furthermore, Dask-ML scales machine learning APIs like Scikit-Learn and XGBoost.
Dask is composed of two parts:
-- Dynamic task scheduling optimized for computation and interactive computational workloads.
+- Dynamic task scheduling optimized for computation and interactive
+ computational workloads.
- Big Data collections like parallel arrays, data frames, and lists that extend common interfaces
like NumPy, Pandas, or Python iterators to larger-than-memory or distributed environments.
These parallel collections run on top of dynamic task schedulers.
@@ -110,9 +115,10 @@ Dask supports several user interfaces:
- Delayed: Parallel function evaluation
- Futures: Real-time parallel function evaluation
-#### Dask Usage
+#### Dask Modules on ZIH Systems
-On ZIH systems, Dask is available as a module. Check available versions and load your preferred one:
+On ZIH systems, Dask is available as a module.
+Check available versions and load your preferred one:
```console
marie@compute$ module spider dask
@@ -129,34 +135,252 @@ marie@compute$ python -c "import dask; print(dask.__version__)"
2021.08.1
```
-The preferred and simplest way to run Dask on ZIH system is using
-[dask-jobqueue](https://jobqueue.dask.org/).
+The preferred way is to use Dask as a separate module as was described above.
+However, you can use it as part of the **Anaconda** module, e.g: `module load Anaconda3`.
-**TODO** create better example with jobqueue
+#### Scheduling by Dask
+
+Whenever you use functions on Dask collections (Dask Array, Dask Bag, etc.), Dask models these as
+single tasks forming larger task graphs in the background without you noticing.
+After Dask generates these task graphs,
+it needs to execute them on parallel hardware.
+This is the job of a task scheduler.
+Please use Distributed scheduler for your
+Dask computations on the cluster and avoid using a Single machine scheduler.
+
+##### Distributed Scheduler
+
+There are a variety of ways to set Distributed scheduler.
+However, `dask.distributed` scheduler will be used for many of them.
+To use the `dask.distributed` scheduler you must set up a Client:
+
+```python
+from dask.distributed import Client
+client = Client(...) # Connect to distributed cluster and override default
+df.x.sum().compute() # This now runs on the distributed system
+```
+
+The idea behind Dask is to scale Python and distribute computation among the workers (multiple
+machines, jobs).
+The preferred and simplest way to run Dask on ZIH systems
+today both for new or experienced users
+is to use **[dask-jobqueue](https://jobqueue.dask.org/)**.
+
+However, Dask-jobqueue is slightly oriented toward
+interactive analysis
+usage, and it might be better to use tools like
+**[Dask-mpi](https://docs.dask.org/en/latest/setup/hpc.html#using-mpi)**
+in some routine batch production workloads.
+
+##### Dask-mpi
+
+You can launch a Dask network using
+`mpirun` or `mpiexec` and the `dask-mpi` command line executable.
+This depends on the [mpi4py library](#mpi4py-mpi-for-python).
+For more detailed information, please check
+[the official documentation](https://docs.dask.org/en/latest/setup/hpc.html#using-mpi).
+
+##### Dask-jobqueue
+
+[Dask-jobqueue](https://jobqueue.dask.org/) can be used as the standard way
+to use dask for most users.
+It allows an easy deployment of Dask Distributed on HPC with Slurm
+or other job queuing systems.
+
+Dask-jobqueue is available as an extension
+for a Dask module (which can be loaded by: `module load dask`).
+
+The availability of the exact packages (such a Dask-jobqueue)
+in the module can be checked by the
+`module whatis <name_of_the_module>` command, e.g. `module whatis dask`.
+
+Moreover, it is possible to install and use `dask-jobqueue`
+in your local python environments.
+You can install Dask-jobqueue with `pip` or `conda`.
+
+###### Example of Using Dask-Jobqueue with SLURMCluster
+
+[Dask-jobqueue](https://jobqueue.dask.org/en/latest/howitworks.html#workers-vs-jobs)
+allows running jobs on the ZIH system
+inside the python code and scale computations over the jobs.
+[Dask-jobqueue](https://jobqueue.dask.org/en/latest/howitworks.html#workers-vs-jobs)
+creates a Dask Scheduler in the Python process
+where the cluster object is instantiated.
+Please check the example of a definition of the cluster object
+for the partition `alpha` (queue at the dask terms) on the ZIH system:
+
+```python
+from dask_jobqueue import SLURMCluster
+
+cluster = SLURMCluster(queue='alpha',
+ cores=8,
+ processes=2,
+ project='p_marie',
+ memory="8GB",
+ walltime="00:30:00")
+
+```
+
+These parameters above specify the characteristics of a
+single job or a single compute node,
+rather than the characteristics of your computation as a whole.
+It hasn’t actually launched any jobs yet.
+For the full computation, you will then ask for a number of
+jobs using the scale command, e.g : `cluster.scale(2)`.
+Thus, you have to specify a `SLURMCluster` by `dask_jobqueue`,
+scale it and use it for your computations. There is an example:
```python
-from dask.distributed import Client, progress
-client = Client(n_workers=4, threads_per_worker=1)
-client
+from distributed import Client
+from dask_jobqueue import SLURMCluster
+from dask import delayed
+
+cluster = SLURMCluster(queue='alpha',
+ cores=8,
+ processes=2,
+ project='p_marie',
+ memory="80GB",
+ walltime="00:30:00",
+ extra=['--resources gpu=1'])
+
+cluster.scale(2) #scale it to 2 workers!
+client = Client(cluster) #command will show you number of workers (python objects corresponds to jobs)
```
-### mpi4py - MPI for Python
+Please have a look at the `extra` parameter in the script above.
+This could be used to specify a
+special hardware availability that the scheduler
+is not aware of, for example, GPUs.
+Please don't forget to specify the name of your project.
+
+The Python code for setting up Slurm clusters
+and scaling clusters can be run by the `srun`
+(but remember that using `srun` directly on the shell
+blocks the shell and launches an
+interactive job) or batch jobs or
+[JupyterHub](../access/jupyterhub.md) with loaded Dask
+(by module or by Python virtual environment).
+
+!!! note
+ The job to run original code (de facto an interface) with a setup should be simple and light.
+ Please don't use a lot of resources for that.
+
+The following example shows using
+Dask by `dask-jobqueue` with `SLURMCluster` and `dask.array`
+for the Monte-Carlo estimation of Pi.
+
+??? example "Example of using SLURMCluster"
+
+ ```python
+ #use of dask-jobqueue for the estimation of Pi by Monte-Carlo method
+
+ import time
+ from time import time, sleep
+ from dask.distributed import Client
+ from dask_jobqueue import SLURMCluster
+ import subprocess as sp
+
+ import dask.array as da
+ import numpy as np
+
+ #setting up the dashboard
+
+ uid = int( sp.check_output('id -u', shell=True).decode('utf-8').replace('\n','') )
+ portdash = 10001 + uid
+
+ #create a Slurm cluster, please specify your project
+
+ cluster = SLURMCluster(queue='alpha', cores=2, project='p_marie', memory="8GB", walltime="00:30:00", extra=['--resources gpu=1'], scheduler_options={"dashboard_address": f":{portdash}"})
-Message Passing Interface (MPI) is a standardized and portable message-passing standard, designed to
-function on a wide variety of parallel computing architectures. The Message Passing Interface (MPI)
-is a library specification that allows HPC to pass information between its various nodes and
-clusters. MPI is designed to provide access to advanced parallel hardware for end-users, library
-writers and tool developers.
+ #submit the job to the scheduler with the number of nodes (here 2) requested:
-mpi4py (MPI for Python) provides bindings of the MPI standard for the Python programming
-language, allowing any Python program to exploit multiple processors.
+ cluster.scale(2)
-mpi4py is based on MPI-2 C++ bindings. It supports almost all MPI calls. This implementation is
-popular on Linux clusters and in the SciPy community. Operations are primarily methods of
-communicator objects. It supports communication of pickle-able Python objects. mpi4py provides
-optimized communication of NumPy arrays.
+ #wait for Slurm to allocate a resources
-mpi4py is included in the SciPy-bundle modules on the ZIH system.
+ sleep(120)
+
+ #check resources
+
+ client = Client(cluster)
+ client
+
+ #real calculations with a Monte Carlo method
+
+ def calc_pi_mc(size_in_bytes, chunksize_in_bytes=200e6):
+ """Calculate PI using a Monte Carlo estimate."""
+
+ size = int(size_in_bytes / 8)
+ chunksize = int(chunksize_in_bytes / 8)
+
+ xy = da.random.uniform(0, 1, size=(size / 2, 2), chunks=(chunksize / 2, 2))
+
+ in_circle = ((xy ** 2).sum(axis=-1) < 1)
+ pi = 4 * in_circle.mean()
+
+ return pi
+
+ def print_pi_stats(size, pi, time_delta, num_workers):
+ """Print pi, calculate offset from true value, and print some stats."""
+ print(f"{size / 1e9} GB\n"
+ f"\tMC pi: {pi : 13.11f}"
+ f"\tErr: {abs(pi - np.pi) : 10.3e}\n"
+ f"\tWorkers: {num_workers}"
+ f"\t\tTime: {time_delta : 7.3f}s")
+
+ #let's loop over different volumes of double-precision random numbers and estimate it
+
+ for size in (1e9 * n for n in (1, 10, 100)):
+
+ start = time()
+ pi = calc_pi_mc(size).compute()
+ elaps = time() - start
+
+ print_pi_stats(size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers))
+
+ #Scaling the Cluster to twice its size and re-run the experiments
+
+ new_num_workers = 2 * len(cluster.scheduler.workers)
+
+ print(f"Scaling from {len(cluster.scheduler.workers)} to {new_num_workers} workers.")
+
+ cluster.scale(new_num_workers)
+
+ sleep(120)
+
+ client
+
+ #Re-run same experiments with doubled cluster
+
+ for size in (1e9 * n for n in (1, 10, 100)):
+
+ start = time()
+ pi = calc_pi_mc(size).compute()
+ elaps = time() - start
+
+ print_pi_stats(size, pi, time_delta=elaps, num_workers=len(cluster.scheduler.workers))
+ ```
+
+Please check the availability of resources that you want to allocate
+by the script for the example above.
+You can do it with `sinfo` command. The script doesn't work
+without available cluster resources.
+
+### Mpi4py - MPI for Python
+
+Message Passing Interface (MPI) is a standardized and
+portable message-passing standard, designed to
+function on a wide variety of parallel computing architectures.
+
+Mpi4py (MPI for Python) provides bindings of the MPI standard for
+the Python programming language,
+allowing any Python program to exploit multiple processors.
+
+Mpi4py is based on MPI-2 C++ bindings. It supports almost all MPI calls.
+It supports communication of pickle-able Python objects.
+Mpi4py provides optimized communication of NumPy arrays.
+
+Mpi4py is included in the SciPy-bundle modules on the ZIH system.
```console
marie@compute$ module load SciPy-bundle/2020.11-foss-2020b
@@ -169,7 +393,7 @@ mpi4py 3.0.3
[...]
```
-Other versions of the package can be found with
+Other versions of the package can be found with:
```console
marie@compute$ module spider mpi4py
@@ -186,7 +410,7 @@ marie@compute$ module spider mpi4py
Names marked by a trailing (E) are extensions provided by another module.
-----------------------------------------------------------------------------------------------------------------------------------------
- For detailed information about a specific "mpi4py" package (including how to load the modules) use the module's full name.
+ For detailed information about a specific "mpi4py" package (including how to load the modules), use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
@@ -194,7 +418,11 @@ Names marked by a trailing (E) are extensions provided by another module.
-----------------------------------------------------------------------------------------------------------------------------------------
```
-Check if mpi4py is running correctly
+Moreover, it is possible to install mpi4py in your local conda
+environment.
+
+The example of mpi4py usage for the verification that
+mpi4py is running correctly can be found below:
```python
from mpi4py import MPI
@@ -202,4 +430,22 @@ comm = MPI.COMM_WORLD
print("%d of %d" % (comm.Get_rank(), comm.Get_size()))
```
-**TODO** verify mpi4py installation
+For the multi-node case, use a script similar to this:
+
+```bash
+#!/bin/bash
+#SBATCH --nodes=2
+#SBATCH --partition=ml
+#SBATCH --tasks-per-node=2
+#SBATCH --cpus-per-task=1
+
+module load modenv/ml
+module load PythonAnaconda/3.6
+
+eval "$(conda shell.bash hook)"
+conda activate /home/marie/conda-virtual-environment/kernel2 && srun python mpi4py_test.py #specify name of your virtual environment
+```
+
+For the verification of the multi-node case,
+you can use the Python code from the previous part
+(with verification of the installation) as a test file.
diff --git a/doc.zih.tu-dresden.de/docs/software/distributed_training.md b/doc.zih.tu-dresden.de/docs/software/distributed_training.md
index 1c84d44b08f6311719802b9e22ce3c6c3af6cfe9..b3c6733bc0c7150eeee561ec450d33a7db27d54a 100644
--- a/doc.zih.tu-dresden.de/docs/software/distributed_training.md
+++ b/doc.zih.tu-dresden.de/docs/software/distributed_training.md
@@ -2,183 +2,346 @@
## Internal Distribution
+Training a machine learning model can be a very time-consuming task.
+Distributed training allows scaling up deep learning tasks,
+so we can train very large models and speed up training time.
+
+There are two paradigms for distributed training:
+
+1. data parallelism:
+each device has a replica of the model and computes over different parts of the data.
+2. model parallelism:
+models are distributed over multiple devices.
+
+In the following, we will stick to the concept of data parallelism because it is a widely-used
+technique.
+There are basically two strategies to train the scattered data throughout the devices:
+
+1. synchronous training: devices (workers) are trained over different slices of the data and at the
+end of each step gradients are aggregated.
+2. asynchronous training:
+all devices are independently trained over the data and update variables asynchronously.
+
### Distributed TensorFlow
-TODO
+[TensorFlow](https://www.tensorflow.org/guide/distributed_training) provides a high-end API to
+train your model and distribute the training on multiple GPUs or machines with minimal code changes.
+
+The primary distributed training method in TensorFlow is `tf.distribute.Strategy`.
+There are multiple strategies that distribute the training depending on the specific use case,
+the data and the model.
+
+TensorFlow refers to the synchronous training as mirrored strategy.
+There are two mirrored strategies available whose principles are the same:
+
+- `tf.distribute.MirroredStrategy` supports the training on multiple GPUs on one machine.
+- `tf.distribute.MultiWorkerMirroredStrategy` for multiple machines, each with multiple GPUs.
+
+The Central Storage Strategy applies to environments where the GPUs might not be able to store
+the entire model:
+
+- `tf.distribute.experimental.CentralStorageStrategy` supports the case of a single machine
+with multiple GPUs.
+
+The CPU holds the global state of the model and GPUs perform the training.
+
+In some cases asynchronous training might be the better choice, for example, if workers differ on
+capability, are down for maintenance, or have different priorities.
+The Parameter Server Strategy is capable of applying asynchronous training:
+
+- `tf.distribute.experimental.ParameterServerStrategy` requires several Parameter Servers and workers.
+
+The Parameter Server holds the parameters and is responsible for updating
+the global state of the models.
+Each worker runs the training loop independently.
+
+??? example "Multi Worker Mirrored Strategy"
+
+ In this case, we will go through an example with Multi Worker Mirrored Strategy.
+ Multi-node training requires a `TF_CONFIG` environment variable to be set which will
+ be different on each node.
+
+ ```console
+ marie@compute$ TF_CONFIG='{"cluster": {"worker": ["10.1.10.58:12345", "10.1.10.250:12345"]}, "task": {"index": 0, "type": "worker"}}' python main.py
+ ```
+
+ The `cluster` field describes how the cluster is set up (same on each node).
+ Here, the cluster has two nodes referred to as workers.
+ The `IP:port` information is listed in the `worker` array.
+ The `task` field varies from node to node.
+ It specifies the type and index of the node.
+ In this case, the training job runs on worker 0, which is `10.1.10.58:12345`.
+ We need to adapt this snippet for each node.
+ The second node will have `'task': {'index': 1, 'type': 'worker'}`.
+
+ With two modifications, we can parallelize the serial code:
+ We need to initialize the distributed strategy:
+
+ ```python
+ strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
+ ```
+
+ And define the model under the strategy scope:
+
+ ```python
+ with strategy.scope():
+ model = resnet.resnet56(img_input=img_input, classes=NUM_CLASSES)
+ model.compile(
+ optimizer=opt,
+ loss='sparse_categorical_crossentropy',
+ metrics=['sparse_categorical_accuracy'])
+ model.fit(train_dataset,
+ epochs=NUM_EPOCHS)
+ ```
+
+ To run distributed training, the training script needs to be copied to all nodes,
+ in this case on two nodes.
+ TensorFlow is available as a module.
+ Check for the version.
+ The `TF_CONFIG` environment variable can be set as a prefix to the command.
+ Now, run the script on the partition `alpha` simultaneously on both nodes:
+
+ ```bash
+ #!/bin/bash
+
+ #SBATCH --job-name=distr
+ #SBATCH --partition=alpha
+ #SBATCH --output=%j.out
+ #SBATCH --error=%j.err
+ #SBATCH --mem=64000
+ #SBATCH --nodes=2
+ #SBATCH --ntasks=2
+ #SBATCH --ntasks-per-node=1
+ #SBATCH --cpus-per-task=14
+ #SBATCH --gres=gpu:1
+ #SBATCH --time=01:00:00
+
+ function print_nodelist {
+ scontrol show hostname $SLURM_NODELIST
+ }
+ NODE_1=$(print_nodelist | awk '{print $1}' | sort -u | head -n 1)
+ NODE_2=$(print_nodelist | awk '{print $1}' | sort -u | tail -n 1)
+ IP_1=$(dig +short ${NODE_1}.taurus.hrsk.tu-dresden.de)
+ IP_2=$(dig +short ${NODE_2}.taurus.hrsk.tu-dresden.de)
+
+ module load modenv/hiera
+ module load modenv/hiera GCC/10.2.0 CUDA/11.1.1 OpenMPI/4.0.5 TensorFlow/2.4.1
+
+ # On the first node
+ TF_CONFIG='{"cluster": {"worker": ["'"${NODE_1}"':33562", "'"${NODE_2}"':33561"]}, "task": {"index": 0, "type": "worker"}}' srun -w ${NODE_1} -N 1 --ntasks=1 --gres=gpu:1 python main_ddl.py &
+
+ # On the second node
+ TF_CONFIG='{"cluster": {"worker": ["'"${NODE_1}"':33562", "'"${NODE_2}"':33561"]}, "task": {"index": 1, "type": "worker"}}' srun -w ${NODE_2} -N 1 --ntasks=1 --gres=gpu:1 python main_ddl.py &
+
+ wait
+ ```
### Distributed PyTorch
-Hint: just copied some old content as starting point
+!!! note
-#### Using Multiple GPUs with PyTorch
+ This section is under construction
-Effective use of GPUs is essential, and it implies using parallelism in
-your code and model. Data Parallelism and model parallelism are effective instruments
-to improve the performance of your code in case of GPU using.
+PyTorch provides multiple ways to achieve data parallelism to train the deep learning models
+efficiently. These models are part of the `torch.distributed` sub-package that ships with the main
+deep learning package.
-The data parallelism is a widely-used technique. It replicates the same model to all GPUs,
-where each GPU consumes a different partition of the input data. You could see this method [here](https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html).
+The easiest method to quickly prototype if the model is trainable in a multi-GPU setting is to wrap
+the existing model with the `torch.nn.DataParallel` class as shown below,
+
+```python
+model = torch.nn.DataParalell(model)
+```
+
+Adding this single line of code to the existing application will let PyTorch know that the model
+needs to be parallelized. But since this method uses threading to achieve parallelism, it fails to
+achieve true parallelism due to the well known issue of Global Interpreter Lock that exists in
+Python. To work around this issue and gain performance benefits of parallelism, the use of
+`torch.nn.DistributedDataParallel` is recommended. This involves little more code changes to set up,
+but further increases the performance of model training. The starting step is to initialize the
+process group by calling the `torch.distributed.init_process_group()` using the appropriate back end
+such as NCCL, MPI or Gloo. The use of NCCL as back end is recommended as it is currently the fastest
+back end when using GPUs.
+
+#### Using Multiple GPUs with PyTorch
-The example below shows how to solve that problem by using model
-parallel, which, in contrast to data parallelism, splits a single model
-onto different GPUs, rather than replicating the entire model on each
-GPU. The high-level idea of model parallel is to place different sub-networks of a model onto different
-devices. As the only part of a model operates on any individual device, a set of devices can
-collectively serve a larger model.
+The example below shows how to solve that problem by using model parallelism, which in contrast to
+data parallelism splits a single model onto different GPUs, rather than replicating the entire
+model on each GPU.
+The high-level idea of model parallelism is to place different sub-networks of a model onto
+different devices.
+As only part of a model operates on any individual device a set of devices can collectively
+serve a larger model.
-It is recommended to use [DistributedDataParallel]
-(https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html),
+It is recommended to use
+[DistributedDataParallel](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html),
instead of this class, to do multi-GPU training, even if there is only a single node.
-See: Use nn.parallel.DistributedDataParallel instead of multiprocessing or nn.DataParallel.
+See: Use `nn.parallel.DistributedDataParallel` instead of multiprocessing or `nn.DataParallel`.
Check the [page](https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead) and
[Distributed Data Parallel](https://pytorch.org/docs/stable/notes/ddp.html#ddp).
-Examples:
+??? example "Parallel Model"
-1\. The parallel model. The main aim of this model to show the way how
-to effectively implement your neural network on several GPUs. It
-includes a comparison of different kinds of models and tips to improve
-the performance of your model. **Necessary** parameters for running this
-model are **2 GPU** and 14 cores (56 thread).
+ The main aim of this model is to show the way how to effectively implement your neural network
+ on multiple GPUs. It includes a comparison of different kinds of models and tips to improve the
+ performance of your model.
+ **Necessary** parameters for running this model are **2 GPU** and 14 cores.
-(example_PyTorch_parallel.zip)
+ Download: [example_PyTorch_parallel.zip (4.2 KB)](misc/example_PyTorch_parallel.zip)
-Remember that for using [JupyterHub service](../access/jupyterhub.md)
-for PyTorch you need to create and activate
-a virtual environment (kernel) with loaded essential modules.
+ Remember that for using [JupyterHub service](../access/jupyterhub.md) for PyTorch, you need to
+ create and activate a virtual environment (kernel) with loaded essential modules.
-Run the example in the same way as the previous examples.
+ Run the example in the same way as the previous examples.
-#### Distributed data-parallel
+#### Distributed Data-Parallel
[DistributedDataParallel](https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel)
(DDP) implements data parallelism at the module level which can run across multiple machines.
Applications using DDP should spawn multiple processes and create a single DDP instance per process.
-DDP uses collective communications in the [torch.distributed]
-(https://pytorch.org/tutorials/intermediate/dist_tuto.html)
-package to synchronize gradients and buffers.
+DDP uses collective communications in the
+[torch.distributed](https://pytorch.org/tutorials/intermediate/dist_tuto.html) package to
+synchronize gradients and buffers.
-The tutorial could be found [here](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html).
+Please also look at the [official tutorial](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html).
-To use distributed data parallelization on ZIH system please use following
-parameters: `--ntasks-per-node` -parameter to the number of GPUs you use
-per node. Also, it could be useful to increase `memomy/cpu` parameters
-if you run larger models. Memory can be set up to:
+To use distributed data parallelism on ZIH systems, please make sure the value of
+parameter `--ntasks-per-node=<N>` equals the number of GPUs you use per node.
+Also, it can be useful to increase `memory/cpu` parameters if you run larger models.
+Memory can be set up to:
-`--mem=250000` and `--cpus-per-task=7` for the partition `ml`.
+- `--mem=250G` and `--cpus-per-task=7` for the partition `ml`.
+- `--mem=60G` and `--cpus-per-task=6` for the partition `gpu2`.
-`--mem=60000` and `--cpus-per-task=6` for the partition `gpu2`.
-
-Keep in mind that only one memory parameter (`--mem-per-cpu` = <MB> or `--mem`=<MB>) can be
-specified
+Keep in mind that only one memory parameter (`--mem-per-cpu=<MB>` or `--mem=<MB>`) can be specified.
## External Distribution
### Horovod
-[Horovod](https://github.com/horovod/horovod) is the open source distributed training
-framework for TensorFlow, Keras, PyTorch. It is supposed to make it easy
-to develop distributed deep learning projects and speed them up with
-TensorFlow.
+[Horovod](https://github.com/horovod/horovod) is the open-source distributed training framework
+for TensorFlow, Keras and PyTorch.
+It makes it easier to develop distributed deep learning projects and speeds them up.
+Horovod scales well to a large number of nodes and has a strong focus on efficient training on
+GPUs.
#### Why use Horovod?
-Horovod allows you to easily take a single-GPU TensorFlow and PyTorch
-program and successfully train it on many GPUs! In
-some cases, the MPI model is much more straightforward and requires far
-less code changes than the distributed code from TensorFlow for
-instance, with parameter servers. Horovod uses MPI and NCCL which gives
-in some cases better results than pure TensorFlow and PyTorch.
+Horovod allows you to easily take a single-GPU TensorFlow and PyTorch program and
+train it on many GPUs!
+In some cases, the MPI model is much more straightforward and requires far less code changes than
+the distributed code from TensorFlow for instance, with parameter servers.
+Horovod uses MPI and NCCL which gives in some cases better results than
+pure TensorFlow and PyTorch.
-#### Horovod as a module
+#### Horovod as Module
-Horovod is available as a module with **TensorFlow** or **PyTorch**for **all** module environments.
+Horovod is available as a module with **TensorFlow** or **PyTorch** for
+**all** module environments.
Please check the [software module list](modules.md) for the current version of the software.
Horovod can be loaded like other software on ZIH system:
-```Bash
-ml av Horovod #Check available modules with Python
-module load Horovod #Loading of the module
+```console
+marie@compute$ module spider Horovod # Check available modules
+------------------------------------------------------------------------------------------------
+ Horovod:
+------------------------------------------------------------------------------------------------
+ Description:
+ Horovod is a distributed training framework for TensorFlow.
+
+ Versions:
+ Horovod/0.18.2-fosscuda-2019b-TensorFlow-2.0.0-Python-3.7.4
+ Horovod/0.19.5-fosscuda-2019b-TensorFlow-2.2.0-Python-3.7.4
+ Horovod/0.21.1-TensorFlow-2.4.1
+[...]
+marie@compute$ module load Horovod/0.19.5-fosscuda-2019b-TensorFlow-2.2.0-Python-3.7.4
```
-#### Horovod installation
-
-However, if it is necessary to use Horovod with **PyTorch** or use
-another version of Horovod it is possible to install it manually. To
-install Horovod you need to create a virtual environment and load the
-dependencies (e.g. MPI). Installing PyTorch can take a few hours and is
-not recommended
-
-**Note:** You could work with simple examples in your home directory but **please use workspaces
-for your study and work projects** (see the Storage concept).
+Or if you want to use Horovod on the partition `alpha`, you can load it with the dependencies:
-Setup:
-
-```Bash
-srun -N 1 --ntasks-per-node=6 -p ml --time=08:00:00 --pty bash #allocate a Slurm job allocation, which is a set of resources (nodes)
-module load modenv/ml #Load dependencies by using modules
-module load OpenMPI/3.1.4-gcccuda-2018b
-module load Python/3.6.6-fosscuda-2018b
-module load cuDNN/7.1.4.18-fosscuda-2018b
-module load CMake/3.11.4-GCCcore-7.3.0
-virtualenv --system-site-packages <location_for_your_environment> #create virtual environment
-source <location_for_your_environment>/bin/activate #activate virtual environment
+```console
+marie@alpha$ module spider Horovod #Check available modules
+marie@alpha$ module load modenv/hiera GCC/10.2.0 CUDA/11.1.1 OpenMPI/4.0.5 Horovod/0.21.1-TensorFlow-2.4.1
```
-Or when you need to use conda:
+#### Horovod Installation
-```Bash
-srun -N 1 --ntasks-per-node=6 -p ml --time=08:00:00 --pty bash #allocate a Slurm job allocation, which is a set of resources (nodes)
-module load modenv/ml #Load dependencies by using modules
-module load OpenMPI/3.1.4-gcccuda-2018b
-module load PythonAnaconda/3.6
-module load cuDNN/7.1.4.18-fosscuda-2018b
-module load CMake/3.11.4-GCCcore-7.3.0
+However, if it is necessary to use another version of Horovod, it is possible to install it
+manually. For that, you need to create a [virtual environment](python_virtual_environments.md) and
+load the dependencies (e.g. MPI).
+Installing TensorFlow can take a few hours and is not recommended.
-conda create --prefix=<location_for_your_environment> python=3.6 anaconda #create virtual environment
+##### Install Horovod for TensorFlow with Python and Pip
-conda activate <location_for_your_environment> #activate virtual environment
-```
+This example shows the installation of Horovod for TensorFlow.
+Adapt as required and refer to the [Horovod documentation](https://horovod.readthedocs.io/en/stable/install_include.html)
+for details.
-Install PyTorch (not recommended)
+```console
+marie@alpha$ HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_TENSORFLOW=1 pip install --no-cache-dir horovod\[tensorflow\]
+[...]
+marie@alpha$ horovodrun --check-build
+Horovod v0.19.5:
-```Bash
-cd /tmp
-git clone https://github.com/pytorch/pytorch #clone PyTorch from the source
-cd pytorch #go to folder
-git checkout v1.7.1 #Checkout version (example: 1.7.1)
-git submodule update --init #Update dependencies
-python setup.py install #install it with python
-```
+Available Frameworks:
+ [X] TensorFlow
+ [ ] PyTorch
+ [ ] MXNet
-##### Install Horovod for PyTorch with python and pip
+Available Controllers:
+ [X] MPI
+ [ ] Gloo
-In the example presented installation for the PyTorch without
-TensorFlow. Adapt as required and refer to the Horovod documentation for
-details.
-
-```Bash
-HOROVOD_GPU_ALLREDUCE=MPI HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 pip install --no-cache-dir horovod
+Available Tensor Operations:
+ [X] NCCL
+ [ ] DDL
+ [ ] CCL
+ [X] MPI
+ [ ] Gloo
```
-##### Verify that Horovod works
-
-```Bash
-python #start python
-import torch #import pytorch
-import horovod.torch as hvd #import horovod
-hvd.init() #initialize horovod
-hvd.size()
-hvd.rank()
-print('Hello from:', hvd.rank())
+If you want to use OpenMPI then specify `HOROVOD_GPU_ALLREDUCE=MPI`.
+To have better performance it is recommended to use NCCL instead of OpenMPI.
+
+##### Verify Horovod Works
+
+```pycon
+>>> import tensorflow
+2021-10-07 16:38:55.694445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
+>>> import horovod.tensorflow as hvd #import horovod
+>>> hvd.init() #initialize horovod
+>>> hvd.size()
+1
+>>> hvd.rank()
+0
+>>> print('Hello from:', hvd.rank())
+Hello from: 0
```
-##### Horovod with NCCL
-
-If you want to use NCCL instead of MPI you can specify that in the
-install command after loading the NCCL module:
-
-```Bash
-module load NCCL/2.3.7-fosscuda-2018b
-HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_GPU_BROADCAST=NCCL HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 pip install --no-cache-dir horovod
-```
+??? example
+
+ Follow the steps in the
+ [official examples](https://github.com/horovod/horovod/tree/master/examples)
+ to parallelize your code.
+ In Horovod, each GPU gets pinned to a process.
+ You can easily start your job with the following bash script with four processes on two nodes:
+
+ ```bash
+ #!/bin/bash
+ #SBATCH --nodes=2
+ #SBATCH --ntasks=4
+ #SBATCH --ntasks-per-node=2
+ #SBATCH --gres=gpu:2
+ #SBATCH --partition=ml
+ #SBATCH --mem=250G
+ #SBATCH --time=01:00:00
+ #SBATCH --output=run_horovod.out
+
+ module load modenv/ml
+ module load Horovod/0.19.5-fosscuda-2019b-TensorFlow-2.2.0-Python-3.7.4
+
+ srun python <your_program.py>
+ ```
+
+ Do not forget to specify the total number of tasks `--ntasks` and the number of tasks per node
+ `--ntasks-per-node` which must match the number of GPUs per node.
diff --git a/doc.zih.tu-dresden.de/docs/software/fem_software.md b/doc.zih.tu-dresden.de/docs/software/fem_software.md
index bd65ea9832462bae475841f2e3ed2fa8193e3355..aa8917ad3c59fbd1c60a349e9030ff830f904234 100644
--- a/doc.zih.tu-dresden.de/docs/software/fem_software.md
+++ b/doc.zih.tu-dresden.de/docs/software/fem_software.md
@@ -1,247 +1,238 @@
# FEM Software
-For an up-to-date list of the installed software versions on our
-cluster, please refer to SoftwareModulesList **TODO LINK** (SoftwareModulesList).
+!!! hint "Its all in the modules"
-## Abaqus
-
-[ABAQUS](http://www.hks.com) **TODO links to realestate site** is a general-purpose finite-element program
-designed for advanced linear and nonlinear engineering analysis
-applications with facilities for linking-in user developed material
-models, elements, friction laws, etc.
-
-Eike Dohmen (from Inst.f. Leichtbau und Kunststofftechnik) sent us the
-attached description of his ABAQUS calculations. Please try to adapt
-your calculations in that way.\<br />Eike is normally a Windows-User and
-his description contains also some hints for basic Unix commands. (
-ABAQUS-SLURM.pdf **TODO LINK** (%ATTACHURL%/ABAQUS-SLURM.pdf) - only in German)
-
-Please note: Abaqus calculations should be started with a batch script.
-Please read the information about the Batch System **TODO LINK ** (BatchSystems)
-SLURM.
-
-The detailed Abaqus documentation can be found at
-abaqus **TODO LINK MISSING** (only accessible from within the
-TU Dresden campus net).
+ All packages described in this section, are organized in so-called modules. To list the available versions of a package and load a
+ particular, e.g., ANSYS, version, invoke the commands
-**Example - Thanks to Benjamin Groeger, Inst. f. Leichtbau und
-Kunststofftechnik) **
-
-1. Prepare an Abaqus input-file (here the input example from Benjamin)
-
-Rot-modell-BenjaminGroeger.inp **TODO LINK** (%ATTACHURL%/Rot-modell-BenjaminGroeger.inp)
-
-2. Prepare a batch script on taurus like this
-
-```
-#!/bin/bash<br>
-### Thanks to Benjamin Groeger, Institut fuer Leichtbau und Kunststofftechnik, 38748<br />### runs on taurus and needs ca 20sec with 4cpu<br />### generates files:
-### yyyy.com
-### yyyy.dat
-### yyyy.msg
-### yyyy.odb
-### yyyy.prt
-### yyyy.sim
-### yyyy.sta
-#SBATCH --nodes=1 ### with >1 node abaqus needs a nodeliste
-#SBATCH --ntasks-per-node=4
-#SBATCH --mem=500 ### memory (sum)
-#SBATCH --time=00:04:00
-### give a name, what ever you want
-#SBATCH --job-name=yyyy
-### you get emails when the job will finished or failed
-### set your right email
-#SBATCH --mail-type=END,FAIL
-#SBATCH --mail-user=xxxxx.yyyyyy@mailbox.tu-dresden.de
-### set your project
-#SBATCH -A p_xxxxxxx
-### Abaqus have its own MPI
-unset SLURM_GTIDS
-### load and start
-module load ABAQUS/2019
-abaqus interactive input=Rot-modell-BenjaminGroeger.inp job=yyyy cpus=4 mp_mode=mpi
+ ```console
+ marie@login$ module avail ANSYS
+ [...]
+ marie@login$ module load ANSYS/<version>
+ ```
-```
+ The section [runtime environment](runtime_environment.md) provides a comprehensive overview
+ on the module system and relevant commands.
-3. Start the batch script (name of our script is
-"batch-Rot-modell-BenjaminGroeger")
+## Abaqus
-```
-sbatch batch-Rot-modell-BenjaminGroeger --->; you will get a jobnumber = JobID (for example 3130522)
-```
+[Abaqus](https://www.3ds.com/de/produkte-und-services/simulia/produkte/abaqus/) is a general-purpose
+finite element method program designed for advanced linear and nonlinear engineering analysis
+applications with facilities for linking-in user developed material models, elements, friction laws,
+etc.
-4. Control the status of the job
+### Guide by User
-```
-squeue -u your_login -->; in column "ST" (Status) you will find a R=Running or P=Pending (waiting for resources)
-```
+Eike Dohmen (from Inst. f. Leichtbau und Kunststofftechnik) sent us the description of his
+Abaqus calculations. Please try to adapt your calculations in that way. Eike is normally a
+Windows user and his description contains also some hints for basic Unix commands:
+[Abaqus-Slurm.pdf (only in German)](misc/abaqus-slurm.pdf).
-## ANSYS
+### General
-ANSYS is a general-purpose finite-element program for engineering
-analysis, and includes preprocessing, solution, and post-processing
-functions. It is used in a wide range of disciplines for solutions to
-mechanical, thermal, and electronic problems. [ANSYS and ANSYS
-CFX](http://www.ansys.com) used to be separate packages in the past and
-are now combined.
+Abaqus calculations should be started using a job file (aka. batch script). Please refer to the
+page covering the [batch system Slurm](../jobs_and_resources/slurm.md) if you are not familiar with
+Slurm or [writing job files](../jobs_and_resources/slurm.md#job-files).
-ANSYS, like all other installed software, is organized in so-called
-modules **TODO LINK** (RuntimeEnvironment). To list the available versions and load a
-particular ANSYS version, type
+??? example "Usage of Abaqus"
-```
-module avail ANSYS
-...
-module load ANSYS/VERSION
-```
+ (Thanks to Benjamin Groeger, Inst. f. Leichtbau und Kunststofftechnik)).
-In general, HPC-systems are not designed for interactive "GUI-working".
-Even so, it is possible to start a ANSYS workbench on Taurus (login
-nodes) interactively for short tasks. The second and recommended way is
-to use batch files. Both modes are documented in the following.
+ 1. Prepare an Abaqus input-file. You can start with the input example from Benjamin:
+ [Rot-modell-BenjaminGroeger.inp](misc/Rot-modell-BenjaminGroeger.inp)
+ 2. Prepare a job file on ZIH systems like this
+ ```bash
+ #!/bin/bash
+ ### needs ca 20 sec with 4cpu
+ ### generates files:
+ ### yyyy.com
+ ### yyyy.dat
+ ### yyyy.msg
+ ### yyyy.odb
+ ### yyyy.prt
+ ### yyyy.sim
+ ### yyyy.sta
+ #SBATCH --nodes=1 # with >1 node Abaqus needs a nodeliste
+ #SBATCH --ntasks-per-node=4
+ #SBATCH --mem=500 # total memory
+ #SBATCH --time=00:04:00
+ #SBATCH --job-name=yyyy # give a name, what ever you want
+ #SBATCH --mail-type=END,FAIL # send email when the job finished or failed
+ #SBATCH --mail-user=<name>@mailbox.tu-dresden.de # set your email
+ #SBATCH -A p_xxxxxxx # charge compute time to your project
+
+
+ # Abaqus has its own MPI
+ unset SLURM_GTIDS
+
+ # load module and start Abaqus
+ module load ABAQUS/2019
+ abaqus interactive input=Rot-modell-BenjaminGroeger.inp job=yyyy cpus=4 mp_mode=mpi
+ ```
+ 3. Start the job file (e.g., name `batch-Rot-modell-BenjaminGroeger.sh`)
+ ```
+ marie@login$ sbatch batch-Rot-modell-BenjaminGroeger.sh # Slurm will provide the Job Id (e.g., 3130522)
+ ```
+ 4. Control the status of the job
+ ```
+ marie@login squeue -u your_login # in column "ST" (Status) you will find a R=Running or P=Pending (waiting for resources)
+ ```
+
+## Ansys
+
+Ansys is a general-purpose finite element method program for engineering analysis, and includes
+preprocessing, solution, and post-processing functions. It is used in a wide range of disciplines
+for solutions to mechanical, thermal, and electronic problems.
+[Ansys and Ansys CFX](http://www.ansys.com) used to be separate packages in the past and are now
+combined.
+
+In general, HPC systems are not designed for interactive working with GUIs. Even so, it is possible to
+start a Ansys workbench on the login nodes interactively for short tasks. The second and
+**recommended way** is to use job files. Both modes are documented in the following.
+
+!!! note ""
+
+ Since the MPI library that Ansys uses internally (Platform MPI) has some problems integrating
+ seamlessly with Slurm, you have to unset the enviroment variable `SLURM_GTIDS` in your
+ environment bevor running Ansysy workbench in interactive andbatch mode.
### Using Workbench Interactively
-For fast things, ANSYS workbench can be invoked interactively on the
-login nodes of Taurus. X11 forwarding needs to enabled when establishing
-the SSH connection. For OpenSSH this option is '-X' and it is valuable
-to use compression of all data via '-C'.
+Ansys workbench (`runwb2`) an be invoked interactively on the login nodes of ZIH systems for short tasks.
+[X11 forwarding](../access/ssh_login.md#x11-forwarding) needs to enabled when establishing the SSH
+connection. For OpenSSH the corresponding option is `-X` and it is valuable to use compression of
+all data via `-C`.
-```
-# Connect to taurus, e.g. ssh -CX
-module load ANSYS/VERSION
-runwb2
+```console
+# SSH connection established using -CX
+marie@login$ module load ANSYS/<version>
+marie@login$ runwb2
```
-If more time is needed, a CPU has to be allocated like this (see topic
-batch systems **TODO LINK** (BatchSystems) for further information):
+If more time is needed, a CPU has to be allocated like this (see
+[batch systems Slurm](../jobs_and_resources/slurm.md) for further information):
+```console
+marie@login$ module load ANSYS/<version>
+marie@login$ srun -t 00:30:00 --x11=first [SLURM_OPTIONS] --pty bash
+[...]
+marie@login$ runwb2
```
-module load ANSYS/VERSION
-srun -t 00:30:00 --x11=first [SLURM_OPTIONS] --pty bash
-runwb2
-```
-
-**Note:** The software NICE Desktop Cloud Visualization (DCV) enables to
-remotly access OpenGL-3D-applications running on taurus using its GPUs
-(cf. virtual desktops **TODO LINK** (Compendium.VirtualDesktops)). Using ANSYS
-together with dcv works as follows:
-
-- Follow the instructions within virtual
- desktops **TODO LINK** (Compendium.VirtualDesktops)
-```
-module load ANSYS
-```
+!!! hint "Better use DCV"
-```
-unset SLURM_GTIDS
-```
+ The software NICE Desktop Cloud Visualization (DCV) enables to
+ remotly access OpenGL-3D-applications running on ZIH systems using its GPUs
+ (cf. [virtual desktops](virtual_desktops.md)).
-- Note the hints w.r.t. GPU support on dcv side
+Ansys can be used under DCV to make use of GPU acceleration. Follow the instructions within
+[virtual desktops](virtual_desktops.md) to set up a DCV session. Then, load a Ansys module, unset
+the environment variable `SLURM_GTIDS`, and finally start the workbench:
-```
-runwb2
+```console
+marie@gpu$ module load ANSYS
+marie@gpu$ unset SLURM_GTIDS
+marie@gpu$ runwb2
```
### Using Workbench in Batch Mode
-The ANSYS workbench (runwb2) can also be used in a batch script to start
-calculations (the solver, not GUI) from a workbench project into the
-background. To do so, you have to specify the -B parameter (for batch
-mode), -F for your project file, and can then either add different
-commands via -E parameters directly, or specify a workbench script file
-containing commands via -R.
+The Ansys workbench (`runwb2`) can also be used in a job file to start calculations (the solver,
+not GUI) from a workbench project into the background. To do so, you have to specify the `-B`
+parameter (for batch mode), `-F` for your project file, and can then either add different commands via
+`-E parameters directly`, or specify a workbench script file containing commands via `-R`.
-**NOTE:** Since the MPI library that ANSYS uses internally (Platform
-MPI) has some problems integrating seamlessly with SLURM, you have to
-unset the enviroment variable SLURM_GTIDS in your job environment before
-running workbench. An example batch script could look like this:
+??? example "Ansys Job File"
+ ```bash
#!/bin/bash
#SBATCH --time=0:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --mem-per-cpu=1000M
+ unset SLURM_GTIDS # Odd, but necessary!
- unset SLURM_GTIDS # Odd, but necessary!
-
- module load ANSYS/VERSION
+ module load ANSYS/<version>
runwb2 -B -F Workbench_Taurus.wbpj -E 'Project.Update' -E 'Save(Overwrite=True)'
#or, if you wish to use a workbench replay file, replace the -E parameters with: -R mysteps.wbjn
+ ```
### Running Workbench in Parallel
-Unfortunately, the number of CPU cores you wish to use cannot simply be
-given as a command line parameter to your runwb2 call. Instead, you have
-to enter it into an XML file in your home. This setting will then be
-used for all your runwb2 jobs. While it is also possible to edit this
-setting via the Mechanical GUI, experience shows that this can be
-problematic via X-Forwarding and we only managed to use the GUI properly
-via DCV **TODO LINK** (DesktopCloudVisualization), so we recommend you simply edit
-the XML file directly with a text editor of your choice. It is located
+Unfortunately, the number of CPU cores you wish to use cannot simply be given as a command line
+parameter to your `runwb2` call. Instead, you have to enter it into an XML file in your `home`
+directory. This setting will then be **used for all** your `runwb2` jobs. While it is also possible
+to edit this setting via the Mechanical GUI, experience shows that this can be problematic via
+X11-forwarding and we only managed to use the GUI properly via [DCV](virtual_desktops.md), so we
+recommend you simply edit the XML file directly with a text editor of your choice. It is located
under:
-'$HOME/.mw/Application Data/Ansys/v181/SolveHandlers.xml'
+`$HOME/.mw/Application Data/Ansys/v181/SolveHandlers.xml`
-(mind the space in there.) You might have to adjust the ANSYS Version
-(v181) in the path. In this file, you can find the parameter
+(mind the space in there.) You might have to adjust the Ansys version
+(here `v181`) in the path to your preferred version. In this file, you can find the parameter
- <MaxNumberProcessors>2</MaxNumberProcessors>
+`<MaxNumberProcessors>2</MaxNumberProcessors>`
-that you can simply change to something like 16 oder 24. For now, you
-should stay within single-node boundaries, because multi-node
-calculations require additional parameters. The number you choose should
-match your used --cpus-per-task parameter in your sbatch script.
+that you can simply change to something like 16 oder 24. For now, you should stay within single-node
+boundaries, because multi-node calculations require additional parameters. The number you choose
+should match your used `--cpus-per-task` parameter in your job file.
## COMSOL Multiphysics
-"[COMSOL Multiphysics](http://www.comsol.com) (formerly FEMLAB) is a
-finite element analysis, solver and Simulation software package for
-various physics and engineering applications, especially coupled
-phenomena, or multiphysics."
-[\[http://en.wikipedia.org/wiki/COMSOL_Multiphysics Wikipedia\]](
- http://en.wikipedia.org/wiki/COMSOL_Multiphysics Wikipedia)
+[COMSOL Multiphysics](http://www.comsol.com) (formerly FEMLAB) is a finite element analysis, solver
+and Simulation software package for various physics and engineering applications, especially coupled
+phenomena, or multiphysics.
-Comsol may be used remotely on ZIH machines or locally on the desktop,
-using ZIH license server.
+COMSOL may be used remotely on ZIH systems or locally on the desktop, using ZIH license server.
-For using Comsol on ZIH machines, the following operating modes (see
-Comsol manual) are recommended:
+For using COMSOL on ZIH systems, we recommend the interactive client-server mode (see COMSOL
+manual).
-- Interactive Client Server Mode
+### Client-Server Mode
-In this mode Comsol runs as server process on the ZIH machine and as
-client process on your local workstation. The client process needs a
-dummy license for installation, but no license for normal work. Using
-this mode is almost undistinguishable from working with a local
-installation. It works well with Windows clients. For this operation
-mode to work, you must build an SSH tunnel through the firewall of ZIH.
-For further information, see the Comsol manual.
+In this mode, COMSOL runs as server process on the ZIH system and as client process on your local
+workstation. The client process needs a dummy license for installation, but no license for normal
+work. Using this mode is almost undistinguishable from working with a local installation. It also works
+well with Windows clients. For this operation mode to work, you must build an SSH tunnel through the
+firewall of ZIH. For further information, please refer to the COMSOL manual.
-Example for starting the server process (4 cores, 10 GB RAM, max. 8
-hours running time):
+### Usage
- module load COMSOL
- srun -c4 -t 8:00 --mem-per-cpu=2500 comsol -np 4 server
+??? example "Server Process"
-- Interactive Job via Batchsystem SLURM
+ Start the server process with 4 cores, 10 GB RAM and max. 8 hours running time using an
+ interactive Slurm job like this:
-<!-- -->
+ ```console
+ marie@login$ module load COMSOL
+ marie@login$ srun -n 1 -c 4 --mem-per-cpu=2500 -t 8:00 comsol -np 4 server
+ ```
- module load COMSOL
- srun -n1 -c4 --mem-per-cpu=2500 -t 8:00 --pty --x11=first comsol -np 4
+??? example "Interactive Job"
+
+ If you'd like to work interactively using COMSOL, you can request for an interactive job with,
+ e.g., 4 cores and 2500 MB RAM for 8 hours and X11 forwarding to open the COMSOL GUI:
+
+ ```console
+ marie@login$ module load COMSOL
+ marie@login$ srun -n 1 -c 4 --mem-per-cpu=2500 -t 8:00 --pty --x11=first comsol -np 4
+ ```
-Man sollte noch schauen, ob das Rendering unter Options -> Preferences
--> Graphics and Plot Windows auf Software-Rendering steht - und dann
-sollte man campusintern arbeiten knnen.
+ Please make sure, that the option *Preferences* --> Graphics --> *Renedering* is set to *software
+ rendering*. Than, you can work from within the campus network.
-- Background Job via Batchsystem SLURM
+??? example "Background Job"
-<!-- -->
+ Interactive working is great for debugging and setting experiments up. But, if you have a huge
+ workload, you should definitively rely on job files. I.e., you put the necessary steps to get
+ the work done into scripts and submit these scripts to the batch system. These two steps are
+ outlined:
+ 1. Create a [job file](../jobs_and_resources/slurm.md#job-files), e.g.
+ ```bash
#!/bin/bash
#SBATCH --time=24:00:00
#SBATCH --nodes=2
@@ -251,21 +242,33 @@ sollte man campusintern arbeiten knnen.
module load COMSOL
srun comsol -mpi=intel batch -inputfile ./MyInputFile.mph
-
-Submit via: `sbatch <filename>`
+ ```
## LS-DYNA
-Both, the shared memory version and the distributed memory version (mpp)
-are installed on all machines.
+[LS-DYNA](https://www.dynamore.de/de) is a general-purpose, implicit and explicit FEM software for
+nonlinear structural analysis. Both, the shared memory version and the distributed memory version
+(`mpp`) are installed on ZIH systems.
+
+You need a job file (aka. batch script) to run the MPI version.
-To run the MPI version on Taurus or Venus you need a batchfile (sumbmit
-with `sbatch <filename>`) like:
+??? example "Minimal Job File"
+ ```bash
#!/bin/bash
- #SBATCH --time=01:00:00 # walltime
- #SBATCH --ntasks=16 # number of processor cores (i.e. tasks)
+ #SBATCH --time=01:00:00 # walltime
+ #SBATCH --ntasks=16 # number of processor cores (i.e. tasks)
#SBATCH --mem-per-cpu=1900M # memory per CPU core
-
+
module load ls-dyna
srun mpp-dyna i=neon_refined01_30ms.k memory=120000000
+ ```
+
+ Submit the job file to the batch system via
+
+ ```console
+ marie@login$ sbatch <filename>
+ ```
+
+ Please refer to the section [Slurm](../jobs_and_resources/slurm.md) for further details and
+ options on the batch system as well as monitoring commands.
diff --git a/doc.zih.tu-dresden.de/docs/software/lo2s.md b/doc.zih.tu-dresden.de/docs/software/lo2s.md
new file mode 100644
index 0000000000000000000000000000000000000000..cf34feccfca15e1e37d5278f30117aaba827e800
--- /dev/null
+++ b/doc.zih.tu-dresden.de/docs/software/lo2s.md
@@ -0,0 +1,141 @@
+# lo2s - Lightweight Node-Level Performance Monitoring
+
+`lo2s` creates parallel OTF2 traces with a focus on both application and system view.
+The traces can contain any of the following information:
+
+* From running threads
+ * Calling context samples based on instruction overflows
+ * The calling context samples are annotated with the disassembled assembler instruction string
+ * The frame pointer-based call-path for each calling context sample
+ * Per-thread performance counter readings
+ * Which thread was scheduled on which CPU at what time
+* From the system
+ * Metrics from tracepoints (e.g., the selected C-state or P-state)
+ * The node-level system tree (CPUs (HW-threads), cores, packages)
+ * CPU power measurements (x86_energy)
+ * Microarchitecture specific metrics (x86_adapt, per package or core)
+ * Arbitrary metrics through plugins (Score-P compatible)
+
+In general, `lo2s` operates either in **process monitoring** or **system monitoring** mode.
+
+With **process monitoring**, all information is grouped by each thread of a monitored process
+group - it shows you *on which CPU is each monitored thread running*. `lo2s` either acts as a
+prefix command to run the process (and also tracks its children) or `lo2s` attaches to a running
+process.
+
+In the **system monitoring** mode, information is grouped by logical CPU - it shows you
+*which thread was running on a given CPU*. Metrics are also shown per CPU.
+
+In both modes, `lo2s` always groups system-level metrics (e.g., tracepoints) by their respective
+system hardware component.
+
+## Usage
+
+Only the basic usage is shown in this Wiki. For a more detailed explanation, refer to the
+[Lo2s website](https://github.com/tud-zih-energy/lo2s).
+
+Before using `lo2s`, set up the correct environment with
+
+```console
+marie@login$ module load lo2s
+```
+
+As `lo2s` is built upon [perf](perf_tools.md), its usage and limitations are very similar to that.
+In particular, you can use `lo2s` as a prefix command just like `perf`. Even some of the command
+line arguments are inspired by `perf`. The main difference to `perf` is that `lo2s` will output
+a [Vampir trace](vampir.md), which allows a full-blown performance analysis almost like
+[Score-P](scorep.md).
+
+To record the behavior of an application, prefix the application run with `lo2s`. We recommend
+using the double dash `--` to prevent mixing command line arguments between `lo2s` and the user
+application. In the following example, we run `lo2s` on the application `sleep 2`.
+
+```console
+marie@compute$ lo2s --no-kernel -- sleep 2
+[ lo2s: sleep 2 (0), 1 threads, 0.014082s CPU, 2.03315s total ]
+[ lo2s: 5 wakeups, wrote 2.48 KiB lo2s_trace_2021-10-12T12-39-06 ]
+```
+
+This will record the application in the `process monitoring mode`. This means, the applications
+process, its forked processes, and threads are recorded and can be analyzed using Vampir.
+The main view will represent each process and thread over time. There will be a metric "CPU"
+indicating for each process, on which CPU it was executed during the runtime.
+
+## Required Permissions
+
+By design, `lo2s` almost exclusively utilizes Linux Kernel facilities such as perf and tracepoints
+to perform the application measurements. For security reasons, these facilities require special
+permissions, in particular `perf_event_paranoid` and read permissions to the `debugfs` under
+`/sys/kernel/debug`.
+
+Luckily, for the `process monitoring mode` the default settings allow you to run `lo2s` just fine.
+All you need to do is pass the `--no-kernel` parameter like in the example above.
+
+For the `system monitoring mode` you can get the required permission with the Slurm parameter
+`--exclusive`. (Note: Regardless of the actual requested processes per node, you will accrue
+cpu-hours as if you had reserved all cores on the node.)
+
+## Memory Requirements
+
+When requesting memory for your jobs, you need to take into account that `lo2s` needs a substantial
+amount of memory for its operation. Unfortunately, the amount of memory depends on the application.
+The amount mainly scales with the number of processes spawned by the traced application. For each
+processes, there is a fixed-sized buffer. This should be fine for a typical HPC application, but
+can lead to extreme cases there the buffers are orders of magnitude larger than the resulting trace.
+For instance, recording a CMake run, which spawns hundreds of processes, each running only for
+a few milliseconds, leaving each buffer almost empty. Still, the buffers needs to be allocated
+and thus require a lot of memory.
+
+Given such a case, we recommend to use the `system monitoring mode` instead, as the memory in this
+mode scales with the number of logical CPUs instead of the number of processes.
+
+## Advanced Topic: System Monitoring
+
+The `system monitoring mode` gives a different view. As the name implies, the focus isn't on processes
+anymore, but the system as a whole. In particular, a trace recorded in this mode will show a timeline
+for each logical CPU of the system. To enable this mode, you need to pass `-a` parameter.
+
+```console
+marie@compute$ lo2s -a
+^C[ lo2s (system mode): monitored processes: 0, 0.136623s CPU, 13.7872s total ]
+[ lo2s (system mode): 36 wakeups, wrote 301.39 KiB lo2s_trace_2021-11-01T09-44-31 ]
+```
+
+Note: As you can read in the above example, `lo2s` monitored zero processes even though it was run
+in the `system monitoring mode`. Certainly, there are more than none processes running on a system.
+However, as the user accounts on our HPC systems are limited to only see their own processes and `lo2s`
+records in the scope of the user, it will only see the users own processes. Hence, in the example
+above, there are no other processes running.
+
+When using the `system monitoring mode` without passing a program, `lo2s` will run indefinitely.
+You can stop the measurement by sending `lo2s` a `SIGINT` signal or hit `ctrl+C`. However, if you pass
+a program, `lo2s` will start that program and run the measurement until the started process finishes.
+Of course, the process and any of its child processes and threads will be visible in the resulting trace.
+
+```console
+marie@compute$ lo2s -a -- sleep 10
+[ lo2s (system mode): sleep 10 (0), 1 threads, monitored processes: 1, 0.133598s CPU, 10.3996s total ]
+[ lo2s (system mode): 39 wakeups, wrote 280.39 KiB lo2s_trace_2021-11-01T09-55-04 ]
+```
+
+Like in the `process monitoring mode`, `lo2s` can also sample instructions in the system monitoring mode.
+You can enable the instruction sampling by passing the parameter `--instruction-sampling` to `lo2s`.
+
+```console
+marie@compute$ lo2s -a --instruction-sampling -- make -j
+[ lo2s (system mode): make -j (0), 268 threads, monitored processes: 286, 258.789s CPU, 445.076s total ]
+[ lo2s (system mode): 3815 wakeups, wrote 39.24 MiB lo2s_trace_2021-10-29T15-08-44 ]
+```
+
+## Advanced Topic: Metric Plugins
+
+`Lo2s` is compatible with [Score-P](scorep.md) metric plugins, but only a subset will work.
+In particular, `lo2s` only supports asynchronous plugins with the per host or once scope.
+You can find a large set of plugins in the [Score-P Organization on GitHub](https://github.com/score-p).
+
+To activate plugins, you can use the same environment variables as with Score-P, or with `LO2S` as
+prefix:
+
+ - LO2S_METRIC_PLUGINS
+ - LO2S_METRIC_PLUGIN
+ - LO2S_METRIC_PLUGIN_PLUGIN
diff --git a/Compendium_attachments/FEMSoftware/Rot-modell-BenjaminGroeger.inp b/doc.zih.tu-dresden.de/docs/software/misc/Rot-modell-BenjaminGroeger.inp
similarity index 100%
rename from Compendium_attachments/FEMSoftware/Rot-modell-BenjaminGroeger.inp
rename to doc.zih.tu-dresden.de/docs/software/misc/Rot-modell-BenjaminGroeger.inp
diff --git a/Compendium_attachments/FEMSoftware/ABAQUS-SLURM.pdf b/doc.zih.tu-dresden.de/docs/software/misc/abaqus-slurm.pdf
similarity index 100%
rename from Compendium_attachments/FEMSoftware/ABAQUS-SLURM.pdf
rename to doc.zih.tu-dresden.de/docs/software/misc/abaqus-slurm.pdf
diff --git a/Compendium_attachments/PyTorch/example_PyTorch_parallel.zip b/doc.zih.tu-dresden.de/docs/software/misc/example_PyTorch_parallel.zip
similarity index 100%
rename from Compendium_attachments/PyTorch/example_PyTorch_parallel.zip
rename to doc.zih.tu-dresden.de/docs/software/misc/example_PyTorch_parallel.zip
diff --git a/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md b/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md
index 1621756f01b01732b675d62dd97a7aa7543c74f2..7b40b9e480755b099c866f0dec2d9a707ea684af 100644
--- a/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md
+++ b/doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md
@@ -230,7 +230,7 @@ Module ORCA/4.2.1-gompi-2019b and 11 dependencies loaded.
## Siesta
-[Siesta](https://www.uam.es/siesta) (Spanish Initiative for Electronic Simulations with
+[Siesta](https://siesta-project.org/siesta) (Spanish Initiative for Electronic Simulations with
Thousands of Atoms) is both a method and its computer program implementation,
to perform electronic structure calculations and ab initio
molecular dynamics simulations of molecules and solids.
diff --git a/doc.zih.tu-dresden.de/docs/software/ngc_containers.md b/doc.zih.tu-dresden.de/docs/software/ngc_containers.md
new file mode 100644
index 0000000000000000000000000000000000000000..f19612d9a3310f869a483c20328d51168317552a
--- /dev/null
+++ b/doc.zih.tu-dresden.de/docs/software/ngc_containers.md
@@ -0,0 +1,161 @@
+# GPU-accelerated Containers for Deep Learning (NGC Containers)
+
+A [container](containers.md) is an executable and portable unit of software.
+On ZIH systems, [Singularity](https://sylabs.io/) is used as a standard container solution.
+
+[NGC](https://developer.nvidia.com/ai-hpc-containers),
+a registry of highly GPU-optimized software,
+has been enabling scientists and researchers by providing regularly updated
+and validated containers of HPC and AI applications.
+NGC containers are **GPU-optimized** containers
+for deep learning, machine learning, visualization:
+
+- Built-in libraries and dependencies;
+- Faster training with Automatic Mixed Precision (AMP);
+- Opportunity to scale up from single-node to multi-node systems;
+- Performance optimized.
+
+!!! note "Advantages of NGC containers"
+ - NGC containers were highly optimized for cluster usage.
+ The performance provided by NGC containers is comparable to the performance
+ provided by the modules on the ZIH system (which is potentially the most performant way).
+ NGC containers are a quick and efficient way to apply the best models
+ on your dataset on a ZIH system;
+ - NGC containers allow using an exact version of the software
+ without installing it with all prerequisites manually.
+ Manual installation can result in poor performance (e.g. using conda to install a software).
+
+## Run NGC Containers on the ZIH System
+
+The first step is a choice of the necessary software (container) to run.
+The [NVIDIA NGC catalog](https://ngc.nvidia.com/catalog)
+contains a host of GPU-optimized containers for deep learning,
+machine learning, visualization, and HPC applications that are tested
+for performance, security, and scalability.
+It is necessary to register to have full access to the catalog.
+
+To find a container that fits the requirements of your task, please check
+the [official examples page](https://github.com/NVIDIA/DeepLearningExamples)
+with the list of main containers with their features and peculiarities.
+
+### Run NGC container on a Single GPU
+
+!!! note
+ Almost all NGC containers can work with a single GPU.
+
+To use NGC containers, it is necessary to understand the main Singularity commands.
+
+If you are not familiar with Singularity's syntax, please find the information on the
+[official page](https://sylabs.io/guides/3.0/user-guide/quick_start.html#interact-with-images).
+However, some main commands will be explained.
+
+Create a container from the image from the NGC catalog.
+(For this example, the alpha is used):
+
+```console
+marie@login$ srun --partition=alpha --nodes=1 --ntasks-per-node=1 --ntasks=1 --gres=gpu:1 --time=08:00:00 --pty --mem=50000 bash
+
+marie@compute$ cd /scratch/ws/<name_of_your_workspace>/containers #please create a Workspace
+
+marie@compute$ singularity pull pytorch:21.08-py3.sif docker://nvcr.io/nvidia/pytorch:21.08-py3
+```
+
+Now, you have a fully functional PyTorch container.
+
+Please pay attention, using `srun` directly on the shell will lead to
+background by using batch jobs.
+For that, you can conveniently put the parameters directly into the job file,
+which you can submit using `sbatch` command.
+
+In the majority of cases, the container doesn't contain the dataset for training models.
+To download the dataset, please follow the
+[instructions](https://github.com/NVIDIA/DeepLearningExamples) for the exact container.
+Also, you can find the instructions in a README file which you can find inside the container:
+
+```console
+marie@compute$ singularity exec pytorch:21.06-py3_beegfs vim /workspace/examples/resnet50v1.5/README.md
+```
+
+It is recommended to run the container with a single command.
+However, for the educational purpose, the separate commands will be presented below:
+
+```console
+marie@login$ srun --partition=alpha --nodes=1 --ntasks-per-node=1 --ntasks=1 --gres=gpu:1 --time=08:00:00 --pty --mem=50000 bash
+```
+
+Run a shell within a container with the `singularity shell` command:
+
+```console
+marie@compute$ singularity shell --nv -B /scratch/imagenet:/data/imagenet pytorch:21.06-py3
+```
+
+The flag `--nv` in the command above was used to enable Nvidia support for GPU usage
+and a flag `-B` for a user-bind path specification.
+
+Run the training inside the container:
+
+```console
+marie@container$ python /workspace/examples/resnet50v1.5/multiproc.py --nnodes=1 --nproc_per_node=1 \
+ --node_rank=0 /workspace/examples/resnet50v1.5/main.py --data-backend dali-cpu \
+ --raport-file raport.json -j16 -p 100 --lr 2.048 --optimizer-batch-size 2048 --warmup 8 \
+ --arch resnet50 -c fanin --label-smoothing 0.1 --lr-schedule cosine --mom 0.875 \
+ --wd 3.0517578125e-05 -b 256 --epochs 90 /data/imagenet
+```
+
+!!! warning
+ Please keep in mind that it is necessary to specify the amount of resources that you use inside
+ the container, especially if you have allocated more resources in the cluster. Regularly, you
+ can do it with flags such as `--nproc_per_node`. You can find more information in the README
+ file inside the container.
+
+As an example, please find the full command to run the ResNet50 model
+on the ImageNet dataset inside the PyTorch container:
+
+```console
+marie@login$ srun --partition=alpha --nodes=1 --ntasks-per-node=1 --ntasks=1 --gres=gpu:1 --time=08:00:00 --pty --mem=50000 \
+ singularity exec --nv -B /scratch/ws/0/anpo879a-ImgNet/imagenet:/data/imagenet pytorch:21.06-py3 \
+ python /workspace/examples/resnet50v1.5/multiproc.py --nnodes=1 --nproc_per_node 1 \
+ --node_rank=0 /workspace/examples/resnet50v1.5/main.py --data-backend dali-cpu --raport-file raport.json \
+ -j16 -p 100 --lr 2.048 --optimizer-batch-size 2048 --warmup 8 --arch resnet50 -c fanin --label-smoothing 0.1 \
+ --lr-schedule cosine --mom 0.875 --wd 3.0517578125e-05 -b 256 --epochs 90 /data/imagenet
+```
+
+### Multi-GPU Usage
+
+The majority of the NGC containers allow you to use multiple GPUs from one node
+to run the model inside the container.
+However, the NGC containers were made by Nvidia for the Nvidia cluster,
+which is not ZIH system.
+Moreover, editing NGC containers requires root privileges,
+which can be done only with [containers](containers.md) on ZIH systems.
+Thus, there is no guarantee that all NGC containers work right out of the box.
+
+However, PyTorch and TensorFlow containers support multi-GPU usage.
+
+An example of using the PyTorch container for the training of the ResNet50 model
+on the classification task on the ImageNet dataset is presented below:
+
+```console
+marie@login$ srun --partition=alpha --nodes=1 --ntasks-per-node=8 --ntasks=8 --gres=gpu:8 --time=08:00:00 --pty --mem=700G bash
+```
+
+```console
+marie@alpha$ singularity exec --nv -B /scratch/ws/0/marie-ImgNet/imagenet:/data/imagenet pytorch:21.06-py3 \
+ python /workspace/examples/resnet50v1.5/multiproc.py --nnodes=1 --nproc_per_node 8 \
+ --node_rank=0 /workspace/examples/resnet50v1.5/main.py --data-backend dali-cpu \
+ --raport-file raport.json -j16 -p 100 --lr 2.048 --optimizer-batch-size 2048 --warmup 8 \
+ --arch resnet50 -c fanin --label-smoothing 0.1 --lr-schedule cosine --mom 0.875 \
+ --wd 3.0517578125e-05 -b 256 --epochs 90 /data/imagenet
+```
+
+Please pay attention to the parameter `--nproc_per_node`.
+The value is equal to 8 because 8 GPUs per node were allocated with `--gres=gpu:8`.
+
+### Multi-node Usage
+
+There are few NGC containers with Multi-node support
+[available](https://github.com/NVIDIA/DeepLearningExamples).
+Moreover, the realization of the multi-node usage depends on the authors
+of the exact container.
+Thus, right now, it is not possible to run NGC containers with multi-node support
+on the ZIH system without changing the source code inside the container.
diff --git a/doc.zih.tu-dresden.de/docs/software/overview.md b/doc.zih.tu-dresden.de/docs/software/overview.md
index f8f4bf32b66c73234ad6db3cb728662e0d33dd7e..9d2d86d7c06989acfcb9415f908fc1453538b6a8 100644
--- a/doc.zih.tu-dresden.de/docs/software/overview.md
+++ b/doc.zih.tu-dresden.de/docs/software/overview.md
@@ -12,7 +12,7 @@ so called dotfiles in your home directory, e.g., `~/.bashrc` or `~/.bash_profile
## Software Environment
There are different options to work with software on ZIH systems: [modules](#modules),
-[JupyterNotebook](#jupyternotebook) and [containers](#containers). Brief descriptions and related
+[Jupyter Notebook](#jupyternotebook) and [containers](#containers). Brief descriptions and related
links on these options are provided below.
!!! note
@@ -21,16 +21,6 @@ links on these options are provided below.
* `scs5` environment for the x86 architecture based compute resources
* and `ml` environment for the Machine Learning partition based on the Power9 architecture.
-According to [What software do I need]**todo link**, first of all, check the [Software module
-list]**todo link**.
-
-<!--Work with the software on ZIH systems could be started only after allocating the resources by [batch-->
-<!--systems]**todo link**.-->
-
-<!--After logging in, you are on one of the login nodes. They are not meant for work, but only for the-->
-<!--login process and short tests. Allocating resources will be done by batch system-->
-<!--[Slurm](../jobs_and_resources/slurm.md).-->
-
## Modules
Usage of software on ZIH systems, e.g., frameworks, compilers, loader and libraries, is
@@ -47,7 +37,7 @@ The [Jupyter Notebook](https://jupyter.org/) is an open-source web application t
documents containing live code, equations, visualizations, and narrative text. There is a
[JupyterHub](../access/jupyterhub.md) service on ZIH systems, where you can simply run your Jupyter
notebook on compute nodes using [modules](#modules), preloaded or custom virtual environments.
-Moreover, you can run a [manually created remote jupyter server](../archive/install_jupyter.md)
+Moreover, you can run a [manually created remote Jupyter server](../archive/install_jupyter.md)
for more specific cases.
## Containers
diff --git a/doc.zih.tu-dresden.de/docs/software/perf_tools.md b/doc.zih.tu-dresden.de/docs/software/perf_tools.md
index 16007698726b0430f84ef20acc80cb9e1766d64d..83398f49cb68a3255e051ae866a3679124559bef 100644
--- a/doc.zih.tu-dresden.de/docs/software/perf_tools.md
+++ b/doc.zih.tu-dresden.de/docs/software/perf_tools.md
@@ -1,8 +1,8 @@
# Introduction
`perf` consists of two parts: the kernel space implementation and the userland tools. This wiki
-entry focusses on the latter. These tools are installed on taurus, and others and provides support
-for sampling applications and reading performance counters.
+entry focusses on the latter. These tools are installed on ZIH systems, and others and provides
+support for sampling applications and reading performance counters.
## Configuration
@@ -34,18 +34,18 @@ Run `perf stat <Your application>`. This will provide you with a general
overview on some counters.
```Bash
-Performance counter stats for 'ls':=
- 2,524235 task-clock # 0,352 CPUs utilized
- 15 context-switches # 0,006 M/sec
- 0 CPU-migrations # 0,000 M/sec
- 292 page-faults # 0,116 M/sec
- 6.431.241 cycles # 2,548 GHz
- 3.537.620 stalled-cycles-frontend # 55,01% frontend cycles idle
- 2.634.293 stalled-cycles-backend # 40,96% backend cycles idle
- 6.157.440 instructions # 0,96 insns per cycle
- # 0,57 stalled cycles per insn
- 1.248.527 branches # 494,616 M/sec
- 34.044 branch-misses # 2,73% of all branches
+Performance counter stats for 'ls':=
+ 2,524235 task-clock # 0,352 CPUs utilized
+ 15 context-switches # 0,006 M/sec
+ 0 CPU-migrations # 0,000 M/sec
+ 292 page-faults # 0,116 M/sec
+ 6.431.241 cycles # 2,548 GHz
+ 3.537.620 stalled-cycles-frontend # 55,01% frontend cycles idle
+ 2.634.293 stalled-cycles-backend # 40,96% backend cycles idle
+ 6.157.440 instructions # 0,96 insns per cycle
+ # 0,57 stalled cycles per insn
+ 1.248.527 branches # 494,616 M/sec
+ 34.044 branch-misses # 2,73% of all branches
0,007167707 seconds time elapsed
```
@@ -142,10 +142,10 @@ If you added a callchain, it also gives you a callchain profile.\<br /> \*Discla
not an appropriate way to gain exact numbers. So this is merely a rough overview and not guaranteed
to be absolutely correct.\*\<span style="font-size: 1em;"> \</span>
-### On Taurus
+### On ZIH systems
-On Taurus, users are not allowed to see the kernel functions. If you have multiple events defined,
-then the first thing you select in `perf report` is the type of event. Press right
+On ZIH systems, users are not allowed to see the kernel functions. If you have multiple events
+defined, then the first thing you select in `perf report` is the type of event. Press right
```Bash
Available samples
@@ -165,7 +165,7 @@ If you'd select cycles, you would get such a screen:
```Bash
Events: 96 cycles
+ 49,13% test_gcc_perf test_gcc_perf [.] main.omp_fn.0
-+ 34,48% test_gcc_perf test_gcc_perf [.]
++ 34,48% test_gcc_perf test_gcc_perf [.]
+ 6,92% test_gcc_perf test_gcc_perf [.] omp_get_thread_num@plt
+ 5,20% test_gcc_perf libgomp.so.1.0.0 [.] omp_get_thread_num
+ 2,25% test_gcc_perf test_gcc_perf [.] main.omp_fn.1
diff --git a/doc.zih.tu-dresden.de/docs/software/pytorch.md b/doc.zih.tu-dresden.de/docs/software/pytorch.md
index 3c2e88a6c9fc209c246ede0e50410771be541c3f..e84f3aac54a88e0984b0da17e3e3527fe37e7b46 100644
--- a/doc.zih.tu-dresden.de/docs/software/pytorch.md
+++ b/doc.zih.tu-dresden.de/docs/software/pytorch.md
@@ -1,11 +1,11 @@
# PyTorch
-[PyTorch](https://pytorch.org/){:target="_blank"} is an open-source machine learning framework.
+[PyTorch](https://pytorch.org/) is an open-source machine learning framework.
It is an optimized tensor library for deep learning using GPUs and CPUs.
-PyTorch is a machine learning tool developed by Facebooks AI division to process large-scale
+PyTorch is a machine learning tool developed by Facebook's AI division to process large-scale
object detection, segmentation, classification, etc.
PyTorch provides a core data structure, the tensor, a multi-dimensional array that shares many
-similarities with Numpy arrays.
+similarities with NumPy arrays.
Please check the software modules list via
@@ -13,9 +13,9 @@ Please check the software modules list via
marie@login$ module spider pytorch
```
-to find out, which PyTorch modules are available on your partition.
+to find out, which PyTorch modules are available.
-We recommend using partitions alpha and/or ml when working with machine learning workflows
+We recommend using partitions `alpha` and/or `ml` when working with machine learning workflows
and the PyTorch library.
You can find detailed hardware specification in our
[hardware documentation](../jobs_and_resources/hardware_overview.md).
@@ -25,7 +25,8 @@ You can find detailed hardware specification in our
On the partition `alpha`, load the module environment:
```console
-marie@login$ srun -p alpha --gres=gpu:1 -n 1 -c 7 --pty --mem-per-cpu=800 bash #Job submission on alpha nodes with 1 gpu on 1 node with 800 Mb per CPU
+# Job submission on alpha nodes with 1 gpu on 1 node with 800 Mb per CPU
+marie@login$ srun -p alpha --gres=gpu:1 -n 1 -c 7 --pty --mem-per-cpu=800 bash
marie@alpha$ module load modenv/hiera GCC/10.2.0 CUDA/11.1.1 OpenMPI/4.0.5 PyTorch/1.9.0
Die folgenden Module wurden in einer anderen Version erneut geladen:
1) modenv/scs5 => modenv/hiera
@@ -34,6 +35,7 @@ Module GCC/10.2.0, CUDA/11.1.1, OpenMPI/4.0.5, PyTorch/1.9.0 and 54 dependencies
```
??? hint "Torchvision on partition `alpha`"
+
On the partition `alpha`, the module torchvision is not yet available within the module
system. (19.08.2021)
Torchvision can be made available by using a virtual environment:
@@ -45,12 +47,13 @@ Module GCC/10.2.0, CUDA/11.1.1, OpenMPI/4.0.5, PyTorch/1.9.0 and 54 dependencies
```
Using the **--no-deps** option for "pip install" is necessary here as otherwise the PyTorch
- version might be replaced and you will run into trouble with the cuda drivers.
+ version might be replaced and you will run into trouble with the CUDA drivers.
On the partition `ml`:
```console
-marie@login$ srun -p ml --gres=gpu:1 -n 1 -c 7 --pty --mem-per-cpu=800 bash #Job submission in ml nodes with 1 gpu on 1 node with 800 Mb per CPU
+# Job submission in ml nodes with 1 gpu on 1 node with 800 Mb per CPU
+marie@login$ srun -p ml --gres=gpu:1 -n 1 -c 7 --pty --mem-per-cpu=800 bash
```
After calling
@@ -62,8 +65,8 @@ marie@login$ module spider pytorch
we know that we can load PyTorch (including torchvision) with
```console
-marie@ml$ module load modenv/ml torchvision/0.7.0-fosscuda-2019b-Python-3.7.4-PyTorch-1.6.0
-Module torchvision/0.7.0-fosscuda-2019b-Python-3.7.4-PyTorch-1.6.0 and 55 dependencies loaded.
+marie@ml$ module load modenv/ml torchvision/0.7.0-fossCUDA-2019b-Python-3.7.4-PyTorch-1.6.0
+Module torchvision/0.7.0-fossCUDA-2019b-Python-3.7.4-PyTorch-1.6.0 and 55 dependencies loaded.
```
Now, we check that we can access PyTorch:
@@ -75,19 +78,24 @@ marie@{ml,alpha}$ python -c "import torch; print(torch.__version__)"
The following example shows how to create a python virtual environment and import PyTorch.
```console
-marie@ml$ mkdir python-environments #create folder
-marie@ml$ which python #check which python are you using
+# Create folder
+marie@ml$ mkdir python-environments
+# Check which python are you using
+marie@ml$ which python
/sw/installed/Python/3.7.4-GCCcore-8.3.0/bin/python
-marie@ml$ virtualenv --system-site-packages python-environments/env #create virtual environment "env" which inheriting with global site packages
+# Create virtual environment "env" which inheriting with global site packages
+marie@ml$ virtualenv --system-site-packages python-environments/env
[...]
-marie@ml$ source python-environments/env/bin/activate #activate virtual environment "env". Example output: (env) bash-4.2$
+# Activate virtual environment "env". Example output: (env) bash-4.2$
+marie@ml$ source python-environments/env/bin/activate
marie@ml$ python -c "import torch; print(torch.__version__)"
```
## PyTorch in JupyterHub
-In addition to using interactive and batch jobs, it is possible to work with PyTorch using JupyterHub.
-The production and test environments of JupyterHub contain Python kernels, that come with a PyTorch support.
+In addition to using interactive and batch jobs, it is possible to work with PyTorch using
+JupyterHub. The production and test environments of JupyterHub contain Python kernels, that come
+with a PyTorch support.
{: align="center"}
@@ -96,3 +104,62 @@ The production and test environments of JupyterHub contain Python kernels, that
For details on how to run PyTorch with multiple GPUs and/or multiple nodes, see
[distributed training](distributed_training.md).
+
+## Migrate PyTorch-script from CPU to GPU
+
+It is recommended to use GPUs when using large training data sets. While TensorFlow automatically
+uses GPUs if they are available, in PyTorch you have to move your tensors manually.
+
+First, you need to import `torch.CUDA`:
+
+```python3
+import torch.CUDA
+```
+
+Then you define a `device`-variable, which is set to 'CUDA' automatically when a GPU is available
+with this code:
+
+```python3
+device = torch.device('CUDA' if torch.CUDA.is_available() else 'cpu')
+```
+
+You then have to move all of your tensors to the selected device. This looks like this:
+
+```python3
+x_train = torch.FloatTensor(x_train).to(device)
+y_train = torch.FloatTensor(y_train).to(device)
+```
+
+Remember that this does not break backward compatibility when you port the script back to a computer
+without GPU, because without GPU, `device` is set to 'cpu'.
+
+### Caveats
+
+#### Moving Data Back to the CPU-Memory
+
+The CPU cannot directly access variables stored on the GPU. If you want to use the variables, e.g.,
+in a `print` statement or when editing with NumPy or anything that is not PyTorch, you have to move
+them back to the CPU-memory again. This then may look like this:
+
+```python3
+cpu_x_train = x_train.cpu()
+print(cpu_x_train)
+...
+error_train = np.sqrt(metrics.mean_squared_error(y_train[:,1].cpu(), y_prediction_train[:,1]))
+```
+
+Remember that, without `.detach()` before the CPU, if you change `cpu_x_train`, `x_train` will also
+be changed. If you want to treat them independently, use
+
+```python3
+cpu_x_train = x_train.detach().cpu()
+```
+
+Now you can change `cpu_x_train` without `x_train` being affected.
+
+#### Speed Improvements and Batch Size
+
+When you have a lot of very small data points, the speed may actually decrease when you try to train
+them on the GPU. This is because moving data from the CPU-memory to the GPU-memory takes time. If
+this occurs, please try using a very large batch size. This way, copying back and forth only takes
+places a few times and the bottleneck may be reduced.
diff --git a/doc.zih.tu-dresden.de/docs/software/virtual_desktops.md b/doc.zih.tu-dresden.de/docs/software/virtual_desktops.md
index 123c323b2d3acc9c24863ca203179f8338da4dce..cb809c3a99022c8ec5c5d3e9a98b96b8533baa0b 100644
--- a/doc.zih.tu-dresden.de/docs/software/virtual_desktops.md
+++ b/doc.zih.tu-dresden.de/docs/software/virtual_desktops.md
@@ -1,89 +1,3 @@
-# Virtual desktops
+# Virtual Desktops
-Use WebVNC or NICE DCV to run GUI applications on HPC resources.
-
-<span class="twiki-macro TABLE" columnwidths="10%,45%,45%"></span>
-
-| | | |
-|----------------|-------------------------------------------------------|-------------------------------------------------|
-| | **WebVNC** | **NICE DCV** |
-| **use case** | all GUI applications that do \<u>not need\</u> OpenGL | only GUI applications that \<u>need\</u> OpenGL |
-| **partitions** | all\* (except partitions with GPUs (gpu2, hpdlf, ml) | dcv |
-
-## Launch a virtual desktop
-
-<span class="twiki-macro TABLE" columnwidths="10%,45%,45%"></span> \|
-**step 1** \| Navigate to \<a href="<https://taurus.hrsk.tu-dresden.de>"
-target="\_blank"><https://taurus.hrsk.tu-dresden.de>\</a>. There is our
-[JupyterHub](../access/jupyterhub.md) instance. \|\| \| **step 2** \|
-Click on the "advanced" tab and choose a preset: \|\|
-
-| | | |
-|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|
-| ^ | **WebVNC** | **DCV** |
-| **step 3** | Optional: Finetune your session with the available SLURM job parameters or assign a certain project or reservation. Then save your settings in a new preset for future use. | |
-| **step 4** | Click on "Spawn". JupyterHub starts now a SLURM job for you. If everything is ready the JupyterLab interface will appear to you. | |
-| **step 5"** | Click on the **button "WebVNC"** to start a virtual desktop. | Click on the \*button "NICE DCV"to start a virtual desktop. |
-| ^ | The virtual desktop starts in a new tab or window. | |
-
-### Demonstration
-
-\<video controls="" width="320" style="border: 1px solid black">\<source
-src="<https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/VirtualDesktops/start-virtual-desktop-dcv.mp4>"
-type="video/mp4">\<source
-src="<https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/VirtualDesktops/start-virtual-desktop-dcv.webm>"
-type="video/webm">\</video>
-
-### Using the quickstart feature
-
-JupyterHub can start a job automatically if the URL contains certain
-parameters.
-
-<span class="twiki-macro TABLE" columnwidths="10%,45%,45%"></span>
-
-| | | |
-|----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| examples | \<a href="<https://taurus.hrsk.tu-dresden.de/jupyter/hub/spawn#/>\~(partition\~'interactive\~cpuspertask\~'2\~mempercpu\~'2583)" target="\_blank" style="font-size: 1.5em">WebVNC\</a> | \<a href="<https://taurus.hrsk.tu-dresden.de/jupyter/hub/spawn#/>\~(partition\~'dcv\~cpuspertask\~'6\~gres\~'gpu\*3a1\~mempercpu\~'2583)" target="\_blank" style="font-size: 1.5em">NICE DCV\</a> |
-| details about the examples | `interactive` partition, 2 CPUs with 2583 MB RAM per core, no GPU | `dcv` partition, 6 CPUs with 2583 MB RAM per core, 1 GPU |
-| link creator | Use the spawn form to set your preferred options. The browser URL will be updated with the corresponding parameters. | |
-
-If you close the browser tabs or windows or log out from your local
-machine, you are able to open the virtual desktop later again - as long
-as the session runs. But please remember that a SLURM job is running in
-the background which has a certain timelimit.
-
-## Reconnecting to a session
-
-In order to reconnect to an active instance of WebVNC, simply repeat the
-steps required to start a session, beginning - if required - with the
-login, then clicking "My server", then by pressing the "+" sign on the
-upper left corner. Provided your server is still running and you simply
-closed the window or logged out without stopping your server, you will
-find your WebVNC desktop the way you left it.
-
-## Terminate a remote session
-
-<span class="twiki-macro TABLE" columnwidths="10%,90%"></span> \| **step
-1** \| Close the VNC viewer tab or window. \| \| **step 2** \| Click on
-File \> Log Out in the JupyterLab main menu. Now you get redirected to
-the JupyterLab control panel. If you don't have your JupyterLab tab or
-window anymore, navigate directly to \<a
-href="<https://taurus.hrsk.tu-dresden.de/jupyter/hub/home>"
-target="\_blank"><https://taurus.hrsk.tu-dresden.de/jupyter/hub/home>\</a>.
-\| \| **step 3** \| Click on "Stop My Server". This cancels the SLURM
-job and terminates your session. \|
-
-### Demonstration
-
-\<video controls="" width="320" style="border: 1px solid black">\<source
-src="<https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/VirtualDesktops/terminate-virtual-desktop-dcv.mp4>"
-type="video/mp4">\<source
-src="<https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/VirtualDesktops/terminate-virtual-desktop-dcv.webm>"
-type="video/webm">\</video>
-
-**Remark:** This does not work if you click on the "Logout"-Btn in your
-virtual desktop. Instead this will just close your DCV session or cause
-a black screen in your WebVNC window without a possibility to recover a
-virtual desktop in the same Jupyter session. The solution for now would
-be to terminate the whole jupyter session and start a new one like
-mentioned above.
+coming soon
diff --git a/doc.zih.tu-dresden.de/docs/software/virtual_machines.md b/doc.zih.tu-dresden.de/docs/software/virtual_machines.md
index c6c660d3c5ac052f3362ad950f6ad395e4420bdf..2527bbe91cbb735824598cc90311b88df2eab808 100644
--- a/doc.zih.tu-dresden.de/docs/software/virtual_machines.md
+++ b/doc.zih.tu-dresden.de/docs/software/virtual_machines.md
@@ -4,21 +4,21 @@ The following instructions are primarily aimed at users who want to build their
[Singularity](containers.md) containers on ZIH systems.
The Singularity container setup requires a Linux machine with root privileges, the same architecture
-and a compatible kernel. If some of these requirements can not be fulfilled, then there is
-also the option of using the provided virtual machines (VM) on ZIH systems.
+and a compatible kernel. If some of these requirements cannot be fulfilled, then there is also the
+option of using the provided virtual machines (VM) on ZIH systems.
-Currently, starting VMs is only possible on partitions `ml` and HPDLF. The VMs on the ML nodes are
+Currently, starting VMs is only possible on partitions `ml` and `hpdlf`. The VMs on the ML nodes are
used to build singularity containers for the Power9 architecture and the HPDLF nodes to build
Singularity containers for the x86 architecture.
## Create a Virtual Machine
-The `--cloud=kvm` Slurm parameter specifies that a virtual machine should be started.
+The Slurm parameter `--cloud=kvm` specifies that a virtual machine should be started.
### On Power9 Architecture
```console
-marie@login$ srun -p ml -N 1 -c 4 --hint=nomultithread --cloud=kvm --pty /bin/bash
+marie@login$ srun --partition=ml --nodes=1 --cpus-per-task=4 --hint=nomultithread --cloud=kvm --pty /bin/bash
srun: job 6969616 queued and waiting for resources
srun: job 6969616 has been allocated resources
bash-4.2$
@@ -27,7 +27,7 @@ bash-4.2$
### On x86 Architecture
```console
-marie@login$ srun -p hpdlf -N 1 -c 4 --hint=nomultithread --cloud=kvm --pty /bin/bash
+marie@login$ srun --partition=hpdlf --nodes=1 --cpus-per-task=4 --hint=nomultithread --cloud=kvm --pty /bin/bash
srun: job 2969732 queued and waiting for resources
srun: job 2969732 has been allocated resources
bash-4.2$
@@ -35,17 +35,17 @@ bash-4.2$
## Access a Virtual Machine
-Since the a security issue on ZIH systems, we restricted the filesystem permissions. Now you have to
-wait until the file `/tmp/${SLURM_JOB_USER}\_${SLURM_JOB_ID}/activate` is created, then you can try
+After a security issue on ZIH systems, we restricted the filesystem permissions. Now, you have to
+wait until the file `/tmp/${SLURM_JOB_USER}_${SLURM_JOB_ID}/activate` is created. Then, you can try
to connect via `ssh` into the virtual machine, but it could be that the virtual machine needs some
-more seconds to boot and start the SSH daemon. So you may need to try the `ssh` command multiple
+more seconds to boot and accept the connection. So you may need to try the `ssh` command multiple
times till it succeeds.
```console
bash-4.2$ cat /tmp/marie_2759627/activate
#!/bin/bash
-if ! grep -q -- "Key for the VM on the partition ml" "/home/rotscher/.ssh/authorized_keys" >& /dev/null; then
+if ! grep -q -- "Key for the VM on the partition ml" "/home/marie/.ssh/authorized_keys" > /dev/null; then
cat "/tmp/marie_2759627/kvm.pub" >> "/home/marie/.ssh/authorized_keys"
else
sed -i "s|.*Key for the VM on the partition ml.*|ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3siZfQ6vQ6PtXPG0RPZwtJXYYFY73TwGYgM6mhKoWHvg+ZzclbBWVU0OoU42B3Ddofld7TFE8sqkHM6M+9jh8u+pYH4rPZte0irw5/27yM73M93q1FyQLQ8Rbi2hurYl5gihCEqomda7NQVQUjdUNVc6fDAvF72giaoOxNYfvqAkw8lFyStpqTHSpcOIL7pm6f76Jx+DJg98sXAXkuf9QK8MurezYVj1qFMho570tY+83ukA04qQSMEY5QeZ+MJDhF0gh8NXjX/6+YQrdh8TklPgOCmcIOI8lwnPTUUieK109ndLsUFB5H0vKL27dA2LZ3ZK+XRCENdUbpdoG2Czz Key for the VM on the partition ml|" "/home/marie/.ssh/authorized_keys"
@@ -71,7 +71,7 @@ We provide [tools](virtual_machines_tools.md) to automate these steps. You may j
The available space inside the VM can be queried with `df -h`. Currently the whole VM has 8 GB and
with the installed operating system, 6.6 GB of available space.
-Sometimes the Singularity build might fail because of a disk out-of-memory error. In this case it
+Sometimes, the Singularity build might fail because of a disk out-of-memory error. In this case, it
might be enough to delete leftover temporary files from Singularity:
```console
@@ -111,4 +111,4 @@ Bootstraps **shub** and **library** should be avoided.
### Transport Endpoint is not Connected
This happens when the SSHFS mount gets unmounted because it is not very stable. It is sufficient to
-run `\~/mount_host_data.sh` again or just the SSHFS command inside that script.
+run `~/mount_host_data.sh` again or just the SSHFS command inside that script.
diff --git a/doc.zih.tu-dresden.de/docs/software/virtual_machines_tools.md b/doc.zih.tu-dresden.de/docs/software/virtual_machines_tools.md
index 0b03ddf927aeed68d8726797ed04db373d24b9b3..fbec2e51bc453cc17e2d131d7229c50ff90aa23f 100644
--- a/doc.zih.tu-dresden.de/docs/software/virtual_machines_tools.md
+++ b/doc.zih.tu-dresden.de/docs/software/virtual_machines_tools.md
@@ -12,12 +12,12 @@ it to ZIH systems for execution.
**This does not work on the partition `ml`** as it uses the Power9 architecture which your
workstation likely doesn't.
-For this we provide a Virtual Machine (VM) on the partition `ml` which allows users to gain root
-permissions in an isolated environment. The workflow to use this manually is described at
-[this page](virtual_machines.md) but is quite cumbersome.
+For this, we provide a Virtual Machine (VM) on the partition `ml` which allows users to gain root
+permissions in an isolated environment. The workflow to use this manually is described for
+[virtual machines](virtual_machines.md) but is quite cumbersome.
-To make this easier two programs are provided: `buildSingularityImage` and `startInVM` which do what
-they say. The latter is for more advanced use cases so you should be fine using
+To make this easier, two programs are provided: `buildSingularityImage` and `startInVM`, which do
+what they say. The latter is for more advanced use cases, so you should be fine using
`buildSingularityImage`, see the following section.
!!! note "SSH key without password"
@@ -28,43 +28,48 @@ they say. The latter is for more advanced use cases so you should be fine using
**The recommended workflow** is to create and test a definition file locally. You usually start from
a base Docker container. Those typically exist for different architectures but with a common name
(e.g. `ubuntu:18.04`). Singularity automatically uses the correct Docker container for your current
-architecture when building. So in most cases you can write your definition file, build it and test
+architecture when building. So, in most cases, you can write your definition file, build it and test
it locally, then move it to ZIH systems and build it on Power9 (partition `ml`) without any further
changes. However, sometimes Docker containers for different architectures have different suffixes,
in which case you'd need to change that when moving to ZIH systems.
## Build a Singularity Container in a Job
-To build a Singularity container on ZIH systems simply run:
+To build a Singularity container for the power9-architecture on ZIH systems simply run:
```console
marie@login$ buildSingularityImage --arch=power9 myContainer.sif myDefinition.def
```
-This command will submit a batch job and immediately return. Note that while Power9 is currently
-the only supported architecture, the parameter is still required. If you want it to block while the
+To build a singularity image on the x86-architecture, run:
+
+```console
+marie@login$ buildSingularityImage --arch=x86 myContainer.sif myDefinition.def
+```
+
+These commands will submit a batch job and immediately return. If you want it to block while the
image is built and see live output, add the option `--interactive`:
```console
marie@login$ buildSingularityImage --arch=power9 --interactive myContainer.sif myDefinition.def
```
-There are more options available which can be shown by running `buildSingularityImage --help`. All
-have reasonable defaults.The most important ones are:
+There are more options available, which can be shown by running `buildSingularityImage --help`. All
+have reasonable defaults. The most important ones are:
* `--time <time>`: Set a higher job time if the default time is not
enough to build your image and your job is canceled before completing. The format is the same as
for Slurm.
* `--tmp-size=<size in GB>`: Set a size used for the temporary
- location of the Singularity container. Basically the size of the extracted container.
+ location of the Singularity container, basically the size of the extracted container.
* `--output=<file>`: Path to a file used for (log) output generated
while building your container.
* Various Singularity options are passed through. E.g.
`--notest, --force, --update`. See, e.g., `singularity --help` for details.
-For **advanced users** it is also possible to manually request a job with a VM (`srun -p ml
+For **advanced users**, it is also possible to manually request a job with a VM (`srun -p ml
--cloud=kvm ...`) and then use this script to build a Singularity container from within the job. In
-this case the `--arch` and other Slurm related parameters are not required. The advantage of using
+this case, the `--arch` and other Slurm related parameters are not required. The advantage of using
this script is that it automates the waiting for the VM and mounting of host directories into it
(can also be done with `startInVM`) and creates a temporary directory usable with Singularity inside
the VM controlled by the `--tmp-size` parameter.
@@ -73,21 +78,22 @@ the VM controlled by the `--tmp-size` parameter.
**Read here if you have problems like "File not found".**
-As the build starts in a VM you may not have access to all your files. It is usually bad practice
+As the build starts in a VM, you may not have access to all your files. It is usually bad practice
to refer to local files from inside a definition file anyway as this reduces reproducibility.
-However common directories are available by default. For others, care must be taken. In short:
+However, common directories are available by default. For others, care must be taken. In short:
-* `/home/$USER`, `/scratch/$USER` are available and should be used `/scratch/\<group>` also works for
-* all groups the users is in `/projects/\<group>` similar, but is read-only! So don't use this to
- store your generated container directly, but rather move it here afterwards
-* /tmp is the VM local temporary directory. All files put here will be lost!
+* `/home/$USER`, `/scratch/$USER` are available and should be used `/scratch/<group>` also works for
+ all groups the users is in
+* `/projects/<group>` similar, but is read-only! So don't use this to store your generated
+ container directly, but rather move it here afterwards
+* `/tmp` is the VM local temporary directory. All files put here will be lost!
If the current directory is inside (or equal to) one of the above (except `/tmp`), then relative paths
for container and definition work as the script changes to the VM equivalent of the current
-directory. Otherwise you need to use absolute paths. Using `~` in place of `$HOME` does work too.
+directory. Otherwise, you need to use absolute paths. Using `~` in place of `$HOME` does work too.
-Under the hood, the filesystem of ZIH systems is mounted via SSHFS at `/host_data`, so if you need any
-other files they can be found there.
+Under the hood, the filesystem of ZIH systems is mounted via SSHFS at `/host_data`. So if you need any
+other files, they can be found there.
There is also a new SSH key named `kvm` which is created by the scripts and authorized inside the VM
to allow for password-less access to SSHFS. This is stored at `~/.ssh/kvm` and regenerated if it
@@ -98,26 +104,32 @@ needs to be re-generated on every script run.
## Start a Job in a VM
-Especially when developing a Singularity definition file it might be useful to get a shell directly
-on a VM. To do so simply run:
+Especially when developing a Singularity definition file, it might be useful to get a shell directly
+on a VM. To do so on the power9-architecture, simply run:
```console
startInVM --arch=power9
```
+To do so on the x86-architecture, run:
+
+```console
+startInVM --arch=x86
+```
+
This will execute an `srun` command with the `--cloud=kvm` parameter, wait till the VM is ready,
mount all folders (just like `buildSingularityImage`, see the Filesystem section above) and come
back with a bash inside the VM. Inside that you are root, so you can directly execute `singularity
build` commands.
-As usual more options can be shown by running `startInVM --help`, the most important one being
+As usual, more options can be shown by running `startInVM --help`, the most important one being
`--time`.
There are two special use cases for this script:
1. Execute an arbitrary command inside the VM instead of getting a bash by appending the command to
- the script. Example: `startInVM --arch=power9 singularity build \~/myContainer.sif \~/myDefinition.de`
+ the script. Example: `startInVM --arch=power9 singularity build ~/myContainer.sif ~/myDefinition.de`
1. Use the script in a job manually allocated via srun/sbatch. This will work the same as when
running outside a job but will **not** start a new job. This is useful for using it inside batch
- scripts, when you already have an allocation or need special arguments for the job system. Again
+ scripts, when you already have an allocation or need special arguments for the job system. Again,
you can run an arbitrary command by passing it to the script.
diff --git a/doc.zih.tu-dresden.de/hackathon.md b/doc.zih.tu-dresden.de/hackathon.md
deleted file mode 100644
index d41781c45455139c62708c708cf42e05babc3b65..0000000000000000000000000000000000000000
--- a/doc.zih.tu-dresden.de/hackathon.md
+++ /dev/null
@@ -1,75 +0,0 @@
-# Hackathon June 2021
-
-The goals for the hackathon are:
-
-* Familiarize main editors (ZIH admin group and domain experts) with new workflow and system
-* Bringing new compendium to life by
- 1. Transferring content from old compendium into new structure and system
- 1. Fixing checks
- 1. Reviewing and updating transferred content
-
-## twiki2md
-
-The script `twiki2md` converts twiki source files into markdown source files using pandoc. It
-outputs the markdown source files according to the old pages tree into subdirectories. The
-output and **starting point for transferring** old content into the new system can be found
-at branch `preview` within directory `twiki2md/root/`.
-
-## Steps
-
-### Familiarize with New Wiki System
-
-* Make sure your are member of the [repository](https://gitlab.hrz.tu-chemnitz.de/zih/hpcsupport/hpc-compendium).
- If not, ask Danny Rotscher for adding you.
-* Clone repository and checkout branch `preview`
-
-```Shell Session
-~ git clone git@gitlab.hrz.tu-chemnitz.de:zih/hpcsupport/hpc-compendium.git
-~ cd hpc-compendium
-~ git checkout preview
-```
-
-* Open terminal and build documentation using `mkdocs`
- * [using mkdocs](README.md#preview-using-mkdocs)
- * [installing dependencies](README.md#install-dependencies)
-
-### Transferring Content
-
-1. Grab a markdown source file from `twiki2md/root/` directory (a topic you are comfortable with)
-1. Find place in new structure according to
-[Typical Project Schedule](https://doc.zih.tu-dresden.de/hpc-wiki/bin/view/Compendium/TypicalProjectSchedule)
-
- * Create new feature branch holding your work `~ git checkout -b <BRANCHNAME>`, whereas branch
- name can be `<FILENAME>` for simplicity
- * Copy reviewed markdown source file to `docs/` directory via
- `~ git mv twiki2md/root/<FILENAME>.md doc.zih.tu-dresden.de/docs/<SUBDIR>/<FILENAME>.md`
- * Update navigation section in `mkdocs.yaml`
-
-1. Commit and push to feature branch via
-
-```Shell Session
-~ git commit docs/<SUBDIR>/<FILENAME>.md mkdocs.yaml -m "MESSAGE"
-~ git push origin <BRANCHNAME>
-```
-
-1. Run checks locally and fix the issues. Otherwise the pipeline will fail.
- * [Check links](README.md#check-links) (There might be broken links which can only be solved
- with ongoing transfer of content.)
- * [Check pages structure](README.md#check-pages-structure)
- * [Markdown Linter](README.md#markdown-linter)
-1. Create
- [merge request](https://gitlab.hrz.tu-chemnitz.de/zih/hpcsupport/hpc-compendium/-/merge_requests)
- against `preview` branch
-
-### Review Content
-
-The following steps are optional in a sense, that the first goal of the hackathon is to transfer all
-old pages into new structure. If this is done, the content of the files need to be reviewed:
-
- * Remove outdated information
- * Update content
- * Apply [writing style](README.md#writing-style)
- * Replace or remove (leftover) html constructs in markdown source file
- * Add ticks for code blocks and command if necessary
- * Fix internal links (mark as todo if necessary)
- * Review and update, remove outdated content
diff --git a/doc.zih.tu-dresden.de/mkdocs.yml b/doc.zih.tu-dresden.de/mkdocs.yml
index 06f4024a6cc4800f28fd967da04f97bc7babeae9..9c30346df2e9f98d5aef67229cd45902ea3bbf42 100644
--- a/doc.zih.tu-dresden.de/mkdocs.yml
+++ b/doc.zih.tu-dresden.de/mkdocs.yml
@@ -32,6 +32,7 @@ nav:
- Singularity Recipes and Hints: software/singularity_recipe_hints.md
- Virtual Machines Tools: software/virtual_machines_tools.md
- Virtual Machines: software/virtual_machines.md
+ - NGC Containers: software/ngc_containers.md
- Applications:
- Licenses: software/licenses.md
- Computational Fluid Dynamics (CFD): software/cfd.md
@@ -64,6 +65,7 @@ nav:
- Debugging: software/debuggers.md
- MPI Error Detection: software/mpi_usage_error_detection.md
- Score-P: software/scorep.md
+ - lo2s: software/lo2s.md
- PAPI Library: software/papi.md
- Pika: software/pika.md
- Perf Tools: software/perf_tools.md
diff --git a/doc.zih.tu-dresden.de/requirements.txt b/doc.zih.tu-dresden.de/requirements.txt
deleted file mode 100644
index 272b09c7c7ffb6b945eaa66e14e2e695f5502f17..0000000000000000000000000000000000000000
--- a/doc.zih.tu-dresden.de/requirements.txt
+++ /dev/null
@@ -1,6 +0,0 @@
-# Documentation static site generator & deployment tool
-mkdocs>=1.1.2
-
-# Add custom theme if not inside a theme_dir
-# (https://github.com/mkdocs/mkdocs/wiki/MkDocs-Themes)
-mkdocs-material>=7.1.0
diff --git a/doc.zih.tu-dresden.de/util/check-bash-syntax.sh b/doc.zih.tu-dresden.de/util/check-bash-syntax.sh
new file mode 100755
index 0000000000000000000000000000000000000000..9f31effee3ebc3380af5ca892047aca6a9357139
--- /dev/null
+++ b/doc.zih.tu-dresden.de/util/check-bash-syntax.sh
@@ -0,0 +1,79 @@
+#!/bin/bash
+
+set -euo pipefail
+
+scriptpath=${BASH_SOURCE[0]}
+basedir=`dirname "$scriptpath"`
+basedir=`dirname "$basedir"`
+
+function usage () {
+ echo "$0 [options]"
+ echo "Search for bash files that have an invalid syntax."
+ echo ""
+ echo "Options:"
+ echo " -a Search in all bash files (default: git-changed files)"
+ echo " -f=FILE Search in a specific bash file"
+ echo " -s Silent mode"
+ echo " -h Show help message"
+}
+
+# Options
+all_files=false
+silent=false
+file=""
+while getopts ":ahsf:" option; do
+ case $option in
+ a)
+ all_files=true
+ ;;
+ f)
+ file=$2
+ shift
+ ;;
+ s)
+ silent=true
+ ;;
+ h)
+ usage
+ exit;;
+ \?) # Invalid option
+ echo "Error: Invalid option."
+ usage
+ exit;;
+ esac
+done
+
+branch="origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME:-preview}"
+
+if [ $all_files = true ]; then
+ echo "Search in all bash files."
+ files=`git ls-tree --full-tree -r --name-only HEAD $basedir/docs/ | grep .sh || true`
+elif [[ ! -z $file ]]; then
+ files=$file
+else
+ echo "Search in git-changed files."
+ files=`git diff --name-only "$(git merge-base HEAD "$branch")" | grep .sh || true`
+fi
+
+
+cnt=0
+for f in $files; do
+ if ! bash -n $f; then
+ if [ $silent = false ]; then
+ echo "Bash file $f has invalid syntax"
+ fi
+ ((cnt=cnt+1))
+ fi
+done
+
+case $cnt in
+ 1)
+ echo "Bash files with invalid syntax: 1 match found"
+ ;;
+ *)
+ echo "Bash files with invalid syntax: $cnt matches found"
+ ;;
+esac
+if [ $cnt -gt 0 ]; then
+ exit 1
+fi
diff --git a/doc.zih.tu-dresden.de/util/check-links.sh b/doc.zih.tu-dresden.de/util/check-links.sh
index e553f9c4828a2286a5f053181dd09eaaa28746ad..49ee7f2dbe876ed8e82c0706631db3479c51fb16 100755
--- a/doc.zih.tu-dresden.de/util/check-links.sh
+++ b/doc.zih.tu-dresden.de/util/check-links.sh
@@ -42,13 +42,18 @@ fi
any_fails=false
files=$(git diff --name-only "$(git merge-base HEAD "$branch")")
+echo "Check files:"
+echo "$files"
+echo ""
for f in $files; do
if [ "${f: -3}" == ".md" ]; then
# do not check links for deleted files
- if [ -e x.txt ]; then
- echo "Checking links for $f"
- if ! $mlc -q -p "$f"; then
- any_fails=true
+ if [ "$f" != "doc.zih.tu-dresden.de/README.md" ]; then
+ if [ -e $f ]; then
+ echo "Checking links for $f"
+ if ! $mlc -q -p "$f"; then
+ any_fails=true
+ fi
fi
fi
fi
diff --git a/doc.zih.tu-dresden.de/util/check-spelling.sh b/doc.zih.tu-dresden.de/util/check-spelling.sh
index 7fa9d2824d4a61ce86ae258d656acfe90c574269..8448d0bbffe534b0fd676dbd00ca82e17e7d167d 100755
--- a/doc.zih.tu-dresden.de/util/check-spelling.sh
+++ b/doc.zih.tu-dresden.de/util/check-spelling.sh
@@ -70,7 +70,8 @@ function isMistakeCountIncreasedByChanges(){
fi
if [ $current_count -gt $previous_count ]; then
echo "-- File $newfile"
- echo "Change increases spelling mistake count (from $previous_count to $current_count)"
+ echo "Change increases spelling mistake count (from $previous_count to $current_count), misspelled/unknown words:"
+ cat "$newfile" | getAspellOutput
any_fails=true
fi
fi
diff --git a/doc.zih.tu-dresden.de/util/grep-forbidden-words.sh b/doc.zih.tu-dresden.de/util/grep-forbidden-words.sh
index 456eb55e192634bf4e159ce0096c83076989f2fc..cfb2b91b57457b701c5b80e76c6346d460cf4602 100755
--- a/doc.zih.tu-dresden.de/util/grep-forbidden-words.sh
+++ b/doc.zih.tu-dresden.de/util/grep-forbidden-words.sh
@@ -18,12 +18,16 @@ i file \+system HDFS
i \<taurus\> taurus\.hrsk /taurus /TAURUS
i \<hrskii\>
i hpc[ -]\+da\>
+i attachurl
+i \<todo\> <!--.*todo.*-->
+i [[:space:]]$
i \(alpha\|ml\|haswell\|romeo\|gpu\|smp\|julia\|hpdlf\|scs5\)-\?\(interactive\)\?[^a-z]*partition
+i \[\s\?\(documentation\|here\|this \(link\|page\|subsection\)\|slides\?\|manpage\)\s\?\]
i work[ -]\+space"
# Whitelisted files will be ignored
# Whitespace separated list with full path
-whitelist=(doc.zih.tu-dresden.de/docs/contrib/content_rules.md)
+whitelist=(doc.zih.tu-dresden.de/README.md doc.zih.tu-dresden.de/docs/contrib/content_rules.md doc.zih.tu-dresden.de/docs/access/ssh_login.md)
function grepExceptions () {
if [ $# -gt 0 ]; then
@@ -44,13 +48,15 @@ function usage () {
echo " -f Search in a specific markdown file"
echo " -s Silent mode"
echo " -h Show help message"
+ echo " -c Show git matches in color"
}
# Options
all_files=false
silent=false
file=""
-while getopts ":ahsf:" option; do
+color=""
+while getopts ":ahsf:c" option; do
case $option in
a)
all_files=true
@@ -62,6 +68,9 @@ while getopts ":ahsf:" option; do
s)
silent=true
;;
+ c)
+ color=" --color=always "
+ ;;
h)
usage
exit;;
@@ -87,7 +96,7 @@ fi
echo "... $files ..."
cnt=0
for f in $files; do
- if [ "$f" != doc.zih.tu-dresden.de/README.md -a "${f: -3}" == ".md" -a -f "$f" ]; then
+ if [ "${f: -3}" == ".md" -a -f "$f" ]; then
if (printf '%s\n' "${whitelist[@]}" | grep -xq $f); then
echo "Skip whitelisted file $f"
continue
@@ -104,7 +113,7 @@ for f in $files; do
grepflag=-i
;;
esac
- if grep -n $grepflag "$pattern" "$f" | grepExceptions "${exceptionPatternsArray[@]}" ; then
+ if grep -n $grepflag $color "$pattern" "$f" | grepExceptions "${exceptionPatternsArray[@]}" ; then
((cnt=cnt+1))
fi
done <<< $exceptionPatterns
diff --git a/doc.zih.tu-dresden.de/util/pre-commit b/doc.zih.tu-dresden.de/util/pre-commit
index 043320f352b923a7e7be96c04de5914960285b65..b86b75d9a07870a68118aa500ee80781e216c56b 100755
--- a/doc.zih.tu-dresden.de/util/pre-commit
+++ b/doc.zih.tu-dresden.de/util/pre-commit
@@ -1,7 +1,4 @@
#!/bin/bash
-exit_ok=yes
-files=$(git diff-index --cached --name-only HEAD)
-
function testPath(){
path_to_test=doc.zih.tu-dresden.de/docs/$1
test -f "$path_to_test" || echo $path_to_test does not exist
@@ -15,29 +12,39 @@ fi
export -f testPath
-for file in $files
-do
- if [ $file == doc.zih.tu-dresden.de/mkdocs.yml ]
+exit_ok=yes
+branch="origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME:-preview}"
+if [ -f "$GIT_DIR/MERGE_HEAD" ]
+then
+ source_hash=`git merge-base HEAD "$branch"`
+else
+ source_hash=`git rev-parse HEAD`
+fi
+#Remove everything except lines beginning with --- or +++
+files=`git diff $source_hash | sed -E -n 's#^(---|\+\+\+) ((/|./)[^[:space:]]+)$#\2#p'`
+#Assume that we have pairs of lines (starting with --- and +++).
+while read oldfile; do
+ read newfile
+
+ if [ "$newfile" == doc.zih.tu-dresden.de/mkdocs.yml ]
then
- echo Testing $file
+ echo Testing "$newfile"
sed -n '/^ *- /s#.*: \([A-Za-z_/]*.md\).*#\1#p' doc.zih.tu-dresden.de/mkdocs.yml | xargs -L1 -I {} bash -c "testPath '{}'"
if [ $? -ne 0 ]
then
exit_ok=no
fi
- elif [[ $file =~ ^doc.zih.tu-dresden.de/(.*.md)$ ]]
+ elif [[ "$newfile" =~ ^b/doc.zih.tu-dresden.de/(.*.md)$ ]]
then
filepattern=${BASH_REMATCH[1]}
- #lint
- echo "Checking linter..."
+ echo "Linting..."
docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)"/doc.zih.tu-dresden.de,target=/docs,type=bind hpc-compendium markdownlint $filepattern
if [ $? -ne 0 ]
then
exit_ok=no
fi
- #link-check
echo "Checking links..."
docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)"/doc.zih.tu-dresden.de,target=/docs,type=bind hpc-compendium markdown-link-check $filepattern
if [ $? -ne 0 ]
@@ -45,9 +52,15 @@ do
exit_ok=no
fi
fi
-done
+done <<< "$files"
+
+echo "Testing syntax of bash files..."
+docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)",target=/docs,type=bind hpc-compendium ./doc.zih.tu-dresden.de/util/check-bash-syntax.sh
+if [ $? -ne 0 ]
+then
+ exit_ok=no
+fi
-#spell-check
echo "Spell-checking..."
docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)",target=/docs,type=bind hpc-compendium ./doc.zih.tu-dresden.de/util/check-spelling.sh
if [ $? -ne 0 ]
@@ -55,7 +68,6 @@ then
exit_ok=no
fi
-#forbidden words checking
echo "Forbidden words checking..."
docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)",target=/docs,type=bind hpc-compendium ./doc.zih.tu-dresden.de/util/grep-forbidden-words.sh
if [ $? -ne 0 ]
diff --git a/doc.zih.tu-dresden.de/wordlist.aspell b/doc.zih.tu-dresden.de/wordlist.aspell
index e484d4e4a72df08f9b53a99fe91ac49f483c6693..6be6d6bef7ea9fa52fd4892c1e7cc65fece62bc9 100644
--- a/doc.zih.tu-dresden.de/wordlist.aspell
+++ b/doc.zih.tu-dresden.de/wordlist.aspell
@@ -1,12 +1,16 @@
personal_ws-1.1 en 203
Abaqus
+ALLREDUCE
Altix
Amber
Amdahl's
analytics
+Analytics
anonymized
+Ansys
APIs
AVX
+awk
BeeGFS
benchmarking
BLAS
@@ -17,28 +21,43 @@ CCM
ccNUMA
centauri
CentOS
+CFX
cgroups
checkpointing
Chemnitz
citable
CLI
+CMake
+COMSOL
conda
+config
+CONFIG
+cpu
CPU
CPUID
+cpus
CPUs
+crossentropy
css
CSV
CUDA
cuDNN
CXFS
dask
+Dask
+dataframes
DataFrames
dataframes
datamover
DataParallel
+dataset
+DCV
+ddl
DDP
DDR
DFG
+dir
+distr
DistributedDataParallel
DMTCP
DNS
@@ -46,6 +65,8 @@ Dockerfile
Dockerfiles
DockerHub
dockerized
+dotfile
+dotfiles
EasyBuild
ecryptfs
engl
@@ -54,6 +75,7 @@ env
EPYC
Espresso
ESSL
+facto
fastfs
FFT
FFTW
@@ -76,10 +98,14 @@ GitHub
GitLab
GitLab's
glibc
+Gloo
gnuplot
+gpu
GPU
GPUs
+gres
GROMACS
+GUIs
hadoop
haswell
HBM
@@ -87,13 +113,16 @@ HDF
HDFS
HDFView
hiera
+horovod
Horovod
+horovodrun
hostname
Hostnames
HPC
HPE
HPL
html
+hvd
hyperparameter
hyperparameters
hyperthreading
@@ -101,8 +130,11 @@ icc
icpc
ifort
ImageNet
+img
Infiniband
+init
inode
+IOPS
IPs
ISA
Itanium
@@ -114,9 +146,11 @@ JupyterHub
JupyterLab
Keras
KNL
+Kunststofftechnik
LAMMPS
LAPACK
lapply
+Leichtbau
LINPACK
Linter
linter
@@ -131,7 +165,9 @@ MathKernel
MathWorks
matlab
MEGWARE
+mem
MiB
+Microarchitecture
MIMD
Miniconda
mkdocs
@@ -143,25 +179,31 @@ modulefile
Montecito
mountpoint
mpi
-mpiCC
+Mpi
mpicc
+mpiCC
mpicxx
mpif
mpifort
mpirun
multicore
+multiphysics
+Multiphysics
multithreaded
MultiThreading
Multithreading
NAMD
natively
nbsp
-nbsp
NCCL
Neptun
NFS
NGC
+nodelist
+NODELIST
NRINGS
+ntasks
+NUM
NUMA
NUMAlink
NumPy
@@ -182,6 +224,7 @@ OpenMPI
openmpi
OpenSSH
Opteron
+OTF
overfitting
Pandarallel
pandarallel
@@ -190,7 +233,9 @@ parallelization
parallelize
parfor
pdf
+perf
Perf
+performant
PESSL
PGI
PiB
@@ -205,9 +250,14 @@ pre
Preload
preloaded
preloading
+preprocessing
PSOCK
Pthreads
+pty
pymdownx
+PythonAnaconda
+pytorch
+PyTorch
Quantum
queue
randint
@@ -215,6 +265,8 @@ reachability
README
reproducibility
requeueing
+resnet
+ResNet
RHEL
Rmpi
rome
@@ -226,11 +278,13 @@ Rsync
runnable
Runtime
runtime
+sacct
salloc
Sandybridge
Saxonid
sbatch
ScaDS
+scalability
scalable
ScaLAPACK
Scalasca
@@ -247,6 +301,7 @@ SHA
SHMEM
SLES
Slurm
+SLURMCluster
SMP
SMT
squeue
@@ -265,6 +320,7 @@ SXM
TBB
TCP
TensorBoard
+tensorflow
TensorFlow
TFLOPS
Theano
@@ -273,10 +329,14 @@ ToDo
todo
toolchain
toolchains
+torchvision
+Torchvision
tracefile
tracefiles
+tracepoints
transferability
Trition
+undistinguishable
unencrypted
uplink
userspace
diff --git a/twiki2md/root/Applications/Electrodynamics.md b/twiki2md/root/Applications/Electrodynamics.md
deleted file mode 100644
index c7b99613df2abfbad8b708d27cae01925a382925..0000000000000000000000000000000000000000
--- a/twiki2md/root/Applications/Electrodynamics.md
+++ /dev/null
@@ -1,17 +0,0 @@
-# Electromagnetic Field Simulation
-
-The following applications are installed at ZIH:
-
-| | | |
-|----------|----------|------------|
-| | **Mars** | **Deimos** |
-| **HFSS** | | 11.0.2 |
-
-## HFSS
-
-[HFSS](http://www.ansoft.com/products/hf/hfss/) is the industry-standard
-simulation tool for 3D full-wave electromagnetic field simulation. HFSS
-provides E- and H-fields, currents, S-parameters and near and far
-radiated field results.
-
--- Main.mark - 2010-01-06
diff --git a/twiki2md/root/Applications/SoftwareModulesList.md b/twiki2md/root/Applications/SoftwareModulesList.md
deleted file mode 100644
index 70feb25fc836e0cee838889ef15cc9ca284d9cfa..0000000000000000000000000000000000000000
--- a/twiki2md/root/Applications/SoftwareModulesList.md
+++ /dev/null
@@ -1,962 +0,0 @@
-### SCS5 Environment
-
-<span class="twiki-macro TABLE">headerrows= 1"</span> <span
-class="twiki-macro EDITTABLE"
-format="| text, 30, Software | text, 40, Kategorie | text, 30, Letzte Änderung | text, 30, SGI-UV | text, 30, Taurus | "
-changerows="on"></span>
-
-| Software | Category | Last change | Venus | Taurus |
-|:--------------------------|:----------|:------------|:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| ABAQUS | cae | 2020-04-30 | \- | 2019 |
-| ABINIT | chem | 2019-07-12 | \- | 8.6.3\<br />8.10.3 |
-| ACE | lib | 2018-11-22 | \- | 6.5.1 |
-| ACTC | lib | 2018-11-22 | \- | 1.1 |
-| AFNI | bio | 2018-11-22 | \- | 20180521 |
-| AMDLibM | perf | 2019-08-20 | \- | 3.4.0 |
-| AMDuProf | perf | 2020-11-26 | \- | 3.3.462\<br />3.1.35 |
-| ANSA | cae | 2021-04-08 | \- | 20.1.4 |
-| ANSYS | tools | 2020-10-23 | \- | 2020R2\<br />19.5\<br />19.2 |
-| ANTLR | tools | 2020-09-29 | \- | 2.7.7 |
-| ANTs | data | 2020-10-06 | \- | 2.3.4 |
-| APR | tools | 2019-11-08 | \- | 1.7.0\<br />1.6.3 |
-| APR-util | tools | 2019-11-08 | \- | 1.6.1 |
-| ASE | chem | 2020-09-29 | \- | 3.19.0\<br />3.18.1\<br />3.16.2 |
-| ATK | vis | 2020-09-02 | \- | 2.34.1\<br />2.28.1 |
-| Advisor | perf | 2018-11-22 | \- | 2018 |
-| Anaconda3 | lang | 2019-10-17 | \- | 2019.03 |
-| AnsysEM | phys | 2018-11-22 | \- | 19.0 |
-| Arrow | data | 2020-12-01 | \- | 0.16.0\<br />0.14.1 |
-| AtomPAW | chem | 2019-09-11 | \- | 4.1.0.6 |
-| Autoconf | devel | 2021-02-17 | \- | 2.69 |
-| Automake | devel | 2021-02-17 | \- | 1.16.2\<br />1.16.1\<br />1.15.1\<br />1.15 |
-| Autotools | devel | 2021-02-17 | \- | 20200321\<br />20180311\<br />20170619\<br />20150215 |
-| Bazel | devel | 2021-03-19 | \- | 3.7.2\<br />3.7.1\<br />3.4.1\<br />0.29.1\<br />0.26.1\<br />0.20.0\<br />0.16.0\<br />0.12.0 |
-| BigDataFrameworkConfigure | devel | 2019-09-16 | \- | 0.0.2\<br />0.0.1 |
-| Bison | lang | 2020-12-11 | \- | 3.7.1\<br />3.5.3\<br />3.3.2\<br />3.0.5\<br />3.0.4\<br />2.7 |
-| Blitz++ | lib | 2019-04-09 | \- | 0.10 |
-| Boost | devel | 2021-04-09 | \- | 1.74.0\<br />1.72.0\<br />1.71.0\<br />1.70.0\<br />1.69.0\<br />1.68.0\<br />1.67.0\<br />1.66.0\<br />1.61.0\<br />1.55.0 |
-| Boost.Python | lib | 2019-02-26 | \- | 1.66.0 |
-| CDO | data | 2020-10-06 | \- | 1.9.8\<br />1.9.4 |
-| CFITSIO | lib | 2019-06-03 | \- | 3.45 |
-| CGAL | numlib | 2020-08-20 | \- | 4.14.3\<br />4.14.1\<br />4.11.1 |
-| CGNS | cae | 2020-10-09 | \- | 3.3.1 |
-| CMake | devel | 2021-02-17 | \- | 3.9.5\<br />3.9.1\<br />3.8.0\<br />3.18.4\<br />3.16.4\<br />3.15.3\<br />3.14.5\<br />3.13.3\<br />3.12.1\<br />3.11.4\<br />3.10.2\<br />3.10.1\<br />3.10.0 |
-| COMSOL | phys | 2021-02-10 | \- | 5.6\<br />5.5\<br />5.4 |
-| CP2K | chem | 2019-12-12 | \- | 6.1\<br />5.1 |
-| CUDA | system | 2021-02-17 | \- | 9.2.88\<br />9.1.85\<br />11.1.1\<br />11.0.2\<br />10.1.243\<br />10.0.130 |
-| CUDAcore | system | 2021-02-17 | \- | 11.1.1\<br />11.0.2 |
-| Check | lib | 2021-02-17 | \- | 0.15.2 |
-| Clang | compiler | 2020-12-09 | \- | 9.0.1\<br />5.0.0 |
-| ClustalW2 | bio | 2018-11-22 | \- | 2.1 |
-| CubeGUI | perf | 2021-01-29 | \- | 4.4.4\<br />4.4 |
-| CubeLib | perf | 2020-10-12 | \- | 4.4.4\<br />4.4 |
-| CubeW | perf | 2018-11-22 | \- | 4.4 |
-| CubeWriter | perf | 2020-10-12 | \- | 4.4.3 |
-| Cython | lang | 2018-11-22 | \- | 0.28.5\<br />0.28.2 |
-| DAMASK | phys | 2020-12-10 | \- | 3.0.0\<br />2.0.3-2992\<br />2.0.3 |
-| DASH | lib | 2018-11-22 | \- | dash |
-| DB | tools | 2021-02-17 | \- | 18.1.40 |
-| DBus | devel | 2020-08-11 | \- | 1.13.8\<br />1.13.6\<br />1.13.12 |
-| DFTB+ | phys | 2021-04-12 | \- | 19.1\<br />18.2\<br />18.1 |
-| DMTCP | tools | 2018-12-06 | \- | 2.5.2\<br />2.5.1 |
-| DOLFIN | math | 2020-12-21 | \- | 2019.1.0\<br />2018.1.0.post1\<br />2017.2.0 |
-| Delft3D | geo | 2020-01-29 | \- | 6.03 |
-| Devel-NYTProf | perf | 2019-08-30 | \- | 6.06 |
-| Doxygen | devel | 2021-03-03 | \- | 1.8.20\<br />1.8.17\<br />1.8.16\<br />1.8.15\<br />1.8.14\<br />1.8.13 |
-| Dyninst | tools | 2019-01-21 | \- | 9.3.2 |
-| ELPA | math | 2021-01-05 | \- | 2019.11.001\<br />2018.11.001\<br />2016.11.001.pre |
-| ELSI | math | 2020-02-27 | \- | 2.5.0 |
-| EMBOSS | bio | 2019-02-11 | \- | 6.6.0 |
-| ESPResSo | phys | 2019-11-19 | \- | 4.1.1 |
-| ETSF_IO | lib | 2020-10-15 | \- | 1.0.4 |
-| EasyBuild | tools | 2021-03-08 | \- | 4.3.3\<br />4.3.2\<br />4.3.1\<br />4.3.0\<br />4.2.2\<br />4.2.0\<br />4.1.2\<br />4.1.1\<br />4.1.0\<br />4.0.1\<br />3.9.4\<br />3.9.3\<br />3.9.2\<br />3.9.1\<br />3.8.1\<br />3.8.0\<br />3.7.1\<br />3.7.0\<br />3.6.2\<br />3.6.1 |
-| Eigen | math | 2021-02-17 | \- | 3.3.8\<br />3.3.7\<br />3.3.4 |
-| Emacs | tools | 2020-02-20 | \- | 25.3 |
-| ErlangOTP | lang | 2019-03-13 | \- | 21.3-no |
-| FFC | math | 2019-02-26 | \- | 2018.1.0 |
-| FFTW | numlib | 2021-02-17 | \- | 3.3.8\<br />3.3.7\<br />3.3.4 |
-| FFmpeg | vis | 2020-08-11 | \- | 4.2.2\<br />4.2.1\<br />4.1.3\<br />4.1\<br />3.4.2 |
-| FIAT | math | 2019-02-26 | \- | 2018.1.0 |
-| FLANN | lib | 2019-08-13 | \- | 1.8.4 |
-| FLTK | vis | 2019-11-08 | \- | 1.3.5\<br />1.3.4 |
-| FSL | bio | 2020-10-30 | \- | 6.0.2\<br />5.0.11 |
-| Flink | devel | 2019-09-16 | \- | 1.9.0\<br />1.8.1 |
-| FoX | lib | 2018-11-22 | \- | 4.1.2 |
-| FreeSurfer | bio | 2020-07-22 | \- | 7.1.1\<br />6.0.0\<br />5.3.0\<br />5.1.0 |
-| FriBidi | vis | 2020-08-11 | \- | 1.0.9\<br />1.0.5\<br />1.0.1 |
-| GAMS | math | 2018-11-28 | \- | 25.1.3 |
-| GCC | compiler | 2020-12-11 | \- | 9.3.0\<br />9.1.0-2.32\<br />8.3.0\<br />8.2.0-2.31.1\<br />8.2.0-2.30\<br />7.3.0-2.30\<br />6.4.0-2.28\<br />10.2.0 |
-| GCCcore | compiler | 2020-12-11 | \- | 9.3.0\<br />9.1.0\<br />8.3.0\<br />8.2.0\<br />7.3.0\<br />6.4.0\<br />6.3.0\<br />10.2.0 |
-| GCL | compiler | 2018-11-22 | \- | 2.6.12 |
-| GDAL | data | 2020-10-06 | \- | 3.0.2\<br />3.0.0\<br />2.2.3 |
-| GDB | debugger | 2020-08-11 | \- | 9.1\<br />8.1 |
-| GDRCopy | lib | 2021-02-17 | \- | 2.1 |
-| GEOS | math | 2020-10-06 | \- | 3.8.0\<br />3.7.2\<br />3.6.2 |
-| GL2PS | vis | 2019-11-08 | \- | 1.4.0 |
-| GLM | lib | 2018-12-11 | \- | 0.9.9.0 |
-| GLPK | tools | 2020-08-28 | \- | 4.65 |
-| GLib | vis | 2020-08-11 | \- | 2.64.1\<br />2.62.0\<br />2.60.1\<br />2.54.3 |
-| GLibmm | vis | 2020-09-24 | \- | 2.49.7 |
-| GMP | math | 2021-02-17 | \- | 6.2.0\<br />6.1.2 |
-| GObject-Introspection | devel | 2020-09-02 | \- | 1.63.1\<br />1.60.1\<br />1.54.1 |
-| GPAW | chem | 2020-01-15 | \- | 19.8.1 |
-| GPAW-setups | chem | 2020-01-15 | \- | 0.9.20000 |
-| GPI2 | base | 2019-05-29 | \- | next-27-05-19\<br />1.3.0 |
-| GROMACS | bio | 2020-10-12 | \- | 2020\<br />2019.4\<br />2018.2 |
-| GSL | numlib | 2020-08-11 | \- | 2.6\<br />2.5\<br />2.4 |
-| GTK+ | vis | 2020-01-13 | \- | 2.24.32 |
-| Gdk-Pixbuf | vis | 2020-09-02 | \- | 2.38.2\<br />2.36.12\<br />2.36.11 |
-| Ghostscript | tools | 2020-08-28 | \- | 9.52\<br />9.50\<br />9.27\<br />9.23\<br />9.22 |
-| GitPython | lib | 2020-04-16 | \- | 3.1.1\<br />3.0.3 |
-| GlobalArrays | lib | 2020-10-12 | \- | 5.7 |
-| Go | compiler | 2019-06-26 | \- | 1.12 |
-| GraphicsMagick | vis | 2019-11-08 | \- | 1.3.33\<br />1.3.31\<br />1.3.28 |
-| Guile | lang | 2020-08-11 | \- | 1.8.8 |
-| Gurobi | math | 2020-12-01 | \- | 9.1.0\<br />9.0.1\<br />8.0.1 |
-| HDF | data | 2018-11-22 | \- | 4.2.13 |
-| HDF5 | data | 2021-03-17 | \- | 1.10.7\<br />1.10.6\<br />1.10.5\<br />1.10.2\<br />1.10.1 |
-| HDFView | vis | 2019-02-20 | \- | 2.14 |
-| Hadoop | devel | 2019-08-29 | \- | 2.7.7 |
-| HarfBuzz | vis | 2020-09-02 | \- | 2.6.4\<br />2.4.0\<br />2.2.0\<br />1.7.5 |
-| Horovod | tools | 2020-01-24 | \- | 0.18.2 |
-| Hyperopt | lib | 2020-02-19 | \- | 0.2.2\<br />0.1.1 |
-| Hypre | numlib | 2020-12-08 | \- | 2.18.2\<br />2.14.0 |
-| ICU | lib | 2021-03-17 | \- | 67.1\<br />66.1\<br />64.2\<br />61.1\<br />56.1 |
-| IPython | tools | 2019-11-19 | \- | 7.7.0\<br />6.4.0\<br />6.3.1 |
-| ImageMagick | vis | 2020-08-28 | \- | 7.0.9-5\<br />7.0.8-46\<br />7.0.8-11\<br />7.0.7-39\<br />7.0.10-1 |
-| Inspector | tools | 2019-01-22 | \- | 2019\<br />2018 |
-| Ipopt | lib | 2018-11-22 | \- | 3.12.11 |
-| JUnit | devel | 2019-02-13 | \- | 4.12 |
-| JasPer | vis | 2020-08-11 | \- | 2.0.14\<br />1.900.1 |
-| Java | lang | 2020-12-02 | \- | 14.0.2\<br />11.0.2\<br />1.8.0-162\<br />1.8.0-161 |
-| JsonCpp | lib | 2021-03-17 | \- | 1.9.4\<br />1.9.3 |
-| Julia | lang | 2020-07-14 | \- | 1.4.2\<br />1.1.1\<br />1.0.2 |
-| Keras | math | 2020-03-20 | \- | 2.3.1\<br />2.2.4\<br />2.2.0 |
-| LAME | data | 2020-08-11 | \- | 3.100 |
-| LAMMPS | chem | 2020-08-11 | \- | 7Aug19\<br />3Mar2020\<br />20180316\<br />12Dec2018 |
-| LLVM | compiler | 2020-09-25 | \- | 9.0.1\<br />9.0.0\<br />8.0.1\<br />7.0.1\<br />6.0.0\<br />5.0.1 |
-| LMDB | lib | 2021-03-17 | \- | 0.9.24 |
-| LS-DYNA | cae | 2020-12-11 | \- | DEV-81069\<br />12.0.0\<br />11.1.0\<br />11.0.0\<br />10.1.0 |
-| LS-Opt | cae | 2019-08-20 | \- | 6.0.0\<br />5.2.1 |
-| LS-PrePost | cae | 2019-05-03 | \- | 4.6\<br />4.5\<br />4.3 |
-| LibTIFF | lib | 2021-02-17 | \- | 4.1.0\<br />4.0.9\<br />4.0.10 |
-| LibUUID | lib | 2019-11-08 | \- | 1.0.3 |
-| Libint | chem | 2019-10-14 | \- | 1.1.6 |
-| LittleCMS | vis | 2020-08-28 | \- | 2.9 |
-| M4 | devel | 2020-12-11 | \- | 1.4.18\<br />1.4.17 |
-| MATIO | lib | 2018-12-11 | \- | 1.5.12 |
-| MATLAB | math | 2021-03-22 | \- | 2021a\<br />2020a\<br />2019b\<br />2018b\<br />2018a\<br />2017a |
-| MDAnalysis | phys | 2020-04-21 | \- | 0.20.1 |
-| METIS | math | 2020-08-20 | \- | 5.1.0 |
-| MPFR | math | 2020-08-20 | \- | 4.0.2\<br />4.0.1 |
-| MUMPS | math | 2020-12-08 | \- | 5.2.1\<br />5.1.2 |
-| MUST | perf | 2019-01-25 | \- | 1.6.0-rc3 |
-| Mako | devel | 2020-08-11 | \- | 1.1.2\<br />1.1.0\<br />1.0.8\<br />1.0.7 |
-| Mathematica | math | 2018-11-22 | \- | 11.3.0\<br />11.2.0 |
-| Maven | devel | 2020-04-29 | \- | 3.6.3 |
-| Maxima | math | 2018-11-22 | \- | 5.42.1 |
-| Mercurial | tools | 2018-11-22 | \- | 4.6.1 |
-| Mesa | vis | 2020-08-11 | \- | 20.0.2\<br />19.1.7\<br />19.0.1\<br />18.1.1\<br />17.3.6 |
-| Meson | tools | 2021-02-17 | \- | 0.55.3\<br />0.53.2\<br />0.51.2\<br />0.50.0 |
-| Mesquite | math | 2020-02-12 | \- | 2.3.0 |
-| Miniconda2 | lang | 2018-11-22 | \- | 4.5.11 |
-| Miniconda3 | lang | 2019-10-17 | \- | 4.5.4 |
-| MongoDB | data | 2019-07-29 | \- | 4.0.3 |
-| NAMD | chem | 2018-11-22 | \- | 2.12 |
-| NASM | lang | 2021-02-17 | \- | 2.15.05\<br />2.14.02\<br />2.13.03\<br />2.13.01 |
-| NCCL | lib | 2021-03-17 | \- | 2.8.3\<br />2.4.8\<br />2.4.2\<br />2.3.7 |
-| NCO | tools | 2020-09-29 | \- | 4.9.3 |
-| NFFT | lib | 2019-08-09 | \- | 3.5.1 |
-| NLTK | data | 2018-12-05 | \- | 3.4 |
-| NLopt | numlib | 2020-08-28 | \- | 2.6.1\<br />2.4.2 |
-| NSPR | lib | 2020-08-11 | \- | 4.25\<br />4.21\<br />4.20 |
-| NSS | lib | 2020-08-11 | \- | 3.51\<br />3.45\<br />3.42.1\<br />3.39 |
-| NWChem | chem | 2018-11-22 | \- | 6.8.revision47\<br />6.6.revision27746 |
-| Nektar++ | math | 2020-02-19 | \- | 5.0.0 |
-| NetLogo | math | 2018-11-22 | \- | 6.0.4-64 |
-| Ninja | tools | 2021-02-17 | \- | 1.9.0\<br />1.10.1\<br />1.10.0 |
-| OPARI2 | perf | 2020-10-12 | \- | 2.0.5\<br />2.0.3 |
-| ORCA | chem | 2020-03-04 | \- | 4.2.1\<br />4.1.1 |
-| OTF2 | perf | 2020-10-12 | \- | 2.2\<br />2.1.1 |
-| Octave | math | 2019-11-08 | \- | 5.1.0 |
-| Octopus | chem | 2020-10-15 | \- | 8.4\<br />10.1 |
-| OpenBLAS | numlib | 2021-02-17 | \- | 0.3.9\<br />0.3.7\<br />0.3.5\<br />0.3.12\<br />0.3.1\<br />0.2.20 |
-| OpenBabel | chem | 2018-11-22 | \- | 2.4.1 |
-| OpenCV | vis | 2020-01-13 | \- | 4.0.1\<br />3.4.1 |
-| OpenFOAM | cae | 2020-09-23 | \- | v2006\<br />v1912\<br />v1806\<br />8\<br />7\<br />6\<br />5.0\<br />4.1\<br />2.3.1 |
-| OpenFOAM-Extend | cae | 2020-02-12 | \- | 4.0 |
-| OpenMPI | mpi | 2021-02-17 | \- | 4.0.5\<br />4.0.4\<br />4.0.3\<br />4.0.1\<br />3.1.6\<br />3.1.4\<br />3.1.3\<br />3.1.2\<br />3.1.1\<br />2.1.5\<br />2.1.2\<br />1.10.7 |
-| OpenMX | phys | 2020-12-08 | \- | 3.9.2 |
-| OpenMolcas | chem | 2020-10-12 | \- | 19.11\<br />18.09 |
-| OpenPGM | system | 2019-11-05 | \- | 5.2.122 |
-| OpenSSL | system | 2019-11-08 | \- | 1.1.1b\<br />1.0.2l |
-| PAPI | perf | 2020-10-12 | \- | 6.0.0\<br />5.7.0\<br />5.6.0 |
-| PCL | vis | 2019-08-13 | \- | 1.9.1 |
-| PCRE | devel | 2021-03-17 | \- | 8.44\<br />8.43\<br />8.41 |
-| PCRE2 | devel | 2020-08-11 | \- | 10.34\<br />10.33\<br />10.32 |
-| PDFCrop | tools | 2018-11-22 | \- | 0.4b |
-| PDT | perf | 2020-10-12 | \- | 3.25.1\<br />3.25 |
-| PETSc | numlib | 2020-12-08 | \- | 3.9.4\<br />3.9.3\<br />3.8.3\<br />3.7.7\<br />3.13.3\<br />3.12.4\<br />3.11.0\<br />3.10.5 |
-| PFFT | numlib | 2020-10-15 | \- | 1.0.8 |
-| PGI | compiler | 2020-02-07 | \- | 19.4\<br />19.10\<br />18.7\<br />18.4\<br />18.10\<br />17.7\<br />17.10 |
-| PLUMED | chem | 2020-08-11 | \- | 2.6.0\<br />2.5.1\<br />2.4.0 |
-| PLY | lib | 2019-02-26 | \- | 3.11 |
-| PMIx | lib | 2021-02-24 | \- | 3.1.5\<br />3.1.1 |
-| PROJ | lib | 2020-10-06 | \- | 6.2.1\<br />6.0.0\<br />5.0.0 |
-| Pandoc | tools | 2019-10-14 | \- | 2.5 |
-| Pango | vis | 2020-09-02 | \- | 1.44.7\<br />1.43.0\<br />1.42.4\<br />1.41.1 |
-| ParMETIS | math | 2020-02-12 | \- | 4.0.3 |
-| ParMGridGen | math | 2020-02-12 | \- | 1.0 |
-| ParaView | vis | 2020-11-12 | \- | 5.9.0-RC1\<br />5.8.0\<br />5.7.0\<br />5.6.2\<br />5.5.2\<br />5.4.1 |
-| Perl | lang | 2021-02-17 | \- | 5.32.0\<br />5.30.2\<br />5.30.0\<br />5.28.1\<br />5.28.0\<br />5.26.1 |
-| Pillow | vis | 2021-02-17 | \- | 8.0.1\<br />7.0.0\<br />6.2.1\<br />5.0.0 |
-| Pillow-SIMD | vis | 2018-11-22 | \- | 5.0.0 |
-| PnetCDF | data | 2019-03-06 | \- | 1.9.0 |
-| PyQt5 | vis | 2018-11-22 | \- | 5.10.1 |
-| PyTorch | devel | 2021-01-04 | \- | 1.6.0\<br />0.3.1 |
-| PyYAML | lib | 2020-04-16 | \- | 5.1.2\<br />5.1\<br />3.13\<br />3.12 |
-| Python | lang | 2021-02-17 | \- | 3.8.6\<br />3.8.2\<br />3.7.4\<br />3.7.2\<br />3.6.6\<br />3.6.4\<br />2.7.18\<br />2.7.16\<br />2.7.15\<br />2.7.14 |
-| Qhull | math | 2019-11-08 | \- | 2019.1\<br />2015.2 |
-| Qt | devel | 2018-11-22 | \- | 4.8.7 |
-| Qt5 | devel | 2020-08-11 | \- | 5.9.3\<br />5.14.1\<br />5.13.1\<br />5.12.3\<br />5.10.1 |
-| QuantumESPRESSO | chem | 2021-01-05 | \- | 6.6\<br />6.5\<br />6.4.1\<br />6.3\<br />6.2 |
-| Qwt | lib | 2020-09-24 | \- | 6.1.4 |
-| R | lang | 2020-10-06 | \- | 4.0.0\<br />3.6.2\<br />3.6.0\<br />3.5.1\<br />3.4.4 |
-| RDFlib | lib | 2020-09-29 | \- | 4.2.2 |
-| RELION | bio | 2019-02-27 | \- | 3.0\<br />2.1 |
-| ROOT | data | 2019-06-03 | \- | 6.14.06 |
-| Ruby | lang | 2019-11-14 | \- | 2.6.3\<br />2.6.1 |
-| SCOTCH | math | 2020-12-08 | \- | 6.0.9\<br />6.0.6\<br />6.0.5\<br />6.0.4\<br />5.1.12b |
-| SCons | devel | 2020-01-24 | \- | 3.1.1\<br />3.0.5\<br />3.0.1 |
-| SHARC | chem | 2019-01-07 | \- | 2.0 |
-| SIONlib | lib | 2020-10-12 | \- | 1.7.6\<br />1.7.4\<br />1.7.2 |
-| SIP | lang | 2018-11-22 | \- | 4.19.8\<br />4.19.12 |
-| SLEPc | numlib | 2020-02-27 | \- | 3.9.2\<br />3.12.2 |
-| SPM | math | 2018-11-22 | \- | 12-r7219 |
-| SQLite | devel | 2021-02-17 | \- | 3.33.0\<br />3.31.1\<br />3.29.0\<br />3.27.2\<br />3.26.0\<br />3.24.0\<br />3.21.0\<br />3.20.1 |
-| STAR-CCM+ | cae | 2021-03-19 | \- | 15.06.008\<br />15.04.010-R8\<br />15.02.007\<br />14.04.011\<br />14.02.012\<br />13.06.012-R8\<br />13.04.011\<br />13.02.013-R8 |
-| SUNDIALS | math | 2018-11-22 | \- | 2.7.0 |
-| SWASH | phys | 2020-03-18 | \- | 6.01\<br />5.01 |
-| SWIG | devel | 2020-10-06 | \- | 4.0.1\<br />3.0.12 |
-| ScaFaCoS | math | 2020-08-11 | \- | 1.0.1 |
-| ScaLAPACK | numlib | 2021-02-17 | \- | 2.1.0\<br />2.0.2 |
-| Scalasca | perf | 2021-02-02 | \- | 2.5\<br />2.4 |
-| SciPy-bundle | lang | 2021-02-17 | \- | 2020.11\<br />2020.03\<br />2019.10\<br />2019.03 |
-| Score-P | perf | 2020-10-19 | \- | 6.0\<br />4.0 |
-| Serf | tools | 2019-11-08 | \- | 1.3.9 |
-| Siesta | phys | 2020-02-27 | \- | 4.1-b4\<br />4.1-b3\<br />4.1 |
-| Six | lib | 2018-11-22 | \- | 1.11.0 |
-| Spark | devel | 2020-10-22 | \- | 3.0.1\<br />2.4.4\<br />2.4.3 |
-| Subversion | tools | 2019-11-08 | \- | 1.9.7\<br />1.12.0 |
-| SuiteSparse | numlib | 2020-09-02 | \- | 5.7.1\<br />5.6.0\<br />5.4.0\<br />5.1.2 |
-| SuperLU | numlib | 2019-01-15 | \- | 5.2.1 |
-| SuperLU_DIST | numlib | 2019-01-16 | \- | 6.1.0 |
-| SuperLU_MT | numlib | 2019-01-16 | \- | 3.1 |
-| Szip | tools | 2021-03-03 | \- | 2.1.1 |
-| Tcl | lang | 2021-02-17 | \- | 8.6.9\<br />8.6.8\<br />8.6.7\<br />8.6.10 |
-| TensorFlow | lib | 2021-03-15 | \- | 2.4.1\<br />2.3.1\<br />2.1.0\<br />2.0.0\<br />1.8.0\<br />1.15.0\<br />1.10.0 |
-| Tk | vis | 2021-02-17 | \- | 8.6.9\<br />8.6.8\<br />8.6.10 |
-| Tkinter | lang | 2021-02-17 | \- | 3.8.6\<br />3.8.2\<br />3.7.4\<br />3.7.2\<br />3.6.6\<br />3.6.4\<br />2.7.15\<br />2.7.14 |
-| TotalView | debugger | 2020-09-24 | \- | 8.14.1-8 |
-| Trilinos | numlib | 2019-02-26 | \- | 12.12.1 |
-| UCX | lib | 2021-02-24 | \- | 1.9.0\<br />1.8.0\<br />1.5.1 |
-| UDUNITS | phys | 2020-08-28 | \- | 2.2.26 |
-| UFL | cae | 2019-02-26 | \- | 2018.1.0 |
-| UnZip | tools | 2021-02-17 | \- | 6.0 |
-| VASP | phys | 2020-05-19 | \- | 5.4.4 |
-| VMD | vis | 2018-11-22 | \- | 1.9.3 |
-| VSEARCH | bio | 2018-11-22 | \- | 2.8.4 |
-| VTK | vis | 2020-10-06 | \- | 8.2.0\<br />8.1.1\<br />8.1.0\<br />5.10.1 |
-| VTune | tools | 2020-04-28 | \- | 2020\<br />2019\<br />2018 |
-| Valgrind | debugger | 2019-11-05 | \- | 3.14.0\<br />3.13.0 |
-| Vampir | tools | 2021-01-13 | \- | unstable\<br />9.9.0\<br />9.8.0\<br />9.7.1\<br />9.7.0\<br />9.6.1\<br />9.5.0\<br />9.11\<br />9.10.0 |
-| Voro++ | math | 2020-08-11 | \- | 0.4.6 |
-| WRF | geo | 2018-12-12 | \- | 3.8.1 |
-| Wannier90 | chem | 2020-12-14 | \- | 2.1.0\<br />2.0.1.1\<br />1.2 |
-| X11 | vis | 2021-02-17 | \- | 20201008\<br />20200222\<br />20190717\<br />20190311\<br />20180604\<br />20180131 |
-| XML-Parser | data | 2018-11-22 | \- | 2.44-01 |
-| XZ | tools | 2021-02-17 | \- | 5.2.5\<br />5.2.4\<br />5.2.3\<br />5.2.2 |
-| YAXT | tools | 2020-10-06 | \- | 0.6.2\<br />0.6.0 |
-| Yasm | lang | 2020-08-11 | \- | 1.3.0 |
-| ZeroMQ | devel | 2019-11-05 | \- | 4.3.2\<br />4.2.5 |
-| Zip | tools | 2021-03-17 | \- | 3.0 |
-| ace | lib | 2018-11-22 | \- | 6.5.0 |
-| ant | devel | 2020-09-02 | \- | 1.10.7\<br />1.10.1 |
-| archspec | tools | 2020-08-11 | \- | 0.1.0 |
-| arpack-ng | numlib | 2019-11-08 | \- | 3.7.0\<br />3.6.1\<br />3.5.0 |
-| asciidoc | base | 2018-11-22 | \- | 8.6.9 |
-| at-spi2-atk | vis | 2020-09-02 | \- | 2.34.1 |
-| at-spi2-core | vis | 2020-09-02 | \- | 2.34.0 |
-| auto_ml | lang | 2019-10-29 | \- | 2.9.9 |
-| basemap | vis | 2019-04-02 | \- | 1.0.7 |
-| binutils | tools | 2020-12-11 | \- | 2.35\<br />2.34\<br />2.32\<br />2.31.1\<br />2.30\<br />2.28\<br />2.27\<br />2.26 |
-| bzip2 | tools | 2021-02-17 | \- | 1.0.8\<br />1.0.6 |
-| cURL | tools | 2021-02-17 | \- | 7.72.0\<br />7.69.1\<br />7.66.0\<br />7.63.0\<br />7.60.0\<br />7.58.0 |
-| cairo | vis | 2020-08-28 | \- | 1.16.0\<br />1.14.12 |
-| cftime | data | 2019-07-17 | \- | 1.0.1 |
-| chrpath | tools | 2018-11-22 | \- | 0.16 |
-| ctags | devel | 2018-11-22 | \- | 5.8 |
-| cuDNN | numlib | 2021-03-17 | \- | 8.0.4.30\<br />7.6.4.38\<br />7.4.2.24\<br />7.1.4.18\<br />7.1.4\<br />7.0.5 |
-| ddt | tools | 2021-04-12 | \- | 20.2.1\<br />20.0.1\<br />18.2.2 |
-| dftd3-lib | chem | 2021-02-17 | \- | 0.9 |
-| dill | data | 2019-10-29 | \- | 0.3.1.1\<br />0.3.1 |
-| double-conversion | lib | 2021-03-17 | \- | 3.1.5\<br />3.1.4 |
-| ecCodes | tools | 2020-10-06 | \- | 2.8.2\<br />2.17.0 |
-| expat | tools | 2021-02-17 | \- | 2.2.9\<br />2.2.7\<br />2.2.6\<br />2.2.5 |
-| flair | vis | 2019-01-25 | \- | 2.3-0 |
-| flair-geoviewer | vis | 2019-01-25 | \- | 2.3-0 |
-| flatbuffers | devel | 2021-03-17 | \- | 1.12.0 |
-| flatbuffers-python | devel | 2021-03-19 | \- | 1.12 |
-| flex | lang | 2020-12-11 | \- | 2.6.4\<br />2.6.3\<br />2.6.0\<br />2.5.39 |
-| fontconfig | vis | 2021-02-17 | \- | 2.13.92\<br />2.13.1\<br />2.13.0\<br />2.12.6 |
-| foss | toolchain | 2021-02-17 | \- | 2020b\<br />2020a\<br />2019b\<br />2019a\<br />2018b\<br />2018a |
-| fosscuda | toolchain | 2021-02-17 | \- | 2020b\<br />2020a\<br />2019b\<br />2019a\<br />2018b |
-| freeglut | lib | 2019-11-08 | \- | 3.0.0 |
-| freetype | vis | 2021-02-17 | \- | 2.9.1\<br />2.9\<br />2.10.3\<br />2.10.1 |
-| future | lib | 2018-11-22 | \- | 0.16.0 |
-| gc | lib | 2020-08-11 | \- | 7.6.12\<br />7.6.10\<br />7.6.0 |
-| gcccuda | toolchain | 2021-02-17 | \- | 2020b\<br />2020a\<br />2019b\<br />2019a\<br />2018b |
-| gettext | tools | 2021-02-17 | \- | 0.21\<br />0.20.1\<br />0.19.8.1\<br />0.19.8 |
-| gflags | devel | 2020-09-29 | \- | 2.2.2 |
-| giflib | lib | 2021-03-17 | \- | 5.2.1 |
-| git | tools | 2021-03-17 | \- | 2.28.0\<br />2.23.0\<br />2.21.0\<br />2.19.1\<br />2.18.0\<br />2.16.1 |
-| git-cola | tools | 2018-11-22 | \- | 3.2 |
-| git-lfs | tools | 2019-06-26 | \- | 2.7.2 |
-| glew | devel | 2020-09-23 | \- | 2.1.0 |
-| glog | devel | 2020-09-29 | \- | 0.4.0 |
-| gmsh | vis | 2019-11-26 | \- | 4.4.1 |
-| gnuplot | vis | 2019-11-08 | \- | 5.2.6\<br />5.2.5\<br />5.2.4\<br />5.2.2 |
-| golf | toolchain | 2018-11-22 | \- | 2018a |
-| gomkl | toolchain | 2019-07-17 | \- | 2019a |
-| gompi | toolchain | 2021-02-17 | \- | 2020b\<br />2020a\<br />2019b\<br />2019a\<br />2018b\<br />2018a |
-| gompic | toolchain | 2021-02-17 | \- | 2020b\<br />2020a\<br />2019b\<br />2019a\<br />2018b |
-| gperf | devel | 2021-02-17 | \- | 3.1 |
-| gperftools | tools | 2018-11-22 | \- | 2.7 |
-| grib_api | data | 2018-11-22 | \- | 1.27.0 |
-| gzip | tools | 2020-08-11 | \- | 1.9\<br />1.8\<br />1.10 |
-| h5py | data | 2020-08-11 | \- | 2.9.0\<br />2.8.0\<br />2.7.1\<br />2.10.0 |
-| help2man | tools | 2020-12-11 | \- | 1.47.8\<br />1.47.6\<br />1.47.4\<br />1.47.16\<br />1.47.12\<br />1.47.10 |
-| hwloc | system | 2021-02-17 | \- | 2.2.0\<br />2.0.3\<br />1.11.8\<br />1.11.12\<br />1.11.11\<br />1.11.10 |
-| hypothesis | tools | 2021-02-17 | \- | 5.41.2\<br />4.44.2 |
-| icc | compiler | 2019-10-11 | \- | 2019.1.144\<br />2019.0.117\<br />2018.3.222\<br />2018.1.163 |
-| iccifort | toolchain | 2020-09-21 | \- | 2020.2.254\<br />2020.1.217\<br />2019.5.281\<br />2019.1.144\<br />2019.0.117\<br />2018.3.222\<br />2018.1.163 |
-| ifort | compiler | 2019-10-11 | \- | 2019.1.144\<br />2019.0.117\<br />2018.3.222\<br />2018.1.163 |
-| iimpi | toolchain | 2020-08-03 | \- | 2020a\<br />2019b\<br />2019a\<br />2018b\<br />2018a |
-| imkl | numlib | 2021-01-05 | \- | 2020.1.217\<br />2019.5.281\<br />2019.1.144\<br />2018.3.222\<br />2018.1.163 |
-| impi | mpi | 2020-08-13 | \- | 2019.7.217\<br />2018.5.288\<br />2018.4.274\<br />2018.3.222\<br />2018.1.163 |
-| intel | toolchain | 2020-08-03 | \- | 2020a\<br />2019b\<br />2019a\<br />2018b\<br />2018a |
-| intltool | devel | 2021-02-17 | \- | 0.51.0 |
-| iomkl | toolchain | 2021-01-05 | \- | 2020a\<br />2019a\<br />2018a |
-| iompi | toolchain | 2021-01-05 | \- | 2020a\<br />2019a\<br />2018a |
-| itac | tools | 2018-11-22 | \- | 2018.3.022 |
-| kim-api | chem | 2020-08-11 | \- | 2.1.3 |
-| libGLU | vis | 2020-08-11 | \- | 9.0.1\<br />9.0.0 |
-| libGridXC | chem | 2020-02-27 | \- | 0.8.5 |
-| libPSML | data | 2020-02-27 | \- | 1.1.8 |
-| libarchive | tools | 2021-02-17 | \- | 3.4.3 |
-| libcerf | math | 2019-11-08 | \- | 1.7\<br />1.5\<br />1.11 |
-| libcint | lib | 2019-01-09 | \- | 3.0.14 |
-| libdap | lib | 2020-09-29 | \- | 3.20.6 |
-| libdrm | lib | 2020-08-11 | \- | 2.4.99\<br />2.4.97\<br />2.4.92\<br />2.4.91\<br />2.4.100 |
-| libelf | devel | 2020-12-11 | \- | 0.8.13 |
-| libepoxy | lib | 2020-09-02 | \- | 1.5.4 |
-| libevent | lib | 2021-02-24 | \- | 2.1.8\<br />2.1.12\<br />2.1.11 |
-| libfabric | lib | 2021-02-17 | \- | 1.11.0 |
-| libffi | lib | 2021-02-17 | \- | 3.3\<br />3.2.1 |
-| libgd | lib | 2019-11-08 | \- | 2.2.5 |
-| libgeotiff | lib | 2020-10-06 | \- | 1.5.1\<br />1.4.2 |
-| libglvnd | lib | 2020-08-11 | \- | 1.2.0 |
-| libharu | lib | 2018-11-22 | \- | 2.3.0 |
-| libiconv | lib | 2021-03-03 | \- | 1.16 |
-| libjpeg-turbo | lib | 2021-02-17 | \- | 2.0.5\<br />2.0.4\<br />2.0.3\<br />2.0.2\<br />2.0.0\<br />1.5.3\<br />1.5.2 |
-| libmatheval | lib | 2020-08-11 | \- | 1.1.11 |
-| libpciaccess | system | 2021-02-17 | \- | 0.16\<br />0.14 |
-| libpng | lib | 2021-02-17 | \- | 1.6.37\<br />1.6.36\<br />1.6.34\<br />1.6.32 |
-| libreadline | lib | 2021-02-17 | \- | 8.0\<br />7.0 |
-| libsigc++ | devel | 2020-09-24 | \- | 2.10.1 |
-| libsndfile | lib | 2020-08-28 | \- | 1.0.28 |
-| libsodium | lib | 2019-11-05 | \- | 1.0.17\<br />1.0.16 |
-| libssh2 | tools | 2018-11-22 | \- | 1.8.0 |
-| libtirpc | lib | 2020-09-29 | \- | 1.2.6 |
-| libtool | lib | 2021-02-17 | \- | 2.4.6 |
-| libunistring | lib | 2020-08-11 | \- | 0.9.7\<br />0.9.10 |
-| libunwind | lib | 2020-08-11 | \- | 1.3.1\<br />1.2.1 |
-| libvdwxc | chem | 2020-01-15 | \- | 0.4.0 |
-| libxc | chem | 2021-01-05 | \- | 4.3.4\<br />4.2.3\<br />3.0.1 |
-| libxml++ | lib | 2020-09-24 | \- | 2.40.1 |
-| libxml2 | lib | 2021-02-17 | \- | 2.9.9\<br />2.9.8\<br />2.9.7\<br />2.9.4\<br />2.9.10 |
-| libxslt | lib | 2020-10-27 | \- | 1.1.34\<br />1.1.33\<br />1.1.32 |
-| libxsmm | math | 2019-10-14 | \- | 1.8.3\<br />1.10 |
-| libyaml | lib | 2020-04-16 | \- | 0.2.2\<br />0.2.1\<br />0.1.7 |
-| likwid | tools | 2020-10-14 | \- | 5.0.1 |
-| lo2s | perf | 2020-01-27 | \- | 1.3.0\<br />1.2.2\<br />1.1.1\<br />1.0.2\<br />1.0.1 |
-| log4cxx | lang | 2020-02-18 | \- | 0.10.0 |
-| lpsolve | math | 2018-11-22 | \- | 5.5.2.5 |
-| lz4 | lib | 2020-08-11 | \- | 1.9.2\<br />1.9.1 |
-| magma | math | 2021-01-04 | \- | 2.5.4\<br />2.3.0 |
-| makedepend | devel | 2018-11-22 | \- | 1.0.5 |
-| matplotlib | vis | 2021-02-17 | \- | 3.3.3\<br />3.2.1\<br />3.1.1\<br />3.0.3\<br />3.0.0\<br />2.1.2 |
-| mkl-dnn | lib | 2018-11-22 | \- | 0.13 |
-| molmod | math | 2020-08-11 | \- | 1.4.5 |
-| motif | vis | 2018-11-22 | \- | 2.3.8 |
-| ncdf4 | math | 2020-10-06 | \- | 1.17 |
-| ncurses | devel | 2021-02-17 | \- | 6.2\<br />6.1\<br />6.0 |
-| netCDF | data | 2021-03-03 | \- | 4.7.4\<br />4.7.1\<br />4.6.2\<br />4.6.1\<br />4.6.0 |
-| netCDF-Fortran | data | 2021-03-04 | \- | 4.5.3\<br />4.5.2\<br />4.4.5\<br />4.4.4 |
-| netcdf4-python | data | 2019-07-17 | \- | 1.4.3 |
-| nettle | lib | 2020-01-24 | \- | 3.5.1\<br />3.4.1\<br />3.4 |
-| nextstrain | bio | 2020-07-20 | \- | 2.0.0.post1 |
-| nfft | math | 2018-11-28 | \- | 3.3.2DLR |
-| nsync | devel | 2021-03-17 | \- | 1.24.0 |
-| numactl | tools | 2021-02-17 | \- | 2.0.13\<br />2.0.12\<br />2.0.11 |
-| numba | lang | 2020-09-25 | \- | 0.47.0 |
-| numeca | cae | 2018-11-22 | \- | all |
-| nvidia-nsight | tools | 2020-01-21 | \- | 2019.3.1 |
-| p7zip | tools | 2018-11-22 | \- | 9.38.1 |
-| parallel | tools | 2020-02-25 | \- | 20190922\<br />20190622\<br />20180822 |
-| patchelf | tools | 2019-08-09 | \- | 0.9 |
-| petsc4py | tools | 2019-02-26 | \- | 3.9.1 |
-| pigz | tools | 2018-11-22 | \- | 2.4 |
-| pixman | vis | 2020-08-28 | \- | 0.38.4\<br />0.38.0\<br />0.34.0 |
-| pkg-config | devel | 2021-02-17 | \- | 0.29.2 |
-| pkgconfig | devel | 2021-03-17 | \- | 1.5.1\<br />1.3.1 |
-| pocl | lib | 2020-02-19 | \- | 1.4 |
-| pompi | toolchain | 2018-11-22 | \- | 2018.04 |
-| protobuf | devel | 2021-03-17 | \- | 3.6.1.2\<br />3.6.1\<br />3.14.0\<br />3.10.0 |
-| protobuf-python | devel | 2021-03-17 | \- | 3.14.0\<br />3.10.0 |
-| pybind11 | lib | 2021-02-17 | \- | 2.6.0\<br />2.4.3\<br />2.2.4 |
-| pyscf | chem | 2019-11-13 | \- | 1.6.1\<br />1.6.0 |
-| pytecplot | data | 2020-04-06 | \- | 1.0.0 |
-| pytest | tools | 2019-02-26 | \- | 3.8.0 |
-| qrupdate | numlib | 2019-11-08 | \- | 1.1.2 |
-| re2c | tools | 2020-08-11 | \- | 1.3\<br />1.2.1 |
-| rgdal | geo | 2019-11-27 | \- | 1.4-4 |
-| rstudio | lang | 2020-02-18 | \- | 1.2.5001\<br />1.2.1335\<br />1.1.456 |
-| scikit-learn | data | 2020-09-24 | \- | 0.21.3 |
-| scorep_plugin_fileparser | perf | 2018-11-22 | \- | 1.3.1 |
-| sf | lib | 2020-10-06 | \- | 0.9-5 |
-| slepc4py | tools | 2019-02-26 | \- | 3.9.0 |
-| snakemake | tools | 2020-04-16 | \- | 5.7.1\<br />5.14.0 |
-| snappy | lib | 2021-03-17 | \- | 1.1.8\<br />1.1.7 |
-| source-highlight | tools | 2018-11-22 | \- | 3.1.8 |
-| spacy | lang | 2018-12-05 | \- | 2.0.18 |
-| spglib | chem | 2019-12-12 | \- | 1.14.1 |
-| tbb | lib | 2020-08-11 | \- | 2020.1\<br />2019-U4\<br />2018-U5 |
-| tcsh | tools | 2018-11-22 | \- | 6.20.00 |
-| tecplot360ex | vis | 2020-07-20 | \- | 2019r1 |
-| texinfo | devel | 2020-08-11 | \- | 6.7\<br />6.6\<br />6.5 |
-| tmux | tools | 2021-04-09 | \- | 3.1c\<br />2.3 |
-| torchvision | vis | 2018-11-22 | \- | 0.2.1 |
-| tqdm | lib | 2020-09-29 | \- | 4.41.1 |
-| typing-extensions | devel | 2021-03-19 | \- | 3.7.4.3 |
-| utf8proc | lib | 2019-11-08 | \- | 2.3.0 |
-| util-linux | tools | 2021-02-17 | \- | 2.36\<br />2.35\<br />2.34\<br />2.33\<br />2.32\<br />2.31.1 |
-| wheel | tools | 2018-11-22 | \- | 0.31.1\<br />0.31.0 |
-| x264 | vis | 2020-08-11 | \- | 20191217\<br />20190925\<br />20190413\<br />20181203\<br />20180128 |
-| x265 | vis | 2020-08-11 | \- | 3.3\<br />3.2\<br />3.0\<br />2.9\<br />2.6 |
-| xbitmaps | devel | 2018-11-22 | \- | 1.1.1 |
-| xmlf90 | data | 2020-02-27 | \- | 1.5.4 |
-| xorg-macros | devel | 2021-02-17 | \- | 1.19.2\<br />1.19.1 |
-| xprop | vis | 2019-11-08 | \- | 1.2.4\<br />1.2.3\<br />1.2.2 |
-| xproto | devel | 2018-11-22 | \- | 7.0.31 |
-| yaff | chem | 2020-08-11 | \- | 1.6.0 |
-| zlib | lib | 2020-12-11 | \- | 1.2.8\<br />1.2.11 |
-| zsh | tools | 2021-01-06 | \- | 5.8 |
-| zstd | lib | 2020-08-11 | \- | 1.4.4 |
-
-### Classic Environment
-
-<span class="twiki-macro TABLE">headerrows= 1"</span> <span
-class="twiki-macro EDITTABLE"
-format="| text, 30, Software | text, 40, Kategorie | text, 30, Letzte Änderung | text, 30, SGI-UV | text, 30, Taurus | "
-changerows="on"></span>
-
-| Software | Category | Last change | Venus | Taurus |
-|:-------------------------|:-------------|:------------|:-----------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| AVS-Express | applications | 2015-06-26 | \- | mpepst8.2\<br />8.2 |
-| FPLO | applications | 2018-04-13 | \- | 18.00-53\<br />14.00-49 |
-| VirtualGL | tools | 2013-10-02 | \- | |
-| abaqus | applications | 2018-04-19 | 2017\<br />2016 | *2018* \<br /> 6.9-EF1\<br />6.13\<br />6.12\<br />2017\<br />2016 |
-| abinit | applications | 2013-11-21 | 7.2.1 | 7.2.1 |
-| ace | lib | 2018-11-22 | \- | 6.3.3 |
-| adolc | libraries | 2014-07-24 | *2.5.0* \<br /> 2.4.1 | *2.5.0* \<br /> 2.4.1 |
-| afni | applications | 2014-07-01 | \- | 2011-12-21-1014 |
-| amber | applications | 2017-10-06 | \- | 15 |
-| ansys | applications | 2018-09-04 | 18.0\<br />17.1\<br />16.1 | 19.0\<br />18.2\<br />18.1\<br />18.0\<br />17.2\<br />17.1\<br />17.0\<br />16.1 |
-| ansysem | applications | 2017-07-20 | \- | 16.0 |
-| asm | tools | 2017-03-17 | \- | 5.2 |
-| autoconf | tools | 2013-10-30 | 2.69 | 2.69 |
-| automake | tools | 2014-09-11 | 1.14\<br />1.12.2\<br />1.12 | 1.14\<br />1.12.2 |
-| autotools | tools | 2017-02-01 | \- | default\<br />2015 |
-| bazel | compilers | 2017-07-13 | \- | 0.5.2 |
-| bison | libraries | 2015-03-11 | \- | 3.0.4 |
-| blcr | tools | 2016-03-02 | \- | |
-| boost | libraries | 2019-03-29 | *1.49* \<br /> 1.69.0\<br />1.54\<br />1.51.0 | *1.54.0* \<br /> 1.66.0\<br />1.65.1\<br />1.65.0\<br />1.64.0\<br />1.63.0\<br />1.62.0\<br />1.61.0\<br />1.60.0\<br />1.59.0\<br />1.58.0\<br />1.57.0\<br />1.56.0\<br />1.55.0\<br />1.49 |
-| bowtie | applications | 2013-01-16 | 0.12.8 | \- |
-| bullxmpi | libraries | 2016-10-14 | \- | *1.2.8.4* \<br /> 1.2.9.2 |
-| casita | tools | 2017-06-08 | \- | 1.9 |
-| cdo | tools | 2013-04-08 | 1.6.0 | \- |
-| ceph | libraries | 2017-01-13 | \- | 11.1 |
-| cereal | libraries | 2016-12-07 | \- | 1.2.1 |
-| cg | libraries | 2015-09-18 | \- | 3.1 |
-| cgal | libraries | 2018-03-08 | \- | 4.11.1 |
-| clFFT | libraries | 2017-07-12 | \- | 2.12.2dev\<br />2.12.2\<br />2.12.1\<br />2.12.0\<br />2.10.0 |
-| clang | compilers | 2018-08-20 | \- | 4.0.0 |
-| cmake | tools | 2019-04-03 | 3.3.1\<br />3.11.4\<br />2.8.2\<br />2.8.12.2\<br />2.8.11 | *3.10.1* \<br /> 3.9.0\<br />3.6.2\<br />3.3.1\<br />2.8.2\<br />2.8.12.2\<br />2.8.11 |
-| collectl | tools | 2017-12-05 | 4.2.0\<br />4.1.2\<br />3.6.7\<br />3.5.1 | 4.2.0\<br />4.1.2\<br />3.6.7\<br />3.5.1 |
-| comsol | applications | 2018-04-20 | \- | *5.3a* \<br /> 5.3 |
-| conn | libraries | 2017-07-12 | \- | 17f |
-| cp2k | applications | 2017-12-05 | 2.5 | *5.1* \<br /> r16298\<br />r15503\<br />r14075\<br />r13178\<br />3.0\<br />2.6.2\<br />2.6\<br />2.4\<br />2.3\<br />130326 |
-| cpufrequtils | libraries | 2017-02-16 | \- | gcc5.3.0 |
-| ctool | libraries | 2013-04-17 | 2.12 | 2.12 |
-| cube | tools | 2018-05-31 | *4.3* | *4.3* \<br /> 4.4 |
-| cuda | libraries | 2018-06-07 | \- | *9.2.88* \<br /> 9.1.85\<br />9.0.176\<br />8.0.61\<br />8.0.44\<br />7.5.18\<br />7.0.28 |
-| curl | libraries | 2013-01-18 | 7.28.1 | \- |
-| cusp | libraries | 2014-04-22 | \- | 0.4.0\<br />0.3.1 |
-| cython | libraries | 2016-08-26 | \- | 0.24.1\<br />0.24\<br />0.19.2 |
-| dalton | applications | 2016-04-07 | \- | 2016.0 |
-| darshan | tools | 2017-09-02 | \- | darshan-3.1.4 |
-| dash | libraries | 2019-02-14 | \- | dash |
-| dataheap | libraries | 2017-01-12 | \- | 1.2\<br />1.1 |
-| ddt | tools | 2021-04-12 | 4.2\<br />4.0\<br />3.2.1 | *18.0.1* \<br /> 6.0.5\<br />6.0 |
-| dftb+ | applications | 2017-06-26 | \- | mpi\<br />1.3\<br />1.2.2 |
-| dmtcp | tools | 2017-10-18 | \- | *2.5.1-ib* \<br /> ib-id |
-| doxygen | tools | 2016-01-27 | \- | 1.8.11\<br />1.7.4 |
-| dune | libraries | 2014-05-28 | \- | 2.2.1 |
-| dyninst | libraries | 2016-02-16 | 8.1.1 | 8.2.1\<br />8.1.1 |
-| eigen | libraries | 2017-03-23 | \- | 3.3.3\<br />3.2.0 |
-| eirods | tools | 2013-12-11 | \- | 3.1 |
-| eman2 | applications | 2017-06-21 | \- | 2.2 |
-| ensight | applications | 2015-07-13 | \- | 10.1.5a\<br />10.0 |
-| extrae | applications | 2017-01-03 | \- | 3.4.1 |
-| fftw | libraries | 2017-03-18 | \- | 3.3.6pl1\<br />3.3.5\<br />3.3.4 |
-| firestarter | applications | 2016-04-20 | \- | 1.4 |
-| flex | lang | 2020-12-11 | \- | 2.5.39 |
-| fme | applications | 2017-04-21 | \- | 2017 |
-| freecad | applications | 2014-05-12 | \- | *0.14* \<br /> 0.13 |
-| freeglut | lib | 2019-11-08 | 2.8.1 | 2.8.1 |
-| freesurfer | applications | 2015-12-04 | \- | *5.3.0* \<br /> 5.1.0 |
-| fsl | libraries | 2018-01-17 | \- | 5.0.5\<br />5.0.4\<br />5.0.10 |
-| ga | libraries | 2015-06-22 | \- | 5.2 |
-| gamess | applications | 2014-12-11 | \- | 2013 |
-| gams | applications | 2017-07-10 | \- | 24.8\<br />24.3.3 |
-| gaussian | applications | 2017-06-01 | g16\<br />g09d01\<br />g09b01\<br />g09\<br />g03 | *g16* \<br /> g09d01\<br />g09b01\<br />g09 |
-| gautomatch | applications | 2017-06-21 | \- | 0.53 |
-| gcc | compilers | 2019-04-02 | 8.3.0\<br />7.1.0\<br />6.3.0\<br />5.5.0\<br />4.9.3\<br />4.9.1\<br />4.8.2\<br />4.8.0\<br />4.7.1 | *7.1.0* \<br /> 6.3.0\<br />6.2.0\<br />5.5.0\<br />5.3.0\<br />5.2.0\<br />5.1.0\<br />4.9.3\<br />4.9.1\<br />4.8.2\<br />4.8.0\<br />4.7.1\<br />4.6.2 |
-| gcl | compilers | 2017-03-13 | \- | 2.6.12 |
-| gcovr | tools | 2015-02-12 | *3.2* | *3.2* |
-| gctf | applications | 2017-06-21 | \- | 0.50 |
-| gdb | tools | 2017-08-03 | 7.9.1 | 7.5 |
-| gdk | tools | 2015-12-14 | \- | 352 |
-| geany | tools | 2014-05-12 | \- | 1.24.1 |
-| ghc | compilers | 2015-01-16 | 7.6.3 | 7.6.3 |
-| git | tools | 2021-03-17 | *1.8.3.1* \<br /> 2.17.1\<br />1.7.7\<br />1.7.4.1\<br />1.7.3.2 | *2.15.1* \<br /> 2.7.3\<br />1.9.0 |
-| glib | libraries | 2016-10-27 | \- | 2.50.1\<br />2.44.1 |
-| gmap | tools | 2018-07-13 | \- | 2018-07-04 |
-| gmock | tools | 2013-10-17 | \- | *1.6.0* |
-| gnuplot | vis | 2019-11-08 | 4.6.1\<br />4.4.0 | 4.6.1\<br />4.4.0 |
-| gpaw | applications | 2016-02-19 | \- | 0.11.0 |
-| gperftools | tools | 2018-11-22 | \- | gperftools-2.6.1.lua |
-| gpi2 | libraries | 2018-04-25 | \- | *git* \<br /> 1.3.0\<br />1.2.2\<br />1.1.0 |
-| gpi2-mpi | libraries | 2015-03-30 | \- | *1.1.1* |
-| gpudevkit | libraries | 2016-07-25 | \- | 352-79 |
-| grads | applications | 2014-08-05 | 2.0.2 | \- |
-| grid | tools | 2014-12-09 | \- | 2012 |
-| gromacs | applications | 2018-01-22 | \- | *5.1.3* \<br /> 5.1.4\<br />5.1.1\<br />5.1\<br />4.6.7\<br />4.5.5\<br />3.3.3 |
-| gsl | libraries | 2015-01-28 | \- | 1.16 |
-| gulp | applications | 2015-10-09 | \- | 4.3 |
-| gurobi | applications | 2017-06-28 | 7.0.2\<br />6.0.4 | 7.0.2\<br />6.0.4 |
-| h5utils | tools | 2014-08-18 | \- | 1.12.1 |
-| haskell-platform | tools | 2015-01-16 | 2013.2.0.0 | 2013.2.0.0 |
-| hdeem | libraries | 2018-01-22 | \- | *deprecated* \<br /> 2.2.20ms\<br />2.2.2\<br />2.2.19ms\<br />2.2.16ms\<br />2.2.15ms\<br />2.2.13ms\<br />2.1.9ms\<br />2.1.5\<br />2.1.4\<br />2.1.10ms |
-| hdf5 | libraries | 2018-02-09 | 1.8.14\<br />1.8.10 | hdfview\<br />1.8.19\<br />1.8.18\<br />1.8.16\<br />1.8.15\<br />1.8.14\<br />1.8.10\<br />1.6.5\<br />1.10.1 |
-| hip | libraries | 2018-08-27 | \- | git |
-| hoomd-blue | applications | 2016-07-29 | \- | 2.0.1 |
-| hpc-x | libraries | 2017-04-10 | \- | 1.8.0 |
-| hpctoolkit | tools | 2013-05-28 | 5.3.2 | \- |
-| hpx | libraries | 2017-09-15 | \- | hpx |
-| htop | tools | 2016-11-04 | 1.0.2 | 1.0.2 |
-| hwloc | system | 2021-02-17 | \- | 1.11.8\<br />1.11.6 |
-| hyperdex | tools | 2015-08-20 | \- | default\<br />1.8.1\<br />1.7.1 |
-| hyperopt | libraries | 2018-03-19 | \- | 0.1 |
-| imagemagick | applications | 2015-03-18 | \- | 6.9.0 |
-| intel | toolchain | 2020-08-03 | *2013* \<br /> 2017.2.174\<br />2016.2.181\<br />2016.1.150\<br />2015.3.187\<br />2015.2.164\<br />2013-sp1\<br />11.1.069 | *2018.1.163* \<br /> 2018.0.128\<br />2017.4.196\<br />2017.2.174\<br />2017.1.132\<br />2017.0.020\<br />2016.2.181\<br />2016.1.150\<br />2015.3.187\<br />2015.2.164\<br />2015.1.133\<br />2013-sp1\<br />2013\<br />12.1\<br />11.1.069 |
-| intelmpi | libraries | 2017-11-21 | \- | *2018.1.163* \<br /> 5.1.3.181\<br />5.1.2.150\<br />5.0.3.048\<br />5.0.1.035\<br />2018.0.128\<br />2017.3.196\<br />2017.2.174\<br />2017.1.132\<br />2017.0.098\<br />2013 |
-| iotop | tools | 2013-07-16 | \- | 0.5 |
-| iotrack | tools | 2013-07-16 | \- | 0.5 |
-| java | tools | 2015-11-17 | jre1.6.0-21\<br />jdk1.8.0-66\<br />jdk1.7.0-25\<br />jdk1.7.0-03 | jdk1.8.0-66\<br />jdk1.7.0-25 |
-| julia | compilers | 2018-05-15 | \- | *0.6.2* \<br /> 0.4.6\<br />0.4.1 |
-| knime | applications | 2017-03-20 | \- | 3.3.1\<br />3.1.0\<br />2.11.3-24\<br />2.11.3 |
-| lammps | applications | 2017-08-31 | 2014sep\<br />2014jun\<br />2013feb | *2016jul* \<br /> 2017aug\<br />2016may\<br />2015aug\<br />2014sep\<br />2014feb\<br />2013feb\<br />2013aug |
-| lbfgsb | libraries | 2013-08-02 | *3.0* \<br /> 2.1 | *3.0* \<br /> 2.1 |
-| libnbc | libraries | 2014-05-28 | \- | 1.1.1 |
-| libssh2 | tools | 2018-11-22 | \- | 1.8.0 |
-| libsvm | tools | 2015-11-20 | \- | 3.20 |
-| libtool | lib | 2021-02-17 | \- | 2.4.2 |
-| libunwind | lib | 2020-09-25 | \- | 1.1 |
-| libxc | chem | 2021-01-05 | \- | 3.0.0\<br />2.2.2 |
-| liggghts | applications | 2014-05-28 | \- | 2.3.8\<br />2.3.2 |
-| llview | tools | 2015-01-28 | \- | |
-| llvm | compilers | 2018-04-20 | \- | *4.0.0* \<br /> ykt\<br />3.9.1\<br />3.7\<br />3.4\<br />3.3.1 |
-| lo2s | perf | 2020-01-27 | \- | 2018-02-13\<br />2017-12-06\<br />2017-08-07 |
-| ls-dyna | applications | 2017-12-05 | *9.0.1* \<br /> 971\<br />7.0 | *10.0.0* \<br /> dev-121559\<br />971\<br />9.0.1\<br />8.1\<br />7.1.2\<br />7.1.1\<br />7.0\<br />6.0 |
-| ls-dyna-usermat | applications | 2016-08-25 | \- | 9.0.1-s\<br />9.0.1-d\<br />7.1.2-d\<br />7.1.1-s\<br />7.1.1-d |
-| ls-prepost | applications | 2016-08-22 | \- | *4.3* |
-| lumerical | applications | 2016-06-01 | \- | fdtd-8.11.422 |
-| m4 | tools | 2013-10-30 | \- | 1.4.16 |
-| m4ri | libraries | 2017-03-27 | \- | 20140914 |
-| make | tools | 2018-02-21 | \- | 4.2 |
-| map | tools | 2016-11-22 | \- | 6.0.5 |
-| mathematica | applications | 2015-10-16 | \- | *10.0* \<br /> 8.0 |
-| matlab | applications | 2019-02-26 | deprecated.lua\<br />2017a.lua\<br />2016b.lua\<br />2015b.lua\<br />2014a.lua\<br />2013a.lua\<br />2012a.lua\<br />2010b.lua\<br />2010a.lua | \- |
-| maxima | applications | 2017-03-15 | \- | 5.39.0 |
-| med | libraries | 2017-09-27 | \- | 3.2.0 |
-| meep | applications | 2015-04-23 | \- | 1.3\<br />1.2.1 |
-| mercurial | tools | 2014-10-22 | 3.1.2 | 3.1.2 |
-| metis | libraries | 2013-12-17 | 5.1.0 | 5.1.0\<br />4.0.3 |
-| mkl | libraries | 2017-05-10 | 2013 | 2017\<br />2015\<br />2013 |
-| modenv | environment | 2020-03-25 | \- | scs5.lua\<br />ml.lua\<br />hiera.lua\<br />classic.lua |
-| mongodb | applications | 2018-03-19 | \- | 3.6.3 |
-| motioncor2 | applications | 2017-06-21 | \- | 01-30-2017 |
-| mpb | applications | 2014-08-19 | \- | 1.4.2 |
-| mpi4py | libraries | 2016-05-02 | 1.3.1 | 2.0.0\<br />1.3.1 |
-| mpirt | libraries | 2017-11-21 | \- | *2018.1.163* \<br /> 5.1.3.181\<br />5.1.2.150\<br />5.0.3.048\<br />5.0.1.035\<br />2018.0.128\<br />2017.3.196\<br />2017.2.174\<br />2017.1.132\<br />2017.0.098\<br />2013 |
-| mumps | libraries | 2017-05-11 | \- | 5.1.1 |
-| must | tools | 2018-02-01 | \- | 1.5.0\<br />1.4.0 |
-| mvapich2 | libraries | 2017-05-23 | \- | 2.2 |
-| mysql | tools | 2013-12-06 | \- | 6.0.11 |
-| namd | applications | 2015-09-08 | \- | *2.10* \<br /> 2.9 |
-| nco | tools | 2013-08-01 | 4.3.0 | \- |
-| nedit | tools | 2013-04-30 | 5.6\<br />5.5 | 5.6\<br />5.5 |
-| netbeans | applications | 2018-03-07 | \- | 8.2 |
-| netcdf | libraries | 2018-02-09 | 4.1.3 | 4.6.0\<br />4.4.0\<br />4.3.3.1\<br />4.1.3 |
-| netlogo | applications | 2017-08-08 | \- | 6.0.1\<br />5.3.0\<br />5.2.0 |
-| nsys | tools | 2018-09-11 | \- | 2018.1.1.36\<br />2018.0.1.173 |
-| numeca | cae | 2018-11-22 | \- | all |
-| nwchem | applications | 2016-02-15 | 6.3 | *6.6* \<br /> custom\<br />6.5patched\<br />6.5\<br />6.3.r2\<br />6.3 |
-| octave | applications | 2018-03-23 | \- | 3.8.1 |
-| octopus | applications | 2017-03-07 | \- | 6.0 |
-| openbabel | applications | 2014-03-07 | 2.3.2 | 2.3.2 |
-| opencl | libraries | 2015-07-13 | \- | 1.2-4.4.0.117 |
-| openems | applications | 2017-08-01 | \- | 0.0.35 |
-| openfoam | applications | 2020-10-15 | \- | *2.3.0* \<br /> v1712\<br />v1706\<br />5.0\<br />4.0\<br />2.4.0\<br />2.3.1\<br />2.2.2 |
-| openmpi | libraries | 2018-02-01 | \- | *1.10.2* \<br /> 3.0.0\<br />2.1.1\<br />2.1.0\<br />1.8.8\<br />1.10.4\<br />1.10.3 |
-| opentelemac | applications | 2017-09-29 | \- | v7p2r3\<br />v7p1r1 |
-| oprofile | tools | 2013-06-05 | 0.9.8 | \- |
-| orca | applications | 2017-07-27 | \- | 4.0.1\<br />4.0.0.2\<br />3.0.3 |
-| otf2 | libraries | 2018-02-12 | \- | *2.0* \<br /> 2.1\<br />1.4\<br />1.3 |
-| papi | libraries | 2017-11-06 | 5.1.0 | 5.5.1\<br />5.4.3\<br />5.4.1 |
-| parallel | tools | 2020-02-25 | \- | 20170222 |
-| paraview | applications | 2016-03-03 | \- | *4.1.0* \<br /> 4.0.1 |
-| parmetis | libraries | 2018-01-12 | \- | 4.0.3 |
-| pasha | applications | 2013-11-14 | 1.0.9 | \- |
-| pathscale | compilers | 2016-03-04 | \- | enzo-6.0.858\<br />enzo-6.0.749 |
-| pdt | tools | 2015-09-18 | 3.18.1 | 3.18.1 |
-| perf | tools | 2016-06-14 | | |
-| perl | applications | 2015-01-29 | 5.20.1\<br />5.12.1 | 5.20.1\<br />5.12.1 |
-| petsc | libraries | 2018-03-19 | *3.3-p6* \<br /> 3.1-p8 | *3.3-p6* \<br /> 3.8.3-64bit\<br />3.8.3\<br />3.4.4\<br />3.4.3\<br />3.3-p7-64bit\<br />3.3-p7\<br />3.2-p7\<br />3.1-p8-p\<br />3.1-p8 |
-| pgi | compilers | 2018-09-07 | 14.9\<br />14.7\<br />14.6\<br />14.3\<br />13.4 | *18.3* \<br /> 17.7\<br />17.4\<br />17.1\<br />16.9\<br />16.5\<br />16.4\<br />16.10\<br />16.1\<br />15.9\<br />14.9 |
-| pigz | tools | 2018-11-22 | \- | 2.3.4 |
-| prope-env | tools | 2017-05-02 | \- | *.1.0* |
-| protobuf | devel | 2021-03-17 | \- | 3.5.0\<br />3.2.0 |
-| pycuda | libraries | 2016-10-28 | \- | *2016.1.2* \<br /> 2013.1.1\<br />2012.1 |
-| pyslurm | libraries | 2017-11-09 | \- | 16.05.8 |
-| python | libraries | 2018-01-17 | 3.6\<br />3.3.0\<br />2.7.5\<br />2.7 | *3.6* \<br /> intelpython3\<br />intelpython2\<br />3.5.2\<br />3.4.3\<br />3.3.0\<br />3.1.2\<br />2.7.6\<br />2.7.5\<br />2.7 |
-| q-chem | applications | 2016-12-12 | \- | 4.4 |
-| qt | libraries | 2016-10-26 | \- | *4.8.1* \<br /> 5.4.1\<br />4.8.7 |
-| quantum_espresso | applications | 2016-09-13 | *5.0.3* \<br /> 5.0.2 | 5.3.0\<br />5.1.2\<br />5.0.3 |
-| r | applications | 2016-02-18 | \- | 3.2.1\<br />2.15.3 |
-| ramdisk | tools | 2016-07-21 | 1.0 | \- |
-| read-nvml-clocks-pci | tools | 2018-02-22 | \- | 1.0 |
-| readex | tools | 2018-06-13 | \- | pre\<br />beta-1806\<br />beta-1805\<br />beta-1804\<br />beta\<br />alpha |
-| redis | tools | 2016-06-28 | \- | 3.2.1 |
-| relion | applications | 2017-06-21 | \- | 2.1 |
-| repoclient | applications | 2017-01-18 | \- | 1.4.1 |
-| ripgrep | tools | 2017-02-17 | \- | 0.3.2 |
-| robinhood | tools | 2017-05-04 | \- | 2.4.3 |
-| root | applications | 2015-02-27 | \- | 6.02.05 |
-| rstudio | lang | 2020-02-18 | \- | 0.98.1103 |
-| ruby | tools | 2014-07-21 | \- | 2.1.2 |
-| samrai | libraries | 2016-04-22 | \- | 3.10.0 |
-| samtools | tools | 2018-07-13 | 0.1.18 | 1.8 |
-| scafes | libraries | 2017-05-05 | *2.3.0* \<br /> 2.2.0\<br />2.0.0\<br />1.0.0 | *2.3.0* \<br /> 2.2.0\<br />2.1.0\<br />2.0.1\<br />2.0.0\<br />1.0.0 |
-| scala | compilers | 2015-06-22 | \- | 2.11.4\<br />2.10.4 |
-| scalapack | libraries | 2013-11-21 | 2.0.2 | \- |
-| scalasca | tools | 2018-02-06 | \- | *2.3.1* |
-| scons | tools | 2015-11-19 | 2.3.4 | 2.4.1\<br />2.3.4 |
-| scorep | tools | 2018-10-01 | 1.3.0 | *3.0* \<br /> try\<br />trunk\<br />ompt\<br />java\<br />dev-io\<br />3.1 |
-| scorep-apapi | libraries | 2018-01-09 | \- | gcc-2018-01-09 |
-| scorep-cpu-energy | libraries | 2017-01-20 | \- | r217\<br />r211\<br />r117\<br />2017-01-20\<br />2016-04-07 |
-| scorep-cpu-id | libraries | 2014-08-13 | \- | r117 |
-| scorep-dataheap | libraries | 2015-07-28 | \- | *2015-07-28* \<br /> r191\<br />r122 |
-| scorep-dev | tools | 2017-07-19 | \- | *05* |
-| scorep-hdeem | libraries | 2018-06-19 | \- | *2016-12-20* \<br /> sync\<br />2017-12-08a\<br />2017-12-08.lua\<br />2016-11-21 |
-| scorep-plugin-x86-energy | libraries | 2018-06-19 | \- | xmpi\<br />intelmpi\<br />2017-09-06\<br />2017-09-05 |
-| scorep-printmetrics | libraries | 2018-02-26 | \- | 2018-02-26 |
-| scorep-uncore | libraries | 2018-06-21 | \- | *2018-01-24* \<br /> 2016-03-29 |
-| scorep_plugin_x86_energy | libraries | 2018-07-04 | \- | intel-2018\<br />gcc-7.1.0\<br />2017-07-14 |
-| scout | compilers | 2015-06-22 | 1.6.0 | 1.6.0 |
-| sed | tools | 2018-02-21 | \- | 4.4 |
-| sftp | tools | 2014-04-10 | \- | 6.6 |
-| shifter | tools | 2016-06-09 | \- | 16.04.0pre1 |
-| siesta | applications | 2017-05-29 | 3.1-pl20 | 4.0\<br />3.2-pl4 |
-| singularity | tools | 2019-02-13 | | ff69c5f3 |
-| sionlib | tools | 2017-06-29 | \- | 1.6.1\<br />1.5.5 |
-| siox | libraries | 2016-10-27 | \- | 2016-10-27\<br />2016-10-26 |
-| spm | applications | 2014-07-09 | \- | 8-r4667 |
-| spm12 | libraries | 2017-07-12 | \- | r6906 |
-| spparks | applications | 2016-06-30 | \- | 2016feb |
-| sqlite3 | libraries | 2016-07-27 | 3.8.2 | 3.8.10 |
-| sra-tools | tools | 2018-07-19 | \- | 2.9.1 |
-| stack | tools | 2016-06-23 | \- | 1.1.2 |
-| star | applications | 2017-10-25 | \- | *12.06* \<br /> 9.06\<br />12.04\<br />12.02\<br />11.02\<br />10.04 |
-| subread | tools | 2018-07-13 | \- | 1.6.2 |
-| suitesparse | libraries | 2017-08-25 | \- | *4.5.4* \<br /> 4.2.1 |
-| superlu | libraries | 2017-08-25 | \- | *5.2.1* |
-| superlu_dist | libraries | 2017-03-21 | \- | 5.1.3 |
-| superlu_mt | libraries | 2017-03-21 | \- | 3.1 |
-| svn | tools | 2016-03-16 | 1.8.11\<br />1.7.3 | *1.9.3* \<br /> 1.8.8\<br />1.8.11 |
-| swig | tools | 2016-04-08 | 2.0 | 3.0.8\<br />2.0 |
-| swipl | tools | 2017-02-28 | \- | 7.4.0-rc2 |
-| tcl | applications | 2017-08-07 | \- | 8.6.6 |
-| tcltk | applications | 2015-02-06 | \- | 8.4.20 |
-| tecplot360 | applications | 2018-05-22 | 2015\<br />2013 | 2018r1\<br />2017r2\<br />2017r1\<br />2016r2\<br />2015r2\<br />2015\<br />2013\<br />2010 |
-| tesseract | libraries | 2016-06-28 | \- | 3.04 |
-| texinfo | devel | 2020-08-11 | \- | 5.2 |
-| theodore | libraries | 2016-05-27 | \- | 1.3 |
-| tiff | libraries | 2013-09-16 | 3.9.2 | 3.9.2 |
-| tinker | applications | 2014-03-04 | \- | 6.3 |
-| tmux | tools | 2021-04-09 | \- | 2.2 |
-| totalview | tools | 2017-11-03 | *2017.2.11* \<br /> 8.9.2-0\<br />8.8.0-1\<br />8.13.0-0\<br />8.11.0-3 | *2017.2.11* \<br /> 8.9.2-0\<br />8.8.0-1\<br />8.13.0-0\<br />8.11.0-3 |
-| trilinos | applications | 2016-04-13 | \- | 12.6.1 |
-| trinityrnaseq | applications | 2013-05-16 | r2013-02-25 | \- |
-| turbomole | applications | 2017-01-10 | 7.1\<br />6.6\<br />6.5 | 7.1\<br />6.6\<br />6.5 |
-| valgrind | tools | 2017-08-08 | 3.8.1 | *3.10.1* \<br /> r15216\<br />3.8.1\<br />3.13.0 |
-| vampir | tools | 2019-03-04 | *9.6.1* \<br /> 9.7\<br />9.5.0\<br />9.4.0\<br />9.3.0\<br />8.5.0 | *9.5.0* \<br /> 9.4.0\<br />9.3.0\<br />9.3\<br />9.2.0\<br />9.1.0\<br />9.0.0\<br />8.5.0\<br />8.4.1\<br />8.3.0 |
-| vampirlive | tools | 2018-02-27 | \- | |
-| vampirtrace | tools | 2016-03-29 | *5.14.4* | *5.14.4* |
-| vampirtrace-plugins | libraries | 2014-08-06 | \- | x86\<br />power-1.1\<br />power-1.0\<br />apapi |
-| vasp | applications | 2017-11-08 | *5.3* \<br /> 5.2 | 5.4.4\<br />5.4.1\<br />5.3 |
-| visit | applications | 2016-12-13 | \- | 2.4.2\<br />2.12.0 |
-| vmd | applications | 2016-12-15 | \- | 1.9.3 |
-| vt_dataheap | libraries | 2014-08-14 | \- | r190 |
-| vtk | libraries | 2016-08-16 | 5.10.1 | 5.10.1 |
-| wannier90 | libraries | 2016-04-21 | 1.2 | 2.0.1\<br />1.2 |
-| wget | tools | 2015-05-12 | 1.16.3 | 1.16.3 |
-| wxwidgets | libraries | 2017-03-15 | \- | 3.0.2 |
-| yade | applications | 2014-05-22 | \- | |
-| zlib | lib | 2020-12-11 | \- | 1.2.8 |
-
-### ML Environment
-
-<span class="twiki-macro TABLE">headerrows= 1"</span> <span
-class="twiki-macro EDITTABLE"
-format="| text, 30, Software | text, 40, Kategorie | text, 30, Letzte Änderung | text, 30, SGI-UV | text, 30, Taurus | "
-changerows="on"></span>
-
-| Software | Category | Last change | Venus | Taurus |
-|:--------------------------|:----------|:------------|:------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| ASE | chem | 2020-09-29 | \- | 3.19.0 |
-| ATK | vis | 2020-10-12 | \- | 2.34.1\<br />2.28.1 |
-| Anaconda3 | lang | 2019-11-13 | \- | 2019.07\<br />2019.03 |
-| Arrow | data | 2020-11-30 | \- | 0.16.0\<br />0.14.1 |
-| Autoconf | devel | 2020-10-14 | \- | 2.69 |
-| Automake | devel | 2020-10-14 | \- | 1.16.1\<br />1.15.1 |
-| Autotools | devel | 2020-10-14 | \- | 20180311\<br />20170619 |
-| Bazel | devel | 2021-03-22 | \- | 3.7.1\<br />3.4.1\<br />2.0.0\<br />1.1.0\<br />0.29.1\<br />0.26.1\<br />0.25.2\<br />0.20.0\<br />0.18.0 |
-| BigDataFrameworkConfigure | devel | 2019-09-16 | \- | 0.0.2\<br />0.0.1 |
-| Bison | lang | 2020-10-14 | \- | 3.5.3\<br />3.3.2\<br />3.0.5\<br />3.0.4 |
-| Boost | devel | 2020-11-30 | \- | 1.71.0\<br />1.70.0\<br />1.69.0\<br />1.67.0\<br />1.66.0 |
-| CMake | devel | 2020-11-02 | \- | 3.9.5\<br />3.9.1\<br />3.16.4\<br />3.15.3\<br />3.13.3\<br />3.12.1\<br />3.11.4\<br />3.10.2 |
-| CUDA | system | 2020-10-14 | \- | 9.2.88\<br />11.0.2\<br />10.1.243\<br />10.1.105 |
-| CUDAcore | system | 2020-10-14 | \- | 11.0.2 |
-| Check | lib | 2020-10-14 | \- | 0.15.2 |
-| Clang | compiler | 2020-02-20 | \- | 9.0.1 |
-| CubeLib | perf | 2020-07-23 | \- | 4.4.4\<br />4.4 |
-| CubeW | perf | 2019-02-04 | \- | 4.4 |
-| CubeWriter | perf | 2020-07-23 | \- | 4.4.3 |
-| DBus | devel | 2019-09-11 | \- | 1.13.8 |
-| Devel-NYTProf | perf | 2019-08-30 | \- | 6.06 |
-| Doxygen | devel | 2019-07-17 | \- | 1.8.14\<br />1.8.13 |
-| EasyBuild | tools | 2021-03-08 | \- | 4.3.3\<br />4.3.2\<br />4.3.1\<br />4.3.0\<br />4.2.2\<br />4.2.0\<br />4.1.2\<br />4.1.1\<br />4.1.0\<br />4.0.1\<br />3.9.4\<br />3.9.3\<br />3.9.2\<br />3.9.1\<br />3.8.1\<br />3.8.0\<br />3.7.1\<br />3.7.0\<br />3.6.2\<br />3.6.1 |
-| Eigen | math | 2020-11-02 | \- | 3.3.7 |
-| FFTW | numlib | 2020-10-14 | \- | 3.3.8\<br />3.3.7\<br />3.3.6 |
-| FFmpeg | vis | 2020-09-29 | \- | 4.2.1\<br />4.1 |
-| Flink | devel | 2019-09-16 | \- | 1.9.0\<br />1.8.1 |
-| FriBidi | lang | 2020-09-29 | \- | 1.0.5 |
-| GCC | compiler | 2020-10-14 | \- | 9.3.0\<br />8.3.0\<br />8.2.0-2.31.1\<br />7.3.0-2.30\<br />6.4.0-2.28 |
-| GCCcore | compiler | 2020-10-14 | \- | 9.3.0\<br />8.3.0\<br />8.2.0\<br />7.3.0\<br />6.4.0 |
-| GDAL | data | 2019-08-19 | \- | 2.2.3 |
-| GDRCopy | lib | 2020-10-14 | \- | 2.1 |
-| GEOS | math | 2019-08-19 | \- | 3.6.2 |
-| GLib | vis | 2020-02-19 | \- | 2.62.0\<br />2.60.1\<br />2.54.3 |
-| GMP | math | 2020-11-02 | \- | 6.2.0\<br />6.1.2 |
-| GObject-Introspection | devel | 2020-10-12 | \- | 1.63.1\<br />1.54.1 |
-| GSL | numlib | 2020-02-19 | \- | 2.6\<br />2.5 |
-| GTK+ | vis | 2019-02-15 | \- | 2.24.32 |
-| Gdk-Pixbuf | vis | 2019-02-15 | \- | 2.36.12 |
-| Ghostscript | tools | 2020-02-19 | \- | 9.50\<br />9.27 |
-| HDF5 | data | 2020-02-19 | \- | 1.10.5\<br />1.10.2\<br />1.10.1 |
-| Hadoop | devel | 2019-09-16 | \- | 2.7.7 |
-| HarfBuzz | vis | 2019-02-15 | \- | 2.2.0 |
-| Horovod | tools | 2020-08-04 | \- | 0.19.5\<br />0.18.2 |
-| Hyperopt | lib | 2020-02-19 | \- | 0.2.2 |
-| ICU | lib | 2020-02-19 | \- | 64.2\<br />61.1\<br />56.1 |
-| ImageMagick | vis | 2020-02-20 | \- | 7.0.9-5\<br />7.0.8-46 |
-| JUnit | devel | 2020-01-21 | \- | 4.12 |
-| JasPer | vis | 2020-02-19 | \- | 2.0.14 |
-| Java | lang | 2020-02-26 | \- | 11.0.6\<br />1.8.0-162\<br />1.8-191-b26 |
-| JsonCpp | lib | 2020-10-30 | \- | 1.9.3 |
-| Keras | math | 2019-06-28 | \- | 2.2.4 |
-| LAME | data | 2020-06-24 | \- | 3.100 |
-| LLVM | compiler | 2020-09-24 | \- | 9.0.0\<br />8.0.1\<br />7.0.1\<br />6.0.0\<br />5.0.1 |
-| LMDB | lib | 2020-10-30 | \- | 0.9.24 |
-| LibTIFF | lib | 2020-02-19 | \- | 4.0.9\<br />4.0.10 |
-| LibUUID | lib | 2019-08-02 | \- | 1.0.3 |
-| LittleCMS | vis | 2020-02-20 | \- | 2.9 |
-| M4 | devel | 2020-10-14 | \- | 1.4.18\<br />1.4.17 |
-| MPFR | math | 2020-06-24 | \- | 4.0.2 |
-| Mako | devel | 2020-02-19 | \- | 1.1.0\<br />1.0.8\<br />1.0.7 |
-| Mesa | vis | 2020-02-19 | \- | 19.1.7\<br />19.0.1\<br />18.1.1\<br />17.3.6 |
-| Meson | tools | 2020-11-03 | \- | 0.55.1\<br />0.51.2\<br />0.50.0 |
-| MongoDB | data | 2019-08-05 | \- | 4.0.3 |
-| NASM | lang | 2020-02-19 | \- | 2.14.02\<br />2.13.03 |
-| NCCL | lib | 2020-02-18 | \- | 2.4.8\<br />2.4.2\<br />2.3.7 |
-| NLopt | numlib | 2020-02-19 | \- | 2.6.1\<br />2.4.2 |
-| NSPR | lib | 2019-09-11 | \- | 4.21 |
-| NSS | lib | 2019-09-11 | \- | 3.42.1 |
-| Ninja | tools | 2020-11-03 | \- | 1.9.0\<br />1.10.0 |
-| OPARI2 | perf | 2020-07-23 | \- | 2.0.5\<br />2.0.3 |
-| OTF2 | perf | 2020-07-23 | \- | 2.2\<br />2.1.1 |
-| OpenBLAS | numlib | 2020-10-14 | \- | 0.3.9\<br />0.3.7\<br />0.3.5\<br />0.3.1\<br />0.2.20 |
-| OpenCV | vis | 2019-02-21 | \- | 4.0.1 |
-| OpenMPI | mpi | 2021-02-10 | \- | 4.0.3\<br />3.1.4\<br />3.1.3\<br />3.1.1 |
-| OpenPGM | system | 2019-09-11 | \- | 5.2.122 |
-| PAPI | perf | 2020-07-23 | \- | 6.0.0\<br />5.6.0 |
-| PCRE | devel | 2020-02-19 | \- | 8.43\<br />8.41 |
-| PCRE2 | devel | 2019-09-11 | \- | 10.33 |
-| PDT | perf | 2020-07-23 | \- | 3.25 |
-| PGI | compiler | 2019-05-14 | \- | 19.4 |
-| PMIx | lib | 2020-10-14 | \- | 3.1.5 |
-| PROJ | lib | 2019-08-19 | \- | 5.0.0 |
-| Pango | vis | 2019-02-15 | \- | 1.42.4 |
-| Perl | lang | 2020-10-14 | \- | 5.30.2\<br />5.30.0\<br />5.28.1\<br />5.28.0\<br />5.26.1 |
-| Pillow | vis | 2020-06-24 | \- | 6.2.1 |
-| PowerAI | data | 2019-12-10 | \- | 1.7.0.a0\<br />1.6.1 |
-| PyTorch | devel | 2020-09-29 | \- | 1.6.0\<br />1.3.1\<br />1.1.0 |
-| PyTorch-Geometric | devel | 2020-09-29 | \- | 1.6.1 |
-| PyYAML | lib | 2020-02-18 | \- | 5.1.2\<br />3.13 |
-| Python | lang | 2020-11-02 | \- | 3.8.2\<br />3.7.4\<br />3.7.2\<br />3.6.6\<br />3.6.4\<br />2.7.16\<br />2.7.15\<br />2.7.14 |
-| PythonAnaconda | lang | 2019-12-10 | \- | 3.7\<br />3.6 |
-| Qt5 | devel | 2019-09-12 | \- | 5.12.3 |
-| R | lang | 2020-08-20 | \- | 3.6.2\<br />3.6.0\<br />3.4.4 |
-| RDFlib | lib | 2020-09-29 | \- | 4.2.2 |
-| SCons | devel | 2019-09-11 | \- | 3.0.5 |
-| SIONlib | lib | 2020-07-23 | \- | 1.7.6 |
-| SQLite | devel | 2020-11-02 | \- | 3.31.1\<br />3.29.0\<br />3.27.2\<br />3.24.0\<br />3.21.0\<br />3.20.1 |
-| SWIG | devel | 2020-10-30 | \- | 4.0.1\<br />3.0.12 |
-| ScaLAPACK | numlib | 2020-10-14 | \- | 2.1.0\<br />2.0.2 |
-| SciPy-bundle | lang | 2020-11-02 | \- | 2020.03\<br />2019.10 |
-| Score-P | perf | 2020-07-23 | \- | 6.0\<br />4.1 |
-| Six | lib | 2019-02-05 | \- | 1.11.0 |
-| Spark | devel | 2020-09-29 | \- | 3.0.1\<br />2.4.4\<br />2.4.3 |
-| SpectrumMPI | mpi | 2019-01-14 | \- | system |
-| Szip | tools | 2020-02-18 | \- | 2.1.1 |
-| Tcl | lang | 2020-11-02 | \- | 8.6.9\<br />8.6.8\<br />8.6.7\<br />8.6.10 |
-| TensorFlow | lib | 2020-10-30 | \- | 2.3.1\<br />2.2.0\<br />2.1.0\<br />2.0.0\<br />1.15.0\<br />1.14.0 |
-| Tk | vis | 2020-02-19 | \- | 8.6.9\<br />8.6.8 |
-| Tkinter | lang | 2020-09-23 | \- | 3.7.4\<br />3.6.6 |
-| UCX | lib | 2020-10-14 | \- | 1.8.0 |
-| UDUNITS | phys | 2020-02-19 | \- | 2.2.26 |
-| UnZip | tools | 2020-11-02 | \- | 6.0 |
-| Vampir | perf | 2020-11-30 | \- | 9.9.0\<br />9.8.0\<br />9.7.1\<br />9.11\<br />9.10.0 |
-| X11 | vis | 2020-11-03 | \- | 20200222\<br />20190717\<br />20190311\<br />20180604\<br />20180131 |
-| XML-Parser | data | 2019-02-15 | \- | 2.44-01 |
-| XZ | tools | 2020-10-14 | \- | 5.2.5\<br />5.2.4\<br />5.2.3 |
-| Yasm | lang | 2020-09-29 | \- | 1.3.0 |
-| ZeroMQ | devel | 2019-09-11 | \- | 4.3.2 |
-| Zip | tools | 2020-07-30 | \- | 3.0 |
-| ant | devel | 2020-10-12 | \- | 1.10.7\<br />1.10.1 |
-| binutils | tools | 2020-10-14 | \- | 2.34\<br />2.32\<br />2.31.1\<br />2.30\<br />2.28 |
-| bokeh | tools | 2020-09-29 | \- | 1.4.0 |
-| bzip2 | tools | 2020-11-02 | \- | 1.0.8\<br />1.0.6 |
-| cURL | tools | 2020-11-02 | \- | 7.69.1\<br />7.66.0\<br />7.63.0\<br />7.60.0\<br />7.58.0 |
-| cairo | vis | 2020-02-19 | \- | 1.16.0\<br />1.14.12 |
-| cftime | data | 2019-07-17 | \- | 1.0.1 |
-| cuDNN | numlib | 2020-02-18 | \- | 7.6.4.38\<br />7.4.2.24\<br />7.1.4.18 |
-| dask | data | 2020-09-29 | \- | 2.8.0 |
-| dill | data | 2019-10-29 | \- | 0.3.1.1 |
-| double-conversion | lib | 2020-08-13 | \- | 3.1.4 |
-| expat | tools | 2020-10-14 | \- | 2.2.9\<br />2.2.7\<br />2.2.6\<br />2.2.5 |
-| flatbuffers | devel | 2020-10-30 | \- | 1.12.0 |
-| flatbuffers-python | devel | 2021-04-10 | \- | 1.12 |
-| flex | lang | 2020-10-14 | \- | 2.6.4\<br />2.6.3 |
-| fontconfig | vis | 2020-11-03 | \- | 2.13.92\<br />2.13.1\<br />2.13.0\<br />2.12.6 |
-| fosscuda | toolchain | 2020-10-14 | \- | 2020a\<br />2019b\<br />2019a\<br />2018b |
-| freetype | vis | 2020-11-03 | \- | 2.9.1\<br />2.9\<br />2.10.1 |
-| future | lib | 2019-02-05 | \- | 0.16.0 |
-| gcccuda | toolchain | 2020-10-14 | \- | 2020a\<br />2019b\<br />2019a\<br />2018b |
-| gettext | tools | 2020-11-03 | \- | 0.20.1\<br />0.19.8.1 |
-| gflags | devel | 2020-06-24 | \- | 2.2.2 |
-| giflib | lib | 2020-10-30 | \- | 5.2.1 |
-| git | tools | 2020-02-18 | \- | 2.23.0\<br />2.18.0 |
-| glog | devel | 2020-06-24 | \- | 0.4.0 |
-| golf | toolchain | 2019-01-14 | \- | 2018a |
-| gompic | toolchain | 2020-10-14 | \- | 2020a\<br />2019b\<br />2019a\<br />2018b |
-| gperf | devel | 2020-11-03 | \- | 3.1 |
-| gsmpi | toolchain | 2019-01-14 | \- | 2018a |
-| gsolf | toolchain | 2019-01-14 | \- | 2018a |
-| h5py | data | 2020-07-30 | \- | 2.8.0\<br />2.10.0 |
-| help2man | tools | 2020-10-14 | \- | 1.47.8\<br />1.47.6\<br />1.47.4\<br />1.47.12 |
-| hwloc | system | 2020-10-14 | \- | 2.2.0\<br />2.0.3\<br />1.11.12\<br />1.11.11\<br />1.11.10 |
-| hypothesis | tools | 2020-06-24 | \- | 4.44.2 |
-| intltool | devel | 2020-11-03 | \- | 0.51.0 |
-| libGLU | vis | 2020-02-19 | \- | 9.0.1\<br />9.0.0 |
-| libdrm | lib | 2020-02-19 | \- | 2.4.99\<br />2.4.97\<br />2.4.92\<br />2.4.91 |
-| libevent | lib | 2020-10-14 | \- | 2.1.8\<br />2.1.11 |
-| libfabric | lib | 2021-02-10 | \- | 1.11.0 |
-| libffi | lib | 2020-11-02 | \- | 3.3\<br />3.2.1 |
-| libgeotiff | lib | 2019-08-19 | \- | 1.4.2 |
-| libjpeg-turbo | lib | 2020-02-19 | \- | 2.0.3\<br />2.0.2\<br />2.0.0\<br />1.5.3 |
-| libpciaccess | system | 2020-10-14 | \- | 0.16\<br />0.14 |
-| libpng | lib | 2020-11-03 | \- | 1.6.37\<br />1.6.36\<br />1.6.34 |
-| libreadline | lib | 2020-10-14 | \- | 8.0\<br />7.0 |
-| libsndfile | lib | 2020-02-19 | \- | 1.0.28 |
-| libsodium | lib | 2019-09-11 | \- | 1.0.17 |
-| libtool | lib | 2020-10-14 | \- | 2.4.6 |
-| libunwind | lib | 2020-09-25 | \- | 1.3.1\<br />1.2.1 |
-| libxml2 | lib | 2020-10-14 | \- | 2.9.9\<br />2.9.8\<br />2.9.7\<br />2.9.4\<br />2.9.10 |
-| libxslt | lib | 2020-10-27 | \- | 1.1.34\<br />1.1.33\<br />1.1.32 |
-| libyaml | lib | 2020-01-24 | \- | 0.2.2\<br />0.2.1 |
-| magma | math | 2020-09-29 | \- | 2.5.1 |
-| matplotlib | vis | 2020-09-23 | \- | 3.1.1\<br />3.0.3 |
-| ncurses | devel | 2020-10-14 | \- | 6.2\<br />6.1\<br />6.0 |
-| netCDF | data | 2019-07-17 | \- | 4.6.1\<br />4.6.0 |
-| netcdf4-python | data | 2019-07-17 | \- | 1.4.3 |
-| nettle | lib | 2020-02-19 | \- | 3.5.1\<br />3.4.1\<br />3.4 |
-| nsync | devel | 2020-10-30 | \- | 1.24.0 |
-| numactl | tools | 2020-10-14 | \- | 2.0.13\<br />2.0.12\<br />2.0.11 |
-| numba | lang | 2020-09-25 | \- | 0.47.0 |
-| pixman | vis | 2020-02-19 | \- | 0.38.4\<br />0.38.0\<br />0.34.0 |
-| pkg-config | devel | 2020-10-14 | \- | 0.29.2 |
-| pkgconfig | devel | 2020-02-18 | \- | 1.5.1\<br />1.3.1 |
-| pocl | lib | 2020-04-22 | \- | 1.4 |
-| protobuf | devel | 2020-01-24 | \- | 3.6.1.2\<br />3.6.1\<br />3.10.0 |
-| protobuf-python | devel | 2020-10-30 | \- | 3.10.0 |
-| pybind11 | lib | 2020-11-02 | \- | 2.4.3 |
-| re2c | tools | 2019-09-11 | \- | 1.1.1 |
-| rstudio | lang | 2020-01-21 | \- | 1.2.5001 |
-| scikit-image | vis | 2020-09-29 | \- | 0.16.2 |
-| scikit-learn | data | 2020-09-24 | \- | 0.21.3 |
-| snappy | lib | 2020-10-30 | \- | 1.1.7 |
-| spleeter | tools | 2020-10-05 | \- | 1.5.4 |
-| torchvision | vis | 2021-03-11 | \- | 0.7.0 |
-| tqdm | lib | 2020-09-29 | \- | 4.41.1 |
-| typing-extensions | devel | 2021-04-10 | \- | 3.7.4.3 |
-| util-linux | tools | 2020-11-03 | \- | 2.35\<br />2.34\<br />2.33\<br />2.32.1\<br />2.32\<br />2.31.1 |
-| wheel | tools | 2019-01-30 | \- | 0.31.1 |
-| x264 | vis | 2020-06-24 | \- | 20190925\<br />20181203 |
-| x265 | vis | 2020-09-29 | \- | 3.2\<br />3.0 |
-| xorg-macros | devel | 2020-10-14 | \- | 1.19.2 |
-| zlib | lib | 2020-10-14 | \- | 1.2.11 |
-| zsh | tools | 2021-01-06 | \- | 5.8 |
diff --git a/twiki2md/root/PerformanceTools/IOTrack.md b/twiki2md/root/PerformanceTools/IOTrack.md
deleted file mode 100644
index f20334c8ead2ae2fcb75e39a7b1096ec524e0cbc..0000000000000000000000000000000000000000
--- a/twiki2md/root/PerformanceTools/IOTrack.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# Introduction
-
-IOTrack is a small tool developed at ZIH that tracks the I/O requests of
-all processes and dumps a statistic per process at the end of the
-program run.
-
-# How it works
-
-On taurus load the module via
-
- module load iotrack
-
-Then, instead of running your normal command, put "iotrack" in front of
-it. So,
-
- python xyz.py arg1 arg2
-
-changes to:
-
- iotrack python xyz.py arg1 arg2
-
-# Technical Details
-
-The functionality is implemented in a library that is preloaded via
-LD_PRELOAD. Thus, this will not work for static binaries.
-
--- Main.MichaelKluge - 2013-07-16
diff --git a/twiki2md/root/SoftwareDevelopment/PerformanceTools.md b/twiki2md/root/SoftwareDevelopment/PerformanceTools.md
deleted file mode 100644
index eb353e8d62612b27761368ce9d6a6326918e1d2f..0000000000000000000000000000000000000000
--- a/twiki2md/root/SoftwareDevelopment/PerformanceTools.md
+++ /dev/null
@@ -1,15 +0,0 @@
-# Performance Tools
-
-- [Score-P](ScoreP) - tool suite for profiling, event tracing, and
- online analysis of HPC applications
-- [VampirTrace](VampirTrace) - recording performance relevant data at
- runtime
-- [Vampir](Vampir) - visualizing performance data from your program
-- [Hardware performance counters - PAPI](PapiLibrary) - generic
- performance counters
-- [perf tools](PerfTools) - general performance statistic
-- [IOTrack](IOTrack) - I/O statistics
-- [EnergyMeasurement](EnergyMeasurement) - energy/power measurements
- on taurus
-
--- Main.mark - 2009-12-16
diff --git a/twiki2md/root/SystemTaurus/EnergyMeasurement.md b/twiki2md/root/SystemTaurus/EnergyMeasurement.md
deleted file mode 100644
index 607263056fd593b5f0ae62474a1c63961c8b31aa..0000000000000000000000000000000000000000
--- a/twiki2md/root/SystemTaurus/EnergyMeasurement.md
+++ /dev/null
@@ -1,310 +0,0 @@
-# Energy Measurement Infrastructure
-
-All nodes of the HPC machine Taurus are equipped with power
-instrumentation that allow the recording and accounting of power
-dissipation and energy consumption data. The data is made available
-through several different interfaces, which will be described below.
-
-## System Description
-
-The Taurus system is split into two phases. While both phases are
-equipped with energy-instrumented nodes, the instrumentation
-significantly differs in the number of instrumented nodes and their
-spatial and temporal granularity.
-
-### Phase 1
-
-In phase one, the 270 Sandy Bridge nodes are equipped with node-level
-power instrumentation that is stored in the Dataheap infrastructure at a
-rate of 1Sa/s and further the energy consumption of a job is available
-in SLURM (see below).
-
-### Phase 2
-
-In phase two, all of the 1456 Haswell DLC nodes are equipped with power
-instrumentation. In addition to the access methods of phase one, users
-will also be able to access the measurements through a C API to get the
-full temporal and spatial resolution, as outlined below:
-
-- ** Blade:**1 kSa/s for the whole node, includes both sockets, DRAM,
- SSD, and other on-board consumers. Since the system is directly
- water cooled, no cooling components are included in the blade
- consumption.
-- **Voltage regulators (VR):** 100 Sa/s for each of the six VR
- measurement points, one for each socket and four for eight DRAM
- lanes (two lanes bundled).
-
-The GPU blades of each Phase as well as the Phase I Westmere partition
-also have 1 Sa/s power instrumentation but have a lower accuracy.
-
-HDEEM is now generally available on all nodes in the "haswell"
-partition.
-
-## Summary of Measurement Interfaces
-
-| Interface | Sensors | Rate | Phase I | Phase II Haswell |
-|:-------------------------------------------|:----------------|:--------------------------------|:--------|:-----------------|
-| Dataheap (C, Python, VampirTrace, Score-P) | Blade, (CPU) | 1 Sa/s | yes | yes |
-| HDEEM\* (C, Score-P) | Blade, CPU, DDR | 1 kSa/s (Blade), 100 Sa/s (VRs) | no | yes |
-| HDEEM Command Line Interface | Blade, CPU, DDR | 1 kSa/s (Blade), 100 Sa/s (VR) | no | yes |
-| SLURM Accounting (sacct) | Blade | Per Job Energy | yes | yes |
-| SLURM Profiling (hdf5) | Blade | up to 1 Sa/s | yes | yes |
-
-Note: Please specify `-p haswell --exclusive` along with your job
-request if you wish to use hdeem.
-
-## Accuracy
-
-HDEEM measurements have an accuracy of 2 % for Blade (node)
-measurements, and 5 % for voltage regulator (CPU, DDR) measurements.
-
-## Command Line Interface
-
-The HDEEM infrastructure can be controlled through command line tools
-that are made available by loading the **hdeem** module. They are
-commonly used on the node under test to start, stop, and query the
-measurement device.
-
-- **startHdeem**: Start a measurement. After the command succeeds, the
- measurement data with the 1000 / 100 Sa/s described above will be
- recorded on the Board Management Controller (BMC), which is capable
- of storing up to 8h of measurement data.
-- **stopHdeem**: Stop a measurement. No further data is recorded and
- the previously recorded data remains available on the BMC.
-- **printHdeem**: Read the data from the BMC. By default, the data is
- written into a CSV file, whose name can be controlled using the
- **-o** argument.
-- **checkHdeem**: Print the status of the measurement device.
-- **clearHdeem**: Reset and clear the measurement device. No further
- data can be read from the device after this command is executed
- before a new measurement is started.
-
-## Integration in Application Performance Traces
-
-The per-node power consumption data can be included as metrics in
-application traces by using the provided metric plugins for Score-P (and
-VampirTrace). The plugins are provided as modules and set all necessary
-environment variables that are required to record data for all nodes
-that are part of the current job.
-
-For 1 Sa/s Blade values (Dataheap):
-
-- [Score-P](ScoreP): use the module **`scorep-dataheap`**
-- [VampirTrace](VampirTrace): use the module
- **vampirtrace-plugins/power-1.1**
-
-For 1000 Sa/s (Blade) and 100 Sa/s (CPU{0,1}, DDR{AB,CD,EF,GH}):
-
-- [Score-P](ScoreP): use the module **\<span
- class="WYSIWYG_TT">scorep-hdeem\</span>**\<br />Note: %ENDCOLOR%This
- module requires a recent version of "scorep/sync-...". Please use
- the latest that fits your compiler & MPI version.**\<br />**
-- [VampirTrace](VampirTrace): not supported
-
-By default, the modules are set up to record the power data for the
-nodes they are used on. For further information on how to change this
-behavior, please use module show on the respective module.
-
- # Example usage with gcc
- % module load scorep/trunk-2016-03-17-gcc-xmpi-cuda7.5
- % module load scorep-dataheap
- % scorep gcc application.c -o application
- % srun ./application
-
-Once the application is finished, a trace will be available that allows
-you to correlate application functions with the component power
-consumption of the parallel application. Note: For energy measurements,
-only tracing is supported in Score-P/VampirTrace. The modules therefore
-disables profiling and enables tracing, please use [Vampir](Vampir) to
-view the trace.
-
-\<img alt="demoHdeem_high_low_vampir_3.png" height="262"
-src="%ATTACHURL%/demoHdeem_high_low_vampir_3.png" width="695" />
-
-%RED%Note<span class="twiki-macro ENDCOLOR"></span>: the power
-measurement modules **`scorep-dataheap`** and **`scorep-hdeem`** are
-dynamic and only need to be loaded during execution. However,
-**`scorep-hdeem`** does require the application to be linked with a
-certain version of Score-P.
-
-By default,** `scorep-dataheap`**records all sensors that are available.
-Currently this is the total node consumption and for Phase II the CPUs.
-**`scorep-hdeem`** also records all available sensors (node, 2x CPU, 4x
-DDR) by default. You can change the selected sensors by setting the
-environment variables:
-
- # For HDEEM
- % export SCOREP_METRIC_HDEEM_PLUGIN=Blade,CPU*
- # For Dataheap
- % export SCOREP_METRIC_DATAHEAP_PLUGIN=localhost/watts
-
-For more information on how to use Score-P, please refer to the
-[respective documentation](ScoreP).
-
-## Access Using Slurm Tools
-
-[Slurm](Slurm) maintains its own database of job information, including
-energy data. There are two main ways of accessing this data, which are
-described below.
-
-### Post-Mortem Per-Job Accounting
-
-This is the easiest way of accessing information about the energy
-consumed by a job and its job steps. The Slurm tool `sacct` allows users
-to query post-mortem energy data for any past job or job step by adding
-the field `ConsumedEnergy` to the `--format` parameter:
-
- $> sacct --format="jobid,jobname,ntasks,submit,start,end,ConsumedEnergy,nodelist,state" -j 3967027
- JobID JobName NTasks Submit Start End ConsumedEnergy NodeList State
- ------------ ---------- -------- ------------------- ------------------- ------------------- -------------- --------------- ----------
- 3967027 bash 2014-01-07T12:25:42 2014-01-07T12:25:52 2014-01-07T12:41:20 taurusi1159 COMPLETED
- 3967027.0 sleep 1 2014-01-07T12:26:07 2014-01-07T12:26:07 2014-01-07T12:26:18 0 taurusi1159 COMPLETED
- 3967027.1 sleep 1 2014-01-07T12:29:06 2014-01-07T12:29:06 2014-01-07T12:29:16 1.67K taurusi1159 COMPLETED
- 3967027.2 sleep 1 2014-01-07T12:33:25 2014-01-07T12:33:25 2014-01-07T12:33:36 1.84K taurusi1159 COMPLETED
- 3967027.3 sleep 1 2014-01-07T12:34:06 2014-01-07T12:34:06 2014-01-07T12:34:11 1.09K taurusi1159 COMPLETED
- 3967027.4 sleep 1 2014-01-07T12:38:03 2014-01-07T12:38:03 2014-01-07T12:39:44 18.93K taurusi1159 COMPLETED
-
-The job consisted of 5 job steps, each executing a sleep of a different
-length. Note that the ConsumedEnergy metric is only applicable to
-exclusive jobs.
-
-###
-
-### Slurm Energy Profiling
-
-The `srun` tool offers several options for profiling job steps by adding
-the `--profile` parameter. Possible profiling options are `All`,
-`Energy`, `Task`, `Lustre`, and `Network`. In all cases, the profiling
-information is stored in an hdf5 file that can be inspected using
-available hdf5 tools, e.g., `h5dump`. The files are stored under
-`/scratch/profiling/` for each job, job step, and node. A description of
-the data fields in the file can be found
-[here](http://slurm.schedmd.com/hdf5_profile_user_guide.html#HDF5). In
-general, the data files contain samples of the current **power**
-consumption on a per-second basis:
-
- $> srun -p sandy --acctg-freq=2,energy=1 --profile=energy sleep 10
- srun: job 3967674 queued and waiting for resources
- srun: job 3967674 has been allocated resources
- $> h5dump /scratch/profiling/jschuch/3967674_0_taurusi1073.h5
- [...]
- DATASET "Energy_0000000002 Data" {
- DATATYPE H5T_COMPOUND {
- H5T_STRING {
- STRSIZE 24;
- STRPAD H5T_STR_NULLTERM;
- CSET H5T_CSET_ASCII;
- CTYPE H5T_C_S1;
- } "Date_Time";
- H5T_STD_U64LE "Time";
- H5T_STD_U64LE "Power";
- H5T_STD_U64LE "CPU_Frequency";
- }
- DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
- DATA {
- (0): {
- "",
- 1389097545, # timestamp
- 174, # power value
- 1
- }
- }
- }
-
-##
-
-## Using the HDEEM C API
-
-Note: Please specify -p haswell --exclusive along with your job request
-if you wish to use hdeem.
-
-Please download the offical documentation at \<font face="Calibri"
-size="2"> [\<font
-color="#0563C1">\<u>http://www.bull.com/download-hdeem-library-reference-guide\</u>\</font>](http://www.bull.com/download-hdeem-library-reference-guide)\</font>
-
-The HDEEM headers and sample code are made available via the hdeem
-module. To find the location of the hdeem installation use
-
- % module show hdeem
- -------------------------------------------------------------------
- /sw/modules/taurus/libraries/hdeem/2.1.9ms:
-
- conflict hdeem
- module-whatis Load hdeem version 2.1.9ms
- prepend-path PATH /sw/taurus/libraries/hdeem/2.1.9ms/include
- setenv HDEEM_ROOT /sw/taurus/libraries/hdeem/2.1.9ms
- -------------------------------------------------------------------
-
-You can find an example of how to use the API under
-\<span>$HDEEM_ROOT/sample.\</span>
-
-## Access Using the Dataheap Infrastructure
-
-In addition to the energy accounting data that is stored by Slurm, this
-information is also written into our local data storage and analysis
-infrastructure called
-[Dataheap](http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/projekte/dataheap/).
-From there, the data can be used in various ways, such as including it
-into application performance trace data or querying through a Python
-interface.
-
-The Dataheap infrastructure is designed to store various types of
-time-based samples from different data sources. In the case of the
-energy measurements on Taurus, the data is stored as a timeline of power
-values which allows the reconstruction of the power and energy
-consumption over time. The timestamps are stored as UNIX timestamps with
-a millisecond granularity. The data is stored for each node in the form
-of `nodename/watts`, e.g., `taurusi1073/watts`. Further metrics might
-already be available or might be added in the future for which
-information is available upon request.
-
-**Note**: The dataheap infrastructure can only be accessed from inside
-the university campus network.
-
-### Using the Python Interface
-
-The module `dataheap/1.0` provides a Python module that can be used to
-query the data in the Dataheap for personalized data analysis. The
-following is an example of how to use the interface:
-
- import time
- import os
- from dhRequest import dhClient
-
- # Connect to the dataheap manager
- dhc = dhClient()
- dhc.connect(os.environ['DATAHEAP_MANAGER_ADDR'], int(os.environ['DATAHEAP_MANAGER_PORT']))
-
- # take timestamps
- tbegin = dhc.getTimeStamp()
- # workload
- os.system("srun -n 6 a.out")
- tend = dhc.getTimeStamp()
-
- # wait for the data to get to the
- # dataheap
- time.sleep(5)
-
- # replace this with name of the node the job ran on
- # Note: use multiple requests if the job used multiple nodes
- countername = "taurusi1159/watts"
-
- # query the dataheap
- integral = dhc.storageRequest("INTEGRAL(%d,%d,\"%s\", 0)"%(tbegin, tend, countername))
- # Remember: timestamps are stored in millisecond UNIX timestamps
- energy = integral/1000
-
- print energy
-
- timeline = dhc.storageRequest("TIMELINE(%d,%d,\"%s\", 0)"%(tbegin, tend, countername))
-
- # output a list of all timestamp/power-value pairs
- print timeline
-
-## More information and Citing
-
-More information can be found in the paper \<a
-href="<http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=7016382>"
-title="HDEEM Paper E2SC 2014">HDEEM: high definition energy efficiency
-monitoring\</a> by Hackenberg et al. Please cite this paper if you are
-using HDEEM for your scientific work.
diff --git a/twiki2md/root/SystemTaurus/RunningNxGpuAppsInOneJob.md b/twiki2md/root/SystemTaurus/RunningNxGpuAppsInOneJob.md
deleted file mode 100644
index 2152522aeaa03d97841a8a9ca1d5ed844a1a0449..0000000000000000000000000000000000000000
--- a/twiki2md/root/SystemTaurus/RunningNxGpuAppsInOneJob.md
+++ /dev/null
@@ -1,85 +0,0 @@
-# Running Multiple GPU Applications Simultaneously in a Batch Job
-
-Keywords: slurm, job, gpu, multiple, instances, application, program,
-background, parallel, serial, concurrently, simultaneously
-
-## Objective
-
-Our starting point is a (serial) program that needs a single GPU and
-four CPU cores to perform its task (e.g. TensorFlow). The following
-batch script shows how to run such a job on the Taurus partition called
-"ml".
-
- #!/bin/bash
- #SBATCH --ntasks=1
- #SBATCH --cpus-per-task=4
- #SBATCH --gres=gpu:1
- #SBATCH --gpus-per-task=1
- #SBATCH --time=01:00:00
- #SBATCH --mem-per-cpu=1443
- #SBATCH --partition=ml
-
- srun some-gpu-application
-
-When srun is used within a submission script, it inherits parameters
-from sbatch, including --ntasks=1, --cpus-per-task=4, etc. So we
-actually implicitly run the following
-
- srun --ntasks=1 --cpus-per-task=4 ... --partition=ml some-gpu-application
-
-Now, our goal is to run four instances of this program concurrently in a
-**single** batch script. Of course we could also start the above script
-multiple times with sbatch, but this is not what we want to do here.
-
-## Solution
-
-In order to run multiple programs concurrently in a single batch
-script/allocation we have to do three things:
-
-1\. Allocate enough resources to accommodate multiple instances of our
-program. This can be achieved with an appropriate batch script header
-(see below).
-
-2\. Start job steps with srun as background processes. This is achieved
-by adding an ampersand at the end of the srun command
-
-3\. Make sure that each background process gets its private resources.
-We need to set the resource fraction needed for a single run in the
-corresponding srun command. The total aggregated resources of all job
-steps must fit in the allocation specified in the batch script header.
-Additionally, the option --exclusive is needed to make sure that each
-job step is provided with its private set of CPU and GPU resources.
-
-The following example shows how four independent instances of the same
-program can be run concurrently from a single batch script. Each
-instance (task) is equipped with 4 CPUs (cores) and one GPU.
-
- #!/bin/bash
- #SBATCH --ntasks=4
- #SBATCH --cpus-per-task=4
- #SBATCH --gres=gpu:4
- #SBATCH --gpus-per-task=1
- #SBATCH --time=01:00:00
- #SBATCH --mem-per-cpu=1443
- #SBATCH --partition=ml
-
- srun --exclusive --gres=gpu:1 --ntasks=1 --cpus-per-task=4 --gpus-per-task=1 --mem-per-cpu=1443 some-gpu-application &
- srun --exclusive --gres=gpu:1 --ntasks=1 --cpus-per-task=4 --gpus-per-task=1 --mem-per-cpu=1443 some-gpu-application &
- srun --exclusive --gres=gpu:1 --ntasks=1 --cpus-per-task=4 --gpus-per-task=1 --mem-per-cpu=1443 some-gpu-application &
- srun --exclusive --gres=gpu:1 --ntasks=1 --cpus-per-task=4 --gpus-per-task=1 --mem-per-cpu=1443 some-gpu-application &
-
- echo "Waiting for all job steps to complete..."
- wait
- echo "All jobs completed!"
-
-In practice it is possible to leave out resource options in srun that do
-not differ from the ones inherited from the surrounding sbatch context.
-The following line would be sufficient to do the job in this example:
-
- srun --exclusive --gres=gpu:1 --ntasks=1 some-gpu-application &
-
-Yet, it adds some extra safety to leave them in, enabling the SLURM
-scheduler to complain if not enough resources in total were specified in
-the header of the batch script.
-
--- Main.HolgerBrunst - 2021-04-16
diff --git a/twiki2md/root/WebHome/Accessibility.md b/twiki2md/root/WebHome/Accessibility.md
deleted file mode 100644
index 022418cf2d58baf9e223c90117e744752bcc762e..0000000000000000000000000000000000000000
--- a/twiki2md/root/WebHome/Accessibility.md
+++ /dev/null
@@ -1,54 +0,0 @@
-# Erklrung zur Barrierefreiheit
-
-Diese Erklrung zur Barrierefreiheit gilt fr die unter
-<https://doc.zih.tu-dresden.de> verffentlichte Website der Technischen
-Universitt Dresden.
-
-Als ffentliche Stelle im Sinne des Barrierefreie-Websites-Gesetz
-(BfWebG) ist die Technische Universitt Dresden bemht, ihre Websites und
-mobilen Anwendungen im Einklang mit den Bestimmungen des
-Barrierefreie-Websites-Gesetz (BfWebG) in Verbindung mit der
-Barrierefreie-Informationstechnik-Verordnung (BITV 2.0) barrierefrei
-zugnglich zu machen.
-
-## Erstellung dieser Erklrung zur Barrierefreiheit
-
-Diese Erklrung wurde am 17.09.2020 erstellt und zuletzt am 17.09.2020
-aktualisiert. Grundlage der Erstellung dieser Erklrung zur
-Barrierefreiheit ist eine am 17.09.2020 von der TU Dresden durchgefhrte
-Selbstbewertung.
-
-## Stand der Barrierefreiheit
-
-Es wurde bisher noch kein BITV-Test fr die Website durchgefhrt. Dieser
-ist bis 30.11.2020 geplant.
-
-## Kontakt
-
-Sollten Ihnen Mngel in Bezug auf die barrierefreie Gestaltung auffallen,
-knnen Sie uns diese ber das Formular [Barriere
-melden](https://tu-dresden.de/barrierefreiheit/barriere-melden)
-mitteilen und im zugnglichen Format anfordern. Alternativ knnen Sie sich
-direkt an die Meldestelle fr Barrieren wenden (Koordinatorin: Mandy
-Weickert, E-Mail: <barrieren@tu-dresden.de>, Telefon: [+49 351
-463-42022](tel:+49-351-463-42022), Fax: [+49 351
-463-42021](tel:+49-351-463-42021), Besucheradresse: Nthnitzer Strae 46,
-APB 1102, 01187 Dresden).
-
-## Durchsetzungsverfahren
-
-Wenn wir Ihre Rckmeldungen aus Ihrer Sicht nicht befriedigend
-bearbeiten, knnen Sie sich an die Schsische Durchsetzungsstelle wenden:
-
-**Beauftragter der Schsischen Staatsregierung fr die Belange von
-Menschen mit Behinderungen**\<br /> Albertstrae 10\<br /> 01097 Dresden
-
-Postanschrift: Archivstrae 1, 01097 Dresden\<br /> E-Mail:
-<info.behindertenbeauftragter@sk.sachsen.de>\<br /> Telefon: [+49 351
-564-12161](tel:+49-351-564-12161)\<br /> Fax: [+49 351
-564-12169](tel:+49-351-564-12169)\<br /> Webseite:
-<https://www.inklusion.sachsen.de>
-
-\<div id="footer"> \</div>
-
--- Main.MatthiasKraeusslein - 2020-09-18
diff --git a/twiki2md/root/WebHome/FurtherDocumentation.md b/twiki2md/root/WebHome/FurtherDocumentation.md
deleted file mode 100644
index 2e6586a43633c20327fae0132b90d3fe73c1bb7c..0000000000000000000000000000000000000000
--- a/twiki2md/root/WebHome/FurtherDocumentation.md
+++ /dev/null
@@ -1,81 +0,0 @@
-# Further Documentation
-
-
-
-## Libraries and Compiler
-
-- <http://www.intel.com/software/products/mkl/index.htm>
-- <http://www.intel.com/software/products/ipp/index.htm>
-- <http://www.ball-project.org/>
-- <http://www.intel.com/software/products/compilers/> - Intel Compiler
- Suite
-- <http://www.pgroup.com/doc> - PGI Compiler
-- <http://pathscale.com/ekopath.html> - PathScale Compilers
-
-## Tools
-
-- <http://www.allinea.com/downloads/userguide.pdf> - Allinea DDT
- Manual
-- <http://www.totalviewtech.com/support/documentation.html> -
- Totalview Documentation
-- <http://www.gnu.org/software/gdb/documentation/> - GNU Debugger
-- <http://vampir-ng.de> - official homepage of Vampir, an outstanding
- tool for performance analysis developed at ZIH.
-- <http://www.fz-juelich.de/zam/kojak/> - homepage of KOJAK at the FZ
- Jlich. Parts of this project are used by Vampirtrace.
-- <http://www.intel.com/software/products/threading/index.htm>
-
-## OpenMP
-
-You will find a lot of information at the following web pages:
-
-- <http://www.openmp.org>
-- <http://www.compunity.org>
-
-## MPI
-
-The following sites may be interesting:
-
-- <http://www.mcs.anl.gov/mpi/> - the MPI homepage.
-- <http://www.mpi-forum.org/> - Message Passing Interface (MPI) Forum
- Home Page
-- <http://www.open-mpi.org/> - the dawn of a new standard for a more
- fail-tolerant MPI.
-- The manual for SGI-MPI (installed on Mars ) can be found at:
-
-<http://techpubs.sgi.com/library/manuals/3000/007-3773-003/pdf/007-3773-003.pdf>
-
-## SGI developer forum
-
-The web sites behind
-<http://www.sgi.com/developers/resources/tech_pubs.html> are full of
-most detailed information on SGI systems. Have a look onto the section
-'Linux Publications'. You will be redirected to the public part of SGI's
-technical publication repository.
-
-- Linux Application Tuning Guide
-- Linux Programmer's Guide, The
-- Linux Device Driver Programmer's Guide
-- Linux Kernel Internals.... and more.
-
-## Intel Itanium
-
-There is a lot of additional material regarding the Itanium CPU:
-
-- <http://www.intel.com/design/itanium/manuals/iiasdmanual.htm>
-- <http://www.intel.com/design/archives/processors/itanium/index.htm>
-- <http://www.intel.com/design/itanium2/documentation.htm>
-
-You will find the following manuals:
-
-- Intel Itanium Processor Floating-Point Software Assistance handler
- (FPSWA)
-- Intel Itanium Architecture Software Developer's Manuals Volume 1:
- Application Architecture
-- Intel Itanium Architecture Software Developer's Manuals Volume 2:
- System Architecture
-- Intel Itanium Architecture Software Developer's Manuals Volume 3:
- Instruction Set
-- Intel Itanium 2 Processor Reference Manual for Software Development
- and Optimization
-- Itanium Architecture Assembly Language Reference Guide
diff --git a/twiki2md/root/WebHome/TypicalProjectSchedule.md b/twiki2md/root/WebHome/TypicalProjectSchedule.md
deleted file mode 100644
index c7b404ea81e20010d505392e1a67caec71f8b678..0000000000000000000000000000000000000000
--- a/twiki2md/root/WebHome/TypicalProjectSchedule.md
+++ /dev/null
@@ -1,546 +0,0 @@
-# Typical project schedule
-
-
-
-## \<span style="font-size: 1em;">0. Application for HPC login\</span>
-
-In order to use the HPC systems installed at ZIH, a project application
-form has to be filled in. The HPC project manager should hold a
-professorship (university) or head a research group. You may also apply
-for the "Schnupperaccount" (trial account) for one year. Check the
-[Access](Access) page for details.
-
-## \<span style="font-size: 1em;">1. Request for resources\</span>
-
-Important note: Taurus is based on the Linux system. Thus for the
-effective work, you should have to know how to work with
-[Linux](https://en.wikipedia.org/wiki/Linux) based systems and [Linux
-Shell](https://ubuntu.com/tutorials/command-line-for-beginners#1-overview).
-Beginners can find a lot of different tutorials on the internet, [for
-example](https://swcarpentry.github.io/shell-novice/).
-
-### \<span style="font-size: 1em;">1.1 How do I determine the required CPU / GPU hours?\</span>
-
-Taurus is focused on data-intensive computing. The cluster is oriented
-on the work with the high parallel code. Please keep it in mind for the
-transfer sequential code from a local machine. So far you will have
-execution time for the sequential program it is reasonable to use
-[Amdahl's law](https://en.wikipedia.org/wiki/Amdahl%27s_law) to roughly
-predict execution time in parallel. Think in advance about the
-parallelization strategy for your project.
-
-### \<span style="font-size: 1em;">1.2 What software do I need? What is already available (in the correct version)?\</span>
-
-The good practice for the HPC clusters is use software and packages
-where parallelization is possible. The open-source software is more
-preferable than proprietary. However, the majority of popular
-programming languages, scientific applications, software, packages
-available or could be installed on Taurus in different ways. First of
-all, check the [Software module list](SoftwareModulesList). There are
-two different software environments: **scs5** (the regular one) and
-**ml** (environment for the Machine Learning partition). Keep in mind
-that Taurus has a Linux based operating system.
-
-## 2. Access to the cluster
-
-### SSH access
-
-%RED%Important note:%ENDCOLOR%\<span style="font-size: 1em;"> ssh to
-Taurus is only possible from \</span> **inside** \<span
-style="font-size: 1em;"> TU Dresden Campus. Users from outside should
-use \</span> **VPN** \<span style="font-size: 1em;"> (see \</span>\<a
-href="<https://tu-dresden.de/zih/dienste/service-katalog/arbeitsumgebung/zugang_datennetz/vpn>"
-target="\_top">here\</a>\<span style="font-size: 1em;">).\</span>
-
-The recommended way to connect to the HPC login servers directly via
-ssh:
-
- ssh <zih-login>@taurus.hrsk.tu-dresden.de
-
-Please put this command in the terminal and replace \<zih-login> with
-your login that you received during the access procedure. Accept the
-host verifying and enter your password. You will be loaded by login
-nodes in your Taurus home directory.
-
-This method requires two conditions: Linux OS, workstation within the
-campus network. For other options and details check the \<a href="Login"
-target="\_blank">Login page\</a>.
-
-Useful links: [Access](Access), [Project Request
-Form](ProjectRequestForm), [Terms Of Use](TermsOfUse)
-
-## 3. Available software, use of the software
-
-According to 1.2, first of all, check the [Software module
-list](SoftwareModulesList). Keep in mind that there are two different
-environments: **scs5** (for the x86 architecture) and **ml**
-(environment for the Machine Learning partition based on the Power9
-architecture).
-
-\<span style="font-size: 1em;">Work with the software on Taurus could be
-started only after allocating the resources by \</span> [batch
-systems](BatchSystems)\<span style="font-size: 1em;">. By default, you
-are in the login nodes. They are not specified for the work, only for
-the login. Allocating resources will be done by batch system \</span>
-[SLURM](Slurm).
-
-There are a lot of different possibilities to work with software on
-Taurus:
-
-**a.** **Modules**
-
-\<span style="font-size: 1em;">The easiest way to start working with
-software is using the \</span>\<a
-href="RuntimeEnvironment#Module_Environments" target="\_blank">Modules
-system\</a>\<span style="font-size: 1em;">. Modules are a way to use
-frameworks, compilers, loader, libraries, and utilities. The module is a
-user interface that provides utilities for the dynamic modification of a
-user's module environment without manual modifications. You could use
-them for **srun**, bath jobs (**sbatch**) and the Jupyterhub.\</span>
-
-**b. JupyterNotebook**
-
-The Jupyter Notebook is an open-source web application that allows
-creating documents containing live code, equations, visualizations, and
-narrative text. \<span style="font-size: 1em;">There is \</span>\<a
-href="JupyterHub" target="\_self">jupyterhub\</a>\<span
-style="font-size: 1em;"> on Taurus, where you can simply run your
-Jupyter notebook on HPC nodes using modules, preloaded or custom virtual
-environments. Moreover, you can run a [manually created remote jupyter
-server](DeepLearning#Jupyter_notebook) for more specific cases.\</span>
-
-**c.** **Containers**
-
-\<span style="font-size: 1em;">Some tasks require using containers. It
-can be done on Taurus by [Singularity](https://sylabs.io/). Details
-could be found in the [following
-chapter](TypicalProjectSchedule#Use_of_containers).\</span>
-
-Useful links: [Libraries](Libraries), [Deep Learning](DeepLearning),
-[Jupyter Hub](JupyterHub), [Big Data
-Frameworks](BigDataFrameworks:ApacheSparkApacheFlinkApacheHadoop),
-[R](DataAnalyticsWithR), [Applications for various fields of
-science](Applications)
-
-## 4. Create a project structure. Data management
-
-Correct organisation of the project structure is a straightforward way
-to the efficient work of the whole team. There have to be rules and
-regulations for working with the project that every member should
-follow. \<span style="font-size: 1em;">The uniformity of the project
-could be achieved by using for each member of a team: the same **data
-storage** or set of them, the same **set of software** (packages,
-libraries etc), **access rights** to project data should be taken into
-account and set up correctly. \</span>
-
-### 4.1 Data storage and management
-
-#### 4.1.1 Taxonomy of file systems
-
-\<span style="font-size: 1em;">As soon as you have access to Taurus you
-have to manage your data. The main [concept](HPCStorageConcept2019) of
-working with data on Taurus is using [Workspaces](WorkSpaces). Use it
-properly:\</span>
-
-- use a `/home` directory for the limited amount of personal data,
- simple examples and the results of calculations. The home directory
- is not a working directory! However, \<span
- class="WYSIWYG_TT">/home\</span> file system is backed up using
- snapshots;
-- use **workspace** as a place for working data (i.e. datasets);
- Recommendations of choosing the correct storage system for workspace
- presented below.
-
-**Recommendations to choose of storage system:**\<span
-style`"font-size: 1em;">For data that seldomly changes but consumes a lot of space, the </span> ==warm_archive=`
-\<span style="font-size: 1em;"> can be used. (Note that this is
-\</span>mounted** read-only**\<span style="font-size: 1em;">on the
-compute nodes). For a series of calculations that works on the same data
-please use a \</span> **scratch**\<span style="font-size: 1em;">based
-workspace. \</span> **SSD** \<span style="font-size: 1em;">, in its
-turn, is the fastest available file system made only for large parallel
-applications running with millions of small I/O (input, output
-operations).\</span>\<span style="font-size: 1em;"> If the batch job
-needs a directory for temporary data then
-\</span>**\<span>SSD\</span>**\<span style="font-size: 1em;"> is a good
-choice as well. The data can be deleted afterwards.\</span>
-
-Note: Keep in mind that every working space has a storage duration (
-i.e. ssd - 30 days). Thus be careful with the expire date otherwise it
-could vanish. The core data of your project should be [backed
-up](FileSystems#Backup_and_snapshots_of_the_file_system) and
-[archived](PreservationResearchData)(for the most
-[important](https://www.dcc.ac.uk/guidance/how-guides/five-steps-decide-what-data-keep)
-data).
-
-#### \<span style="font-size: 1em;">4.1.2 Backup \</span>
-
-\<span
-style`"font-size: 1em;">The backup is a crucial part of any project. Organize it at the beginning of the project. If you will lose/delete your data in the "no back up" file systems it can not be restored! The b</span><span style="font-size: 13px;">ackup on Taurus is </span><b style="font-size: 1em;">only </b><span style="font-size: 13px;">available in the </span> =/home`
-\<span style`"font-size: 13px;"> and the </span> =/projects` \<span
-style="font-size: 13px;"> file systems! Backed up files could be
-restored by the user. Details could be found
-[here](FileSystems#Backup_and_snapshots_of_the_file_system).\</span>
-
-#### 4.1.3 Folder structure and organizing data
-
-\<span style="font-size: 1em;">Organizing of living data using the file
-system helps for consistency and structuredness of the project.
-\</span>\<span style="font-size: 1em;">We recommend following the rules
-for your work regarding:\</span>
-
-- Organizing the data: Never change the original data; Automatize the
- organizing the data; Clearly separate intermediate and final output
- in the filenames; Carry identifier and original name along in your
- analysis pipeline; Make outputs clearly identifiable; Document your
- analysis steps.
-- Naming Data: Keep s\<span style="font-size: 1em;">hort, but
- meaningful names; \</span>\<span style="font-size: 1em;">Keep
- standard file endings; File names dont replace documentation and
- metadata; Use standards of your discipline; \</span>\<span
- style="font-size: 1em;">Make rules for your project, document and
- keep them (See the \</span> [README
- recommendations](TypicalProjectSchedule#README_recommendation)
- below)
-
-\<span style="font-size: 1em;">This is the example of an organisation
-(hierarchical) for the folder structure. Use it as a visual illustration
-of the above:\</span>
-
-\<img align="justify" alt="Organizing_Data-using_file_systems.png"
-height="161" src="%ATTACHURL%/Organizing_Data-using_file_systems.png"
-title="Organizing_Data-using_file_systems.png" width="300" />
-
-Keep in mind [input-process-output
-pattern](https://en.wikipedia.org/wiki/IPO_model#Programming) for the
-work with folder structure.
-
-#### 4.1.4 README recommendation
-
-In general, [README](https://en.wikipedia.org/wiki/README) is just
-simple general information of software/project that exists in the same
-directory/repository of the project. README is used to explain the
-details project and the **structure** of the project/folder in a short
-way. We recommend using readme as for entire project as for every
-important folder in the project.
-
-Example of the structure for the README:\<br />\<span style="font-size:
-1em;">Think first: What is calculated why? (Description); \</span>\<span
-style="font-size: 1em;">What is expected? (software and version)\<br
-/>\</span>\<span style="font-size: 1em;">Example text file\<br
-/>\</span>\<span style="font-size: 1em;"> Title:\<br />\</span>\<span
-style="font-size: 1em;"> User:\<br />\</span>\<span style="font-size:
-1em;"> Date:\<br />\</span>\<span style="font-size: 1em;">
-Description:\<br />\</span>\<span style="font-size: 1em;"> software:\<br
-/>\</span>\<span style="font-size: 1em;"> version:\</span>
-
-#### 4.1.5 Metadata
-
-Another important aspect is the
-[Metadata](http://dublincore.org/resources/metadata-basics/). It is
-sufficient to use
-[Metadata](PreservationResearchData#Why_should_I_add_Meta_45Data_to_my_data_63)
-for your project on Taurus. [Metadata
-standards](https://en.wikipedia.org/wiki/Metadata_standard) will help to
-do it easier (i.e. [Dublin core](https://dublincore.org/),
-[OME](https://www.openmicroscopy.org/))
-
-#### 4.1.6 Data hygiene
-
-Don't forget about data hygiene: Classify your current data into
-critical (need it now), necessary (need it later) or unnecessary
-(redundant, trivial or obsolete); Track and classify data throughout its
-lifecycle (from creation, storage and use to sharing, archiving and
-destruction); Erase the data you dont need throughout its lifecycle.
-
-### \<span style="font-size: 1em;">4.2 Software packages\</span>
-
-As was written before the module concept is the basic concept for using
-software on Taurus. Uniformity of the project has to be achieved by
-using the same set of software on different levels. It could be done by
-using environments. There are two types of environments should be
-distinguished: runtime environment (the project level, use scripts to
-load [modules](RuntimeEnvironment)), Python virtual environment. The
-concept of the environment will give an opportunity to use the same
-version of the software on every level of the project for every project
-member.
-
-#### Private individual and project modules files
-
-[Private individual and project module
-files](RuntimeEnvironment#Private_Project_Module_Files)\<span
-style="font-size: 1em;"> will be discussed in [chapter
-](TypicalProjectSchedule#A_7._Use_of_specific_software_40packages_44_libraries_44_etc_41)\</span>
-[7](TypicalProjectSchedule#A_7._Use_of_specific_software_40packages_44_libraries_44_etc_41)\<span
-style="font-size: 1em;">. Project modules list is a powerful instrument
-for effective teamwork.\</span>
-
-#### Python virtual environment
-
-If you are working with the Python then it is crucial to use the virtual
-environment on Taurus. The main purpose of Python virtual environments
-(don't mess with the software environment for modules) is to create an
-isolated environment for Python projects (self-contained directory tree
-that contains a Python installation for a particular version of Python,
-plus a number of additional packages).
-
-**Vitualenv (venv)** is a standard Python tool to create isolated Python
-environments. We recommend using venv to work with Tensorflow and
-Pytorch on Taurus. It has been integrated into the standard library
-under the \<a href="<https://docs.python.org/3/library/venv.html>"
-target="\_blank">venv module\</a>. **Conda** is the second way to use a
-virtual environment on the Taurus. \<a
-href="<https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>"
-target="\_blank">Conda\</a> is an open-source package management system
-and environment management system from the Anaconda.
-
-[Detailed information](Python#Virtual_environment) about using the
-virtual environment.
-
-### \<span style="font-size: 1em;">4.3 Application software availability\</span>
-
-Software created for the purpose of the project should be available for
-all members of the group. The instruction of how to use the software:
-installation of packages, compilation etc should be documented and gives
-the opportunity to comfort efficient and safe work.
-
-### 4.4 Access rights
-
-The concept of **permissions** and **ownership** is crucial in Linux.
-See the
-[HPC-introduction](%PUBURL%/Compendium/WebHome/HPC-Introduction.pdf?t=1602081321)
-slides for the understanding of the main concept. Standard Linux
-changing permission command(i.e `chmod`) valid for Taurus as well. The
-**group** access level contains members of your project group. Be
-careful with 'write' permission and never allow to change the original
-data.
-
-Useful links: [Data Management](DataManagement), [File
-Systems](FileSystems), [Get Started with HPC-DA](GetStartedWithHPCDA),
-[Project Management](ProjectManagement), [Preservation research
-data](PreservationResearchData)
-
-## 5. Data moving
-
-### 5.1 Moving data to/from the HPC machines
-
-To copy data to/from the HPC machines, the Taurus [export
-nodes](ExportNodes) should be used as a preferred way. There are three
-possibilities to exchanging data between your local machine (lm) and the
-HPC machines (hm):\<span> **SCP, RSYNC, SFTP**. \</span>\<span
-style`"font-size: 1em;">Type following commands in the terminal of the local machine. The </span> ==SCP=`
-\<span style="font-size: 1em;"> command was used for the following
-example.\</span>
-
-#### Copy data from lm to hm
-
- scp <file> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location> #Copy file from your local machine. For example: scp helloworld.txt mustermann@taurusexport.hrsk.tu-dresden.de:/scratch/ws/mastermann-Macine_learning_project/
-
- scp -r <directory> <zih-user>@taurusexport.hrsk.tu-dresden.de:<target-location> #Copy directory from your local machine.
-
-#### Copy data from hm to lm
-
- scp <zih-user>@taurusexport.hrsk.tu-dresden.de:<file> <target-location> #Copy file. For example: scp mustermann@taurusexport.hrsk.tu-dresden.de:/scratch/ws/mastermann-Macine_learning_project/helloworld.txt /home/mustermann/Downloads
-
- scp -r <zih-user>@taurusexport.hrsk.tu-dresden.de:<directory> <target-location> #Copy directory
-
-### 5.2 Moving data inside the HPC machines. Datamover
-
-The best way to transfer data inside the Taurus is the \<a
-href="DataMover" target="\_blank">datamover\</a>. It is the special data
-transfer machine provides the best data speed. To load, move, copy etc.
-files from one file system to another file system, you have to use
-commands with **dt** prefix, such as: **\<span>dtcp, dtwget, dtmv, dtrm,
-dtrsync, dttar, dtls. \</span>**\<span style="font-size: 1em;">These
-commands submit a job to the data transfer machines that execute the
-selected command. Except for the '\</span>\<span style="font-size:
-1em;">dt'\</span>\<span style="font-size: 1em;"> prefix, their syntax is
-the same as the shell command without the '\</span>\<span
-style="font-size: 1em;">dt\</span>\<span style="font-size:
-1em;">'\</span>**.**
-
-Keep in mind: The warm_archive is not writable for jobs. However, you
-can store the data in the warm archive with the datamover.
-
-Useful links: [Data Mover](DataMover), [Export Nodes](ExportNodes)
-
-## 6. Use of hardware
-
-To run the software, do some calculations or compile your code compute
-nodes have to be used. Login nodes which are using for login can not be
-used for your computations. Submit your tasks (by using
-[jobs](https://en.wikipedia.org/wiki/Job_(computing))) to compute nodes.
-The [SLURM](Slurm) (scheduler to handle your jobs) is using on Taurus
-for this purposes. [HPC
-Introduction](%PUBURL%/Compendium/WebHome/HPC-Introduction.pdf) is a
-good resource to get started with it.
-
-### 6.1 What do I need a CPU or GPU?
-
-The main difference between CPU and GPU architecture is that a CPU is
-designed to handle a wide range of tasks quickly, but are limited in the
-concurrency of tasks that can be running. While GPUs can process data
-much faster than a CPU due to massive parallelism (but the amount of
-data which single GPU's core can handle is small), GPUs are not as
-versatile as CPUs.
-
-### 6.2 Selection of suitable hardware
-
-Available [hardware](HardwareTaurus): Normal compute nodes (Haswell\[
-[64,128,256](SystemTaurus#Run_45time_and_Memory_Limits)\], Broadwell,
-[Rome](RomeNodes)), Large [SMP nodes](SDFlex), Accelerator(GPU) nodes:
-(gpu2 partition, [ml partition](Power9)).
-
-The exact partition could be specified by `-p` flag with the srun
-command or in your batch job.
-
-Majority of the basic task could be done on the conventional nodes like
-a Haswell. SLURM will automatically select a suitable partition
-depending on your memory and --gres (gpu) requirements. If you do not
-specify the partition most likely you will be addressed to the Haswell
-partition (1328 nodes in total).
-
-#### Parallel jobs:
-
-**MPI jobs**: For MPI jobs typically allocates one core per task.
-Several nodes could be allocated if it is necessary. SLURM will
-automatically find suitable hardware. Normal compute nodes are perfect
-for this task.
-
-**OpenMP jobs**: An SMP-parallel job can only run **within a node**, so
-it is necessary to include the options **-N 1** and **-n 1**. Using
---cpus-per-task N SLURM will start one task and you will have N CPUs.
-The maximum number of processors for an SMP-parallel program is 896 on
-Taurus ( [SMP](SDFlex) island).
-
-**GPUs** partitions are best suited for **repetitive** and
-**highly-parallel** computing tasks. If you have a task with potential
-[data
-parallelism](https://en.wikipedia.org/wiki/Data_parallelism#:~:text=Data%20parallelism%20is%20parallelization%20across,on%20each%20element%20in%20parallel.)
-most likely that you need the GPUs. Beyond video rendering, GPUs excel
-in tasks such as machine learning, financial simulations and risk
-modelling. Use the gpu2 and ml partition only if you need GPUs!
-Otherwise using the x86 partitions (e.g Haswell) most likely would be
-more beneficial.
-
-**Interactive jobs**: SLURM can forward your X11 credentials to the
-first (or even all) node for a job with the --x11 option. To use an
-interactive job you have to specify -X flag for the ssh login.
-
-### 6.3 Interactive vs. sbatch
-
-However, using srun directly on the shell will lead to blocking and
-launch an interactive job. Apart from short test runs, it is
-**recommended to launch your jobs into the background by using batch
-jobs**. For that, you can conveniently put the parameters directly into
-the job file which you can submit using `sbatch` \[options\] \<job
-file>.
-
-### 6.4 Processing of data for input and output
-
-Pre-processing and post-processing of the data is a crucial part for the
-majority of data-dependent projects. The quality of this work influence
-on the computations. However, pre- and post-processing in many cases can
-be done completely or partially on a local pc and then
-[transferred](TypicalProjectSchedule#A_5._Data_moving) to the Taurus.
-Please use Taurus for the computation-intensive tasks.
-
-Useful links: [Batch Systems](BatchSystems), [Hardware
-Taurus](HardwareTaurus), [HPC-DA](HPCDA), [Slurm](Slurm)
-
-## 7. Use of specific software (packages, libraries, etc)
-
-### 7.1 Modular system
-
-The modular concept is the easiest way to work with the software on
-Taurus. It allows to user to switch between different versions of
-installed programs and provides utilities for the dynamic modification
-of a user's environment. The information can be found
-[here](RuntimeEnvironment#Modules).
-
-#### Private project and user modules files
-
-[Private project module
-files](RuntimeEnvironment#Private_Project_Module_Files)\<span
-style="font-size: 1em;"> allow you to load your group-wide installed
-software into your environment and to handle different versions. It
-allows creating your own software environment for the project. You can
-create a list of modules that will be loaded for every member of the
-team. It gives opportunity on unifying work of the team and defines the
-reproducibility of results. Private modules can be loaded like other
-modules with \</span>\<span class="WYSIWYG_TT">module load\</span>\<span
-style="font-size: 1em;">.\</span>
-
-[Private user module
-files](RuntimeEnvironment#Private_User_Module_Files) allow you to load
-your own installed software into your environment. It works in the same
-manner as to project modules but for your private use.
-
-### 7.2 Use of containers
-
-[Containerization](https://www.ibm.com/cloud/learn/containerization)
-encapsulating or packaging up software code and all its dependencies to
-run uniformly and consistently on any infrastructure. On Taurus
-[Singularity](https://sylabs.io/) used as a standard container solution.
-Singularity enables users to have full control of their environment.
-This means that you dont have to ask an HPC support to install anything
-for you - you can put it in a Singularity container and run! As opposed
-to Docker (the most famous container solution), Singularity is much more
-suited to being used in an HPC environment and more efficient in many
-cases. Docker containers can easily be used in Singularity. Information
-about the use of Singularity on Taurus can be found [here](Container).
-
-\<span style="font-size: 1em;">In some cases using Singularity requires
-a Linux machine with root privileges (e.g. using the ml partition), the
-same architecture and a compatible kernel. For many reasons, users on
-Taurus cannot be granted root permissions. A solution is a Virtual
-Machine (VM) on the ml partition which allows users to gain root
-permissions in an isolated environment. There are two main options on
-how to work with VM on Taurus:\<br />\</span>\<span style="font-size:
-1em;">1. \</span> [VM tools](VMTools)\<span style="font-size: 1em;">.
-Automative algorithms for using virtual machines;\<br />\</span>\<span
-style="font-size: 1em;">2. \</span> [Manual method](Cloud)\<span
-style="font-size: 1em;">. It required more operations but gives you more
-flexibility and reliability.\<br />\</span>\<span style="font-size:
-1em;">Additional Information: Examples of the definition for the
-Singularity container (\</span>
-[here](SingularityExampleDefinitions)\<span style="font-size: 1em;">)
-and some hints (\</span> [here](SingularityRecipeHints)\<span
-style="font-size: 1em;">).\</span>
-
-Useful links: [Containers](Container), [Custom EasyBuild
-Environment](CustomEasyBuildEnvironment), [Cloud](Cloud)
-
-## 8. Structuring experiments
-
-- \<p>Input data\</p>
-- \<p>Calculation results\</p>
-- \<p>Log files\</p>
-- \<p>Submission scripts (examples / code for survival)\</p>
-
-## What if everything didn't help?
-
-### Create a ticket: how do I do that?
-
-The best way to ask about the help is to create a ticket. In order to do
-that you have to write a message to the <hpcsupport@zih.tu-dresden.de>
-with a detailed description of your problem. If it possible please add
-logs, used environment and write a minimal executable example for the
-purpose to recreate the error or issue.
-
-### \<span style="font-size: 1em;">Communication with HPC support\</span>
-
-There is the HPC support team who is responsible for the support of HPC
-users and stable work of the cluster. You could find the
-[details](https://tu-dresden.de/zih/hochleistungsrechnen/support) in the
-right part of any page of the compendium. However, please, before the
-contact with the HPC support team check the documentation carefully
-(starting points: [ main page](WebHome), [HPC-DA](HPCDA)), use a
-[search](WebSearch) and then create a ticket. The ticket is a preferred
-way to solve the issue, but in some terminable cases, you can call to
-ask for help.
-
-Useful link: [Further Documentation](FurtherDocumentation)
-
-\<span style="font-size: 1em;">-- Main.AndreiPolitov -
-2020-09-14\</span>