Merge branch 'merge-p-in-m' into 'main'

Merge preview into main

See merge request !366

doc.zih.tu-dresden.de/README.md

@@ -465,6 +465,8 @@ should be highlighted, etc. Code examples, longer than half screen height should
 **TODO** Guide [Issue #14](#14)
 
 * Capitalize headings, e.g. *Exclusive Reservation of Hardware*
+* Give keywords in link texts, e.g. [Code Blocks](#code-blocks-and-syntax-highlighting) is more
+  descriptive than [this subsection](#code-blocks-and-syntax-highlighting)
 
 ### Spelling and Technical Wording
 

doc.zih.tu-dresden.de/docs/access/ssh_login.md

@@ -13,7 +13,8 @@ For more information on our VPN and how to set it up, please visit the correspon
 
 ## Connecting from Linux
 
-Please use an up-to-date SSH client. The login nodes accept the following encryption algorithms:
+SSH establishes secure connections using authentication and encryption. Thus, please use an
+up-to-date SSH client. The login nodes accept the following encryption algorithms:
 
 * `aes128-ctr`
 * `aes192-ctr`
@@ -23,58 +24,111 @@ Please use an up-to-date SSH client. The login nodes accept the following encryp
 * `chacha20-poly1305@openssh.com`
 * `chacha20-poly1305@openssh.com`
 
-### SSH Session
+### Before Your First Connection
 
-If your workstation is within the campus network, you can connect to the HPC login nodes directly.
+We suggest to create an SSH key pair before you work with the ZIH systems. This ensures high
+connection security.
 
 ```console
-marie@local$ ssh <zih-login>@taurus.hrsk.tu-dresden.de
+marie@local$ mkdir -p ~/.ssh
+marie@local$ ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
+Generating public/private ed25519 key pair.
+Enter passphrase (empty for no passphrase):
+Enter same passphrase again:
+[...]
 ```
 
-If you connect for the fist time, the client will ask you to verify the host by its fingerprint:
+Type in a passphrase for the protection of your key. The passphrase should be **non-empty**.
+Copy the public key to the ZIH system (Replace placeholder `marie` with your ZIH login):
 
 ```console
-marie@local$: ssh <zih-login>@taurus.hrsk.tu-dresden.de
+marie@local$ ssh-copy-id -i ~/.ssh/id_ed25519.pub marie@taurus.hrsk.tu-dresden.de
 The authenticity of host 'taurus.hrsk.tu-dresden.de (141.30.73.104)' can't be established.
 RSA key fingerprint is SHA256:HjpVeymTpk0rqoc8Yvyc8d9KXQ/p2K0R8TJ27aFnIL8.
 Are you sure you want to continue connecting (yes/no)?
 ```
 
 Compare the shown fingerprint with the [documented fingerprints](key_fingerprints.md). Make sure
-they match. Than you can accept by typing `y` or `yes`.
+they match. Then you can accept by typing `yes`.
 
-### X11-Forwarding
+!!! info
+    If `ssh-copy-id` is not available, you need to do additional steps:
 
-If you plan to use an application with graphical user interface (GUI), you need to enable
-X11-forwarding for the connection. Add the option `-X` or `-XC` to your SSH command. The `-C` enables
-compression which usually improves usability in this case).
+    ```console
+    marie@local$ scp ~/.ssh/id_ed25519.pub marie@taurus.hrsk.tu-dresden.de:
+    The authenticity of host 'taurus.hrsk.tu-dresden.de (141.30.73.104)' can't be established.
+    RSA key fingerprint is SHA256:HjpVeymTpk0rqoc8Yvyc8d9KXQ/p2K0R8TJ27aFnIL8.
+    Are you sure you want to continue connecting (yes/no)?
+    ```
+
+    After that, you need to manually copy the key to the right place:
+
+    ```console
+    marie@local$ ssh marie@taurus.hrsk.tu-dresden.de
+    [...]
+    marie@login$ mkdir -p ~/.ssh
+    marie@login$ touch ~/.ssh/authorized_keys
+    marie@login$ cat id_ed25519.pub >> ~/.ssh/authorized_keys
+    ```
+
+#### Configuring Default Parameters for SSH
+
+After you have copied your key to the ZIH system, you should be able to connect using:
 
 ```console
-marie@local$ ssh -XC <zih-login>@taurus.hrsk.tu-dresden.de
+marie@local$ ssh marie@taurus.hrsk.tu-dresden.de
+[...]
+marie@login$ exit
 ```
 
-!!! info
+However, you can make this more comfortable if you prepare an SSH configuration on your local
+workstation. Navigate to the subdirectory `.ssh` in your home directory and open the file `config`
+(`~/.ssh/config`) in your favorite editor. If it does not exist, create it. Put the following lines
+in it (you can omit lines starting with `#`):
+
+```bash
+Host taurus
+  HostName taurus.hrsk.tu-dresden.de
+  #Put your ZIH-Login after keyword "User":
+  User marie
+  #Path to private key:
+  IdentityFile ~/.ssh/id_ed25519
+  #Don't try other keys if you have more:
+  IdentitiesOnly yes
+  #Enable X11 forwarding for graphical applications and compression. You don't need parameter -X and -C when invoking ssh then.
+  ForwardX11 yes
+  Compression yes
+```
 
-    Also consider to use a [DCV session](desktop_cloud_visualization.md) for remote desktop
-    visualization at ZIH systems.
+Afterwards, you can connect to the ZIH system using:
 
-### Password-Less SSH
+```console
+marie@local$ ssh taurus
+```
+
+### X11-Forwarding
 
-Of course, password-less SSH connecting is supported at ZIH. All public SSH keys for ZIH systems
-have to be generated following these rules:
+If you plan to use an application with graphical user interface (GUI), you need to enable
+X11-forwarding for the connection. If you use the SSH configuration described above, everything is
+already prepared and you can simply use:
 
-  * The **ED25519** algorithm has to be used, e.g., `ssh-keygen -t ed25519`
-  * A **non-empty** passphrase for the private key must be set.
+```console
+marie@local$ ssh taurus
+```
 
-The generated public key is usually saved at `~/.ssh/id_ed25519` at your local system. To allow for
-password-less SSH connection to ZIH systems, it has to be added to the file `.ssh/authorized_keys` within
-your home directory `/home/<zih-login>/` at ZIH systems.
+If you have omitted the last two lines in the default configuration above, you need to add the
+option `-X` or `-XC` to your SSH command. The `-C` enables compression which usually improves
+usability in this case:
 
 ```console
-marie@local$ ssh -i id-ed25519 <zih-login>@taurus.hrsk.tu-dresden.de
-Enter passphrase for key 'id-ed25519':
+marie@local$ ssh -XC taurus
 ```
 
+!!! info
+
+    Also consider to use a [DCV session](desktop_cloud_visualization.md) for remote desktop
+    visualization at ZIH systems.
+
 ## Connecting from Windows
 
 We recommend one of the following applications:

doc.zih.tu-dresden.de/docs/data_lifecycle/workspaces.md

@@ -4,13 +4,13 @@ Storage systems differ in terms of capacity, streaming bandwidth, IOPS rate, etc
 efficiency don't allow to have it all in one. That is why fast parallel filesystems at ZIH have
 restrictions with regards to **age of files** and [quota](quotas.md). The mechanism of workspaces
 enables users to better manage their HPC data.
-<!--Workspaces are primarily login-related.-->
-The concept of "workspaces" is common and used at a large number of HPC centers.
+
+The concept of workspaces is common and used at a large number of HPC centers.
 
 !!! note
 
-    A workspace is a directory, with an associated expiration date, created on behalf of a user in a
-    certain storage system.
+    A **workspace** is a directory, with an associated expiration date, created on behalf of a user
+    in a certain filesystem.
 
 Once the workspace has reached its expiration date, it gets moved to a hidden directory and enters a
 grace period. Once the grace period ends, the workspace is deleted permanently. The maximum lifetime
@@ -30,8 +30,8 @@ times.
 
 To list all available filesystems for using workspaces use:
 
-```bash
-zih$ ws_find -l
+```console
+marie@login$ ws_find -l
 Available filesystems:
 scratch
 warm_archive
@@ -43,8 +43,8 @@ beegfs_global0
 
 To list all workspaces you currently own, use:
 
-```bash
-zih$ ws_list
+```console
+marie@login$ ws_list
 id: test-workspace
      workspace directory  : /scratch/ws/0/marie-test-workspace
      remaining time       : 89 days 23 hours
@@ -59,8 +59,8 @@ id: test-workspace
 To create a workspace in one of the listed filesystems use `ws_allocate`. It is necessary to specify
 a unique name and the duration of the workspace.
 
-```bash
-ws_allocate: [options] workspace_name duration
+```console
+marie@login$ ws_allocate: [options] workspace_name duration
 
 Options:
   -h [ --help]               produce help message
@@ -74,13 +74,12 @@ Options:
   -u [ --username ] arg      username
   -g [ --group ]             group workspace
   -c [ --comment ] arg       comment
-
 ```
 
 !!! example
 
-    ```bash
-    zih$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90
+    ```console
+    marie@login$ ws_allocate -F scratch -r 7 -m marie.testuser@tu-dresden.de test-workspace 90
     Info: creating workspace.
     /scratch/ws/marie-test-workspace
     remaining extensions  : 10
@@ -95,22 +94,22 @@ days with an email reminder for 7 days before the expiration.
     Setting the reminder to `7` means you will get a reminder email on every day starting `7` prior
     to expiration date.
 
-### Extention of a Workspace
+### Extension of a Workspace
 
 The lifetime of a workspace is finite. Different filesystems (storage systems) have different
 maximum durations. A workspace can be extended multiple times, depending on the filesystem.
 
-| Storage system (use with parameter -F ) | Duration, days | Extensions | Remarks |
-|:------------------------------------------:|:----------:|:-------:|:---------------------------------------------------------------------------------------:|
-| `ssd`                                       | 30 | 10 | High-IOPS filesystem (`/lustre/ssd`) on SSDs.                                          |
-| `beegfs`                                     | 30 | 2 | High-IOPS filesystem (`/lustre/ssd`) onNVMes.                                          |
-| `scratch`                                    | 100 | 2 | Scratch filesystem (/scratch) with high streaming bandwidth, based on spinning disks |
-| `warm_archive`                               | 365 | 2 | Capacity filesystem based on spinning disks                                          |
+| Filesystem (use with parameter `-F`) | Duration, days | Extensions | Remarks |
+|:------------------------------------:|:----------:|:-------:|:-----------------------------------:|
+| `ssd`                                | 30         | 2       | High-IOPS filesystem (`/lustre/ssd`) on SSDs. |
+| `beegfs`                             | 30         | 2       | High-IOPS filesystem (`/lustre/ssd`) on NVMes. |
+| `scratch`                            | 100        | 10      | Scratch filesystem (`/scratch`) with high streaming bandwidth, based on spinning disks |
+| `warm_archive`                       | 365        | 2       | Capacity filesystem based on spinning disks   |
 
-To extend your workspace use the following command:
+To extent your workspace use the following command:
 
-```
-zih$ ws_extend -F scratch test-workspace 100      #extend the workspace for 100 days
+```console
+marie@login$ ws_extend -F scratch test-workspace 100
 Info: extending workspace.
 /scratch/ws/marie-test-workspace
 remaining extensions  : 1
@@ -122,39 +121,46 @@ remaining time in days: 100
     With the `ws_extend` command, a new duration for the workspace is set. The new duration is not
     added!
 
-This means when you extend a workspace that expires in 90 days with the `ws_extend -F scratch
-my-workspace 40`, it will now expire in 40 days **not** 130 days.
+This means when you extend a workspace that expires in 90 days with the command
+
+```console
+marie@login$ ws_extend -F scratch my-workspace 40
+```
+
+it will now expire in 40 days **not** 130 days.
 
 ### Deletion of a Workspace
 
 To delete a workspace use the `ws_release` command. It is mandatory to specify the name of the
 workspace and the filesystem in which it is located:
 
-`ws_release -F <filesystem> <workspace name>`
+```console
+marie@login$ ws_release -F <filesystem> <workspace name>
+```
 
 ### Restoring Expired Workspaces
 
 At expiration time your workspace will be moved to a special, hidden directory. For a month (in
 warm_archive: 2 months), you can still restore your data into an existing workspace.
 
-!!!Warning
+!!! warning
 
     When you release a workspace **by hand**, it will not receive a grace period and be
     **permanently deleted** the **next day**. The advantage of this design is that you can create
     and release workspaces inside jobs and not swamp the filesystem with data no one needs anymore
     in the hidden directories (when workspaces are in the grace period).
 
-Use:
+Use
 
-```
-ws_restore -l -F scratch
+```console
+marie@login$ ws_restore -l -F scratch
 ```
 
 to get a list of your expired workspaces, and then restore them like that into an existing, active
 workspace 'new_ws':
 
-```
-ws_restore -F scratch marie-test-workspace-1234567 new_ws
+```console
+marie@login$ ws_restore -F scratch marie-test-workspace-1234567 new_ws
 ```
 
 The expired workspace has to be specified by its full name as listed by `ws_restore -l`, including
@@ -174,8 +180,8 @@ workspaces within in the directory `DIR`. Calling this command will do the follo
 
 - The directory `DIR` will be created if necessary.
 - Links to all personal workspaces will be managed:
-  - Create links to all available workspaces if not already present.
-  - Remove links to released workspaces.
+    - Create links to all available workspaces if not already present.
+    - Remove links to released workspaces.
 
 **Remark**: An automatic update of the workspace links can be invoked by putting the command
 `ws_register DIR` in your personal `shell` configuration file (e.g., `.bashrc`).
@@ -198,8 +204,9 @@ A batch job needs a directory for temporary data. This can be deleted afterwards
     #SBATCH --ntasks=1
     #SBATCH --cpus-per-task=24
 
-    module load modenv/classic
-    module load gaussian
+    module purge
+    module load modenv/hiera
+    module load Gaussian
 
     COMPUTE_DIR=gaussian_$SLURM_JOB_ID
     export GAUSS_SCRDIR=$(ws_allocate -F ssd $COMPUTE_DIR 7)
@@ -218,8 +225,8 @@ Likewise, other jobs can use temporary workspaces.
 For a series of jobs or calculations that work on the same data, you should allocate a workspace
 once, e.g., in `scratch` for 100 days:
 
-```
-zih$ ws_allocate -F scratch my_scratchdata 100
+```console
+marie@login$ ws_allocate -F scratch my_scratchdata 100
 Info: creating workspace.
 /scratch/ws/marie-my_scratchdata
 remaining extensions  : 2
@@ -234,8 +241,8 @@ chmod g+wrx /scratch/ws/marie-my_scratchdata
 
 And verify it with:
 
-```
-zih $ ls -la /scratch/ws/marie-my_scratchdata
+```console
+marie@login$ ls -la /scratch/ws/marie-my_scratchdata
 total 8
 drwxrwx--- 2 marie    hpcsupport 4096 Jul 10 09:03 .
 drwxr-xr-x 5 operator adm        4096 Jul 10 09:01 ..
@@ -247,8 +254,8 @@ For data that seldom changes but consumes a lot of space, the warm archive can b
 this is mounted read-only on the compute nodes, so you cannot use it as a work directory for your
 jobs!
 
-```
-zih$ ws_allocate -F warm_archive my_inputdata 365
+```console
+marie@login$ ws_allocate -F warm_archive my_inputdata 365
 /warm_archive/ws/marie-my_inputdata
 remaining extensions  : 2
 remaining time in days: 365
@@ -259,10 +266,10 @@ remaining time in days: 365
     The warm archive is not built for billions of files. There
     is a quota for 100.000 files per group. Please archive data.
 
-To see your active quota use:
+To see your active quota use
 
-```
-qinfo quota /warm_archive/ws/
+```console
+marie@login$ qinfo quota /warm_archive/ws/
 ```
 
 Note that the workspaces reside under the mountpoint `/warm_archive/ws/` and not `/warm_archive`

doc.zih.tu-dresden.de/docs/software/nanoscale_simulations.md

@@ -230,7 +230,7 @@ Module ORCA/4.2.1-gompi-2019b and 11 dependencies loaded.
 
 ## Siesta
 
-[Siesta](https://www.uam.es/siesta) (Spanish Initiative for Electronic Simulations with
+[Siesta](https://siesta-project.org/siesta) (Spanish Initiative for Electronic Simulations with
 Thousands of Atoms) is both a method and its computer program implementation,
 to perform electronic structure calculations and ab initio
 molecular dynamics simulations of molecules and solids.

doc.zih.tu-dresden.de/util/check-bash-syntax.sh

+#!/bin/bash
+
+set -euo pipefail
+
+scriptpath=${BASH_SOURCE[0]}
+basedir=`dirname "$scriptpath"`
+basedir=`dirname "$basedir"`
+
+function usage () {
+  echo "$0 [options]"
+  echo "Search for bash files that have an invalid syntax."
+  echo ""
+  echo "Options:"
+  echo "  -a           Search in all bash files (default: git-changed files)" 
+  echo "  -f=FILE      Search in a specific bash file" 
+  echo "  -s           Silent mode"
+  echo "  -h           Show help message"
+}
+
+# Options
+all_files=false
+silent=false
+file=""
+while getopts ":ahsf:" option; do
+ case $option in
+   a)
+     all_files=true
+     ;;
+   f)
+     file=$2
+     shift
+     ;;
+   s)
+     silent=true
+     ;;
+   h)
+     usage
+     exit;;
+   \?) # Invalid option
+     echo "Error: Invalid option."
+     usage
+     exit;;
+ esac
+done
+
+branch="origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME:-preview}"
+
+if [ $all_files = true ]; then
+  echo "Search in all bash files."
+  files=$(git ls-tree --full-tree -r --name-only HEAD $basedir/docs/ | grep .sh)
+elif [[ ! -z $file ]]; then
+  files=$file
+else
+  echo "Search in git-changed files."
+  files=`git diff --name-only "$(git merge-base HEAD "$branch")" | grep .sh`
+fi
+
+
+cnt=0
+for f in $files; do
+  if ! bash -n $f; then
+    if [ $silent = false ]; then
+      echo "Bash file $f has invalid syntax"
+    fi
+    ((cnt=cnt+1))
+  fi
+done
+
+case $cnt in
+  1)
+    echo "Bash files with invalid syntax: 1 match found"
+  ;;
+  *)
+    echo "Bash files with invalid syntax: $cnt matches found"
+  ;;
+esac
+if [ $cnt -gt 0 ]; then
+  exit 1
+fi

doc.zih.tu-dresden.de/util/grep-forbidden-words.sh

@@ -18,12 +18,14 @@ i	file \+system	HDFS
 i	\<taurus\>	taurus\.hrsk	/taurus	/TAURUS
 i	\<hrskii\>
 i	hpc[ -]\+da\>
+i	ATTACHURL
 i	\(alpha\|ml\|haswell\|romeo\|gpu\|smp\|julia\|hpdlf\|scs5\)-\?\(interactive\)\?[^a-z]*partition
+i	\[\s\?\(documentation\|here\|this \(link\|page\|subsection\)\|slides\?\|manpage\)\s\?\]
 i	work[ -]\+space"
 
 # Whitelisted files will be ignored
 # Whitespace separated list with full path
-whitelist=(doc.zih.tu-dresden.de/docs/contrib/content_rules.md)
+whitelist=(doc.zih.tu-dresden.de/README.md doc.zih.tu-dresden.de/docs/contrib/content_rules.md doc.zih.tu-dresden.de/docs/access/ssh_login.md)
 
 function grepExceptions () {
   if [ $# -gt 0 ]; then
@@ -87,7 +89,7 @@ fi
 echo "... $files ..."
 cnt=0
 for f in $files; do
-  if [ "$f" != doc.zih.tu-dresden.de/README.md -a "${f: -3}" == ".md" -a -f "$f" ]; then
+  if [ "${f: -3}" == ".md" -a -f "$f" ]; then
     if (printf '%s\n' "${whitelist[@]}" | grep -xq $f); then
       echo "Skip whitelisted file $f"
       continue

doc.zih.tu-dresden.de/util/pre-commit

 #!/bin/bash
-exit_ok=yes
-files=$(git diff-index --cached --name-only HEAD)
-
 function testPath(){
 path_to_test=doc.zih.tu-dresden.de/docs/$1
 test -f "$path_to_test" || echo $path_to_test does not exist
@@ -15,29 +12,39 @@ fi
 
 export -f testPath
 
-for file in $files
-do
-  if [ $file == doc.zih.tu-dresden.de/mkdocs.yml ]
+exit_ok=yes
+branch="origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME:-preview}"
+if [ -f "$GIT_DIR/MERGE_HEAD" ]
+then
+  source_hash=`git merge-base HEAD "$branch"`
+else
+  source_hash=`git rev-parse HEAD`
+fi
+#Remove everything except lines beginning with --- or +++
+files=`git diff $source_hash | sed -E -n 's#^(---|\+\+\+) ((/|./)[^[:space:]]+)$#\2#p'`
+#Assume that we have pairs of lines (starting with --- and +++).
+while read oldfile; do
+  read newfile
+  
+  if [ "$newfile" == doc.zih.tu-dresden.de/mkdocs.yml ]
   then
-    echo Testing $file
+    echo Testing "$newfile"
     sed -n '/^ *- /s#.*: \([A-Za-z_/]*.md\).*#\1#p' doc.zih.tu-dresden.de/mkdocs.yml | xargs -L1 -I {} bash -c "testPath '{}'"
     if [ $? -ne 0 ]
     then
       exit_ok=no
     fi
-  elif [[ $file =~ ^doc.zih.tu-dresden.de/(.*.md)$ ]]
+  elif [[ "$newfile" =~ ^b/doc.zih.tu-dresden.de/(.*.md)$ ]]
   then
     filepattern=${BASH_REMATCH[1]}
 
-    #lint
-    echo "Checking linter..."
+    echo "Linting..."
     docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)"/doc.zih.tu-dresden.de,target=/docs,type=bind hpc-compendium markdownlint $filepattern
     if [ $? -ne 0 ]
     then
       exit_ok=no
     fi
 
-    #link-check
     echo "Checking links..."
     docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)"/doc.zih.tu-dresden.de,target=/docs,type=bind hpc-compendium markdown-link-check $filepattern
     if [ $? -ne 0 ]
@@ -45,9 +52,15 @@ do
       exit_ok=no
     fi
   fi
-done
+done <<< "$files"
+
+echo "Testing syntax of bash files..."
+docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)",target=/docs,type=bind hpc-compendium ./doc.zih.tu-dresden.de/util/check-bash-syntax.sh
+if [ $? -ne 0 ]
+then
+  exit_ok=no
+fi
 
-#spell-check
 echo "Spell-checking..."
 docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)",target=/docs,type=bind hpc-compendium ./doc.zih.tu-dresden.de/util/check-spelling.sh
 if [ $? -ne 0 ]
@@ -55,7 +68,6 @@ then
   exit_ok=no
 fi
 
-#forbidden words checking
 echo "Forbidden words checking..."
 docker run --name=hpc-compendium --rm -w /docs --mount src="$(pwd)",target=/docs,type=bind hpc-compendium ./doc.zih.tu-dresden.de/util/grep-forbidden-words.sh
 if [ $? -ne 0 ]