Merge branch 'DA_neustrukturierung' of...

Merge branch 'DA_neustrukturierung' of https://gitlab.hrz.tu-chemnitz.de/zih/hpcsupport/hpc-compendium into DA_neustrukturierung

.gitlab-ci.yml

@@ -45,7 +45,7 @@ Check spelling for changed md-files:
     stage: test
     script:
         - docker run --rm -w /src -e CI_MERGE_REQUEST_TARGET_BRANCH_NAME "${DOCKER_IMAGE}"
-          doc.zih.tu-dresden.de/util/check-spelling-changes.sh
+          doc.zih.tu-dresden.de/util/check-spelling.sh
     only: [ merge_requests ]
 
 Check links for changed md-files:

doc.zih.tu-dresden.de/docs/archive/install_jupyter.md

@@ -67,7 +67,7 @@ deactivate it also manually) and install Jupyter packages for this python enviro
 source activate jnb conda install jupyter
 ```
 
-If you need to adjust the configuration, you should create the template. Generate configuration 
+If you need to adjust the configuration, you should create the template. Generate configuration
 files for Jupyter notebook server:
 
 ```Bash

doc.zih.tu-dresden.de/docs/data_lifecycle/overview.md

@@ -29,7 +29,7 @@ properly:
 It is important to design your data workflow according to characteristics, like I/O footprint
 (bandwidth/IOPS) of the application, size of the data, (number of files,) and duration of the
 storage to efficiently use the provided storage and filesystems.
-The page [filesystems](file_systems.md) holds a comprehensive documentation on the different 
+The page [filesystems](file_systems.md) holds a comprehensive documentation on the different
 filesystems.
 <!--In general, the mechanisms of
 so-called--> <!--[Workspaces](workspaces.md) are compulsory for all HPC users to store data for a
@@ -172,4 +172,5 @@ changing permission command (i.e `chmod`) valid for ZIH system as well. The **gr
 contains members of your project group. Be careful with 'write' permission and never allow to change
 the original data.
 
-Useful links: [Data Management]**todo link**, [Filesystems]**todo link**, [Project Management]**todo link**, [Preservation research data[**todo link**
+Useful links: [Data Management]**todo link**, [Filesystems]**todo link**,
+[Project Management]**todo link**, [Preservation research data[**todo link**

doc.zih.tu-dresden.de/docs/software/machine_learning.md

@@ -142,7 +142,7 @@ still need to download some datasets use [DataMover](../../data_transfer/data_mo
 ### The ImageNet dataset
 
 The ImageNet project is a large visual database designed for use in visual object recognition
-software research. In order to save space in the file system by avoiding to have multiple duplicates
+software research. In order to save space in the filesystem by avoiding to have multiple duplicates
 of this lying around, we have put a copy of the ImageNet database (ILSVRC2012 and ILSVR2017) under
 `/scratch/imagenet` which you can use without having to download it again. For the future,
 the ImageNet dataset will be available in warm_archive. ILSVR2017 also includes a dataset for

doc.zih.tu-dresden.de/docs/software/tensorflow.md

@@ -122,8 +122,8 @@ tf_upgrade_v2 utility to help transition legacy code to the new API.
 
 ## Keras
 
-[Keras](https://keras.io) is a high-level neural network API, written in Python and capable of running on
-top of TensorFlow. Please check the software modules list via
+[Keras](https://keras.io) is a high-level neural network API, written in Python and capable
+of running on top of TensorFlow. Please check the software modules list via
 
 ```console
 marie@compute$ module spider Keras

doc.zih.tu-dresden.de/util/check-spelling-changes.sh

-#!/bin/bash
-
-set -euo pipefail
-
-scriptpath=${BASH_SOURCE[0]}
-basedir=`dirname "$scriptpath"`
-basedir=`dirname "$basedir"`
-wordlistfile=$(realpath $basedir/wordlist.aspell)
-
-function getNumberOfAspellOutputLines(){
-  cat - | aspell -p "$wordlistfile" --ignore 2 -l en_US list --mode=markdown | sort -u | wc -l
-}
-
-branch="preview"
-if [ -n "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" ]; then
-  branch="origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
-fi
-
-any_fails=false
-
-source_hash=`git merge-base HEAD "$branch"`
-#Remove everything except lines beginning with --- or +++
-files=`git diff $source_hash | sed -n 's/^[-+]\{3,3\} //p'`
-#echo "$files"
-#echo "-------------------------"
-#Assume that we have pairs of lines (starting with --- and +++).
-while read oldfile; do
-  read newfile
-  if [ "${newfile: -3}" == ".md" ]; then
-    if [[ $newfile == *"accessibility.md"* ||
-          $newfile == *"data_protection_declaration.md"* ||
-          $newfile == *"legal_notice.md"* ]]; then
-      echo "Skip $newfile"
-    else
-      echo "Check $newfile"
-      if [ "$oldfile" == "/dev/null" ]; then
-        #Added files should not introduce new spelling mistakes
-        previous_count=0
-      else
-        previous_count=`git show "$source_hash:${oldfile:2}" | getNumberOfAspellOutputLines`
-      fi
-      if [ "$newfile" == "/dev/null" ]; then
-        #Deleted files do not contain any spelling mistakes
-        current_count=0
-      else
-        #Remove the prefix "b/"
-        newfile=${newfile:2}
-        current_count=`cat "$newfile" | getNumberOfAspellOutputLines`
-      fi
-      if [ $current_count -gt $previous_count ]; then
-        echo "-- File $newfile"
-        echo "Change increases spelling mistake count (from $previous_count to $current_count)"
-        any_fails=true
-      fi
-    fi
-  fi
-done <<< "$files"
-
-if [ "$any_fails" == true ]; then
-  exit 1
-fi

doc.zih.tu-dresden.de/util/check-spelling.sh

 #!/bin/bash
 
+set -euo pipefail
+
 scriptpath=${BASH_SOURCE[0]}
 basedir=`dirname "$scriptpath"`
 basedir=`dirname "$basedir"`
-wordlistfile=$basedir/wordlist.aspell
-acmd="aspell -p $wordlistfile --ignore 2 -l en_US list --mode=markdown"
-
-function spell_check () {
-  file_to_check=$1
-  ret=$(cat "$file_to_check" | $acmd)
-  if [ ! -z "$ret" ]; then
-    echo "-- File $file_to_check"
-    echo "$ret" | sort -u
-  fi
-}
+wordlistfile=$(realpath $basedir/wordlist.aspell)
+branch="origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME:-preview}"
+aspellmode=
+if aspell dump modes | grep -q markdown; then
+  aspellmode="--mode=markdown"
+fi
 
 function usage() {
   cat <<-EOF
 usage: $0 [file]
-Outputs all words of the file (or, if no argument given, all files in the current directory, recursively), that the spell checker cannot recognize.
+If file is given, outputs all words of the file, that the spell checker cannot recognize.
+If file is omitted, checks whether any changed file contains more unrecognizable words than before the change.
 If you are sure a word is correct, you can put it in $wordlistfile.
 EOF
 }
 
+function getAspellOutput(){
+  aspell -p "$wordlistfile" --ignore 2 -l en_US $aspellmode list | sort -u
+}
+
+function getNumberOfAspellOutputLines(){
+  getAspellOutput | wc -l
+}
+
+function isMistakeCountIncreasedByChanges(){
+  any_fails=false
+
+  #Unfortunately, sort depends on locale and docker does not provide much.
+  #Therefore, it uses bytewise comparison. We avoid problems with the command tr.
+  if ! sed 1d "$wordlistfile" | tr [:upper:] [:lower:] | sort -C; then
+    echo "Unsorted wordlist in $wordlistfile"
+    any_fails=true
+  fi
+
+  source_hash=`git merge-base HEAD "$branch"`
+  #Remove everything except lines beginning with --- or +++
+  files=`git diff $source_hash | sed -E -n 's#^(---|\+\+\+) ((/|./)[^[:space:]]+)$#\2#p'`
+  #echo "$files"
+  #echo "-------------------------"
+  #Assume that we have pairs of lines (starting with --- and +++).
+  while read oldfile; do
+    read newfile
+    if [ "${newfile: -3}" == ".md" ]; then
+      if [[ $newfile == *"accessibility.md"* ||
+            $newfile == *"data_protection_declaration.md"* ||
+            $newfile == *"legal_notice.md"* ]]; then
+        echo "Skip $newfile"
+      else
+        echo "Check $newfile"
+        if [ "$oldfile" == "/dev/null" ]; then
+          #Added files should not introduce new spelling mistakes
+          previous_count=0
+        else
+          previous_count=`git show "$source_hash:${oldfile:2}" | getNumberOfAspellOutputLines`
+        fi
+        if [ "$newfile" == "/dev/null" ]; then
+          #Deleted files do not contain any spelling mistakes
+          current_count=0
+        else
+          #Remove the prefix "b/"
+          newfile=${newfile:2}
+          current_count=`cat "$newfile" | getNumberOfAspellOutputLines`
+        fi
+        if [ $current_count -gt $previous_count ]; then
+          echo "-- File $newfile"
+          echo "Change increases spelling mistake count (from $previous_count to $current_count)"
+          any_fails=true
+        fi
+      fi
+    fi
+  done <<< "$files"
+
+  if [ "$any_fails" == true ]; then
+    return 1
+  fi
+  return 0
+}
+
 if [ $# -eq 1 ]; then
   case $1 in
   help | -help | --help)
@@ -30,13 +90,11 @@ if [ $# -eq 1 ]; then
     exit
   ;;
   *)
-    spell_check $1
+    cat "$1" | getAspellOutput
   ;;
   esac
 elif [ $# -eq 0 ]; then
-  for i in `find -name \*.md`; do
-  spell_check $i
-  done
+  isMistakeCountIncreasedByChanges
 else
   usage
 fi

doc.zih.tu-dresden.de/wordlist.aspell

-personal_ws-1.1 en 154 
-APIs
+personal_ws-1.1 en 1805 
 Altix
+analytics
+APIs
+BeeGFS
+benchmarking
+broadwell
+bsub
+ccNUMA
+centauri
+citable
+conda
 CPU
 CPUs
 CUDA
+cuDNN
 CXFS
-DDP
-DFG
+dask
 Dask
+dataframes
 DataFrames
 DataParallel
+DDP
+DFG
 DistributedDataParallel
 DockerHub
-ESSL
 EasyBuild
+env
+ESSL
+fastfs
+filesystem
+Filesystem
+filesystems
+Filesystems
 Flink
+foreach
 Fortran
 GFLOPS
+gfortran
+gnuplot
+Gnuplot
 GPU
 GPUs
-Gnuplot
-HDFS
-HPC
+hadoop
+haswell
 Haswell
+HDFS
 Horovod
+hostname
+HPC
+hyperparameter
 Hyperparameter
+hyperparameters
+icc
+icpc
+ifort
 ImageNet
 Infiniband
+IOPS
 Itanium
+jobqueue
+jpg
 Jupyter
 JupyterHub
 JupyterLab
 Keras
-LSF
+lapply
 LoadLeveler
+lsf
+LSF
+Mathematica
 MEGWARE
 MIMD
+Miniconda
 MKL
 MNIST
-MPI
-Mathematica
-Miniconda
 Montecito
+mountpoint
+mpi
+MPI
+mpicc
+mpiCC
+mpicxx
+mpif
+mpifort
+mpirun
+multicore
+multithreaded
 NCCL
+Neptun
 NFS
 NRINGS
 NUMA
 NUMAlink
-Neptun
 NumPy
-OPARI
+OME
 OmniOpt
+OPARI
 OpenACC
 OpenCL
 OpenMP
+openmpi
 OpenMPI
 Opteron
+overfitting
 PAPI
+parallelization
+parallelize
+pdf
 PESSL
 PGI
-PSOCK
+pipelining
+png
 PowerAI
+ppc
+PSOCK
+randint
+README
+Rmpi
+rome
+romeo
 RSA
 RStudio
-Rmpi
+salloc
+Saxonid
+sbatch
+ScaDS
+Scalasca
+scancel
+Scikit
+SciPy
+scontrol
+scp
 SGI
 SHA
 SHMEM
 SLES
+Slurm
 SMP
 SMT
+squeue
+srun
 SSD
+stderr
+stdout
 SUSE
-Saxonid
-ScaDS
-Scalasca
-SciPy
-Scikit
-Slurm
 TBB
 TCP
-TFLOPS
-TODO
 TensorBoard
 TensorFlow
+TFLOPS
 Theano
+tmp
+todo
 ToDo
+TODO
+transferability
 Trition
 Vampir
-XArray
-XGBoost
-XLC
-XLF
-ZIH
-analytics
-benchmarking
-broadwell
-bsub
-ccNUMA
-centauri
-citable
-conda
-cuDNN
-dask
-dataframes
-env
-fastfs
-foreach
-gfortran
-gnuplot
-hadoop
-haswell
-hyperparameter
-hyperparameters
-icc
-icpc
-ifort
-jobqueue
-jpg
-lapply
-lsf
-mpi
-mpiCC
-mpicc
-mpicxx
-mpif
-mpifort
-mpirun
-multicore
-openmpi
-overfitting
-parallelization
-parallelize
-pdf
-pipelining
-png
-ppc
-randint
-rome
-romeo
-salloc
-sbatch
-scancel
-scontrol
-scp
-squeue
-srun
-tmp
-transferability
 vectorization
 venv
 virtualenv
 workspace
 workspaces
-stdout
-stderr
-multithreaded
-hostname
-Filesystems
-IOPS
-OME
-README
-filesystem
-filesystems
-todo
+XArray
+XGBoost
+XLC
+XLF
+ZIH