From 28321be4fe314e1b39dc367d2d6f11ac628ceda7 Mon Sep 17 00:00:00 2001
From: Martin Schroschk <martin.schroschk@tu-dresden.de>
Date: Mon, 27 Sep 2021 11:58:34 +0200
Subject: [PATCH] Merge recipes and hints into one page
---
.../docs/software/containers.md | 4 +-
.../singularity_example_definitions.md | 110 -----------------
.../docs/software/singularity_recipe_hints.md | 115 +++++++++++++++++-
doc.zih.tu-dresden.de/mkdocs.yml | 3 +-
doc.zih.tu-dresden.de/wordlist.aspell | 1 +
5 files changed, 117 insertions(+), 116 deletions(-)
delete mode 100644 doc.zih.tu-dresden.de/docs/software/singularity_example_definitions.md
diff --git a/doc.zih.tu-dresden.de/docs/software/containers.md b/doc.zih.tu-dresden.de/docs/software/containers.md
index 5f93b1bb1..46157daf5 100644
--- a/doc.zih.tu-dresden.de/docs/software/containers.md
+++ b/doc.zih.tu-dresden.de/docs/software/containers.md
@@ -34,8 +34,8 @@ execution. Follow the instructions for [locally install Singularity](#local-inst
[container creation](#container-creation). Moreover, existing Docker container can easily be
converted, which is documented [here](#importing-a-docker-container).
-If you are already familar with Singularity, you might be more intressted in our [singularity recipe
-hints](singularity_recipe_hints.md).
+If you are already familar with Singularity, you might be more intressted in our [singularity
+recipes and hints](singularity_recipe_hints.md).
<!--ZIH wiki sites:-->
diff --git a/doc.zih.tu-dresden.de/docs/software/singularity_example_definitions.md b/doc.zih.tu-dresden.de/docs/software/singularity_example_definitions.md
deleted file mode 100644
index 28fe94a9d..000000000
--- a/doc.zih.tu-dresden.de/docs/software/singularity_example_definitions.md
+++ /dev/null
@@ -1,110 +0,0 @@
-# Singularity Example Definitions
-
-## Basic example
-
-A usual workflow to create Singularity Definition consists of the
-following steps:
-
-- Start from base image
-- Install dependencies
- - Package manager
- - Other sources
-- Build & Install own binaries
-- Provide entrypoints & metadata
-
-An example doing all this:
-
-```Bash
-Bootstrap: docker
-From: alpine
-
-%post
- . /.singularity.d/env/10-docker*.sh
-
- apk add g++ gcc make wget cmake
-
- wget https://github.com/fmtlib/fmt/archive/5.3.0.tar.gz
- tar -xf 5.3.0.tar.gz
- mkdir build && cd build
- cmake ../fmt-5.3.0 -DFMT_TEST=OFF
- make -j$(nproc) install
- cd ..
- rm -r fmt-5.3.0*
-
- cat hello.cpp
-#include <fmt/format.h>
-
-int main(int argc, char** argv){
- if(argc == 1) fmt::print("No arguments passed!\n");
- else fmt::print("Hello {}!\n", argv[1]);
-}
-EOF
-
- g++ hello.cpp -o hello -lfmt
- mv hello /usr/bin/hello
-
-%runscript
- hello "$@"
-
-%labels
- Author Alexander Grund
- Version 1.0.0
-
-%help
- Display a greeting using the fmt library
-
- Usage:
- ./hello
-```
-
-## CUDA + CuDNN + OpenMPI
-
-- Chosen CUDA version depends on installed driver of host
-- OpenMPI needs PMI for SLURM integration
-- OpenMPI needs CUDA for GPU copy-support
-- OpenMPI needs ibverbs libs for Infiniband
-- openmpi-mca-params.conf required to avoid warnings on fork (OK on
- taurus)
-- Environment variables SLURM_VERSION, OPENMPI_VERSION can be set to
- choose different version when building the container
-
-```
-Bootstrap: docker
-From: nvidia/cuda-ppc64le:10.1-cudnn7-devel-ubuntu18.04
-
-%labels
- Author ZIH
- Requires CUDA driver 418.39+.
-
-%post
- . /.singularity.d/env/10-docker*.sh
-
- apt-get update
- apt-get install -y cuda-compat-10.1
- apt-get install -y libibverbs-dev ibverbs-utils
- # Install basic development tools
- apt-get install -y gcc g++ make wget python
- apt-get autoremove; apt-get clean
-
- cd /tmp
-
- : ${SLURM_VERSION:=17-02-11-1}
- wget https://github.com/SchedMD/slurm/archive/slurm-${SLURM_VERSION}.tar.gz
- tar -xf slurm-${SLURM_VERSION}.tar.gz
- cd slurm-slurm-${SLURM_VERSION}
- ./configure --prefix=/usr/ --sysconfdir=/etc/slurm --localstatedir=/var --disable-debug
- make -C contribs/pmi2 -j$(nproc) install
- cd ..
- rm -rf slurm-*
-
- : ${OPENMPI_VERSION:=3.1.4}
- wget https://download.open-mpi.org/release/open-mpi/v${OPENMPI_VERSION%.*}/openmpi-${OPENMPI_VERSION}.tar.gz
- tar -xf openmpi-${OPENMPI_VERSION}.tar.gz
- cd openmpi-${OPENMPI_VERSION}/
- ./configure --prefix=/usr/ --with-pmi --with-verbs --with-cuda
- make -j$(nproc) install
- echo "mpi_warn_on_fork = 0" >> /usr/etc/openmpi-mca-params.conf
- echo "btl_openib_warn_default_gid_prefix = 0" >> /usr/etc/openmpi-mca-params.conf
- cd ..
- rm -rf openmpi-*
-```
diff --git a/doc.zih.tu-dresden.de/docs/software/singularity_recipe_hints.md b/doc.zih.tu-dresden.de/docs/software/singularity_recipe_hints.md
index 0b568ffeb..b8304b57d 100644
--- a/doc.zih.tu-dresden.de/docs/software/singularity_recipe_hints.md
+++ b/doc.zih.tu-dresden.de/docs/software/singularity_recipe_hints.md
@@ -1,6 +1,117 @@
-# Singularity Recipe Hints
+# Singularity Recipes and Hints
-## GUI (X11) Applications
+## Example Definitions
+
+### Basic Example
+
+A usual workflow to create Singularity Definition consists of the following steps:
+
+* Start from base image
+* Install dependencies
+ * Package manager
+ * Other sources
+* Build and install own binaries
+* Provide entry points and metadata
+
+An example doing all this:
+
+```bash
+Bootstrap: docker
+From: alpine
+
+%post
+ . /.singularity.d/env/10-docker*.sh
+
+ apk add g++ gcc make wget cmake
+
+ wget https://github.com/fmtlib/fmt/archive/5.3.0.tar.gz
+ tar -xf 5.3.0.tar.gz
+ mkdir build && cd build
+ cmake ../fmt-5.3.0 -DFMT_TEST=OFF
+ make -j$(nproc) install
+ cd ..
+ rm -r fmt-5.3.0*
+
+ cat hello.cpp
+#include <fmt/format.h>
+
+int main(int argc, char** argv){
+ if(argc == 1) fmt::print("No arguments passed!\n");
+ else fmt::print("Hello {}!\n", argv[1]);
+}
+EOF
+
+ g++ hello.cpp -o hello -lfmt
+ mv hello /usr/bin/hello
+
+%runscript
+ hello "$@"
+
+%labels
+ Author Alexander Grund
+ Version 1.0.0
+
+%help
+ Display a greeting using the fmt library
+
+ Usage:
+ ./hello
+```
+
+### CUDA + CuDNN + OpenMPI
+
+* Chosen CUDA version depends on installed driver of host
+* OpenMPI needs PMI for Slurm integration
+* OpenMPI needs CUDA for GPU copy-support
+* OpenMPI needs `ibverbs` library for Infiniband
+* `openmpi-mca-params.conf` required to avoid warnings on fork (OK on ZIH systems)
+* Environment variables `SLURM_VERSION` and `OPENMPI_VERSION` can be set to choose different
+ version when building the container
+
+```bash
+Bootstrap: docker
+From: nvidia/cuda-ppc64le:10.1-cudnn7-devel-ubuntu18.04
+
+%labels
+ Author ZIH
+ Requires CUDA driver 418.39+.
+
+%post
+ . /.singularity.d/env/10-docker*.sh
+
+ apt-get update
+ apt-get install -y cuda-compat-10.1
+ apt-get install -y libibverbs-dev ibverbs-utils
+ # Install basic development tools
+ apt-get install -y gcc g++ make wget python
+ apt-get autoremove; apt-get clean
+
+ cd /tmp
+
+ : ${SLURM_VERSION:=17-02-11-1}
+ wget https://github.com/SchedMD/slurm/archive/slurm-${SLURM_VERSION}.tar.gz
+ tar -xf slurm-${SLURM_VERSION}.tar.gz
+ cd slurm-slurm-${SLURM_VERSION}
+ ./configure --prefix=/usr/ --sysconfdir=/etc/slurm --localstatedir=/var --disable-debug
+ make -C contribs/pmi2 -j$(nproc) install
+ cd ..
+ rm -rf slurm-*
+
+ : ${OPENMPI_VERSION:=3.1.4}
+ wget https://download.open-mpi.org/release/open-mpi/v${OPENMPI_VERSION%.*}/openmpi-${OPENMPI_VERSION}.tar.gz
+ tar -xf openmpi-${OPENMPI_VERSION}.tar.gz
+ cd openmpi-${OPENMPI_VERSION}/
+ ./configure --prefix=/usr/ --with-pmi --with-verbs --with-cuda
+ make -j$(nproc) install
+ echo "mpi_warn_on_fork = 0" >> /usr/etc/openmpi-mca-params.conf
+ echo "btl_openib_warn_default_gid_prefix = 0" >> /usr/etc/openmpi-mca-params.conf
+ cd ..
+ rm -rf openmpi-*
+```
+
+## Hints
+
+### GUI (X11) Applications
Running GUI applications inside a singularity container is possible out of the box. Check the
following definition:
diff --git a/doc.zih.tu-dresden.de/mkdocs.yml b/doc.zih.tu-dresden.de/mkdocs.yml
index f9b63ed0d..d53d4adb6 100644
--- a/doc.zih.tu-dresden.de/mkdocs.yml
+++ b/doc.zih.tu-dresden.de/mkdocs.yml
@@ -30,8 +30,7 @@ nav:
- Python Virtual Environments: software/python_virtual_environments.md
- Containers:
- Singularity: software/containers.md
- - Singularity Recipe Hints: software/singularity_recipe_hints.md
- - Singularity Example Definitions: software/singularity_example_definitions.md
+ - Singularity Recipes and Hints: software/singularity_recipe_hints.md
- VM tools: software/vm_tools.md
- Applications:
- Licenses: software/licenses.md
diff --git a/doc.zih.tu-dresden.de/wordlist.aspell b/doc.zih.tu-dresden.de/wordlist.aspell
index 34aaae848..410e426d8 100644
--- a/doc.zih.tu-dresden.de/wordlist.aspell
+++ b/doc.zih.tu-dresden.de/wordlist.aspell
@@ -139,6 +139,7 @@ PiB
Pika
pipelining
png
+PMI
PowerAI
ppc
PSOCK
--
GitLab