From 8cb2c0315f4898022a93c209cabec71f0b602543 Mon Sep 17 00:00:00 2001
From: Elias Werner <eliwerner3@googlemail.com>
Date: Fri, 27 Aug 2021 12:56:47 +0200
Subject: [PATCH] added keras part to tensorflow and added datasets to ml
overview
---
.../docs/software/machine_learning.md | 32 +++++++++++++++----
.../docs/software/tensorflow.md | 14 +++++++-
2 files changed, 39 insertions(+), 7 deletions(-)
diff --git a/doc.zih.tu-dresden.de/docs/software/machine_learning.md b/doc.zih.tu-dresden.de/docs/software/machine_learning.md
index 7ca0f0350..85a784103 100644
--- a/doc.zih.tu-dresden.de/docs/software/machine_learning.md
+++ b/doc.zih.tu-dresden.de/docs/software/machine_learning.md
@@ -46,9 +46,12 @@ marie@romeo$ module load modenv/scs5
### Python and Virtual Environments
-Python users should use a [virtual environment](python_virtual_environments.md) when conducting machine learning tasks via console.
-In case of using [sbatch files](../jobs_and_resources/batch_systems.md) to send your job you usually
-don't need a virtual environment.
+Python users should use a [virtual environment](python_virtual_environments.md) when conducting
+machine learning tasks via console.
+
+??? hint
+ In case of using [sbatch files](../jobs_and_resources/batch_systems.md)
+ to send your job you usually don't need a virtual environment.
For more details on machine learning or data science with Python see [here](data_analytics_with_python.md).
@@ -94,9 +97,9 @@ In the following example, we build a Singularity container with TensorFlow from
start it:
```console
-marie@login$ srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty --mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container.
-marie@ml$ singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version
-marie@ml$ singularity run --nv my-ML-container.sif #run my-ML-container.sif container with support of the Nvidia's GPU. You could also entertain with your container by commands: singularity shell, singularity exec
+marie@login$ srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty --mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container.
+marie@ml$ singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version
+marie@ml$ singularity run --nv my-ML-container.sif #run my-ML-container.sif container supporting the Nvidia's GPU. You can also work with your container by: singularity shell, singularity exec
```
## Additional Libraries for Machine Learning
@@ -128,3 +131,20 @@ The following HPC related software is installed on all nodes:
| IBM XLF Compiler | /opt/ibm/xlf/ |
| IBM ESSL | /opt/ibmmath/essl/ |
| IBM PESSL | /opt/ibmmath/pessl/ |
+
+## Datasets for Machine Learning
+
+There are many different datasets designed for research purposes. If you would like to download some
+of them, keep in mind that many machine learning libraries have direct access to public datasets
+without downloading it, e.g. [TensorFlow Datasets](https://www.tensorflow.org/datasets). If you
+still need to download some datasets use [DataMover](../../data_transfer/data_mover).
+
+### The ImageNet dataset
+
+The ImageNet project is a large visual database designed for use in visual object recognition
+software research. In order to save space in the file system by avoiding to have multiple duplicates
+of this lying around, we have put a copy of the ImageNet database (ILSVRC2012 and ILSVR2017) under
+`/scratch/imagenet` which you can use without having to download it again. For the future,
+the ImageNet dataset will be available in warm_archive. ILSVR2017 also includes a dataset for
+recognition objects from a video. Please respect the corresponding
+[Terms of Use](https://image-net.org/download.php).
diff --git a/doc.zih.tu-dresden.de/docs/software/tensorflow.md b/doc.zih.tu-dresden.de/docs/software/tensorflow.md
index 0206f3ad1..7f03a743a 100644
--- a/doc.zih.tu-dresden.de/docs/software/tensorflow.md
+++ b/doc.zih.tu-dresden.de/docs/software/tensorflow.md
@@ -75,7 +75,7 @@ the notebook by pre-loading a specific TensorFlow module:
??? hint
You can also define your own Jupyter kernel for more specific tasks. Please read there
- documentation about JupyterHub, Jupyter kernels and virtual environments
+ documentation about JupyterHub, Jupyter kernels and virtual environments
[here](../../access/jupyterhub/#creating-and-using-your-own-environment).
## TensorFlow in Containers
@@ -120,3 +120,15 @@ tf.disable_v2_behavior() #instead of "import tensorflow as tf"
To make the transition to TF 2.0 as seamless as possible, the TensorFlow team has created the
tf_upgrade_v2 utility to help transition legacy code to the new API.
+
+## Keras
+
+[Keras](keras.io) is a high-level neural network API, written in Python and capable of running on
+top of TensorFlow. Please check the software modules list via
+
+```console
+marie@compute$ module spider Keras
+```
+
+to find out, which Keras modules are available on your partition. TensorFlow should be automatically
+loaded as a dependency. After loading the module, you can use Keras as usual.
--
GitLab