Skip to content
Snippets Groups Projects
Commit 8cb2c031 authored by Elias Werner's avatar Elias Werner
Browse files

added keras part to tensorflow and added datasets to ml overview

parent b57d1d2e
No related branches found
No related tags found
5 merge requests!333Draft: update NGC containers,!322Merge preview into main,!319Merge preview into main,!279Draft: Machine Learning restructuring,!258Data Analytics restructuring
...@@ -46,9 +46,12 @@ marie@romeo$ module load modenv/scs5 ...@@ -46,9 +46,12 @@ marie@romeo$ module load modenv/scs5
### Python and Virtual Environments ### Python and Virtual Environments
Python users should use a [virtual environment](python_virtual_environments.md) when conducting machine learning tasks via console. Python users should use a [virtual environment](python_virtual_environments.md) when conducting
In case of using [sbatch files](../jobs_and_resources/batch_systems.md) to send your job you usually machine learning tasks via console.
don't need a virtual environment.
??? hint
In case of using [sbatch files](../jobs_and_resources/batch_systems.md)
to send your job you usually don't need a virtual environment.
For more details on machine learning or data science with Python see [here](data_analytics_with_python.md). For more details on machine learning or data science with Python see [here](data_analytics_with_python.md).
...@@ -94,9 +97,9 @@ In the following example, we build a Singularity container with TensorFlow from ...@@ -94,9 +97,9 @@ In the following example, we build a Singularity container with TensorFlow from
start it: start it:
```console ```console
marie@login$ srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty --mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container. marie@login$ srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty --mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container.
marie@ml$ singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version marie@ml$ singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version
marie@ml$ singularity run --nv my-ML-container.sif #run my-ML-container.sif container with support of the Nvidia's GPU. You could also entertain with your container by commands: singularity shell, singularity exec marie@ml$ singularity run --nv my-ML-container.sif #run my-ML-container.sif container supporting the Nvidia's GPU. You can also work with your container by: singularity shell, singularity exec
``` ```
## Additional Libraries for Machine Learning ## Additional Libraries for Machine Learning
...@@ -128,3 +131,20 @@ The following HPC related software is installed on all nodes: ...@@ -128,3 +131,20 @@ The following HPC related software is installed on all nodes:
| IBM XLF Compiler | /opt/ibm/xlf/ | | IBM XLF Compiler | /opt/ibm/xlf/ |
| IBM ESSL | /opt/ibmmath/essl/ | | IBM ESSL | /opt/ibmmath/essl/ |
| IBM PESSL | /opt/ibmmath/pessl/ | | IBM PESSL | /opt/ibmmath/pessl/ |
## Datasets for Machine Learning
There are many different datasets designed for research purposes. If you would like to download some
of them, keep in mind that many machine learning libraries have direct access to public datasets
without downloading it, e.g. [TensorFlow Datasets](https://www.tensorflow.org/datasets). If you
still need to download some datasets use [DataMover](../../data_transfer/data_mover).
### The ImageNet dataset
The ImageNet project is a large visual database designed for use in visual object recognition
software research. In order to save space in the file system by avoiding to have multiple duplicates
of this lying around, we have put a copy of the ImageNet database (ILSVRC2012 and ILSVR2017) under
`/scratch/imagenet` which you can use without having to download it again. For the future,
the ImageNet dataset will be available in warm_archive. ILSVR2017 also includes a dataset for
recognition objects from a video. Please respect the corresponding
[Terms of Use](https://image-net.org/download.php).
...@@ -75,7 +75,7 @@ the notebook by pre-loading a specific TensorFlow module: ...@@ -75,7 +75,7 @@ the notebook by pre-loading a specific TensorFlow module:
??? hint ??? hint
You can also define your own Jupyter kernel for more specific tasks. Please read there You can also define your own Jupyter kernel for more specific tasks. Please read there
documentation about JupyterHub, Jupyter kernels and virtual environments documentation about JupyterHub, Jupyter kernels and virtual environments
[here](../../access/jupyterhub/#creating-and-using-your-own-environment). [here](../../access/jupyterhub/#creating-and-using-your-own-environment).
## TensorFlow in Containers ## TensorFlow in Containers
...@@ -120,3 +120,15 @@ tf.disable_v2_behavior() #instead of "import tensorflow as tf" ...@@ -120,3 +120,15 @@ tf.disable_v2_behavior() #instead of "import tensorflow as tf"
To make the transition to TF 2.0 as seamless as possible, the TensorFlow team has created the To make the transition to TF 2.0 as seamless as possible, the TensorFlow team has created the
tf_upgrade_v2 utility to help transition legacy code to the new API. tf_upgrade_v2 utility to help transition legacy code to the new API.
## Keras
[Keras](keras.io) is a high-level neural network API, written in Python and capable of running on
top of TensorFlow. Please check the software modules list via
```console
marie@compute$ module spider Keras
```
to find out, which Keras modules are available on your partition. TensorFlow should be automatically
loaded as a dependency. After loading the module, you can use Keras as usual.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment