diff --git a/doc.zih.tu-dresden.de/docs/software/machine_learning.md b/doc.zih.tu-dresden.de/docs/software/machine_learning.md index 535fcf508e725995c62efde0806a11773dae0574..dc280d7c66ff5be947bcddbfc2a954c614d10c83 100644 --- a/doc.zih.tu-dresden.de/docs/software/machine_learning.md +++ b/doc.zih.tu-dresden.de/docs/software/machine_learning.md @@ -47,8 +47,11 @@ marie@romeo$ module load modenv/scs5 ### Python and Virtual Environments Python users should use a [virtual environment](python_virtual_environments.md) when conducting -machine learning tasks via console. In case of using [sbatch files](../jobs_and_resources/batch_systems.md) -to send your job you usually don't need a virtual environment. +machine learning tasks via console. + +??? hint + In case of using [sbatch files](../jobs_and_resources/batch_systems.md) + to send your job you usually don't need a virtual environment. For more details on machine learning or data science with Python see [here](data_analytics_with_python.md). @@ -93,9 +96,9 @@ In the following example, we build a Singularity container with TensorFlow from start it: ```console -marie@login$ srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty --mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container. -marie@ml$ singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version -marie@ml$ singularity run --nv my-ML-container.sif #run my-ML-container.sif container with support of the Nvidia's GPU. You could also entertain with your container by commands: singularity shell, singularity exec +marie@login$ srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty --mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container. +marie@ml$ singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version +marie@ml$ singularity run --nv my-ML-container.sif #run my-ML-container.sif container supporting the Nvidia's GPU. You can also work with your container by: singularity shell, singularity exec ``` ## Additional Libraries for Machine Learning @@ -127,3 +130,20 @@ The following HPC related software is installed on all nodes: | IBM XLF Compiler | `/opt/ibm/xlf/` | | IBM ESSL | `/opt/ibmmath/essl/` | | IBM PESSL | `/opt/ibmmath/pessl/` | + +## Datasets for Machine Learning + +There are many different datasets designed for research purposes. If you would like to download some +of them, keep in mind that many machine learning libraries have direct access to public datasets +without downloading it, e.g. [TensorFlow Datasets](https://www.tensorflow.org/datasets). If you +still need to download some datasets use [DataMover](../../data_transfer/data_mover). + +### The ImageNet dataset + +The ImageNet project is a large visual database designed for use in visual object recognition +software research. In order to save space in the file system by avoiding to have multiple duplicates +of this lying around, we have put a copy of the ImageNet database (ILSVRC2012 and ILSVR2017) under +`/scratch/imagenet` which you can use without having to download it again. For the future, +the ImageNet dataset will be available in warm_archive. ILSVR2017 also includes a dataset for +recognition objects from a video. Please respect the corresponding +[Terms of Use](https://image-net.org/download.php). diff --git a/doc.zih.tu-dresden.de/docs/software/tensorflow.md b/doc.zih.tu-dresden.de/docs/software/tensorflow.md index 493c6187fd290b525808b02b96182aa3df938dce..7f03a743ac6acba93bcc02e1c3c4b703bfe9cb38 100644 --- a/doc.zih.tu-dresden.de/docs/software/tensorflow.md +++ b/doc.zih.tu-dresden.de/docs/software/tensorflow.md @@ -120,3 +120,15 @@ tf.disable_v2_behavior() #instead of "import tensorflow as tf" To make the transition to TF 2.0 as seamless as possible, the TensorFlow team has created the tf_upgrade_v2 utility to help transition legacy code to the new API. + +## Keras + +[Keras](keras.io) is a high-level neural network API, written in Python and capable of running on +top of TensorFlow. Please check the software modules list via + +```console +marie@compute$ module spider Keras +``` + +to find out, which Keras modules are available on your partition. TensorFlow should be automatically +loaded as a dependency. After loading the module, you can use Keras as usual.