6 merge requests!333Draft: update NGC containers,!322Merge preview into main,!319Merge preview into main,!279Draft: Machine Learning restructuring,!268Update ML branch with the content from DA,!258Data Analytics restructuring
The inscription (env) at the beginning of each line represents that now you are in the virtual
The inscription (env) at the beginning of each line represents that now you are in the virtual
environment.
environment.
Note: However in case of using [sbatch files](link) to send your job you usually don't need a
Note: However in case of using [sbatch files](link) to send your job you usually don't need a
virtual environment.
virtual environment.
## Machine Learning with Jupyter
## Machine Learning with Jupyter
The [Jupyter Notebook](https://jupyter.org/) is an open-source web application that allows you to
The [Jupyter Notebook](https://jupyter.org/) is an open-source web application that allows you to
create documents containing live code, equations, visualizations, and narrative text. [JupyterHub](../access/jupyterhub.md)
create documents containing live code, equations, visualizations, and narrative text. [JupyterHub](../access/jupyterhub.md)
allows to work with machine learning frameworks (e.g. TensorFlow or Pytorch) on Taurus and to run
allows to work with machine learning frameworks (e.g. TensorFlow or Pytorch) on Taurus and to run
your Jupyter notebooks on HPC nodes.
your Jupyter notebooks on HPC nodes.
After accessing JupyterHub, you can start a new session and configure it. For machine learning
After accessing JupyterHub, you can start a new session and configure it. For machine learning
purposes, select either **Alpha** or **ML** partition and the resources, your application requires.
purposes, select either **Alpha** or **ML** partition and the resources, your application requires.
## Machine Learning with Containers
## Machine Learning with Containers
Some machine learning tasks require using containers. In the HPC domain, the [Singularity](https://singularity.hpcng.org/)
Some machine learning tasks require using containers. In the HPC domain, the [Singularity](https://singularity.hpcng.org/)
container system is a widely used tool. Docker containers can also be used by Singularity. You can
container system is a widely used tool. Docker containers can also be used by Singularity. You can
find further information on working with containers on ZIH systems [here](containers.md)
find further information on working with containers on ZIH systems [here](containers.md)
There are two sources for containers for Power9 architecture with
There are two sources for containers for Power9 architecture with
...
@@ -119,14 +117,12 @@ page of the container.
...
@@ -119,14 +117,12 @@ page of the container.
In the following example, we build a Singularity container with TensorFlow from the DockerHub and
In the following example, we build a Singularity container with TensorFlow from the DockerHub and
start it:
start it:
```console
```console
marie@login$srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty--mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container.
marie@login$srun -p ml -N 1 --gres=gpu:1 --time=02:00:00 --pty--mem-per-cpu=8000 bash #allocating resourses from ml nodes to start the job to create a container.
marie@ml$singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version
marie@ml$singularity build my-ML-container.sif docker://ibmcom/tensorflow-ppc64le #create a container from the DockerHub with the last TensorFlow version
marie@ml$singularity run --nv my-ML-container.sif #run my-ML-container.sif container with support of the Nvidia's GPU. You could also entertain with your container by commands: singularity shell, singularity exec
marie@ml$singularity run --nv my-ML-container.sif #run my-ML-container.sif container with support of the Nvidia's GPU. You could also entertain with your container by commands: singularity shell, singularity exec
```
```
## Additional Libraries for Machine Learning
## Additional Libraries for Machine Learning
The following NVIDIA libraries are available on all nodes:
The following NVIDIA libraries are available on all nodes: