diff --git a/doc.zih.tu-dresden.de/docs/archive/power_ai.md b/doc.zih.tu-dresden.de/docs/archive/power_ai.md deleted file mode 100644 index dc0fa59b3fc53e180bd620dde71df5597c33298f..0000000000000000000000000000000000000000 --- a/doc.zih.tu-dresden.de/docs/archive/power_ai.md +++ /dev/null @@ -1,82 +0,0 @@ -# PowerAI Documentation Links - -There are different documentation sources for users to learn more about -the PowerAI Framework for Machine Learning. In the following the links -are valid for PowerAI version 1.5.4 - -## General Overview: - -- \<a - href="<https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.5.3/welcome/welcome.htm>" - target="\_blank" title="Landing Page">Landing Page\</a> (note that - you can select different PowerAI versions with the drop down menu - "Change Product or version") -- \<a - href="<https://developer.ibm.com/linuxonpower/deep-learning-powerai/>" - target="\_blank" title="PowerAI Developer Portal">PowerAI Developer - Portal \</a>(Some Use Cases and examples) -- \<a - href="<https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.5.4/navigation/pai_software_pkgs.html>" - target="\_blank" title="Included Software Packages">Included - Software Packages\</a> (note that you can select different PowerAI - versions with the drop down menu "Change Product or version") - -## Specific User Howtos. Getting started with...: - -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted.htm>" - target="\_blank" title="Getting Started with PowerAI">PowerAI\</a> -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_caffe.html>" - target="\_blank" title="Caffe">Caffe\</a> -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_tensorflow.html?view=kc>" - target="\_blank" title="Tensorflow">TensorFlow\</a> -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_tensorflow_prob.html?view=kc>" - target="\_blank" title="Tensorflow Probability">TensorFlow - Probability\</a>\<br />This release of PowerAI includes TensorFlow - Probability. TensorFlow Probability is a library for probabilistic - reasoning and statistical analysis in TensorFlow. -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_tensorboard.html?view=kc>" - target="\_blank" title="Tensorboard">TensorBoard\</a> -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_snapml.html>" - target="\_blank">Snap ML\</a>\<br />This release of PowerAI includes - Snap Machine Learning (Snap ML). Snap ML is a library for training - generalized linear models. It is being developed at IBM with the - vision to remove training time as a bottleneck for machine learning - applications. Snap ML supports many classical machine learning - models and scales gracefully to data sets with billions of examples - or features. It also offers distributed training, GPU acceleration, - and supports sparse data structures. -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_pytorch.html>" - target="\_blank">PyTorch\</a>\<br />This release of PowerAI includes - the community development preview of PyTorch 1.0 (rc1). PowerAI's - PyTorch includes support for IBM's Distributed Deep Learning (DDL) - and Large Model Support (LMS). -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_caffe2ONNX.html>" - target="\_blank">Caffe2 and ONNX\</a>\<br />This release of PowerAI - includes a Technology Preview of Caffe2 and ONNX. Caffe2 is a - companion to PyTorch. PyTorch is great for experimentation and rapid - development, while Caffe2 is aimed at production environments. ONNX - (Open Neural Network Exchange) provides support for moving models - between those frameworks. -- \<a - href="<https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_ddl.html?view=kc>" - target="\_blank" title="Distributed Deep Learning">Distributed Deep - Learning\</a> (DDL). \<br />Works up to 4 TaurusML worker nodes. - (Larger models with more nodes are possible with PowerAI Enterprise) - -## PowerAI Container - -We have converted the official Docker container to Singularity. Here is -a documentation about the Docker base container, including a table with -the individual software versions of the packages installed within the -container: - -- \<a href="<https://hub.docker.com/r/ibmcom/powerai/>" - target="\_blank">PowerAI Docker Container Docu\</a> diff --git a/doc.zih.tu-dresden.de/docs/software/machine_learning.md b/doc.zih.tu-dresden.de/docs/software/machine_learning.md index 85a784103d12604afc4126d2b322255a783db2ec..3e439a0ffa086b0220bb5e7ca1e23552a04742eb 100644 --- a/doc.zih.tu-dresden.de/docs/software/machine_learning.md +++ b/doc.zih.tu-dresden.de/docs/software/machine_learning.md @@ -14,19 +14,25 @@ The main feature of the nodes is the ability to work with the support that allows a total bandwidth with up to 300 gigabytes per second (GB/sec). Each node on the ml partition has 6x Tesla V-100 GPUs. You can find a detailed specification of the partition [here](../jobs_and_resources/power9.md). -**Note:** The ML partition is based on the Power9 architecture, which means that the software built -for x86_64 will not work on this partition. Also, users need to use the modules which are -specially made for the ml partition (from modenv/ml). +!!! note + The ML partition is based on the Power9 architecture, which means that the software built + for `x86_64` will not work on this partition. Also, users need to use the modules which are + specially made for the `ml` partition (from `modenv/ml`). ### Modules -On the **ML** partition load the module environment: +On the `ml` partition load the module environment: ```console marie@login$ srun -p ml --gres=gpu:1 -n 1 -c 7 --pty --mem-per-cpu=8000 bash #Job submission in ml nodes with 1 gpu on 1 node with 8000 Mb per CPU marie@ml$ module load modenv/ml #example output: The following have been reloaded with a version change: 1) modenv/scs5 => modenv/ml ``` +### Power AI + +There are tools provided by IBM, that work on `ml` partition and are related to AI tasks. +For more information see [here](power_ai.md). + ## Alpha partition Another partition for machine learning tasks is Alpha. It is mainly dedicated to [ScaDS.AI](https://scads.ai/) @@ -58,7 +64,7 @@ For more details on machine learning or data science with Python see [here](data ### R R also supports machine learning via console. It does not require a virtual environment due to a -different package managment. +different package management. For more details on machine learning or data science with R see [here](../data_analytics_with_r/#r-console). diff --git a/doc.zih.tu-dresden.de/docs/software/power_ai.md b/doc.zih.tu-dresden.de/docs/software/power_ai.md new file mode 100644 index 0000000000000000000000000000000000000000..f5d3c136ffcb4e761d20badefb59afda996775a5 --- /dev/null +++ b/doc.zih.tu-dresden.de/docs/software/power_ai.md @@ -0,0 +1,57 @@ +# PowerAI Documentation Links + +There are different documentation sources for users to learn more about +the PowerAI Framework for Machine Learning. In the following the links +are valid for PowerAI version 1.5.4. + +!!! warning + The information provided here is available from IBM and can be used on `ml` partition only! + +## General Overview + +- [PowerAI Introduction](https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.5.3/welcome/welcome.htm) + (note that you can select different PowerAI versions with the drop down menu + "Change Product or version") +- [PowerAI Developer Portal](https://developer.ibm.com/linuxonpower/deep-learning-powerai/) + (Some Use Cases and examples) +- [Included Software Packages](https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.5.4/navigation/pai_software_pkgs.html) + (note that you can select different PowerAI versions with the drop down menu "Change Product + or version") + +## Specific User Guides + +- [Getting Started with PowerAI](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted.htm) +- [Caffe](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_caffe.html) +- [TensorFlow](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_tensorflow.html?view=kc) +- [TensorFlow Probability](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_tensorflow_prob.html?view=kc) + This release of PowerAI includes TensorFlow Probability. TensorFlow Probability is a library + for probabilistic reasoning and statistical analysis in TensorFlow. +- [TensorBoard](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_tensorboard.html?view=kc) +- [Snap ML](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_snapml.html) + This release of PowerAI includes Snap Machine Learning (Snap ML). Snap ML is a library for + training generalized linear models. It is being developed at IBM with the + vision to remove training time as a bottleneck for machine learning + applications. Snap ML supports many classical machine learning + models and scales gracefully to data sets with billions of examples + or features. It also offers distributed training, GPU acceleration, + and supports sparse data structures. +- [PyTorch](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_pytorch.html) + This release of PowerAI includes + the community development preview of PyTorch 1.0 (rc1). PowerAI's + PyTorch includes support for IBM's Distributed Deep Learning (DDL) + and Large Model Support (LMS). +- [Caffe2 and ONNX](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_caffe2ONNX.html) + This release of PowerAI includes a Technology Preview of Caffe2 and ONNX. Caffe2 is a + companion to PyTorch. PyTorch is great for experimentation and rapid + development, while Caffe2 is aimed at production environments. ONNX + (Open Neural Network Exchange) provides support for moving models + between those frameworks. +- [Distributed Deep Learning](https://www.ibm.com/support/knowledgecenter/SS5SF7_1.5.4/navigation/pai_getstarted_ddl.html?view=kc) + Distributed Deep Learning (DDL). Works on up to 4 nodes on `ml` partition. + +## PowerAI Container + +We have converted the official Docker container to Singularity. Here is +a documentation about the Docker base container, including a table with +the individual software versions of the packages installed within the +container: [PowerAI Docker Container](https://hub.docker.com/r/ibmcom/powerai/). diff --git a/doc.zih.tu-dresden.de/mkdocs.yml b/doc.zih.tu-dresden.de/mkdocs.yml index eda2bde18672ae9f73a3a1b6922353ef8db2043c..782c943e79346da0e5d2902c097450b7a6f1a7d9 100644 --- a/doc.zih.tu-dresden.de/mkdocs.yml +++ b/doc.zih.tu-dresden.de/mkdocs.yml @@ -47,6 +47,7 @@ nav: - Tensorboard: software/tensorboard.md - Distributed Training: software/distributed_training.md - Hyperparameter Optimization (OmniOpt): software/hyperparameter_optimization.md + - PowerAI: software/power_ai.md - Data Analytics: - Overview: software/data_analytics.md - Data Analytics with R: software/data_analytics_with_r.md