Skip to content
Snippets Groups Projects
Commit abed86d5 authored by Taras Lazariv's avatar Taras Lazariv
Browse files

Correct in TensorflowContainer and TensorflowJupyter

parent 68398dc4
No related branches found
No related tags found
3 merge requests!322Merge preview into main,!319Merge preview into main,!105Tensorflow
Hide whitespace changes
Inline Side-by-side
doc.zih.tu-dresden.de/docs/software/TensorFlowContainerOnHPCDA.md
+ 2
− 2
View file @ abed86d5
...@@ -44,10 +44,10 @@ is a Virtual Machine (VM) on the ml partition which allows users to gain ...@@ -44,10 +44,10 @@ is a Virtual Machine (VM) on the ml partition which allows users to gain
root permissions in an isolated environment. There are two main options root permissions in an isolated environment. There are two main options
on how to work with VM on Taurus: on how to work with VM on Taurus:
1\. [VM tools](VMTools). Automative algorithms for using virtual 1\. [VM tools](VMTools.md). Automative algorithms for using virtual
machines; machines;
2\. [Manual method](Cloud). It required more operations but gives you 2\. [Manual method](Cloud.md). It required more operations but gives you
more flexibility and reliability. more flexibility and reliability.
Short algorithm to run the virtual machine manually: Short algorithm to run the virtual machine manually:
......
doc.zih.tu-dresden.de/docs/software/TensorFlowOnJupyterNotebook.md
+ 43
− 77
View file @ abed86d5
...@@ -3,8 +3,6 @@ ...@@ -3,8 +3,6 @@
%RED%Note: This page is under construction<span %RED%Note: This page is under construction<span
class="twiki-macro ENDCOLOR"></span> class="twiki-macro ENDCOLOR"></span>
Disclaimer: This page dedicates a specific question. For more general Disclaimer: This page dedicates a specific question. For more general
questions please check the JupyterHub webpage. questions please check the JupyterHub webpage.
...@@ -21,8 +19,7 @@ with HPC or Linux. \</span> ...@@ -21,8 +19,7 @@ with HPC or Linux. \</span>
basic knowledge about Python, SLURM system and the Jupyter notebook. basic knowledge about Python, SLURM system and the Jupyter notebook.
\<span style="font-size: 1em;"> **This page aims** to introduce users on \<span style="font-size: 1em;"> **This page aims** to introduce users on
how to start working with TensorFlow on the \</span>\<a href="HPCDA" how to start working with TensorFlow on the [HPCDA](../jobs/HPCDA.md) system - part
target="\_self">HPC-DA\</a>\<span style="font-size: 1em;"> system - part
of the TU Dresden HPC system with a graphical interface.\</span> of the TU Dresden HPC system with a graphical interface.\</span>
## Get started with Jupyter notebook ## Get started with Jupyter notebook
...@@ -38,13 +35,10 @@ work Jupyter notebooks using HPC. \</span> ...@@ -38,13 +35,10 @@ work Jupyter notebooks using HPC. \</span>
- \<span style="font-size: 1em;">There is \</span>**\<a - \<span style="font-size: 1em;">There is \</span>**\<a
href="JupyterHub" target="\_self">jupyterhub\</a>** on Taurus, where href="JupyterHub" target="\_self">jupyterhub\</a>** on Taurus, where
you can simply run your Jupyter notebook on HPC nodes. JupyterHub is you can simply run your Jupyter notebook on HPC nodes. JupyterHub is
available here: \<a available [here](https://taurus.hrsk.tu-dresden.de/jupyter)
href="<https://taurus.hrsk.tu-dresden.de/jupyter>"
target="\_top"><https://taurus.hrsk.tu-dresden.de/jupyter>\</a>
- For more specific cases you can run a manually created **remote - For more specific cases you can run a manually created **remote
jupyter server.** \<span style="font-size: 1em;"> You can find the jupyter server.** \<span style="font-size: 1em;"> You can find the
manual server setup \</span>\<a href="DeepLearning" manual server setup [here](DeepLearning.md).
target="\_blank">here.\</a>
\<span style="font-size: 13px;">Keep in mind that with Jupyterhub you \<span style="font-size: 13px;">Keep in mind that with Jupyterhub you
can't work with some special instruments. However general data analytics can't work with some special instruments. However general data analytics
...@@ -69,28 +63,24 @@ environment](https://docs.python.org/3/glossary.html#term-virtual-environment) ...@@ -69,28 +63,24 @@ environment](https://docs.python.org/3/glossary.html#term-virtual-environment)
is a self-contained directory tree that contains a Python installation is a self-contained directory tree that contains a Python installation
for a particular version of Python, plus several additional packages. At for a particular version of Python, plus several additional packages. At
its core, the main purpose of Python virtual environments is to create its core, the main purpose of Python virtual environments is to create
an isolated environment for Python projects. \<span style="font-size: an isolated environment for Python projects. Python virtual environment is
1em;">Python virtual environment is the main method to work with Deep the main method to work with Deep Learning software as TensorFlow on the
Learning software as TensorFlow on the \</span>\<a href="HPCDA" [HPCDA](../jobs/HPCDA.md) system.
target="\_self">HPC-DA\</a>\<span style="font-size: 1em;">
system.\</span>
### Conda and Virtualenv ### Conda and Virtualenv
There are two methods of how to work with virtual environments on There are two methods of how to work with virtual environments on
Taurus.\<br />**Vitualenv (venv)**\<span style="font-size: 1em;"> is a Taurus. **Vitualenv (venv)** is a
standard Python tool to create isolated Python environments. We standard Python tool to create isolated Python environments. We
recommend using venv to work with Tensorflow and Pytorch on Taurus. It recommend using venv to work with Tensorflow and Pytorch on Taurus. It
has been integrated into the standard library under the \</span>\<a has been integrated into the standard library under
href="<https://docs.python.org/3/library/venv.html>" the [venv](https://docs.python.org/3/library/venv.html).
target="\_blank">venv module\</a>\<span style="font-size: 1em;">.
However, if you have reasons (previously created environments etc) you However, if you have reasons (previously created environments etc) you
could easily use conda. The conda is the second way to use a virtual could easily use conda. The conda is the second way to use a virtual
environment on the Taurus. \</span>\<a environment on the Taurus.
href="<https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>" [Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)
target="\_blank">Conda\</a>\<span style="font-size: 1em;"> is an is an open-source package management system and environment management system
open-source package management system and environment management system from the Anaconda.
from the Anaconda.\</span>
**Note:** Keep in mind that you **can not** use conda for working with **Note:** Keep in mind that you **can not** use conda for working with
the virtual environments previously created with Vitualenv tool and vice the virtual environments previously created with Vitualenv tool and vice
...@@ -123,11 +113,11 @@ Now you can check the working capacity of the current environment. ...@@ -123,11 +113,11 @@ Now you can check the working capacity of the current environment.
### Install Ipykernel ### Install Ipykernel
Ipykernel is an interactive Python shell and a Jupyter kernel to work Ipykernel is an interactive Python shell and a Jupyter kernel to work
with Python code in Jupyter notebooks. \<span style="font-size: with Python code in Jupyter notebooks. The IPython kernel is the Python
1em;">The IPython kernel is the Python execution backend for Jupyter. execution backend for Jupyter. The Jupyter Notebook
\</span>\<span style="font-size: 1em;">The Jupyter Notebook automatically ensures that the IPython kernel is available.
automatically ensures that the IPython kernel is available.\</span>
```
(env) bash-4.2$ pip install ipykernel #example output: Collecting ipykernel (env) bash-4.2$ pip install ipykernel #example output: Collecting ipykernel
... ...
#example output: Successfully installed ... ipykernel-5.1.0 ipython-7.5.0 ... #example output: Successfully installed ... ipykernel-5.1.0 ipython-7.5.0 ...
...@@ -136,6 +126,7 @@ automatically ensures that the IPython kernel is available.\</span> ...@@ -136,6 +126,7 @@ automatically ensures that the IPython kernel is available.\</span>
#example output: Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/env #example output: Installed kernelspec my-kernel in .../.local/share/jupyter/kernels/env
[install now additional packages for your notebooks] [install now additional packages for your notebooks]
```
Deactivate the virtual environment Deactivate the virtual environment
...@@ -154,17 +145,13 @@ with jupyterhub and tensorflow models. It can be useful and instructive ...@@ -154,17 +145,13 @@ with jupyterhub and tensorflow models. It can be useful and instructive
to start your acquaintance with Tensorflow and HPC-DA system from these to start your acquaintance with Tensorflow and HPC-DA system from these
simple examples. simple examples.
You can use a [remote Jupyter server](DeepLearning) or \<a You can use a [remote Jupyter server](JupyterHub.md). For simplicity, we
href="JupyterHub" target="\_blank">Jupyterhub\</a>. For simplicity, we
will recommend using Jupyterhub for our examples. will recommend using Jupyterhub for our examples.
JupyterHub is available here: \<a JupyterHub is available [here](https://taurus.hrsk.tu-dresden.de/jupyter)
href="<https://taurus.hrsk.tu-dresden.de/jupyter>"
target="\_top"><https://taurus.hrsk.tu-dresden.de/jupyter>\</a>
Please check updates and details \<a href="JupyterHub" Please check updates and details [JupyterHub](JupyterHub.md). However,
target="\_blank">JupyterHub page\</a>. However, the general pipeline can the general pipeline can be briefly explained as follows.
be briefly explained as follows.
After logging, you can start a new session and configure it. There are After logging, you can start a new session and configure it. There are
simple and advanced forms to set up your session. On the simple form, simple and advanced forms to set up your session. On the simple form,
...@@ -172,7 +159,7 @@ you have to choose the "IBM Power (ppc64le)" architecture. You can ...@@ -172,7 +159,7 @@ you have to choose the "IBM Power (ppc64le)" architecture. You can
select the required number of CPUs and GPUs. For the acquaintance with select the required number of CPUs and GPUs. For the acquaintance with
the system through the examples below the recommended amount of CPUs and the system through the examples below the recommended amount of CPUs and
1 GPU will be enough. With the advanced form, you can use the 1 GPU will be enough. With the advanced form, you can use the
configuration with 1 GPU and 7 CPUs. To access for all your workspaces configuration with 1 GPU and 7 CPUs. To access all your workspaces
use " / " in the workspace scope. use " / " in the workspace scope.
You need to download the file with a jupyter notebook that already You need to download the file with a jupyter notebook that already
...@@ -180,31 +167,26 @@ contains all you need for the start of the work. Please put the file ...@@ -180,31 +167,26 @@ contains all you need for the start of the work. Please put the file
into your previously created virtual environment in your working into your previously created virtual environment in your working
directory or use the kernel for your notebook. directory or use the kernel for your notebook.
\<span style="font-size: 1em;">Note: You could work with simple examples Note: You could work with simple examples in your home directory but according to
in your home directory but according to \</span>\<a [new storage concept](../data_management/HPCStorageConcept2019.md) please use
href="HPCStorageConcept2019" target="\_blank">New storage [workspaces](../data_management/workspaces.md) for your study and work projects**.
concept\</a>**\<span style="font-size: 1em;"> please use \</span>\<a For this reason, you have to use advanced options and put "/" in "Workspace scope" field.
href="WorkSpaces" target="\_blank">workspaces\</a>**\<span
style="font-size: 1em;">** for your study and work projects**. For this
reason, you have to use advanced options and put "/" in "Workspace
scope" field.\</span>
To download the first example (from the list below) into your previously To download the first example (from the list below) into your previously
created virtual environment you could use the following command: created virtual environment you could use the following command:
```
ws_list ws_list
cd <name_of_your_workspace> #go to workspace cd <name_of_your_workspace> #go to workspace
wget https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Mnistmodel.zip wget https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Mnistmodel.zip
unzip Example_TensorFlow_Automobileset.zip unzip Example_TensorFlow_Automobileset.zip
```
\<span style="font-size: 1em;">Also, you could use kernels for all Also, you could use kernels for all notebooks, not only for them which placed
notebooks, not only for them which placed in your virtual environment. in your virtual environment. See the [jupyterhub](JupyterHub.md) page.
See the \</span>\<a href="JupyterHub"
target="\_blank">jupyterhub\</a>\<span style="font-size: 1em;">
page.\</span>
Examples: ### Examples:
1\. Simple MNIST model. The MNIST database is a large database of 1\. Simple MNIST model. The MNIST database is a large database of
handwritten digits that is commonly used for \<a handwritten digits that is commonly used for \<a
...@@ -217,12 +199,9 @@ Keras allow us to import and download the MNIST dataset directly from ...@@ -217,12 +199,9 @@ Keras allow us to import and download the MNIST dataset directly from
their API. Recommended parameters for running this model is 1 GPU and 7 their API. Recommended parameters for running this model is 1 GPU and 7
cores (28 thread) cores (28 thread)
[https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Mnistmodel.zip](%ATTACHURL%/Mnistmodel.zip) [doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Mnistmodel.zip]**todo**(Mnistmodel.zip)
####
\<span style="color: #222222; font-size: 1.154em;">Running the ### Running the model
model\</span>
\<span style="font-size: 1em;">Documents are organized with tabs and a \<span style="font-size: 1em;">Documents are organized with tabs and a
very versatile split-screen feature. On the left side of the screen, you very versatile split-screen feature. On the left side of the screen, you
...@@ -233,12 +212,10 @@ command for running one cell Shift+Enter'. Also, you could run all cells ...@@ -233,12 +212,10 @@ command for running one cell Shift+Enter'. Also, you could run all cells
with the command 'run all cells' how presented on the picture with the command 'run all cells' how presented on the picture
below\</span> below\</span>
\<img alt="Screenshot_from_2019-09-03_15-20-16.png" height="250" **todo** \<img alt="Screenshot_from_2019-09-03_15-20-16.png" height="250"
src="%ATTACHURL%/Screenshot_from_2019-09-03_15-20-16.png" src="Screenshot_from_2019-09-03_15-20-16.png"
title="Screenshot_from_2019-09-03_15-20-16.png" width="436" /> title="Screenshot_from_2019-09-03_15-20-16.png" width="436" />
####
#### Additional advanced models #### Additional advanced models
1\. A simple regression model uses [Automobile 1\. A simple regression model uses [Automobile
...@@ -249,7 +226,7 @@ model created to present how to work with a jupyter notebook for the ...@@ -249,7 +226,7 @@ model created to present how to work with a jupyter notebook for the
TensorFlow models. Recommended parameters for running this model is 1 TensorFlow models. Recommended parameters for running this model is 1
GPU and 7 cores (28 thread) GPU and 7 cores (28 thread)
[https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Example_TensorFlow_Automobileset.zip](%ATTACHURL%/Example_TensorFlow_Automobileset.zip) [doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Example_TensorFlow_Automobileset.zip]**todo**(Example_TensorFlow_Automobileset.zip)
2\. The regression model uses the 2\. The regression model uses the
[dataset](https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data) [dataset](https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data)
...@@ -258,7 +235,7 @@ The data set contains almost 50 thousand on instances and therefore ...@@ -258,7 +235,7 @@ The data set contains almost 50 thousand on instances and therefore
needs more computational effort. Recommended parameters for running this needs more computational effort. Recommended parameters for running this
model is 1 GPU and 7 cores (28 threads) model is 1 GPU and 7 cores (28 threads)
[https://doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Example_TensorFlow_Meteo_airport.zip](%ATTACHURL%/Example_TensorFlow_Meteo_airport.zip) [doc.zih.tu-dresden.de/hpc-wiki/pub/Compendium/TensorFlowOnJupyterNotebook/Example_TensorFlow_Meteo_airport.zip]**todo**(Example_TensorFlow_Meteo_airport.zip)
**Note**: All examples created only for study purposes. The main aim is **Note**: All examples created only for study purposes. The main aim is
to introduce users of the HPC-DA system of TU-Dresden with TensorFlow to introduce users of the HPC-DA system of TU-Dresden with TensorFlow
...@@ -266,21 +243,10 @@ and Jupyter notebook. Examples do not pretend to completeness or ...@@ -266,21 +243,10 @@ and Jupyter notebook. Examples do not pretend to completeness or
science's significance. Feel free to improve the models and use them for science's significance. Feel free to improve the models and use them for
your study. your study.
-- Main.AndreiPolitov - 2019-08-27 - [Mnistmodel.zip]**todo**(Mnistmodel.zip): Mnistmodel.zip
- [Example_TensorFlow_Automobileset.zip]**todo**(Example_TensorFlow_Automobileset.zip):
- [Mnistmodel.zip](%ATTACHURL%/Mnistmodel.zip): Mnistmodel.zip
- [Example_TensorFlow_Automobileset.zip](%ATTACHURL%/Example_TensorFlow_Automobileset.zip):
Example_TensorFlow_Automobileset.zip Example_TensorFlow_Automobileset.zip
- [Example_TensorFlow_Meteo_airport.zip](%ATTACHURL%/Example_TensorFlow_Meteo_airport.zip): - [Example_TensorFlow_Meteo_airport.zip]**todo**(Example_TensorFlow_Meteo_airport.zip):
Example_TensorFlow_Meteo_airport.zip Example_TensorFlow_Meteo_airport.zip
- [Example_TensorFlow_3D_road_network.zip](%ATTACHURL%/Example_TensorFlow_3D_road_network.zip): - [Example_TensorFlow_3D_road_network.zip]**todo**(Example_TensorFlow_3D_road_network.zip):
Example_TensorFlow_3D_road_network.zip \<div style="visibility: Example_TensorFlow_3D_road_network.zip
visible; left: -318px; top: 2579px; opacity: 1;"> \</div> \<div
style="visibility: visible; left: 73px; top: 3248px; opacity: 1;">
\</div>
\<div style="visibility: visible; left: 46px; top: 3464px; opacity: 1;">
\</div> \<div id="gtx-anchor" style="position: absolute; visibility:
hidden; left: 236.9px; top: 1861.9px; width: 65.8px; height: 14.4px;">
\</div> \<div style="visibility: visible; left: -289px; top: 1886px;
opacity: 1;"> \</div>
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Nutzungsbedingungen | Datenschutzerklärung | Barrierefreiheitserklärung