Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
hpc-compendium
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Package registry
Container Registry
Model registry
Operate
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ZIH
hpcsupport
hpc-compendium
Commits
24ec0536
Commit
24ec0536
authored
3 years ago
by
Martin Schroschk
Browse files
Options
Downloads
Patches
Plain Diff
Fix line lenght; minor issues
parent
64099f1e
No related branches found
No related tags found
3 merge requests
!412
Manual attempt to merge preview into main
,
!402
Solved issue-194. Added missing information regarding usage of SSH config for...
,
!388
Resolve "Missing a tutorial about how to get a PyTorch to GPUs"
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
doc.zih.tu-dresden.de/docs/software/pytorch.md
+34
-24
34 additions, 24 deletions
doc.zih.tu-dresden.de/docs/software/pytorch.md
with
34 additions
and
24 deletions
doc.zih.tu-dresden.de/docs/software/pytorch.md
+
34
−
24
View file @
24ec0536
# PyTorch
[
PyTorch
](
https://pytorch.org/
)
{:target="_blank"}
is an open-source machine learning framework.
[
PyTorch
](
https://pytorch.org/
)
is an open-source machine learning framework.
It is an optimized tensor library for deep learning using GPUs and CPUs.
PyTorch is a machine learning tool developed by Facebooks AI division to process large-scale
object detection, segmentation, classification, etc.
...
...
@@ -13,9 +13,9 @@ Please check the software modules list via
marie@login$
module spider pytorch
```
to find out, which PyTorch modules are available
on your partition
.
to find out, which PyTorch modules are available.
We recommend using partitions alpha and/or ml when working with machine learning workflows
We recommend using partitions
`
alpha
`
and/or
`
ml
`
when working with machine learning workflows
and the PyTorch library.
You can find detailed hardware specification in our
[
hardware documentation
](
../jobs_and_resources/hardware_overview.md
)
.
...
...
@@ -25,7 +25,8 @@ You can find detailed hardware specification in our
On the partition
`alpha`
, load the module environment:
```
console
marie@login$
srun
-p
alpha
--gres
=
gpu:1
-n
1
-c
7
--pty
--mem-per-cpu
=
800 bash
#Job submission on alpha nodes with 1 gpu on 1 node with 800 Mb per CPU
#
Job submission on alpha nodes with 1 gpu on 1 node with 800 Mb per CPU
marie@login$
srun
-p
alpha
--gres
=
gpu:1
-n
1
-c
7
--pty
--mem-per-cpu
=
800 bash
marie@alpha$
module load modenv/hiera GCC/10.2.0 CUDA/11.1.1 OpenMPI/4.0.5 PyTorch/1.9.0
Die folgenden Module wurden in einer anderen Version erneut geladen:
1) modenv/scs5 =>
modenv/hiera
...
...
@@ -34,6 +35,7 @@ Module GCC/10.2.0, CUDA/11.1.1, OpenMPI/4.0.5, PyTorch/1.9.0 and 54 dependencies
```
??? hint "Torchvision on partition
`alpha`
"
On the partition `alpha`, the module torchvision is not yet available within the module
system. (19.08.2021)
Torchvision can be made available by using a virtual environment:
...
...
@@ -50,7 +52,8 @@ Module GCC/10.2.0, CUDA/11.1.1, OpenMPI/4.0.5, PyTorch/1.9.0 and 54 dependencies
On the partition
`ml`
:
```
console
marie@login$
srun
-p
ml
--gres
=
gpu:1
-n
1
-c
7
--pty
--mem-per-cpu
=
800 bash
#Job submission in ml nodes with 1 gpu on 1 node with 800 Mb per CPU
#
Job submission
in
ml nodes with 1 gpu on 1 node with 800 Mb per CPU
marie@login$
srun
-p
ml
--gres
=
gpu:1
-n
1
-c
7
--pty
--mem-per-cpu
=
800 bash
```
After calling
...
...
@@ -75,19 +78,24 @@ marie@{ml,alpha}$ python -c "import torch; print(torch.__version__)"
The following example shows how to create a python virtual environment and import PyTorch.
```
console
marie@ml$
mkdir
python-environments
#create folder
marie@ml$
which python
#check which python are you using
#
Create folder
marie@ml$
mkdir
python-environments
#
Check which python are you using
marie@ml$
which python
/sw/installed/Python/3.7.4-GCCcore-8.3.0/bin/python
marie@ml$
virtualenv
--system-site-packages
python-environments/env
#create virtual environment "env" which inheriting with global site packages
#
Create virtual environment
"env"
which inheriting with global site packages
marie@ml$
virtualenv
--system-site-packages
python-environments/env
[...]
marie@ml$
source
python-environments/env/bin/activate
#activate virtual environment "env". Example output: (env) bash-4.2$
#
Activate virtual environment
"env"
.
Example output:
(
env
)
bash-4.2
$
marie@ml$
source
python-environments/env/bin/activate
marie@ml$
python
-c
"import torch; print(torch.__version__)"
```
## PyTorch in JupyterHub
In addition to using interactive and batch jobs, it is possible to work with PyTorch using JupyterHub.
The production and test environments of JupyterHub contain Python kernels, that come with a PyTorch support.
In addition to using interactive and batch jobs, it is possible to work with PyTorch using
JupyterHub. The production and test environments of JupyterHub contain Python kernels, that come
with a PyTorch support.

{: align="center"}
...
...
@@ -99,8 +107,8 @@ For details on how to run PyTorch with multiple GPUs and/or multiple nodes, see
## Migrate PyTorch-script from CPU to GPU
It is recommended to use GPUs when using large training data sets. While TensorFlow automatically
uses GPUs if they are available, in
PyTorch you have to move your tensors manually.
It is recommended to use GPUs when using large training data sets. While TensorFlow automatically
uses GPUs if they are available, in
PyTorch you have to move your tensors manually.
First, you need to import
`torch.cuda`
:
...
...
@@ -108,7 +116,8 @@ First, you need to import `torch.cuda`:
import torch.cuda
```
Then you define a
`device`
-variable, which is set to 'cuda' automatically when a GPU is available with this code:
Then you define a
`device`
-variable, which is set to 'cuda' automatically when a GPU is available
with this code:
```
python3
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
...
...
@@ -121,15 +130,16 @@ x_train = torch.FloatTensor(x_train).to(device)
y_train = torch.FloatTensor(y_train).to(device)
```
Remember that this does not break backward compatibility when you port the script back to a computer
without GPU, because without GPU,
`device`
is set to 'cpu'.
Remember that this does not break backward compatibility when you port the script back to a computer
without GPU, because without GPU,
`device`
is set to 'cpu'.
### Caveats
#### Moving Data Back to the CPU-Memory
The CPU cannot directly access variables stored on the GPU. If you want to use the variables, e.g. in a
`print`
-statement or
when editing with NumPy or anything that is not PyTorch, you have to move them back to the CPU-memory again. This then may look like this:
The CPU cannot directly access variables stored on the GPU. If you want to use the variables, e.g.,
in a
`print`
statement or when editing with NumPy or anything that is not PyTorch, you have to move
them back to the CPU-memory again. This then may look like this:
```
python3
cpu_x_train = x_train.cpu()
...
...
@@ -138,8 +148,8 @@ print(cpu_x_train)
error_train = np.sqrt(metrics.mean_squared_error(y_train[:,1].cpu(), y_prediction_train[:,1]))
```
Remember that, without
`.detach()`
before the CPU, if you change
`cpu_x_train`
,
`x_train`
will also
be changed.
If you want to treat them independently, use
Remember that, without
`.detach()`
before the CPU, if you change
`cpu_x_train`
,
`x_train`
will also
be changed.
If you want to treat them independently, use
```
python3
cpu_x_train = x_train.detach().cpu()
...
...
@@ -149,7 +159,7 @@ Now you can change `cpu_x_train` without `x_train` being affected.
#### Speed Improvements and Batch Size
When you have a lot of very small data points, the speed may actually decrease when you try to train
them on the GPU.
This is because moving data from the CPU-memory to the GPU-memory takes time. If
this occurs, please try using
a very large batch size. This way, copying back and forth only takes
places a few times and the bottleneck may
be reduced.
When you have a lot of very small data points, the speed may actually decrease when you try to train
them on the GPU.
This is because moving data from the CPU-memory to the GPU-memory takes time. If
this occurs, please try using
a very large batch size. This way, copying back and forth only takes
places a few times and the bottleneck may
be reduced.
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment