Skip to content
Snippets Groups Projects
Commit 8fe0544d authored by Elias Werner's avatar Elias Werner
Browse files

first draft for python virtual environments, structured and tidy up

parent e89b49af
No related branches found
No related tags found
5 merge requests!333Draft: update NGC containers,!322Merge preview into main,!319Merge preview into main,!279Draft: Machine Learning restructuring,!258Data Analytics restructuring
Hide whitespace changes
Inline Side-by-side
doc.zih.tu-dresden.de/docs/software/python_virtual_environments.md
+ 89
− 105
View file @ 8fe0544d
# Python Virtual Environments
## TODO
Link to this page from other DA/ML topics.
## copied from alpha_centauri.md
??? comment
copied from `alpha_centauri.md`. Please remove there if this article is finished
Virtual environments allow users to install additional python packages and create an isolated
run-time environment. We recommend using `virtualenv` for this purpose.
```console
marie@login$ srun --partition=alpha-interactive --nodes=1 --cpus-per-task=1 --gres=gpu:1 --time=01:00:00 --pty bash
marie@alpha$ mkdir python-environments # please use workspaces
marie@alpha$ module load modenv/hiera GCC/10.2.0 CUDA/11.1.1 OpenMPI/4.0.5 PyTorch
Module GCC/10.2.0, CUDA/11.1.1, OpenMPI/4.0.5, PyTorch/1.9.0 and 54 dependencies loaded.
marie@alpha$ which python
/sw/installed/Python/3.8.6-GCCcore-10.2.0/bin/python
marie@alpha$ pip list
[...]
marie@alpha$ virtualenv --system-site-packages python-environments/my-torch-env
created virtual environment CPython3.8.6.final.0-64 in 42960ms
creator CPython3Posix(dest=~/python-environments/my-torch-env, clear=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=~/.local/share/virtualenv)
added seed packages: pip==21.1.3, setuptools==57.2.0, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
marie@alpha$ source python-environments/my-torch-env/bin/activate
(my-torch-env) marie@alpha$ pip install torchvision
[...]
Installing collected packages: torchvision
Successfully installed torchvision-0.10.0
[...]
(my-torch-env) marie@alpha$ python -c "import torchvision; print(torchvision.__version__)"
0.10.0+cu102
(my-torch-env) marie@alpha$ deactivate
```
run-time environment. We recommend using `virtualenv` for this purpose. In your virtual environment,
you can use packages from the (Complete List of Modules) or if you didn't find what you need you can
install required packages with the command: `pip install`. With the command `pip freeze`, you can
see a list of all installed packages and their versions.
## copied from python.md
??? comment
clear up the following. Maybe leave only conda stuff...
There are two methods of how to work with virtual environments on
ZIH system:
There are two methods of how to work with virtual environments on ZIH systems:
1. **virtualenv** is a standard Python tool to create isolated Python environments.
It is the preferred interface for
......@@ -55,77 +17,99 @@ virtual environments on ZIH system. conda is an open-source package
management system and environment management system from Anaconda. The
conda manager is included in all versions of Anaconda and Miniconda.
**Note:** Keep in mind that you **cannot** use virtualenv for working
with the virtual environments previously created with conda tool and
vice versa! Prefer virtualenv whenever possible.
!!! warning
Keep in mind that you **cannot** use virtualenv for working
with the virtual environments previously created with conda tool and
vice versa! Prefer virtualenv whenever possible.
## Python virtual environment
This example shows how to start working with **virtualenv** and Python virtual environment (using
the module system). At first we use an interactive job and create a directory for the virtual
environment:
```console
marie@login$ srun -p alpha -N 1 -n 1 -c 7 --mem-per-cpu=5772 --gres=gpu:1 --time=04:00:00 --pty bash #Job submission in ml nodes with 1 gpu on 1 node.
marie@alpha$ mkdir python-environments #Optional: Create folder. Please use Workspaces!
```
Now we check available Python modules and load the preferred version:
```console
marie@alpha$ ml av Python #Check the available modules with Python
marie@alpha$ module load Python #Load default Python. Example output: Module Python/3.7 4-GCCcore-8.3.0 with 7 dependencies loaded
marie@alpha$ which python #Check which python are you using
```
Then create the virtual environment and activate it:
```console
marie@alpha$ virtualenv --system-site-packages python-environments/envtest #Create virtual environment
marie@alpha$ source python-environments/envtest/bin/activate #Activate virtual environment. Example output: (envtest) bash-4.2$
```
This example shows how to start working
with **virtualenv** and Python virtual environment (using the module system)
Now you can work in this isolated environment, without interferring with other tasks running on the
system. Note that the inscription (env) at the beginning of each line represents that you are in
the virtual environment. You can deactivate the environment as follows:
```Bash
srun -p ml -N 1 -n 1 -c 7 --mem-per-cpu=5772 --gres=gpu:1 --time=04:00:00 --pty bash #Job submission in ml nodes with 1 gpu on 1 node.
```console
(envtest) marie@alpha$ deactivate #Leave the virtual environment
```
mkdir python-environments # Optional: Create folder. Please use Workspaces!
## Conda virtual environment
module load modenv/ml # Changing the environment. Example output: The following have been reloaded with a version change: 1 modenv/scs5 => modenv/ml
ml av Python #Check the available modules with Python
module load Python #Load default Python. Example output: Module Python/3.7 4-GCCcore-8.3.0 with 7 dependencies loaded
which python #Check which python are you using
virtualenv --system-site-packages python-environments/envtest #Create virtual environment
source python-environments/envtest/bin/activate #Activate virtual environment. Example output: (envtest) bash-4.2$
python #Start python
This example shows how to start working with **conda** and virtual environment (with using module
system). At first we use an interactive job and create a directory for the conda virtual
environment:
from time import gmtime, strftime
print(strftime("%Y-%m-%d %H:%M:%S", gmtime())) #Example output: 2019-11-18 13:54:16
deactivate #Leave the virtual environment
```console
marie@login$ srun -p ml -N 1 -n 1 -c 7 --mem-per-cpu=5772 --gres=gpu:1 --time=04:00:00 --pty bash #Job submission in ml nodes with 1 gpu on 1 node.
marie@alpha$ mkdir conda-virtual-environments #create a folder
```
The [virtualenv](https://virtualenv.pypa.io/en/latest/) Python module (Python 3) provides support
for creating virtual environments with their own site-directories, optionally isolated from system
site directories. Each virtual environment has its own Python binary (which matches the version of
the binary that was used to create this environment) and can have its own independent set of
installed Python packages in its site directories. This allows you to manage separate package
installations for different projects. It essentially allows us to create a virtual isolated Python
installation and install packages into that virtual installation. When you switch projects, you can
simply create a new virtual environment and not have to worry about breaking the packages installed
in other environments.
In your virtual environment, you can use packages from the (Complete List of
Modules) or if you didn't find what you need you can install required packages
with the command: `pip install`. With the command `pip freeze`, you can see a list of all installed
packages and their versions.
This example shows how to start working with **conda** and virtual
environment (with using module system)
```Bash
srun -p ml -N 1 -n 1 -c 7 --mem-per-cpu=5772 --gres=gpu:1 --time=04:00:00 --pty bash # Job submission in ml nodes with 1 gpu on 1 node.
module load modenv/ml
mkdir conda-virtual-environments #create a folder
cd conda-virtual-environments #go to folder
which python #check which python are you using
module load PythonAnaconda/3.6 #load Anaconda module
which python #check which python are you using now
conda create -n conda-testenv python=3.6 #create virtual environment with the name conda-testenv and Python version 3.6
conda activate conda-testenv #activate conda-testenv virtual environment
conda deactivate #Leave the virtual environment
Then we load Anaconda, create an environment in our directory and activate the environment:
```console
marie@alpha$ module load Anaconda3 #load Anaconda module
marie@alpha$ conda create --prefix conda-virtual-environments/conda-testenv python=3.6 #create virtual environment with Python version 3.6
marie@alpha$ conda activate conda-virtual-environments/conda-testenv #activate conda-testenv virtual environment
```
You can control where a conda environment
lives by providing a path to a target directory when creating the
environment. For example, the following command will create a new
environment in a workspace located in `scratch`
Now you can work in this isolated environment, without interferring with other tasks running on the
system. Note that the inscription (env) at the beginning of each line represents that you are in
the virtual environment. You can deactivate the conda environment as follows:
```Bash
conda create --prefix /scratch/ws/<name_of_your_workspace>/conda-virtual-environment/<name_of_your_environment>
```console
(conda-testenv) marie@alpha$ conda deactivate #Leave the virtual environment
```
Please pay attention,
using srun directly on the shell will lead to blocking and launch an
interactive job. Apart from short test runs, it is **recommended to
launch your jobs into the background by using Slurm**. For that, you can conveniently put
the parameters directly into the job file which you can submit using
`sbatch [options] <job file>.`
TODO: Link to this page from other DA/ML topics. insert link in alpha centauri
??? example
This is an example on alpha partition. The example creates a virtual environment, and installs
the package `torchvision` with pip.
```console
marie@login$ srun --partition=alpha-interactive -N=1 --gres=gpu:1 --time=01:00:00 --pty bash
marie@alpha$ mkdir python-environments # please use workspaces
marie@alpha$ module load modenv/hiera GCC/10.2.0 CUDA/11.1.1 OpenMPI/4.0.5 PyTorch
Module GCC/10.2.0, CUDA/11.1.1, OpenMPI/4.0.5, PyTorch/1.9.0 and 54 dependencies loaded.
marie@alpha$ which python
/sw/installed/Python/3.8.6-GCCcore-10.2.0/bin/python
marie@alpha$ pip list
[...]
marie@alpha$ virtualenv --system-site-packages python-environments/my-torch-env
created virtual environment CPython3.8.6.final.0-64 in 42960ms
creator CPython3Posix(dest=~/python-environments/my-torch-env, clear=False, global=True)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=~/.local/share/virtualenv)
added seed packages: pip==21.1.3, setuptools==57.2.0, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
marie@alpha$ source python-environments/my-torch-env/bin/activate
(my-torch-env) marie@alpha$ pip install torchvision
[...]
Installing collected packages: torchvision
Successfully installed torchvision-0.10.0
[...]
(my-torch-env) marie@alpha$ python -c "import torchvision; print(torchvision.__version__)"
0.10.0+cu102
(my-torch-env) marie@alpha$ deactivate
```
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
Nutzungsbedingungen | Datenschutzerklärung | Barrierefreiheitserklärung