Skip to content
Snippets Groups Projects
Commit 313ef99d authored by Martin Schroschk's avatar Martin Schroschk
Browse files

Fix workspace spaces

parent ffa6cca4
No related branches found
No related tags found
2 merge requests!938Automated merge from preview to main,!936Update to Five-Cluster-Operation
......@@ -41,7 +41,7 @@ marie@login$ # source $FOAM_CSH
OUTFILE="Output"
module load OpenFOAM
source $FOAM_BASH
cd /horse/ws/1/marie-example-workspace # work directory using workspace
cd /horse/ws/marie-example-workspace # work directory using workspace
srun pimpleFoam -parallel > "$OUTFILE"
```
......@@ -61,7 +61,7 @@ geometry and mesh generator cfx5pre, and the post-processor cfx5post.
#SBATCH --mail-type=ALL
module load ANSYS
cd /horse/ws/1/marie-example-workspace # work directory using workspace
cd /horse/ws/marie-example-workspace # work directory using workspace
cfx-parallel.sh -double -def StaticMixer.def
```
......
......@@ -54,7 +54,7 @@ once:
```console
marie@login$ ws_allocate EasyBuild 50
marie@login$ ws_list | grep 'directory.*EasyBuild'
workspace directory : /data/horse/ws/1/marie-EasyBuild
workspace directory : /data/horse/ws/marie-EasyBuild
```
**Step 2:** Allocate nodes. You can do this with interactive jobs (see the example below) and/or
......
......@@ -211,7 +211,7 @@ from the [MUST documentation v1.7.2](https://hpc.rwth-aachen.de/must/files/Docum
html files. Copy the files to your local host, e.g.
```console
marie@local$ scp -r dataport1.hpc.tu-dresden.de:/data/horse/ws/0/marie-must/{MUST_Output-files,MUST_Output.html}
marie@local$ scp -r dataport1.hpc.tu-dresden.de:/data/horse/ws/marie-must/{MUST_Output-files,MUST_Output.html}
```
and open the file `MUST_Output.html` using a webbrowser. Alternativly, you can open the html file with a
......
......@@ -93,7 +93,7 @@ For runs with [Slurm](../jobs_and_resources/slurm.md), please use a script like
module load modenv/classic
module load gamess
rungms.slurm cTT_M_025.inp /data/horse/ws/0/marie-gamess
rungms.slurm cTT_M_025.inp /data/horse/ws/marie-gamess
# the third parameter is the location of your horse directory
```
......
......@@ -54,9 +54,9 @@ Create a container from the image from the NGC catalog.
```console
marie@login.alpha$ srun --nodes=1 --ntasks-per-node=1 --ntasks=1 --gres=gpu:1 --time=08:00:00 --pty --mem=50000 bash
[...]
marie@alpha$ cd /data/horse/ws/<name_of_your_workspace>/containers #please create a Workspace
[...]
marie@alpha$ singularity pull pytorch:21.08-py3.sif docker://nvcr.io/nvidia/pytorch:21.08-py3
```
......@@ -113,7 +113,7 @@ on the ImageNet dataset inside the PyTorch container:
```console
marie@login.alpha$ srun --nodes=1 --ntasks-per-node=1 --ntasks=1 --gres=gpu:1 --time=08:00:00 --pty --mem=50000 \
singularity exec --nv -B /data/horse/ws/0/anpo879a-ImgNet/imagenet:/data/imagenet pytorch:21.06-py3 \
singularity exec --nv -B /data/horse/ws/anpo879a-ImgNet/imagenet:/data/imagenet pytorch:21.06-py3 \
python /workspace/examples/resnet50v1.5/multiproc.py --nnodes=1 --nproc_per_node 1 \
--node_rank=0 /workspace/examples/resnet50v1.5/main.py --data-backend dali-cpu --raport-file raport.json \
-j16 -p 100 --lr 2.048 --optimizer-batch-size 2048 --warmup 8 --arch resnet50 -c fanin --label-smoothing 0.1 \
......@@ -140,7 +140,7 @@ marie@login.alpha$ srun --nodes=1 --ntasks-per-node=8 --ntasks=8 --gres=gpu:8 --
```
```console
marie@alpha$ singularity exec --nv -B /data/horse/ws/0/marie-ImgNet/imagenet:/data/imagenet pytorch:21.06-py3 \
marie@alpha$ singularity exec --nv -B /data/horse/ws/marie-ImgNet/imagenet:/data/imagenet pytorch:21.06-py3 \
python /workspace/examples/resnet50v1.5/multiproc.py --nnodes=1 --nproc_per_node 8 \
--node_rank=0 /workspace/examples/resnet50v1.5/main.py --data-backend dali-cpu \
--raport-file raport.json -j16 -p 100 --lr 2.048 --optimizer-batch-size 2048 --warmup 8 \
......
......@@ -50,11 +50,11 @@ Then create the virtual environment and activate it.
```console
marie@compute$ ws_allocate python_virtual_environment 1
Info: creating workspace.
/horse/ws/1/marie-python_virtual_environment
/horse/ws/marie-python_virtual_environment
[...]
marie@compute$ virtualenv --system-site-packages /horse/ws/1/marie-python_virtual_environment/env #Create virtual environment
marie@compute$ virtualenv --system-site-packages /horse/ws/marie-python_virtual_environment/env #Create virtual environment
[...]
marie@compute$ source /horse/ws/1/marie-python_virtual_environment/env/bin/activate #Activate virtual environment. Example output: (env) bash-4.2$
marie@compute$ source /horse/ws/marie-python_virtual_environment/env/bin/activate #Activate virtual environment. Example output: (env) bash-4.2$
```
Now you can work in this isolated environment, without interfering with other
......@@ -73,7 +73,7 @@ the environment as follows:
```console
marie@login.alpha$ srun --nodes=1 --gres=gpu:1 --time=01:00:00 --pty bash
marie@alpha$ ws_allocate my_python_virtualenv 100 # use a workspace for the environment
marie@alpha$ cd /horse/ws/1/marie-my_python_virtualenv
marie@alpha$ cd /horse/ws/marie-my_python_virtualenv
marie@alpha$ module load modenv/hiera GCC/10.2.0 CUDA/11.1.1 OpenMPI/4.0.5 PyTorch/1.9.0
Module GCC/10.2.0, CUDA/11.1.1, OpenMPI/4.0.5, PyTorch/1.9.0 and 54 dependencies loaded.
marie@alpha$ which python
......@@ -113,9 +113,9 @@ packages from the file:
```console
marie@compute$ module load Python #Load default Python
[...]
marie@compute$ virtualenv --system-site-packages /horse/ws/1/marie-python_virtual_environment/env_post #Create virtual environment
marie@compute$ virtualenv --system-site-packages /horse/ws/marie-python_virtual_environment/env_post #Create virtual environment
[...]
marie@compute$ source /horse/ws/1/marie-python_virtual_environment/env/bin/activate #Activate virtual environment. Example output: (env_post) bash-4.2$
marie@compute$ source /horse/ws/marie-python_virtual_environment/env/bin/activate #Activate virtual environment. Example output: (env_post) bash-4.2$
(env_post) marie@compute$ pip install -r requirements.txt #Install packages from the created requirements.txt file
```
......@@ -147,7 +147,7 @@ directory for the conda virtual environment:
```console
marie@compute$ ws_allocate conda_virtual_environment 1
Info: creating workspace.
/horse/ws/1/marie-conda_virtual_environment
/horse/ws/marie-conda_virtual_environment
[...]
```
......@@ -156,8 +156,8 @@ environment:
```console
marie@compute$ module load Anaconda3 #load Anaconda module
marie@compute$ conda create --prefix /horse/ws/1/marie-conda_virtual_environment/conda-env python=3.6 #create virtual environment with Python version 3.6
marie@compute$ conda activate /horse/ws/1/marie-conda_virtual_environment/conda-env #activate conda-env virtual environment
marie@compute$ conda create --prefix /horse/ws/marie-conda_virtual_environment/conda-env python=3.6 #create virtual environment with Python version 3.6
marie@compute$ conda activate /horse/ws/marie-conda_virtual_environment/conda-env #activate conda-env virtual environment
```
Now you can work in this isolated environment, without interfering with other
......@@ -185,7 +185,7 @@ can deactivate the conda environment as follows:
```console
marie@login.alpha$ srun --nodes=1 --gres=gpu:1 --time=01:00:00 --pty bash
marie@alpha$ ws_allocate my_conda_virtualenv 100 # use a workspace for the environment
marie@alpha$ cd /horse/ws/1/marie-my_conda_virtualenv
marie@alpha$ cd /horse/ws/marie-my_conda_virtualenv
marie@alpha$ module load Anaconda3
Module Anaconda3/2021.11 loaded.
marie@alpha$ conda create --prefix my-torch-env python=3.8
......@@ -250,8 +250,8 @@ Recreate the conda virtual environment with the packages from the created
`environment.yml` file:
```console
marie@compute$ mkdir /horse/ws/1/marie-conda_virtual_environment/conda-env #Create directory for environment
marie@compute$ mkdir /horse/ws/marie-conda_virtual_environment/conda-env #Create directory for environment
marie@compute$ module load Anaconda3 #Load Anaconda
marie@compute$ conda config --set channel_priority strict
marie@compute$ conda env create --prefix /horse/ws/1/marie-conda_virtual_environment/conda-env --file environment.yml #Create conda env in directory with packages from environment.yml file
marie@compute$ conda env create --prefix /horse/ws/marie-conda_virtual_environment/conda-env --file environment.yml #Create conda env in directory with packages from environment.yml file
```
......@@ -63,7 +63,7 @@ already have an event data directory, you can skip that step.
```console
marie@compute$ ws_allocate -F /data/horse tensorboard_logdata 1
Info: creating workspace.
/data/horse/ws/1/marie-tensorboard_logdata
/data/horse/ws/marie-tensorboard_logdata
[...]
```
......@@ -72,7 +72,7 @@ accessible for TensorBoard. Please find further information on the official [Ten
Then, you can start TensorBoard and pass the directory of the event data:
```console
marie@compute$ tensorboard --logdir /data/horse/ws/1/marie-tensorboard_logdata --bind_all
marie@compute$ tensorboard --logdir /data/horse/ws/marie-tensorboard_logdata --bind_all
[...]
TensorBoard 2.3.0 at http://taurusi8034.taurus.hrsk.tu-dresden.de:6006/
[...]
......
......@@ -72,13 +72,13 @@ import TensorFlow:
!!! example
```console
marie@power$ ws_allocate -F /data/horse python_virtual_environment 1
marie@power$ ws_allocate -F horse python_virtual_environment 1
Info: creating workspace.
/data/horse/ws/1/python_virtual_environment
/data/horse/ws/python_virtual_environment
[...]
marie@power$ which python #check which python are you using
/sw/installed/Python/3.7.2-GCCcore-8.2.0
marie@power$ virtualenv --system-site-packages /data/horse/ws/1/marie-python_virtual_environment/env
marie@power$ virtualenv --system-site-packages /data/horse/ws/marie-python_virtual_environment/env
[...]
marie@power$ source /data/horse/ws/marie-python_virtual_environment/env/bin/activate
marie@power$ python -c "import tensorflow as tf; print(tf.__version__)"
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment