NGC containers documentation

NGC containers documentation MR !339 (merged)

@mark--tu-dresden.de @rotscher--tu-dresden.de Regarding NGC-containers documentation progress

changed the description

• Could please provide some further information about the performance differences between all your tests?
• Could please also provide some documentation, how you run the different tests?

Hi all @rotscher--tu-dresden.de @mark--tu-dresden.de.

I have done some tests regarding use NGC containers for the alpha partition and using NGC in general.

During the tests, I have discovered that NGC containers have changed a lot of structure and code inside exact NGC containers (PyTorch for example). From Pytorch 21.03 to PyTorch 21.06 they have changed code regarding the parallelization for example. As far as I understand, the way that authors of various containers with PyTorch used parallelization function is different and will be difficult to suggest one way of parallelization for different containers. The documentation regarding the advanced option is not perfect as well. I have made tests with PyTorch 21.06 for the classification task (downgraded ImageNet dataset) with ResNet. The main metrics were the number of images per second and latency. the main problem so far is that I can run some containers on 2 nodes (tnx for the solution from Danny) but with the PyTorch container I didn't succeed (I have tried different approaches). They are using Apex. I can run the container on the 1-8 GPUs from one node but that's all. I have compared the results obtained for the same configuration for the container and for the same code run on the node by srun interactively.

The command for the one node 8 GPUS with NGC container:

srun -p alpha-interactive --nodes 1 --ntasks 8 --gres=gpu:8 --time=08:00:00 --pty --mem=500000 bash

singularity shell --nv -B /warm_archive/ws/anpo879a-Imagenet/Imagenet_light/ImageNet/ImageNet:/data/imagenet pytorch:21.06-py3

python /workspace/examples/resnet50v1.5/multiproc.py --nproc_per_node=8 /workspace/examples/resnet50v1.5/main.py --arch resnet50 -c fanin --label-smoothing 0.1 --epochs=20 -b=64 /data/imagenet

For some reason, I was able to run it for 8 GPUs only like this with shell. Exec doesn't work

The command for the one node 1 GPUS with NGC container:

srun -p alpha-interactive --nodes 1 --ntasks 1 --gres=gpu:1 --time=02:00:00 --pty --mem=500000 singularity exec --nv -B /warm_archive/ws/anpo879a-Imagenet/Imagenet_light/ImageNet/ImageNet:/data/imagenet pytorch:21.06-py3 python /workspace/examples/resnet50v1.5/main.py --arch resnet50 -c fanin --label-smoothing 0.1 --epochs=20 -b=64 /data/imagenet

the comparison of the results shows that the results for interactive code running and container are pretty close regarding the performance. The latency for the containers is worse. The parallelization gives some improvements regarding the bandwidth (imgs/s) but this topic is required further research.

compsrison_pytorch_gpu3.ods

@rotscher--tu-dresden.de Dear Danny could you help me please solve the issue with the use of NGC containers for multiple nodes. I have no idea how to run it on multiple nodes.

Commands and errors:

According to previous experience with multiple gpu that's the most promissing way:

srun -p alpha --nodes 2 --ntasks 8 --gres=gpu:4 --time=08:00:00 --pty --mem=500000 bash

singularity shell --nv -B /warm_archive/ws/anpo879a-Imagenet/Imagenet_light/ImageNet/ImageNet:/data/imagenet pytorch:21.06-py3

python /workspace/examples/resnet50v1.5/multiproc.py --nnodes 2 --nproc_per_node 4 /workspace/examples/resnet50v1.5/main.py --arch resnet50 -c fanin --label-smoothing 0.1 --epochs=20 -b=64 /data/imagenet

Output:

['/opt/conda/bin/python', '-u', '/workspace/examples/resnet50v1.5/main.py', '--arch', 'resnet50', '-c', 'fanin', '--label-smoothing', '0.1', '--epochs=20', '-b=64', '/data/imagenet']

['/opt/conda/bin/python', '-u', '/workspace/examples/resnet50v1.5/main.py', '--arch', 'resnet50', '-c', 'fanin', '--label-smoothing', '0.1', '--epochs=20', '-b=64', '/data/imagenet']

['/opt/conda/bin/python', '-u', '/workspace/examples/resnet50v1.5/main.py', '--arch', 'resnet50', '-c', 'fanin', '--label-smoothing', '0.1', '--epochs=20', '-b=64', '/data/imagenet']

['/opt/conda/bin/python', '-u', '/workspace/examples/resnet50v1.5/main.py', '--arch', 'resnet50', '-c', 'fanin', '--label-smoothing', '0.1', '--epochs=20', '-b=64', '/data/imagenet']

and no response after that

The same behaviour can be detected with singularity exec

salloc:

salloc -p alpha-interactive --nodes 2 --ntasks 8 --gres=gpu:4 --time=08:00:00 --mem=500000 singularity exec --nv -B /wet_light/ImageNet/ImageNet:/data/imagenet pytorch:21.06-py3 python /workspace/examples/resnet50v1.5/multiproc.py --nnodes 2 --nproc_per_noain.py --arch resnet50 -c fanin --label-smoothing 0.1 --epochs=20 -b=64 /data/imagenet

Output:

INFO: Could not find any nv files on this host! ERROR : Failed to create mount namespace: Operation not permitted

changed milestone to %Umstellung

mentioned in merge request !333 (closed)

Hi all! Dear @rotscher--tu-dresden.de @s3248973--tu-dresden.de can one of you review my NGC container documentationplease? I'm done with it from my point of view.

salloc -p alpha-interactive --nodes 2 --ntasks 8 --gres=gpu:4 --time=08:00:00 --mem=500000 singularity exec --nv -B /wet_light/ImageNet/ImageNet:/data/imagenet pytorch:21.06-py3 python /workspace/examples/resnet50v1.5/multiproc.py --nnodes 2 --nproc_per_noain.py --arch resnet50 -c fanin --label-smoothing 0.1 --epochs=20 -b=64 /data/imagenet

salloc only creates an allocation. To actually run something ON the nodes (and not on the login node) you need to use srun. Of course the (CPU-only) Login nodes don't have any nvidia driver files hence the error

changed the description

mentioned in merge request !339 (merged)

mentioned in commit 2b55262e

closed with merge request !339 (merged)