Skip to content
Snippets Groups Projects

Update the distributed_training.md Pytorch section

1 file
+ 2
2
Compare changes
  • Side-by-side
  • Inline
@@ -159,8 +159,8 @@ Python. To work around this issue and gain performance benefits of parallelism,
@@ -159,8 +159,8 @@ Python. To work around this issue and gain performance benefits of parallelism,
`torch.nn.DistributedDataParallel` is recommended. This involves little more code changes to set up,
`torch.nn.DistributedDataParallel` is recommended. This involves little more code changes to set up,
but further increases the performance of model training. The starting step is to initialize the
but further increases the performance of model training. The starting step is to initialize the
process group by calling the `torch.distributed.init_process_group()` using the appropriate backend
process group by calling the `torch.distributed.init_process_group()` using the appropriate backend
such as 'nccl', 'mpi' or 'gloo'. The use of 'nccl' as backend is recommended as it is currently the
such as NCCL, MPI or Gloo. The use of NCCL as back end is recommended as it is currently the fastest
fastest backend when using GPUs.
back end when using GPUs.
#### Using Multiple GPUs with PyTorch
#### Using Multiple GPUs with PyTorch
Loading