diff --git a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md b/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md index c52cd15c8c297adf5840e401e3944e0e32244059..bc07ba923a74c8afc85589871c6d89ff47ce0dd5 100644 --- a/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md +++ b/doc.zih.tu-dresden.de/docs/jobs_and_resources/slurm.md @@ -301,6 +301,54 @@ provide a comprehensive collection of job examples. * Submisson: `marie@login$ sbatch batch_script.sh` * Run with fewer MPI tasks: `marie@login$ sbatch --ntasks=14 batch_script.sh` +## Heterogeneous Jobs + +A heterogeneous job consists of several job components, all of which can have individual job +options. In particular, different components can use resources from different Slurm partitions. +One example for this setting is an MPI application consisting of a master process with a huge memory +footprint and worker processes requiring GPU support. + +The `salloc`, `sbatch` and `srun` commands can all be used to submit heterogeneous jobs. Resource +specifications for each component of the heterogeneous job should be separated with ":" character. +Running a job step on a specific component is supported by the option `--het-group`. + +```console +marie@login$ salloc --ntasks 1 --cpus-per-task 4 --partition <partition> --mem=200G : \ + --ntasks 8 --cpus-per-task 1 --gres=gpu:8 --mem=80G --partition <partition> +[...] +marie@login$ srun ./my_application <args for master tasks> : ./my_application <args for worker tasks> +``` + +Heterogeneous jobs can also be defined in job files. There, it is required to separate multiple +components by a line containing the directive `"#SBATCH hetjob`. + +```bash +#!/bin/bash + +#SBATCH --ntasks 1 +#SBATCH --cpus 4 +#SBATCH --partition <partition> +#SBATCH --mem=200G +#SBATCH hetjob # required to separate groups +#SBATCH --ntasks 8 +#SBATCH --cpus 1 +#SBATCH --gres=gpu:8 +#SBATCH --mem=80G +#SBATCH --partition <partition> + +srun ./my_application <args for master tasks> : ./my_application <args for worker tasks> + +# or as an alternative +srun ./my_application <args for master tasks> & +srun --het-group=1 ./my_application <args for worker tasks> & +wait +``` + +### Limitations + +Due to the way scheduling algorithm works it is required that each component has to be allocated on +a different node. Furthermore, job arrays of heterogeneous jobs are not supported. + ## Manage and Control Jobs ### Job and Slurm Monitoring