From 6df7e77f2e915728cc2192692537f94467f1ba1a Mon Sep 17 00:00:00 2001
From: Morris Jette <jette@schedmd.com>
Date: Thu, 23 Aug 2018 18:58:52 -0600
Subject: [PATCH] Expose new cons_tres documentation

New --gpu options in the commands and some configuration file changes
---
 doc/man/man1/salloc.1     | 396 +++++++++++++++++++------------------
 doc/man/man1/sbatch.1     | 396 +++++++++++++++++++------------------
 doc/man/man1/scontrol.1   |  26 +--
 doc/man/man1/srun.1       | 397 +++++++++++++++++++-------------------
 doc/man/man5/gres.conf.5  |  22 +--
 doc/man/man5/slurm.conf.5 | 208 ++++++++------------
 6 files changed, 689 insertions(+), 756 deletions(-)

diff --git a/doc/man/man1/salloc.1 b/doc/man/man1/salloc.1
index d02939d3217..cd2672eb323 100644
--- a/doc/man/man1/salloc.1
+++ b/doc/man/man1/salloc.1
@@ -375,13 +375,13 @@ Accounting is able to poll for job information. As a result not all of
 accounting information will be present.
 .RE
 
-.\ .TP
-.\ \fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
-.\ Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
-.\ allocated GPU.
-.\ Requires the \fB\-\-gpus\fR option.
-.\ Not compatible with the \fB\-\-cpus\-per\-task\fR option.
-.\ 
+.TP
+\fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
+Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
+allocated GPU.
+Requires the \fB\-\-gpus\fR option.
+Not compatible with the \fB\-\-cpus\-per\-task\fR option.
+
 .TP
 \fB\-c\fR, \fB\-\-cpus\-per\-task\fR=<\fIncpus\fR>
 Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
@@ -522,121 +522,121 @@ Submit the job with the specified \fIgroup\fR's group access permissions.
 In the default Slurm configuration, this option is only valid when used
 by the user root.
 
-.\ .TP
-.\ \fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the total number of GPUs required for the job.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus=volta:3,kepler:1".
-.\ See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpu\-bind\fR=<\fItype\fR>
-.\ Bind tasks to specific GPUs.
-.\ By default every spawned task can access every GPU allocated to the job.
-.\ 
-.\ Supported \fItype\fR options:
-.\ .RS
-.\ .TP 10
-.\ \fBclosest\fR
-.\ Bind each task to the GPU(s) which are closest.
-.\ In a NUMA environment, each task may be bound to more than one GPU (i.e.
-.\ all GPUs in that NUMA environment).
-.\ .TP
-.\ \fBmap_gpu:<list>\fR
-.\ Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
-.\ <gpu_id_for_task_0>,<gpu_id_for_task_1>,... GPU IDs are interpreted as decimal
-.\ values unless they are preceded with '0x' in which case they interpreted as
-.\ hexadecimal values. If the number of tasks (or ranks) exceeds the number of
-.\ elements in this list, elements in the list will be reused as needed starting
-.\ from the beginning of the list. To simplify support for large task counts,
-.\ the lists may follow a map with an asterisk and repetition count.
-.\ For example "map_cpu:0*4,1*4".
-.\ Not supported unless the entire node is allocated to the job.
-.\ .TP
-.\ \fBmask_gpu:<list>\fR
-.\ Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
-.\ <gpu_mask_for_task_0>,<gpu_mask_for_task_1>,... The mapping is specified for
-.\ a node and identical mapping is applied to the tasks on every node (i.e. the
-.\ lowest task ID on each node is mapped to the first mask specified in the list,
-.\ etc.). GPU masks are always interpreted as hexadecimal values but can be
-.\ preceded with an optional '0x'. Not supported unless the entire node is
-.\ allocated to the job. To simplify support for large task counts, the lists
-.\ may follow a map with an asterisk and repetition count.
-.\ For example "mask_gpu:0x0f*4,0xf0*4".
-.\ Not supported unless the entire node is allocated to the job.
-.\ .RE
-.\ 
-.\ .TP
-.\ \fB\-\-gpu\-freq\fR=[<\fItype\fR]=\fIvalue\fR>[:<\fItype\fR=\fIvalue\fR>]
-.\ Request that GPUs allocated to the job are configured with specific voltage
-.\ and/or frequency values.
-.\ This option can be used to independently configure the GPU and its memory
-.\ frequencies.
-.\ In some cases, system power caps may override the requested values.
-.\ The field \fItype\fR can be "memory" or "voltage".
-.\ If \fItype\fR is not specified. the GPU frequency is implied.
-.\ The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
-.\ a numeric value (megahertz or volts).
-.\ If the specified numeric value is not possible, a value as close as
-.\ possible will be used. See below for definition of the values.
-.\ Examples of use include "\-\-gpu\-freq\fR=medium:memory=high:voltage=high" and
-.\ \-\-gpu\-freq\fR=450".
-.\ 
-.\ Supported \fIvalue\fR definitions:
-.\ .RS
-.\ .TP 10
-.\ \fBlow\fR
-.\ the lowest available frequency or voltage
-.\ .TP
-.\ \fBmedium\fR
-.\ attempts to set a frequency  or voltage in the middle of the available range
-.\ .TP
-.\ \fBhigh\fR
-.\ the highest available frequency or voltage
-.\ .TP
-.\ \fBhighm1\fR
-.\ (high minus one) will select the next highest available frequency or voltage
-.\ .RE
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-node\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each node included in
-.\ the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus\-per\-node=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-node=volta:3,kepler:1".
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-socket\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each socket included in
-.\ the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus\-per\-socket=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-socket=volta:3,kepler:1".
-.\ Requires job to specify a sockets per node count ( \-\-sockets\-per\-node).
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-node\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-task\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each task to be spawned
-.\ in the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ This option requires the specification of a task count.
-.\ For example "\-\-gpus\-per\-task=volta:1".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-task=volta:3,kepler:1".
-.\ Requires job to specify a task count (\-\-nodes).
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-node\fR options.
-.\ 
+.TP
+\fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the total number of GPUs required for the job.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus=volta:3,kepler:1".
+See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpu\-bind\fR=<\fItype\fR>
+Bind tasks to specific GPUs.
+By default every spawned task can access every GPU allocated to the job.
+
+Supported \fItype\fR options:
+.RS
+.TP 10
+\fBclosest\fR
+Bind each task to the GPU(s) which are closest.
+In a NUMA environment, each task may be bound to more than one GPU (i.e.
+all GPUs in that NUMA environment).
+.TP
+\fBmap_gpu:<list>\fR
+Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
+<gpu_id_for_task_0>,<gpu_id_for_task_1>,... GPU IDs are interpreted as decimal
+values unless they are preceded with '0x' in which case they interpreted as
+hexadecimal values. If the number of tasks (or ranks) exceeds the number of
+elements in this list, elements in the list will be reused as needed starting
+from the beginning of the list. To simplify support for large task counts,
+the lists may follow a map with an asterisk and repetition count.
+For example "map_cpu:0*4,1*4".
+Not supported unless the entire node is allocated to the job.
+.TP
+\fBmask_gpu:<list>\fR
+Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
+<gpu_mask_for_task_0>,<gpu_mask_for_task_1>,... The mapping is specified for
+a node and identical mapping is applied to the tasks on every node (i.e. the
+lowest task ID on each node is mapped to the first mask specified in the list,
+etc.). GPU masks are always interpreted as hexadecimal values but can be
+preceded with an optional '0x'. Not supported unless the entire node is
+allocated to the job. To simplify support for large task counts, the lists
+may follow a map with an asterisk and repetition count.
+For example "mask_gpu:0x0f*4,0xf0*4".
+Not supported unless the entire node is allocated to the job.
+.RE
+
+.TP
+\fB\-\-gpu\-freq\fR=[<\fItype\fR]=\fIvalue\fR>[:<\fItype\fR=\fIvalue\fR>]
+Request that GPUs allocated to the job are configured with specific voltage
+and/or frequency values.
+This option can be used to independently configure the GPU and its memory
+frequencies.
+In some cases, system power caps may override the requested values.
+The field \fItype\fR can be "memory" or "voltage".
+If \fItype\fR is not specified. the GPU frequency is implied.
+The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
+a numeric value (megahertz or volts).
+If the specified numeric value is not possible, a value as close as
+possible will be used. See below for definition of the values.
+Examples of use include "\-\-gpu\-freq\fR=medium:memory=high:voltage=high" and
+\-\-gpu\-freq\fR=450".
+
+Supported \fIvalue\fR definitions:
+.RS
+.TP 10
+\fBlow\fR
+the lowest available frequency or voltage
+.TP
+\fBmedium\fR
+attempts to set a frequency  or voltage in the middle of the available range
+.TP
+\fBhigh\fR
+the highest available frequency or voltage
+.TP
+\fBhighm1\fR
+(high minus one) will select the next highest available frequency or voltage
+.RE
+
+.TP
+\fB\-\-gpus\-per\-node\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each node included in
+the job's resource allocation.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus\-per\-node=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-node=volta:3,kepler:1".
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpus\-per\-socket\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each socket included in
+the job's resource allocation.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus\-per\-socket=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-socket=volta:3,kepler:1".
+Requires job to specify a sockets per node count ( \-\-sockets\-per\-node).
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-node\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpus\-per\-task\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each task to be spawned
+in the job's resource allocation.
+An optional GPU type specification can be supplied.
+This option requires the specification of a task count.
+For example "\-\-gpus\-per\-task=volta:1".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-task=volta:3,kepler:1".
+Requires job to specify a task count (\-\-nodes).
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-node\fR options.
+
 .TP
 \fB\-\-gres\fR=<\fIlist\fR>
 Specifies a comma delimited list of generic consumable resources.
@@ -909,12 +909,9 @@ Default value is \fBDefMemPerNode\fR and the maximum value is
 seen using the \fBscontrol show config\fR command.
 This parameter would generally be used if whole nodes
 are allocated to jobs (\fBSelectType=select/linear\fR).
-Also see \fB\-\-mem\-per\-cpu\fR.
-The \fB\-\-mem\fR and \fB\-\-mem\-per\-cpu\fR options are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
+Also see fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR.
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
 
 NOTE: A memory size specification of zero is treated as a special case and
 grants the job access to all of the memory on each node.
@@ -949,27 +946,24 @@ are allocated to jobs (\fBSelectType=select/cons_res\fR).
 If resources are allocated by the core, socket or whole nodes; the number
 of CPUs allocated to a job may be higher than the task count and the value
 of \fB\-\-mem\-per\-cpu\fR should be adjusted accordingly.
+Also see \fB\-\-mem\fR and fB\-\-mem\-per\-gpu\fR.
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
+
+.TP
+\fB\-\-mem\-per\-gpu\fR=<\fIsize[units]\fR>
+Minimum memory required per allocated GPU.
+Default units are megabytes unless the SchedulerParameters configuration
+parameter includes the "default_gbytes" option for gigabytes.
+Different units can be specified using the suffix [K|M|G|T].
+Default value is \fBDefMemPerGPU\fR and is available on both a global and
+per partition basis.
+If configured, the parameters can be seen using the \fBscontrol show config\fR
+and \fBscontrol show partition\fR commands.
 Also see \fB\-\-mem\fR.
-\fB\-\-mem\fR and \fB\-\-mem\-per\-cpu\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see \fB\-\-mem\fR and fB\-\-mem\-per\-gpu\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
-
-.\ .TP
-.\ \fB\-\-mem\-per\-gpu\fR=<\fIsize[units]\fR>
-.\ Minimum memory required per allocated GPU.
-.\ Default units are megabytes unless the SchedulerParameters configuration
-.\ parameter includes the "default_gbytes" option for gigabytes.
-.\ Different units can be specified using the suffix [K|M|G|T].
-.\ Default value is \fBDefMemPerGPU\fR and is available on both a global and
-.\ per partition basis.
-.\ If configured, the parameters can be seen using the \fBscontrol show config\fR
-.\ and \fBscontrol show partition\fR commands.
-.\ Also see \fB\-\-mem\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
-.\ 
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
+
 .TP
 \fB\-\-mem\-bind\fR=[{\fIquiet,verbose\fR},]\fItype\fR
 Bind tasks to memory. Used only when the task/affinity plugin is enabled
@@ -1625,9 +1619,9 @@ Same as \fB\-C\fR, \fB\-\-constraint\fR
 .TP
 \fBSALLOC_CORE_SPEC\fR
 Same as \fB\-\-core\-spec\fR
-.\ .TP
-.\ \fBSALLOC_CPUS_PER_GPU\fR
-.\ Same as \fB\-\-cpus\-per\-gpu\fR
+.TP
+\fBSALLOC_CPUS_PER_GPU\fR
+Same as \fB\-\-cpus\-per\-gpu\fR
 .TP
 \fBSALLOC_DEBUG\fR
 Same as \fB\-v, \-\-verbose\fR
@@ -1637,21 +1631,21 @@ Same as \fB\-\-delay\-boot\fR
 .TP
 \fBSALLOC_EXCLUSIVE\fR
 Same as \fB\-\-exclusive\fR
-.\ .TP
-.\ \fBSALLOC_GPUS\fR
-.\ Same as \fB\-G, \-\-gpus\fR
-.\ .TP
-.\ \fBSALLOC_GPU_BIND\fR
-.\ Same as \fB\-\-gpu\-bind\fR
-.\ .TP
-.\ \fBSALLOC_GPU_FREQ\fR
-.\ Same as \fB\-\-gpu\-freq\fR
-.\ .TP
-.\ \fBSALLOC_GPUS_PER_NODE\fR
-.\ Same as \fB\-\-gpus\-per\-node\fR
-.\ .TP
-.\ \fBSALLOC_GPUS_PER_TASK\fR
-.\ Same as \fB\-\-gpus\-per\-task\fR
+.TP
+\fBSALLOC_GPUS\fR
+Same as \fB\-G, \-\-gpus\fR
+.TP
+\fBSALLOC_GPU_BIND\fR
+Same as \fB\-\-gpu\-bind\fR
+.TP
+\fBSALLOC_GPU_FREQ\fR
+Same as \fB\-\-gpu\-freq\fR
+.TP
+\fBSALLOC_GPUS_PER_NODE\fR
+Same as \fB\-\-gpus\-per\-node\fR
+.TP
+\fBSALLOC_GPUS_PER_TASK\fR
+Same as \fB\-\-gpus\-per\-task\fR
 .TP
 \fBSALLOC_GRES_FLAGS\fR
 Same as \-\-gres\-flags\fR
@@ -1670,9 +1664,9 @@ Same as \fB\-K\fR, \fB\-\-kill\-command\fR
 .TP
 \fBSALLOC_MEM_BIND\fR
 Same as \fB\-\-mem\-bind\fR
-.\ .TP
-.\ \fBSALLOC_MEM_PER_GPU\fR
-.\ Same as \fB\-\-mem\-per\-gpu\fR
+.TP
+\fBSALLOC_MEM_PER_GPU\fR
+Same as \fB\-\-mem\-per\-gpu\fR
 .TP
 \fBSALLOC_NETWORK\fR
 Same as \fB\-\-network\fR
@@ -1758,10 +1752,10 @@ for each component.
 .TP
 \fBSLURM_CLUSTER_NAME\fR
 Name of the cluster on which the job is executing.
-.\ .TP
-.\ \fBSLURM_CPUS_PER_GPU\fR
-.\ Number of CPUs requested per allocated GPU.
-.\ Only set if the \fB\-\-cpus\-per\-gpu\fR option is specified.
+.TP
+\fBSLURM_CPUS_PER_GPU\fR
+Number of CPUs requested per allocated GPU.
+Only set if the \fB\-\-cpus\-per\-gpu\fR option is specified.
 .TP
 \fBSLURM_CPUS_PER_TASK\fR
 Number of CPUs requested per task.
@@ -1769,30 +1763,30 @@ Only set if the \fB\-\-cpus\-per\-task\fR option is specified.
 .TP
 \fBSLURM_DISTRIBUTION\fR
 Only set if the \fB\-m, \-\-distribution\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS\fR
-.\ Number of GPUs requested.
-.\ Only set if the \fB\-G, \-\-gpus\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPU_BIND\fR
-.\ Requested binding of tasks to GPU.
-.\ Only set if the \fB\-\-gpu\-bind\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPU_FREQ\fR
-.\ Requested GPU frequency and voltage.
-.\ Only set if the \fB\-\-gpu\-freq\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_NODE\fR
-.\ Requested GPU count per allocated node.
-.\ Only set if the \fB\-\-gpus\-per\-node\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_SOCKET\fR
-.\ Requested GPU count per allocated socket.
-.\ Only set if the \fB\-\-gpus\-per\-socket\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_TASK\fR
-.\ Requested GPU count per allocated task.
-.\ Only set if the \fB\-\-gpus\-per\-task\fR option is specified.
+.TP
+\fBSLURM_GPUS\fR
+Number of GPUs requested.
+Only set if the \fB\-G, \-\-gpus\fR option is specified.
+.TP
+\fBSLURM_GPU_BIND\fR
+Requested binding of tasks to GPU.
+Only set if the \fB\-\-gpu\-bind\fR option is specified.
+.TP
+\fBSLURM_GPU_FREQ\fR
+Requested GPU frequency and voltage.
+Only set if the \fB\-\-gpu\-freq\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_NODE\fR
+Requested GPU count per allocated node.
+Only set if the \fB\-\-gpus\-per\-node\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_SOCKET\fR
+Requested GPU count per allocated socket.
+Only set if the \fB\-\-gpus\-per\-socket\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_TASK\fR
+Requested GPU count per allocated task.
+Only set if the \fB\-\-gpus\-per\-task\fR option is specified.
 .TP
 \fBSLURM_JOB_ACCOUNT\fR
 Account name associated of the job allocation.
@@ -1845,10 +1839,10 @@ Set to "quiet" otherwise.
 .TP
 \fBSLURM_MEM_PER_CPU\fR
 Same as \fB\-\-mem\-per\-cpu\fR
-.\ .TP
-.\ \fBSLURM_MEM_PER_GPU\fR
-.\ Requested memory per allocated GPU.
-.\ Only set if the \fB\-\-mem\-per\-gpu\fR option is specified.
+.TP
+\fBSLURM_MEM_PER_GPU\fR
+Requested memory per allocated GPU.
+Only set if the \fB\-\-mem\-per\-gpu\fR option is specified.
 .TP
 \fBSLURM_MEM_PER_NODE\fR
 Same as \fB\-\-mem\fR
diff --git a/doc/man/man1/sbatch.1 b/doc/man/man1/sbatch.1
index 36336deb01b..cadf6e52a1b 100644
--- a/doc/man/man1/sbatch.1
+++ b/doc/man/man1/sbatch.1
@@ -419,13 +419,13 @@ Accounting is able to poll for job information. As a result not all of
 accounting information will be present.
 .RE
 
-.\ .TP
-.\ \fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
-.\ Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
-.\ allocated GPU.
-.\ Requires the \fB\-\-gpus\fR option.
-.\ Not compatible with the \fB\-\-cpus\-per\-task\fR option.
-.\ 
+.TP
+\fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
+Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
+allocated GPU.
+Requires the \fB\-\-gpus\fR option.
+Not compatible with the \fB\-\-cpus\-per\-task\fR option.
+
 .TP
 \fB\-c\fR, \fB\-\-cpus\-per\-task\fR=<\fIncpus\fR>
 Advise the Slurm controller that ensuing job steps will require \fIncpus\fR
@@ -607,121 +607,121 @@ If \fBsbatch\fR is run as root, and the \fB\-\-gid\fR option is used,
 submit the job with \fIgroup\fR's group access permissions.  \fIgroup\fR
 may be the group name or the numerical group ID.
 
-.\ .TP
-.\ \fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the total number of GPUs required for the job.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus=volta:3,kepler:1".
-.\ See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpu\-bind\fR=<\fItype\fR>
-.\ Bind tasks to specific GPUs.
-.\ By default every spawned task can access every GPU allocated to the job.
-.\ 
-.\ Supported \fItype\fR options:
-.\ .RS
-.\ .TP 10
-.\ \fBclosest\fR
-.\ Bind each task to the GPU(s) which are closest.
-.\ In a NUMA environment, each task may be bound to more than one GPU (i.e.
-.\ all GPUs in that NUMA environment).
-.\ .TP
-.\ \fBmap_gpu:<list>\fR
-.\ Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
-.\ <gpu_id_for_task_0>,<gpu_id_for_task_1>,... GPU IDs are interpreted as decimal
-.\ values unless they are preceded with '0x' in which case they interpreted as
-.\ hexadecimal values. If the number of tasks (or ranks) exceeds the number of
-.\ elements in this list, elements in the list will be reused as needed starting
-.\ from the beginning of the list. To simplify support for large task counts,
-.\ the lists may follow a map with an asterisk and repetition count.
-.\ For example "map_cpu:0*4,1*4".
-.\ Not supported unless the entire node is allocated to the job.
-.\ .TP
-.\ \fBmask_gpu:<list>\fR
-.\ Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
-.\ <gpu_mask_for_task_0>,<gpu_mask_for_task_1>,... The mapping is specified for
-.\ a node and identical mapping is applied to the tasks on every node (i.e. the
-.\ lowest task ID on each node is mapped to the first mask specified in the list,
-.\ etc.). GPU masks are always interpreted as hexadecimal values but can be
-.\ preceded with an optional '0x'. Not supported unless the entire node is
-.\ allocated to the job. To simplify support for large task counts, the lists
-.\ may follow a map with an asterisk and repetition count.
-.\ For example "mask_gpu:0x0f*4,0xf0*4".
-.\ Not supported unless the entire node is allocated to the job.
-.\ .RE
-.\ 
-.\ .TP
-.\ \fB\-\-gpu\-freq\fR=[<\fItype\fR]=\fIvalue\fR>[:<\fItype\fR=\fIvalue\fR>]
-.\ Request that GPUs allocated to the job are configured with specific voltage
-.\ and/or frequency values.
-.\ This option can be used to independently configure the GPU and its memory
-.\ frequencies.
-.\ In some cases, system power caps may override the requested values.
-.\ The field \fItype\fR can be "memory" or "voltage".
-.\ If \fItype\fR is not specified. the GPU frequency is implied.
-.\ The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
-.\ a numeric value (megahertz or volts).
-.\ If the specified numeric value is not possible, a value as close as
-.\ possible will be used. See below for definition of the values.
-.\ Examples of use include "\-\-gpu\-freq\fR=medium:memory=high:voltage=high" and
-.\ \-\-gpu\-freq\fR=450".
-.\ 
-.\ Supported \fIvalue\fR definitions:
-.\ .RS
-.\ .TP 10
-.\ \fBlow\fR
-.\ the lowest available frequency or voltage
-.\ .TP
-.\ \fBmedium\fR
-.\ attempts to set a frequency  or voltage in the middle of the available range
-.\ .TP
-.\ \fBhigh\fR
-.\ the highest available frequency or voltage
-.\ .TP
-.\ \fBhighm1\fR
-.\ (high minus one) will select the next highest available frequency or voltage
-.\ .RE
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-node\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each node included in
-.\ the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus\-per\-node=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-node=volta:3,kepler:1".
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-socket\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each socket included in
-.\ the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus\-per\-socket=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-socket=volta:3,kepler:1".
-.\ Requires job to specify a sockets per node count ( \-\-sockets\-per\-node).
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-node\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-task\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each task to be spawned
-.\ in the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ This option requires the specification of a task count.
-.\ For example "\-\-gpus\-per\-task=volta:1".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-task=volta:3,kepler:1".
-.\ Requires job to specify a task count (\-\-nodes).
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-node\fR options.
-.\ 
+.TP
+\fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the total number of GPUs required for the job.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus=volta:3,kepler:1".
+See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpu\-bind\fR=<\fItype\fR>
+Bind tasks to specific GPUs.
+By default every spawned task can access every GPU allocated to the job.
+
+Supported \fItype\fR options:
+.RS
+.TP 10
+\fBclosest\fR
+Bind each task to the GPU(s) which are closest.
+In a NUMA environment, each task may be bound to more than one GPU (i.e.
+all GPUs in that NUMA environment).
+.TP
+\fBmap_gpu:<list>\fR
+Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
+<gpu_id_for_task_0>,<gpu_id_for_task_1>,... GPU IDs are interpreted as decimal
+values unless they are preceded with '0x' in which case they interpreted as
+hexadecimal values. If the number of tasks (or ranks) exceeds the number of
+elements in this list, elements in the list will be reused as needed starting
+from the beginning of the list. To simplify support for large task counts,
+the lists may follow a map with an asterisk and repetition count.
+For example "map_cpu:0*4,1*4".
+Not supported unless the entire node is allocated to the job.
+.TP
+\fBmask_gpu:<list>\fR
+Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
+<gpu_mask_for_task_0>,<gpu_mask_for_task_1>,... The mapping is specified for
+a node and identical mapping is applied to the tasks on every node (i.e. the
+lowest task ID on each node is mapped to the first mask specified in the list,
+etc.). GPU masks are always interpreted as hexadecimal values but can be
+preceded with an optional '0x'. Not supported unless the entire node is
+allocated to the job. To simplify support for large task counts, the lists
+may follow a map with an asterisk and repetition count.
+For example "mask_gpu:0x0f*4,0xf0*4".
+Not supported unless the entire node is allocated to the job.
+.RE
+
+.TP
+\fB\-\-gpu\-freq\fR=[<\fItype\fR]=\fIvalue\fR>[:<\fItype\fR=\fIvalue\fR>]
+Request that GPUs allocated to the job are configured with specific voltage
+and/or frequency values.
+This option can be used to independently configure the GPU and its memory
+frequencies.
+In some cases, system power caps may override the requested values.
+The field \fItype\fR can be "memory" or "voltage".
+If \fItype\fR is not specified. the GPU frequency is implied.
+The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
+a numeric value (megahertz or volts).
+If the specified numeric value is not possible, a value as close as
+possible will be used. See below for definition of the values.
+Examples of use include "\-\-gpu\-freq\fR=medium:memory=high:voltage=high" and
+\-\-gpu\-freq\fR=450".
+
+Supported \fIvalue\fR definitions:
+.RS
+.TP 10
+\fBlow\fR
+the lowest available frequency or voltage
+.TP
+\fBmedium\fR
+attempts to set a frequency  or voltage in the middle of the available range
+.TP
+\fBhigh\fR
+the highest available frequency or voltage
+.TP
+\fBhighm1\fR
+(high minus one) will select the next highest available frequency or voltage
+.RE
+
+.TP
+\fB\-\-gpus\-per\-node\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each node included in
+the job's resource allocation.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus\-per\-node=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-node=volta:3,kepler:1".
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpus\-per\-socket\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each socket included in
+the job's resource allocation.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus\-per\-socket=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-socket=volta:3,kepler:1".
+Requires job to specify a sockets per node count ( \-\-sockets\-per\-node).
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-node\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpus\-per\-task\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each task to be spawned
+in the job's resource allocation.
+An optional GPU type specification can be supplied.
+This option requires the specification of a task count.
+For example "\-\-gpus\-per\-task=volta:1".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-task=volta:3,kepler:1".
+Requires job to specify a task count (\-\-nodes).
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-node\fR options.
+
 .TP
 \fB\-\-gres\fR=<\fIlist\fR>
 Specifies a comma delimited list of generic consumable resources.
@@ -996,12 +996,9 @@ Default value is \fBDefMemPerNode\fR and the maximum value is
 seen using the \fBscontrol show config\fR command.
 This parameter would generally be used if whole nodes
 are allocated to jobs (\fBSelectType=select/linear\fR).
-Also see \fB\-\-mem\-per\-cpu\fR.
-\fB\-\-mem\fR and \fB\-\-mem\-per\-cpu\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
+Also see fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR.
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
 
 NOTE: A memory size specification of zero is treated as a special case and
 grants the job access to all of the memory on each node.
@@ -1035,27 +1032,24 @@ are allocated to jobs (\fBSelectType=select/cons_res\fR).
 If resources are allocated by the core, socket or whole nodes; the number
 of CPUs allocated to a job may be higher than the task count and the value
 of \fB\-\-mem\-per\-cpu\fR should be adjusted accordingly.
+Also see \fB\-\-mem\fR and fB\-\-mem\-per\-gpu\fR.
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
+
+.TP
+\fB\-\-mem\-per\-gpu\fR=<\fIsize[units]\fR>
+Minimum memory required per allocated GPU.
+Default units are megabytes unless the SchedulerParameters configuration
+parameter includes the "default_gbytes" option for gigabytes.
+Different units can be specified using the suffix [K|M|G|T].
+Default value is \fBDefMemPerGPU\fR and is available on both a global and
+per partition basis.
+If configured, the parameters can be seen using the \fBscontrol show config\fR
+and \fBscontrol show partition\fR commands.
 Also see \fB\-\-mem\fR.
-\fB\-\-mem\fR and \fB\-\-mem\-per\-cpu\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see \fB\-\-mem\fR and fB\-\-mem\-per\-gpu\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
-
-.\ .TP
-.\ \fB\-\-mem\-per\-gpu\fR=<\fIsize[units]\fR>
-.\ Minimum memory required per allocated GPU.
-.\ Default units are megabytes unless the SchedulerParameters configuration
-.\ parameter includes the "default_gbytes" option for gigabytes.
-.\ Different units can be specified using the suffix [K|M|G|T].
-.\ Default value is \fBDefMemPerGPU\fR and is available on both a global and
-.\ per partition basis.
-.\ If configured, the parameters can be seen using the \fBscontrol show config\fR
-.\ and \fBscontrol show partition\fR commands.
-.\ Also see \fB\-\-mem\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
-.\ 
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
+
 .TP
 \fB\-\-mem\-bind\fR=[{\fIquiet,verbose\fR},]\fItype\fR
 Bind tasks to memory. Used only when the task/affinity plugin is enabled
@@ -1878,9 +1872,9 @@ Same as \fB\-C\fR, \fB\-\-constraint\fR
 .TP
 \fBSBATCH_CORE_SPEC\fR
 Same as \fB\-\-core\-spec\fR
-.\ .TP
-.\ \fBSBATCH_CPUS_PER_GPU\fR
-.\ Same as \fB\-\-cpus\-per\-gpu\fR
+.TP
+\fBSBATCH_CPUS_PER_GPU\fR
+Same as \fB\-\-cpus\-per\-gpu\fR
 .TP
 \fBSBATCH_DEBUG\fR
 Same as \fB\-v, \-\-verbose\fR
@@ -1899,21 +1893,21 @@ Same as \fB\-\-export\fR
 .TP
 \fBSBATCH_GET_USER_ENV\fR
 Same as \fB\-\-get\-user\-env\fR
-.\ .TP
-.\ \fBSBATCH_GPUS\fR
-.\ Same as \fB\-G, \-\-gpus\fR
-.\ .TP
-.\ \fBSBATCH_GPU_BIND\fR
-.\ Same as \fB\-\-gpu\-bind\fR
-.\ .TP
-.\ \fBSBATCH_GPU_FREQ\fR
-.\ Same as \fB\-\-gpu\-freq\fR
-.\ .TP
-.\ \fBSBATCH_GPUS_PER_NODE\fR
-.\ Same as \fB\-\-gpus\-per\-node\fR
-.\ .TP
-.\ \fBSBATCH_GPUS_PER_TASK\fR
-.\ Same as \fB\-\-gpus\-per\-task\fR
+.TP
+\fBSBATCH_GPUS\fR
+Same as \fB\-G, \-\-gpus\fR
+.TP
+\fBSBATCH_GPU_BIND\fR
+Same as \fB\-\-gpu\-bind\fR
+.TP
+\fBSBATCH_GPU_FREQ\fR
+Same as \fB\-\-gpu\-freq\fR
+.TP
+\fBSBATCH_GPUS_PER_NODE\fR
+Same as \fB\-\-gpus\-per\-node\fR
+.TP
+\fBSBATCH_GPUS_PER_TASK\fR
+Same as \fB\-\-gpus\-per\-task\fR
 .TP
 \fBSBATCH_GRES_FLAGS\fR
 Same as \-\-gres\-flags\fR
@@ -1932,9 +1926,9 @@ Same as \fB\-J, \-\-job\-name\fR
 .TP
 \fBSBATCH_MEM_BIND\fR
 Same as \fB\-\-mem\-bind\fR
-.\ .TP
-.\ \fBSBATCH_MEM_PER_GPU\fR
-.\ Same as \fB\-\-mem\-per\-gpu\fR
+.TP
+\fBSBATCH_MEM_PER_GPU\fR
+Same as \fB\-\-mem\-per\-gpu\fR
 .TP
 \fBSBATCH_NETWORK\fR
 Same as \fB\-\-network\fR
@@ -2067,10 +2061,10 @@ Name of the cluster on which the job is executing.
 .TP
 \fBSLURM_CPUS_ON_NODE\fR
 Number of CPUS on the allocated node.
-.\ .TP
-.\ \fBSLURM_CPUS_PER_GPU\fR
-.\ Number of CPUs requested per allocated GPU.
-.\ Only set if the \fB\-\-cpus\-per\-gpu\fR option is specified.
+.TP
+\fBSLURM_CPUS_PER_GPU\fR
+Number of CPUs requested per allocated GPU.
+Only set if the \fB\-\-cpus\-per\-gpu\fR option is specified.
 .TP
 \fBSLURM_CPUS_PER_TASK\fR
 Number of cpus requested per task.
@@ -2078,30 +2072,30 @@ Only set if the \fB\-\-cpus\-per\-task\fR option is specified.
 .TP
 \fBSLURM_DISTRIBUTION\fR
 Same as \fB\-m, \-\-distribution\fR
-.\ .TP
-.\ \fBSLURM_GPUS\fR
-.\ Number of GPUs requested.
-.\ Only set if the \fB\-G, \-\-gpus\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPU_BIND\fR
-.\ Requested binding of tasks to GPU.
-.\ Only set if the \fB\-\-gpu\-bind\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPU_FREQ\fR
-.\ Requested GPU frequency and voltage.
-.\ Only set if the \fB\-\-gpu\-freq\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_NODE\fR
-.\ Requested GPU count per allocated node.
-.\ Only set if the \fB\-\-gpus\-per\-node\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_SOCKET\fR
-.\ Requested GPU count per allocated socket.
-.\ Only set if the \fB\-\-gpus\-per\-socket\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_TASK\fR
-.\ Requested GPU count per allocated task.
-.\ Only set if the \fB\-\-gpus\-per\-task\fR option is specified.
+.TP
+\fBSLURM_GPUS\fR
+Number of GPUs requested.
+Only set if the \fB\-G, \-\-gpus\fR option is specified.
+.TP
+\fBSLURM_GPU_BIND\fR
+Requested binding of tasks to GPU.
+Only set if the \fB\-\-gpu\-bind\fR option is specified.
+.TP
+\fBSLURM_GPU_FREQ\fR
+Requested GPU frequency and voltage.
+Only set if the \fB\-\-gpu\-freq\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_NODE\fR
+Requested GPU count per allocated node.
+Only set if the \fB\-\-gpus\-per\-node\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_SOCKET\fR
+Requested GPU count per allocated socket.
+Only set if the \fB\-\-gpus\-per\-socket\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_TASK\fR
+Requested GPU count per allocated task.
+Only set if the \fB\-\-gpus\-per\-task\fR option is specified.
 .TP
 \fBSLURM_GTIDS\fR
 Global task IDs running on this node.  Zero  origin and comma separated.
@@ -2146,10 +2140,10 @@ Node local task ID for the process within a job.
 .TP
 \fBSLURM_MEM_PER_CPU\fR
 Same as \fB\-\-mem\-per\-cpu\fR
-.\ .TP
-.\ \fBSLURM_MEM_PER_GPU\fR
-.\ Requested memory per allocated GPU.
-.\ Only set if the \fB\-\-mem\-per\-gpu\fR option is specified.
+.TP
+\fBSLURM_MEM_PER_GPU\fR
+Requested memory per allocated GPU.
+Only set if the \fB\-\-mem\-per\-gpu\fR option is specified.
 .TP
 \fBSLURM_MEM_PER_NODE\fR
 Same as \fB\-\-mem\fR
diff --git a/doc/man/man1/scontrol.1 b/doc/man/man1/scontrol.1
index 332b9507755..f289aca61f8 100644
--- a/doc/man/man1/scontrol.1
+++ b/doc/man/man1/scontrol.1
@@ -1380,19 +1380,19 @@ Hidden partitions will by default not be reported by Slurm APIs
 or commands.
 Possible values are "YES" and "NO".
 
-.\ .TP
-.\ \fIJobDefaults\fP=<specs>
-.\ Specify job default values using a comma delimited list of "key=value" pairs.
-.\ Supported keys include
-.\ .RS
-.\ .TP 14
-.\ \fIDefCpuPerGPU\fP
-.\ Default number of CPUs per allocated GPU.
-.\ .TP
-.\ \fIDefMemPerGPU\fP
-.\ Default memory limit (in megabytes) per allocated GPU.
-.\ .RE
-.\ 
+.TP
+\fIJobDefaults\fP=<specs>
+Specify job default values using a comma delimited list of "key=value" pairs.
+Supported keys include
+.RS
+.TP 14
+\fIDefCpuPerGPU\fP
+Default number of CPUs per allocated GPU.
+.TP
+\fIDefMemPerGPU\fP
+Default memory limit (in megabytes) per allocated GPU.
+.RE
+
 .TP
 \fIMaxMemPerCPU\fP=<MB>
 Set the maximum memory to be allocated per CPU for jobs in this partition.
diff --git a/doc/man/man1/srun.1 b/doc/man/man1/srun.1
index ce98dee69cd..e0f873b8c3c 100644
--- a/doc/man/man1/srun.1
+++ b/doc/man/man1/srun.1
@@ -579,13 +579,13 @@ accounting information will be present.
 This option applies to job and step allocations.
 .RE
 
-.\ .TP
-.\ \fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
-.\ Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
-.\ allocated GPU.
-.\ Requires the \fB\-\-gpus\fR option.
-.\ Not compatible with the \fB\-\-cpus\-per\-task\fR option.
-.\ 
+.TP
+\fB\-\-cpus\-per\-gpu\fR=<\fIncpus\fR>
+Advise Slurm that ensuing job steps will require \fIncpus\fR processors per
+allocated GPU.
+Requires the \fB\-\-gpus\fR option.
+Not compatible with the \fB\-\-cpus\-per\-task\fR option.
+
 .TP
 \fB\-c\fR, \fB\-\-cpus\-per\-task\fR=<\fIncpus\fR>
 Request that \fIncpus\fR be allocated \fBper process\fR. This may be
@@ -792,121 +792,121 @@ submit the job with \fIgroup\fR's group access permissions.  \fIgroup\fR
 may be the group name or the numerical group ID. This option applies to
 job allocations.
 
-.\ .TP
-.\ \fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the total number of GPUs required for the job.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus=volta:3,kepler:1".
-.\ See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpu\-bind\fR=<\fItype\fR>
-.\ Bind tasks to specific GPUs.
-.\ By default every spawned task can access every GPU allocated to the job.
-.\ 
-.\ Supported \fItype\fR options:
-.\ .RS
-.\ .TP 10
-.\ \fBclosest\fR
-.\ Bind each task to the GPU(s) which are closest.
-.\ In a NUMA environment, each task may be bound to more than one GPU (i.e.
-.\ all GPUs in that NUMA environment).
-.\ .TP
-.\ \fBmap_gpu:<list>\fR
-.\ Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
-.\ <gpu_id_for_task_0>,<gpu_id_for_task_1>,... GPU IDs are interpreted as decimal
-.\ values unless they are preceded with '0x' in which case they interpreted as
-.\ hexadecimal values. If the number of tasks (or ranks) exceeds the number of
-.\ elements in this list, elements in the list will be reused as needed starting
-.\ from the beginning of the list. To simplify support for large task counts,
-.\ the lists may follow a map with an asterisk and repetition count.
-.\ For example "map_cpu:0*4,1*4".
-.\ Not supported unless the entire node is allocated to the job.
-.\ .TP
-.\ \fBmask_gpu:<list>\fR
-.\ Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
-.\ <gpu_mask_for_task_0>,<gpu_mask_for_task_1>,... The mapping is specified for
-.\ a node and identical mapping is applied to the tasks on every node (i.e. the
-.\ lowest task ID on each node is mapped to the first mask specified in the list,
-.\ etc.). GPU masks are always interpreted as hexadecimal values but can be
-.\ preceded with an optional '0x'. Not supported unless the entire node is
-.\ allocated to the job. To simplify support for large task counts, the lists
-.\ may follow a map with an asterisk and repetition count.
-.\ For example "mask_gpu:0x0f*4,0xf0*4".
-.\ Not supported unless the entire node is allocated to the job.
-.\ .RE
-.\ 
-.\ .TP
-.\ \fB\-\-gpu\-freq\fR=[<\fItype\fR]=\fIvalue\fR>[:<\fItype\fR=\fIvalue\fR>]
-.\ Request that GPUs allocated to the job are configured with specific voltage
-.\ and/or frequency values.
-.\ This option can be used to independently configure the GPU and its memory
-.\ frequencies.
-.\ In some cases, system power caps may override the requested values.
-.\ The field \fItype\fR can be "memory" or "voltage".
-.\ If \fItype\fR is not specified. the GPU frequency is implied.
-.\ The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
-.\ a numeric value (megahertz or volts).
-.\ If the specified numeric value is not possible, a value as close as
-.\ possible will be used. See below for definition of the values.
-.\ Examples of use include "\-\-gpu\-freq\fR=medium:memory=high:voltage=high" and
-.\ \-\-gpu\-freq\fR=450".
-.\ 
-.\ Supported \fIvalue\fR definitions:
-.\ .RS
-.\ .TP 10
-.\ \fBlow\fR
-.\ the lowest available frequency or voltage
-.\ .TP
-.\ \fBmedium\fR
-.\ attempts to set a frequency  or voltage in the middle of the available range
-.\ .TP
-.\ \fBhigh\fR
-.\ the highest available frequency or voltage
-.\ .TP
-.\ \fBhighm1\fR
-.\ (high minus one) will select the next highest available frequency or voltage
-.\ .RE
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-node\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each node included in
-.\ the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus\-per\-node=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-node=volta:3,kepler:1".
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-socket\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each socket included in
-.\ the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ For example "\-\-gpus\-per\-socket=volta:3".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-socket=volta:3,kepler:1".
-.\ Requires job to specify a sockets per node count ( \-\-sockets\-per\-node).
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-node\fR and
-.\ \fB\-\-gpus\-per\-task\fR options.
-.\ 
-.\ .TP
-.\ \fB\-\-gpus\-per\-task\fR=[<\fitype\fR>:]<\fInumber\fR>
-.\ Specify the number of GPUs required for the job on each task to be spawned
-.\ in the job's resource allocation.
-.\ An optional GPU type specification can be supplied.
-.\ This option requires the specification of a task count.
-.\ For example "\-\-gpus\-per\-task=volta:1".
-.\ Multiple options can be requested in a comma separated list, for example:
-.\ "\-\-gpus\-per\-task=volta:3,kepler:1".
-.\ Requires job to specify a task count (\-\-nodes).
-.\ See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
-.\ \fB\-\-gpus\-per\-node\fR options.
-.\ 
+.TP
+\fB\-G\fR, \fB\-\-gpus\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the total number of GPUs required for the job.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus=volta:3,kepler:1".
+See also the \fB\-\-gpus\-per\-node\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpu\-bind\fR=<\fItype\fR>
+Bind tasks to specific GPUs.
+By default every spawned task can access every GPU allocated to the job.
+
+Supported \fItype\fR options:
+.RS
+.TP 10
+\fBclosest\fR
+Bind each task to the GPU(s) which are closest.
+In a NUMA environment, each task may be bound to more than one GPU (i.e.
+all GPUs in that NUMA environment).
+.TP
+\fBmap_gpu:<list>\fR
+Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
+<gpu_id_for_task_0>,<gpu_id_for_task_1>,... GPU IDs are interpreted as decimal
+values unless they are preceded with '0x' in which case they interpreted as
+hexadecimal values. If the number of tasks (or ranks) exceeds the number of
+elements in this list, elements in the list will be reused as needed starting
+from the beginning of the list. To simplify support for large task counts,
+the lists may follow a map with an asterisk and repetition count.
+For example "map_cpu:0*4,1*4".
+Not supported unless the entire node is allocated to the job.
+.TP
+\fBmask_gpu:<list>\fR
+Bind by setting GPU masks on tasks (or ranks) as specified where <list> is
+<gpu_mask_for_task_0>,<gpu_mask_for_task_1>,... The mapping is specified for
+a node and identical mapping is applied to the tasks on every node (i.e. the
+lowest task ID on each node is mapped to the first mask specified in the list,
+etc.). GPU masks are always interpreted as hexadecimal values but can be
+preceded with an optional '0x'. Not supported unless the entire node is
+allocated to the job. To simplify support for large task counts, the lists
+may follow a map with an asterisk and repetition count.
+For example "mask_gpu:0x0f*4,0xf0*4".
+Not supported unless the entire node is allocated to the job.
+.RE
+
+.TP
+\fB\-\-gpu\-freq\fR=[<\fItype\fR]=\fIvalue\fR>[:<\fItype\fR=\fIvalue\fR>]
+Request that GPUs allocated to the job are configured with specific voltage
+and/or frequency values.
+This option can be used to independently configure the GPU and its memory
+frequencies.
+In some cases, system power caps may override the requested values.
+The field \fItype\fR can be "memory" or "voltage".
+If \fItype\fR is not specified. the GPU frequency is implied.
+The \fIvalue\fR field can either be "low", "medium", "high", "highm1" or
+a numeric value (megahertz or volts).
+If the specified numeric value is not possible, a value as close as
+possible will be used. See below for definition of the values.
+Examples of use include "\-\-gpu\-freq\fR=medium:memory=high:voltage=high" and
+\-\-gpu\-freq\fR=450".
+
+Supported \fIvalue\fR definitions:
+.RS
+.TP 10
+\fBlow\fR
+the lowest available frequency or voltage
+.TP
+\fBmedium\fR
+attempts to set a frequency  or voltage in the middle of the available range
+.TP
+\fBhigh\fR
+the highest available frequency or voltage
+.TP
+\fBhighm1\fR
+(high minus one) will select the next highest available frequency or voltage
+.RE
+
+.TP
+\fB\-\-gpus\-per\-node\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each node included in
+the job's resource allocation.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus\-per\-node=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-node=volta:3,kepler:1".
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpus\-per\-socket\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each socket included in
+the job's resource allocation.
+An optional GPU type specification can be supplied.
+For example "\-\-gpus\-per\-socket=volta:3".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-socket=volta:3,kepler:1".
+Requires job to specify a sockets per node count ( \-\-sockets\-per\-node).
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-node\fR and
+\fB\-\-gpus\-per\-task\fR options.
+
+.TP
+\fB\-\-gpus\-per\-task\fR=[<\fitype\fR>:]<\fInumber\fR>
+Specify the number of GPUs required for the job on each task to be spawned
+in the job's resource allocation.
+An optional GPU type specification can be supplied.
+This option requires the specification of a task count.
+For example "\-\-gpus\-per\-task=volta:1".
+Multiple options can be requested in a comma separated list, for example:
+"\-\-gpus\-per\-task=volta:3,kepler:1".
+Requires job to specify a task count (\-\-nodes).
+See also the \fB\-\-gpus\fR, \fB\-\-gpus\-per\-socket\fR and
+\fB\-\-gpus\-per\-node\fR options.
+
 .TP
 \fB\-\-gres\fR=<\fIlist\fR>
 Specifies a comma delimited list of generic consumable resources.
@@ -1287,12 +1287,9 @@ are allocated to jobs (\fBSelectType=select/linear\fR).
 Specifying a memory limit of zero for a job step will restrict the job step
 to the amount of memory allocated to the job, but not remove any of the job's
 memory allocation from being available to other job steps.
-Also see \fB\-\-mem\-per\-cpu\fR.
-\fB\-\-mem\fR and \fB\-\-mem\-per\-cpu\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
+Also see fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR.
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
 
 NOTE: A memory size specification of zero is treated as a special case and
 grants the job access to all of the memory on each node.
@@ -1332,28 +1329,24 @@ of \fB\-\-mem\-per\-cpu\fR should be adjusted accordingly.
 Specifying a memory limit of zero for a job step will restrict the job step
 to the amount of memory allocated to the job, but not remove any of the job's
 memory allocation from being available to other job steps.
+Also see \fB\-\-mem\fR and fB\-\-mem\-per\-gpu\fR.
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
+
+.TP
+\fB\-\-mem\-per\-gpu\fR=<\fIsize[units]\fR>
+Minimum memory required per allocated GPU.
+Default units are megabytes unless the SchedulerParameters configuration
+parameter includes the "default_gbytes" option for gigabytes.
+Different units can be specified using the suffix [K|M|G|T].
+Default value is \fBDefMemPerGPU\fR and is available on both a global and
+per partition basis.
+If configured, the parameters can be seen using the \fBscontrol show config\fR
+and \fBscontrol show partition\fR commands.
 Also see \fB\-\-mem\fR.
-\fB\-\-mem\fR and \fB\-\-mem\-per\-cpu\fR are mutually exclusive. This option
-applies to job and step allocations.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see \fB\-\-mem\fR and fB\-\-mem\-per\-gpu\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
-
-.\ .TP
-.\ \fB\-\-mem\-per\-gpu\fR=<\fIsize[units]\fR>
-.\ Minimum memory required per allocated GPU.
-.\ Default units are megabytes unless the SchedulerParameters configuration
-.\ parameter includes the "default_gbytes" option for gigabytes.
-.\ Different units can be specified using the suffix [K|M|G|T].
-.\ Default value is \fBDefMemPerGPU\fR and is available on both a global and
-.\ per partition basis.
-.\ If configured, the parameters can be seen using the \fBscontrol show config\fR
-.\ and \fBscontrol show partition\fR commands.
-.\ Also see \fB\-\-mem\fR.
-.\ The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
-.\ options are mutually exclusive.
-.\ 
+The \fB\-\-mem\fR, \fB\-\-mem\-per\-cpu\fR and fB\-\-mem\-per\-gpu\fR
+options are mutually exclusive.
+
 .TP
 \fB\-\-mem\-bind\fR=[{\fIquiet,verbose\fR},]\fItype\fR
 Bind tasks to memory. Used only when the task/affinity plugin is enabled
@@ -2531,9 +2524,9 @@ Same as \fB\-\-cpu\-bind\fR
 .TP
 \fBSLURM_CPU_FREQ_REQ\fR
 Same as \fB\-\-cpu\-freq\fR.
-.\ .TP
-.\ \fBSLURM_CPUS_PER_GPU\fR
-.\ Same as \fB\-\-cpus\-per\-gpu\fR
+.TP
+\fBSLURM_CPUS_PER_GPU\fR
+Same as \fB\-\-cpus\-per\-gpu\fR
 .TP
 \fBSLURM_CPUS_PER_TASK\fR
 Same as \fB\-c, \-\-cpus\-per\-task\fR
@@ -2578,21 +2571,21 @@ is used and resources are not currently available.
 This can be used by a script to distinguish application exit codes from
 various Slurm error conditions.
 Also see \fBSLURM_EXIT_ERROR\fR.
-.\ .TP
-.\ \fBSLURM_GPUS\fR
-.\ Same as \fB\-G, \-\-gpus\fR
-.\ .TP
-.\ \fBSLURM_GPU_BIND\fR
-.\ Same as \fB\-\-gpu\-bind\fR
-.\ .TP
-.\ \fBSLURM_GPU_FREQ\fR
-.\ Same as \fB\-\-gpu\-freq\fR
-.\ .TP
-.\ \fBSLURM_GPUS_PER_NODE\fR
-.\ Same as \fB\-\-gpus\-per\-node\fR
-.\ .TP
-.\ \fBSLURM_GPUS_PER_TASK\fR
-.\ Same as \fB\-\-gpus\-per\-task\fR
+.TP
+\fBSLURM_GPUS\fR
+Same as \fB\-G, \-\-gpus\fR
+.TP
+\fBSLURM_GPU_BIND\fR
+Same as \fB\-\-gpu\-bind\fR
+.TP
+\fBSLURM_GPU_FREQ\fR
+Same as \fB\-\-gpu\-freq\fR
+.TP
+\fBSLURM_GPUS_PER_NODE\fR
+Same as \fB\-\-gpus\-per\-node\fR
+.TP
+\fBSLURM_GPUS_PER_TASK\fR
+Same as \fB\-\-gpus\-per\-task\fR
 .TP
 \fBSLURM_GRES_FLAGS\fR
 Same as \-\-gres\-flags\fR
@@ -2629,9 +2622,9 @@ Same as \fB\-\-mem\-bind\fR
 .TP
 \fBSLURM_MEM_PER_CPU\fR
 Same as \fB\-\-mem\-per\-cpu\fR
-.\ .TP
-.\ \fBSLURM_MEM_PER_GPU\fR
-.\ Same as \fB\-\-mem\-per\-gpu\fR
+.TP
+\fBSLURM_MEM_PER_GPU\fR
+Same as \fB\-\-mem\-per\-gpu\fR
 .TP
 \fBSLURM_MEM_PER_NODE\fR
 Same as \fB\-\-mem\fR
@@ -2808,10 +2801,10 @@ Note the select/linear plugin allocates entire nodes to
 jobs, so the value indicates the total count of CPUs on the node.
 For the select/cons_res plugin, this number indicates the number of cores
 on this node allocated to the job.
-.\ .TP
-.\ \fBSLURM_CPUS_PER_GPU\fR
-.\ Number of CPUs requested per allocated GPU.
-.\ Only set if the \fB\-\-cpus\-per\-gpu\fR option is specified.
+.TP
+\fBSLURM_CPUS_PER_GPU\fR
+Number of CPUs requested per allocated GPU.
+Only set if the \fB\-\-cpus\-per\-gpu\fR option is specified.
 .TP
 \fBSLURM_CPUS_PER_TASK\fR
 Number of cpus requested per task.
@@ -2820,30 +2813,30 @@ Only set if the \fB\-\-cpus\-per\-task\fR option is specified.
 \fBSLURM_DISTRIBUTION\fR
 Distribution type for the allocated jobs. Set the distribution
 with \-m, \-\-distribution.
-.\ .TP
-.\ \fBSLURM_GPUS\fR
-.\ Number of GPUs requested.
-.\ Only set if the \fB\-G, \-\-gpus\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPU_BIND\fR
-.\ Requested binding of tasks to GPU.
-.\ Only set if the \fB\-\-gpu\-bind\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPU_FREQ\fR
-.\ Requested GPU frequency and voltage.
-.\ Only set if the \fB\-\-gpu\-freq\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_NODE\fR
-.\ Requested GPU count per allocated node.
-.\ Only set if the \fB\-\-gpus\-per\-node\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_SOCKET\fR
-.\ Requested GPU count per allocated socket.
-.\ Only set if the \fB\-\-gpus\-per\-socket\fR option is specified.
-.\ .TP
-.\ \fBSLURM_GPUS_PER_TASK\fR
-.\ Requested GPU count per allocated task.
-.\ Only set if the \fB\-\-gpus\-per\-task\fR option is specified.
+.TP
+\fBSLURM_GPUS\fR
+Number of GPUs requested.
+Only set if the \fB\-G, \-\-gpus\fR option is specified.
+.TP
+\fBSLURM_GPU_BIND\fR
+Requested binding of tasks to GPU.
+Only set if the \fB\-\-gpu\-bind\fR option is specified.
+.TP
+\fBSLURM_GPU_FREQ\fR
+Requested GPU frequency and voltage.
+Only set if the \fB\-\-gpu\-freq\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_NODE\fR
+Requested GPU count per allocated node.
+Only set if the \fB\-\-gpus\-per\-node\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_SOCKET\fR
+Requested GPU count per allocated socket.
+Only set if the \fB\-\-gpus\-per\-socket\fR option is specified.
+.TP
+\fBSLURM_GPUS_PER_TASK\fR
+Requested GPU count per allocated task.
+Only set if the \fB\-\-gpus\-per\-task\fR option is specified.
 .TP
 \fBSLURM_GTIDS\fR
 Global task IDs running on this node.
@@ -2901,10 +2894,10 @@ Sort free cache pages (run zonesort on Intel KNL nodes).
 .TP
 \fBSLURM_MEM_BIND_VERBOSE\fR
 \-\-mem\-bind verbosity (quiet,verbose).
-.\ .TP
-.\ \fBSLURM_MEM_PER_GPU\fR
-.\ Requested memory per allocated GPU.
-.\ Only set if the \fB\-\-mem\-per\-gpu\fR option is specified.
+.TP
+\fBSLURM_MEM_PER_GPU\fR
+Requested memory per allocated GPU.
+Only set if the \fB\-\-mem\-per\-gpu\fR option is specified.
 .TP
 \fBSLURM_JOB_NODES\fR
 Total number of nodes in the job's resource allocation.
diff --git a/doc/man/man5/gres.conf.5 b/doc/man/man5/gres.conf.5
index 7b614556c5d..aef6ddc57eb 100644
--- a/doc/man/man5/gres.conf.5
+++ b/doc/man/man5/gres.conf.5
@@ -88,14 +88,14 @@ of each specific resource on each node. Otherwise Slurm will only track a
 count of allocated resources rather than the state of each individual device
 file.
 
-.\ .TP
-.\ \fBLinks\fR
-.\ If multiple devices should be co-scheduled whenever possible, then specify a
-.\ link number on both devices that have the same value.
-.\ Devices may have multiple links and those numberic values should be
-.\ identified in a comma separated list.
-.\ A typical use case would be to identify GPUs having LNLink connectivity.
-.\ This is an optional value.
+.TP
+\fBLinks\fR
+If multiple devices should be co-scheduled whenever possible, then specify a
+link number on both devices that have the same value.
+Devices may have multiple links and those numberic values should be
+identified in a comma separated list.
+A typical use case would be to identify GPUs having LNLink connectivity.
+This is an optional value.
 .\
 .TP
 \fBName\fR
@@ -164,9 +164,9 @@ Name=bandwidth Count=20M
 \. .br
 \. Name=gpu Type=tesla File=/dev/nvidia1 COREs=0,1 Links=0
 \. .br
-.\ Name=gpu Type=tesla  File=/dev/nvidia2 COREs=2,3 Links=1
-.\ .br
-.\ Name=gpu Type=tesla  File=/dev/nvidia3 COREs=2,3 Links=1
+Name=gpu Type=tesla  File=/dev/nvidia2 COREs=2,3 Links=1
+.br
+Name=gpu Type=tesla  File=/dev/nvidia3 COREs=2,3 Links=1
 \. 
 .LP
 .br
diff --git a/doc/man/man5/slurm.conf.5 b/doc/man/man5/slurm.conf.5
index 668f8ec2751..6b007b47023 100644
--- a/doc/man/man5/slurm.conf.5
+++ b/doc/man/man5/slurm.conf.5
@@ -597,34 +597,29 @@ including state, job ids and allocated nodes counter.
 Slurmctld triggers
 .RE
 
-.\ .TP
-.\ \fBDefCpusPerGPU\fR
-.\ Default count of CPUs allocated per allocated GPU.
-.\ 
+.TP
+\fBDefCpusPerGPU\fR
+Default count of CPUs allocated per allocated GPU.
+
 .TP
 \fBDefMemPerCPU\fR
 Default real memory size available per allocated CPU in megabytes.
 Used to avoid over\-subscribing memory and causing paging.
 \fBDefMemPerCPU\fR would generally be used if individual processors
-are allocated to jobs (\fBSelectType=select/cons_res\fR).
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ are allocated to jobs (\fBSelectType=select/cons_res\fR or
-.\ \fBSelectType=select/cons_tres\fR).
+are allocated to jobs (\fBSelectType=select/cons_res\fR or
+\fBSelectType=select/cons_tres\fR).
 The default value is 0 (unlimited).
-Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
-\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
-.\ \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
-.\ mutually exclusive.
+Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
+\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
+mutually exclusive.
 
-.\ .TP
-.\ \fBDefMemPerGPU\fR
-.\ Default real memory size available per allocated GPU in megabytes.
-.\ The default value is 0 (unlimited).
-.\ Also see \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR.
-.\ \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
-.\ mutually exclusive.
+.TP
+\fBDefMemPerGPU\fR
+Default real memory size available per allocated GPU in megabytes.
+The default value is 0 (unlimited).
+Also see \fBDefMemPerCPU\fR and \fBDefMemPerNode\fR.
+\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
+mutually exclusive.
 
 .TP
 \fBDefMemPerNode\fR
@@ -635,12 +630,9 @@ are allocated to jobs (\fBSelectType=select/linear\fR) and
 resources are over\-subscribed (\fBOverSubscribe=yes\fR or
 \fBOverSubscribe=force\fR).
 The default value is 0 (unlimited).
-Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
-\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
-.\ \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
-.\ mutually exclusive.
+Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
+\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are
+mutually exclusive.
 
 .TP
 \fBDefaultStorageHost\fR
@@ -802,12 +794,9 @@ Consider which value you want to be used for scheduling purposes.
 \fB0\fR
 Base scheduling decisions upon the actual configuration of each individual
 node except that the node's processor count in Slurm's configuration must
-match the actual hardware configuration if \fBPreemptMode=suspend,gang\fR
-or \fBSelectType=select/cons_res\fR are configured (both of those plugins
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ match the actual hardware configuration if \fBPreemptMode=suspend,gang\fR,
-.\ \fBSelectType=select/cons_res\fR or \fBSelectType=select/cons_tres\fR are
-.\ configured (all of those plugins
+match the actual hardware configuration if \fBPreemptMode=suspend,gang\fR,
+\fBSelectType=select/cons_res\fR or \fBSelectType=select/cons_tres\fR are
+configured (all of those plugins
 maintain resource allocation information using bitmaps for the cores in the
 system and must remain static, while the node's memory and disk space can
 be established later).
@@ -1380,14 +1369,10 @@ Also see \fBFirstJobId\fR.
 Maximum real memory size available per allocated CPU in megabytes.
 Used to avoid over\-subscribing memory and causing paging.
 \fBMaxMemPerCPU\fR would generally be used if individual processors
-are allocated to jobs (\fBSelectType=select/cons_res\fR).
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ are allocated to jobs (\fBSelectType=select/cons_res\fR or
-.\ \fBSelectType=select/cons_tres\fR).
+are allocated to jobs (\fBSelectType=select/cons_res\fR or
+\fBSelectType=select/cons_tres\fR).
 The default value is 0 (unlimited).
-Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerNode\fR.
+Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerNode\fR.
 \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.
 
 NOTE: If a job specifies a memory per CPU limit that exceeds this system limit,
@@ -1765,9 +1750,7 @@ Job preemption rules are specified by Quality Of Service (QOS) specifications
 in the Slurm database.
 This option is not compatible with \fBPreemptMode=OFF\fR.
 A configuration of \fBPreemptMode=SUSPEND\fR is only supported by the
-\fBselect/cons_res\fR plugin.
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ \fBSelectType=select/cons_res\fR and \fBSelectType=select/cons_tres\fR plugins.
+\fBSelectType=select/cons_res\fR and \fBSelectType=select/cons_tres\fR plugins.
 .RE
 
 .TP
@@ -2511,14 +2494,10 @@ nodes that are in use.
 This will tend to leave currently idle resources available for backfilling
 longer running jobs, but may result in allocations having less than optimal
 network topology.
-This option is currently only supported by the select/cons_res plugin
-(or select/cray with SelectTypeParameters set to "OTHER_CONS_RES",
-which layers the select/cray plugin over the select/cons_res plugin).
-.\ FIXME: REMOVE THREE LINES ABOVE
-.\ This option is currently only supported by the select/cons_res and
-.\ select/cons_tres plugins (or select/cray with SelectTypeParameters set to
-.\ "OTHER_CONS_RES" or "OTHER_CONS_TRES", which layers the select/cray plugin
-.\ over the select/cons_res or select/cons_tres plugin respectively).
+This option is currently only supported by the select/cons_res and
+select/cons_tres plugins (or select/cray with SelectTypeParameters set to
+"OTHER_CONS_RES" or "OTHER_CONS_TRES", which layers the select/cray plugin
+over the select/cons_res or select/cons_tres plugin respectively).
 .TP
 \fBbf_continue\fR
 The backfill scheduler periodically releases locks in order to permit other
@@ -2846,10 +2825,8 @@ fail instead of using the cached env.  This will also implicitly imply the
 requeue_setup_env_fail option as well.
 .TP
 \fBpack_serial_at_end\fR
-If used with the select/cons_res plugin then put serial jobs at the end of
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ If used with the select/cons_res or select/cons_tres plugin,
-.\ then put serial jobs at the end of
+If used with the select/cons_res or select/cons_tres plugin,
+then put serial jobs at the end of
 the available nodes rather than using a best fit algorithm.
 This may reduce resource fragmentation for some workloads.
 .TP
@@ -2867,20 +2844,16 @@ Also see the \fBbf_max_job_part\fR option.
 Specify how many attempts should be made in reording preemptable jobs to
 minimize the count of jobs preempted.
 The default value is 1. High values may adversely impact performance.
-The logic to support this option is only available in the select/cons_res plugin.
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ The logic to support this option is only available in the select/cons_res and
-.\ select/cons_tres plugins.
+The logic to support this option is only available in the select/cons_res and
+select/cons_tres plugins.
 .TP
 \fBpreempt_strict_order\fR
 If set, then execute extra logic in an attempt to preempt only the lowest
 priority jobs.
 It may be desirable to set this configuration parameter when there are multiple
 priorities of preemptable jobs.
-The logic to support this option is only available in the select/cons_res plugin.
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ The logic to support this option is only available in the select/cons_res and
-.\ select/cons_tres plugins.
+The logic to support this option is only available in the select/cons_res and
+select/cons_tres plugins.
 .TP
 \fBpreempt_youngest_first\fR
 If set, then the preemption sorting algorithm will be changed to sort by the
@@ -3050,13 +3023,13 @@ for allocating resources to single CPU jobs only.
 Highly optimized for maximum throughput.
 NOTE: SPANK environment variables are NOT propagated to the job's \fBEpilog\fR
 program.
-.\ .TP
-.\ \fBselect/cons_tres\fR
-.\ The resources (cores, memory, GPUs and all other trackable resources) within
-.\ a node are individually allocated as consumable resources.
-.\ Note that whole nodes can be allocated to jobs for selected
-.\ partitions by using the \fIOverSubscribe=Exclusive\fR option.
-.\ See the partition \fBOverSubscribe\fR parameter for more information.
+.TP
+\fBselect/cons_tres\fR
+The resources (cores, memory, GPUs and all other trackable resources) within
+a node are individually allocated as consumable resources.
+Note that whole nodes can be allocated to jobs for selected
+partitions by using the \fIOverSubscribe=Exclusive\fR option.
+See the partition \fBOverSubscribe\fR parameter for more information.
 .RE
 
 .TP
@@ -3069,12 +3042,9 @@ The only supported options for \fBSelectType=select/linear\fR are
 prevents memory over subscription with job preemption or gang scheduling.
 By default \fBSelectType=select/linear\fR allocates whole nodes to jobs without
 considering their memory consumption.
-By default \fBSelectType=select/cons_res\fR, \fBSelectType=select/cray\fR, and
-\fBSelectType=select/serial\fR use \fBCR_CPU\fR, which allocates CPU (threads)
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ By default \fBSelectType=select/cons_res\fR, \fBSelectType=select/cray\fR,
-.\ \fBSelectType=select/serial\fR, and \fBSelectType=select/cons_tres\fR, use
-.\ \fBCR_CPU\fR, which allocates CPU (threads)
+By default \fBSelectType=select/cons_res\fR, \fBSelectType=select/cray\fR,
+\fBSelectType=select/serial\fR, and \fBSelectType=select/cons_tres\fR, use
+\fBCR_CPU\fR, which allocates CPU (threads)
 to jobs without considering their memory consumption.
 .RS
 .TP
@@ -3085,11 +3055,11 @@ The following options are supported for \fBSelectType=select/cray\fR:
 Layer the select/cons_res plugin under the select/cray plugin, the default is
 to layer on select/linear.  This also allows all the options available for
 \fBSelectType=select/cons_res\fR.
-.\ .TP
-.\ \fBOTHER_CONS_TRES\fR
-.\ Layer the select/cons_tres plugin under the select/cray plugin, the default is
-.\ to layer on select/linear.  This also allows all the options available for
-.\ \fBSelectType=select/cons_tres\fR.
+.TP
+\fBOTHER_CONS_TRES\fR
+Layer the select/cons_tres plugin under the select/cray plugin, the default is
+to layer on select/linear.  This also allows all the options available for
+\fBSelectType=select/cons_tres\fR.
 .TP
 \fBNHC_ABSOLUTELY_NO\fR
 Never run the node health check. Implies NHC_NO and NHC_NO_STEPS as well.
@@ -3104,10 +3074,8 @@ after each allocation.  This also sets NHC_NO_STEPS, so the NHC will never run
 except when nodes have been left with unkillable steps.
 .RE
 .TP
-The following options are supported by the \fBSelectType=select/cons_res\fR plugin:
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ The following options are supported by the \fBSelectType=select/cons_res\fR
-.\ and \fBSelectType=select/cons_tres\fR plugins:
+The following options are supported by the \fBSelectType=select/cons_res\fR
+and \fBSelectType=select/cons_tres\fR plugins:
 .RS
 .TP
 \fBCR_CPU\fR
@@ -4415,10 +4383,8 @@ plus \fBCPUs\fR; do not configure \fBSockets\fR, \fBCoresPerSocket\fR or
 A job can execute a one task per thread from within one job step or
 execute a distinct job step on each of the threads.
 Note also if you are running with more than 1 thread per core and running
-the select/cons_res plugin you will want to set the SelectTypeParameters
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ the select/cons_res or select/cons_tres plugin then you will want to set
-.\ the SelectTypeParameters
+the select/cons_res or select/cons_tres plugin then you will want to set
+the SelectTypeParameters
 variable to something other than CR_CPU to avoid unexpected results.
 The default value is 1.
 
@@ -4736,34 +4702,29 @@ specification will utilize this partition.
 Possible values are "YES" and "NO".
 The default value is "NO".
 
-.\ .TP
-.\ \fBDefCpusPerGPU\fR
-.\ Default count of CPUs allocated per allocated GPU.
-.\ 
+.TP
+\fBDefCpusPerGPU\fR
+Default count of CPUs allocated per allocated GPU.
+
 .TP
 \fBDefMemPerCPU\fR
 Default real memory size available per allocated CPU in megabytes.
 Used to avoid over\-subscribing memory and causing paging.
 \fBDefMemPerCPU\fR would generally be used if individual processors
-are allocated to jobs (\fBSelectType=select/cons_res\fR).
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ are allocated to jobs (\fBSelectType=select/cons_res\fR or
-.\ \fBSelectType=select/cons_tres\fR).
+are allocated to jobs (\fBSelectType=select/cons_res\fR or
+\fBSelectType=select/cons_tres\fR).
 If not set, the \fBDefMemPerCPU\fR value for the entire cluster will be used.
-Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
-\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
-.\ \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
-.\ exclusive.
-
-.\ .TP
-.\ \fBDefMemPerGPU\fR
-.\ Default real memory size available per allocated GPU in megabytes.
-.\ Also see \fBDefMemPerCPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
-.\ \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
-.\ exclusive.
-.\ 
+Also see \fBDefMemPerGPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
+\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
+exclusive.
+
+.TP
+\fBDefMemPerGPU\fR
+Default real memory size available per allocated GPU in megabytes.
+Also see \fBDefMemPerCPU\fR, \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
+\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
+exclusive.
+
 .TP
 \fBDefMemPerNode\fR
 Default real memory size available per allocated node in megabytes.
@@ -4773,12 +4734,9 @@ are allocated to jobs (\fBSelectType=select/linear\fR) and
 resources are over\-subscribed (\fBOverSubscribe=yes\fR or
 \fBOverSubscribe=force\fR).
 If not set, the \fBDefMemPerNode\fR value for the entire cluster will be used.
-Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
-\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
-.\ FIXME: REMOVE TWO LINES ABOVE
-.\ Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
-.\ \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
-.\ exclusive.
+Also see \fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBMaxMemPerCPU\fR.
+\fBDefMemPerCPU\fR, \fBDefMemPerGPU\fR and \fBDefMemPerNode\fR are mutually
+exclusive.
 
 .TP
 \fBDenyAccounts\fR
@@ -4863,10 +4821,8 @@ partition/queue.
 Maximum real memory size available per allocated CPU in megabytes.
 Used to avoid over\-subscribing memory and causing paging.
 \fBMaxMemPerCPU\fR would generally be used if individual processors
-are allocated to jobs (\fBSelectType=select/cons_res\fR).
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ are allocated to jobs (\fBSelectType=select/cons_res\fR or
-.\ \fBSelectType=select/cons_tres\fR).
+are allocated to jobs (\fBSelectType=select/cons_res\fR or
+\fBSelectType=select/cons_tres\fR).
 If not set, the \fBMaxMemPerCPU\fR value for the entire cluster will be used.
 Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
 \fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.
@@ -4945,10 +4901,8 @@ For more information see the following web pages:
 .RS
 .TP 12
 \fBEXCLUSIVE\fR
-Allocates entire nodes to jobs even with \fBSelectType=select/cons_res\fR configured.
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ Allocates entire nodes to jobs even with \fBSelectType=select/cons_res\fR or
-.\ \fBSelectType=select/cons_tres\fR configured.
+Allocates entire nodes to jobs even with \fBSelectType=select/cons_res\fR or
+\fBSelectType=select/cons_tres\fR configured.
 Jobs that run in partitions with "OverSubscribe=EXCLUSIVE" will have
 exclusive access to all allocated nodes.
 .TP
@@ -4966,9 +4920,7 @@ on the partition if started due to job preemption. For example, a configuration
 of \fIOverSubscribe=FORCE:1\fR will only permit one job per resources normally,
 but a second job can be started if done so through preemption based upon QOS.
 The use of \fIPreemptType=QOS\fR and \fIPreemptType=Suspend\fR only applies
-with \fBSelectType=select/cons_res\fR.
-.\ FIXME: REMOVE ONE LINE ABOVE
-.\ with \fBSelectType=select/cons_res\fR or \fBSelectType=select/cons_tres\fR.
+with \fBSelectType=select/cons_res\fR or \fBSelectType=select/cons_tres\fR.
 .TP
 \fBYES\fR
 Makes all resources in the partition available for sharing upon request by
-- 
GitLab