cgroup.conf.5

.TH "cgroup.conf" "5" "Slurm Configuration File" "November 2015" "Slurm Configuration File"

.SH "NAME"
cgroup.conf \- Slurm configuration file for the cgroup support

.SH "DESCRIPTION"

\fBcgroup.conf\fP is an ASCII file which defines parameters used by
Slurm's Linux cgroup related plugins.
The file location can be modified at system build time using the
DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF
environment variable. The file will always be located in the
same directory as the \fBslurm.conf\fP file.
.LP
Parameter names are case insensitive.
Any text following a "#" in the configuration file is treated
as a comment through the end of that line.
Changes to the configuration file take effect upon restart of
Slurm daemons, daemon receipt of the SIGHUP signal, or execution
of the command "scontrol reconfigure" unless otherwise noted.

.LP
For general Slurm Cgroups information, see the Cgroups Guide at
<http://slurm.schedmd.com/cgroups.html>.

.LP
The following cgroup.conf parameters are defined to control the general behavior
of Slurm cgroup plugins.

.TP
\fBCgroupAutomount\fR=<yes|no>
Slurm cgroup plugins require valid and functional cgroup subsystem to be mounted
under /sys/fs/cgroup/<subsystem_name>.
When launched, plugins check their subsystem availability. If not available,
the plugin launch fails unless CgroupAutomount is set to yes. In that case, the
plugin will first try to mount the required subsystems.

.TP
\fBCgroupMountpoint\fR=\fIPATH\fR
Specify the \fIPATH\fR under which cgroups should be mounted. This
should be a writable directory which will contain cgroups mounted
one per subsystem. The default \fIPATH\fR is /sys/fs/cgroup.

.TP
\fBCgroupReleaseAgentDir\fR=<path_to_release_agent_directory>
Used to tune the cgroup system behavior. This parameter identifies the location
of the directory containing Slurm cgroup release_agent files.

.SH "TASK/CGROUP PLUGIN"

.LP
The following cgroup.conf parameters are defined to control the behavior
of this particular plugin:

.TP
\fBAllowedDevicesFile\fR=<path_to_allowed_devices_file>
If the ConstrainDevices field is set to "yes" then this file has to be used to declare
the devices that need to be allowed by default for all the jobs. The current implementation
of cgroup devices subsystem works as a whitelist of entries, which means that in order to
isolate the access of a job upon particular devices we need to allow the access on all
the devices, supported by default and then deny on those that the job does not have the
permission to use. The default value is "/etc/slurm/cgroup_allowed_devices_file.conf". The syntax of
the file accepts one device per line and it permits lines like /dev/sda* or /dev/cpu/*/*.
See also an example of this file in etc/cgroup_allowed_devices_file.conf.example.

.TP
\fBAllowedRAMSpace\fR=<number>
Constrain the job cgroup RAM to this percentage of the allocated memory.
The percentage supplied may be expressed as floating point
number, e.g. 98.5. If the \fBAllowedRAMSpace\fR limit is exceeded, the
job steps will be killed and a warning message will be written to standard
error.  Also see \fBConstrainRAMSpace\fR.
The default value is 100.

.TP
\fBAllowedSwapSpace\fR=<number>
Constrain the job cgroup swap space to this percentage of the allocated
memory.  The default value is 0, which means that RAM+Swap will be limited
to \fBAllowedRAMSpace\fR. The supplied percentage may be expressed as a
floating point number, e.g. 50.5.  If the limit is exceeded, the job steps
will be killed and a warning message will be written to standard error.
Also see \fBConstrainSwapSpace\fR.

.TP
\fBConstrainCores\fR=<yes|no>
If configured to "yes" then constrain allowed cores to the subset of
allocated resources. It uses the cpuset subsystem.
The default value is "no".

.TP
\fBConstrainDevices\fR=<yes|no>
If configured to "yes" then constrain the job's allowed devices based on GRES
allocated resources. It uses the devices subsystem for that.
The default value is "no".

.TP
\fBConstrainKmemSpace\fR=<yes|no>
If configured to "yes" then constrain the job's Kmem RAM usage. In addition to
RAM usage. Only takes effect if ConstrainRAMSpace is set to "yes".
The default value is "yes",
in which case the job's Kmem limit will be set to its RAM limit.

.TP
\fBConstrainRAMSpace\fR=<yes|no>
If configured to "yes" then constrain the job's RAM usage by setting
the memory soft limit to the allocated memory and the hard limit to
the allocated memory * \fBAllowedRAMSpace\fR.  The default value is "no", in
which case the job's RAM limit will be set to its swap space limit if
\fBConstrainSwapSpace\fR is set to "yes".
Also see \fBAllowedSwapSpace\fR, \fBAllowedRAMSpace\fR and
\fBConstrainSwapSpace\fR.

.TP
\fBConstrainSwapSpace\fR=<yes|no>
If configured to "yes" then constrain the job's swap space usage.
The default value is "no". Note that when set to "yes" and
ConstrainRAMSpace is set to "no", \fBAllowedRAMSpace\fR is automatically set
to 100% in order to limit the RAM+Swap amount to 100% of job's requirement
plus the percent of allowed swap space. This amount is thus set to both
RAM and RAM+Swap limits. This means that in that particular case,
ConstrainRAMSpace is automatically enabled with the same limit than the one
used to constrain swap space.
Also see \fBAllowedSwapSpace\fR.

.TP
\fBMaxRAMPercent\fR=\fIPERCENT\fR
Set an upper bound in percent of total RAM on the RAM constraint for a job.
This will be the memory constraint applied to jobs that are not explicitly
allocated memory by Slurm (i.e. Slurm's select plugin is not configured to manage
memory allocations). The \fIPERCENT\fR may be an arbitrary floating
point number. The default value is 100.

.TP
\fBMaxSwapPercent\fR=\fIPERCENT\fR
Set an upper bound (in percent of total RAM) on the amount of RAM+Swap
that may be used for a job. This will be the swap limit applied to jobs
on systems where memory is not being explicitly allocated to job. The
\fIPERCENT\fR may be an arbitrary floating point number between 0 and 100.
The default value is 100.

.TP
\fBMinRAMSpace\fR=<number>
Set a lower bound (in MB) on the memory limits defined by
\fBAllowedRAMSpace\fR and \fBAllowedSwapSpace\fR. This prevents
accidentally creating a memory cgroup with such a low limit that slurmstepd
is immediately killed due to lack of RAM. The default limit is 30M.

.TP
\fBTaskAffinity\fR=<yes|no>
If configured to "yes" then set a default task affinity to bind each step
task to a subset of the allocated cores using \fBsched_setaffinity\fP.
The default value is "no".
Note: This feature requires the Portable Hardware Locality (hwloc) library
to be installed.

.SH "DISTRIBUTION\-SPECIFIC NOTES"

.LP
Debian and derivatives (e.g. Ubuntu) usually exclude the memory and memsw (swap)
cgroups by default. To include them, add the following parameters to the kernel
command line: \fBcgroup_enable=memory swapaccount=1\fR
.LP
This can usually be placed in /etc/default/grub inside the
\fBGRUB_CMDLINE_LINUX\fR variable. A command such as update\-grub must be run
after updating the file.

.SH "EXAMPLE"
.LP
.br
###
.br
# Slurm cgroup support configuration file
.br
###
.br
CgroupAutomount=yes
.br
CgroupReleaseAgentDir="/etc/slurm/cgroup"
.br
ConstrainCores=yes
.br
#

.SH "COPYING"
Copyright (C) 2010\-2012 Lawrence Livermore National Security.
Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
.br
Copyright (C) 2010\-2015 SchedMD LLC.
.LP
This file is part of Slurm, a resource management program.
For details, see <http://slurm.schedmd.com/>.
.LP
Slurm is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your option)
any later version.
.LP
Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
details.

.SH "SEE ALSO"
.LP
\fBslurm.conf\fR(5)