slurm.conf.5

.TH "slurm.conf" "5" "April 2008" "slurm.conf 1.3" "Slurm configuration file"
.SH "NAME"
slurm.conf \- Slurm configuration file 
.SH "DESCRIPTION"
\fB/etc/slurm.conf\fP is an ASCII file which describes general SLURM
configuration information, the nodes to be managed, information about
how those nodes are grouped into partitions, and various scheduling
parameters associated with those partitions. This file should be
consistent across all nodes in the cluster.
.LP
The file location can be modified at system build time using the
DEFAULT_SLURM_CONF parameter. In addition, you can use the
\fBSLURM_CONF\fR environment variable to override the built\-in
location of this file. The SLURM daemons also allow you to override
both the built\-in and environment\-provided location using the "\-f"
option on the command line.
.LP
Note the while SLURM daemons create log files and other files as needed, 
it treats the lack of parent directories as a fatal error. 
This prevents the daemons from running if critical file systems are
not mounted and will minimize the risk of cold\-starting (starting 
without preserving jobs).
.LP
The contents of the file are case insensitive except for the names of nodes 
and partitions. Any text following a "#" in the configuration file is treated 
as a comment through the end of that line. 
The size of each line in the file is limited to 1024 characters.
Changes to the configuration file take effect upon restart of 
SLURM daemons, daemon receipt of the SIGHUP signal, or execution 
of the command "scontrol reconfigure" unless otherwise noted.
.LP
If a line begins with the word "Include" followed by whitespace
and then a file name, that file will be included inline with the current
configuration file.
.LP
The overall configuration parameters available include:

.TP
\fBAccountingStorageEnforce\fR
If set to a non-zero value and the user, partition, account association is not 
defined for a job in the accounting database then prevent the job from being 
executed.
The default value is zero.

.TP
\fBAccountingStorageHost\fR
Define the name of the host where the database is running we are going
to store the accounting data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageHost\fR.

.TP
\fBAccountingStorageLoc\fR
Specifies the location of the file or database where accounting 
records are written.
Also see \fBDefaultStorageLoc\fR.

.TP
\fBAccountingStoragePass\fR
Define the password used to gain access to the database to store the
accounting data.
Only used for database type storage plugins, ignored otherwise.
In the case of Slurm DBD (Data Base Daemon) with Munge authentication this can be 
configured to use a Munge daemon specifically configured to provide authentication 
between clusters while the default Munge daemon provides authentication within a cluster. 
In that case, \fBAccountingStoragePass\fR should specify the named port to be used
for communications with the alternate Munge daemon (e.g. 
"/var/run/munge/global.socket.2"). The default value is NULL.
Also see \fBDefaultStoragePass\fR.

.TP
\fBAccountingStoragePort\fR
Define the port the database server is listening on where we are going
to store the accounting data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePort\fR.

.TP
\fBAccountingStorageType\fR
Define the accounting storage mechanism type.
Acceptable values at present include 
"accounting_storage/filetxt", "accounting_storage/gold",
"accounting_storage/mysql", "accounting_storage/none", 
"accounting_storage/pgsql", and "accounting_storage/slurmdbd".
The value "accounting_storage/filetxt" indicates that accounting records
will be written to a the file specified by the 
\fBAccountingStorageLoc\fR parameter.
The value "accounting_storage/gold" indicates that account records
will be written to Gold
(http://www.clusterresources.com/pages/products/gold-allocation-manager.php),
which maintains its own database.
The value "accounting_storage/mysql" indicates that accounting records
should be written to a MySQL database specified by the 
\fBAccountingStorageLoc\fR parameter.
The default value is "accounting_storage/none", which means that
account records are not maintained. 
The value "accounting_storage/pgsql" indicates that accounting records
should be written to a PostgreSQL database specified by the 
\fBAccountingStorageLoc\fR parameter.
The value "accounting_storage/slurmdbd" indicates that accounting records
will be written to SlurmDDB, which manages an underlying MySQL or 
PostgreSQL database. See "man slurmdbd" for more information.
Also see \fBDefaultStorageType\fR.

.TP
\fBAccountingStorageUser\fR
Define the name of the user we are going to connect to the database
with to store the accounting data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageUser\fR.

.TP
\fBAuthType\fR
Define the authentication method for communications between SLURM 
components. 
Acceptable values at present include "auth/none", "auth/authd", 
and "auth/munge".
The default value is "auth/none", which means the UID included in 
communication messages is not verified. 
This may be fine for testing purposes, but 
\fBdo not use "auth/none" if you desire any security\fR.
"auth/authd" indicates that Brett Chun's authd is to be used (see
"http://www.theether.org/authd/" for more information, Note that
authd is no longer actively supported).
"auth/munge" indicates that LLNL's MUNGE is to be used
(this is the best supported authentication mechanism for SLURM, 
see "http://home.gna.org/munge/" for more information).
All SLURM daemons and commands must be terminated prior to changing 
the value of \fBAuthType\fR and later restarted (SLURM jobs can be 
preserved).

.TP
\fBBackupAddr\fR
Name that \fBBackupController\fR should be referred to in 
establishing a communications path. This name will 
be used as an argument to the gethostbyname() function for 
identification. For example, "elx0000" might be used to designate 
the ethernet address for node "lx0000". 
By default the \fBBackupAddr\fR will be identical in value to 
\fBBackupController\fR.

.TP
\fBBackupController\fR
The name of the machine where SLURM control functions are to be 
executed in the event that \fBControlMachine\fR fails. This node
may also be used as a compute server if so desired. It will come into service 
as a controller only upon the failure of ControlMachine and will revert 
to a "standby" mode when the ControlMachine becomes available once again. 
This should be a node name without the full domain name (e.g. "lx0002"). 
While not essential, it is recommended that you specify a backup controller.
See  the \fBRELOCATING CONTROLLERS\fR section if you change this.

.TP
\fBCacheGroups\fR
If set to 1, the slurmd daemon will  cache /etc/groups entries.
This can improve performance for highly parallel jobs if NIS servers
are used and unable to respond very quickly.
The default value is 0 to disable caching group data.

.TP
\fBCheckpointType\fR
Define the system\-initiated checkpoint method to be used for user jobs. 
The slurmctld daemon must be restarted for a change in \fBCheckpointType\fR 
to take effect. 
Acceptable values at present include
"checkpoint/aix" (only on AIX systems),
"checkpoint/ompi" (requires OpenMPI version 1.3 or higher),
"checkpoint/xlch" (for XLCH, requires that SlurmUser be root), and
"checkpoint/none".
The default value is "checkpoint/none".

.TP
\fBClusterName\fR
The name by which this SLURM managed cluster is known for accounting 
purposes. This is needed distinguish between accounting data from 
multiple clusters being recorded in a single database.

.TP
\fBControlAddr\fR
Name that \fBControlMachine\fR should be referred to in 
establishing a communications path. This name will 
be used as an argument to the gethostbyname() function for 
identification. For example, "elx0000" might be used to designate 
the ethernet address for node "lx0000". 
By default the \fBControlAddr\fR will be identical in value to 
\fBControlMachine\fR.

.TP
\fBControlMachine\fR
The name of the machine where SLURM control functions are executed. 
This should be a node name without the full domain name (e.g. "lx0001"). 
This value must be specified.
See  the \fBRELOCATING CONTROLLERS\fR section if you change this.

.TP
\fBCryptoType\fR
Define the cryptographic signature tool to be used in the creation of 
job step credentials.
The slurmctld daemon must be restarted for a change in \fBCryptoType\fR
to take effect.
Acceptable values at present include "crypto/munge" and "crypto/openssl".
OpenSSL offers the best performance and is available with an 
Apache style open source license.
Munge is a little slower, but is available under the Gnu General Public 
License (GPL).
The default value is "crypto/openssl".

.TP
\fBDefMemPerTask\fR
Default real memory size available per task in MegaBytes. 
Used to avoid over\-subscribing memory and causing paging.
Also see \fBMaxMemPerTask\fR.
The default value is 0 (unlimited).

.TP
\fBDefaultStorageHost\fR
Define the name of the host where the database is running and used to
to store the accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStorageHost\fR and \fBJobCompHost\fR.

.TP
\fBDefaultStorageLoc\fR
Specifies the location of the file or database where accounting 
and job completion records are written.
Also see \fBAccountingStorageLoc\fR and \fBJobCompLoc\fR.

.TP
\fBDefaultStoragePass\fR
Define the password used to gain access to the database to store the
accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStoragePass\fR and \fBJobCompPass\fR.

.TP
\fBDefaultStoragePort\fR
Define the port the database server is listening on where we are going
to store the accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStoragePort\fR and \fBJobCompPort\fR.

.TP
\fBDefaultStorageType\fR
.ad l
Define the accounting and job completion storage mechanism type.
Acceptable values at present include 
"filetxt", "gold", "mysql", "none", "pgsql", and "slurmdbd".
The value "filetxt" indicates that records will be written to a the file.
The value "gold" indicates that records will be written to Gold
(http://www.clusterresources.com/pages/products/gold-allocation-manager.php),
which maintains its own database.
The value "mysql" indicates that accounting records will be written to 
a mysql database.
The default value is "none", which means that records are not maintained. 
The value "pgsql" indicates that records will be written to a postresql 
database.
The value "slurmdbd" indicates that records will be written to SlurmDbd,
which maintains its own database. See "man slurmdbd for more information".
Also see \fBAccountingStorageType\fR  and \fBJobCompType\fR.
.ad

.TP
\fBDefaultStorageUser\fR
Define the name of the user we are going to connect to the database
with to store the accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStorageUser\fR and \fBJobCompUser\fR.

.TP
\fBEpilog\fR
Fully qualified pathname of a script to execute as user root on every 
node when a user's job completes (e.g. "/usr/local/slurm/epilog"). This may 
be used to purge files, disable user login, etc. By default there is no epilog.

.TP
\fBEpilogMsgTime\fR
The number of microseconds the the slurmctld daemon requires to process
an epilog completion message from the slurmd dameons. This parameter can
be used to prevent a burst of epilog completion messages from being sent 
at the same time which should help prevent lost messages and improve 
throughput for large jobs.
The default value is 2000 microseconds.
For a 1000 node job, this spreads the epilog completion messages out over
two seconds.

.TP
\fBFastSchedule\fR
Controls how a nodes configuration specifications in slurm.conf are used.
If the number of node configuration entries in the configuration file
is significantly lower than the number of nodes, setting FastSchedule to
1 will permit much faster scheduling decisions to be made.
(The scheduler can just check the values in a few configuration records
instead of possibly thousands of node records. If a job can't be initiated
immediately, the scheduler may execute these tests repeatedly.)
Note that on systems with hyper\-threading, the processor count
reported by the node will be twice the actually processor count.
Consider which value you want to be used for scheduling purposes.
.RS
.TP 5
\fB1\fR (default)
Consider the configuration of each node to be that specified in the
configuration file and any node with less
than the configured resources will be set DOWN.
.TP
\fB0\fR
Base scheduling decisions upon the actual configuration of 
each individual node. 
.TP
\fB2\fR
Consider the configuration of each node to be that specified in the 
slurm.conf configuration file and any node with less resources 
than configured will not be set DOWN. 
This can be useful for testing purposes.
.RE

.TP
\fBFirstJobId\fR
The job id to be used for the first submitted to SLURM without a 
specific requested value. Job id values generated will incremented by 1 
for each subsequent job. This may be used to provide a meta\-scheduler 
with a job id space which is disjoint from the interactive jobs. 
The default value is 1.

.TP
\fBGetEnvTimeout\fR
Used for Moab scheduled jobs only. Controls how long job should wait
in seconds for loading the user's environment before attempting to 
load it from a cache file. Applies when the srun or sbatch 
\fI--get-user-env\fR option is used. Default value is 2 seconds.

.TP
\fBHealthCheckInterval\fR
The interval in seconds between executions of \fBHealthCheckProgram\fR.
The default value is zero, which disables execution.

.TP
\fBHealthCheckProgram\fR
Fully qualified pathname of a script to execute as user root periodically 
on all compute nodes that are not in the DOWN state. This may be used to 
verify the node is fully operational and DRAIN the it otherwise.
The interval is controlled using the \fBHealthCheckInterval\fR parameter.
Note that the \fBHealthCheckProgram\fR will be executed at the same time 
on all nodes to minimize its impact upon parallel programs.
This program is will be killed if it does not terminate normally within
60 seconds. 
By default, no program will be executed.

.TP
\fBInactiveLimit\fR
The interval, in seconds, a job or job step is permitted to be inactive
before it is terminated. A job or job step is considered inactive if 
the associated srun command is not responding to slurm daemons. This 
could be due to the termination of the srun command or the program 
being is a stopped state. A batch job is considered inactive if it 
has no active job steps (e.g. periods of pre\- and post\-processing).
This limit permits defunct jobs to be purged in a timely fashion 
without waiting for their time limit to be reached.
This value should reflect the possibility that the srun command may
stopped by a debugger or considerable time could be required for batch 
job pre\- and post\-processing. 
This limit is ignored for jobs running in partitions with the 
\fBRootOnly\fR flag set (the scheduler running as root will be 
responsible for the job).
The default value is unlimited (zero). 
May not exceed 65533.

.TP
\fBJobAcctGatherType\fR
Define the job accounting mechanism type.
Acceptable values at present include "jobacct_gather/aix" (for AIX operating
system), "jobacct_gather/linux" (for Linux operating system) and "jobacct_gather/none"
(no accounting data collected).
The default value is "jobacct_gather/none".
In order to use the \fBsacct\fR tool, "jobacct_gather/aix" or "jobacct_gather/linux" 
must be configured.

.TP
\fBJobAcctGatherFrequency\fR
Define the job accounting sampling interval.
For jobacct_gather/none this parameter is ignored.
For  jobacct_gather/aix and jobacct_gather/linux the parameter is a number is 
seconds between sampling job state.
The default value is 30 seconds. 
A value of zero disables real the periodic job sampling and provides accounting 
information only on job termination (reducing SLURM interference with the job).

.TP
\fBJobCompHost\fR
Define the name of the host where the database is running and used
to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageHost\fR.

.TP
\fBJobCompLoc\fR
The interpretation of this value depends upon the logging mechanism 
specified by the \fBJobCompType\fR parameter either a filename or a 
database name. 
Also see \fBDefaultStorageLoc\fR.

.TP
\fBJobCompPass\fR
Define the password used to gain access to the database to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePass\fR.

.TP
\fBJobCompPort\fR
Define the port the database server is listening on where we are going
to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePort\fR.

.TP
\fBJobCompType\fR
Define the job completion logging mechanism type.
Acceptable values at present include "jobcomp/none", "jobcomp/filetxt", 
"jobcomp/mysql", "jobcomp/pgsql", "jobcomp/script"and "jobcomp/slurmdbd".
The default value is "jobcomp/none", which means that upon job completion 
the record of the job is purged from the system. 
The value "jobcomp/filetxt" indicates that a record of the job should be 
written to a text file specified by the \fBJobCompLoc\fR parameter.
The value "jobcomp/mysql" indicates that a record of the job should be 
written to a mysql database specified by the \fBJobCompLoc\fR parameter.
The value "jobcomp/pgsql" indicates that a record of the job should be 
written to a postgresql database specified by the \fBJobCompLoc\fR parameter.
The value "jobcomp/script" indicates that a script specified by the 
\fBJobCompLoc\fR parameter is to be executed with environment variables 
indicating the job information.
The value "jobcomp/slurmdbd" indicates that job completion records
will be written to SlurmDbd, which maintains its own database. See 
"man slurmdbd" for more information.
Also see \fBDefaultStorageType\fR.

.TP
\fBJobCompUser\fR
Define the name of the user we are going to connect to the database
with to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageUser\fR.

.TP
\fBJobCredentialPrivateKey\fR
Fully qualified pathname of a file containing a private key used for 
authentication by Slurm daemons.
This parameter is ignored if \fBCryptType=munge\fR.

.TP
\fBJobCredentialPublicCertificate\fR
Fully qualified pathname of a file containing a public key used for 
authentication by Slurm daemons.
This parameter is ignored if \fBCryptType=munge\fR.

.TP
\fBJobFileAppend\fR
This option controls what to do if a job's output or error file 
exist when the job is started. 
If \fBJobFileAppend\fR is set to a value of 1, then append to 
the existing file.
By default, any existing file is truncated.

.TP
\fBJobRequeue\fR
This option controls what to do by default after a node failure. 
If \fBJobRequeue\fR is set to a value of 1, then any job running 
on the failed node will be requeued for execution on different nodes.
If \fBJobRequeue\fR is set to a value of 0, then any job running 
on the failed node will be terminated.
Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR 
option to change the default behavior for individual jobs.
The default value is 1.

.TP
\fBKillWait\fR
The interval, in seconds, given to a job's processes between the 
SIGTERM and SIGKILL signals upon reaching its time limit. 
If the job fails to terminate gracefully 
in the interval specified, it will be forcibly terminated. 
The default value is 30 seconds.
May not exceed 65533.

.TP
\fBLicenses\fR
Specification of licenses (or other resources available on all 
nodes of the cluster) which can be allocated to jobs.
License names can optionally be followed by an asterisk 
and count with a default count of one.
Multiple license names should be comma separated (e.g.
"Licenses=foo*4,bar").
Note that SLURM prevents jobs from being scheduled if their 
required license specification is not available.
SLURM does not prevent jobs from using licenses that are
not explicitly listed in the job submission specification.

.TP
\fBMailProg\fR
Fully qualified pathname to the program used to send email per user request.
The default value is "/bin/mail".

.TP
\fBMaxJobCount\fR
The maximum number of jobs SLURM can have in its active database 
at one time. Set the values of \fBMaxJobCount\fR and \fBMinJobAge\fR 
to insure the slurmctld daemon does not exhaust its memory or other 
resources. Once this limit is reached, requests to submit additional 
jobs will fail. The default value is 5000 jobs. This value may not 
be reset via "scontrol reconfig". It only takes effect upon restart 
of the slurmctld daemon.
May not exceed 65533.

.TP
\fBMaxMemPerTask\fR
Maximum real memory size available per task in MegaBytes. 
Used to avoid over\-subscribing memory and causing paging.
Also see \fBDefMemPerTask\fR.
The default value is 0 (unlimited).

.TP
\fBMessageTimeout\fR
Time permitted for a round\-trip communication to complete
in seconds. Default value is 10 seconds. For systems with 
shared nodes, the slurmd daemon could be paged out and 
necessitate higher values.

.TP
\fBMinJobAge\fR
The minimum age of a completed job before its record is purged from 
SLURM's active database. Set the values of \fBMaxJobCount\fR and 
\fBMinJobAge\fR to insure the slurmctld daemon does not exhaust 
its memory or other resources. The default value is 300 seconds. 
A value of zero prevents any job record purging.
May not exceed 65533.

.TP
\fBMpiDefault\fR
Identifies the default type of MPI to be used. 
Srun may override this configuration parameter in any case.
Currently supported versions include: 
\fBmpichgm\fR, 
\fBmvapich\fR,
\fBnone\fR (default, which works for many other versions of MPI including 
LAM MPI and Open MPI).

.TP
\fBPluginDir\fR
Identifies the places in which to look for SLURM plugins. 
This is a colon\-separated list of directories, like the PATH 
environment variable. 
The default value is "/usr/local/lib/slurm".

.TP
\fBPlugStackConfig\fR
Location of the config file for SLURM stackable plugins that use
the Stackable Plugin Architecture for Node job (K)control (SPANK).
This provides support for a highly configurable set of plugins to
be called before and/or after execution of each task spawned as
part of a user's job step.  Default location is "plugstack.conf"
in the same directory as the system slurm.conf. For more information
on SPANK plugins, see the \fBspank\fR(8) manual.

.TP
\fBPrivateData\fR
If non-zero then users are unable to view jobs or job steps belonging 
to other users (except for SlurmUser or root, who can view all jobs).
The default value is "0", permitting any user to view any jobs or 
job steps.

.TP
\fBProctrackType\fR
Identifies the plugin to be used for process tracking. 
The slurmd daemon uses this mechanism to identify all processes 
which are children of processes it spawns for a user job. 
The slurmd daemon must be restarted for a change in ProctrackType
to take effect.
NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to 
identify all processes associated with a job since processes 
can become a child of the init process (when the parent process 
terminates) or change their process group. 
To reliably track all processes, one of the other mechanisms 
utilizing kernel modifications is preferable. 
NOTE: "proctrack/linuxproc" is not compatible with "switch/elan."
Acceptable values at present include:
.RS
.TP 
\fBproctrack/aix\fR which uses an AIX kernel extension and is 
the default for AIX systems
.TP
\fBproctrack/linuxproc\fR which uses linux process tree using 
parent process IDs
.TP
\fBproctrack/rms\fR which uses Quadrics kernel patch and is the 
default if "SwitchType=switch/elan" 
.TP
\fBproctrack/sgi_job\fR which uses SGI's Process Aggregates (PAGG)
kernel module, see \fIhttp://oss.sgi.com/projects/pagg/\fR 
for more information 
.TP
\fBproctrack/pgid\fR which uses process group IDs and is the 
default for all other systems
.RE

.TP
\fBProlog\fR
Fully qualified pathname of a script for the slurmd to execute whenever
it is asked to run a job step from a new job allocation.  (e.g.
"/usr/local/slurm/prolog").  The slurmd executes the script before starting
the job step.  This may be used to purge files, enable user login, etc.
By default there is no prolog. Any configured script is expected to 
complete execution quickly (in less time than \fBMessageTimeout\fR).

NOTE:  The Prolog script is ONLY run on any individual
node when it first sees a job step from a new allocation; it does not
run the Prolog immediately when an allocation is granted.  If no job steps
from an allocation are run on a node, it will never run the Prolog for that
allocation.  The Epilog, on the other hand, always runs on every node of an
allocation when the allocation is released.

.TP
\fBPropagatePrioProcess\fR
Setting \fBPropagatePrioProcess\fR to "1", will cause a users job to run
with the same priority (aka nice value) as the users process which
launched the job on the submit node.
If set to "0", or left unset, the users job will inherit the
scheduling priority from the slurm daemon.

.TP
\fBPropagateResourceLimits\fR
A list of comma separated resource limit names.
The slurmd daemon uses these names to obtain the associated (soft) limit
values from the users process environment on the submit node.
These limits are then propagated and applied to the jobs that
will run on the compute nodes. 
This parameter can be useful when system limits vary among nodes.
Any resource limits that do not appear in the list are not propagated.
However, the user can override this by specifying which resource limits
to propagate with the srun commands "\-\-propagate" option.
If neither of the 'propagate resource limit' parameters are specified, then
the default action is to propagate all limits.
Only one of the parameters, either
\fBPropagateResourceLimits\fR or \fBPropagateResourceLimitsExcept\fR,
may be specified.
The following limit names are supported by Slurm (although some 
options may not be supported on some systems):
.RS
.TP 10
\fBALL\fR
All limits listed below
.TP
\fBAS\fR
The maximum address space for a processes
.TP
\fBCORE\fR
The maximum size of core file
.TP
\fBCPU\fR
The maximum amount of CPU time
.TP
\fBDATA\fR
The maximum size of a process's data segment
.TP
\fBFSIZE\fR
The maximum size of files created
.TP
\fBMEMLOCK\fR
The maximum size that may be locked into memory
.TP
\fBNOFILE\fR
The maximum number of open files
.TP
\fBNPROC\fR
The maximum number of processes available
.TP
\fBRSS\fR
The maximum resident set size
.TP
\fBSTACK\fR
The maximum stack size
.RE

.TP
\fBPropagateResourceLimitsExcept\fR
A list of comma separated resource limit names.
By default, all resource limits will be propagated, (as described by
the \fBPropagateResourceLimits\fR parameter), except for the limits
appearing in this list.   The user can override this by specifying which
resource limits to propagate with the srun commands "\-\-propagate" option.
See \fBPropagateResourceLimits\fR above for a list of valid limit names.

.TP
\fBResumeProgram\fR
SLURM supports a mechanism to reduce power consumption on nodes that 
remain idle for an extended period of time. 
This is typically accomplished by reducing voltage and frequency. 
\fBResumeProgram\fR is the program that will be executed when a node 
in power save mode is assigned work to perform.
The program executes as \fBSlurmUser\fR.
The argument to the program will be the names of nodes to
be removed from power savings mode (using SLURM's hostlist
expression format).
By default no program is run.
Related configuration options include \fBResumeRate\fR, \fBSuspendRate\fR,
\fBSuspendTime\fR, \fBSuspendProgram\fR, \fBSuspendExcNodes\fR, and
\fBSuspendExcParts\fR.
More information is available at the SLURM web site
(https://computing.llnl.gov/linux/slurm/power_save.html).

.TP
\fBResumeRate\fR
The rate at which nodes in power save mode are returned to normal 
operation by \fBResumeProgram\fR. 
The value is number of nodes per minute and it can be used to prevent 
power surges if a large number of nodes in power save mode are 
assigned work at the same time (e.g. a large job starts).
A value of zero results in no limits being imposed.
The default value is 60 nodes per minute.
Related configuration options include \fBResumeProgram\fR, \fBSuspendRate\fR,
\fBSuspendTime\fR, \fBSuspendProgram\fR, \fBSuspendExcNodes\fR, and
\fBSuspendExcParts\fR.

.TP
\fBReturnToService\fR
If set to 1, then a non\-responding (DOWN) node will become available 
for use upon registration. Note that DOWN node's state will be changed 
only if it was set DOWN due to being non\-responsive. If the node was 
set DOWN for any other reason (low memory, prolog failure, epilog 
failure, etc.), its state will not automatically be changed.  The 
default value is 0, which means that a node will remain in the 
DOWN state until a system administrator explicitly changes its state
(even if the slurmd daemon registers and resumes communications).

.TP
\fBSchedulerPort\fR
The port number on which slurmctld should listen for connection requests.
This value is only used by the Maui Scheduler (see \fBSchedulerType\fR).
The default value is 7321.

.TP
\fBSchedulerRootFilter\fR
Identifies whether or not \fBRootOnly\fR partitions should be filtered from
any external scheduling activities. If set to 0, then \fBRootOnly\fR partitions
are treated like any other partition. If set to 1, then \fBRootOnly\fR
partitions are exempt from any external scheduling activities. The
default value is 1. Currently only used by the built\-in backfill
scheduling module "sched/backfill" (see \fBSchedulerType\fR).

.TP
\fBSchedulerTimeSlice\fR
Number of seconds in each time slice when \fBSchedulerType=sched/gang\fR.
The default value is 30.

.TP
\fBSchedulerType\fR
Identifies the type of scheduler to be used. Acceptable values include 
"sched/builtin" for the built\-in FIFO scheduler, 
"sched/backfill" for a backfill scheduling module to augment 
the default FIFO scheduling, 
"sched/gang" for gang scheduler (time\-slicing of parallel jobs),
"sched/hold" to hold all newly arriving jobs if a file "/etc/slurm.hold" 
exists otherwise use the built\-in FIFO scheduler, and 
"sched/wiki" for the Wiki interface to the Maui Scheduler. 
The default value is "sched/builtin".
Backfill scheduling will initiate lower\-priority jobs if doing 
so does not delay the expected initiation time of any higher 
priority job. 
Note that this backfill scheduler implementation is relatively 
simple. It does not support partitions configured to to share 
resources (run multiple jobs on the same nodes) or support 
jobs requesting specific nodes.
When initially setting the value to "sched/wiki", any pending jobs 
must have their priority set to zero (held).
When changing the value from "sched/wiki", all pending jobs 
should have their priority change from zero to some large number.
The \fBscontrol\fR command can be used to change job priorities.
The \fBslurmctld\fR daemon must be restarted for a change in 
scheduler type to become effective.

.TP
\fBSelectType\fR
Identifies the type of resource selection algorithm to be used. 
Acceptable values include 
.RS
.TP
\fBselect/linear\fR
for allocation of entire nodes assuming a
one\-dimensional array of nodes in which sequentially ordered 
nodes are preferable. 
This is the default value for non\-BlueGene systems. 
.TP
\fBselect/cons_res\fR
The resources within a node are individually allocated as
consumable resources. 
Note that whole nodes can be allocated to jobs for selected 
partitions by using the \fIShared=Exclusive\fR option.
See the partition \fBShared\fR parameter for more information.
.TP
\fBselect/bluegene\fR
for a three\-dimensional BlueGene system. 
The default value is "select/bluegene" for BlueGene systems.
.RE

.TP
\fBSelectTypeParameters\fR
The permitted values of \fBSelectTypeParameters\fR depend upon the 
configured value of \fBSelectType\fR.
\fBSelectType=select/bluegene\fR supports no \fBSelectTypeParameters\fR.
The only supported option for \fBSelectType=select/linear\fR is 
\fBCR_Memory\fR, which treats memory as a consumable resource and 
prevents memory over subscription with job preemption or gang scheduling.
The following values are supported for \fBSelectType=select/cons_res\fR:
.RS
.TP
\fBCR_CPU\fR
CPUs are consumable resources.
There is no notion of sockets, cores or threads.
On a multi\-core system, each core will be considered a CPU.
On a multi\-core and hyperthreaded system, each thread will be
considered a CPU.
On single\-core systems, each CPUs will be considered a CPU.
.TP
\fBCR_CPU_Memory\fR
CPUs and memory are consumable resources.
.TP
\fBCR_Core\fR
Cores are consumable resources.
.TP
\fBCR_Core_Memory\fR
Cores and memory are consumable resources.
.TP
\fBCR_Socket\fR
Sockets are consumable resources.
.TP
\fBCR_Socket_Memory\fR
Memory and CPUs are consumable resources.
.TP
\fBCR_Memory\fR
Memory is a consumable resource.
NOTE: This implies \fIShared=YES\fR or \fIShared=FORCE\fR for all partitions.
.RE

.TP
\fBSlurmUser\fR
The name of the user that the \fBslurmctld\fR daemon executes as. 
For security purposes, a user other than "root" is recommended.
The default value is "root". 

.TP
\fBSlurmctldDebug\fR
The level of detail to provide \fBslurmctld\fR daemon's logs. 
Values from 0 to 7 are legal, with `0' being "quiet" operation and `7' 
being insanely verbose.
The default value is 3.

.TP
\fBSlurmctldLogFile\fR
Fully qualified pathname of a file into which the \fBslurmctld\fR daemon's 
logs are written.
The default value is none (performs logging via syslog).

.TP
\fBSlurmctldPidFile\fR
Fully qualified pathname of a file into which the  \fBslurmctld\fR daemon 
may write its process id. This may be used for automated signal processing.
The default value is "/var/run/slurmctld.pid".

.TP
\fBSlurmctldPort\fR
The port number that the SLURM controller, \fBslurmctld\fR, listens 
to for work. The default value is SLURMCTLD_PORT as established at system 
build time. If none is explicitly specified, it will be set to 6817.  
NOTE: Either \fBslurmctld\fR and \fBslurmd\fR daemons must not 
execute on the same nodes or the values of \fBSlurmctldPort\fR and 
\fBSlurmdPort\fR must be different.

.TP
\fBSlurmctldTimeout\fR
The interval, in seconds, that the backup controller waits for the 
primary controller to respond before assuming control. 
The default value is 120 seconds.
May not exceed 65533.

.TP
\fBSlurmdDebug\fR
The level of detail to provide \fBslurmd\fR daemon's logs. 
Values from 0 to 7 are legal, with `0' being "quiet" operation and `7' being 
insanely verbose.
The default value is 3.

.TP
\fBSlurmdLogFile\fR
Fully qualified pathname of a file into which the  \fBslurmd\fR daemon's 
logs are written.
The default value is none (performs logging via syslog).
Any "%h" within the name is replaced with the hostname on which the 
\fBslurmd\fR is running.

.TP
\fBSlurmdPidFile\fR
Fully qualified pathname of a file into which the  \fBslurmd\fR daemon may write 
its process id. This may be used for automated signal processing.
The default value is "/var/run/slurmd.pid".

.TP
\fBSlurmdPort\fR
The port number that the SLURM compute node daemon, \fBslurmd\fR, listens 
to for work. The default value is SLURMD_PORT as established at system 
build time. If none is explicitly specified, its value will be 6818. 
NOTE: Either slurmctld and slurmd daemons must not execute
on the same nodes or the values of \fBSlurmctldPort\fR and \fBSlurmdPort\fR
must be different.

.TP
\fBSlurmdSpoolDir\fR
Fully qualified pathname of a directory into which the \fBslurmd\fR
daemon's state information and batch job script information are written. This
must be a common pathname for all nodes, but should represent a directory which
is local to each node (reference a local file system). The default value
is "/var/spool/slurmd." \fBNOTE\fR: This directory is also used to store
\fBslurmd\fR's
shared memory lockfile, and \fBshould not be changed\fR unless the system
is being cleanly restarted. If the location of \fBSlurmdSpoolDir\fR is
changed and \fBslurmd\fR is restarted, the new daemon will attach to a
different shared memory region and lose track of any running jobs.

.TP
\fBSlurmdTimeout\fR
The interval, in seconds, that the SLURM controller waits for \fBslurmd\fR 
to respond before configuring that node's state to DOWN. 
The default value is 300 seconds.
A value of zero indicates the node will not be tested by \fBslurmctld\fR to 
confirm the state of \fBslurmd\fR, the node will not be automatically set to 
a DOWN state indicating a non\-responsive \fBslurmd\fR, and some other tool 
will take responsibility for monitoring the state of each compute node 
and its \fBslurmd\fR daemon.
The value may not exceed 65533.

.TP
\fBSrunEpilog\fR
Fully qualified pathname of an executable to be run by srun following the
completion of a job step.  The command line arguments for the executable will
be the command and arguments of the job step.  This configuration parameter
may be overridden by srun's \fB\-\-epilog\fR parameter.

.TP
\fBSrunProlog\fR
Fully qualified pathname of an executable to be run by srun prior to the
launch of a job step.  The command line arguments for the executable will
be the command and arguments of the job step.  This configuration parameter
may be overridden by srun's \fB\-\-prolog\fR parameter.

.TP
\fBStateSaveLocation\fR
Fully qualified pathname of a directory into which the SLURM controller, 
\fBslurmctld\fR, saves its state (e.g. "/usr/local/slurm/checkpoint"). 
SLURM state will saved here to recover from system failures.
\fBSlurmUser\fR must be able to create files in this directory.
If you have a \fBBackupController\fR configured, this location should be 
readable and writable by both systems. 
Since all running and pending job information is stored here, the use of 
a reliable file system (e.g. RAID) is recommended.
The default value is "/tmp".
If any slurm daemons terminate abnormally, their core files will also be written 
into this directory.

.TP
\fBSuspendExcNodes\fR
Specifies the nodes which are to not be placed in power save mode, even 
if the node remains idle for an extended period of time.
Use SLURM's hostlist expression to identify nodes.
By default no nodes are excluded.
Related configuration options include \fBResumeProgram\fR, \fBResumeRate\fR,
\fBSuspendProgram\fR, \fBSuspendRate\fR, \fBSuspendTime\fR and
\fBSuspendExcParts\fR.

.TP
\fBSuspendExcParts\fR
Specifies the partitions whose nodes are to not be placed in power save 
mode, even if the node remains idle for an extended period of time.
Multiple partitions can be identified and separated by commas.
By default no nodes are excluded.
Related configuration options include \fBResumeProgram\fR, \fBResumeRate\fR,
\fBSuspendProgram\fR, \fBSuspendRate\fR, \fBSuspendTime\fR and
\fBSuspendExcNodes\fR.

.TP
\fBSuspendProgram\fR
\fBSuspendProgram\fR is the program that will be executed when a node
remains idle for an extended period of time.
This program is expected to place the node into some power save mode.
The program executes as \fBSlurmUser\fR.
The argument to the program will be the names of nodes to
be placed into power savings mode (using SLURM's hostlist
expression format).
By default no program is run.
Related configuration options include \fBResumeProgram\fR, \fBResumeRate\fR,
\fBSuspendRate\fR, \fBSuspendTime\fR, \fBSuspendExcNodes\fR, and
\fBSuspendExcParts\fR.

.TP
\fBSuspendRate\fR
The rate at which nodes are place into power save mode by \fBSuspendProgram\fR.