slurm.conf.5

.TH "slurm.conf" "5" "March 2009" "slurm.conf 2.0" "Slurm configuration file"

.SH "NAME"
slurm.conf \- Slurm configuration file 
.SH "DESCRIPTION"
\fB/etc/slurm.conf\fP is an ASCII file which describes general SLURM
configuration information, the nodes to be managed, information about
how those nodes are grouped into partitions, and various scheduling
parameters associated with those partitions. This file should be
consistent across all nodes in the cluster.
.LP
The file location can be modified at system build time using the
DEFAULT_SLURM_CONF parameter. In addition, you can use the
\fBSLURM_CONF\fR environment variable to override the built\-in
location of this file. The SLURM daemons also allow you to override
both the built\-in and environment\-provided location using the "\-f"
option on the command line.
.LP
Note the while SLURM daemons create log files and other files as needed, 
it treats the lack of parent directories as a fatal error. 
This prevents the daemons from running if critical file systems are
not mounted and will minimize the risk of cold\-starting (starting 
without preserving jobs).
.LP
The contents of the file are case insensitive except for the names of nodes 
and partitions. Any text following a "#" in the configuration file is treated 
as a comment through the end of that line. 
The size of each line in the file is limited to 1024 characters.
Changes to the configuration file take effect upon restart of 
SLURM daemons, daemon receipt of the SIGHUP signal, or execution 
of the command "scontrol reconfigure" unless otherwise noted.
.LP
If a line begins with the word "Include" followed by whitespace
and then a file name, that file will be included inline with the current
configuration file.
.LP
The overall configuration parameters available include:

.TP
\fBAccountingStorageEnforce\fR
If set to a non-zero value and the user, partition, account association is not 
defined for a job in the accounting database then prevent the job from being 
executed. This needs to be set to '2' if you the association limits will also 
be enforced.  If set to anything else limits of associations will not be 
enforced.
The default value is zero.

.TP
\fBAccountingStorageHost\fR
Define the name of the host where the database is running we are going
to store the accounting data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageHost\fR.

.TP
\fBAccountingStorageLoc\fR
Specifies the location of the file or database where accounting 
records are written.
Also see \fBDefaultStorageLoc\fR.

.TP
\fBAccountingStoragePass\fR
Define the password used to gain access to the database to store the
accounting data.
Only used for database type storage plugins, ignored otherwise.
In the case of Slurm DBD (Data Base Daemon) with Munge authentication this can be 
configured to use a Munge daemon specifically configured to provide authentication 
between clusters while the default Munge daemon provides authentication within a cluster. 
In that case, \fBAccountingStoragePass\fR should specify the named port to be used
for communications with the alternate Munge daemon (e.g. 
"/var/run/munge/global.socket.2"). The default value is NULL.
Also see \fBDefaultStoragePass\fR.

.TP
\fBAccountingStoragePort\fR
Define the port the database server is listening on where we are going
to store the accounting data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePort\fR.

.TP
\fBAccountingStorageType\fR
Define the accounting storage mechanism type.
Acceptable values at present include 
"accounting_storage/mysql", "accounting_storage/none", 
"accounting_storage/pgsql", and "accounting_storage/slurmdbd".
The value "accounting_storage/filetxt" indicates that accounting records
will be written to a the file specified by the 
\fBAccountingStorageLoc\fR parameter.
The value "accounting_storage/mysql" indicates that accounting records
should be written to a MySQL database specified by the 
\fBAccountingStorageLoc\fR parameter.
The default value is "accounting_storage/none", which means that
account records are not maintained. 
The value "accounting_storage/pgsql" indicates that accounting records
should be written to a PostgreSQL database specified by the 
\fBAccountingStorageLoc\fR parameter.  This plugin is not complete and 
should not be used if wanting to use associations.  It will however work with
basic accounting of jobs and job steps.  If interested in 
completing please email slurm-dev@lists.llnl.gov.
The value "accounting_storage/slurmdbd" indicates that accounting records
will be written to SlurmDDB, which manages an underlying MySQL or 
PostgreSQL database. See "man slurmdbd" for more information.
Also see \fBDefaultStorageType\fR.

.TP
\fBAccountingStorageUser\fR
Define the name of the user we are going to connect to the database
with to store the accounting data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageUser\fR.

.TP
\fBAuthType\fR
Define the authentication method for communications between SLURM 
components. 
Acceptable values at present include "auth/none", "auth/authd", 
and "auth/munge".
The default value is "auth/munge".
"auth/nume includes the UID in each communication, but it is not verified. 
This may be fine for testing purposes, but 
\fBdo not use "auth/none" if you desire any security\fR.
"auth/authd" indicates that Brett Chun's authd is to be used (see
"http://www.theether.org/authd/" for more information, Note that
authd is no longer actively supported).
"auth/munge" indicates that LLNL's MUNGE is to be used
(this is the best supported authentication mechanism for SLURM, 
see "http://home.gna.org/munge/" for more information).
All SLURM daemons and commands must be terminated prior to changing 
the value of \fBAuthType\fR and later restarted (SLURM jobs can be 
preserved).

.TP
\fBBackupAddr\fR
Name that \fBBackupController\fR should be referred to in 
establishing a communications path. This name will 
be used as an argument to the gethostbyname() function for 
identification. For example, "elx0000" might be used to designate 
the Ethernet address for node "lx0000". 
By default the \fBBackupAddr\fR will be identical in value to 
\fBBackupController\fR.

.TP
\fBBackupController\fR
The name of the machine where SLURM control functions are to be 
executed in the event that \fBControlMachine\fR fails. This node
may also be used as a compute server if so desired. It will come into service 
as a controller only upon the failure of ControlMachine and will revert 
to a "standby" mode when the ControlMachine becomes available once again. 
This should be a node name without the full domain name (e.g. "lx0002"). 
While not essential, it is recommended that you specify a backup controller.
See  the \fBRELOCATING CONTROLLERS\fR section if you change this.

.TP
\fBBatchStartTimeout\fR
The maximum time (in seconds) that a batch job is permitted for 
launching before being considered missing and releasing the 
allocation. The default value is 10 (seconds). Larger values may 
be required if more time is required to execute the \fBProlog\fR, 
loading user environment variables (for Moab spawned jobs), or the 
slurmd daemon gets paged from memory.

.TP
\fBCacheGroups\fR
If set to 1, the slurmd daemon will  cache /etc/groups entries.
This can improve performance for highly parallel jobs if NIS servers
are used and unable to respond very quickly.
The default value is 0 to disable caching group data.

.TP
\fBCheckpointType\fR
Define the system\-initiated checkpoint method to be used for user jobs. 
The slurmctld daemon must be restarted for a change in \fBCheckpointType\fR 
to take effect. 
Acceptable values at present include
"checkpoint/aix" (only on AIX systems),
"checkpoint/ompi" (requires OpenMPI version 1.3 or higher),
"checkpoint/xlch" (for XLCH, requires that SlurmUser be root), and
"checkpoint/none".
The default value is "checkpoint/none".

.TP
\fBClusterName\fR
The name by which this SLURM managed cluster is known for accounting 
purposes. This is needed distinguish between accounting data from 
multiple clusters being recorded in a single database.

.TP
\fBCompleteWait\fR
The time, in seconds, given for a job to remain in COMPLETING state
before any any additional jobs are scheduled. 
If set to zero, pending jobs will be started as soon as possible.
Since a COMPLETING job's resources are released for use by other 
jobs as soon as the \fBEpilog\fR completes on each individual node, 
this can result in very fragmented resource allocations. 
To provide jobs with the minimum response time, a value of zero is 
recommended (no waiting).
To minimize fragmentation of resources, a value equal to \fBKillWait\fR 
plus two is recommended. 
In that case, setting \fBKillWait\fR to a small value may be beneficial.
The default value of \fBCompleteWait\fR is zero seconds.
The value may not exceed 65533.

.TP
\fBControlAddr\fR
Name that \fBControlMachine\fR should be referred to in 
establishing a communications path. This name will 
be used as an argument to the gethostbyname() function for 
identification. For example, "elx0000" might be used to designate 
the Ethernet address for node "lx0000". 
By default the \fBControlAddr\fR will be identical in value to 
\fBControlMachine\fR.

.TP
\fBControlMachine\fR
The name of the machine where SLURM control functions are executed
as returned by the \fIgethostname()\fR function the cut at the first dot
or the \fIhostname \-s\fR command (e.g. use "tux001" rather than "tux001.my.com"). 
This value must be specified.
See the \fBRELOCATING CONTROLLERS\fR section if you change this.

.TP
\fBCryptoType\fR
Define the cryptographic signature tool to be used in the creation of 
job step credentials.
The slurmctld daemon must be restarted for a change in \fBCryptoType\fR
to take effect.
Acceptable values at present include "crypto/munge" and "crypto/openssl".
The default value is "crypto/munge".

.TP
\fBDebugFlags\fR
Define specific subsystems which should provide more detailed event logging.
Multiple subsystems can be specified with comma separators. 
Valid subsystems available today (with more to come) include:
.RS
.TP 15
\fBCPU_Bind\fR
CPU binding details for jobs and steps
.TP
\fBSteps\fR
Slurmctld resource allocation for job steps
.TP
\fBTriggers\fR
Slurmctld triggers
.TP
\fBWiki\fR
Sched/wiki and wiki2 communications
.RE

.TP
\fBDefMemPerCPU\fR
Default real memory size available per allocated CPU in MegaBytes. 
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerCPU\fR would generally be used if individual processors
are alocated to jobs (\fBSelectType=select/cons_res\fR). 
The default value is 0 (unlimited).
Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
NOTE: Enforcement of memory limits currently requires enabling of 
accounting, which samples memory use on a periodic basis (data need
not be stored, just collected).

.TP
\fBDefMemPerNode\fR
Default real memory size available per allocated node in MegaBytes.
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerNode\fR would generally be used if whole nodes
are alocated to jobs (\fBSelectType=select/linear\fR) and 
resources are shared (\fBShared=yes\fR or \fBShared=force\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
NOTE: Enforcement of memory limits currently requires enabling of 
accounting, which samples memory use on a periodic basis (data need
not be stored, just collected).

.TP
\fBDefaultStorageHost\fR
Define the name of the host where the database is running and used to
to store the accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStorageHost\fR and \fBJobCompHost\fR.

.TP
\fBDefaultStorageLoc\fR
Specifies the location of the file or database where accounting 
and job completion records are written.
Also see \fBAccountingStorageLoc\fR and \fBJobCompLoc\fR.

.TP
\fBDefaultStoragePass\fR
Define the password used to gain access to the database to store the
accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStoragePass\fR and \fBJobCompPass\fR.

.TP
\fBDefaultStoragePort\fR
Define the port the database server is listening on where we are going
to store the accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStoragePort\fR and \fBJobCompPort\fR.

.TP
\fBDefaultStorageType\fR
Define the accounting and job completion storage mechanism type.
Acceptable values at present include 
"filetxt", "mysql", "none", "pgsql", and "slurmdbd".
The value "filetxt" indicates that records will be written to a the file.
.ad
The value "mysql" indicates that accounting records will be written to 
a mysql database.
The default value is "none", which means that records are not maintained. 
The value "pgsql" indicates that records will be written to a postresql 
database.
The value "slurmdbd" indicates that records will be written to SlurmDbd,
which maintains its own database. See "man slurmdbd for more information".
Also see \fBAccountingStorageType\fR  and \fBJobCompType\fR.

.TP
\fBDefaultStorageUser\fR
Define the name of the user we are going to connect to the database
with to store the accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStorageUser\fR and \fBJobCompUser\fR.

.TP
\fBDisableRootJobs\fR
If set to "YES" then user root will be prevented from running any jobs.
The default value is "NO", meaning user root will be able to execute jobs.
\fBDisableRootJobs\fR may also be set by partition.

.TP
\fBEnforcePartLimits\fR
If set to "YES" then jobs which exceed a partition's size and/or time limits
will be rejected at submission time. If set to "NO" then the job will be 
accepted and remain queued until the partition limits are altered.
The default value is "NO".

.TP
\fBEpilog\fR
Fully qualified pathname of a script to execute as user root on every 
node when a user's job completes (e.g. "/usr/local/slurm/epilog"). This may 
be used to purge files, disable user login, etc. 
By default there is no epilog.
See \fBProlog and Epilog Scripts\fR for more information.

.TP
\fBEpilogMsgTime\fR
The number of microseconds the the slurmctld daemon requires to process
an epilog completion message from the slurmd dameons. This parameter can
be used to prevent a burst of epilog completion messages from being sent 
at the same time which should help prevent lost messages and improve 
throughput for large jobs.
The default value is 2000 microseconds.
For a 1000 node job, this spreads the epilog completion messages out over
two seconds.

.TP
\fBEpilogSlurmctld\fR
Fully qualified pathname of a program for the slurmctld to execute 
upon termination of a job allocation (e.g.
"/usr/local/slurm/epilog_controller"). 
The program executes as SlurmUser, which gives it permission to drain 
nodes and requeue the job if a failure occurs or cancel the job if appropriate.
The program can be used to reboot nodes or perform other work to prepare 
resources for use. 
See \fBProlog and Epilog Scripts\fR for more information.

.TP
\fBFastSchedule\fR
Controls how a nodes configuration specifications in slurm.conf are used.
If the number of node configuration entries in the configuration file
is significantly lower than the number of nodes, setting FastSchedule to
1 will permit much faster scheduling decisions to be made.
(The scheduler can just check the values in a few configuration records
instead of possibly thousands of node records.)
Note that on systems with hyper\-threading, the processor count
reported by the node will be twice the actually processor count.
Consider which value you want to be used for scheduling purposes.
.RS
.TP 5
\fB1\fR (default)
Consider the configuration of each node to be that specified in the
configuration file and any node with less than the configured resources 
will be set DOWN.
.TP
\fB0\fR
Base scheduling decisions upon the actual configuration of each individual 
node except that the node's processor count in SLURM's configuration must
match the actual hardware configuration if \fBSchedulerType=sched/gang\fR 
or \fBSelectType=select/cons_res\fR are configured (both of those plugins
maintain resource allocation information using bitmaps for the cores in the
system and must remain static, while the node's memory and disk space can
be esblished later).
.TP
\fB2\fR
Consider the configuration of each node to be that specified in the 
slurm.conf configuration file and any node with less resources 
than configured will \fBnot\fR be set DOWN. 
This can be useful for testing purposes.
.RE

.TP
\fBFirstJobId\fR
The job id to be used for the first submitted to SLURM without a 
specific requested value. Job id values generated will incremented by 1 
for each subsequent job. This may be used to provide a meta\-scheduler 
with a job id space which is disjoint from the interactive jobs. 
The default value is 1.

.TP
\fBGetEnvTimeout\fR
Used for Moab scheduled jobs only. Controls how long job should wait
in seconds for loading the user's environment before attempting to 
load it from a cache file. Applies when the srun or sbatch 
\fI--get-user-env\fR option is used. If set to 0 then always load 
the user's environment from the cache file.
The default value is 2 seconds.

.TP
\fBHealthCheckInterval\fR
The interval in seconds between executions of \fBHealthCheckProgram\fR.
The default value is zero, which disables execution.

.TP
\fBHealthCheckProgram\fR
Fully qualified pathname of a script to execute as user root periodically 
on all compute nodes that are not in the DOWN state. This may be used to 
verify the node is fully operational and DRAIN the node or send email
if a problem is detected. 
Any action to be taken must be explicitly performed by the program
(e.g. execute "scontrol update NodeName=foo State=drain Reason=tmp_file_system_full"
to drain a node).
The interval is controlled using the \fBHealthCheckInterval\fR parameter.
Note that the \fBHealthCheckProgram\fR will be executed at the same time 
on all nodes to minimize its impact upon parallel programs.
This program is will be killed if it does not terminate normally within
60 seconds. 
By default, no program will be executed.

.TP
\fBInactiveLimit\fR
The interval, in seconds, a job or job step is permitted to be inactive
before it is terminated. A job or job step is considered inactive if 
the associated srun command is not responding to slurm daemons. This 
could be due to the termination of the srun command or the program 
being is a stopped state. A batch job is considered inactive if it 
has no active job steps (e.g. periods of pre\- and post\-processing).
This limit permits defunct jobs to be purged in a timely fashion 
without waiting for their time limit to be reached.
This value should reflect the possibility that the srun command may
stopped by a debugger or considerable time could be required for batch 
job pre\- and post\-processing. 
This limit is ignored for jobs running in partitions with the 
\fBRootOnly\fR flag set (the scheduler running as root will be 
responsible for the job).
The default value is unlimited (zero). 
May not exceed 65533.

.TP
\fBJobAcctGatherType\fR
Define the job accounting mechanism type.
Acceptable values at present include "jobacct_gather/aix" (for AIX operating
system), "jobacct_gather/linux" (for Linux operating system) and "jobacct_gather/none"
(no accounting data collected).
The default value is "jobacct_gather/none".
In order to use the \fBsacct\fR tool, "jobacct_gather/aix" or "jobacct_gather/linux" 
must be configured.

.TP
\fBJobAcctGatherFrequency\fR
Define the job accounting sampling interval.
For jobacct_gather/none this parameter is ignored.
For  jobacct_gather/aix and jobacct_gather/linux the parameter is a number is 
seconds between sampling job state.
The default value is 30 seconds. 
A value of zero disables real the periodic job sampling and provides accounting 
information only on job termination (reducing SLURM interference with the job).

.TP
\fBJobCheckpointDir\fR
Set the default directory used to store job checkpoint files.
The default value is "/var/slurm/checkpoint".

.TP
\fBJobCompHost\fR
Define the name of the host where the database is running and used
to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageHost\fR.

.TP
\fBJobCompLoc\fR
The interpretation of this value depends upon the logging mechanism 
specified by the \fBJobCompType\fR parameter either a filename or a 
database name. 
Also see \fBDefaultStorageLoc\fR.

.TP
\fBJobCompPass\fR
Define the password used to gain access to the database to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePass\fR.

.TP
\fBJobCompPort\fR
Define the port the database server is listening on where we are going
to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePort\fR.

.TP
\fBJobCompType\fR
Define the job completion logging mechanism type.
Acceptable values at present include "jobcomp/none", "jobcomp/filetxt", 
"jobcomp/mysql", "jobcomp/pgsql", and "jobcomp/script"".
The default value is "jobcomp/none", which means that upon job completion 
the record of the job is purged from the system.  If using the accounting 
infrastructure this plugin may not be of interest since the information 
here is redundant.
The value "jobcomp/filetxt" indicates that a record of the job should be 
written to a text file specified by the \fBJobCompLoc\fR parameter.
The value "jobcomp/mysql" indicates that a record of the job should be 
written to a mysql database specified by the \fBJobCompLoc\fR parameter.
The value "jobcomp/pgsql" indicates that a record of the job should be 
written to a postgresql database specified by the \fBJobCompLoc\fR parameter.
The value "jobcomp/script" indicates that a script specified by the 
\fBJobCompLoc\fR parameter is to be executed with environment variables 
indicating the job information.

.TP
\fBJobCompUser\fR
Define the name of the user we are going to connect to the database
with to store the job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageUser\fR.

.TP
\fBJobCredentialPrivateKey\fR
Fully qualified pathname of a file containing a private key used for 
authentication by Slurm daemons.
This parameter is ignored if \fBCryptType=munge\fR.

.TP
\fBJobCredentialPublicCertificate\fR
Fully qualified pathname of a file containing a public key used for 
authentication by Slurm daemons.
This parameter is ignored if \fBCryptType=munge\fR.

.TP
\fBJobFileAppend\fR
This option controls what to do if a job's output or error file 
exist when the job is started. 
If \fBJobFileAppend\fR is set to a value of 1, then append to 
the existing file.
By default, any existing file is truncated.

.TP
\fBJobRequeue\fR
This option controls what to do by default after a node failure. 
If \fBJobRequeue\fR is set to a value of 1, then any job running 
on the failed node will be requeued for execution on different nodes.
If \fBJobRequeue\fR is set to a value of 0, then any job running 
on the failed node will be terminated.
Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR 
option to change the default behavior for individual jobs.
The default value is 1.

.TP
\fBKillOnBadExit\fR
If set to 1, the job will be terminated immediately when one of the 
processes is crashed or aborted. With default value of 0, if one of 
the processes is crashed or aborted the other processes will continue 
to run.

.TP
\fBKillWait\fR
The interval, in seconds, given to a job's processes between the 
SIGTERM and SIGKILL signals upon reaching its time limit. 
If the job fails to terminate gracefully in the interval specified, 
it will be forcibly terminated. 
The default value is 30 seconds.
The value may not exceed 65533.

.TP
\fBLicenses\fR
Specification of licenses (or other resources available on all 
nodes of the cluster) which can be allocated to jobs.
License names can optionally be followed by an asterisk 
and count with a default count of one.
Multiple license names should be comma separated (e.g.
"Licenses=foo*4,bar").
Note that SLURM prevents jobs from being scheduled if their 
required license specification is not available.
SLURM does not prevent jobs from using licenses that are
not explicitly listed in the job submission specification.

.TP
\fBMailProg\fR
Fully qualified pathname to the program used to send email per user request.
The default value is "/bin/mail".

.TP
\fBMaxJobCount\fR
The maximum number of jobs SLURM can have in its active database 
at one time. Set the values of \fBMaxJobCount\fR and \fBMinJobAge\fR 
to insure the slurmctld daemon does not exhaust its memory or other 
resources. Once this limit is reached, requests to submit additional 
jobs will fail. The default value is 5000 jobs. This value may not 
be reset via "scontrol reconfig". It only takes effect upon restart 
of the slurmctld daemon.
May not exceed 65533.

.TP
\fBMaxMemPerCPU\fR
Maximum real memory size available per allocated CPU in MegaBytes. 
Used to avoid over\-subscribing memory and causing paging.
\fBMaxMemPerCPU\fR would generally be used if individual processors
are alocated to jobs (\fBSelectType=select/cons_res\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
\fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.
NOTE: Enforcement of memory limits currently requires enabling of 
accounting, which samples memory use on a periodic basis (data need
not be stored, just collected).

.TP
\fBMaxMemPerNode\fR
Maximum real memory size available per allocated node in MegaBytes.
Used to avoid over\-subscribing memory and causing paging.
\fBMaxMemPerNode\fR would generally be used if whole nodes
are alocated to jobs (\fBSelectType=select/linear\fR) and
resources are shared (\fBShared=yes\fR or \fBShared=force\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
\fBMaxMemPerCPU\fR and \fBMaxMemPerNode\fR are mutually exclusive.
NOTE: Enforcement of memory limits currently requires enabling of 
accounting, which samples memory use on a periodic basis (data need
not be stored, just collected).

.TP
\fBMessageTimeout\fR
Time permitted for a round\-trip communication to complete
in seconds. Default value is 10 seconds. For systems with 
shared nodes, the slurmd daemon could be paged out and 
necessitate higher values.

.TP
\fBMinJobAge\fR
The minimum age of a completed job before its record is purged from 
SLURM's active database. Set the values of \fBMaxJobCount\fR and 
\fBMinJobAge\fR to insure the slurmctld daemon does not exhaust 
its memory or other resources. The default value is 300 seconds. 
A value of zero prevents any job record purging.
May not exceed 65533.

.TP
\fBMpiDefault\fR
Identifies the default type of MPI to be used. 
Srun may override this configuration parameter in any case.
Currently supported versions include: 
\fBmpichgm\fR, 
\fBmvapich\fR,
\fBnone\fR (default, which works for many other versions of MPI including 
LAM MPI and Open MPI).

.TP
\fBMpiParams\fR
MPI parameters. 
Used to identify ports used by OpenMPI only and the input format is
"ports=12000\-12999" to identify a range of communcation ports to be used.

.TP
\fBOverTimeLimit\fR
Number of minutes by which a job can exceed its time limit before 
being cancelled. 
The configured job time limit is treated as a \fIsoft\fR limit.
Adding \fBOverTimeLimit\fR to the \fIsoft\fR limit provides a \fIhard\fR
limit, at which point the job is cancelled.
This is particularly useful for backfill scheduling, which bases upon
each job's soft time limit.
The default value is zero.
Man not exceed exceed 65533 minutes.
A value of "UNLIMITED" is also supported.

.TP
\fBPluginDir\fR
Identifies the places in which to look for SLURM plugins. 
This is a colon\-separated list of directories, like the PATH 
environment variable. 
The default value is "/usr/local/lib/slurm".

.TP
\fBPlugStackConfig\fR
Location of the config file for SLURM stackable plugins that use
the Stackable Plugin Architecture for Node job (K)control (SPANK).
This provides support for a highly configurable set of plugins to
be called before and/or after execution of each task spawned as
part of a user's job step.  Default location is "plugstack.conf"
in the same directory as the system slurm.conf. For more information
on SPANK plugins, see the \fBspank\fR(8) manual.

.TP
\fBPriorityDecayHalfLife\fR
This controls how long prior resource use is considered in determining
how over\- or under\-serviced an association is (user, bank account and 
cluster) in determining job priority.
Applicable only if PriorityType=priority/multifactor.
The units are minutes and the default value is 7 days.

.TP
\fBPriorityFavorSmall\fR
Specifies that small jobs should be given preferencial scheduling priority.
Applicable only if PriorityType=priority/multifactor.
Supported values are "YES" and "NO". 
The default value is "NO".

.TP
\fBPriorityMaxAge\fR
Specifies the job age which will be given the maximum age factor in computing
priority. For example, a value of 30 minutes would result in all jobs over
30 minutes old would get the same age\-based priority.
Applicable only if PriorityType=priority/multifactor.
The units are minutes and the default value is 7 days.

.TP
\fBPriorityType\fR
This specifies the plugin to be used in establishing a job's scheduling
priority. Supported values are "priority/basic" (jobs are prioritized
by order of arrival, also suitable for sched/wiki and sched/wiki2) and
"priority/multifactor" (jobs are prioritized based upon size, age, 
fair\-share of allocation, etc). 
The default value is "priority/basic".

.TP
\fBPriorityWeightAge\fR
An integer value specifying how important a job's age is in computing 
its priority.
Applicable only if PriorityType=priority/multifactor.
The default value is 1.

.TP
\fBPriorityWeightFairshare\fR
An integer value specifying how important a job's age is in computing 
its priority.
Applicable only if PriorityType=priority/multifactor.
The default value is 1.

.TP
\fBPriorityWeightJobSize\fR
An integer value specifying how important a job's size is in computing 
its priority.
Applicable only if PriorityType=priority/multifactor.
The default value is 1.

.TP
\fBPriorityWeightPartition\fR
An integer value specifying how important a job's partition is in computing 
its priority.
Applicable only if PriorityType=priority/multifactor.
The default value is 1.

.TP
\fBPriorityWeightQOS\fR
An integer value specifying how important a job's Quality Of Service is 
in computing its priority.
Applicable only if PriorityType=priority/multifactor.
The default value is 1.

.TP
\fBPrivateData\fR
This controls what type of information is hidden from regular users.
By default, all information is visible to all users.
User \fBSlurmUser\fR and \fBroot\fR can always view all information.
Multiple values may be specified with a comma separator.
Acceptable values include:
.RS
.TP
\fBaccounts\fR 
(NON-SLURMDBD ACCOUNTING ONLY) prevents users from viewing any account 
definitions unless they are coordinators of them.
.TP
\fBjobs\fR
prevents users from viewing jobs or job steps belonging
to other users. (NON-SLURMDBD ACCOUNTING ONLY) prevents users from viewing 
job records belonging to other users unless they are coordinators of 
the association running the job when using sacct.
.TP
\fBnodes\fR
prevents users from viewing node state information.
.TP
\fBpartitions\fR 
prevents users from viewing partition state information.
.TP
\fBusers\fR 
(NON-SLURMDBD ACCOUNTING ONLY) prevents users from viewing 
information of any user other than themselves, this also makes it so users can 
only see associations they deal with.  
Coordinators can see associations of all users they are coordinator of, 
but can only see themselves when listing users.
.TP
\fBusage\fR 
(NON-SLURMDBD ACCOUNTING ONLY) prevents users from viewing 
usage of any other user.  This applys to sreport.
.RE


.TP
\fBProctrackType\fR
Identifies the plugin to be used for process tracking. 
The slurmd daemon uses this mechanism to identify all processes 
which are children of processes it spawns for a user job. 
The slurmd daemon must be restarted for a change in ProctrackType
to take effect.
NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to 
identify all processes associated with a job since processes 
can become a child of the init process (when the parent process 
terminates) or change their process group. 
To reliably track all processes, one of the other mechanisms 
utilizing kernel modifications is preferable. 
NOTE: "proctrack/linuxproc" is not compatible with "switch/elan."
Acceptable values at present include:
.RS
.TP 
\fBproctrack/aix\fR which uses an AIX kernel extension and is 
the default for AIX systems
.TP
\fBproctrack/linuxproc\fR which uses linux process tree using 
parent process IDs
.TP
\fBproctrack/rms\fR which uses Quadrics kernel patch and is the 
default if "SwitchType=switch/elan" 
.TP
\fBproctrack/sgi_job\fR which uses SGI's Process Aggregates (PAGG)
kernel module, see \fIhttp://oss.sgi.com/projects/pagg/\fR 
for more information 
.TP
\fBproctrack/pgid\fR which uses process group IDs and is the 
default for all other systems
.RE

.TP
\fBProlog\fR
Fully qualified pathname of a program for the slurmd to execute 
whenever it is asked to run a job step from a new job allocation (e.g.
"/usr/local/slurm/prolog").  The slurmd executes the script before starting
the first job step.  This may be used to purge files, enable user login, etc.
By default there is no prolog. Any configured script is expected to 
complete execution quickly (in less time than \fBMessageTimeout\fR).
See \fBProlog and Epilog Scripts\fR for more information.

.TP
\fBPrologSlurmctld\fR
Fully qualified pathname of a program for the slurmctld to execute 
before granting a new job allocation (e.g.
"/usr/local/slurm/prolog_controller"). 
The program executes as SlurmUser, which gives it permission to drain 
nodes and requeue the job if a failure occurs or cancel the job if appropriate.
The program can be used to reboot nodes or perform other work to prepare 
resources for use. 
See \fBProlog and Epilog Scripts\fR for more information.

.TP
\fBPropagatePrioProcess\fR
Setting \fBPropagatePrioProcess\fR to "1", will cause a users job to run
with the same priority (aka nice value) as the users process which
launched the job on the submit node.
If set to "0", or left unset, the users job will inherit the
scheduling priority from the slurm daemon.

.TP
\fBPropagateResourceLimits\fR
A list of comma separated resource limit names.
The slurmd daemon uses these names to obtain the associated (soft) limit
values from the users process environment on the submit node.
These limits are then propagated and applied to the jobs that
will run on the compute nodes. 
This parameter can be useful when system limits vary among nodes.
Any resource limits that do not appear in the list are not propagated.
However, the user can override this by specifying which resource limits
to propagate with the srun commands "\-\-propagate" option.
If neither of the 'propagate resource limit' parameters are specified, then
the default action is to propagate all limits.
Only one of the parameters, either
\fBPropagateResourceLimits\fR or \fBPropagateResourceLimitsExcept\fR,
may be specified.
The following limit names are supported by Slurm (although some 
options may not be supported on some systems):
.RS
.TP 10
\fBALL\fR
All limits listed below
.TP
\fBNONE\fR
No limits listed below
.TP
\fBAS\fR
The maximum address space for a processes
.TP
\fBCORE\fR
The maximum size of core file
.TP
\fBCPU\fR
The maximum amount of CPU time
.TP
\fBDATA\fR
The maximum size of a process's data segment
.TP
\fBFSIZE\fR
The maximum size of files created
.TP
\fBMEMLOCK\fR
The maximum size that may be locked into memory
.TP
\fBNOFILE\fR
The maximum number of open files
.TP
\fBNPROC\fR
The maximum number of processes available
.TP
\fBRSS\fR
The maximum resident set size
.TP
\fBSTACK\fR
The maximum stack size
.RE

.TP
\fBPropagateResourceLimitsExcept\fR
A list of comma separated resource limit names.
By default, all resource limits will be propagated, (as described by
the \fBPropagateResourceLimits\fR parameter), except for the limits
appearing in this list.   The user can override this by specifying which
resource limits to propagate with the srun commands "\-\-propagate" option.
See \fBPropagateResourceLimits\fR above for a list of valid limit names.

.TP
\fBResumeProgram\fR
SLURM supports a mechanism to reduce power consumption on nodes that 
remain idle for an extended period of time. 
This is typically accomplished by reducing voltage and frequency or powering
the node down. 
\fBResumeProgram\fR is the program that will be executed when a node 
in power save mode is assigned work to perform.
For reasons of reliability, \fBResumeProgram\fR may execute more than once
for a node.
If \fBResumeProgram\fR is unable to restore a node to service, it should
requeue any node associated with the node and set the node state to DRAIN.
The program executes as \fBSlurmUser\fR.
The argument to the program will be the names of nodes to
be removed from power savings mode (using SLURM's hostlist
expression format).
By default no program is run.
Related configuration options include \fBResumeRate\fR, \fBSuspendRate\fR,
\fBSuspendTime\fR, \fBSuspendProgram\fR, \fBSuspendExcNodes\fR, and
\fBSuspendExcParts\fR.
More information is available at the SLURM web site
(https://computing.llnl.gov/linux/slurm/power_save.html).

.TP
\fBResumeRate\fR
The rate at which nodes in power save mode are returned to normal 
operation by \fBResumeProgram\fR. 
The value is number of nodes per minute and it can be used to prevent 
power surges if a large number of nodes in power save mode are 
assigned work at the same time (e.g. a large job starts).
A value of zero results in no limits being imposed.
The default value is 300 nodes per minute.
Related configuration options include \fBResumeProgram\fR, \fBSuspendRate\fR,
\fBSuspendTime\fR, \fBSuspendProgram\fR, \fBSuspendExcNodes\fR, and
\fBSuspendExcParts\fR.

.TP
\fBResvOverRun\fR
Describes how long a job already running in a reservation should be 
permitted to execute after the end time of the reservation has been 
reached. 
The time period is specified in minutes and the default value is 0
(kill the job immediately).
The value may not exceed 65533 minutes, although a value of "UNLIMITED"
is supported to permit a job to run indefinitely after its reservation
is terminated.

.TP
\fBReturnToService\fR
Controls when a DOWN node will be returned to service. 
The default value is 0.
Supported values include
.RS
.TP 4
\fB0\fR
A node will remain in the DOWN state until a system administrator
explicitly changes its state (even if the slurmd daemon registers
and resumes communications).
.TP
\fB1\fR
A non\-responding (DOWN) node will become available for use upon
registration. Note that DOWN node's state will be changed only if