Newer
Older
.TH "slurm.conf" "5" "October 2013" "slurm.conf 13.12" "Slurm configuration file"
\fBslurm.conf\fP is an ASCII file which describes general SLURM
configuration information, the nodes to be managed, information about
how those nodes are grouped into partitions, and various scheduling
parameters associated with those partitions. This file should be
consistent across all nodes in the cluster.
The file location can be modified at system build time using the
DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF
environment variable. The SLURM daemons also allow you to override
both the built\-in and environment\-provided location using the "\-f"
option on the command line.
The contents of the file are case insensitive except for the names of nodes
and partitions. Any text following a "#" in the configuration file is treated
as a comment through the end of that line.
Changes to the configuration file take effect upon restart of
SLURM daemons, daemon receipt of the SIGHUP signal, or execution
of the command "scontrol reconfigure" unless otherwise noted.
If a line begins with the word "Include" followed by whitespace
Christopher J. Morrone
committed
and then a file name, that file will be included inline with the current
configuration file.
.LP
Note on file permissions:
.LP
The \fIslurm.conf\fR file must be readable by all users of SLURM, since it
is used by many of the SLURM commands. Other files that are defined
in the \fIslurm.conf\fR file, such as log files and job accounting files,
may need to be created/owned by the user "SlurmUser" to be successfully
accessed. Use the "chown" and "chmod" commands to set the ownership
and permissions appropriately.
See the section \fBFILE AND DIRECTORY PERMISSIONS\fR for information
about the various files and directories used by SLURM.
.SH "PARAMETERS"
.LP
The overall configuration parameters available include:
.TP
\fBAccountingStorageBackupHost\fR
The name of the backup machine hosting the accounting storage database.
If used with the accounting_storage/slurmdbd plugin, this is where the backup
slurmdbd would be running.
Only used for database type storage plugins, ignored otherwise.
This controls what level of association\-based enforcement to impose
on job submissions. Valid options are any combination of
\fIassociations\fR, \fIlimits\fR, \fInojobs\fR, \fInosteps\fR, \fIqos\fR, \fIsafe\fR, and \fIwckeys\fR, or
\fIall\fR for all things.
If limits, qos, or wckeys are set, associations will automatically be set.
If wckeys is set, TrackWCKey will automatically be set.
If safe is set, limits and associations will automatically be set.
If nojobs is set nosteps will automatically be set.
By enforcing Associations no new job is allowed to run unless a corresponding
association exists in the system. If limits are enforced users can be
limited by association to whatever job size or run time limits are defined.
If nojobs is set Slurm will not account for any jobs or steps on the system,
like wise if nosteps is set Slurm will not account for any steps ran limits
will still be enforced.
If safe is enforced a job will only be launched against an association or qos
that has a GrpCPUMins limit set if the job will be able to run to completion.
Without this option set, jobs will be launched as long as their usage
hasn't reached the cpu-minutes limit which can lead to jobs being
launched but then killed when the limit is reached.
With qos and/or wckeys enforced jobs will not be scheduled unless a valid qos
and/or workload characterization key is specified.
When \fBAccountingStorageEnforce\fR is changed, a restart of the slurmctld
daemon is required (not just a "scontrol reconfig").
The name of the machine hosting the accounting storage database.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageHost\fR.
.TP
\fBAccountingStorageLoc\fR
The fully qualified file name where accounting records are written
when the \fBAccountingStorageType\fR is "accounting_storage/filetxt"
or else the name of the database where accounting records are stored when the
\fBAccountingStorageType\fR is a database.
Also see \fBDefaultStorageLoc\fR.
.TP
\fBAccountingStoragePass\fR
The password used to gain access to the database to store the
accounting data. Only used for database type storage plugins, ignored
otherwise. In the case of SLURM DBD (Database Daemon) with MUNGE
authentication this can be configured to use a MUNGE daemon
specifically configured to provide authentication between clusters
while the default MUNGE daemon provides authentication within a
cluster. In that case, \fBAccountingStoragePass\fR should specify the
named port to be used for communications with the alternate MUNGE
daemon (e.g. "/var/run/munge/global.socket.2"). The default value is
NULL. Also see \fBDefaultStoragePass\fR.
The listening port of the accounting storage database server.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStoragePort\fR.
.TP
\fBAccountingStorageType\fR
The accounting storage mechanism type. Acceptable values at
present include "accounting_storage/filetxt",
"accounting_storage/mysql", "accounting_storage/none"
and "accounting_storage/slurmdbd". The
"accounting_storage/filetxt" value indicates that accounting records
will be written to the file specified by the
\fBAccountingStorageLoc\fR parameter. The "accounting_storage/mysql"
value indicates that accounting records will be written to a MySQL or
MariaDB database specified by the \fBAccountingStorageLoc\fR parameter.
The "accounting_storage/slurmdbd" value indicates that accounting records
will be written to the SLURM DBD, which manages an underlying MySQL
database. See "man slurmdbd" for more information. The
default value is "accounting_storage/none" and indicates that account
records are not maintained.
Note: The filetxt plugin records only a limited subset of accounting
information and will prevent some sacct options from proper operation.
Also see \fBDefaultStorageType\fR.
The user account for accessing the accounting storage database.
Only used for database type storage plugins, ignored otherwise.
Also see \fBDefaultStorageUser\fR.
.TP
\fBAccountingStoreJobComment\fR
If set to "YES" then include the job's comment field in the job
complete message sent to the Accounting Storage database. The default
is "YES".
.TP
\fBAcctGatherNodeFreq\fR
The AcctGather plugins sampling interval for node accounting.
For AcctGather plugin values of none, this parameter is ignored.
of seconds between node accounting samples. For the
acct_gather_energy/rapl plugin, set a value less
than 300 because the counters may overflow beyond this rate.
The default value is zero. This value disables accounting sampling
for nodes. Note: The accounting sampling interval for jobs is
determined by the value of \fBJobAcctGatherFrequency\fR.
.TP
\fBAcctGatherEnergyType\fR
Identifies the plugin to be used for energy consumption accounting.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
energy consumption data for jobs and nodes. The collection of energy
consumption data takes place on node level, hence only in case of exclusive
job allocation the energy consumption measurements will reflect the jobs
real consumption. In case of node sharing between jobs the reported consumed
energy per job (through sstat or sacct) will not reflect the real energy
consumed by the jobs.
Configurable values at present are:
.RS
.TP 20
\fBacct_gather_energy/none\fR
No energy consumption data is collected.
.TP
\fBacct_gather_energy/ipmi\fR
Energy consumption data is collected from the Baseboard Management Controller
(BMC) using the Intelligent Platform Management Interface (IPMI).
.TP
\fBacct_gather_energy/rapl\fR
Energy consumption data is collected from hardware sensors using the Running
Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the
execution of the command "sudo modprobe msr".
Yiannis Georgiou
committed
.TP
\fBAcctGatherInfinibandType\fR
Identifies the plugin to be used for infiniband network traffic accounting.
The plugin is activated only when profiling on hdf5 files is activated and
the user asks for network data collection for jobs through --profile=Network
(or =All). The collection of network traffic data takes place on node level,
hence only in case of exclusive job allocation the collected values will
reflect the jobs real traffic. All network traffic data are logged on hdf5 files
per job on each node. No storage on the Slurm database takes place.
Configurable values at present are:
.RS
.TP 20
\fBacct_gather_infiniband/none\fR
No infiniband network data are collected.
.TP
\fBacct_gather_infiniband/ofed\fR
Infiniband network traffic data are collected from the hardware monitoring
counters of Infiniband devices through the OFED library.
.RE
.TP
\fBAcctGatherFilesystemType\fR
Identifies the plugin to be used for filesystem traffic accounting.
The plugin is activated only when profiling on hdf5 files is activated and
the user asks for filesystem data collection for jobs through \-\-profile=Network
(or =All). The collection of filesystem traffic data takes place on node level,
hence only in case of exclusive job allocation the collected values will
reflect the jobs real traffic. All filesystem traffic data are logged on hdf5 files
per job on each node. No storage on the Slurm database takes place.
Configurable values at present are:
.RS
.TP 20
\fBacct_gather_filesystem/none\fR
No filesystem data are collected.
.TP
\fBacct_gather_filesystem/lustre\fR
Lustre filesystem traffic data are collected from the counters found in
Yiannis Georgiou
committed
.TP
\fBAcctGatherProfileType\fR
Identifies the plugin to be used for detailed job profiling.
The jobacct_gather plugin and slurmd daemon call this plugin to collect
Yiannis Georgiou
committed
detailed data such as I/O counts, memory usage, or energy consumption for jobs
and nodes. There are interfaces in this plugin to collect data as step start
and completion, task start and completion, and at the account gather
Yiannis Georgiou
committed
frequency. The data collected at the node level is related to jobs only in
case of exclusive job allocation.
Configurable values at present are:
.RS
.TP 20
\fBacct_gather_profile/none\fR
No profile data is collected.
.TP
\fBacct_gather_profile/hdf5\fR
This enables the HDF5 plugin. The directory where the profile files
are stored and which values are collected are configured in the
acct_gather.conf file.
The authentication method for communications between SLURM
components.
Acceptable values at present include "auth/none", "auth/authd",
and "auth/munge".
The default value is "auth/munge".
"auth/none" includes the UID in each communication, but it is not verified.
This may be fine for testing purposes, but
\fBdo not use "auth/none" if you desire any security\fR.
"auth/authd" indicates that Brett Chun's authd is to be used (see
"http://www.theether.org/authd/" for more information. Note that
authd is no longer actively supported).
"auth/munge" indicates that LLNL's MUNGE is to be used
(this is the best supported authentication mechanism for SLURM,
see "http://munge.googlecode.com/" for more information).
All SLURM daemons and commands must be terminated prior to changing
the value of \fBAuthType\fR and later restarted (SLURM jobs can be
preserved).
\fBBackupAddr\fR
The name that \fBBackupController\fR should be referred to in
establishing a communications path. This name will
be used as an argument to the gethostbyname() function for
identification. For example, "elx0000" might be used to designate
the Ethernet address for node "lx0000".
By default the \fBBackupAddr\fR will be identical in value to
\fBBackupController\fR.
The name of the machine where SLURM control functions are to be
executed in the event that \fBControlMachine\fR fails. This node
may also be used as a compute server if so desired. It will come into service
as a controller only upon the failure of ControlMachine and will revert
to a "standby" mode when the ControlMachine becomes available once again.
This should be a node name without the full domain name. I.e., the hostname
returned by the \fIgethostname()\fR function cut at the first dot (e.g. use
"tux001" rather than "tux001.my.com").
While not essential, it is recommended that you specify a backup controller.
See the \fBRELOCATING CONTROLLERS\fR section if you change this.
.TP
\fBBatchStartTimeout\fR
The maximum time (in seconds) that a batch job is permitted for
launching before being considered missing and releasing the
allocation. The default value is 10 (seconds). Larger values may be
required if more time is required to execute the \fBProlog\fR, load
user environment variables (for Moab spawned jobs), or if the slurmd
daemon gets paged from memory.
Christopher J. Morrone
committed
\fBCacheGroups\fR
If set to 1, the slurmd daemon will cache /etc/groups entries.
Christopher J. Morrone
committed
This can improve performance for highly parallel jobs if NIS servers
are used and unable to respond very quickly.
The default value is 0 to disable caching group data.
Christopher J. Morrone
committed
.TP
\fBCheckpointType\fR
The system\-initiated checkpoint method to be used for user jobs.
The slurmctld daemon must be restarted for a change in \fBCheckpointType\fR
to take effect.
Supported values presently include:
.RS
.TP 18
\fBcheckpoint/aix\fR
.TP
\fBcheckpoint/blcr\fR
Berkeley Lab Checkpoint Restart (BLCR).
NOTE: If a file is found at sbin/scch (relative to the SLURM installation
location), it will be executed upon completion of the checkpoint. This can
be a script used for managing the checkpoint files.
NOTE: SLURM's BLCR logic only supports batch jobs.
.TP
\fBcheckpoint/none\fR
no checkpoint support (default)
.TP
\fBcheckpoint/ompi\fR
OpenMPI (version 1.3 or higher)
.TP
\fBcheckpoint/poe\fR
for use with IBM POE (Parallel Operating Environment) only
.RE
The name by which this SLURM managed cluster is known in the
accounting database. This is needed distinguish accounting records
when multiple clusters report to the same database. Because of limitations
in some databases, any upper case letters in the name will be silently mapped
to lower case. In order to avoid confusion, it is recommended that the name
be lower case.
.TP
\fBCompleteWait\fR
The time, in seconds, given for a job to remain in COMPLETING state
before any additional jobs are scheduled.
If set to zero, pending jobs will be started as soon as possible.
Since a COMPLETING job's resources are released for use by other
jobs as soon as the \fBEpilog\fR completes on each individual node,
this can result in very fragmented resource allocations.
To provide jobs with the minimum response time, a value of zero is
To minimize fragmentation of resources, a value equal to \fBKillWait\fR
plus two is recommended.
In that case, setting \fBKillWait\fR to a small value may be beneficial.
The default value of \fBCompleteWait\fR is zero seconds.
The value may not exceed 65533.
\fBControlAddr\fR
Name that \fBControlMachine\fR should be referred to in
establishing a communications path. This name will
be used as an argument to the gethostbyname() function for
identification. For example, "elx0000" might be used to designate
the Ethernet address for node "lx0000".
By default the \fBControlAddr\fR will be identical in value to
\fBControlMachine\fR.
The short hostname of the machine where SLURM control functions are
executed (i.e. the name returned by the command "hostname \-s", use
"tux001" rather than "tux001.my.com").
This value must be specified.
In order to support some high availability architectures, multiple
hostnames may be listed with comma separators and one \fBControlAddr\fR
must be specified. The high availability system must insure that the
slurmctld daemon is running on only one of these hosts at a time.
See the \fBRELOCATING CONTROLLERS\fR section if you change this.
.TP
\fBCryptoType\fR
The cryptographic signature tool to be used in the creation of
job step credentials.
The slurmctld daemon must be restarted for a change in \fBCryptoType\fR
to take effect.
Acceptable values at present include "crypto/munge" and "crypto/openssl".
The default value is "crypto/munge".
.TP
\fBDebugFlags\fR
Defines specific subsystems which should provide more detailed event logging.
Multiple subsystems can be specified with comma separators.
Valid subsystems available today (with more to come) include:
.TP 17
\fBBackfill\fR
Backfill scheduler details
.TP
\fBBGBlockAlgo\fR
BlueGene block selection details
.TP
\fBBGBlockAlgoDeep\fR
BlueGene block selection, more details
.TP
\fBBGBlockPick\fR
BlueGene block selection for jobs
.TP
\fBBGBlockWires\fR
BlueGene block wiring (switch state details)
.TP
\fBCPU_Bind\fR
CPU binding details for jobs and steps
\fBEnergy\fR
AcctGatherEnergy debug info
.TP
\fBExtSensors\fR
External Sensors debug info
.TP
\fBFrontEnd\fR
Front end node details
.TP
\fBGres\fR
Generic resource details
.TP
\fBGang\fR
Gang scheduling details
.TP
\fBJobContainer\fR
Job container plugin details
.TP
\fBNO_CONF_HASH\fR
Do not log when the slurm.conf files differs between SLURM daemons
.TP
\fBPriority\fB
Job prioritization
.TP
\fBReservation\fB
Advanced reservations
.TP
\fBSelectType\fR
Resource selection plugin
.TP
\fBSteps\fR
Slurmctld resource allocation for job steps
\fBTriggers\fR
Slurmctld triggers
.TP
\fBWiki\fR
Sched/wiki and wiki2 communications
.RE
.TP
\fBDefMemPerCPU\fR
Default real memory size available per allocated CPU in MegaBytes.
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerCPU\fR would generally be used if individual processors
are allocated to jobs (\fBSelectType=select/cons_res\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerNode\fR and \fBMaxMemPerCPU\fR.
\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
NOTE: Enforcement of memory limits currently requires enabling of
accounting, which samples memory use on a periodic basis (data need
not be stored, just collected).
.TP
\fBDefMemPerNode\fR
Default real memory size available per allocated node in MegaBytes.
Used to avoid over\-subscribing memory and causing paging.
\fBDefMemPerNode\fR would generally be used if whole nodes
are allocated to jobs (\fBSelectType=select/linear\fR) and
resources are shared (\fBShared=yes\fR or \fBShared=force\fR).
The default value is 0 (unlimited).
Also see \fBDefMemPerCPU\fR and \fBMaxMemPerNode\fR.
\fBDefMemPerCPU\fR and \fBDefMemPerNode\fR are mutually exclusive.
NOTE: Enforcement of memory limits currently requires enabling of
accounting, which samples memory use on a periodic basis (data need
not be stored, just collected).
The default name of the machine hosting the accounting storage and
job completion databases.
Only used for database type storage plugins and when the
\fBAccountingStorageHost\fR and \fBJobCompHost\fR have not been
defined.
The fully qualified file name where accounting records and/or job
completion records are written when the \fBDefaultStorageType\fR is
"filetxt" or the name of the database where accounting records and/or job
completion records are stored when the \fBDefaultStorageType\fR is a
database.
Also see \fBAccountingStorageLoc\fR and \fBJobCompLoc\fR.
.TP
\fBDefaultStoragePass\fR
The password used to gain access to the database to store the
accounting and job completion data.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStoragePass\fR and \fBJobCompPass\fR.
.TP
\fBDefaultStoragePort\fR
The listening port of the accounting storage and/or job completion
database server.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStoragePort\fR and \fBJobCompPort\fR.
.TP
\fBDefaultStorageType\fR
The accounting and job completion storage mechanism type. Acceptable
values at present include "filetxt", "mysql" and "none".
The value "filetxt" indicates that records will be written to a file.
The value "mysql" indicates that accounting records will be written to a MySQL
or MariaDB database.
The default value is "none", which means that records are not maintained.
Also see \fBAccountingStorageType\fR and \fBJobCompType\fR.
The user account for accessing the accounting storage and/or job
completion database.
Only used for database type storage plugins, ignored otherwise.
Also see \fBAccountingStorageUser\fR and \fBJobCompUser\fR.
.TP
\fBDisableRootJobs\fR
If set to "YES" then user root will be prevented from running any jobs.
The default value is "NO", meaning user root will be able to execute jobs.
\fBDisableRootJobs\fR may also be set by partition.
.TP
\fBDynallocPort\fR
Socket port used for MapReduce dynamic allocation communications.
Used only by the slurmctld/dynalloc plugin.
.TP
\fBEnforcePartLimits\fR
If set to "YES" then jobs which exceed a partition's size and/or time limits
will be rejected at submission time. If set to "NO" then the job will be
accepted and remain queued until the partition limits are altered.
The default value is "NO".
NOTE: If set, then a job's QOS can not be used to exceed partition limits.
Moe Jette
committed
.TP
Fully qualified pathname of a script to execute as user root on every
node when a user's job completes (e.g. "/usr/local/slurm/epilog"). A
glob pattern (See \fBglob\fR(7)) may also be used to run more than
one epilog script (e.g. "/etc/slurm/epilog.d/*"). The Epilog script
or scripts may be used to purge files, disable user login, etc.
By default there is no epilog.
See \fBProlog and Epilog Scripts\fR for more information.
.TP
\fBEpilogMsgTime\fR
The number of microseconds that the slurmctld daemon requires to process
an epilog completion message from the slurmd dameons. This parameter can
be used to prevent a burst of epilog completion messages from being sent
at the same time which should help prevent lost messages and improve
throughput for large jobs.
The default value is 2000 microseconds.
For a 1000 node job, this spreads the epilog completion messages out over
two seconds.
.TP
\fBEpilogSlurmctld\fR
Fully qualified pathname of a program for the slurmctld to execute
upon termination of a job allocation (e.g.
"/usr/local/slurm/epilog_controller").
The program executes as SlurmUser, which gives it permission to drain
nodes and requeue the job if a failure occurs (See scontrol(1)).
Exactly what the program does and how it accomplishes this is completely at
the discretion of the system administrator.
Information about the job being initiated, it's allocated nodes, etc. are
passed to the program using environment variables.
See \fBProlog and Epilog Scripts\fR for more information.
.TP
\fBExtSensorsFreq\fR
The external sensors plugin sampling interval.
If \fBExtSensorsType=ext_sensors/none\fR, this parameter is ignored.
For all other values of \fBExtSensorsType\fR, this parameter is the number
of seconds between external sensors samples for hardware components (nodes,
switches, etc.) The default value is zero. This value disables external
sensors sampling. Note: This parameter does not affect external sensors
data collection for jobs/steps.
.TP
\fBExtSensorsType\fR
Identifies the plugin to be used for external sensors data collection.
Slurmctld calls this plugin to collect external sensors data for jobs/steps
and hardware components. In case of node sharing between jobs the reported
values per job/step (through sstat or sacct) may not be accurate. See also
"man ext_sensors.conf".
Configurable values at present are:
.RS
.TP 20
\fBext_sensors/none\fR
No external sensors data is collected.
.TP
\fBext_sensors/rrd\fR
External sensors data is collected from the RRD database.
.RE
Controls how a node's configuration specifications in slurm.conf are used.
If the number of node configuration entries in the configuration file
is significantly lower than the number of nodes, setting FastSchedule to
1 will permit much faster scheduling decisions to be made.
(The scheduler can just check the values in a few configuration records
instead of possibly thousands of node records.)
Note that on systems with hyper\-threading, the processor count
reported by the node will be twice the actual processor count.
Consider which value you want to be used for scheduling purposes.
.RS
.TP 5
\fB1\fR (default)
Consider the configuration of each node to be that specified in the
slurm.conf configuration file and any node with less than the
configured resources will be set DOWN.
.TP
\fB0\fR
Base scheduling decisions upon the actual configuration of each individual
node except that the node's processor count in SLURM's configuration must
match the actual hardware configuration if \fBSchedulerType=sched/gang\fR
or \fBSelectType=select/cons_res\fR are configured (both of those plugins
maintain resource allocation information using bitmaps for the cores in the
system and must remain static, while the node's memory and disk space can
be established later).
.TP
\fB2\fR
Consider the configuration of each node to be that specified in the
slurm.conf configuration file and any node with less than the
configured resources will \fBnot\fR be set DOWN.
This can be useful for testing purposes.
.RE
The job id to be used for the first submitted to SLURM without a
specific requested value. Job id values generated will incremented by 1
for each subsequent job. This may be used to provide a meta\-scheduler
with a job id space which is disjoint from the interactive jobs.
Also see \fBMaxJobId\fR
.TP
\fBGetEnvTimeout\fR
Used for Moab scheduled jobs only. Controls how long job should wait
in seconds for loading the user's environment before attempting to
load it from a cache file. Applies when the srun or sbatch
\fI\-\-get\-user\-env\fR option is used. If set to 0 then always load
the user's environment from the cache file.
The default value is 2 seconds.
\fBGresTypes\fR
A comma delimited list of generic resources to be managed.
These generic resources may have an associated plugin available to provide
additional functionality.
No generic resources are managed by default.
Moe Jette
committed
Insure this parameter is consistent across all nodes in the cluster for
proper operation.
The slurmctld daemon must be restarted for changes to this parameter to become
effective.
.TP
\fBGroupUpdateForce\fR
If set to a non\-zero value, then information about which users are members
of groups allowed to use a partition will be updated periodically, even when
there have been no changes to the /etc/group file.
Otherwise group member information will be updated periodically only after the
/etc/group file is updated
Also see the \fBGroupUpdateTime\fR parameter.
.TP
\fBGroupUpdateTime\fR
Controls how frequently information about which users are members of groups
allowed to use a partition will be updated.
The time interval is given in seconds with a default value of 600 seconds and
a maximum value of 4095 seconds.
A value of zero will prevent periodic updating of group membership information.
Also see the \fBGroupUpdateForce\fR parameter.
.TP
\fBHealthCheckInterval\fR
The interval in seconds between executions of \fBHealthCheckProgram\fR.
The default value is zero, which disables execution.
.TP
\fBHealthCheckNodeState\fR
Identify what node states should execute the \fBHealthCheckProgram\fR.
Multiple state values may be specified with a comma separator.
The default value is ANY to execute on nodes in any state.
.RS
.TP 12
\fBALLOC\fR
Run on nodes in the ALLOC state (all CPUs allocated).
.TP
\fBANY\fR
Run on nodes in any state.
.TP
\fBIDLE\fR
Run on nodes in the IDLE state.
.TP
\fBMIXED\fR
Run on nodes in the MIXED state (some CPUs idle and other CPUs allocated).
.RE
.TP
\fBHealthCheckProgram\fR
Fully qualified pathname of a script to execute as user root periodically
on all compute nodes that are \fBnot\fR in the NOT_RESPONDING state. This
program may be used to verify the node is fully operational and DRAIN the node
or send email if a problem is detected.
Any action to be taken must be explicitly performed by the program
(e.g. execute
"scontrol update NodeName=foo State=drain Reason=tmp_file_system_full"
to drain a node).
The execution interval is controlled using the \fBHealthCheckInterval\fR
parameter.
Note that the \fBHealthCheckProgram\fR will be executed at the same time
on all nodes to minimize its impact upon parallel programs.
This program is will be killed if it does not terminate normally within
By default, no program will be executed.
The interval, in seconds, after which a non\-responsive job allocation
command (e.g. \fBsrun\fR or \fBsalloc\fR) will result in the job being
terminated. If the node on which the command is executed fails or the
command abnormally terminates, this will terminate its job allocation.
This option has no effect upon batch jobs.
When setting a value, take into consideration that a debugger using \fBsrun\fR
to launch an application may leave the \fBsrun\fR command in a stopped state
for extended periods of time.
This limit is ignored for jobs running in partitions with the
\fBRootOnly\fR flag set (the scheduler running as root will be
responsible for the job).
The default value is unlimited (zero) and may not exceed 65533 seconds.
\fBJobAcctGatherType\fR
The job accounting mechanism type.
Acceptable values at present include "jobacct_gather/aix" (for AIX operating
system), "jobacct_gather/linux" (for Linux operating system),
"jobacct_gather/cgroup" (experimental) and "jobacct_gather/none"
The default value is "jobacct_gather/none".
"jobacct_gather/cgroup" is an experimental plugin for the Linux operating system
that uses cgroups to collect accounting statistics. The plugin collects the
following statistics: From the cgroup memory subsystem: memory.usage_in_bytes
(reported as 'pages') and rss from memory.stat (reported as 'rss'). From the
cgroup cpuacct subsystem: user cpu time and system cpu time. No value
is provided by cgroups for virtual memory size ('vsize').
In order to use the \fBsstat\fR tool, "jobacct_gather/aix", "jobacct_gather/linux",
or "jobacct_gather/cgroup" must be configured.
.br
\fBNOTE:\fR Changing this configuration parameter changes the contents of
the messages between Slurm daemons. Any previously running job steps are
managed by a slurmstepd daemon that will persist through the lifetime of
that job step and not change it's communication prototol. Only change this
configuration parameter when there are no running job steps.
\fBJobAcctGatherFrequency\fR
The job accounting and profiling sampling intervals.
The supported format is follows:
.RS
.TP 12
\fBJobAcctGatherFrequency=\fR\fI<datatype>\fR\fB=\fR\fI<interval>\fR
where \fI<datatype>\fR=\fI<interval>\fR specifies the task sampling
interval for the jobacct_gather plugin or a
sampling interval for a profiling type by the
acct_gather_profile plugin. Multiple,
comma-separated \fI<datatype>\fR=\fI<interval>\fR intervals
may be specified. Supported datatypes are as follows:
.RS
.TP
\fBtask=\fI<interval>\fR
where \fI<interval>\fR is the task sampling interval in seconds
for the jobacct_gather plugins and for task
profiling by the acct_gather_profile plugin.
.TP
\fBenergy=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for energy profiling using the acct_gather_energy plugin
.TP
\fBnetwork=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for infiniband profiling using the acct_gather_infiniband
plugin.
.TP
\fBfilesystem=\fI<interval>\fR
where \fI<interval>\fR is the sampling interval in seconds
for filesystem profiling using the acct_gather_filesystem
plugin.
.TP
.RE
.RE
The default value for task sampling interval
is 30 seconds. The default value for all other intervals is 0.
An interval of 0 disables sampling of the specified type.
If the task sampling interval is 0, accounting
information is collected only at job termination (reducing SLURM
interference with the job).
.br
.br
Smaller (non\-zero) values have a greater impact upon job performance,
but a value of 30 seconds is not likely to be noticeable for
applications having less than 10,000 tasks.
.br
.br
Users can independently override each interval on a per job basis using the
\fB\-\-acctg\-freq\fR option when submitting the job.
.RE
.TP
\fBJobAcctGatherParams\fR
Arbitrary parameters for the job account gather plugin
Acceptable values at present include:
.RS
.TP 20
fB\NoShared\fR
Exclude shared memory from accounting.
.RE
.TP
\fBJobCheckpointDir\fR
Specifies the default directory for storing or reading job checkpoint
information. The data stored here is only a few thousand bytes per job
and includes information needed to resubmit the job request, not job's
memory image. The directory must be readable and writable by
\fBSlurmUser\fR, but not writable by regular users. The job memory images
may be in a different location as specified by \fB\-\-checkpoint\-dir\fR
option at job submit time or scontrol's \fBImageDir\fR option.
.TP
The name of the machine hosting the job completion database.
Only used for database type storage plugins, ignored otherwise.
.TP
The fully qualified file name where job completion records are written
when the \fBJobCompType\fR is "jobcomp/filetxt" or the database where
job completion records are stored when the \fBJobCompType\fR is a
database.
.TP
The password used to gain access to the database to store the job
completion data.
Only used for database type storage plugins, ignored otherwise.
.TP
The listening port of the job completion database server.
Only used for database type storage plugins, ignored otherwise.
.TP
\fBJobCompType\fR
The job completion logging mechanism type.
Acceptable values at present include "jobcomp/none", "jobcomp/filetxt",
"jobcomp/mysql", and "jobcomp/script"".
The default value is "jobcomp/none", which means that upon job completion
the record of the job is purged from the system. If using the accounting
infrastructure this plugin may not be of interest since the information
here is redundant.
The value "jobcomp/filetxt" indicates that a record of the job should be
written to a text file specified by the \fBJobCompLoc\fR parameter.
The value "jobcomp/mysql" indicates that a record of the job should be
written to a MySQL or MariaDB database specified by the \fBJobCompLoc\fR
parameter.
The value "jobcomp/script" indicates that a script specified by the
\fBJobCompLoc\fR parameter is to be executed with environment variables
indicating the job information.
.TP
\fBJobCompUser\fR
The user account for accessing the job completion database.
Only used for database type storage plugins, ignored otherwise.
.TP
Identifies the plugin to be used for job tracking.
The slurmd daemon must be restarted for a change in JobContainerType
to take effect.
NOTE: The \fBJobContainerType\fR applies to a job allocation, while
\fBProctrackType\fR applies to job steps.
Acceptable values at present include:
.RS
.TP 20
used only for Cray systems (CNCU = Compute Node Clean Up)
used for all other system types
.RE
Fully qualified pathname of a file containing a private key used for
authentication by SLURM daemons.
This parameter is ignored if \fBCryptoType=crypto/munge\fR.
.TP
\fBJobCredentialPublicCertificate\fR
Fully qualified pathname of a file containing a public key used for
authentication by SLURM daemons.
This parameter is ignored if \fBCryptoType=crypto/munge\fR.
.TP
\fBJobFileAppend\fR
This option controls what to do if a job's output or error file
exist when the job is started.
If \fBJobFileAppend\fR is set to a value of 1, then append to
the existing file.
By default, any existing file is truncated.
.TP
\fBJobRequeue\fR
This option controls what to do by default after a node failure.
If \fBJobRequeue\fR is set to a value of 1, then any batch job running
on the failed node will be requeued for execution on different nodes.
If \fBJobRequeue\fR is set to a value of 0, then any job running
on the failed node will be terminated.
Use the \fBsbatch\fR \fI\-\-no\-requeue\fR or \fI\-\-requeue\fR
option to change the default behavior for individual jobs.
The default value is 1.
.TP
\fBJobSubmitPlugins\fR
A comma delimited list of job submission plugins to be used.
The specified plugins will be executed in the order listed.
These are intended to be site\-specific plugins which can be used to set
default job parameters and/or logging events.
Sample plugins available in the distribution include "all_partitions", "cnode",
"defaults", "logging", "lua", and "partition".
For examples of use, see the SLURM code in "src/plugins/job_submit" and
"contribs/lua/job_submit*.lua" then modify the code to satisfy your needs.
SLURM can be configured to use multiple job_submit plugins if desired,
however the lua plugin will only execute one lua script named "job_submit.lua"
and located in default script directory (typically the subdirectory "etc" of
the installation directory).
No job submission plugins are used by default.
Morris Jette
committed
.TP
\fBKeepAliveTime\fR
Specifies how long sockets communications used between the srun command and its
slurmstepd process are kept alive after disconnect.
Longer values can be used to improve reliability of communications in the
event of network failures.
The default value leaves the system default value.
The value may not exceed 65533.
.TP
If set to 1, the job will be terminated immediately when one of the
processes is crashed or aborted. With the default value of 0, if one of
the processes is crashed or aborted the other processes will continue
to run. The user can override this configuration parameter by using srun's
\fB\-K\fR, \fB\-\-kill\-on\-bad\-exit\fR.
.TP
The interval, in seconds, given to a job's processes between the
SIGTERM and SIGKILL signals upon reaching its time limit.
If the job fails to terminate gracefully in the interval specified,
it will be forcibly terminated.
The default value is 30 seconds.