<body><h1>Accounting</h1><p>SLURM collects accounting information for every job and job step
<h1>Accounting</h1>
<p>SLURM collects accounting information for every job and job step
executed.
executed.
Information is available about both currently executing jobs and
Information is available about both currently executing jobs and
jobs which have already terminated and can be viewed using the
jobs which have already terminated and can be viewed using the
...
@@ -250,62 +251,227 @@ This value must be specified.</li>
...
@@ -250,62 +251,227 @@ This value must be specified.</li>
<li><b>StorageUser</b>:
<li><b>StorageUser</b>:
Define the name of the user we are going to connect to the database
Define the name of the user we are going to connect to the database
with to store the job accounting data.</li>
with to store the job accounting data.</li>
</ul><big><big><spanstyle="font-weight: bold;">Tools</span></big></big><br><br>There are two tools available to work with accounting data,
</ul>
sacct and sacctmgr. Both of these tools will get or set data
through the SlurmDBD daemon.
<h2>Tools</h2>
<p>There are two tools available to work with accounting data,
<b>sacct</b> and <b>sacctmgr</b>.
Both of these tools will get or set data through the SlurmDBD daemon.
Sacct is used to generate accounting report for both running and
Sacct is used to generate accounting report for both running and
completed jobs.
completed jobs.
Sacctmgr is used to manage associations in the database:
Sacctmgr is used to manage associations in the database:
add or remove clusters, add or remove users, etc.
add or remove clusters, add or remove users, etc.
See the man pages for each command for more information.
See the man pages for each command for more information.</p>
<br><br>Web interfaces with graphical output is currently under
<p>Web interfaces with graphical output is currently under
development and should be available in the summer of 2008.
development and should be available in the summer of 2008.
A tool to report node state information is also under development.<br><h2>Database Configuration</h2>
A tool to report node state information is also under development.</p>
<h2>Database Configuration</h2>
<p>Accounting records are maintained based upon what we refer
<p>Accounting records are maintained based upon what we refer
to as an <i>Association</i>,
to as an <i>Association</i>,
which consists of four elements: cluster, account, and user names. For
which consists of four elements: cluster, account, and user names. For
more flexibility in accounting the association can also include a
more flexibility in accounting the association can also include a
partition name, but it is not necissary. Use the <i>sacctmgr</i>
partition name, but it is not necessary. Use the <i>sacctmgr</i>
command to create and manage these records. There is an order to set up
command to create and manage these records. There is an order to set up
accounting associations. You must define clusters before you add
accounting associations. You must define clusters before you add
accounts and you must add accounts before you can add users. </p><p>When adding clusters to the system you only need to run... </p><p> <spanstyle="font-family: Bitstream Charter;">sacctmgr add cluster Snowflake</span></p><p>To add accounts to clusters you can do something like this...</p><p> <spanstyle="font-family: Bitstream Charter;">sacctmgr add account<spanstyle="font-family: Bitstream Charter;"></span>none,test Description="none" Organization="none" Cluster=Snowflake</span></p><p>This will add accounts <spanstyle="font-style: italic;">none</span> and <spanstyle="font-style: italic;">test </span>to cluster Snowflake
accounts and you must add accounts before you can add users. </p>
If you have more clusters you want to add these accounts to you
<p>For example, to add a cluster named "snowflake" to the database
execute this line:</p>
<pre>
sacctmgr add cluster snowflake
</pre>
<p>Add accounts "none" and "test" to cluster "snowflake" with an execute
<p>If you have more clusters you want to add these accounts to you
can either not specify a cluster, which will add the accounts to all
can either not specify a cluster, which will add the accounts to all
clusters in the system, or common separate the cluster names you want
clusters in the system, or comma separate the cluster names you want
to add to in the cluster option. As you may have noticed you can
to add to in the cluster option.
add many different accounts at the same time by common separating the
Note that multiple accounts can be added at the same at the same time
names. You need to specify the Description of the account and the
by comma separating the names.
organization which it belongs. These terms can be used to display
Some Description of the account and the organization which it belongs
accounting reports later. Accounts may be arranged in a hierarchical fashion, for example accounts <i>chemistry</i> and <i>physics</i> may be
must be specified.
children of the account <i>science</i>.
These terms can be used later to generated accounting reports.
The hierarchy may have an arbitrary depth. To do this one only needs to specify the <spanstyle="font-style: italic;">parent='' </span>option to the add account line, for instance if you want to do the example above...</p><pstyle="font-family: Bitstream Charter;">sacctmgr add account science Description="science accounts" Organization=science<br>sacctmgr add account chemistry,physics parent=science Description="physical sciences" Organization=science<br></p><p>Now, to add users to accounts you can run...</p><p> <spanstyle="font-family: Bitstream Charter;">sacctmgr add user da default=test <br></span></p><p>This
Accounts may be arranged in a hierarchical fashion, for example accounts
will add user da to the system, and add associations to account test on
<i>chemistry</i> and <i>physics</i> may be children of the account <i>science</i>.
all clusters test exists for user da. This will enable user da to run
The hierarchy may have an arbitrary depth.
jobs in account test on those clusters. For instance if
To do this one only needs to specify the <i>parent='' </i>option in the add
AccountingStorageEnforce=1 in the slurm.conf of Snowflake da would be
account line.
allowed to run in account test, and any other ones we add him to in the
For the example above execute</p>
future but not any other accounts. Account <spanstyle="font-style: italic;">test</span> will be the default if he doesn't specify one in a srun line. </p><p>Partition
<pre>
names can also be added to an add user command with the
For example, to permit user <i>da</i> to execute jobs on all clusters
with a default account of <i>test</i> execute:</p>
<pre>
sacctmgr add user da default=test
</pre>
<p>If <b>AccountingStorageEnforce=1</b> is configured in the slurm.conf of
the cluster <i>snowflake</i> then user <i>da</i> would be
allowed to run in account <i>test</i> and any other accounts added
in the future.
Any attepmt to use other accounts will result in the job being
aborted.
Account <i>test</i> will be the default if he doesn't specify one in a
srun line.</p>
<p>Partition names can also be added to an "add user" command with the
Partition='partitionname' option to specify an association specific to
Partition='partitionname' option to specify an association specific to
a slurm partition. </p><h2>Cluster Options</h2>When either adding or modifying a cluster these are the options you can use with sacctmgr:<ul><li><spanstyle="font-weight: bold;">Name=:</span> Cluster name<spanstyle="font-weight: bold;"></span></li><li><spanstyle="font-weight: bold;">Fairshare=:</span> Used for determining priority (used in later development)</li><li><spanstyle="font-weight: bold;">MaxJobs=:</span> Limit number of jobs a user can run in this account (used in later development)</li><li><spanstyle="font-weight: bold;">MaxNodes=: </span>Limit number of nodes a user can allocate in this account (used in later development)</li><li><spanstyle="font-weight: bold;">MaxWall=: </span>Limit wall clock time a job can run (used in later development)</li><li><spanstyle="font-weight: bold;">MaxCPUSecs=:</span> Limit cpu seconds a job can run (used in later development)</li></ul><h2>Account Options</h2>When either adding or modifying an account these are the options you can use with sacctmgr:<br><ul><li><spanstyle="font-weight: bold;">Description=:</span> Description of the account. (Required when creating)</li><li><spanstyle="font-weight: bold;">Organization=: </span>Organization of the account. (Required when creating)</li><li><spanstyle="font-weight: bold;">Name=:</span> Name of account</li><li><spanstyle="font-weight: bold;">Cluster=:</span> Only add this account to these clusters.</li><li><spanstyle="font-weight: bold;">Parent=:</span> Make this account a child of this other account.</li><li><spanstyle="font-weight: bold;">QOS=:</span> Quality of Service (used in later development)</li><li><spanstyle="font-weight: bold;">Fairshare=:</span> Used for determining priority (used in later development)</li><li><spanstyle="font-weight: bold;">MaxJobs=:</span> Limit number of jobs a user can run in this account (used in later development)</li><li><spanstyle="font-weight: bold;">MaxNodes=: </span>Limit number of nodes a user can allocate in this account (used in later development)</li><li><spanstyle="font-weight: bold;">MaxWall=: </span>Limit wall time a job can run (used in later development)</li><li><spanstyle="font-weight: bold;">MaxCPUSecs=:</span> Limit cpu seconds a job can run (used in later development)</li></ul><h2>User Options</h2>When either adding or modifying a user these are the options you can use with sacctmgr:<br><ul><li><spanstyle="font-weight: bold;">Name=: </span>User name</li><li><spanstyle="font-weight: bold;">DefaultAccount=:</span> Default account for the user, used when a user doesn't specify an account on job submit. (Required when creating)</li><li><spanstyle="font-weight: bold;">AdminLevel=: </span>This field is used to allow a user to add accounting privileges to this user. Valid options are <spanstyle="font-style: italic;">None, Operator </span>(can add, modify,<spanstyle="font-style: italic;"></span>and remove users, and add other operators)<spanstyle="font-style: italic;">, </span>and<spanstyle="font-style: italic;"> Admin<spanstyle="font-style: italic;"><spanstyle="font-style: italic;"><spanstyle="font-style: italic;"></span></span></span></span>(In addition to operator privileges these users can add, modify, and remove accounts and clusters).</li><li><spanstyle="font-weight: bold;">Account=: </span>Account(s) to add user to. </li><li><spanstyle="font-weight: bold;">Cluster=: </span>Only add to accounts on these clusters.</li><li><spanstyle="font-weight: bold;">Partition=:</span> Name of partition this association is for.</li><li><spanstyle="font-weight: bold;">QOS=:</span> Quality of Service (used in later development)</li>
a slurm partition.</p>
<li><spanstyle="font-weight: bold;">Fairshare=:</span> Used for determining priority (used in later development)</li><li><spanstyle="font-weight: bold;">MaxJobs=:</span> Limit number of jobs a user can run in this account (used in later development)</li><li><spanstyle="font-weight: bold;">MaxNodes=: </span>Limit number of nodes a user can allocate in this account (used in later development)</li><li><spanstyle="font-weight: bold;">MaxWall=: </span>Limit wall time a job can run (used in later development)</li><li><spanstyle="font-weight: bold;">MaxCPUSecs=:</span> Limit cpu seconds a job can run (used in later development)</li></ul><big><big><spanstyle="font-weight: bold;">Limit enforcement</span></big></big><br><br>When limits are developed they will work in this order...<br>If
a user has a limit set SLURM will read in those, if not we will refer
<!-- For future use
to the account associated with the job. If the account doesn't
<h2>Cluster Options</h2>
have the limit set we will refer to the cluster's limits. If the
cluster doesn't have the limit set no limit will be enforced.<br><big><big><br><spanstyle="font-weight: bold;">Modifying entities<br></span></big></big><br>When modifying entities, you can specify many different options in SQL like fashion, using key words like <spanstyle="font-style: italic;">where, </span>and <spanstyle="font-style: italic;">set.</span> The line<br><br><spanstyle="font-family: Bitstream Charter;">sacctmgr modify <spanstyle="font-style: italic;">entity</span> set <spanstyle="font-style: italic;">options</span> where <spanstyle="font-style: italic;">options</span><br><br></span>The example...<br><br><spanstyle="font-family: Bitstream Charter;">sacctmgr modify user set default=none where default=test fairshare=2<br><br></span>will change all users with default account test, and fairshare of 2 to account none. <br><br>Once
<p>When either adding or modifying a cluster, these are the options
an entity has been add/modified/removed the change is sent to the
available with sacctmgr:
appropriate slurmctld and will be available to be used instantly. <br><br><big><big><spanstyle="font-weight: bold;">Removing entities</span><brstyle="font-weight: bold;"></big></big><br>When removing entities, you can issue a line similar to the modify example above only removing the set options.<br><br><spanstyle="font-family: Bitstream Charter;">sacctmgr remove user where default=test fairshare=2<br><br></span>This will remove all users with default account test, and fairshare of 2.<br>
<ul>
<li><b>Name=</b> Cluster name</li>
<bigstyle="font-weight: bold;"><big>Node State Information</big></big>
<li><b>Fairshare=</b> Used for determining priority</li>
<br><br>Node state information is also recorded in the database.
<li><b>MaxJobs=</b> Limit number of jobs a user can run in this account</li>
<li><b>MaxNodes=</b>Limit number of nodes a user can allocate in this
account</li>
<li><b>MaxWall=</b>Limit wall clock time a job can run</li>
<li><b>MaxCPUSecs=</b> Limit cpu seconds a job can run</li>
</ul>
!-->
<h2>Account Options</h2>
<p>When either adding or modifying an account, the following sacctmgr
options are available:
<ul>
<li><b>Description=</b> Description of the account. (Required on creation)</li>
<li><b>Organization=</b>Organization of the account. (Required on creation)</li>
<li><b>Name=</b> Name of account</li>
<li><b>Cluster=</b> Only add this account to these clusters.
The account is added to all defined clusters by default.</li>
<li><b>Parent=</b> Make this account a child of this other account.</li>
<!-- For future use
<li><b>QOS=</b> Quality of Service</li>
<li><b>Fairshare=</b> Used for determining priority</li>
<li><b>MaxJobs=</b> Limit number of jobs a user can run in this account</li>
<li><b>MaxNodes=</b>Limit number of nodes a user can allocate in this account</li>
<li><b>MaxWall=</b>Limit wall time a job can run</li>
<li><b>MaxCPUSecs=</b> Limit cpu seconds a job can run</li>
!-->
</ul>
<h2>User Options</h2>
<p>When either adding or modifying a user, the following sacctmgr
options are available:
<ul>
<li><b>Name=</b> User name</li>
<li><b>DefaultAccount=</b> Default account for the user, used when no account
is specified when a job is sumbitted. (Required on creation)</li>
<li><b>AdminLevel=</b> This field is used to allow a user to add accounting
privileges to this user. Valid options are
<ul>
<li>None</li>
<li>Operator: can add, modify,and remove users, and add other operators)</li>
<li>Admin: In addition to operator privileges these users can add, modify,
and remove accounts and clusters</li>
</ul>
<li><b>Account=</b> Account(s) to add user to</li>
<li><b>Cluster=</b> Only add to accounts on these clusters (default is all clusters)</li>
<li><b>Partition=</b> Name of Slurm partition this association applies to</li>
<!-- For future use
<li><b>QOS=</b> Quality of Service</li>
<li><b>Fairshare=</b> Used for determining priority</li>
<li><b>MaxJobs=</b> Limit number of jobs a user can run in this account</li>
<li><b>MaxNodes=</b> Limit number of nodes a user can allocate in this account</li>
<li><b>MaxWall=</b> Limit wall time a job can run</li>
<li><b>MaxCPUSecs=</b> Limit cpu seconds a job can run</li>
!-->
</ul>
<!-- For future use
<h2>Limit enforcement</h2>
<p>When limits are developed they will work in this order...
If a user has a limit set SLURM will read in those,
if not we will refer to the account associated with the job.
If the account doesn't have the limit set we will refer to
the cluster's limits.
If the cluster doesn't have the limit set no limit will be enforced.
!-->
<h2>Modifying Entities</h2>
<p>When modifying entities, you can specify many different options in
SQL-like fashion, using key words like <i>where</i> and <i>set</i>.
A typical execute line has the following form:
<pre>
sacctmgr modify <entity> set <options> where <options>
</pre>
<p>For example:</p>
<pre>
sacctmgr modify user set default=none where default=test
</pre>
<p>will change all users with a default account of "test" to account "none".
Once an entity has been added, modified or removed, the change is
sent to the appropriate SLURM daemons and will be available for use
instantly.</p>
<h2>Removing Entities</h2>
<p>Removing entities using an execute line similar to the modify example above,
but without the set options.
For example, remove all users with a default account "test" using the following
execute line:</p>
<pre>
sacctmgr remove user where default=test
</pre>
<h2>Node State Information</h2>
<p>Node state information is also recorded in the database.
Whenever a node goes DOWN or becomes DRAINED that event is
Whenever a node goes DOWN or becomes DRAINED that event is
logged along with the node's <i>Reason</i> field.
logged along with the node's <i>Reason</i> field.
This can be used to generate various reports.<spanstyle="font-weight: bold;"></span><spanstyle="font-weight: bold;"></span><br><br><big><big><spanstyle="font-weight: bold;"><br></span></big></big><ul>
This can be used to generate various reports.
<p style="text-align: center;">Last modified 25 March 2008</p>
<p style="text-align: center;">Last modified 25 March 2008</p>
