From 00f57b3ee7d90d4c920084979e267542ae1d192c Mon Sep 17 00:00:00 2001
From: Danny Auble <da@llnl.gov>
Date: Mon, 2 Mar 2009 21:41:08 +0000
Subject: [PATCH] svn merge -r16727:16742
https://eris.llnl.gov/svn/slurm/branches/slurm-1.3
---
doc/html/accounting.shtml | 203 ++++++++++++------
doc/man/man1/sreport.1 | 12 +-
.../mysql/accounting_storage_mysql.c | 10 -
src/plugins/proctrack/rms/proctrack_rms.c | 2 +-
src/sacctmgr/account_functions.c | 8 +-
src/sacctmgr/association_functions.c | 7 +-
src/sacctmgr/cluster_functions.c | 2 +-
src/sacctmgr/sacctmgr.c | 12 +-
src/sacctmgr/user_functions.c | 11 +-
src/slurmctld/acct_policy.c | 2 +-
10 files changed, 179 insertions(+), 90 deletions(-)
diff --git a/doc/html/accounting.shtml b/doc/html/accounting.shtml
index 4f67cb5242b..30cfe52eb63 100644
--- a/doc/html/accounting.shtml
+++ b/doc/html/accounting.shtml
@@ -15,6 +15,7 @@ jobs which have already terminated and can be viewed using the
<b>sacct</b> can also report resource usage for individual tasks,
which can be useful to detect load imbalance between the tasks.
The <b>sstat</b> tool can be used to status a currently running job.
+It also can give you valuable information about imbalance between tasks.
The <b>sreport</b> can be used to generate reports based upon all jobs
executed in a particular time interval.</p>
@@ -44,12 +45,16 @@ database optionally using either
or SlurmDBD for added security.</li>
</ul>
-<p>The use of sacct or sstat to view information about completed jobs
-is dependent upon both JobAcctGatherType and AccountingStorageType
+<p>The use of sacct to view information about jobs
+is dependent upon AccountingStorageType
being configured to collect and store that information.
The use of sreport is dependent upon some database being
used to store that information.</p>
+<p>The use of sacct or sstat to view information about resource usage
+ within jobs is dependent upon both JobAcctGatherType and AccountingStorageType
+ being configured to collect and store that information.</p>
+
<p>Storing the accounting information into text files is
very simple. Just configure the appropriate plugin (e.g.
<i>AccountingStorageType=accounting_storage/filetxt</i> and/or
@@ -212,20 +217,35 @@ Note that you can write accounting information to SlurmDBD
while job completion records are written to a text file or
not maintained at all.
If you don't set the configuration parameters that begin
-with "JobComp" then job completion records will not be recorded.</p>
+with "AccountingStorage" then accounting information will not be
+referenced or recorded.</p>
<ul>
<li><b>AccountingStorageEnforce</b>:
-If you want to prevent users from running jobs if their <i>association</i>
-is not in the database, then set this to "1".
+This option contains a comma separated list of options you may want to
+ enforce. The valid options are
+<ul>
+<li>associations - This will prevent users from running jobs if
+their <i>association</i> is not in the database. This option will
+prevent users from accessing invalid accounts.
+</li>
+<li>limits - This will enforce limits set to associations. By setting
+ this option the 'associations' option is also set.
+</li>
+<li>wckeys - This will prevent users from running jobs under a wckey
+ that they don't have access to. By using this option the
+ 'associations' option is also set. The 'TrackWCKey' option is also
+ set to true.
+</li>
+</ul>
(NOTE: The association is a combination of cluster, account,
user names and optional partition name.)
+<br>
Without AccountingStorageEnforce being set (the default behavior)
-jobs will be executed based upon policies configured in SLURM on each cluster.
-This option will prevent users from accessing invalid accounts.
-Setting this to "2" will also cause association limits to be
-enforced. When set to "1" association limits will not be
-enforced. It is a good idea to run in this mode when running a
+jobs will be executed based upon policies configured in SLURM on each
+cluster.
+<br>
+It is a good idea to run without the option 'limits' set when running a
scheduler on top of slurm, like Moab, that does not update in real
time their limits per association.</li>
@@ -245,6 +265,13 @@ Set to "accounting_storage/slurmdbd".</li>
<li><b>ClusterName</b>:
Set to a unique name for each Slurm-managed cluster so that
accounting records from each can be identified.</li>
+<li><b>TrackWCKey</b>:
+Boolean. If you want to track wckeys (Workload Characterization Key)
+ of users. A Wckey is an orthogonal way to do accounting against
+ maybe a group of unrelated accounts. WCKeys can be definded using
+ sacctmgr add wckey 'name'. When a job is run use srun --wckey and
+ time will be summed up for this wckey.
+</li>
</ul>
<h2>SlurmDBD Configuration</h2>
@@ -446,8 +473,8 @@ with a default account of <i>test</i> execute:</p>
sacctmgr add user da default=test
</pre>
-<p>If <b>AccountingStorageEnforce=1</b> is configured in the slurm.conf of
-the cluster <i>snowflake</i> then user <i>da</i> would be
+<p>If <b>AccountingStorageEnforce=associations</b> is configured in
+the slurm.conf of the cluster <i>snowflake</i> then user <i>da</i> would be
allowed to run in account <i>test</i> and any other accounts added
in the future.
Any attempt to use other accounts will result in the job being
@@ -459,7 +486,6 @@ the job submission command.</p>
Partition='partitionname' option to specify an association specific to
a slurm partition.</p>
-<!-- For future use
<h2>Cluster Options</h2>
<p>When either adding or modifying a cluster, these are the options
@@ -467,48 +493,28 @@ available with sacctmgr:
<ul>
<li><b>Name=</b> Cluster name</li>
-<li><b>Fairshare=</b> Used for determining priority</li>
-
-<li><b>MaxJobs=</b> Limit number of jobs a user can run in this account</li>
-
-<li><b>MaxNodes=</b>Limit number of nodes a user can allocate in this
-account</li>
-
-<li><b>MaxWall=</b>Limit wall clock time a job can run</li>
-
-<li><b>MaxCPUSecs=</b> Limit cpu seconds a job can run</li>
</ul>
-!-->
<h2>Account Options</h2>
<p>When either adding or modifying an account, the following sacctmgr
options are available:
<ul>
-<li><b>Description=</b> Description of the account. (Required on creation)</li>
-
-<li><b>Organization=</b>Organization of the account. (Required on creation)</li>
-
-<li><b>Name=</b> Name of account</li>
-
<li><b>Cluster=</b> Only add this account to these clusters.
The account is added to all defined clusters by default.</li>
-<li><b>Parent=</b> Make this account a child of this other account.</li>
-
-<!-- For future use
-<li><b>QOS=</b> Quality of Service</li>
-
-<li><b>Fairshare=</b> Used for determining priority</li>
+<li><b>Description=</b> Description of the account. (Default is
+ account name)</li>
-<li><b>MaxJobs=</b> Limit number of jobs a user can run in this account</li>
+<li><b>Name=</b> Name of account</li>
-<li><b>MaxNodes=</b>Limit number of nodes a user can allocate in this account</li>
+<li><b>Organization=</b>Organization of the account. (Default is
+ parent account unless parent account is root then organization is
+ set to the account name.)</li>
-<li><b>MaxWall=</b>Limit wall time a job can run</li>
+<li><b>Parent=</b> Make this account a child of this other account
+ (already added).</li>
-<li><b>MaxCPUSecs=</b> Limit cpu seconds a job can run</li>
-!-->
</ul>
<h2>User Options</h2>
@@ -517,10 +523,7 @@ The account is added to all defined clusters by default.</li>
options are available:
<ul>
-<li><b>Name=</b> User name</li>
-
-<li><b>DefaultAccount=</b> Default account for the user, used when no account
-is specified when a job is submitted. (Required on creation)</li>
+<li><b>Account=</b> Account(s) to add user to</li>
<li><b>AdminLevel=</b> This field is used to allow a user to add accounting
privileges to this user. Valid options are
@@ -531,28 +534,20 @@ privileges to this user. Valid options are
and remove accounts and clusters</li>
</ul>
-<li><b>Account=</b> Account(s) to add user to</li>
-
<li><b>Cluster=</b> Only add to accounts on these clusters (default is all clusters)</li>
-<li><b>Partition=</b> Name of Slurm partition this association applies to</li>
-
-<!-- For future use
-<li><b>QOS=</b> Quality of Service</li>
-
-<li><b>Fairshare=</b> Used for determining priority</li>
+<li><b>DefaultAccount=</b> Default account for the user, used when no account
+is specified when a job is submitted. (Required on creation)</li>
-<li><b>MaxJobs=</b> Limit number of jobs a user can run in this account</li>
+<li><b>DefaultWCKey=</b> Default wckey for the user, used when no wckey
+is specified when a job is submitted. (Only used when tracking wckeys.)</li>
-<li><b>MaxNodes=</b> Limit number of nodes a user can allocate in this account</li>
+<li><b>Name=</b> User name</li>
-<li><b>MaxWall=</b> Limit wall time a job can run</li>
+<li><b>Partition=</b> Name of Slurm partition this association applies to</li>
-<li><b>MaxCPUSecs=</b> Limit cpu seconds a job can run</li>
-!-->
</ul>
-<!-- For future use
<h2>Limit enforcement</h2>
<p>When limits are developed they will work in this order...
@@ -561,7 +556,95 @@ if not we will refer to the account associated with the job.
If the account doesn't have the limit set we will refer to
the cluster's limits.
If the cluster doesn't have the limit set no limit will be enforced.
-!-->
+<p>All of the above entities can include limits as described below...
+
+<ul>
+
+<li><b>Fairshare=</b> Used for determining priority. Essentially
+ this is the amount of claim this association and it's childern have
+ to the above system.</li>
+</li>
+
+<!-- For future use
+<li><b>GrpCPUMins=</b> A hard limit of cpu minutes to be used by jobs
+ running from this association and its childern. If this limit is
+ reached all jobs running in this group will be killed, and no new
+ jobs will be allowed to run.
+</li>
+-->
+
+<!-- For future use
+<li><b>GrpCPUs=</b> The total count of cpus able to be used at any given
+ time from jobs running from this association and its childern. If
+ this limit is reached new jobs will be queued but only allowed to
+ run after resources have been relinquished from this group.
+</li>
+-->
+
+<li><b>GrpJobs=</b> The total number of jobs able to run at any given
+ time from this association and its childern. If
+ this limit is reached new jobs will be queued but only allowed to
+ run after previous jobs complete from this group.
+</li>
+
+<li><b>GrpNodes=</b> The total count of nodes able to be used at any given
+ time from jobs running from this association and its childern. If
+ this limit is reached new jobs will be queued but only allowed to
+ run after resources have been relinquished from this group.
+</li>
+
+<li><b>GrpSubmitJobs=</b> The total number of jobs able to be submitted
+ to the system at any given time from this association and its childern. If
+ this limit is reached new submission requests will be denied until
+ previous jobs complete from this group.
+</li>
+
+<li><b>GrpWall=</b> The maximum wall clock time any job submitted to
+ this group can run for. If this limit is reached submission requests
+ will be denied.
+</li>
+
+<!-- For future use
+<li><b>MaxCPUMinsPerJob=</b> A limit of cpu minutes to be used by jobs
+ running from this association. If this limit is
+ reached the job will be killed will be allowed to run.
+</li>
+-->
+
+<!-- For future use
+<li><b>MaxCPUsPerJob=</b> The maximum size in cpus any given job can
+ have from this association. If this limit is reached the job will
+ be denied at submission.
+</li>
+-->
+
+<li><b>MaxJobs=</b> The total number of jobs able to run at any given
+ time from this association. If this limit is reached new jobs will
+ be queued but only allowed to run after previous jobs complete from
+ this association.
+</li>
+
+<li><b>MaxNodesPerJob=</b> The maximum size in nodes any given job can
+ have from this association. If this limit is reached the job will
+ be denied at submission.
+</li>
+
+<li><b>MaxSubmitJobs=</b> The maximum number of jobs able to be submitted
+ to the system at any given time from this association. If
+ this limit is reached new submission requests will be denied until
+ previous jobs complete from this association.
+</li>
+
+<li><b>MaxWallDurationPerJob=</b> The maximum wall clock time any job
+ submitted to this association can run for. If this limit is reached
+ the job will be denied at submission.
+</li>
+
+<li><b>QOS=</b> comma separated list of QOS's this association is
+ able to run.
+
+</li>
+</ul>
<h2>Modifying Entities</h2>
diff --git a/doc/man/man1/sreport.1 b/doc/man/man1/sreport.1
index 1ff369b2444..489ee722d0f 100644
--- a/doc/man/man1/sreport.1
+++ b/doc/man/man1/sreport.1
@@ -291,21 +291,21 @@ List of users jobs to include in report. Default is all.
\fBCluster\fP
AccountUtilizationByUser
UserUtilizationByAccount
- - Accounts, Cluster, CPUCount, Login, Proper, Used
+ \- Accounts, Cluster, CPUCount, Login, Proper, Used
UserUtilizationByWckey
WCKeyUtilizationByUser
- - Cluster, CPUCount, Login, Proper, Used, Wckey
+ \- Cluster, CPUCount, Login, Proper, Used, Wckey
Utilization
- - Allocated, Cluster, CPUCount, Down, Idle, Overcommited,
+ \- Allocated, Cluster, CPUCount, Down, Idle, Overcommited,
PlannedDown, Reported, Reserved
\fBJob\fP
Sizes
- - Account, Cluster
+ \- Account, Cluster
\fBUser\fP
TopUsage
- - Account, Cluster, Login, Proper, Used
+ \- Account, Cluster, Login, Proper, Used
.TP
All commands and options are case-insensitive.
@@ -330,7 +330,7 @@ the specified 24 hour day of February 23, 2009, by user gore1
Report cluster account utilization by user in the LC account on
cluster zeus
.TP
-\fBsreport user topusage start=2/16/09 end=2/23/09 -t percent account=lc\fP
+\fBsreport user topusage start=2/16/09 end=2/23/09 \-t percent account=lc\fP
Report top usage in percent of the lc account during the specified week
.TP
diff --git a/src/plugins/accounting_storage/mysql/accounting_storage_mysql.c b/src/plugins/accounting_storage/mysql/accounting_storage_mysql.c
index a6f8bd090a0..091c8bf9a49 100644
--- a/src/plugins/accounting_storage/mysql/accounting_storage_mysql.c
+++ b/src/plugins/accounting_storage/mysql/accounting_storage_mysql.c
@@ -2572,7 +2572,6 @@ static int _mysql_acct_check_tables(MYSQL *db_conn)
{ "control_host", "tinytext not null default ''" },
{ "control_port", "mediumint not null default 0" },
{ "rpc_version", "mediumint not null default 0" },
- { "valid_qos", "blob" },
{ NULL, NULL}
};
@@ -7739,14 +7738,12 @@ extern List acct_storage_p_get_clusters(mysql_conn_t *mysql_conn, uid_t uid,
"control_host",
"control_port",
"rpc_version",
- "valid_qos",
};
enum {
CLUSTER_REQ_NAME,
CLUSTER_REQ_CH,
CLUSTER_REQ_CP,
CLUSTER_REQ_VERSION,
- CLUSTER_REQ_VALID_QOS,
CLUSTER_REQ_COUNT
};
@@ -7833,13 +7830,6 @@ empty:
cluster->control_host = xstrdup(row[CLUSTER_REQ_CH]);
cluster->control_port = atoi(row[CLUSTER_REQ_CP]);
cluster->rpc_version = atoi(row[CLUSTER_REQ_VERSION]);
- cluster->valid_qos_list = list_create(slurm_destroy_char);
-
- if(row[CLUSTER_REQ_VALID_QOS] && row[CLUSTER_REQ_VALID_QOS][0])
- slurm_addto_char_list(assoc->qos_list,
- row[CLUSTER_REQ_VALID_QOS]+1);
- else
- list_append(cluster->valid_qos_list, xstrdup("all"));
}
mysql_free_result(result);
diff --git a/src/plugins/proctrack/rms/proctrack_rms.c b/src/plugins/proctrack/rms/proctrack_rms.c
index 8378aa40264..0e56f034963 100644
--- a/src/plugins/proctrack/rms/proctrack_rms.c
+++ b/src/plugins/proctrack/rms/proctrack_rms.c
@@ -94,7 +94,7 @@ extern int slurm_container_create (slurmd_job_t *job)
/*
* Return a handle to an existing prgid or create a new one
*/
- if (rms_getprgid (job->jmgr_pid, &prgid) < 0) {
+ if (rms_getprgid ((int) job->jmgr_pid, &prgid) < 0) {
int fd = _prg_destructor_fork();
/* Use slurmd job-step manager's pid as a unique identifier */
prgid = job->jmgr_pid;
diff --git a/src/sacctmgr/account_functions.c b/src/sacctmgr/account_functions.c
index 72e05bd84e6..99a4e6e3cea 100644
--- a/src/sacctmgr/account_functions.c
+++ b/src/sacctmgr/account_functions.c
@@ -102,7 +102,9 @@ static int _set_cond(int *start, int argc, char *argv[],
|| !strncasecmp (argv[i], "Names",
MAX(command_len, 1))
|| !strncasecmp (argv[i], "Accounts",
- MAX(command_len, 1))) {
+ MAX(command_len, 1))
+ || !strncasecmp (argv[i], "Acct",
+ MAX(command_len, 4))) {
if(!assoc_cond->acct_list) {
assoc_cond->acct_list =
list_create(slurm_destroy_char);
@@ -340,7 +342,9 @@ static int _set_rec(int *start, int argc, char *argv[],
|| !strncasecmp (argv[i], "Account",
MAX(command_len, 1))
|| !strncasecmp (argv[i], "Names",
- MAX(command_len, 1))) {
+ MAX(command_len, 1))
+ || !strncasecmp (argv[i], "Acct",
+ MAX(command_len, 4))) {
if(acct_list)
slurm_addto_char_list(acct_list, argv[i]+end);
diff --git a/src/sacctmgr/association_functions.c b/src/sacctmgr/association_functions.c
index 2b5a0c44a2a..649011d272b 100644
--- a/src/sacctmgr/association_functions.c
+++ b/src/sacctmgr/association_functions.c
@@ -109,7 +109,9 @@ static int _set_cond(int *start, int argc, char *argv[],
list_iterator_destroy(itr);
set = 1;
} else if (!strncasecmp (argv[i], "Accounts",
- MAX(command_len, 2))) {
+ MAX(command_len, 2))
+ || !strncasecmp (argv[i], "Acct",
+ MAX(command_len, 4))) {
if(!assoc_cond->acct_list)
assoc_cond->acct_list =
list_create(slurm_destroy_char);
@@ -374,7 +376,8 @@ extern int sacctmgr_list_association(int argc, char *argv[])
field = xmalloc(sizeof(print_field_t));
- if(!strncasecmp("Account", object, MAX(command_len, 1))) {
+ if(!strncasecmp("Account", object, MAX(command_len, 1))
+ || !strncasecmp (argv[i], "Acct", MAX(command_len, 4))) {
field->type = PRINT_ACCOUNT;
field->name = xstrdup("Account");
if(tree_display)
diff --git a/src/sacctmgr/cluster_functions.c b/src/sacctmgr/cluster_functions.c
index 631582ff15b..a5613b3eac2 100644
--- a/src/sacctmgr/cluster_functions.c
+++ b/src/sacctmgr/cluster_functions.c
@@ -212,7 +212,7 @@ static int _set_rec(int *start, int argc, char *argv[],
" Bad MaxWall time format: %s\n",
argv[i]);
}
- } else if (!strncasecmp (argv[i], "QosLevel",
+ } else if (!strncasecmp (argv[i], "QOSLevel",
MAX(command_len, 1))) {
if(!assoc->qos_list)
assoc->qos_list =
diff --git a/src/sacctmgr/sacctmgr.c b/src/sacctmgr/sacctmgr.c
index f1963f759f0..cfd0ce54822 100644
--- a/src/sacctmgr/sacctmgr.c
+++ b/src/sacctmgr/sacctmgr.c
@@ -502,7 +502,8 @@ static void _add_it (int argc, char *argv[])
acct_storage_g_commit(db_conn, 0);
/* First identify the entity to add */
- if (strncasecmp (argv[0], "Account", MAX(command_len, 1)) == 0) {
+ if (strncasecmp (argv[0], "Account", MAX(command_len, 1)) == 0
+ || !strncasecmp (argv[0], "Acct", MAX(command_len, 4))) {
error_code = sacctmgr_add_account((argc - 1), &argv[1]);
} else if (strncasecmp (argv[0], "Cluster", MAX(command_len, 2)) == 0) {
error_code = sacctmgr_add_cluster((argc - 1), &argv[1]);
@@ -589,7 +590,8 @@ static void _show_it (int argc, char *argv[])
acct_storage_g_commit(db_conn, 0);
/* First identify the entity to list */
- if (strncasecmp (argv[0], "Accounts", MAX(command_len, 2)) == 0) {
+ if (strncasecmp (argv[0], "Accounts", MAX(command_len, 2)) == 0
+ || !strncasecmp (argv[0], "Acct", MAX(command_len, 4))) {
error_code = sacctmgr_list_account((argc - 1), &argv[1]);
} else if (strncasecmp (argv[0], "Associations",
MAX(command_len, 2)) == 0) {
@@ -649,7 +651,8 @@ static void _modify_it (int argc, char *argv[])
acct_storage_g_commit(db_conn, 0);
/* First identify the entity to modify */
- if (strncasecmp (argv[0], "Accounts", MAX(command_len, 1)) == 0) {
+ if (strncasecmp (argv[0], "Accounts", MAX(command_len, 1)) == 0
+ || !strncasecmp (argv[0], "Acct", MAX(command_len, 4))) {
error_code = sacctmgr_modify_account((argc - 1), &argv[1]);
} else if (strncasecmp (argv[0], "Clusters",
MAX(command_len, 1)) == 0) {
@@ -693,7 +696,8 @@ static void _delete_it (int argc, char *argv[])
acct_storage_g_commit(db_conn, 0);
/* First identify the entity to delete */
- if (strncasecmp (argv[0], "Accounts", MAX(command_len, 1)) == 0) {
+ if (strncasecmp (argv[0], "Accounts", MAX(command_len, 1)) == 0
+ || !strncasecmp (argv[0], "Acct", MAX(command_len, 4))) {
error_code = sacctmgr_delete_account((argc - 1), &argv[1]);
} else if (strncasecmp (argv[0], "Clusters",
MAX(command_len, 2)) == 0) {
diff --git a/src/sacctmgr/user_functions.c b/src/sacctmgr/user_functions.c
index 784326ecd2c..8496b000a06 100644
--- a/src/sacctmgr/user_functions.c
+++ b/src/sacctmgr/user_functions.c
@@ -113,7 +113,9 @@ static int _set_cond(int *start, int argc, char *argv[],
argv[i]+end))
u_set = 1;
} else if (!strncasecmp (argv[i], "Account",
- MAX(command_len, 2))) {
+ MAX(command_len, 2))
+ || !strncasecmp (argv[i], "Acct",
+ MAX(command_len, 4))) {
if(!assoc_cond->acct_list) {
assoc_cond->acct_list =
list_create(slurm_destroy_char);
@@ -705,7 +707,9 @@ extern int sacctmgr_add_user(int argc, char *argv[])
slurm_addto_char_list(assoc_cond->user_list,
argv[i]+end);
} else if (!strncasecmp (argv[i], "Accounts",
- MAX(command_len, 2))) {
+ MAX(command_len, 2))
+ || !strncasecmp (argv[i], "Acct",
+ MAX(command_len, 4))) {
slurm_addto_char_list(assoc_cond->acct_list,
argv[i]+end);
} else if (!strncasecmp (argv[i], "AdminLevel",
@@ -1539,7 +1543,8 @@ extern int sacctmgr_list_user(int argc, char *argv[])
command_len = strlen(object);
field = xmalloc(sizeof(print_field_t));
- if(!strncasecmp("Account", object, MAX(command_len, 2))) {
+ if(!strncasecmp("Account", object, MAX(command_len, 2))
+ || !strncasecmp (argv[i], "Acct", MAX(command_len, 4))) {
field->type = PRINT_ACCOUNT;
field->name = xstrdup("Account");
field->len = 10;
diff --git a/src/slurmctld/acct_policy.c b/src/slurmctld/acct_policy.c
index 3c80affc2fc..21c895b245b 100644
--- a/src/slurmctld/acct_policy.c
+++ b/src/slurmctld/acct_policy.c
@@ -304,7 +304,7 @@ extern bool acct_policy_job_runnable(struct job_record *job_ptr)
/* we don't need to check submit_jobs here */
/* FIX ME: Once we start tracking time of running jobs
- * we will need toupdate the amount of time we have
+ * we will need to update the amount of time we have
* used and check against that here. When we start
* keeping track of time we will also need to come up
* with a way to refresh the time.
--
GitLab