Switch/nrt - Properly track usage of CAU and RDMA resources with multiple
tasks per compute node. Previous logic would allocate resources once per
task and then deallocate once per node, leaking CMA and RDMA resources
and preventing their use by future jobs.

If a job is held, then only release it with the "scontrol release <jobid>"
command rather than a simple reset of the job's priority. This is needed to
support job arrays better. Otherwise a priority reset of a job array
would free all requeued/held jobs from that job array rather than
leaving them held.

in 2.0 :)

Previously partition priority was only considered when used as a
component of a job's priority with the priority/multifactor plugin.
Now the partition priority is considered first, as documented,
and the job priority is considered second.
bug 764

Add --priority option to the salloc, sbatch and srun commands.

This code was originally put here to enforce checks to make sure jobs
didn't go over the limit.  If they didn't request the amount then we set
the limit and worked off that as if it were a request.

If we do this now we could get jobs deigned which would cancel the job at
submit with a very unrelated note as to why the job failed.

Since we now check this these limits after the node selection this isn't
needed.

In addition to accepting a job ID argument to the hold and release
commands also accept a job name (e.g. "scontrol hold my.bash")

Add a job's exit state (COMPLETED, FAILED, etc) and exit code to
email message
bug 737

Added job reason of "SchedTimeout" if the scheduler was not able to reach
the job to attempt scheduling it.

Added SchedulerParameter of batch_sched_delay to permit many batch jobs
to be submitted between each scheduling attempt to reduce overhead of
scheduling logic.

Set "Reason" field for all elements of a job array on short-circuited
scheduling for job arrays.
bug 748

Jobs dependent upon multiple other jobs may start prematurely, after
only some of the depdendencies are satisfied.
bug 746

Arguments of "abe" and "aeb" both failed.
bug 743

of limited non NPC jobs.

This reverts commit 3b676075.

plugin and provide more information in the message.

together with the remote IP address and port.

On switch resource allocation failure, free partial allocation.
Failure mode was CAU could be allocated on some nodes, but not
others. The CAU allocated on nodes and switches up to the failure
point were never released.

Don't block scheduling of entire job array if it could run in multiple
partitions.
bug 726

ephemeral port.

Requeue batch job if Munge is down and credential can not be created.
bug 727

Do not overwrite existing reason for node being down or drained.
Previously draining a node would always overwrite any previous
"reason".
bug 724

the current settings.

This reverts commit 2a72aa51.

Conflicts:
	NEWS

Decided to take this out and wait until 14.11 to fix this correctly instead
of this bandaid.

Previously if a job was dependent upon a job array and the first
element of that job array was no longer in slurmctld (completed
and the job record purged) and the slurmctld restarted or was
reconfigured, the dependent job would start rather than waiting
for all elements of the job array to complete.
bug 714

Add squeue -L/--licenses option to filter jobs by license names
bug 720

successive release requests.