This creates an empty directory, since we no longer bother
copying the job script or environment as of 28b7f853.

** CID 175193: (FORWARD_NULL)

Theoretically we shouldn't have a job_desc_msg_t without an associated
part_record, but just in case let's harden the code.

Introduced in previous commit: 24365514.

Initially job mem limits were tested at submission time through
_validate_min_mem_partition() -> _valid_pn_min_mem(), but not tested
again at scheduling time, thus leading to jobs incorrectly being scheduled
against partitions where the job exceeded their MaxMemPer* limit
(which can in turn be inherited from the system wide limit too).

NOTE: New WAIT_PN_MEM_LIMIT job_state_reason enum component added to support
this new waiting reason.

Bug 2291.

checked on submit.

This only mattered when submitting a job to multiple partitions.

Bug 4066

on node 0.

Bug 4035

Add the xassert() that the inline version has to _remove_job_hash().

Coverity CID 174399

This insures that the batch script can have appropriate environment
  variables set for all components (i.e. the node list, cpu count,
  etc. for all pack groups).

When the "scontrol update jobid=#" command specifies a pack job
leader, then modify all components of the pack job. To modiify only
the pack job leader, specify "scontrol update jobid=#+0".

Modify scontrol job hold/release and update to operate with heterogeneous
      job id specification (e.g. "scontrol hold 123+4").

Doing so would break the current scheduling logic.

This situation would be caused by an invalid user request, not a slurmctld error

In all of these cases, the input account name is NULL, so there
  should never be a failure. In every case, the returned association
  pointer is checked anyway.
Coverity CID 44719, 44720, 44721

Coverity, CID 45150

srun jobs that could start immediately and requested multiple partitions
didn't run in the highest priority partition if the highest priority
partition wasn't listed first.

It's possible that the scontrol show jobs will show the partition list
in priority order now that the job's partition list gets sorted by
priority.

Bug 4015

If a pack job is only partitially allocated resources (likely due
  due to limits), deallocate resources from those components which
  have been started and requeue them.

This change adds a new function and moves some logic around so that
  limits can be tested on a pack job as a whole (that logic still
  needs to be developed).

Don't requeue a batch pack job component that is not found node zero of
the allocation. Only the first pack job component is expected to have a
running script.

Bug 3953

This removes several define statements with different names in
  various functions

It wasn't doing it for origin jobs.

Previously remote jobs would be removed from the job_list as quickly as
possible to prevent collisions with requeued jobs and to clear up the
jobs and the orign job would stay around till minjobage on the origin.
But the origin job didn't have the details from the job that ran on a
remote cluster.

Now just don't show revoked jobs. The origin tracking job will remain as
revoked and not shown and the remote job will hang around for display
till minjobage. scontrol show jobs will show the job from the cluster
that ran the job. The job is requeuable as long as the origin job is
still in the origin cluster's job_list.

Just check for the revoked state instead of checking if it's a tracker
job since an origin job will be revoked if it can't run on the origin or
if it's running on a remote cluster.

No longer require part write lock.

Replace with direct references to the struct.

This allows the origin to be able to sync up jobs after it has been
down.

When a non-origin cluster is removed:
- running jobs remain - fed_details removed so it can't call home.
- origin cluster removes tracking job for running jobs
- pending jobs are removed.
- pending srun/sallocs don't get notified.
- other clusters remove removed cluster from viable and active sibs

When an origin cluster is removed:
- all pending jobs are removed from all clusters that had job.
- pending srun/sallocs are notified of termination
- running jobs remain.

bug 3757

more than 1 partition or when the partition is changed with scontrol.

Bug 3849

Continuation of b9719be2

CID: 170772, 170773
Introduced by commit: 250378c2

Bug 3502.

Bug 3888

Prevent segfault from pointer dereference to the QOS that is
being deleted.

Fix to commit 3e8aa451.