Fixes problem running very large number of simultaneous jobs when
their inactivity limit was reached due to a backlog of message.

data structure. This eliminates risks associated with re-reading slurm.conf.

responding. Wait for tasks to complete or administrator to set DOWN.

slurmctld to exit.

with SIGCONT.

requests can get backed up in a queue while the job actually completes).

away at any time anyway.

src/api/free_msg.c.

queued allocation request is satisfied. Srun just has a stub to catch
and log the message while using polling to notice the allocation has
been made.

and/or job step(s) will have their resources de-allocated and be killed.
A resource allocation will not be release unless no job steps are active
for at least InactiveLimit seconds. DPCS jobs will be subject to this
forced de-allocation if they remain inactive for an extended period of
time, which can get SLURM and DPCS back in sync if DPCS does a cold-start.

Send multiple SIGALRMs if needed and deal with possible abort of
a thread.

scalability. An arbitrary number of requests may be queued and they
are processed one per second until the queue is empty or pending
requests were last attempted recently (configuration parameters set
to 60 seconds as a minimum retry interval).

when the job does not exist).

to job_kill request and slurmctld leaves node and job in COMPLETING state
until the slurmd issues an EPILOG_COMPLETE RPC on each node. This permits
better support for non-killable processes and/or long-running epilog scripts.
Several minor changes in node registration handling and slurmctld agent
logic to better address a flood of incomming RPC (typically when system
restarts).
(gnats:268)

batch_job_launch RPC, then deallocate those resources and requeue the job.
If a node registers and fails to show a batch job that should have a
script running there (node zero of allocation), then consider the job
complete.

"Can't connect to node" with every ping failure.

Changed the logging level of a few other message.

until the previous one completes. This avoids having too many cycles
active (and a bunch of threads too). Ping_nodes control functions moved
to a new module.

Perform some general code clean-up in those areas.

while fail/if too many then fatal/sleep and retry

out message was sent (e.g., slurmd down, msg sent to slurmd, slurmd up
and registers, msg previously sent to slurmd times out).

rather than letting agent go off the end of an array.

   failed (presumably due to unkillable processes)
 o retry failed JOB_KILL rpcs

error).

RPC in agent.

   instead of just kill_wait seconds because slurmd sleeps for kill_wait
   seconds, so therefore slurmctld would never recv a reply.

   slurm_send_recv_node_msg(), slurm_send_recv_rc_msg(), etc.
 o Fixed fd leak in agent.c using slurm_send_recv_rc_msg() w/ timeout.

reference if a reconfigure RPC was active at the same time.

This would occur for example if we killed a job that never started any
steps on a node.

improved performance in production.