scalability. An arbitrary number of requests may be queued and they
are processed one per second until the queue is empty or pending
requests were last attempted recently (configuration parameters set
to 60 seconds as a minimum retry interval).

when the job does not exist).

to job_kill request and slurmctld leaves node and job in COMPLETING state
until the slurmd issues an EPILOG_COMPLETE RPC on each node. This permits
better support for non-killable processes and/or long-running epilog scripts.
Several minor changes in node registration handling and slurmctld agent
logic to better address a flood of incomming RPC (typically when system
restarts).
(gnats:268)

batch_job_launch RPC, then deallocate those resources and requeue the job.
If a node registers and fails to show a batch job that should have a
script running there (node zero of allocation), then consider the job
complete.

"Can't connect to node" with every ping failure.

Changed the logging level of a few other message.

until the previous one completes. This avoids having too many cycles
active (and a bunch of threads too). Ping_nodes control functions moved
to a new module.

Perform some general code clean-up in those areas.

while fail/if too many then fatal/sleep and retry

out message was sent (e.g., slurmd down, msg sent to slurmd, slurmd up
and registers, msg previously sent to slurmd times out).

rather than letting agent go off the end of an array.

   failed (presumably due to unkillable processes)
 o retry failed JOB_KILL rpcs

error).

RPC in agent.

   instead of just kill_wait seconds because slurmd sleeps for kill_wait
   seconds, so therefore slurmctld would never recv a reply.

   slurm_send_recv_node_msg(), slurm_send_recv_rc_msg(), etc.
 o Fixed fd leak in agent.c using slurm_send_recv_rc_msg() w/ timeout.

reference if a reconfigure RPC was active at the same time.

This would occur for example if we killed a job that never started any
steps on a node.

improved performance in production.

slurmctld sends REQUEST_KILL_TIMELIMIT when job reaches its time limit,
change job state to TIMEOUT | COMPLETING and node state to COMPLETING.

Add COMPLETING node and job states.
Major restructuring of slurmctld node and job state transition code.

its nodes are put in state COMPLETING (used to be IDLE) and a revoke
credential RPC is issued to all slurmd's. When the slurmd's respond
(after running the epilog and confirming that all tasks are completed),
the node's state is changed to IDLE and other jobs can run.

Initialize/free/set alloc_sid and alloc_node in API functions.
Pack/unpack/free new elements in job descriptor RPCs.
Load/dump/pack new elements into job table records.

Add support in slurmctld for SlurmctldDebug on startup and reconfig RPC.

Slurmctld add node list to job step (leave bitmap too, faster)	Jette
Slurmctld fix node state when job keeps running and node fails	Jette
Slurmctld job requests with min-max resource requirement pair	Jette
Slurmctld job requests with excluded node list (--exclude)	Jette

job time limit. Defined pack/unpack/free functions for RPC.
Slurmctld modified to broadcast REQUEST_UPDATE_JOB_TIME RPCs.