The gprof tool is showing most time is being consumed by the bit_test()
  function as called from the select plugin, which in turn was called
  by the backfill scheduler. These changes replace the for loop end-points.
  Previous logic tested for all possible nodes. The new logic identifes
  the first and last bit set in the node bitmap and uses those end-points
  instead. Node the logic to find the first and last bits set starts off
  with a word-based search (testing for a 64-bit zero value rather than
  testing each individual bit). The net result is a small performance
  improvement.
bug 2588

burst_buffer/cray - Decrement job's prolog_running counter if pre_run fails.
bug 2621

If a job is no longer in configuring state, then clear the prolog_running
  counter on slurmctld restart or reconfigure.
bug 2621

For case where job can't start and there are no running jobs
to remove in order to establish estimated start time.

When determining when a pending job will be able to start, rather
  than testing after removing each running job and trying to schedule
  the pending jobs, remove multiple jobs that all end about the
  same time before testing. This reduces the number of calls to
  the job placement logic, which is time consuming.

list_peek_next(), like list_next() but WITHOUT advancing the pointer

Document and log cases where max jobs per user or partition is
  equal or greater than the max jobs test. In that case, a single
  user can easily stop all backfill scheduling.

Fix for job "--contiguous" option that could cause job allocation/launch
    failure or slurmctld crash.
bug 2573

This reverts commit f559a55c.

constraints mattered in a job.

Details include:
A job doesn't request memory but the system is running
with CR_*MEMORY with no default memory limit and the job requests nodes
with features of different sizes.  Previously the order of constraints
mattered where the smaller memory node would need to be requested first
or the job would fail.

Bug 2608

Previous logic would get an account and/or QOS time limit and use
  that value to overwrite the incoming RPC's NO_VAL value, which
  would change a job's time limit when changing an unrelated
  field (e.g. priority, QOS, etc.).
bug 2610

Prevent use of NULL pointer and SEGV when changing a job's QOS when
  the slurmdbd is not configured.

Fix backfill scheduler race condition that could cause invalid pointer in
    select/cons_res plugin. Bug introduced in 15.08.9, commit:
    efd9d35e

The scenario is as follows
1. Backfill scheduler is running, then releases locks
2. Main scheduling loop starts a job "A"
3. Backfill scheduler resumes, finds job "A" in its queue and
   resets it's partition pointer.
4. Job "A" completes and tries to remove resource allocation record
   from select/cons_res data structure, but fails to find it because
   it is looking in the table for the wrong partition.
5. Job "A" record gets purged from slurmctld
6. Select/cons_res plugin attempts to operate on resource allocation
   data structure, finds pointer into the now purged data structure
   of job "A" and aborts or gets SEGV
Bug 2603

misleading.

canceled while launching.

Power/cray: Don't specify NID list to Cray APIs. If any of those nodes are
    not in a ready state, the API returned an error for ALL nodes rather than
    valid data for nodes in ready state.
bug 2332

and retries are done making the error message a little misleading.

rollup would effectively never run again.

bug 2575

and sort of bug 2596

There was a subtle bug in how tasks were bound to CPUs which could result
in an "infinite loop" error. The problem was various socket/core/threasd
calculations were based upon the resources allocated to a step rather than
all resources on the node and rounding errors could occur. Consider for
example a node with 2 sockets, 6 cores per socket and 2 threads per core.
On the idle node, a job requesting 14 CPUs is submitted. That job would
be allocted 4 cores on the first socket and 3 cores on the second socket.
The old logic would get the number of sockets for the job at 2 and the
number of cores at 7, then calculate the number of cores per socket at
7/2 or 3 (rounding down to an integer). The logic layouting out tasks
would bind the first 3 cores on each socket to the job then not find any
remaining cores, report the "infinite loop" error to the user, and run
the job without one of the expected cores. The problem gets even worse
when there are some allocated cores on a node. In a more extreme case,
a job might be allocated 6 cores on one socket and 1 core on a second
socket. In that case, 3 of that job's cores would be unused.
bug 2502

This is a revision to commit 1ed38f26
The root problem is that a pthread is passed an argument which is
a pointer to a variable on the stack. If that variable is over-written,
the signal number recieved will be garbage, and that bad signal
number will be interpretted by srun to possible abort the request.

The previous commit obviously fixed a problem, but introduced a different
set of problems. This will be pursued later, perhaps in version 16.05.

With some configurations and systems, errors of the following sort were
occuring:
task/cgroup: task[1] infinite loop broken while trying to provision compute elements using block
task/cgroup: task[1] unable to set taskset '0x0'

burst_buffer/cray - If the pre-run operation fails then don't issue
    duplicate job cancel/requeue unless the job is still in run state. Prevents
    jobs hung in COMPLETING state.
bug 2587

Running "scontrol reconfig" releases resources for jobs waiting for
  the completion of Node Health Check so that other jobs can run.
  Cray says to always wait for NHC to complete, but in extreme
  cases that can be 2 hours, during which the entire resource
  allocation for a job may be unusable. Per advice from NERSC,
  the logic to release resources is unchanged, but logging is
  added here.