we came across the following error message in the slurmctld logs when
using non-consumable resources:

error: gres/potion: job 39 dealloc of node node1 bad node_offset 0 count
is 0

The error comes from _job_dealloc():

node_gres_data=0x7f8a18000b70, node_offset=0, gres_name=0x1999e00
"potion", job_id=46,
    node_name=0x1987ab0 "node1") at gres.c:3980
(job_gres_list=0x199b7c0, node_gres_list=0x199bc38, node_offset=0,
job_id=46,
    node_name=0x1987ab0 "node1") at gres.c:4190
job_ptr=0x19e9d50, pre_err=0x7f8a31353cb0 "_will_run_test", remove_all=true)
    at select_linear.c:2091
bitmap=0x7f8a18001ad0, min_nodes=1, max_nodes=1, max_share=1, req_nodes=1,
    preemptee_candidates=0x0, preemptee_job_list=0x7f8a2f910c40) at
select_linear.c:3176
bitmap=0x7f8a18001ad0, min_nodes=1, max_nodes=1, req_nodes=1, mode=2,
    preemptee_candidates=0x0, preemptee_job_list=0x7f8a2f910c40,
exc_core_bitmap=0x0) at select_linear.c:3390
bitmap=0x7f8a18001ad0, min_nodes=1, max_nodes=1, req_nodes=1, mode=2,
    preemptee_candidates=0x0, preemptee_job_list=0x7f8a2f910c40,
exc_core_bitmap=0x0) at node_select.c:588
avail_bitmap=0x7f8a2f910d38, min_nodes=1, max_nodes=1, req_nodes=1,
exc_core_bitmap=0x0)
    at backfill.c:367

The cause of this problem is that _node_state_dup() in gres.c does not
duplicate the no_consume flag.
The cr_ptr passed to _rm_job_from_nodes() is created with _dup_cr()
which calls _node_state_dup().

Below is a simple patch to fix the problem. A "future-proof" alternative
might be to memcpy() from gres_ptr to new_gres and
only handle pointers separately.

Bug 1471

runs certain sreport reports.

Added "--mail=stage_out" option to job submission commands to notify user
when burst buffer state out is complete.

Add SLURM_JOB_CONSTAINTS to environment variables available to the Prolog.
bug 1458

Add job credential to "Run Prolog" RPC used with a configuration of
PrologFlags=alloc. This allows the Prolog to be passed identification of
GPUs allocated to the job.

Add SLURM_JOB_GPUS environment variable to those available in Prolog.
Also add list of environment variables available in the various
prologs and epilogs on the web page.
bug 1458

in the runjob_mux plugin.

Bug 1461
Commit: 2e2d924e

Bug 1446

don't insert a new row in the event table.

Bug 1321

uid's are 0 when associations are loaded.

 Fix segfault in controller when deleting a user association of a user which had been previously removed from the system.

 Bug 1238

Fix slurmctld initialization problem which could cause requeue of the last
task in a job array to fail if executed prior to the slurmctld loading
the maximum size of a job array into a variable in the job_mgr.c module.

Fix slurmctld job recovery logic which could cause the last task in a job
array to be lost on restart.

In order to support inter-cluster job dependencies, the MaxJobID
configuration parameter default value has been reduced from 4,294,901,760
to 2,147,418,112 and it's maximum value is now 2,147,463,647.
ANY JOBS WITH A JOB ID ABOVE 2,147,463,647 WILL BE PURGED WHEN SLURM IS
UPGRADED FROM AN OLDER VERSION!

 -- Add job submission command options: --sicp (available for inter-cluster
    dependencies) and --power (specify power management options) to salloc,
    sbatch, and srun commands.
 -- Add DebugFlags option of SICP (inter-cluster option logging).
 -- Added DebugFlags value "SICP".
 -- Added job_descriptor field of "cluster".

event  REQUEUED to slurmdbd.

Previously it was not possible to distinguish between a job needing
exclusive nodes and the default job/partition configuration.

Fix job array logic that can cause slurmctld to abort.
bug 1426

Enable CUDA v7.0+ use with a Slurm configuration of TaskPlugin=task/cgroup
ConstrainDevices=yes (in cgroup.conf). With that configuration
CUDA_VISIBLE_DEVICES will start at 0 rather than the device number.
bug 1421

debug2 instead of info.

SLURM_JOB_PARTITION to be the one in which the job started.

If using proctrack/cgroup and gres/gpu, always start CUDA_VISIBLE_DEVICES
environment variable numbering at 0.
bug 1421

reservations.

This is an add on to commit 2e5142ef.  Servicing Bug 1418.

the partition in which the job runs.

Front End systems.