NEWS

This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.

* Changes in Slurm 21.08.0rc2
=============================
 -- Load bf_when_last_cycle from job state only if protocol version >= 21.08.
 -- Docs - remove man3 section entirely.
 -- Enable OCI container execution support as a technical preview.
 -- Set step memory when using MemPerGPU or DefMemPerGPU. Previously a step's
    memory was not set even when it requested --mem-per-gpu and at least one
    GPU.

* Changes in Slurm 21.08.0rc1
=============================
 -- slurmrestd - add v0.0.37 OpenAPI plugin.
 -- slurmrestd/v0.0.37 - rename standard_in -> standard_input.
 -- slurmrestd/v0.0.37 - rename standard_out -> standard_output.
 -- slurmdbd - Improve log messages of more time than possible on rollups.
 -- Changed the --format handling for negative field widths (left justified)
    to apply to the column headers as well as the printed fields.
 -- Add LimitFactor to the QOS. A float that is factored into an associations
    [Grp|Max]TRES limits.  For example, if the LimitFactor is 2, then an
    association with a GrpTRES of 30 CPUs, would be allowed to allocate 60
    CPUs when running under this QOS.
 -- slurmrestd - Pass SLURM_NO_CHANGE_IN_DATA to client as 403 (Not Modified).
 -- slurmrestd/v0.0.37 - Add update_time field to Jobs query to allow clients
    to only get jobs list based on change timestamp.
 -- Reset job eligible time when job is manually held.
 -- Add DEBUG_FLAG_JAG to improve logging related to job account gathering.
 -- Convert logging in account_gather/common to DEBUG_FLAG_JAG.
 -- Add more logging for jag_common_poll_data() when prec_extra() called.
 -- slurmrestd/v0.0.37 - add API to fetch reservation(s) info.
 -- Catch more errors in task/cgroup initalization and cleanup to avoid allowing
    jobs to start when cgroups failure to configure correctly.
 -- Fix cgroup ns detection when using containers (e.g. LXC or Docker).
 -- Reset job's next_step_id counter to 0 after being requeued.
 -- Make scontrol exit with non-zero status after failing to delete a partition
    or reservation.
 -- Make NtasksPerTRES optional in slurm_sprint_job_info().
 -- slurmrestd/v0.0.37 - Add update_time field to nodes query to allow clients
    to only get nodes list based on change timestamp.
 -- common/parse_config - catch and propagate return codes when handling a match
    on a key-value pattern. This implies error codes detected in the handlers
    are now not ignored and users of _handle_keyvalue_match() can fatal().
 -- common/hostlist - fix hostlist_delete_nth() xassert() upper bound check.
 -- API change: Removed slurm_kill_job_msg and modified the function signature
    for slurm_kill_job2. slurm_kill_job2 should be used instead of
    slurm_kill_job_msg.
 -- Fix non-zero exit code for scontrol ping when all controllers are down.
 -- Enforce a valid configuration for AccountingStorageEnforce in slurm.conf.
    If the configuration is invalid, then an error message will be printed and
    the command or daemon (including slurmctld) will not run.
 -- slurmrestd/v0.0.37 - Add update_time field to partitions/reservations query
    to allow clients to only get the entities list when something changed.
 -- slurmdbd.service - add "After" relationship to all common names for MariaDB
    to reduce startup delays.
 -- slurmrestd/v0.0.37 - Correct displaying node states that are UNKNOWN.
 -- slurmrestd/v0.0.37 - Add flags to node states.
 -- Fix first job on fresh cluster not being assigned JobId=1 (or FirstJobId).
 -- squeue - make it so --nodelist is sensitive to --clusters.
 -- squeue - do --nodelist node validation in the same order as listing.
 -- Removed AccountingStoreJobComment option.  Please update your config to use
    AccountingStoreFlags=job_comment instead.
 -- AccountingStoreFlags=job_script allows you to store the job's batch script.
 -- AccountingStoreFlags=job_env allows you to store the job's env vars.
 -- Add sacct -o SubmitLine to get the submit line of a job/step.
 -- Removed DefaultStorage{Host,Loc,Pass,Port,Type,User} options.
 -- Fix NtasksPerTRES delimiter from : to = in scontrol show job output.
 -- Removed CacheGroups, CheckpointType, JobCheckpointDir, MemLimitEnforce,
    SchedulerPort, SchedulerRootFilter options.
 -- Make job accounting queries use consistent timeframes with and w/o jobs.
 -- --cpus-per-task and --threads-per-core now imply --exact.
    This fixes issues where steps would be allocated the wrong number of CPUs.
 -- configure: the --with option handling has been made consistent across the
    various optional libraries. Specifying --with-foo=/path/to/foo will only
    check that directory for the applicable library (rather than, in some cases,
    falling back to the default directories), and will always error the build
    if the library is not found (instead of a mix of error messages and non-
    fatal warning messages).
 -- configure: replace --with-rmsi_dir option with proper handling for
    --with-rsmi=dir.
 -- Pass additional job environment variables to MailProg.
 -- Add SLURM_JOB_WORK_DIR to Prolog, Epilog.
 -- Removed sched/hold plugin.
 -- Fix srun overwriting SLURM_SUBMIT_DIR and SLURM_SUBMIT_HOST when within an
    existing allocation.
 -- step_ctx code has been removed from the api.
 -- cli_filter/lua, jobcomp/lua, job_submit/lua now load their scripts from the
    same directory as the slurm.conf file (and thus now will respect changes
    to the SLURM_CONF environment variable).
 -- SPANK - call slurm_spank_init if defined without slurm_spank_slurmd_exit in
    slurmd context.
 -- job_container/tmpfs - Remove need for .active file to allow salloc without
    an interactive step to work.
 -- Add new 'PLANNED' state to a node to represent when the backfill scheduler
    has it planned to be used in the future instead of showing as 'IDLE'.
 -- slurmd - Delay background node registration on every failure up to 128s on
    startup.
 -- slurmctld - Always notify slurmd that node registration was accepted to
    avoid slurmd needless attempting to re-register if there is configuration
    issue.
 -- Put node into "INVAL" state upon registering with an invalid node
    configuration. Node must register with a valid configuration to continue.
 -- Make --cpu-bind=threads default for --threads-per-core -- cli and env can
    override.
 -- jobcomp/elasticsearch - Use data_t to serialize data. The plugin now has the
    JSON-C library as a prerequisite.
 -- scrontab - create the temporary file under the TMPDIR environment variable
    (if set), otherwise continue to use TmpFS as configured in slurm.conf.
 -- Add LastBusyTime to "scontrol show nodes" and slurmrestd nodes output,
    which represents the time the node last had jobs on it.
 -- slurmd - allow multiple comma-separated controllers to be specified in
    configless mode with --conf-server
 -- sacctmgr - changed column headings to "ParentID" and "ParentName" instead
    of "Par ID" and "Par Name" respectively.
 -- Perl API - make sure man pages are installed under the --prefix given to
    configure.
 -- Manually powering down of nodes with scontrol now ignores
    SuspendExc<Nodes|Parts>.
 -- SALLOC_THREADS_PER_CORE and SBATCH_THREADS_PER_CORE have been added as
    input environment variables for salloc and sbatch, respectively. They do
    the same thing as --threads-per-core.
 -- Distinguish queued reboot requests (REBOOT) from issued reboots (REBOOT^).
 -- Set the maximum number of open files per process to 4096 to avoid
    performance issues when closing the entire range with closeall().
 -- auth/jwt - add support for RS256 tokens.
 -- Relax reservation purge due to any invalid uid after creation time.
 -- Reject srun that requests both --exclusive and --overlap.
 -- service files - change dependency to network-online rather than just
    network to ensure DNS and other services are available.
 -- RSMI: Fix incorrect PCI BDF bits.
 -- plugins/cli_filter - Convert to using data_t to serialize JSON.
 -- Fix testing array job after regaining locks in backfill.
 -- Don't display node's comment with "scontrol show nodes" unless set.
 -- Add "Extra" field to node to store extra information other than a comment.
 -- scrontab - Use /tmp instead of TmpFS if TMPDIR is not set.
 -- Add ResumeTimeout, SuspendTimeout and SuspendTime to Partitions.
 -- sreport - change to sorting TopUsage by the --tres option.
 -- slurmrestd - do not run allow operation as SlurmUser/root by default.
 -- Allow map_cpu and mask_cpu for non-whole node allocation.
 -- TaskPluginParam=verbose is now treated as a default. Previously it would be
    applied regardless of the job specifying a --cpu-bind.
 -- Add "node_reg_mem_percent" SlurmctldParameter to define percentage of
    memory nodes are allowed to register with.
 -- Show correct number of SocketsPerBoard in slurmd -C with hwloc2.
 -- Alter sreport's cluster utilization report column name from
    'Reserved' to 'Planned' to match the nomenclature of the 'Planned' node.
 -- Add StateComplete format option to sinfo to show base_state+flags.
 -- "scontrol show node" now shows State as base_state+flags instead of
    shortened state with flags appended. eg. IDLE# -> IDLE+POWERING_UP.
    Also "POWER" state flag string is "POWERED_DOWN".
 -- slurmd/req - add missing job_env_t's het_job_id initialization off the
    request in _rpc_{abort,terminate}_job(). This caused problems for Native
    Cray builds when joining a CNCU job_container plugin with Epilog configured.
 -- Fix joining a CNCU job_container on a Native Cray build before executing the
    UnkillableStepProgram for a HetJob step.
 -- slurmrestd/v0.0.35 - Plugin has been tagged as deprecated.
 -- srun - Job steps requiring more cores than available to be rejected unless
    '--overlap' is specificed.
 -- Add bf_node_space_size to SchedulerParameters.
 -- Add scontrol update node state=POWER_DOWN_FORCE and POWER_DOWN_ASAP as new
    ways to power off and reset especially CLOUD nodes.
 -- Define and separate node power state transitions. Previously a powering
    down node was in both states, POWERING_OFF and POWERED_OFF. These are now
    separated.
 -- Create a new process called slurmscriptd which runs PrologSlurmctld and
    EpilogSlurmctld. This avoids fork() calls from slurmctld, and can avoid
    performance issues if the slurmctld has a large memory footprint.
 -- Added new Script option to DebugFlags for debugging slurmscriptd.
 -- scrontab - add ability to update crontab from a file or standard input.
 -- scrontab - add ability to set and expand variables.
 -- Pass JSON of job to node mappings to ResumeProgram.
 -- If running steps in an allocation with CR_PACK_NODE or -mpack the srun will
    only attempt to allocate as much as needed from the allocation instead
    of always trying to allocate every node in the allocation.
 -- Jobs that request the whole node now check to see if any gres are allocated.
 -- Rename SbcastParameters to BcastParameters.
 -- Make srun sensitive to BcastParameters.
 -- RSMI: Add gres_links_create_empty() and preserve RSMI enumeration order.
 -- GPUs: Use index instead of dev_num for CUDA_VISIBLE_DEVICES
 -- Don't run epilog on nodes if job never launched.
 -- QOS accrue limits only apply to the job QOS, not partition QOS.
 -- Add --gpu-bind=per_task:<gpus_per_task> option, --gpus-per-task will now
    set this option by default.
 -- Treat any return code from SPANK plugin that is not SLURM_SUCCESS to be an
    error or rejection.
 -- Print the statistics for extern step adopted processes in sstat.
 -- Fix SLURM_NODE_ALIASES to work for ipv6 node addrs.
 -- Add support for automatically detecting and broadcasting executable shared
    object dependencies for sbcast and srun --bcast.
 -- Delay steps when memory already used instead of rejecting step request.

* Changes in Slurm 20.11.9
==========================
 -- burst_buffer - add missing common directory to the Makefile SUBDIRS.
 -- sacct - fix truncation when printing jobidraw field.
 -- GRES - Fix loading state of jobs using --gpus to request gpus.
 -- Fix minor logic error in health check node state output
 -- Fix GCC 11.1 compiler warnings.

* Changes in Slurm 20.11.8
==========================
 -- slurmctld - fix erroneous "StepId=CORRUPT" messages in error logs.
 -- Correct the error given when auth plugin fails to pack a credential.
 -- Fix unused-variable compiler warning on FreeBSD in fd_resolve_path().
 -- acct_gather_filesystem/lustre - only emit collection error once per step.
 -- srun - leave SLURM_DIST_UNKNOWN as default for --interactive.
 -- Add GRES environment variables (e.g., CUDA_VISIBLE_DEVICES) into the
    interactive step, the same as is done for the batch step.
 -- Fix various potential deadlocks when altering objects in the database
    dealing with every cluster in the database.
 -- slurmrestd - handle slurmdbd connection failures without segfaulting.
 -- slurmrestd - fix segfault for searches in slurmdb/v0.0.36/jobs.
 -- slurmrestd - remove (non-functioning) users query parameter for
    slurmdb/v0.0.36/jobs from openapi.json
 -- slurmrestd - fix segfault in slurmrestd db/jobs with numeric queries
 -- slurmrestd - add argv handling for job/submit endpoint.
 -- srun - fix broken node step allocation in a heterogeneous allocation.
 -- Fail step creation if -n is not multiple of --ntasks-per-gpu.
 -- job_container/tmpfs - Fix slowdown on teardown.
 -- Fix problem with SlurmctldProlog where requeued jobs would never launch.
 -- job_container/tmpfs - Fix issue when restarting slurmd where the namespace
    mount points could disappear.
 -- sacct - avoid truncating JobId at 34 characters.
 -- scancel - fix segfault when --wckey filtering option is used.
 -- select/cons_tres - Fix memory leak.
 -- Prevent file descriptor leak in job_container/tmpfs on slurmd restart.
 -- slurmrestd/dbv0.0.36 - Fix values dumped in job state/current and
    job step state.
 -- slurmrestd/dbv0.0.36 - Correct description for previous state property.
 -- perlapi/libslurmdb - expose tres_req_str to job hash.
 -- scrontab - close and reopen temporary crontab file to deal with editors
    that do not change the original file, but instead write out then rename
    a new file.
 -- sstat - fix linking so that it will work when --without-shared-libslurm
    was used to build Slurm.
 -- Clear allocated cpus for running steps in a job before handling requested
    nodes on new step.
 -- Don't reject a step if not enough nodes are available. Instead, defer the
    step until enough nodes are available to satisfy the request.
 -- Don't reject a step if it requests at least one specific node that is
    already allocated to another step. Instead, defer the step until the
    requested node(s) become available.
 -- slurmrestd - add description for slurmdb/job endpoint.
 -- Better handling of --mem=0.
 -- Ignore DefCpuPerGpu when --cpus-per-task given.
 -- sacct - fix segfault when printing StepId (or when using --long).

* Changes in Slurm 20.11.7
==========================
 -- slurmd - handle configless failures gracefully instead of hanging
    indefinitely.
 -- select/cons_tres - fix Dragonfly topology not selecting nodes in the same
    leaf switch when it should as well as requests with --switches option.
 -- Fix issue where certain step requests wouldn't run if the first node in the
    job allocation was full and there were idle resources on other nodes in
    the job allocation.
 -- Fix deadlock issue with <Prolog|Epilog>Slurmctld.
 -- torque/qstat - fix printf error message in output.
 -- When adding associations or wckeys avoid checking multiple times a user or
    cluster name.
 -- Fix wrong jobacctgather information on a step on multiple nodes
    due to timeouts sending its the information gathered on its node.
 -- Fix missing xstrdup which could result in slurmctld segfault on array jobs.
 -- Fix security issue in PrologSlurmctld and EpilogSlurmctld by always
    prepending SPANK_ to all user-set environment variables. CVE-2021-31215.

* Changes in Slurm 20.11.6
==========================
 -- Fix sacct assert with the --qos option.
 -- Use pkg-config --atleast-version instead of --modversion for systemd.
 -- common/fd - fix getsockopt() call in fd_get_socket_error().
 -- Properly handle the return from fd_get_socket_error() in _conn_readable().
 -- cons_res - Fix issue where running jobs were not taken into consideration
    when creating a reservation.
 -- Avoid a deadlock between job_list for_each and assoc QOS_LOCK.
 -- Fix TRESRunMins usage for partition qos on restart/reconfig.
 -- Fix printing of number of tasks on a completed job that didn't request
    tasks.
 -- Fix updating GrpTRESRunMins when decrementing job time is bigger than it.
 -- Make it so we handle multithreaded allocations correctly when doing
    --exclusive or --core-spec allocations.
 -- Fix incorrect round-up division in _pick_step_cores
 -- Use appropriate math to adjust cpu counts when --ntasks-per-core=1.
 -- cons_tres - Fix consideration of power downed nodes.
 -- cons_tres - Fix DefCpuPerGPU, increase cpus-per-task to match with
    gpus-per-task * cpus-per-gpu.
 -- Fix under-cpu memory auto-adjustment when MaxMemPerCPU is set.
 -- Make it possible to override CR_CORE_DEFAULT_DIST_BLOCK.
 -- Perl API - fix retrieving/storing of slurm_step_id_t in job_step_info_t.
 -- Recover state of burst buffers when slurmctld is restarted to avoid skipping
    burst buffer stages.
 -- Fix race condition in burst buffer plugin which caused a burst buffer
    in stage-in to not get state saved if slurmctld stopped.
 -- auth/jwt - print an error if jwt_file= has not been set in slurmdbd.
 -- Fix RESV_DEL_HOLD not being a valid state when using squeue --states.
 -- Add missing squeue selectable states in valid states error message.
 -- Fix scheduling last array task multiple times on error, causing segfault.
 -- Fix issue where a step could be allocated more memory than the job when
    dealing with --mem-per-cpu and --threads-per-core.
 -- Fix removing qos from assoc with -= can lead to assoc with no qos
 -- auth/jwt - fix segfault on invalid credential in slurmdbd due to
    missing validate_slurm_user() function in context.
 -- Fix single Port= not being applied to range of nodes in slurm.conf
 -- Fix Jobs not requesting a tres are not starting because of that tres limit.
 -- acct_gather_energy/rapl - fix AveWatts calculation.
 -- job_container/tmpfs - Fix issues with cleanup and slurmd restarting on
    running jobs.

* Changes in Slurm 20.11.5
==========================
 -- Fix main scheduler bug where bf_hetjob_prio truncates SchedulerParameters.
 -- Fix sacct not displaying UserCPU, SystemCPU and TotalCPU for large times.
 -- scrontab - fix to return the correct index for a bad #SCRON option.
 -- scrontab - fix memory leak when invalid option found in #SCRON line.
 -- Add errno for when a user requests multiple partitions and they are using
    partition based associations.
 -- Fix issue where a job could run in a wrong partition when using
    EnforcePartLimits=any and partition based associations.
 -- Remove possible deadlock when adding associations/wckeys in multiple
    threads.
 -- When using PrologFlags=alloc make sure the correct Slurm version is set
    in the credential.
 -- When sending a job a warning signal make sure we always send SIGCONT
    beforehand.
 -- Fix issue where a batch job would continue running if a prolog failed on a
    node that wasn't the batch host and requeuing was disabled.
 -- Fix issue where sometimes salloc/srun wouldn't get a message about a prolog
    failure in the job's stdout.
 -- Requeue or kill job on a prolog failure when PrologFlags is not set.
 -- Fix race condition causing node reboots to get requeued before
    ResumeTimeout expires.
 -- Preserve node boot_req_time on reconfigure.
 -- Preserve node power_save_req_time on reconfigure.
 -- Fix node reboots being queued and issued multiple times and preventing the
    reboot to time out.
 -- Fix debug message related to GrpTRESRunMin (AssocGrpCPURunMinutesLimit).
 -- Fix run_command to exit correctly if track_script kills the calling thread.
 -- Only requeue a job when the PrologSlurmctld returns nonzero.
 -- When a job is signaled with SIGKILL make sure we flush all
    prologs/setup scripts.
 -- Handle burst buffer scripts if the job is canceled while stage_in is
    happening.
 -- When shutting down the slurmctld make note to ignore error message when
    we have to kill a prolog/setup script we are tracking.
 -- scrontab - add support for the --open-mode option.
 -- acct_gather_profile/influxdb - avoid segfault on plugin shutdown if setup
    has not completed successfully.
 -- Reduce delay in starting salloc allocations when running with prologs.
 -- Fix issue passing open fd's with [send|recv]msg.
 -- Alter AllocNodes check to work if the allocating node's domain doesn't
    match the slurmctld's. This restores the pre-20.11 behavior.
 -- Fix slurmctld segfault if jobs from a prior version had the now-removed
    INVALID_DEPEND state flag set and were allowed to run in 20.11.
 -- Add job_container/tmpfs plugin to give a method to provide a private /tmp
    per job.
 -- Set the correct core affinity when using AutoDetect.
 -- Start relying on the conf again in xcpuinfo_mac_to_abs().
 -- Fix global_last_rollup assignment on job resizing.
 -- slurmrestd - hand over connection context on _on_message_complete().
 -- slurmrestd - mark "environment" as required for job submissions in schema.
 -- slurmrestd - Disable credential reuse on the same TCP connection. Pipelined
    HTTP connections will have to provide authentication with every request.
 -- Avoid data conversion error on NULL strings in data_get_string_converted().
 -- Handle situation where slurmctld is too slow processing
    REQUEST_COMPLETE_BATCH_SCRIPT and it gets resent from the slurmstepd.
 -- Fix sacct crashing with the --qos option

* Changes in Slurm 20.11.4
==========================
 -- Fix node selection for advanced reservations with features.
 -- mpi/pmix: Handle pipe failure better when using ucx.
 -- mpi/pmix: include PMIX_NODEID for each process entry.
 -- Fix job getting rejected after being requeued on same node that died.
 -- job_submit/lua - add "network" field.
 -- Fix situations when a reoccuring reservation could erroneously skip a
    period.
 -- Ensure that a reservations [pro|epi]log are ran on reoccuring reservations.
 -- Fix threads-per-core memory allocation issue when using CR_CPU_MEMORY.
 -- Fix scheduling issue with --gpus.
 -- Fix gpu allocations that request --cpus-per-task.
 -- mpi/pmix: fixed print messages for all PMIXP_* macros
 -- Add mapping for XCPU to --signal option.
 -- Fix regression in 20.11 that prevented a full pass of the main scheduler
    from ever executing.
 -- Work around a glibc bug in which "0" is incorrectly printed as "nan"
    which will result in corrupted association state on restart.
 -- Fix regression in 20.11 which made slurmd incorrectly attempt to find the
    parent slurmd address when not applicable and send incorrect reverse-tree
    info to the slurmstepd.
 -- Fix cgroup ns detection when using containers (e.g. LXC or Docker).
 -- scrontab - change temporary file handling to work with emacs.

* Changes in Slurm 20.11.3
==========================
 -- Fix segfault when parsing bad "#SBATCH hetjob" directive.
 -- Allow countless gpu:<type> node GRES specifications in slurm.conf.
 -- PMIx - Don't set UCX_MEM_MMAP_RELOC for older version of UCX (pre 1.5).
 -- Don't green-light any GPU validation when core conversion fails.
 -- Allow updates to a reservation in the database that starts in the future.
 -- Better check/handling of primary key collision in reservation table.
 -- Improve reported error and logging in _build_node_list().
 -- Fix uninitialized variable in _rpc_file_bcast() which could lead to an
    incorrect error return from sbcast / srun --bcast.
 -- mpi/cray_shasta - fix use-after-free on error in _multi_prog_parse().
 -- Cray - Handle setting correct prefix for cpuset cgroup with respects to
    expected_usage_in_bytes.  This fixes Cray's OOM killer.
 -- mpi/pmix: Fix PMIx_Abort support.
 -- Don't reject jobs allocating more cores than tasks with MaxMemPerCPU.
 -- Fix false error message complaining about oversubscribe in cons_tres.
 -- scrontab - fix parsing of empty lines.
 -- Fix regression causing spank_process_option errors to be ignored.
 -- Avoid making multiple interactive steps.
 -- Fix corner case issues where step creation should fail.
 -- Fix job rejection when --gres is less than --gpus.
 -- Fix regression causing spank prolog/epilog not to be called unless the
    spank plugin was loaded in slurmd context.
 -- Fix regression preventing SLURM_HINT=nomultithread from being used
    to set defaults for salloc->srun, sbatch->srun sequence.
 -- Reject job credential if non-superuser sets the LAUNCH_NO_ALLOC flag.
 -- Make it so srun --no-allocate works again.
 -- jobacct_gather/linux - Don't count memory on tasks that have already
    finished.
 -- Fix 19.05/20.02 batch steps talking with a 20.11 slurmctld.
 -- jobacct_gather/common - Do not process jobacct's with same taskid when
    calling prec_extra.
 -- Cleanup all tracked jobacct tasks when extern step child process finishes.
 -- slurmrestd/dbv0.0.36 - Correct structure of dbv0.0.36_tres_list.
 -- Fix regression causing task/affinity and task/cgroup to be out of sync when
    configured ThreadsPerCore is different than the physical threads per core.
 -- Fix situation when --gpus is given but not max nodes (-N1-1) in a job
    allocation.
 -- Interactive step - ignore cpu bind and mem bind options, and do not set
    the associated environment variables which lead to unexpected behavior
    from srun commands launched within the interactive step.
 -- Handle exit code from pipe when using UCX with PMIx.
 -- Partially revert changes made in 20.11.0 to srun step behavior. This change
    required steps to explicitly request resources, otherwise the minimal set
    of resources required would be assigned. This broke OpenMPI's 'mpiexec'
    alongside other use cases. This reverts the behavior such that all resources
    on a node are assigned to the job step by default.
 -- srun - add a new --exact option, and deprecate the --whole option (which has
    been restored as the default behavior).

* Changes in Slurm 20.11.2
==========================
 -- Fix older versions of sacct not working with 20.11.
 -- Fix slurmctld crash when using a pre-20.11 srun in a job allocation.
 -- Correct logic problem in _validate_user_access.
 -- Fix libpmi to initialize Slurm configuration correctly.

* Changes in Slurm 20.11.1
==========================
 -- Fix spelling of "overcomited" to "overcomitted" in sreport's cluster
    utilization report.
 -- Silence debug message about shutting down backup controllers if none are
    configured.
 -- Don't create interactive srun until PrologSlurmctld is done.
 -- Fix fd symlink path resolution.
 -- Fix slurmctld segfault on subnode reservation restore after node
    configuration change.
 -- Fix resource allocation response message environment allocation size.
 -- Ensure that details->env_sup is NULL terminated.
 -- select/cray_aries - Correctly remove jobs/steps from blades using NPC.
 -- cons_tres - Avoid max_node_gres when entire node is allocated with
    --ntasks-per-gpu.
 -- Allow NULL arg to data_get_type().
 -- In sreport have usage for a reservation contain all jobs that ran in the
    reservation instead of just the ones that ran in the time specified. This
    matches the report for the reservation is not truncated for a time period.
 -- Fix issue with sending wrong batch step id to a < 20.11 slurmd.
 -- Add a job's alloc_node to lua for job modification and completion.
 -- Fix regression getting a slurmdbd connection through the perl API.
 -- Stop the extern step terminate monitor right after proctrack_g_wait().
 -- Fix removing the normalized priority of assocs.
 -- slurmrestd/v0.0.36 - Use correct name for partition field:
    "min nodes per job" -> "min_nodes_per_job".
 -- slurmrestd/v0.0.36 - Add node comment field.
 -- Fix regression marking cloud nodes as "unexpectedly rebooted" after
    multiple boots.
 -- Fix slurmctld segfault in _slurm_rpc_job_step_create().
 -- slurmrestd/v0.0.36 - Filter node states against NODE_STATE_BASE to avoid
    the extended states all being reported as "invalid".
 -- Fix race that can prevent the prolog for a requeued job from running.
 -- cli_filter - add "type" to readily distinguish between the CLI command in
    use.
 -- smail - reduce sleep before seff to 5 seconds.
 -- Ensure SPANK prolog and epilog run without an explicit PlugStackConfig.
 -- Disable MySQL automatic reconnection.
 -- Fix allowing "b" after memory unit suffixes.
 -- Fix slurmctld segfault with reservations without licenses.
 -- Due to internal restructuring ahead of the 20.11 release, applications
    calling libslurm MUST call slurm_init(NULL) before any API calls.
    Otherwise the API call is likely to fail due to libslurm's internal
    configuration not being available.
 -- slurm.spec - allow custom paths for PMIx and UCX install locations.
 -- Use rpath if enabled when testing for Mellanox's UCX libraries.
 -- slurmrestd/dbv0.0.36 - Change user query for associations to optional.
 -- slurmrestd/dbv0.0.36 - Change account query for associations to optional.
 -- mpi/pmix - change the error handler error message to be more useful.
 -- Add missing connection in acct_storage_p_{clear_stats, reconfig, shutdown}.
 -- Perl API - fix issue when running in configless mode.
 -- nss_slurm - avoid deadlock when stray sockets are found.
 -- Display correct value for ScronParameters in 'scontrol show config'.

* Changes in Slurm 20.11.0
==========================
 -- x11 forwarding: fix race on setup that prevented X11 forwarding from
    working within the new Interactive Step.
 -- Fix various Coverity issues.
 -- cons_tres - Fix DefCpuPerGPU
 -- Make it so you can have a job with multiple partitions and multiple
    reservations.
 -- Fix primary controller assert when shutting down backup controllers.
 -- Enforce invalid argument combinations with --ntasks-per-gpu
 -- slurmrestd/auth_local - Verify username on slurm_rest_auth_p_apply()
 -- Fix requeue of job on node failure.
 -- Prevent a job from requesting too much memory if it
    requests MEM_PER_CPUS and --threads-per-core < the number of threads
    on a core.
 -- slurmrestd - Avoid sending close header after body in
    _operations_router_reject().
 -- slurmrestd - Set new job environment for SLURM_JOB_NAME, SLURM_OPEN_MODE,
    SLURM_JOB_DEPENDENCY, SLURM_PROFILE, SLURM_ACCTG_FREQ, SLURM_NETWORK and
    SLURM_CPU_FREQ_REQ to match sbatch.
 -- slurmrestd - Avoid defaulting open_mode to append for job submission.
 -- Fix "scontrol takeover [backup]" hangs when specifying a backup > 1.
 -- salloc now waits for PrologSlurmctld to finish before entering the shell.

* Changes in Slurm 20.11.0rc2
==============================
 -- MySQL - Remove potential race condition when sending updates to a cluster
    and commit_delay used.
 -- Fixed regression in rc1 where sinfo et al would not show a node in a resv
    state.
 -- select/linear will now allocate up to nodes RealMemory when configured with
    SelectTypeParameters=CR_Memory and --mem=0 specified. Previous behavior was
    no memory accouted and no memory limits implied to job.
 -- Remove unneeded lock check from running the slurmctld prolog for a job.
 -- Fix duplicate key error on clean starts after slurmctld is killed.
 -- Avoid double free of step_record_t in the slurmctld when node is removed
    from config.
 -- Zero out step_record_t's magic when freed.
 -- Fix sacctmgr clearing QosLevel when trailing comma is used.
 -- slurmrestd - fix a fatal() error when connecting over IPv6.
 -- slurmrestd - add API to interface with slurmdbd.
 -- mpi/cray_shasta - fix PMI port parsing for non-contiguous port ranges.
 -- squeue and sinfo -O no longer repeat the last suffix specified.
 -- cons_tres - fix regression regarding gpus with --cpus-per-task.
 -- Avoid non-async-signal-safe functions calls in X11 fowarding which can
    lead to the extern step terminating unexpectedly.
 -- Don't send job completion email for revoked federation jobs.
 -- Fix device or resource busy errors on cgroup cleanup on older kernels.
 -- Avoid binding to IPv6 wildcard address in slurmd if IPv6 is not explicitly
    enabled.
 -- Make ntasks_per_gres work with cpus_per_task.
 -- Various alterations in reference to ntasks_per_tres.
 -- slurmrestd - multiple changes to make Slurm's OpenAPI spec compatible with
    https://openapi-generator.tech/.
 -- nss_slurm - avoid loading slurm.conf to avoid issues on configless systems,
    or systems with config files loaded on shared storage.
 -- scrontab - add cli_filter hooks.
 -- job_submit/lua - expose a "cron_job" flag to identify jobs submitted
    through scrontab.
 -- PMIx - fix potential buffer overflows from use of unpackmem().
    CVE-2020-27745.
 -- X11 forwarding - fix potential leak of the magic cookie when sent as an
    argument to the xauth command. CVE-2020-27746.

* Changes in Slurm 20.11.0rc1
==============================
 -- Fix corner case issue with interrupted resource allocation requests.
 -- Pack all gres information in the slurmd to send to the stepd to help
    reduce calls in the stepd to read gres.conf.
 -- The example systemd unit files have been changed to the "simple" type of
    operation, and the daemon will now run in the foreground within systemd
    instead of daemonizing itself.
 -- Add --gpu-bind=mask_gpu reusability functionality if tasks > elements.
 -- Add separate unversion libslurm_pmi.so library to use with libpmi.so.
 -- Configurations including CR_Socket and AllowSpecResourcesUsage=NO will now
    fatal if there are no allocatable sockets due to core specialization.
 -- Make sacct get UID from database instead of from the username and a
    system call. Add --use-local-uid option to sacct to use old behavior.
 -- Limit number of jobs updated by as_mysql_flush_jobs_on_cluster() to avoid
    boot loop failures in slurmdbd.
 -- Add Autodetect option to NodeName line in gres.conf to override the global
    Autodetect option.
 -- Add NetworkRaw debugflag.
 -- Make REQUEST_LAUNCH_PROLOG handler fail if PrologFlags includes Contain and
    the credential has already expired when setting the memory limits.
 -- Reject jobs that request more nodes than provided in job credential if
    PrologFlags includes Contain.
 -- Slurmdbd is now set to fatal if slurmdbd.conf file isn't owned by SlurmUser
    or it's mode is not set to 0600.
 -- libsrun/opt - use slurm_option_reset() when ignoring ntasks_per_node.
 -- Removed "regression" script from testsuite. Please use regression.py.
 -- Avoid communication issues if TreeWidth greatly exceeds the node count
    for a job.
 -- accounting_storage/filetxt has been removed as an option.
 -- Update and validate reservations after loading from state save.
 -- Update and validate reservations after setting node to down, drain or
    updating node state.
 -- Change reservation selection order to attempt to reserve unreserved nodes
    first, followed by reserved nodes under OVERLAP|MAINT reservations, and
    finally all nodes in the partition for MAINT reservations.
 -- Add [Accounting]StorageParameters slurm[dbd].conf parameter.
 -- Improve detection and logging of incompatible options involving the
    REPLACE[_DOWN] flags when creating/updating reservations.
 -- Export SLURMD_NODENAME envvar to HealthCheckProgram.
 -- SlurmctldParameters=user_resv_delete which allows any user able to run
    in a reservation to delete it.
 -- Set default unit when parsing #BSUB -M to KB to match LSF documentation.
 -- slurmrestd - fatal() when accept() returns an unexpected result.
 -- slurmrestd - Parse multiple OpenAPI specifications for path resolution.
 -- slurmrestd - Add v0.0.36 OpenAPI plugin.
 -- slurmrestd/v0.0.36 - Add error schema.
 -- slurmrestd/v0.0.36 - return array of nodes instead of dictionary.
 -- slurmrestd/v0.0.36 - return array of partitions instead of dictionary
 -- slurmrestd/v0.0.36 - return -1 (integer) instead of INFINITE (as a string)
 -- slurmrestd/v0.0.36 - return array of pings instead of dictionary
 -- slurmrestd/v0.0.36 - Simplify possible signals for canceling jobs.
 -- slurmrestd/v0.0.36 - Simplify exclusive for jobs submissions.
 -- slurmrestd/v0.0.36 - Simplify nodes for jobs submissions.
 -- slurmrestd/v0.0.36 - Use "/slurm/v0.0.36/" as server instead of "/" to
    simply naming for clients.
 -- Add 'scontrol update res=name skip' to skip the current/next reoccurring
    reservation.
 -- Add ability for reservations to be accessed by Linux Groups.
 -- Let users submit to multiple reservations as they can partitions.
 -- Report a wider range of error codes for sbcast when opening a file.
 -- Rename acct_gather_energy/cray_aries to acct_gather_energy/pm_counters.
 -- Removed gres_alloc and gres_req from job_record_t.Tres should be used
    instead.
 -- The JobCompLoc URL endpoint when the JobCompType=jobcomp/elasticsearch
    plugin is enabled is now fully configurable and the plugin no longer appends
    a hardcoded "/slurm/jobcomp" index and type suffix to it.
 -- Add check to the reservation create/update logic to prevent an inconsistent
    state without nodes and with no ANY_NODES flag with either Licenses,
    BurstBuffer and/or Watts.
 -- slurmrestd - allow the host to be optional when specifying the address to
    listen on.
 -- slurmrestd - Log numerical service name when referencing host port pairs.
 -- slurmrestd - Log host port information in RFC3986 format.
 -- sview - Remove (long-broken) batch job submission option.
 -- Dynamic Future Nodes - slurmds started with -F[<feature>] will be
    associated with a nodename in Slurm that matches the same hardware
    configuration.
 -- SlurmctldParameters=cloud_reg_addrsa - Cloud nodes automatically get
    NodeAddr and NodeHostname set from slurmd registration.
 -- SlurmctldParameters=power_save[_min]_interval - Configure how often the
    power save module looks to do work.
 -- Add CLOUD state to sinfo --state filter list.
 -- Add ability for sinfo state filtering to require all listed states.
 -- Add the "Reserved" license count to 'scontrol show licenses'.
 -- Don't display MailUser/MailType in scontrol show jobs if mail won't be sent.
 -- Throw an error and ignore CpuSpecList if it cannot be translated to bitmap
    of number of CPUs size.
 -- Validate at submission that --hint is mutully exclusive with --cpu-bind,
    --ntasks-per-core, --threads-per-core or -B.
 -- Make --exclusive the default with srun as a step adding --overlap to
    reverse behavior.
 -- Add --whole option to srun to allocate all resources on a node
    in an allocation.
 -- Allow --threads-per-core to influence task layout/binding.
 -- Remove support for "default_gbytes" option from SchedulerParameters.
 -- gres.conf - Add new MultipleFiles configuration entry to allow a single
    GRES to manage multiple device files simultaneously.
 -- Fix scontrol write config to output OverSubscribe intead of Shared.
 -- job_submit/lua - print/access oversubscribe variable with "oversubscribe".
 -- Remove SallocDefaultCommand option.
 -- Add support for an "Interactive Step", designed to be used with salloc to
    launch a terminal on an allocated compute node automatically.
 -- Add time specification: "now-<x>" (i.e. subtract from the present)
 -- Add IPv6 support. Must be explicitly enabled with EnableIPv6 in
    CommunicationParameters.
 -- Add LaunchParameters=mpir_use_nodeaddr configuration option.
 -- Allow use of a target directory with "srun --bcast", and change the default
    filename to include the node name as well.
 -- Set -fno-omit-frame-pointer compiler flag.
 -- Add --mail-type=INVALID_DEPEND option to salloc, sbatch, and srun.
 -- Fix passing names with commas to the slurmdbd.
 -- squeue - put sorted start times of "N/A" or 0 at the end of the list.
 -- Add correspond_after_task_cnt to SchedulerParameters
 -- Fix node's not being considered unresponsive/down for ResumeTimeout after
    reboot or power_up.
 -- Change "scontrol reboot ASAP" to use next_state=resume logic.
 -- Exclude HetJobs from GANG scheduling operations.
 -- Add scrontab as a new command.
 -- Enable -lnodes=#:gpus=# in #PBS/qsub -l nodes syntax.
 -- Add admin-settable "Comment" field to each Node.
 -- Fix show runaway and/on hidden partitions for >= Operator.
 -- Add --ntasks-per-gpu option.
 -- Add --gpu-bind=single option.

* Changes in Slurm 20.02.8
==========================

* Changes in Slurm 20.02.7
==========================
 -- cons_tres - Fix DefCpuPerGPU
 -- select/cray_aries - Correctly remove jobs/steps from blades using NPC.
 -- Fix false positive oom-kill events on extern step termination when
    jobacct_gather/cgroup configured.
 -- Ensure SPANK prolog and epilog run without an explicit PlugStackConfig.
 -- Fix missing xstrdup which could result in slurmctld segfault on array jobs.
 -- Fix security issue in PrologSlurmctld and EpilogSlurmctld by always
    prepending SPANK_ to all user-set environment variables. CVE-2021-31215.

* Changes in Slurm 20.02.6
==========================
 -- Fix sbcast --fanout option.
 -- Tighten up keyword matching for --dependency.
 -- Fix "squeue -S P" not sorting by partition name.
 -- Fix segfault in slurmctld if group resolution fails during job credential
    creation.
 -- sacctmgr - Honor PreserveCaseUser when creating users with load command.
 -- Avoid attempting to schedule jobs on magnetic reservations when they aren't
    allowed.
 -- Always make sure we clear the magnetic flag from a job.
 -- In backfill avoid NULL pointer dereference.
 -- Fix Segfault at end of slurmctld if you have a magnetic reservation and
    you shutdown the slurmctld.
 -- Silence security warning when a Slurm is trying a job for a
    magnetic reservation.
 -- Have sacct exit correctly when a user/group id isn't valid.
 -- Remove extra \n from invalid user/group id error message.
 -- Detect when extern steps trigger OOM events and mark extern step correctly.
 -- pam_slurm_adopt - permit root access to the node before reading the config
    file, which will give root a chance to fix the config if missing or broken.
 -- Reset DefMemPerCPU, MaxMemPerCPU, and TaskPluginParam (among other minor
    flags) on reconfigure.
 -- Fix incorrect memory handling of mail_user when updating mail_type=none.
 -- Handle mail_user and mail_type independently.
 -- Fix thread-safety issue with assoc_mgr_get_admin_level().
 -- Ignore step features if equal to job features
 -- Fix slurmstepd segfault caused by incorrect strtok() usage.
 -- CRAY - Remove unneeded ATP spank plugin from ansible playbook.
 -- Fix core selection for exclusive step on nodes where CPUs == Cores.
 -- Fix topology aware scheduling reservations.
 -- Fix loading cpus_per_task on a job from state file.
 -- When a partition has no nodes fix estimate of max cpus possible on a job
    trying to run there.
 -- In cons_tres fix sorting functions to handle node/topo weight
    correctly.
 -- Fix regression in 20.02.5 where you couldn't request contraints with a
    simple & and a count.
 -- Limit the number of threads for servicing emails.
 -- Avoid possible double init race condition in assoc_mgr_lock().
 -- Add missing locks in slurm_cred_handle_reissue().
 -- Add missing locks in slurm_cred_revoked().
 -- Fix slurmctld segfault due to tight reconfigure RPC requests by serializing
    the RPC handler processing logic.
 -- Use _exit() instead exit() after fork().
 -- Perl API - fix hang reading config in configless environments.
 -- slurmrestd - request detailed node information to populate GRES fields.
 -- slurmrestd - request detailed job information to populate GRES fields.
 -- Fix job license update bug on array tasks or hetjob components.
 -- Fix job partition update bug on array tasks or hetjob components.
 -- Fix slurmctld segfault on _pick_best_nodes() when processing a job request
    with XOR'd constraints and no nodeset has the feature.
 -- Fix job requests rejected with incorrect NODE_CONFIG_UNAVAIL when nodes are
    actually only busy due to an overlapping MAINT reservation.
 -- Fix sacctmgr allowing the deletion of a user's default account.
 -- Fix srun and other Slurm commands running within a "configless" salloc
    terminal.
 -- MySQL - Correctly handle QOS deletion from assocation tables.
 -- Fix update of First_Cores flag in a reservation.
 -- Fix parsing of update reservation flags.
 -- Fix --switches for cons_tres.
 -- Retry connection on ETIMEDOUT in slurm_send_addr_recv_msgs.
 -- Fix wait for RPC_PROLOG_LAUNCH notification 2*MessageTimeout.
 -- Have slurm_send_addr_recv_msgs conn_timeout to match rpc_wait in slurmd.
 -- pam_slurm_adopt - operate correctly even if ConstrainRAMSpace is not
    enabled on the node by falling back to the cpuset, devices, or freezer
    subsystem instead.
 -- slurmrestd - use memmove() instead of memcpy() in string manipulation
    to avoid bugs related to overlapping memory regions.
 -- slurmrestd - avoid xassert() failure on duplicated headers in request.
 -- Remove stale 'ReqNodeNotAvail, Reserved for maintenance' message from
    pending jobs after a maintenance reservation ended.
 -- MySQL - Stop steps from printing when outside time range.
 -- Fixed kmem limit calculation to use MaxKmemPercent correctly.
 -- Fix initialization of cpuset.mems/cpus on uid cgroup subdir.
 -- MySQL - Remove potential race condition when sending updates to a cluster
    and commit_delay used.
 -- Avoid double free of step_record_t in the slurmctld when node is removed
    from config.
 -- cons_tres - fix regression regarding gpus with --cpus-per-task.
 -- Don't send job completion email for revoked federation jobs.
 -- PMIx - fix potential buffer overflows from use of unpackmem().
    CVE-2020-27745.
 -- X11 forwarding - fix potential leak of the magic cookie when sent as an
    argument to the xauth command. CVE-2020-27746.

* Changes in Slurm 20.02.5
==========================
 -- Fix leak of TRESRunMins when job time is changed with --time-min
 -- pam_slurm - explicitly initialize slurm config to support configless mode.
 -- scontrol - Fix exit code when creating/updating reservations with wrong
    Flags.
 -- When a GRES has a no_consume flag, report 0 for allocated.
 -- Fix cgroup cleanup by jobacct_gather/cgroup.
 -- When creating reservations/jobs don't allow counts on a feature unless
    using an XOR.
 -- Improve number of boards discovery
 -- Fix updating a reservation NodeCnt on a zero-count reservation.
 -- slurmrestd - provide an explicit error messages when PSK auth fails.
 -- cons_tres - fix job requesting single gres per-node getting two or more
    nodes with less CPUs than requested per-task.
 -- cons_tres - fix calculation of cores when using gres and cpus-per-task.
 -- cons_tres - fix job not getting access to socket without GPU or with less
    than --gpus-per-socket when not enough cpus available on required socket
    and not using --gres-flags=enforce binding.
 -- Fix HDF5 type version build error.
 -- Fix creation of CoreCnt only reservations when the first node isn't
    available.
 -- Fix wrong DBD Agent queue size in sdiag when using accounting_storage/none.
 -- Improve job constraints XOR option logic.
 -- Fix preemption of hetjobs when needed nodes not in leader component.
 -- Fix wrong bit_or() messing potential preemptor jobs node bitmap, causing
    bad node deallocations and even allocation of nodes from other partitions.
 -- Fix double-deallocation of preempted non-leader hetjob components.
 -- slurmdbd - prevent truncation of the step nodelists over 4095.
 -- Fix nodes remaining in drain state state after rebooting with ASAP option.

* Changes in Slurm 20.02.4
==========================
 -- srun - suppress job step creation warning message when waiting on
    PrologSlurmctld.
 -- slurmrestd - fix incorrect return values in data_list_for_each() functions.
 -- mpi/pmix - fix issue where HetJobs could fail to launch.
 -- slurmrestd - set content-type header in responses.
 -- Fix cons_res GRES overallocation for --gres-flags=disable-binding.
 -- Fix cons_res incorrectly filtering cores with respect to GRES locality for
    --gres-flags=disable-binding requests.
 -- Fix regression where a dependency on multiple jobs in a single array using
    underscores would only add the first job.
 -- slurmrestd - fix corrupted output due to incorrect use of memcpy().
 -- slurmrestd - address a number of minor Coverity warnings.
 -- Handle retry failure when slurmstepd is communicating with srun correctly.
 -- Fix jobacct_gather possibly duplicate stats when _is_a_lwp error shows up.
 -- Fix tasks binding to GRES which are closest to the allocated CPUs.
 -- Fix AMD GPU ROCM 3.5 support.
 -- Fix handling of job arrays in sacct when querying specific steps.
 -- slurmrestd - avoid fallback to local socket authentication if JWT
    authentication is ill-formed.
 -- slurmrestd - restrict ability of requests to use different authentication
    plugins.
 -- slurmrestd - unlink named unix sockets before closing.
 -- slurmrestd - fix invalid formatting in openapi.json.
 -- Fix batch jobs stuck in CF state on FrontEnd mode.
 -- Add a separate explicit error message when rejecting changes to active node
    features.
 -- cons_common/job_test - fix slurmctld SIGABRT due to double-free.
 -- Fix updating reservations to set the duration correctly if updating the
    start time.
 -- Fix update reservation to promiscuous mode.
 -- Fix override of job tasks count to max when ntasks-per-node present.
 -- Fix min CPUs per node not being at least CPUs per task requested.
 -- Fix CPUs allocated to match CPUs requested when requesting GRES and
    threads per core equal to one.
 -- Fix NodeName config parsing with Boards and without CPUs.
 -- Ensure SLURM_JOB_USER and SLURM_JOB_UID are set in SrunProlog/Epilog.
 -- Fix error messages for certain invalid salloc/sbatch/srun options.
 -- pmi2 - clean up sockets at step termination.
 -- Fix 'scontrol hold' to work with 'JobName'.
 -- sbatch - handle --uid/--gid in #SBATCH directives properly.
 -- Fix race condition in job termination on slurmd.
 -- Print specific error messages if trying to run use certain
    priority/multifactor factors that cannot work without SlurmDBD.
 -- Avoid partial GRES allocation when --gpus-per-job is not satisfied.
 -- Cray - Avoid referencing a variable outside of it's correct scope when
    dealing with creating steps within a het job.
 -- slurmrestd - correctly handle larger addresses from accept().
 -- Avoid freeing wrong pointer with SlurmctldParameters=max_dbd_msg_action
    with another option after that.
 -- Restore MCS label when suspended job is resumed.
 -- Fix insufficient lock levels.
 -- slurmrestd - use errno from job submission.
 -- Fix "user" filter for sacctmgr show transactions.
 -- Fix preemption logic.
 -- Fix no_consume GRES for exclusive (whole node) requests.
 -- Fix regression in 20.02 that caused an infinite loop in slurmctld when
    requesting --distribution=plane for the job.
 -- Fix parsing of the --distribution option.
 -- Add CONF READ_LOCK to _handle_fed_send_job_sync.
 -- prep/script - always call slurmctld PrEp callback in _run_script().
 -- Fix node estimation for jobs that use GPUs or --cpus-per-task.
 -- Fix jobcomp, job_submit and cli_filter Lua implementation plugins causing
    slurmctld and/or job submission CLI tools segfaults due to bad return
    handling when the respective Lua script failed to load.
 -- Fix propagation of gpu options through hetjob components.
 -- Add SLURM_CLUSTERS environment variable to scancel.
 -- Fix packing/unpacking of "unlinked" jobs.
 -- Connect slurmstepd's stderr to srun for steps launched with --pty.
 -- Handle MPS correctly when doing exclusive allocations.
 -- slurmrestd - fix compiling against libhttpparser in a non-default path.
 -- slurmrestd - avoid compilation issues with libhttpparser < 2.6.
 -- Fix compile issues when compiling slurmrestd without --enable-debug.
 -- Reset idle time on a reservation that is getting purged.
 -- Fix reoccurring reservations that have Purge_comp= to keep correct
    duration if they are purged.
 -- scontrol - changed the "PROMISCUOUS" flag to "MAGNETIC"
 -- Early return from epilog_set_env in case of no_consume.
 -- Fix cons_common/job_test start time discovery logic to prevent skewed
    results between "will run test" executions.
 -- Ensure TRESRunMins limits are maintained during "scontrol reconfigure".
 -- Improve error message when host lookup fails.

* Changes in Slurm 20.02.3
==========================
 -- Factor in ntasks-per-core=1 with cons_tres.
 -- Fix formatting in error message in cons_tres.
 -- Fix calling stat on a NULL variable.
 -- Fix minor memory leak when using reservations with flags=first_cores.
 -- Fix gpu bind issue when CPUs=Cores and ThreadsPerCore > 1 on a node.
 -- Fix --mem-per-gpu for heterogenous --gres requests.
 -- Fix slurmctld load order in load_all_part_state().
 -- Fix race condition not finding jobacct gather task cgroup entry.
 -- Suppress error message when selecting nodes on disjoint topologies.
 -- Improve performance of _pack_default_job_details() with large number of job
    arguments.
 -- Fix archive loading previous to 17.11 jobs per-node req_mem.
 -- Fix regresion validating that --gpus-per-socket requires --sockets-per-node
    for steps. Should only validate allocation requests.
 -- error() instead of fatal() when parsing an invalid hostlist.
 -- nss_slurm - fix potential deadlock in slurmstepd on overloaded systems.
 -- cons_tres - fix --gres-flags=enforce-binding and related --cpus-per-gres.
 -- cons_tres - Allocate lowest numbered cores when filtering cores with gres.
 -- Fix getting system counts for named GRES/TRES.
 -- MySQL - Fix for handing typed GRES for association rollups.
 -- Fix step allocations when tasks_per_core > 1.
 -- Fix allocating more GRES than requested when asking for multiple GRES types.

* Changes in Slurm 20.02.2
==========================
 -- Fix slurmctld segfault when checking no_consume GRES node allocation counts.
 -- Fix resetting of cloud_dns on a reconfigure.
 -- squeue - change output for dependency column to use "(null)" instead of ""
    for no dependncies as documented in the man page, and used by other columns.
 -- Clear node_cnt_wag after job update.
 -- Fix regression where AccountingStoreJobComment was not defaulting to 'yes'.
 -- Send registration message immediately after a node is resumed.
 -- Cray - Fix hetjobs when using only a single component in the step launch.
 -- Cray - Fix hetjobs launched without component 0.
 -- Cray - Quiet cookies missing message which is expected on for hetjobs.
 -- Fix handling of -m/--distribution options for across socket/2nd level by
    task/affinity plugin.
 -- Fix grp_node_bitmap error when slurmctld started before slurmdbd.
 -- Fix scheduling issue when there are not enough nodes available to run a job
    resulting in possible job starvation.
 -- Make it so mpi/cray_shasta appears in srun --mpi=list
 -- Don't requeue jobs that have been explicitly canceled.
 -- Fix error message for a regular user trying to update licenses on a running
    job.
 -- Fix backup slurmctld handling for logrotation via SIGUSR2.
 -- Fix reservation feature specification when looking for inactive features
    after active features fails.
 -- Prevent misleading error messages for reservation creation.
 -- Print message in scontrol when a request fails for not having enough nodes.
 -- Fix duplicate output in sacct with multiple resv events.
 -- auth/jwt - return correct gid for a given user. This was incorrectly
    assuming the users's primary group name matched their username.
 -- slurmrestd - permit non-SlurmUser/root job submission.
 -- Use host IP if hostname unknown for job submission for allocating node.
 -- Fix issue with primary_slurmdbd_resumed_operation trigger not happening
    on slurmctld restart.
 -- Fix race in acct_gather_interconnect/ofed on step termination.
 -- Fix typo of SlurmctldProlog -> PrologSlurmctld in error message.
 -- slurm.spec - add SuSE-specific dependencies for optional slurmrestd package.
 -- Fix FreeBSD build issues.
 -- Fixed sbatch not processing --ignore-pbs in batch script.
 -- Don't clear the qos_id of an invalid QOS.
 -- Allow a job that was once FAIL_[QOS|ACCOUNT] to be eligible again if
    the qos|account limitation is remedied.
 -- Fix core reservations using the FLEX flag to allow use of resources
    outside of the reservation allocation.
 -- Fix MPS without File with 1 GPU, and without GPUs.
 -- Add FreeBSD support to proctrack/pgid plugin.
 -- Fix remote dependency testing for meta job in job array.
 -- Fix preemption when dealing with a job array.
 -- Don't send remote non-pending singleton dependencies on federation update.
 -- slurmrestd - fix crash on empty query.
 -- Fix race condition which could lead to invalid references in backfill.
 -- Fix edge case in _remove_job_hash().
 -- Fix exit code when using --cluster/-M client options.
 -- Fix compilation issues in GCC10.
 -- Fix invalid references when federated job is revoked while in backfill loop.
 -- Fix distributing job steps across idle nodes within a job.
 -- Fix detected floating reservation overlapping.
 -- Break infinite loop in cons_tres dealing with incorrect tasks per tres
    request resulting in slurmctld hang.
 -- Send the current (not the previous) reason for a pending job to client
    commands like squeue/scontrol.
 -- Fix incorrect lock levels for select_g_reconfigure().
 -- Handle hidden nodes correctly in slurmrestd.
 -- Allow sacctmgr to use MaxSubmitP[U|A] as format options.
 -- Fix segfault when trying to delete a corrupted association.
 -- Fix setting ntasks-per-core when using --multithread.
 -- Only override job wait reason to priority if Reason=None or