NEWS

This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.

* Changes in Slurm 21.08.9
==========================
 -- Fix cross-endian communication.
 -- Fix slurmd segfault when seeing authentication issues alongside a failed
    message forward.
 -- Fix resuming nodes not part of an allocation.
 -- Fix "--gpu-bind=single:" having wrong env variables.
 -- Fix logic estimating when a job can start based on ANY_NODES / LICENSE_ONLY
    reservations that could lead to an infinite loop in backfill.
 -- Fix a segfault that may happen on gpu configured as no_consume.
 -- Fix regression which prevented a cons_tres gpu job to be submitted to a
    cons_tres cluster from a non-con_tres cluster.
 -- select/linear - fix regression introduced in 21.08.6 which prevented step
    requests with --threads-per-core or --hint=nomultithread from being created.

* Changes in Slurm 21.08.8-2
============================
 -- Fix communication forwarding when running with a mix of patch and unpatched
    slurmd processes.

* Changes in Slurm 21.08.8
==========================
 -- openapi/dbv0.0.37 - fix slurmrestd fatal() when deleting an association.
 -- Allow scontrol update <job> Gres=... to not require "gres:".
 -- Fix inconsistent reboot message appending behavior.
 -- Fix incorrect reason_time and reason_uid on reboot message.
 -- Fix "scontrol reboot" clearing node reason on ResumeTimeout.
 -- Fix ResumeTimeout error message missing when node already has reason set.
 -- Avoid "running with local config" error when conf server is provided by DNS.
 -- openapi/v0.0.37 - resolve job user name when not sent by slurmctld.
 -- openapi/dbv0.0.37 - Correct OpenAPI specification for diag request.
 -- Ignore power_down request when node is already powering down.
 -- CVE-2022-29500 - Prevent credential abuse.
 -- CVE-2022-29501 - Prevent abuse of REQUEST_FORWARD_DATA.
 -- CVE-2022-29502 - Correctly validate io keys.

* Changes in Slurm 21.08.7
==========================
 -- openapi/v0.0.37 - correct calculation for bf_queue_len_mean in /diag.
 -- Optimize sending down nodes in maintenance mode to the database when
    removing reservations.
 -- Avoid shrinking a reservation when overlapping with downed nodes.
 -- Fix 'planned time' in rollups for jobs that were still pending when the
    rollup happened.
 -- Prevent new elements from a job array from causing rerollups.
 -- Only check TRES limits against current usage for TRES requested by the job.
 -- Do not allocate shared gres (MPS) in whole-node allocations
 -- Fix minor memory leak when dealing with configless setups.
 -- Constrain slurmstepd to job/step cgroup like in previous versions of Slurm.
 -- Fix warnings on 32-bit compilers related to printf() formats.
 -- Fix memory leak when freeing kill_job_msg_t.
 -- Fix memory leak when using data_t.
 -- Fix reconfigure issues after disabling/reenabling the GANG PreemptMode.
 -- Fix race condition where a cgroup was being deleted while another step
    was creating it.
 -- Set the slurmd port correctly if multi-slurmd
 -- openapi/v0.0.37 - Fix misspelling of account_gather_frequency in spec.
 -- openapi/v0.0.37 - Fix misspelling of cluster_constraint in spec.
 -- Fix FAIL mail not being sent if a job was cancelled due to preemption.
 -- slurmrestd - move debug logs for HTTP handling to be gated by debugflag
    NETWORK to avoid unnecessary logging of communication contents.
 -- Fix issue with bad memory access when shrinking running steps.
 -- Fix various issues with internal job accounting with GRES when jobs are
    shrunk.
 -- Fix ipmi polling on slurmd reconfig or restart.
 -- Fix srun crash when reserved ports are being used and het step fails
    to launch.
 -- openapi/dbv0.0.37 - fix DELETE execution path on /user/{user_name}.
 -- slurmctld - Properly requeue all components of a het job if PrologSlurmctld
    fails.
 -- rlimits - remove final calls to limit nofiles to 4096 but to instead use
    the max possible nofiles in slurmd and slurmdbd.
 -- Fix slurmctld memory leak after a reconfigure with configless.
 -- Fix slurmd memory leak when fetching configless files.
 -- Allow the DBD agent to load large messages (up to MAX_BUF_SIZE) from state.
 -- Fix minor memory leak with cleaning up the extern step.
 -- Fix potential deadlock during slurmctld restart when there is a completing
    job.
 -- slurmstepd - reduce user requested soft rlimits when they are above max
    hard rlimits to avoid rlimit request being completely ignored and
    processes using default limits.
 -- Fix memory leaks when job/step specifies a container.
 -- Fix Slurm user commands displaying available features as active features
    when no features were active.
 -- Don't power down nodes that are rebooting.
 -- Clear pending node reboot on power down request.
 -- Ignore node registrations while node is powering down.
 -- Don't reboot any node that is power<ing|ed> down.
 -- Don't allow a node to reboot if it's marked for power down.
 -- Fix issuing reboot and downing when rebooting a powering up node.
 -- Clear DRAIN on node after failing to resume before ResumeTimeout.
 -- Prevent repeating power down if node fails to resume before ResumeTimeout.
 -- Fix federated cloud node communication with srun and cloud_dns.
 -- Fix jobs being scheduled on nodes marked to be powered_down when idle.
 -- Fix problem where a privileged user could not view array tasks specified by
    <array_job_id>_<task_id> when PrivateData had the jobs value set.

* Changes in Slurm 21.08.6
==========================
 -- Handle typed shared GRES better in accounting.
 -- Fix plugin_name definitions in a number of plugins to improve logging.
 -- Close sbcast file transfers when job is cancelled.
 -- job_submit/lua - allow mail_type and mail_user fields to be modified.
 -- scrontab - fix handling of --gpus and --ntasks-per-gpu options.
 -- sched/backfill - fix job_queue_rec_t memory leak.
 -- Fix magnetic reservation logic in both main and backfill schedulers.
 -- job_container/tmpfs - fix memory leak when using InitScript.
 -- slurmrestd / openapi - fix memory leaks.
 -- Fix slurmctld segfault due to job array resv_list double free.
 -- Fix multi-reservation job testing logic.
 -- Fix slurmctld segfault due to insufficient job reservation parse validation.
 -- Fix main and backfill schedulers handling for already rejected job array.
 -- sched/backfill - restore resv_ptr after yielding locks.
 -- acct_gather_energy/xcc - appropriately close and destroy the IPMI context.
 -- Protect slurmstepd from making multiple calls to the cleanup logic.
 -- Prevent slurmstepd segfault at cleanup time in mpi_fini().
 -- Fix slurmctld sometimes hanging if shutdown while PrologSlurmctld or
    EpilogSlurmctld were running and PrologEpilogTimeout is set in slurm.conf.
 -- Fix affinity of the batch step if batch host is different than the first
    node in the allocation.
 -- slurmdbd - fix segfault after multiple failover/failback operations.
 -- Fix jobcomp filetxt job selection condition.
 -- Fix -f flag of sacct not being used.
 -- Select cores for job steps according to the socket distribution. Previously,
    sockets were always filled before selecting cores from the next socket.
 -- Keep node in Future state if epilog completes while in Future state.
 -- Fix erroneous --constraint behavior by preventing multiple sets of brackets.
 -- Make ResetAccrueTime update the job's accrue_time to now.
 -- Fix sattach initialization with configless mode.
 -- Revert packing limit checks affecting pmi2.
 -- sacct - fixed assertion failure when using -c option and a federation
    display
 -- Fix issue that allowed steps to overallocate the job's memory.
 -- Fix the sanity check mode of AutoDetect so that it actually works.
 -- Fix deallocated nodes that didn't actually launch a job from waiting for
    Epilogslurmctld to complete before clearing completing node's state.
 -- Job should be in a completing state if EpilogSlurmctld when being requeued.
 -- Fix job not being requeued properly if all node epilog's completed before
    EpilogSlurmctld finished.
 -- Keep job completing until EpilogSlurmctld is completed even when "downing"
    a node.
 -- Fix handling reboot with multiple job features.
 -- Fix nodes getting powered down when creating new partitions.
 -- Fix bad bit_realloc which potentially could lead to bad memory access.
 -- slurmctld - remove limit on the number of open files.
 -- Fix bug where job_state file of size above 2GB wasn't saved without any
    error message.
 -- Fix various issues with no_consume gres.
 -- Fix regression in 21.08.0rc1 where job steps failed to launch on systems
    that reserved a CPU in a cgroup outside of Slurm (for example, on systems
    with WekaIO).
 -- Fix OverTimeLimit not being reset on scontrol reconfigure when it is
    removed from slurm.conf.
 -- serializer/yaml - use dynamic buffer to allow creation of YAML outputs
    larger than 1MiB.
 -- Fix minor memory leak affecting openapi users at process termination.
 -- Fix batch jobs not resolving the username when nss_slurm is enabled.
 -- slurmrestd - Avoid slurmrestd ignoring invalid HTTP method if the response
    serialized without error.
 -- openapi/dbv0.0.37 - Correct conditional that caused the diag output to
    give an internal server error status on success.
 -- Make --mem-bind=sort work with task_affinity
 -- Fix sacctmgr to set MaxJobsAccruePer{User|Account} and MinPrioThres in
    sacctmgr add qos, modify already worked correctly.
 -- job_container/tmpfs - avoid printing extraneous error messages in Prolog
    and Epilog, and when the job completes.
 -- Fix step CPU memory allocation with --threads-per-core without --exact.
 -- Remove implicit --exact when --threads-per-core or --hint=nomultithread
    is used.
 -- Do not allow a step to request more threads per core than the
    allocation did.
 -- Remove implicit --exact when --cpus-per-task is used.

* Changes in Slurm 21.08.5
==========================
 -- Fix issue where typeless GRES node updates were not immediately reflected.
 -- Fix setting the default scrontab job working directory so that it's the home
    of the different user (-u <user>) and not that of root or SlurmUser editor.
 -- Fix stepd not respecting SlurmdSyslogDebug.
 -- Fix concurrency issue with squeue.
 -- Fix job start time not being reset after launch when job is packed onto
    already booting node.
 -- Fix updating SLURM_NODE_ALIASES for jobs packed onto powering up nodes.
 -- Cray - Fix issues with starting hetjobs.
 -- auth/jwks - Print fatal() message when jwks is configured but file could
    not be opened.
 -- If sacctmgr has an association with an unknown qos as the default qos
    print 'UNKN-###' instead of leaving a blank name.
 -- Correctly determine task count when giving --cpus-per-gpu, --gpus and
    --ntasks-per-node without task count.
 -- slurmctld - Fix places where the global last_job_update was not being set
    to the time of update when a job's reason and description were updated.
 -- slurmctld - Fix case where a job submitted with more than one partition
    would not have its reason updated while waiting to start.
 -- Fix memory leak in node feature rebooting.
 -- Fix time limit permanetly set to 1 minute by backfill for job array tasks
    higher than the first with QOS NoReserve flag and PreemptMode configured.
 -- Fix sacct -N to show jobs that started in the current second
 -- Fix issue on running steps where both SLURM_NTASKS_PER_TRES and
    SLURM_NTASKS_PER_GPU are set.
 -- Handle oversubscription request correctly when also requesting
    --ntasks-per-tres.
 -- Correctly detect when a step requests bad gres inside an allocation.
 -- slurmstepd - Correct possible deadlock when UnkillableStepTimeout triggers.
 -- srun - use maximum number of open files while handling job I/O.
 -- Fix writing to Xauthority files on root_squash NFS exports, which was
    preventing X11 forwarding from completing setup.
 -- Fix regression in 21.08.0rc1 that broke --gres=none.
 -- Fix srun --cpus-per-task and --threads-per-core not implicitly setting
    --exact. It was meant to work this way in 21.08.
 -- Fix regression in 21.08.0 that broke dynamic future nodes.
 -- Fix dynamic future nodes remembering active state on restart.
 -- Fix powered down nodes getting stuck in COMPLETING+POWERED_DOWN when job is
    cancelled before nodes are powering up.

* Changes in Slurm 21.08.4
==========================
 -- Fix potential deadlock when using PMI v1.
 -- Fix tight loop sending DBD_SEND_MULT_JOB_START when the slurmctld has an
    issue talking correctly to the DBD.
 -- Fix memory leak in step creation.
 -- Fix potential deadlock when shutting down slurmctld.
 -- Fix regression in 21.08 where multi-node steps that requested MemPerCPU
    were not counted against the job's memory allocation on some nodes.
 -- Fix issue with select/cons_tres and the partition limit MaxCpusPerNode where
    the limit was enforced for one less CPU than the configured value.
 -- jobacct_gather/common - compare Pss to Rss after scaling Pss to Rss units.
 -- Fix SLURM_NODE_ALIASES in RPC Prolog for batch jobs.
 -- Fix regression in 21.08 where slurmd and slurmstepd were not constrained
    with CpuSpecList or CoreSpecCount.
 -- Fix cloud jobs running without powering up nodes after a reconfig/restart.
 -- CVE-2021-43337 - Fix security issue with new AccountingStoreFlags=job_script
    and job_env options where users could request scripts and environments they
    should not have been permitted to access.

* Changes in Slurm 21.08.3
==========================
 -- Return error to sacctmgr when running 'sacctmgr archive load' and the load
    fails due to an invalid or corrupted file.
 -- slurmctld/gres_ctld - fix deallocation of typed GRES without device.
 -- scrontab - fix capturing the cronspec request in the job script.
 -- openapi/dbv0.0.37 - Add missing method POST for /associations/.
 -- If ALTER TABLE was already run, continue with database upgrade.
 -- slurmstepd - Gracefully handle RunTimeQuery returning no output.
 -- srun - automatically handle issues with races to listen() on an ephemeral
    socket, and suppress otherwise needless error messages.
 -- Schedule sooner after Epilog completion with SchedulerParameters=defer.
 -- Improve performance for AccountingStoreFlags=job_env.
 -- Expose missing SLURMD_NODENAME and SLURM_NODEID to TaskEpilog environment.
 -- Bring slurm_completion.sh up to date with changes to commands.
 -- Fix issue where burst buffer stage-in could only start for one job in a job
    array per scheduling cycle instead of bb_array_stage_cnt jobs per scheduling
    cycle.
 -- Fix checking if the dependency is the same job for array jobs.
 -- Fix checking for circular dependencies with job arrays.
 -- Restore dependent job pointers on slurmctld startup to avoid race.
 -- openapi/v0.0.37 - Allow strings for JobIds instead of only numerical JobIds
    for GET, DELETE, and POST job methods.
 -- openapi/dbv0.0.36 - Gracefully handle missing associations.
 -- openapi/dbv0.0.36 - Avoid restricting job association lookups to only
    default associations.
 -- openapi/dbv0.0.37 - Gracefully handle missing associations.
 -- openapi/dbv0.0.37 - Avoid restricting job association lookups to only
    default associations.
 -- Fix error in GPU frequency validation logic.
 -- Fix regression in 21.08.1 that broke federated jobs.
 -- Correctly handle requested GRES when used in job arrays.
 -- Fix error in pmix logic dealing with the incorrect size of buffer.
 -- Fix handling of no_consume GRES, add it to allocated job allocated TRES.
 -- Fix issue with typed GRES without Files= (bitmap).
 -- Fix job_submit/lua support for 'gres' which is now stored as a 'tres'
    when requesting jobs so needs a 'gres' prefix.
 -- Fix regression where MPS would not deallocate from the node properly.
 -- Fix --gpu-bind=verbose to work correctly.
 -- Do not deny --constraint with special operators "[]()|*" when no changeable
    features are requested, but continue to deny --constraint with special
    operators when changeable features are requested.
 -- openapi/v0.0.{35,36,37} - prevent merging the slurmrestd environment
    alongside a new job submission.
 -- openapi/dbv0.0.36 - Correct tree position of dbv0.0.36_job_step.
 -- openapi/dbv0.0.37 - Correct tree position of dbv0.0.37_job_step.
 -- openapi/v0.0.37 - enable job priority field for job submissions and updates.
 -- openapi/v0.0.37 - request node states query includes MIXED state instead of
    only allocated.
 -- mpi/pmix - avoid job hanging until the time limit on PMIx agent failures.
 -- Correct inverted logic where reduced version matching applied to non-SPANK
    plugins where it should have only applied to SPANK plugins.
 -- Fix issues where prologs would run in serial without PrologFlags=serial.
 -- Make sure a job coming in is initially considered for magnetic reservations.
 -- PMIx v1.1.4 and below are no longer supported.
 -- Add comment to service files about disabling logging through journald.
 -- Add SLURM_NODE_ALIASES env to RPC Prolog (PrologFlags=alloc) environment.
 -- Limit max_script_size to 512 MB.
 -- Fix shutdown of slurmdbd plugin to correctly notice when the agent thread
    finishes.
 -- slurmdbd - fix issue with larger batch script files being sent to SlurmDBD
    with AccountingStoreFlags=job_script that can lead to accounting data loss
    as the resulting RPC generated can exceed internal limits and won't be
    sent, preventing further communication with SlurmDBD.
    This issue is indicated by "error: Invalid msg_size" in your log files.
 -- Fix compile issue with --without-shared-libslurm.

* Changes in Slurm 21.08.2
==========================
 -- slurmctld - fix how the max number of cores on a node in a partition are
    calculated when the partition contains multi-socket nodes. This in turn
    corrects certain jobs node count estimations displayed client-side.
 -- job_submit/cray_aries - fix "craynetwork" GRES specification after changes
    introduced in 21.08.0rc1 that made TRES always have a type prefix.
 -- Ignore nonsensical check in the slurmd for [Pro|Epi]logSlurmctld.
 -- Fix writing to stderr/syslog when systemd runs slurmctld in the foreground.
 -- Fix locking around log level setting routines.
 -- Fix issue with updating job started with node range.
 -- Fix issue with nodes not clearing state in the database when the slurmctld
    is started with clean-start.
 -- Fix hetjob components > 1 timing out due to InactiveLimit.
 -- Fix sprio printing -nan for normalized association priority if
    PriorityWeightAssoc was not defined.
 -- Disallow FirstJobId=0.
 -- Preserve job start info in the database for a requeued job that hadn't
    registered the first time in the database yet.
 -- Only send one message on prolog failure from the slurmd.
 -- Remove support for TaskAffinity=yes in cgroup.conf.
 -- accounting_storage/mysql - fix issue where querying jobs via sacct
    --whole-hetjob=yes or slurmrestd (which automatically includes this flag)
    could in some cases return more records than expected.
 -- Fix issue for preemption of job array task that makes afterok dependency
    fail. Additionally, send emails when requeueing happens due to preemption.
 -- Fix sending requeue mail type.
 -- Properly resize a job's GRES bitmaps and counts when resizing the job.
 -- Fix node being able to transition to CLOUD state from non-cloud state.
 -- Fix regression introduced in 21.08.0rc1 which broke a step's ability to
    inherit GRES from the job when the step didn't request GRES but the job did.
 -- Fix errors in logic when picking nodes based on bracketed anded constraints.
    This also enforces the requirement to have a count when using such
    constraints.
 -- Handle job resize better in the database.
 -- Exclude currently running, resized jobs from the runaway jobs list.
 -- Make it possible to shrink a job more than once.

* Changes in Slurm 21.08.1
==========================
 -- Fix potential memory leak if a problem happens while allocating GRES for
    a job.
 -- If an overallocation of GRES happens terminate the creation of a job.
 -- AutoDetect=nvml: Fatal if no devices found in MIG mode.
 -- slurm.spec - fix querying for PMIx and UCX version.
 -- Print federation and cluster sacctmgr error messages to stderr.
 -- Fix off by one error in --gpu-bind=mask_gpu.
 -- Fix statement condition in http_parser autoconf macro.
 -- Fix statement condition in netloc autoconf macro.
 -- Add --gpu-bind=none to disable gpu binding when using --gpus-per-task.
 -- Handle the burst buffer state "alloc-revoke" which previously would not
    display in the job correctly.
 -- Fix issue in the slurmstepd SPANK prolog/epilog handler where configuration
    values were used before being initialized.
 -- Restore a step's ability to utilize all of an allocations memory if --mem=0.
 -- Fix --cpu-bind=verbose garbage taskid.
 -- Fix cgroup task affinity issues from garbage taskid info.
 -- Make gres_job_state_validate() client logging behavior as before 44466a4641.
 -- Fix steps with --hint overriding an allocation with --threads-per-core.
 -- Require requesting a GPU if --mem-per-gpu is requested.
 -- Return error early if a job is requesting --ntasks-per-gpu and no gpus or
    task count.
 -- Properly clear out pending step if unavailable to run with available
    resources.
 -- Kill all processes spawned by burst_buffer.lua including decendents.
 -- openapi/v0.0.{35,36,37} - Avoid setting default values of min_cpus,
    job name, cwd, mail_type, and contiguous on job update.
 -- openapi/v0.0.{35,36,37} - Clear user hold on job update if hold=false.
 -- Prevent CRON_JOB flag from being cleared when loading job state.
 -- sacctmgr - Fix deleting WCKeys when not specifying a cluster.
 -- Fix getting memory for a step when the first node in the step isn't the
    first node in the allocation.
 -- Make SelectTypeParameters=CR_Core_Memory default for cons_tres and cons_res.
 -- Correctly handle mutex unlocks in the gres code if failures happen.
 -- Give better error message if -m plane is given with no size.
 -- Fix --distribution=arbitrary for salloc.
 -- Fix jobcomp/script regression introduced in 21.08.0rc1 0c75b9ac9d.
 -- Only send the batch node in the step_hostlist in the job credential.
 -- When setting affinity for the batch step don't assume the batch host is node
    0.
 -- In task/affinity better checking for node existence when laying out
    affinity.
 -- slurmrestd - fix job submission with auth/jwt.
 -- Fix 'scontrol setdebug' not respecting slurm.conf logging configurations.

* Changes in Slurm 21.08.0
==========================
 -- Restored --gpu-bind=single:<ntasks> to check core affinity like
    --gpu-bind=closest does. This removal of this behavior only was in rc2.
 -- slurmd - Fix assert failure on initialization due to bad node name.
 -- Fix error codes in cgroup/v1.
 -- Don't destroy the memory step outside fini, which leads to a double destroy
    causing an error message.
 -- Add support for lua 5.4.
 -- Force cgroup.clone_children to 0 in slurm cgroup directories. This caused
    issues in task cpuset plugin in systems with it enabled by default.
 -- Clear GRES HAS_TYPE flag when removing type name.
 -- Environment flags in gres.conf now override flags set by AutoDetect.
 -- Environment flags in gres.conf now apply to subsequent gres.conf lines where
    Environment flags are not set.
 -- Set missing job_uid and job_gid members when preparing a kill_job_msg_t in
    abort_job_on_node(), abort_job_on_nodes() and kill_job_on_node().
 -- Fix swappiness not being set in cgroups.
 -- Fix coordinators for new subaccounts.
 -- Fix coordinators when adding existing users with PrivateData=users.
 -- slurmctld - do not attempt to relinquish control to self.
 -- openapi/v0.0.37 - Honor kill_on_invalid_dependency as job parameter.
 -- Check max_gres when doing step allocation, fix for regression in rc2.
 -- SPANK plugins are now required to match the current Slurm version, and must
    be recompiled for each new Slurm release.
 -- node_features/helpers - add ExecTime configuration option.
 -- srun - Fix force termination with -X.
 -- On slurmctld restart set node typed GRES counts correctly.
 -- Fix places where a step wasn't allocated in the slurmctld but wasn't ever
    removed from the job.
 -- Fix step allocation memory when using --threads-per-core.
 -- Fix step allocations to consume all threads on a core when using
    threads-per-core.
 -- Add check to validate cpu request on a step if --threads-per-core is given
    and it is less than what the core on the node has in the allocation.
 -- Fix issue where a step could request more gres than the job had and the step
    would hang forever. This bug was only introduced in 21.08.0rc2.
 -- Only print \r\n for logging messages on stderr when --pty has been
    explicitly requested.
 -- Relax check on SPANK plugins to only require Slurm major + minor versions
    to match.
 -- job_container/tmpfs - delegate handling of /dev/shm to the extern step
    so new step launches will be attached correctly even after the slurmd
    process has been restarted.
 -- Limit the wait time in proctrack_g_wait() to UnkillableStepTimeout instead
    of a hardcoded value of 256 seconds, and limit the delay between tests to a
    maximum of 32 seconds.
 -- fatal() on start if using job_container/tmpfs without PrologFlags=Contian.

* Changes in Slurm 21.08.0rc2
=============================
 -- Load bf_when_last_cycle from job state only if protocol version >= 21.08.
 -- Docs - remove man3 section entirely.
 -- Enable OCI container execution support as a technical preview.
 -- Set step memory when using MemPerGPU or DefMemPerGPU. Previously a step's
    memory was not set even when it requested --mem-per-gpu and at least one
    GPU.
 -- Add cli_filter.lua support in configless mode.
 -- Remove some of the MAX macro usages with return codes.
 -- Check that the step requests at least as many gres as nodes.
 -- sacct - Add --json and --yaml arguments.
 -- squeue - Add --json and --yaml arguments.
 -- sinfo - Add --json and --yaml arguments.
 -- Make job's SLURM_JOB_GPUS print global GPU IDs instead of MIG unique_ids.
 -- Fix miscounting of GPU envs in prolog/epilog if MultipleFiles was used.
 -- Support MIGs in prolog/epilog's CUDA_VISIBLE_DEVICES & co.
 -- Add SLURM_JOB_GPUS back into Prolog; add it to Epilog.
 -- Fix issue where the original executable, not the bcast'd version, was
    executed with 'srun --bcast'.
 -- Fix memory leaks when cleaning up config_response_msg_t.
 -- sacct - print '-' header correctly for fields over 53-characters wide.
 -- openapi/dbv0.0.37 - replace "REST" with "Slurm OpenAPI" for plugin_name.
 -- openapi/v0.0.37 - replace "REST" with "Slurm OpenAPI" for plugin_name.
 -- configless - fix segfault on 'scontrol reconfigure'.
 -- Fix bad reference to step_ptr->job_ptr after step_ptr is freed.
 -- Fix bad reference to config_for_clients when shutting down slurmctld.
 -- Fix memory leak on gres_list_alloc when freeing steps.
 -- Fix memory leak on kill_job_msg_t->work_dir when freeing struct.
 -- Fix memory leak on slurmdb_step_rec_t->submit_line when freeing struct.
 -- Fix memory leak in the slurmdbd when requesting wckeys from all clusters.
 -- Use FREE_NULL_LIST instead of list_destroy.
 -- Fix bad pointer reference after bad unpack of jobacctinfo_t.
 -- If we made are running an interactive session we need to force track_steps.
 -- Disable OPOST flag when using --pty to avoid issues with Emac.
 -- Fix issue where extra bonus core was allocated in some situations.
 -- Avoid putting gres with count of 0 on a TRES req/alloc.
 -- Fix determining if a reservation is used or not.
 -- Fix memory in requested TRES when --mem-per-gpu is used.
 -- Changed ReqMem field in sacct to match memory from ReqTRES.
 -- Changed --gpu-bind=single:<ntasks> to no longer check core affinity like
    --gpu-bind=closest does. This consequently affects --ntasks-per-gpu.

* Changes in Slurm 21.08.0rc1
=============================
 -- slurmrestd - add v0.0.37 OpenAPI plugin.
 -- slurmrestd/v0.0.37 - rename standard_in -> standard_input.
 -- slurmrestd/v0.0.37 - rename standard_out -> standard_output.
 -- slurmdbd - Improve log messages of more time than possible on rollups.
 -- Changed the --format handling for negative field widths (left justified)
    to apply to the column headers as well as the printed fields.
 -- Add LimitFactor to the QOS. A float that is factored into an associations
    [Grp|Max]TRES limits.  For example, if the LimitFactor is 2, then an
    association with a GrpTRES of 30 CPUs, would be allowed to allocate 60
    CPUs when running under this QOS.
 -- slurmrestd - Pass SLURM_NO_CHANGE_IN_DATA to client as 403 (Not Modified).
 -- slurmrestd/v0.0.37 - Add update_time field to Jobs query to allow clients
    to only get jobs list based on change timestamp.
 -- Reset job eligible time when job is manually held.
 -- Add DEBUG_FLAG_JAG to improve logging related to job account gathering.
 -- Convert logging in account_gather/common to DEBUG_FLAG_JAG.
 -- Add more logging for jag_common_poll_data() when prec_extra() called.
 -- slurmrestd/v0.0.37 - add API to fetch reservation(s) info.
 -- Catch more errors in task/cgroup initalization and cleanup to avoid allowing
    jobs to start when cgroups failure to configure correctly.
 -- Fix cgroup ns detection when using containers (e.g. LXC or Docker).
 -- Reset job's next_step_id counter to 0 after being requeued.
 -- Make scontrol exit with non-zero status after failing to delete a partition
    or reservation.
 -- Make NtasksPerTRES optional in slurm_sprint_job_info().
 -- slurmrestd/v0.0.37 - Add update_time field to nodes query to allow clients
    to only get nodes list based on change timestamp.
 -- common/parse_config - catch and propagate return codes when handling a match
    on a key-value pattern. This implies error codes detected in the handlers
    are now not ignored and users of _handle_keyvalue_match() can fatal().
 -- common/hostlist - fix hostlist_delete_nth() xassert() upper bound check.
 -- API change: Removed slurm_kill_job_msg and modified the function signature
    for slurm_kill_job2. slurm_kill_job2 should be used instead of
    slurm_kill_job_msg.
 -- Fix non-zero exit code for scontrol ping when all controllers are down.
 -- Enforce a valid configuration for AccountingStorageEnforce in slurm.conf.
    If the configuration is invalid, then an error message will be printed and
    the command or daemon (including slurmctld) will not run.
 -- slurmrestd/v0.0.37 - Add update_time field to partitions/reservations query
    to allow clients to only get the entities list when something changed.
 -- slurmdbd.service - add "After" relationship to all common names for MariaDB
    to reduce startup delays.
 -- slurmrestd/v0.0.37 - Correct displaying node states that are UNKNOWN.
 -- slurmrestd/v0.0.37 - Add flags to node states.
 -- Fix first job on fresh cluster not being assigned JobId=1 (or FirstJobId).
 -- squeue - make it so --nodelist is sensitive to --clusters.
 -- squeue - do --nodelist node validation in the same order as listing.
 -- Removed AccountingStoreJobComment option.  Please update your config to use
    AccountingStoreFlags=job_comment instead.
 -- AccountingStoreFlags=job_script allows you to store the job's batch script.
 -- AccountingStoreFlags=job_env allows you to store the job's env vars.
 -- Add sacct -o SubmitLine to get the submit line of a job/step.
 -- Removed DefaultStorage{Host,Loc,Pass,Port,Type,User} options.
 -- Fix NtasksPerTRES delimiter from : to = in scontrol show job output.
 -- Removed CacheGroups, CheckpointType, JobCheckpointDir, MemLimitEnforce,
    SchedulerPort, SchedulerRootFilter options.
 -- Make job accounting queries use consistent timeframes with and w/o jobs.
 -- --cpus-per-task and --threads-per-core now imply --exact.
    This fixes issues where steps would be allocated the wrong number of CPUs.
 -- configure: the --with option handling has been made consistent across the
    various optional libraries. Specifying --with-foo=/path/to/foo will only
    check that directory for the applicable library (rather than, in some cases,
    falling back to the default directories), and will always error the build
    if the library is not found (instead of a mix of error messages and non-
    fatal warning messages).
 -- configure: replace --with-rmsi_dir option with proper handling for
    --with-rsmi=dir.
 -- Pass additional job environment variables to MailProg.
 -- Add SLURM_JOB_WORK_DIR to Prolog, Epilog.
 -- Removed sched/hold plugin.
 -- Fix srun overwriting SLURM_SUBMIT_DIR and SLURM_SUBMIT_HOST when within an
    existing allocation.
 -- step_ctx code has been removed from the api.
 -- cli_filter/lua, jobcomp/lua, job_submit/lua now load their scripts from the
    same directory as the slurm.conf file (and thus now will respect changes
    to the SLURM_CONF environment variable).
 -- SPANK - call slurm_spank_init if defined without slurm_spank_slurmd_exit in
    slurmd context.
 -- job_container/tmpfs - Remove need for .active file to allow salloc without
    an interactive step to work.
 -- Add new 'PLANNED' state to a node to represent when the backfill scheduler
    has it planned to be used in the future instead of showing as 'IDLE'.
 -- slurmd - Delay background node registration on every failure up to 128s on
    startup.
 -- slurmctld - Always notify slurmd that node registration was accepted to
    avoid slurmd needless attempting to re-register if there is configuration
    issue.
 -- Put node into "INVAL" state upon registering with an invalid node
    configuration. Node must register with a valid configuration to continue.
 -- Make --cpu-bind=threads default for --threads-per-core -- cli and env can
    override.
 -- jobcomp/elasticsearch - Use data_t to serialize data. The plugin now has the
    JSON-C library as a prerequisite.
 -- scrontab - create the temporary file under the TMPDIR environment variable
    (if set), otherwise continue to use TmpFS as configured in slurm.conf.
 -- Add LastBusyTime to "scontrol show nodes" and slurmrestd nodes output,
    which represents the time the node last had jobs on it.
 -- slurmd - allow multiple comma-separated controllers to be specified in
    configless mode with --conf-server
 -- sacctmgr - changed column headings to "ParentID" and "ParentName" instead
    of "Par ID" and "Par Name" respectively.
 -- Perl API - make sure man pages are installed under the --prefix given to
    configure.
 -- Manually powering down of nodes with scontrol now ignores
    SuspendExc<Nodes|Parts>.
 -- SALLOC_THREADS_PER_CORE and SBATCH_THREADS_PER_CORE have been added as
    input environment variables for salloc and sbatch, respectively. They do
    the same thing as --threads-per-core.
 -- Distinguish queued reboot requests (REBOOT) from issued reboots (REBOOT^).
 -- Set the maximum number of open files per process to 4096 to avoid
    performance issues when closing the entire range with closeall().
 -- auth/jwt - add support for RS256 tokens.
 -- Relax reservation purge due to any invalid uid after creation time.
 -- Reject srun that requests both --exclusive and --overlap.
 -- service files - change dependency to network-online rather than just
    network to ensure DNS and other services are available.
 -- RSMI: Fix incorrect PCI BDF bits.
 -- plugins/cli_filter - Convert to using data_t to serialize JSON.
 -- Fix testing array job after regaining locks in backfill.
 -- Don't display node's comment with "scontrol show nodes" unless set.
 -- Add "Extra" field to node to store extra information other than a comment.
 -- scrontab - Use /tmp instead of TmpFS if TMPDIR is not set.
 -- Add ResumeTimeout, SuspendTimeout and SuspendTime to Partitions.
 -- sreport - change to sorting TopUsage by the --tres option.
 -- slurmrestd - do not run allow operation as SlurmUser/root by default.
 -- Allow map_cpu and mask_cpu for non-whole node allocation.
 -- TaskPluginParam=verbose is now treated as a default. Previously it would be
    applied regardless of the job specifying a --cpu-bind.
 -- Add "node_reg_mem_percent" SlurmctldParameter to define percentage of
    memory nodes are allowed to register with.
 -- Show correct number of SocketsPerBoard in slurmd -C with hwloc2.
 -- Alter sreport's cluster utilization report column name from
    'Reserved' to 'Planned' to match the nomenclature of the 'Planned' node.
 -- Add StateComplete format option to sinfo to show base_state+flags.
 -- "scontrol show node" now shows State as base_state+flags instead of
    shortened state with flags appended. eg. IDLE# -> IDLE+POWERING_UP.
    Also "POWER" state flag string is "POWERED_DOWN".
 -- slurmd/req - add missing job_env_t's het_job_id initialization off the
    request in _rpc_{abort,terminate}_job(). This caused problems for Native
    Cray builds when joining a CNCU job_container plugin with Epilog configured.
 -- Fix joining a CNCU job_container on a Native Cray build before executing the
    UnkillableStepProgram for a HetJob step.
 -- slurmrestd/v0.0.35 - Plugin has been tagged as deprecated.
 -- srun - Job steps requiring more cores than available to be rejected unless
    '--overlap' is specificed.
 -- Add bf_node_space_size to SchedulerParameters.
 -- Add scontrol update node state=POWER_DOWN_FORCE and POWER_DOWN_ASAP as new
    ways to power off and reset especially CLOUD nodes.
 -- Define and separate node power state transitions. Previously a powering
    down node was in both states, POWERING_OFF and POWERED_OFF. These are now
    separated.
 -- Create a new process called slurmscriptd which runs PrologSlurmctld and
    EpilogSlurmctld. This avoids fork() calls from slurmctld, and can avoid
    performance issues if the slurmctld has a large memory footprint.
 -- Added new Script option to DebugFlags for debugging slurmscriptd.
 -- scrontab - add ability to update crontab from a file or standard input.
 -- scrontab - add ability to set and expand variables.
 -- Pass JSON of job to node mappings to ResumeProgram.
 -- If running steps in an allocation with CR_PACK_NODE or -mpack the srun will
    only attempt to allocate as much as needed from the allocation instead
    of always trying to allocate every node in the allocation.
 -- Jobs that request the whole node now check to see if any gres are allocated.
 -- Rename SbcastParameters to BcastParameters.
 -- Make srun sensitive to BcastParameters.
 -- RSMI: Add gres_links_create_empty() and preserve RSMI enumeration order.
 -- GPUs: Use index instead of dev_num for CUDA_VISIBLE_DEVICES
 -- Don't run epilog on nodes if job never launched.
 -- QOS accrue limits only apply to the job QOS, not partition QOS.
 -- Add --gpu-bind=per_task:<gpus_per_task> option, --gpus-per-task will now
    set this option by default.
 -- Treat any return code from SPANK plugin that is not SLURM_SUCCESS to be an
    error or rejection.
 -- Print the statistics for extern step adopted processes in sstat.
 -- Fix SLURM_NODE_ALIASES to work for ipv6 node addrs.
 -- Add support for automatically detecting and broadcasting executable shared
    object dependencies for sbcast and srun --bcast.
 -- Delay steps when memory already used instead of rejecting step request.

* Changes in Slurm 20.11.10
===========================
 -- Fix cross-endian communication.
 -- Fix slurmd segfault when seeing authentication issues alongside a failed
    message forward.

* Changes in Slurm 20.11.9
==========================
 -- burst_buffer - add missing common directory to the Makefile SUBDIRS.
 -- sacct - fix truncation when printing jobidraw field.
 -- GRES - Fix loading state of jobs using --gpus to request gpus.
 -- Fix minor logic error in health check node state output
 -- Fix GCC 11.1 compiler warnings.
 -- Delay steps when memory already used instead of rejecting step request.
 -- Fix memory leak in the slurmdbd when requesting wckeys from all clusters.
 -- Fix determining if a reservation is used or not.
 -- openapi/v0.0.35 - Honor kill_on_invalid_dependency as job parameter.
 -- openapi/v0.0.36 - Honor kill_on_invalid_dependency as job parameter.
 -- Fix various issues dealing with updates on magnetic reservations that could
    lead to abort slurmctld.
 -- openapi/v0.0.36 - Avoid setting default values of min_cpus, job name, cwd,
    mail_type, and contiguous on job update.
 -- openapi/v0.0.36 - Clear user hold on job update if hold=false.
 -- Fix slurmctld segfault due to a bit_test() call with a MAINT+ANY_NODES
    reservation NULL node_bitmap.
 -- Fix slurmctld segfault due to a bit_copy() call with a REPLACE+ANY_NODES
    reservation NULL node_bitmap.
 -- Fix error in GPU frequency validation logic.
 -- Fix error in pmix logic dealing with the incorrect size of buffer.
 -- PMIx v1.1.4 and below are no longer supported.
 -- Fix shutdown of slurmdbd plugin to correctly notice when the agent thread
    finishes.
 -- Fix slurmctld segfault due to job array --batch features double free.
 -- CVE-2022-29500 - Prevent credential abuse.
 -- CVE-2022-29501 - Prevent abuse of REQUEST_FORWARD_DATA.

* Changes in Slurm 20.11.8
==========================
 -- slurmctld - fix erroneous "StepId=CORRUPT" messages in error logs.
 -- Correct the error given when auth plugin fails to pack a credential.
 -- Fix unused-variable compiler warning on FreeBSD in fd_resolve_path().
 -- acct_gather_filesystem/lustre - only emit collection error once per step.
 -- srun - leave SLURM_DIST_UNKNOWN as default for --interactive.
 -- Add GRES environment variables (e.g., CUDA_VISIBLE_DEVICES) into the
    interactive step, the same as is done for the batch step.
 -- Fix various potential deadlocks when altering objects in the database
    dealing with every cluster in the database.
 -- slurmrestd - handle slurmdbd connection failures without segfaulting.
 -- slurmrestd - fix segfault for searches in slurmdb/v0.0.36/jobs.
 -- slurmrestd - remove (non-functioning) users query parameter for
    slurmdb/v0.0.36/jobs from openapi.json
 -- slurmrestd - fix segfault in slurmrestd db/jobs with numeric queries
 -- slurmrestd - add argv handling for job/submit endpoint.
 -- srun - fix broken node step allocation in a heterogeneous allocation.
 -- Fail step creation if -n is not multiple of --ntasks-per-gpu.
 -- job_container/tmpfs - Fix slowdown on teardown.
 -- Fix problem with SlurmctldProlog where requeued jobs would never launch.
 -- job_container/tmpfs - Fix issue when restarting slurmd where the namespace
    mount points could disappear.
 -- sacct - avoid truncating JobId at 34 characters.
 -- scancel - fix segfault when --wckey filtering option is used.
 -- select/cons_tres - Fix memory leak.
 -- Prevent file descriptor leak in job_container/tmpfs on slurmd restart.
 -- slurmrestd/dbv0.0.36 - Fix values dumped in job state/current and
    job step state.
 -- slurmrestd/dbv0.0.36 - Correct description for previous state property.
 -- perlapi/libslurmdb - expose tres_req_str to job hash.
 -- scrontab - close and reopen temporary crontab file to deal with editors
    that do not change the original file, but instead write out then rename
    a new file.
 -- sstat - fix linking so that it will work when --without-shared-libslurm
    was used to build Slurm.
 -- Clear allocated cpus for running steps in a job before handling requested
    nodes on new step.
 -- Don't reject a step if not enough nodes are available. Instead, defer the
    step until enough nodes are available to satisfy the request.
 -- Don't reject a step if it requests at least one specific node that is
    already allocated to another step. Instead, defer the step until the
    requested node(s) become available.
 -- slurmrestd - add description for slurmdb/job endpoint.
 -- Better handling of --mem=0.
 -- Ignore DefCpuPerGpu when --cpus-per-task given.
 -- sacct - fix segfault when printing StepId (or when using --long).

* Changes in Slurm 20.11.7
==========================
 -- slurmd - handle configless failures gracefully instead of hanging
    indefinitely.
 -- select/cons_tres - fix Dragonfly topology not selecting nodes in the same
    leaf switch when it should as well as requests with --switches option.
 -- Fix issue where certain step requests wouldn't run if the first node in the
    job allocation was full and there were idle resources on other nodes in
    the job allocation.
 -- Fix deadlock issue with <Prolog|Epilog>Slurmctld.
 -- torque/qstat - fix printf error message in output.
 -- When adding associations or wckeys avoid checking multiple times a user or
    cluster name.
 -- Fix wrong jobacctgather information on a step on multiple nodes
    due to timeouts sending its the information gathered on its node.
 -- Fix missing xstrdup which could result in slurmctld segfault on array jobs.
 -- Fix security issue in PrologSlurmctld and EpilogSlurmctld by always
    prepending SPANK_ to all user-set environment variables. CVE-2021-31215.

* Changes in Slurm 20.11.6
==========================
 -- Fix sacct assert with the --qos option.
 -- Use pkg-config --atleast-version instead of --modversion for systemd.
 -- common/fd - fix getsockopt() call in fd_get_socket_error().
 -- Properly handle the return from fd_get_socket_error() in _conn_readable().
 -- cons_res - Fix issue where running jobs were not taken into consideration
    when creating a reservation.
 -- Avoid a deadlock between job_list for_each and assoc QOS_LOCK.
 -- Fix TRESRunMins usage for partition qos on restart/reconfig.
 -- Fix printing of number of tasks on a completed job that didn't request
    tasks.
 -- Fix updating GrpTRESRunMins when decrementing job time is bigger than it.
 -- Make it so we handle multithreaded allocations correctly when doing
    --exclusive or --core-spec allocations.
 -- Fix incorrect round-up division in _pick_step_cores
 -- Use appropriate math to adjust cpu counts when --ntasks-per-core=1.
 -- cons_tres - Fix consideration of power downed nodes.
 -- cons_tres - Fix DefCpuPerGPU, increase cpus-per-task to match with
    gpus-per-task * cpus-per-gpu.
 -- Fix under-cpu memory auto-adjustment when MaxMemPerCPU is set.
 -- Make it possible to override CR_CORE_DEFAULT_DIST_BLOCK.
 -- Perl API - fix retrieving/storing of slurm_step_id_t in job_step_info_t.
 -- Recover state of burst buffers when slurmctld is restarted to avoid skipping
    burst buffer stages.
 -- Fix race condition in burst buffer plugin which caused a burst buffer
    in stage-in to not get state saved if slurmctld stopped.
 -- auth/jwt - print an error if jwt_file= has not been set in slurmdbd.
 -- Fix RESV_DEL_HOLD not being a valid state when using squeue --states.
 -- Add missing squeue selectable states in valid states error message.
 -- Fix scheduling last array task multiple times on error, causing segfault.
 -- Fix issue where a step could be allocated more memory than the job when
    dealing with --mem-per-cpu and --threads-per-core.
 -- Fix removing qos from assoc with -= can lead to assoc with no qos
 -- auth/jwt - fix segfault on invalid credential in slurmdbd due to
    missing validate_slurm_user() function in context.
 -- Fix single Port= not being applied to range of nodes in slurm.conf
 -- Fix Jobs not requesting a tres are not starting because of that tres limit.
 -- acct_gather_energy/rapl - fix AveWatts calculation.
 -- job_container/tmpfs - Fix issues with cleanup and slurmd restarting on
    running jobs.

* Changes in Slurm 20.11.5
==========================
 -- Fix main scheduler bug where bf_hetjob_prio truncates SchedulerParameters.
 -- Fix sacct not displaying UserCPU, SystemCPU and TotalCPU for large times.
 -- scrontab - fix to return the correct index for a bad #SCRON option.
 -- scrontab - fix memory leak when invalid option found in #SCRON line.
 -- Add errno for when a user requests multiple partitions and they are using
    partition based associations.
 -- Fix issue where a job could run in a wrong partition when using
    EnforcePartLimits=any and partition based associations.
 -- Remove possible deadlock when adding associations/wckeys in multiple
    threads.
 -- When using PrologFlags=alloc make sure the correct Slurm version is set
    in the credential.
 -- When sending a job a warning signal make sure we always send SIGCONT
    beforehand.
 -- Fix issue where a batch job would continue running if a prolog failed on a
    node that wasn't the batch host and requeuing was disabled.
 -- Fix issue where sometimes salloc/srun wouldn't get a message about a prolog
    failure in the job's stdout.
 -- Requeue or kill job on a prolog failure when PrologFlags is not set.
 -- Fix race condition causing node reboots to get requeued before
    ResumeTimeout expires.
 -- Preserve node boot_req_time on reconfigure.
 -- Preserve node power_save_req_time on reconfigure.
 -- Fix node reboots being queued and issued multiple times and preventing the
    reboot to time out.
 -- Fix debug message related to GrpTRESRunMin (AssocGrpCPURunMinutesLimit).
 -- Fix run_command to exit correctly if track_script kills the calling thread.
 -- Only requeue a job when the PrologSlurmctld returns nonzero.
 -- When a job is signaled with SIGKILL make sure we flush all
    prologs/setup scripts.
 -- Handle burst buffer scripts if the job is canceled while stage_in is
    happening.
 -- When shutting down the slurmctld make note to ignore error message when
    we have to kill a prolog/setup script we are tracking.
 -- scrontab - add support for the --open-mode option.
 -- acct_gather_profile/influxdb - avoid segfault on plugin shutdown if setup
    has not completed successfully.
 -- Reduce delay in starting salloc allocations when running with prologs.
 -- Fix issue passing open fd's with [send|recv]msg.
 -- Alter AllocNodes check to work if the allocating node's domain doesn't
    match the slurmctld's. This restores the pre-20.11 behavior.
 -- Fix slurmctld segfault if jobs from a prior version had the now-removed
    INVALID_DEPEND state flag set and were allowed to run in 20.11.
 -- Add job_container/tmpfs plugin to give a method to provide a private /tmp
    per job.
 -- Set the correct core affinity when using AutoDetect.
 -- Start relying on the conf again in xcpuinfo_mac_to_abs().
 -- Fix global_last_rollup assignment on job resizing.
 -- slurmrestd - hand over connection context on _on_message_complete().
 -- slurmrestd - mark "environment" as required for job submissions in schema.
 -- slurmrestd - Disable credential reuse on the same TCP connection. Pipelined
    HTTP connections will have to provide authentication with every request.
 -- Avoid data conversion error on NULL strings in data_get_string_converted().
 -- Handle situation where slurmctld is too slow processing
    REQUEST_COMPLETE_BATCH_SCRIPT and it gets resent from the slurmstepd.
 -- Fix sacct crashing with the --qos option

* Changes in Slurm 20.11.4
==========================
 -- Fix node selection for advanced reservations with features.
 -- mpi/pmix: Handle pipe failure better when using ucx.
 -- mpi/pmix: include PMIX_NODEID for each process entry.
 -- Fix job getting rejected after being requeued on same node that died.
 -- job_submit/lua - add "network" field.
 -- Fix situations when a reoccuring reservation could erroneously skip a
    period.
 -- Ensure that a reservations [pro|epi]log are ran on reoccuring reservations.
 -- Fix threads-per-core memory allocation issue when using CR_CPU_MEMORY.
 -- Fix scheduling issue with --gpus.
 -- Fix gpu allocations that request --cpus-per-task.
 -- mpi/pmix: fixed print messages for all PMIXP_* macros
 -- Add mapping for XCPU to --signal option.
 -- Fix regression in 20.11 that prevented a full pass of the main scheduler
    from ever executing.
 -- Work around a glibc bug in which "0" is incorrectly printed as "nan"
    which will result in corrupted association state on restart.
 -- Fix regression in 20.11 which made slurmd incorrectly attempt to find the
    parent slurmd address when not applicable and send incorrect reverse-tree
    info to the slurmstepd.
 -- Fix cgroup ns detection when using containers (e.g. LXC or Docker).
 -- scrontab - change temporary file handling to work with emacs.

* Changes in Slurm 20.11.3
==========================
 -- Fix segfault when parsing bad "#SBATCH hetjob" directive.
 -- Allow countless gpu:<type> node GRES specifications in slurm.conf.
 -- PMIx - Don't set UCX_MEM_MMAP_RELOC for older version of UCX (pre 1.5).
 -- Don't green-light any GPU validation when core conversion fails.
 -- Allow updates to a reservation in the database that starts in the future.
 -- Better check/handling of primary key collision in reservation table.
 -- Improve reported error and logging in _build_node_list().
 -- Fix uninitialized variable in _rpc_file_bcast() which could lead to an
    incorrect error return from sbcast / srun --bcast.
 -- mpi/cray_shasta - fix use-after-free on error in _multi_prog_parse().
 -- Cray - Handle setting correct prefix for cpuset cgroup with respects to
    expected_usage_in_bytes.  This fixes Cray's OOM killer.
 -- mpi/pmix: Fix PMIx_Abort support.
 -- Don't reject jobs allocating more cores than tasks with MaxMemPerCPU.
 -- Fix false error message complaining about oversubscribe in cons_tres.
 -- scrontab - fix parsing of empty lines.
 -- Fix regression causing spank_process_option errors to be ignored.
 -- Avoid making multiple interactive steps.
 -- Fix corner case issues where step creation should fail.
 -- Fix job rejection when --gres is less than --gpus.
 -- Fix regression causing spank prolog/epilog not to be called unless the
    spank plugin was loaded in slurmd context.
 -- Fix regression preventing SLURM_HINT=nomultithread from being used
    to set defaults for salloc->srun, sbatch->srun sequence.
 -- Reject job credential if non-superuser sets the LAUNCH_NO_ALLOC flag.
 -- Make it so srun --no-allocate works again.
 -- jobacct_gather/linux - Don't count memory on tasks that have already
    finished.
 -- Fix 19.05/20.02 batch steps talking with a 20.11 slurmctld.
 -- jobacct_gather/common - Do not process jobacct's with same taskid when
    calling prec_extra.
 -- Cleanup all tracked jobacct tasks when extern step child process finishes.
 -- slurmrestd/dbv0.0.36 - Correct structure of dbv0.0.36_tres_list.
 -- Fix regression causing task/affinity and task/cgroup to be out of sync when
    configured ThreadsPerCore is different than the physical threads per core.
 -- Fix situation when --gpus is given but not max nodes (-N1-1) in a job
    allocation.
 -- Interactive step - ignore cpu bind and mem bind options, and do not set
    the associated environment variables which lead to unexpected behavior
    from srun commands launched within the interactive step.
 -- Handle exit code from pipe when using UCX with PMIx.
 -- Partially revert changes made in 20.11.0 to srun step behavior. This change
    required steps to explicitly request resources, otherwise the minimal set
    of resources required would be assigned. This broke OpenMPI's 'mpiexec'
    alongside other use cases. This reverts the behavior such that all resources
    on a node are assigned to the job step by default.
 -- srun - add a new --exact option, and deprecate the --whole option (which has
    been restored as the default behavior).

* Changes in Slurm 20.11.2
==========================
 -- Fix older versions of sacct not working with 20.11.
 -- Fix slurmctld crash when using a pre-20.11 srun in a job allocation.
 -- Correct logic problem in _validate_user_access.
 -- Fix libpmi to initialize Slurm configuration correctly.

* Changes in Slurm 20.11.1
==========================
 -- Fix spelling of "overcomited" to "overcomitted" in sreport's cluster
    utilization report.
 -- Silence debug message about shutting down backup controllers if none are
    configured.
 -- Don't create interactive srun until PrologSlurmctld is done.
 -- Fix fd symlink path resolution.
 -- Fix slurmctld segfault on subnode reservation restore after node
    configuration change.
 -- Fix resource allocation response message environment allocation size.
 -- Ensure that details->env_sup is NULL terminated.
 -- select/cray_aries - Correctly remove jobs/steps from blades using NPC.
 -- cons_tres - Avoid max_node_gres when entire node is allocated with
    --ntasks-per-gpu.
 -- Allow NULL arg to data_get_type().
 -- In sreport have usage for a reservation contain all jobs that ran in the
    reservation instead of just the ones that ran in the time specified. This
    matches the report for the reservation is not truncated for a time period.
 -- Fix issue with sending wrong batch step id to a < 20.11 slurmd.
 -- Add a job's alloc_node to lua for job modification and completion.
 -- Fix regression getting a slurmdbd connection through the perl API.
 -- Stop the extern step terminate monitor right after proctrack_g_wait().
 -- Fix removing the normalized priority of assocs.
 -- slurmrestd/v0.0.36 - Use correct name for partition field:
    "min nodes per job" -> "min_nodes_per_job".
 -- slurmrestd/v0.0.36 - Add node comment field.
 -- Fix regression marking cloud nodes as "unexpectedly rebooted" after
    multiple boots.
 -- Fix slurmctld segfault in _slurm_rpc_job_step_create().
 -- slurmrestd/v0.0.36 - Filter node states against NODE_STATE_BASE to avoid
    the extended states all being reported as "invalid".
 -- Fix race that can prevent the prolog for a requeued job from running.
 -- cli_filter - add "type" to readily distinguish between the CLI command in
    use.
 -- smail - reduce sleep before seff to 5 seconds.
 -- Ensure SPANK prolog and epilog run without an explicit PlugStackConfig.
 -- Disable MySQL automatic reconnection.
 -- Fix allowing "b" after memory unit suffixes.
 -- Fix slurmctld segfault with reservations without licenses.
 -- Due to internal restructuring ahead of the 20.11 release, applications
    calling libslurm MUST call slurm_init(NULL) before any API calls.
    Otherwise the API call is likely to fail due to libslurm's internal
    configuration not being available.
 -- slurm.spec - allow custom paths for PMIx and UCX install locations.
 -- Use rpath if enabled when testing for Mellanox's UCX libraries.
 -- slurmrestd/dbv0.0.36 - Change user query for associations to optional.
 -- slurmrestd/dbv0.0.36 - Change account query for associations to optional.
 -- mpi/pmix - change the error handler error message to be more useful.
 -- Add missing connection in acct_storage_p_{clear_stats, reconfig, shutdown}.
 -- Perl API - fix issue when running in configless mode.