NEWS

This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.

* Changes in Slurm 22.05.3
==========================
 -- job_container/tmpfs - cleanup containers even when the .ns file isn't
    mounted anymore.
 -- Ignore the bf_licenses option if using sched/builtin.
 -- Do not clear the job's requested QOS (qos_id) when ineligible due to QOS.
 -- Emit error and add fail-safe when job's qos_id changes unexpectedly.
 -- Fix timeout value in log.
 -- openapi/v0.0.38 - fix setting of DefaultTime when dumping a partition.
 -- openapi/dbv0.0.38 - correct parsing association QOS field.
 -- Fix LaunchParameters=mpir_use_nodeaddr.
 -- Fix various edge cases where accrue limits could be exceeded or cause
    underflow error messages.
 -- Fix issue where a job requesting --ntasks and --nodes could be wrongly
    rejected when spanning heterogeneous nodes.
 -- openapi/v0.0.38 - detect when partition PreemptMode is disabled
 -- openapi/v0.0.38 - add QOS flag to handle partition PreemptMode=within
 -- Add total_cpus and total_nodes values to the partition list in
    the job_submit/lua plugin.
 -- openapi/dbv0.0.38 - reject and error on invalid flag values in well defined
    flag fields.
 -- openapi/dbv0.0.38 - correct QOS preempt_mode flag requests being silently
    ignored.
 -- accounting_storage/mysql - allow QOS preempt_mode flag updates when GANG
    mode is requested.
 -- openapi/dbv0.0.38 - correct QOS flag modifications request being silently
    ignored.
 -- sacct/sinfo/squeue - use openapi/[db]v0.0.38 for --json and --yaml modes.
 -- Improve error messages when using configless and fetching the config fails.
 -- Fix segfault when reboot_from_controller is configured and scontrol reboot
    is used.
 -- Fix regression which prevented a cons_tres gpu job to be submitted to a
    cons_tres cluster from a non-con_tres cluster.
 -- openapi/dbv0.0.38 - correct association QOS list parsing for updates.
 -- Fix rollup incorrectly divying up unused reservation time between
    associations.
 -- slurmrestd - add SLURMRESTD_SECURITY=disable_unshare_files environment
    variable.

* Changes in Slurm 22.05.2
==========================
 -- Fix a segfault in slurmctld when requesting gres in job arrays.
 -- Prevent jobs from launching on newly powered up nodes that register with
    invalid config.
 -- Fix a segfault when there's no memory.swap.current interface in cgroup/v2.
 -- Fix memleak in cgroup/v2.

* Changes in Slurm 22.05.1
==========================
 -- Flush the list of Include config files on SIGHUP.
 -- Fix and update Slurm completion script.
 -- jobacct_gather/cgroup - Add VMem support both for cgroup v1 and v2.
 -- Allow subset of node state transitions when node is in INVAL state.
 -- Remove INVAL state from cloud node after being powered down.
 -- When showing reason UID in scontrol show node, use the authenticated UID
    instead of the login UID.
 -- Fix calculation of reservation's NodeCnt when using dynamic nodes.
 -- Add SBATCH_{ERROR,INPUT,OUTPUT} input environment variables for --error,
    --input and --output options respectively.
 -- Prevent oversubscription of licenses by the backfill scheduler when not
    using the new "bf_licenses" option.
 -- Jobs with multiple nodes in a heterogeneous cluster now have access to all
    the memory on each node by using --mem=0. Previously the memory limit was
    set by the node with the least amount of memory.
 -- Don't limit the size of TaskProlog output (previously TaskProlog output was
    limited to 4094 characters per line, which limited the size of exported
    environment variables or logging to the task).
 -- Fix usage of possibly uninitialized buffer in proctrack/cgroup.
 -- Fix memleak in proctrack/cgroup proctrack_p_wait.
 -- Fix cloud/remote het srun jobs.
 -- Fix a segfault that may happen on gpu configured as no_consume.

* Changes in Slurm 22.05.0
==========================
 -- openapi/v0.0.38 - add group name to job info
 -- openapi/v0.0.38 - add container field to job description.
 -- openapi/v0.0.38 - add container to job submission.
 -- openapi/[db]v0.0.38 - add container field to job description.
 -- openapi/dbv0.0.38 - fix bug where QOS update set various limits to 0.
 -- openapi/dbv0.0.38 - properly initialize wckey record.
 -- openapi/dbv0.0.38 - properly initialize cluster record.
 -- openapi/dbv0.0.38 - gracefully update existing QOSs.
 -- Fix x11 forwarding with job_container/tmpfs and without home_xauthority.
 -- Fix reconfig of dynamic nodes with gres.
 -- Fix corner case issues when removing assocs/qos' while jobs are completing
    and a reconfigure happens.
 -- Add SlurmdParameters=numa_node_as_socket to use the numa node as a socket.
 -- Avoid creating and referencing NULL script/env hash entries when not storing
    the job script or job environment.
 -- slurmd - If a het component job fails to launch due to a node prolog or
    other failure, properly handle the whole het job.
 -- Fix possible race condition while handling forwarded messages.
 -- Add new SchedulerParameters option "bf_licenses" to track licenses as
    within the backfill scheduler.
 -- Fix resuming nodes not part of an allocation.
 -- Minor memory leak fixes.
 -- Use accept4() and pipe2() to ensure new file descriptors are always set
    with close-on-exec flag.
 -- slurmctld - allow users to request tokens for themselves by name.
 -- srun - adjust output of "--mpi=list".
 -- Fix "--gpu-bind=single:" having wrong env variables.
 -- Sync missing pieces in slurmctld SIGHUP handler with respect to
    REQUEST_RECONFIGURE RPC handler.
 -- Fixed the behavior of the --states=all option in squeue to actually show
    all of the states.
 -- job_container - Don't constrain a job using --no-allocate.
 -- Fix slurmctld segfault when a step requests a gres without a file.
 -- Fix slurmctld segfault when cleaning up an overlapping step.
 -- Dynamic node fixes and updates.
 -- Added error logging when a node goes into an invalid state so that the user
    can still see the reason even if the user set a custom reason beforehand.
 -- Remove redundant message when libdbus is found during configure.
 -- Fix pmix to honor SrunPortRange.
 -- Fix issue with ntasks_per_socket and ntasks_per_core with --exclusive
 -- openapi/v0.0.38 - change job parsing error message to reflect reality.
 -- Avoid segfault/assert when --thread-spec used with config_overrides
 -- Don't send SIGTERM to prematurely in run_command.
 -- Add logging to inform where networking errors are occurring on slurmd end.
 -- openapi/dbv0.0.38 - Enforce GET /association/ to dump only a single
    association.
 -- openapi/dbv0.0.38 - Enforce DELETE /association/ to delete only a single
    association.
 -- openapi/dbv0.0.38 - Add parameters to filter to GET /associations/.
 -- openapi/dbv0.0.38 - Add DELETE /associations using parameters to set filters.
 -- openapi/dbv0.0.38 - Fix plural form of delete association response schema.

* Changes in Slurm 22.05.0rc1
=============================
 -- gres/gpu - Avoid stripping GRES type field during normalization if any
    other GRES have a defined type field.
 -- burst_buffer/datawarp - free bb_job after stage-out or teardown are done.
 -- acct_gather_energy_rsmi has been renamed acct_gather_energy_gpu.
 -- Remove support for (non-functional) --cpu-bind=boards.
 -- accounting_storage/mysql - ensure new non-HetJobs have het_job_offset NO_VAL
    in the database and fix the same field when retrieving older bad records.
 -- PreemptMode now works as a condition for qos in sacctmgr.
 -- scancel - add "--me" option.
 -- gres/gpu - Fix configured/system-detected GRES match for some combinations
    for which the sorting affected the expected selection result.
 -- openapi/v0.0.38 - Fork existing openapi/v0.0.37 plugin.
 -- openapi/dbv0.0.38 - Fork existing openapi/dbv0.0.37 plugin.
 -- openapi/v0.0.35 - Plugin has been removed.
 -- scrontab - Don't accept extra letters after a '@' repeating pattern.
 -- openapi/dbv0.0.38 - Add missing method POST for /associations/.
 -- Make DefMemPerCPU/MaxMemPerCPU and DefMemPerNode/MaxMemPerNode precedence
    at the global level the same as in partition level, and print an error
    if both of a pair are set.
 -- openapi/v0.0.38 - Allow strings for JobIds instead of only numerical JobIds
    for GET, DELETE, and POST job methods.
 -- openapi/v0.0.38 - enable job priority field for job submissions and updates.
 -- openapi/v0.0.38 - request node states query includes MIXED state instead of
    only allocated.
 -- openapi/dbv0.0.38 - Correct tree position of dbv0.0.38_job_step.
 -- Enforce all Slurm plugin requirements for plugins specified with absolute
    path.
 -- Added --prefer option at job submission to allow for 'soft' constraints.
 -- Remove support for PMIx 1.x
 -- slurmrestd - Unlink unix sockets before binding to avoid errors where
    previous run of slurmrestd did not properly cleanup the sockets.
 -- Add extra 'EnvironmentFile=-/etc/default/$service' setting to service files.
 -- slurmctld/agent - spawn more agent threads in a single _agent_retry().
 -- openapi/v0.0.38 - change job response types to more specific types than
    generic string.
 -- srun - refuse to run on malformed SPANK environment variable.
 -- Allow jobs to pack onto nodes already rebooting with the desired features.
 -- Reset job start time after nodes are rebooted.
 -- Node features (if any) are passed to RebootProgram if run from slurmctld.
 -- Fix sending multiple features to RebootProgram.
 -- If a task/srun prolog fails don't allow the step to continue running.
 -- Fail srun when using invalid --cpu-bind options.
 -- cgroup/v1 - Set swappiness at job level instead of at root level.
 -- Fix issues where a step's GRES request could never be satisfied but the step
    remained pending forever instead of being rejected.
 -- openapi/v0.0.38 - Remove errant extra space after JOB_CPUS_SET flag.
 -- slurmrestd - refuse to run with gid 0, or under SlurmUser's gid.
 -- Storing batch scripts and env vars are now in indexed tables using
    substantially less disk space.  Those storing scripts in 21.08 will all
    be moved and indexed automatically.
 -- Run MailProg through slurmscriptd instead of directly fork+exec()'ing
    from slurmctld.
 -- Add acct_gather_interconnect/sysfs plugin.
 -- Fix gpus spanning more resources than needed when usng --cpus-per-gpu.
 -- burst_buffer plugins - err_msg added to bb_p_job_validate().
 -- burst_buffer plugins - Send user specific error message if permission
    denied due to AllowUsers or DenyUsers in burst_buffer.conf.
 -- Fatal if the mutually-exclusive JobAcctGatherParams options of UsePss and
    NoShared are both defined.
 -- Future and Cloud nodes are treated as "Planned Down" in usage reports.
 -- Add "condflags=open" to sacctmgr show events to return open/currently down
    events.
 -- Skip --mem-per-gpu impact on GRES availability when CR_MEMORY not set.
 -- Add reservation start time as a new schedulers sorting factor for magnetic
    or multi-reservation job queue records.
 -- Improve Dynamic Future node startup to load in config with mapped nodename.
 -- Add new shard plugin for sharing gpus but not with mps.
 -- acct_gather_energy/xcc - add support for Lenovo SD650 V2.
 -- Remove cgroup_allowed_devices_file.conf support.
 -- sacct -f flag implies -c flag.
 -- Node state flags (DRAIN, FAILED, POWERING UP, etc.) will be cleared now if
    node state is updated to FUTURE.
 -- Clear GRES environment variables for jobs or steps that request --gres=none.
 -- slurmctld - Avoid requiring environment to be set when container job is
    specified.
 -- Move the KeepAliveTime option into CommunicationParameters.
 -- slurm.spec - stop explicitly packaging pkgconfig directory to avoid a
    conflict with the pkgconfig package.
 -- Fix sacctmgr load dump printing incorrect default QOS when no value is set.
 -- Fix very early logging in the controller when daemonized
 -- Fix possibility of primary group going missing when running jobs submitted
    after a user has switched primary groups (eg: with sg).
 -- Make it so srun no longer inherits SLURM_CPUS_PER_TASK. It now will only
    inherit SRUN_CPUS_PER_TASK.
 -- Make HetJob signaling requests using the HetJobId+HetJobOffset format behave
    the same as with direct JobId requests with regards to job state and
    whole_hetjob considerations.
 -- scancel - avoid issuing a RPC for each HetJob non-leader components hit by
    using filters (not using jobids) when a RPC is issued for their leader.
 -- Make slurmctld call the JobCompPlugin set location operation on SIGUSR2. As
    a relevant consequence, the filetxt plugin reopens the file for potential
    logrotation.
 -- openapi/v0.0.38 - added fields bf_table_size and bf_table_size_mean to
    diag query.
 -- Handle scontrol setdebug[flags] and scontrol reconfigure in slurmscriptd.
 -- openapi/v0.0.38 - add /licenses endpoint.
 -- slurmrestd - only compile JWT authentication plugin if libjwt is present.
 -- Remove connect_timeout and timeout options from JobCompParams as there's no
    longer a connectivity check happening in the jobcomp/elasticsearch plugin
    when setting the location off of JobCompLoc.
 -- Add %n for NodeName substitution in SlurmdSpoolDir in nss_slurm.conf.
 -- job_container/tmpfs - add basic environment variables to InitScript.
 -- openapi/dbv0.0.38 - split security entries in openapi.json.
 -- openapi/v0.0.38 - Add schema for Slurm meta in responses.
 -- openapi/dbv0.0.38 - Add schema for Slurm meta in responses.
 -- openapi/v0.0.38 - Correct field "errno" to "error_number" in schema for
    errors.
 -- openapi/dbv0.0.38 - add failure response contents in openapi.json.
 -- openapi/v0.0.38 - add failure response contents in openapi.json.
 -- openapi/dbv0.0.38 - add requestBody for /users/ in openapi.json.
 -- openapi/dbv0.0.38 - add requestBody for /accounts/ in openapi.json.
 -- openapi/dbv0.0.38 - add missing response field "removed_associations"
    in openapi.json
 -- openapi/dbv0.0.38 - Correct type from object to array for user assocations
    list in openapi.json.
 -- openapi/dbv0.0.38 - Add missing field for tres minutes per qos in qos list
    in openapi.json.
 -- openapi/dbv0.0.38 - Add missing field for name in qos list in openapi.json.
 -- openapi/dbv0.0.38 - Correct tres field in qos list in openapi.json.
 -- openapi/dbv0.0.38 - Correct het job details types in jobs in openapi.json
 -- openapi/dbv0.0.38 - Correct task type for job steps in openapi.json.
 -- openapi/dbv0.0.38 - Sync fields for errors to source in openapi.json.
 -- openapi/dbv0.0.38 - Add missing field for adding clusters in clusters list
    in openapi.json.
 -- openapi/v0.0.38 - correct parameter styles from "simple" to "form".
 -- openapi/dbv0.0.38 - correct parameter styles from "simple" to "form".
 -- openapi - add flags to slurm_openapi_p_get_specification().
 -- openapi/v0.0.38 - use new operationId generation to ensure uniqueness.
 -- openapi/dbv0.0.38 - use new operationId generation to ensure uniqueness.
 -- slurmstepd - avoid possible race condition while updating step return code
    during job startup.
 -- slurmstepd - report and log I/O setup failure using corresponding error
    code during job start.
 -- Do not run the job if PrologSlurmctld times out.
 -- Fix SPANK plugin calls slurm_spank_job_prolog and slurm_spank_job_epilog
    not properly respecting PrologEpilogTimeout.
 -- Add support for 'scontrol delete <ENTITY> <ID>' format alongside the already
    expected 'scontrol delete <ENTITY>=<ID>'.
 -- Fix task/cgroup plugin which was not adding stepd to memory cgroup.
 -- preempt/qos - add support for WITHIN mode to allow for preemption between
    jobs within the same qos.
 -- Fix sattach for interactive step.
 -- Fix srun -Z when using nss_slurm.
 -- Avoid memory leak in srun -Z.
 -- openapi/dbv0.0.38 - disable automatic lookup of all HetJob components on
    specific job lookups.
 -- Add support for hourly reoccurring reservations.
 -- openapi/v0.0.38 - Fix misspelling of job parameter time_minimum.
 -- openapi/v0.0.38 - Fix misspelling of job parameter cpu_binding_hint.
 -- openapi/v0.0.38 - Fix misspelling of job parameter mcs_label
 -- openapi/dbv0.0.38 - Fix issue where association's QOS list consisted of IDs
    instead of names.
 -- Allow nodes to be dynamically added and removed from the system.
 -- srun --overlap now allows the step to share all resources (CPUs, memory, and
    GRES), where previously --overlap only allowed the step to share CPUs with
    other steps.
 -- openapi/v0.0.38 - new format for how core and socket allocations are dumped
    for jobs.
 -- openapi/v0.0.38 - add RPC call statistics to diag endpoint.
 -- openapi/[db]v0.0.36 - plugins have been marked as deprecated.
 -- sacct - allocations made by srun will now always display the allocation and
    step(s). Previously, the allocation and step were combined when possible.
 -- Steps now allocate gres according to topo data.
 -- Add validation of numbers provided to --gpu-bind=map_gpu and
    --gpu-bind=mask_gpu=.
 -- Fatal error if CgroupReleaseAgentDir is configured in cgroup.conf. The
    option has long been obsolete.
 -- cons_tres - change definition of the "least loaded node" (LLN) to the
    node with the greatest ratio of available cpus to total cpus.
 -- Fatal if more than one burst buffer plugin is configured.
 -- Added keepaliveinterval and keepaliveprobes options to
    CommunicationParameters.
 -- Correctly failover to backup slurmdbd if the primary slurmdbd node crashes.
 -- Add support to ship Include configuration files with configless.
 -- Fix issues with track_script cleanup.
 -- Add new max_token_lifespan limit to AuthAltParameters.
 -- sacctmgr - allow Admins to update AdminComment and SystemComment fields.
 -- slurmd - Cache node features at startup and scontrol reconfig. It avoids
    executing node_features/helpers scripts in the middle of a job. This clould
    potentially affect job performance.
 -- Pass and use alias_list through credential instead of environment variable.
 -- Add ability to get host addresses from nss_slurm.
 -- Enable reverse fanout for cloud+alias_list jobs.
 -- Disallow slurm.conf node configurations with NodeName=ALL.
 -- Add support to delete/update nodes by specifying nodesets or the 'ALL'
    keyword alongside the delete/update node message nodelist expression (i.e.
    'scontrol delete/update NodeName=ALL' or 'scontrol delete/update
    NodeName=ns1,nodes[1-3]').
 -- Add support for PMIx v4
 -- Add support for PMIx v5
 -- Correctly load potentially NULL values from slurmdbd archive files.
 -- Add slurmdbd.conf option AllowNoDefAcct to remove requirement for users to
    have a default account.
 -- Fix issues related to users with very large uids.
 -- common/openapi - fix bug populating methods for parameter-less endpoint.
 -- slurmrestd/operations - fix memory leak when resolving bad path.
 -- sacctmgr - improve performance of query generation for archive load.
 -- sattach - allow connecting to interactive steps with JOBID.interactive
 -- Fix for --gpus-per-task parsing when using --cpus-per-gpu and multiple gpu
    types in the same request.
 -- rest_auth/local - always log when new slurmdbd connection fails.
 -- openapi/dbv0.0.38 - gracefully update existing associations.
 -- Improve auto-detection of pmix
 -- openapi/dbv0.0.38 - Add missing method POST for /qos/.
 -- scrontab - On errors, print 1-index line numbers instead of 0-indexing them.
 -- openapi/dbv0.0.38 - Correct OpenAPI specification for diag request.
 -- openapi/dbv0.0.38 - reject requests with incorrect TRES specification.
 -- sacctmgr - reject requests with incorrect TRES specification.
 -- Correct issue where conversion to/from JSON/YAML may have resulted in empty
    strings being replaced with strings containing "null" instead.
 -- Attempt to requeue jobs terminated by slurm.conf changes (node vanish, node
    socket/core change, etc). Processes may still be running on excised nodes.
    Admin should take precautions when removing nodes that have jobs on running
    on them.
 -- Fix race with task/cgroup memory and jobacctgather/cgroup, the first was
    removing the pid from the task_X cgroup directory.
 -- openapi/dbv0.0.38 - set default account org and desc to be account name.
 -- openapi/dbv0.0.38 - Allow strings for JobId instead of only numerical JobId
    for GET job methods.
 -- Add switch/hpe_slingshot plugin.
 -- cloud_reg_addrs - use hostname from slurmd rather than from munge
    credential for NodeHostName.
 -- Store assoc usage in parent assoc when deleted to preserve account
    fairshare.
 -- CVE-2022-29500 - Prevent credential abuse.
 -- CVE-2022-29501 - Prevent abuse of REQUEST_FORWARD_DATA.
 -- CVE-2022-29502 - Correctly validate io keys.
 -- openapi/dbv0.0.38 - set with_deleted to false by default for /user[s].
 -- openapi/dbv0.0.38 - add with_deleted input parameter to GET /user[s].
 -- openapi/dbv0.0.38 - add deleted flag to /user[s] output.
 -- openapi/dbv0.0.38 - set with_deleted to false by default for GET /qos.
 -- openapi/dbv0.0.38 - add with_deleted input to GET /qos.
 -- openapi/dbv0.0.38 - set with_deleted to false by default for GET /account[s]
 -- openapi/dbv0.0.38 - add with_deleted input to GET /account[s].
 -- Include k12 hash of the RPC message body in the auth/munge tokens to
    provide for additional communication resiliency.
 -- Allow job steps to continue to launch during OverTimeLimit.
 -- slurmctld - avoid crash when attempting to load job without a script.
 -- Fix exclusive jobs breaking MaxCPUsPerNode limit
 -- Error if ConstrainSwapSpace is set and system doesn't have the kernel with
    CONFIG_MEMCG_SWAP set. This will avoid bad configurations.
 -- Add SlurmdParameters=numa_node_as_socket.
 -- Fix incorrect node allocation when using multiple counts with --constraint
    and --ntasks-per-node.

* Changes in Slurm 21.08.9
==========================
 -- Fix cross-endian communication.
 -- Fix slurmd segfault when seeing authentication issues alongside a failed
    message forward.
 -- Fix resuming nodes not part of an allocation.
 -- Fix "--gpu-bind=single:" having wrong env variables.
 -- Fix logic estimating when a job can start based on ANY_NODES / LICENSE_ONLY
    reservations that could lead to an infinite loop in backfill.
 -- Fix a segfault that may happen on gpu configured as no_consume.
 -- Fix regression which prevented a cons_tres gpu job to be submitted to a
    cons_tres cluster from a non-con_tres cluster.

* Changes in Slurm 21.08.8-2
============================
 -- Fix communication forwarding when running with a mix of patch and unpatched
    slurmd processes.

* Changes in Slurm 21.08.8
==========================
 -- openapi/dbv0.0.37 - fix slurmrestd fatal() when deleting an association.
 -- Allow scontrol update <job> Gres=... to not require "gres:".
 -- Fix inconsistent reboot message appending behavior.
 -- Fix incorrect reason_time and reason_uid on reboot message.
 -- Fix "scontrol reboot" clearing node reason on ResumeTimeout.
 -- Fix ResumeTimeout error message missing when node already has reason set.
 -- Avoid "running with local config" error when conf server is provided by DNS.
 -- openapi/v0.0.37 - resolve job user name when not sent by slurmctld.
 -- openapi/dbv0.0.37 - Correct OpenAPI specification for diag request.
 -- Ignore power_down request when node is already powering down.
 -- CVE-2022-29500 - Prevent credential abuse.
 -- CVE-2022-29501 - Prevent abuse of REQUEST_FORWARD_DATA.
 -- CVE-2022-29502 - Correctly validate io keys.

* Changes in Slurm 21.08.7
==========================
 -- openapi/v0.0.37 - correct calculation for bf_queue_len_mean in /diag.
 -- Optimize sending down nodes in maintenance mode to the database when
    removing reservations.
 -- Avoid shrinking a reservation when overlapping with downed nodes.
 -- Fix 'planned time' in rollups for jobs that were still pending when the
    rollup happened.
 -- Prevent new elements from a job array from causing rerollups.
 -- Only check TRES limits against current usage for TRES requested by the job.
 -- Do not allocate shared gres (MPS) in whole-node allocations
 -- Fix minor memory leak when dealing with configless setups.
 -- Constrain slurmstepd to job/step cgroup like in previous versions of Slurm.
 -- Fix warnings on 32-bit compilers related to printf() formats.
 -- Fix memory leak when freeing kill_job_msg_t.
 -- Fix memory leak when using data_t.
 -- Fix reconfigure issues after disabling/reenabling the GANG PreemptMode.
 -- Fix race condition where a cgroup was being deleted while another step
    was creating it.
 -- Set the slurmd port correctly if multi-slurmd
 -- openapi/v0.0.37 - Fix misspelling of account_gather_frequency in spec.
 -- openapi/v0.0.37 - Fix misspelling of cluster_constraint in spec.
 -- Fix FAIL mail not being sent if a job was cancelled due to preemption.
 -- slurmrestd - move debug logs for HTTP handling to be gated by debugflag
    NETWORK to avoid unnecessary logging of communication contents.
 -- Fix issue with bad memory access when shrinking running steps.
 -- Fix various issues with internal job accounting with GRES when jobs are
    shrunk.
 -- Fix ipmi polling on slurmd reconfig or restart.
 -- Fix srun crash when reserved ports are being used and het step fails
    to launch.
 -- openapi/dbv0.0.37 - fix DELETE execution path on /user/{user_name}.
 -- slurmctld - Properly requeue all components of a het job if PrologSlurmctld
    fails.
 -- rlimits - remove final calls to limit nofiles to 4096 but to instead use
    the max possible nofiles in slurmd and slurmdbd.
 -- Fix slurmctld memory leak after a reconfigure with configless.
 -- Fix slurmd memory leak when fetching configless files.
 -- Allow the DBD agent to load large messages (up to MAX_BUF_SIZE) from state.
 -- Fix minor memory leak with cleaning up the extern step.
 -- Fix potential deadlock during slurmctld restart when there is a completing
    job.
 -- slurmstepd - reduce user requested soft rlimits when they are above max
    hard rlimits to avoid rlimit request being completely ignored and
    processes using default limits.
 -- Fix memory leaks when job/step specifies a container.
 -- Fix Slurm user commands displaying available features as active features
    when no features were active.
 -- Don't power down nodes that are rebooting.
 -- Clear pending node reboot on power down request.
 -- Ignore node registrations while node is powering down.
 -- Don't reboot any node that is power<ing|ed> down.
 -- Don't allow a node to reboot if it's marked for power down.
 -- Fix issuing reboot and downing when rebooting a powering up node.
 -- Clear DRAIN on node after failing to resume before ResumeTimeout.
 -- Prevent repeating power down if node fails to resume before ResumeTimeout.
 -- Fix federated cloud node communication with srun and cloud_dns.
 -- Fix jobs being scheduled on nodes marked to be powered_down when idle.
 -- Fix problem where a privileged user could not view array tasks specified by
    <array_job_id>_<task_id> when PrivateData had the jobs value set.

* Changes in Slurm 21.08.6
==========================
 -- Handle typed shared GRES better in accounting.
 -- Fix plugin_name definitions in a number of plugins to improve logging.
 -- Close sbcast file transfers when job is cancelled.
 -- job_submit/lua - allow mail_type and mail_user fields to be modified.
 -- scrontab - fix handling of --gpus and --ntasks-per-gpu options.
 -- sched/backfill - fix job_queue_rec_t memory leak.
 -- Fix magnetic reservation logic in both main and backfill schedulers.
 -- job_container/tmpfs - fix memory leak when using InitScript.
 -- slurmrestd / openapi - fix memory leaks.
 -- Fix slurmctld segfault due to job array resv_list double free.
 -- Fix multi-reservation job testing logic.
 -- Fix slurmctld segfault due to insufficient job reservation parse validation.
 -- Fix main and backfill schedulers handling for already rejected job array.
 -- sched/backfill - restore resv_ptr after yielding locks.
 -- acct_gather_energy/xcc - appropriately close and destroy the IPMI context.
 -- Protect slurmstepd from making multiple calls to the cleanup logic.
 -- Prevent slurmstepd segfault at cleanup time in mpi_fini().
 -- Fix slurmctld sometimes hanging if shutdown while PrologSlurmctld or
    EpilogSlurmctld were running and PrologEpilogTimeout is set in slurm.conf.
 -- Fix affinity of the batch step if batch host is different than the first
    node in the allocation.
 -- slurmdbd - fix segfault after multiple failover/failback operations.
 -- Fix jobcomp filetxt job selection condition.
 -- Fix -f flag of sacct not being used.
 -- Select cores for job steps according to the socket distribution. Previously,
    sockets were always filled before selecting cores from the next socket.
 -- Keep node in Future state if epilog completes while in Future state.
 -- Fix erroneous --constraint behavior by preventing multiple sets of brackets.
 -- Make ResetAccrueTime update the job's accrue_time to now.
 -- Fix sattach initialization with configless mode.
 -- Revert packing limit checks affecting pmi2.
 -- sacct - fixed assertion failure when using -c option and a federation
    display
 -- Fix issue that allowed steps to overallocate the job's memory.
 -- Fix the sanity check mode of AutoDetect so that it actually works.
 -- Fix deallocated nodes that didn't actually launch a job from waiting for
    Epilogslurmctld to complete before clearing completing node's state.
 -- Job should be in a completing state if EpilogSlurmctld when being requeued.
 -- Fix job not being requeued properly if all node epilog's completed before
    EpilogSlurmctld finished.
 -- Keep job completing until EpilogSlurmctld is completed even when "downing"
    a node.
 -- Fix handling reboot with multiple job features.
 -- Fix nodes getting powered down when creating new partitions.
 -- Fix bad bit_realloc which potentially could lead to bad memory access.
 -- slurmctld - remove limit on the number of open files.
 -- Fix bug where job_state file of size above 2GB wasn't saved without any
    error message.
 -- Fix various issues with no_consume gres.
 -- Fix regression in 21.08.0rc1 where job steps failed to launch on systems
    that reserved a CPU in a cgroup outside of Slurm (for example, on systems
    with WekaIO).
 -- Fix OverTimeLimit not being reset on scontrol reconfigure when it is
    removed from slurm.conf.
 -- serializer/yaml - use dynamic buffer to allow creation of YAML outputs
    larger than 1MiB.
 -- Fix minor memory leak affecting openapi users at process termination.
 -- Fix batch jobs not resolving the username when nss_slurm is enabled.
 -- slurmrestd - Avoid slurmrestd ignoring invalid HTTP method if the response
    serialized without error.
 -- openapi/dbv0.0.37 - Correct conditional that caused the diag output to
    give an internal server error status on success.
 -- Make --mem-bind=sort work with task_affinity
 -- Fix sacctmgr to set MaxJobsAccruePer{User|Account} and MinPrioThres in
    sacctmgr add qos, modify already worked correctly.
 -- job_container/tmpfs - avoid printing extraneous error messages in Prolog
    and Epilog, and when the job completes.
 -- Fix step CPU memory allocation with --threads-per-core without --exact.
 -- Remove implicit --exact when --threads-per-core or --hint=nomultithread
    is used.
 -- Do not allow a step to request more threads per core than the
    allocation did.
 -- Remove implicit --exact when --cpus-per-task is used.

* Changes in Slurm 21.08.5
==========================
 -- Fix issue where typeless GRES node updates were not immediately reflected.
 -- Fix setting the default scrontab job working directory so that it's the home
    of the different user (-u <user>) and not that of root or SlurmUser editor.
 -- Fix stepd not respecting SlurmdSyslogDebug.
 -- Fix concurrency issue with squeue.
 -- Fix job start time not being reset after launch when job is packed onto
    already booting node.
 -- Fix updating SLURM_NODE_ALIASES for jobs packed onto powering up nodes.
 -- Cray - Fix issues with starting hetjobs.
 -- auth/jwks - Print fatal() message when jwks is configured but file could
    not be opened.
 -- If sacctmgr has an association with an unknown qos as the default qos
    print 'UNKN-###' instead of leaving a blank name.
 -- Correctly determine task count when giving --cpus-per-gpu, --gpus and
    --ntasks-per-node without task count.
 -- slurmctld - Fix places where the global last_job_update was not being set
    to the time of update when a job's reason and description were updated.
 -- slurmctld - Fix case where a job submitted with more than one partition
    would not have its reason updated while waiting to start.
 -- Fix memory leak in node feature rebooting.
 -- Fix time limit permanetly set to 1 minute by backfill for job array tasks
    higher than the first with QOS NoReserve flag and PreemptMode configured.
 -- Fix sacct -N to show jobs that started in the current second
 -- Fix issue on running steps where both SLURM_NTASKS_PER_TRES and
    SLURM_NTASKS_PER_GPU are set.
 -- Handle oversubscription request correctly when also requesting
    --ntasks-per-tres.
 -- Correctly detect when a step requests bad gres inside an allocation.
 -- slurmstepd - Correct possible deadlock when UnkillableStepTimeout triggers.
 -- srun - use maximum number of open files while handling job I/O.
 -- Fix writing to Xauthority files on root_squash NFS exports, which was
    preventing X11 forwarding from completing setup.
 -- Fix regression in 21.08.0rc1 that broke --gres=none.
 -- Fix srun --cpus-per-task and --threads-per-core not implicitly setting
    --exact. It was meant to work this way in 21.08.
 -- Fix regression in 21.08.0 that broke dynamic future nodes.
 -- Fix dynamic future nodes remembering active state on restart.
 -- Fix powered down nodes getting stuck in COMPLETING+POWERED_DOWN when job is
    cancelled before nodes are powering up.

* Changes in Slurm 21.08.4
==========================
 -- Fix potential deadlock when using PMI v1.
 -- Fix tight loop sending DBD_SEND_MULT_JOB_START when the slurmctld has an
    issue talking correctly to the DBD.
 -- Fix memory leak in step creation.
 -- Fix potential deadlock when shutting down slurmctld.
 -- Fix regression in 21.08 where multi-node steps that requested MemPerCPU
    were not counted against the job's memory allocation on some nodes.
 -- Fix issue with select/cons_tres and the partition limit MaxCpusPerNode where
    the limit was enforced for one less CPU than the configured value.
 -- jobacct_gather/common - compare Pss to Rss after scaling Pss to Rss units.
 -- Fix SLURM_NODE_ALIASES in RPC Prolog for batch jobs.
 -- Fix regression in 21.08 where slurmd and slurmstepd were not constrained
    with CpuSpecList or CoreSpecCount.
 -- Fix cloud jobs running without powering up nodes after a reconfig/restart.
 -- CVE-2021-43337 - Fix security issue with new AccountingStoreFlags=job_script
    and job_env options where users could request scripts and environments they
    should not have been permitted to access.

* Changes in Slurm 21.08.3
==========================
 -- Return error to sacctmgr when running 'sacctmgr archive load' and the load
    fails due to an invalid or corrupted file.
 -- slurmctld/gres_ctld - fix deallocation of typed GRES without device.
 -- scrontab - fix capturing the cronspec request in the job script.
 -- openapi/dbv0.0.37 - Add missing method POST for /associations/.
 -- If ALTER TABLE was already run, continue with database upgrade.
 -- slurmstepd - Gracefully handle RunTimeQuery returning no output.
 -- srun - automatically handle issues with races to listen() on an ephemeral
    socket, and suppress otherwise needless error messages.
 -- Schedule sooner after Epilog completion with SchedulerParameters=defer.
 -- Improve performance for AccountingStoreFlags=job_env.
 -- Expose missing SLURMD_NODENAME and SLURM_NODEID to TaskEpilog environment.
 -- Bring slurm_completion.sh up to date with changes to commands.
 -- Fix issue where burst buffer stage-in could only start for one job in a job
    array per scheduling cycle instead of bb_array_stage_cnt jobs per scheduling
    cycle.
 -- Fix checking if the dependency is the same job for array jobs.
 -- Fix checking for circular dependencies with job arrays.
 -- Restore dependent job pointers on slurmctld startup to avoid race.
 -- openapi/v0.0.37 - Allow strings for JobIds instead of only numerical JobIds
    for GET, DELETE, and POST job methods.
 -- openapi/dbv0.0.36 - Gracefully handle missing associations.
 -- openapi/dbv0.0.36 - Avoid restricting job association lookups to only
    default associations.
 -- openapi/dbv0.0.37 - Gracefully handle missing associations.
 -- openapi/dbv0.0.37 - Avoid restricting job association lookups to only
    default associations.
 -- Fix error in GPU frequency validation logic.
 -- Fix regression in 21.08.1 that broke federated jobs.
 -- Correctly handle requested GRES when used in job arrays.
 -- Fix error in pmix logic dealing with the incorrect size of buffer.
 -- Fix handling of no_consume GRES, add it to allocated job allocated TRES.
 -- Fix issue with typed GRES without Files= (bitmap).
 -- Fix job_submit/lua support for 'gres' which is now stored as a 'tres'
    when requesting jobs so needs a 'gres' prefix.
 -- Fix regression where MPS would not deallocate from the node properly.
 -- Fix --gpu-bind=verbose to work correctly.
 -- Do not deny --constraint with special operators "[]()|*" when no changeable
    features are requested, but continue to deny --constraint with special
    operators when changeable features are requested.
 -- openapi/v0.0.{35,36,37} - prevent merging the slurmrestd environment
    alongside a new job submission.
 -- openapi/dbv0.0.36 - Correct tree position of dbv0.0.36_job_step.
 -- openapi/dbv0.0.37 - Correct tree position of dbv0.0.37_job_step.
 -- openapi/v0.0.37 - enable job priority field for job submissions and updates.
 -- openapi/v0.0.37 - request node states query includes MIXED state instead of
    only allocated.
 -- mpi/pmix - avoid job hanging until the time limit on PMIx agent failures.
 -- Correct inverted logic where reduced version matching applied to non-SPANK
    plugins where it should have only applied to SPANK plugins.
 -- Fix issues where prologs would run in serial without PrologFlags=serial.
 -- Make sure a job coming in is initially considered for magnetic reservations.
 -- PMIx v1.1.4 and below are no longer supported.
 -- Add comment to service files about disabling logging through journald.
 -- Add SLURM_NODE_ALIASES env to RPC Prolog (PrologFlags=alloc) environment.
 -- Limit max_script_size to 512 MB.
 -- Fix shutdown of slurmdbd plugin to correctly notice when the agent thread
    finishes.
 -- slurmdbd - fix issue with larger batch script files being sent to SlurmDBD
    with AccountingStoreFlags=job_script that can lead to accounting data loss
    as the resulting RPC generated can exceed internal limits and won't be
    sent, preventing further communication with SlurmDBD.
    This issue is indicated by "error: Invalid msg_size" in your log files.
 -- Fix compile issue with --without-shared-libslurm.

* Changes in Slurm 21.08.2
==========================
 -- slurmctld - fix how the max number of cores on a node in a partition are
    calculated when the partition contains multi-socket nodes. This in turn
    corrects certain jobs node count estimations displayed client-side.
 -- job_submit/cray_aries - fix "craynetwork" GRES specification after changes
    introduced in 21.08.0rc1 that made TRES always have a type prefix.
 -- Ignore nonsensical check in the slurmd for [Pro|Epi]logSlurmctld.
 -- Fix writing to stderr/syslog when systemd runs slurmctld in the foreground.
 -- Fix locking around log level setting routines.
 -- Fix issue with updating job started with node range.
 -- Fix issue with nodes not clearing state in the database when the slurmctld
    is started with clean-start.
 -- Fix hetjob components > 1 timing out due to InactiveLimit.
 -- Fix sprio printing -nan for normalized association priority if
    PriorityWeightAssoc was not defined.
 -- Disallow FirstJobId=0.
 -- Preserve job start info in the database for a requeued job that hadn't
    registered the first time in the database yet.
 -- Only send one message on prolog failure from the slurmd.
 -- Remove support for TaskAffinity=yes in cgroup.conf.
 -- accounting_storage/mysql - fix issue where querying jobs via sacct
    --whole-hetjob=yes or slurmrestd (which automatically includes this flag)
    could in some cases return more records than expected.
 -- Fix issue for preemption of job array task that makes afterok dependency
    fail. Additionally, send emails when requeueing happens due to preemption.
 -- Fix sending requeue mail type.
 -- Properly resize a job's GRES bitmaps and counts when resizing the job.
 -- Fix node being able to transition to CLOUD state from non-cloud state.
 -- Fix regression introduced in 21.08.0rc1 which broke a step's ability to
    inherit GRES from the job when the step didn't request GRES but the job did.
 -- Fix errors in logic when picking nodes based on bracketed anded constraints.
    This also enforces the requirement to have a count when using such
    constraints.
 -- Handle job resize better in the database.
 -- Exclude currently running, resized jobs from the runaway jobs list.
 -- Make it possible to shrink a job more than once.

* Changes in Slurm 21.08.1
==========================
 -- Fix potential memory leak if a problem happens while allocating GRES for
    a job.
 -- If an overallocation of GRES happens terminate the creation of a job.
 -- AutoDetect=nvml: Fatal if no devices found in MIG mode.
 -- slurm.spec - fix querying for PMIx and UCX version.
 -- Print federation and cluster sacctmgr error messages to stderr.
 -- Fix off by one error in --gpu-bind=mask_gpu.
 -- Fix statement condition in http_parser autoconf macro.
 -- Fix statement condition in netloc autoconf macro.
 -- Add --gpu-bind=none to disable gpu binding when using --gpus-per-task.
 -- Handle the burst buffer state "alloc-revoke" which previously would not
    display in the job correctly.
 -- Fix issue in the slurmstepd SPANK prolog/epilog handler where configuration
    values were used before being initialized.
 -- Restore a step's ability to utilize all of an allocations memory if --mem=0.
 -- Fix --cpu-bind=verbose garbage taskid.
 -- Fix cgroup task affinity issues from garbage taskid info.
 -- Make gres_job_state_validate() client logging behavior as before 44466a4641.
 -- Fix steps with --hint overriding an allocation with --threads-per-core.
 -- Require requesting a GPU if --mem-per-gpu is requested.
 -- Return error early if a job is requesting --ntasks-per-gpu and no gpus or
    task count.
 -- Properly clear out pending step if unavailable to run with available
    resources.
 -- Kill all processes spawned by burst_buffer.lua including decendents.
 -- openapi/v0.0.{35,36,37} - Avoid setting default values of min_cpus,
    job name, cwd, mail_type, and contiguous on job update.
 -- openapi/v0.0.{35,36,37} - Clear user hold on job update if hold=false.
 -- Prevent CRON_JOB flag from being cleared when loading job state.
 -- sacctmgr - Fix deleting WCKeys when not specifying a cluster.
 -- Fix getting memory for a step when the first node in the step isn't the
    first node in the allocation.
 -- Make SelectTypeParameters=CR_Core_Memory default for cons_tres and cons_res.
 -- Correctly handle mutex unlocks in the gres code if failures happen.
 -- Give better error message if -m plane is given with no size.
 -- Fix --distribution=arbitrary for salloc.
 -- Fix jobcomp/script regression introduced in 21.08.0rc1 0c75b9ac9d.
 -- Only send the batch node in the step_hostlist in the job credential.
 -- When setting affinity for the batch step don't assume the batch host is node
    0.
 -- In task/affinity better checking for node existence when laying out
    affinity.
 -- slurmrestd - fix job submission with auth/jwt.
 -- Fix 'scontrol setdebug' not respecting slurm.conf logging configurations.

* Changes in Slurm 21.08.0
==========================
 -- Restored --gpu-bind=single:<ntasks> to check core affinity like
    --gpu-bind=closest does. This removal of this behavior only was in rc2.
 -- slurmd - Fix assert failure on initialization due to bad node name.
 -- Fix error codes in cgroup/v1.
 -- Don't destroy the memory step outside fini, which leads to a double destroy
    causing an error message.
 -- Add support for lua 5.4.
 -- Force cgroup.clone_children to 0 in slurm cgroup directories. This caused
    issues in task cpuset plugin in systems with it enabled by default.
 -- Clear GRES HAS_TYPE flag when removing type name.
 -- Environment flags in gres.conf now override flags set by AutoDetect.
 -- Environment flags in gres.conf now apply to subsequent gres.conf lines where
    Environment flags are not set.
 -- Set missing job_uid and job_gid members when preparing a kill_job_msg_t in
    abort_job_on_node(), abort_job_on_nodes() and kill_job_on_node().
 -- Fix swappiness not being set in cgroups.
 -- Fix coordinators for new subaccounts.
 -- Fix coordinators when adding existing users with PrivateData=users.
 -- slurmctld - do not attempt to relinquish control to self.
 -- openapi/v0.0.37 - Honor kill_on_invalid_dependency as job parameter.
 -- Check max_gres when doing step allocation, fix for regression in rc2.
 -- SPANK plugins are now required to match the current Slurm version, and must
    be recompiled for each new Slurm release.
 -- node_features/helpers - add ExecTime configuration option.
 -- srun - Fix force termination with -X.
 -- On slurmctld restart set node typed GRES counts correctly.
 -- Fix places where a step wasn't allocated in the slurmctld but wasn't ever
    removed from the job.
 -- Fix step allocation memory when using --threads-per-core.
 -- Fix step allocations to consume all threads on a core when using
    threads-per-core.
 -- Add check to validate cpu request on a step if --threads-per-core is given
    and it is less than what the core on the node has in the allocation.
 -- Fix issue where a step could request more gres than the job had and the step
    would hang forever. This bug was only introduced in 21.08.0rc2.
 -- Only print \r\n for logging messages on stderr when --pty has been
    explicitly requested.
 -- Relax check on SPANK plugins to only require Slurm major + minor versions
    to match.
 -- job_container/tmpfs - delegate handling of /dev/shm to the extern step
    so new step launches will be attached correctly even after the slurmd
    process has been restarted.
 -- Limit the wait time in proctrack_g_wait() to UnkillableStepTimeout instead
    of a hardcoded value of 256 seconds, and limit the delay between tests to a
    maximum of 32 seconds.
 -- fatal() on start if using job_container/tmpfs without PrologFlags=Contian.

* Changes in Slurm 21.08.0rc2
=============================
 -- Load bf_when_last_cycle from job state only if protocol version >= 21.08.
 -- Docs - remove man3 section entirely.
 -- Enable OCI container execution support as a technical preview.
 -- Set step memory when using MemPerGPU or DefMemPerGPU. Previously a step's
    memory was not set even when it requested --mem-per-gpu and at least one
    GPU.
 -- Add cli_filter.lua support in configless mode.
 -- Remove some of the MAX macro usages with return codes.
 -- Check that the step requests at least as many gres as nodes.
 -- sacct - Add --json and --yaml arguments.
 -- squeue - Add --json and --yaml arguments.
 -- sinfo - Add --json and --yaml arguments.
 -- Make job's SLURM_JOB_GPUS print global GPU IDs instead of MIG unique_ids.
 -- Fix miscounting of GPU envs in prolog/epilog if MultipleFiles was used.
 -- Support MIGs in prolog/epilog's CUDA_VISIBLE_DEVICES & co.
 -- Add SLURM_JOB_GPUS back into Prolog; add it to Epilog.
 -- Fix issue where the original executable, not the bcast'd version, was
    executed with 'srun --bcast'.
 -- Fix memory leaks when cleaning up config_response_msg_t.
 -- sacct - print '-' header correctly for fields over 53-characters wide.
 -- openapi/dbv0.0.37 - replace "REST" with "Slurm OpenAPI" for plugin_name.
 -- openapi/v0.0.37 - replace "REST" with "Slurm OpenAPI" for plugin_name.
 -- configless - fix segfault on 'scontrol reconfigure'.
 -- Fix bad reference to step_ptr->job_ptr after step_ptr is freed.
 -- Fix bad reference to config_for_clients when shutting down slurmctld.
 -- Fix memory leak on gres_list_alloc when freeing steps.
 -- Fix memory leak on kill_job_msg_t->work_dir when freeing struct.
 -- Fix memory leak on slurmdb_step_rec_t->submit_line when freeing struct.
 -- Fix memory leak in the slurmdbd when requesting wckeys from all clusters.
 -- Use FREE_NULL_LIST instead of list_destroy.
 -- Fix bad pointer reference after bad unpack of jobacctinfo_t.
 -- If we made are running an interactive session we need to force track_steps.
 -- Disable OPOST flag when using --pty to avoid issues with Emac.
 -- Fix issue where extra bonus core was allocated in some situations.
 -- Avoid putting gres with count of 0 on a TRES req/alloc.
 -- Fix determining if a reservation is used or not.
 -- Fix memory in requested TRES when --mem-per-gpu is used.
 -- Changed ReqMem field in sacct to match memory from ReqTRES.
 -- Changed --gpu-bind=single:<ntasks> to no longer check core affinity like
    --gpu-bind=closest does. This consequently affects --ntasks-per-gpu.

* Changes in Slurm 21.08.0rc1
=============================
 -- slurmrestd - add v0.0.37 OpenAPI plugin.
 -- slurmrestd/v0.0.37 - rename standard_in -> standard_input.
 -- slurmrestd/v0.0.37 - rename standard_out -> standard_output.
 -- slurmdbd - Improve log messages of more time than possible on rollups.
 -- Changed the --format handling for negative field widths (left justified)
    to apply to the column headers as well as the printed fields.
 -- Add LimitFactor to the QOS. A float that is factored into an associations
    [Grp|Max]TRES limits.  For example, if the LimitFactor is 2, then an
    association with a GrpTRES of 30 CPUs, would be allowed to allocate 60
    CPUs when running under this QOS.
 -- slurmrestd - Pass SLURM_NO_CHANGE_IN_DATA to client as 403 (Not Modified).
 -- slurmrestd/v0.0.37 - Add update_time field to Jobs query to allow clients
    to only get jobs list based on change timestamp.
 -- Reset job eligible time when job is manually held.
 -- Add DEBUG_FLAG_JAG to improve logging related to job account gathering.
 -- Convert logging in account_gather/common to DEBUG_FLAG_JAG.
 -- Add more logging for jag_common_poll_data() when prec_extra() called.
 -- slurmrestd/v0.0.37 - add API to fetch reservation(s) info.
 -- Catch more errors in task/cgroup initalization and cleanup to avoid allowing
    jobs to start when cgroups failure to configure correctly.
 -- Fix cgroup ns detection when using containers (e.g. LXC or Docker).
 -- Reset job's next_step_id counter to 0 after being requeued.
 -- Make scontrol exit with non-zero status after failing to delete a partition
    or reservation.
 -- Make NtasksPerTRES optional in slurm_sprint_job_info().
 -- slurmrestd/v0.0.37 - Add update_time field to nodes query to allow clients
    to only get nodes list based on change timestamp.
 -- common/parse_config - catch and propagate return codes when handling a match
    on a key-value pattern. This implies error codes detected in the handlers
    are now not ignored and users of _handle_keyvalue_match() can fatal().
 -- common/hostlist - fix hostlist_delete_nth() xassert() upper bound check.
 -- API change: Removed slurm_kill_job_msg and modified the function signature
    for slurm_kill_job2. slurm_kill_job2 should be used instead of
    slurm_kill_job_msg.
 -- Fix non-zero exit code for scontrol ping when all controllers are down.
 -- Enforce a valid configuration for AccountingStorageEnforce in slurm.conf.
    If the configuration is invalid, then an error message will be printed and
    the command or daemon (including slurmctld) will not run.
 -- slurmrestd/v0.0.37 - Add update_time field to partitions/reservations query
    to allow clients to only get the entities list when something changed.
 -- slurmdbd.service - add "After" relationship to all common names for MariaDB
    to reduce startup delays.
 -- slurmrestd/v0.0.37 - Correct displaying node states that are UNKNOWN.
 -- slurmrestd/v0.0.37 - Add flags to node states.
 -- Fix first job on fresh cluster not being assigned JobId=1 (or FirstJobId).
 -- squeue - make it so --nodelist is sensitive to --clusters.
 -- squeue - do --nodelist node validation in the same order as listing.
 -- Removed AccountingStoreJobComment option.  Please update your config to use
    AccountingStoreFlags=job_comment instead.
 -- AccountingStoreFlags=job_script allows you to store the job's batch script.
 -- AccountingStoreFlags=job_env allows you to store the job's env vars.
 -- Add sacct -o SubmitLine to get the submit line of a job/step.
 -- Removed DefaultStorage{Host,Loc,Pass,Port,Type,User} options.
 -- Fix NtasksPerTRES delimiter from : to = in scontrol show job output.
 -- Removed CacheGroups, CheckpointType, JobCheckpointDir, MemLimitEnforce,
    SchedulerPort, SchedulerRootFilter options.
 -- Make job accounting queries use consistent timeframes with and w/o jobs.
 -- --cpus-per-task and --threads-per-core now imply --exact.
    This fixes issues where steps would be allocated the wrong number of CPUs.
 -- configure: the --with option handling has been made consistent across the
    various optional libraries. Specifying --with-foo=/path/to/foo will only
    check that directory for the applicable library (rather than, in some cases,
    falling back to the default directories), and will always error the build
    if the library is not found (instead of a mix of error messages and non-
    fatal warning messages).
 -- configure: replace --with-rmsi_dir option with proper handling for
    --with-rsmi=dir.
 -- Pass additional job environment variables to MailProg.
 -- Add SLURM_JOB_WORK_DIR to Prolog, Epilog.
 -- Removed sched/hold plugin.
 -- Fix srun overwriting SLURM_SUBMIT_DIR and SLURM_SUBMIT_HOST when within an
    existing allocation.
 -- step_ctx code has been removed from the api.
 -- cli_filter/lua, jobcomp/lua, job_submit/lua now load their scripts from the
    same directory as the slurm.conf file (and thus now will respect changes
    to the SLURM_CONF environment variable).
 -- SPANK - call slurm_spank_init if defined without slurm_spank_slurmd_exit in
    slurmd context.
 -- job_container/tmpfs - Remove need for .active file to allow salloc without
    an interactive step to work.
 -- Add new 'PLANNED' state to a node to represent when the backfill scheduler
    has it planned to be used in the future instead of showing as 'IDLE'.
 -- slurmd - Delay background node registration on every failure up to 128s on
    startup.
 -- slurmctld - Always notify slurmd that node registration was accepted to
    avoid slurmd needless attempting to re-register if there is configuration
    issue.
 -- Put node into "INVAL" state upon registering with an invalid node
    configuration. Node must register with a valid configuration to continue.
 -- Make --cpu-bind=threads default for --threads-per-core -- cli and env can
    override.
 -- jobcomp/elasticsearch - Use data_t to serialize data. The plugin now has the
    JSON-C library as a prerequisite.
 -- scrontab - create the temporary file under the TMPDIR environment variable
    (if set), otherwise continue to use TmpFS as configured in slurm.conf.
 -- Add LastBusyTime to "scontrol show nodes" and slurmrestd nodes output,
    which represents the time the node last had jobs on it.
 -- slurmd - allow multiple comma-separated controllers to be specified in
    configless mode with --conf-server
 -- sacctmgr - changed column headings to "ParentID" and "ParentName" instead
    of "Par ID" and "Par Name" respectively.
 -- Perl API - make sure man pages are installed under the --prefix given to
    configure.
 -- Manually powering down of nodes with scontrol now ignores
    SuspendExc<Nodes|Parts>.
 -- SALLOC_THREADS_PER_CORE and SBATCH_THREADS_PER_CORE have been added as
    input environment variables for salloc and sbatch, respectively. They do
    the same thing as --threads-per-core.
 -- Distinguish queued reboot requests (REBOOT) from issued reboots (REBOOT^).
 -- Set the maximum number of open files per process to 4096 to avoid
    performance issues when closing the entire range with closeall().
 -- auth/jwt - add support for RS256 tokens.
 -- Relax reservation purge due to any invalid uid after creation time.
 -- Reject srun that requests both --exclusive and --overlap.
 -- service files - change dependency to network-online rather than just
    network to ensure DNS and other services are available.
 -- RSMI: Fix incorrect PCI BDF bits.
 -- plugins/cli_filter - Convert to using data_t to serialize JSON.
 -- Fix testing array job after regaining locks in backfill.
 -- Don't display node's comment with "scontrol show nodes" unless set.
 -- Add "Extra" field to node to store extra information other than a comment.
 -- scrontab - Use /tmp instead of TmpFS if TMPDIR is not set.
 -- Add ResumeTimeout, SuspendTimeout and SuspendTime to Partitions.
 -- sreport - change to sorting TopUsage by the --tres option.
 -- slurmrestd - do not run allow operation as SlurmUser/root by default.
 -- Allow map_cpu and mask_cpu for non-whole node allocation.
 -- TaskPluginParam=verbose is now treated as a default. Previously it would be
    applied regardless of the job specifying a --cpu-bind.
 -- Add "node_reg_mem_percent" SlurmctldParameter to define percentage of
    memory nodes are allowed to register with.
 -- Show correct number of SocketsPerBoard in slurmd -C with hwloc2.
 -- Alter sreport's cluster utilization report column name from
    'Reserved' to 'Planned' to match the nomenclature of the 'Planned' node.
 -- Add StateComplete format option to sinfo to show base_state+flags.
 -- "scontrol show node" now shows State as base_state+flags instead of
    shortened state with flags appended. eg. IDLE# -> IDLE+POWERING_UP.
    Also "POWER" state flag string is "POWERED_DOWN".
 -- slurmd/req - add missing job_env_t's het_job_id initialization off the
    request in _rpc_{abort,terminate}_job(). This caused problems for Native
    Cray builds when joining a CNCU job_container plugin with Epilog configured.
 -- Fix joining a CNCU job_container on a Native Cray build before executing the
    UnkillableStepProgram for a HetJob step.
 -- slurmrestd/v0.0.35 - Plugin has been tagged as deprecated.
 -- srun - Job steps requiring more cores than available to be rejected unless
    '--overlap' is specificed.