NEWS

This file describes changes in recent versions of SLURM. It primarily
documents those changes that are of interest to users and admins.

* Changes in SLURM 1.4.0-pre10
==============================
 -- Critical bug fix in task/affinity when the CoresPerSocket is greater
    than the ThreadsPerCore (invalid memory reference).
 -- Add DebugFlag parameter of "Wiki" to log sched/wiki and wiki2 
    communications in greater detail.
 -- Add "-d <slurmstepd_path>" as an option to the slurmd daemon to
    specifying a non-stardard slurmstepd file, used  for testing purposes.
 -- Minor cleanup to crypto/munge plugin.
    - Restrict uid allowed to decode job credentials in crypto/munge
    - Get slurm user id early in crypto/munge
    - Remove buggy error code handling in crypto/munge
 -- Added sprio command - works only with the priority/multifactor plugin
 -- Add real topology plugin infrastructure (it was initially added 
    directly into slurmctld code). To specify topology information,
    set TopologyType=topology/tree and add configuration information
    to a new file called topology.conf. See "man topology.conf" or
    topology.html web page for details.
 -- Set "/proc/self/oom_adj" for slurmd and slurmstepd daemons based upon
    the values of SLURMD_OOM_ADJ and SLURMSTEPD_OOM_ADJ environment 
    variables. This can be used to prevent daemons being killed when
    a node's memory is exhausted. Based upon patch by Hongjia  Cao, NUDT.
 -- Fix several bugs with respect to cpuset logic in task/affinity plugin.

* Changes in SLURM 1.4.0-pre9
=============================
 -- OpenMPI users only: Add srun logic to automatically recreate and 
    re-launch a job step if the step fails with a reserved port conflict.
 -- Added TopologyPlugin configuration parameter.
 -- Added switch topology data structure to slurmctld (for use by select 
    plugin) add load it based upon new slurm.conf parameters: SwitchName, 
    Nodes, Switches and LinkSpeed.
 -- Modify select/linear and select/cons_res plugins to optimize resource
    allocation with respect to network topology.
 -- Added  support for new configuration parameter EpilogSlurmctld (executed 
    by slurmctld daemon).
 -- Added checkpoint/blcr plugin, SLURM now support job checkpoint/restart 
    using BLCR. Patch from Hongjia Cao, NUDT, China.
 -- Made a variety of new environment variables available to PrologSlurmctld
    and EpilogSlurmctld. See the "Prolog and Epilog Scripts" section of the 
    slurm.conf man page for details.
 -- NOTE: Cold-start (without preserving state) required for upgrade from 
    version 1.4.0-pre8.

* Changes in SLURM 1.4.0-pre8
=============================
 -- In order to create a new partition using the scontrol command, use
    the "create" option rather than "update" (which will only operate
    upon partitions that already exist).
 -- Added environment variable SLURM_RESTART_COUNT to batch jobs to
    indicated the count of job restarts made.
 -- Added sacctmgr command "show config".
 -- Added the scancel option --nodelist to cancel any jobs running on a
    given list of nodes.
 -- Add partition-specific DefaultTime (default time limit for jobs, 
    if not specified use MaxTime for the partition. Patch from Par
    Andersson, National Supercomputer Centre, Sweden.
 -- Add support for the scontrol command to be able change the Weight
    associated with nodes. Patch from Krishnakumar Ravi[KK] (HP).
 -- Add DebugFlag configuration option of "CPU_Bind" for detailed CPU
    binding information to be logged.
 -- Fix some significant bugs in task binding logic (possible infinite loops
    and memory corruption).
 -- Add new node state flag of NODE_STATE_MAINT indicating the node is in
    a reservation of type MAINT.
 -- Modified task/affinity plugin to automatically bind tasks to sockets,
    cores, or threads as appropriated based upon resource allocation and
    task count. User can override with srun's --cpu_bind option. 
 -- Fix bug in backfill logic for select/cons_res plugin, resulted in 
    error "cons_res:_rm_job_from_res: node_state mis-count".
 -- Add logic go bind a batch job to the resources allocated to that job.
 -- Add configuration parameter MpiParams for (future) OpenMPI port 
    management. Add resv_port_cnt and resv_ports fields to the job step 
    data structures. Add environment variable SLURM_STEP_RESV_PORTS to
    show what ports are reserved for a job step.
 -- Add support for SchedulerParameters=interval=<sec> to control the time
    interval between executions of the backfill scheduler logic.
 -- Preserve record of last job ID in use even when doing a cold-start unless
    there is no job state file or there is a change in its format (which only 
    happens when there is a change in SLURM's major or minor version number: 
    v1.3 -> v1.4).
 -- Added new configuration parameter KillOnBadExit to kill a job step as soon
    as any task of a job step exits with a non-zero exit code. Patch based
    on work from Eric Lin, Bull.
 -- Add spank plugin calls for use by salloc and sbatch command, see 
    "man spank" for details.
 -- NOTE: Cold-start (without preserving state) required for upgrade from 
    version 1.4.0-pre7.

* Changes in SLURM 1.4.0-pre7
=============================
 -- Bug fix for preemption with select/cons_res when there are no idle nodes.
 -- Bug fix for use of srun options --exclusive and --cpus-per-task together
    for job step resource allocation (tracking of cpus in use was bad).
 -- Added the srun option --preserve-env to pass the current values of 
    environment variables SLURM_NNODES and SLURM_NPROCS through to the 
    executable, rather than computing them from commandline parameters.
 -- For select/cons_res or sched/gang only: Validate a job's resource 
    allocation socket and core count on each allocated node. If the node's
    configuration has been changed, then abort the job.
 -- For select/cons_res or sched/gang only: Disable updating a node's 
    processor count if FastSchedule=0. Administrators must set a valid
    processor count although the memory and disk space configuration can
    be loaded from the compute node when it starts.
 -- Add configure option "--disable-iso8601" to disable SLURM use of ISO 8601
    time format at the time of SLURM build. Default output for all commands
    is now ISO 8601 (yyyy-mm-ddThh:mm:ss).
 -- Add support for scontrol to explicity power a node up or down using the
    configured SuspendProg and ResumeProg programs.
 -- Fix book select/cons_res logic for tracking the number of allocated
    CPUs on a node when a partition's Shared value is YES or FORCE.
 -- Added configure options "--enable-cray-xt" and "--with-apbasil=PATH" for
    eventual support of Cray-XT systems.

* Changes in SLURM 1.4.0-pre6
=============================
 -- Fix job preemption when sched/gang and select/linear are configured with
    non-sharing partitions.
 -- In select/cons_res insure that required nodes have available resources.

* Changes in SLURM 1.4.0-pre5
=============================
 -- Correction in setting of SLURM_CPU_BIND environment variable.
 -- Rebuild slurmctld's job select_jobinfo->node_bitmap on restart/reconfigure
    of the daemon rather than restoring the bitmap since the nodes in a system
    can change (be added or removed).
 -- Add configuration option "--with-cpusetdir=PATH" for non-standard 
    locations.
 -- Get new multi-core data structures working on BlueGene systems.
 -- Modify PMI_Get_clique_ranks() to return an array of integers rather 
    than a char * to satisfy PMI standard. Correct logic in 
    PMI_Get_clique_size() for when srun --overcommit option is used.
 -- Fix bug in select/cons_res, allocated a job all of the processors on a 
    node when the --exclusive option is specified as a job submit option.
 -- Add NUMA cpu_bind support to the task affinity plugin. Binds tasks to
    a set of CPUs that belong NUMA locality domain with the appropriate
    --cpu-bind option (ldoms, rank_ldom, map_ldom, and mask_ldom), see
    "man srun" for more information.

* Changes in SLURM 1.4.0-pre4
=============================
 -- For task/affinity, force jobs to use a particular task binding by setting
    the TaskPluginParam configuration parameter rather than slurmd's
    SLURM_ENFORCED_CPU_BIND environment variable.
 -- Enable full preemption of jobs by partition with select/cons_res 
    (cons_res_preempt.patch from Chris Holmes, HP).
 -- Add configuration parameter DebugFlags to provide detailed logging for
    specific subsystems (steps and triggers so far).
 -- srun's --no-kill option is passed to slurmctld so that a job step is 
    killed even if the node where srun executes goes down (unless the 
    --no-kill option is used, previous termination logic would fail if 
    srun was not responding).
 -- Transfer a job step's core bitmap from the slurmctld to the slurmd
    within the job step credential.
 -- Add cpu_bind, cpu_bind_type, mem_bind and mem_bind_type to job allocation
    request and job_details structure in slurmctld. Add support to --cpu_bind
    and --mem_bind options from salloc and sbatch commands.

* Changes in SLURM 1.4.0-pre3
=============================
 -- Internal changes: CPUs per node changed from 32-bit to 16-bit size.
    Node count fields changed from 16-bit to 32-bit size in some structures.
 -- Remove select plugin functions select_p_get_extra_jobinfo(),
    select_p_step_begin() and select_p_step_fini().
 -- Remove the following slurmctld job structure fields: num_cpu_groups,
    cpus_per_node, cpu_count_reps, alloc_lps_cnt, alloc_lps, and used_lps.
    Use equivalent fields in new "select_job" structure, which is filled
    in by the select plugins.
 -- Modify mem_per_task in job step request from 16-bit to 32-bit size.
    Use new "select_job" structure for the job step's memory management.
 -- Add core_bitmap_job to slurmctld's job step structure to identify
    which specific cores are allocated to the step.
 -- Add new configuration option OverTimeLimit to permit jobs to exceed 
    their (soft) time limit by a configurable amount. Backfill scheduling
    will be based upon the soft time limit.
 -- Remove select_g_get_job_cores(). That data is now within the slurmctld's
    job structure.

* Changes in SLURM 1.4.0-pre2
=============================
 -- Remove srun's --ctrl-comm-ifhn-addr option (for PMI/MPICH2). It is no
    longer needed.
 -- Modify power save mode so that nodes can be powered off when idle. See
    https://computing.llnl.gov/linux/slurm/power_save.html or 
    "man slurm.conf" (SuspendProgram and related parameters) for more 
    information.
 -- Added configuration parameter PrologSlurmctld, which can be used to boot
    nodes into a particular state for each job. See "man slurm.conf" for 
    details.
 -- Add configuration parameter CompleteTime to control how long to wait for 
    a job's completion before allocating already released resources to pending
    jobs. This can be used to reduce fragmentation of resources. See
    "man slurm.conf" for details.
 -- Make default CryptoType=crypto/munge. OpenSSL is now completely optional.
 -- Make default AuthType=auth/munge rather than auth/none.
 -- Change output format of "sinfo -R" from "%35R %N" to "%50R %N".