NEWS

This file describes changes in recent versions of SLURM. It primarily
documents those changes that are of interest to users and admins.

* Changes in SLURM 1.3.2
========================
 -- Get --ntasks-per-node option working for sbatch command.
 -- BLUEGENE: Added logic to give back a best block on overlapped mode 
    in test_only mode
 -- BLUEGENE: Updated debug info and man pages for better help with the 
    numpsets option and to fail correctly with bad image request for building
    blocks.
 -- In sched/wiki and sched/wiki2 properly support Slurm license consumption
    (job state reported as "Hold" when required licenses are not available).
 -- In sched/wiki2 JobWillRun command, don't return an error code if the job(s)
    can not be started at that time. Just return an error message (from 
    Doug Wightman, CRI).
 -- Fix bug if sched/wiki or sched/wiki2 are configured and no job comment is 
    set.
 -- scontrol modified to report partition partition's "DisableRootJobs" value.
 -- Fix bug in setting host address for PMI communications (mpich2 only).
 -- Fix for memory size accounting on some architectures.
 -- NOTE: We needed to change an RPC from version 1.3.0. You must upgrade 
    all nodes in a cluster from v1.3.0 to v1.3.1 at the same time.

* Changes in SLURM 1.3.1
========================
 -- Correct logic for processing batch job's memory limit enforcement.
 -- Fix bug that was setting a job's requeue value on any update of the 
    job using the "scontrol update" command. The invalid value of an 
    updated job prevents it's recovery when slurmctld restarts.
 -- Add support for cluster-wide consumable resources. See "Licenses"
    parameter in slurm.conf man page and "--licenses" option in salloc, 
    sbatch and srun man pages.
 -- Major changes in select/cons_res to support FastSchedule=2 with more
    resources configured than actually exist (useful for testing purposes).
 -- Modify srun --test-only response to include expected initiation time 
    for a job as well as the nodes to be allocated and processor count
    (for use by Moab).
 -- Correct sched/backfill to properly honor job dependencies.
 -- Correct select/cons_res logic to allocate CPUs properly if there is
    more than one thread per core (previously failed to allocate all cores).
 -- Correct select/linear logic in shared job count (was off by 1).
 -- Add support for job preeption based upon partition priority (in sched/gang,
    preempt.patch from Chris Holmes, HP).
 -- Added much better logic for mysql accounting.  
 -- Finished all basic functionality for sacctmgr.
 -- Added load file logic to sacctmgr for setting up a cluster in one step.
 -- NOTE: We needed to change an RPC from version 1.3.0. You must upgrade 
    all nodes in a cluster from v1.3.0 to v1.3.1 at the same time.
 -- NOTE: Work is currently underway to improve placement of jobs for gang
    scheduling and preemption.
 -- NOTE: Work is underway to provide additional tools for reporting 
    accounting information.

* Changes in SLURM 1.3.0
========================
 -- In sched/wiki2, add processor count to JOBWILLRUN response.
 -- Add event trigger for node entering DRAINED state.
 -- Build properly without OpenSSL installed (OpenSSL is recommended, but not 
    required).
 -- Added slurmdbd, and modified accounting_storage plugin to talk to it. 
    Allowing multiple slurm systems to securly store and gather information
    not only about jobs, but the system also. See accounting web page for more
    information.    

* Changes in SLURM 1.3.0-pre11
==============================
 -- Restructure the sbcast RPC to take advantage of larger buffers available
    in Slurm v1.3 RPCs.
 -- Fix several memory leaks.
 -- In scontrol, show job's Requeue value, permit change of Requeue and Comment
    values.
 -- In slurmctld job record, add QOS (quality of service) value for accounting
    purposes with Maui and Moab.
 -- Log to a job's stderr when it is being cancelled explicitly or upon reaching
    it's time limit.
 -- Only permit a job's account to be changed while that job is PENDING.
 -- Fix race condition in job suspend/resume (slurmd.sus_res.patch from HP).

* Changes in SLURM 1.3.0-pre10
==============================
 -- Add support for node-specific "arch" (architecture) and "os" (operating 
    system) fields. These fields are set based upon values reported by the
    slurmd daemon on each compute node using SLURM_ARCH and SLURM_OS environment 
    variables (if set, the uname function otherwise) and are intended to support
    changes in real time changes in operating system. These values are reported
    by "scontrol show node" plus the sched/wiki and sched/wiki2 plugins for Maui
    and Moab respectively.
 -- In sched/wiki and sched/wiki2: add HostFormat and HidePartitionJobs to 
    "scontrol show config" SCHEDULER_CONF output.
 -- In sched/wiki2: accept hostname expression as input for GETNODES command.
 -- Add JobRequeue configuration parameter and --requeue option to the sbatch
    command.
 -- Add HealthCheckInterval and HealthCheckProgram configuration parameters.
 -- Add SlurmDbdAddr, SlurmDbdAuthInfo and SlurmDbdPort configuration parameters.
 -- Modify select/linear to achieve better load leveling with gang scheduler.
 -- Develop the sched/gang plugin to support select/linear and
    select/cons_res. If sched/gang is enabled and Shared=FORCE is configured
    for a partition, this plugin will gang-schedule or "timeslice" jobs that
    share common resources within the partition. Note that resources that are
    shared across partitions are not gang-scheduled.
 -- Add EpilogMsgTime configuration parameter. See "man slurm.conf" for details.
 -- Increase default MaxJobCount configuration parameter from 2000 to 5000. 
 -- Move all database common files from src/common to new lib in src/database.
 -- Move sacct to src/accounting added sacctmgr for scontrol like operations 
    to accounting infrastructure.
 -- Basic functions of sacctmgr in place to make for administration of 
    accounting.
 -- Moved clusteracct_storage plugin to accounting_storage plugin,
    jobacct_storage is still it's own plugin for now.
 -- Added template for slurm php extention.
 -- Add infrastructure to support allocation of cluster-wide licenses to jobs.
    Full support will be added some time after version 1.3.0 is released.
 -- In sched/wiki2 with select/bluegene, add support for WILLRUN command
    to accept multiple jobs with start time specifications.

* Changes in SLURM 1.3.0-pre9
=============================
 -- Add spank support to sbatch. Note that spank_local_user() will be called 
    with step_layout=NULL and gid=SLURM_BATCH_SCRIPT and spank_fini() will 
    be called immediately afterwards.
 -- Made configure use mysql_config to find location of mysql database install
    Removed bluegene specific information from the general database tables.
 -- Re-write sched/backfill to utilize new will-run logic in the select 
    plugins. It now supports select/cons_res and all job options (required
    nodes, excluded nodes, contiguous, etc.).
 -- Modify scheduling logic to better support overlapping partitions.
 -- Add --task-mem option and remove --job-mem option from srun, salloc, and 
    sbatch commands. Enforce step memory limit, if specified and there is
    no job memory limit specified (--mem). Also see DefMemPerTask and
    MaxMemPerTask in "man slurm.conf". Enforcement is dependent upon job
    accounting being enabled with non-zero value for JoabAcctGatherFrequency.
 -- Change default node tmp_disk size to zero (for diskless nodes).

* Changes in SLURM 1.3.0-pre8
=============================
 -- Modify how strings are packed in the RPCs, Maximum string size 
    increased from 64KB (16-bit size field) to 4GB (32-bit size field).
 -- Fix bug that prevented time value of "INFINITE" from being processed.
 -- Added new srun/sbatch option "--open-mode" to control how output/error 
    files are opened ("t" for truncate, "a" for append).
 -- Added checkpoint/xlch plugin for use with XLCH (Hongjia Cao, NUDT).
 -- Added srun option --checkpoint-path for use with XLCH (Hongjia Cao, NUDT).
 -- Added new srun/salloc/sbatch option "--acctg-freq" for user control over 
    accounting data collection polling interval.
 -- In sched/wiki2 add support for hostlist expression use in GETNODES command
    with HostFormat=2 in the wiki.conf file.
 -- Added new scontrol option "setdebug" that can change the slurmctld daemons
    debug level at any time (Hongjia Cao, NUDT).
 -- Track total total suspend time for jobs and steps for accounting purposes.
 -- Add version information to partition state file.
 -- Added 'will-run' functionality to all of the select plugins (bluegene,
    linear, and cons_res) to return node list and time job can start based 
    on other jobs running.
 -- Major restructuring of node selection logic. select/linear now supports
    partition max_share parameter and tries to match like size jobs on the 
    same nodes to improve gang scheduling performance. Also supports treating 
    memory as consumable resource for job preemption and  gang scheduling if 
    SelectTypeParameter=CR_Memory in slurm.conf.
 -- BLUEGENE: Reorganized bluegene plugin for maintainability sake.
 -- Major restructuring of data structures in select/cons_res.
 -- Support job, node and partition names of arbitrary size.
 -- Fix bug that caused slurmd to hang when using select/linear with
    task/affinity.

* Changes in SLURM 1.3.0-pre7
=============================
 -- Fix a bug in the processing of srun's --exclusive option for a job step.

* Changes in SLURM 1.3.0-pre6
=============================
 -- Add support for configurable number of jobs to share resources using the 
    partition Shared parameter in slurm.conf (e.g. "Shared=FORCE:3" for two 
    jobs to share the resources). From Chris Holmes, HP.
 -- Made salloc use api instead of local code for message handling.

* Changes in SLURM 1.3.0-pre5
=============================
 -- Add select_g_reconfigure() function to node changes in slurmctld configuration
    that can impact node scheduling.
 -- scontrol to set/get partition's MaxTime and job's Timelimit in minutes plus
    new formats: min:sec, hr:min:sec, days-hr:min:sec, days-hr, etc.
 -- scontrol "notify" command added to send message to stdout of srun for 
    specified job id.
 -- For BlueGene, make alpha part of node location specification be case insensitive.
 -- Report scheduler-plugin specific configuration information with the 
    "scontrol show configuration" command on the SCHEDULER_CONF line. This
    information is not found in the "slurm.conf" file, but a scheduler plugin 
    specific configuration (e.g. "wiki.conf").
 -- sview partition information reported now includes partition priority.
 -- Expand job dependency specification to support concurrent execution, 
    testing of job exit status and multiple job IDs.

* Changes in SLURM 1.3.0-pre4
=============================
 -- Job step launch in srun is now done from the slurm api's all further
    modifications to job launch should be done there.
 -- Add new partition configuration parameter Priority. Add job count to 
    Shared parameter.
 -- Add new configuration parameters DefMemPerTask, MaxMemPerTask, and