NEWS

This file describes changes in recent versions of SLURM. It primarily
documents those changes that are of interest to users and admins.

* Changes in SLURM 1.3.1
========================
 -- Correction in calculation of a job's memory used for use in enforcement
    of a job's memory limit.
 -- Correct logic for processing batch job's memory limit enforcement.
 -- Fix bug that was setting a job's requeue value on any update of the 
    job using the "scontrol update" command. The invalid value of an 
    updated job prevents it's recovery when slurmctld restarts.
 -- Add support for cluster-wide consumable resources. See "Licenses"
    parameter in slurm.conf man page and "--licenses" option in salloc, 
    sbatch and srun man pages.

* Changes in SLURM 1.3.0
========================
 -- In sched/wiki2, add processor count to JOBWILLRUN response.
 -- Add event trigger for node entering DRAINED state.
 -- Build properly without OpenSSL installed (OpenSSL is recommended, but not 
    required).
 -- Added slurmdbd, and modified accounting_storage plugin to talk to it. 
    Allowing multiple slurm systems to securly store and gather information
    not only about jobs, but the system also. See accounting web page for more
    information.    

* Changes in SLURM 1.3.0-pre11
==============================
 -- Restructure the sbcast RPC to take advantage of larger buffers available
    in Slurm v1.3 RPCs.
 -- Fix several memory leaks.
 -- In scontrol, show job's Requeue value, permit change of Requeue and Comment
    values.
 -- In slurmctld job record, add QOS (quality of service) value for accounting
    purposes with Maui and Moab.
 -- Log to a job's stderr when it is being cancelled explicitly or upon reaching
    it's time limit.
 -- Only permit a job's account to be changed while that job is PENDING.
 -- Fix race condition in job suspend/resume (slurmd.sus_res.patch from HP).

* Changes in SLURM 1.3.0-pre10
==============================
 -- Add support for node-specific "arch" (architecture) and "os" (operating 
    system) fields. These fields are set based upon values reported by the
    slurmd daemon on each compute node using SLURM_ARCH and SLURM_OS environment 
    variables (if set, the uname function otherwise) and are intended to support
    changes in real time changes in operating system. These values are reported
    by "scontrol show node" plus the sched/wiki and sched/wiki2 plugins for Maui
    and Moab respectively.
 -- In sched/wiki and sched/wiki2: add HostFormat and HidePartitionJobs to 
    "scontrol show config" SCHEDULER_CONF output.
 -- In sched/wiki2: accept hostname expression as input for GETNODES command.
 -- Add JobRequeue configuration parameter and --requeue option to the sbatch
    command.
 -- Add HealthCheckInterval and HealthCheckProgram configuration parameters.
 -- Add SlurmDbdAddr, SlurmDbdAuthInfo and SlurmDbdPort configuration parameters.
 -- Modify select/linear to achieve better load leveling with gang scheduler.
 -- Develop the sched/gang plugin to support select/linear and
    select/cons_res. If sched/gang is enabled and Shared=FORCE is configured
    for a partition, this plugin will gang-schedule or "timeslice" jobs that
    share common resources within the partition. Note that resources that are
    shared across partitions are not gang-scheduled.
 -- Add EpilogMsgTime configuration parameter. See "man slurm.conf" for details.
 -- Increase default MaxJobCount configuration parameter from 2000 to 5000. 
 -- Move all database common files from src/common to new lib in src/database.
 -- Move sacct to src/accounting added sacctmgr for scontrol like operations 
    to accounting infrastructure.
 -- Basic functions of sacctmgr in place to make for administration of 
    accounting.
 -- Moved clusteracct_storage plugin to accounting_storage plugin,
    jobacct_storage is still it's own plugin for now.
 -- Added template for slurm php extention.
 -- Add infrastructure to support allocation of cluster-wide licenses to jobs.
    Full support will be added some time after version 1.3.0 is released.
 -- In sched/wiki2 with select/bluegene, add support for WILLRUN command
    to accept multiple jobs with start time specifications.

* Changes in SLURM 1.3.0-pre9
=============================
 -- Add spank support to sbatch. Note that spank_local_user() will be called 
    with step_layout=NULL and gid=SLURM_BATCH_SCRIPT and spank_fini() will 
    be called immediately afterwards.
 -- Made configure use mysql_config to find location of mysql database install
    Removed bluegene specific information from the general database tables.
 -- Re-write sched/backfill to utilize new will-run logic in the select 
    plugins. It now supports select/cons_res and all job options (required
    nodes, excluded nodes, contiguous, etc.).
 -- Modify scheduling logic to better support overlapping partitions.
 -- Add --task-mem option and remove --job-mem option from srun, salloc, and 
    sbatch commands. Enforce step memory limit, if specified and there is
    no job memory limit specified (--mem). Also see DefMemPerTask and
    MaxMemPerTask in "man slurm.conf". Enforcement is dependent upon job
    accounting being enabled with non-zero value for JoabAcctGatherFrequency.
 -- Change default node tmp_disk size to zero (for diskless nodes).

* Changes in SLURM 1.3.0-pre8
=============================
 -- Modify how strings are packed in the RPCs, Maximum string size 
    increased from 64KB (16-bit size field) to 4GB (32-bit size field).
 -- Fix bug that prevented time value of "INFINITE" from being processed.
 -- Added new srun/sbatch option "--open-mode" to control how output/error 
    files are opened ("t" for truncate, "a" for append).
 -- Added checkpoint/xlch plugin for use with XLCH (Hongjia Cao, NUDT).
 -- Added srun option --checkpoint-path for use with XLCH (Hongjia Cao, NUDT).
 -- Added new srun/salloc/sbatch option "--acctg-freq" for user control over 
    accounting data collection polling interval.
 -- In sched/wiki2 add support for hostlist expression use in GETNODES command
    with HostFormat=2 in the wiki.conf file.
 -- Added new scontrol option "setdebug" that can change the slurmctld daemons
    debug level at any time (Hongjia Cao, NUDT).
 -- Track total total suspend time for jobs and steps for accounting purposes.
 -- Add version information to partition state file.
 -- Added 'will-run' functionality to all of the select plugins (bluegene,
    linear, and cons_res) to return node list and time job can start based 
    on other jobs running.
 -- Major restructuring of node selection logic. select/linear now supports
    partition max_share parameter and tries to match like size jobs on the 
    same nodes to improve gang scheduling performance. Also supports treating 
    memory as consumable resource for job preemption and  gang scheduling if 
    SelectTypeParameter=CR_Memory in slurm.conf.
 -- BLUEGENE: Reorganized bluegene plugin for maintainability sake.
 -- Major restructuring of data structures in select/cons_res.
 -- Support job, node and partition names of arbitrary size.
 -- Fix bug that caused slurmd to hang when using select/linear with
    task/affinity.

* Changes in SLURM 1.3.0-pre7
=============================
 -- Fix a bug in the processing of srun's --exclusive option for a job step.

* Changes in SLURM 1.3.0-pre6
=============================
 -- Add support for configurable number of jobs to share resources using the 
    partition Shared parameter in slurm.conf (e.g. "Shared=FORCE:3" for two 
    jobs to share the resources). From Chris Holmes, HP.
 -- Made salloc use api instead of local code for message handling.

* Changes in SLURM 1.3.0-pre5
=============================
 -- Add select_g_reconfigure() function to node changes in slurmctld configuration
    that can impact node scheduling.
 -- scontrol to set/get partition's MaxTime and job's Timelimit in minutes plus
    new formats: min:sec, hr:min:sec, days-hr:min:sec, days-hr, etc.
 -- scontrol "notify" command added to send message to stdout of srun for 
    specified job id.
 -- For BlueGene, make alpha part of node location specification be case insensitive.
 -- Report scheduler-plugin specific configuration information with the 
    "scontrol show configuration" command on the SCHEDULER_CONF line. This
    information is not found in the "slurm.conf" file, but a scheduler plugin 
    specific configuration (e.g. "wiki.conf").
 -- sview partition information reported now includes partition priority.
 -- Expand job dependency specification to support concurrent execution, 
    testing of job exit status and multiple job IDs.

* Changes in SLURM 1.3.0-pre4
=============================
 -- Job step launch in srun is now done from the slurm api's all further
    modifications to job launch should be done there.
 -- Add new partition configuration parameter Priority. Add job count to 
    Shared parameter.
 -- Add new configuration parameters DefMemPerTask, MaxMemPerTask, and 
    SchedulerTimeSlice.
 -- In sched/wiki2, return REJMESSAGE with details on why a job was 
    requeued (e.g. what node failed).

* Changes in SLURM 1.3.0-pre3
=============================
 -- Remove slaunch command
 -- Added srun option "--checkpoint=time" for job step to automatically be 
    checkpointed on a period basis.
 -- Change behavior of "scancel -s KILL <jobid>" to send SIGKILL to all job
    steps rather than cancelling the job. This now matches the behavior of
    all other signals. "scancel <jobid>" still cancels the job and all steps.
 -- Add support for new job step options --exclusive and --immediate. Permit
    job steps to be queued when resources are not available within an existing 
    job allocation to dedicate the resources to the job step. Useful for
    executing simultaneous job steps. Provides resource management both at 
    the level of jobs and job steps.
 -- Add support for feature count in job constraints, for example
    srun --nodes=16 --constraint=graphics*4 ...
    Based upon work by Kumar Krishna (HP, India).
 -- Add multi-core options to salloc and sbatch commands (sbatch.patch and
    cleanup.patch from Chris Holmes, HP).
 -- In select/cons_res properly release resources allocated to job being 
    suspended (rmbreak.patch, from Chris Holmes, HP).
 -- Removed database and jobacct plugin replaced with jobacct_storage 
    and jobacct_gather for easier hooks for further expansion of the
    jobacct plugin.

* Changes in SLURM 1.3.0-pre2
=============================
 -- Added new srun option --pty to start job with pseudo terminal attached 
    to task 0 (all other tasks have I/O discarded)
 -- Disable user specifying jobid when sched/wiki2 configured (needed for 
    Moab releases until early 2007).
 -- Report command, args and working directory for batch jobs with 
    "scontrol show job".

* Changes in SLURM 1.3.0-pre1
=============================
 -- !!! SRUN CHANGES !!!
    The srun options -A/--allocate, -b/--batch, and -a/--attach have been
    removed!  That functionality is now available in the separate commands
    salloc, sbatch, and sattach, respectively.
 -- Add new node state FAILING plus trigger for when node enters that state.
 -- Add new configuration paramter "PrivateData". This can be used to 
    prevent a user from seeing jobs or job steps belonging to other users.
 -- Added configuration parameters for node power save mode: ResumeProgram
    ResumeRate, SuspendExcNodes, SuspendExcParts, SuspendProgram and 
    SuspendRate.
 -- Slurmctld maintains the IP address (rather than hostname) for srun 
    communications. This fixes some possible network routing issues.
 -- Added global database plugin.  Job accounting and Job completion are the 
    first to use it.  Follow documentation to add more to the plugin.
 -- Removed no-longer-needed jobacct/common/common_slurmctld.c since that is
    replaced by the database plugin.
 -- Added new configuration parameter: CryptoType.
    Moved existing digital signature logic into new plugin: crypto/openssl.
    Added new support for crypto/munge (available with GPL license).

* Changes in SLURM 1.2.26
=========================
 -- Correct number of sockets/cores/threads reported by slurmd (from
    Par Andersson, National Supercomputer Centre, Sweden).
 -- Update libpmi linking so that libslurm is not required for PMI use
    (from Steven McDougal, SiCortex).
 -- In srun and sbatch, do not check the PATH env var if an absolute pathname 
    of the program is specified (previously reported an error if no PATH).

* Changes in SLURM 1.2.25
=========================
 -- Bug fix for setting exit code in accounting for batch script.
 -- Add salloc option, --no-shell (for LSF).
 -- Added new options for sacct output
 -- mvapich: Ensure MPIRUN_ID is unique for all job steps within a job.
    (Fixes crashes when running multiple job steps within a job on one node)
 -- Prevent "scontrol show job" from failing with buffer overflow when a job 
    has a very long Comment field.
 -- Make certain that a job step is purged when a job has been completed.
    Previous versions could have the job step persist if an allocated node
    went DOWN and the slurmctld restarted.
 -- Fix bug in sbcast that can cause communication problems for large files.
 -- Add sbcast option -t/--timeout and SBCAST_TIMEOUT environment variable 
    to control message timeout.
 -- Add threaded agent to manage a queue of Gold update requests for 
    performance reasons.
 -- Add salloc options --chdir and --get-user-env (for Moab).
 -- Modify scontrol update to support job comment changes.
 -- Do not clear a DRAINED node's reason field when slurmctld restarts.
 -- Do not cancel a pending job if Moab or Maui try to start it on unusable nodes.
    Leave the job queued.
 -- Add --requeue option to srun and sbatch (these undocumented options have no
    effect in slurm v1.2, but are legitimate options in slurm v1.3).
 
* Changes in SLURM 1.2.24
=========================
 -- In sched/wiki and sched/wiki2, support non-zero UPDATE_TIME specification
    for GETNODES and GETJOBS commands.
 -- Bug fix for sending accounting information multiple times for same 
    info.  patch from Hongjia Cao (NUDT).
 -- BLUEGENE - try FILE pointer rotation logic to avoid core dump on 
    bridge log rotate
 -- Spread out in time the EPILOG_COMPLETE messages from slurmd to slurmctld
    to avoid message congestions and retransmission.

* Changes in SLURM 1.2.23
=========================
 -- Fix for libpmi to not export unneeded variables like xstr*
 -- BLUEGENE - added per partition dynamic block creation
 -- fix infinite loop bug in sview when there were multiple partitions
 -- Send message to srun command when a job is requeued due to node failure.
    Note this will be overwritten in the output file unless JobFileAppend
    is set in slurm.conf. In slurm version 1.3, srun's --open-mode=append
    option will offer this control for each job.
 -- Change a node's default TmpDisk from 1MB to 0MB and change job's default 
    disk space requirement from 1MB to 0MB.
 -- In sched/wiki (Maui scheduler) specify a QOS (quality of service) by 
    specifying an account of the form "qos-name".
 -- In select/linear, fix bug in scheduling required nodes that already have
    a job running on them (req.load.patch from Chris Holmes, HP).
 -- For use with Moab only: change timeout for srun/sbatch --get-user-env 
    option to 2 secs, don't get DISPLAY environment variables, but explicitly 
    set ENVIRONMENT=BATCH and HOSTNAME to the execution host of the batch script.
 -- Add configuration parameter GetEnvTimeout for use with Moab. See
    "man slurm.conf" for details.
 -- Modify salloc and sbatch to accept both "--tasks" and "--ntasks" as 
    equival