Newer
Older
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and admins.
* Changes in Slurm 14.03.6
==========================
-- Added examples to demonstrate the use of the sacct -T option to the man
page.
-- Fix for regression in 14.03.5 with sacctmgr load when Parent has "'"
around it
-- Update comments in sacctmgr dump header
* Changes in Slurm 14.03.5
==========================
-- If a srun runs in an exclusive allocation and doesn't use the entire
allocation and CR_PACK_NODES is set layout tasks appropriately.
-- Correct Shared field in job state information seen by scontrol, sview, etc.
-- Print Slurm error string in scontrol update job and reset the Slurm errno
before each call to the API.
-- Fix task/cgroup to handle -mblock:fcyclic correctly
-- Fix for core-based advanced reservations where the distribution of cores
across nodes is not even.
-- Fix issue where association maxnodes wouldn't be evaluated correctly if a
QOS had a GrpNodes set.
-- GRES fix with multiple files defined per line in gres.conf.
-- When a job is requeued make sure accounting marks it as such.
-- Print the state of requeued job as REQUEUED.
-- Fix if a job's partition was taken away from it don't allow a requeue.
-- Make sure we lock on the conf when sending slurmd's conf to the slurmstepd.
-- Fix issue with sacctmgr 'load' not able to gracefully handle bad formatted
file.
-- sched/backfill: Correct job start time estimate with advanced reservations.
-- Error message added when in proctrack/cgroup the step freezer path isn't
able to be destroyed for debug.
-- Added extra index's into the database for better performance when
deleting users.
-- Fix issue with wckeys when tracking wckeys, but not enforcing them,
you could get multiple '*' wckeys.
-- Fix bug which could report to squeue the wrong partition for a running job
that is submitted to multiple partitions.
-- Report correct CPU count allocated to job when allocated whole node even if
not using all CPUs.
-- If job's constraints cannot be satisfied put it in pending state with reason
BadConstraints and don't remove it.
-- sched/backfill - If job started with infinite time limit, set its end_time
one year in the future.
-- Clear QOS GrpUsedCPUs when resetting raw usage if QOS is not using any cpus.
-- Remove log message left over from debugging.
-- When using CR_PACK_NODES fix make --ntasks-per-node work correctly.
-- Report correct partition associated with a step if the job is submitted to
multiple partitions.
-- Fix to allow removing of preemption from a QOS
-- If the proctrack plugins fail to destroy the job container print an error
message and avoid to loop forever, give up after 120 seconds.
-- Make srun obey POSIX convention and increase the exit code by 128 when the
process terminated by a signal.
-- Sanity check for acct_gather_energy/rapl
-- If the proctrack plugins fail to destroy the job container print an error
message and avoid to loop forever, give up after 120 seconds.
-- If the sbatch command specifies the option --signal=B:signum sent the signal
to the batch script only.
-- If we cancel a task and we have no other exit code send the signal and
exit code.
-- Added note about InnoDB storage engine being used with MySQL.
-- Set the job exit code when the job is signaled and set the log level to
debug2() when processing an already completed job.
-- Reset diagnostics time stamp when "sdiag --reset" is called.
-- squeue and scontrol to report a job's "shared" value based upon partition
options rather than reporting "unknown" if job submission does not use
--exclusive or --shared option.
-- task/cgroup - Fix cpuset binding for batch script.
-- sched/backfill - Fix anomaly that could result in jobs being scheduled out
of order.
-- Expand pseudo-terminal size data structure field sizes from 8 to 16 bits.
-- Set the job exit code when the job is signaled and set the log level to
-- Distinguish between two identical error messages.
-- If using accounting_storage/mysql directly without a DBD fix issue with
start of requeued jobs.
-- If a job fails because of batch node failure and the job is requeued and an
epilog complete message comes from that node do not process the batch step
information since the job has already been requeued because the epilog
script running isn't guaranteed in this situation.
-- Change message to note a NO_VAL for return code could of come from node
failure as well as interactive user.
-- Modify test4.5 to only look at one partition instead of all of them.
-- Fix sh5util -u to accept username different from the user that runs the
command.
-- Corrections to man pages:salloc.1 sbatch.1 srun.1 nonstop.conf.5
slurm.conf.5.
-- Have sacctmgr dump cluster handle situations where users or such have
special characters in their names like ':'
-- Fix issue where not enforcing QOS but a partition either allows or denies
them.
-- CRAY - Make switch/cray default when running on a Cray natively.
-- CRAY - Make job_container/cncu default when running on a Cray natively.
-- Disable job time limit change if it's preemption is in progress.
-- Correct logic to properly enforce job preemption GraceTime.
-- Fix sinfo -R to print each down/drained node once, rather than once per
partition.
-- If a job has non-responding node, retry job step create rather than
returning with DOWN node error.
-- Support SLURM_CONF path which does not have "slurm.conf" as the file name.
-- CRAY - make job_container/cncu default when running on a Cray natively
-- Fix issue where batch cpuset wasn't looked at correctly in
jobacct_gather/cgroup.
-- Correct squeue's job node and CPU counts for requeued jobs.
-- Correct SelectTypeParameters=CR_LLN with job selecition of specific nodes.
-- Only if ALL of their partitions are hidden will a job be hidden by default.
-- Run EpilogSlurmctld for a job is killed during slurmctld reconfiguration.
-- Close window with srun if waiting for an allocation and while printing
something you also get a signal which would produce deadlock.
-- Add SelectTypeParameters option of CR_PACK_NODES to pack a job's tasks
tightly on its allocated nodes rather than distributing them evenly across
the allocated nodes.
-- cpus-per-task support: Try to pack all CPUs of each tasks onto one socket.
Previous logic could spread the tasks CPUs across multiple sockets.
-- Add new distribution method fcyclic so when a task is using multiple cpus
it can bind cyclically across sockets.
-- task/affinity - When using --hint=nomultithread only bind to the first
thread in a core.
-- Make cgroup task layout (block | cyclic) method mirror that of
task/affinity.
-- If TaskProlog sets SLURM_PROLOG_CPU_MASK reset affinity for that task
based on the mask given.
-- Keep supporting 'srun -N x --pty bash' for historical reasons.
-- If EnforcePartLimits=Yes and QOS job is using can override limits, allow
it.
-- Fix issues if partition allows or denies account's or QOS' and either are
not set.
-- If a job requests a partition and it doesn't allow a QOS or account the
job is requesting pend unless EnforcePartLimits=Yes. Before it would
always kill the job at submit.
-- Fix format output of scontrol command when printing node state.
-- Improve the clean up of cgroup hierarchy when using the
jobacct_gather/cgroup plugin.
-- Added SchedulerParameters value of Ignore_NUMA.
-- Fix issues with code when using automake 1.14.1
-- select/cons_res plugin: Fix memory leak related to job preemption.
-- After reconfig rebuild the job node counters only for jobs that have
not finished yet, otherwise if requeued the job may enter an invalid
COMPLETING state.
-- Do not purge the script and environment files for completed jobs on
slurmctld reconfiguration or restart (they might be later requeued).
-- scontrol now accepts the option job=xxx or jobid=xxx for the requeue,
requeuehold and release operations.
-- task/cgroup - fix to bind batch job in the proper CPUs.
-- Added strigger option of -N, --noheader to not print the header when
displaying a list of triggers.
-- Modify strigger to accept arguments to the program to execute when an
event trigger occurs.
-- Attempt to create duplicate event trigger now generates ESLURM_TRIGGER_DUP
("Duplicate event trigger").
-- Treat special characters like %A, %s etc. literally in the file names
when specified escaped e.g. sbatch -o /home/zebra\\%s will not expand
%s as the stepid of the running job.
-- CRAYALPS - Add better support for CLE 5.2 when running Slurm over ALPS.
-- Test time when job_state file was written to detect multiple primary
slurmctld daemons (e.g. both backup and primary are functioning as
primary and there is a split brain problem).
-- Fix scontrol to accept update jobid=# numtasks=#
-- If the backup slurmctld assumes primary status, then do NOT purge any
job state files (batch script and environment files) and do not re-use them.
This may indicate that multiple primary slurmctld daemons are active (e.g.
both backup and primary are functioning as primary and there is a split
brain problem).
-- Set correct error code when requeuing a completing/pending job
-- When checking for if dependency of type afterany, afterok and afternotok
don't clear the dependency if the job is completing.
-- Cleanup the JOB_COMPLETING flag and eventually requeue the job when the
last epilog completes, either slurmd epilog or slurmctld epilog, whichever
comes last.
-- When attempting to requeue a job distinguish the case in which the job is
JOB_COMPLETING or already pending.
-- When reconfiguring the controller don't restart the slurmctld epilog if it
is already running.
-- Email messages for job array events print now use the job ID using the
format "#_# (#)" rather than just the internal job ID.
-- Set the number of free licenses to be 0 if the global license count
decreases and total is less than in use.
-- Add DebugFlag of BackfillMap. Previously a DebugFlag value of Backfill
logged information about what it was doing plus a map of expected resouce
use in the future. Now that very verbose resource use map is only logged
with a DebugFlag value of BackfillMap
-- Modify the description of -E and -S option of sacct command as point in time
'before' or 'after' the database records are returned.
-- Correct support for partition with Shared=YES configuration.
-- If job requests --exclusive then do not use nodes which have any cores in an
advanced reservation. Also prevents case where nodes can be shared by other
jobs.
-- For "scontrol --details show job" report the correct CPU_IDs when thre are
multiple threads per core (we are translating a core bitmap to CPU IDs).
-- If DebugFlags=Protocol is configured in slurm.conf print details of the
connection, ip address and port accepted by the controller.
-- Fix minor memory leak when reading in incomplete node data checkpoint file.
Loading
Loading full blame...