Newer
Older
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and administrators.
* Changes in Slurm 15.08.2
==========================
-- Fix for tracking node state when jobs that have been allocated exclusive
access to nodes (i.e. entire nodes) and later relinquish some nodes. Nodes
would previously appear partly allocated and prevent use by other jobs.
-- Correct some cgroup paths ("step_batch" vs. "step_4294967294", "step_exter"
vs. "step_extern", and "step_extern" vs. "step_4294967295").
-- Fix advanced reservation core selection logic with network topology.
-- MYSQL - Remove restriction to have to be at least an operator to query TRES
values.
-- For pending jobs have sacct print 0 for nnodes instead of the bogus 2.
-- Fix for tracking node state when jobs that have been allocated exclusive
access to nodes (i.e. entire nodes) and later relinquish some nodes. Nodes
would previously appear partly allocated and prevent use by other jobs.
-- Fix updating job in db after extending job's timelimit past partition's
timelimit.
-- Fix srun -I<timeout> from flooding the controller with step create requests.
-- Requeue/hold batch job launch request if job already running (possible if
node went to DOWN state, but jobs remained active).
-- If a job's CPUs/task ratio is increased due to configured MaxMemPerCPU,
then increase it's allocated CPU count in order to enforce CPU limits.
-- Don't mark powered down node as not responding. This could be triggered by
race condition of the node suspend and ping logic, preventing use of the
node.
-- Don't requeue RPC going out from slurmctld to DOWN nodes (can generate
repeating communication errors).
-- Propagate sbatch "--dist=plane=#" option to srun.
-- Add acct_gather_energy/ibmaem plugin for systems with IBM Systems Director
Active Energy Manager.
-- Fix spec file to look for mariadb or mysql devel packages for build
requirements.
-- MySQL - Improve the code with asking for jobs in a suspended state.
-- Fix slurcmtld allowing root to see job steps using squeues -s.
-- Do not send burst buffer stage out email unless the job uses burst buffers.
-- Fix sacct to not return all jobs if the -j option is given with a trailing
','.
-- Permit job_submit plugin to set a job's priority.
-- Fix occasional srun segfault.
-- Fix issue with sacct, printing 0_0 for array's that had finished in the
database but the start record hadn't made it yet.
-- sacctmgr - Don't allow default account associations to be removed
from a user.
-- Fix sacct -j, (nothing but a comma) to not return all jobs.
-- Fixed slurmctld not sending cold-start messages correctly to the database
when a cold-start (-c) happens to the slurmctld.
Brian Christiansen
committed
-- Fix case where if the backup slurmdbd has existing connections when it gives
up control that the it would be killed.
* Changes in Slurm 15.08.1
==========================
-- Fix test21.30 and 21.34 to check grpwall better.
-- Add time to the partition QOS the job is running on instead of just the
job QOS.
-- Print usage for GrpJobs, GrpSubmitJobs and GrpWall even if there is no
limit.
-- If AccountingEnforce=safe is set make sure a job can finish before going
over the limit with grpwall on a QOS or association.
-- burst_buffer/cray - Major updates based upon recent Cray changes.
-- Improve job state reason string when required nodes not available.
-- Fix missing else when packing an update partition message
Brian Christiansen
committed
-- Fix srun from inheriting the SLURM_CPU_BIND and SLURM_MEM_BIND environment
variables when running in an existing srun (e.g. an srun within an salloc).
-- Fix missing else when packing an update partition message.
-- Use more flexible mechnanism to find json installation.
-- Make sure safe_limits was initialized before processing limits in the
slurmctld.
-- Fix for burst_buffer/cray to parse type option correctly.
-- Fix memory error and version number in the nonstop plugin and reservation
code.
-- When requesting GRES in a step check for correct variable for the count.
-- Fix issue with GRES in steps so that if you have multiple exclusive steps
and you use all the GRES up instead of reporting the configuration isn't
available you hold the requesting step until the GRES is available.
-- MYSQL - Change debug to print out with DebugFlags=DB_Step instead of debug4
-- Simplify code when user is selecting a job/step/array id and removed
anomaly when only asking for 1 (task_id was never set to INFINITE).
-- MYSQL - If user is requesting various task_ids only return requested steps.
-- Fix issue when tres cnt for energy is 0 for total reported.
-- Resolved scalability issues of power adaptive scheduling with layouts.
-- Burst_buffer/cray bug - Fix teardown race condition that can result in
infinite loop.
-- Job "--reboot" option automatically, set's exclusive node mode.
-- Fix memory leak when using PrologFlags=Alloc.
-- Fix truncation of job reason in squeue.
-- If a node is in DOWN or DRAIN state, leave it unavailable for allocation
when powered down.
-- Update the slurm.conf man page documenting better nohold_on_prolog_fail
variable.
-- Don't trucate task ID information in "squeue --array/-r" or "sview".
-- Fix a bug which caused scontrol to core dump when releasing or
holding a job by name.
-- Fix unit conversion bug in slurmd which caused wrong memory calculation
for cgroups.
-- Fix issue with GRES in steps so that if you have multiple exclusive steps
and you use all the GRES up instead of reporting the configuration isn't
available you hold the requesting step until the GRES is available.
-- Fix slurmdbd backup to use DbdAddr when contacting the primary.
-- Fix error in MPI documentation.
-- Fix to handle arrays with respect to number of jobs submitted. Previously
only 1 job was accounted (against MaxSubmitJob) for when an array was
submitted.
-- Correct counting for job array limits, job count limit underflow possible
when master cancellation of master job record.
-- Combine 2 _valid_uid_gid functions into a single function to avoid
diversion.
-- Pending job array records will be combined into single line by default,
even if started and requeued or modified.
-- Fix sacct --format=nnodes to print out correct information for pending
jobs.
-- Make is so 'scontrol update job 1234 qos='' will set the qos back to
the default qos for the association.
-- Add [Alloc|Req]Nodes to sacct to be more like cpus.
-- Fix sacct documentation about [Alloc|Req]TRES
-- Put node count in TRES string for steps.
-- Fix issue with wrong protocol version when using the srun --no-allocate
option.
-- Fix TRES counts on GRES on a clean start of the slurmctld.
-- Add ability to change a job array's maximum running task count:
"scontrol update jobid=# arraytaskthrottle=#"
* Changes in Slurm 15.08.0
==========================
-- Fix issue with frontend systems (outside ALPs or BlueGene) where srun
wouldn't get the correct protocol version to launch a step.
-- Fix for message aggregation return rpcs where none of the messages are
intended for the head of the tree.
-- Fix segfault in sreport when there was no response from the dbd.
-- ALPS - Fix compile to not link against -ljob and -lexpat with every lib
or binary.
-- Fix testing for CR_Memory when CR_Memory and CR_ONE_TASK_PER_CORE are used
with select/linear.
-- When restarting or reconfiging the slurmctld, if job is completing handle
accounting correctly to avoid meaningless errors about overflow.
-- Add AccountingStorageTRES to scontrol show config
-- MySQL - Fix minor memory leak if a connection ever goes away whist using it.
-- ALPS - Make it so srun --hint=nomultithread works correctly.
-- Make MaxTRESPerUser work in sacctmgr.
-- Fix handling of requeued jobs with steps that are still finishing.
-- Cleaner copy for PriorityWeightTRES, it also fixes a core dump when trying
to free it otherwise.
-- Add environment variables SLURM_ARRAY_TASK_MAX, SLURM_ARRAY_TASK_MIN,
SLURM_ARRAY_TASK_STEP for job arrays.
-- Fix srun to use the NoInAddrAny TopologyParam option.
-- Change QOS flag name from PartitionQOS to OverPartQOS to be a better
description.
-- Make complete_batch_script RPC work with message aggregation.
-- Do not count slurmctld threads waiting in a "throttle" lock against the
daemon's thread limit as they are not contending for resources.
-- Modify slurmctld outgoing RPC logic to support more parallel tasks (up to
85 RPCs and 256 pthreads; the old logic supported up to 21 RPCs and 256
threads). This change can dramatically improve performance for RPCs
operating on small node counts.
-- Increase total backfill scheduler run time in stats_info_response_msg data
structure from 32 to 64 bits in order to prevent overflow.
-- Add NoInAddrAny option to TopologyParam in the slurm.conf which allows to
bind to the interface of return of gethostname instead of any address on
the node which avoid RSIP issues in Cray systems. This is most likely
useful in other systems as well.
-- Fix memory leak in Slurm::load_jobs perl api call.
-- Added --noconvert option to sacct, sstat, squeue and sinfo which allows
values to be displayed in their original unit types (e.g. 2048M won't be
converted to 2G).
-- Fix spelling of node_rescrs to node_resrcs in Perl API.
-- Fix node state race condition, UNKNOWN->IDLE without configuration info.
-- Cray: Disable LDAP references from slurmstepd on job launch due for
improved scalability.
-- Remove srun "read header error" due to application termination race
condition.
-- Optimize sacct queries with additional db indexes.
-- Add SLURM_TOPO_LEN env variable for scontrol show topology.
-- Add free_mem to node information.
-- Fix abort of batch launch if prolog is running, wait for prolog instead.
Brian Christiansen
committed
-- Fix case where job would get the wrong cpu count when using
--ntasks-per-core and --cpus-per-task together.
-- Add TRESBillingWeights to partitions in slurm.conf which allows taking into
consideration any TRES Type when calculating the usage of a job.
-- Add PriorityWeightTRES slurm.conf option to be able to configure priority
factors for TRES types.
* Changes in Slurm 15.08.0pre6
==============================
-- Add scontrol options to view and modify layouts tables.
-- Add MsgAggregationParams which controls a reverse tree to the slurmctld
which can be used to aggregate messages to the slurmctld into a single
message to reduce communication to the slurmctld. Currently only epilog
complete messages and node registration messages use this logic.
-- Add sacct and squeue options to print trackable resources.
-- Add sacctmgr option to display trackable resources.
-- If an salloc or srun command is executed on a "front-end" configuration,
that job will be assigned a slurmd shepherd daemon on the same host as used
Loading
Loading full blame...