Newer
Older
This file describes changes in recent versions of Slurm. It primarily
documents those changes that are of interest to users and admins.
* Changes in Slurm 2.6.8
========================
-- Add support for Torque/PBS job array options and environment variables.
-- Fix issue where jobs still pending after a reservation would remain
in waiting reason ReqNodeNotAvail.
-- Update last_job_update when a job's state_reason was modified.
-- Free job_ptr->state_desc where ever state_reason is set.
-- Fixed sacct.1 and srun.1 manual pages which contains a hyphen where
a minus sign for options was intended.
-- sinfo - Make sure if partition name is long and the default the last char
doesn't get chopped off.
-- task/affinity - Protect against zero divide when simulating more hardware
than you really have.
-- NRT - Fix issue with 1 node jobs. It turns out the network does need to
be setup for 1 node jobs.
-- Fix recovery of job dependency on task of job array when slurmctld restarts.
-- mysql - Fix invalid memory reference.
* Changes in Slurm 2.6.7
========================
-- Properly enforce a job's cpus-per-task option when a job's allocation is
constrained on some nodes by the mem-per-cpu option.
-- Correct the slurm.conf man pages and checkpoint_blcr.html page
describing that jobs must be drained from cluster before deploying
-- Fix issue where if using munge and munge wasn't running and a slurmd
-- Update srun.1 man page documenting the PMI2 support.
-- Fix slurmctld core dump when a jobs gets its QOS updated but there
is not a corresponding association.
-- If a job requires specific nodes and can not run due to those nodes being
busy, the main scheduling loop will block those specific nodes rather than
the entire queue/partition.
-- Fix minor memory leak when updating a job's name.
-- Fix minor memory leak when updating a reservation on a partition using "ALL"
nodes.
-- Fix minor memory leak when adding a reservation with a nodelist and core
count.
-- Update sacct man page description of job states.
-- BGQ - Fix minor memory leak when selecting blocks that can't immediately be
placed.
-- Fixed minor memory leak in backfill scheduler.
-- MYSQL - Fixed memory leak when querying clusters.
-- MYSQL - Fix when updating QOS on an association.
-- NRT - Fix to supply correct error messages to poe/pmd when a launch fails.
-- Add SLURM_STEP_ID to Prolog environment.
-- Add support for SchedulerParameters value of bf_max_job_start that limits
the total number of jobs that can be started in a single iteration of the
backfill scheduler.
-- Don't print negative number when dealing with large memory sizes with
sacct.
-- Fix sinfo output so that host in state allocated and mixed will not be
merged together.
-- GRES: Avoid crash if GRES configurations is inconstent.
-- Make S_SLURM_RESTART_COUNT item available to SPANK.
-- Munge plugins - Add sleep between retries if can't connect to socket.
-- Fix the database query to return all pending jobs in a given time interval.
-- switch/nrt - Correct logic to get dynamic window count.
-- Remove need to use job->ctx_params in the launch plugin, just to simplify
code.
-- NRT - Fix possible memory leak if using multiple adapters.
-- NRT - Fix issue where there are more than NRT_MAXADAPTERS on a system.
-- NRT - Increase Max number of adapters from 8 -> 9
-- NRT - Initialize missing variables when the PMD is starting a job.
-- NRT - Fix issue where we are launching hosts out of numerical order,
this would cause pmd's to hang.
-- NRT - Change xmalloc's to malloc just to be safe.
-- NRT - Sanity check to make sure a jobinfo is there before packing.
-- Add missing options to the print of TaskPluginParam.
-- Fix a couple of issues with scontrol reconfig and adding nodes to
slurm.conf. Rebooting daemons after adding nodes to the slurm.conf
is highly recommended.
* Changes in Slurm 2.6.6
========================
-- sched/backfill - Fix bug that could result in failing to reserve resources
for high priority jobs.
-- Correct job RunTime if requeued from suspended state.
-- Reset job priority from zero (held) on manual resume from suspend state.
-- If FastSchedule=0 then do not DOWN a node with low memory or disk size.
-- Update sshare.1 man page making it consistent with sacctmgr.1.
-- Do not reset a job's priority when the slurmctld restarts if previously
set to some specific value.
-- sview - Fix regression where the Node tab wasn't able to add/remove columns.
-- Fix slurmstepd lock when job terminates inside the infiniband
network traffic accounting plugin.
-- Correct the documentation to read filesystem instead of Lustre. Update
the srun help.
-- Fix the acct_gather_filesystem_lustre.c to compute the Lustre accounting
data correctly accumulating differences between sampling intervals.
Fix the data structure mismatch between acct_gather_filesystem_lustre.c
and slurm_jobacct_gather.h which caused the hdf5 plugin to log incorrect
data.
-- Don't allow PMI_TIME to be zero which will cause floating exception.
-- Fix purging of old reservation errors in database.
-- MYSQL - If starting the plugin and the database isn't up attempt to
connect in a loop instead of producing a fatal.
-- BLUEGENE - If IONodesPerMP changes in bluegene.conf recalculate bitmaps
based on ionode count correctly on slurmctld restart.
-- Fix step allocation when some CPUs are not available due to memory limits.
This happens when one step is active and using memory that blocks the
scheduling of another step on a portion of the CPUs needed. The new step
is now delayed rather than aborting with "Requested node configuration is
not available".
-- Make sure node limits get assessed if no node count was given in request.
-- Removed obsolete slurm_terminate_job() API.
-- Update documentation about QOS limits
-- Retry task exit message from slurmstepd to srun on message timeout.
-- Correction to logic reserving all nodes in a specified partition.
-- Added support for selecting AMD GPU by setting GPU_DEVICE_ORDINAL env var.
-- Properly enforce GrpSubmit limit for job arrays.
-- CRAY - fix issue with using CR_ONE_TASK_PER_CORE
-- CRAY - fix memory leak when using accelerators
* Changes in Slurm 2.6.5
========================
-- Correction to hostlist parsing bug introduced in v2.6.4 for hostlists with
more than one numeric range in brackets (e.g. rack[0-3]_blade[0-63]").
-- Add notification if using proctrack/cgroup and task/cgroup when oom hits.
-- Corrections to advanced reservation logic with overlapping jobs.
-- job_submit/lua - add cpus_per_task field to those available.
-- Add cpu_load to the node information available using the Perl API.
-- Correct a job's GRES allocation data in accounting records for non-Cray
systems.
-- Substantial performance improvement for systems with Shared=YES or FORCE
and large numbers of running jobs (replace bubble sort with quick sort).
-- proctrack/cgroup - Add locking to prevent race condition where one job step
is ending for a user or job at the same time another job stepsis starting
and the user or job container is deleted from under the starting job step.
-- Fixed sh5util loop when there are no node-step files.
-- Fix race condition on batch job termination that could result in a job exit
code of 0xfffffffe if the slurmd on node zero registers its active jobs at
the same time that slurmstepd is recording the job's exit code.
-- Correct logic returning remaining job dependencies in job information
reported by scontrol and squeue. Eliminates vestigial descriptors with
no job ID values (e.g. "afterany").
-- Improve performance of REQUEST_JOB_INFO_SINGLE RPC by removing unnecessary
locks and use hash function to find the desired job.
-- jobcomp/filetxt - Reopen the file when slurmctld daemon is reconfigured
or gets SIGHUP.
-- Remove notice of CVE with very old/deprecated versions of Slurm in
news.html.
-- Fix if hwloc_get_nbobjs_by_type() returns zero core count (set to 1).
-- Added ApbasilTimeout parameter to the cray.conf configuration file.
-- Handle in the API if parts of the node structure are NULL.
-- Fix srun hang when IO fails to start at launch.
-- Fix for GRES bitmap not matching the GRES count resulting in abort
(requires manual resetting of GRES count, changes to gres.conf file,
and slurmd restarts).
-- Modify sview to better support job arrays.
-- Modify squeue to support longer job ID values (for many job array tasks).
-- Fix race condition in authentication credential creation that could corrupt
memory. (NOTE: This race condition has existed since 2003 and would be
exceedingly rare.)
-- Slurmstepd variable initialization - Without this patch, free() is called
on a random memory location (i.e. whatever is on the stack), which can
result in slurmstepd dying and a completed job not being purged in a
timely fashion.
-- Fix slurmstepd race condition when separate threads are reading and
modifying the job's environment, which can result in the slurmstepd failing
with an invalid memory reference.
-- Fix erroneous error messages when running gang scheduling.
-- Fix minor memory leak.
-- scontrol modified to suspend, resume, hold, uhold, or release multiple
jobs in a space separated list.
-- Minor debug error when a connection goes away at the end of a job.
-- Validate return code from calls to slurm_get_peer_addr
-- BGQ - Fix issues with making sure all cnodes are accounted for when mulitple
steps cause multiple cnodes in one allocation to go into error at the
same time.
-- scontrol show job - Correct NumNodes value calculated based upon job
specifications.
-- BGQ - Fix issue if user runs multiple sub-block jobs inside a multiple
midplane block that starts on a higher coordinate than it ends (i.e if a
block has midplanes [0010,0013] 0013 is the start even though it is
listed second in the hostlist).
-- BGQ - Add midplane to the total_cnodes used in the runjob_mux plugin
for better debug.
-- Update AllocNodes paragraph in slurm.conf.5.
* Changes in Slurm 2.6.4
========================
-- Honor ntasks-per-node option with exclusive node allocations.
-- sched/backfill - Prevent invalid memory reference if bf_continue option is
configured and slurm is reconfigured during one of the sleep cycles or if
there are any changes to the partition configuration or if the normal
scheduler runs and starts a job that the backfill scheduler is actively
working on.
-- Update man pages information about acct-freq and JobAcctGatherFrequency
to reflect only the latest supported format.
-- Minor document update to include note about PrivateData=Usage for the
slurm.conf when using the DBD.
Loading
Loading full blame...