In SLURM verstion 2.4, we now schedule jobs at priority=1 and no longer treat
it as a special case.

In cray/srun wrapper, only include aprun "-q" option when srun "--quiet"
option is used.

Here's what seems to have happened:

- A job was pending, waiting for resources.
- slurm.conf was changed to remove some nodes, and a scontrol reconfigure was done.
- As a result of the reconfigure, the pending job became non-runnable, due to "Requested node configuration is not available". The scheduler set the job state to JOB_FAILED and called delete_job_details.
- scontrol reconfigure was done again.
- read_slurm_conf called _restore_job_dependencies.
- _restore_job_dependencies called build_feature_list for each job in the job list
- When build_feature_list tried to reference the now deleted job details for the failed job, it got a segmentation fault.

The problem was reported by a customer on Slurm 2.2.7.  I have not been able to reproduce it on 2.4.0-pre3, although the relevant code looks the same. There may be a timing window. The attached patch attempts to fix the problem by adding a check to _restore_job_dependencies.  If the job state is JOB_FAILED, the job is skipped.

Regards,
Martin

This is an alternative solutionh to bug316980fix.patch

Added default_time field in partition records in Lua job submit
plugin.

Added a new Lua library name to try loading with dlopen() in Lua
based plugins.

Only report "gres/<name> lacks File parameter" if some nodes define
File AND this node does not AND (new part here) the GRES count on
this node is non-zero

If a time value to be printed (e.g. job run time) is negative then
print the value as "INVALID" rather than with negative numbers
(e.g. "-123--12:-12:-12").

Change default SchedulerParameters max_switch_wait field value from 60 to
300 seconds.

Conflicts:
	src/slurmd/slurmstepd/multi_prog.c
	src/slurmd/slurmstepd/multi_prog.h