Do a call of select_g_reconfigure() on Cray systems

immediately before attempting to schedule jobs.
04_Cray-INVENTORY-directly-before-schedule.diff

select/cray: do the inventory immediately before each schedule

This puts the Basil inventory immediately before each (backfill) schedule. 

Having considered multiple alternatives, this is the most robust and least
wasteful solution. The reason is that ALPS keeps internal node state, which
can be changed
 * by the administrator (xtprocadmin),
 * by the node health checker programs (setting some nodes into 'suspect'),
 * by ALPS itself.

Tracking this periodically, e.g. every HealthCheckInterval, may mean to miss
some state changes. The result would not be a crash, but a subsequently
failed ALPS reservation, which would require to undo some of the slurm state.

FIXME: since we are not using this, we have not yet considered wiki/wiki2.
       Possible places to update these are:
       - run periodic checkes every HealthCheckInterval (sub-optimal),
       - change plugins/sched/wiki{,2}/get_nodes.c

src/plugins/sched/backfill/backfill.c

@@ -480,6 +480,18 @@ static int _attempt_backfill(void)
 	}
 	this_sched_timeout = sched_timeout;
 
+#ifdef HAVE_NATIVE_CRAY
+	/*
+	 * Run a Basil Inventory immediately before setting up the schedule
+	 * plan, to avoid race conditions caused by ALPS node state change.
+	 * Needs to be done with the node-state lock taken.
+	 */
+	if (select_g_reconfigure()) {
+		debug4("backfill: not scheduling due to ALPS");
+		return SLURM_SUCCESS;
+	}
+#endif
+
 	if (slurm_get_root_filter())
 		filter_root = true;
 

src/slurmctld/job_scheduler.c

@@ -386,6 +386,20 @@ extern int schedule(uint32_t job_limit)
 		return SLURM_SUCCESS;
 	}
 
+#ifdef HAVE_NATIVE_CRAY
+	/*
+	 * Run a Basil Inventory immediately before scheduling, to avoid
+	 * race conditions caused by ALPS node state change (caused e.g.
+	 * by the node health checker).
+	 * This relies on the above write lock for the node state.
+	 */
+	if (select_g_reconfigure()) {
+		unlock_slurmctld(job_write_lock);
+		debug4("sched: not scheduling due to ALPS");
+		return SLURM_SUCCESS;
+	}
+#endif
+
 	failed_parts = xmalloc(sizeof(struct part_record *) *
 			       list_count(part_list));
 	save_avail_node_bitmap = bit_copy(avail_node_bitmap);