Notify user if he tries to start a job step with a higher pack group
  value than the number available in the job allocation. Previous
  logic would abort the srun.

Modify sbcast command and srun's --bcast option to support heterogeneous
      jobs.
bug 4099

Set SLURM_NTASKS environment variable to reflect global task count
(needed by MPI).

Set SLURM_PROCID environment variable to reflect global task rank
(needed by MPI).

If srun lacks application specification for some component, the next one
      specified will be used for earlier components.

When the "scontrol update jobid=#" command specifies a pack job
leader, then modify all components of the pack job. To modiify only
the pack job leader, specify "scontrol update jobid=#+0".

Properly set up debugger symbols if an srun command tries to
  launch multiple, different applications in different components
  of a heterogeneous job (i.e. a different application in each
  pack job component).

Add pack_job_id and pack_job_offset to accounting database.
Modified sacct to accept pack job ID specification using "#+#" notation.
Modified sstat to accept pack job ID specification using "#+#" notation.

Adds assocation and QOS limits for the pack job as a whole

This is a work in progress, not ready for use yet.

Underlying logic not yet available, just the new option parsing and
  documentation.

Fix some parsing issues with respect to memory management, avoid
  memory over-writing and double free
Create one step per pack job (no work on launch yet)

Document which srun-specific options will be propagated between
  job steps rather than reset. Also propagate checkpoint options
  for consistency with sbatch.

This will route a heterogeneous job submitted to multiple clusters
  to the one cluster able to start all compoments fastest.

Work still needed on scheduling, launch, env vars, etc.
This just creates the job records

Still need to document the API, flesh out slurmctld response and
  more work on sbatch logic

Make _opt_default() function in salloc and sbatch reetrant.
Re-initializing variables used to hold changable fields for
each component of the heterogeneous job.