Better default memory assignement behavior
For Spark, Flink, Hadoop, etc, where we have to explicitly mention master and worker memory; a better way to assign default values should be developed. Currently, for:
- Spark: 100% memory is assigned to Worker, 4g is assigned to Executor, 4g is assigned to Driver
- Flink: No memory is derived/assigned from the slurm job
- Hadoop: No memory is derived/assigned from the slurm job
There are many cases such distributed frameworks could be set. But we can cover one case as default functionality. We can cover the case where only single node is used for launching. So we can use 20% of Node memory for master processes and 80% for worker processes. User can later modify these values if he/she wants to use frameworks in multi-node case.