Accurately estimate the memory available on the executor

[calebwin]

Right now we have a fixed maximum memory usage per worker based on the instance type - and because this isn't accurate we end up assuming that only 30% of it is available which leads to worse performance than needed. Instead, we should have each job that is created, compute the minimum free_memory = begin GC.gc(true); Sys.free_memory end across all workers using min_free_memory = MPI.Reduce(min, free_memory, 0, MPI.COMM_WORLD) and on the head node (where the MPI.Comm_rank(MPI.COMM_WORLD) == 0) we should send this to the client and then send this max_worker_memory to each call to evaluate which stores it in JobInfo.

[cailinw]

We should ensure that this is not an issue
https://discourse.julialang.org/t/how-to-get-current-julia-process-memory-usage/41734