[JS-3] Resource sharing across jobs #6

wynot12 · 2017-12-22T06:16:22Z

Resolves #3.

This PR introduces resource-sharing across jobs in JobServer.
Specifically, there are two new components:

ResourcePool: a resource pool for JobServer to maintain whole cluster resource.
SchedulerImpl: a new scheduler implementation that immediately launches incoming jobs, simply using all executors

Additional changes:

Dolphin on JobServer only supports PS-collocation mode.
Removes FIFOScheduler, an old scheduler implementation, since we will use it anymore.
named parameter's short names are renamed since there are conflicts across packages.
the number of worker threads in pregel and dolphin can be configured through command-line; otherwise, the system will use Runtime.getRuntime().availableProcessors().

…c-jobserver

* introduce combiner * Minor fix (grammar, comments)

wynot12 · 2017-12-22T06:17:10Z

common/src/main/java/edu/snu/cay/common/param/Parameters.java

@@ -82,6 +82,11 @@
  public final class OptimizationBenefitThreshold implements Name<Double> {
  }

+  @NamedParameter(doc = "Whether the hyper-thread is enabled, which determines the proper number of trainer threads.",
+      short_name = "hyper_thread_enabled", default_value = "false")
+  public final class HyperThreadEnabled implements Name<Boolean> {


It has been moved from DolphinParameters, since Pregel will use it, too.

wynot12 · 2017-12-22T06:17:57Z

jobserver/bin/run_addinteger.sh

@@ -14,7 +14,7 @@
 # limitations under the License.

 # EXAMPLE USAGE
-# ./run_addinteger.sh -num_workers 3 -number_workers 3 -number_servers 2 -max_num_epochs 10 -num_mini_batches 15 -num_worker_blocks 15 -delta 4 -num_keys 100 -num_training_data 50 -num_test_data 5 -compute_time_ms 30 -max_num_eval_local 5 -input run_addinteger.sh -optimizer edu.snu.cay.dolphin.optimizer.impl.EmptyPlanOptimizer -optimization_interval_ms 3000 -delay_after_optimization_ms 10000 -server_metric_flush_period_ms 1000 -num_initial_batch_metrics_to_skip 3 -min_num_required_batch_metrics 3
+# ./run_addinteger.sh -number_workers 3 -num_dolphin_workers 3 -num_servers 2 -max_num_epochs 10 -num_mini_batches 15 -num_worker_blocks 15 -delta 4 -num_keys 100 -num_training_data 50 -num_test_data 5 -compute_time_ms 30 -max_num_eval_local 5 -input run_addinteger.sh -optimizer edu.snu.cay.dolphin.optimizer.impl.EmptyPlanOptimizer -optimization_interval_ms 3000 -delay_after_optimization_ms 10000 -server_metric_flush_period_ms 1000 -num_initial_batch_metrics_to_skip 3 -min_num_required_batch_metrics 3


Note that number_workers is for example parameter (ExampleParameters.NumWorkers).

wynot12 · 2017-12-22T06:18:32Z

jobserver/bin/run_gbt.sh

@@ -15,7 +15,7 @@

 # EXAMPLE USAGE
 # Classification
-# ./run_gbt.sh -num_workers 2 -number_servers 1 -local true -input sample_gbt -max_num_eval_local 3 -test_data_path file://$(pwd)/sample_gbt_test -max_num_epochs 50 -num_mini_batches 10 -num_worker_blocks 10 -features 784  -lambda 0 -gamma 0 -max_depth_of_tree 5 -leaf_min_size 3 -num_keys 5 -timeout 200000 -num_trainer_threads 2 -optimizer edu.snu.cay.dolphin.optimizer.impl.EmptyPlanOptimizer -optimization_interval_ms 3000 -delay_after_optimization_ms 10000 -metadata_path file://$(pwd)/sample_gbt.meta -opt_benefit_threshold 0.1 -server_metric_flush_period_ms 1000 -moving_avg_window_size 0 -metric_weight_factor 0.0 -num_initial_batch_metrics_to_skip 3 -min_num_required_batch_metrics 3
+# ./run_gbt.sh -num_dolphin_workers 2 -num_servers 1 -local true -input sample_gbt -max_num_eval_local 3 -test_data_path file://$(pwd)/sample_gbt_test -max_num_epochs 50 -num_mini_batches 10 -num_worker_blocks 10 -features 784  -lambda 0 -gamma 0 -max_depth_of_tree 5 -leaf_min_size 3 -num_keys 5 -timeout 200000 -num_trainer_threads 2 -optimizer edu.snu.cay.dolphin.optimizer.impl.EmptyPlanOptimizer -optimization_interval_ms 3000 -delay_after_optimization_ms 10000 -metadata_path file://$(pwd)/sample_gbt.meta -opt_benefit_threshold 0.1 -server_metric_flush_period_ms 1000 -moving_avg_window_size 0 -metric_weight_factor 0.0 -num_initial_batch_metrics_to_skip 3 -min_num_required_batch_metrics 3


num_workers -> num_dolphin_workers is the only change.

wynot12 · 2017-12-22T06:19:31Z

jobserver/bin/start_jobserver.sh

@@ -14,7 +14,7 @@
 # limitations under the License.

 # EXAMPLE USAGE
-# ./start_jobserver.sh -num_total_resources 10 -max_num_eval_local 10 -local true -timeout 300000 -scheduler edu.snu.cay.jobserver.driver.FIFOJobScheduler
+# ./start_jobserver.sh -max_num_eval_local 5 -local true -timeout 300000 -scheduler edu.snu.cay.jobserver.driver.SchedulerImpl -num_executors 5 -executor_mem_size 128 -executor_num_cores 1 -executor_num_tasklets 4 -handler_queue_size 1024 -sender_queue_size 1024 -handler_num_threads 2 -sender_num_threads 2


JobServer requires executor spec arguments, since it will spawn all executors upon start.

wynot12 · 2017-12-22T06:21:43Z

jobserver/bin/submit_gbt.sh

@@ -15,7 +15,7 @@

 # EXAMPLE USAGE
 # Classification
-# ./submit_gbt.sh -num_workers 2 -number_servers 1 -input file://$(pwd)/sample_gbt -test_data_path file://$(pwd)/sample_gbt_test -max_num_epochs 50 -num_mini_batches 10 -num_worker_blocks 10 -init_step_size 0.1 -features 784  -lambda 0 -gamma 0 -max_depth_of_tree 5 -leaf_min_size 3 -num_keys 5 -num_trainer_threads 2 -metadata_path file://$(pwd)/sample_gbt.meta -server_metric_flush_period_ms 1000
+# ./submit_gbt.sh -input file://$(pwd)/sample_gbt -test_data_path file://$(pwd)/sample_gbt_test -max_num_epochs 50 -num_mini_batches 10 -num_worker_blocks 10 -features 784 -lambda 0 -gamma 0 -max_depth_of_tree 5 -leaf_min_size 3 -num_keys 5 -num_trainer_threads 2 -metadata_path file://$(pwd)/sample_gbt.meta -server_metric_flush_period_ms 1000


All submit scripts do not need executor-request parameters.
Scheduler will determine which and how many executors for a job.

wynot12 · 2017-12-22T06:27:58Z

jobserver/src/main/java/edu/snu/cay/dolphin/core/master/DolphinMaster.java

@@ -168,13 +168,12 @@ public MasterSideMsgHandler getMsgHandler() {
  public void start(final List<AllocatedExecutor> servers, final List<AllocatedExecutor> workers,
                    final AllocatedTable modelTable, final AllocatedTable trainingDataTable) {
    try {
-      servers.forEach(server -> metricManager.startMetricCollection(server.getId(), getServerMetricConf()));
+      // TODO #5: tasklet-level metric collection


I've just removed metric collection for server tasklets, since we don't use metrics from servers.
But multiple worker tasklet from different jobs may incur problems, I'll fix the problem in a PR for #5.

Actually we don't need even server tasklet at all. But I'd like to minimize the code change. So let's tackle this issue when it becomes urgent.

wynot12 · 2017-12-22T06:30:03Z

jobserver/src/main/java/edu/snu/cay/dolphin/jobserver/DolphinJobEntity.java


-    return executorGroups;
-  }
+    // support collocation only


Here's the only assumption of PS-collocation.

yunseong

Thanks for the hard work! The PR looks great.
I'll merge once tests pass (triggered by my minor fix).

wynot12 added 30 commits December 11, 2017 02:45

commit

40e9aa5

merge pregel into the dolphin module

82155de

Change pregel not to use CentComm

56505bf

generic jobserver that runs pregel and dolphin jobs together

5eff8f3

commit

0ed686a

merge pregel into the dolphin module

f518199

Change pregel not to use CentComm

f303b5f

generic jobserver that runs pregel and dolphin jobs together

c111169

Merge branch 'generic-jobserver' of github.com:snuspl/cay into generi…

6c23e7b

…c-jobserver

[CAY-1208] Introduce combiner in Pregel (#1279)

59938fd

* introduce combiner * Minor fix (grammar, comments)

Merge branch 'master' into generic-jobserver

b724369

Merge branch 'master' into generic-jobserver

d5869f3

shortest-path app launch files

0f6d5ca

minor refine

618a389

reorg packages

87d26b3

jobserver module

225a612

nit

888c017

fix style error

b1f9600

Merge branch 'master' into generic-jobserver

e036201

minor

ff6e658

restore package name ps to dolphin

c44a23e

commit

824bfc3

nit

5dff1ee

Decoule table/executor setup from JobDispatcher

d181563

Merge branch 'generic-jobserver' into generic-jobserver-resource-sharing

ea06c04

Merge branch 'master' into generic-jobserver-resource-sharing

ecc70f3

Fix concurrency bug in pregel

3991aa3

resolve remaining conflicts

3d129c0

minor updates

1e4b080

ready for PR

17240cd

wynot12 added the jobserver label Dec 22, 2017

wynot12 requested a review from yunseong December 22, 2017 06:16

wynot12 commented Dec 22, 2017

View reviewed changes

minor modification

5d6f0a4

yunseong approved these changes Dec 22, 2017

View reviewed changes

yunseong merged commit ec97ff1 into master Dec 22, 2017

yunseong deleted the generic-jobserver-resource-sharing branch December 22, 2017 11:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JS-3] Resource sharing across jobs #6

[JS-3] Resource sharing across jobs #6

wynot12 commented Dec 22, 2017 •

edited by yunseong

Loading

wynot12 Dec 22, 2017

wynot12 Dec 22, 2017

wynot12 Dec 22, 2017

wynot12 Dec 22, 2017

wynot12 Dec 22, 2017

wynot12 Dec 22, 2017 •

edited

Loading

wynot12 Dec 22, 2017

yunseong left a comment

[JS-3] Resource sharing across jobs #6

[JS-3] Resource sharing across jobs #6

Conversation

wynot12 commented Dec 22, 2017 • edited by yunseong Loading

wynot12 Dec 22, 2017

Choose a reason for hiding this comment

wynot12 Dec 22, 2017

Choose a reason for hiding this comment

wynot12 Dec 22, 2017

Choose a reason for hiding this comment

wynot12 Dec 22, 2017

Choose a reason for hiding this comment

wynot12 Dec 22, 2017

Choose a reason for hiding this comment

wynot12 Dec 22, 2017 • edited Loading

Choose a reason for hiding this comment

wynot12 Dec 22, 2017

Choose a reason for hiding this comment

yunseong left a comment

Choose a reason for hiding this comment

wynot12 commented Dec 22, 2017 •

edited by yunseong

Loading

wynot12 Dec 22, 2017 •

edited

Loading