cluster: optimise workflow start-up time

[diegodelemos]

When a new workflow is created in REANA, the REANA-Workflow-Engines (CWL, Yadage and serial) are created on demand (more details here). Because of this, the delay between the reana-client run and the real workflow starting up depends on, at least, two different factors which could be optimised:

• (1) Kubernetes job scheduling (Workflow-Engines are spawned as Kubernetes jobs)
• (2) Actual code inside the workflow engines

So if we take a little walk-through: when REANA-Workflow-Controller receives a request to start a workflow, it will delegate on Kubernetes to spawn a job (1) running those commands to start the real workflow instances. Once the job is created, a CLI application is initialised and the instructions are executed (2).

To address (1), job scheduling

Investigate how to tweak Kubernetes, a good starting point is https://kubernetes.io/blog/2017/08/kubernetes-meets-high-performance/.

To address (2), workflow engines code

All REANA-Workflow-Engines (CWL, Yadage and serial) expose their functionality through a CLI interface, i.e. CWL engine snippet:

...
@click.command()
@click.option('--workflow-uuid',
              required=True,
              help='UUID of workflow to be run.')
@click.option('--workflow-workspace',
              required=True,
              help='Name of workspace in which workflow should run.')
@click.option('--workflow-json',
              help='JSON representation of workflow object to be run.',
              callback=load_json)
@click.option('--workflow-parameters',
              help='JSON representation of parameters received by'
                   ' the workflow.',
              callback=load_json)
@click.option('--operational-options',
              help='Options to be passed to the workflow engine'
                   ' (i.e. caching).',
              callback=load_json)
def run_cwl_workflow(workflow_uuid, workflow_workspace,
                     workflow_json=None,
                     workflow_parameters=None,
                     operational_options={}):
...

See for example REANA-Workflow-Engine-CWL command line interface.

We should profile this code and find out what is the bottleneck. One of the problems that might causing the startup of the workflow to be a bit slow is Click initialisation. We can optimise Click or switch to argparse.