RFC: clarify the role of "code"

[lukasheinrich]

To reiterate the 4 questions:

1. What is your input data?
2. What is your environment?
3. Which code analyses it?
4. Which steps did you take?

the role of "code" is not really clear yet. One the one hand it is being treated just like the data by being bind-mounted in, on the other hand a lot of the analysis code (if not all of it) is already captured in the environment (i.e. docker image)

I think it would be useful to collect thoughts to clarify this. In my understanding

code (from 3) +environment (from 4)-> runtime (container_image or filesystem)

there are multiple ways to arrive at this image

• build the image via Dockerfile
• various source2image solutions like

buildpacks (https://buildpacks.io)
source2image (https://github.com/openshift/source-to-image.git)
repo2docker (https://github.com/jupyter/repo2docker)
ad-hoc mounting in the data (only works for dynamic languages -- which is what we do now)
(docker run -v code:/code)

For a workflow there might be multple code bases (for each workflow), multiple images, etc. So we should try to define this carefully

Once the and then the data (from 1) ) is made available by mounting in a directory and running the step in 4) :

ctr run -v data:/data container_image "./mycommand.sh /data/in.root /data/out.root"

[lukasheinrich]

I think for me the priority would be that in the REANA codebase, these things are separated already at the code level, such that the code factors the concerns appropriately. I.e. we should bind mount data and code separately.

[tiborsimko]

Quoting some of my past musings from the chatroom:

We distinguish between the "compile-time code" that builds the container, so
to speak, and the "run-time code" that can be mounted "live" just for the
execution time. Think R environment and running small R scripts, or ROOT
environment and running small macros to produce plots. In this use case, the
user code does not necessarily need to build a new image, it can be
"mounted" into an already existing runtime environment. The use case is a
researcher with simple needs that may not know much about containers and
images and workflows. In this use case, we are basically avoiding the need
to build images.... Make sense if many people can easily work on top of
already-prepared environment image. There are pros and cons obviously. (And
the more common use case would be to work on top of alerady-prepared base
image, creating Dockerfile with ADD . /code during image build time, where
CWL workflow graphs would look naturally pretty...)

Another quote:

Regarding mounting code, for REANA we have foreseen two use cases: (1) users
using images prepared by others; (2) users writing their own images. For the
former use case, we are actually recommending to mount the code and inputs
into the container [...] The assumption is that the number of
people-using-images may be significantly higher than the number of
people-creating-images, so mounting code may simplify things for the
newcomers. 

Another quote:

We'd distinguish between compile-time code (that is helping to build and
define the environment) and the runtime code (that only uses it). Hence the
"code" section outside the "environment" section, in case the said code does
not contribute anything to "build" the environment, but simply "uses" a
provided environment. The distinction can be useful in case you have 1-2
researchers "providing" environments and 10-15 researchers "using" them,
without modification needs... That kind of compile-time vs run-time code.

Since the time of these musings, the build-time-code vs run-time-code
distinction still holds, obviously -- as you also describe it.

However one of major novelties of REANA v0.3.0 is that we have abolished the
separation between code and data runtime inputs in the reana.yaml structure
and also in the CLI usage scenarios, so that the old:

$ reana-client code upload ./mycode.C
$ reana-client inputs upload ./mydata.csv

is now simply:

$ reana-client upload ./mycode.C 
$ reana-client upload ./mydata.csv

Basically, the users are still fully free to select any kind of directory
structure they want for their analyses -- to separate code and data, to not
separate them, to introduce subdirectories etc -- the REANA CLI won't try to map
these into "code" or "data" category anymore. All the code organisation
techniques would work the same.

This will hopefully alleviate possible confusions regarding build-time-code vs
run-time-code.

P.S. See also reanahub/reana-demo-worldpopulation#23
that discusses what might be a good default for various demo examples.

P.S. Note that REANA v0.2.0 was combining code and data inputs together and into
a single mount under the hood anyway...