The road towards a Library Sharing MVP

[radeusgd]

We want to allow our users to share their projects as libraries. I will analyze two approaches that we can take to get us to this state in the minimal amount of work - relying on the existing infrastructure we have for this purpose, and listing the things that need to be added.

The workflow we want to support

1. The user works on a project and creates functions within it using Ctrl+G.
2. The user wants to use these functions from another project.
3. The user clicks 'Share' (or 'Publish' or some appropriate option) on the project (in the dashboard or in the top bar). Possibly we may allow the user to select whether to share only for their own projects (Only me) or on the organization level (Organization). Public sharing could be a next step.
4. Now, the user can switch to some other project. There the user can open a list of available libraries (this list will include any libraries that the user Shared).
5. The user can select a library from this list and choose to add it to the current project.
6. From now on, the (non-private) functions from that added library should show up in the Component Browser and the user should be able to use them.
7. Extra feature that was mentioned:
   i. the ability to somehow access libraries the user has 'Published' in the Cloud from within a local Enso installation.

Possible solutions

We have 2 possible approaches we can take to make the workflow described above possible.

Our library resolution system contains two concepts of library sources: local user-editable libraries on the filesystem, and libraries published in a repository. The two solutions rely on either of these systems to provide an MVP.

'Local' solution

Enso is able to automatically include libraries from $ENSO_HOME/libraries if the package.yaml contains prefer-local-libraries: true.

In this scenario:

• Publishing a library amounts to copying it into $ENSO_HOME/libraries (if needed we can have more than one directory for local libraries, we just need to override $ENSO_LIBRARY_PATH).
• Discovering libraries can be done by querying the Language Server's library/listLocal endpoint.
• Adding a library to a project amounts to adding import <namespace>.<name> or from <namespace>.<name> import all to the Main.enso file.

This scenario has limited support for sharing a library from Enso Cloud into local Enso installation. To achieve it, we would need the IDE to be able to query libraries available in the cloud 'directory' and download them into the local $ENSO_HOME/libraries.

'Repository' solution

This uses the dedicated infrastructure for sharing libraries, thus it may require a bit more support but is also much more future proof.

In this scenario:

• Publishing a library amounts to uploading the project files into some place that is accessible through HTTP.
  • The library repository is prepared to keep library versions - the structure is root/<namespace>/<name>/<version> (see docs).
  • We already have a Library Uploader which does all necessary packaging.
• Libraries available for the user need to be defined in an edition file, accessible over HTTP.
• Discovering libraries can be done by reading the specific edition file and querying the Language Server's editions/listDefinedLibraries.
• Adding a library to a project amounts to adding import <namespace>.<name> or from <namespace>.<name> import all to the Main.enso file, as long as it is defined in the current edition.

In this scenario we can use exactly the same logic for uploading and downloading the libraries on both Cloud and Local installations. Alternatively, in the cloud we can speed things up slightly by avoiding HTTP requests and pre-populating library caches directly on the mounted volumes - but that is rather an optimization that may be done later and is not strictly needed for the MVP.

The major difference between this and the Local solution is how we discover available libraries. In the Local solution we simply look at what projects are available on the filesystem in directories determined by ENSO_LIBRARY_PATH. In this solution, the libraries are determined by the edition and can come from various repositories.

What we need to get this working

Common functionality

Both solutions will need some common part in terms of the UI:

• a button somewhere allowing to Share/Publish a project, together with some dialog allowing to customize any options (a dialog telling the user that any data associated with the project is also published was mentioned);
• a list of available libraries together with buttons that allow to add the library to a project;
• the capability to add a library (usually just appending an import statement at the beginning of Main.enso).

Moreover, both will need some additional support for authentication, especially to support accessing private libraries from the Local installations. We will need to extend the system for downloading libraries to be able to specify additional headers containing any necessary authentication tokens, probably set via environment variables. For that we will need:

• the IDE to pass the authentication tokens to the Language Server; through environment variables or a protocol message;
• the library resolver / uploader; any component that needs the authentication - to be able to set the additional headers for its requests.

'Local' solution

1. The cloud needs to expose some shared storage mounted at $ENSO_HOME/libraries (or some other location), used for storing libraries.
   i. It can provide one or more volumes, e.g. we could have one volume for Personal (Only me) libraries and another one for Organization-level libraries (shared between users). Each of these volumes need to be mounted for each instance run by the user.
   ii. If there are more volumes, we need to set ENSO_LIBRARY_PATH to allow Enso to discover them.
2. The cloud needs logic that will copy the data of the current project into one of these shared library volumes when the user clicks 'Publish'.
3. The local IDE will need a similar, although simpler, logic to copy the current project data into $ENSO_HOME/libraries when the user clicks 'Publish'.
4. This solution relies on resolving local libraries so we need to:
   i. either ensure that new projects get prefer-local-libraries: true,
   ii. or change the resolution logic so that even if prefer-local-libraries: false, if a library is not found in the edition, a local lookup is still performed (IMO this is better).
5. If we want the extra feature of sharing libraries from Cloud to local Enso installation, we need some additional way for the user to access the shared storage volume (i.e. through S3). However, contrary to the 'Repository' solution, the whole point of 'Local' solution means we cannot rely on the Enso built-in downloader, so we'd need the IDE to somehow download any new libraries shared from Cloud and put them on the Local filesystem.
   i. This is extra hacky. I think the Local solution is a great MVP that can be developed very quickly, without this feature. If we want this extra feature though, the 'Repository' solution would be a much better choice IMHO.

'Repository' solution

1. We need to expose a shared storage (likely S3 buckets) for storing the published libraries.
   i. We probably want to have separate buckets for each user and organization, to keep private libraries private.
2. We need an ability to upload a project to the bucket.
   i. The client-side uploader that handles all necessary packaging is already implemented.
   ii. We need some 'other side' to receive these packages.
   • We have a simple server implemented in Node.JS, but it is rather not production ready.
   • We'd probably need some AWS Lambda that can receive the files and forward them into an appropriate S3 bucket.
3. We need some way to define an edition for each user that will contain the libraries 'currently' available to them.
   i. We can use an AWS lambda that would gather libraries available to the particular user (e.g. listing objects in their 'personal' bucket and merging these with the objects available in the user organization's bucket), and generate an edition file from these.
4. Currently the edition files are assumed to be immutable, so they are downloaded only once and cached. We can easily disable the caching, so that the edition is re-downloaded every time the project is opened to ensure that we get a list of most up-to-date libraries.
   i. The edition file is not meant to be re-loadable during runtime, so the newly added libraries only show up when the project is restarted. This is by design (if the edition file changed a version of a particular library, we would need to unload the old library version and load a new one which is extremely hard to do in a live environment that may already be using this library).
   ii. For the longer term, the idea was indeed that editions were supposed to be immutable. But we can easily disable this temporarily. We can also (for future) create a notion of stable editions that are immutable once published, and unstable editions that are expected to be updated and need to be re-downloaded.
5. Updating libraries:
   i. We first thought that for first version we could keep a pinned 1.0.0 version and overwrite library files.
   ii. That will not work well, because again - the library system assumes that library versions are immutable. So once a version 1.0.0 of a library is downloaded, it is cached locally and will not be redownloaded unless the cache is manually purged.
   • On the cloud, we could circumvent this by purging the cache on each run (although that will also mean the libraries are always redownloaded which can increase the startup time).
   • I don't think there is a good solution for this locally.
     iii. Thus, I think instead, we should always increment the minor version number when uploading a new version (1.0.0, 1.0.1, 1.0.2, ...). For now, our logic for generating the edition file listing libraries available to the user can always look for the highest version number available in the bucket and list that. By increasing the numbers for each new version, we will ensure that the library is re-downloaded when a new version is published.
   • IMO this does not increase the complexity. All we need is to bump the version number before upload and ensure the logic for generating the edition file uses the highest available number.
   • For the MVP we'd ignore the problem of inter-version dependencies and just use latest available version of each available library.
6. Reporting progress
   i. Since this solution relies more on downloading libraries thru HTTP; and likely the first version will not retain caches in the cloud (a very useful optimization, but not strictly necessary for MVP), it would be good to notify the user about pending downloads.
   ii. All functionality is already there in the Language Server. We would just need the IDE to handle the task/* messages (especially task/progress-update) and display some kind of spinner/progress bar indicating the pending action.
   iii. This is not strictly necessary for the MVP to work, but I think it would be a helpful feature to make the UX better.

[radeusgd]

RFC @wdanilo @jdunkerley @JaroslavTulach @hubertp

My 2 cents:

The local solution requires less work for a minimal solution - it requires the bare minimum of moving project files around and a slight adaptation of resolution logic. It can give us very basic sharing very quickly - but then we hit a wall. It is much more 'hacky' and will need to be redone once we want to include any versioning. If we want any more advanced features, like sharing from cloud to local installations or vice-versa, the Local solution does not hold much benefits anymore and becomes more complex than necessary - so at this point I think the Repository one may be a better pick.

In general, I'd choose the Local one if we want to get a very simple MVP very quickly.
For any more advanced features (like sharing between Cloud and Local), the Repository seems like a better pick - it is much more future-proof and it does not require that much more work to get a most basic MVP up.

[JaroslavTulach]

Local Solution, point 5 - isn't an option. We shall not hack it. We need Repository Solution to implement this point 5 functionality. However I don't see a reason why not start implementing Local Solution, points 1 to 4 and only then start implementing the Repository Solution when needed. Thus I'd say MVP is Local Solution, points 1 to 4

Whether to go with Local Solution or directly start with Repository Solution depends on the cloud guys: do they prefer mounting for now or they want full featured HTTP server?

[PabloBuchu]

I vote for Repository Solution. More work but is scales better and align with the idea of creating public marketplace for users functions / projects. In Cloud we are also preparing to move out from ec2 based architecture towards ecs and ephemeral storages so we prefer https server.

[JaroslavTulach]

Here is a short how to describing setup of local node.js based server, creating necessary configuration files, uploading a project as a library to the local server and finally using the uploaded library from another project. Everything seems to work.

[JaroslavTulach]

When thinking about the library sharing system, I'd like to mention that it'd be beneficial if the IDE could provide a repository with its own libraries and use them in visualizations.

[radeusgd]

I think this is a separate concern from user library sharing, but definitely needs a good solution as well.

IMHO a simple solution could be:

1. Ensure our editions specify:

   - name: Standard.Visualization
     version: local
   

   making sure that the locally available version is used instead of a version specific to the engine version (like 2023.123).
2. The IDE can contain the library as part of its distribution.
3. The path to this local version of Standard.Visualization should be added to ENSO_LIBRARY_PATH.
   i. This will probably need a slight change of logic in how ENSO_LIBRARY_PATH works, since currently overriding it discards the defaults, but we may want to add something to the path without overriding the defaults. Maybe we need an ENSO_EXTENDED_LIBRARY_PATH.

Alternatively, we can do:

1. Fixed static version for the library:

   - name: Standard.Visualization
     version: 1.0.0-IDE
   

2. Distribute it as above as part of IDE distribution.
3. Ensure that IDE sets ENSO_AUXILIARY_LIBRARY_CACHES variable to a directory that will contain its distribution Standard.Visualization library.
4. We should ensure that Standard.Visualization is no longer distributed as part of engine distribution, but now as part of IDE distribution.

I guess the latter may be even better.