Persist messages over application restarts

[jleach]

As part of creating a highly available mediator we want to be able to persist messages over restarts of the pod. This should include a technology capable of scaling as we implement other high availability features such as horizontal pod scaling.

[jleach]

@genaris I was thinking of using this to base the functionality of this feature. My though was to just use Redis or the existing PostgreSQL as the message store for now. Thoughts

Cc: @TimoGlastra

[TimoGlastra]

Redis (AFAIK) is more for caching, and there's not always a guarantee messages will be persisted. So I lean more towards something like PostgreSQL. I think using the libraries from @genaris make a lot of sense.

Would be great if we can do this is coordination with @genaris to make sure we keep all the important features they implemented.

[TimoGlastra]

Also -- I think we need to update this repo to a monorepo structure to allow deployment of packages to npm.

We usually take this appraoch nowadays:

• apps/* -> all the apps for the repo. In this case the mediator will be the only app
• packages/* -> all the packages (that can be dependent upon by apps) for the repo. We can add the pacakges from @genaris his repo to this.

One thing I want to be cautious of is that we make sure we combine all the packages in a clean manner. We are combining work from Animo/Credebl/2060 and such things tend to become messy if not enough time is spent on cleanup.

I can help with the monorepo setup, as I have done this numerous times already for public and internal projects (set up apps, and release pipelines for the packages)

[jleach]

@TimoGlastra @genaris Should we bring the changes in here or just use them from a published (from 2060.io) npm repository? Also, I did have some questions about the complexity of that package. Rather that just having the mediator talk to the persistent storage, it abstracts this behind a server. It also uses both Redis and Mongo which gives us three backing stores once you include Postgres. It for sure gets the job done but I wonder if we need to go from 1 service to 4 just to persist messages?