Create updated compatibility tests of tools that will run on PRs

[alyssadai]

Context

As we start having more nodes in our network that are maintained by others, and also we are hosting more services, it becomes increasingly important to make sure that we are intentional about introducing breaking changes and have plans in place on how to deal with them. The first step here is to make sure that we know if we are about to unintentionally introduce a breaking change, and then not do that.

Why

We have an existing simple compatibility test that runs on a cronjob and checks that all nightly tools and latest tools do not produce errors when launched using our full stack recipe.

However, to avoid releasing breaking changes unintentionally that will result in an incompatible deployment or break our public services, we should have a test that runs before changes are merged (i.e., on PRs) that will signal to us if we are safe to release the change or not.

Outcomes

• We have compatibility tests that run:
  • automatically on PRs
    • once with the PR branch version of tool + latest of all others, to give a signal if it’s safe to release after/on PR merge
    • once with PR branch version of tool + nightly of all others
  • once a day / on some cronjob, with all latest versions as a sanity check
    • if this passes, should ideally write current (confirmed) versions of tools to a file
• For now, the tests will run on the GH runner, and internally spin up a stack including all services (deployment recipe + CLI)
• Eventually, we want to move the tool stack that tests will run on to a staging area

What

• [ ]