Process and infrastructure for automatic nightly test execution

[lukasz-zimnoch]

We have to design the process and configure the infrastructure to be able to continuously deploy the whole system and execute end-to-end tests in a fully automatic way regardless of the host environment.

A possible solution could be:

1. Prepare a VM image using Packer consisting of the OS and all prerequisites required to run local-setup scripts.
2. Use Vagrant to set up a VM using the prepared image and run local-setup installation scripts with all auxiliary software.
3. Execute E2E tests on the running VM.

[pdyraga]

Why don't we use Docker just like we do for unit tests?

[lukasz-zimnoch]

Why don't we use Docker just like we do for unit tests?

First of all, using Packer and Vagrant doesn't mean we can't use Docker. This is a very flexible solution because Packer has multiple builders and Vagrant has multiple providers which can be used. I propose the VM-based approach which looks like:

Ubuntu box (VM image) -> Packer with Vagrant builder- > Ubuntu box + local-setup prerequisites -> Vagrant with Virtualbox provider -> Running VM with local-setup installed

But the pipeline can be also:

Ubuntu image -> Packer with Docker builder -> Ubuntu image + local-setup prerequisites -> Vagrant with Docker provider -> Running container with local-setup installed

My reasoning for the VM-based approach is as follows:

Local-setup env is far more complicated and much bigger than simple unit tests. There are multiple services and dependencies between them. Docker is primarily designed to use the service-per-container approach. Of course, it is possible to run multiple services but in our case, that will be probably more complicated than using a VM. Also, if we use Docker we will be also forced to use the Docker-in-Docker approach for Bitcoin Core and Electrum which can bring a new set of problems. Also, local-setup is not only for testing but can be also used for local development, and using a separate VM for development here gives more customization options.

[pdyraga]

Also, local-setup is not only for testing but can be also used for local development, and using a separate VM for development here gives more customization options.

Side note - without reading the rest of your comment - I think we should consider redoing local-setup into system-tests. All the work we are doing here right now is to support automatic end-to-end tests. It started as a local development setup repo but escalated quickly into something else.

[pdyraga]

I'll also tag @sthompson22 and @Shadowfiend for this issue, I am sure they will have some good ideas.

[sthompson22]

3. Execute E2E tests on the running VM.

Can we clarify what E2E tests mean here?

[sthompson22]

Quick thought here, want to write it down before I forget:

It would be ideal if the test suite that gets executed nightly also gets executed against the live Ropsten env. Reason is that having the black box and live env run the same tests may help identify issues that occur only in a live env/ at scale.

This is a bit hand wavy, depends on the tests that re run and how we surface the results / errors etc...just wanted to get it out there.

[sthompson22]

On nightly vs PR runs. Is there a particular reason we wouldn't run this setup against a PR, rather than a nightly build? I could see the build/test time being an issue..but if we have a failure on some test run in the night, we have to go through each of the PR's that were merged between last pass and failure?

Granted, the type of failure should hopefully help narrow where the issue is.

[pdyraga]

It would be ideal if the test suite that gets executed nightly also gets executed against the live Ropsten env. Reason is that having the black box and live env run the same tests may help identify issues that occur only in a live env/ at scale.

This is a bit hand wavy, depends on the tests that re run and how we surface the results / errors etc...just wanted to get it out there.

That's the plan 🙌 We want to start with a simple GCP-scheduled deposit+redemption flow (captured here keep-network/tbtc#703) and evolve it over time.

On nightly vs PR runs. Is there a particular reason we wouldn't run this setup against a PR, rather than a nightly build? I could see the build/test time being an issue..but if we have a failure on some test run in the night, we have to go through each of the PR's that were merged between last pass and failure?

Granted, the type of failure should hopefully help narrow where the issue is.

Would love to have it for each PR but setting up the environment and executing the test is... I think at least on hour? It doesn't have to be a nightly build, we can keep it executing in a loop if we want to.