[CI] Integrate E2E tests with GitHub CI #152

okdas · 2023-11-07T01:43:28Z

Summary

Allows to automatically run an E2E test on GitHub PR.

Human Summary

Changed GitHub workflows to run tests and a build in parallel;
Added a Kubernetes Job to run the test on the remote cluster;
Added E2E_DEBUG_OUTPUT environment variable to be able to troubleshoot E2E test failures;

AI Summary

Summary generated by Reviewpad on 10 Nov 23 21:16 UTC

This pull request introduces several changes across multiple files. Here is a summary of the changes:

A new file named "run-tests.yml" is added, which includes a new workflow for running tests. The workflow is triggered on push events to the "main" branch and pull requests. Concurrency settings are configured to group workflow runs based on the workflow and branch or pull request. Environment variables "GKE_CLUSTER" and "GKE_ZONE" are set to specific values. The workflow contains a job named "go-test" that runs on the "ubuntu-latest" environment. Several steps are added to the job, including installing the "ignite" tool, checking out the repository with a specified fetch depth, setting up Go with a specified version, installing CI dependencies, generating protobufs, generating mocks, running golangci-lint, and running tests with the "go_test" make command.
The Dockerfile.dev has changes related to the installation of vim and less using apt-get. The file path for moving the poktrolld binary is changed to /usr/bin/ignite and /usr/bin/poktrolld. Additionally, port 8546 is exposed in addition to port 8545.
The "node.go" file diff adds a new variable called "defaultDebugOutput" and a new condition to set its value based on an environment variable. Additionally, a debug output is printed if the "defaultDebugOutput" variable is true. These changes seem to be related to debugging and adding more control over the output in the code.
The .gitignore file diff includes changes such as a deleted entry, modified entries, and added entries. Line 32 has a deleted entry: localnet/*/config/*.json. Two lines starting with localnet/poktrolld/config/ have been added: !localnet/poktrolld/config/client.toml and !localnet/poktrolld/config/config.toml. The entry for ignoring Mac OS files .DS_Store remains unchanged.
The labels and actions configuration for PRs in a repository is updated. Changes include updating the label and comment when the devnet-test-e2e label is added or removed from a PR, updating the label to be removed when devnet is removed from a PR, and updating the label to be removed when the push-image label is removed.
A new file named "run-e2e-test-job-template.yaml" is added, defining a Kubernetes Job that runs end-to-end tests for a Pocket Network application. The Job runs a container using the image "ghcr.io/pokt-network/poktrolld" with specified environment variables. It mounts two volumes for storing configuration files and keys. The Job has a TTL of 120 seconds after it finishes, and the restart policy is set to Never. The backoff limit is set to 0, meaning it will not retry failed attempts.
The reviewdog.yml file has changes related to concurrency configuration for the workflow, defining the concurrency group based on the workflow and branch name, and enabling the cancellation of in-progress jobs. There is also a comment about blocking certain comments to ensure the merging to the main branch is not affected.
A new shell script named "run-e2e-test.sh" is added, which sets up a sequence of checks and actions to run end-to-end tests in a Kubernetes environment. It waits for a pod with the matching image SHA and purpose to become ready, creates a job to run the e2e tests, monitors the logs and status of the pod, and deletes the job based on the pod's status and exit code.
The file diff involves renaming the file from "go.yml" to "main-build.yml", updating the workflow name from "Ignite build & test" to "Main build", modifying the concurrency group to include the workflow name, changing the job name for building the container to "build-push-container", updating the Go version to "1.20.10", removing steps for generating mocks and running golangci-lint, changing the Docker image name from "pocketd" to "poktrolld", modifying the step for copying binaries to include the ignite binary and the poktrolld binary, and adding a new job named "run-e2e-tests" that runs E2E tests based on the "devnet-test-e2e" label.

Overall, this pull request introduces a range of changes related to running tests, Dockerfile modifications, debugging, gitignore configuration, labels and actions for PRs, Kubernetes job templates, reviewdog configuration, shell script for e2e testing, and workflow updates.

Issue

[Testing] E2E Test Automation #18

Type of change

Select one or more:

Testing

Run all unit tests: make go_develop_and_test
Verify Localnet manually: See the instructions [here](TODO: add link to instructions)

Sanity Checklist

I have tested my changes using the available tooling
I have performed a self-review of my own code
I have commented my code, updated documentation and left TODOs throughout the codebase

github-actions · 2023-11-07T01:43:40Z

The CI will now also run the e2e tests on devnet, which increases the time it takes to complete all CI checks.

github-actions · 2023-11-07T01:45:24Z

The image is going to be pushed after the next commit. If you want to run an e2e test, it is necessary to push another commit. You can use make trigger_ci to push an empty commit.

github-actions · 2023-11-07T02:46:33Z

Dockerfile.dev

@@ -15,7 +15,8 @@ COPY . /poktroll

 WORKDIR /poktroll

-RUN mv /poktroll/bin/poktrolld /usr/bin/poktrolld
+# TODO_IN_THIS_PR: move to poktroll binary across the board including helm charts


[linter-name (fail-on-found)] _{reported by reviewdog 🐶}
# TODO_IN_THIS_PR: move to poktroll binary across the board including helm charts

Olshansk

No blockers, just some comments. I don't really know how to review something like this given that it works 😅

Great job! I know how difficult all these configs, tests, permissions, etc are!!!

.github/workflows-helpers/run-e2e-test.sh

Olshansk · 2023-11-14T04:36:28Z

.github/workflows-helpers/run-e2e-test.sh

+fi
+done
+
+# Create a job to run the e2e tests


Great comments throughout!

thanks @openai

e2e/tests/node.go

Olshansk · 2023-11-14T04:42:07Z

.github/workflows-helpers/run-e2e-test-job-template.yaml

+      restartPolicy: Never
+      volumes:
+      - configMap:
+          defaultMode: 420


Olshansk · 2023-11-14T04:42:57Z

.github/workflows-helpers/run-e2e-test.sh

@@ -0,0 +1,55 @@
+# Check if the pod with the matching image SHA and purpose is ready


Not a blocker but just a question.

Is the plan to keep this bash script as is? I know you've used things like argoCD and other DAG like workflow builders, so was just surprised to have a simple bash script like it today.

@Olshansk, if this will cause issues, we'll switch. Even with Argo Workflows with DAG, we are going to need some bash scripting. I'm not a fan of them, but we don't have the permissions/connectivity restraints we used to have on previous infrastructure so it seemed as an OK solution.

.github/workflows/run-tests.yml

Co-authored-by: Daniel Olshansky <[email protected]>

…d-error * pokt/main: [CI] Integrate E2E tests with GitHub CI (#152)

* relayer/cli: More tiny comment updates Added a couple more comments Update some comments and TODOs Update the names and references to queryNode/sequencerNode/fullNode etc Update pkg/relayer/cmd/cmd.go [Test] Updating `relay.feature` to run curl command to enable E2E Relay Test (#178) Updated comments for post 177+179 work for okdas Update OpenAPI spec Update .gitignore chore: update comment chore: move shared dependency setup logic to shared pkg chore: cleanup flags and dependencies for appgateserver cmd [Supplier] chore: improve supplier not found error message (#183) [CI] Integrate E2E tests with GitHub CI (#152)

…-update * merge/e2e_test/relay_x_relayer_cli: [LocalNet] Run Relayer and AppGateServer (#179) [Relay] E2E Relay Gaps (#177) More tiny comment updates Added a couple more comments Update some comments and TODOs Update the names and references to queryNode/sequencerNode/fullNode etc Update pkg/relayer/cmd/cmd.go [Test] Updating `relay.feature` to run curl command to enable E2E Relay Test (#178) Updated comments for post 177+179 work for okdas Update OpenAPI spec Update .gitignore chore: update comment chore: move shared dependency setup logic to shared pkg chore: cleanup flags and dependencies for appgateserver cmd [Supplier] chore: improve supplier not found error message (#183) [CI] Integrate E2E tests with GitHub CI (#152)

* add more debug output * Empty commit * symlink to pocketd for now * rename pocketd * --wip-- [skip ci] * removing pocketd completely * also remove comment * rename repo * test the permissions * test permissions * set project_id: ${{ secrets.GKE_PROTOCOL_PROJECT }} * maybe that will help * test perms * try that * test job * try ls * add checkout * separate the jobs * ignite is needed for tests * avoid collision on name * backoffLimit 0 * wait for pod * change envs * add wait for sequencer before running a test * fix jq * fail fast * move to a script * try this * image sha to image tag * dont use make targets then * troubleshoot * pre-populate the variable * add debug output * install more stuff into the container * try this * cleanup * devnet-test-e2e label everywhere * add env * Update .github/workflows-helpers/run-e2e-test.sh Co-authored-by: Daniel Olshansky <[email protected]> * Update .github/workflows/run-tests.yml Co-authored-by: Daniel Olshansky <[email protected]> * requested changes --------- Co-authored-by: Daniel Olshansky <[email protected]>

add more debug output

c458279

okdas added the devnet-e2e-test label Nov 7, 2023

okdas added this to the Shannon TestNet milestone Nov 7, 2023

okdas self-assigned this Nov 7, 2023

github-actions bot added the devnet label Nov 7, 2023

okdas added the push-image CI related - pushes images to ghcr.io label Nov 7, 2023

okdas added 2 commits November 6, 2023 18:07

Empty commit

c933eb2

symlink to pocketd for now

1f2d44d

github-actions bot reviewed Nov 7, 2023

View reviewed changes

okdas and others added 19 commits November 7, 2023 13:49

rename pocketd

341c7c5

--wip-- [skip ci]

0fff5bd

removing pocketd completely

8ff6e3d

also remove comment

0ebb318

Merge branch 'main' into issue/18-e2e-tests-automation

d356a84

rename repo

9874e6a

test the permissions

0ab073b

test permissions

2937c5f

set project_id: ${{ secrets.GKE_PROTOCOL_PROJECT }}

38fff23

maybe that will help

0e2f159

test perms

2c2faa7

try that

216652b

test job

e7a14c7

Merge branch 'main' into issue/18-e2e-tests-automation

b3c8860

try ls

7f78a1c

add checkout

8b3eaf5

separate the jobs

45844a2

ignite is needed for tests

af01e2a

avoid collision on name

4f4b208

okdas added 6 commits November 10, 2023 11:16

troubleshoot

037b3a9

pre-populate the variable

4424f56

add debug output

191b0d4

install more stuff into the container

57419d1

try this

c29a74b

cleanup

1885c4c

okdas added devnet-test-e2e and removed devnet-e2e-test labels Nov 10, 2023

github-actions bot removed the devnet label Nov 10, 2023

devnet-test-e2e label everywhere

763f4ff

okdas added the devnet label Nov 10, 2023

okdas and others added 2 commits November 10, 2023 13:09

Merge branch 'main' into issue/18-e2e-tests-automation

2e4355e

add env

dc4ce63

okdas marked this pull request as ready for review November 11, 2023 00:02

okdas requested a review from Olshansk November 11, 2023 00:02

Olshansk approved these changes Nov 14, 2023

View reviewed changes

okdas and others added 4 commits November 14, 2023 16:26

Update .github/workflows-helpers/run-e2e-test.sh

726a96c

Co-authored-by: Daniel Olshansky <[email protected]>

Update .github/workflows/run-tests.yml

f486920

Co-authored-by: Daniel Olshansky <[email protected]>

requested changes

65305ff

Merge branch 'main' into issue/18-e2e-tests-automation

ad816cb

okdas requested a review from Olshansk November 15, 2023 00:45

okdas merged commit 16c6ebc into main Nov 15, 2023
8 checks passed

okdas deleted the issue/18-e2e-tests-automation branch November 15, 2023 00:45

bryanchriswhite added a commit that referenced this pull request Nov 15, 2023

Merge remote-tracking branch 'pokt/main' into chore/supplier-not-foun…

9176fca

…d-error * pokt/main: [CI] Integrate E2E tests with GitHub CI (#152)

bryanchriswhite removed push-image CI related - pushes images to ghcr.io devnet-test-e2e labels May 16, 2024

github-actions bot removed the devnet label May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Integrate E2E tests with GitHub CI #152

[CI] Integrate E2E tests with GitHub CI #152

okdas commented Nov 7, 2023 •

edited

Loading

github-actions bot commented Nov 7, 2023

github-actions bot commented Nov 7, 2023

github-actions bot Nov 7, 2023

Olshansk left a comment

Olshansk Nov 14, 2023

okdas Nov 15, 2023

Olshansk Nov 14, 2023

Olshansk Nov 14, 2023

okdas Nov 15, 2023

		@@ -0,0 +1,55 @@
		# Check if the pod with the matching image SHA and purpose is ready

[CI] Integrate E2E tests with GitHub CI #152

[CI] Integrate E2E tests with GitHub CI #152

Conversation

okdas commented Nov 7, 2023 • edited Loading

Summary

Human Summary

AI Summary

Summary generated by Reviewpad on 10 Nov 23 21:16 UTC

Issue

Type of change

Testing

Sanity Checklist

github-actions bot commented Nov 7, 2023

github-actions bot commented Nov 7, 2023

github-actions bot Nov 7, 2023

Choose a reason for hiding this comment

Olshansk left a comment

Choose a reason for hiding this comment

Olshansk Nov 14, 2023

Choose a reason for hiding this comment

okdas Nov 15, 2023

Choose a reason for hiding this comment

Olshansk Nov 14, 2023

Choose a reason for hiding this comment

Olshansk Nov 14, 2023

Choose a reason for hiding this comment

okdas Nov 15, 2023

Choose a reason for hiding this comment

okdas commented Nov 7, 2023 •

edited

Loading