ci: add performance script to PR jobs #3274

BlairCurrey · 2025-02-05T19:16:38Z

Changes proposed in this pull request

Decouples testenv from integration tests such that it can be re-used for performance tests in CI
- MockASE handles seeding and hosting integration endpoints required for rafiki to properly function (rates, webhooks). This is located in the integration tests in main so I moved it out for re-use when running performance tests.
- Performance tests in main run against our localenv. I changed it to be configurable by parameterizing the run script and test and reading from a config to get the correct environment details (network name, urls). So now you could run performance tests locally against the testenv without interference with your localenv. And can still run against localenv if you want to use the grafana dashboard.
Adds github action job to run k6 test and upsert comment to PR with result

Context

Checklist

Related issues linked using fixes #number
Tests added/updated
Make sure that all checks pass
Bruno collection updated (if necessary)
Documentation issue created with user-docs label (if necessary)
OpenAPI specs updated (if necessary)

- currently requires running tests to setup db (waiting to avoid actually running tests and spinning down tests/env) - loses ability to run k6 against rafiki w/ telemetry

This reverts commit 1a6c0e6.

- in preparation for switching performance test to target testenv - ensured integration tests pass

- supports reuse in setup script for performance tests which will allow running performance test against testenv - also bumps graphql pacakge version across monorepo. added to test-lib which caused some type errors in backend and auth due to a version mismatch

netlify · 2025-02-05T19:17:09Z

✅ Deploy Preview for brilliant-pasca-3e80ec ready!

Name	Link
🔨 Latest commit	`d4e11ea`
🔍 Latest deploy log	https://app.netlify.com/sites/brilliant-pasca-3e80ec/deploys/67c0b9f97f2d6a000836ecb0
😎 Deploy Preview	https://deploy-preview-3274--brilliant-pasca-3e80ec.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

- not sure if config outpute by script will reflect options passed in, probably not. config in gh pr comment may not be accurate.

BlairCurrey · 2025-02-12T19:45:09Z

.github/workflows/node-build.yml

+      - name: Post/Update Performance Test Results as PR Comment
+        if: always()
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |


I am in-lining the javascript here because I found it easier than passing everything from the actions context in and it was just easier for development. We can move this to a file and just run it with node if we prefer though.

BlairCurrey · 2025-02-13T18:59:52Z

.github/workflows/node-build.yml

+      - name: Run performance tests
+        id: perf_test
+        run: |
+          pnpm --filter performance run-tests:testenv -k --vus 4 --duration 1m | tee performance.log
+        continue-on-error: true # Don't fail here, allows showing logs
+
+      - name: Print MASE logs
+        if: always()
+        run: cat mases.log
+
+      - name: Post/Update Performance Test Results as PR Comment


There is a grafana/k6 action that runs the tests and exports to grafana cloud and leaves a comment here with a link:
https://grafana.com/blog/2024/07/15/performance-testing-with-grafana-k6-and-github-actions/

I opted for doing it more manually as a start - was simpler to get going. And even with that action we might need to manual write/edit a comment if we want the actual metrics in the PR comment (looks like the comment just leaves a link). But that could probably be mitigated by setting good thresholds (fail if iterations < x etc. ) and just use the job for pass/fail and venture into grafana when it fails for more details.

It does appear like the free tier is fairly generous and it probably gives us a lot of stuff for free like a centralized view of the history of runs. Open to reworking if we want that but I think this gives us what we're looking for for now.

This is the comment generated: #3274 (comment)

It just upserts with the latest results and test logs.

BlairCurrey · 2025-02-13T19:06:29Z

.github/workflows/node-build.yml

+      - name: Run performance tests
+        id: perf_test
+        run: |
+          pnpm --filter performance run-tests:testenv -k --vus 4 --duration 1m | tee performance.log


What do we want the test config to be?

I opted for 4VUs and 1M after testing some different values. I think we want to:

not make the duration longer than necessary. Just dont want to extend the time it takes CI to finish if we can help it. So I prefered more VUs for a shorter time to some extent. There may still be some room to increase duration if we want. This is taking ~3m while the docker builds are taking 4-5m

reliably complete the tests without failures as a baseline. But approaching the point where we do see issues. I saw some errors at 5VUs so I backed off.

I think the hardware is determined by runs-on: ubuntu-22.04 and should always be the same across runs.

BlairCurrey · 2025-02-13T19:10:12Z

test/performance/scripts/create-outgoing-payments.js

+export function handleSummary(data) {
+  const requestsPerSecond = data.metrics.http_reqs.values.rate
+  const iterationsPerSecond = data.metrics.iterations.values.rate
+  const failedRequests = data.metrics.http_req_failed.values.passes
+  const failureRate = data.metrics.http_req_failed.values.rate
+  const requests = data.metrics.http_reqs.values.count
+
+  const summaryText = `
+  **Test Configuration**:
+  - VUs: ${options.vus}
+  - Duration: ${options.duration}
+
+  **Test Metrics**:
+  - Requests/s: ${requestsPerSecond.toFixed(2)}
+  - Iterations/s: ${iterationsPerSecond.toFixed(2)}
+  - Failed Requests: ${failureRate.toFixed(2)}% (${failedRequests} of ${requests})
+    `
+
+  return {
+    // Preserve standard output w/ textSummary
+    stdout: textSummary(data, { enableColors: false }),
+    'k6-test-summary.txt': summaryText // saves to file
+  }
+}


this is how we're saving the results for use in the comment. the stdout will preserve the normal output and the k6-test-summary.txt saves the details we want to report.

BlairCurrey · 2025-02-13T19:11:24Z

test/performance/scripts/run-tests.sh

lots of changes here but it basically just makes the target environment configurable.

njlie · 2025-02-21T20:09:43Z

test/performance/README.md

+  and a process for the Mock Account Servicing Entities:
+
+  ```sh
+  pnpm --filter perfromance start-mases


typo: perfromance -> performance

njlie · 2025-02-21T20:21:59Z

test/performance/config/local.env

@@ -0,0 +1,8 @@
+C9_BACKEND_HOST="cloud-nine-wallet-backend"
+HLB_BACKEND_HOST="happy-life-bank-backend"
+C9_AUTH_HOST="cloud-nine-wallet-backend"


Isn't the auth host here supposed to be (cloud-nine-wallet/happy-life-bank)-auth?

Yes, fixed. Thanks.

njlie · 2025-02-21T20:22:20Z

test/performance/config/test.env

@@ -0,0 +1,8 @@
+C9_BACKEND_HOST="cloud-nine-wallet-test-backend"
+HLB_BACKEND_HOST="happy-life-bank-test-backend"
+C9_AUTH_HOST="cloud-nine-wallet-test-backend"


Ditto here of https://github.com/interledger/rafiki/pull/3274/files#r1966124607

BlairCurrey added 4 commits February 5, 2025 11:46

feat: poc run performance against testenv

1a6c0e6

- currently requires running tests to setup db (waiting to avoid actually running tests and spinning down tests/env) - loses ability to run k6 against rafiki w/ telemetry

Revert "feat: poc run performance against testenv"

f433b57

This reverts commit 1a6c0e6.

refactor: mv testenv to ./test from ./test/integration

0eeab9b

- in preparation for switching performance test to target testenv - ensured integration tests pass

github-actions bot added type: tests Testing related pkg: backend Changes in the backend package. labels Feb 5, 2025

chore: generate gql

2ef91de

github-actions bot added the type: source Changes business logic label Feb 5, 2025

BlairCurrey added 4 commits February 6, 2025 16:07

feat(performance): make environment configurable

fbad4e2

feat(performance): make run and test script configurable per env

713cd81

fix(performance): up liquidity, matching localenv

07095f9

ci: add performance test

b7022d2

github-actions bot added the type: ci Changes to the CI label Feb 7, 2025

chore: fix eslint error

da89299

BlairCurrey changed the title ~~Bc/3240/performance ci~~ ci: add performance script to PR jobs Feb 7, 2025

BlairCurrey added 14 commits February 7, 2025 12:17

chore: debug test in ci

6c33b43

chore: debug test in ci

001eda5

chore: debug test in ci

6f52bc9

chore: debug test in ci

4388003

chore: debug test in ci

38ab11b

chore: debug test in ci

2789afd

chore: debug test in ci

f29a751

fix: script typo

e8b09b8

chore: debug test in ci

143ab10

chore: debug test in ci

81a1a12

chore: debug test in ci

ea8e812

chore: debug test in ci

c7587e7

chore: debug test in ci

036b023

chore(auth): gql generate

f27d966

BlairCurrey added 5 commits February 12, 2025 12:49

fix: override test config in ci

8a358db

- not sure if config outpute by script will reflect options passed in, probably not. config in gh pr comment may not be accurate.

ci: test failure

a7ce6c9

chore: ignore test summary

3d7f998

ci: un-fail step

e498465

ci: tune test config

747aa83

BlairCurrey commented Feb 12, 2025

View reviewed changes

BlairCurrey added 8 commits February 12, 2025 14:50

ci: tune test config

c5b279d

ci: tune test config

4a923db

fix: 401 bug for duplicate reqs

27d9d81

ci: tune test config

a829619

docs: update performance readme

f17acd8

docs: update README

cd83210

docs: rm inaccurate info

128160c

ci: tune test config

248ee43

interledger deleted a comment from github-actions bot Feb 13, 2025

BlairCurrey commented Feb 13, 2025

View reviewed changes

fix: return to orginal duration

b910e1a

BlairCurrey commented Feb 13, 2025

View reviewed changes

BlairCurrey marked this pull request as ready for review February 13, 2025 19:14

BlairCurrey requested review from mkurapov, njlie and golobitch February 13, 2025 19:17

njlie reviewed Feb 21, 2025

View reviewed changes

BlairCurrey added 3 commits February 27, 2025 13:48

chore: fix typos

bb64af6

chore: rm console.log

0d72824

Merge branch 'main' into bc/3240/performance-ci

d4e11ea

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add performance script to PR jobs #3274

ci: add performance script to PR jobs #3274

BlairCurrey commented Feb 5, 2025 •

edited

Loading

netlify bot commented Feb 5, 2025 •

edited

Loading

BlairCurrey Feb 12, 2025

BlairCurrey Feb 13, 2025

BlairCurrey Feb 13, 2025

BlairCurrey Feb 13, 2025 •

edited

Loading

BlairCurrey Feb 13, 2025

BlairCurrey Feb 13, 2025

njlie Feb 21, 2025

njlie Feb 21, 2025 •

edited

Loading

BlairCurrey Feb 27, 2025

njlie Feb 21, 2025

ci: add performance script to PR jobs #3274

Are you sure you want to change the base?

ci: add performance script to PR jobs #3274

Conversation

BlairCurrey commented Feb 5, 2025 • edited Loading

Changes proposed in this pull request

Context

Checklist

netlify bot commented Feb 5, 2025 • edited Loading

✅ Deploy Preview for brilliant-pasca-3e80ec ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BlairCurrey Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

njlie Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BlairCurrey commented Feb 5, 2025 •

edited

Loading

netlify bot commented Feb 5, 2025 •

edited

Loading

BlairCurrey Feb 13, 2025 •

edited

Loading

njlie Feb 21, 2025 •

edited

Loading