Cypress e2e initial infra

[marcosnav]

Intent

Fixes #2508

PR to have the initial building steps of infra for e2e tests, aiming for consistent reproducibility.

Only includes three tests so far to serve as examples and starting point for more to come:

1. Common checks, basically that the extension is installed and can be opened.
2. Create a new credential and load existing credentials.
3. Deployment of static content.

To try this, just pull the branch and just test/e2e/dev, this command will do the necessary setup and open Cypress.
... and just test/e2e/stop to tear down the containers.

This is not set to run in CI, yet. A separate GH ticket will be created, to determine specifics of adding this e2e process to CI.

Type of Change

• • Bug Fix
• • New Feature
• • Breaking Change
• • Documentation
• • Refactor
• • Tooling

Approach

• Leverage on Docker infra to spin up a Connect and code-server instances.
• Cypress will target these instances for tests.
• Cypress not running in Docker for local runs, a container for Cypress will be needed only for CI, to be addressed in a separate GH issue.

User Impact

There is not a direct impact for the end user.

Automated Tests

Yes

Directions for Reviewers

Try it out with just test/e2e/dev, and just test/e2e/stop to teardown the containers.

Checklist

• I have updated CHANGELOG.md to cover notable changes.

[dotNomad]

Still looking through this, but had a few questions so far

[sagerb]

I attempted to run the three specs, after selecting the Google browser. Looks to be a fundamental issue, any help would be appreciaed.

• common: ran successfullly
• credentials: failed with an error:
  
  (console output is)

Error:       CypressError: `cy.exec('rsconnect bootstrap --raw --jwt-keypath bootstrap-secret.key --server http://localhost:3939')` failed because the command exited with a non-zero code.

Pass `{failOnNonZeroExit: false}` to ignore exit code failures.

Information about the failure:
Code: 1

Stderr:
Traceback (most recent call last):
  File "/Users/billsager/.pyenv/versions/3.8.2/bin/rsconnect", line 5, in <module>
    from rsconnect.main import cli
  File "/Users/billsager/.pyenv/versions/3.8...

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Deployments Section`
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:119310:77)
    at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1777:23)
    at Promise._settlePromiseFromHandler (http://localhost:8080/__cypress/runner/cypress_runner.js:1489:31)
    at Promise._settlePromise (http://localhost:8080/__cypress/runner/cypress_runner.js:1546:18)
    at Promise._settlePromise0 (http://localhost:8080/__cypress/runner/cypress_runner.js:1591:10)
    at Promise._settlePromises (http://localhost:8080/__cypress/runner/cypress_runner.js:1671:18)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:2362:25)
From previous event:
    at Promise.longStackTracesCaptureStackTrace [as _captureStackTrace] (http://localhost:8080/__cypress/runner/cypress_runner.js:3456:19)
    at Promise._then (http://localhost:8080/__cypress/runner/cypress_runner.js:1209:17)
    at Promise._passThrough (http://localhost:8080/__cypress/runner/cypress_runner.js:4080:17)
    at Promise.lastly.Promise.finally (http://localhost:8080/__cypress/runner/cypress_runner.js:4089:17)
    at Object.onRunnableRun (http://localhost:8080/__cypress/runner/cypress_runner.js:162647:53)
    at $Cypress.action (http://localhost:8080/__cypress/runner/cypress_runner.js:41540:28)
    at Runnable.run (http://localhost:8080/__cypress/runner/cypress_runner.js:145229:13)
    at next (http://localhost:8080/__cypress/runner/cypress_runner.js:155014:10)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:155058:5)
    at timeslice (http://localhost:8080/__cypress/runner/cypress_runner.js:145569:27)
From Your Spec Code:
    at Context.eval (webpack://publisher-e2e/./support/commands.js:74:5)
    at wrapped (http://localhost:8080/__cypress/runner/cypress_runner.js:137875:43)
index-DwhYwRtt.js:5316 Yielded:     Object

• deployments: failed with an error:
  
  (console output is)

Error:       CypressError: `cy.exec('rsconnect bootstrap --raw --jwt-keypath bootstrap-secret.key --server http://localhost:3939')` failed because the command exited with a non-zero code.

Pass `{failOnNonZeroExit: false}` to ignore exit code failures.

Information about the failure:
Code: 1

Stderr:
Traceback (most recent call last):
  File "/Users/billsager/.pyenv/versions/3.8.2/bin/rsconnect", line 5, in <module>
    from rsconnect.main import cli
  File "/Users/billsager/.pyenv/versions/3.8...

Because this error occurred during a `before each` hook we are skipping the remaining tests in the current suite: `Deployments Section`
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:119310:77)
    at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1777:23)
    at Promise._settlePromiseFromHandler (http://localhost:8080/__cypress/runner/cypress_runner.js:1489:31)
    at Promise._settlePromise (http://localhost:8080/__cypress/runner/cypress_runner.js:1546:18)
    at Promise._settlePromise0 (http://localhost:8080/__cypress/runner/cypress_runner.js:1591:10)
    at Promise._settlePromises (http://localhost:8080/__cypress/runner/cypress_runner.js:1671:18)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:2362:25)
From previous event:
    at Promise.longStackTracesCaptureStackTrace [as _captureStackTrace] (http://localhost:8080/__cypress/runner/cypress_runner.js:3456:19)
    at Promise._then (http://localhost:8080/__cypress/runner/cypress_runner.js:1209:17)
    at Promise._passThrough (http://localhost:8080/__cypress/runner/cypress_runner.js:4080:17)
    at Promise.lastly.Promise.finally (http://localhost:8080/__cypress/runner/cypress_runner.js:4089:17)
    at Object.onRunnableRun (http://localhost:8080/__cypress/runner/cypress_runner.js:162647:53)
    at $Cypress.action (http://localhost:8080/__cypress/runner/cypress_runner.js:41540:28)
    at Runnable.run (http://localhost:8080/__cypress/runner/cypress_runner.js:145229:13)
    at next (http://localhost:8080/__cypress/runner/cypress_runner.js:155014:10)
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:155058:5)
    at timeslice (http://localhost:8080/__cypress/runner/cypress_runner.js:145569:27)
From Your Spec Code:
    at Context.eval (webpack://publisher-e2e/./support/commands.js:74:5)
    at wrapped (http://localhost:8080/__cypress/runner/cypress_runner.js:137875:43)
index-DwhYwRtt.js:5316 Yielded:     {stdout: '', stderr: `Traceback (most recent call last):\n  File "/Users/…bssl.1.1.dylib' (no such file, not in dyld cache)`, code: 1}

[marcosnav]

@sagerb The error points to the command used to bootstrap an admin user API key, that is done through rsconnect bootstrap, you may need to update your local version of rsconnect.

Thanks for trying it out!

[dotNomad]

I attempted to run just/e2e/dev and got the Cypress UI.

I first ran into an error where I didn't have rsconnect so I created a Python venv and installed rsconnect-python which got me past that.

Next I ran into an error in the Credentials test:

The text didn't match because of a URL validation error.

The deployments spec also did not pass for me because of Cypress env BOOTSTRAP_ADMIN_API_KEY is empty. Cannot set admin credentials.

Is there additional setup I'm missing?

[marcosnav]

@dotNomad

It could be that not having rsconnect from the beginning left some bad state

Could you try the following?:

just test/e2e/stop
just test/e2e/dev

Also, it could be that the Connect instance of the e2e tests is not running. You may have to double check that there is no other Connect instance running with the same ports that the e2e Connect container wants to pick up.

[dotNomad]

It could be that not having rsconnect from the beginning left some bad state

I didn't run test/e2e/stop in between the failures and the re-test after getting rsconnect working, but this time I did.

No difference - I'm still seeing both of the errors above.