Change recommendation for ci-build-id?

[rbayliss]

Hello!
Thanks for this action! It looks like Github has recently added context support for github.run_attempt. This action currently has to do backflips with the GH API in order to generate a unique (but predictable) ci-build-id property, but with the addition of run_attempt, I think you could probably deprecate that complexity. Eg:

- name: E2E Tests
  uses: cypress-io/github-action@v2
  with:
    record: true
    parallel: true
    ci-build-id: "${{ github.repository }}-${{ github.run_id }}-${{ github.run_attempt}}"

I think this build ID would be guaranteed unique across all of Github for every run, even for reruns. github.run_attempt hasn't made the GHA documentation yet, so I'm not sure if it's stable enough to recommend right now, but I've verified that it's present for all of the current "Github Provided" runners.

[MikeMcC399]

• This needs to be revisited. See also Outdated README warning for parallel re-runs #1074

[cgraham-rs]

@rbayliss run_attempt is now included in the documentation. Have you implemented this and found it to be reliable? It would appear to fully solve the potential issue with Github Re-run failed jobs running 0 tests and emitting a false pass.

[rbayliss]

Hey @cgraham-rs, yep - I've used this and it's 100% reliable for my purposes (including across reruns).

[MikeMcC399]

• There is a feature request Re-run only failed tests (jobs) for Cloud parallel recording #1290 open which describes the issue with "Re-run failed jobs" using the parallel option.
• Is there something missing which still needs to be addressed for this issue?

[cgraham-rs]

@MikeMcC399 there's still an issue where if you use Github Re-run failed jobs Cypress will run 0 tests and emit a terrible, terrible false pass since nothing was run thus nothing failed. Nobody will ever notice unless they happen to look at the details of the run in the CI logs.

I believe this is caused by the current Robust custom build id documentation suggesting creating a separate CI job to compose ci-build-id This means when using Re-run failed jobs it will NOT re-run the separate setup job, thus it re-uses the previous ci-build-id and... I guess Cypress sees nothing new to run so it skips all tests?

Migrating the "Robust custom build id" into the actual test job and ensuring it's unique on each run and NOT time-based should solve that problem as a new ci-build-id will get created each time the job is re-run, including when using Re-run failed jobs. Chaining ${{ github.run_id }}-${{ github.run_attempt}} into the ci-build-id seems like an excellent solution as the run_attempt will be unique on each re-run.

It looks like this issue was originally documented in #574 but closed as a dupe of #531 which is not correct as that addresses how it re-runs ALL tests instead of JUST the failed tests which is a completely separate issue.

[cgraham-rs]

Hey @cgraham-rs, yep - I've used this and it's 100% reliable for my purposes (including across reruns).

Confirmed.

[MikeMcC399]

@cgraham-rs

https://github.com/cypress-io/github-action/blob/master/README.md#record-test-results-on-cypress-cloud says:

We recommend passing the GITHUB_TOKEN secret (created by the GH Action automatically) as an environment variable. This will allow correctly identifying every build and avoid confusion when re-running a build.

I tested this (without using the option ci-build-id) and found that selecting "Re-run failed jobs" runs all Cypress tests again and correctly reports successes and failures.

I'm unsure about the need for ci-build-id. The discussion would benefit from input coming from the Cypress Cloud team at this stage.

[jennifer-shehane]

@cgraham-rs This behavior would indicate the ci-build-id not being unique. You can see the generated ci-build-id in the Cloud in the properties tab. I wouldn't expect users to need to pass one, as we do try to generate it ourselves, as Mike is also reiterating.

[cgraham-rs]

@jennifer-shehane the Robust custom build id documentation instructs us to manually craft a ci-build-id as a separate job from the test job. Though it mentions ...If the user re-runs the workflow a new unique build id is generated... which is true. But causes problems when a user chooses to Re-run failed jobs which would ONLY re-run the test job and not the setup job, thus the test job has the same ci-build-id from the previous run.

I don't recall why we chose to implement a custom ci-build-id but my assumption is it was to solve some CI problem.