GITBOOK-662: Disable Reruns

.gitbook/includes/retries.md

@@ -0,0 +1,9 @@
+---
+title: Retries
+---
+
+{% hint style="warning" %}
+### Retries
+
+Retries negatively impact the accuracy of detection and reporting because they mask flaky failures. When adopting Trunk Flaky Tests, disable retried when testing in CI and prefer detecting then [quarantining](../../flaky-tests/quarantining.md) tests.
+{% endhint %}

flaky-tests/detection.md

@@ -42,6 +42,8 @@ A test’s health status transitions between broken, flaky, and healthy as new t
 
 This means if a test is healthy, it can transition into a broken or flaky status after new results appear that show failures. This also means if a test that was previously labeled as broken or flaky sees consistently passing runs, it can transition into a healthy test. 
 
+{% include "../.gitbook/includes/retries.md" %}
+
 <figure><picture><source srcset="../.gitbook/assets/status-history-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/status-history-light.png" alt=""></picture><figcaption></figcaption></figure>
 
 Tests may transition between flaky, broken, and healthy states multiple times over their lifetime. You can see previous changes in the detected health status of a test under Status History, as well as an explanation for why it was detected to have a new state.

flaky-tests/get-started/README.md

@@ -14,6 +14,8 @@ You must create an account and Trunk organization to complete the steps below.
 
 ### 2. Configure your test framework to output compatible test results
 
+{% include "../../.gitbook/includes/retries.md" %}
+
 For Trunk to start detecting flaky tests, it needs to ingest the results from all of your CI test runs. To get started, configure your test framework (jest, pytest, gtest, etc) to output results. You'll do this only for CI jobs, _not_ for running tests locally.
 
 Check out our guides to many of the most popular test frameworks: [frameworks](frameworks/ "mention")

flaky-tests/get-started/frameworks/android.md

@@ -34,6 +34,12 @@ Output from `./gradlew connectedAndroidTest` will be written to
 
 `path_to_your_project/module_name/build/test-results/connected/`
 
+## Disable Retries
+
+You need to disable [automatic retries](https://source.android.com/docs/core/tests/tradefed/testing/through-tf/auto-retry) if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+You can disable retries by using the option: `--retry-strategy NO_RETRY`.
+
 ## Next Step
 
 JUnit test reports for Android projects are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/bazel.md

@@ -26,6 +26,12 @@ By default the JUnit output will be written to the `bazel-testlogs` output direc
 
 For example, a target named `//app/component:test` will generate a JUnit file at `bazel-testlogs/app/component/test.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+Disable retries if you're retrying tests using the `--flaky_test_attempts` command line option or retrying in your test runner.
+
 ## Next Step
 
 JUnit files generated with Bazel are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/cucumber.md

@@ -28,6 +28,12 @@ cucumber tests/test.feature --format junit:output/report.xml
 
 JUnit files will be written according to the path passed to the `--format` option. In the above example, a JUnit file will be written to `output/report.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+You can disable retry by omitting the `--retry` CLI option or removing retry from your configuration file.
+
 ## Next Step
 
 JUnit files generated with Cucumber are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/cypress.md

@@ -38,6 +38,18 @@ module.exports = defineConfig({
 
 The JUnit report location is specified by the `mochaFile` property in your Cypress config. In the above example, the file will be at `results/junit.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+You can disable retries by setting `retries: 0`in your Cypress config file.
+
+```javascript
+module.exports = defineConfig({
+  retries: 1,
+})
+```
+
 ## Next Step
 
 JUnit files generated with Cypress are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/gotestsum.md

@@ -30,6 +30,12 @@ gotestsum --junitfile junit.xml
 
 `gotestsum` will write a JUnit test report to the file specified by the `--junitfile` argument. In the example above, the JUnit report would be written to `junit.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.\
+\
+If you're using a package like [**retry**](https://pkg.go.dev/github.com/hashicorp/consul/sdk/testutil/retry), disable it to get more accurate results from Trunk.
+
 ## Next Step
 
 JUnit files generated with `gotestsum` are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/jasmine.md

@@ -54,6 +54,12 @@ jasmine.getEnv().addReporter(junitReporter)
 
 Jasmine will generate a JUnit report at the location specified by the `savePath` property. In the examples above, the JUnit report will be written to a directory named `test_reports/`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+If you're using a package like [protractor-flake](https://www.npmjs.com/package/protractor-flake), disable it to get more accurate results from Trunk.
+
 ## Next Step
 
 JUnit files generated with Jasmine are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/jest.md

@@ -45,6 +45,12 @@ Update your Jest config to add `jest-junit` as a reporter:
 
 The `outputDirectory` and `outputName` specify the location of the JUnit test report. In the example above, the JUnit would be at `./test_results/report.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+If you have retries configured using the [jest.retryTimes method](https://jestjs.io/docs/jest-object#jestretrytimesnumretries-options), disable them for more accurate results.
+
 ## Next Step
 
 JUnit files generated with Jest are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/karma.md

@@ -44,6 +44,12 @@ module.exports = function(config) {
 
 The `outputDir` and `outputFile` specify the location of the JUnit test report. In the example above, the JUnit would be at `./test_results/$browserName/report.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+Karma doesn't support retries out of the box, but if you implemented retries, remember to disable them.
+
 ## Next Step <a href="#next-step" id="next-step"></a>
 
 JUnit files generated with Karma are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/maven.md

@@ -48,6 +48,12 @@ Configure the `maven-surefire-plugin` plugin in your `pom.xml` file:
 
 The JUnit report will be in the `target/surefire-reports` directory.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests. 
+
+If you're using the Surefire Plugin to retry tests, you can disable it by omitting the `-Dsurefire.rerunFailingTestsCount` option.
+
 ## Next Step
 
 JUnit files generated with Maven are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/minitest.md

@@ -35,6 +35,12 @@ Minitest::Reporters.use! Minitest::Reporters::JUnitReporter.new(:reports_dir =>
 
 This will automatically write all test results to JUnit XML files in the `results` directory.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+Minitest doesn't support retries out of the box, but if you implemented retries or imported a package, remember to disable them.
+
 ## Next Step
 
 JUnit files generated with `minitest` are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/mocha.md

@@ -39,6 +39,12 @@ var mocha = new Mocha({
 
 The resulting JUnit XML file will be written to the location specified by the `mochaFile` property in `reporterOptions`. In the example above, the results would be at `./test_results/junit.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+You can disable retry by omitting the `--retries` CLI option and [removing retries for individual tests](https://mochajs.org/#retry-tests).
+
 ## Next Step
 
 JUnit files generated with Mocha are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/phpunit.md

@@ -28,6 +28,12 @@ phpunit ./tests --log-junit junit.xml
 
 The JUnit report is written to the location specified by `--log-junit`. In the example above, the test results will be written to `./junit.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+PHPUnit doesn't support retries out of the box, but if you implemented retries, remember to disable them.
+
 ## Next Step
 
 JUnit files generated with PHPUnit are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/playwright.md

@@ -34,8 +34,14 @@ export default defineConfig({
 
 ## 2. Output Location
 
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+## Disable Retries
+
 The JUnit file will be written to the `outputFile` specified in the configuration. In the example above, the results will be written to `junit.xml`.
 
+You can disable retries in Playwright by omitting the `--retries` command line option and [removing retries in your `playwright.config.ts` file](https://playwright.dev/docs/test-retries#retries).
+
 ## Next Step
 
 JUnit files generated with Playwright are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/pytest.md

@@ -28,6 +28,12 @@ pytest --junit-xml=junit.xml
 
 The test results JUnit report will be written to the location specified by the `--junit-xml` argument. In the example above, it would be at `./junit.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests. 
+
+Omit the [ `--lf` or `--ff` options](https://docs.pytest.org/en/stable/how-to/cache.html) if you've previously configured your CI with these options to disable retries.
+
 ## Next Step
 
 JUnit files generated with pytest are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/rspec.md

@@ -34,6 +34,12 @@ rspec --format RspecJunitFormatter --out junit.xml
 
 The JUnit report will be written to the location specified by the `--out` argument. In the example above, the report would be at `./junit.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests. 
+
+If you have a step in CI to rerun failed tests with the `--only-failures` option, or you're using a package like [rspec-retry](https://github.com/NoRedInk/rspec-retry), remember to disable them.
+
 ## Next Step
 
 JUnit files generated with RSpec are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/rust.md

@@ -35,6 +35,12 @@ cargo nextest run --profile ci
 
 When using a profile with JUnit support configured, a JUnit report will be written out to `target/nextest/ci/junit.xml` within the workspace root.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests. 
+
+Omit the `--retries` option.
+
 ## Next Step
 
 JUnit files generated with `cargo-nextest` are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/swift-testing.md

@@ -28,6 +28,12 @@ swift test --xunit-output junit.xml
 
 The test results JUnit report will be written to the location specified by the `--xunit-output` argument. In the example above, it would be at `./junit.xml`.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+Swift Testing doesn't support retries out of the box, but if you implemented retries or imported a package, remember to disable them.
+
 ## Next Step
 
 JUnit files generated with Swift Test are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/vitest.md

@@ -36,6 +36,12 @@ export default defineConfig({
 
 The JUnit report will be written to the `outputFile` specified in the Vite config. In the example above, the report would be written to `./junit.xml.`
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+If you've enabled retries, you can disable them following the [Vitest docs](https://vitest.dev/api/) for more accurate results.
+
 ## Next Step
 
 JUnit files generated with Vitest are compatible with Trunk Flaky Tests. See [CI Providers](https://docs.trunk.io/flaky-tests/get-started/ci-providers) for a guide on how to upload test results to Trunk.

flaky-tests/get-started/frameworks/xctest.md

@@ -44,6 +44,12 @@ This will produce a `build/reports/junit.xml` output file.
 
 `xcbeautify` will use the name of file the tests are in as the name of the output `<testsuite>` and the function name as the `name` attribute of each `<testcase>` element.
 
+## Disable Retries
+
+You need to disable automatic retries if you previously enabled them. Retries compromise the accurate detection of flaky tests.
+
+If you run tests in CI with [the `-retry-tests-on-failure` option](https://keith.github.io/xcode-man-pages/xcodebuild.1.html#retry-tests-on-failure), disable it for more accurate results.
+
 ## Next Step
 
 Once you've configured your test runner to output JUnit XML, you're ready to modify your CI test jobs to actually upload test results to Trunk. See [CI Providers](../ci-providers/) for instructions to do this for the CI system you use.