Multiple Fixtures Running In Parallel with Concurrency Enabled

[codambro]

What is your Scenario?

I am running multiple fixtures as a test run, with concurrency enabled.

What is the Current behavior?

If the concurrency level is greater than the number of tests in a fixture, or a test finishes and a concurrently open browser is available, the next fixture starts before the previous fixture finishes.

What is the Expected behavior?

Based on this documentation:
https://testcafe.io/documentation/403626/guides/intermediate-guides/run-tests-concurrently

TestCafe does not start a new fixture until after it completes the last after hook.

I am expecting the entire first fixture to finish before the next fixture starts. Even disableConcurrency per fixture did not help

What is the public URL of the test page? (attach your complete example)

no test page

What is your TestCafe test code?

sample1.ts:

fixture("Fixture 1").disableConcurrency

test("Test 1", async t => {
    console.log("Starting Test 1");
    await t.wait(60000);
    console.log("Finishing Test 1");
})

sample2.ts:

fixture("Fixture 2").disableConcurrency

test("Test 2", async t => {
    console.log("Starting Test 2");
    await t.wait(60000);
    console.log("Finishing Test 2");
})

Console output:

 Running tests in:
 - Chrome 120.0.0.0 / macOS 10.15.7
 - Chrome 120.0.0.0 / macOS 10.15.7
 - Chrome 120.0.0.0 / macOS 10.15.7

 Fixture 1
Starting Test 1
Starting Test 2
Finishing Test 1
 ✓ Test 1

 Fixture 2
Finishing Test 2
 ✓ Test 2


 2 passed (1m 05s)

Your complete configuration file

{
  "src": ["tests/**/*.ts", "!tests/Inference/*.ts", "!tests/API/*.ts"],
  "compilerOptions": {
    "typescript": {
      "configPath": "tsconfig.json"
    }
  },
  "quarantineMode": {
    "successThreshold": 1,
    "attemptLimit": 3
  },
  "reporter": [
    {
      "name": "xunit",
      "output": "results/reports/report.xml"
    },
    {
      "name": "qa-xray-json",
      "output": "results/reports/xray.report.json"
    },
    {
      "name": "html",
      "output": "results/reports/report.html"
    },
    {
      "name": "spec"
    }
  ],
  "screenshots": {
    "path": "results/screenshots/",
    "takeOnFails": true,
    "pathPattern": "${USERAGENT}/${FIXTURE}_${TEST}_${QUARANTINE_ATTEMPT}.png",
    "fullPage": true,
    "thumbnails": false
  },
  "videoPath": "results/videos/",
  "videoOptions": {
    "failedOnly": true,
    "pathPattern": "${USERAGENT}/${FIXTURE}_${TEST}.mp4"
  },
  "disablePageCaching": true,
  "skipUncaughtErrors": true,
  "skipJsErrors": true,
  "pageLoadTimeout": 60000,
  "pageRequestTimeout": 60000,
  "assertionTimeout": 90000,
  "concurrency": 3
}

Your complete test report

No response

Screenshots

No response

Steps to Reproduce

1. Run testcafe with two fixtures and concurrency enabled

TestCafe version

3.5.0

Node.js version

No response

Command-line arguments

using testcafe runner api

Browser name(s) and version(s)

No response

Platform(s) and version(s)

No response

Other

No response

[Bayheck]

Hello,
This behavior is intended.
disableConcurrency disables concurrent test execution for a specific fixture.
Concurrent test execution is not suitable for tests that can only run in a certain order.

The documentation reference implies that fixtures have 'after hooks', however, you don't use 'after hooks' in your example.

[codambro]

I tried that as well, both fixture .after and test .after hooks didn't make a difference:

fixture("Fixture 1").after(async ctx => {
    console.log("Fixture 1 after hook");
}).disableConcurrency;

test("Test 1", async t => {
    console.log("Starting Test 1");
    await t.wait(30000);
    console.log("Finished Test 1");
}).after(async t => {
    console.log("Test 1 after hook");
});

Test 2 still started before fixture 1 completed. Also I don't expect disableConcurrency to make a difference here. Just added it to try and see if it helped but it did not.

[Bayheck]

Thank you for your response,
It turns out that this was a documentation issue.
So, as mentioned earlier, concurrent test execution is not suitable for tests that can only run in a certain order.

[codambro]

I know this is old, but is there any workaround for it?

I have two fixtures. Each contains tests that can run in parallel amongst themselves. But having both fixtures in parallel causes problems. I'd really like to run one fixture after the other, but the tests within them in parallel.