Bug: No way to affect the order in which test cases are run

[torokati44]

Description of the issue

There is no way to affect the order in which the test cases are executed.

It would be nice to be able to do this, see:
#905 (comment)

Also:
ruffle-rs/ruffle#17961 (comment)
ruffle-rs/ruffle#15950

Expected outcome

Tests are executed in the order in which they are listed - at least optionally.

Actual result

Test cases are always sorted alphabetically.

Nextest version

0.9.81

Additional context

No response

[sunshowers]

This would be very valuable, I agree -- I think I can cook up something basic this weekend. (This would improve a situation at work as well.)

[joonas]

I just wanted to echo I would love to be able to customize (or at the least, shuffle) the order in which tests run, recently ran into some test ordering related issues that this should help iron out 🙂

[sunshowers]

Sadly due to an emergency my free time has completely evaporated. I'd still like to get to this.

[Estus-Dev]

If it's within the scope of this issue: It would be very nice just to be able to order which crate's tests are run first. (I'm running the vanilla test runner right now and a friend pointed me to this issue/project and suggested I ask here)

I'm developing an emulator in a multi-crate workspace, and I have two relevant crates: my core and my CPU. The CPU doesn't depend on anything in the workspace, but my core depends on the CPU. If a single CPU test fails, I don't care about the results of testing the core because I expect most of those tests to now fail.

Additionally, the CPU tests are pretty fast, just looping over some JSON data to test each of the 500 instructions 1,000 times each. The core's tests on the other hand are substantially slower. They initialize a full emulator and run a test ROM to completion, then compare state at the end. I'd estimate each test ROM performs at least as much work as all the CPU tests together.

So when I run the tests and it runs the slower core tests first (failing most of them) and then early exits before running the CPU tests, it's a waste of time on every front, and misleading to boot. It'd be much more useful to run the CPU tests first, then the core tests only if the CPU tests pass.

[sunshowers]

All right -- we realized this would be beneficial at my workplace, so nextest 0.9.91 has support for this: https://nexte.st/docs/configuration/test-priorities/

(As you might imagine, most of the work here is in testing and documenting the feature, not in writing the code itself :) )