Test target depends on all executables

[gsauthof]

Say I have a meson build that defines several executables including a unittest-suite executable. Then I define one test for the one unittest executable.

Example:

executable('some_frontend', 'foo_etc.cc')
executable('some_frontend2', 'foo_etc2.cc')
# some more executable definitions
ut = executable('ut', 'some_sources.cc')
test('unittest', ut)

When I execute:

$ ninja-build ut
$ ninja-build test

Expected behavior: The ninja-build test just executes the unittest executable since the test only depends on that executable.

Actual behavior: The ninja-build test also builds all other executables that are defined in the meson build file.

This is caused because meson generates the ninja test target like this:

build test: CUSTOM_COMMAND all PHONY

The expected behavior would require the generation of something like:

build test: CUSTOM_COMMAND ut PHONY

[nirbheek]

This is indeed how make check works with Autotools, and we should probably make it work like this. However, we have to think about how to do this because this will break people's setups.

[jpakkane]

No. The current behaviour is there by design and will not be changed. Ever! In Meson "ninja test" always means running all tests with everything up to date. If I had an euro for every case where people have run tests without everything being up to date because "I know that that I only need to build X for this" and then failing spectacularly I could spend the rest of my life under a very comfortable tree in the Bahamas.

That being said what we can do, and AFAICR Nirbheek has already looked into this, is to make this possible with Mesontest. So you could do the equivalent of "mesontest --only-rebuild-what-is-needed testname" which would bring that target and everything it depends on up to date. This is still dangerous and has failure potential, but since it is explicit, people are prepared for weird issues.

[nirbheek]

If I had an euro for every case where people have run tests without everything being up to date because "I know that that I only need to build X for this" and then failing spectacularly I could spend the rest of my life under a very comfortable tree in the Bahamas.

The whole point of Meson is that you can actually ensure that this will not happen. What's the point of tracking all dependencies carefully if we just throw all that information away and build all the targets before running tests?

So you could do the equivalent of "mesontest --only-rebuild-what-is-needed testname" which would bring that target and everything it depends on up to date. This is still dangerous and has failure potential, but since it is explicit, people are prepared for weird issues.

Honestly, while running specific tests, we should definitely not build everything that is out of date (like we do right now). It is a total and complete waste of time in larger projects. In fact, the decision to always build everything has caused people to not bother with properly specifying dependencies in test() and made the transition to sanity worse.

[jpakkane]

The whole point of Meson is that you can actually ensure that this will not happen.

This can not be detected reliably. If you have a test that does internal file system globbing or something else it has a secret dependency. Or it may call a built executable that is linked against some shared library that is updated but the other executable is not relinked because it was not listed as a dependency of the test. Or any of the myriad of ways to have a non-obvious or hidden dependency. The only reliable way is to make sure everything is up to date.

[nirbheek]

If you have a test that does internal file system globbing or something else it has a secret dependency.

custom targets can do that too, and yet we handle those just fine with keyword arguments.

Or it may call a built executable that is linked against some shared library that is updated but the other executable is not relinked because it was not listed as a dependency of the test.

Same thing with custom targets, and yet we handle those fine.

Or any of the myriad of ways to have a non-obvious or hidden dependency.

All those are handled just fine with every other meson function!

The only reliable way is to make sure everything is up to date.

Yes, let's give up and always rebuild everything for every target including custom targets. It's the only way to be absolutely sure.

Tests are no different from custom targets or run targets, and there is no reason to treat them specially, especially given how many people expect them to not behave specially. People are going to keep filing bug reports about this till we change it.

[jpakkane]

There are two different issues here. The first one is adding functionality so people can run a subset of tests and we only rebuild those targets that are explicitly mentioned (the exe, its direct dependencies, other targets given on the command line, extra dependency targets). This is good and useful functionality, which we can add to mesontest, no doubt about that. We might even add something like mesontest --minimal-rebuild which does exactly what the original bug message requested.

The second issue is what should ninja test do by default. The current semantics are "make sure everything is up to date and run tests". This is a good default because it is the reliable thing to do.

If sometime in the future we change the default from current to minimal rebuilds (which I consider very unlikely) having it in the mesontest command as an optional thing is good because it allows people to work out the kinks in an opt-in fashion.

[nirbheek]

I find this position completely reasonable. Thanks for discussing this with me!

[gsauthof]

FWIW,

• building everything when having coverage data collection enabled also introduces noise in the baseline that you have to filter out again
• the test target generated by cmake/ctest also doesn't depend on all executables
• I haven't met anyone so far who complained or was burned by the expected behavior I've described
• it is pretty standard to run tests in a clean continuous integration environment (e.g. in a newly instantiated container - like with travis CI) - thus when you really miss to add a dependency on needed additional executables your test just fails

In any case, perhaps you want to update the test() section the reference manual such that the current design is documented.

[jpakkane]

I haven't met anyone so far who [...] was burned by the expected behavior I've described

I have, that is the main reason the current behavior is what it is.

[bonzini]

This is intentional as "ninja test" is a simplified workflow, and I am closing the issue because now the tests track their dependencies in introspection data and meson test uses that when starting individual test. See #7902 and #10413.