ninja backend: don't rebuild all tests by default

When running `ninja all` we shouldn't build testsuite programs as these
might not be wanted e.g. in order to just install the project. We do
want them to be built when running `ninja test`. Since meson 0.63 we
actually have a dedicated ninja alias for test dependencies -- move
these from the "all" rule to the dedicated test/benchmark rules.

Closes: mesonbuild#1704
Closes: mesonbuild#1949
Closes: mesonbuild#2518
Closes: mesonbuild#3662
Closes: mesonbuild#5728
Closes: mesonbuild#5867
Closes: mesonbuild#6511
Closes: mesonbuild#11317
Closes: mesonbuild#13378

docs/markdown/snippets/test_dependencies.md

@@ -0,0 +1,29 @@
+## Test targets no longer built by default
+
+`meson test` and the `ninja all` rule have been reworked to no longer force
+unnecessary rebuilds.
+
+`meson test` was invoking `ninja all` due to a bug if the chosen set of tests
+had no build dependencies. The behavior is now the same as when tests do have
+build dependencies, i.e. to only build the actual set of targets that are used
+by the test. This change could cause failures when upgrading to Meson 1.7.0, if
+the dependencies are not specified correctly in meson.build. Using `ninja test`
+has always been guaranteed to "do the right thing" and rebuild `all` as well;
+this continues to work.
+
+`ninja all` does not rebuild all tests anymore; it should be noted that this
+change means test programs are no longer guaranteed to have been built,
+depending on whether those test programs were *also* defined to build by
+default / marked as installable. This avoids building test-only binaries as
+part of installing the project (`ninja && ninja install`), which is unnecessary
+and has no use case.
+
+Some users might have been relying on the "all" target building test
+dependencies in combination with `meson test --no-rebuild` in order to skip
+calling out to ninja when running tests. This might break with this change
+because, when given `--no-rebuild`, Meson provides no guarantee that test
+dependencies are present and up to date. The recommended workflow is to use
+either `ninja test` or `ninja && meson test` but, if you wish to build test
+programs and dependencies in a separate stage, you can use for example `ninja
+all meson-test-prereq meson-benchmark-prereq` before `meson test --no-rebuild`.
+These prereq targets have been available since meson 0.63.0.

mesonbuild/backend/ninjabackend.py

@@ -1327,7 +1327,7 @@ def generate_tests(self) -> None:
             cmd += ['--no-stdsplit']
         if self.environment.coredata.get_option(OptionKey('errorlogs')):
             cmd += ['--print-errorlogs']
-        elem = self.create_phony_target('test', 'CUSTOM_COMMAND', ['all', 'PHONY'])
+        elem = self.create_phony_target('test', 'CUSTOM_COMMAND', ['all', 'meson-test-prereq', 'PHONY'])
         elem.add_item('COMMAND', cmd)
         elem.add_item('DESC', 'Running all tests')
         elem.add_item('pool', 'console')
@@ -1337,7 +1337,7 @@ def generate_tests(self) -> None:
         cmd = self.environment.get_build_command(True) + [
             'test', '--benchmark', '--logbase',
             'benchmarklog', '--num-processes=1', '--no-rebuild']
-        elem = self.create_phony_target('benchmark', 'CUSTOM_COMMAND', ['all', 'PHONY'])
+        elem = self.create_phony_target('benchmark', 'CUSTOM_COMMAND', ['all', 'meson-benchmark-prereq', 'PHONY'])
         elem.add_item('COMMAND', cmd)
         elem.add_item('DESC', 'Running benchmark suite')
         elem.add_item('pool', 'console')
@@ -3740,10 +3740,6 @@ def generate_ending(self) -> None:
                 ('meson-test-prereq', self.get_testlike_targets()),
                 ('meson-benchmark-prereq', self.get_testlike_targets(True))]:
             targetlist = []
-            # These must also be built by default.
-            # XXX: Sometime in the future these should be built only before running tests.
-            if targ == 'all':
-                targetlist.extend(['meson-test-prereq', 'meson-benchmark-prereq'])
             for t in deps.values():
                 # Add the first output of each target to the 'all' target so that
                 # they are all built

run_project_tests.py

@@ -712,7 +712,14 @@ def _run_test(test: TestDef,
     # Build with subprocess
     def build_step() -> None:
         build_start = time.time()
-        pc, o, _ = Popen_safe(compile_commands + dir_args, cwd=test_build_dir, stderr=subprocess.STDOUT)
+
+        if backend is Backend.ninja:
+            # FIXME: meson test inprocess does not handle running ninja via StringIO
+            targets = ['all', 'meson-test-prereq', 'meson-benchmark-prereq']
+        else:
+            targets = []
+
+        pc, o, _ = Popen_safe(compile_commands + dir_args + targets, cwd=test_build_dir, stderr=subprocess.STDOUT)
         testresult.add_step(BuildStep.build, o, '', '', time.time() - build_start)
         if should_fail == 'build':
             if pc.returncode != 0: