nixos-install cannot install nixos-test-driver package with fileset src

[Yarny0]

Describe the bug

There is apparently something strange happening when a package that is using src = lib.fileset is built during a nixos-install invocation. The symptom is that src ends up as an empty directory (failing the build), but I can't see why.

Steps To Reproduce

1. Clone nixpkgs, checkout current nixos-unstable (I used this commit for my tests).

2. Apply the following patch

   diff --git a/nixos/tests/installer.nix b/nixos/tests/installer.nix
   index d57866c9f52c..ea041202ed6e 100644
   --- a/nixos/tests/installer.nix
   +++ b/nixos/tests/installer.nix
   @@ -29,7 +29,7 @@ let
            documentation.enable = false;
   
            # To ensure that we can rebuild the grub configuration on the nixos-rebuild
   -        system.extraDependencies = with pkgs; [ stdenvNoCC ];
   +        system.extraDependencies = with pkgs; [ stdenvNoCC pkgs.nixosTests.installer.simple.driver ];
   
            ${optionalString systemdStage1 "boot.initrd.systemd.enable = true;"}
   
   @@ -570,9 +570,9 @@ let
          nodes = let
            commonConfig = {
              # builds stuff in the VM, needs more juice
   -          virtualisation.diskSize = 8 * 1024;
   +          virtualisation.diskSize = 8 * 1024*3;
              virtualisation.cores = 8;
   -          virtualisation.memorySize = 2048;
   +          virtualisation.memorySize = 2048*3;
   
              # both installer and target need to use the same drive
              virtualisation.diskImage = "./target.qcow2";
   @@ -640,6 +640,11 @@ let
                # curl's tarball, we see what it's trying to download
                curl
              ]
   +
   +          ++ [ nixos-render-docs ]
   +          ++ (with (pkgs.callPackage ../lib/test-driver {}); nativeBuildInputs ++ propagatedBuildInputs)
   +          ++ (with jq; buildInputs ++ nativeBuildInputs ++ [ src ])
   +          ++ (with python3Packages; [ jq pyflakes ])
              ++ optionals (bootLoader == "grub") (let
                zfsSupport = extraInstallerConfig.boot.supportedFilesystems.zfs or false;
              in [

   This should modify the installer.simple vm test such the installed system also contains the nixos-test-driver-1.1 package. The additional extraDependencies are needed to build the test driver package inside the vm without network access.

3. Run nix build -f ${NIXPKGS_CHECKOUT_PATH} nixosTests.installer.simple.

   The test fails: vm-test-run-installer-simple> installer # ERROR Source /build/source does not appear to be a Python project: no pyproject.toml or setup.py

4. Revert the commit b3ad661 while keeping the patch above.

5. Run nix build -f ${NIXPKGS_CHECKOUT_PATH} nixosTests.installer.simple again.

   The test succeeds, it will also install nixos-test-driver-1.1 as an extraDependency on the target system as expected.

Expected behavior

The patched test should succeed in all circumstances, i.e., even with b3ad661 applied.

Additional context

This behaviour stumps me -- I investigated as far as my imagination allowed me to do, but I can't see any root cause for what is happening here.

Here are some additional observations:

• The direct cause of the build failure is that the src is effectively empty. (I discovered that after investigating the failing derivation with nix derivation show and looking into the derivation inputs.)

• Internally, nixos-install calls

  nix-build --out-link /mnt/${SOME_TEMP_NAME}/system --store /mnt --extra-substituters 'auto?trusted=1' '<nixpkgs/nixos>' -A system -I nixos-config=/mnt/etc/nixos/configuration.nix
  

  which attempts to build the broken package. This also fails if called by hand. If one removes the --store and --extra-substituters arguments, hence calls

  nix-build --out-link /mnt/${SOME_TEMP_NAME}/system '<nixpkgs/nixos>' -A system -I nixos-config=/mnt/etc/nixos/configuration.nix
  

  the build succeeds. So it seems nix interprets the src of the problematic derivation differently, depending on those arguments.

• I recall that if the symlink /mnt/nix/var/nix/profiles/per-user/root/channels is removed and replaced with a directory with the same content (I did this with rsync -lr channels-1-link/ channels/), then nixos-install succeeds. This suggests that symlink handling is contributing to the problem. However, I'm unable to reproduce it.

I encountered this because I have a vm test that installs a machine and includes the test drivers in order to pull in its dependencies. The driver itself isn't strictly needed, so I already found a workaround to pull in those dependencies by other means. Yet I'm reporting my observations here as I suspect this problem might come up again in other circumstances where it might be even more suprising and possibly harder to work around.

Notify maintainers

Notifying fileset author @infinisil .

Sorry! I see it entirely possible that this behaviour is caused by something else (maybe even a bug in nix iteself), and the fileset mechanism is just uncovering it. Yet you might better understand the difference in what filesets are doing compared to a simple ./., and I don't know anyone else to ping.

---

Add a 👍 reaction to issues you find important.

[infinisil]

This is a great report, thank you! I briefly looked into it, but I'm pretty much as clueless 😅

However, this very much reminds me of NixOS/nix#9852, which could be related

[Yarny0]

I think I came a lot closer to the root cause. Here's what I observed by placing trace calls throughout lib/fileset/* and lib/sources.nix:

• The fileset turns up as empty directory because the nonEmtpy filter function in lib/fileset/internal.nix (defined as part of _toSourceFilter) constantly returns false.
• The reason for this is that the paths handed over to nonEmpty are not subpaths of the base path (that is transported in fileset._internalBaseComponents).
• This is apprently caused by builtins.path. That function is called indirectly from fileset.toSource (via lib.sources.cleanSourceWith`). It is supposed to call the filter function for each directory entry within its root path (that is the base path from the fileset), however:
• If the base path is part of a nix store path that is present in /nix/store and in /mnt/nix/store, the base path is /nix/store/hash-somewhere/base, but the path that is given to the filter is /mnt/nix/store/hash-somewhere/base/some-file. The fileset's filter function correcly rejects that file as it is not a subpath of the base path.

So it appears that the second nix store that is given by --store very much contributes to the problem, whereas symlinks seem to be unrelated. Note that builtins.filterSource has the same behaviour.

To me this feels like a bug in builtins.path. I opened a bug report over at the Nix repository: NixOS/nix#11503 . Depending on how it is assessed there I will close the report at hand.

Unfortunatelly I currently don't see a proper workaround for this problem, particularly none that might be implemented within the fileset library to make things like fileset.toSource { root = ./.; fileset = ...; } work reliably in the face of the --store argument.

[Yarny0]

Given NixOS/nix#11503 (comment) it looks like a bug in Nix indeed, so I think we can close the issue at hand. Thanks to @infinisil for agonizing over this!

[infinisil]

@Yarny0 Thank you for the detailed report!