[Nix#9605] preferLocalBuild + IFD causes Nix to prefer local builds even when the system doesn't match #86

New issue

Open

opened 2024-03-16 06:44:50 +00:00 by lix-bot · 0 comments

lix-bot commented

2024-03-16 06:44:50 +00:00

Member

Upstream-Issue: NixOS/nix#9605

Describe the bug

If you use import-from-derivation (IFD) to build a derivation at evaluation time and the derivation specifies preferLocalBuild = true; then Nix will prefer to build the derivation on the local system (instead of a remote builder) even if the derivation's system doesn't match the local system. However, if you don't use IFD (i.e. build the derivation by itself in isolation at build time) then Nix respects the derivation's specified system and won't build it locally if the local system doesn't match.

The original context for this is that we have a build that used haskell.lib.packageSourceOverrides (which uses import-from-derivation under the hood via callCabal2nix) and when we ran the same build for a different system using --system then Nix would still try to build the derivation locally despite system mismatch and then the build fails. However, if we disabled local builds (e.g. using --max-jobs 0) then Nix correctly delegates the build to a remote builder.

Steps To Reproduce

You can reproduce this by saving the following code to flake.nix:

{ inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/23.11";

    flake-utils.url = "github:numtide/flake-utils/v1.0.0";
  };

  outputs = { nixpkgs, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        config = { };

        overlay = self: super: {
          hello =
            let
              identityDerivation = self.runCommand "derivation-to-import"
                { preferLocalBuild = true; }
                ''
                ${self.coreutils}/bin/echo 'x: x' > "$out"
                '';

             id = import identityDerivation;
           in
             id super.hello;
        };

        overlays = [ overlay ];

        pkgs = import nixpkgs { inherit config system overlays; };

      in
        { packages.default = pkgs.hello; }
    );
}

… then if you nix build that for a system other than the current one (my local system is aarch64-darwin) then it will fail like this:

$ nix build --system x86_64-linux             
error:
       … while evaluating the attribute 'hello'

         at /nix/store/z8m0sgbjj55zaw96m3q3b1jlm0nr8a6b-source/flake.nix:13:11:

           12|         overlay = self: super: {
           13|           hello =
             |           ^
           14|             let

       … while calling the 'import' builtin

         at /nix/store/z8m0sgbjj55zaw96m3q3b1jlm0nr8a6b-source/flake.nix:21:19:

           20|
           21|              id = import identityDerivation;
             |                   ^
           22|            in

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: builder for '/nix/store/wbk0wpr8rwf49lqs1w6hdnzfrhxia1jx-derivation-to-import.drv' failed with exit code 126;
       last 1 log lines:
       > /nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin/bash: /nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin/bash: cannot execute binary file
       For full logs, run 'nix log /nix/store/wbk0wpr8rwf49lqs1w6hdnzfrhxia1jx-derivation-to-import.drv'.

This will fail regardless of whether you have configured a remote builder that supports the desired system (e.g. x86_64-linux in the above example) because Nix (incorrectly) prefers to build this derivation locally (and the local system doesn't support the derivation's specified system).

However, if you do any of the following then the build will succeed:

If you change preferLocalBuild to false then it will succeed
If you add --max-jobs 0 to the command (disabling local builds) then it will succeed
If you nix-store --realise /nix/store/…-derivation-to-import.drv then it will succeed

That last bullet point is the primary reason I consider this a bug in Nix and not, say, Nixpkgs, since I'd expect that whether or not a .drv successfully realizes should be independent of whether it is realized at evaluation time (i.e. import from derivation) or build time.

Expected behavior

I expect the original nix build command to succeed. More generally, I expect any derivation to realize the same regardless of whether it is realized at evaluation time or not.

nix-env --version output

nix-env (Nix) 2.18.1

Additional context

For reference, the Nix manual has this to say about preferLocalBuild:

If this attribute is set to true and distributed building is enabled, then, if possible, the derivation will be built locally instead of forwarded to a remote machine. This is appropriate for trivial builders where the cost of doing a download or remote build would exceed the cost of building locally.

… emphasis on "if possible". Nix should only prefer building locally if the local system matches.

Priorities

Add 👍 to issues you find important.

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/9605 **Describe the bug** If you use import-from-derivation (IFD) to build a derivation at evaluation time and the derivation specifies `preferLocalBuild = true;` then Nix will prefer to build the derivation on the local system (instead of a remote builder) even if the derivation's system doesn't match the local system. However, if you don't use IFD (i.e. build the derivation by itself in isolation at build time) then Nix respects the derivation's specified system and won't build it locally if the local system doesn't match. The original context for this is that we have a build that used `haskell.lib.packageSourceOverrides` (which uses import-from-derivation under the hood via `callCabal2nix`) and when we ran the same build for a different system using `--system` then Nix would still try to build the derivation locally despite system mismatch and then the build fails. However, if we disabled local builds (e.g. using `--max-jobs 0`) then Nix correctly delegates the build to a remote builder. **Steps To Reproduce** You can reproduce this by saving the following code to `flake.nix`: ```nix { inputs = { nixpkgs.url = "github:NixOS/nixpkgs/23.11"; flake-utils.url = "github:numtide/flake-utils/v1.0.0"; }; outputs = { nixpkgs, flake-utils, ... }: flake-utils.lib.eachDefaultSystem (system: let config = { }; overlay = self: super: { hello = let identityDerivation = self.runCommand "derivation-to-import" { preferLocalBuild = true; } '' ${self.coreutils}/bin/echo 'x: x' > "$out" ''; id = import identityDerivation; in id super.hello; }; overlays = [ overlay ]; pkgs = import nixpkgs { inherit config system overlays; }; in { packages.default = pkgs.hello; } ); } ``` … then if you `nix build` that for a system other than the current one (my local system is `aarch64-darwin`) then it will fail like this: ```ShellSession $ nix build --system x86_64-linux error: … while evaluating the attribute 'hello' at /nix/store/z8m0sgbjj55zaw96m3q3b1jlm0nr8a6b-source/flake.nix:13:11: 12| overlay = self: super: { 13| hello = | ^ 14| let … while calling the 'import' builtin at /nix/store/z8m0sgbjj55zaw96m3q3b1jlm0nr8a6b-source/flake.nix:21:19: 20| 21| id = import identityDerivation; | ^ 22| in (stack trace truncated; use '--show-trace' to show the full trace) error: builder for '/nix/store/wbk0wpr8rwf49lqs1w6hdnzfrhxia1jx-derivation-to-import.drv' failed with exit code 126; last 1 log lines: > /nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin/bash: /nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin/bash: cannot execute binary file For full logs, run 'nix log /nix/store/wbk0wpr8rwf49lqs1w6hdnzfrhxia1jx-derivation-to-import.drv'. ``` This will fail regardless of whether you have configured a remote builder that supports the desired `system` (e.g. `x86_64-linux` in the above example) because Nix (incorrectly) prefers to build this derivation locally (and the local system doesn't support the derivation's specified system). However, if you do any of the following then the build will succeed: - If you change `preferLocalBuild` to `false` then it will succeed - If you add `--max-jobs 0` to the command (disabling local builds) then it will succeed - If you `nix-store --realise /nix/store/…-derivation-to-import.drv` then it will succeed That last bullet point is the primary reason I consider this a bug in Nix and not, say, Nixpkgs, since I'd expect that whether or not a `.drv` successfully realizes should be independent of whether it is realized at evaluation time (i.e. import from derivation) or build time. **Expected behavior** I expect the original `nix build` command to succeed. More generally, I expect any derivation to realize the same regardless of whether it is realized at evaluation time or not. **`nix-env --version` output** ``` nix-env (Nix) 2.18.1 ``` **Additional context** For reference, the Nix manual has this to say about [`preferLocalBuild`](https://nixos.org/manual/nix/stable/language/advanced-attributes#adv-attr-preferLocalBuild): > If this attribute is set to true and [distributed building is enabled](https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds), then, if possible, the derivation will be built locally instead of forwarded to a remote machine. This is appropriate for trivial builders where the cost of doing a download or remote build would exceed the cost of building locally. … emphasis on "if possible". Nix should only prefer building locally if the local system matches. **Priorities** Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).