Allow overriding the tarball fetcher #62

Closed

enobayram wants to merge 1 commit from enobayram/master into master

enobayram commented

2023-11-08 12:54:31 +00:00

(Migrated from github.com)

This change allows the caller to replace the fetchTarball used for fetching the inputs of the flake with pkgs.fetchzip, which works as a drop-in replacement. The difference is that builtins.fetchTarball does the fetching at Nix evaluation time, while pkgs.fetchzip does the fetching at build time. The distinction becomes crucial when one attempts to build a flake output inside a recursive-nix derivation. A vanilla nix build ${someFlake} or a nix-build default.nix using the current flake-compat will fail inside a recursive Nix when fetchTarball attempts to download the tarball, but the following derivation (using the flake-compat from this PR) will successfully build (on a system with recursive-nix):

let
  flake-compat = fetchTarball {
    url = "https://github.com/enobayram/flake-compat/archive/6ef9397736efb0f6075188aa3488022d1b85c11d.tar.gz";
    sha256 = "sha256:0sviihalf7lrlg4pfajywck50hi01ksax613kz6mldafb5i2g3cc";
  };
  pkgs = import <nixpkgs> {};
  buildInRec-nix = pkgs.writeText "buildInRec.nix" ''
    {flake}: let
      src = "''${flake}";
      fetchTarball = (import ${pkgs.path} {}).fetchzip;
      defaultNix = (import ${flake-compat} { inherit src fetchTarball; }).defaultNix;
      in defaultNix
  '';
  someExampleFlake = fetchTarball {
    url = "https://github.com/cachix/cachix/archive/67487d306ac5e8725e5eefa4ef50706723b986c5.tar.gz";
    sha256 = "sha256:1ks0ikha2qvr6bcypfzkgjgirvqaw1sxfzdkz9axkfy91jnp5szd";
  };
in pkgs.runCommand "recursive-flake"
  {
    requiredSystemFeatures = [ "recursive-nix" ];
    buildInputs = [ pkgs.nix ];
  }
  ''
    ln -s $(
      nix-build ${buildInRec-nix} \
        --arg flake ${someExampleFlake}
        -A packages.${builtins.currentSystem}.default
    ) > $out
  ''

In the above Nix expression, if you remove the fetchTarball argument from line 11, then the build will fail with:

       error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/tarball/970a59bd19eff3752ce552935687100c46e820a5': Couldn't resolve host name (6)

Building a flake recursively like this allows us to hide its evaluation inside a derivation, which can be a world of difference in some cases (see this discussion thread I've opened a few months ago where I've explained in detail how bad this can get).

This approach essentially allows us to get a kind of nix flake output evaluation caching as part of a larger Nix evaluation.

This change allows the caller to replace the `fetchTarball` used for fetching the inputs of the flake with `pkgs.fetchzip`, which works as a drop-in replacement. The difference is that `builtins.fetchTarball` does the fetching at Nix evaluation time, while `pkgs.fetchzip` does the fetching at build time. The distinction becomes crucial when one attempts to build a flake output inside a `recursive-nix` derivation. A vanilla `nix build ${someFlake}` or a `nix-build default.nix` using the current `flake-compat` will fail inside a recursive Nix when `fetchTarball` attempts to download the tarball, but the following derivation (using the `flake-compat` from this PR) will successfully build (on a system with `recursive-nix`): ``` nix let flake-compat = fetchTarball { url = "https://github.com/enobayram/flake-compat/archive/6ef9397736efb0f6075188aa3488022d1b85c11d.tar.gz"; sha256 = "sha256:0sviihalf7lrlg4pfajywck50hi01ksax613kz6mldafb5i2g3cc"; }; pkgs = import <nixpkgs> {}; buildInRec-nix = pkgs.writeText "buildInRec.nix" '' {flake}: let src = "''${flake}"; fetchTarball = (import ${pkgs.path} {}).fetchzip; defaultNix = (import ${flake-compat} { inherit src fetchTarball; }).defaultNix; in defaultNix ''; someExampleFlake = fetchTarball { url = "https://github.com/cachix/cachix/archive/67487d306ac5e8725e5eefa4ef50706723b986c5.tar.gz"; sha256 = "sha256:1ks0ikha2qvr6bcypfzkgjgirvqaw1sxfzdkz9axkfy91jnp5szd"; }; in pkgs.runCommand "recursive-flake" { requiredSystemFeatures = [ "recursive-nix" ]; buildInputs = [ pkgs.nix ]; } '' ln -s $( nix-build ${buildInRec-nix} \ --arg flake ${someExampleFlake} -A packages.${builtins.currentSystem}.default ) > $out '' ``` In the above Nix expression, if you remove the `fetchTarball` argument from line 11, then the build will fail with: ``` error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/tarball/970a59bd19eff3752ce552935687100c46e820a5': Couldn't resolve host name (6) ``` Building a flake recursively like this allows us to hide its evaluation inside a derivation, which can be a world of difference in some cases (see [this discussion thread](https://discourse.nixos.org/t/flakes-and-recursive-nix/32277) I've opened a few months ago where I've explained in detail how bad this can get). This approach essentially allows us to get a kind of nix flake output evaluation caching as part of a larger Nix evaluation.