[Nix#9456] Non-flake file input records recursive hash in `flake.lock` but using flat hash in store path (mismatched hash mode) #95

New issue

Open

opened 2024-03-16 06:44:52 +00:00 by lix-bot · 0 comments

lix-bot commented

2024-03-16 06:44:52 +00:00

Member

Upstream-Issue: NixOS/nix#9456

Describe the bug

As in the title.

Steps To Reproduce

Given this flake.nix:

{
  inputs.some-file = {
    flake = false;
    url = "https://raw.githubusercontent.com/NixOS/nix/7304806241fe4f72c5f33a5a929d675c8342fabd/flake.nix";
  };

  outputs = { some-file, ... }: {
    foo = some-file.outPath;
  };
}

Run nix eval .#foo gives /nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source. The generated flake.lock is:

{
  "nodes": {
    "root": {
      "inputs": {
        "some-file": "some-file"
      }
    },
    "some-file": {
      "flake": false,
      "locked": {
        "narHash": "sha256-D4N+MuIfbF+WogLprhLa5iQACL0tU1OKVqFhkQNf10A=",
        "type": "file",
        "url": "https://raw.githubusercontent.com/NixOS/nix/7304806241fe4f72c5f33a5a929d675c8342fabd/flake.nix"
      },
      "original": {
        "type": "file",
        "url": "https://raw.githubusercontent.com/NixOS/nix/7304806241fe4f72c5f33a5a929d675c8342fabd/flake.nix"
      }
    }
  },
  "root": "root",
  "version": 7
}

Note that the evaluated some-file.outPath and the locked narHash are using different hash mode. The former is a path to flat content hash FOD, while the latter is a recursive NAR hash. This can be verified by:

$ nix hash file /nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source
sha256-/UF2l1qK3ojTAtvZHWs7PvFXCF/3VlBIeWYLGv4zLwg=
$ nix eval --expr '(derivation { name = "source"; system = "dummy"; builder = "dummy"; outputHashMode = "flat"; outputHash = "sha256-/UF2l1qK3ojTAtvZHWs7PvFXCF/3VlBIeWYLGv4zLwg="; }).outPath'
"/nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source"
$ nix hash path /nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source
sha256-D4N+MuIfbF+WogLprhLa5iQACL0tU1OKVqFhkQNf10A=

Mismatched hash modes caused various effects:

The store path can only be calculated after the content is downloaded, and is not possible with flake.lock alone, unlike other recursive hashed inputs.
As long as the fetch cache (~/.config/nix/fetcher-cache-v1.sqlite) is invalidated, there's no way to evaluate the flake, even though the input /nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source is intact. Because you cannot calculate the store path itself!
This can be observed by rm ~/.config/nix/fetcher-cache-v1.sqlite*; echo >>flake.nix; nix eval .#foo. The input is downloaded and locked, but is still refetched every time.
Even though the nar hash <-> store path relationship is maintained in database ValidPath. hash column (narHash) is not indexed. So it's practically impossible to get store path given nar hash of a flat hash FOD without a full table scan.
This makes downstream tools (eg. nil, see the link below) unable to check the existence of flake input, because Nix itself cannot either.

Expected behavior

Possible solutions are:

Record flat hash (instead of, or coexist with narHash) inside flake.lock for inputs whose type="file".
Also record storePath for all inputs in flake.lock.

Personally I prefer solution (2). It also provides a cheap way to get locked dependency paths without nix flake archive which currently dumps the full flake directory.

nix-env --version output

nix (Nix) 2.18.1

Additional context

Observed in https://github.com/oxalica/nil/issues/113, thanks to @MathiasSven.

May be related: #9303.

Priorities

Add 👍 to issues you find important.

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/9456 **Describe the bug** As in the title. **Steps To Reproduce** Given this `flake.nix`: ```nix { inputs.some-file = { flake = false; url = "https://raw.githubusercontent.com/NixOS/nix/7304806241fe4f72c5f33a5a929d675c8342fabd/flake.nix"; }; outputs = { some-file, ... }: { foo = some-file.outPath; }; } ``` Run `nix eval .#foo` gives `/nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source`. The generated `flake.lock` is: ```json { "nodes": { "root": { "inputs": { "some-file": "some-file" } }, "some-file": { "flake": false, "locked": { "narHash": "sha256-D4N+MuIfbF+WogLprhLa5iQACL0tU1OKVqFhkQNf10A=", "type": "file", "url": "https://raw.githubusercontent.com/NixOS/nix/7304806241fe4f72c5f33a5a929d675c8342fabd/flake.nix" }, "original": { "type": "file", "url": "https://raw.githubusercontent.com/NixOS/nix/7304806241fe4f72c5f33a5a929d675c8342fabd/flake.nix" } } }, "root": "root", "version": 7 } ``` Note that the evaluated `some-file.outPath` and the locked `narHash` are using different hash mode. The former is a path to flat content hash FOD, while the latter is a recursive NAR hash. This can be verified by: ```console $ nix hash file /nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source sha256-/UF2l1qK3ojTAtvZHWs7PvFXCF/3VlBIeWYLGv4zLwg= $ nix eval --expr '(derivation { name = "source"; system = "dummy"; builder = "dummy"; outputHashMode = "flat"; outputHash = "sha256-/UF2l1qK3ojTAtvZHWs7PvFXCF/3VlBIeWYLGv4zLwg="; }).outPath' "/nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source" $ nix hash path /nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source sha256-D4N+MuIfbF+WogLprhLa5iQACL0tU1OKVqFhkQNf10A= ``` Mismatched hash modes caused various effects: 1. The store path can only be calculated after the content is downloaded, and is not possible with `flake.lock` alone, unlike other recursive hashed inputs. 2. As long as the fetch cache (`~/.config/nix/fetcher-cache-v1.sqlite`) is invalidated, there's no way to evaluate the flake, even though the input `/nix/store/z3ra5xqj5ac87ccs5r3604afi13ar1rj-source` is intact. Because you cannot calculate the store path itself! This can be observed by `rm ~/.config/nix/fetcher-cache-v1.sqlite*; echo >>flake.nix; nix eval .#foo`. The input is downloaded and locked, but is still refetched every time. 3. Even though the nar hash <-> store path relationship is maintained in database `ValidPath`. [`hash` column (narHash) is not indexed](https://github.com/NixOS/nix/blob/f25c06d7a3289343887d09761c82356a3b6b441b/src/libstore/schema.sql#L4). So it's practically impossible to get store path given nar hash of a flat hash FOD without a full table scan. 4. This makes downstream tools (eg. `nil`, see the link below) unable to check the existence of flake input, because Nix itself cannot either. **Expected behavior** Possible solutions are: 1. Record flat hash (instead of, or coexist with `narHash`) inside `flake.lock` for inputs whose type="file". 2. Also record `storePath` for all inputs in `flake.lock`. Personally I prefer solution (2). It also provides a cheap way to get locked dependency paths without `nix flake archive` which currently dumps the full flake directory. **`nix-env --version` output** `nix (Nix) 2.18.1` **Additional context** Observed in https://github.com/oxalica/nil/issues/113, thanks to @MathiasSven. May be related: #9303. **Priorities** Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).