lix-project/lix

Fork 57

[Nix#10104] `fetchTree` performs fetch even when a `narHash` is provided #52

New issue

Open

opened 2024-03-16 06:44:39 +00:00 by lix-bot · 2 comments

lix-bot commented

2024-03-16 06:44:39 +00:00

Member

Upstream-Issue: NixOS/nix#10104

Describe the bug

According to the docs, when you add narHash to a fetchTree call, the result won't be fetched if it's already available: NixOS/nix@864fc85fc8/src/libexpr/primops/fetchTree.cc (L218)

Well, that's not happening.

Steps To Reproduce

The 1st time you fetch, it downloads ✅:

➤ nix eval --no-eval-cache --debug --expr 'builtins.fetchTree {type="github"; owner="OCA"; repo="account-analytic"; rev="bc8dbde0477634d64915d027093756defcf0ebc3"; narHash="sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc=";}'
did not find cache entry for '{"rev":"bc8dbde0477634d64915d027093756defcf0ebc3","type":"git-tarball"}'
did not find cache entry for '{"name":"source","type":"tarball","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}'
did not find cache entry for '{"name":"source","type":"file","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}'
downloading 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'...
starting download of https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz
finished download of 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'; curl status = 0, HTTP status = 200, body = 1089274 bytes
performing daemon worker op: 39
acquiring write lock on '/nix/var/nix/temproots/575565'
performing daemon worker op: 7
performing daemon worker op: 26
{ lastModified = 1697447248; lastModifiedDate = "20231016090728"; narHash = "sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc="; outPath = "/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source"; rev = "bc8dbde0477634d64915d027093756defcf0ebc3"; shortRev = "bc8dbde"; }
download thread shutting down

The next time you download, it says "using cache entry". That's suspicious 🕵️‍♂️. It shouldn't need any cache because there's a narHash parameter passed:

➤ nix eval --no-eval-cache --debug --expr 'builtins.fetchTree {type="github"; owner="OCA"; repo="account-analytic"; rev="bc8dbde0477634d64915d027093756defcf0ebc3"; narHash="sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc=";}'
performing daemon worker op: 11
acquiring write lock on '/nix/var/nix/temproots/575669'
performing daemon worker op: 1
using cache entry '{"rev":"bc8dbde0477634d64915d027093756defcf0ebc3","type":"git-tarball"}' -> '{"lastModified":1697447248,"rev":"bc8dbde0477634d64915d027093756defcf0ebc3"}', '/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source'
performing daemon worker op: 26
{ lastModified = 1697447248; lastModifiedDate = "20231016090728"; narHash = "sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc="; outPath = "/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source"; rev = "bc8dbde0477634d64915d027093756defcf0ebc3"; shortRev = "bc8dbde"; }

Let's remove cache and retry. We can see how fetches again. ❌ It shouldn't!:

➤ rm -rf ~/.cache/nix

➤ nix eval --no-eval-cache --debug --expr 'builtins.fetchTree {type="github"; owner="OCA"; repo="account-analytic"; rev="bc8dbde0477634d64915d027093756defcf0ebc3"; narHash="sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc=";}'
did not find cache entry for '{"rev":"bc8dbde0477634d64915d027093756defcf0ebc3","type":"git-tarball"}'
did not find cache entry for '{"name":"source","type":"tarball","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}'
did not find cache entry for '{"name":"source","type":"file","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}'
downloading 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'...
starting download of https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz
finished download of 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'; curl status = 0, HTTP status = 200, body = 1089274 bytes
performing daemon worker op: 39
acquiring write lock on '/nix/var/nix/temproots/575717'
performing daemon worker op: 7
performing daemon worker op: 26
{ lastModified = 1697447248; lastModifiedDate = "20231016090728"; narHash = "sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc="; outPath = "/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source"; rev = "bc8dbde0477634d64915d027093756defcf0ebc3"; shortRev = "bc8dbde"; }
download thread shutting down

Expected behavior

When calling with a NAR hash, it shouldn't need to even check the cache. Nix should just check the store and skip the fetching/caching completely.

nix-env --version output
nix-env (Nix) 2.18.1

Additional context

This might be duplicate of https://github.com/NixOS/nix/issues/9077 or https://github.com/NixOS/nix/issues/9570, but I'm not sure so I open separately.

Priorities

Add 👍 to issues you find important.

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/10104 **Describe the bug** According to the docs, when you add `narHash` to a `fetchTree` call, the result won't be fetched if it's already available: https://github.com/NixOS/nix/blob/864fc85fc88ff092725ba99907611b2b8d2205fb/src/libexpr/primops/fetchTree.cc#L218 Well, that's not happening. **Steps To Reproduce** The 1st time you fetch, it downloads ✅: ```shell ➤ nix eval --no-eval-cache --debug --expr 'builtins.fetchTree {type="github"; owner="OCA"; repo="account-analytic"; rev="bc8dbde0477634d64915d027093756defcf0ebc3"; narHash="sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc=";}' did not find cache entry for '{"rev":"bc8dbde0477634d64915d027093756defcf0ebc3","type":"git-tarball"}' did not find cache entry for '{"name":"source","type":"tarball","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}' did not find cache entry for '{"name":"source","type":"file","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}' downloading 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'... starting download of https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz finished download of 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'; curl status = 0, HTTP status = 200, body = 1089274 bytes performing daemon worker op: 39 acquiring write lock on '/nix/var/nix/temproots/575565' performing daemon worker op: 7 performing daemon worker op: 26 { lastModified = 1697447248; lastModifiedDate = "20231016090728"; narHash = "sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc="; outPath = "/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source"; rev = "bc8dbde0477634d64915d027093756defcf0ebc3"; shortRev = "bc8dbde"; } download thread shutting down ``` The next time you download, it says "using cache entry". That's suspicious 🕵️‍♂️. It shouldn't need any cache because there's a `narHash` parameter passed: ```shell ➤ nix eval --no-eval-cache --debug --expr 'builtins.fetchTree {type="github"; owner="OCA"; repo="account-analytic"; rev="bc8dbde0477634d64915d027093756defcf0ebc3"; narHash="sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc=";}' performing daemon worker op: 11 acquiring write lock on '/nix/var/nix/temproots/575669' performing daemon worker op: 1 using cache entry '{"rev":"bc8dbde0477634d64915d027093756defcf0ebc3","type":"git-tarball"}' -> '{"lastModified":1697447248,"rev":"bc8dbde0477634d64915d027093756defcf0ebc3"}', '/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source' performing daemon worker op: 26 { lastModified = 1697447248; lastModifiedDate = "20231016090728"; narHash = "sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc="; outPath = "/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source"; rev = "bc8dbde0477634d64915d027093756defcf0ebc3"; shortRev = "bc8dbde"; } ``` Let's remove cache and retry. We can see how fetches again. ❌ It shouldn't!: ```shell ➤ rm -rf ~/.cache/nix ➤ nix eval --no-eval-cache --debug --expr 'builtins.fetchTree {type="github"; owner="OCA"; repo="account-analytic"; rev="bc8dbde0477634d64915d027093756defcf0ebc3"; narHash="sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc=";}' did not find cache entry for '{"rev":"bc8dbde0477634d64915d027093756defcf0ebc3","type":"git-tarball"}' did not find cache entry for '{"name":"source","type":"tarball","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}' did not find cache entry for '{"name":"source","type":"file","url":"https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz"}' downloading 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'... starting download of https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz finished download of 'https://github.com/OCA/account-analytic/archive/bc8dbde0477634d64915d027093756defcf0ebc3.tar.gz'; curl status = 0, HTTP status = 200, body = 1089274 bytes performing daemon worker op: 39 acquiring write lock on '/nix/var/nix/temproots/575717' performing daemon worker op: 7 performing daemon worker op: 26 { lastModified = 1697447248; lastModifiedDate = "20231016090728"; narHash = "sha256-sD75xS86JRr7q1be+U8FMI7ijDj4tgexlyNGVEa8fpc="; outPath = "/nix/store/q3fw9j28hdlyal51xgqlrac9vcb59p2f-source"; rev = "bc8dbde0477634d64915d027093756defcf0ebc3"; shortRev = "bc8dbde"; } download thread shutting down ``` **Expected behavior** When calling with a NAR hash, it shouldn't need to even check the cache. Nix should just check the store and skip the fetching/caching completely. **`nix-env --version` output** nix-env (Nix) 2.18.1 **Additional context** This might be duplicate of https://github.com/NixOS/nix/issues/9077 or https://github.com/NixOS/nix/issues/9570, but I'm not sure so I open separately. **Priorities** Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).

lix-bot added the

bug

imported

labels

2024-03-16 06:44:39 +00:00

jade added the

Area/fetching

label

2024-10-20 01:05:05 +00:00

lix-project referenced this issue from a commit

2025-01-29 18:11:12 +00:00

Merge pull request #52 from jsoo1/gc-dont-gc

commentatorforall commented

2025-11-23 21:44:38 +00:00

Member

according to upstream issue, this has been fixed in cppnix 2.19; needs to be checked for persistence

commentatorforall added the

awaiting

contributors

label

2025-11-23 21:45:06 +00:00

tom-hubrecht commented

2025-11-23 23:03:35 +00:00

Member

It's a little more complicated than this, fetchTree will only check the store and bypass the cache when narHash is present, and the fetcher type deems that enough info is given (which info varies per type), it is in the hasAllInfo method, which:

for git type, needs lastModified and either shallow = true, revCount to be set or have no ref
for github type, needs a rev and lastModified
for tarball and path is always true
for mercurial, have default as a ref or revCount to be set

It's a little more complicated than this, `fetchTree` will only check the store and bypass the cache when `narHash` is present, and the fetcher type deems that enough info is given (which info varies per type), it is in the `hasAllInfo` method, which: - for `git` type, needs `lastModified` and either `shallow = true`, `revCount` to be set or have no `ref` - for `github` type, needs a `rev` and `lastModified` - for `tarball` and `path` is always true - for `mercurial`, have `default` as a ref or `revCount` to be set