lix-project/lix

Fork 14

[Nix#8720] `builtins.fetchurl` does not try to refetch changed contents of `file://` URI's #131

New issue

Open

opened 2024-03-16 06:45:04 +00:00 by lix-bot · 0 comments

lix-bot commented

2024-03-16 06:45:04 +00:00

Member

Upstream-Issue: NixOS/nix#8720

Describe the bug

Once builtins.fetchurl fetched a file:// URI, it seems to assume it never changes and does not try to rehash it. After changing the file and clearing the hash, the hash of the old file is given in the hash mismatch error, not the one of the changed file.

This work is sponsored by Antithesis ✨

Steps To Reproduce

Create a file and fetch it with fetchurl:

$ echo a > f
$ hash=$(nix-hash --flat --base32 --type sha256 f)
$ echo "$hash"
1ir5rz261phggayppq7565k0jxg47zq3rkkvbq332gc04b2qyhl7
$ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }'
"/nix/store/kpbjaryhvz5xckkvry7fnj4zmpnslp61-f"

Change the contents of the file and try to fetch it with fetchurl again:

$ echo b > f
$ hash=$(nix-hash --flat --base32 --type sha256 f)
$ echo "$hash"
0gw1k5p8z07ahqlyfk9dd4fz1qn29g32zbxsf97rbzdni6cq4qq2
$ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }'
    error: hash mismatch in file downloaded from 'file:///home/tweagysil/antithesis/nixpkgs/f':
      specified: sha256:0gw1k5p8z07ahqlyfk9dd4fz1qn29g32zbxsf97rbzdni6cq4qq2
      got:       sha256:1ir5rz261phggayppq7565k0jxg47zq3rkkvbq332gc04b2qyhl7

Notice how it errors, saying to use the hash of the previous contents of the file.

Workarounds

We can work around this by removing the old store path:

$ nix-store --delete /nix/store/kpbjaryhvz5xckkvry7fnj4zmpnslp61-f
finding garbage collector roots...
deleting '/nix/store/kpbjaryhvz5xckkvry7fnj4zmpnslp61-f'
deleting unused links...
note: currently hard linking saves 15891.46 MiB
1 store paths deleted, 0.00 MiB freed

$ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }'
"/nix/store/d36kr8s4rxvz6w4ynw9a2s067g2k8af9-f"

Notice how the import of the new file works now.

Another workaround is to add a dummy URI parameter and remove it again:

$ echo c > f
$ hash=$(nix-hash --flat --base32 --type sha256 f)
$ echo "$hash"
0y0lk9vxl2f9q9274f3vnvazycjgq9mvpgprqdrllmycy0ayg9d3
$ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }'
       error: hash mismatch in file downloaded from 'file:///home/tweagysil/antithesis/nixpkgs/f':
         specified: sha256:0y0lk9vxl2f9q9274f3vnvazycjgq9mvpgprqdrllmycy0ayg9d3
         got:       sha256:0gw1k5p8z07ahqlyfk9dd4fz1qn29g32zbxsf97rbzdni6cq4qq2
❯ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}?x=10"; sha256 = "'$hash'"; }'
"/nix/store/kyybs2sb2p4r3ryq2s46hnrdla9h43fc-f"
❯ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }'
"/nix/store/kyybs2sb2p4r3ryq2s46hnrdla9h43fc-f"

And another workaround is to change the name of the store path:

$ echo d > f
$ hash=$(nix-hash --flat --base32 --type sha256 f)
$ echo "$hash"
1glqmhg21jhpif0fxmvdbn4kj26ljbxvl4v8swm355p93gnbwx4d
$ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }'
       error: hash mismatch in file downloaded from 'file:///home/tweagysil/antithesis/nixpkgs/f':
         specified: sha256:1glqmhg21jhpif0fxmvdbn4kj26ljbxvl4v8swm355p93gnbwx4d
         got:       sha256:0y0lk9vxl2f9q9274f3vnvazycjgq9mvpgprqdrllmycy0ayg9d3
$ nix-instantiate --eval -E 'builtins.fetchurl { name = "g"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }'
"/nix/store/8fpq49vd9qr28cjghdpch1ab7kawwi7r-g"

I suspect some overly ambitious URI caching is going on somewhere.

Expected behavior

No workaround should be required, it should just work.

`nix-env --version` output

nix-env (Nix) 2.15.1

Priorities

Add 👍 to issues you find important.

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/8720 ### Describe the bug Once `builtins.fetchurl` fetched a `file://` URI, it seems to assume it never changes and does not try to rehash it. After changing the file and clearing the hash, the hash of the old file is given in the hash mismatch error, not the one of the changed file. This work is sponsored by [Antithesis](https://antithesis.com/) :sparkles: ### Steps To Reproduce Create a file and fetch it with `fetchurl`: ``` $ echo a > f $ hash=$(nix-hash --flat --base32 --type sha256 f) $ echo "$hash" 1ir5rz261phggayppq7565k0jxg47zq3rkkvbq332gc04b2qyhl7 $ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }' "/nix/store/kpbjaryhvz5xckkvry7fnj4zmpnslp61-f" ``` Change the contents of the file and try to fetch it with `fetchurl` again: ``` $ echo b > f $ hash=$(nix-hash --flat --base32 --type sha256 f) $ echo "$hash" 0gw1k5p8z07ahqlyfk9dd4fz1qn29g32zbxsf97rbzdni6cq4qq2 $ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }' error: hash mismatch in file downloaded from 'file:///home/tweagysil/antithesis/nixpkgs/f': specified: sha256:0gw1k5p8z07ahqlyfk9dd4fz1qn29g32zbxsf97rbzdni6cq4qq2 got: sha256:1ir5rz261phggayppq7565k0jxg47zq3rkkvbq332gc04b2qyhl7 ``` Notice how it errors, saying to use the hash of the previous contents of the file. ### Workarounds We can work around this by removing the old store path: ``` $ nix-store --delete /nix/store/kpbjaryhvz5xckkvry7fnj4zmpnslp61-f finding garbage collector roots... deleting '/nix/store/kpbjaryhvz5xckkvry7fnj4zmpnslp61-f' deleting unused links... note: currently hard linking saves 15891.46 MiB 1 store paths deleted, 0.00 MiB freed $ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }' "/nix/store/d36kr8s4rxvz6w4ynw9a2s067g2k8af9-f" ``` Notice how the import of the new file works now. Another workaround is to add a dummy URI parameter and remove it again: ``` $ echo c > f $ hash=$(nix-hash --flat --base32 --type sha256 f) $ echo "$hash" 0y0lk9vxl2f9q9274f3vnvazycjgq9mvpgprqdrllmycy0ayg9d3 $ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }' error: hash mismatch in file downloaded from 'file:///home/tweagysil/antithesis/nixpkgs/f': specified: sha256:0y0lk9vxl2f9q9274f3vnvazycjgq9mvpgprqdrllmycy0ayg9d3 got: sha256:0gw1k5p8z07ahqlyfk9dd4fz1qn29g32zbxsf97rbzdni6cq4qq2 ❯ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}?x=10"; sha256 = "'$hash'"; }' "/nix/store/kyybs2sb2p4r3ryq2s46hnrdla9h43fc-f" ❯ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }' "/nix/store/kyybs2sb2p4r3ryq2s46hnrdla9h43fc-f" ``` And another workaround is to change the name of the store path: ``` $ echo d > f $ hash=$(nix-hash --flat --base32 --type sha256 f) $ echo "$hash" 1glqmhg21jhpif0fxmvdbn4kj26ljbxvl4v8swm355p93gnbwx4d $ nix-instantiate --eval -E 'builtins.fetchurl { name = "f"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }' error: hash mismatch in file downloaded from 'file:///home/tweagysil/antithesis/nixpkgs/f': specified: sha256:1glqmhg21jhpif0fxmvdbn4kj26ljbxvl4v8swm355p93gnbwx4d got: sha256:0y0lk9vxl2f9q9274f3vnvazycjgq9mvpgprqdrllmycy0ayg9d3 $ nix-instantiate --eval -E 'builtins.fetchurl { name = "g"; url = "file://${toString ./f}"; sha256 = "'$hash'"; }' "/nix/store/8fpq49vd9qr28cjghdpch1ab7kawwi7r-g" ``` I suspect some overly ambitious URI caching is going on somewhere. ### Expected behavior No workaround should be required, it should just work. ### `nix-env --version` output nix-env (Nix) 2.15.1 ### Priorities Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).

lix-bot added the

bug

imported

labels 2024-03-16 06:45:04 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#131

No description provided.

[Nix#8720] builtins.fetchurl does not try to refetch changed contents of file:// URI's #131