`libstore` check for changed target URI may be overzealous #822

New issue

Open

opened 2025-05-08 22:53:19 +00:00 by magneticflux0 · 1 comment

magneticflux0 commented

2025-05-08 22:53:19 +00:00

Describe the bug

While serving jhk2qycb2gl5xjmp3fdp3fhdcni08k9q.narinfo, the garnix cache issued a 302 Found/"Moved Temporarily" redirect from https://cache.garnix.io/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q.narinfo to https://old-cache.garnix.io/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q.narinfo, which subsequently 404'd.
This triggered the check for changed URLs here, which was originally added by this commit.

The intention of the original commit: "immutable url changing implies that the immutable url we got previously was wrong, which is probably a server bug" is correct for the hash in the path changing, but I don't think it should consider the other URI components. It might not even need to consider anything but the final path segment containing the hash, but I'm not sure how easy parsing that out would be.

Steps To Reproduce

I originally encountered this issue resolving /nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2. Running nix path-info --json /nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2 | jq yields:

[
  {
    "deriver": "/nix/store/7jbi4zg290h6cpyg7kkvrs6pj8507yg6-nautilus-47.2.drv",
    "narHash": "sha256-S0hAaBW4qJ5O0juxH2LcoI7VMfL2nQZlXYUJjoByQXM=",
    "narSize": 14458112,
    "path": "/nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2",
    "references": [
      "/nix/store/0f3z5ixg7v9wxsf27p3pc7clsycvxyry-gdk-pixbuf-2.42.12",
      "/nix/store/209a73blm5p97vl9qx1xzh1qkl5g0gc1-gexiv2-0.14.3",
      "/nix/store/21r6fbzp31gafh0zbxv7nc4z0bhmjbfp-wayland-1.23.1",
      "/nix/store/25nkjd6y6rc7ap82frgqlga0sshvb3mx-librsvg-2.59.2",
      "/nix/store/35nvkdwk3qx63sif1c8fv9fdcr12cy8k-gtk4-4.16.12",
      "/nix/store/3q509a68aqbnh3dlv7qjkp6q7nfj7l2g-glib-networking-2.80.1",
      "/nix/store/4s50lxhz3gks66dx7m17laz58x9sy5c8-localsearch-3.8.2",
      "/nix/store/6kg26a99l5bq20zbk1c6a70brma5j84b-gsettings-desktop-schemas-47.1",
      "/nix/store/73gqa6192dkdalb6bcz54cqswcfqhvf3-gnome-desktop-44.1",
      "/nix/store/7idgdwzfkl83m0jifr5plahfg55jppm6-libjxl-0.11.1",
      "/nix/store/9b1y83zz0nk2w8c4x6ig0lnd4kfyckqy-pango-1.56.2",
      "/nix/store/9vrc87acgdzhpqkmfsihf0jir3006mzy-gstreamer-1.26.0",
      "/nix/store/am4ypiz0kjj7diynlk56p3gjxjrsyw89-shared-mime-info-2.4",
      "/nix/store/cyavy4kslnhb23p9mbbfmmfb7kgb0bws-webp-pixbuf-loader-0.2.6",
      "/nix/store/dfq2z3j0krkvjjza5hywvx1g7vv67vdm-gobject-introspection-1.82.0",
      "/nix/store/g3s0z9r7m1lsfxdk8bj88nw8k8q3dmmg-glibc-2.40-66",
      "/nix/store/g5h4kh5jy4648fwqyd43fmp2j90c0k08-libnotify-0.8.4",
      "/nix/store/grxanfsjw37kv7q7hnkwjf0pl1imh9h9-dconf-0.40.0-lib",
      "/nix/store/hcf1varsgkgc9r0l8x3yhjfgp5jlm62h-tinysparql-3.8.2",
      "/nix/store/j0sb2c91h2z7azfp1lp7jq5vh7bl6677-libportal-gtk4-0.9.1",
      "/nix/store/jfn5cq536hfa3dv9p3px6l89b79rwc19-graphene-1.10.8",
      "/nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2",
      "/nix/store/jsb94zxbl6n1ljxl3mwbb3lmzsbznnyj-gnome-user-share-47.2",
      "/nix/store/k1h6h1h02dkhcy0qfiz09njpf91xpxy8-gst-plugins-base-1.26.0",
      "/nix/store/lsya04bhx5bx7nlij201slxjw5kzzy8a-cairo-1.18.2",
      "/nix/store/mglvz0wh458afkvyk7xq1jlhcljjrynl-libcloudproviders-0.3.6",
      "/nix/store/qvya5walwxj7f37j2yclr0mf6z1c4lim-libX11-1.8.12",
      "/nix/store/v92zj1h355zqxfrph394hzmnpr5bd7qa-gnome-autoar-0.4.5",
      "/nix/store/vn0hbbmvc02gi5mczx2wpk5xc5ap00ma-harfbuzz-10.2.0",
      "/nix/store/ysm6ybv02ms2v6lzsx7fnqi2cy937l9x-glib-2.82.5",
      "/nix/store/z3w1grl7jzrnkzzhcww2gckh3mlsqwj6-libadwaita-1.6.4",
      "/nix/store/zlvdm56mss012a8vz8q9c9z2d0qcslrw-gobject-introspection-wrapped-1.82.0"
    ],
    "registrationTime": 1746735146,
    "signatures": [
      "private-nix-cache-1:lj/ifjxio52ZS5qpJPckpn1arSelhYvwDk6nj2kg8UoV7GNyt8SrddNS/CUSKamUHYqbkH8aG6nCMU4WLsN0DQ=="
    ],
    "valid": true
  }
]

but I'm not sure how to cause the check to trigger again. I tried running nix path-info --json /nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2 --store https://cache.garnix.io/ | jq, but that only yielded

this path will be fetched (0.00 MiB download, 13.79 MiB unpacked):
  /nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2
[
  {
    "path": "/nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2",
    "valid": false
  }
]

Expected behavior

Ideally, the changed target check should be as specific as possible to allow cache providers to redirect to equivalent hashes in different locations.

`nix --version` output

❯ nix --version
nix (Lix, like Nix) 2.92.0
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/mitchell/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/mitchell/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/mitchell/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/mitchell/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/store/q94fg6p8v05kgjmx1nvbpd91s8jzwb3i-gnome-settings-daemon-47.2/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/k2jqygqxply71hbg0cjybrq40k37klli-lix-2.92.0/share

Additional context

Thanks to Sönke Hahn and Alex David for helping figure out what was going on from the garnix side!

## Describe the bug While serving `jhk2qycb2gl5xjmp3fdp3fhdcni08k9q.narinfo`, the garnix cache issued a 302 Found/"Moved Temporarily" redirect from `https://cache.garnix.io/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q.narinfo` to `https://old-cache.garnix.io/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q.narinfo`, which subsequently 404'd. This triggered the check for changed URLs [here](https://git.lix.systems/lix-project/lix/src/commit/885dde9c3d65ffe90bc8411f143a06c4c62f46e1/lix/libstore/filetransfer.cc#L915), which was originally added by [this commit](https://git.lix.systems/lix-project/lix/commit/212a14bb1f2f8b844c314d75ac31f77785ba6b20). The intention of the original commit: "immutable url changing implies that the immutable url we got previously was wrong, which is probably a server bug" is correct for the hash in the _path_ changing, but I don't think it should consider the other URI components. It might not even need to consider anything but the final path segment containing the hash, but I'm not sure how easy parsing that out would be. ## Steps To Reproduce I originally encountered this issue resolving `/nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2`. Running `nix path-info --json /nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2 | jq` yields: ```json [ { "deriver": "/nix/store/7jbi4zg290h6cpyg7kkvrs6pj8507yg6-nautilus-47.2.drv", "narHash": "sha256-S0hAaBW4qJ5O0juxH2LcoI7VMfL2nQZlXYUJjoByQXM=", "narSize": 14458112, "path": "/nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2", "references": [ "/nix/store/0f3z5ixg7v9wxsf27p3pc7clsycvxyry-gdk-pixbuf-2.42.12", "/nix/store/209a73blm5p97vl9qx1xzh1qkl5g0gc1-gexiv2-0.14.3", "/nix/store/21r6fbzp31gafh0zbxv7nc4z0bhmjbfp-wayland-1.23.1", "/nix/store/25nkjd6y6rc7ap82frgqlga0sshvb3mx-librsvg-2.59.2", "/nix/store/35nvkdwk3qx63sif1c8fv9fdcr12cy8k-gtk4-4.16.12", "/nix/store/3q509a68aqbnh3dlv7qjkp6q7nfj7l2g-glib-networking-2.80.1", "/nix/store/4s50lxhz3gks66dx7m17laz58x9sy5c8-localsearch-3.8.2", "/nix/store/6kg26a99l5bq20zbk1c6a70brma5j84b-gsettings-desktop-schemas-47.1", "/nix/store/73gqa6192dkdalb6bcz54cqswcfqhvf3-gnome-desktop-44.1", "/nix/store/7idgdwzfkl83m0jifr5plahfg55jppm6-libjxl-0.11.1", "/nix/store/9b1y83zz0nk2w8c4x6ig0lnd4kfyckqy-pango-1.56.2", "/nix/store/9vrc87acgdzhpqkmfsihf0jir3006mzy-gstreamer-1.26.0", "/nix/store/am4ypiz0kjj7diynlk56p3gjxjrsyw89-shared-mime-info-2.4", "/nix/store/cyavy4kslnhb23p9mbbfmmfb7kgb0bws-webp-pixbuf-loader-0.2.6", "/nix/store/dfq2z3j0krkvjjza5hywvx1g7vv67vdm-gobject-introspection-1.82.0", "/nix/store/g3s0z9r7m1lsfxdk8bj88nw8k8q3dmmg-glibc-2.40-66", "/nix/store/g5h4kh5jy4648fwqyd43fmp2j90c0k08-libnotify-0.8.4", "/nix/store/grxanfsjw37kv7q7hnkwjf0pl1imh9h9-dconf-0.40.0-lib", "/nix/store/hcf1varsgkgc9r0l8x3yhjfgp5jlm62h-tinysparql-3.8.2", "/nix/store/j0sb2c91h2z7azfp1lp7jq5vh7bl6677-libportal-gtk4-0.9.1", "/nix/store/jfn5cq536hfa3dv9p3px6l89b79rwc19-graphene-1.10.8", "/nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2", "/nix/store/jsb94zxbl6n1ljxl3mwbb3lmzsbznnyj-gnome-user-share-47.2", "/nix/store/k1h6h1h02dkhcy0qfiz09njpf91xpxy8-gst-plugins-base-1.26.0", "/nix/store/lsya04bhx5bx7nlij201slxjw5kzzy8a-cairo-1.18.2", "/nix/store/mglvz0wh458afkvyk7xq1jlhcljjrynl-libcloudproviders-0.3.6", "/nix/store/qvya5walwxj7f37j2yclr0mf6z1c4lim-libX11-1.8.12", "/nix/store/v92zj1h355zqxfrph394hzmnpr5bd7qa-gnome-autoar-0.4.5", "/nix/store/vn0hbbmvc02gi5mczx2wpk5xc5ap00ma-harfbuzz-10.2.0", "/nix/store/ysm6ybv02ms2v6lzsx7fnqi2cy937l9x-glib-2.82.5", "/nix/store/z3w1grl7jzrnkzzhcww2gckh3mlsqwj6-libadwaita-1.6.4", "/nix/store/zlvdm56mss012a8vz8q9c9z2d0qcslrw-gobject-introspection-wrapped-1.82.0" ], "registrationTime": 1746735146, "signatures": [ "private-nix-cache-1:lj/ifjxio52ZS5qpJPckpn1arSelhYvwDk6nj2kg8UoV7GNyt8SrddNS/CUSKamUHYqbkH8aG6nCMU4WLsN0DQ==" ], "valid": true } ] ``` but I'm not sure how to cause the check to trigger again. I tried running `nix path-info --json /nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2 --store https://cache.garnix.io/ | jq`, but that only yielded ``` this path will be fetched (0.00 MiB download, 13.79 MiB unpacked): /nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2 [ { "path": "/nix/store/jhk2qycb2gl5xjmp3fdp3fhdcni08k9q-nautilus-47.2", "valid": false } ] ``` ## Expected behavior Ideally, the changed target check should be as specific as possible to allow cache providers to redirect to equivalent hashes in different locations. ## `nix --version` output ``` ❯ nix --version nix (Lix, like Nix) 2.92.0 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/mitchell/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/mitchell/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/mitchell/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/mitchell/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/store/q94fg6p8v05kgjmx1nvbpd91s8jzwb3i-gnome-settings-daemon-47.2/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/k2jqygqxply71hbg0cjybrq40k37klli-lix-2.92.0/share ``` ## Additional context Thanks to Sönke Hahn and Alex David for helping figure out what was going on from the garnix side!

magneticflux0 added the

bug

label

2025-05-08 22:53:19 +00:00

pennae commented

2025-05-09 12:59:49 +00:00

Owner

that's working exactly as intended. immutable urls are supposed to be immutable, and you are breaking that contract. moving a cache is a good reason to break the contract, but it's still no less broken.

perhaps more importantly though: immutable urls aren't intended for caches, but for flake inputs. cache contents are already immutably identified by their content hash (as you noted), but flake inputs referring to eg a tarball export of the main branch of some git repo are not (and the (completely broken) immutable url protocol extension allows us to get a stable url for such an unstable revision specification anyway)

that's working exactly as intended. immutable urls are supposed to be *immutable*, and you are breaking that contract. moving a cache is a good reason to break the contract, but it's still no less broken. perhaps more importantly though: immutable urls aren't intended for caches, but for flake inputs. cache contents are already immutably identified by their content hash (as you noted), but flake inputs referring to eg a tarball export of the `main` branch of some git repo are not (and the (completely broken) immutable url protocol extension allows us to get a stable url for such an unstable revision specification anyway)