Store path is present in `ValidPaths` in `db.sqlite` but is not present in the nix store

r-vdp commented

2026-02-04 13:49:05 +00:00

Describe the bug

For the second time now, I have a build failure like this:

error:
       … while setting up the build environment

       error: getting attributes of path "/nix/store/yfpwx89vv1m72cr4ym9bs32znd4xyvzc-Cargo.lock": No such file or directory

However, the path in question should be in my nix store according to the nix DB:

➜ sudo sqlite3 /nix/var/nix/db/db.sqlite -header "select * from ValidPaths where path = '/nix/store/yfpwx89vv1m72cr4ym9bs32znd4xyvzc-Cargo.lock' limit 3"
id|path|hash|registrationTime|deriver|narSize|ultimate|sigs|ca
5000576|/nix/store/yfpwx89vv1m72cr4ym9bs32znd4xyvzc-Cargo.lock|sha256:eff0b05f7cb436cfbc9953da3c6602a4876c687d35385bf462e53c60aa2bc1ff|1757449495||16480|||fixed:r:sha256:1zy15fm60g75cbs5nf1mgml6r1x409k3rnjkk6ycydmlgigv1w7g

Manually fetching the path from another machine and copying it into the nix store, fixes the issue.

The path in question is coming from here: github.com/NixOS/nixpkgs@f613dc8375/pkgs/by-name/sw/switch-to-configuration-ng/package.nix (L16)

Is lix somehow getting confused and deleting this path because it thinks that it's a lock file for the derivation?

Steps To Reproduce

I didn't yet figure out how to reproduce it, but it has happened twice so far, both times when doing large builds with a couple of thousand derivations.

`nix --version` output

nix (Lix, like Nix) 2.95.0-pre20260203-dev_aa89604
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/ramses/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/ramses/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/ramses/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/ramses/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/adx0vdqg8nl1ng9sw69df01rr9fczzrl-lix-2.95.0-pre20260203-dev_aa89604/share

## Describe the bug For the second time now, I have a build failure like this: ``` error: … while setting up the build environment error: getting attributes of path "/nix/store/yfpwx89vv1m72cr4ym9bs32znd4xyvzc-Cargo.lock": No such file or directory ``` However, the path in question should be in my nix store according to the nix DB: ``` ➜ sudo sqlite3 /nix/var/nix/db/db.sqlite -header "select * from ValidPaths where path = '/nix/store/yfpwx89vv1m72cr4ym9bs32znd4xyvzc-Cargo.lock' limit 3" id|path|hash|registrationTime|deriver|narSize|ultimate|sigs|ca 5000576|/nix/store/yfpwx89vv1m72cr4ym9bs32znd4xyvzc-Cargo.lock|sha256:eff0b05f7cb436cfbc9953da3c6602a4876c687d35385bf462e53c60aa2bc1ff|1757449495||16480|||fixed:r:sha256:1zy15fm60g75cbs5nf1mgml6r1x409k3rnjkk6ycydmlgigv1w7g ``` Manually fetching the path from another machine and copying it into the nix store, fixes the issue. The path in question is coming from here: https://github.com/NixOS/nixpkgs/blob/f613dc8375de96702c5809acb2235eb46bcc4201/pkgs/by-name/sw/switch-to-configuration-ng/package.nix#L16 Is lix somehow getting confused and deleting this path because it thinks that it's a lock file for the derivation? ## Steps To Reproduce I didn't yet figure out how to reproduce it, but it has happened twice so far, both times when doing large builds with a couple of thousand derivations. ## `nix --version` output ``` nix (Lix, like Nix) 2.95.0-pre20260203-dev_aa89604 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/ramses/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/ramses/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/ramses/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/ramses/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/adx0vdqg8nl1ng9sw69df01rr9fczzrl-lix-2.95.0-pre20260203-dev_aa89604/share ```

r-vdp added the

bug

label

2026-02-04 13:49:05 +00:00

raito added the

Area/store

Affects/Only nightly

labels

2026-02-04 14:12:20 +00:00

raito commented

2026-02-04 14:16:19 +00:00

Owner

@tom-hubrecht ran into this as well, but without a reproducer, we will be in trouble to figure out the root cause. Any help on this is appreciated.

r-vdp commented

2026-02-04 15:45:16 +00:00

Author

Yeah, I tried for a bit but I couldn't figure out yet how to reproduce it exactly.
In the case of @tom-hubrecht, was it also a derivation with a name ending in .lock ?

Yeah, I tried for a bit but I couldn't figure out yet how to reproduce it exactly. In the case of @tom-hubrecht, was it also a derivation with a name ending in `.lock` ?

raito commented

2026-02-04 15:45:48 +00:00

Owner

@r-vdp exactly the same file I think

raito commented

2026-02-04 15:46:23 +00:00

Owner

No I lied:

error: opening file '/nix/store/2img2fdpkr60nyc5ki385hdbyck4y9fc-python3.13-aiohttp-3.13.2.drv.chroot/etc/passwd': No such file or directory

on Tom's side.

No I lied: > error: opening file '/nix/store/2img2fdpkr60nyc5ki385hdbyck4y9fc-python3.13-aiohttp-3.13.2.drv.chroot/etc/passwd': No such file or directory on Tom's side.

raito commented

2026-02-04 15:47:06 +00:00

Owner

@r-vdp Have you verified your store entirely btw?

r-vdp commented

2026-02-04 15:48:10 +00:00

Author

Those chroot directories are created during a build but are not registered as store paths though, right? Because in that case it's not the same issue.

👍 1

pennae commented

2026-02-04 15:56:14 +00:00

Owner

it may still be the same issue because all paths that are spuriously disappearing are in the store and subject to GC deletion

pennae commented

2026-02-04 16:07:37 +00:00

Owner

@r-vdp do you have auto-gc enabled? @raito does tom?

tom-hubrecht commented

2026-02-04 16:22:06 +00:00

Member

@pennae wrote in #1119 (comment):

@r-vdp do you have auto-gc enabled? @raito does tom?

It's a bit of a weird setup on my part, it was in our CI that uses daemonless stores through containers, with 4 of them trying to build that derivation that was failing, so when a build failed if it deleted the lock before the chroot there might have been a race

@pennae wrote in https://git.lix.systems/lix-project/lix/issues/1119#issuecomment-17430: > @r-vdp do you have auto-gc enabled? @raito does tom? It's a bit of a weird setup on my part, it was in our CI that uses daemonless stores through containers, with 4 of them trying to build that derivation that was failing, so when a build failed if it deleted the lock before the chroot there might have been a race

pennae added the

release-blocker

label

2026-02-04 16:26:41 +00:00

r-vdp commented

2026-02-04 19:09:03 +00:00

Author

➜ sudo systemd-run --pty --wait --collect -p Delegate=yes -p DelegateSubgroup=supervisor -p Environment="NIX_REMOTE=local" nix-store --verify
Running as unit: run-p475949-i4668435.service; invocation ID: 2e8802a5a1f248f19e863e8a23718654
Press ^] three times within 1s to disconnect TTY.
reading the Nix store...
checking path existence...
path '/nix/store/3hqq60czfhx1fdkwjdgmlaxq002mb7s8-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/68iwcilv272mjm0w04ygwlsgw3883qh9-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/73961ikx5fz1mid548pbgygxbni079hg-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/7dd49ax4i9dxgm3djccs9lh4fy1fvf56-Gemfile.lock' disappeared, but it still has valid referrers!
path '/nix/store/bf99l8ilr0y3p87mibsbswrish8y53h8-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/ci3mkcf32cpnfh57bm689x3n8mhfy6dj-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/dj92890xijb4n0yyzawgdsx3p574f0y1-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/fiywbd64ac0a19z8a2082hkjmlf8lrb3-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/gfnv9n4h835fggiv4frwm1jvcpqn4nkr-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/gfxypyjzhi44gaf5n1wfnnfy76gni95z-Gemfile.lock' disappeared, but it still has valid referrers!
path '/nix/store/pa6sl46npsr1m2br1fjmllbpw59ziz6h-flake.lock' disappeared, but it still has valid referrers!
path '/nix/store/swi2diza6kch6m2gpf50c9h79zg9b6q9-Gemfile.lock' disappeared, but it still has valid referrers!
path '/nix/store/vl65m3gp12qkbh2qffzm12f7k77pqdvr-Cargo.lock' disappeared, but it still has valid referrers!
path '/nix/store/yigw6zvx0n0cf6d7w2l3mnfcn88kcr87-Cargo.lock' disappeared, but it still has valid referrers!
warning: not all store errors were fixed

``` ➜ sudo systemd-run --pty --wait --collect -p Delegate=yes -p DelegateSubgroup=supervisor -p Environment="NIX_REMOTE=local" nix-store --verify Running as unit: run-p475949-i4668435.service; invocation ID: 2e8802a5a1f248f19e863e8a23718654 Press ^] three times within 1s to disconnect TTY. reading the Nix store... checking path existence... path '/nix/store/3hqq60czfhx1fdkwjdgmlaxq002mb7s8-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/68iwcilv272mjm0w04ygwlsgw3883qh9-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/73961ikx5fz1mid548pbgygxbni079hg-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/7dd49ax4i9dxgm3djccs9lh4fy1fvf56-Gemfile.lock' disappeared, but it still has valid referrers! path '/nix/store/bf99l8ilr0y3p87mibsbswrish8y53h8-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/ci3mkcf32cpnfh57bm689x3n8mhfy6dj-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/dj92890xijb4n0yyzawgdsx3p574f0y1-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/fiywbd64ac0a19z8a2082hkjmlf8lrb3-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/gfnv9n4h835fggiv4frwm1jvcpqn4nkr-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/gfxypyjzhi44gaf5n1wfnnfy76gni95z-Gemfile.lock' disappeared, but it still has valid referrers! path '/nix/store/pa6sl46npsr1m2br1fjmllbpw59ziz6h-flake.lock' disappeared, but it still has valid referrers! path '/nix/store/swi2diza6kch6m2gpf50c9h79zg9b6q9-Gemfile.lock' disappeared, but it still has valid referrers! path '/nix/store/vl65m3gp12qkbh2qffzm12f7k77pqdvr-Cargo.lock' disappeared, but it still has valid referrers! path '/nix/store/yigw6zvx0n0cf6d7w2l3mnfcn88kcr87-Cargo.lock' disappeared, but it still has valid referrers! warning: not all store errors were fixed ```

r-vdp commented

2026-02-04 19:09:17 +00:00

Author

I think that there is a clear pattern there...

r-vdp commented

2026-02-04 19:11:49 +00:00

Author

@pennae wrote in #1119 (comment):

@r-vdp do you have auto-gc enabled? @raito does tom?

What do you mean by auto-gc exactly? I have a timer that runs nix-collect-garbage --delete-older-than 14d once a week, and I have this in my nix.conf:

max-free = 1000000000
min-free = 128000000

@pennae wrote in https://git.lix.systems/lix-project/lix/issues/1119#issuecomment-17430: > @r-vdp do you have auto-gc enabled? @raito does tom? What do you mean by auto-gc exactly? I have a timer that runs `nix-collect-garbage --delete-older-than 14d` once a week, and I have this in my nix.conf: ``` max-free = 1000000000 min-free = 128000000 ```

raito commented

2026-02-04 19:17:50 +00:00

Owner

AFAIK, these .lock things are local path files which appears in inputSrcs of drvs, right? Or are those FODs or something else?

AFAIK, these .lock things are local path files which appears in `inputSrcs` of drvs, right? Or are those FODs or something else?

r-vdp commented

2026-02-04 19:33:59 +00:00

Author

For the Cargo.lock files, they are local paths that are in the inputSrcs of the fetchCargoVendor FOD which itself is in the inputDrvs of the actual rust application derivation, which is a runtime dep of my nixos closure.

For the Cargo.lock files, they are local paths that are in the `inputSrcs` of the `fetchCargoVendor` FOD which itself is in the `inputDrvs` of the actual rust application derivation, which is a runtime dep of my nixos closure.

👍 1

Rows
Columns

Store path is present in ValidPaths in db.sqlite but is not present in the nix store #1119

Describe the bug

Steps To Reproduce

nix --version output

Store path is present in `ValidPaths` in `db.sqlite` but is not present in the nix store #1119

`nix --version` output