Lix broken during nixos-rebuild #883

Closed
opened 2025-06-27 02:45:31 +00:00 by randomnetcat · 11 comments

Describe the bug

I have two machines (one x64 and one aarch64) that appear to have borked their Lix installations during nixos-rebuild. The journals of their most recent attempts to run nixos-upgrade are attached. In both cases, Lix appears to have entered a corrupt state during the upgrade, as at the end of each upgrade, Lix complains that it is unable to find libeditline. There appear to have been no errors prior to the upgrade attempt.

After the upgrade attempt, Lix on both hosts fails with the error nix: error while loading shared libraries: libeditline.so.1: cannot open shared object file: No such file or directory.

Taking one of the hosts as an example, we find that the nix binary has in its rpath the lib folder of the same derivation and that liblixcmd.so has a libeditline output in its rpath. However, the libeditline output no longer exists:

[root@shaw:~]# patchelf --print-rpath /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/bin/lix
/nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/lib:/nix/store/iwml852wwzlq34a6wg5vq0n2ylg7iqx3-capnproto-1.0.2/lib:/nix/store/i5zd3ha7wjnxnshsqisqarm8gxrfgc70-boehm-gc-8.2.8/lib:/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66/lib:/nix/store/l7d6vwajpfvgsd3j4cr25imd1mzb7d1d-gcc-14.3.0-lib/lib

[root@shaw:~]# patchelf --print-rpath /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/lib/liblixcmd.so 
/nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/lib:/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1/lib:/nix/store/gfw7f9lv4lmxnx3h5988hcyvl7wi5zav-lowdown-1.3.2-lib/lib:/nix/store/iwml852wwzlq34a6wg5vq0n2ylg7iqx3-capnproto-1.0.2/lib:/nix/store/i5zd3ha7wjnxnshsqisqarm8gxrfgc70-boehm-gc-8.2.8/lib:/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66/lib:/nix/store/l7d6vwajpfvgsd3j4cr25imd1mzb7d1d-gcc-14.3.0-lib/lib

[root@shaw:~]# ls /nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1/lib
ls: cannot access '/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1/lib': No such file or directory

The relevant output does not appear to have been invalidly garbage collected, leading me to believe the error is, in fact, occurring during nixos-rebuild:

[root@shaw:~]# journalctl -u nix-gc --grep dpih77kxyw9kvyyncdj4abfx59padqik --since 2025-06-01
-- Boot ea1d1978f2b549e3833e88ed2fc764c9 --
-- Boot fb86cbf8f9fa4c8da149effb453a2abd --
Jun 20 04:51:07 shaw nix-gc-start[1965]: deleting '/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1'
-- Boot fb5b0a0e50a0469cb48d376123ae7ff4 --
-- Boot f1d83291e9884ff4989991aa48a12857 --
Jun 23 03:15:33 shaw nix-gc-start[9777]: deleting '/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1'

(This editline output first became referenced by the Lix derivation after this garbage-collection run, later on June 23.)

The result is that the store is corrupt, with an extant output referencing an output that has been somehow deleted.

My nix configurations are available at https://github.com/randomnetcat/nix-configs if this helps with reproduction.

I apologize that I don't have anything more specific to add, but I do not have time to attempt to reproduce this issue in, e.g., a VM. However, I am happy to answer any questions.

Steps To Reproduce

I... have no idea.

Expected behavior

Lix does not break during nixos-rebuild.

nix --version output

[root@shaw:~]# realpath "$(which nix)"
/nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/bin/nix

[root@shaw:~]# nix --version
nix: error while loading shared libraries: libeditline.so.1: cannot open shared object file: No such file or directory

Output from a different (working) machine with the same version of Lix:

[randomcat@carter:~]$ /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/bin/nix --version
nix (Lix, like Nix) 2.94.0-dev-pre20250624-42e2bd0
System type: x86_64-linux
Additional system types: aarch64-linux, i686-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/randomcat/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/randomcat/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/randomcat/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/randomcat/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/store/r5ypl97bb730gqjq7mrcifrm92lzyi2k-gnome-settings-daemon-48.1/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/share
## Describe the bug I have two machines (one x64 and one aarch64) that appear to have borked their Lix installations during `nixos-rebuild`. The journals of their most recent attempts to run `nixos-upgrade` are attached. In both cases, Lix appears to have entered a corrupt state during the upgrade, as at the end of each upgrade, Lix complains that it is unable to find `libeditline`. There appear to have been no errors prior to the upgrade attempt. After the upgrade attempt, Lix on both hosts fails with the error `nix: error while loading shared libraries: libeditline.so.1: cannot open shared object file: No such file or directory`. Taking one of the hosts as an example, we find that the `nix` binary has in its rpath the `lib` folder of the same derivation and that `liblixcmd.so` has a `libeditline` output in its rpath. However, the `libeditline` output no longer exists: ``` [root@shaw:~]# patchelf --print-rpath /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/bin/lix /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/lib:/nix/store/iwml852wwzlq34a6wg5vq0n2ylg7iqx3-capnproto-1.0.2/lib:/nix/store/i5zd3ha7wjnxnshsqisqarm8gxrfgc70-boehm-gc-8.2.8/lib:/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66/lib:/nix/store/l7d6vwajpfvgsd3j4cr25imd1mzb7d1d-gcc-14.3.0-lib/lib [root@shaw:~]# patchelf --print-rpath /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/lib/liblixcmd.so /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/lib:/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1/lib:/nix/store/gfw7f9lv4lmxnx3h5988hcyvl7wi5zav-lowdown-1.3.2-lib/lib:/nix/store/iwml852wwzlq34a6wg5vq0n2ylg7iqx3-capnproto-1.0.2/lib:/nix/store/i5zd3ha7wjnxnshsqisqarm8gxrfgc70-boehm-gc-8.2.8/lib:/nix/store/q4wq65gl3r8fy746v9bbwgx4gzn0r2kl-glibc-2.40-66/lib:/nix/store/l7d6vwajpfvgsd3j4cr25imd1mzb7d1d-gcc-14.3.0-lib/lib [root@shaw:~]# ls /nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1/lib ls: cannot access '/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1/lib': No such file or directory ``` The relevant output does not appear to have been invalidly garbage collected, leading me to believe the error is, in fact, occurring during `nixos-rebuild`: ``` [root@shaw:~]# journalctl -u nix-gc --grep dpih77kxyw9kvyyncdj4abfx59padqik --since 2025-06-01 -- Boot ea1d1978f2b549e3833e88ed2fc764c9 -- -- Boot fb86cbf8f9fa4c8da149effb453a2abd -- Jun 20 04:51:07 shaw nix-gc-start[1965]: deleting '/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1' -- Boot fb5b0a0e50a0469cb48d376123ae7ff4 -- -- Boot f1d83291e9884ff4989991aa48a12857 -- Jun 23 03:15:33 shaw nix-gc-start[9777]: deleting '/nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1' ``` (This `editline` output first became referenced by the Lix derivation after this garbage-collection run, later on June 23.) The result is that the store is corrupt, with an extant output referencing an output that has been somehow deleted. My nix configurations are available at https://github.com/randomnetcat/nix-configs if this helps with reproduction. I apologize that I don't have anything more specific to add, but I do not have time to attempt to reproduce this issue in, e.g., a VM. However, I am happy to answer any questions. ## Steps To Reproduce I... have no idea. ## Expected behavior Lix does not break during `nixos-rebuild`. ## `nix --version` output ``` [root@shaw:~]# realpath "$(which nix)" /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/bin/nix [root@shaw:~]# nix --version nix: error while loading shared libraries: libeditline.so.1: cannot open shared object file: No such file or directory ``` Output from a **different** (working) machine with the same version of Lix: ``` [randomcat@carter:~]$ /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/bin/nix --version nix (Lix, like Nix) 2.94.0-dev-pre20250624-42e2bd0 System type: x86_64-linux Additional system types: aarch64-linux, i686-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/randomcat/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/randomcat/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/randomcat/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/randomcat/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/store/r5ypl97bb730gqjq7mrcifrm92lzyi2k-gnome-settings-daemon-48.1/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/34f39kbxg07nhfmp52nrqc518ydfhqkm-lix-2.94.0-dev-pre20250624-42e2bd0/share ```
Owner

Can you share more information about what filesystem for the store do you have on the machine that exhibit the bug?

Can you share more information about what filesystem for the store do you have on the machine that exhibit the bug?
Owner

yep, this is definitely a bug in lix. it happens when a derivation that already has some valid output on the system is rebuilt for its other outputs, in this case the other outputs are added to the store and the pre-existing outputs are deleted due to a logic error. quite bad.

yep, this is definitely a bug in lix. it happens when a derivation that already has some valid output on the system is rebuilt for its other outputs, in this case the other outputs are added to the store and the pre-existing outputs are deleted due to a logic error. quite bad.
Owner

since the problematic code was backported to all stable releases as part of the CVE fix chain it's all but guaranteed that stable is affected as well

since the problematic code was backported to all stable releases as part of the CVE fix chain it's all but guaranteed that stable is affected as well
Owner

We will perform an emergency release (probably with other papercut fixes) to fix that.
Additionally, we will offer guidance and tools to repair systems that might have been broken by this.

We will perform an emergency release (probably with other papercut fixes) to fix that. Additionally, we will offer guidance and tools to repair systems that might have been broken by this.
Owner

I just ran into this. I was able to fix it by setting LD_PRELOAD to some existing libeditline.so.1 in my Nix store, and running Lix as root to bypass the daemon. All in all:

$ sudo env LD_PRELOAD=/nix/store/pr2xhavyz59p98mpjndyn4acqbliw1fa-editline-1.17.1/lib/libeditline.so.1 nix-store --repair-path /nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1 --store local

(The --store local was unnecessary of course, but I'm putting it here for history since #18)

I just ran into this. I was able to fix it by setting `LD_PRELOAD` to some existing `libeditline.so.1` in my Nix store, and running Lix as root to bypass the daemon. All in all: ```bash $ sudo env LD_PRELOAD=/nix/store/pr2xhavyz59p98mpjndyn4acqbliw1fa-editline-1.17.1/lib/libeditline.so.1 nix-store --repair-path /nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1 --store local ``` (The `--store local` was unnecessary of course, but I'm putting it here for history since #18)
Member

This issue was mentioned on Gerrit on the following CLs:

  • commit message in cl/3491 ("libstore: don't delete already valid outputs after build")
  • commit message in cl/3494 ("libstore: don't delete already valid outputs after build")
  • commit message in cl/3496 ("libstore: don't delete already valid outputs after build")
  • commit message in cl/3498 ("libstore: don't delete already valid outputs after build")
  • commit message in cl/3513 ("version: 2.92.2 -> 2.92.3")
  • commit message in cl/3515 ("version: 2.91.2 -> 2.91.3")
<!-- GERRIT_LINKBOT: {"cls": [{"backlink": "https://gerrit.lix.systems/c/lix/+/3491", "number": 3491, "kind": "commit message"}, {"backlink": "https://gerrit.lix.systems/c/lix/+/3494", "number": 3494, "kind": "commit message"}, {"backlink": "https://gerrit.lix.systems/c/lix/+/3496", "number": 3496, "kind": "commit message"}, {"backlink": "https://gerrit.lix.systems/c/lix/+/3498", "number": 3498, "kind": "commit message"}, {"backlink": "https://gerrit.lix.systems/c/lix/+/3513", "number": 3513, "kind": "commit message"}, {"backlink": "https://gerrit.lix.systems/c/lix/+/3515", "number": 3515, "kind": "commit message"}], "cl_meta": {"3491": {"change_title": "libstore: don't delete already valid outputs after build"}, "3494": {"change_title": "libstore: don't delete already valid outputs after build"}, "3496": {"change_title": "libstore: don't delete already valid outputs after build"}, "3498": {"change_title": "libstore: don't delete already valid outputs after build"}, "3513": {"change_title": "version: 2.92.2 -> 2.92.3"}, "3515": {"change_title": "version: 2.91.2 -> 2.91.3"}}} --> This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/3491](https://gerrit.lix.systems/c/lix/+/3491) ("libstore: don't delete already valid outputs after build") * commit message in [cl/3494](https://gerrit.lix.systems/c/lix/+/3494) ("libstore: don't delete already valid outputs after build") * commit message in [cl/3496](https://gerrit.lix.systems/c/lix/+/3496) ("libstore: don't delete already valid outputs after build") * commit message in [cl/3498](https://gerrit.lix.systems/c/lix/+/3498) ("libstore: don't delete already valid outputs after build") * commit message in [cl/3513](https://gerrit.lix.systems/c/lix/+/3513) ("version: 2.92.2 -> 2.92.3") * commit message in [cl/3515](https://gerrit.lix.systems/c/lix/+/3515) ("version: 2.91.2 -> 2.91.3")
Owner

@qyriad wrote in #883 (comment):

I just ran into this. I was able to fix it by setting LD_PRELOAD to some existing libeditline.so.1 in my Nix store, and running Lix as root to bypass the daemon. All in all:

$ sudo env LD_PRELOAD=/nix/store/pr2xhavyz59p98mpjndyn4acqbliw1fa-editline-1.17.1/lib/libeditline.so.1 nix-store --repair-path /nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1 --store local

(The --store local was unnecessary of course, but I'm putting it here for history since #18)

A previous Lix would probably work as well if you still have one in the past. If you can create a new section in the blog post for the CVE to inform about this known issue and provide these recommendations in the blog post, that'd be awesome!

@qyriad wrote in https://git.lix.systems/lix-project/lix/issues/883#issuecomment-12780: > I just ran into this. I was able to fix it by setting `LD_PRELOAD` to some existing `libeditline.so.1` in my Nix store, and running Lix as root to bypass the daemon. All in all: > > ```bash > $ sudo env LD_PRELOAD=/nix/store/pr2xhavyz59p98mpjndyn4acqbliw1fa-editline-1.17.1/lib/libeditline.so.1 nix-store --repair-path /nix/store/dpih77kxyw9kvyyncdj4abfx59padqik-editline-1.17.1 --store local > ``` > > (The `--store local` was unnecessary of course, but I'm putting it here for history since #18) A previous Lix would probably work as well if you still have one in the past. If you can create a new section in the blog post for the CVE to inform about this known issue and provide these recommendations in the blog post, that'd be awesome!
Owner

We will be releasing the fixes ASAP for this as part of:

  • Lix 2.91.3
  • Lix 2.92.3
  • Lix 2.93.2

Thank you for bearing with us and sorry for the inconveniences.

We will be releasing the fixes ASAP for this as part of: - Lix 2.91.3 - Lix 2.92.3 - Lix 2.93.2 Thank you for bearing with us and sorry for the inconveniences.
Author

Thank you all for the very quick fix!

Thank you all for the very quick fix!
Member

Hey, I've ran into the same issue. I have one machine that does all the builds, and other machines just pull their built configs, apparently only the build host had problems.
This is the commit that caused the breakage (all configs and pipeline logs should be publicly visible, let me know if not): e621f23afd

Not sure if it's just a case of "Lix version with the bug built version without the bug, so the build was borked, and there's nothing here to see", or something worth looking into.
The LD_PRELOAD hack with --repair-path that @qyriad shared worked to recover, so now my system is back and running.

Please let me know if there's any more information that you need. Thanks!

Hey, I've ran into the same issue. I have one machine that does all the builds, and other machines just pull their built configs, apparently only the build host had problems. This is the commit that caused the breakage (all configs and pipeline logs should be publicly visible, let me know if not): https://github.com/ramonacat/monorepo/commit/e621f23afdb5368b9dc089312edd4a38ce6644d8 Not sure if it's just a case of "Lix version with the bug built version without the bug, so the build was borked, and there's nothing here to see", or something worth looking into. The `LD_PRELOAD` hack with `--repair-path` that @qyriad shared worked to recover, so now my system is back and running. Please let me know if there's any more information that you need. Thanks!
Owner

@raito wrote in #883 (comment):

We will be releasing the fixes ASAP for this as part of:

  • Lix 2.91.3
  • Lix 2.92.3
  • Lix 2.93.2

Thank you for bearing with us and sorry for the inconveniences.

The fixes are now released! Thank you for your patience.

@raito wrote in https://git.lix.systems/lix-project/lix/issues/883#issuecomment-12788: > We will be releasing the fixes ASAP for this as part of: > > * Lix 2.91.3 > * Lix 2.92.3 > * Lix 2.93.2 > > Thank you for bearing with us and sorry for the inconveniences. The fixes are now released! Thank you for your patience.
Sign in to join this conversation.
No milestone
No project
No assignees
6 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#883
No description provided.