lix-project/lix

Fork 65

Upgrading an existing install deleted `nix-daemon.service` and broke everything #1189

New issue

Open

opened 2026-04-22 10:15:29 +00:00 by blokyk · 5 comments

blokyk commented

2026-04-22 10:15:29 +00:00

Describe the bug

TL;DR: I tried to upgrade a 2.95.1 Lix install to 2.95.1 (yes, i'm stupid), and it failed in the middle of nix-enving the new install, after having uninstalled the old nix and nix-daemon.service, completely breaking my Lix install. I had to completely uninstall nix using /nix/nix-installer uninstaller and then reinstall everything.

Steps To Reproduce

maybe I'm stupid and did something wrong, but running

$ curl -sSf -L "https://git.lix.systems/lix-project/lix/raw/tag/2.95.1/misc/upgrade-lix.sh" | sudo --preserve-env=PATH bash -s -- 2.95.1

on an existing 2.95.1 (i forgot i'd already upgraded) config uninstalled (among other things) nix-daemon.service and then was unable to start it again, completely breaking everything. this also means i don't have nix in my path anymore, so i can't take the "upgrade" path anymore, and instead have to reinstall everything. (i tried to just rm /nix/receipt.json && /nix/nix-installer install, but for some reason it fails, complaining about either error: you don't have sufficient rights to use this command or error: could not connect to any lix socket (tried /nix/var/nix/daemon-socket/socket) when trying to --load-db.)

Here is the original installer error, after which nix was unusable:

$ curl -sSf -L "https://git.lix.systems/lix-project/lix/raw/tag/2.95.1/misc/upgrade-lix.sh" | sudo --preserve-env=PATH bash -s -- 2.95.1

+ exec /nix/var/nix/profiles/default/bin/nix run --extra-experimental-features 'nix-command flakes' --extra-substituters https://cache.lix.systems --extra-trusted-public-keys cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o= 'git+https://git.lix.systems/lix-project/lix?ref=refs/tags/2.95.1' --extra-experimental-features 'nix-command flakes' upgrade-nix --extra-substituters https://cache.lix.systems --extra-trusted-public-keys cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o=
uninstalling 'lix-2.95.1'
building '/nix/store/a09wbm6bn585g6z7rd2pjw389yar4pz8-user-environment.drv'...
warning: Ignoring setting 'bash-prompt-prefix' because experimental feature 'nix-command' is not enabled
installing 'lix-2.95.1'
error: cannot open connection to remote store 'daemon': error: read failed: Connection reset by peer
error: program '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env' failed with exit code 1

For completeness, I've also attached the full terminal session as nix-install.log (yes, there's a lot of repetition in there, sorry ;-;)

Expected behavior

Either the upgrade is idempotent or it just refuses to upgrade because it detects that it is already the same version. In any case, it doesn't... explode.

`nix --version` output

N/A, but i'm on an x86_64 Ubuntu 24.04.4

Additional context

Add any other context about the problem here.

## Describe the bug TL;DR: I tried to upgrade a 2.95.1 Lix install to 2.95.1 (yes, i'm stupid), and it failed in the middle of `nix-env`ing the new install, after having uninstalled the old `nix` and `nix-daemon.service`, completely breaking my Lix install. I had to completely uninstall nix using `/nix/nix-installer uninstaller` and then reinstall everything. ## Steps To Reproduce maybe I'm stupid and did something wrong, but running ```sh $ curl -sSf -L "https://git.lix.systems/lix-project/lix/raw/tag/2.95.1/misc/upgrade-lix.sh" | sudo --preserve-env=PATH bash -s -- 2.95.1 ``` on an existing 2.95.1 (i forgot i'd already upgraded) config uninstalled (among other things) `nix-daemon.service` and then was unable to start it again, completely breaking everything. this also means i don't have nix in my path anymore, so i can't take the "upgrade" path anymore, and instead have to reinstall everything. (i tried to just `rm /nix/receipt.json && /nix/nix-installer install`, but for some reason it fails, complaining about either `error: you don't have sufficient rights to use this command` or `error: could not connect to any lix socket (tried /nix/var/nix/daemon-socket/socket)` when trying to `--load-db`.) Here is the original installer error, after which nix was unusable: ```sh $ curl -sSf -L "https://git.lix.systems/lix-project/lix/raw/tag/2.95.1/misc/upgrade-lix.sh" | sudo --preserve-env=PATH bash -s -- 2.95.1 + exec /nix/var/nix/profiles/default/bin/nix run --extra-experimental-features 'nix-command flakes' --extra-substituters https://cache.lix.systems --extra-trusted-public-keys cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o= 'git+https://git.lix.systems/lix-project/lix?ref=refs/tags/2.95.1' --extra-experimental-features 'nix-command flakes' upgrade-nix --extra-substituters https://cache.lix.systems --extra-trusted-public-keys cache.lix.systems:aBnZUw8zA7H35Cz2RyKFVs3H4PlGTLawyY5KRbvJR8o= uninstalling 'lix-2.95.1' building '/nix/store/a09wbm6bn585g6z7rd2pjw389yar4pz8-user-environment.drv'... warning: Ignoring setting 'bash-prompt-prefix' because experimental feature 'nix-command' is not enabled installing 'lix-2.95.1' error: cannot open connection to remote store 'daemon': error: read failed: Connection reset by peer error: program '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env' failed with exit code 1 ``` For completeness, I've also attached the full terminal session as `nix-install.log` (yes, there's a lot of repetition in there, sorry ;-;) ## Expected behavior Either the upgrade is idempotent or it just refuses to upgrade because it detects that it is already the same version. In any case, it doesn't... explode. ## `nix --version` output N/A, but i'm on an x86_64 Ubuntu 24.04.4 ## Additional context Add any other context about the problem here.

blokyk added the

bug

label

2026-04-22 10:15:29 +00:00

qyriad commented

2026-04-22 10:28:01 +00:00

Owner

(Opened from lix-project/lix-installer#79)

qyriad self-assigned this

2026-04-22 10:28:06 +00:00

qyriad added the

labels

2026-04-22 10:32:11 +00:00

qyriad commented

2026-04-22 11:06:20 +00:00

Owner

In my test container, I was able to recover with:

$ sudo /nix/var/nix/profiles/per-user/root/profile-1-link/bin/nix-env --profile /nix/var/nix/profiles/per-user/root/profile --rollback

Where profile-1-link was the previous generation:

$ ls -lah /nix/var/nix/profiles/per-user/root/
Permissions Size User Group Date Modified Name
lrwxrwxrwx     - root root  22 Apr 13:00  profile -> profile-1-link/
lrwxrwxrwx     - root root  22 Apr 12:49  profile-1-link -> /nix/store/ym9zadvh3d2fdd8hx8y5sylgs1hcsnvv-user-environment/
lrwxrwxrwx     - root root  22 Apr 12:51  profile-2-link -> /nix/store/lsp1lmy8rr3yhgrl0vdkwpnqa1i9b15f-user-environment/

Not exactly an obvious recovery path, but at least the previous profile is intact.

In my test container, I was able to recover with: ```bash $ sudo /nix/var/nix/profiles/per-user/root/profile-1-link/bin/nix-env --profile /nix/var/nix/profiles/per-user/root/profile --rollback ``` Where `profile-1-link` was the previous generation: ```bash $ ls -lah /nix/var/nix/profiles/per-user/root/ Permissions Size User Group Date Modified Name lrwxrwxrwx - root root 22 Apr 13:00 profile -> profile-1-link/ lrwxrwxrwx - root root 22 Apr 12:49 profile-1-link -> /nix/store/ym9zadvh3d2fdd8hx8y5sylgs1hcsnvv-user-environment/ lrwxrwxrwx - root root 22 Apr 12:51 profile-2-link -> /nix/store/lsp1lmy8rr3yhgrl0vdkwpnqa1i9b15f-user-environment/ ``` Not exactly an obvious recovery path, but at least the previous profile is intact.

blokyk commented

2026-04-22 11:23:38 +00:00

Author

iirc, /nix/var/nix/profiles was basically empty for me except for a manifest.nix somewhere (probably in default/, though i could be misremembering). i don't remember if per-user/root/profile existed or if i checked it, but if you're seeing it in your tests i probably just missed it. honestly i was a little panicked given the installer's failures and the fact the daemon service was gone, so i just thought the easiest path would be uninstalling (and i didn't even know about /nix/nix-installer repair ;-;)

iirc, `/nix/var/nix/profiles` was basically empty for me except for a `manifest.nix` somewhere (probably in `default/`, though i could be misremembering). i don't remember if `per-user/root/profile` existed or if i checked it, but if you're seeing it in your tests i probably just missed it. honestly i was a little panicked given the installer's failures and the fact the daemon service was gone, so i just thought the easiest path would be uninstalling (and i didn't even know about `/nix/nix-installer repair` ;-;)

qyriad commented

2026-04-22 11:41:02 +00:00

Owner

Ah, I see what's happening.

$ sudo --preserve-env=PATH /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix upgrade-nix -vv
found Nix in '/nix/var/nix/profiles/per-user/root/profile/bin'
found profile '/nix/var/nix/profiles/per-user/root/profile'
upgrading Nix in profile '/nix/var/nix/profiles/per-user/root/profile'
querying latest Nix version...
downloading 'https://releases.lix.systems/manifest.nix'...
downloading '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1'......
verifying that '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1' works......
running /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env --uninstall /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env --profile /nix/var/nix/profiles/per-user/root/profile
uninstalling 'lix-2.95.1'
running /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env --profile /nix/var/nix/profiles/per-user/root/profile --install /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1 --no-sandbox
installing 'lix-2.95.1'
error: cannot open connection to remote store 'daemon': error: read failed: Connection reset by peer
error: program '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env' failed with exit code 1

nix upgrade-nix calls nix-env to remove the old profile, which removes /nix/var/nix/profiles/default/lib/systemd/system/nix-daemon{@.service,.socket}. Then it calls nix-env again to install the new profile, which tries to connect to the running nix-daemon.socket and create a new instance of nix-daemon@.service, except now /etc/systemd/nix-daemon{@.service,.socket} are symlinks with invalid targets, so the daemon connection fails.

In other words, now that we are fully socket activated, nix upgrade-nix removes the ability to use the daemon partway through.

cc @pennae

I know that we want to remove upgrade-nix anyway, because, well, stuff like this. Should we make it execute nix-env with --store local or something, though, to prevent this failure mode?

Ah, I see what's happening. ```bash $ sudo --preserve-env=PATH /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix upgrade-nix -vv found Nix in '/nix/var/nix/profiles/per-user/root/profile/bin' found profile '/nix/var/nix/profiles/per-user/root/profile' upgrading Nix in profile '/nix/var/nix/profiles/per-user/root/profile' querying latest Nix version... downloading 'https://releases.lix.systems/manifest.nix'... downloading '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1'...... verifying that '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1' works...... running /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env --uninstall /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env --profile /nix/var/nix/profiles/per-user/root/profile uninstalling 'lix-2.95.1' running /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env --profile /nix/var/nix/profiles/per-user/root/profile --install /nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1 --no-sandbox installing 'lix-2.95.1' error: cannot open connection to remote store 'daemon': error: read failed: Connection reset by peer error: program '/nix/store/slkbw0cg08jhfsax6kc0bf6rvw7rkl26-lix-2.95.1/bin/nix-env' failed with exit code 1 ``` `nix upgrade-nix` calls `nix-env` to remove the old profile, which removes `/nix/var/nix/profiles/default/lib/systemd/system/nix-daemon{@.service,.socket}`. Then it calls `nix-env` again to install the new profile, which tries to connect to the running `nix-daemon.socket` and create a new instance of `nix-daemon@.service`, except now `/etc/systemd/nix-daemon{@.service,.socket}` are symlinks with invalid targets, so the daemon connection fails. In other words, now that we are fully socket activated, `nix upgrade-nix` removes the ability to use the daemon partway through. cc @pennae I know that we want to remove `upgrade-nix` anyway, because, well, stuff like this. Should we make it execute `nix-env` with `--store local` or something, though, to prevent this failure mode?

pennae commented

2026-04-22 12:02:08 +00:00

Owner

I know that we want to remove upgrade-nix anyway, because, well, stuff like this. Should we make it execute nix-env with --store local or something, though, to prevent this failure mode?

absolutely! upgrading while the daemon is still running and might be in use by something else is a recipe for disaster, especially if that upgrade changes the sqlite db schema (which luckily hasn't happened in a long time now). upgrades should take down the daemon until they're done just to be safe

> I know that we want to remove `upgrade-nix` anyway, because, well, stuff like this. Should we make it execute `nix-env` with `--store local` or something, though, to prevent this failure mode? absolutely! upgrading while the daemon is still running and might be in use by something else is a recipe for disaster, especially if that upgrade changes the sqlite db schema (which luckily hasn't happened in a long time now). upgrades should take down the daemon until they're done just to be safe