lix-project/lix
22
82
Fork 55
Code Issues 529 Code Review (Gerrit) Projects 8 Releases Packages Wiki Activity

Profile conflict error in nix upgrade-nix when upgrading from 2.20 #411

New issue
Closed
opened 2024-06-23 18:27:41 +00:00 by jade · 14 comments
jade commented 2024-06-23 18:27:41 +00:00
Owner
Copy link

still not sure what exact circumstances lead to this and which upgrade-nix but it seems to happen to our upgrade-nix so :(

building '/nix/store/9gd7xvjhdfsxjp5w9rkv04zh384y4hpy-user-environment.drv'...
error: Unable to build profile. There is a conflict for the following files:

         /nix/store/4xwvps7bnachw9686ac22jhczn04i44b-nix-2.20.5/bin/nix-prefetch-url
         /nix/store/z1bdccwsk34iv491aygh0mm1lgpf7yy1-lix-2.90.0-rc1/bin/nix-prefetch-url
error: builder for '/nix/store/9gd7xvjhdfsxjp5w9rkv04zh384y4hpy-user-environment.drv' failed with exit code 1
error: program '/nix/store/z1bdccwsk34iv491aygh0mm1lgpf7yy1-lix-2.90.0-rc1/bin/nix-env' failed with exit code 100
still not sure what exact circumstances lead to this and which upgrade-nix but it seems to happen to our upgrade-nix so :( ``` building '/nix/store/9gd7xvjhdfsxjp5w9rkv04zh384y4hpy-user-environment.drv'... error: Unable to build profile. There is a conflict for the following files: /nix/store/4xwvps7bnachw9686ac22jhczn04i44b-nix-2.20.5/bin/nix-prefetch-url /nix/store/z1bdccwsk34iv491aygh0mm1lgpf7yy1-lix-2.90.0-rc1/bin/nix-prefetch-url error: builder for '/nix/store/9gd7xvjhdfsxjp5w9rkv04zh384y4hpy-user-environment.drv' failed with exit code 1 error: program '/nix/store/z1bdccwsk34iv491aygh0mm1lgpf7yy1-lix-2.90.0-rc1/bin/nix-env' failed with exit code 100 ```
jade commented 2024-07-05 13:36:16 +00:00
Author
Owner
Copy link

cc @qyriad

This happens also with nix-2.11 -> lix-2.90.0-rc1 on macOS.

cc @qyriad This happens also with nix-2.11 -> lix-2.90.0-rc1 on macOS.
lunaphied commented 2024-07-07 20:23:57 +00:00
Owner
Copy link

This occurs because when using a nix-env style profile, nix upgrade-nix simply runs nix-env --install with the newly provided derivation into the provided profile.

This works fine if both derivations have the same name (such as a normal upgrade from nix-<old.version> to nix-<new.version>), as with the flags provided, the install will simply replace the old version with the new (like an upgrade however missing the version resolution logic, a curiously underdocumented behavior). But in this case they don't, as one of them is named 'lix'. They both have the same default priority in the profile and provide the same file names, so the symlink step of building the environment fails.

With a nix profile style environment, Lix's nix upgrade-nix logic was improved to support this type of upgrade and searches for a package that provides bin/nix-env as a heuristic to identify the Nix implementation to replace, it then itself handles creating a new profile generation with the change and switching to it. Lix retains the older "just use nix-env directly" logic for older profiles, so Lix would have this error upgrading to a CppNix as well with those profiles.

Unfortunately we can't actually fix this case, as the underlying behavior is in CppNix , we'd need to add the same heuristic method to the nix-env style profile handling to fix Lix, and then get those changes merged into CppNix, and considering that we don't plan on having upgrade-nix be the proper way to install Lix, this seems unrealistic.

This occurs because when using a `nix-env` style profile, `nix upgrade-nix` simply runs `nix-env --install` with the newly provided derivation into the provided profile. This works fine if both derivations have the same name (such as a normal upgrade from `nix-<old.version>` to `nix-<new.version>`), as with the flags provided, the install will simply replace the old version with the new (like an upgrade however missing the version resolution logic, a curiously underdocumented behavior). But in this case they don't, as one of them is named 'lix'. They both have the same default priority in the profile and provide the same file names, so the symlink step of building the environment fails. With a `nix profile` style environment, Lix's `nix upgrade-nix` logic was improved to support this type of upgrade and searches for a package that provides `bin/nix-env` as a heuristic to identify the Nix implementation to replace, it then itself handles creating a new profile generation with the change and switching to it. Lix retains the older "just use `nix-env` directly" logic for older profiles, so Lix would have this error upgrading to a CppNix as well with those profiles. Unfortunately we can't actually fix this case, as the underlying behavior is in *CppNix* , we'd need to add the same heuristic method to the `nix-env` style profile handling to fix Lix, and then get those changes merged into CppNix, and considering that we don't plan on having `upgrade-nix` be the proper way to install Lix, this seems unrealistic.
jade commented 2024-07-10 18:19:07 +00:00
Author
Owner
Copy link

But this is unusable, since AFAICT every installer uses the nix-env style. So we do have to actually care about this bug, unfortunately.

However, this might be best achieved by just nuking upgrade-nix and making it tell you to use the installer, assuming it can repair existing installations in-place.

But this is *unusable*, since AFAICT every installer uses the `nix-env` style. So we do have to actually care about this bug, unfortunately. However, this might be best achieved by just nuking upgrade-nix and making it tell you to use the installer, assuming it can repair existing installations in-place.
lunaphied commented 2024-07-10 22:34:24 +00:00
Owner
Copy link

However, this might be best achieved by just nuking upgrade-nix and making it tell you to use the installer, assuming it can repair existing installations in-place.

Yes this is what I meant about us (Lix) not considering nix upgrade-nix to be a reasonable way of upgrading to Lix. From what I can tell there is literally no way for us to fix this without patching Nix itself, which would be even more confusing as some copies of Nix will work and others won't.

We can ofc patch Lix to handle this case but it won't matter for Lix->Lix updates anyway, since it will replace the older Lix with the newer Lix as both have the package name lix, the issue is only caused because CppNix's package name is nix and so when you install Lix you get lix as the package name and both are installed rather than simply replacing the old nix package with a new nix package. If you already have a lix installed, the new lix package will safely replace it, even with nix-env style profiles, so we don't have to worry about that.

Afaik we consider the Lix installer to be the way to do this, it will need to be modified to remove any existing installs of Nix first if it's currently trying to do upgrades through nix upgrade-nix or similar. This will literally only affect Nix->Lix upgrades, Lix->Lix upgrades will work fine with nix-env style profiles as is.

I consider the correct way to upgrade a system to Lix to be either using NixOS and our various recommended approaches there, or installing fresh using the Lix installer. I haven't touched the installer code so I don't know how happy it will handle an existing store, if it doesn't handle that well we should update the Lix installer to better handle that case.

> However, this might be best achieved by just nuking upgrade-nix and making it tell you to use the installer, assuming it can repair existing installations in-place. Yes this is what I meant about us (Lix) not considering `nix upgrade-nix` to be a reasonable way of upgrading to Lix. From what I can tell there is literally no way for us to fix this without patching Nix itself, which would be even more confusing as *some* copies of Nix will work and others won't. We can ofc patch Lix to handle this case but it won't matter for Lix->Lix updates anyway, since it will replace the older Lix with the newer Lix as both have the package name `lix`, the issue is only caused because CppNix's package name is `nix` and so when you install Lix you get `lix` as the package name and both are installed rather than simply replacing the old `nix` package with a new `nix` package. If you already have a `lix` installed, the new `lix` package will safely replace it, even with `nix-env` style profiles, so we don't have to worry about that. Afaik we consider the Lix installer to be the way to do this, it will need to be modified to remove any existing installs of Nix first if it's currently trying to do upgrades through `nix upgrade-nix` or similar. This will literally only affect Nix->Lix upgrades, Lix->Lix upgrades will work fine with `nix-env` style profiles as is. I consider the correct way to upgrade a system to Lix to be either using NixOS and our various recommended approaches there, or installing fresh using the Lix installer. I haven't touched the installer code so I don't know how happy it will handle an existing store, if it doesn't handle that well we should update the Lix installer to better handle that case.
jade commented 2024-07-11 11:01:56 +00:00
Author
Owner
Copy link

If we think this is indeed the case, we should fix the website documentation to not suggest nix upgrade-nix anymore. But upgrade-nix would be sound if we just used the nix profile hack also for nix-env.

If we think this is indeed the case, we should fix the website documentation to not suggest `nix upgrade-nix` anymore. But `upgrade-nix` *would* be sound if we just used the `nix profile` hack also for `nix-env`.
ethnt commented 2024-07-11 13:54:22 +00:00
Copy link

@lunaphied I also ran into this — am I correct in assuming that the workaround for this is to uninstall Nix and then reinstall using the Lix install script?

@lunaphied I also ran into this — am I correct in assuming that the workaround for this is to uninstall Nix and then reinstall using the Lix install script?
lunaphied commented 2024-07-11 14:31:57 +00:00
Owner
Copy link

@lunaphied I also ran into this — am I correct in assuming that the workaround for this is to uninstall Nix and then reinstall using the Lix install script?

@ethnt Correct

If we think this is indeed the case, we should fix the website documentation to not suggest nix upgrade-nix anymore. But upgrade-nix would be sound if we just used the nix profile hack also for nix-env.

That would only fix Lix->Nix, Nix->Lix can never work unless we get that code merged to CppNix itself, and then there'll be versions of CppNix that have the code and work, and versions that don't.

> @lunaphied I also ran into this — am I correct in assuming that the workaround for this is to uninstall Nix and then reinstall using the Lix install script? @ethnt Correct > If we think this is indeed the case, we should fix the website documentation to not suggest `nix upgrade-nix` anymore. But `upgrade-nix` *would* be sound if we just used the `nix profile` hack also for `nix-env`. That would only fix Lix->Nix, Nix->Lix can never work unless we get that code merged to CppNix itself, and then there'll be versions of CppNix that have the code and work, and versions that don't.
ian-h-chamberlain commented 2024-07-11 17:31:17 +00:00
Copy link

Hi, just found this issue and wanted to check if this is also expected for Lix -> Lix upgrades (it sounds unexpected, from reading the previous discussion). I tried to upgrade from 2.90.0-beta.1 and saw a similar message:

$ nix upgrade-nix
installing 'lix-2.90.0'
warning: Ignoring the client-specified setting 'sandbox', because it is a restricted setting and you are not a trusted user
building '/nix/store/j2b3lapkbqr99544b7q3508r7y1j6nqq-user-environment.drv'...
error: Unable to build profile. There is a conflict for the following files:

         /nix/store/dajhmsik4z5s2nn9gbiv8bxv1j641bl4-nix-2.90.0-beta.1/bin/nix
         /nix/store/rp7y16q2py2n9y19jvxkjr83lp77bh7y-lix-2.90.0/bin/nix
error: builder for '/nix/store/j2b3lapkbqr99544b7q3508r7y1j6nqq-user-environment.drv' failed with exit code 1;
       last 4 log lines:
       > error: Unable to build profile. There is a conflict for the following files:
       >
       >          /nix/store/dajhmsik4z5s2nn9gbiv8bxv1j641bl4-nix-2.90.0-beta.1/bin/nix
       >          /nix/store/rp7y16q2py2n9y19jvxkjr83lp77bh7y-lix-2.90.0/bin/nix
       For full logs, run 'nix log /nix/store/j2b3lapkbqr99544b7q3508r7y1j6nqq-user-environment.drv'.
error: program '/nix/store/dajhmsik4z5s2nn9gbiv8bxv1j641bl4-nix-2.90.0-beta.1/bin/nix-env' failed with exit code 1

Maybe this is just an artifact of how beta.1 was installed (lix-installer 0.17.1), and I'm assuming from reading e.g. #267 that this isn't really a planned workflow long-term anyway? I mainly just wanted to confirm this is part of the same problem (and same workaround) and not something that deserves a separate issue.

Hi, just found this issue and wanted to check if this is also expected for Lix -> Lix upgrades (it sounds unexpected, from reading the previous discussion). I tried to upgrade from `2.90.0-beta.1` and saw a similar message: ``` $ nix upgrade-nix installing 'lix-2.90.0' warning: Ignoring the client-specified setting 'sandbox', because it is a restricted setting and you are not a trusted user building '/nix/store/j2b3lapkbqr99544b7q3508r7y1j6nqq-user-environment.drv'... error: Unable to build profile. There is a conflict for the following files: /nix/store/dajhmsik4z5s2nn9gbiv8bxv1j641bl4-nix-2.90.0-beta.1/bin/nix /nix/store/rp7y16q2py2n9y19jvxkjr83lp77bh7y-lix-2.90.0/bin/nix error: builder for '/nix/store/j2b3lapkbqr99544b7q3508r7y1j6nqq-user-environment.drv' failed with exit code 1; last 4 log lines: > error: Unable to build profile. There is a conflict for the following files: > > /nix/store/dajhmsik4z5s2nn9gbiv8bxv1j641bl4-nix-2.90.0-beta.1/bin/nix > /nix/store/rp7y16q2py2n9y19jvxkjr83lp77bh7y-lix-2.90.0/bin/nix For full logs, run 'nix log /nix/store/j2b3lapkbqr99544b7q3508r7y1j6nqq-user-environment.drv'. error: program '/nix/store/dajhmsik4z5s2nn9gbiv8bxv1j641bl4-nix-2.90.0-beta.1/bin/nix-env' failed with exit code 1 ``` Maybe this is just an artifact of how `beta.1` was installed (`lix-installer 0.17.1`), and I'm assuming from reading e.g. #267 that this isn't really a planned workflow long-term anyway? I mainly just wanted to confirm this is part of the same problem (and same workaround) and not something that deserves a separate issue.
jade commented 2024-07-12 14:51:18 +00:00
Author
Owner
Copy link

Yep, same problem, beta 1 had a derivation name of "nix" so you see the same symptom.

Yep, same problem, beta 1 had a derivation name of "nix" so you see the same symptom.
👍 1
ikselven commented 2024-07-12 15:28:44 +00:00
Copy link

I've found a workaround for the Unable to build profile error, though I'm not sure, if it broke something along the way:

oldnixbin="$(readlink -f "$(which nix)")"

sudo nix profile remove nix
sudo "${oldnixbin}" profile install "git+https://git.lix.systems/lix-project/lix?ref=refs/tags/2.90.0#nix"
I've found a workaround for the `Unable to build profile` error, though I'm not sure, if it broke something along the way: ``` oldnixbin="$(readlink -f "$(which nix)")" sudo nix profile remove nix sudo "${oldnixbin}" profile install "git+https://git.lix.systems/lix-project/lix?ref=refs/tags/2.90.0#nix" ```
lunaphied commented 2024-07-12 15:38:04 +00:00
Owner
Copy link

Yes that is absolutely the correct workaround

Yes that is absolutely the correct workaround
qyriad commented 2024-07-12 18:55:56 +00:00
Owner
Copy link

If you're getting a profile conflict with Lix's upgrade-nix, that means that it's using the wrong nix-env (either from CppNix or from a version of Lix that didn't have this fix). Lix's upgrade-nix is supposed to use the nix-env from the Lix that's currently being invoked, so the nix run $lix -- upgrade-nix pattern works. If it's not, that means our "find nix-env" logic wasn't implemented correctly.

If you're getting a profile conflict with Lix's `upgrade-nix`, that means that it's using the *wrong* `nix-env` (either from CppNix or from a version of Lix that didn't have this fix). Lix's `upgrade-nix` is *supposed* to use the `nix-env` from the Lix that's currently being invoked, so the `nix run $lix -- upgrade-nix` pattern works. If it's not, that means our "find nix-env" logic wasn't implemented correctly.
qyriad reopened this issue 2024-07-12 18:55:58 +00:00
qyriad commented 2024-07-12 18:56:22 +00:00
Owner
Copy link

This being said, we agree that upgrade-nix really should just be killed, and we should just use the installer for upgrades.

This being said, we agree that `upgrade-nix` really should just be killed, and we should just use the installer for upgrades.
lix-bot commented 2024-07-12 22:28:38 +00:00
Member
Copy link

This issue was mentioned on Gerrit on the following CLs:

  • commit message in cl/1604 ("nix3-upgrade-nix: always use the /new/ nix-env to perform the installation")
<!-- GERRIT_LINKBOT: {"cls": [{"backlink": "https://gerrit.lix.systems/c/lix/+/1604", "number": 1604, "kind": "commit message"}], "cl_meta": {"1604": {"change_title": "nix3-upgrade-nix: always use the /new/ nix-env to perform the installation"}}} --> This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/1604](https://gerrit.lix.systems/c/lix/+/1604) ("nix3-upgrade-nix: always use the /new/ nix-env to perform the installation")
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ci-config
sb/pennae/ci-config
release-2.93
release-2.91
release-2.92
sb/raito/darwin-kqueue-disablement
sb/raito/hydra-merge
sb/raito/phantom-referrers-gc
repl-overlay-example
reuse-compliance
release-2.90
sb/pennae/io-rewrite
git-series/activity-cleanup
sb/tom-hubrecht/primops.cc-v2
sb/bacchanalia/cache-strlen%topic=substr-perf
sb/qyriad/beta-1
sb/qyriad/2.90-beta.0
sb/rbt/test-branch
sb/rbt/justfile
sb/rbt/repl-overlays
sb/qyriad/maint-meson
sb/qyriad/maint/meson
sb/puck/meson-c-api
sb/jade/burn-it-all
dependabot/github_actions/zeebe-io/backport-action-2.4.1
dependabot/github_actions/cachix/install-nix-action-25
dependabot/github_actions/actions/labeler-5
dependabot/github_actions/cachix/cachix-action-14
2.93.3
2.92.3
2.93.2
2.91.3
2.92.2
2.91.2
2.93.1
2.93.0
2.92.0
2.91.1
2.91.0
2.90.0
2.90.0-rc1
2.90-beta.1
2.90-beta.0
Labels
Clear labels   
Affects/CppNix

Probably affects CppNix. This is a best-effort tag for troubles in old code; it exists to inform the CppNix team about problems they might care about also.

  
Affects/Nightly

Found in Lix nightly.

  
Affects/Only nightly

Tag for regressions in Lix nightly. We intend to fix all of these prior to releases.

  
Affects/Stable

Affects the stable release of Lix. Usually such issues affect nightly too but there is no guarantee.

  
Area/build-packaging

regarding the building or packaging of Lix itself

  
Area/cli

The command line interface of Lix

  
Area/evaluator

libexpr and anything else related to the evaluation of Nix expressions

  
Area/fetching

Fetching sources or other materials from the World Wide Web in Lix

  
Area/flakes

   
Area/language

language design, features, extensions, etc

  
Area/lix ci

Lix's own CI and deployment pipeline

  
Area/nix-eval-jobs

nix-eval-jobs for Lix

  
Area/profiles

profiles: nix-env, nix profile, and installer flavors all alike

  
Area/protocol

The protocol, used for communicating between the client and the daemon

  
Area/releng

Release engineering

  
Area/remote-builds

remote builds

  
Area/repl

nix repl

  
Area/repl/debugger

The --debugger that brings up a Lix repl

  
Area/store

libstore and anything else relating to the Nix store

  
bug

it's a bug

  
Context
contributors
This issue might require some context that isn't written down. It might be possible to handle as a person who already worked on this area.

  
Context
drive-by
This issue contains all the necessary information to do a drive-by contribution: that is, you don't have to talk with anyone at all to successfully solve it!

  
Context
maintainers
This issue requires a lot of know-how. Attention from maintainers is needed to proceed.

  
Context
RFD
This issue needs discussion from multiple maintainers and interested parties: it's a Request For Discussion.

  
crash 💥

Lix crashes (possibly bad error handling)

  
Cross Compilation

   
devx

development experience issues

  
docs

requires or is about documentation changes

  
Downstream Dependents

regarding an external tool's usage with Lix (e.g. nix-eval-jobs)

  
E/easy

bug is relatively simple and could simply be fixed with some minor effort

  
E/hard

this issue should probably be solved with close involvement of an expert

  
E/help wanted

maintainers may not get to this soon, but we want it fixed

  
E/reproducible

This bug is reproducible and obvious.

  
E/requires rearchitecture

this bug cannot be fixed properly without fixing a known architectural flaw

  
Feature/S3

S3 support in Lix

  
imported

Issue imported from upstream nix (please do not remove this label, it will cause dupes if import is rerun)

  
Language/Bash

   
Language/C++

   
Language/NixLang

   
Language/Python

   
Language/Rust

   
Needs Langver

We can't get to this until we implement language versioning

  
OS/Linux

Affects specifically Lix on Linux

  
OS/macOS

Affects specifically Lix on macOS

  
performance

make lix go faster

  
regression

This is a regression since some previous version of Lix

  
release-blocker

Blocks public release

  
stability

Fixing this makes the software more stable

  
Status
blocked
Blocked on something

  
Status
invalid
not our bug, duplicate, similar

  
Status
postponed
closed but should maybe look at later (generally with S-wontfix)

  
Status
wontfix
We chose not to change this (see issue comments for more detail).

  
testing

Tests that should be added

  
testing/flakey

Tests that are unreliable/flakey (not to be confused with flakes that test)

  
Topic/Large Scale Installations

Large scale installations of Lix, e.g. NixOS Hydra, Lix CI, big corporate installations etc.

  
ux

makes the user experience better :)

No labels
Affects/CppNix
Affects/Nightly
Affects/Only nightly
Affects/Stable
Area/build-packaging
Area/cli
Area/evaluator
Area/fetching
Area/flakes
Area/language
Area/lix ci
Area/nix-eval-jobs
Area/profiles
Area/protocol
Area/releng
Area/remote-builds
Area/repl
Area/repl/debugger
Area/store
bug
Context
contributors
Context
drive-by
Context
maintainers
Context
RFD
crash 💥
Cross Compilation
devx
docs
Downstream Dependents
E/easy
E/hard
E/help wanted
E/reproducible
E/requires rearchitecture
Feature/S3
imported
Language/Bash
Language/C++
Language/NixLang
Language/Python
Language/Rust
Needs Langver
OS/Linux
OS/macOS
performance
regression
release-blocker
stability
Status
blocked
Status
invalid
Status
postponed
Status
wontfix
testing
testing/flakey
Topic/Large Scale Installations
ux
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
7 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lix-project/lix#411
No description provided.