lix-project/lix
22
79
Fork 54
Code Issues 516 Code Review (Gerrit) Projects 8 Releases Packages Wiki Activity

(in)compatibility of nix store subcommand with CppNix (new add vs. old add-file and add-path subcommands) #780

New issue
Open
opened 2025-03-29 22:10:44 +00:00 by wamserma · 6 comments
wamserma commented 2025-03-29 22:10:44 +00:00
Copy link

Describe the bug

With https://github.com/NixOS/nix/pull/9357 CppNix merged the add-file and add-path subcommands of nix store into a singe add subcommand. This has not been reflected in Lix (d169c092fc/lix/nix/add-to-store.cc as of opening this issue).

This breaks downstream tooling such as nix-playground, which uses the unified command: c1a5c79f4a/nix_playground/build.py (L44)

Steps To Reproduce

  1. Run touch delicious.icecream && nix store add delicious.icecream
  2. Get error: 'add' is not a recognised command

Expected behavior

  1. Run touch delicious.icecream && nix store add delicious.icecream
  2. Get /nix/store/kphh26dlb2w2mhydjqaiqn02zcxgz0j2-delicious.icecream (produced by CppNix 2.24.12 nix store add delicious.icecream)

nix --version output

nix (Lix, like Nix) 2.91.1

## Describe the bug With https://github.com/NixOS/nix/pull/9357 CppNix merged the `add-file` and `add-path` subcommands of `nix store` into a singe `add` subcommand. This has not been reflected in Lix (https://git.lix.systems/lix-project/lix/src/commit/d169c092fc28838a253be136d17fe7de1292c728/lix/nix/add-to-store.cc as of opening this issue). This breaks downstream tooling such as nix-playground, which uses the unified command: https://github.com/LaunchPlatform/nix-playground/blob/c1a5c79f4abbfdd57ee01cd498532519f18a3f55/nix_playground/build.py#L44 ## Steps To Reproduce 1. Run `touch delicious.icecream && nix store add delicious.icecream` 2. Get `error: 'add' is not a recognised command` ## Expected behavior 1. Run `touch delicious.icecream && nix store add delicious.icecream` 2. Get `/nix/store/kphh26dlb2w2mhydjqaiqn02zcxgz0j2-delicious.icecream` (produced by CppNix 2.24.12 `nix store add delicious.icecream`) ## `nix --version` output nix (Lix, like Nix) 2.91.1
jade commented 2025-03-30 00:30:45 +00:00
Owner
Copy link

I am not sure what the correct way to handle this is. It's possible to pick the change if it looks reasonable, but it sucks to be dragged into making changes just because CppNix did them, even if they're a good idea. Having this unstable CLI be the locus of user complaints whenever it differs from CppNix by either option of not changing or changing in ways that break scripts is untenable in the long term.

The ideal solution here is to stop making compatibility promises that prevent us having control over our destiny.

Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable.

The downstream software in question was never compatible with 2.18 since that file didn't exist at all.

I am not sure what the correct way to handle this is. It's possible to pick the change if it looks reasonable, but it sucks to be dragged into making changes just because CppNix did them, even if they're a good idea. Having this unstable CLI be the locus of user complaints whenever it differs from CppNix by either option of not changing *or* changing in ways that break scripts is untenable in the long term. The ideal solution here is to stop making compatibility promises that prevent us having control over our destiny. Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it *sounds* reasonable. The downstream software in question was never compatible with 2.18 since that file didn't exist at all.
pennae commented 2025-03-30 10:15:30 +00:00
Owner
Copy link

Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable.

please do not, because it will regress something. the new command unnecessarily exposes implementation details that'll prevent us making meaningful improvements to the store layer without breaking the commands again.

> Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable. please do not, because it will regress something. the new command unnecessarily exposes implementation details that'll prevent us making meaningful improvements to the store layer without breaking the commands *again*.
wamserma commented 2025-03-30 12:05:29 +00:00
Author
Copy link

The ideal solution here is to stop making compatibility promises that prevent us having control over our destiny.

This was merely an observation and by no means seen as a broken promise. Especially given that this stuff is still marked as experimental.

Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable.

TBH, I am still undecided how this command should be implemented. I can somewhat understand why add-path has been taken as default for add and add-files is now available as add --flat, but I do not like the ergonomics of having nix command subcommand --flag. The change itself is rather trivial (just adopting the RegisterCommand calls at the of the source file).

w.r.t. to regressions: the subcommand in question was not available for long, so probably has low adoption and CppNix at least added deprecation warnings in their patch instead of doing a hard switch.

> The ideal solution here is to stop making compatibility promises that prevent us having control over our destiny. This was merely an observation and by no means seen as a broken promise. Especially given that this stuff is still marked as experimental. > Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable. TBH, I am still undecided how this command should be implemented. I can somewhat understand why `add-path` has been taken as default for `add` and `add-files` is now available as `add --flat`, but I do not like the ergonomics of having `nix command subcommand --flag`. The change itself is rather trivial (just adopting the RegisterCommand calls at the of the source file). w.r.t. to regressions: the subcommand in question was not available for long, so probably has low adoption and CppNix at least added deprecation warnings in their patch instead of doing a hard switch.
wamserma commented 2025-03-30 12:06:03 +00:00
Author
Copy link

The ideal solution here is to stop making compatibility promises that prevent us having control over our destiny.

This was merely an observation and by no means seen as a broken promise. Especially given that this stuff is still marked as experimental.

Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable.

TBH, I am still undecided how this command should be implemented. I can somewhat understand why add-path has been taken as default for add and add-files is now available as add --flat, but I do not like the ergonomics of having nix command subcommand --flag. The change itself is rather trivial (just adopting the RegisterCommand calls at the of the source file).

w.r.t. to regressions: the subcommand in question was not available for long, so probably has low adoption and CppNix at least added deprecation warnings in their patch instead of doing a hard switch.

> The ideal solution here is to stop making compatibility promises that prevent us having control over our destiny. This was merely an observation and by no means seen as a broken promise. Especially given that this stuff is still marked as experimental. > Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable. TBH, I am still undecided how this command should be implemented. I can somewhat understand why `add-path` has been taken as default for `add` and `add-files` is now available as `add --flat`, but I do not like the ergonomics of having `nix command subcommand --flag`. The change itself is rather trivial (just adopting the RegisterCommand calls at the of the source file). w.r.t. to regressions: the subcommand in question was not available for long, so probably has low adoption and CppNix at least added deprecation warnings in their patch instead of doing a hard switch.
jade commented 2025-03-30 15:04:17 +00:00
Owner
Copy link

@pennae wrote in #780 (comment):

Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable.

please do not, because it will regress something. the new command unnecessarily exposes implementation details that'll prevent us making meaningful improvements to the store layer without breaking the commands again.

Oh? As far as I can tell the way it's been written today leaks approximately the same amount: the mode flag exists already and allows either recursive, flat or text modes. That being said, there being three modes for adding single files to the store is kind of pointless since every store path has a nar hash already and adding a single file probably could just make three hard links for compatibility with any of the non nar modes. Anyway I'm not sure of the specifics of the improvements you're planning.

@pennae wrote in https://git.lix.systems/lix-project/lix/issues/780#issuecomment-9989: > > Probably the fix to this one is picking the change, assuming it doesn't regress anything, since it sounds reasonable. > > please do not, because it will regress something. the new command unnecessarily exposes implementation details that'll prevent us making meaningful improvements to the store layer without breaking the commands _again_. Oh? As far as I can tell the way it's been written today leaks approximately the same amount: the mode flag exists already and allows either recursive, flat or text modes. That being said, there being three modes for adding single files to the store is kind of pointless since every store path has a nar hash already and adding a single file probably could just make three hard links for compatibility with any of the non nar modes. Anyway I'm not sure of the specifics of the improvements you're planning.
pennae commented 2025-03-30 17:00:41 +00:00
Owner
Copy link

exposing nar as the canonical format to create path hashes nails us down to not changing this unless we want to break the abi. creating compatibility hardlinks will not be possible for directories, creating symlinks will be wrong, and in any case we'll be unable to distinguish a compat-linked object from an object that was added with an old method and will have to deal with tremendously complicated GC (since every object may now exist multiple times, opening multiple race windows even once we've dealt with everything else)

we could of course sidestep this somewhat by sticking to the old recursive/flat wording without specifying behavior, but then we're still incompatible and have gained nothing except breaking things for no reason

exposing nar as the canonical format to create path hashes nails us down to not changing this unless we want to break the abi. creating compatibility hardlinks will not be possible for directories, creating symlinks will be wrong, and in any case we'll be unable to distinguish a compat-linked object from an object that was added with an old method and will have to deal with tremendously complicated GC (since every object may now exist multiple times, opening multiple race windows even once we've dealt with everything else) we could of course sidestep this somewhat by sticking to the old recursive/flat wording without specifying behavior, but then we're still incompatible and have gained nothing except breaking things for no reason
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ci-config
release-2.91
release-2.92
release-2.93
sb/raito/darwin-kqueue-disablement
sb/raito/hydra-merge
sb/pennae/ci-config
sb/raito/phantom-referrers-gc
repl-overlay-example
reuse-compliance
release-2.90
sb/pennae/io-rewrite
git-series/activity-cleanup
sb/tom-hubrecht/primops.cc-v2
sb/bacchanalia/cache-strlen%topic=substr-perf
sb/qyriad/beta-1
sb/qyriad/2.90-beta.0
sb/rbt/test-branch
sb/rbt/justfile
sb/rbt/repl-overlays
sb/qyriad/maint-meson
sb/qyriad/maint/meson
sb/puck/meson-c-api
sb/jade/burn-it-all
dependabot/github_actions/zeebe-io/backport-action-2.4.1
dependabot/github_actions/cachix/install-nix-action-25
dependabot/github_actions/actions/labeler-5
dependabot/github_actions/cachix/cachix-action-14
2.93.3
2.92.3
2.93.2
2.91.3
2.92.2
2.91.2
2.93.1
2.93.0
2.92.0
2.91.1
2.91.0
2.90.0
2.90.0-rc1
2.90-beta.1
2.90-beta.0
Labels
Clear labels   
Affects/CppNix

Probably affects CppNix. This is a best-effort tag for troubles in old code; it exists to inform the CppNix team about problems they might care about also.

  
Affects/Nightly

Found in Lix nightly.

  
Affects/Only nightly

Tag for regressions in Lix nightly. We intend to fix all of these prior to releases.

  
Affects/Stable

Affects the stable release of Lix. Usually such issues affect nightly too but there is no guarantee.

  
Area/build-packaging

regarding the building or packaging of Lix itself

  
Area/cli

The command line interface of Lix

  
Area/evaluator

libexpr and anything else related to the evaluation of Nix expressions

  
Area/fetching

Fetching sources or other materials from the World Wide Web in Lix

  
Area/flakes

   
Area/language

language design, features, extensions, etc

  
Area/lix ci

Lix's own CI and deployment pipeline

  
Area/nix-eval-jobs

nix-eval-jobs for Lix

  
Area/profiles

profiles: nix-env, nix profile, and installer flavors all alike

  
Area/protocol

The protocol, used for communicating between the client and the daemon

  
Area/releng

Release engineering

  
Area/remote-builds

remote builds

  
Area/repl

nix repl

  
Area/repl/debugger

The --debugger that brings up a Lix repl

  
Area/store

libstore and anything else relating to the Nix store

  
bug

it's a bug

  
Context
contributors
This issue might require some context that isn't written down. It might be possible to handle as a person who already worked on this area.

  
Context
drive-by
This issue contains all the necessary information to do a drive-by contribution: that is, you don't have to talk with anyone at all to successfully solve it!

  
Context
maintainers
This issue requires a lot of know-how. Attention from maintainers is needed to proceed.

  
Context
RFD
This issue needs discussion from multiple maintainers and interested parties: it's a Request For Discussion.

  
crash 💥

Lix crashes (possibly bad error handling)

  
Cross Compilation

   
devx

development experience issues

  
docs

requires or is about documentation changes

  
Downstream Dependents

regarding an external tool's usage with Lix (e.g. nix-eval-jobs)

  
E/easy

bug is relatively simple and could simply be fixed with some minor effort

  
E/hard

this issue should probably be solved with close involvement of an expert

  
E/help wanted

maintainers may not get to this soon, but we want it fixed

  
E/reproducible

This bug is reproducible and obvious.

  
E/requires rearchitecture

this bug cannot be fixed properly without fixing a known architectural flaw

  
imported

Issue imported from upstream nix (please do not remove this label, it will cause dupes if import is rerun)

  
Language/Bash

   
Language/C++

   
Language/NixLang

   
Language/Python

   
Language/Rust

   
Needs Langver

We can't get to this until we implement language versioning

  
OS/Linux

Affects specifically Lix on Linux

  
OS/macOS

Affects specifically Lix on macOS

  
performance

make lix go faster

  
regression

This is a regression since some previous version of Lix

  
release-blocker

Blocks public release

  
stability

Fixing this makes the software more stable

  
Status
blocked
Blocked on something

  
Status
invalid
not our bug, duplicate, similar

  
Status
postponed
closed but should maybe look at later (generally with S-wontfix)

  
Status
wontfix
We chose not to change this (see issue comments for more detail).

  
testing

Tests that should be added

  
testing/flakey

Tests that are unreliable/flakey (not to be confused with flakes that test)

  
Topic/Large Scale Installations

Large scale installations of Lix, e.g. NixOS Hydra, Lix CI, big corporate installations etc.

  
ux

makes the user experience better :)

No labels
Affects/CppNix
Affects/Nightly
Affects/Only nightly
Affects/Stable
Area/build-packaging
Area/cli
Area/evaluator
Area/fetching
Area/flakes
Area/language
Area/lix ci
Area/nix-eval-jobs
Area/profiles
Area/protocol
Area/releng
Area/remote-builds
Area/repl
Area/repl/debugger
Area/store
bug
Context
contributors
Context
drive-by
Context
maintainers
Context
RFD
crash 💥
Cross Compilation
devx
docs
Downstream Dependents
E/easy
E/hard
E/help wanted
E/reproducible
E/requires rearchitecture
imported
Language/Bash
Language/C++
Language/NixLang
Language/Python
Language/Rust
Needs Langver
OS/Linux
OS/macOS
performance
regression
release-blocker
stability
Status
blocked
Status
invalid
Status
postponed
Status
wontfix
testing
testing/flakey
Topic/Large Scale Installations
ux
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#780
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?