lix-project/lix
22
79
Fork 54
Code Issues 518 Code Review (Gerrit) Projects 8 Releases Packages Wiki Activity

RFD: Remove recursive-nix #767

New issue
Closed
opened 2025-03-25 23:54:08 +00:00 by jade · 14 comments
jade commented 2025-03-25 23:54:08 +00:00
Owner
Copy link

We want to remove the recursive-nix experimental feature and all associated code. Recursive Nix is a feature where the derivation builder creates a Nix daemon socket that through a complex series of operations which is running against the host Nix store. The purpose of this is that you can do nix evaluations and nix builds inside of a derivation and have the results appear in the outer nix store.

It's used for dynamic derivations, where a derivation produces a build plan which references other store paths, which currently have to use recursive-nix to create said store paths. This is not the final design and CppNix is driving the dev process of dyndrvs, not us; it would be replaced with varlink or something else before stabilization.

Here's why:

  • It puts the legacy nix daemon protocol as part of stable derivation ABI. This is unacceptable and means it will never be stabilized.
  • There is broad agreement including with the CppNix team that dynamic-derivations will NOT ship without a replacement to recursive-nix, so there is no compat hazard to removing it except for people who are trying to do dyndrvs on lix (haven't heard of anyone).
  • It's not "can you run nix inside a derivation builder". You can do that still. You can even do that with nested user namespaces and a fake nix store if you really need to.
  • It's a structural problem when we want to refactor the sandbox setup code to not know what a nix is, which we want to do to be able to simplify it and maybe even RIIR.

Is anyone using recursive-nix on Lix? What are you using it for? Is there some way that we can accommodate your use case?

CL: https://gerrit.lix.systems/c/lix/+/2872

We want to remove the `recursive-nix` experimental feature and all associated code. Recursive Nix is a feature where the derivation builder creates a Nix daemon socket that through a complex series of operations which is running against the *host* Nix store. The purpose of this is that you can do nix evaluations and nix builds inside of a derivation and have the results appear in the outer nix store. It's used for dynamic derivations, where a derivation produces a build plan which references other store paths, which currently have to use recursive-nix to create said store paths. This is not the final design and CppNix is driving the dev process of dyndrvs, not us; it would be replaced with varlink or something else before stabilization. Here's why: - It puts the legacy nix daemon protocol as part of stable derivation ABI. This is unacceptable and means it will *never* be stabilized. - There is broad agreement including with the CppNix team that dynamic-derivations will NOT ship without a replacement to `recursive-nix`, so there is no compat hazard to removing it except for people who are trying to do dyndrvs on lix (haven't heard of anyone). - It's *not* "can you run nix inside a derivation builder". You can do that still. You can even do that with nested user namespaces and a fake nix store if you really need to. - It's a structural problem when we want to refactor the sandbox setup code to not know what a nix is, which we want to do to be able to simplify it and maybe even RIIR. Is anyone using recursive-nix on Lix? What are you using it for? Is there some way that we can accommodate your use case? CL: https://gerrit.lix.systems/c/lix/+/2872
👍 4
lix-bot commented 2025-03-25 23:54:26 +00:00
Member
Copy link

This issue was mentioned on Gerrit on the following CLs:

  • comment in cl/2872 ("chore: drop experimental feature recursive-nix")
  • commit message in cl/2872 ("chore: drop experimental feature recursive-nix")
<!-- GERRIT_LINKBOT: {"cls": [{"backlink": "https://gerrit.lix.systems/c/lix/+/2872", "number": 2872, "kind": "comment"}, {"backlink": "https://gerrit.lix.systems/c/lix/+/2872", "number": 2872, "kind": "commit message"}], "cl_meta": {"2872": {"change_title": "chore: drop experimental feature `recursive-nix`"}}} --> This issue was mentioned on Gerrit on the following CLs: * comment in [cl/2872](https://gerrit.lix.systems/c/lix/+/2872) ("chore: drop experimental feature `recursive-nix`") * commit message in [cl/2872](https://gerrit.lix.systems/c/lix/+/2872) ("chore: drop experimental feature `recursive-nix`")
lunaphied commented 2025-03-26 02:56:41 +00:00
Owner
Copy link

except for people who are trying to do dyndrvs on lix (haven't heard of anyone).

considering that it's still disabled as it is on CppNix, and even if it wasn't it's still probably bugged, honestly we could remove both at this point. Bit of a hot take though, but I suspect we'll want to do something different for dyndrvs anyway.

> except for people who are trying to do dyndrvs on lix (haven't heard of anyone). considering that it's still disabled as it is on CppNix, and even if it wasn't it's still probably bugged, honestly we could remove both at this point. Bit of a hot take though, but I suspect we'll want to do something different for dyndrvs anyway.
👍 1
pennae commented 2025-03-26 11:00:22 +00:00
Owner
Copy link

recursive nix has been such a huge pain in our tail for anything async-related, it isn't (can't be?) implemented an platforms that aren't linux without completely disabling the sandbox (and thus invalidating the entire concept). let's get rid of it.

recursive nix has been *such* a huge pain in our tail for anything async-related, it isn't (can't be?) implemented an platforms that aren't linux without completely disabling the sandbox (and thus invalidating the entire concept). let's get rid of it.
kloenk commented 2025-03-26 14:40:21 +00:00
Member
Copy link

For me it's always a this (recursive-nix) feature looks interesting, and somehow teaching tools to create smaller derivation to cache through nix derivations sounds interesting. But never actually used it as there is not really tooling around it and so never got around to it. So sad to see it go, but also will not actually miss anything

For me it's always a this (recursive-nix) feature looks interesting, and somehow teaching tools to create smaller derivation to cache through nix derivations sounds interesting. But never actually used it as there is not really tooling around it and so never got around to it. So sad to see it go, but also will not actually miss anything
benaryorg commented 2025-04-26 00:35:43 +00:00
Copy link

Since I'm not very active in community spaces I only noticed this because my pipelines against Lix main started failing all of a sudden, so I don't really know where else to ask; with the removal of this feature is there still a way to run tools like nix-diff (generally everything that has to query the store to produce its output) inside a derivation?
I've been using this for a while now with recursive-nix to diff two .drvs, which I did struggle to emulate otherwise before opting for recursive-nix.

Since I'm not very active in community spaces I only noticed this because my pipelines against Lix *main* started failing all of a sudden, so I don't really know where else to ask; with the removal of this feature is there still a way to run tools like *nix-diff* (generally everything that has to query the store to produce its output) inside a derivation? I've been using this for a while now with *recursive-nix* to diff two `.drv`s, which I did struggle to emulate otherwise before opting for *recursive-nix*.
jade commented 2025-04-26 00:59:26 +00:00
Author
Owner
Copy link

yes, you can use --store local or NIX_REMOTE=local or similar, theoretically, which should convince the lix to not try the daemon. i haven't tested this though. it might blow up for bad reasons.

yes, you can use --store local or NIX_REMOTE=local or similar, theoretically, which should convince the lix to not try the daemon. i haven't tested this though. it might blow up for bad reasons.
pennae commented 2025-04-26 10:41:05 +00:00
Owner
Copy link

nixos itself uses nix in the sandbox with a store to build its option docs: 3afc8f4712/nixos/modules/misc/documentation.nix (L131)

nixos itself uses nix in the sandbox with a store to build its option docs: https://github.com/NixOS/nixpkgs/blob/3afc8f47128378578094bdd00e8d2e0c9eb0f307/nixos/modules/misc/documentation.nix#L131
benaryorg commented 2025-04-26 16:20:02 +00:00
Copy link

Thanks everyone for the input.

After taking a stab at it again I remember what wall I ran into back then; while nix-diff seems to work just fine without recursive-nix (as long as you throw it against the .drv files directly), nvd calls the Nix CLI to query about references. Since /nix/var doesn't exist to begin with thore are parts of the tooling which fall face flat with EPERM trying to create it, and bending NIX_STATE_DIR yields a writable but empty storedb giving either "not in store" or "invalid" for the store paths and their .drv files respectively.
Moving the entire NIX_STORE_DIR as per № 3 of the OP of course would mean rebuilding which is not feasible in that scenario.

So the workaround for all of this is a mix of direct .drv files for nix-diff and loading the store path from the closure info in the same way NixOS ISO/squashfs builds rebuild the store from the closure info paths embedded in the squashfs for nvd (though I guess this'd also work for nix-diff.…?).

(just in case anyone opens this issue trying to achieve something similar; using NIX_STATE_DIR and nix-store --load-db should be able to get you most of the way for a lot of applications)

With this change I can actually rip recursive-nix out of my cursed setup making it ever so slightly less cursed.
Thanks everyone.

Thanks everyone for the input. After taking a stab at it again I remember what wall I ran into back then; while *nix-diff* seems to work just fine without *recursive-nix* (as long as you throw it against the `.drv` files **directly**), *nvd* calls the Nix CLI to [query about references](https://git.sr.ht/~khumba/nvd/tree/b1f5f07a3bf0c6212a03600b875ea512166e32d8/item/src/nvd#L298). Since */nix/var* doesn't exist to begin with thore are parts of the tooling which fall face flat with EPERM trying to create it, and bending *NIX_STATE_DIR* yields a writable but empty storedb giving either "not in store" or "invalid" for the store paths and their `.drv` files respectively. Moving the entire *NIX_STORE_DIR* as per [№ 3 of the OP](https://git.lix.systems/lix-project/lix/issues/767#issue-13149) of course would mean rebuilding which is not feasible in that scenario. So the workaround for all of this is a mix of direct `.drv` files for *nix-diff* and loading the store path from the closure info in the same way [NixOS ISO/squashfs builds rebuild the store](https://github.com/NixOS/nixpkgs/blob/494e8180e3ea322088dbfe5fee714888c902be27/nixos/modules/installer/cd-dvd/iso-image.nix#L1012) from the [closure info paths embedded in the squashfs](https://github.com/NixOS/nixpkgs/blob/2debd4ebfe3a0aa1b2a8cc6b400cbe06675b34b3/nixos/lib/make-btrfs-fs.nix#L25) for *nvd* (though I guess this'd also work for *nix-diff*.…?). (just in case anyone opens this issue trying to achieve something similar; using *NIX_STATE_DIR* and `nix-store --load-db` should be able to get you most of the way for a lot of applications) With [this change](https://git.shell.bsocat.net/infra/commit/?h=e431269f72be653b4ae5632818e63885e4fd4e1c) I can actually rip *recursive-nix* out of my cursed setup making it ever so slightly less cursed. Thanks everyone.
🎉 1
raito commented 2025-04-28 21:01:29 +00:00
Owner
Copy link

This was finally done, feel free to comment after this issue is closed if you want to add your remarks. We are aware of a distribution in https://discourse.nixos.org/t/opinionated-patchset-with-recursive-nix-planned-for-lix/63494 that plans to offer recursive-nix back.
Check them out if you really need that feature on the top of Lix.

This was finally done, feel free to comment after this issue is closed if you want to add your remarks. We are aware of a distribution in https://discourse.nixos.org/t/opinionated-patchset-with-recursive-nix-planned-for-lix/63494 that plans to offer `recursive-nix` back. Check them out if you really need that feature on the top of Lix.
raito closed this issue 2025-04-28 21:01:29 +00:00
bacchanalia commented 2025-05-29 18:47:20 +00:00
Member
Copy link

I'm in the process of switching my organization over to lix now, and we a have been actively using recursive-nix. We evaluate a large number of configurations in CI to check that they still evaluate for a PR and I was using recursive-nix to offload that work to our farm of remote builders, in a way that integrates nicely with our CI and it simplifies managing resources on our builders because it just uses build slots like everything else.

I'm in the process of switching my organization over to lix now, and we a have been actively using recursive-nix. We evaluate a large number of configurations in CI to check that they still evaluate for a PR and I was using recursive-nix to offload that work to our farm of remote builders, in a way that integrates nicely with our CI and it simplifies managing resources on our builders because it just uses build slots like everything else.
raito commented 2025-05-30 09:59:57 +00:00
Owner
Copy link

@bacchanalia wrote in #767 (comment):

I'm in the process of switching my organization over to lix now, and we a have been actively using recursive-nix. We evaluate a large number of configurations in CI to check that they still evaluate for a PR and I was using recursive-nix to offload that work to our farm of remote builders, in a way that integrates nicely with our CI and it simplifies managing resources on our builders because it just uses build slots like everything else.

I'm not deeply familiar with your CI system, but would you know why one of the following two options are not feasible in your case:

  • remove the use of nix build in the entrypoint of your CI so that you only run the inner nix build wrapped in the derivations?
  • add an impure sandbox path via /etc/nix/nix.conf pointing to the Nix's ambient daemon so that you can run nix build inside will reuse the parent daemon socket, this is similar to what recursive-nix does without all the store paths restriction and special casing that you do not seem to be using

Let us know if we are still missing a story in our understanding of your usage.

@bacchanalia wrote in https://git.lix.systems/lix-project/lix/issues/767#issuecomment-12318: > I'm in the process of switching my organization over to lix now, and we a have been actively using recursive-nix. We evaluate a large number of configurations in CI to check that they still evaluate for a PR and I was using recursive-nix to offload that work to our farm of remote builders, in a way that integrates nicely with our CI and it simplifies managing resources on our builders because it just uses build slots like everything else. I'm not deeply familiar with your CI system, but would you know why one of the following two options are not feasible in your case: - remove the use of `nix build` in the entrypoint of your CI so that you only run the inner `nix build` wrapped in the derivations? - add an impure sandbox path via `/etc/nix/nix.conf` pointing to the Nix's ambient daemon so that you can run `nix build` inside will reuse the parent daemon socket, this is similar to what `recursive-nix` does without all the store paths restriction and special casing that you do not seem to be using Let us know if we are still missing a story in our understanding of your usage.
bacchanalia commented 2025-06-06 22:13:31 +00:00
Member
Copy link

@raito I was trying to get eval working within a build by exposing the socket to the sandbox as you suggested, but I'm getting
error: creating directory '/nix/var': Permission denied even if I include all the way up to /nix/var as an extra-sandbox-path.
(also, I'm not sure what you mean by the first suggestion?)

@raito I was trying to get eval working within a build by exposing the socket to the sandbox as you suggested, but I'm getting `error: creating directory '/nix/var': Permission denied` even if I include all the way up to `/nix/var` as an extra-sandbox-path. (also, I'm not sure what you mean by the first suggestion?)
jade commented 2025-06-07 02:16:54 +00:00
Author
Owner
Copy link

/nix/var should be something that's overridable by environment variable, check the hastily written environment variable list in testing.md

/nix/var should be something that's overridable by environment variable, check the hastily written environment variable list in testing.md
bacchanalia commented 2025-06-10 05:21:18 +00:00
Member
Copy link

I managed to get it working after removing the group check in daemon.cc, setting noChroot = true; on the derivation. I was unable to get it to work with extra-sandbox-paths, I think due to paths required for eval not being available in the chroot.

I managed to get it working after removing the group check in [daemon.cc](https://git.lix.systems/lix-project/lix/src/branch/main/lix/nix/daemon.cc#L264), setting `noChroot = true;` on the derivation. I was unable to get it to work with `extra-sandbox-paths`, I think due to paths required for eval not being available in the chroot.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ci-config
release-2.91
release-2.92
release-2.93
sb/raito/darwin-kqueue-disablement
sb/raito/hydra-merge
sb/pennae/ci-config
sb/raito/phantom-referrers-gc
repl-overlay-example
reuse-compliance
release-2.90
sb/pennae/io-rewrite
git-series/activity-cleanup
sb/tom-hubrecht/primops.cc-v2
sb/bacchanalia/cache-strlen%topic=substr-perf
sb/qyriad/beta-1
sb/qyriad/2.90-beta.0
sb/rbt/test-branch
sb/rbt/justfile
sb/rbt/repl-overlays
sb/qyriad/maint-meson
sb/qyriad/maint/meson
sb/puck/meson-c-api
sb/jade/burn-it-all
dependabot/github_actions/zeebe-io/backport-action-2.4.1
dependabot/github_actions/cachix/install-nix-action-25
dependabot/github_actions/actions/labeler-5
dependabot/github_actions/cachix/cachix-action-14
2.93.3
2.92.3
2.93.2
2.91.3
2.92.2
2.91.2
2.93.1
2.93.0
2.92.0
2.91.1
2.91.0
2.90.0
2.90.0-rc1
2.90-beta.1
2.90-beta.0
Labels
Clear labels   
Affects/CppNix

Probably affects CppNix. This is a best-effort tag for troubles in old code; it exists to inform the CppNix team about problems they might care about also.

  
Affects/Nightly

Found in Lix nightly.

  
Affects/Only nightly

Tag for regressions in Lix nightly. We intend to fix all of these prior to releases.

  
Affects/Stable

Affects the stable release of Lix. Usually such issues affect nightly too but there is no guarantee.

  
Area/build-packaging

regarding the building or packaging of Lix itself

  
Area/cli

The command line interface of Lix

  
Area/evaluator

libexpr and anything else related to the evaluation of Nix expressions

  
Area/fetching

Fetching sources or other materials from the World Wide Web in Lix

  
Area/flakes

   
Area/language

language design, features, extensions, etc

  
Area/lix ci

Lix's own CI and deployment pipeline

  
Area/nix-eval-jobs

nix-eval-jobs for Lix

  
Area/profiles

profiles: nix-env, nix profile, and installer flavors all alike

  
Area/protocol

The protocol, used for communicating between the client and the daemon

  
Area/releng

Release engineering

  
Area/remote-builds

remote builds

  
Area/repl

nix repl

  
Area/repl/debugger

The --debugger that brings up a Lix repl

  
Area/store

libstore and anything else relating to the Nix store

  
bug

it's a bug

  
Context
contributors
This issue might require some context that isn't written down. It might be possible to handle as a person who already worked on this area.

  
Context
drive-by
This issue contains all the necessary information to do a drive-by contribution: that is, you don't have to talk with anyone at all to successfully solve it!

  
Context
maintainers
This issue requires a lot of know-how. Attention from maintainers is needed to proceed.

  
Context
RFD
This issue needs discussion from multiple maintainers and interested parties: it's a Request For Discussion.

  
crash 💥

Lix crashes (possibly bad error handling)

  
Cross Compilation

   
devx

development experience issues

  
docs

requires or is about documentation changes

  
Downstream Dependents

regarding an external tool's usage with Lix (e.g. nix-eval-jobs)

  
E/easy

bug is relatively simple and could simply be fixed with some minor effort

  
E/hard

this issue should probably be solved with close involvement of an expert

  
E/help wanted

maintainers may not get to this soon, but we want it fixed

  
E/reproducible

This bug is reproducible and obvious.

  
E/requires rearchitecture

this bug cannot be fixed properly without fixing a known architectural flaw

  
imported

Issue imported from upstream nix (please do not remove this label, it will cause dupes if import is rerun)

  
Language/Bash

   
Language/C++

   
Language/NixLang

   
Language/Python

   
Language/Rust

   
Needs Langver

We can't get to this until we implement language versioning

  
OS/Linux

Affects specifically Lix on Linux

  
OS/macOS

Affects specifically Lix on macOS

  
performance

make lix go faster

  
regression

This is a regression since some previous version of Lix

  
release-blocker

Blocks public release

  
stability

Fixing this makes the software more stable

  
Status
blocked
Blocked on something

  
Status
invalid
not our bug, duplicate, similar

  
Status
postponed
closed but should maybe look at later (generally with S-wontfix)

  
Status
wontfix
We chose not to change this (see issue comments for more detail).

  
testing

Tests that should be added

  
testing/flakey

Tests that are unreliable/flakey (not to be confused with flakes that test)

  
Topic/Large Scale Installations

Large scale installations of Lix, e.g. NixOS Hydra, Lix CI, big corporate installations etc.

  
ux

makes the user experience better :)

No labels
Affects/CppNix
Affects/Nightly
Affects/Only nightly
Affects/Stable
Area/build-packaging
Area/cli
Area/evaluator
Area/fetching
Area/flakes
Area/language
Area/lix ci
Area/nix-eval-jobs
Area/profiles
Area/protocol
Area/releng
Area/remote-builds
Area/repl
Area/repl/debugger
Area/store
bug
Context
contributors
Context
drive-by
Context
maintainers
Context
RFD
crash 💥
Cross Compilation
devx
docs
Downstream Dependents
E/easy
E/hard
E/help wanted
E/reproducible
E/requires rearchitecture
imported
Language/Bash
Language/C++
Language/NixLang
Language/Python
Language/Rust
Needs Langver
OS/Linux
OS/macOS
performance
regression
release-blocker
stability
Status
blocked
Status
invalid
Status
postponed
Status
wontfix
testing
testing/flakey
Topic/Large Scale Installations
ux
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
8 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#767
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?