lix-project/lix

Fork 56

Meta-bug: `builtins.derivation` replacement design #217

New issue

Open

opened 2024-04-02 20:50:45 +00:00 by jade · 5 comments

jade commented

2024-04-02 20:50:45 +00:00

Owner

builtins.derivation has several intractable legacy problems:

Mixing env-vars with control attributes for the nixpkgs builder
Mixing control attributes for the nix builder with ones for the nixpkgs builder.
Control attributes (like __noChroot) don't say who they are for: are they for scheduling? are they for the builder daemon? are they secretly requests for some feature to be available to the builder? are they for cache upload attrs (e.g. "this is debuginfo, don't retain it as long").
structuredAttrs didn't really get adopted, which would fix parts of this.
builtins.derivationStrict has better semantics, but is not used in practice
There is no way to feature detect features of the builder to require that the thing is actually scheduled on a machine that can do it (though this might be intractable to fix, in general).

The current interface to it needs to be redesigned to specify better who everything is for, which then makes it much easier to figure out as well where it should go in the derivation format, etc, etc, etc.

It would not require significant (dangerous, at least) rearchitecture to get started on this issue, and would be preferable over adding anything to the list of special control attrs, which might be incompatible with upstream C++Nix.

Here is a pad (freely editable for everyone; consider writing who you are next to your comments) for working on a replacement API design: https://pad.lix.systems/lix-builtins-derivation

Related issues:

`builtins.derivation` has several intractable legacy problems: - Mixing env-vars with control attributes for the nixpkgs builder - Mixing control attributes for *the nix builder* with ones for the nixpkgs builder. - Control attributes (like `__noChroot`) don't say who they are for: are they for scheduling? are they for the builder daemon? are they secretly requests for some feature to be available to the builder? are they for cache upload attrs (e.g. "this is debuginfo, don't retain it as long"). - structuredAttrs didn't really get adopted, which would fix parts of this. - `builtins.derivationStrict` has better semantics, but is not used in practice - There is no way to feature detect features of the builder to require that the thing is actually scheduled on a machine that can do it (though this might be intractable to fix, in general). The current interface to it needs to be redesigned to specify better *who* everything is for, which then makes it much easier to figure out as well where it should go in the derivation format, etc, etc, etc. It would not require significant (dangerous, at least) rearchitecture to get started on this issue, and would be preferable over adding anything to the list of special control attrs, which might be incompatible with upstream C++Nix. Here is a pad (freely editable for everyone; consider writing who you are next to your comments) for working on a replacement API design: https://pad.lix.systems/lix-builtins-derivation Related issues: - https://git.lix.systems/lix-project/lix/issues/121 - https://git.lix.systems/lix-project/lix/issues/84 - https://git.lix.systems/lix-project/lix/issues/73 - https://git.lix.systems/lix-project/lix/issues/63

jade added the

Context

RFD

label

2024-04-02 20:50:50 +00:00

pennae commented

2024-04-02 20:57:31 +00:00

Owner

comments of note:

derivationStrict is used, it's the backend for derivation and has all the same problems
structuredAttrs are very slowly gaining traction, having nixpkgs folks push in that direction may be helpful

comments of note: - `derivationStrict` *is* used, it's the backend for `derivation` and has all the same problems - `structuredAttrs` are *very* slowly gaining traction, having nixpkgs folks push in that direction may be helpful

jade commented

2024-04-02 21:03:06 +00:00

Author

Owner

Right, but the semantics of derivationStrict are better (according to @puck at least, iwrc), and so its lack of direct use is sort of strange, and the fact that builtins.derivation as a wrapper exists is also kind of strange.

Right, but the semantics of `derivationStrict` are better (according to @puck at least, iwrc), and so its lack of direct use is sort of strange, and the fact that `builtins.derivation` as a wrapper exists is also kind of strange.

pennae commented

2024-04-02 21:05:51 +00:00

Owner

derivation x is basically x: let s = derivationStrict x; in x // { type = "derivation"; inherit (s) outPath drvPath; } with some sugar coat. it has all the same problems as derivation by virtue of being the problem with derivation; there are some semantic bits in the strict variant that are pleasant but they're unrelated to the problem being discussed here

`derivation x` is *basically* `x: let s = derivationStrict x; in x // { type = "derivation"; inherit (s) outPath drvPath; }` with some sugar coat. it has all the same problems as `derivation` by virtue of *being* the problem with `derivation`; there are some semantic bits in the strict variant that are pleasant but they're unrelated to the problem being discussed here

jade referenced this issue

2024-05-05 04:23:53 +00:00

Do something about allowNewPrivileges setting #265

roberth commented

2024-05-29 21:57:30 +00:00

Member

These points largely agree with nix#9774. (Ignoring the bit about meta perhaps)
I'd like to keep in touch about this, and perhaps collaborate?

Also I was very close to making Nixpkgs use derivationStrict directly in nixpkgs#217243 cleanAttrs, before deciding not to take that broader direction. I believe it should be fine for Nixpkgs to do that still. It'd improve performance slightly.

These points largely agree with [nix#9774](https://github.com/NixOS/nix/issues/9774). (Ignoring the bit about `meta` perhaps) I'd like to keep in touch about this, and perhaps collaborate? Also I was very close to making Nixpkgs use `derivationStrict` directly in [nixpkgs#217243 cleanAttrs](https://github.com/NixOS/nixpkgs/pull/217243), before deciding not to take that broader direction. I believe it should be fine for Nixpkgs to do that still. It'd improve performance slightly.

lix-project referenced this issue from a commit

2025-01-29 18:11:13 +00:00

Merge pull request #217 from nix-community/update_flake_lock_action

jade referenced this issue

2025-07-28 02:20:10 +00:00

sandbox setting is a horrifying field of landmines of implementation complexity #936

ericson2314 commented

2025-07-30 23:02:16 +00:00

My view is that it is better to first fix the underlying issues at the store layer, and then as a "victory lap" make the better derivations primop in the language.

https://github.com/NixOS/nix/pull/10760 This is the final PR in my long saga to factor out all the env var hacks, and structured attrs __json hacks, and make the Derivation data type ready for a better builtin (and better on-disk format). I strongly encourage Lix to cherry-pick the relevant commits.

If we do that (or you all otherwise get your internal data structures ready), I think a next step could be thinking about the successor for buildDerivation in a new protocol, e.g. as used by Hydra. A new protocol (that is designed with extensibility/versioning in mind) is a good opportunity to try things out and make things better with something that isn't exposed, but isn't as visible low/churn as Nix code / Nixpkgs.

My view is that it is better to first fix the underlying issues at the store layer, and then as a "victory lap" make the better derivations primop in the language. https://github.com/NixOS/nix/pull/10760 This is the final PR in my long saga to factor out all the env var hacks, and structured attrs `__json` hacks, and make the `Derivation` data type ready for a better builtin (and better on-disk format). I strongly encourage Lix to cherry-pick the relevant commits. If we do that (or you all otherwise get your internal data structures ready), I think a next step could be thinking about the successor for `buildDerivation` in a new protocol, e.g. as used by Hydra. A new protocol (that is designed with extensibility/versioning in mind) is a good opportunity to try things out and make things better with something that isn't exposed, but isn't as visible low/churn as Nix code / Nixpkgs.