Have a nixpkgs overlay to avoid enumerating packages

raito commented

2025-09-21 10:26:52 +00:00

Owner

Re: lix-project/lix#989

cc @dwt @yu-re-ka

Re: lix-project/lix#989 cc @dwt @yu-re-ka

raito referenced this issue from lix-project/lix

2025-09-21 10:27:11 +00:00

Change nixpgks overlay so I do not have to enumerate all the packages that are included #989

raito commented

2025-09-21 10:28:20 +00:00

Author

Owner

@dwt wrote in lix-project/lix#989 (comment):

@raito @yu-re-ka Thinking about this a bit further, the plans you outline seem like they require quite a bit of know how in nixpkgs and perhaps even quite a bit clout in the nixpkgs project to get merged.

Would you oppose some incremental improvement? I am thinking of refining the package set, so it is possible to easily apply all packages within it as an overlay yourself. Something like:
overlay = final: prev: final.lixPackageSet.latest
To my understanding that would require modifying the current set to get rid of the lix package inside it, as well as the storeDir, stateDir and confDir variable.

Yeah, I don't think I am a fan of getting rid of it that way because it introduces a lot of churn for our users.

What about:

overlay = final: prev: prev.lixPackageSet.stable.lix.overlay # or .stable.overlay

? where overlay is a passthru argument.

@dwt wrote in https://git.lix.systems/lix-project/lix/issues/989#issuecomment-14942: > @raito @yu-re-ka Thinking about this a bit further, the plans you outline seem like they require quite a bit of know how in nixpkgs and perhaps even quite a bit clout in the nixpkgs project to get merged. > > Would you oppose some incremental improvement? I am thinking of refining the package set, so it is possible to easily apply all packages within it as an overlay yourself. Something like: > > ```nix > overlay = final: prev: final.lixPackageSet.latest > ``` > > To my understanding that would require modifying the current set to get rid of the `lix` package inside it, as well as the `storeDir`, `stateDir` and `confDir` variable. Yeah, I don't think I am a fan of getting rid of it that way because it introduces a lot of churn for our users. What about: ```nix overlay = final: prev: prev.lixPackageSet.stable.lix.overlay # or .stable.overlay ``` ? where `overlay` is a `passthru` argument.

dwt commented

2025-09-23 07:44:57 +00:00

I would really like

overlay = final: prev: prev.lixPackageSet.stable.overlay

if nixpkgs has the stomach to do it. To me that sounds like it would bring together most of the advantages of the various approaches.

I am always not sure with overlays, but my current understanding of them would have written this overlay to come from final:

overlay = final: prev: final.lixPackageSet.stable.overlay

Is this necessary to prevent an infinite recursion here, or is there just much more to learn about overlays?

I would really like ```nix overlay = final: prev: prev.lixPackageSet.stable.overlay ``` if nixpkgs has the stomach to do it. To me that sounds like it would bring together most of the advantages of the various approaches. I am always not sure with overlays, but my current understanding of them would have written this overlay to come from final: ```nix overlay = final: prev: final.lixPackageSet.stable.overlay ``` Is this necessary to prevent an infinite recursion here, or is there just much more to learn about overlays?

raito commented

2025-09-23 15:02:12 +00:00

Author

Owner

I would have loved it and tried hard to get it in the PR I opened in nixpkgs.

Unfortunately, this cannot work in a module system configuration, observe:

{ ... }: {
   nixpkgs.overlays = [ (final: prev: final.lixPackageSet.stable.overlay) ];
}

A definition of overlay would be… ?

overlay = {
    nix-review = nix-review.override { nix = self.lix; };
}

Notice self, this refers to final actually here. Notice nix-review, it refers to final.nix-review too.

But nixpkgs.overlays will change the final, therefore: final.nix-review will change, therefore infinite recursion.

Let's fix that:

overlay = final: prev: {
    nix-review = prev.nix-review.override { nix = final.lix; }
 }

Now, we can do: nixpkgs.overlays = [ XXX.lixPackageSets.stable.overlay ];, what XXX… ?!
OK, let's do: nixpkgs.overlays = [ (final: prev: prev.lixPackageSets.stable.overlay final prev) ];, this won't work because prev will change as the overlay evolve!

Also, notice that we used final.lix, not self.lix, so it's not even dependent upon stable or latest!

In a module system configuration, we may want to have an option lix.packages = pkgs.lixPackageSets.stable;, this means that we have a dependency in pkgs, if we add a nixpkgs.overlays, pkgs will change, therefore the lix.packages will change and this will infinitely recurse again.

Conclusion: I cannot find a way to have an overlay defined inside Nixpkgs that can be applied efficiently in NixOS (or nix-darwin).

The only way I see to achieve this is to have a well-known path like <nixpkgs/overlays/lix> and import it outside of the lifecycle of the module system. This doesn't even solve the "I'd like to install the right overlay for the right package set".

e.g.

nixpkgs.overlays = [ (import <nixpkgs/overlays/lix> { }) ]; # notice, no dependency in config.lix.packages.

To solve that, we would need to make the choice of the package set a string.

lix.packages = "stable";

By doing that, we could do: nixpkgs.overlays = [ (import <nixpkgs/overlays/lix> { inherit (config.lix) packages; }) ];

The implementation of that overlay would do final.lixPackageSets.${packages}, so we lose the possibility to pass arbitrary package sets or extend the scope.

Overall, I am not sure I can figure out an ideal solution, but I need to read Yureka's views again about this.

I would have loved it and tried hard to get it in the PR I opened in nixpkgs. Unfortunately, this cannot work in a module system configuration, observe: ``` { ... }: { nixpkgs.overlays = [ (final: prev: final.lixPackageSet.stable.overlay) ]; } ``` A definition of overlay would be… ? ``` overlay = { nix-review = nix-review.override { nix = self.lix; }; } ``` Notice `self`, this refers to `final` actually here. Notice `nix-review`, it refers to `final.nix-review` too. But `nixpkgs.overlays` will change the `final`, therefore: `final.nix-review` will change, therefore infinite recursion. --- Let's fix that: ``` overlay = final: prev: { nix-review = prev.nix-review.override { nix = final.lix; } } ``` Now, we can do: `nixpkgs.overlays = [ XXX.lixPackageSets.stable.overlay ];`, what `XXX`… ?! OK, let's do: `nixpkgs.overlays = [ (final: prev: prev.lixPackageSets.stable.overlay final prev) ];`, this won't work because `prev` will change as the overlay evolve! Also, notice that we used `final.lix`, not `self.lix`, so it's not even dependent upon `stable` or `latest`! --- In a module system configuration, we may want to have an option `lix.packages = pkgs.lixPackageSets.stable;`, this means that we have a dependency in `pkgs`, if we add a `nixpkgs.overlays`, `pkgs` will *change*, therefore the `lix.packages` will change and this will infinitely recurse again. --- Conclusion: I cannot find a way to have an overlay defined inside Nixpkgs that can be applied efficiently in NixOS (or nix-darwin). The only way I see to achieve this is to have a well-known path like `<nixpkgs/overlays/lix>` and import it outside of the lifecycle of the module system. This **doesn't** even solve the "I'd like to install the right overlay for the right package set". e.g. ``` nixpkgs.overlays = [ (import <nixpkgs/overlays/lix> { }) ]; # notice, no dependency in config.lix.packages. ``` To solve that, we would need to make the choice of the package set a string. ``` lix.packages = "stable"; ``` By doing that, we could do: `nixpkgs.overlays = [ (import <nixpkgs/overlays/lix> { inherit (config.lix) packages; }) ];` The implementation of that overlay would do `final.lixPackageSets.${packages}`, so we lose the possibility to pass arbitrary package sets or extend the scope. Overall, I am not sure I can figure out an ideal solution, but I need to read Yureka's views again about this.

raito commented

2025-09-23 15:03:14 +00:00

Author

Owner

lix-project/lix#989 (comment) seems indeed the way forward to improve things but this requires cooperation with CppNix people.

I personally don't have the spoons for it, but I would happily help and enable anything on the Lix side to make this happen.

https://git.lix.systems/lix-project/lix/issues/989#issuecomment-14902 seems indeed the way forward to improve things but this requires cooperation with CppNix people. I personally don't have the spoons for it, but I would happily help and enable anything on the Lix side to make this happen.

dwt commented

2025-09-25 06:32:05 +00:00

Yeah, I think I slowly understand why @yu-re-ka made the suggestion she did. How would it look if we incrementally worked in that direction? Would we start with config.lix.packageSet and then later rename that to config.nix.packageSet?

My naive view is that this should not involve collaboration of cppnix, as this is all happening in nixpkgs? When we move to config.nix.packageSet we would need some forwards to be in place, but that shouldn't be a big deal?

Yeah, I think I slowly understand why @yu-re-ka made the suggestion she did. How would it look if we incrementally worked in that direction? Would we start with `config.lix.packageSet` and then later rename that to `config.nix.packageSet`? My naive view is that this should not involve collaboration of cppnix, as this is all happening in nixpkgs? When we move to `config.nix.packageSet` we would need some forwards to be in place, but that shouldn't be a big deal?

yu-re-ka commented

2025-09-25 11:54:35 +00:00

Member

The problem is that users are always going to attempt using the pkgs.colmena / pkgs.nix-review / ... from the all-packages package set first, and they will not get the version built against their chosen Nix/Lix version.
Without removing the packages from all-packages, we really don't get any benefit over the current state (where users can already spell out pkgs.lixPackageSets.latest.<package>).

Otherwise it could still be implemented in steps though:

Add a pkgs/top-level/nix-impl-packageset.nix which directly includes the appropriate callPackage calls (instead of overriding the packages from all-packages, since we ultimately want to remove them)
Convince the relevant people that it makes sense to use the same packagesFor on CppNix (to choose between different versions of CppNix etc.)
remove the packages from all-packages.nix / by-name and point to the relevant packagesets

At any point in between: Introduce a NixOS / nix-darwin option used to reference the user-chosen Nix/Lix packageSet

The problem is that users are always going to attempt using the pkgs.colmena / pkgs.nix-review / ... from the all-packages package set first, and they will not get the version built against their chosen Nix/Lix version. Without removing the packages from all-packages, we really don't get any benefit over the current state (where users can already spell out pkgs.lixPackageSets.latest.\<package>). Otherwise it could still be implemented in steps though: - Add a pkgs/top-level/nix-impl-packageset.nix which directly includes the appropriate `callPackage` calls (instead of overriding the packages from all-packages, since we ultimately want to remove them) - Convince the relevant people that it makes sense to use the same packagesFor on CppNix (to choose between different versions of CppNix etc.) - remove the packages from all-packages.nix / by-name and point to the relevant packagesets At any point in between: Introduce a NixOS / nix-darwin option used to reference the user-chosen Nix/Lix packageSet

Rows
Columns

Have a nixpkgs overlay to avoid enumerating packages #94