shared understanding of phases with nixpkgs

dwt commented

2025-08-26 18:58:16 +00:00

When trying to debug a nix build, it's possible to start the nix shell and if you know where to look you can figure out all the phases this build has. Then you can start them in the right order by invoking the *Phase shell functions / scripts.

This works, but is quite magic.

Describe the solution you'd like

We already have nix develop --unpack and friends, as well as the more generic nix develop --phase $SOME_PHASE. But we do not have a way to get the list of phases in the right order.

This is what I'm proposing: a flag that just prints those in the order that they need to be executed to make the build happen. Then you can manually run and re-run those phases as often as you need to debug them.

Describe alternatives you've considered

Better documentation of how this works is probably still a good idea. Good, but still very slow when you have to re-create the develop shell every few seconds because you have to change the actual package for debugging.
Make it easier to print this list from the develop shell: The nix develop --phase $PHASE has the advantage that it is easy to run, even while you are working on (and changing) the nixpkgs package.

What do you think?

## Is your feature request related to a problem? Please describe. When trying to debug a nix build, it's possible to start the nix shell and if you know where to look you can figure out all the phases this build has. Then you can start them in the right order by invoking the `*Phase` shell functions / scripts. This works, but is quite magic. ## Describe the solution you'd like We already have `nix develop --unpack` and friends, as well as the more generic `nix develop --phase $SOME_PHASE`. But we do not have a way to get the list of phases in the right order. This is what I'm proposing: a flag that just prints those in the order that they need to be executed to make the build happen. Then you can manually run and re-run those phases as often as you need to debug them. ## Describe alternatives you've considered - Better documentation of how this works is probably still a good idea. Good, but still very slow when you have to re-create the develop shell every few seconds because you have to change the actual package for debugging. - Make it easier to print this list from the develop shell: The `nix develop --phase $PHASE` has the advantage that it is easy to run, even while you are working on (and changing) the nixpkgs package. What do you think?

jade commented

2025-08-26 19:07:59 +00:00

Owner

I think: this is a layering violation. I am not sure how responsibility should be passed off to nixpkgs, but we should be very careful about how we implement this so that we are not tangling ourselves in nixpkgs in nix develop any more than necessary. I am in general opposed in principle to implementing this in Lix as it is likely to cause pain to both us and nixpkgs due to the layering violation surface area.

Maybe a printPhases function should exist in stdenv? Should stdenv have a way of jamming some debugging shell hooks into nix develop shells?

N.B. I know that nix develop, even more than nix-shell (somehow?) is the layering violation command. Adding more of those adds more fragility and difficulty for nixpkgs to do its job.

I think: this is a layering violation. I am not sure *how* responsibility should be passed off to nixpkgs, but we should be very careful about how we implement this so that we are not tangling ourselves in nixpkgs in `nix develop` any more than necessary. I am in general opposed in principle to implementing this in Lix as it is likely to cause pain to both us and nixpkgs due to the layering violation surface area. Maybe a `printPhases` function should exist in stdenv? Should stdenv have a way of jamming some debugging shell hooks into `nix develop` shells? N.B. I know that `nix develop`, even more than `nix-shell` (somehow?) *is* the layering violation command. Adding more of those adds more fragility and difficulty for nixpkgs to do its job.

jade added the

Area/cli

label

2025-08-26 19:08:06 +00:00

qyriad added the

ux

label

2025-08-26 20:00:38 +00:00

qyriad commented

2025-08-26 20:14:14 +00:00

Owner

Lix actually has no idea what phases there are:

script += fmt("runHook %1%Phase\n", *phase);

All the existing phase flags like --unpack are hardcoded, and won't even work in cases where a setup hook adds a phase that semantically performs that function but is under a different name (did you know pytest doesn't run in checkPhase or installCheckPhase?). Lix just assumes that for --phase foo the bash command runHook foo is enough, but setup hooks that add or change phases don't all work the same way and I would be shocked if this wasn't already broken for various phases in Nixpkgs.

I think this is a hard sell for Lix, and stdenv's phase handling could really use the extra friendliness already.

Lix actually has no idea what phases there are: https://git.lix.systems/lix-project/lix/src/commit/d5970d8a8bf78a3efbf072c4f85ea5cfde6d94d6/lix/nix/develop.cc#L574 All the existing phase flags like `--unpack` are hardcoded, and won't even work in cases where a setup hook adds a phase that semantically performs that function but is under a different name (did you know `pytest` doesn't run in `checkPhase` or `installCheckPhase`?). Lix just assumes that for `--phase foo` the bash command `runHook foo` is enough, but setup hooks that add or change phases don't all work the same way and I would be shocked if this **wasn't** already broken for various phases in Nixpkgs. I think this is a hard sell for Lix, and stdenv's phase handling could *really* use the extra friendliness already.

raito commented

2025-08-26 20:53:44 +00:00

Owner

Maybe if we wanted to look at what would be a way forward for the semantic need of this issue: how can nixpkgs communicate to Nix information about its available phases, planned phases, ongoing phases and so on?

If a standard format is described and if we had (we don't) a standard protocol for the sandbox to report metadata to the driver subdaemon, we could do nifty things.

I surmise that getting right the standard format and what information is useful at the stdenv and nixpkgs level is going to be some long work (not a quick win, for sure; but a very much needed win.)
At the same time, Lix will be able to get progress on RPC and perhaps, we will get all the pieces to connect it together.

In other words:

Design a standard data interexchange meant for the sandbox and the driver subdaemon regarding metadata (nixpkgs)
Design a standard protocol to send it to the sandbox (Lix)
Capture that information in Lix and expose it at RPC boundaries and some CLI boundaries (Lix)

Maybe if we wanted to look at what would be a way forward for the semantic need of this issue: how can nixpkgs communicate to Nix information about its available phases, planned phases, ongoing phases and so on? If a standard format is described and if we had (we don't) a standard protocol for the sandbox to report metadata to the driver subdaemon, we could do nifty things. I surmise that getting right the standard format and what information is useful at the stdenv and nixpkgs level is going to be some long work (not a quick win, for sure; but a very much needed win.) At the same time, Lix will be able to get progress on RPC and perhaps, we will get all the pieces to connect it together. In other words: - Design a standard data interexchange meant for the sandbox and the driver subdaemon regarding metadata (nixpkgs) - Design a standard protocol to send it to the sandbox (Lix) - Capture that information in Lix and expose it at RPC boundaries and some CLI boundaries (Lix)

jade commented

2025-08-27 06:51:07 +00:00

Owner

nixpkgs does already send phase information of what phase it is currently on to $NIX_LOG_FD, but that's only if it is actually conducting a build and it does not include information about all involved phases. The reporter is mostly asking about how to inspect it without conducting a build, but I realize it would be really valuable in a build as well.

nixpkgs *does* already send phase information of what phase it is currently on to $NIX_LOG_FD, but that's only if it is actually conducting a build and it does not include information about all involved phases. The reporter is mostly asking about how to inspect it without conducting a build, but I realize it would be really valuable in a build as well.

👍 1

pennae commented

2025-08-30 17:26:21 +00:00

Owner

in the current nixpkgs architecture it is in general impossible to know which phases will be run without running them since hooks are loaded dynamically and may or may not inject themselves into a build depending on circumstances lix cannot inspect any earlier than the hook scripts can. a printPhases or something in nixpkgs would sidestep this by turning a real build into a fake one that only prints the phases, but that too comes at the cost of realizing all inputs and is thus probably not a good idea.

in our view this is first and foremost a nixpkgs architecture problem, not a lix problem. if we do turn this into a lix problem we should think about more general solutions that let us lift the phases array into a lix build graph.

in the current nixpkgs architecture it is in general *impossible* to know which phases will be run without running them since hooks are loaded dynamically and may or may not inject themselves into a build depending on circumstances lix cannot inspect any earlier than the hook scripts can. a `printPhases` or something in nixpkgs would sidestep this by turning a real build into a fake one that only prints the phases, but that too comes at the cost of realizing *all* inputs and is thus probably not a good idea. in our view this is first and foremost a nixpkgs architecture problem, not a lix problem. if we *do* turn this into a lix problem we should think about more general solutions that let us lift the phases array into a lix build graph.

dwt commented

2025-08-31 13:38:18 +00:00

Author

Getting the lix phases lifted into a lix build graph would be Wonderfull. That would neatly facilitate re-running and developing on failing phases.

But that also sounds like a huge Chunk of work, that nixpkgs would have to participate in. Or am I missing something?

Getting the lix phases lifted into a lix build graph would be Wonderfull. That would neatly facilitate re-running and developing on failing phases. But that also sounds like a huge Chunk of work, that nixpkgs would have to participate in. Or am I missing something?

jade commented

2025-08-31 13:47:10 +00:00

Owner

Correct assessment; this issue can basically be closed WONTFIX due to the extended timelines on which it is remotely likely for that to happen. It's pretty pie in the sky for implementation complexity, current dev capacity, and high coordination complexity.

Your immediate question is likely to need to be answered by nixpkgs without any lix involvement.

Correct assessment; this issue can basically be closed WONTFIX due to the extended timelines on which it is remotely likely for that to happen. It's pretty pie in the sky for implementation complexity, current dev capacity, and high coordination complexity. Your immediate question is likely to need to be answered by nixpkgs without any lix involvement.

jade

2025-08-31 13:47:40 +00:00