lix-project/lix

Fork 65

RFD: remove `build-hook` setting #911

New issue

Open

opened 2025-07-10 18:14:41 +00:00 by pennae · 7 comments

pennae commented

2025-07-10 18:14:41 +00:00

Owner

build-hook and its associated nix __build-remote are extremely cursed. each remote build uses its own build-hook process, which holds on to a single local store connection and another single remote store connection (which itself usually holds on to a single ssh process). scheduling of remote builders is complicated immensely by this because it all happens through file locks, the additional localstore connection causes overhead, and the behavior differences of ssh and ssh-ng remotes are a major headache.

we should probably just remove build-hook altogether. an rpc-based system will not have any use for build hooks as they exist today, and what little can be gained from build hooks can also be got from plugins that add custom store url schemes. we haven't been able to find any use of build-hook out there either.

this is not about removing pre-build-hook or post-build-hook, just build-hook. we've already deprecated build-hook a while ago.

`build-hook` and its associated `nix __build-remote` are extremely cursed. each remote build uses its own build-hook process, which holds on to a single local store connection and another single remote store connection (which itself usually holds on to a single ssh process). scheduling of remote builders is complicated immensely by this because it all happens through file locks, the additional localstore connection causes overhead, and the behavior differences of ssh and ssh-ng remotes are a major headache. we should probably just remove `build-hook` altogether. an rpc-based system will not have any use for build hooks as they exist today, and what little can be gained from build hooks can also be got from plugins that add custom store url schemes. we haven't been able to find *any* use of `build-hook` out there either. this is *not* about removing `pre-build-hook` or `post-build-hook`, just `build-hook`. we've already deprecated `build-hook` a while ago.

pennae added the

Context

RFD

label

2025-07-10 18:14:41 +00:00

raito commented

2025-11-18 02:33:20 +00:00

Owner

should we get this done in 2.95.0? if so, please add release blocker to it

pennae commented

2025-11-18 12:55:18 +00:00

Author

Owner

not entirely sure yet. we'll add it as a blocker just so we don't forget about it, if 2.95 ends up taking too long we can also postpone this a bit

pennae added the

Release Blocking

Urgent

label

2025-11-18 12:55:27 +00:00

pennae added this to the 2.97 milestone

2025-12-01 14:51:11 +00:00

pennae modified the milestone from 2.97 to 2.95

2025-12-01 14:52:00 +00:00

pennae removed the

Release Blocking

Urgent

label

2026-01-30 16:10:58 +00:00

pennae removed this from the 2.95 milestone

2026-02-09 14:21:43 +00:00

lix-bot commented

2026-03-16 14:01:54 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

comment in cl/5380 ("Kill build hook with SIGTERM instead of SIGKILL")

This issue was mentioned on Gerrit on the following CLs: * comment in [cl/5380](https://gerrit.lix.systems/c/lix/+/5380) ("Kill build hook with SIGTERM instead of SIGKILL")

lisanna-dettwyler commented

2026-03-16 14:10:28 +00:00

I do want to note that custom store url schemes are not really sufficient to replace build hooks, because that doesn't end up copying the result back to the local store, resulting in a state tracking problem where the custom store needs to somehow keep track of which store paths are on which remote machine in between your nix build --store custom:// and nix copy --from custom:// invocations. Acting like a NIX_REMOTE is better for this, which will get easier to do after the Nix GSoC project to make the daemon connection handler pluggable by libnix-store users (but it's still not a great replacement).

I do want to note that custom store url schemes are not really sufficient to replace build hooks, because that doesn't end up copying the result back to the local store, resulting in a state tracking problem where the custom store needs to somehow keep track of which store paths are on which remote machine in between your `nix build --store custom://` and `nix copy --from custom://` invocations. Acting like a `NIX_REMOTE` is better for this, which will get easier to do after the Nix GSoC project to make the daemon connection handler pluggable by libnix-store users (but it's still not a great replacement).

pennae commented

2026-03-16 14:17:32 +00:00

Author

Owner

when we remove the hook it will be the daemon's job to copy outputs from builder stores to the local store once a build has completed as part of finalization of a remote build (which currently just amounts to "wait to the build hook to exit"). our goal is to have the daemon open rpc connections to other daemons directly; in that model if you want something that behaves like a custom build hook does today you'd set up a local proxy that does all your custom processing and forwards to other stores as needed (including the final copy process). rather than make the connection handler pluggable we'll make the daemon itself trivial to proxy.

when we remove the hook it will be the daemon's job to copy outputs from builder stores to the local store once a build has completed as part of finalization of a remote build (which currently just amounts to "wait to the build hook to exit"). our goal is to have the daemon open rpc connections to other daemons *directly*; in that model if you want something that behaves like a custom build hook does today you'd set up a local proxy that does all your custom processing and forwards to other stores as needed (including the final copy process). rather than make the connection handler pluggable we'll make the daemon itself trivial to proxy.

lisanna-dettwyler commented

2026-03-16 14:23:02 +00:00

Is there an issue (aside from this one) where I can track the daemon proxying design / implementation?

pennae commented

2026-03-16 14:28:03 +00:00

Author

Owner

not yet, we're still preparing to actually start the work on this. initial cls should start showing up in the coming weeks, from there it'll likely be a lot of experimentation under new xp features until we find something that works well and isn't likely to bite us in the tail any time soon