Lix currently does a lot of NAR processing for binary caches that it should delegate #578

New issue

Open

opened 2024-11-19 08:49:48 +00:00 by jade · 0 comments

jade commented

2024-11-19 08:49:48 +00:00

Owner

In a similar way to the serve protocol becoming legacy cruft because something else would do the job better, this functionality probably should be moved outside of Lix core.

Arguably, even, fancy functionality for s3 caches should really be in an external project that does a better job of focusing on them.

That being said, there are a few barriers that I think have stopped this from happening already:

Lack of awareness of the necessary Nix/Lix internals to implement it
Disconnection between those deeply involved in the store layer and those who want more things from it
Poor extension points in Lix/Nix to implement services using the store: it is not really obvious how to implement efficiently without rewriting about half of nix copy and probably doing a lot of shelling out; though the latter probably is less bad than one might imagine. Nevertheless it is not great to not be able to reuse the graph visiting algorithms inside the Lix implementation and so on.

@pennae wanted to delete the debuginfo mapping functionality, but I suspect it is in active usage and is probably currently mandatory for Hydra (h-n-o scale Hydra, at least), in spite of really being more reasonable as a post-processing step or so implemented on the binary cache side or as a Fancy S3 Client for Making Good Binary Caches.

Lix contains things from a .ls file generator (fine, I suppose), to a debuginfo map generator for dwarffs (pretty serious layering violation), which were implemented in Nix since it was the most expedient place and probably nobody was there to stop it at the time, but it really should be part of the *binary cache* implementation since any serious binary cache needs more than what Lix provides. These also work for directory binary caches, even! In a similar way to the serve protocol becoming legacy cruft because something else would do the job better, this functionality probably should be moved outside of Lix core. Arguably, even, fancy functionality for s3 caches should really be in an external project that does a better job of focusing on them. That being said, there are a few barriers that I think have stopped this from happening already: - Lack of awareness of the necessary Nix/Lix internals to implement it - Disconnection between those deeply involved in the store layer and those who want more things from it - Poor extension points in Lix/Nix to implement services using the store: it is not really obvious how to implement efficiently without rewriting about half of `nix copy` and probably doing a lot of shelling out; though the latter probably is less bad than one might imagine. Nevertheless it is not great to not be able to reuse the graph visiting algorithms inside the Lix implementation and so on. @pennae wanted to delete the debuginfo mapping functionality, but I suspect it is in active usage and is probably currently mandatory for Hydra (h-n-o scale Hydra, at least), in spite of really being more reasonable as a post-processing step or so implemented on the binary cache side or as a Fancy S3 Client for Making Good Binary Caches.