Quickly add a non-leaf path to the nix store

lolbinarycat commented

2024-10-21 22:00:42 +00:00

I am the developer of nix-inst. one of the primary goals of the project is performance, however, there is a major bottleneck:

the only way to create a store object that contains references to other store objects is to build a derivation.

simply using nix-store --add with a directory containing symlinks to the nix store will not work correctly with garbage collection.

Describe the solution you'd like

A nix-store flag that makes --add recurse into the directory and look for symlinks to store paths

Describe alternatives you've considered

change the default behavior of nix-store, add a new subcommand, or gate this feature behind the new cli.

Additional context

this would allow me to nearly double the performance of nix-inst on lix systems.

## Is your feature request related to a problem? Please describe. I am the developer of [nix-inst](https://codeberg.org/binarycat/nix-inst/). one of the primary goals of the project is performance, however, there is a major bottleneck: the only way to create a store object that contains references to other store objects is to build a derivation. simply using `nix-store --add` with a directory containing symlinks to the nix store will not work correctly with garbage collection. ## Describe the solution you'd like A `nix-store` flag that makes `--add` recurse into the directory and look for symlinks to store paths ## Describe alternatives you've considered change the default behavior of `nix-store`, add a new subcommand, or gate this feature behind the new cli. ## Additional context this would allow me to nearly double the performance of nix-inst on lix systems.

pennae commented

2024-10-21 22:22:59 +00:00

Owner

you are looking for nix-store --import. you will have to create the format it expects manually because we have no writers for it, and considering that the thing we think you are asking for (scanning any directory for references and adding those as dependencies) is fundamentally unsound that's about as good as it gets.

you are looking for `nix-store --import`. you will have to create the format it expects manually because we have no writers for it, and considering that the thing we think you are asking for (scanning any directory for references and adding those as dependencies) is fundamentally unsound that's about as good as it gets.

lolbinarycat commented

2024-10-25 15:42:16 +00:00

Author

the thing we think you are asking for (scanning any directory for references and adding those as dependencies) is fundamentally unsound

how so? do you just mean in terms of garbage collection, or...?

> the thing we think you are asking for (scanning any directory for references and adding those as dependencies) is fundamentally unsound how so? do you just mean in terms of garbage collection, or...?

pennae commented

2024-10-25 16:06:02 +00:00

Owner

how so? do you just mean in terms of garbage collection, or...?

without a bound on which references to scan for the scan result depends on the current state of the store, which is entirely against the stated goal of making store objects reproducible.

> how so? do you just mean in terms of garbage collection, or...? without a bound on which references to scan for the scan result depends on the current state of the store, which is entirely against the stated goal of making store objects reproducible.

lolbinarycat commented

2024-10-25 17:53:42 +00:00

Author

without a bound on which references to scan for the scan result depends on the current state of the store, which is entirely against the stated goal of making store objects reproducible.

do you think i mean just any file containing a store path? i'm specifically referring to absolute symlinks to /nix/store.

just scan all symlinks (without following symlinks!), and error if any of them point to a non-existent store path.

if an "upper bound" is needed, i can provide that, i mean, i can provide a full list of references too, even.

nix-store --import requires building an intermediate NAR archive, which is an undesirable amount of I/O.

> without a bound on which references to scan for the scan result depends on the current state of the store, which is entirely against the stated goal of making store objects reproducible. do you think i mean just any file containing a store path? i'm specifically referring to absolute symlinks to /nix/store. just scan all symlinks (without following symlinks!), and error if any of them point to a non-existent store path. if an "upper bound" is needed, i can provide that, i mean, i can provide a full list of references too, even. `nix-store --import` requires building an intermediate NAR archive, which is an undesirable amount of I/O.

pennae commented

2024-10-25 18:06:57 +00:00

Owner

just scan all symlinks (without following symlinks!), and error if any of them point to a non-existent store path.

that changes absolutely nothing about the soundness issue. no restriction on the path set makes the entire thing not reproducible, which is not something we should add as a feature.

if an "upper bound" is needed, i can provide that, i mean, i can provide a full list of references too, even.

great! format that as a dump archive and pipe it into nix-store --import. that archive doesn't need to hit the disk, and whether you do the reading and scanning or lix does is immaterial.

> just scan all symlinks (without following symlinks!), and error if any of them point to a non-existent store path. that changes absolutely nothing about the soundness issue. no restriction on the path set makes the entire thing not reproducible, which is not something we should add as a feature. > if an "upper bound" is needed, i can provide that, i mean, i can provide a full list of references too, even. great! format that as a dump archive and pipe it into `nix-store --import`. that archive doesn't need to hit the disk, and whether you do the reading and scanning or lix does is immaterial.

lolbinarycat commented

2024-10-25 18:32:50 +00:00

Author

ok, anywhere i can actually find a description of the format used by nix-store --import? i tried reading the relevant code but C++ code is really hard to grep through due to lack of a fn keyword.

ok, anywhere i can actually find a description of the format used by `nix-store --import`? i tried reading the relevant code but C++ code is really hard to grep through due to lack of a fn keyword.

pennae commented

2024-10-25 18:43:49 +00:00

Owner

unfortunately we do not have good documentation on the formats for neither NARs nor the dump archive format of nix-store --import. at this point your best bet is to read archive.cc and (i'm guessing) import-export.cc for the relevant functions.

unfortunately we do not have good documentation on the formats for neither NARs nor the dump archive format of `nix-store --import`. at this point your best bet is to read `archive.cc` and (i'm guessing) `import-export.cc` for the relevant functions.

jade commented

2024-11-10 02:48:36 +00:00

Owner

if this information isn't arriving too late to be useful to you, we have an implementation of a writer for that format as evil_nar.py or a similarly named Python file in the lix repo, along with a NAR implementation.

jade commented

2024-11-10 02:50:20 +00:00

Owner

but yes, i think this should be wontfix because we can generate the desired thing very precisely today using an implementation of the import format. reference scanning assumes that we already know the subset of store paths that might appear and thus the idea as proposed cannot be implemented.

but yes, i think this should be wontfix because we can generate the desired thing very precisely today using an implementation of the import format. reference scanning assumes that we already know the subset of store paths that *might* appear and thus the idea as proposed cannot be implemented.

jade closed this issue

2024-11-10 02:50:21 +00:00

jade added the

Status

wontfix

label 2024-11-10 02:50:36 +00:00

Quickly add a non-leaf path to the nix store #555

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context