Support P2P binary cache

Ghost commented

2024-05-09 21:54:20 +00:00

Sometimes the binary cache is down, slow, or otherwise hampered. This makes downloads either slow or impossible.

Describe the solution you'd like

Support a substituter that returns P2P links to binary content e.g IPFS addresses, magnet links, etc.

Describe alternatives you've considered

Host my own binary cache? But I don't know how, and others won't be able to take advantage of it anyway due to the centralised nature.

Additional context

Some people started and stopped implementing it on nix but never finished it (maybe Mic92 or K900?). There might be 2 implementation routes allowing protocol handlers to be configured in nix.conf (lix.conf?)

internal protocol handler (e.g use libtorrent to download torrents, or use IPFS API to download stuff)
call out to external binary with a certain protocol (CLI, unixsocket, ...) that takes over downloading the .nar (or whatever) to a certain location which is then copied into the store

The later might carry a security and allow putting arbitrary stuff into the store, but you know the ins and outs of this better, so you might already have a mitigation in mind.

## Is your feature request related to a problem? Please describe. Sometimes the binary cache is down, slow, or otherwise hampered. This makes downloads either slow or impossible. ## Describe the solution you'd like Support a substituter that returns P2P links to binary content e.g IPFS addresses, magnet links, etc. ## Describe alternatives you've considered Host my own binary cache? But I don't know how, and others won't be able to take advantage of it anyway due to the centralised nature. ## Additional context Some people started and stopped implementing it on `nix` but never finished it (maybe Mic92 or K900?). There might be 2 implementation routes allowing protocol handlers to be configured in `nix.conf` (`lix.conf?`) - internal protocol handler (e.g use `libtorrent` to download torrents, or use IPFS API to download stuff) - call out to external binary with a certain protocol (CLI, unixsocket, ...) that takes over downloading the `.nar` (or whatever) to a certain location which is then copied into the store The later might carry a security and allow putting arbitrary stuff into the store, but you know the ins and outs of this better, so you might already have a mitigation in mind.

pennae commented

2024-05-09 22:21:55 +00:00

Owner

that would be really nice to have, but realistically we won't be able to add new substituters any time soon with the current state of the code. there's a stopgap available for now though: local http proxies that turn the substituter protocol nix understands into literally anything else. (in fact one of the things we::horrors would like to do is to do that even for the currently built-in substituters, such that only the nix store protocol remains in the core and substituters can be treated as simple remote stores that can build anything, but don't have any actual build slots)

that would be really nice to have, but realistically we won't be able to add new substituters any time soon with the current state of the code. there's a stopgap available for now though: local http proxies that turn the substituter protocol nix understands into literally anything else. (in fact one of the things we::horrors would like to do is to do that even for the currently built-in substituters, such that only the nix store protocol remains in the core and substituters can be treated as simple remote stores that can build anything, but don't have any actual *build* slots)

Ghost commented

2024-05-09 22:33:27 +00:00

Author

Is there some kind documentation for the substituter protocol? Is it possible to point lix to a HTTP address where the .nar can be retrieved? As a stopgap for IPFS, for example, the proxy / local substituter could return a localhost:8888/ipfs/{cid}[/{path}][?{params}] URL (as described here) that will trigger IPFS download the requested file.

I call it stopgap because IPFS would download the file to its own store, then provide it to lix that'll in turn put that into its store, thus doubling the space. A solution for the future could be worked on e.g linking or hard-linking to the IPFS store, but that's for later down the road.

substituters can be treated as simple remote stores that can build anything, but don't have any actual build slots

Could you explain this a little more? I don't understand what you mean by "build anything" and not having any actual build slots. I thought builders took a derivation, built it, and stored the output into the cache, then communicated with substituters to update the mapping of derivation hash --> output hash / download location.

Is there some kind documentation for the substituter protocol? Is it possible to point `lix` to a HTTP address where the `.nar` can be retrieved? As a stopgap for IPFS, for example, the proxy / local substituter could return a `localhost:8888/ipfs/{cid}[/{path}][?{params}]` URL ([as described here](https://docs.ipfs.tech/reference/http/gateway/#api)) that will trigger IPFS download the requested file. I call it stopgap because IPFS would download the file to its own store, then provide it to `lix` that'll in turn put that into its store, thus doubling the space. A solution for the future could be worked on e.g linking or hard-linking to the IPFS store, but that's for later down the road. > substituters can be treated as simple remote stores that can build anything, but don't have any actual build slots Could you explain this a little more? I don't understand what you mean by "build anything" and not having any actual build slots. I thought builders took a derivation, built it, and stored the output into the cache, then communicated with substituters to update the mapping of derivation hash --> output hash / download location.

jade commented

2024-05-09 22:37:56 +00:00

Owner

yeah ^^ the issue with the idea of having an IPFS/etc substituter is that currently architecturally this use case would have to be in the core, which is not a pleasant place to write code in the first place.

A local HTTP server implementing the Nix store protocol should make your job much easier in implementing this idea as a prototype without any changes to Lix, and it will let you do it today so you can evaluate how well it works in practice. Also, I don't think that the way that torrents work is at all conducive to nix substitution as a use case, which is currently architecturally very horrible for a protocol that often takes several seconds to start a download, see #19.

The earliest that this feature could possibly be added is probably about a year, and that is highly optimistic on our pace of code improvement. I am marking this as WONTFIX, but it can and perhaps should be re-evaluated in the changed conditions in a year or so.

yeah ^^ the issue with the idea of having an IPFS/etc substituter is that currently architecturally this use case would have to be in the core, which is not a pleasant place to write code in the first place. A local HTTP server implementing the Nix store protocol should make your job much easier in implementing this idea as a prototype without any changes to Lix, and it will let you do it today so you can evaluate how well it works in practice. Also, I don't think that the way that torrents work is at all conducive to nix substitution as a use case, which is currently architecturally very horrible for a protocol that often takes several seconds to start a download, see https://git.lix.systems/lix-project/lix/issues/19. The earliest that this feature could possibly be added is probably about a year, and that is highly optimistic on our pace of code improvement. I am marking this as WONTFIX, but it can and perhaps should be re-evaluated in the changed conditions in a year or so.

jade closed this issue

2024-05-09 22:37:56 +00:00

jade added the

Status

wontfix

Area/store

labels 2024-05-09 22:38:09 +00:00

raito commented

2024-05-09 22:40:49 +00:00

Owner

@loveisgrief Note that https://github.com/tvlfyi/tvix/tree/canon/nar-bridge is an interesting example of such a substituter that you could rework to plug a IPFS/etc. way of doing the bidding. It's used as compatibility glue with a "Tvix store" which has a different model than a classical Nix store.

Ghost commented

2024-05-10 07:49:05 +00:00

Author

Thanks both. I wish the substituter protocol were documented, but maybe I'll have to take care of that on nixlang.wiki or something.

Thanks again for the quick responses.

Thanks both. I wish the substituter protocol were documented, but maybe I'll have to take care of that on nixlang.wiki or something. Thanks again for the quick responses.

qyriad added the

Status

postponed

label 2024-05-10 18:23:39 +00:00

jade commented

2024-05-10 21:21:15 +00:00

Owner

Thanks both. I wish the substituter protocol were documented, but maybe I'll have to take care of that on nixlang.wiki or something.

If you want to write up documentation for it and send it to us as a patch, we would take it. There's a doc/manual/src/protocols/ directory in the docs in Lix where it would reasonably fit.

> Thanks both. I wish the substituter protocol were documented, but maybe I'll have to take care of that on nixlang.wiki or something. If you want to write up documentation for it and send it to us as a patch, we would take it. There's a `doc/manual/src/protocols/` directory in the docs in Lix where it would reasonably fit.

Support P2P binary cache #289

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context