moraxyc/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
lix/src/libstore/remote-store.hh

218 lines
6.4 KiB
C++
Raw Normal View History

Use "#pragma once" to prevent repeated header file inclusion
2012-07-18 18:59:03 +00:00
#pragma once
Ensure all headers have `#pragma once` and are in API docs `///@file` makes them show up in the internal API dos. A tiny few were missing `#pragma once`.
2023-04-01 03:18:41 +00:00
///@file
* Oops.
2006-11-30 18:35:50 +00:00
Pool<T>: Allow a maximum pool size
2016-02-23 15:40:16 +00:00
#include <limits>
* Oops.
2006-11-30 18:35:50 +00:00
#include <string>
#include "store-api.hh"
Factor out a `GcStore` interface Starts progress on #5729. The idea is that we should not have these default methods throwing "unimplemented". This is a small step in that direction. I kept `addTempRoot` because it is a no-op, rather than failure. Also, as a practical matter, it is called all over the place, while doing other tasks, so the downcasting would be annoying. Maybe in the future I could move the "real" `addTempRoot` to `GcStore`, and the existing usecases use a `tryAddTempRoot` wrapper to downcast or do nothing, but I wasn't sure whether that was a good idea so with a bias to less churn I didn't do it yet.
2022-03-01 18:31:36 +00:00
#include "gc-store.hh"
Factor out a `LogStore` interface Continue progress on #5729. Just as I hoped, this uncovered an issue: the daemon protocol is missing a way to query build logs. This doesn't effect `unix://`, but does effect `ssh://`. A FIXME is left for this, so we come back to it later.
2022-03-08 18:20:39 +00:00
#include "log-store.hh"
* Oops.
2006-11-30 18:35:50 +00:00
namespace nix {
* When NIX_REMOTE is set to "slave", fork off nix-worker in slave mode. Presumably nix-worker would be setuid to the Nix store user. The worker performs all operations on the Nix store and database, so the caller can be completely unprivileged. This is already much more secure than the old setuid scheme, since the worker doesn't need to do Nix expression evaluation and so on. Most importantly, this means that it doesn't need to access any user files, with all resulting security risks; it only performs pure store operations. Once this works, it is easy to move to a daemon model that forks off a worker for connections established through a Unix domain socket. That would be even more secure.
2006-11-30 19:54:43 +00:00
class Pipe;
class Pid;
struct FdSink;
struct FdSource;
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
template<typename T> class Pool;
RemoteStore: Close connection if an exception occurs Fixes #2075.
2018-10-16 21:36:15 +00:00
struct ConnectionHandle;
* When NIX_REMOTE is set to "slave", fork off nix-worker in slave mode. Presumably nix-worker would be setuid to the Nix store user. The worker performs all operations on the Nix store and database, so the caller can be completely unprivileged. This is already much more secure than the old setuid scheme, since the worker doesn't need to do Nix expression evaluation and so on. Most importantly, this means that it doesn't need to access any user files, with all resulting security risks; it only performs pure store operations. Once this works, it is easy to move to a daemon model that forks off a worker for connections established through a Unix domain socket. That would be even more secure.
2006-11-30 19:54:43 +00:00
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
struct RemoteStoreConfig : virtual StoreConfig
* Oops.
2006-11-30 18:35:50 +00:00
{
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
using StoreConfig::StoreConfig;
* Oops.
2006-11-30 18:35:50 +00:00
Improve store setting descriptions / Markdown formatting
2023-03-22 13:23:36 +00:00
const Setting<int> maxConnections{(StoreConfig*) this, 1, "max-connections",
"Maximum number of concurrent connections to the Nix daemon."};
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
Improve store setting descriptions / Markdown formatting
2023-03-22 13:23:36 +00:00
const Setting<unsigned int> maxConnectionAge{(StoreConfig*) this,
std::numeric_limits<unsigned int>::max(),
"max-connection-age",
"Maximum age of a connection before it is closed."};
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
};
Finish converting existing comments for internal API docs (#8146) * Finish converting existing comments for internal API docs 99% of this was just reformatting existing comments. Only two exceptions: - Expanded upon `BuildResult::status` compat note - Split up file-level `symbol-table.hh` doc comments to get per-definition docs Also fixed a few whitespace goofs, turning leading tabs to spaces and removing trailing spaces. Picking up from #8133 * Fix two things from comments * Use triple-backtick not indent for `dumpPath` * Convert GNU-style `\`..'` quotes to markdown style in API docs This will render correctly.
2023-04-07 13:55:28 +00:00
/**
* \todo RemoteStore is a misnomer - should be something like
* DaemonStore.
*/
Factor out a `LogStore` interface Continue progress on #5729. Just as I hoped, this uncovered an issue: the daemon protocol is missing a way to query build logs. This doesn't effect `unix://`, but does effect `ssh://`. A FIXME is left for this, so we come back to it later.
2022-03-08 18:20:39 +00:00
class RemoteStore : public virtual RemoteStoreConfig,
public virtual Store,
public virtual GcStore,
public virtual LogStore
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
{
public:
RemoteStore: Add option to drop old connections from the pool This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14 16:10:38 +00:00
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
RemoteStore(const Params & params);
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
* Oops.
2006-11-30 18:35:50 +00:00
/* Implementations of abstract store API methods. */
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
bool isValidPathUncached(const StorePath & path) override;
* Oops.
2006-11-30 18:35:50 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryValidPaths(const StorePathSet & paths,
Replace a few bool flags with enums Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-06-28 16:11:01 +00:00
SubstituteFlag maybeSubstitute = NoSubstitute) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryAllValidPaths() override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void queryPathInfoUncached(const StorePath & path,
queryPathInfoUncached(): Return const ValidPathInfo
2018-09-25 16:54:16 +00:00
Callback<std::shared_ptr<const ValidPathInfo>> callback) noexcept override;
* Oops.
2006-11-30 18:35:50 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void queryReferrers(const StorePath & path, StorePathSet & referrers) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryValidDerivers(const StorePath & path) override;
nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settings So if a path is not garbage solely because it's reachable from a root due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store -q --roots’ now shows that root.
2012-12-20 17:41:44 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryDerivationOutputs(const StorePath & path) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Rename drv output querying functions - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
std::map<std::string, std::optional<StorePath>> queryPartialDerivationOutputMap(const StorePath & path) override;
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
std::optional<StorePath> queryPathFromHashPart(const std::string & hashPart) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet querySubstitutablePaths(const StorePathSet & paths) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Add StorePathCAMap for querySubstitutablePathInfos I’m not 100% sure this is wanted since it kind of makes everything have to know about ca even if they don’t really want to. But it also make things easier in dealing with looking up ca.
2020-06-17 19:03:05 +00:00
void querySubstitutablePathInfos(const StorePathCAMap & paths,
SubstitutablePathInfos & infos) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Finish converting existing comments for internal API docs (#8146) * Finish converting existing comments for internal API docs 99% of this was just reformatting existing comments. Only two exceptions: - Expanded upon `BuildResult::status` compat note - Split up file-level `symbol-table.hh` doc comments to get per-definition docs Also fixed a few whitespace goofs, turning leading tabs to spaces and removing trailing spaces. Picking up from #8133 * Fix two things from comments * Use triple-backtick not indent for `dumpPath` * Convert GNU-style `\`..'` quotes to markdown style in API docs This will render correctly.
2023-04-07 13:55:28 +00:00
/**
* Add a content-addressable store path. `dump` will be drained.
*/
Style fixes
2020-09-22 09:40:19 +00:00
ref<const ValidPathInfo> addCAToStore(
Source & dump,
addToStoreFromDump(): Take std::string_view
2022-02-15 14:08:06 +00:00
std::string_view name,
Style fixes
2020-09-22 09:40:19 +00:00
ContentAddressMethod caMethod,
Derivations can output "text-hashed" data In particular, this means that derivations can output derivations. But that ramification isn't (yet!) useful as we would want, since there is no way to have a dependent derivation that is itself a dependent derivation.
2020-10-12 23:51:23 +00:00
HashType hashType,
Style fixes
2020-09-22 09:40:19 +00:00
const StorePathSet & references,
RepairFlag repair);
Refactor wopAddToStore to make wopAddTextToStore obsolete
2020-09-17 18:19:15 +00:00
Finish converting existing comments for internal API docs (#8146) * Finish converting existing comments for internal API docs 99% of this was just reformatting existing comments. Only two exceptions: - Expanded upon `BuildResult::status` compat note - Split up file-level `symbol-table.hh` doc comments to get per-definition docs Also fixed a few whitespace goofs, turning leading tabs to spaces and removing trailing spaces. Picking up from #8133 * Fix two things from comments * Use triple-backtick not indent for `dumpPath` * Convert GNU-style `\`..'` quotes to markdown style in API docs This will render correctly.
2023-04-07 13:55:28 +00:00
/**
* Add a content-addressable store path. Does not support references. `dump` will be drained.
*/
addToStoreFromDump(): Take std::string_view
2022-02-15 14:08:06 +00:00
StorePath addToStoreFromDump(Source & dump, std::string_view name,
addToStore, addToStoreFromDump: refactor: pass refs by const reference Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 09:24:49 +00:00
FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair, const StorePathSet & references = StorePathSet()) override;
Overhaul wopAddToStore
2020-09-17 17:27:11 +00:00
Make 'nix copy --to daemon' run in constant memory
2018-03-21 22:42:21 +00:00
void addToStore(const ValidPathInfo & info, Source & nar,
Remove 'accessor' from addToStore() This is only used by hydra-queue-runner and it's better to implement it there.
2020-07-13 15:37:44 +00:00
RepairFlag repair, CheckSigsFlag checkSigs) override;
Add a Store::addToStore() variant that accepts a NAR As a side effect, this ensures that signatures are propagated when copying paths between stores. Also refactored import/export to make use of this.
2016-05-04 11:36:54 +00:00
Low-latency closure copy This adds a new store operation 'addMultipleToStore' that reads a number of NARs and ValidPathInfos from a Source, allowing any number of store paths to be copied in a single call. This is much faster on high-latency links when copying a lot of small files, like .drv closures. For example, on a connection with an 50 ms delay: Before: $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \ /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \ --derivation --no-check-sigs real 0m57.868s user 0m0.103s sys 0m0.056s After: real 0m0.690s user 0m0.017s sys 0m0.011s
2021-07-26 11:31:09 +00:00
void addMultipleToStore(
Source & source,
RepairFlag repair,
CheckSigsFlag checkSigs) override;
Restore the "low-latency" ssh copying
2022-06-08 12:03:46 +00:00
void addMultipleToStore(
Indentation
2022-08-22 13:30:38 +00:00
PathsSource & pathsToCopy,
Activity & act,
RepairFlag repair,
CheckSigsFlag checkSigs) override;
Restore the "low-latency" ssh copying
2022-06-08 12:03:46 +00:00
Remove std::string alias (for real this time) Also use std::string_view in a few more places.
2022-02-25 15:00:00 +00:00
StorePath addTextToStore(
std::string_view name,
std::string_view s,
const StorePathSet & references,
RepairFlag repair) override;
* Oops.
2006-11-30 18:35:50 +00:00
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 15:36:51 +00:00
void registerDrvOutput(const Realisation & info) override;
Expose an async interface for `queryRealisation` Doesn’t change much so far because everything is still using it synchronously, but should allow the binary cache to fetch stuff in parallel
2021-10-27 09:36:51 +00:00
void queryRealisationUncached(const DrvOutput &,
Callback<std::shared_ptr<const Realisation>> callback) noexcept override;
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 15:36:51 +00:00
buildPaths(): Add an evalStore argument With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue #5025.
2021-07-19 13:43:08 +00:00
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override;
* Oops.
2006-11-30 18:35:50 +00:00
Make `KeyedBuildResult`, `BuildResult` like before, and fix bug another way In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I realized since derivation goals' wanted outputs can "grow" due to overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called by `Worker::makeDerivationGoalCommon`), the previous bug fix had an unfortunate side effect of causing more pointless rebuilds. In paticular, we have this situation: 1. Goal made from `DerivedPath::Built { foo, {a} }`. 2. Goal gives on on substituting, starts building. 3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just modified original goal. 4. Though the goal had gotten as far as building, so all outputs were going to be produced, `addWantedOutputs` no longer knows that and so the goal is flagged to be restarted. This might sound far-fetched with input-addressed drvs, where we usually basically have all our goals "planned out" before we start doing anything, but with CA derivation goals and especially RFC 92, where *drv resolution* means goals are created after some building is completed, it is more likely to happen. So the first thing to do was restore the clearing of `wantedOutputs` we used to do, and then filter the outputs in `buildPathsWithResults` to only get the ones we care about. But fix also has its own side effect in that the `DerivedPath` in the `BuildResult` in `DerivationGoal` cannot be trusted; it is merely the *first* `DerivedPath` for which this goal was originally created. To remedy this, I made `BuildResult` be like it was before, and instead made `KeyedBuildResult` be a subclass wit the path. Only `buildPathsWithResults` returns `KeyedBuildResult`s, everything else just becomes like it was before, where the "key" is unambiguous from context. I think separating the "primary key" field(s) from the other fields is good practical in general anyways. (I would like to do the same thing for `ValidPathInfo`.) Among other things, it allows constructions like `std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and just precludes the possibility of those duplicate keys being out of sync. We might leverage the above someday to overload `buildPathsWithResults` to take a *set* of return a *map* per the above. ----- Unfortunately, we need to avoid C++20 strictness on designated initializers. (BTW https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html this offers some new syntax for this use-case. Hopefully this will be adopted and we can eventually use it.) No having that yet, maybe it would be better to not make `KeyedBuildResult` a subclass to just avoid this. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2022-03-25 01:26:07 +00:00
std::vector<KeyedBuildResult> buildPathsWithResults(
Add Store::buildPathsWithResults() This function is like buildPaths(), except that it returns a vector of BuildResults containing the exact statuses and output paths of each derivation / substitution. This is convenient for functions like Installable::build(), because they then don't need to do another series of calls to get the outputs of CA derivations. It's also a precondition to impure derivations, where we *can't* query the output of those derivations since they're not stored in the Nix database. Note that PathSubstitutionGoal can now also return a BuildStatus.
2022-03-08 18:50:46 +00:00
const std::vector<DerivedPath> & paths,
BuildMode buildMode,
std::shared_ptr<Store> evalStore) override;
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 15:57:40 +00:00
BuildMode buildMode) override;
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void ensurePath(const StorePath & path) override;
* Oops.
2006-11-30 18:35:50 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void addTempRoot(const StorePath & path) override;
* Move addTempRoot() to the store API, and add another function syncWithGC() to allow clients to register GC roots without needing write access to the global roots directory or the GC lock.
2006-12-02 16:41:36 +00:00
Shut up clang warnings
2015-09-17 23:22:06 +00:00
void addIndirectRoot(const Path & path) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
findRoots(): Add 'censor' parameter This is less brittle than filtering paths after the fact in nix-daemon.
2019-03-14 12:50:07 +00:00
Roots findRoots(bool censor) override;
* Allow unprivileged users to run the garbage collector and to do `nix-store --delete'. But unprivileged users are not allowed to ignore liveness. * `nix-store --delete --ignore-liveness': ignore the runtime roots as well.
2006-12-05 02:18:46 +00:00
Shut up clang warnings
2015-09-17 23:22:06 +00:00
void collectGarbage(const GCOptions & options, GCResults & results) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Shut up clang warnings
2015-09-17 23:22:06 +00:00
void optimiseStore() override;
Replace a few bool flags with enums Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-06-28 16:11:01 +00:00
bool verifyStore(bool checkContents, RepairFlag repair) override;
Add an 'optimiseStore' remote procedure call.
2014-09-01 20:21:42 +00:00
Support `repairPath` on most stores. More progress on issue #5729 The method trivially generalizes to be store-implementation-agnostic, in fact. However, we force it to continue to be unimplemented with `RemoteStore` and `LegacySSHStore` because the implementation we'd get via the generalization is probably not the one users expect. This keeps our hands untied to do it right going forward. For more about the tension between the scheduler logic being store-type-agnostic and remote stores doing their own scheduling, see issues #5025 and #5056.
2023-04-13 14:38:35 +00:00
/**
* The default instance would schedule the work on the client side, but
* for consistency with `buildPaths` and `buildDerivation` it should happen
* on the remote side.
*
* We make this fail for now so we can add implement this properly later
* without it being a breaking change.
*/
void repairPath(const StorePath & path) override
{ unsupported("repairPath"); }
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void addSignatures(const StorePath & storePath, const StringSet & sigs) override;
Add "nix copy-sigs" command This imports signatures from one store into another. E.g. $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/ imported 595 signatures
2016-04-05 13:30:22 +00:00
Rename Buildable
2021-04-05 13:48:18 +00:00
void queryMissing(const std::vector<DerivedPath> & targets,
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet & willBuild, StorePathSet & willSubstitute, StorePathSet & unknown,
unsigned long long -> uint64_t
2020-07-30 11:10:49 +00:00
uint64_t & downloadSize, uint64_t & narSize) override;
Implement RemoteStore::queryMissing() This provides a significant speedup, e.g. 64 s -> 12 s for nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test on a cold local and CloudFront cache. The alternative is to use lots of concurrent daemon connections but that seems wasteful.
2017-04-06 16:40:19 +00:00
Add command 'nix store copy-log' Fixes #5222.
2022-01-17 18:45:21 +00:00
void addBuildLog(const StorePath & drvPath, std::string_view log) override;
nix store ping: Report Nix daemon version Fixes #5952.
2022-01-25 20:14:27 +00:00
std::optional<std::string> getVersion() override;
build-remote: Fix fallback to other machines when connecting fails Opening an SSHStore or LegacySSHStore does not actually establish a connection, so the try/catch block here did nothing. Added a Store::connect() method to test whether a connection can be established.
2017-05-02 12:18:46 +00:00
void connect() override;
Store: expose the protocol version used by a store
2018-08-30 21:28:47 +00:00
unsigned int getProtocol() override;
Add `Store::isTrustedClient()` This function returns true or false depending on whether the Nix client is trusted or not. Mostly relevant when speaking to a remote store with a daemon. We include this information in `nix ping store` and `nix doctor` Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2022-12-26 20:21:08 +00:00
std::optional<TrustedFlag> isTrustedClient() override;
RemoteStore: Add option to drop old connections from the pool This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14 16:10:38 +00:00
void flushBadConnections();
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
struct Connection
{
FdSink to;
FdSource from;
unsigned int daemonVersion;
Add `Store::isTrustedClient()` This function returns true or false depending on whether the Nix client is trusted or not. Mostly relevant when speaking to a remote store with a daemon. We include this information in `nix ping store` and `nix doctor` Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2022-12-26 20:21:08 +00:00
std::optional<TrustedFlag> remoteTrustsUs;
nix store ping: Report Nix daemon version Fixes #5952.
2022-01-25 20:14:27 +00:00
std::optional<std::string> daemonNixVersion;
RemoteStore: Add option to drop old connections from the pool This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14 16:10:38 +00:00
std::chrono::time_point<std::chrono::steady_clock> startTime;
* Some hackery to propagate the worker's stderr and exceptions to the client.
2006-12-03 02:08:13 +00:00
Add ssh store implementation
2016-09-02 18:31:38 +00:00
virtual ~Connection();
Pool<T>: Allow a maximum pool size
2016-02-23 15:40:16 +00:00
Shut down write side before draining the read side This is important if the remote side *does* execute nix-store/nix-daemon successfully, but stdout is polluted (e.g. because the remote user's bashrc script prints something to stdout). In that case we have to shutdown the write side to force the remote nix process to exit.
2021-09-23 16:01:04 +00:00
virtual void closeWrite() = 0;
RemoteStore::addToStore(): Fix race between stderrThread and NAR writer As pointed out by @B4dM4n, the call to to.flush() on stderrThread is unsafe because the NAR writer thread is also writing to 'to'. Fixes #3943.
2020-08-27 12:48:08 +00:00
std::exception_ptr processStderr(Sink * sink = 0, Source * source = 0, bool flush = true);
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
};
* Refactoring.
2006-12-04 13:28:14 +00:00
Cache connection failures
2017-03-03 18:35:34 +00:00
ref<Connection> openConnectionWrapper();
Use `RemoteStore` to open connection for proxying daemon Removes duplicate websocket opening code, and also means we should be able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
protected:
Factor out the unix domain socket-specific code from RemoteStore
2016-09-02 18:15:04 +00:00
virtual ref<Connection> openConnection() = 0;
void initConnection(Connection & conn);
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
ref<Pool<Connection>> connections;
* Pass various options to the worker so that flags like -K or -j work in multi-user Nix (NIX-72). * Client/worker: exchange a protocol version number for future compatibility.
2007-09-18 09:11:20 +00:00
ssh-ng: Don't forward options to the daemon. This can be iterated on and currently leaves out settings we know we want to forward, but it fixes #1713 and fixes #1935 and isn't fundamentally broken like the status quo. Future changes are suggested in a comment.
2018-03-05 12:42:15 +00:00
virtual void setOptions(Connection & conn);
Make the flake options work when using the daemon When setting flake-local options (with the `nixConfig` field), forward these options to the daemon in case we’re using one. This is necessary in particular for options like `binary-caches` or `post-build-hook` to make sense. Fix <https://github.com/NixOS/nix/commit/343239fc8a1993f707a990c2cd54a41f1fa3de99#r44356843>
2021-11-05 15:17:49 +00:00
void setOptions() override;
RemoteStore: Close connection if an exception occurs Fixes #2075.
2018-10-16 21:36:15 +00:00
ConnectionHandle getConnection();
remote-store.hh: ConnectionHandle is struct, minor fix warning
2018-10-26 17:34:29 +00:00
friend struct ConnectionHandle;
RemoteStore: Close connection if an exception occurs Fixes #2075.
2018-10-16 21:36:15 +00:00
gracefully handle old daemon versions Add a fallback path in `queryPartialDerivationOutputMap` for daemons that don't support it. Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this is needed to handle the fallback path.
2020-09-07 09:26:09 +00:00
virtual ref<FSAccessor> getFSAccessor() override;
virtual void narFromPath(const StorePath & path, Sink & sink) override;
Factor out the unix domain socket-specific code from RemoteStore
2016-09-02 18:15:04 +00:00
private:
Cache connection failures
2017-03-03 18:35:34 +00:00
std::atomic_bool failed{false};
Add Store::buildPathsWithResults() This function is like buildPaths(), except that it returns a vector of BuildResults containing the exact statuses and output paths of each derivation / substitution. This is convenient for functions like Installable::build(), because they then don't need to do another series of calls to get the outputs of CA derivations. It's also a precondition to impure derivations, where we *can't* query the output of those derivations since they're not stored in the Nix database. Note that PathSubstitutionGoal can now also return a BuildStatus.
2022-03-08 18:50:46 +00:00
void copyDrvsFromEvalStore(
const std::vector<DerivedPath> & paths,
std::shared_ptr<Store> evalStore);
Factor out the unix domain socket-specific code from RemoteStore
2016-09-02 18:15:04 +00:00
};
* Pass various options to the worker so that flags like -K or -j work in multi-user Nix (NIX-72). * Client/worker: exchange a protocol version number for future compatibility.
2007-09-18 09:11:20 +00:00
* Oops.
2006-11-30 18:35:50 +00:00
}
Copy permalink