lunaphied/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
lix/src/libstore/remote-store.hh

168 lines
4.9 KiB
C++
Raw Normal View History

Use "#pragma once" to prevent repeated header file inclusion
2012-07-18 18:59:03 +00:00
#pragma once
* Oops.
2006-11-30 18:35:50 +00:00
Pool<T>: Allow a maximum pool size
2016-02-23 15:40:16 +00:00
#include <limits>
* Oops.
2006-11-30 18:35:50 +00:00
#include <string>
#include "store-api.hh"
namespace nix {
* When NIX_REMOTE is set to "slave", fork off nix-worker in slave mode. Presumably nix-worker would be setuid to the Nix store user. The worker performs all operations on the Nix store and database, so the caller can be completely unprivileged. This is already much more secure than the old setuid scheme, since the worker doesn't need to do Nix expression evaluation and so on. Most importantly, this means that it doesn't need to access any user files, with all resulting security risks; it only performs pure store operations. Once this works, it is easy to move to a daemon model that forks off a worker for connections established through a Unix domain socket. That would be even more secure.
2006-11-30 19:54:43 +00:00
class Pipe;
class Pid;
struct FdSink;
struct FdSource;
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
template<typename T> class Pool;
RemoteStore: Close connection if an exception occurs Fixes #2075.
2018-10-16 21:36:15 +00:00
struct ConnectionHandle;
* When NIX_REMOTE is set to "slave", fork off nix-worker in slave mode. Presumably nix-worker would be setuid to the Nix store user. The worker performs all operations on the Nix store and database, so the caller can be completely unprivileged. This is already much more secure than the old setuid scheme, since the worker doesn't need to do Nix expression evaluation and so on. Most importantly, this means that it doesn't need to access any user files, with all resulting security risks; it only performs pure store operations. Once this works, it is easy to move to a daemon model that forks off a worker for connections established through a Unix domain socket. That would be even more secure.
2006-11-30 19:54:43 +00:00
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
struct RemoteStoreConfig : virtual StoreConfig
* Oops.
2006-11-30 18:35:50 +00:00
{
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
using StoreConfig::StoreConfig;
* Oops.
2006-11-30 18:35:50 +00:00
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
const Setting<int> maxConnections{(StoreConfig*) this, 1,
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
"max-connections", "maximum number of concurrent connections to the Nix daemon"};
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
const Setting<unsigned int> maxConnectionAge{(StoreConfig*) this, std::numeric_limits<unsigned int>::max(),
RemoteStore: Add option to drop old connections from the pool This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14 16:10:38 +00:00
"max-connection-age", "number of seconds to reuse a connection"};
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
};
/* FIXME: RemoteStore is a misnomer - should be something like
DaemonStore. */
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 15:33:12 +00:00
class RemoteStore : public virtual RemoteStoreConfig, public virtual Store
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
{
public:
RemoteStore: Add option to drop old connections from the pool This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14 16:10:38 +00:00
ssh-ng: Don't set CPU affinity on the remote Fixes #3138.
2019-10-11 16:48:59 +00:00
virtual bool sameMachine() = 0;
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
RemoteStore(const Params & params);
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
* Oops.
2006-11-30 18:35:50 +00:00
/* Implementations of abstract store API methods. */
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
bool isValidPathUncached(const StorePath & path) override;
* Oops.
2006-11-30 18:35:50 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryValidPaths(const StorePathSet & paths,
Replace a few bool flags with enums Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-06-28 16:11:01 +00:00
SubstituteFlag maybeSubstitute = NoSubstitute) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryAllValidPaths() override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void queryPathInfoUncached(const StorePath & path,
queryPathInfoUncached(): Return const ValidPathInfo
2018-09-25 16:54:16 +00:00
Callback<std::shared_ptr<const ValidPathInfo>> callback) noexcept override;
* Oops.
2006-11-30 18:35:50 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void queryReferrers(const StorePath & path, StorePathSet & referrers) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryValidDerivers(const StorePath & path) override;
nix-store -q --roots: Respect the gc-keep-outputs/gc-keep-derivations settings So if a path is not garbage solely because it's reachable from a root due to the gc-keep-outputs or gc-keep-derivations settings, ‘nix-store -q --roots’ now shows that root.
2012-12-20 17:41:44 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryDerivationOutputs(const StorePath & path) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Rename drv output querying functions - `queryDerivationOutputMapAssumeTotal` -> `queryPartialDerivationOutputMap` - `queryDerivationOutputMapAssumeTotal` -> `queryDerivationOutputMap`
2020-08-20 14:12:51 +00:00
std::map<std::string, std::optional<StorePath>> queryPartialDerivationOutputMap(const StorePath & path) override;
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
std::optional<StorePath> queryPathFromHashPart(const std::string & hashPart) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet querySubstitutablePaths(const StorePathSet & paths) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Add StorePathCAMap for querySubstitutablePathInfos I’m not 100% sure this is wanted since it kind of makes everything have to know about ca even if they don’t really want to. But it also make things easier in dealing with looking up ca.
2020-06-17 19:03:05 +00:00
void querySubstitutablePathInfos(const StorePathCAMap & paths,
SubstitutablePathInfos & infos) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Document addCAToStore/addToStoreFromDump source drainage Also checked that all usages satisfy the requirement and removed dead code.
2020-09-18 09:58:45 +00:00
/* Add a content-addressable store path. `dump` will be drained. */
Style fixes
2020-09-22 09:40:19 +00:00
ref<const ValidPathInfo> addCAToStore(
Source & dump,
const string & name,
ContentAddressMethod caMethod,
const StorePathSet & references,
RepairFlag repair);
Refactor wopAddToStore to make wopAddTextToStore obsolete
2020-09-17 18:19:15 +00:00
Document addCAToStore/addToStoreFromDump source drainage Also checked that all usages satisfy the requirement and removed dead code.
2020-09-18 09:58:45 +00:00
/* Add a content-addressable store path. Does not support references. `dump` will be drained. */
Overhaul wopAddToStore
2020-09-17 17:27:11 +00:00
StorePath addToStoreFromDump(Source & dump, const string & name,
FileIngestionMethod method = FileIngestionMethod::Recursive, HashType hashAlgo = htSHA256, RepairFlag repair = NoRepair) override;
Make 'nix copy --to daemon' run in constant memory
2018-03-21 22:42:21 +00:00
void addToStore(const ValidPathInfo & info, Source & nar,
Remove 'accessor' from addToStore() This is only used by hydra-queue-runner and it's better to implement it there.
2020-07-13 15:37:44 +00:00
RepairFlag repair, CheckSigsFlag checkSigs) override;
Add a Store::addToStore() variant that accepts a NAR As a side effect, this ensures that signatures are propagated when copying paths between stores. Also refactored import/export to make use of this.
2016-05-04 11:36:54 +00:00
Low-latency closure copy This adds a new store operation 'addMultipleToStore' that reads a number of NARs and ValidPathInfos from a Source, allowing any number of store paths to be copied in a single call. This is much faster on high-latency links when copying a lot of small files, like .drv closures. For example, on a connection with an 50 ms delay: Before: $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \ /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \ --derivation --no-check-sigs real 0m57.868s user 0m0.103s sys 0m0.056s After: real 0m0.690s user 0m0.017s sys 0m0.011s
2021-07-26 11:31:09 +00:00
void addMultipleToStore(
Source & source,
RepairFlag repair,
CheckSigsFlag checkSigs) override;
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePath addTextToStore(const string & name, const string & s,
const StorePathSet & references, RepairFlag repair) override;
* Oops.
2006-11-30 18:35:50 +00:00
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 15:36:51 +00:00
void registerDrvOutput(const Realisation & info) override;
std::optional<const Realisation> queryRealisation(const DrvOutput &) override;
buildPaths(): Add an evalStore argument With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue #5025.
2021-07-19 13:43:08 +00:00
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override;
* Oops.
2006-11-30 18:35:50 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 15:57:40 +00:00
BuildMode buildMode) override;
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void ensurePath(const StorePath & path) override;
* Oops.
2006-11-30 18:35:50 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void addTempRoot(const StorePath & path) override;
* Move addTempRoot() to the store API, and add another function syncWithGC() to allow clients to register GC roots without needing write access to the global roots directory or the GC lock.
2006-12-02 16:41:36 +00:00
Shut up clang warnings
2015-09-17 23:22:06 +00:00
void addIndirectRoot(const Path & path) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Shut up clang warnings
2015-09-17 23:22:06 +00:00
void syncWithGC() override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
findRoots(): Add 'censor' parameter This is less brittle than filtering paths after the fact in nix-daemon.
2019-03-14 12:50:07 +00:00
Roots findRoots(bool censor) override;
* Allow unprivileged users to run the garbage collector and to do `nix-store --delete'. But unprivileged users are not allowed to ignore liveness. * `nix-store --delete --ignore-liveness': ignore the runtime roots as well.
2006-12-05 02:18:46 +00:00
Shut up clang warnings
2015-09-17 23:22:06 +00:00
void collectGarbage(const GCOptions & options, GCResults & results) override;
addToStore(): Take explicit name argument
2015-03-25 16:06:12 +00:00
Shut up clang warnings
2015-09-17 23:22:06 +00:00
void optimiseStore() override;
Replace a few bool flags with enums Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-06-28 16:11:01 +00:00
bool verifyStore(bool checkContents, RepairFlag repair) override;
Add an 'optimiseStore' remote procedure call.
2014-09-01 20:21:42 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void addSignatures(const StorePath & storePath, const StringSet & sigs) override;
Add "nix copy-sigs" command This imports signatures from one store into another. E.g. $ nix copy-sigs -r /run/current-system -s https://cache.nixos.org/ imported 595 signatures
2016-04-05 13:30:22 +00:00
Rename Buildable
2021-04-05 13:48:18 +00:00
void queryMissing(const std::vector<DerivedPath> & targets,
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet & willBuild, StorePathSet & willSubstitute, StorePathSet & unknown,
unsigned long long -> uint64_t
2020-07-30 11:10:49 +00:00
uint64_t & downloadSize, uint64_t & narSize) override;
Implement RemoteStore::queryMissing() This provides a significant speedup, e.g. 64 s -> 12 s for nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test on a cold local and CloudFront cache. The alternative is to use lots of concurrent daemon connections but that seems wasteful.
2017-04-06 16:40:19 +00:00
build-remote: Fix fallback to other machines when connecting fails Opening an SSHStore or LegacySSHStore does not actually establish a connection, so the try/catch block here did nothing. Added a Store::connect() method to test whether a connection can be established.
2017-05-02 12:18:46 +00:00
void connect() override;
Store: expose the protocol version used by a store
2018-08-30 21:28:47 +00:00
unsigned int getProtocol() override;
RemoteStore: Add option to drop old connections from the pool This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14 16:10:38 +00:00
void flushBadConnections();
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
struct Connection
{
FdSink to;
FdSource from;
unsigned int daemonVersion;
RemoteStore: Add option to drop old connections from the pool This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14 16:10:38 +00:00
std::chrono::time_point<std::chrono::steady_clock> startTime;
* Some hackery to propagate the worker's stderr and exceptions to the client.
2006-12-03 02:08:13 +00:00
Add ssh store implementation
2016-09-02 18:31:38 +00:00
virtual ~Connection();
Pool<T>: Allow a maximum pool size
2016-02-23 15:40:16 +00:00
Shut down write side before draining the read side This is important if the remote side *does* execute nix-store/nix-daemon successfully, but stdout is polluted (e.g. because the remote user's bashrc script prints something to stdout). In that case we have to shutdown the write side to force the remote nix process to exit.
2021-09-23 16:01:04 +00:00
virtual void closeWrite() = 0;
RemoteStore::addToStore(): Fix race between stderrThread and NAR writer As pointed out by @B4dM4n, the call to to.flush() on stderrThread is unsafe because the NAR writer thread is also writing to 'to'. Fixes #3943.
2020-08-27 12:48:08 +00:00
std::exception_ptr processStderr(Sink * sink = 0, Source * source = 0, bool flush = true);
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
};
* Refactoring.
2006-12-04 13:28:14 +00:00
Cache connection failures
2017-03-03 18:35:34 +00:00
ref<Connection> openConnectionWrapper();
Use `RemoteStore` to open connection for proxying daemon Removes duplicate websocket opening code, and also means we should be able to to ssh-ssh-... daemon relays, not just uds-uds-... ones.
2020-08-19 19:34:47 +00:00
protected:
Factor out the unix domain socket-specific code from RemoteStore
2016-09-02 18:15:04 +00:00
virtual ref<Connection> openConnection() = 0;
void initConnection(Connection & conn);
RemoteStore: Make thread-safe This allows a RemoteStore object to be used safely from multiple threads concurrently. It will make multiple daemon connections if necessary. Note: pool.hh and sync.hh have been copied from the Hydra source tree.
2016-02-23 14:00:59 +00:00
ref<Pool<Connection>> connections;
* Pass various options to the worker so that flags like -K or -j work in multi-user Nix (NIX-72). * Client/worker: exchange a protocol version number for future compatibility.
2007-09-18 09:11:20 +00:00
ssh-ng: Don't forward options to the daemon. This can be iterated on and currently leaves out settings we know we want to forward, but it fixes #1713 and fixes #1935 and isn't fundamentally broken like the status quo. Future changes are suggested in a comment.
2018-03-05 12:42:15 +00:00
virtual void setOptions(Connection & conn);
RemoteStore: Close connection if an exception occurs Fixes #2075.
2018-10-16 21:36:15 +00:00
ConnectionHandle getConnection();
remote-store.hh: ConnectionHandle is struct, minor fix warning
2018-10-26 17:34:29 +00:00
friend struct ConnectionHandle;
RemoteStore: Close connection if an exception occurs Fixes #2075.
2018-10-16 21:36:15 +00:00
gracefully handle old daemon versions Add a fallback path in `queryPartialDerivationOutputMap` for daemons that don't support it. Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this is needed to handle the fallback path.
2020-09-07 09:26:09 +00:00
virtual ref<FSAccessor> getFSAccessor() override;
virtual void narFromPath(const StorePath & path, Sink & sink) override;
Factor out the unix domain socket-specific code from RemoteStore
2016-09-02 18:15:04 +00:00
private:
Cache connection failures
2017-03-03 18:35:34 +00:00
std::atomic_bool failed{false};
Factor out the unix domain socket-specific code from RemoteStore
2016-09-02 18:15:04 +00:00
};
* Pass various options to the worker so that flags like -K or -j work in multi-user Nix (NIX-72). * Client/worker: exchange a protocol version number for future compatibility.
2007-09-18 09:11:20 +00:00
* Oops.
2006-11-30 18:35:50 +00:00
}
Copy permalink