aleksanaa/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
lix/src/libstore/legacy-ssh-store.cc

379 lines
13 KiB
C++
Raw Normal View History

Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
#include "archive.hh"
#include "pool.hh"
#include "remote-store.hh"
#include "serve-protocol.hh"
#include "store-api.hh"
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
#include "path-with-outputs.hh"
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
#include "worker-protocol.hh"
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
#include "ssh.hh"
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
#include "derivations.hh"
Move Callback into its own header This gets rid of the inclusion of <future> in util.hh, cutting compilation time by ~20s (CPU time). Issue #4045.
2020-09-21 16:40:11 +00:00
#include "callback.hh"
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
namespace nix {
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
struct LegacySSHStoreConfig : virtual StoreConfig
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
using StoreConfig::StoreConfig;
Add `(StoreConfig*)` casts to work around a GCC bug Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was already there in the code but was accidentally removed in the last commits
2020-09-15 07:10:37 +00:00
const Setting<int> maxConnections{(StoreConfig*) this, 1, "max-connections", "maximum number of concurrent SSH connections"};
const Setting<Path> sshKey{(StoreConfig*) this, "", "ssh-key", "path to an SSH private key"};
distributed builds: load remote builder host key from the machines file This is already used by Hydra, and is very useful when materializing a remote builder list from service discovery. This allows the service discovery tool to only sync one file instead of two.
2021-02-25 01:52:22 +00:00
const Setting<std::string> sshPublicHostKey{(StoreConfig*) this, "", "base64-ssh-public-host-key", "The public half of the host's SSH key"};
Add `(StoreConfig*)` casts to work around a GCC bug Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was already there in the code but was accidentally removed in the last commits
2020-09-15 07:10:37 +00:00
const Setting<bool> compress{(StoreConfig*) this, false, "compress", "whether to compress the connection"};
const Setting<Path> remoteProgram{(StoreConfig*) this, "nix-store", "remote-program", "path to the nix-store executable on the remote system"};
const Setting<std::string> remoteStore{(StoreConfig*) this, "", "remote-store", "URI of the store on the remote system"};
Add a name to the stores So that it can be printed by `nix describe-stores`
2020-09-14 12:04:02 +00:00
const std::string name() override { return "Legacy SSH Store"; }
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
};
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 15:33:12 +00:00
struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Store
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
{
build-remote: Ugly hackery to get build logs to work The build hook mechanism expects build log output to go to file descriptor 4, so do that.
2017-05-02 10:01:46 +00:00
// Hack for getting remote build log output.
Add `(StoreConfig*)` casts to work around a GCC bug Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was already there in the code but was accidentally removed in the last commits
2020-09-15 07:10:37 +00:00
// Intentionally not in `LegacySSHStoreConfig` so that it doesn't appear in
// the documentation
const Setting<int> logFD{(StoreConfig*) this, -1, "log-fd", "file descriptor to which SSH's stderr is connected"};
build-remote: Ugly hackery to get build logs to work The build hook mechanism expects build log output to go to file descriptor 4, so do that.
2017-05-02 10:01:46 +00:00
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
struct Connection
{
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
std::unique_ptr<SSHMaster::Connection> sshConn;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
FdSink to;
FdSource from;
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
int remoteVersion;
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
bool good = true;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
};
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
std::string host;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
ref<Pool<Connection>> connections;
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
SSHMaster master;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Properly filter the stores according to their declared uriSchemes When opening a store, only try the stores whose `uriSchemes()` include the current one
2020-09-11 09:11:05 +00:00
static std::set<std::string> uriSchemes() { return {"ssh"}; }
Make the store plugins more introspectable Directly register the store classes rather than a function to build an instance of them. This gives the possibility to introspect static members of the class or choose different ways of instantiating them.
2020-09-08 12:50:23 +00:00
Properly filter the stores according to their declared uriSchemes When opening a store, only try the stores whose `uriSchemes()` include the current one
2020-09-11 09:11:05 +00:00
LegacySSHStore(const string & scheme, const string & host, const Params & params)
Correctly call all the parent contructors of the stores Using virtual inheritance means that only the default constructors of the parent classes will be called, which isn't what we want
2020-09-11 09:06:18 +00:00
: StoreConfig(params)
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 15:33:12 +00:00
, LegacySSHStoreConfig(params)
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
, Store(params)
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
, host(host)
, connections(make_ref<Pool<Connection>>(
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
std::max(1, (int) maxConnections),
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
[this]() { return openConnection(); },
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
[](const ref<Connection> & r) { return r->good; }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
))
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
, master(
host,
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
sshKey,
distributed builds: load remote builder host key from the machines file This is already used by Hydra, and is very useful when materializing a remote builder list from service discovery. This allows the service discovery tool to only sync one file instead of two.
2021-02-25 01:52:22 +00:00
sshPublicHostKey,
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
// Use SSH master only if using more than 1 connection.
connections->capacity() > 1,
build-remote: Ugly hackery to get build logs to work The build hook mechanism expects build log output to go to file descriptor 4, so do that.
2017-05-02 10:01:46 +00:00
compress,
logFD)
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
}
ref<Connection> openConnection()
{
auto conn = make_ref<Connection>();
LegacySSHStore: Add remote-store option This is primarily useful for testing, e.g. $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 16:07:46 +00:00
conn->sshConn = master.startCommand(
Doh
2018-08-06 15:27:08 +00:00
fmt("%s --serve --write", remoteProgram)
LegacySSHStore: Add remote-store option This is primarily useful for testing, e.g. $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 16:07:46 +00:00
+ (remoteStore.get() == "" ? "" : " --store " + shellEscape(remoteStore.get())));
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
conn->to = FdSink(conn->sshConn->in.get());
conn->from = FdSource(conn->sshConn->out.get());
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
try {
conn->to << SERVE_MAGIC_1 << SERVE_PROTOCOL_VERSION;
conn->to.flush();
SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin" Instead of error: serialised integer 7161674624452356180 is too large for type 'j' we now get error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.' Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 15:48:52 +00:00
StringSink saved;
try {
TeeSource tee(conn->from, saved);
unsigned int magic = readInt(tee);
if (magic != SERVE_MAGIC_2)
throw Error("'nix-store --serve' protocol mismatch from '%s'", host);
} catch (SerialisationError & e) {
Shut down write side before draining the read side This is important if the remote side *does* execute nix-store/nix-daemon successfully, but stdout is polluted (e.g. because the remote user's bashrc script prints something to stdout). In that case we have to shutdown the write side to force the remote nix process to exit.
2021-09-23 16:01:04 +00:00
/* In case the other side is waiting for our input,
close it. */
SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin" Instead of error: serialised integer 7161674624452356180 is too large for type 'j' we now get error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.' Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 15:48:52 +00:00
conn->sshConn->in.close();
auto msg = conn->from.drain();
throw Error("'nix-store --serve' protocol mismatch from '%s', got '%s'",
host, chomp(*saved.s + msg));
}
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
conn->remoteVersion = readInt(conn->from);
if (GET_PROTOCOL_MAJOR(conn->remoteVersion) != 0x200)
Replace Unicode quotes in user-facing strings by ASCII Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 11:27:57 +00:00
throw Error("unsupported 'nix-store --serve' protocol version on '%s'", host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
} catch (EndOfFile & e) {
Replace Unicode quotes in user-facing strings by ASCII Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 11:27:57 +00:00
throw Error("cannot connect to '%1%'", host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
return conn;
};
string getUri() override
{
Properly filter the stores according to their declared uriSchemes When opening a store, only try the stores whose `uriSchemes()` include the current one
2020-09-11 09:11:05 +00:00
return *uriSchemes().begin() + "://" + host;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void queryPathInfoUncached(const StorePath & path,
queryPathInfoUncached(): Return const ValidPathInfo
2018-09-25 16:54:16 +00:00
Callback<std::shared_ptr<const ValidPathInfo>> callback) noexcept override
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
Simplify the callback mechanism
2018-03-27 20:16:01 +00:00
try {
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
auto conn(connections->get());
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
/* No longer support missing NAR hash */
assert(GET_PROTOCOL_MINOR(conn->remoteVersion) >= 4);
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
debug("querying remote host '%s' for info on '%s'", host, printStorePath(path));
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to << cmdQueryPathInfos << PathSet{printStorePath(path)};
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
conn->to.flush();
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
auto p = readString(conn->from);
if (p.empty()) return callback(nullptr);
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
auto path2 = parseStorePath(p);
assert(path == path2);
/* Hash will be set below. FIXME construct ValidPathInfo at end. */
auto info = std::make_shared<ValidPathInfo>(path, Hash::dummy);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
PathSet references;
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
auto deriver = readString(conn->from);
if (deriver != "")
info->deriver = parseStorePath(deriver);
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
info->references = worker_proto::read(*this, conn->from, Phantom<StorePathSet> {});
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
readLongLong(conn->from); // download size
info->narSize = readLongLong(conn->from);
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
{
LegacySSHStore: Include signatures etc.
2017-09-08 14:55:27 +00:00
auto s = readString(conn->from);
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
if (s == "")
throw Error("NAR hash is now mandatory");
info->narHash = Hash::parseAnyPrefixed(s);
LegacySSHStore: Include signatures etc.
2017-09-08 14:55:27 +00:00
}
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
info->ca = parseContentAddressOpt(readString(conn->from));
info->sigs = readStrings<StringSet>(conn->from);
LegacySSHStore: Include signatures etc.
2017-09-08 14:55:27 +00:00
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
auto s = readString(conn->from);
assert(s == "");
Simplify the callback mechanism
2018-03-27 20:16:01 +00:00
callback(std::move(info));
} catch (...) { callback.rethrow(); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make 'nix copy --to ssh://...' run in constant memory
2018-03-21 22:12:22 +00:00
void addToStore(const ValidPathInfo & info, Source & source,
Remove 'accessor' from addToStore() This is only used by hydra-queue-runner and it's better to implement it there.
2020-07-13 15:37:44 +00:00
RepairFlag repair, CheckSigsFlag checkSigs) override
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
debug("adding path '%s' to remote host '%s'", printStorePath(info.path), host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
auto conn(connections->get());
fix `error: unknown serve command 9`
2018-08-30 23:00:01 +00:00
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 5) {
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
conn->to
<< cmdAddToStoreNar
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< printStorePath(info.path)
<< (info.deriver ? printStorePath(*info.deriver) : "")
Remove optionality in ValidPathInfo::narInfo
2020-08-05 18:42:48 +00:00
<< info.narHash.to_string(Base16, false);
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
worker_proto::write(*this, conn->to, info.references);
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
<< info.registrationTime
<< info.narSize
<< info.ultimate
<< info.sigs
WIP
2020-06-02 00:37:43 +00:00
<< renderContentAddress(info.ca);
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
try {
copyNAR(source, conn->to);
} catch (...) {
conn->good = false;
throw;
}
conn->to.flush();
} else {
conn->to
<< cmdImportPaths
<< 1;
try {
copyNAR(source, conn->to);
} catch (...) {
conn->good = false;
throw;
}
conn->to
<< exportMagic
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< printStorePath(info.path);
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
worker_proto::write(*this, conn->to, info.references);
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to
<< (info.deriver ? printStorePath(*info.deriver) : "")
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
<< 0
<< 0;
conn->to.flush();
}
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
if (readInt(conn->from) != 1)
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
throw Error("failed to add path '%s' to remote host '%s'", printStorePath(info.path), host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void narFromPath(const StorePath & path, Sink & sink) override
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
auto conn(connections->get());
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to << cmdDumpStorePath << printStorePath(path);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
conn->to.flush();
Make 'nix copy --from ssh://...' run in constant memory For instance, this reduced the memory consumption of $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a from 632 MiB to 16 MiB.
2018-03-21 21:56:02 +00:00
copyNAR(conn->from, sink);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
std::optional<StorePath> queryPathFromHashPart(const std::string & hashPart) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("queryPathFromHashPart"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePath addToStore(const string & name, const Path & srcPath,
Replace some `bool recursive` with a new `FileIngestionMethod` enum
2020-03-29 05:04:55 +00:00
FileIngestionMethod method, HashType hashAlgo,
addToStore, addToStoreFromDump: refactor: pass refs by const reference Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 09:24:49 +00:00
PathFilter & filter, RepairFlag repair, const StorePathSet & references) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("addToStore"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePath addTextToStore(const string & name, const string & s,
const StorePathSet & references, RepairFlag repair) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("addTextToStore"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
private:
void putBuildSettings(Connection & conn)
{
conn.to
<< settings.maxSilentTime
<< settings.buildTimeout;
if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 2)
conn.to
<< settings.maxLogSize;
if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 3)
conn.to
<< settings.buildRepeat
<< settings.enforceDeterminism;
nix-store --serve: pass on `settings.keepFailed` from SSH store When doing e.g. nix-build -A package --keep-failed --option \ builders \ 'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel' this doesn't work properly because this build-setting is ignored. I changed this behavior by passing the `settings.keepFailed` through the serve-protocol to remote machines to make sure that I can introspect the build-directory (which is particularly helpful when I have to look at a `config.log` from a failed build for instance).
2021-08-23 21:47:29 +00:00
if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 7) {
conn.to << ((int) settings.keepFailed);
}
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
}
public:
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
BuildMode buildMode) override
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
{
auto conn(connections->get());
conn->to
<< cmdBuildDerivation
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< printStorePath(drvPath);
writeDerivation(conn->to, *this, drv);
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
putBuildSettings(*conn);
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
conn->to.flush();
BuildResult status;
status.status = (BuildResult::Status) readInt(conn->from);
conn->from >> status.errorMsg;
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 3)
conn->from >> status.timesBuilt >> status.isNonDeterministic >> status.startTime >> status.stopTime;
LegacySSHStore: Send back the new realisations To allow it to build ca derivations remotely
2021-01-26 09:50:44 +00:00
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 6) {
status.builtOutputs = worker_proto::read(*this, conn->from, Phantom<DrvOutputs> {});
}
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
return status;
}
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
buildPaths(): Add an evalStore argument With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue #5025.
2021-07-19 13:43:08 +00:00
void buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
{
buildPaths(): Add an evalStore argument With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue #5025.
2021-07-19 13:43:08 +00:00
if (evalStore && evalStore.get() != this)
throw Error("building on an SSH store is incompatible with '--eval-store'");
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
auto conn(connections->get());
conn->to << cmdBuildPaths;
Strings ss;
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
for (auto & p : drvPaths) {
Rename Buildable
2021-04-05 13:48:18 +00:00
auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p);
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
std::visit(overloaded {
`std::visit` by reference I had started the trend of doing `std::visit` by value (because a type error once mislead me into thinking that was the only form that existed). While the optomizer in principle should be able to deal with extra coppying or extra indirection once the lambdas inlined, sticking with by reference is the conventional default. I hope this might even improve performance.
2021-09-30 21:31:21 +00:00
[&](const StorePathWithOutputs & s) {
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
ss.push_back(s.to_string(*this));
},
`std::visit` by reference I had started the trend of doing `std::visit` by value (because a type error once mislead me into thinking that was the only form that existed). While the optomizer in principle should be able to deal with extra coppying or extra indirection once the lambdas inlined, sticking with by reference is the conventional default. I hope this might even improve performance.
2021-09-30 21:31:21 +00:00
[&](const StorePath & drvPath) {
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
throw Error("wanted to fetch '%s' but the legacy ssh protocol doesn't support merely substituting drv files via the build paths command. It would build them instead. Try using ssh-ng://", printStorePath(drvPath));
},
}, sOrDrvPath);
}
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
conn->to << ss;
putBuildSettings(*conn);
conn->to.flush();
BuildResult result;
result.status = (BuildResult::Status) readInt(conn->from);
if (!result.success()) {
conn->from >> result.errorMsg;
throw Error(result.status, result.errorMsg);
}
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void ensurePath(const StorePath & path) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("ensurePath"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void computeFSClosure(const StorePathSet & paths,
StorePathSet & out, bool flipDirection = false,
LegacySSHStore: Provide a faster implementation of computeFSClosure() This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 10:44:01 +00:00
bool includeOutputs = false, bool includeDerivers = false) override
{
if (flipDirection || includeDerivers) {
Store::computeFSClosure(paths, out, flipDirection, includeOutputs, includeDerivers);
return;
}
auto conn(connections->get());
conn->to
<< cmdQueryClosure
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< includeOutputs;
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
worker_proto::write(*this, conn->to, paths);
LegacySSHStore: Provide a faster implementation of computeFSClosure() This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 10:44:01 +00:00
conn->to.flush();
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
for (auto & i : worker_proto::read(*this, conn->from, Phantom<StorePathSet> {}))
Remove StorePath::clone() and related functions
2020-06-16 20:20:18 +00:00
out.insert(i);
LegacySSHStore: Provide a faster implementation of computeFSClosure() This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 10:44:01 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryValidPaths(const StorePathSet & paths,
Replace a few bool flags with enums Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-06-28 16:11:01 +00:00
SubstituteFlag maybeSubstitute = NoSubstitute) override
copyPaths(): Use queryValidPaths() to reduce SSH latency
2017-03-16 12:50:01 +00:00
{
auto conn(connections->get());
conn->to
<< cmdQueryValidPaths
<< false // lock
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< maybeSubstitute;
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
worker_proto::write(*this, conn->to, paths);
copyPaths(): Use queryValidPaths() to reduce SSH latency
2017-03-16 12:50:01 +00:00
conn->to.flush();
nix::worker_proto -> worker_proto
2020-09-30 00:41:18 +00:00
return worker_proto::read(*this, conn->from, Phantom<StorePathSet> {});
copyPaths(): Use queryValidPaths() to reduce SSH latency
2017-03-16 12:50:01 +00:00
}
build-remote: Fix fallback to other machines when connecting fails Opening an SSHStore or LegacySSHStore does not actually establish a connection, so the try/catch block here did nothing. Added a Store::connect() method to test whether a connection can be established.
2017-05-02 12:18:46 +00:00
void connect() override
{
auto conn(connections->get());
}
Store: expose the protocol version used by a store
2018-08-30 21:28:47 +00:00
unsigned int getProtocol() override
{
auto conn(connections->get());
return conn->remoteVersion;
}
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 15:36:51 +00:00
Expose an async interface for `queryRealisation` Doesn’t change much so far because everything is still using it synchronously, but should allow the binary cache to fetch stuff in parallel
2021-10-27 09:36:51 +00:00
void queryRealisationUncached(const DrvOutput &,
Callback<std::shared_ptr<const Realisation>> callback) noexcept override
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 15:36:51 +00:00
// TODO: Implement
{ unsupported("queryRealisation"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
};
Remove static variable name clashes This was useful for an experiment with building Nix as a single compilation unit. It's not very useful otherwise but also doesn't hurt...
2020-10-06 11:36:55 +00:00
static RegisterStoreImplementation<LegacySSHStore, LegacySSHStoreConfig> regLegacySSHStore;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Copy permalink