qyriad/lix
1
0
Fork 0
forked from lix-project/lix
Code Pull requests Activity
lix/src/libstore/legacy-ssh-store.cc

423 lines
13 KiB
C++
Raw Normal View History

Improve store setting descriptions / Markdown formatting
2023-03-22 13:23:36 +00:00
#include "ssh-store-config.hh"
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
#include "archive.hh"
#include "pool.hh"
#include "remote-store.hh"
#include "serve-protocol.hh"
Move `BuildResult` defintion to its own header Just like we did for `ValidPathInfo` in d92d4f85a5c8a2a2385c084500a8b6bd54b54e6c.
2022-03-01 19:43:07 +00:00
#include "build-result.hh"
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
#include "store-api.hh"
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
#include "path-with-outputs.hh"
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
#include "worker-protocol.hh"
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
#include "ssh.hh"
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
#include "derivations.hh"
Move Callback into its own header This gets rid of the inclusion of <future> in util.hh, cutting compilation time by ~20s (CPU time). Issue #4045.
2020-09-21 16:40:11 +00:00
#include "callback.hh"
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
namespace nix {
Improve store setting descriptions / Markdown formatting
2023-03-22 13:23:36 +00:00
struct LegacySSHStoreConfig : virtual CommonSSHStoreConfig
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
Improve store setting descriptions / Markdown formatting
2023-03-22 13:23:36 +00:00
using CommonSSHStoreConfig::CommonSSHStoreConfig;
Fix SSHStore
2023-03-23 08:35:35 +00:00
const Setting<Path> remoteProgram{(StoreConfig*) this, "nix-store", "remote-program",
"Path to the `nix-store` executable on the remote machine."};
Improve store setting descriptions / Markdown formatting
2023-03-22 13:23:36 +00:00
const Setting<int> maxConnections{(StoreConfig*) this, 1, "max-connections",
"Maximum number of concurrent SSH connections."};
Add a name to the stores So that it can be printed by `nix describe-stores`
2020-09-14 12:04:02 +00:00
Support per-store Markdown documentation
2023-03-21 13:03:05 +00:00
const std::string name() override { return "SSH Store"; }
std::string doc() override
{
return
#include "legacy-ssh-store.md"
;
}
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
};
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 15:33:12 +00:00
struct LegacySSHStore : public virtual LegacySSHStoreConfig, public virtual Store
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
{
build-remote: Ugly hackery to get build logs to work The build hook mechanism expects build log output to go to file descriptor 4, so do that.
2017-05-02 10:01:46 +00:00
// Hack for getting remote build log output.
Add `(StoreConfig*)` casts to work around a GCC bug Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80431 that was already there in the code but was accidentally removed in the last commits
2020-09-15 07:10:37 +00:00
// Intentionally not in `LegacySSHStoreConfig` so that it doesn't appear in
// the documentation
const Setting<int> logFD{(StoreConfig*) this, -1, "log-fd", "file descriptor to which SSH's stderr is connected"};
build-remote: Ugly hackery to get build logs to work The build hook mechanism expects build log output to go to file descriptor 4, so do that.
2017-05-02 10:01:46 +00:00
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
struct Connection
{
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
std::unique_ptr<SSHMaster::Connection> sshConn;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
FdSink to;
FdSource from;
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
int remoteVersion;
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
bool good = true;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
};
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
std::string host;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
ref<Pool<Connection>> connections;
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
SSHMaster master;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Properly filter the stores according to their declared uriSchemes When opening a store, only try the stores whose `uriSchemes()` include the current one
2020-09-11 09:11:05 +00:00
static std::set<std::string> uriSchemes() { return {"ssh"}; }
Make the store plugins more introspectable Directly register the store classes rather than a function to build an instance of them. This gives the possibility to introspect static members of the class or choose different ways of instantiating them.
2020-09-08 12:50:23 +00:00
Remove std::string alias (for real this time) Also use std::string_view in a few more places.
2022-02-25 15:00:00 +00:00
LegacySSHStore(const std::string & scheme, const std::string & host, const Params & params)
Correctly call all the parent contructors of the stores Using virtual inheritance means that only the default constructors of the parent classes will be called, which isn't what we want
2020-09-11 09:06:18 +00:00
: StoreConfig(params)
Fix clang build
2023-03-23 08:11:15 +00:00
, CommonSSHStoreConfig(params)
Overhaul store subclassing We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-20 15:33:12 +00:00
, LegacySSHStoreConfig(params)
Separate store configs from the implems Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-10 08:55:51 +00:00
, Store(params)
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
, host(host)
, connections(make_ref<Pool<Connection>>(
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
std::max(1, (int) maxConnections),
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
[this]() { return openConnection(); },
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
[](const ref<Connection> & r) { return r->good; }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
))
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
, master(
host,
Add a Config class to simplify adding configuration settings The typical use is to inherit Config and add Setting<T> members: class MyClass : private Config { Setting<int> foo{this, 123, "foo", "the number of foos to use"}; Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"}; MyClass() : Config(readConfigFile("/etc/my-app.conf")) { std::cout << foo << "\n"; // will print 123 unless overriden } }; Currently, this is used by Store and its subclasses for store parameters. You now get a warning if you specify a non-existant store parameter in a store URI.
2017-04-13 13:55:38 +00:00
sshKey,
distributed builds: load remote builder host key from the machines file This is already used by Hydra, and is very useful when materializing a remote builder list from service discovery. This allows the service discovery tool to only sync one file instead of two.
2021-02-25 01:52:22 +00:00
sshPublicHostKey,
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
// Use SSH master only if using more than 1 connection.
connections->capacity() > 1,
build-remote: Ugly hackery to get build logs to work The build hook mechanism expects build log output to go to file descriptor 4, so do that.
2017-05-02 10:01:46 +00:00
compress,
logFD)
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
}
ref<Connection> openConnection()
{
auto conn = make_ref<Connection>();
LegacySSHStore: Add remote-store option This is primarily useful for testing, e.g. $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 16:07:46 +00:00
conn->sshConn = master.startCommand(
Doh
2018-08-06 15:27:08 +00:00
fmt("%s --serve --write", remoteProgram)
LegacySSHStore: Add remote-store option This is primarily useful for testing, e.g. $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' ...
2018-08-03 16:07:46 +00:00
+ (remoteStore.get() == "" ? "" : " --store " + shellEscape(remoteStore.get())));
Improve SSH handling * Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03 18:05:50 +00:00
conn->to = FdSink(conn->sshConn->in.get());
conn->from = FdSource(conn->sshConn->out.get());
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
try {
conn->to << SERVE_MAGIC_1 << SERVE_PROTOCOL_VERSION;
conn->to.flush();
SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin" Instead of error: serialised integer 7161674624452356180 is too large for type 'j' we now get error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.' Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 15:48:52 +00:00
StringSink saved;
try {
TeeSource tee(conn->from, saved);
unsigned int magic = readInt(tee);
if (magic != SERVE_MAGIC_2)
throw Error("'nix-store --serve' protocol mismatch from '%s'", host);
} catch (SerialisationError & e) {
Shut down write side before draining the read side This is important if the remote side *does* execute nix-store/nix-daemon successfully, but stdout is polluted (e.g. because the remote user's bashrc script prints something to stdout). In that case we have to shutdown the write side to force the remote nix process to exit.
2021-09-23 16:01:04 +00:00
/* In case the other side is waiting for our input,
close it. */
SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin" Instead of error: serialised integer 7161674624452356180 is too large for type 'j' we now get error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.' Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 15:48:52 +00:00
conn->sshConn->in.close();
auto msg = conn->from.drain();
throw Error("'nix-store --serve' protocol mismatch from '%s', got '%s'",
Get rid of std::shared_ptr<std::string> and ref<std::string> These were needed back in the pre-C++11 era because we didn't have move semantics. But now we do.
2022-01-17 21:20:05 +00:00
host, chomp(saved.s + msg));
SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin" Instead of error: serialised integer 7161674624452356180 is too large for type 'j' we now get error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.' Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 15:48:52 +00:00
}
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
conn->remoteVersion = readInt(conn->from);
if (GET_PROTOCOL_MAJOR(conn->remoteVersion) != 0x200)
Replace Unicode quotes in user-facing strings by ASCII Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 11:27:57 +00:00
throw Error("unsupported 'nix-store --serve' protocol version on '%s'", host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
} catch (EndOfFile & e) {
Replace Unicode quotes in user-facing strings by ASCII Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 11:27:57 +00:00
throw Error("cannot connect to '%1%'", host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
return conn;
};
Remove std::string alias (for real this time) Also use std::string_view in a few more places.
2022-02-25 15:00:00 +00:00
std::string getUri() override
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
Properly filter the stores according to their declared uriSchemes When opening a store, only try the stores whose `uriSchemes()` include the current one
2020-09-11 09:11:05 +00:00
return *uriSchemes().begin() + "://" + host;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void queryPathInfoUncached(const StorePath & path,
queryPathInfoUncached(): Return const ValidPathInfo
2018-09-25 16:54:16 +00:00
Callback<std::shared_ptr<const ValidPathInfo>> callback) noexcept override
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
Simplify the callback mechanism
2018-03-27 20:16:01 +00:00
try {
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
auto conn(connections->get());
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
/* No longer support missing NAR hash */
assert(GET_PROTOCOL_MINOR(conn->remoteVersion) >= 4);
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
debug("querying remote host '%s' for info on '%s'", host, printStorePath(path));
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to << cmdQueryPathInfos << PathSet{printStorePath(path)};
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
conn->to.flush();
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
auto p = readString(conn->from);
if (p.empty()) return callback(nullptr);
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
auto path2 = parseStorePath(p);
assert(path == path2);
/* Hash will be set below. FIXME construct ValidPathInfo at end. */
auto info = std::make_shared<ValidPathInfo>(path, Hash::dummy);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
auto deriver = readString(conn->from);
if (deriver != "")
info->deriver = parseStorePath(deriver);
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
info->references = WorkerProto<StorePathSet>::read(*this, conn->from);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
readLongLong(conn->from); // download size
info->narSize = readLongLong(conn->from);
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
{
LegacySSHStore: Include signatures etc.
2017-09-08 14:55:27 +00:00
auto s = readString(conn->from);
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
if (s == "")
throw Error("NAR hash is now mandatory");
info->narHash = Hash::parseAnyPrefixed(s);
LegacySSHStore: Include signatures etc.
2017-09-08 14:55:27 +00:00
}
Use "raw pattern" for content address types We weren't because this ancient PR predated it! This is actually a new version of the pattern which addresses some issues identified in #7479.
2023-03-30 21:12:49 +00:00
info->ca = ContentAddress::parseOpt(readString(conn->from));
Minimize the usage of `Hash::dummy`
2020-08-06 18:31:48 +00:00
info->sigs = readStrings<StringSet>(conn->from);
LegacySSHStore: Include signatures etc.
2017-09-08 14:55:27 +00:00
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
auto s = readString(conn->from);
assert(s == "");
Simplify the callback mechanism
2018-03-27 20:16:01 +00:00
callback(std::move(info));
} catch (...) { callback.rethrow(); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make 'nix copy --to ssh://...' run in constant memory
2018-03-21 22:12:22 +00:00
void addToStore(const ValidPathInfo & info, Source & source,
Remove 'accessor' from addToStore() This is only used by hydra-queue-runner and it's better to implement it there.
2020-07-13 15:37:44 +00:00
RepairFlag repair, CheckSigsFlag checkSigs) override
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
debug("adding path '%s' to remote host '%s'", printStorePath(info.path), host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
auto conn(connections->get());
fix `error: unknown serve command 9`
2018-08-30 23:00:01 +00:00
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 5) {
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
conn->to
<< cmdAddToStoreNar
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< printStorePath(info.path)
<< (info.deriver ? printStorePath(*info.deriver) : "")
Remove optionality in ValidPathInfo::narInfo
2020-08-05 18:42:48 +00:00
<< info.narHash.to_string(Base16, false);
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
workerProtoWrite(*this, conn->to, info.references);
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
<< info.registrationTime
<< info.narSize
<< info.ultimate
<< info.sigs
WIP
2020-06-02 00:37:43 +00:00
<< renderContentAddress(info.ca);
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
try {
copyNAR(source, conn->to);
} catch (...) {
conn->good = false;
throw;
}
conn->to.flush();
} else {
conn->to
<< cmdImportPaths
<< 1;
try {
copyNAR(source, conn->to);
} catch (...) {
conn->good = false;
throw;
}
conn->to
<< exportMagic
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< printStorePath(info.path);
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
workerProtoWrite(*this, conn->to, info.references);
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to
<< (info.deriver ? printStorePath(*info.deriver) : "")
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 19:10:03 +00:00
<< 0
<< 0;
conn->to.flush();
}
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
if (readInt(conn->from) != 1)
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
throw Error("failed to add path '%s' to remote host '%s'", printStorePath(info.path), host);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void narFromPath(const StorePath & path, Sink & sink) override
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
{
auto conn(connections->get());
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
conn->to << cmdDumpStorePath << printStorePath(path);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
conn->to.flush();
Make 'nix copy --from ssh://...' run in constant memory For instance, this reduced the memory consumption of $ nix copy --from ssh://localhost --to ~/my-nix /nix/store/1n7x0yv8vq6zi90hfmian84vdhd04bgp-blender-2.79a from 632 MiB to 16 MiB.
2018-03-21 21:56:02 +00:00
copyNAR(conn->from, sink);
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
std::optional<StorePath> queryPathFromHashPart(const std::string & hashPart) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("queryPathFromHashPart"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Remove std::string alias (for real this time) Also use std::string_view in a few more places.
2022-02-25 15:00:00 +00:00
StorePath addToStore(
std::string_view name,
const Path & srcPath,
FileIngestionMethod method,
HashType hashAlgo,
PathFilter & filter,
RepairFlag repair,
const StorePathSet & references) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("addToStore"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Remove std::string alias (for real this time) Also use std::string_view in a few more places.
2022-02-25 15:00:00 +00:00
StorePath addTextToStore(
std::string_view name,
std::string_view s,
const StorePathSet & references,
RepairFlag repair) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("addTextToStore"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
private:
void putBuildSettings(Connection & conn)
{
conn.to
<< settings.maxSilentTime
<< settings.buildTimeout;
if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 2)
conn.to
<< settings.maxLogSize;
if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 3)
conn.to
Remove repeat and enforce-determinism options These only functioned if a very narrow combination of conditions held: - The result path does not yet exist (--check did not result in repeated builds), AND - The result path is not available from any configured substituters, AND - No remote builders that can build the path are available. If any of these do not hold, a derivation would be built 0 or 1 times regardless of the repeat option. Thus, remove it to avoid confusion.
2022-09-26 18:55:56 +00:00
<< 0 // buildRepeat hasn't worked for ages anyway
<< 0;
nix-store --serve: pass on `settings.keepFailed` from SSH store When doing e.g. nix-build -A package --keep-failed --option \ builders \ 'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel' this doesn't work properly because this build-setting is ignored. I changed this behavior by passing the `settings.keepFailed` through the serve-protocol to remote machines to make sure that I can introspect the build-directory (which is particularly helpful when I have to look at a `config.log` from a failed build for instance).
2021-08-23 21:47:29 +00:00
if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 7) {
conn.to << ((int) settings.keepFailed);
}
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
}
public:
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
BuildMode buildMode) override
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
{
auto conn(connections->get());
conn->to
<< cmdBuildDerivation
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< printStorePath(drvPath);
writeDerivation(conn->to, *this, drv);
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
putBuildSettings(*conn);
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
conn->to.flush();
Make `KeyedBuildResult`, `BuildResult` like before, and fix bug another way In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I realized since derivation goals' wanted outputs can "grow" due to overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called by `Worker::makeDerivationGoalCommon`), the previous bug fix had an unfortunate side effect of causing more pointless rebuilds. In paticular, we have this situation: 1. Goal made from `DerivedPath::Built { foo, {a} }`. 2. Goal gives on on substituting, starts building. 3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just modified original goal. 4. Though the goal had gotten as far as building, so all outputs were going to be produced, `addWantedOutputs` no longer knows that and so the goal is flagged to be restarted. This might sound far-fetched with input-addressed drvs, where we usually basically have all our goals "planned out" before we start doing anything, but with CA derivation goals and especially RFC 92, where *drv resolution* means goals are created after some building is completed, it is more likely to happen. So the first thing to do was restore the clearing of `wantedOutputs` we used to do, and then filter the outputs in `buildPathsWithResults` to only get the ones we care about. But fix also has its own side effect in that the `DerivedPath` in the `BuildResult` in `DerivationGoal` cannot be trusted; it is merely the *first* `DerivedPath` for which this goal was originally created. To remedy this, I made `BuildResult` be like it was before, and instead made `KeyedBuildResult` be a subclass wit the path. Only `buildPathsWithResults` returns `KeyedBuildResult`s, everything else just becomes like it was before, where the "key" is unambiguous from context. I think separating the "primary key" field(s) from the other fields is good practical in general anyways. (I would like to do the same thing for `ValidPathInfo`.) Among other things, it allows constructions like `std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and just precludes the possibility of those duplicate keys being out of sync. We might leverage the above someday to overload `buildPathsWithResults` to take a *set* of return a *map* per the above. ----- Unfortunately, we need to avoid C++20 strictness on designated initializers. (BTW https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html this offers some new syntax for this use-case. Hopefully this will be adopted and we can eventually use it.) No having that yet, maybe it would be better to not make `KeyedBuildResult` a subclass to just avoid this. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2022-03-25 01:26:07 +00:00
BuildResult status;
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
status.status = (BuildResult::Status) readInt(conn->from);
conn->from >> status.errorMsg;
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 3)
conn->from >> status.timesBuilt >> status.isNonDeterministic >> status.startTime >> status.stopTime;
LegacySSHStore: Send back the new realisations To allow it to build ca derivations remotely
2021-01-26 09:50:44 +00:00
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 6) {
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
auto builtOutputs = WorkerProto<DrvOutputs>::read(*this, conn->from);
Introduce `SingleDrvOutputs` In many cases we are dealing with a collection of realisations, they are all outputs of the same derivation. In that case, we don't need "derivation hashes modulos" to be part of our map key, because the output names alone will be unique. Those hashes are still part of the realisation proper, so we aren't loosing any information, we're just "normalizing our schema" by narrowing the "primary key". Besides making our data model a bit "tighter" this allows us to avoid a double `for` loop in `DerivationGoal::waiteeDone`. The inner `for` loop was previously just to select the output we cared about without knowing its hash. Now we can just select the output by name directly. Note that neither protocol is changed as part of this: we are still transferring `DrvOutputs` over the wire for `BuildResult`s. I would only consider revising this once #6223 is merged, and we can mention protocol versions inside factored-out serialization logic. Until then it is better not change anything because it would come a the cost of code reuse.
2023-04-14 22:18:32 +00:00
for (auto && [output, realisation] : builtOutputs)
status.builtOutputs.insert_or_assign(
std::move(output.outputName),
std::move(realisation));
LegacySSHStore: Send back the new realisations To allow it to build ca derivations remotely
2021-01-26 09:50:44 +00:00
}
Implement LegacySSHStore::buildDerivation() This makes LegacySSHStore usable by build-remote and hydra-queue-runner.
2017-05-01 14:08:13 +00:00
return status;
}
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
buildPaths(): Add an evalStore argument With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue #5025.
2021-07-19 13:43:08 +00:00
void buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
{
buildPaths(): Add an evalStore argument With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue #5025.
2021-07-19 13:43:08 +00:00
if (evalStore && evalStore.get() != this)
throw Error("building on an SSH store is incompatible with '--eval-store'");
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
auto conn(connections->get());
conn->to << cmdBuildPaths;
Strings ss;
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
for (auto & p : drvPaths) {
Rename Buildable
2021-04-05 13:48:18 +00:00
auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p);
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
std::visit(overloaded {
`std::visit` by reference I had started the trend of doing `std::visit` by value (because a type error once mislead me into thinking that was the only form that existed). While the optomizer in principle should be able to deal with extra coppying or extra indirection once the lambdas inlined, sticking with by reference is the conventional default. I hope this might even improve performance.
2021-09-30 21:31:21 +00:00
[&](const StorePathWithOutputs & s) {
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
ss.push_back(s.to_string(*this));
},
`std::visit` by reference I had started the trend of doing `std::visit` by value (because a type error once mislead me into thinking that was the only form that existed). While the optomizer in principle should be able to deal with extra coppying or extra indirection once the lambdas inlined, sticking with by reference is the conventional default. I hope this might even improve performance.
2021-09-30 21:31:21 +00:00
[&](const StorePath & drvPath) {
Use `BuildableReq` for `buildPaths` and `ensurePath` This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-02 03:50:41 +00:00
throw Error("wanted to fetch '%s' but the legacy ssh protocol doesn't support merely substituting drv files via the build paths command. It would build them instead. Try using ssh-ng://", printStorePath(drvPath));
},
}, sOrDrvPath);
}
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
conn->to << ss;
putBuildSettings(*conn);
conn->to.flush();
Make `KeyedBuildResult`, `BuildResult` like before, and fix bug another way In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I realized since derivation goals' wanted outputs can "grow" due to overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called by `Worker::makeDerivationGoalCommon`), the previous bug fix had an unfortunate side effect of causing more pointless rebuilds. In paticular, we have this situation: 1. Goal made from `DerivedPath::Built { foo, {a} }`. 2. Goal gives on on substituting, starts building. 3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just modified original goal. 4. Though the goal had gotten as far as building, so all outputs were going to be produced, `addWantedOutputs` no longer knows that and so the goal is flagged to be restarted. This might sound far-fetched with input-addressed drvs, where we usually basically have all our goals "planned out" before we start doing anything, but with CA derivation goals and especially RFC 92, where *drv resolution* means goals are created after some building is completed, it is more likely to happen. So the first thing to do was restore the clearing of `wantedOutputs` we used to do, and then filter the outputs in `buildPathsWithResults` to only get the ones we care about. But fix also has its own side effect in that the `DerivedPath` in the `BuildResult` in `DerivationGoal` cannot be trusted; it is merely the *first* `DerivedPath` for which this goal was originally created. To remedy this, I made `BuildResult` be like it was before, and instead made `KeyedBuildResult` be a subclass wit the path. Only `buildPathsWithResults` returns `KeyedBuildResult`s, everything else just becomes like it was before, where the "key" is unambiguous from context. I think separating the "primary key" field(s) from the other fields is good practical in general anyways. (I would like to do the same thing for `ValidPathInfo`.) Among other things, it allows constructions like `std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and just precludes the possibility of those duplicate keys being out of sync. We might leverage the above someday to overload `buildPathsWithResults` to take a *set* of return a *map* per the above. ----- Unfortunately, we need to avoid C++20 strictness on designated initializers. (BTW https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html this offers some new syntax for this use-case. Hopefully this will be adopted and we can eventually use it.) No having that yet, maybe it would be better to not make `KeyedBuildResult` a subclass to just avoid this. Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2022-03-25 01:26:07 +00:00
BuildResult result;
Define `LegacySSHStore::buildPaths` using `cmdBuildPaths` Evidentally this was never implemented because Nix switched to using `buildDerivation` exclusively before `build-remote.pl` was rewritten. The `nix-copy-ssh` test (already) tests this.
2020-08-12 03:13:17 +00:00
result.status = (BuildResult::Status) readInt(conn->from);
if (!result.success()) {
conn->from >> result.errorMsg;
throw Error(result.status, result.errorMsg);
}
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void ensurePath(const StorePath & path) override
unsupported(): Show the name of the unsupported operation
2019-01-18 12:34:23 +00:00
{ unsupported("ensurePath"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
Push `getFSAccessor` `unsupported(...)` down `Store` class hierarchy More progress on issue #5729. Instead of having it by the default method in `Store` itself, have it be the implementation in `DummyStore` and `LegacySSHStore`. Then just the implementations which fail to provide the method pay the "penalty" of dealing with the icky `unimplemented` function for non-compliance. Combined with my other recent PRs, this finally makes `Store` have no `unsupported` calls!
2023-04-13 17:38:12 +00:00
virtual ref<FSAccessor> getFSAccessor() override
{ unsupported("getFSAccessor"); }
Support `repairPath` on most stores. More progress on issue #5729 The method trivially generalizes to be store-implementation-agnostic, in fact. However, we force it to continue to be unimplemented with `RemoteStore` and `LegacySSHStore` because the implementation we'd get via the generalization is probably not the one users expect. This keeps our hands untied to do it right going forward. For more about the tension between the scheduler logic being store-type-agnostic and remote stores doing their own scheduling, see issues #5025 and #5056.
2023-04-13 14:38:35 +00:00
/**
* The default instance would schedule the work on the client side, but
* for consistency with `buildPaths` and `buildDerivation` it should happen
* on the remote side.
*
* We make this fail for now so we can add implement this properly later
* without it being a breaking change.
*/
void repairPath(const StorePath & path) override
{ unsupported("repairPath"); }
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
void computeFSClosure(const StorePathSet & paths,
StorePathSet & out, bool flipDirection = false,
LegacySSHStore: Provide a faster implementation of computeFSClosure() This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 10:44:01 +00:00
bool includeOutputs = false, bool includeDerivers = false) override
{
if (flipDirection || includeDerivers) {
Store::computeFSClosure(paths, out, flipDirection, includeOutputs, includeDerivers);
return;
}
auto conn(connections->get());
conn->to
<< cmdQueryClosure
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< includeOutputs;
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
workerProtoWrite(*this, conn->to, paths);
LegacySSHStore: Provide a faster implementation of computeFSClosure() This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 10:44:01 +00:00
conn->to.flush();
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
for (auto & i : WorkerProto<StorePathSet>::read(*this, conn->from))
Remove StorePath::clone() and related functions
2020-06-16 20:20:18 +00:00
out.insert(i);
LegacySSHStore: Provide a faster implementation of computeFSClosure() This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 10:44:01 +00:00
}
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
StorePathSet queryValidPaths(const StorePathSet & paths,
Replace a few bool flags with enums Functions like copyClosure() had 3 bool arguments, which creates a severe risk of mixing up arguments. Also, implement copyClosure() using copyPaths().
2017-06-28 16:11:01 +00:00
SubstituteFlag maybeSubstitute = NoSubstitute) override
copyPaths(): Use queryValidPaths() to reduce SSH latency
2017-03-16 12:50:01 +00:00
{
auto conn(connections->get());
conn->to
<< cmdQueryValidPaths
<< false // lock
Make the Store API more type-safe Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-12-05 18:11:09 +00:00
<< maybeSubstitute;
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
workerProtoWrite(*this, conn->to, paths);
copyPaths(): Use queryValidPaths() to reduce SSH latency
2017-03-16 12:50:01 +00:00
conn->to.flush();
Revert "Revert "Use template structs instead of phantoms"" This is the more typically way to do [Argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging generic serializers in C++. It makes the relationship between the `read` and `write` methods more clear and rigorous, and also looks more familiar to users coming from other languages that do not have C++'s libertine ad-hoc overloading. I am returning to this because during the review in https://github.com/NixOS/nix/pull/6223, it came up as something that would make the code easier to read --- easier today hopefully already, but definitely easier if we were have multiple codified protocols with code sharing between them as that PR seeks to accomplish. If I recall correctly, the main criticism of this the first time around (in 2020) was that having to specify the type when writing, e.g. `WorkerProto<MyType>::write`, was too verbose and cumbersome. This is now addressed with the `workerProtoWrite` wrapper function. This method is also the way `nlohmann::json`, which we have used for a number of years now, does its serializers, for what its worth. This reverts commit 45a0ed82f089158a79c8c25ef844c55e4a74fc35. That commit in turn reverted 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2023-05-18 02:04:59 +00:00
return WorkerProto<StorePathSet>::read(*this, conn->from);
copyPaths(): Use queryValidPaths() to reduce SSH latency
2017-03-16 12:50:01 +00:00
}
build-remote: Fix fallback to other machines when connecting fails Opening an SSHStore or LegacySSHStore does not actually establish a connection, so the try/catch block here did nothing. Added a Store::connect() method to test whether a connection can be established.
2017-05-02 12:18:46 +00:00
void connect() override
{
auto conn(connections->get());
}
Store: expose the protocol version used by a store
2018-08-30 21:28:47 +00:00
unsigned int getProtocol() override
{
auto conn(connections->get());
return conn->remoteVersion;
}
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 15:36:51 +00:00
Add `Store::isTrustedClient()` This function returns true or false depending on whether the Nix client is trusted or not. Mostly relevant when speaking to a remote store with a daemon. We include this information in `nix ping store` and `nix doctor` Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2022-12-26 20:21:08 +00:00
/**
* The legacy ssh protocol doesn't support checking for trusted-user.
* Try using ssh-ng:// instead if you want to know.
*/
std::optional<TrustedFlag> isTrustedClient() override
{
return std::nullopt;
}
Expose an async interface for `queryRealisation` Doesn’t change much so far because everything is still using it synchronously, but should allow the binary cache to fetch stuff in parallel
2021-10-27 09:36:51 +00:00
void queryRealisationUncached(const DrvOutput &,
Callback<std::shared_ptr<const Realisation>> callback) noexcept override
Store metadata about drv outputs realisations For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-10-08 15:36:51 +00:00
// TODO: Implement
{ unsupported("queryRealisation"); }
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
};
Remove static variable name clashes This was useful for an experiment with building Nix as a single compilation unit. It's not very useful otherwise but also doesn't hurt...
2020-10-06 11:36:55 +00:00
static RegisterStoreImplementation<LegacySSHStore, LegacySSHStoreConfig> regLegacySSHStore;
Add a LegacySSHStore that uses nix-store --serve This is useful for nix-copy-closure.
2017-02-07 18:28:40 +00:00
}
Copy permalink