Commit graph

135 commits

Author SHA1 Message Date
Eelco Dolstra 4dda1f92aa Add command 'nix store copy-log'
Fixes #5222.
2022-01-18 14:08:49 +01:00
Eelco Dolstra d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string>
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
2022-01-18 11:12:30 +01:00
Eelco Dolstra 9747ea84b4 Remove CPU locking
This was already accidentally disabled in ba87b08. It also no longer
appears to be beneficial, and in fact slow things down, e.g. when
evaluating a NixOS system configuration:

  elapsed time:       median =      3.8170  mean =      3.8202  stddev =      0.0195  min =      3.7894  max =      3.8600  [rejected, p=0.00000, Δ=0.36929±0.02513]
2021-12-22 15:56:25 +01:00
Eelco Dolstra bc57b3854e
Merge pull request #5695 from obsidiansystems/tidy-logging
Tidy up the logging
2021-12-09 15:31:08 +01:00
John Ericson 95157b4e66 Push wopAddToStore old style stream adapters into smaller scopes
This doesn't fix the bug, but makes the code less difficult to read.
Also improve the comments, now that it is clear what part is needed in
each code path.
2021-11-30 22:10:58 +00:00
John Ericson 70a717f7a8 Nix daemon stream old wopAddToStore
No more buffering in string.
2021-11-30 21:55:53 +00:00
John Ericson 1f15441103 Tidy up the logging
Use the macros more, so we properly skip work when the log level
excludes. Also log the daemon operation number on the daemon side.
2021-11-30 20:23:13 +00:00
Eelco Dolstra 67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev 9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
Eelco Dolstra a7d4f3411e Merge remote-tracking branch 'origin/master' into non-blocking-gc 2021-10-28 14:56:55 +02:00
Eelco Dolstra 6e684d1b87 daemon: Accept 'repeat' setting from untrusted users
Fixes #5352.
2021-10-27 13:09:13 +02:00
regnat af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Alexander Bantyev b9234142f5
addToStore, addToStoreFromDump: add references argument
Allow to pass a set of references to be added as info to the added paths.
2021-10-23 21:30:51 +03:00
Eelco Dolstra 9947f1646a Remove syncWithGC() 2021-10-13 12:12:44 +02:00
John Ericson 242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
regnat b2bce915ad Don’t accept experimental features from the client
If the client sends an “experimental features” setting, just ignore it
2021-08-30 10:48:05 +02:00
Eelco Dolstra fe1f34fa60 Low-latency closure copy
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.

For example, on a connection with an 50 ms delay:

Before:

  $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
    /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
    --derivation --no-check-sigs
  real    0m57.868s
  user    0m0.103s
  sys     0m0.056s

After:

  real    0m0.690s
  user    0m0.017s
  sys     0m0.011s
2021-07-26 13:31:09 +02:00
Eelco Dolstra 3d9de41a5b Hacky fast closure copying mechanism 2021-07-22 09:59:51 +02:00
regnat a8416866cf Always send the realisations as JSON
Align all the worker protocol with `buildDerivation` which inlines the
realisations as one opaque json blob.
That way we don’t have to bother changing the remote store protocol
when the definition of `Realisation` changes, as long as we keep the
json backwards-compatible
2021-05-19 11:45:16 +02:00
Maximilian Bosch e5951a6b2f
Bump version number for DerivedPath changes
I guess I misunderstood John's initial explanation about why wildcards
for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26
also has version 1.29, but misses the wildcard[2]. So bumping seems to
be the right call.

[1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464
[2] 255d145ba7
2021-05-03 01:12:23 +02:00
regnat ed29610cc6 Fix registerDrvOutput with the daemon
Resolve a protocol issue that caused the daemon to endlessly wait for
some information that the client doesn't ever send
2021-04-13 13:58:22 +02:00
John Ericson 9b805d36ac Rename Buildable 2021-04-05 09:52:25 -04:00
John Ericson 255d145ba7 Use BuildableReq for buildPaths and ensurePath
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {}
}` could mean "build `brvPath`" or "substitute `drvPath`" depending on
context.

It also brings the internals closer in line to the new CLI, by
generalizing the `Buildable` type is used there and makes that
distinction already.

In doing so, relegate `StorePathWithOutputs` to being a type just for
backwards compatibility (CLI and RPC).
2021-04-05 08:33:00 -04:00
John Ericson 32f4454b9f Move StorePathWithOutput utilities out of store class
These are by no means part of the notion of a store, but rather are
things that happen to use stores. (Or put another way, there's no way
we'd make them virtual methods any time soon.) It's better to move them
out of that too-big class then.

Also, this helps us remove StorePathWithOutputs from the Store interface
altogether next commit.
2021-04-05 08:31:37 -04:00
John Ericson f44206e719 Fix typos in the last PR #4656 2021-03-22 15:18:48 +00:00
John Ericson 9d309de0de Clean up serialization for BuildResult
A few versioning mistakes were corrected:

- In 27b5747ca7, Daemon protocol had some
  version `>= 0xc` that should have been `>= 0x1c`, or `28` since the
  other conditions used decimal.

- In a2b69660a9, legacy SSH gated new CAS
  info on version 6, but version 5 in the server. It is now 6
  everywhere.

Additionally, legacy ssh was sending over more metadata than the daemon
one was. The daemon now sends that data too.

CC @regnat

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2021-03-22 14:57:41 +00:00
regnat 27b5747ca7 RemoteStore: Send back the new realisations
To allow it to build ca derivations remotely
2021-02-23 08:04:03 +01:00
regnat 58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Matthew Bauer b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Eelco Dolstra 1b79b5b983 read(): Use char * instead of unsigned char *
This gets rid of some pointless casts.
2020-12-02 14:17:27 +01:00
Eelco Dolstra faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Maximilian Bosch 3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng://
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
2020-11-05 20:12:37 +01:00
Eelco Dolstra ff4dea63c9 Generalize extra-* settings
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
2020-10-29 18:17:39 +01:00
Eelco Dolstra c43e882f54 Serialize exceptions from the daemon to the client 2020-10-07 17:13:54 +02:00
John Ericson 69afaeace3 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-30 00:42:28 +00:00
John Ericson b759701652 nix::worker_proto -> worker_proto 2020-09-30 00:41:18 +00:00
John Ericson 45a0ed82f0 Revert "Use template structs instead of phantoms"
This reverts commit 9ab07e99f5.
2020-09-30 00:39:06 +00:00
John Ericson 993229cdaf Deduplicate basic derivation goals too
See comments for security concerns.

Also optimize goal creation by not traversing map twice.
2020-09-22 17:13:59 +00:00
John Ericson e9fc2031f0 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 14:18:31 +00:00
Eelco Dolstra 35a0ac1838 Style fixes 2020-09-22 11:40:19 +02:00
John Ericson b92d3b2edd Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-09-22 00:45:55 +00:00
Robert Hensing fa08db5c4c wopAddToStore: return ValidPathInfo
A ValidPathInfo is created anyway. By returning it we can save a
roundtrip and we have a nicer interface.
2020-09-21 07:55:47 +02:00
Robert Hensing 7c68264085 wopAddToStore: add RepairFlag 2020-09-21 07:55:47 +02:00
Robert Hensing ecc8088cb7 wopAddToStore: Throw to clarify unused refs
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-21 07:55:47 +02:00
Robert Hensing e34fe47d0c Overhaul wopAddToStore 2020-09-21 07:54:05 +02:00
Robert Hensing 14b30b3f3d Move FramedSource and FramedSink, extract withFramedSink 2020-09-17 20:21:04 +02:00
Robert Hensing 9ee3122ec7 Remove redundant import 2020-09-17 20:21:04 +02:00
Eelco Dolstra a59e77d9e5 nix-daemon: Lower verbosity of restricted setting warning
Fixes #3992.
2020-09-14 13:48:51 +02:00
John Ericson c265e0ea82 Merge remote-tracking branch 'upstream/master' into templated-daemon-protocol 2020-08-20 20:24:22 +00:00
Eelco Dolstra 4d77513d97
Merge pull request #3859 from obsidiansystems/drv-outputs-map-allow-missing
`queryDerivationOutputMap` no longer assumes all outputs have a mapping
2020-08-20 16:49:23 +02:00