Merge remote-tracking branch 'upstream/master' into path-info

doc/manual/src/release-notes/rl-2.4.md

@@ -1,8 +1,498 @@
-# Release 2.4 (202X-XX-XX)
+# Release 2.4 (2021-10-XX)
 
-  - It is now an error to modify the `plugin-files` setting via a
+This is the first release in more than two years and is the result of
-    command-line flag that appears after the first non-flag argument
+more than 2800 commits from 195 contributors since release 2.3.
-    to any command, including a subcommand to `nix`. For example,
+
+## Highlights
+
+* Nix's **error messages** have been improved a lot. For instance,
+  evaluation errors now point out the location of the error:
+
+  ```
+  $ nix build
+  error: undefined variable 'bzip3'
+
+         at /nix/store/449lv242z0zsgwv95a8124xi11sp419f-source/flake.nix:88:13:
+
+             87|           [ curl
+             88|             bzip3 xz brotli editline
+               |             ^
+             89|             openssl sqlite
+  ```
+
+* The **`nix` command** has seen a lot of work and is now almost at
+  feature parity with the old command-line interface (the `nix-*`
+  commands). It aims to be [more modern, consistent and pleasant to
+  use](../contributing/cli-guideline.md) than the old CLI. It is still
+  marked as experimental but its interface should not change much
+  anymore in future releases.
+
+* **Flakes** are a new format to package Nix-based projects in a more
+  discoverable, composable, consistent and reproducible way. A flake
+  is just a repository or tarball containing a file named `flake.nix`
+  that specifies dependencies on other flakes and returns any Nix
+  assets such as packages, Nixpkgs overlays, NixOS modules or CI
+  tests. The new `nix` CLI is primarily based around flakes; for
+  example, a command like `nix run nixpkgs#hello` runs the `hello`
+  application from the `nixpkgs` flake.
+
+  Flakes are currently marked as experimental. For an introduction,
+  see [this blog
+  post](https://www.tweag.io/blog/2020-05-25-flakes/). For detailed
+  information about flake syntax and semantics, see the [`nix flake`
+  manual page](../command-ref/new-cli/nix3-flake.md).
+
+* Nix's store can now be **content-addressed**, meaning that the hash
+  component of a store path is the hash of the path's
+  contents. Previously Nix could only build **input-addressed** store
+  paths, where the hash is computed from the derivation dependency
+  graph. Content-addressing allows deduplication, early cutoff in
+  build systems, and unprivileged closure copying. This is still [an
+  experimental
+  feature](https://discourse.nixos.org/t/content-addressed-nix-call-for-testers/12881).
+
+* The Nix manual has been converted into Markdown, making it easier to
+  contribute. In addition, every `nix` subcommand now has a manual
+  page, documenting every option.
+
+* A new setting that allows **experimental features** to be enabled
+  selectively. This allows us to merge unstable features into Nix more
+  quickly and do more frequent releases.
+
+## Other features
+
+* There are many new `nix` subcommands:
+
+  - `nix develop` is intended to replace `nix-shell`. It has a number
+    of new features:
+
+    * It automatically sets the output environment variables (such as
+      `$out`) to writable locations (such as `./outputs/out`).
+
+    * It can store the environment in a profile. This is useful for
+      offline work.
+
+    * It can run specific phases directly. For instance, `nix develop
+      --build` runs `buildPhase`.
+
+    - It allows dependencies in the Nix store to be "redirected" to
+      arbitrary directories using the `--redirect` flag. This is
+      useful if you want to hack on a package *and* some of its
+      dependencies at the same time.
+
+  - `nix print-dev-env` prints the environment variables and bash
+    functions defined by a derivation. This is useful for users of
+    other shells than bash (especially with `--json`).
+
+  - `nix shell` was previously named `nix run` and is intended to
+    replace `nix-shell -p`, but without the `stdenv` overhead. It
+    simply starts a shell where some packages have been added to
+    `$PATH`.
+
+  - `nix run` (not to be confused with the old subcommand that has
+    been renamed to `nix shell`) runs an "app", a flake output that
+    specifies a command to run, or an eponymous program from a
+    package. For example, `nix run nixpkgs#hello` runs the `hello`
+    program from the `hello` package in `nixpkgs`.
+
+  - `nix flake` is the container for flake-related operations, such as
+    creating a new flake, querying the contents of a flake or updating
+    flake lock files.
+
+  - `nix registry` allows you to query and update the flake registry,
+    which maps identifiers such as `nixpkgs` to concrete flake URLs.
+
+  - `nix profile` is intended to replace `nix-env`. Its main advantage
+    is that it keeps track of the provenance of installed packages
+    (e.g. exactly which flake version a package came from). It also
+    has some helpful subcommands:
+
+    * `nix profile history` shows what packages were added, upgraded
+      or removed between each version of a profile.
+
+    * `nix profile diff-closures` shows the changes between the
+      closures of each version of a profile. This allows you to
+      discover the addition or removal of dependencies or size
+      changes.
+
+    **Warning**: after a profile has been updated using `nix profile`,
+    it is no longer usable with `nix-env`.
+
+  - `nix store diff-closures` shows the differences between the
+    closures of two store paths in terms of the versions and sizes of
+    dependencies in the closures.
+
+  - `nix store make-content-addressable` rewrites an arbitrary closure
+    to make it content-addressed. Such paths can be copied into other
+    stores without requiring signatures.
+
+  - `nix bundle` uses the [`nix-bundle`
+    program](https://github.com/matthewbauer/nix-bundle) to convert a
+    closure into a self-extracting executable.
+
+  - Various other replacements for the old CLI, e.g. `nix store gc`,
+    `nix store delete`, `nix store repair`, `nix nar dump-path`, `nix
+    store prefetch-file`, `nix store prefetch-tarball`, `nix key` and
+    `nix daemon`.
+
+* Nix now has an **evaluation cache** for flake outputs. For example,
+  a second invocation of the command `nix run nixpkgs#firefox` will
+  not need to evaluate the `firefox` attribute because it's already in
+  the evaluation cache. This is made possible by the hermetic
+  evaluation model of flakes.
+
+* The new `--offline` flag disables substituters and causes all
+  locally cached tarballs and repositories to be considered
+  up-to-date.
+
+* The new `--refresh` flag causes all locally cached tarballs and
+  repositories to be considered out-of-date.
+
+* Many `nix` subcommands now have a `--json` option to produce
+  machine-readable output.
+
+* `nix repl` has a new `:doc` command to show documentation about
+  builtin functions (e.g. `:doc builtins.map`).
+
+* Binary cache stores now have an option `index-debug-info` to create
+  an index of DWARF debuginfo files for use by
+  [`dwarffs`](https://github.com/edolstra/dwarffs).
+
+* To support flakes, Nix now has an extensible mechanism for fetching
+  source trees. Currently it has the following backends:
+
+  * Git repositories
+
+  * Mercurial repositories
+
+  * GitHub and GitLab repositories (an optimisation for faster
+    fetching than Git)
+
+  * Tarballs
+
+  * Arbitrary directories
+
+  The fetcher infrastructure is exposed via flake input specifications
+  and via the `fetchTree` built-in.
+
+* **Languages changes**: the only new language feature is that you can
+  now have antiquotations in paths, e.g. `./${foo}` instead of `./. +
+  foo`.
+
+* **New built-in functions**:
+
+  - `builtins.fetchTree` allows fetching a source tree using any
+    backends supported by the fetcher infrastructure. It subsumes the
+    functionality of existing built-ins like `fetchGit`,
+    `fetchMercurial` and `fetchTarball`.
+
+  - `builtins.getFlake` fetches a flake and returns its output
+    attributes. This function should not be used inside flakes! Use
+    flake inputs instead.
+
+  - `builtins.floor` and `builtins.ceil` round a floating-point number
+    down and up, respectively.
+
+* Experimental support for recursive Nix. This means that Nix
+  derivations can now call Nix to build other derivations. This is not
+  in a stable state yet and not well
+  [documented](https://github.com/NixOS/nix/commit/c4d7c76b641d82b2696fef73ce0ac160043c18da).
+
+* The new experimental feature `no-url-literals` disables URL
+  literals. This helps to implement [RFC
+  45](https://github.com/NixOS/rfcs/pull/45).
+
+* Nix now uses `libarchive` to decompress and unpack tarballs and zip
+  files, so `tar` is no longer required.
+
+* The priority of substituters can now be overridden using the
+  `priority` substituter setting (e.g. `--substituters
+  'http://cache.nixos.org?priority=100 daemon?priority=10'`).
+
+* `nix edit` now supports non-derivation attributes, e.g. `nix edit
+  .#nixosConfigurations.bla`.
+
+* The `nix` command now provides command line completion for `bash`,
+  `zsh` and `fish`. Since the support for getting completions is built
+  into `nix`, it's easy to add support for other shells.
+
+* The new `--log-format` flag selects what Nix's output looks like. It
+  defaults to a terse progress indicator. There is a new
+  `internal-json` output format for use by other programs.
+
+* `nix eval` has a new `--apply` flag that applies a function to the
+  evaluation result.
+
+* `nix eval` has a new `--write-to` flag that allows it to write a
+  nested attribute set of string leaves to a corresponding directory
+  tree.
+
+* Memory improvements: many operations that add paths to the store or
+  copy paths between stores now run in constant memory.
+
+* Many `nix` commands now support the flag `--derivation` to operate
+  on a `.drv` file itself instead of its outputs.
+
+* There is a new store called `dummy://` that does not support
+  building or adding paths. This is useful if you want to use the Nix
+  evaluator but don't have a Nix store.
+
+* The `ssh-ng://` store now allows substituting paths on the remote,
+  as `ssh://` already did.
+
+* When auto-calling a function with an ellipsis, all arguments are now
+  passed.
+
+* New `nix-shell` features:
+
+  - It preserves the `PS1` environment variable if
+    `NIX_SHELL_PRESERVE_PROMPT` is set.
+
+  - With `-p`, it passes any `--arg`s as Nixpkgs arguments.
+
+  - Support for structured attributes.
+
+* `nix-prefetch-url` has a new `--executable` flag.
+
+* On `x86_64` systems, [`x86_64` microarchitecture
+  levels](https://lwn.net/Articles/844831/) are mapped to additional
+  system types (e.g. `x86_64-v1-linux`).
+
+* The new `--eval-store` flag allows you to use a different store for
+  evaluation than for building or storing the build result. This is
+  primarily useful when you want to query whether something exists in
+  a read-only store, such as a binary cache:
+
+  ```
+  # nix path-info --json --store https://cache.nixos.org \
+    --eval-store auto nixpkgs#hello
+  ```
+
+  (Here `auto` indicates the local store.)
+
+* The Nix daemon has a new low-latency mechanism for copying
+  closures. This is useful when building on remote stores such as
+  `ssh-ng://`.
+
+* Plugins can now register `nix` subcommands.
+
+## Incompatible changes
+
+* The `nix` command is now marked as an experimental feature. This
+  means that you need to add
+
+  > experimental-features = nix-command
+
+  to your `nix.conf` if you want to use it, or pass
+  `--extra-experimental-features nix-command` on the command line.
+
+* The old `nix run` has been renamed to `nix shell` (and there is a
+  new `nix run` that does something else, as described above).
+
+* It is now an error to modify the `plugin-files` setting via a
+  command-line flag that appears after the first non-flag argument to
+  any command, including a subcommand to `nix`. For example,
   `nix-instantiate default.nix --plugin-files ""` must now become
   `nix-instantiate --plugin-files "" default.nix`.
-  - Plugins that add new `nix` subcommands are now actually respected.
+
+* We no longer release source tarballs. If you want to build from
+  source, please build from the tags in the Git repository.
+
+## Contributors
+
+This release has contributions from
+Adam Höse,
+Albert Safin,
+Alex Kovar,
+Alex Zero,
+Alexander Bantyev,
+Alexandre Esteves,
+Alyssa Ross,
+Anatole Lucet,
+Anders Kaseorg,
+Andreas Rammhold,
+Antoine Eiche,
+Antoine Martin,
+Arnout Engelen,
+Arthur Gautier,
+aszlig,
+Ben Burdette,
+Benjamin Hipple,
+Bernardo Meurer,
+Björn Gohla,
+Bjørn Forsman,
+Bob van der Linden,
+Brian Leung,
+Brian McKenna,
+Brian Wignall,
+Bruce Toll,
+Bryan Richter,
+Calle Rosenquist,
+Calvin Loncaric,
+Carlo Nucera,
+Carlos D'Agostino,
+Chaz Schlarp,
+Christian Höppner,
+Christian Kampka,
+Chua Hou,
+Chuck,
+Cole Helbling,
+Daiderd Jordan,
+Dan Callahan,
+Dani,
+Daniel Fitzpatrick,
+Danila Fedorin,
+Daniël de Kok,
+Danny Bautista,
+DavHau,
+David McFarland,
+Dima,
+Domen Kožar,
+Dominik Schrempf,
+Dominique Martinet,
+dramforever,
+Dustin DeWeese,
+edef,
+Eelco Dolstra,
+Emilio Karakey,
+Emily,
+Eric Culp,
+Ersin Akinci,
+Fabian Möller,
+Farid Zakaria,
+Federico Pellegrin,
+Finn Behrens,
+Florian Franzen,
+Félix Baylac-Jacqué,
+Gabriel Gonzalez,
+Geoff Reedy,
+Georges Dubus,
+Graham Christensen,
+Greg Hale,
+Greg Price,
+Gregor Kleen,
+Gregory Hale,
+Griffin Smith,
+Guillaume Bouchard,
+Harald van Dijk,
+illustris,
+Ivan Zvonimir Horvat,
+Jade,
+Jake Waksbaum,
+jakobrs,
+James Ottaway,
+Jan Tojnar,
+Janne Heß,
+Jaroslavas Pocepko,
+Jarrett Keifer,
+Jeremy Schlatter,
+Joachim Breitner,
+Joe Hermaszewski,
+Joe Pea,
+John Ericson,
+Jonathan Ringer,
+Josef Kemetmüller,
+Joseph Lucas,
+Jude Taylor,
+Julian Stecklina,
+Julien Tanguy,
+Jörg Thalheim,
+Kai Wohlfahrt,
+keke,
+Keshav Kini,
+Kevin Quick,
+Kevin Stock,
+Kjetil Orbekk,
+Krzysztof Gogolewski,
+kvtb,
+Lars Mühmel,
+Leonhard Markert,
+Lily Ballard,
+Linus Heckemann,
+Lorenzo Manacorda,
+Lucas Desgouilles,
+Lucas Franceschino,
+Lucas Hoffmann,
+Luke Granger-Brown,
+Madeline Haraj,
+Marwan Aljubeh,
+Mat Marini,
+Mateusz Piotrowski,
+Matthew Bauer,
+Matthew Kenigsberg,
+Mauricio Scheffer,
+Maximilian Bosch,
+Michael Adler,
+Michael Bishop,
+Michael Fellinger,
+Michael Forney,
+Michael Reilly,
+mlatus,
+Mykola Orliuk,
+Nathan van Doorn,
+Naïm Favier,
+ng0,
+Nick Van den Broeck,
+Nicolas Stig124 Formichella,
+Niels Egberts,
+Niklas Hambüchen,
+Nikola Knezevic,
+oxalica,
+p01arst0rm,
+Pamplemousse,
+Patrick Hilhorst,
+Paul Opiyo,
+Pavol Rusnak,
+Peter Kolloch,
+Philipp Bartsch,
+Philipp Middendorf,
+Piotr Szubiakowski,
+Profpatsch,
+Puck Meerburg,
+Ricardo M. Correia,
+Rickard Nilsson,
+Robert Hensing,
+Robin Gloster,
+Rodrigo,
+Rok Garbas,
+Ronnie Ebrin,
+Rovanion Luckey,
+Ryan Burns,
+Ryan Mulligan,
+Ryne Everett,
+Sam Doshi,
+Sam Lidder,
+Samir Talwar,
+Samuel Dionne-Riel,
+Sebastian Ullrich,
+Sergei Trofimovich,
+Sevan Janiyan,
+Shao Cheng,
+Shea Levy,
+Silvan Mosberger,
+Stefan Frijters,
+Stefan Jaax,
+sternenseemann,
+Steven Shaw,
+Stéphan Kochen,
+SuperSandro2000,
+Suraj Barkale,
+Taeer Bar-Yam,
+Thomas Churchman,
+Théophane Hufschmitt,
+Timothy DeHerrera,
+Timothy Klim,
+Tobias Möst,
+Tobias Pflug,
+Tom Bereknyei,
+Travis A. Everett,
+Ujjwal Jain,
+Vladimír Čunát,
+Wil Taylor,
+Will Dietz,
+Yaroslav Bolyukin,
+Yestin L. Harrison,
+YI,
+Yorick van Pelt,
+Yuriy Taraday and
+zimbatm.

src/libcmd/command.cc

@@ -203,10 +203,10 @@ void MixProfile::updateProfile(const BuiltPaths & buildables)
 
     for (auto & buildable : buildables) {
         std::visit(overloaded {
-            [&](BuiltPath::Opaque bo) {
+            [&](const BuiltPath::Opaque & bo) {
                 result.push_back(bo.path);
             },
-            [&](BuiltPath::Built bfd) {
+            [&](const BuiltPath::Built & bfd) {
                 for (auto & output : bfd.outputs) {
                     result.push_back(output.second);
                 }

src/libcmd/installables.cc

@@ -697,13 +697,13 @@ std::shared_ptr<Installable> SourceExprCommand::parseInstallable(
 BuiltPaths getBuiltPaths(ref<Store> evalStore, ref<Store> store, const DerivedPaths & hopefullyBuiltPaths)
 {
     BuiltPaths res;
-    for (auto & b : hopefullyBuiltPaths)
+    for (const auto & b : hopefullyBuiltPaths)
         std::visit(
             overloaded{
-                [&](DerivedPath::Opaque bo) {
+                [&](const DerivedPath::Opaque & bo) {
                     res.push_back(BuiltPath::Opaque{bo.path});
                 },
-                [&](DerivedPath::Built bfd) {
+                [&](const DerivedPath::Built & bfd) {
                     OutputPathMap outputs;
                     auto drv = evalStore->readDerivation(bfd.drvPath);
                     auto outputHashes = staticOutputHashes(*evalStore, drv); // FIXME: expensive
@@ -823,10 +823,10 @@ StorePathSet toDerivations(
 {
     StorePathSet drvPaths;
 
-    for (auto & i : installables)
+    for (const auto & i : installables)
-        for (auto & b : i->toDerivedPaths())
+        for (const auto & b : i->toDerivedPaths())
             std::visit(overloaded {
-                [&](DerivedPath::Opaque bo) {
+                [&](const DerivedPath::Opaque & bo) {
                     if (!useDeriver)
                         throw Error("argument '%s' did not evaluate to a derivation", i->what());
                     auto derivers = store->queryValidDerivers(bo.path);
@@ -835,7 +835,7 @@ StorePathSet toDerivations(
                     // FIXME: use all derivers?
                     drvPaths.insert(*derivers.begin());
                 },
-                [&](DerivedPath::Built bfd) {
+                [&](const DerivedPath::Built & bfd) {
                     drvPaths.insert(bfd.drvPath);
                 },
             }, b.raw());

src/libexpr/primops.cc

@@ -1180,7 +1180,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
         // hash per output.
         auto hashModulo = hashDerivationModulo(*state.store, Derivation(drv), true);
         std::visit(overloaded {
-            [&](Hash h) {
+            [&](Hash & h) {
                 for (auto & i : outputs) {
                     auto outPath = state.store->makeOutputPath(i, h, drvName);
                     drv.env[i] = state.store->printStorePath(outPath);
@@ -1192,11 +1192,11 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
                         });
                 }
             },
-            [&](CaOutputHashes) {
+            [&](CaOutputHashes &) {
                 // Shouldn't happen as the toplevel derivation is not CA.
                 assert(false);
             },
-            [&](DeferredHash _) {
+            [&](DeferredHash &) {
                 for (auto & i : outputs) {
                     drv.outputs.insert_or_assign(i,
                         DerivationOutput {

src/libstore/build/entry-points.cc

@@ -11,12 +11,12 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
     Worker worker(*this, evalStore ? *evalStore : *this);
 
     Goals goals;
-    for (auto & br : reqs) {
+    for (const auto & br : reqs) {
         std::visit(overloaded {
-            [&](DerivedPath::Built bfd) {
+            [&](const DerivedPath::Built & bfd) {
                 goals.insert(worker.makeDerivationGoal(bfd.drvPath, bfd.outputs, buildMode));
             },
-            [&](DerivedPath::Opaque bo) {
+            [&](const DerivedPath::Opaque & bo) {
                 goals.insert(worker.makePathSubstitutionGoal(bo.path, buildMode == bmRepair ? Repair : NoRepair));
             },
         }, br.raw());

src/libstore/build/local-derivation-goal.cc

@@ -1094,10 +1094,10 @@ void LocalDerivationGoal::writeStructuredAttrs()
 static StorePath pathPartOfReq(const DerivedPath & req)
 {
     return std::visit(overloaded {
-        [&](DerivedPath::Opaque bo) {
+        [&](const DerivedPath::Opaque & bo) {
             return bo.path;
         },
-        [&](DerivedPath::Built bfd)  {
+        [&](const DerivedPath::Built & bfd)  {
             return bfd.drvPath;
         },
     }, req.raw());
@@ -2155,8 +2155,8 @@ void LocalDerivationGoal::registerOutputs()
                 /* Since we'll use the already installed versions of these, we
                    can treat them as leaves and ignore any references they
                    have. */
-                [&](AlreadyRegistered _) { return StringSet {}; },
+                [&](const AlreadyRegistered &) { return StringSet {}; },
-                [&](PerhapsNeedToRegister refs) {
+                [&](const PerhapsNeedToRegister & refs) {
                     StringSet referencedOutputs;
                     /* FIXME build inverted map up front so no quadratic waste here */
                     for (auto & r : refs.refs)
@@ -2192,11 +2192,11 @@ void LocalDerivationGoal::registerOutputs()
         };
 
         std::optional<StorePathSet> referencesOpt = std::visit(overloaded {
-            [&](AlreadyRegistered skippedFinalPath) -> std::optional<StorePathSet> {
+            [&](const AlreadyRegistered & skippedFinalPath) -> std::optional<StorePathSet> {
                 finish(skippedFinalPath.path);
                 return std::nullopt;
             },
-            [&](PerhapsNeedToRegister r) -> std::optional<StorePathSet> {
+            [&](const PerhapsNeedToRegister & r) -> std::optional<StorePathSet> {
                 return r.refs;
             },
         }, outputReferencesIfUnregistered.at(outputName));
@@ -2306,7 +2306,7 @@ void LocalDerivationGoal::registerOutputs()
         };
 
         ValidPathInfo newInfo = std::visit(overloaded {
-            [&](DerivationOutputInputAddressed output) {
+            [&](const DerivationOutputInputAddressed & output) {
                 /* input-addressed case */
                 auto requiredFinalPath = output.path;
                 /* Preemptively add rewrite rule for final hash, as that is
@@ -2322,14 +2322,14 @@ void LocalDerivationGoal::registerOutputs()
                 static_cast<PathReferences<StorePath> &>(newInfo0) = rewriteRefs();
                 return newInfo0;
             },
-            [&](DerivationOutputCAFixed dof) {
+            [&](const DerivationOutputCAFixed & dof) {
                 auto newInfo0 = newInfoFromCA(DerivationOutputCAFloating {
                     .method = dof.hash.method,
                     .hashType = dof.hash.hash.type,
                 });
 
                 /* Check wanted hash */
-                Hash & wanted = dof.hash.hash;
+                const Hash & wanted = dof.hash.hash;
                 assert(newInfo0.ca);
                 auto got = getContentAddressHash(*newInfo0.ca);
                 if (wanted != got) {

src/libstore/content-address.cc

@@ -31,11 +31,11 @@ std::string makeFixedOutputCA(FileIngestionMethod method, const Hash & hash)
 std::string renderContentAddress(ContentAddress ca)
 {
     return std::visit(overloaded {
-        [](TextHash th) {
+        [](TextHash & th) {
             return "text:"
                 + th.hash.to_string(Base32, true);
         },
-        [](FixedOutputHash fsh) {
+        [](FixedOutputHash & fsh) {
             return "fixed:"
                 + makeFileIngestionPrefix(fsh.method)
                 + fsh.hash.to_string(Base32, true);
@@ -46,10 +46,10 @@ std::string renderContentAddress(ContentAddress ca)
 std::string renderContentAddressMethod(ContentAddressMethod cam)
 {
     return std::visit(overloaded {
-        [](TextHashMethod &th) {
+        [](TextHashMethod & th) {
             return std::string{"text:"} + printHashType(htSHA256);
         },
-        [](FixedOutputHashMethod &fshm) {
+        [](FixedOutputHashMethod & fshm) {
             return "fixed:" + makeFileIngestionPrefix(fshm.fileIngestionMethod) + printHashType(fshm.hashType);
         }
     }, cam);
@@ -107,12 +107,12 @@ ContentAddress parseContentAddress(std::string_view rawCa) {
 
     return std::visit(
         overloaded {
-            [&](TextHashMethod thm) {
+            [&](TextHashMethod & thm) {
                 return ContentAddress(TextHash {
                     .hash = Hash::parseNonSRIUnprefixed(rest, htSHA256)
                 });
             },
-            [&](FixedOutputHashMethod fohMethod) {
+            [&](FixedOutputHashMethod & fohMethod) {
                 return ContentAddress(FixedOutputHash {
                     .method = fohMethod.fileIngestionMethod,
                     .hash = Hash::parseNonSRIUnprefixed(rest, std::move(fohMethod.hashType)),
@@ -140,10 +140,10 @@ std::string renderContentAddress(std::optional<ContentAddress> ca)
 Hash getContentAddressHash(const ContentAddress & ca)
 {
     return std::visit(overloaded {
-        [](TextHash th) {
+        [](const TextHash & th) {
             return th.hash;
         },
-        [](FixedOutputHash fsh) {
+        [](const FixedOutputHash & fsh) {
             return fsh.hash;
         },
     }, ca);
@@ -151,10 +151,10 @@ Hash getContentAddressHash(const ContentAddress & ca)
 
 ContentAddressWithReferences caWithoutRefs(const ContentAddress & ca) {
     return std::visit(overloaded {
-        [&](TextHash h) -> ContentAddressWithReferences {
+        [&](const TextHash & h) -> ContentAddressWithReferences {
             return TextInfo { h, {}};
         },
-        [&](FixedOutputHash h) -> ContentAddressWithReferences {
+        [&](const FixedOutputHash & h) -> ContentAddressWithReferences {
             return FixedOutputInfo { h, {}};
         },
     }, ca);

src/libstore/daemon.cc

@@ -395,13 +395,13 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
                 FramedSource source(from);
                 // TODO this is essentially RemoteStore::addCAToStore. Move it up to Store.
                 return std::visit(overloaded {
-                    [&](TextHashMethod &_) {
+                    [&](TextHashMethod &) {
                         // We could stream this by changing Store
                         std::string contents = source.drain();
                         auto path = store->addTextToStore(name, contents, refs, repair);
                         return store->queryPathInfo(path);
                     },
-                    [&](FixedOutputHashMethod &fohm) {
+                    [&](FixedOutputHashMethod & fohm) {
                         if (!refs.empty())
                             throw UnimplementedError("cannot yet have refs with flat or nar-hashed data");
                         auto path = store->addToStoreFromDump(source, name, fohm.fileIngestionMethod, fohm.hashType, repair);

src/libstore/derivations.cc

@@ -10,18 +10,18 @@ namespace nix {
 std::optional<StorePath> DerivationOutput::path(const Store & store, std::string_view drvName, std::string_view outputName) const
 {
     return std::visit(overloaded {
-        [](DerivationOutputInputAddressed doi) -> std::optional<StorePath> {
+        [](const DerivationOutputInputAddressed & doi) -> std::optional<StorePath> {
             return { doi.path };
         },
-        [&](DerivationOutputCAFixed dof) -> std::optional<StorePath> {
+        [&](const DerivationOutputCAFixed & dof) -> std::optional<StorePath> {
             return {
                 dof.path(store, drvName, outputName)
             };
         },
-        [](DerivationOutputCAFloating dof) -> std::optional<StorePath> {
+        [](const DerivationOutputCAFloating & dof) -> std::optional<StorePath> {
             return std::nullopt;
         },
-        [](DerivationOutputDeferred) -> std::optional<StorePath> {
+        [](const DerivationOutputDeferred &) -> std::optional<StorePath> {
             return std::nullopt;
         },
     }, output);
@@ -332,22 +332,22 @@ string Derivation::unparse(const Store & store, bool maskOutputs,
         if (first) first = false; else s += ',';
         s += '('; printUnquotedString(s, i.first);
         std::visit(overloaded {
-            [&](DerivationOutputInputAddressed doi) {
+            [&](const DerivationOutputInputAddressed & doi) {
                 s += ','; printUnquotedString(s, maskOutputs ? "" : store.printStorePath(doi.path));
                 s += ','; printUnquotedString(s, "");
                 s += ','; printUnquotedString(s, "");
             },
-            [&](DerivationOutputCAFixed dof) {
+            [&](const DerivationOutputCAFixed & dof) {
                 s += ','; printUnquotedString(s, maskOutputs ? "" : store.printStorePath(dof.path(store, name, i.first)));
                 s += ','; printUnquotedString(s, dof.hash.printMethodAlgo());
                 s += ','; printUnquotedString(s, dof.hash.hash.to_string(Base16, false));
             },
-            [&](DerivationOutputCAFloating dof) {
+            [&](const DerivationOutputCAFloating & dof) {
                 s += ','; printUnquotedString(s, "");
                 s += ','; printUnquotedString(s, makeFileIngestionPrefix(dof.method) + printHashType(dof.hashType));
                 s += ','; printUnquotedString(s, "");
             },
-            [&](DerivationOutputDeferred) {
+            [&](const DerivationOutputDeferred &) {
                 s += ','; printUnquotedString(s, "");
                 s += ','; printUnquotedString(s, "");
                 s += ','; printUnquotedString(s, "");
@@ -420,13 +420,13 @@ DerivationType BasicDerivation::type() const
     std::optional<HashType> floatingHashType;
     for (auto & i : outputs) {
         std::visit(overloaded {
-            [&](DerivationOutputInputAddressed _) {
+            [&](const DerivationOutputInputAddressed &) {
                inputAddressedOutputs.insert(i.first);
             },
-            [&](DerivationOutputCAFixed _) {
+            [&](const DerivationOutputCAFixed &) {
                 fixedCAOutputs.insert(i.first);
             },
-            [&](DerivationOutputCAFloating dof) {
+            [&](const DerivationOutputCAFloating & dof) {
                 floatingCAOutputs.insert(i.first);
                 if (!floatingHashType) {
                     floatingHashType = dof.hashType;
@@ -435,7 +435,7 @@ DerivationType BasicDerivation::type() const
                         throw Error("All floating outputs must use the same hash type");
                 }
             },
-            [&](DerivationOutputDeferred _) {
+            [&](const DerivationOutputDeferred &) {
                deferredIAOutputs.insert(i.first);
             },
         }, i.second.output);
@@ -538,15 +538,15 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m
         const auto & res = pathDerivationModulo(store, i.first);
         std::visit(overloaded {
             // Regular non-CA derivation, replace derivation
-            [&](Hash drvHash) {
+            [&](const Hash & drvHash) {
                 inputs2.insert_or_assign(drvHash.to_string(Base16, false), i.second);
             },
-            [&](DeferredHash deferredHash) {
+            [&](const DeferredHash & deferredHash) {
                 isDeferred = true;
                 inputs2.insert_or_assign(deferredHash.hash.to_string(Base16, false), i.second);
             },
             // CA derivation's output hashes
-            [&](CaOutputHashes outputHashes) {
+            [&](const CaOutputHashes & outputHashes) {
                 std::set<std::string> justOut = { "out" };
                 for (auto & output : i.second) {
                     /* Put each one in with a single "out" output.. */
@@ -572,17 +572,17 @@ std::map<std::string, Hash> staticOutputHashes(Store & store, const Derivation &
 {
     std::map<std::string, Hash> res;
     std::visit(overloaded {
-        [&](Hash drvHash) {
+        [&](const Hash & drvHash) {
             for (auto & outputName : drv.outputNames()) {
                 res.insert({outputName, drvHash});
             }
         },
-        [&](DeferredHash deferredHash) {
+        [&](const DeferredHash & deferredHash) {
             for (auto & outputName : drv.outputNames()) {
                 res.insert({outputName, deferredHash.hash});
             }
         },
-        [&](CaOutputHashes outputHashes) {
+        [&](const CaOutputHashes & outputHashes) {
             res = outputHashes;
         },
     }, hashDerivationModulo(store, drv, true));
@@ -666,22 +666,22 @@ void writeDerivation(Sink & out, const Store & store, const BasicDerivation & dr
     for (auto & i : drv.outputs) {
         out << i.first;
         std::visit(overloaded {
-            [&](DerivationOutputInputAddressed doi) {
+            [&](const DerivationOutputInputAddressed & doi) {
                 out << store.printStorePath(doi.path)
                     << ""
                     << "";
             },
-            [&](DerivationOutputCAFixed dof) {
+            [&](const DerivationOutputCAFixed & dof) {
                 out << store.printStorePath(dof.path(store, drv.name, i.first))
                     << dof.hash.printMethodAlgo()
                     << dof.hash.hash.to_string(Base16, false);
             },
-            [&](DerivationOutputCAFloating dof) {
+            [&](const DerivationOutputCAFloating & dof) {
                 out << ""
                     << (makeFileIngestionPrefix(dof.method) + printHashType(dof.hashType))
                     << "";
             },
-            [&](DerivationOutputDeferred) {
+            [&](const DerivationOutputDeferred &) {
                 out << ""
                     << ""
                     << "";

src/libstore/derived-path.cc

@@ -24,8 +24,8 @@ StorePathSet BuiltPath::outPaths() const
 {
     return std::visit(
         overloaded{
-            [](BuiltPath::Opaque p) { return StorePathSet{p.path}; },
+            [](const BuiltPath::Opaque & p) { return StorePathSet{p.path}; },
-            [](BuiltPath::Built b) {
+            [](const BuiltPath::Built & b) {
                 StorePathSet res;
                 for (auto & [_, path] : b.outputs)
                     res.insert(path);
@@ -94,8 +94,8 @@ RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const
     RealisedPath::Set res;
     std::visit(
         overloaded{
-            [&](BuiltPath::Opaque p) { res.insert(p.path); },
+            [&](const BuiltPath::Opaque & p) { res.insert(p.path); },
-            [&](BuiltPath::Built p) {
+            [&](const BuiltPath::Built & p) {
                 auto drvHashes =
                     staticOutputHashes(store, store.readDerivation(p.drvPath));
                 for (auto& [outputName, outputPath] : p.outputs) {

src/libstore/legacy-ssh-store.cc

@@ -247,6 +247,10 @@ private:
             conn.to
                 << settings.buildRepeat
                 << settings.enforceDeterminism;
+
+        if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 7) {
+            conn.to << ((int) settings.keepFailed);
+        }
     }
 
 public:
@@ -289,10 +293,10 @@ public:
         for (auto & p : drvPaths) {
             auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p);
             std::visit(overloaded {
-                [&](StorePathWithOutputs s) {
+                [&](const StorePathWithOutputs & s) {
                     ss.push_back(s.to_string(*this));
                 },
-                [&](StorePath drvPath) {
+                [&](const StorePath & drvPath) {
                     throw Error("wanted to fetch '%s' but the legacy ssh protocol doesn't support merely substituting drv files via the build paths command. It would build them instead. Try using ssh-ng://", printStorePath(drvPath));
                 },
             }, sOrDrvPath);

src/libstore/local-store.cc

@@ -681,7 +681,7 @@ void LocalStore::checkDerivationOutputs(const StorePath & drvPath, const Derivat
     std::optional<Hash> h;
     for (auto & i : drv.outputs) {
         std::visit(overloaded {
-            [&](DerivationOutputInputAddressed doia) {
+            [&](const DerivationOutputInputAddressed & doia) {
                 if (!h) {
                     // somewhat expensive so we do lazily
                     auto temp = hashDerivationModulo(*this, drv, true);
@@ -693,14 +693,14 @@ void LocalStore::checkDerivationOutputs(const StorePath & drvPath, const Derivat
                         printStorePath(drvPath), printStorePath(doia.path), printStorePath(recomputed));
                 envHasRightPath(doia.path, i.first);
             },
-            [&](DerivationOutputCAFixed dof) {
+            [&](const DerivationOutputCAFixed & dof) {
                 StorePath path = makeFixedOutputPath(drvName, { dof.hash, {} });
                 envHasRightPath(path, i.first);
             },
-            [&](DerivationOutputCAFloating _) {
+            [&](const DerivationOutputCAFloating &) {
                 /* Nothing to check */
             },
-            [&](DerivationOutputDeferred) {
+            [&](const DerivationOutputDeferred &) {
             },
         }, i.second.output);
     }

src/libstore/misc.cc

@@ -165,7 +165,7 @@ void Store::queryMissing(const std::vector<DerivedPath> & targets,
         }
 
         std::visit(overloaded {
-          [&](DerivedPath::Built bfd) {
+          [&](const DerivedPath::Built & bfd) {
             if (!isValidPath(bfd.drvPath)) {
                 // FIXME: we could try to substitute the derivation.
                 auto state(state_.lock());
@@ -198,7 +198,7 @@ void Store::queryMissing(const std::vector<DerivedPath> & targets,
                 mustBuildDrv(bfd.drvPath, *drv);
 
           },
-          [&](DerivedPath::Opaque bo) {
+          [&](const DerivedPath::Opaque & bo) {
 
             if (isValidPath(bo.path)) return;
 

src/libstore/path-with-outputs.cc

@@ -31,14 +31,14 @@ std::vector<DerivedPath> toDerivedPaths(const std::vector<StorePathWithOutputs>
 std::variant<StorePathWithOutputs, StorePath> StorePathWithOutputs::tryFromDerivedPath(const DerivedPath & p)
 {
     return std::visit(overloaded {
-        [&](DerivedPath::Opaque bo) -> std::variant<StorePathWithOutputs, StorePath> {
+        [&](const DerivedPath::Opaque & bo) -> std::variant<StorePathWithOutputs, StorePath> {
             if (bo.path.isDerivation()) {
                 // drv path gets interpreted as "build", not "get drv file itself"
                 return bo.path;
             }
             return StorePathWithOutputs { bo.path };
         },
-        [&](DerivedPath::Built bfd) -> std::variant<StorePathWithOutputs, StorePath> {
+        [&](const DerivedPath::Built & bfd) -> std::variant<StorePathWithOutputs, StorePath> {
             return StorePathWithOutputs { bfd.drvPath, bfd.outputs };
         },
     }, p.raw());

src/libstore/remote-store.cc

@@ -529,13 +529,13 @@ ref<const ValidPathInfo> RemoteStore::addCAToStore(
         if (repair) throw Error("repairing is not supported when building through the Nix daemon protocol < 1.25");
 
         std::visit(overloaded {
-            [&](TextHashMethod thm) -> void {
+            [&](const TextHashMethod & thm) -> void {
                 std::string s = dump.drain();
                 conn->to << wopAddTextToStore << name << s;
                 worker_proto::write(*this, conn->to, references);
                 conn.processStderr();
             },
-            [&](FixedOutputHashMethod fohm) -> void {
+            [&](const FixedOutputHashMethod & fohm) -> void {
                 conn->to
                     << wopAddToStore
                     << name
@@ -706,10 +706,10 @@ static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, cons
         for (auto & p : reqs) {
             auto sOrDrvPath = StorePathWithOutputs::tryFromDerivedPath(p);
             std::visit(overloaded {
-                [&](StorePathWithOutputs s) {
+                [&](const StorePathWithOutputs & s) {
                     ss.push_back(s.to_string(store));
                 },
-                [&](StorePath drvPath) {
+                [&](const StorePath & drvPath) {
                     throw Error("trying to request '%s', but daemon protocol %d.%d is too old (< 1.29) to request a derivation file",
                         store.printStorePath(drvPath),
                         GET_PROTOCOL_MAJOR(conn->daemonVersion),

src/libstore/serve-protocol.hh

@@ -5,7 +5,7 @@ namespace nix {
 #define SERVE_MAGIC_1 0x390c9deb
 #define SERVE_MAGIC_2 0x5452eecb
 
-#define SERVE_PROTOCOL_VERSION (2 << 8 | 6)
+#define SERVE_PROTOCOL_VERSION (2 << 8 | 7)
 #define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
 #define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
 

src/libstore/store-api.cc

@@ -208,10 +208,10 @@ StorePath Store::makeFixedOutputPathFromCA(const StorePathDescriptor & desc) con
 {
     // New template
     return std::visit(overloaded {
-        [&](TextInfo ti) {
+        [&](const TextInfo & ti) {
             return makeTextPath(desc.name, ti);
         },
-        [&](FixedOutputInfo foi) {
+        [&](const FixedOutputInfo & foi) {
             return makeFixedOutputPath(desc.name, foi);
         }
     }, desc.info);
@@ -1150,13 +1150,13 @@ std::optional<StorePathDescriptor> ValidPathInfo::fullStorePathDescriptorOpt() c
     return StorePathDescriptor {
         .name = std::string { path.name() },
         .info = std::visit(overloaded {
-            [&](TextHash th) {
+            [&](const TextHash & th) {
                 TextInfo info { th };
                 assert(!hasSelfReference);
                 info.references = references;
                 return ContentAddressWithReferences { info };
             },
-            [&](FixedOutputHash foh) {
+            [&](const FixedOutputHash & foh) {
                 FixedOutputInfo info { foh };
                 info.references = static_cast<PathReferences<StorePath>>(*this);
                 return ContentAddressWithReferences { info };
@@ -1218,11 +1218,11 @@ ValidPathInfo::ValidPathInfo(
       , narHash(narHash)
 {
     std::visit(overloaded {
-        [this](TextInfo ti) {
+        [this](const TextInfo & ti) {
             this->references = ti.references;
             this->ca = TextHash { std::move(ti) };
         },
-        [this](FixedOutputInfo foi) {
+        [this](const FixedOutputInfo & foi) {
             *(static_cast<PathReferences<StorePath> *>(this)) = foi.references;
             this->ca = FixedOutputHash { (FixedOutputHash) std::move(foi) };
         },

src/nix-store/nix-store.cc

@@ -807,6 +807,9 @@ static void opServe(Strings opFlags, Strings opArgs)
             settings.enforceDeterminism = readInt(in);
             settings.runDiffHook = true;
         }
+        if (GET_PROTOCOL_MINOR(clientVersion) >= 7) {
+            settings.keepFailed = (bool) readInt(in);
+        }
         settings.printRepeatedBuilds = false;
     };
 

src/nix/build.cc

@@ -66,12 +66,12 @@ struct CmdBuild : InstallablesCommand, MixDryRun, MixJSON, MixProfile
                 for (const auto & [_i, buildable] : enumerate(buildables)) {
                     auto i = _i;
                     std::visit(overloaded {
-                        [&](BuiltPath::Opaque bo) {
+                        [&](const BuiltPath::Opaque & bo) {
                             std::string symlink = outLink;
                             if (i) symlink += fmt("-%d", i);
                             store2->addPermRoot(bo.path, absPath(symlink));
                         },
-                        [&](BuiltPath::Built bfd) {
+                        [&](const BuiltPath::Built & bfd) {
                             for (auto & output : bfd.outputs) {
                                 std::string symlink = outLink;
                                 if (i) symlink += fmt("-%d", i);

src/nix/log.cc

@@ -35,10 +35,10 @@ struct CmdLog : InstallableCommand
         RunPager pager;
         for (auto & sub : subs) {
             auto log = std::visit(overloaded {
-                [&](DerivedPath::Opaque bo) {
+                [&](const DerivedPath::Opaque & bo) {
                     return sub->getBuildLog(bo.path);
                 },
-                [&](DerivedPath::Built bfd) {
+                [&](const DerivedPath::Built & bfd) {
                     return sub->getBuildLog(bfd.drvPath);
                 },
             }, b.raw());

src/nix/profile.cc

@@ -269,11 +269,11 @@ struct CmdProfileInstall : InstallablesCommand, MixDefaultProfile
                     ProfileElement element;
 
                     std::visit(overloaded {
-                        [&](BuiltPath::Opaque bo) {
+                        [&](const BuiltPath::Opaque & bo) {
                             pathsToBuild.push_back(bo);
                             element.storePaths.insert(bo.path);
                         },
-                        [&](BuiltPath::Built bfd) {
+                        [&](const BuiltPath::Built & bfd) {
                             // TODO: Why are we querying if we know the output
                             // names already? Is it just to figure out what the
                             // default one is?

src/nix/show-derivation.cc

@@ -65,18 +65,18 @@ struct CmdShowDerivation : InstallablesCommand
                     auto & outputName = _outputName; // work around clang bug
                     auto outputObj { outputsObj.object(outputName) };
                     std::visit(overloaded {
-                        [&](DerivationOutputInputAddressed doi) {
+                        [&](const DerivationOutputInputAddressed & doi) {
                             outputObj.attr("path", store->printStorePath(doi.path));
                         },
-                        [&](DerivationOutputCAFixed dof) {
+                        [&](const DerivationOutputCAFixed & dof) {
                             outputObj.attr("path", store->printStorePath(dof.path(*store, drv.name, outputName)));
                             outputObj.attr("hashAlgo", dof.hash.printMethodAlgo());
                             outputObj.attr("hash", dof.hash.hash.to_string(Base16, false));
                         },
-                        [&](DerivationOutputCAFloating dof) {
+                        [&](const DerivationOutputCAFloating & dof) {
                             outputObj.attr("hashAlgo", makeFileIngestionPrefix(dof.method) + printHashType(dof.hashType));
                         },
-                        [&](DerivationOutputDeferred) {},
+                        [&](const DerivationOutputDeferred &) {},
                     }, output.output);
                 }
             }

tests/build-remote.sh

@@ -53,3 +53,16 @@ nix path-info --store $TEST_ROOT/machine3 --all \
   | grep -v builder-build-remote-input-1.sh \
   | grep -v builder-build-remote-input-2.sh \
   | grep builder-build-remote-input-3.sh
+
+# Behavior of keep-failed
+out="$(nix-build 2>&1 failing.nix \
+  --builders "$(join_by '; ' "${builders[@]}")"  \
+  --keep-failed \
+  --store $TEST_ROOT/machine0 \
+  -j0 \
+  --arg busybox $busybox)" || true
+
+[[ "$out" =~ .*"note: keeping build directory".* ]]
+
+build_dir="$(grep "note: keeping build" <<< "$out" | sed -E "s/^(.*)note: keeping build directory '(.*)'(.*)$/\2/")"
+[[ "foo" = $(<"$build_dir"/bar) ]]

tests/failing.nix

@@ -0,0 +1,22 @@
+{ busybox }:
+with import ./config.nix;
+let
+
+  mkDerivation = args:
+    derivation ({
+      inherit system;
+      builder = busybox;
+      args = ["sh" "-e" args.builder or (builtins.toFile "builder-${args.name}.sh" "if [ -e .attrs.sh ]; then source .attrs.sh; fi; eval \"$buildCommand\"")];
+    } // removeAttrs args ["builder" "meta"])
+    // { meta = args.meta or {}; };
+in
+{
+
+  failing = mkDerivation {
+    name = "failing";
+    buildCommand = ''
+      echo foo > bar
+      exit 1
+    '';
+  };
+}