Commit graph

9777 commits

Author SHA1 Message Date
regnat 4d45839499 Fix the detection of already built drv outputs
PRs #4370 and #4348 had a bad interaction in that the second broke the fist
one in a not trivial way.

The issue was that since #4348 the logic for detecting whether a
derivation output is already built requires some logic that was specific
to the `LocalStore`.

It happens though that most of this logic could be upstreamed to any `Store`,
which is what this commit does.
2020-12-17 11:35:24 +01:00
Eelco Dolstra ae3c3e3bb2
Merge pull request #4370 from NixOS/ca/more-precise-build-noop
Better detect when `buildPaths` would be a no-op
2020-12-16 14:54:04 +01:00
Eelco Dolstra fdc8e7cbe7
Merge pull request #4373 from NixOS/ca/fix-queryPartialDrvOutputMap-when-no-derivation
Don't ignore an absent drv file in queryPartialDrvOutputMap
2020-12-16 14:52:39 +01:00
regnat cac8d5b742 Don't ignore an absent drv file in queryPartialDrvOutputMap
This ignore was here because `queryPartialDrvOutputMap` was used both
1. as a cache to avoid having to re-read the derivation (when gc-ing for
example), and
2. as the source of truth for ca realisations

The use-case 2. required it to be able to work even when the derivation
wasn't there anymore (see https://github.com/NixOS/nix/issues/4138).
However, this use-case is now handled by `queryRealisation`, meaning
that we can safely error out if the derivation isn't there anymore
2020-12-16 13:36:17 +01:00
Eelco Dolstra 3765174691
Merge pull request #4348 from NixOS/ca/use-hashmodulo
Use the hash modulo in the derivation outputs
2020-12-16 12:48:44 +01:00
Eelco Dolstra c64208d6f8
Merge pull request #4371 from NixOS/ca/fix-remote-store-registerDrvOutput
Fix BinaryCacheStore::registerDrvOutput
2020-12-16 12:47:32 +01:00
regnat 962b82ef25 Fix BinaryCacheStore::registerDrvOutput
Was crashing because coercing a json document into a string is only
valid if the json is a string, otherwise we need to call `.dump()`
2020-12-16 10:54:36 +01:00
regnat 6e899278d3 Better detect when buildPaths would be a no-op
`buildPaths` can be called even for stores where it's not defined in case it's
bound to be a no-op.
The “no-op detection” mechanism was only detecting the case wher `buildPaths`
was called on a set of (non-drv) paths that were already present on the store.

This commit extends this mechanism to also detect the case where `buildPaths`
is called on a set of derivation outputs which are already built on the store.

This only works with the ca-derivations flag. It could be possible to
extend this to also work without it, but it would add quite a bit of
complexity, and it's not used without it anyways.
2020-12-16 10:36:16 +01:00
Eelco Dolstra e3ddffb27e
Merge pull request #4361 from tweag/fix-binary-caches-addTextToStore
Fix `addTextToStore` for binary caches
2020-12-15 12:23:10 +01:00
regnat 44c3fbc6e0 Fix addTextToStore for binary caches
Because of a too eager refactoring, `addTextToStore` used to throw an
error because the input wasn't a valid nar.

Partially revert that refactoring to wrap the text into a proper nar
(using `dumpString`) to make this method work again
2020-12-15 09:38:19 +01:00
Eelco Dolstra f2f60bf5d6
Merge pull request #4330 from NixOS/ca/properly-store-outputs
Properly store the outputs of CA derivations − take 2
2020-12-14 15:01:23 +01:00
Eelco Dolstra 27b5ff354e
Merge pull request #4351 from Ma27/json-errtrace
primops/fromJSON: add error position in case of parse error
2020-12-14 11:27:44 +01:00
Maximilian Bosch f890830b33
primops/fromJSON: add error position in case of parse error
This makes it easier to track down where invalid JSON was passed to
`builtins.fromJSON`.
2020-12-13 13:55:32 +01:00
Eelco Dolstra 92438c70d2
Merge pull request #4352 from jonringer/allow-private-caches
treat s3 permission errors as file-not-found
2020-12-13 13:39:20 +01:00
regnat e9b39f6004 Restrict the operations on drv outputs in recursive Nix
There's currently no way to properly filter them, so disallow them
altogether instead.
2020-12-11 21:17:25 +01:00
regnat bab1cda0e6 Use the hash modulo in the derivation outputs
Rather than storing the derivation outputs as `drvPath!outputName` internally,
store them as `drvHashModulo!outputName` (or `outputHash!outputName` for
fixed-output derivations).

This makes the storage slightly more opaque, but enables an earlier
cutoff in cases where a fixed-output dependency changes (but keeps the
same output hash) − same as what we already do for input-addressed
derivations.
2020-12-11 21:17:23 +01:00
regnat 8914e01e37 Store the realisations as JSON in the binary cache
Fix #4332
2020-12-11 21:05:09 +01:00
regnat 3ac9d74eb1 Rework the db schema for derivation outputs
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes #4138)
2020-12-11 20:41:32 +01:00
regnat 58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Michael Bishop 63b3536f50
treat s3 permission errors as file-not-found
Signed-off-by: Jonathan Ringer <jonringer117@gmail.com>
2020-12-11 09:49:24 -08:00
Eelco Dolstra 9c143c411b
Merge pull request #4350 from NixOS/ca/fix-build-with-nix-command
Fix the `nix` command with CA derivations
2020-12-11 15:01:49 +01:00
regnat eb45308109 Fix the nix command with CA derivations
Prevents a crash because most `nix` subcommands assumed that derivations
know their output path, which isn't the case for CA derivations
2020-12-11 10:28:09 +01:00
Eelco Dolstra c6a1bcd0ec nix store make-content-addressable: Show rewritten path 2020-12-10 17:11:56 +01:00
Eelco Dolstra a8f533b664 Add lvlNotice log level
This is like syslog's LOG_NOTICE: "normal, but significant,
condition".
2020-12-10 16:41:24 +01:00
Eelco Dolstra 29fbc3cf31
Merge pull request #4343 from tweag/fix-osx-ci
Use no substituers by default in the tests
2020-12-09 20:38:06 +01:00
regnat 5286310e59 Use no substituers by default in the tests
Otherwise https://cache.nixos.org is chosen by default, causing the OSX
testsuite to hang inside the sandbox.

(In a way, this is probably rugging an actual bug under the carpet as
Nix should be able to gracefully timeout in such a case, but that's
beyond mac OSX-fu)
2020-12-09 14:53:45 +01:00
Eelco Dolstra 253571e4ec
Merge pull request #4342 from tweag/fix-remote-build-hook
fix remote build hook
2020-12-09 12:40:00 +01:00
Eelco Dolstra eb458ad1b2
Merge pull request #4319 from Ma27/store-v6-addr
libstore/openStore: fix stores with IPv6 addresses
2020-12-09 12:38:04 +01:00
Maximilian Bosch 93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
regnat c87267c2a4 Store the final drv outputs in memory when building remotely
The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
2020-12-09 10:45:12 +01:00
regnat ee7c94fa1b Test the post-build-hook with remote builders
Regression test for #4245
2020-12-09 10:45:12 +01:00
regnat 6758e65612 Revert "Re-query for the derivation outputs in the post-build-hook"
This reverts commit 1b1e076033.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
Eelco Dolstra 82e5511594
Merge pull request #4325 from tweag/hide-local-store-sql-statements
Hide the sqlite statements declarations for the local store
2020-12-08 14:04:16 +01:00
regnat c0f21f08f8 Hide the sqlite statements declarations for the local store
These have no need to be in the public interface and it causes spurious
rebuilds each time one wants to add or remove a new statement.
2020-12-08 13:29:13 +01:00
Eelco Dolstra 500161b970
Merge pull request #4326 from tweag/fix-post-build-hook-and-remote-builders
Re-query for the derivation outputs in the post-build-hook
2020-12-08 11:47:47 +01:00
regnat 1b1e076033 Re-query for the derivation outputs in the post-build-hook
We can't assume that the runtime state knows about them as they might have
been built remotely, in which case we must query the db again to get
them.
2020-12-08 11:11:02 +01:00
Eelco Dolstra 97dc44f3d6
Merge pull request #4305 from matthewbauer/rosetta2-check
Check for rosetta 2 support before installing x86_64-darwin Nix
2020-12-07 15:59:56 +01:00
Eelco Dolstra 8a06edbf7e
Merge pull request #4321 from matthewbauer/always-default-cache-nixos-org
Always default to cache.nixos.org even when different nix store dir
2020-12-07 11:05:07 +01:00
Matthew Bauer aa07502009 Always default to cache.nixos.org even when different nix store dir
Since 0744f7f, it is now useful to have cache.nixos.org in substituers
even if /nix/store is not the Nix Store Dir. This can always be
overridden via configuration, though.
2020-12-06 23:04:42 -06:00
Eelco Dolstra 1d1a85eb0a
Merge pull request #4318 from matthewbauer/add-slash-to-trusted-binary-cache
Canonicalize binary caches with ‘/’ when one is missing
2020-12-05 10:17:17 +01:00
Eelco Dolstra a5d85d07fa
Merge pull request #4316 from stephank/aws-sdk-compat
Fix compatibility with newer AWS SDKs
2020-12-05 10:16:19 +01:00
Matthew Bauer b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Matthew Bauer 692549c542 Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:28:09 -06:00
Stéphan Kochen e20a3ec756 Fix compatibility with newer AWS SDKs
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-04 19:36:09 +01:00
Eelco Dolstra 45645b6f3b
Merge pull request #4314 from tweag/less-noisy-make-doc
Make `make install` less noisy
2020-12-04 15:02:37 +01:00
regnat 5f66edf245 Make make install less noisy
Remove the printing and useless output of a couple of commands when running `make install`
2020-12-04 14:50:47 +01:00
Eelco Dolstra d94239c979
Merge pull request #4311 from matthewbauer/static-nix-support
Include static "nix" binary in Hydra build products
2020-12-04 12:33:21 +01:00
Matthew Bauer be09af8002 Include static "nix" binary in Hydra build products
This allows users to get Nix from Hydra via a stable url like
https://hydra.nixos.org/build/132078238/download/1/nix
2020-12-03 18:05:18 -06:00
Eelco Dolstra f337aa7099 Split 'nix store add-to-store' into 'add-path' and 'add-file'
This makes it consistent with 'nix hash <path|file>'.
2020-12-04 00:59:24 +01:00
Eelco Dolstra 8df58eae4c
Merge pull request #3858 from edolstra/group-commands
Group 'nix' subcommands
2020-12-03 23:55:28 +01:00