Commit graph

3008 commits

Author SHA1 Message Date
regnat b8f7177a7b Properly fail when trying to register an incoherent realisation 2021-06-23 11:27:14 +02:00
Eelco Dolstra 4a5aa1dbf6
Merge pull request #4838 from NixOS/ca/recursively-substitute-realisations
Recursively substitute the realisations
2021-06-23 10:33:25 +02:00
Eelco Dolstra 26d2c62225
Merge pull request #4906 from NixOS/collect-garbage-ca
Make `computeFSClosure` ca-aware
2021-06-23 10:07:08 +02:00
Eelco Dolstra d9a43d3137
Merge pull request #4905 from NixOS/ca-derivations-machine-feature
Add a ca-derivations required machine feature
2021-06-23 10:05:53 +02:00
Eelco Dolstra 4f9508c3b5
Merge pull request #4836 from NixOS/ca/track-drvoutput-dependencies-2-le-retour
Track the dependencies of CA realisations
2021-06-23 10:03:31 +02:00
regnat ed0e21a88d Fix indentation 2021-06-23 08:16:34 +02:00
regnat 7c96a76dd7 Reformat the sql statements 2021-06-23 08:16:34 +02:00
Théophane Hufschmitt 8d09a4f9a0 Remove a useless string split
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
Théophane Hufschmitt c13d7d0b97 Pass more values by reference
Rather than copying them around everywhere

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
regnat dcabb46124 Shorten a stupidly long sql query name 2021-06-21 16:28:06 +02:00
Eelco Dolstra 610baf359a
Merge pull request #4833 from NixOS/ca/json-realisations-in-worker-protocol
Always send the realisations as JSON
2021-06-21 16:12:17 +02:00
regnat 3784c66a46 Remove a possible existing store path when building CA derivations
In case a previous interrupted build left a garbage path laying around,
remove it before trying to move the path to its final location.

Fix #4858
2021-06-21 15:29:15 +02:00
Eelco Dolstra 4202a3bc4e UDSRemoteStore: Support the 'root' store parameter
Useful when we're using a daemon with a chroot store, e.g.

  $ NIX_DAEMON_SOCKET_PATH=/tmp/chroot/nix/var/nix/daemon-socket/socket nix-daemon --store /tmp/chroot

Then the client can now connect with

  $ nix build --store unix:///tmp/chroot/nix/var/nix/daemon-socket/socket?root=/tmp/chroot nixpkgs#hello
2021-06-18 17:04:11 +02:00
regnat 96d7170e12 Don’t check the deriver field on computeFSClosure
That doesn’t really make sense with CA derivations (and wasn’t even
really correct before because of FO derivations, though that probably
didn’t matter much in practice)
2021-06-12 12:24:53 +02:00
regnat 7ac038fa4b Make computeFSClosure ca-aware
Fix #4820 by preventing nix-collect garbage from crashing if
`keep-outputs` or `keep-derivations` is true
2021-06-11 09:26:49 +02:00
regnat 7c077d2a0f Add a ca-derivations required machine feature
Make ca-derivations require a `ca-derivations` machine feature, and
ca-aware builders expose it.

That way, a network of builders can mix ca-aware and non-ca-aware
machines, and the scheduler will send them in the right place.
2021-06-11 09:12:53 +02:00
regnat 5985b8b527 Check the CA hash when importing stuff in the local store
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
2021-06-01 15:09:24 +02:00
Eelco Dolstra 48396d940e
Merge pull request #4866 from alyssais/libdl
Only link with libdl on Linux
2021-06-01 11:50:50 +02:00
Eelco Dolstra caef6f4314
Merge pull request #4734 from p01arst0rm/fix-s3-ifdef
unified macro style for ENABLE_S3
2021-06-01 11:30:21 +02:00
Alyssa Ross c57ab17687
Only link with libdl on Linux
Linux is (as far as I know) the only mainstream operating system that
requires linking with libdl for dlopen.  On BSD, libdl doesn't exist,
so on non-FreeBSD BSDs linking will currently fail.  On macOS, it's
apparently just a symlink to libSystem (macOS libc), presumably
present for compatibility with things that assume Linux.

So the right thing to do here is to only add -ldl on Linux, not to add
it for everything that isn't FreeBSD.
2021-06-01 08:05:21 +00:00
regnat a22755721b Recursively substitute the realisations
Make sure that whenever we substitute a realisation, we also substitute
its entire closure
2021-05-26 18:44:17 +02:00
regnat 1f3ff0d193 Aso track the output path of the realisation dependencies 2021-05-26 17:09:21 +02:00
regnat cb46d70794 Add a db migration script 2021-05-26 16:59:09 +02:00
regnat 63ebfc73c5 Make copyPaths copy the whole realisations closure
Otherwise registering the realisations on the remote side might fail as
it now expects a complete closure
2021-05-26 16:59:09 +02:00
regnat 8c30acc3e8 Properly track the drvoutput references when building 2021-05-26 16:59:09 +02:00
regnat af3afd25ea Add a method to compute the closure of a realisation
Only considers the closure in term of `Realisation`, ignores all the
opaque inputs.

Dunno whether that’s the nicest solution, need to think it through a bit
2021-05-26 16:59:09 +02:00
regnat eca6ff06d6 Store the realisation deps on the local store 2021-05-26 16:59:09 +02:00
regnat 7ce0441d80 Add a dependencies field to DrvOutputInfo
Currently never used, nor set but will be useful shortly
2021-05-26 16:59:09 +02:00
regnat 7616268812 Always send the realisations as JSON
Align all the worker protocol with `buildDerivation` which inlines the
realisations as one opaque json blob.
That way we don’t have to bother changing the remote store protocol
when the definition of `Realisation` changes, as long as we keep the
json backwards-compatible
2021-05-26 16:59:09 +02:00
regnat 129384bcf3 Remove the remaining occurenceses of a NarHash modulo 2021-05-26 09:39:29 +02:00
regnat 79ae9e4558 Make the Nar hash non modulo
It makes much more sense to have the Nar hash be a plain straight hash
rather than a hash modulo
2021-05-25 10:58:43 +02:00
regnat a8416866cf Always send the realisations as JSON
Align all the worker protocol with `buildDerivation` which inlines the
realisations as one opaque json blob.
That way we don’t have to bother changing the remote store protocol
when the definition of `Realisation` changes, as long as we keep the
json backwards-compatible
2021-05-19 11:45:16 +02:00
regnat 184558834a Extract a generic computeClosure function
Move the `closure` logic of `computeFSClosure` to its own (templated) function.

This doesn’t bring much by itself (except for the ability to properly
test the “closure” functionality independently from the rest), but it
allows reusing it (in particular for the realisations which will require
a very similar closure computation)
2021-05-19 11:44:58 +02:00
Eelco Dolstra 6849ae82de
Merge pull request #4818 from NixOS/ca/cli-use-builtpaths
Enforce the use of properly built paths in libcmd
2021-05-17 16:15:40 +02:00
regnat 2105084645 Enfore the use of properly built paths in libcmd
Replace `DerivedPathWithHints` by a new `BuiltPath` type that serves as
a proof that the corresponding path has been built.
2021-05-17 08:45:08 +02:00
Emily 559a504da7 sandbox: allow SystemVersionCompat.plist on Darwin
For whatever reason, many programs trying to access SystemVersion.plist
also open SystemVersionCompat.plist; this includes Python code and
coreutils’ `cat(1)` (but not the native macOS `/bin/cat`). Illustratory
`dtruss(1m)` output:

    open("/System/Library/CoreServices/SystemVersion.plist\0", 0x0, 0x0)		 = 3 0
    open("/System/Library/CoreServices/SystemVersionCompat.plist\0", 0x0, 0x0)		 = 4 0

I assume this is a Big Sur change relating to the 10.16.x/11.x
version compatibility divide and that it’s something along the lines of
a hook inside libSystem.

Fixes a lot of sandboxed package builds under Big Sur.
2021-05-17 01:24:31 +01:00
Eelco Dolstra 340f831ebe
Merge pull request #4798 from matthewbauer/relock-wait-for-build-slot-goals
Relock wait for build slot goals
2021-05-13 13:07:35 +02:00
Eelco Dolstra 89a4ede92b
Merge pull request #4801 from matthewbauer/fix-tokenize-output-names
Fix tokenize output names in DerivedPath
2021-05-13 13:06:29 +02:00
Matthew Bauer 8c7e043de2 Fix tokenize output names in drv
This should fix the issue described in
https://discourse.nixos.org/t/derivation-does-not-have-wanted-outputs-dev-out/12905.

Specifically, we get an error of

  error: derivation '/nix/store/_.drv' does not have wanted outputs 'dev,out'

when a path like /nix/store/_.drv!dev,out is sent to the daemon.
2021-05-12 21:40:28 -05:00
Matthew Bauer 9a14335845 Relock wait for build slot goals
When we don’t have enough free job slots to run a goal, we put it in
the waitForBuildSlot list & unlock its output locks. This will
continue from where we left off (tryLocalBuild). However, we need the
locks to get reacquired when/if the goal ever restarts. So, we need to
send it back through tryToBuild to get reqacquire those locks.

I think this bug was introduced in
https://github.com/NixOS/nix/pull/4570. It leads to some builds
starting without proper locks.
2021-05-12 12:15:32 -05:00
regnat ec613603ba DerivedPathWithHints -> BuiltPath
Just a renaming for now
2021-05-12 16:19:51 +02:00
Lorenzo Manacorda 4029f4b05b doc: mention Priority for substituters 2021-05-12 13:27:05 +02:00
Eelco Dolstra 7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
regnat d5d19582ef Simplify the realisations disk cache 2021-05-10 17:47:14 +02:00
regnat ab96c1ee50 Remove useless parents
I never remember the exact syntax of the `switch` statement
2021-05-10 17:36:49 +02:00
regnat b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
regnat bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat dadfbce318 Fix the double-slash in the realisations path
Make sure that we always access the realisations under
`binaryCacheUrl/realisations` and not `binaryCacheUrl//realisations`

Fix #4766
2021-05-04 10:35:34 +02:00
Eelco Dolstra 40378fbcba
Merge pull request #4761 from emilazy/issue-4658-mark-impure-host-deps-as-optional
Mark `__impureHostDeps` paths as optional
2021-05-03 16:13:36 +02:00
Maximilian Bosch e5951a6b2f
Bump version number for DerivedPath changes
I guess I misunderstood John's initial explanation about why wildcards
for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26
also has version 1.29, but misses the wildcard[2]. So bumping seems to
be the right call.

[1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464
[2] 255d145ba7
2021-05-03 01:12:23 +02:00