Commit graph

1085 commits

Author SHA1 Message Date
John Ericson 2dd11f0780 Reenable previously failing trustless remote builder tests 2021-02-27 05:33:47 +00:00
John Ericson e547fe12d1 Merge branch 'restore-test-build-remote-ca-fixed' into trustless-remote-builder-simple 2021-02-27 03:53:22 +00:00
John Ericson 5b42e5b177 Restore now-working build-remote-content-addressed-fixed test
This was

 - Added in dbf96e10ec.

 - Commented out in 07975979aa, which I
   believe only reached master by mistake.

 - Deleted in c32168c9bc, when
   `tests/build-hook-ca.nix` was reused for a new test.

But the test works, and we ought to have it.
2021-02-26 16:32:52 +00:00
Eelco Dolstra 94637cd7e5
Merge pull request #4477 from NixOS/ca/build-remote
Build ca derivations remotely
2021-02-26 16:54:44 +01:00
Eelco Dolstra 453c3a603f nix flake update: Recreate the lock file
This is probably what most people expect it to do. Fixes #3781.

There is a new command 'nix flake lock' that has the old behaviour of
'nix flake update', i.e. it just adds missing lock file entries unless
overriden using --update-input.
2021-02-26 14:55:54 +01:00
regnat aead35531a Add a test for the copy of CA paths 2021-02-25 17:19:18 +01:00
John Ericson 7d703f2106 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2021-02-24 17:58:43 +00:00
Shea Levy 98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Eelco Dolstra a878c448d8
Merge pull request #4551 from danieldk/system-features-compute-level
Add x86_64 compute levels as system features
2021-02-23 14:30:10 +01:00
regnat c32168c9bc Test the remote building of ca derivations 2021-02-23 08:04:03 +01:00
Daniël de Kok 2de232d2b3 Add x86_64 compute levels as additional system types
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.

Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:

https://gitlab.com/x86-psABIs/x86-64-ABI
https://lwn.net/Articles/844831/

This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:

$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
2021-02-22 09:11:15 +01:00
regnat be1b5c4e59 Test the garbage collection of CA derivations
Simple test to ensure that `nix-build && nix-collect-garbage &&
nix-build -j0` works as it should
2021-02-19 15:48:31 +01:00
John Ericson b0d52855dc Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2021-02-12 05:51:53 +00:00
Graham Christensen f2245091d0
Revert "narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash" 2021-02-09 12:26:41 -05:00
Eelco Dolstra ee3846b587
Merge pull request #4464 from tweag/nar-narhash-addressed
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
2021-02-09 14:47:39 +01:00
Eelco Dolstra 0187838e2e Add a trace to readLine() failures
Hopefully this helps to diagnose 'error: unexpected EOF reading a
line' on macOS.
2021-02-05 12:18:11 +01:00
Eelco Dolstra 488a826842
Merge pull request #4467 from edolstra/error-formatting
Improve error formatting
2021-01-25 12:50:57 +01:00
Eelco Dolstra 680d8a5b86
Merge pull request #4387 from obsidiansystems/non-local-store-build
Make `nix-build --store whatever` work
2021-01-25 12:24:23 +01:00
John Ericson 5738b08233 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2021-01-22 16:22:00 +00:00
John Ericson 8c07ed1dda Improve documentation and test and requested 2021-01-22 15:58:58 +00:00
Eelco Dolstra 55849e153e Change error position formatting
It's now

  at /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix:7:7:

instead of

  at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix

The new format is more standard and clickable.
2021-01-21 11:02:09 +01:00
adisbladis 144cad9069
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
This change is to simplify [Trustix](https://github.com/tweag/trustix) indexing and makes it possible to reconstruct this URL regardless of the compression used.

In particular this means that 7c2e9ca597/contrib/nix/nar/nar.go (L61-L71) can be removed and only the bits that are required to establish trust needs to be published in the Trustix build logs.
2021-01-21 10:32:56 +01:00
Eelco Dolstra d04d8463fa
Merge pull request #4281 from lilyball/shebang
Escape filename given to nix-shell in shebang mode
2021-01-20 10:00:10 +01:00
John Ericson 0027b05a15 Merge remote-tracking branch 'upstream/master' into non-local-store-build 2021-01-15 02:01:24 +00:00
Eelco Dolstra 7a472a76d4 Add 'nix daemon' command 2021-01-14 00:05:04 +01:00
Eelco Dolstra d33eca8539 Rename 'nix store sign-paths' to 'nix store sign' 2021-01-13 23:32:37 +01:00
Eelco Dolstra 6254b1f5d2 Add 'nix store delete' command 2021-01-11 19:46:59 +01:00
Eelco Dolstra fdcd62eec5 Add 'nix store gc' command 2021-01-10 23:29:14 +01:00
Eelco Dolstra 0df69d96e0 Make sodium a required dependency 2021-01-06 17:56:53 +01:00
Eelco Dolstra 9374c2baea Add commands for generating secret/public keys 2021-01-06 17:49:31 +01:00
John Ericson fed1237246 Test nix-build with non-local-store --store
Just a few small things needed fixing!
2020-12-23 22:42:06 +00:00
John Ericson 29bd63e990 Test nix-instantiate with binary cache store
Trying to make sure it work with obscurers stores.
2020-12-23 22:42:06 +00:00
Eelco Dolstra f4a9fb67da Merge branch 'git-rev-error' of https://github.com/Ma27/nix into master 2020-12-22 16:39:32 +01:00
Eelco Dolstra 75efa42134 Move <nix/fetchurl.nix> into the nix binary
This makes the statically linked nix binary just work, without needing
any additional files.
2020-12-22 14:43:20 +01:00
Maximilian Bosch 897ae235fc
tests/fetchGit: test behavior of allRefs = true; 2020-12-22 12:18:10 +01:00
John Ericson bd96403da6 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2020-12-20 14:02:12 +00:00
regnat 58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Eelco Dolstra 9c143c411b
Merge pull request #4350 from NixOS/ca/fix-build-with-nix-command
Fix the `nix` command with CA derivations
2020-12-11 15:01:49 +01:00
regnat eb45308109 Fix the nix command with CA derivations
Prevents a crash because most `nix` subcommands assumed that derivations
know their output path, which isn't the case for CA derivations
2020-12-11 10:28:09 +01:00
regnat 5286310e59 Use no substituers by default in the tests
Otherwise https://cache.nixos.org is chosen by default, causing the OSX
testsuite to hang inside the sandbox.

(In a way, this is probably rugging an actual bug under the carpet as
Nix should be able to gracefully timeout in such a case, but that's
beyond mac OSX-fu)
2020-12-09 14:53:45 +01:00
regnat ee7c94fa1b Test the post-build-hook with remote builders
Regression test for #4245
2020-12-09 10:45:12 +01:00
Eelco Dolstra f337aa7099 Split 'nix store add-to-store' into 'add-path' and 'add-file'
This makes it consistent with 'nix hash <path|file>'.
2020-12-04 00:59:24 +01:00
Eelco Dolstra ea2062a2d9 Move most store-related commands to 'nix store' 2020-12-03 23:22:22 +01:00
Eelco Dolstra ef583303f0 Move NAR-related commands to 'nix nar' 2020-12-03 18:09:02 +01:00
Eelco Dolstra b2d6c6161e Move 'nix hash-*' and 'nix to-*' to 'nix hash'
From the 'nix' UX review.
2020-12-03 17:55:55 +01:00
Eelco Dolstra 0bd060f23a
Merge pull request #4308 from tweag/properly-test-early-cutoff
Properly test the early cutoff for CA derivations
2020-12-03 14:45:29 +01:00
regnat 8ad72b1f1c Properly test early cutoff with CA derivations
Build things with a different seed each time to make sure that it works
despite the different drvs
2020-12-03 13:31:07 +01:00
Eelco Dolstra df552a2645 nix eval: Add option to write a directory
This is useful for generating the nix manpages, but it may have other
applications (like generating configuration files without a Nix store).
2020-12-02 23:23:23 +01:00
Eelco Dolstra e5cf501c77
Merge pull request #4284 from tweag/fixed-output-depending-on-ca
Allow fixed-output derivations to depend on (floating) content-addressed ones
2020-12-01 20:25:41 +01:00
regnat 9bd8184f1f Allow fixed-output derivations to depend on (floating) content-addressed ones
Fix an overlook of https://github.com/NixOS/nix/pull/4056
2020-11-27 15:39:24 +01:00