Commit graph

1111 commits

Author SHA1 Message Date
Eelco Dolstra 26d2c62225
Merge pull request #4906 from NixOS/collect-garbage-ca
Make `computeFSClosure` ca-aware
2021-06-23 10:07:08 +02:00
Eelco Dolstra d9a43d3137
Merge pull request #4905 from NixOS/ca-derivations-machine-feature
Add a ca-derivations required machine feature
2021-06-23 10:05:53 +02:00
regnat 3784c66a46 Remove a possible existing store path when building CA derivations
In case a previous interrupted build left a garbage path laying around,
remove it before trying to move the path to its final location.

Fix #4858
2021-06-21 15:29:15 +02:00
regnat a3ce88725b Add a test for the gc with CA derivations
Also add a small architecture to easily run CA-enabled tests
2021-06-15 12:11:31 +02:00
regnat 7c077d2a0f Add a ca-derivations required machine feature
Make ca-derivations require a `ca-derivations` machine feature, and
ca-aware builders expose it.

That way, a network of builders can mix ca-aware and non-ca-aware
machines, and the scheduler will send them in the right place.
2021-06-11 09:12:53 +02:00
regnat d12b12a15b Let nix flake check keep going when keep-going is set
When the `keep-going` option is set to `true`, make `nix flake check`
continue as much as it can before failing.

The UI isn’t perfect as-it-is as all the lines currently start with a
mostly useless `error (ignored): error:` prefix, but I’m not sure what
the best output would be, so I’ll leave it as-it-is for the time being

(This is a bit hijacking the `keep-going` flag as it’s supposed to be a
build-time only thing. But I think it’s faire to reuse it here).

Fix https://github.com/NixOS/nix/issues/4450
2021-06-02 11:13:12 +02:00
regnat 5985b8b527 Check the CA hash when importing stuff in the local store
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
2021-06-01 15:09:24 +02:00
Timothy Klim 4da9ec772c Add .tar.zst support for TarballInputScheme 2021-05-29 16:03:26 +07:00
regnat f46adb783c Add a test for nix run with CA derivations 2021-05-17 15:10:48 +02:00
Eelco Dolstra 7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
Eelco Dolstra 8768398d5f
Merge pull request #4787 from Ma27/builtins-ceil-floor
Implement `builtins.floor` and `builtins.ceil` using the C library functions internally
2021-05-10 20:37:27 +02:00
Maximilian Bosch 7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
regnat b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
regnat bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat 6ea9c65aec fixup! Add a regression test for #4725 2021-04-23 09:34:16 +02:00
regnat b1711071d1 Add a regression test for #4725 2021-04-22 20:07:02 +02:00
Eelco Dolstra 76980a1f3d Merge branch 'build-with-strictDeps' of https://github.com/hercules-ci/nix 2021-04-15 14:15:21 +02:00
Eelco Dolstra 6fb7582413 Merge branch 'libarchive-decompress' of https://github.com/serokell/nix 2021-04-15 13:39:04 +02:00
Domen Kožar c2dfda007e
add tests for zstd compression 2021-04-09 23:13:08 +02:00
Robert Hensing c3090bc6fd tests/*: show when tests are skipped 2021-03-31 08:30:12 +02:00
Eelco Dolstra e7810665a7
Merge pull request #4676 from bjornfor/fix-fetchgit-ref-head
fetchGit: don't prefix "refs/heads/" on ref = "HEAD"
2021-03-30 12:07:05 +02:00
Bjørn Forsman f2a799b16d tests: check that builtins.fetchGit { ..., ref = "HEAD"; } works 2021-03-30 11:39:37 +02:00
Eelco Dolstra 3ab5e8a391
Merge pull request #4239 from tweag/test-against-old-daemon
Add a CI check to ensure compatibility with an old daemon
2021-03-29 16:15:36 +02:00
Eelco Dolstra dd77f71afe LocalBinaryCacheStore::upsertFile(): Fix race
When multiple threads try to upsert the same file, this could fail.

Fixes #4667.
2021-03-26 17:10:15 +01:00
Samuel Dionne-Riel bf07581497 tests: Test .config stays clean with XDG_CONFIG_HOME set 2021-03-19 15:20:47 -04:00
Samuel Dionne-Riel 1765711b68 tests/config: Fix config test configuration
First, "XDG_CONFIG_HOME" shouldn't be named "home", as it may be
confusing compared with `$HOME`, which an upcoming test will be using.

Then, using a fixed location for the test is problematic. Use
`$TEST_ROOT` instead.
2021-03-19 15:20:47 -04:00
Eelco Dolstra 66fa1c7375 Merge 'nix flake {info,list-inputs}' into 'nix flake metadata'
Fixes #4613.
2021-03-16 17:19:04 +01:00
regnat be60c9ef50 Fix the db-migration test 2021-03-16 14:21:41 +01:00
regnat 81df1b5c68 Remove the remote-store-old-daemon test
Doesn't make sense anymore with the new setup
2021-03-16 14:21:40 +01:00
regnat a0866c8ea4 Make the tests (optionnally) run in another derivation
That way we can run them without rebuilding Nix
2021-03-16 14:21:39 +01:00
regnat eab9cdbd75 Add a test for the migration of the db between versions 2021-03-16 14:20:41 +01:00
regnat 5716345adf Add a test ensuring compatibility with an old daemon
This requires adding `nix` to its own closure which is a bit unfortunate,
but as it is optional (the test will be disabled if `OUTER_NIX` is unset) it
shouldn't be too much of an issue.

(Ideally this should go in another derivation so that we can build Nix and run
the test independently, but as the tests are running in the same derivation
as the build it's a bit complicated to do so).
2021-03-16 14:20:41 +01:00
sternenseemann 5869b3025d tests/local.mk: fix missing newline escape
Fixes syntax error introduced in 54ced9072b.
2021-03-16 08:25:04 +01:00
Eelco Dolstra c0073f6268
Merge pull request #4580 from obsidiansystems/restore-test-build-remote-ca-fixed
Restore now-working build-remote-content-addressed-fixed test
2021-03-15 16:48:28 +01:00
regnat 54ced9072b Check the signatures when copying store paths around
Broken atm
2021-03-15 16:35:14 +01:00
Eelco Dolstra 306c154632
Merge pull request #4592 from NixOS/ca/remote-cache
Substitute content-addressed derivations
2021-03-15 16:22:42 +01:00
Kjetil Orbekk 92a234322f Add test for git+file with bare repository 2021-03-01 09:03:25 -05:00
regnat 93b5a59b67 Add a test for the remote caching of CA derivations 2021-03-01 14:00:17 +01:00
regnat 259d6778ef Move the CA tests to a sub-directory
Requires a slight update to the test infra to work properly, but
having the possibility to group tests that way makes the whole thing
quite cleaner imho
2021-03-01 11:08:01 +01:00
John Ericson 5b42e5b177 Restore now-working build-remote-content-addressed-fixed test
This was

 - Added in dbf96e10ec.

 - Commented out in 07975979aa, which I
   believe only reached master by mistake.

 - Deleted in c32168c9bc, when
   `tests/build-hook-ca.nix` was reused for a new test.

But the test works, and we ought to have it.
2021-02-26 16:32:52 +00:00
Eelco Dolstra 94637cd7e5
Merge pull request #4477 from NixOS/ca/build-remote
Build ca derivations remotely
2021-02-26 16:54:44 +01:00
Eelco Dolstra 453c3a603f nix flake update: Recreate the lock file
This is probably what most people expect it to do. Fixes #3781.

There is a new command 'nix flake lock' that has the old behaviour of
'nix flake update', i.e. it just adds missing lock file entries unless
overriden using --update-input.
2021-02-26 14:55:54 +01:00
regnat aead35531a Add a test for the copy of CA paths 2021-02-25 17:19:18 +01:00
Shea Levy 98d1b64400
Initialize plugins after handling initial command line flags
This is technically a breaking change, since attempting to set plugin
files after the first non-flag argument will now throw an error. This
is acceptable given the relative lack of stability in a plugin
interface and the need to tie the knot somewhere once plugins can
actually define new subcommands.
2021-02-24 08:22:17 -05:00
Eelco Dolstra a878c448d8
Merge pull request #4551 from danieldk/system-features-compute-level
Add x86_64 compute levels as system features
2021-02-23 14:30:10 +01:00
regnat c32168c9bc Test the remote building of ca derivations 2021-02-23 08:04:03 +01:00
Daniël de Kok 2de232d2b3 Add x86_64 compute levels as additional system types
When performing distributed builds of machine learning packages, it
would be nice if builders without the required SIMD instructions can
be excluded as build nodes.

Since x86_64 has accumulated a large number of different instruction
set extensions, listing all possible extensions would be unwieldy.
AMD, Intel, Red Hat, and SUSE have recently defined four different
microarchitecture levels that are now part of the x86-64 psABI
supplement and will be used in glibc 2.33:

https://gitlab.com/x86-psABIs/x86-64-ABI
https://lwn.net/Articles/844831/

This change uses libcpuid to detect CPU features and then uses them to
add the supported x86_64 levels to the additional system types. For
example on a Ryzen 3700X:

$ ~/aps/bin/nix -vv --version | grep "Additional system"
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
2021-02-22 09:11:15 +01:00
regnat be1b5c4e59 Test the garbage collection of CA derivations
Simple test to ensure that `nix-build && nix-collect-garbage &&
nix-build -j0` works as it should
2021-02-19 15:48:31 +01:00
Graham Christensen f2245091d0
Revert "narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash" 2021-02-09 12:26:41 -05:00
Eelco Dolstra ee3846b587
Merge pull request #4464 from tweag/nar-narhash-addressed
narinfo: Change NAR URLs to be addressed on the NAR hash instead of the compressed hash
2021-02-09 14:47:39 +01:00