Commit graph

1146 commits

Author SHA1 Message Date
regnat 8707773965 Properly lock the builds of CA derivations
Make sure that we can’t build the same derivation twice at the same
time.

Fix https://github.com/NixOS/nix/issues/5029
2021-07-20 06:57:56 +02:00
regnat c05bdef020 Forward the whole Nix config to the repl subprocesses
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the nix subprocesses in the repl
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available.

This is required for example to make `nix repl` work with a custom
`--store`
2021-07-16 09:48:45 +02:00
Maximilian Bosch 04cd2da84c
Merge branch 'master' into structured-attrs-shell
Conflicts:
        src/nix/develop.cc
        src/nix/get-env.sh
        tests/shell.nix
2021-07-12 15:49:39 +02:00
Eelco Dolstra 86fb01c4be nix print-dev-env: Add --json flag 2021-07-09 12:10:48 +02:00
Eelco Dolstra e50408bd31 nix develop: Add a test for bash functions 2021-07-09 01:18:44 +02:00
Eelco Dolstra 9fc7da1e08 Add test for #4992 2021-07-09 01:18:44 +02:00
Eelco Dolstra b1cfe8f984 nix develop: Add a test for arrays 2021-07-09 00:32:55 +02:00
Eelco Dolstra 8648143120
Merge pull request #4988 from NixOS/fetchgit-name-attribute
Add a name attribute to the fetchers
2021-07-08 14:33:49 +02:00
regnat 7e5c79a2d2 Forbid the name attribute for fetchTree
We need to support it for the “old” fetch* functions for backwards
compatibility, but we don’t need it for fetchTree (as it’s a new
function).
Given that changing the `name` messes-up the content hashing, we can
just forbid passing a custom `name` argument to it
2021-07-08 13:53:52 +02:00
Eelco Dolstra 6060ea1b0e Add tests/dummy 2021-07-07 11:30:35 +02:00
Eelco Dolstra 02dd6bb610 tests/check.sh: Fix a race
Fixes this random failure:

    error: hash mismatch in fixed-output derivation '/tmp/nix-shell.EUgAVU/nix-test/tests/check/store/sfps3l3c5n7dabpx34kigxnfhmrwk2h6-dummy.drv':
             specified: sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
                got:    sha256-0qhPS4tlCTfsj3PNi+LHSt1akRumTfJ0WO2CKdqASiY=

which happens because multiple tests were writing to ./dummy.
2021-07-07 10:48:47 +02:00
Eelco Dolstra e37ecd1282 Merge branch 'balsoft/registry' of https://github.com/serokell/nix 2021-07-07 09:57:15 +02:00
regnat e4b082a52b allow fetchMercurial to take a name argument 2021-07-06 08:43:06 +02:00
regnat a487a652ed allow fetchTarball to take a name argument 2021-07-06 08:42:47 +02:00
regnat 83615fcf8f Allow fetchGit to take a name argument
Fix #3388
2021-07-06 08:41:52 +02:00
Michael Fellinger 087c5f5325
Fix devShell handling of env values including @ and % 2021-07-05 12:05:06 +02:00
Alexander Bantyev d8ad6f1c10
Add tests for --registry and second arg for nix registry pin 2021-07-02 16:10:57 +03:00
Eelco Dolstra f5320299dd
Merge pull request #4937 from NixOS/ca/make-the-tests-useful
Make the CA tests actually test something
2021-06-28 16:06:49 +02:00
Eelco Dolstra bf68c693dc tests: Get rid of some result symlinks
Fixes

  error: cannot create symlink '/home/eelco/Dev/nix/tests/result'; already exists
2021-06-25 11:17:19 +02:00
regnat 7746cb13dc Make CA derivations compatible with recursive Nix
Add an access-control list to the realisations in recursive-nix (similar
to the already existing one for store paths), so that we can build
content-addressed derivations in the restricted store.

Fix #4353
2021-06-24 14:53:10 +02:00
regnat be7a4a6a13 Make the post-build-hook also run for unresolved CA derivations
Fix #4837
2021-06-24 11:41:57 +02:00
regnat 01a3f4d7ec Fix the CA gc test
Broken by https://github.com/NixOS/nix/issues/4936
2021-06-23 17:37:29 +02:00
regnat fd3f5e9085 Make the CA tests actuall test CA derivations
Fix a mistake in config.nix that was preventing
`NIX_TESTS_CA_BY_DEFAULT` from having any meaningful effect
2021-06-23 17:36:50 +02:00
Eelco Dolstra 0a535dd5ac
Merge pull request #4839 from NixOS/ca/gracefully-handle-duplicate-realisations
Gracefully handle duplicate realisations
2021-06-23 11:50:18 +02:00
Eelco Dolstra f9f773b332
Merge pull request #4908 from NixOS/ca/fix-nix-develop
Make `nix develop` work with CA derivations
2021-06-23 11:27:58 +02:00
Eelco Dolstra 7945055c63
Merge pull request #4842 from NixOS/ca/fix-nix-shell
Make `nix-shell` support content-addressed derivations
2021-06-23 11:26:22 +02:00
regnat a5df669bc6 Add a test for the “two glibc” issue 2021-06-23 11:18:31 +02:00
Eelco Dolstra 4a5aa1dbf6
Merge pull request #4838 from NixOS/ca/recursively-substitute-realisations
Recursively substitute the realisations
2021-06-23 10:33:25 +02:00
Eelco Dolstra 26d2c62225
Merge pull request #4906 from NixOS/collect-garbage-ca
Make `computeFSClosure` ca-aware
2021-06-23 10:07:08 +02:00
Eelco Dolstra d9a43d3137
Merge pull request #4905 from NixOS/ca-derivations-machine-feature
Add a ca-derivations required machine feature
2021-06-23 10:05:53 +02:00
Maximilian Bosch 27ce722638
Prefix env vars for attrs.* files with NIX_ 2021-06-22 19:45:08 +02:00
Maximilian Bosch 3504c811a5
Add testcase for nix develop with __structuredAttrs 2021-06-22 19:15:57 +02:00
Maximilian Bosch 3944a120ec
Set environment variables for .attrs.json & .attrs.sh
This way no derivation has to expect that these files are in the `cwd`
during the build. This is problematic for `nix-shell` where these files
would have to be inserted into the nix-shell's `cwd` which can become
problematic with e.g. recursive `nix-shell`.

To remain backwards-compatible, the location inside the build sandbox
will be kept, however using these files directly should be deprecated
from now on.
2021-06-22 19:15:57 +02:00
Maximilian Bosch 3b5429aec1
Source complete env in nix-shell with __structuredAttrs = true;
This is needed to push the adoption of structured attrs[1] forward. It's
now checked if a `__json` exists in the environment-map of the derivation
to be openend in a `nix-shell`.

Derivations with structured attributes enabled also make use of a file
named `.attrs.json` containing every environment variable represented as
JSON which is useful for e.g. `exportReferencesGraph`[2]. To
provide an environment similar to the build sandbox, `nix-shell` now
adds a `.attrs.json` to `cwd` (which is mostly equal to the one in the
build sandbox) and removes it using an exit hook when closing the shell.

To avoid leaking internals of the build-process to the `nix-shell`, the
entire logic to generate JSON and shell code for structured attrs was
moved into the `ParsedDerivation` class.

[1] https://nixos.mayflower.consulting/blog/2020/01/20/structured-attrs/
[2] https://nixos.org/manual/nix/unstable/expressions/advanced-attributes.html#advanced-attributes
2021-06-22 19:15:57 +02:00
regnat 3784c66a46 Remove a possible existing store path when building CA derivations
In case a previous interrupted build left a garbage path laying around,
remove it before trying to move the path to its final location.

Fix #4858
2021-06-21 15:29:15 +02:00
regnat a3ce88725b Add a test for the gc with CA derivations
Also add a small architecture to easily run CA-enabled tests
2021-06-15 12:11:31 +02:00
regnat 2cf591a134 Make nix develop work with CA derivations
Fix #4823
2021-06-11 13:35:13 +02:00
regnat 56605b4688 Make nix-shell support content-addressed derivations
Resolve the derivation before trying to load its environment −
essentially reproducing what the build loop does − so that we can
effectively access our dependencies (and not just their placeholders).

Fix #4821
2021-06-11 13:32:49 +02:00
regnat 7c077d2a0f Add a ca-derivations required machine feature
Make ca-derivations require a `ca-derivations` machine feature, and
ca-aware builders expose it.

That way, a network of builders can mix ca-aware and non-ca-aware
machines, and the scheduler will send them in the right place.
2021-06-11 09:12:53 +02:00
regnat d12b12a15b Let nix flake check keep going when keep-going is set
When the `keep-going` option is set to `true`, make `nix flake check`
continue as much as it can before failing.

The UI isn’t perfect as-it-is as all the lines currently start with a
mostly useless `error (ignored): error:` prefix, but I’m not sure what
the best output would be, so I’ll leave it as-it-is for the time being

(This is a bit hijacking the `keep-going` flag as it’s supposed to be a
build-time only thing. But I think it’s faire to reuse it here).

Fix https://github.com/NixOS/nix/issues/4450
2021-06-02 11:13:12 +02:00
regnat 5985b8b527 Check the CA hash when importing stuff in the local store
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
2021-06-01 15:09:24 +02:00
Timothy Klim 4da9ec772c Add .tar.zst support for TarballInputScheme 2021-05-29 16:03:26 +07:00
regnat a22755721b Recursively substitute the realisations
Make sure that whenever we substitute a realisation, we also substitute
its entire closure
2021-05-26 18:44:17 +02:00
regnat f46adb783c Add a test for nix run with CA derivations 2021-05-17 15:10:48 +02:00
Eelco Dolstra 7f9759b18d
Merge pull request #4781 from NixOS/locally_cache_the_remote_realisations
Add a realisations disk cache
2021-05-10 20:37:57 +02:00
Eelco Dolstra 8768398d5f
Merge pull request #4787 from Ma27/builtins-ceil-floor
Implement `builtins.floor` and `builtins.ceil` using the C library functions internally
2021-05-10 20:37:27 +02:00
Maximilian Bosch 7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
regnat b66234134f Add a realisations disk cache
Similar to the nar-info disk cache (and using the same db).
This makes rebuilds muuch faster.

- This works regardless of the ca-derivations experimental feature.
  I could modify the logic to not touch the db if the flag isn’t there,
  but given that this is a trash-able local cache, it doesn’t seem to be
  really worth it.
- We could unify the `NARs` and `Realisation` tables to only have one
  generic kv table. This is left as an exercise to the reader.
- I didn’t update the cache db version number as the new schema just
  adds a new table to the previous one, so the db will be transparently
  migrated and is backwards-compatible.

Fix #4746
2021-05-06 17:38:20 +02:00
regnat bf485dcf46 Properly normalize the content-addressed paths
Make sure that their timestamp are always normalized.
Otherwise, strange − and non-deterministic − things might happen, like
https://github.com/NixOS/nixpkgs/issues/121813

Fix #4775
2021-05-05 21:00:08 +02:00
regnat 6ea9c65aec fixup! Add a regression test for #4725 2021-04-23 09:34:16 +02:00