Commit graph

1745 commits

Author SHA1 Message Date
Alex Ameen 2d1d81114d
Add parseFlakeRef and flakeRefToString builtins (#8670)
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.

I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.

I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-25 17:43:33 +00:00
Naïm Favier 570a1a3ad7
parser: merge nested dynamic attributes
Fixes https://github.com/NixOS/nix/issues/7115
2023-07-21 17:14:03 +02:00
John Ericson 453c4be93c
Merge pull request #8680 from NixLayeredStore/test-groups
Introduce notion of a test group, use for CA tests
2023-07-19 11:17:57 -04:00
Robert Hensing 0e3a7e34a0
Merge pull request #8506 from corngood/ssh-master
Pass NIX_SSHOPTS when checking for an ssh master connection.
2023-07-18 15:47:57 +02:00
John Ericson 259e328de8 Introduce notion of a test group, use for CA tests
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.

We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.

I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-18 09:31:13 -04:00
Théophane Hufschmitt a8d5bb5e7e
Merge pull request #8342 from NixLayeredStore/best-effort-supplementary-groups
Best effort supplementary groups
2023-07-17 20:58:17 +02:00
John Ericson 0f7242ff87 Test nested sandboxing, and make nicer error
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.

After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.

Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.

Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-14 09:40:24 -04:00
cidkidnix adb28d4a26 move unset NIX_STORE_DIR in supplementary-groups.sh
to inside the unshare
2023-07-13 14:23:24 -05:00
cidkidnix 1a13757880 Add comment regarding the unset of NIX_STORE_DIR
in build-remote.sh and supplementary-groups.sh
2023-07-13 14:18:12 -05:00
John Ericson 84c4e6f0ac Revert "Skip build-remote-trustless unless sandbox is supported."
This reverts commit 41412dc4ae.
2023-07-13 15:06:50 -04:00
John Ericson 9e64f24340 Revert "Check _NIX_TEST_NO_SANDBOX when setting _canUseSandbox."
This reverts commit c1d39de1fb.
2023-07-13 15:06:34 -04:00
John Ericson ddc0a2050b Merge remote-tracking branch 'upstream/master' into best-effort-supplementary-groups 2023-07-13 15:06:08 -04:00
John Ericson e072e18475 Fix race condition in the language tests
When we pipe to `>(...)` like that, we unfortunately don't wait for the
process to finish. Better to just substitute the file.

Also, use the "unified" diff output that people (including myself) are
more familiar with, thanks to Git.
2023-07-13 08:09:03 -04:00
Mathnerd314 c70484454f Expanded test suite
* Lang now verifies errors and parse output

* Some new miscellaneous tests

* Easy way to update the tests

* Document workflow in manual

* Use `!` not `~` as separater char for sed

  It is confusing to use `~` when we are talking about paths and home
  directories!

* Test test suite itself (`test/lang-test/infra.sh`)

Additionally, run shellcheck on `tests/lang.sh` to help ensure it is
correct, now that is is more complex.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-11 21:43:09 -04:00
Ben Radford 41412dc4ae
Skip build-remote-trustless unless sandbox is supported. 2023-07-11 12:52:59 +01:00
Ben Radford c1d39de1fb
Check _NIX_TEST_NO_SANDBOX when setting _canUseSandbox. 2023-07-11 12:08:33 +01:00
Ben Radford 07dabcc90e
Always attempt setgroups but allow failure to be ignored. 2023-07-11 10:44:05 +01:00
Ben Radford 25b20b4ad2
Merge remote-tracking branch 'origin/master' into best-effort-supplementary-groups 2023-07-11 09:38:34 +01:00
John Ericson c2c8187118 Fix test file name
It's UTF-8, not UFT-8.
2023-07-10 20:58:19 -04:00
John Ericson 028b26a77f
Merge pull request #8370 from hercules-ci/fetchClosure-input-addressed
`fetchClosure`: input addressed and pure
2023-07-09 23:41:22 -04:00
Robert Hensing 9fc82de493 signing.sh: Revert test improvement because it fails on GHA + macOS 2023-07-07 15:37:09 +02:00
Eelco Dolstra 5fbfbb4c7c Fix test 2023-07-03 12:23:57 +02:00
Robert Hensing a6c17097d2 tests: Don't install test-libstoreconsumer program
Sorry about that.
Fixes https://github.com/NixOS/nix/issues/8616
2023-06-30 23:36:27 +02:00
Robert Hensing fefb947132 tests/signing.sh: Check signature checking error message
We should check error messages, so that we know the command fails for
the right reason.
Alternatively, a mere typo can run the test undetected.
2023-06-30 18:23:44 +02:00
Robert Hensing 1db81f7107 tests/fetchClosure: Improve coverage of new and some existing flows 2023-06-30 18:23:44 +02:00
Robert Hensing 40052c7613 fetchClosure: Docs and error message improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-30 18:23:42 +02:00
Robert Hensing ea30f152b7 fetchClosure: Allow input addressed paths in pure mode
When explicitly requested by the caller, as suggested in the meeting
(https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324)

> @edolstra: { toPath } vs { fromPath } is too implicit

I've opted for the `inputAddressed = true` requirement, because it
we did not agree on renaming the path attributes.

> @roberth: more explicit
> @edolstra: except for the direction; not immediately clear in which direction the rewriting happens

This is in fact the most explicit syntax and a bit redundant, which is
good, because that redundancy lets us deliver an error message that
reminds expression authors that CA provides a better experience to
their users.
2023-06-30 18:22:43 +02:00
Eelco Dolstra a0c617348b
Merge pull request #8589 from jfroche/sign-paths-as-allowed-user
Allow to sign path as unprivileged user
2023-06-30 13:13:42 +02:00
Jean-François Roche 80c9259756 Allow to sign path as unprivileged user
User can now sign path as unprivileged/allowed user

refs #1708
2023-06-27 18:31:31 +02:00
Maximilian Bosch 559fd7ffe7
nix flake check: improve error message if overlay is not a lambda (#8582)
* nix flake check: improve error message if overlay is not a lambda

Suppose you have an overlay like this

    {
      inputs = { /* ... */ };
      outputs = { flake-utils, ... }: flake-utils.lib.eachDefaultSystem
        (system: {
          overlays.default = final: prev: {

          };
        });
    }

then `nix flake check` (correctly) fails because `overlays` are supposed
to have the structure `overlays.<name> = final: prev: exp`. However, the
error-message is a little bit counter-intuitive:

    error: overlay does not take an argument named 'final'

While one might guess where the error actually comes from because the
trace above says `… while checking the overlay 'overlays.x86_64-linux'`
this is still pretty confusing because it complains about an argument
not being named `final` even though that's evidently the case.

With this change, the error-message actually makes it clear what's
wrong:

    [ma27@carsten:~/Projects/nix/tmp]$ nix flake check --extra-experimental-features 'nix-command flakes' path:$(pwd)
    error:
           … while checking flake output 'overlays'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           … while checking the overlay 'overlays.x86_64-linux'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           error: overlay is not a lambda, but a set instead
2023-06-27 14:58:29 +02:00
Théophane Hufschmitt 60f06a1714
Merge pull request #5385 from Enzime/add/dirty-rev
Add `dirtyRev` and `dirtyShortRev` to `fetchGit`
2023-06-24 14:55:31 +02:00
Michael Hoang a7b49086c7 Add dirtyRev and dirtyShortRev to fetchGit
Fixes #4682
2023-06-24 14:17:25 +10:00
John Ericson 97df060588 Better document build failure exit codes
- Improved API docs from comment

- Exit codes are for `nix-build`, not just `nix-store --release`

- Make note in tests so the magic numbers are not surprising

Picking up where #8387 left off.
2023-06-22 14:29:45 -04:00
John Ericson 2291232dc1
Merge pull request #8387 from layus/fix-tests
Check exact error codes in linux-sandbox.sh
2023-06-22 13:56:03 -04:00
Guillaume Maudoux 3a20c7c46c
Update tests/linux-sandbox.sh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-20 22:51:29 +02:00
Ben Radford 6ae35534b7
Support opening local store with database on read-only filesystem (#8356)
Previously it was not possible to open a local store when its database is on a read-only filesystem. Obviously a store on a read-only filesystem cannot be modified, but it would still be useful to be able to query it.

This change adds a new read-only setting to LocalStore. When set to true, Nix will skip operations that fail when the database is on a read-only filesystem (acquiring big-lock, schema migration, etc), and the store database will be opened in immutable mode.

Co-authored-by: Ben Radford <benradf@users.noreply.github.com>
Co-authored-by: cidkidnix <cidkidnix@protonmail.com>
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-20 11:34:09 +02:00
Eelco Dolstra e503eadafc
Merge pull request #8477 from edolstra/tarball-flake-redirects
Tarball flake improvements
2023-06-16 18:03:50 +02:00
Eelco Dolstra 713836112c
Merge pull request #8517 from hercules-ci/fix-build-hook-error-for-lib-users
Fix build hook error for libstore library users
2023-06-16 13:20:50 +02:00
Robert Hensing d2696cdd1e Fix build hook error for libstore library users
A library shouldn't require changes to the caller's argument handling,
especially if it doesn't have to, and indeed we don't have to.

This changes the lookup order to prioritize the hardcoded path to nix
if it exists. The static executable still finds itself through /proc
and the like.
2023-06-15 14:32:00 +02:00
Valentin Gagarin c453719d6e rename files referring to antiquotation
since we renamed this to string interpolation, file names should be
fixed up as well
2023-06-15 02:29:31 +02:00
John Ericson 946cd9e3f9
Merge pull request #8351 from obsidiansystems/delete-profiles-tests-docs
Expanding tests and docs relating to deleting profiles
2023-06-15 01:47:21 +02:00
John Ericson d4a2ced9cb Split out nix-collect-garbage -d test to new file
Good for test parallelism, and separation of concerns (core GC vs
profiles deleting).
2023-06-14 19:01:07 -04:00
John Ericson ca5752d4fa Add another case to the nix-collect-garbage -d test 2023-06-14 19:01:07 -04:00
Michal Sojka a0c4d58549 Remove RegisterPrimOp constructor without support for documentation
The remaining constructor RegisterPrimOp::RegisterPrimOp(Info && info)
allows specifying the documentation in .args and .doc members of the
Info structure.

Commit 8ec1ba0210 removed all uses of the removed constructor in the
nix binary. Here, we remove the constructor completely as well as its
use in a plugin test. According to #8515, we didn't promis to maintain
compatibility with external plugins.

Fixes #8515
2023-06-14 22:37:52 +02:00
John Ericson 05eb06a1de
Merge pull request #8490 from flox/stdin_handling
fix: Do not apply default installables when using --stdin
2023-06-14 20:41:22 +02:00
John Ericson ff905cb796
Merge pull request #4803 from ShamrockLee/nix-channel-list-generations
Add `nix-channel --list-generations`
2023-06-14 18:30:35 +02:00
John Ericson 61a3e1f2e2
Merge pull request #4282 from tweag/fix-ca-hash-rewriting
fix the hash rewriting for ca-derivations
2023-06-14 18:25:00 +02:00
Eelco Dolstra 1ad3328c5e Allow tarball URLs to redirect to a lockable immutable URL
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
2023-06-13 14:17:45 +02:00
David McFarland 5454fdcceb Add test of explicit ssh control path in nix-copy test
This highlights a problem caused by SSHMaster::isMasterRunning returning
false when NIX_SSHOPTS contains -oControlPath.
2023-06-13 00:54:52 -03:00
Tom Bereknyei 0e3849dc65 test: add test for non-defaulting for stding installable input 2023-06-12 08:40:17 -04:00