Commit graph

2005 commits

Author SHA1 Message Date
eldritch horrors 2633ca3f88 Merge pull request #9288 from edolstra/fix-nar-access-test
Fix nar-access test on macOS

(cherry picked from commit e9a857e4189241d23111e18e284ab7ac9d56b798)
Change-Id: I2fd2ec3e6ccb88350bdcd2ba15ccfd77275399c9
2024-03-04 05:27:18 +01:00
eldritch horrors 7ff1dca1fa Merge pull request #9247 from obsidiansystems/derivation-test-with-files
Turn derivation unit tests into unit characterization tests

(cherry picked from commit a6e587923c9d5d716fe0f0049bed96d1cc210bff)
Change-Id: Ia2a2e65aabfee8d5d52142b8fdaacbae4a27242c
2024-03-04 05:21:10 +01:00
eldritch horrors ea10088703 Merge pull request #9216 from obsidiansystems/addDrvOutputDependencies-pre
Add `builtins.addDrvOutputDependencies`

(cherry picked from commit a58d7f143ec995a45745c2176bfebcc3e011db58)
Change-Id: Ia5a1790bf29dfaf29287cc35cdae6b6d650e7a83
2024-03-04 05:06:56 +01:00
eldritch horrors 032eff7f69 Merge pull request #8470 from ncfavier/shebang-single-quotes
nix-shell: support single quotes in shebangs, fix whitespace parsing
(cherry picked from commit 3b99c6291377cbd22607896af9dfafa857d2f2dc)
Change-Id: I2a431b21c3467eefa1ef95d5a36d672f45b6937a
2024-03-04 05:06:32 +01:00
eldritch horrors 7f590ea709 Merge pull request #6223 from obsidiansystems/worker-proto-with-version
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info

(cherry picked from commit 8b68bbb77745fda0d14939b6c23d31cc89da41ce)
Change-Id: Ia3d3b9fbaf9f0ae62ab225020b7d14790e793655
2024-03-04 04:59:31 +01:00
Robert Hensing ab40b2c5d0 Merge pull request #9157 from obsidiansystems/protocol-versions
Add protocol versions to `{Worker,Serve}Proto::*Conn`

(cherry picked from commit 4d17c59d8d059a5b39f1d1da2b58f2ec8da44861)
Change-Id: I497af39deb792e50c157a1305d8c9e722798740b
2024-03-04 04:43:35 +01:00
eldritch horrors 5ddd1a9166 Merge pull request #9168 from obsidiansystems/fix-lang-tests-read-only
Fix language tests a bit

(cherry picked from commit e92cac789fc84b116a03db04a5f7de8df48e34cd)
Change-Id: I527e94c5af0921ff2559ce5a4ac7635be791c22e
2024-03-04 04:42:21 +01:00
eldritch horrors d28a6618a8 Merge pull request #9167 from obsidiansystems/pre-overhaul-completions
Improve tests and docs prior to refactoring completions

(cherry picked from commit 5442d9b47298389918d1f38d20f768a80ffc2369)
Change-Id: Ief99ac2cd9c92981a9a522d15b9c3daf99182c9d
2024-03-04 04:38:33 +01:00
eldritch horrors e12e9f2452 Merge pull request #9137 from obsidiansystems/serve-protocol
Introduce separate Serve protocol serialisers

(cherry picked from commit d070d8b7460f412a657745698dba291c66792402)
Change-Id: Ibcc8639e8997bcd07f7a5318330a147bcadc411b
2024-03-04 04:37:05 +01:00
eldritch horrors 6897e238bd Merge pull request #9099 from obsidiansystems/common-proto
Factor out bits of the worker protocol to use elsewhere

(cherry picked from commit 4b1a97338f517f45e6169d3d8845c5caa5724e97)
Change-Id: If93afa0f8b1cf9b0e705b34fa71e6fd708752758
2024-03-04 04:36:58 +01:00
eldritch horrors fd1299cef3 Merge pull request #9106 from Ericson2314/positive-source-filtering
Use positive source filtering for the standalone functional tests job and Perl bindings

(cherry picked from commit 6b6bd9003062c86a49d4384381941cf57f269c45)
Change-Id: I896be67654f893d543ed6beb5d0d0d6c6d36e027
2024-03-04 04:36:42 +01:00
eldritch horrors 4517de00cb Merge pull request #8905 from hercules-ci/no-unknown-location
Don't print unknown locations unless requested for dev purposes

(cherry picked from commit 3dd4475826dff8052fef42d4600b61e6b91950ac)
Change-Id: I04a91277d1d9d09f5c1bf4a28fc99f0702b161e5
2024-03-04 04:36:38 +01:00
eldritch horrors c36ba796dd Merge pull request #9096 from NixOS/dont-rebuild-twice-on-ci
Don't run the tests that require building if we're not building

(cherry picked from commit 8cfa582f436db8066eff74cb084990367e014ce6)
Change-Id: I6085ca6107349669407340d7a5e52639a2febc90
2024-03-04 04:36:33 +01:00
eldritch horrors 7e966819b5 Merge pull request #9098 from obsidiansystems/test-proto
Enable most of the third `BuildResult` worker protocol test

(cherry picked from commit d344c112f772282bacacd4c66a75df4022d16e12)
Change-Id: I7b2b72aa84c19a6069f9c12128d901262db6f91c
2024-03-04 04:36:28 +01:00
eldritch horrors ea0008deea Merge pull request #9094 from obsidiansystems/test-proto
Test the rest of the worker protocol serializers

(cherry picked from commit 2f1c16dfa2378fd8616bff1b9b7cd0b4d42af69b)
Change-Id: Idfd72d32b21d14a260e02f65531d287cef7464d2
2024-03-04 04:36:23 +01:00
eldritch horrors f17e7b1855 Merge pull request #8923 from obsidiansystems/test-proto
Unit test some worker protocol serializers

(cherry picked from commit c6faef61a6f31c71146aee5d88168e861df9a22a)
Change-Id: I99e36f5f17eb7642211a4e42a16b143424f164b4
2024-03-04 04:36:14 +01:00
Eelco Dolstra bef68e53b9 Fix symlink handling
This restores the symlink handling behaviour prior to
94812cca98.

Fixes #9298.

(cherry picked from commit 31ebc6028b3682969d86a7b39ae87131c41cc604)
2024-01-21 20:53:30 +00:00
Shea Levy ba48ab4b95
Merge branch '2.18-maintenance' into ifd-buildStore-2.18 2024-01-11 07:21:16 -05:00
Shea Levy e7c2b35827
Build IFD in the build store when using eval-store.
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.

Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.

(cherry picked from commit c3942ef85ffbd83391410fbf012f1de366d2463c)
2024-01-11 06:34:27 -05:00
Shea Levy be208d8e78
remote-store test: Break out IFD expression into a separate file
(cherry picked from commit 9cb287657bec5a969d8bb1678d598d9fa820e60b)
2024-01-11 06:30:02 -05:00
John Ericson 615bd655e5 Add missing -pthread for test support libraries
This is good in general (see how the other libraries also have long had
it, since 49fe9592a4) but in particular
needed to fix the NetBSD build.

(cherry picked from commit b23273f6a29c725646b3523b1c35a0ae4a84ef61)
2023-12-05 23:44:41 +00:00
John Ericson f7f37035c8 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

(cherry picked from commit 91b6833686a6a6d9eac7f3f66393ec89ef1d3b57)
(cherry picked from commit a61e42adb528b3d40ce43e07c79368d779a8b624)
2023-12-01 13:05:03 -05:00
John Ericson 30dcc19d1f Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```

(cherry picked from commit 68c81c737571794f7246db53fb4774e94fcf4b7e)
2023-12-01 12:06:43 -05:00
John Ericson 72b65981f9 Revert "Adapt scheduler to work with dynamic derivations"
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.

Mostly fixed #9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.

(cherry picked from commit 8440afbed756254784d9fea3eaab06649dffd390)
2023-10-02 15:05:23 +00:00
Robert Hensing 860f64c345 pathExists: isDir when endswith /.
(cherry picked from commit f8a3893e8d77ce4a6e23719a0b2d88464cb84b9c)
2023-10-01 13:58:20 +01:00
Cole Helbling f3005632c4 Re-enable systemd-nspawn test
It was disabled in c6953d1ff6 because
a recent Nixpkgs bump brought in a new systemd which changed how
systemd-nspawn worked.

As far as I can tell, the issue was caused by this upstream systemd
commit:
b71a0192c0

Bind-mounting the host's `/sys` and `/proc` into the container's
`/run/host/{sys,proc}` fixes the issue and allows the test to succeed.

(cherry picked from commit 883092e3f78d4efb1066a2e24e343b307035a04c)
2023-09-20 17:03:47 +00:00
Eelco Dolstra b6b2a0aea9 Use "touch -h"
https://hydra.nixos.org/build/235888160

This is needed because Nixpkgs now contains dangling symlinks
(pkgs/test/nixpkgs-check-by-name/tests/symlink-invalid/pkgs/by-name/fo/foo/foo.nix).
2023-09-19 17:21:07 +02:00
Eelco Dolstra c6953d1ff6 Disable systemd-nspawn test
This is broken because of a change in systemd in NixOS 23.05. It fails
with

  Failed to mount proc (type proc) on /proc (MS_NOSUID|MS_NODEV|MS_NOEXEC ""): Operation not permitted
2023-09-19 17:03:21 +02:00
Eelco Dolstra c451b48993 Merge remote-tracking branch 'origin/master' into p/flake-update 2023-09-19 13:33:56 +02:00
John Ericson 80d7994f52 Special-case error message to add extra information
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.

The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.

The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.

There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson 7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
Eelco Dolstra 919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Eelco Dolstra b88784278f
Merge pull request #8869 from hercules-ci/fix-issue-8838-pathExists-isDir
Fix #8838, pathExists: isDir when ends with `/ `
2023-09-01 13:15:54 +02:00
Robert Hensing be3362e747 Fix nix-copy test 2023-08-30 19:35:02 -04:00
Eelco Dolstra 5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Robert Hensing 1e08e12d81 pathExists: isDir when endswith /
Fixes https://github.com/NixOS/nix/issues/8838
2023-08-25 17:18:37 +02:00
Robert Hensing d2e6cfa075 tests/lang/eval-okay-pathexists: Add cases 2023-08-25 17:17:33 +02:00
John Ericson 50f40ac4c0
Merge pull request #8829 from obsidiansystems/build-dynamic-derivations
Adapt scheduler to work with dynamic derivations
2023-08-25 11:13:15 -04:00
tomberek b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
John Ericson dd9f816b29
Merge pull request #8661 from hercules-ci/test-reformat-error-message
tests: Reformat exit code error message
2023-08-25 10:17:15 -04:00
John Ericson 5e3986f59c Adapt scheduler to work with dynamic derivations
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).

This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.

The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-25 10:01:25 -04:00
Guillaume Girol 925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Eelco Dolstra 81045f243f Tarball trees: Propagate lastModified
This makes them behave consistently with GitHub/GitLab flakes.
2023-08-22 21:51:26 +02:00
Robert Hensing 75243c9693
test/flakes/follow-paths.sh: Quote
Co-authored-by: Alex Ameen <alex.ameen.tx@gmail.com>
2023-08-18 14:46:13 +02:00
Felix Uhl 17ceec3a91 Test repl formatting with and without :p 2023-08-17 13:03:43 +02:00
Vertex 20d9c672d1
Update tests/flakes/follow-paths.sh
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-15 10:10:27 +01:00
Alex Zero 1ef8008ca7 Fix follow path checking at depths greater than 2
We need to recurse into the input tree to handle follows paths that
trarverse multiple inputs that may or may not be follow paths
themselves.
2023-08-14 18:55:46 +01:00
John Ericson 44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
tomberek 010dc7958e
Merge pull request #8369 from obsidiansystems/inductive-derived-path
Make the Derived Path family of types inductive for dynamic derivations
2023-08-11 08:50:22 -05:00
John Ericson 60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Peter Waller 4b1bd822ac Try to realise CA derivations during queryMissing
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.

Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.

Fixes #8615.

Signed-off-by: Peter Waller <p@pwaller.net>
2023-08-09 20:57:04 +01:00
Théophane Hufschmitt d00fe5f225
Merge pull request #8805 from tweag/fix-add-to-store-existing
[V2] Fix misread of source if path is already valid
2023-08-08 14:57:45 +02:00
Théophane Hufschmitt afac001c39 Test the parallel copy over ssh-ng
Regression test for https://github.com/NixOS/nix/issues/6253
2023-08-08 11:55:09 +02:00
Eelco Dolstra 5624777988
Merge pull request #8786 from Ma27/fix-why-depends-precise
nix/why-depends: fix output of `--precise`
2023-08-07 19:32:49 +02:00
Théophane Hufschmitt 4999f42a70
Merge pull request #8322 from tweag/stabilize-discard-references
Stabilize `discard-references`
2023-08-07 17:35:02 +02:00
Théophane Hufschmitt ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
Maximilian Bosch 7c09104a94
nix/why-depends: fix output of --precise
I haven't checked when this was exactly introduced, but on Nix 2.16 I
realized that the additional lines inserted when using `--precise` are
completely separated from the tree:

    nix why-depends /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2 /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0 --precise --extra-experimental-features nix-command
    /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2
        → /nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0
            → /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0
    └───bin/evcxr: …':'}.PATH=${PATH/':''/nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0/bin'':'/':'}…
        └───bin/cpp: …k disable=SC2193.[[ "/nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0/bin/cpp" = *++ ]] &&…

This is apparently because `std::cout` is buffered and flushed in the
end whereas the rest of the output isn't. The fix is rather simple, just
use `logger->cout` as it's already the case for the rest of the code.

This way we also don't need to insert additional newlines in the `hits`
map since that's something the logger takes care of.

Also added a small test to make sure that the layout of this is somehow
tested to reduce the risk of further regressions here.
2023-08-04 23:11:08 +02:00
Eelco Dolstra d9e7758f47 Don't require .tar/.zip extension for tarball flakerefs
Special-casing the file name is rather ugly, so we shouldn't do
that. So now any {file,http,https} URL is handled by
TarballInputScheme, except for non-flake inputs (i.e. inputs that have
the attribute `flake = false`).
2023-08-01 16:07:20 +02:00
Robert Hensing 33d58a90c2 toJSON: Add attribute path to trace 2023-07-31 13:02:54 +02:00
Alex Ameen 2d1d81114d
Add parseFlakeRef and flakeRefToString builtins (#8670)
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.

I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.

I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-25 17:43:33 +00:00
Naïm Favier 570a1a3ad7
parser: merge nested dynamic attributes
Fixes https://github.com/NixOS/nix/issues/7115
2023-07-21 17:14:03 +02:00
John Ericson 453c4be93c
Merge pull request #8680 from NixLayeredStore/test-groups
Introduce notion of a test group, use for CA tests
2023-07-19 11:17:57 -04:00
Robert Hensing 0e3a7e34a0
Merge pull request #8506 from corngood/ssh-master
Pass NIX_SSHOPTS when checking for an ssh master connection.
2023-07-18 15:47:57 +02:00
John Ericson 259e328de8 Introduce notion of a test group, use for CA tests
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.

We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.

I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-18 09:31:13 -04:00
Théophane Hufschmitt a8d5bb5e7e
Merge pull request #8342 from NixLayeredStore/best-effort-supplementary-groups
Best effort supplementary groups
2023-07-17 20:58:17 +02:00
John Ericson 0f7242ff87 Test nested sandboxing, and make nicer error
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.

After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.

Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.

Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-14 09:40:24 -04:00
cidkidnix adb28d4a26 move unset NIX_STORE_DIR in supplementary-groups.sh
to inside the unshare
2023-07-13 14:23:24 -05:00
cidkidnix 1a13757880 Add comment regarding the unset of NIX_STORE_DIR
in build-remote.sh and supplementary-groups.sh
2023-07-13 14:18:12 -05:00
John Ericson 84c4e6f0ac Revert "Skip build-remote-trustless unless sandbox is supported."
This reverts commit 41412dc4ae.
2023-07-13 15:06:50 -04:00
John Ericson 9e64f24340 Revert "Check _NIX_TEST_NO_SANDBOX when setting _canUseSandbox."
This reverts commit c1d39de1fb.
2023-07-13 15:06:34 -04:00
John Ericson ddc0a2050b Merge remote-tracking branch 'upstream/master' into best-effort-supplementary-groups 2023-07-13 15:06:08 -04:00
John Ericson e072e18475 Fix race condition in the language tests
When we pipe to `>(...)` like that, we unfortunately don't wait for the
process to finish. Better to just substitute the file.

Also, use the "unified" diff output that people (including myself) are
more familiar with, thanks to Git.
2023-07-13 08:09:03 -04:00
Mathnerd314 c70484454f Expanded test suite
* Lang now verifies errors and parse output

* Some new miscellaneous tests

* Easy way to update the tests

* Document workflow in manual

* Use `!` not `~` as separater char for sed

  It is confusing to use `~` when we are talking about paths and home
  directories!

* Test test suite itself (`test/lang-test/infra.sh`)

Additionally, run shellcheck on `tests/lang.sh` to help ensure it is
correct, now that is is more complex.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-11 21:43:09 -04:00
Ben Radford 41412dc4ae
Skip build-remote-trustless unless sandbox is supported. 2023-07-11 12:52:59 +01:00
Ben Radford c1d39de1fb
Check _NIX_TEST_NO_SANDBOX when setting _canUseSandbox. 2023-07-11 12:08:33 +01:00
Ben Radford 07dabcc90e
Always attempt setgroups but allow failure to be ignored. 2023-07-11 10:44:05 +01:00
Ben Radford 25b20b4ad2
Merge remote-tracking branch 'origin/master' into best-effort-supplementary-groups 2023-07-11 09:38:34 +01:00
John Ericson c2c8187118 Fix test file name
It's UTF-8, not UFT-8.
2023-07-10 20:58:19 -04:00
John Ericson 028b26a77f
Merge pull request #8370 from hercules-ci/fetchClosure-input-addressed
`fetchClosure`: input addressed and pure
2023-07-09 23:41:22 -04:00
Robert Hensing 9fc82de493 signing.sh: Revert test improvement because it fails on GHA + macOS 2023-07-07 15:37:09 +02:00
Robert Hensing 3b3822ea1d tests: Reformat exit code error message
Now looks like:

Expected exit code '123' but got '0' from command 'echo' 'hi'
2023-07-07 15:08:25 +02:00
Eelco Dolstra 5fbfbb4c7c Fix test 2023-07-03 12:23:57 +02:00
Robert Hensing a6c17097d2 tests: Don't install test-libstoreconsumer program
Sorry about that.
Fixes https://github.com/NixOS/nix/issues/8616
2023-06-30 23:36:27 +02:00
Robert Hensing fefb947132 tests/signing.sh: Check signature checking error message
We should check error messages, so that we know the command fails for
the right reason.
Alternatively, a mere typo can run the test undetected.
2023-06-30 18:23:44 +02:00
Robert Hensing 1db81f7107 tests/fetchClosure: Improve coverage of new and some existing flows 2023-06-30 18:23:44 +02:00
Robert Hensing 40052c7613 fetchClosure: Docs and error message improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-30 18:23:42 +02:00
Robert Hensing ea30f152b7 fetchClosure: Allow input addressed paths in pure mode
When explicitly requested by the caller, as suggested in the meeting
(https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324)

> @edolstra: { toPath } vs { fromPath } is too implicit

I've opted for the `inputAddressed = true` requirement, because it
we did not agree on renaming the path attributes.

> @roberth: more explicit
> @edolstra: except for the direction; not immediately clear in which direction the rewriting happens

This is in fact the most explicit syntax and a bit redundant, which is
good, because that redundancy lets us deliver an error message that
reminds expression authors that CA provides a better experience to
their users.
2023-06-30 18:22:43 +02:00
Eelco Dolstra a0c617348b
Merge pull request #8589 from jfroche/sign-paths-as-allowed-user
Allow to sign path as unprivileged user
2023-06-30 13:13:42 +02:00
Jean-François Roche 80c9259756 Allow to sign path as unprivileged user
User can now sign path as unprivileged/allowed user

refs #1708
2023-06-27 18:31:31 +02:00
Maximilian Bosch 559fd7ffe7
nix flake check: improve error message if overlay is not a lambda (#8582)
* nix flake check: improve error message if overlay is not a lambda

Suppose you have an overlay like this

    {
      inputs = { /* ... */ };
      outputs = { flake-utils, ... }: flake-utils.lib.eachDefaultSystem
        (system: {
          overlays.default = final: prev: {

          };
        });
    }

then `nix flake check` (correctly) fails because `overlays` are supposed
to have the structure `overlays.<name> = final: prev: exp`. However, the
error-message is a little bit counter-intuitive:

    error: overlay does not take an argument named 'final'

While one might guess where the error actually comes from because the
trace above says `… while checking the overlay 'overlays.x86_64-linux'`
this is still pretty confusing because it complains about an argument
not being named `final` even though that's evidently the case.

With this change, the error-message actually makes it clear what's
wrong:

    [ma27@carsten:~/Projects/nix/tmp]$ nix flake check --extra-experimental-features 'nix-command flakes' path:$(pwd)
    error:
           … while checking flake output 'overlays'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           … while checking the overlay 'overlays.x86_64-linux'

             at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:

                3|   outputs = { ... }: {
                4|     overlays.x86_64-linux.snens = final: prev: {
                 |     ^
                5|       kek = throw "snens";

           error: overlay is not a lambda, but a set instead
2023-06-27 14:58:29 +02:00
Théophane Hufschmitt 60f06a1714
Merge pull request #5385 from Enzime/add/dirty-rev
Add `dirtyRev` and `dirtyShortRev` to `fetchGit`
2023-06-24 14:55:31 +02:00
Michael Hoang a7b49086c7 Add dirtyRev and dirtyShortRev to fetchGit
Fixes #4682
2023-06-24 14:17:25 +10:00
John Ericson 97df060588 Better document build failure exit codes
- Improved API docs from comment

- Exit codes are for `nix-build`, not just `nix-store --release`

- Make note in tests so the magic numbers are not surprising

Picking up where #8387 left off.
2023-06-22 14:29:45 -04:00
John Ericson 2291232dc1
Merge pull request #8387 from layus/fix-tests
Check exact error codes in linux-sandbox.sh
2023-06-22 13:56:03 -04:00
Guillaume Maudoux 3a20c7c46c
Update tests/linux-sandbox.sh
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-20 22:51:29 +02:00
Ben Radford 6ae35534b7
Support opening local store with database on read-only filesystem (#8356)
Previously it was not possible to open a local store when its database is on a read-only filesystem. Obviously a store on a read-only filesystem cannot be modified, but it would still be useful to be able to query it.

This change adds a new read-only setting to LocalStore. When set to true, Nix will skip operations that fail when the database is on a read-only filesystem (acquiring big-lock, schema migration, etc), and the store database will be opened in immutable mode.

Co-authored-by: Ben Radford <benradf@users.noreply.github.com>
Co-authored-by: cidkidnix <cidkidnix@protonmail.com>
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-20 11:34:09 +02:00
Eelco Dolstra e503eadafc
Merge pull request #8477 from edolstra/tarball-flake-redirects
Tarball flake improvements
2023-06-16 18:03:50 +02:00
Eelco Dolstra 713836112c
Merge pull request #8517 from hercules-ci/fix-build-hook-error-for-lib-users
Fix build hook error for libstore library users
2023-06-16 13:20:50 +02:00
Robert Hensing d2696cdd1e Fix build hook error for libstore library users
A library shouldn't require changes to the caller's argument handling,
especially if it doesn't have to, and indeed we don't have to.

This changes the lookup order to prioritize the hardcoded path to nix
if it exists. The static executable still finds itself through /proc
and the like.
2023-06-15 14:32:00 +02:00
Valentin Gagarin c453719d6e rename files referring to antiquotation
since we renamed this to string interpolation, file names should be
fixed up as well
2023-06-15 02:29:31 +02:00
John Ericson 946cd9e3f9
Merge pull request #8351 from obsidiansystems/delete-profiles-tests-docs
Expanding tests and docs relating to deleting profiles
2023-06-15 01:47:21 +02:00
John Ericson d4a2ced9cb Split out nix-collect-garbage -d test to new file
Good for test parallelism, and separation of concerns (core GC vs
profiles deleting).
2023-06-14 19:01:07 -04:00
John Ericson ca5752d4fa Add another case to the nix-collect-garbage -d test 2023-06-14 19:01:07 -04:00
Michal Sojka a0c4d58549 Remove RegisterPrimOp constructor without support for documentation
The remaining constructor RegisterPrimOp::RegisterPrimOp(Info && info)
allows specifying the documentation in .args and .doc members of the
Info structure.

Commit 8ec1ba0210 removed all uses of the removed constructor in the
nix binary. Here, we remove the constructor completely as well as its
use in a plugin test. According to #8515, we didn't promis to maintain
compatibility with external plugins.

Fixes #8515
2023-06-14 22:37:52 +02:00
John Ericson 05eb06a1de
Merge pull request #8490 from flox/stdin_handling
fix: Do not apply default installables when using --stdin
2023-06-14 20:41:22 +02:00
John Ericson ff905cb796
Merge pull request #4803 from ShamrockLee/nix-channel-list-generations
Add `nix-channel --list-generations`
2023-06-14 18:30:35 +02:00
John Ericson 61a3e1f2e2
Merge pull request #4282 from tweag/fix-ca-hash-rewriting
fix the hash rewriting for ca-derivations
2023-06-14 18:25:00 +02:00
Eelco Dolstra 1ad3328c5e Allow tarball URLs to redirect to a lockable immutable URL
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
2023-06-13 14:17:45 +02:00
David McFarland 5454fdcceb Add test of explicit ssh control path in nix-copy test
This highlights a problem caused by SSHMaster::isMasterRunning returning
false when NIX_SSHOPTS contains -oControlPath.
2023-06-13 00:54:52 -03:00
Tom Bereknyei 0e3849dc65 test: add test for non-defaulting for stding installable input 2023-06-12 08:40:17 -04:00
Eelco Dolstra 381a32981b
Merge branch 'master' into angerman/mac-fix-recursive-nix 2023-06-09 13:06:47 +02:00
Andrea Bedini 3c78920f73
Parse TOML timestamps (#8120)
Currently `fromTOML` throws an exception when encountering a timestamp
since the Nix language lacks a way to represent them.

This patch changes this beaviour and makes `fromTOML` parse timestamps as
attrsets of the format

  { _type = "timestamp"; value = "1979-05-27T07:32:00Z"; }

This is guarded by an experimental feature flag to leave room for iterating on the representation.
2023-06-09 11:53:18 +02:00
Eelco Dolstra f5c6b29940 Fix SourcePath::resolveSymlinks()
This fixes handling of symlinks that start with '..', and symlink
targets that contain symlinks themselves.
2023-06-06 11:24:10 +02:00
Théophane Hufschmitt 0101ce0d96 Test nix-channel --list-generations
Rough test, but the feature is a fairly trivial addition on top of
`nix-profile --list-generations`, so it should be enough
2023-06-02 10:21:30 +02:00
polykernel a382919d7d
primops: lazy evaluation of replaceStrings replacements
The primop `builtins.replaceStrings` currently always strictly evaluates the
replacement strings, however time and space are wasted for their computation
if the corresponding pattern do not occur in the input string. This commit
makes the evaluation of the replacement strings lazy by deferring their
evaluation to when the corresponding pattern are matched and memoize the result
for efficient retrieval on subsequent matches.

The testcases for replaceStrings was updated to check for lazy evaluation
of the replacements. A note was also added in the release notes to
document the behavior change.
2023-05-25 18:35:23 -04:00
Théophane Hufschmitt d0cecbe877 Disable the fetchClosure test for old daemons
Broken because of the change introduced by #4282
2023-05-24 15:35:46 +02:00
Guillaume Maudoux f0233f3a3f Further refactor linux-sandbox.sh and fix tee usage 2023-05-23 10:05:56 +02:00
Peter Becich a420ccc6a8
nix flake check: skip derivations for foreign systems (#7759)
`nix flake show` now skips derivations for foreign systems: https://github.com/NixOS/nix/pull/6988

This commit borrows from that to implement the same behavior for `nix flake check`.

See "nix flake check breaks on IFD in multi-platform flake" https://github.com/NixOS/nix/issues/4265
2023-05-23 06:59:44 +02:00
Guillaume Maudoux b9c2f834ee Check exact error codes in linux-sandbox.sh 2023-05-22 23:39:31 +02:00
Guillaume Maudoux 5a98dd0b39 Add tests for bind mount of SSL certs in sandbox 2023-05-22 02:32:09 +02:00
John Ericson 32dc77ba5d
Merge pull request #8349 from tweag/fix-control-master
Fix ControlMaster behaviour
2023-05-17 12:17:09 -04:00
Alexander Bantyev 992e2ed0cf
Add a test for ControlMaster 2023-05-17 11:34:45 +04:00
John Ericson 5fd161189d
Merge pull request #8346 from tweag/fix-nix-profile-install-conflict-segfault
Fix the segfault on `nix profile install` with conflict
2023-05-16 15:49:43 -04:00
John Ericson 0a715ff9cf
Merge pull request #8154 from tweag/delete-old-on-all-profiles-dir
undefined
2023-05-16 09:51:41 -04:00
Théophane Hufschmitt e997512523 Fix the printing of the installables on nix profile install conflict
- If the element comes from a flake, print the full flakeref (with the
  fragment part) and not just the reference to the flake itself
- If the element doesn't come from a flake, print its store path(s)

This is a bit too verbose, but has the advantages of being correct (and
not crashing), so it's strictly better than the previous situation

Fix https://github.com/NixOS/nix/issues/8284
2023-05-16 11:41:58 +02:00
John Ericson 754ced4a3f Avoid out links in supplementary groups test
This gets in the way of the tests running in parallel.
2023-05-15 17:49:28 -04:00
John Ericson d8ef0c9495 Add some tests for drop-supplementary-groups 2023-05-15 17:41:51 -04:00
John Ericson d2162e7acd Make more string values work as installables
As discussed in #7417, it would be good to make more string values work
as installables. That is to say, if an installable refers to a value,
and the value is a string, it used to not work at all, since #7484, it
works somewhat, and this PR make it work some more.

The new cases that are added for `BuiltPath` contexts:

- Fixed input- or content-addressed derivation:

  ```
  nix-repl> hello.out.outPath
  "/nix/store/jppfl2bp1zhx8sgs2mgifmsx6dv16mv2-hello-2.12"

  nix-repl> :p builtins.getContext hello.out.outPath
  { "/nix/store/c7jrxqjhdda93lhbkanqfs07x2bzazbm-hello-2.12.drv" = { outputs = [ "out" ]; }; }

  The string matches the specified single output of that derivation, so
  it should also be valid.

- Floating content-addressed derivation:

  ```
  nix-repl> (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
  "/1a08j26xqc0zm8agps8anxpjji410yvsx4pcgyn4bfan1ddkx2g0"

  nix-repl> :p builtins.getContext (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
  { "/nix/store/qc645pyf9wl37c6qvqzaqkwsm1gp48al-hello-2.12.drv" = { outputs = [ "out" ]; }; }
  ```

  The string is not a path but a placeholder, however it also matches
  the context, and because it is a CA derivation we have no better
  option. This should also be valid.

We may also want to think about richer attrset based values (also
discussed in that issue and #6507), but this change "completes" our
string-based building blocks, from which the others can be desugared
into or at least described/document/taught in terms of.

Progress towards #7417

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-15 09:03:38 -04:00
Théophane Hufschmitt e97e9e9f00 test the garbage collection with the old profile dir
Regression test for https://github.com/NixOS/nix/issues/8294
2023-05-15 11:58:49 +02:00
John Ericson a93110ab19 Fix nix print-dev-env & nix develop with drv paths
Fixes #8309

This regression was because both `CmdDevelop` and `CmdPrintDevEnv` were
switched to be `InstallableValueCommand` subclasses, but actually
neither should have been.

The `nixpkgsFlakeRef` method should indeed not be on the base
installable class, because "flake refs" and "nixpkgs" are not
installable-wide notions, but that doesn't mean these commands should
only accept installable values.
2023-05-10 11:29:45 -04:00
John Ericson 53a1354acf
Merge pull request #3959 from obsidiansystems/ca-drv-exotic
Derivations can output "text-hashed" data
2023-05-10 10:41:59 -04:00
Théophane Hufschmitt 85ff212051
Merge pull request #7721 from yorickvP/post-build-hook
Also pass unwanted outputs to post-build-hook
2023-05-10 14:30:42 +02:00
Alex Ameen 82d1d74a85
quote subshell expansion in tests/eval.sh 2023-05-09 10:06:26 -05:00
Alex Ameen 82296f8113
prevent double quotation 2023-05-09 09:59:18 -05:00
Alex Ameen b72bc4a972
libexpr: quote reserved keys when printing
This fixes a bug in commands like `nix eval' which would emit invalid attribute
sets if they contained reserved keywords such as "assert", "let", etc.

These keywords will not be quoted when printed, making them valid expressions.
All keywords recognized by the lexer are quoted except "or", which does not
require quotation.
2023-05-09 09:45:12 -05:00
John Ericson b5d9ef0a4c
Merge pull request #3921 from obsidiansystems/trustless-remote-builder-simple
Trustless remote building for input-addressed drvs
2023-05-08 10:43:37 -04:00
John Ericson 278c94d607 Rename a few things in new tests
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-05-08 08:01:58 -04:00
Yorick d1ff33d2d6
tests/post-hook: remove TODO and --derivation upload 2023-05-08 12:58:59 +02:00
Yorick 5e332aa503
tests: copying only the out paths is not enough information for CA 2023-05-08 12:58:59 +02:00
Yorick 869fb1a2f6
tests/post-hook: test to see if all outputs are passed
fe5509df caused only wanted outputs to be passed to the
post-build-hook, which resulted in paths being built
without ever going into the hook.

This commit adds a (currently failing) test for this.
2023-05-08 12:43:56 +02:00
Moritz Angermann 0e18254aa8
Fix shutdown behavior and resource management for recursive-nix on macOS
Previously, we relied on the `shutdown()` function to terminate `accept()`
calls on a listening socket. However, this approach did not work on macOS as
the waiting `accept()` call is not considered a connected socket, resulting in
an `ENOTCONN` error. Instead, we now close the listening socket to terminate
the `accept()` call.

Additionally, we fixed a resource management issue where we set the
`daemonSocket` variable to -1, triggering resource cleanup and causing the
`stopDaemon` function to be called twice. This resulted in errors as the socket
was already closed by the time the second `stopDaemon` call was made. Instead of
setting `daemonSocket` to -1, we now release the socket using the `release()`
method on a unique pointer. This properly transfers ownership and allows for
correct resource cleanup.

These changes ensure proper behavior and resource management for the
recursive-nix feature on macOS.
2023-04-25 09:39:05 +08:00
Eelco Dolstra 01232358ff Merge remote-tracking branch 'origin/master' into source-path 2023-04-24 13:20:36 +02:00
John Ericson 969def696a Fix typo in tests 2023-04-19 20:47:23 -04:00
John Ericson e26662709e Add a more interesting test
In this one, we don't just output an existing derivation as is, but
modify it first.
2023-04-19 20:36:33 -04:00
John Ericson 3eb343754e Move test/recursive.sh nix expr to file
I found it hard to read as a big string literal.
2023-04-19 19:36:05 -04:00
John Ericson f3a31b14db Make tests/dyn-drv test dir 2023-04-19 18:49:50 -04:00
John Ericson 61d3e64fd0 Require daemon version for text hashing test 2023-04-19 17:24:55 -04:00
John Ericson 76baaeb341 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-19 11:32:14 -04:00
John Ericson 668377f217 TextHashMethod -> TextIngestionMethod, gate with XP feature
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-17 19:02:45 -04:00
John Ericson f56c4a5bdf Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 18:10:12 -04:00
John Ericson aa74c7b0bc Gate experimental features in DerivationOutput::fromJSON
This is an entry point for outside data, so we need to check enabled
experimental features here.
2023-04-17 17:36:12 -04:00
John Ericson ab5ca608bf Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 13:54:48 -04:00
Robert Hensing 64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
John Ericson d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00
John Ericson 2c8475600d Fix some issues with experimental config settings
Issues:

1. Features gated on disabled experimental settings should warn and be
   ignored, not silently succeed.

2. Experimental settings in the same config "batch" (file or env var)
   as the enabling of the experimental feature should work.

3. For (2), the order should not matter.

These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 12:41:04 -04:00
John Ericson e12efa3654 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 10:16:57 -04:00
John Ericson e95db8f2b9 nix-testing -> daemon-trust-override
And only enable in the tests that need it. This makes it less of a
sledgehammer.
2023-04-17 09:35:43 -04:00
John Ericson b1343e8ad1 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 09:27:17 -04:00
Robert Hensing cb2615cf47 Merge remote-tracking branch 'upstream/master' into source-path 2023-04-17 11:41:50 +02:00
Robert Hensing 9af9c260fc
Merge pull request #8193 from hercules-ci/dry-strings
Deduplicate string literal rendering, fix 4909
2023-04-17 11:19:40 +02:00
John Ericson 615c25b0dd Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-12 00:00:12 -04:00
John Ericson 450e5ec618 Do not gate or hide experimental settings
This is somewhat hacky fix just for 2.15. I unintentionally hid them
from the manual, when no one wanted to hide them that (including
myself). I also required the experimental feature to be enabled in an
order-dependent way, which is not good.

The simplest fix for this immanent release is just to always show them,
and always allow them to be set.

Effectively undoes some changes from aa663b7e89
2023-04-11 10:56:48 -04:00
Eelco Dolstra 45ca4e6432
Merge pull request #8158 from tweag/harden-profiles-test
Fix the flaky `nix-profile` test
2023-04-11 12:12:09 +02:00
Robert Hensing 4e0804c920 Deduplicate string literal rendering, fix 4909 2023-04-09 22:42:20 +02:00
John Ericson 51c8ffbc28 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-08 18:49:26 -04:00
John Ericson 38ae7c2891 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 11:13:35 -04:00
John Ericson 6e1e15ffec Fix it! 2023-04-07 11:13:23 -04:00
John Ericson c036de086e Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 09:40:36 -04:00
John Ericson 59e0728714 Create nix derivation add command
Also refine `nix derivation show`'s docs very slightly.
2023-04-07 08:34:58 -04:00
John Ericson 2b98af2e62 nix show-derivation -> nix derivation show 2023-04-07 08:34:58 -04:00
matthewcroughan 9207f94582 Add Store::isTrustedClient()
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.

We include this information in `nix ping store` and `nix doctor`

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2023-04-06 19:59:57 -04:00
Eelco Dolstra 94812cca98 Backport SourcePath from the lazy-trees branch
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
2023-04-06 13:15:50 +02:00
Théophane Hufschmitt 7076d37047 Fix the flaky nix-profile test
Exclude the `error (ignored)` from the message that is checked by the
install conflict test.

Fix https://github.com/NixOS/nix/issues/8140
2023-04-03 20:47:21 +02:00
Eelco Dolstra 81491e1379
Merge pull request #8042 from lheckemann/alt-lockfiles
Allow specifying alternative paths for reading/writing flake locks
2023-04-03 19:28:09 +02:00
John Ericson 4a0b893d5e Stuctured command stability
Prior to this, there was an ad-hoc whitelist in `main.cc`. Now, every
command states its stability.

In a future PR, we will adjust the manual to take advantage of this new
information in the JSON.
(It will be easier to do that once we have some experimental feature
docs to link too; see #5930 and #7798.)
2023-04-03 11:48:21 -04:00
Robert Hensing f3a6de6ba9
Merge pull request #8148 from hercules-ci/fix-issue-8119-printValue-tBlackhole-abort
eval: Fix crash on missing printValue tBlackhole case
2023-04-03 16:05:55 +02:00
Robert Hensing 1c55544a42 eval: Fix crash on missing printValue tBlackhole case
Fixes #8119
2023-04-03 15:32:42 +02:00
Théophane Hufschmitt 70bb7b7289
Merge pull request #7610 from obsidiansystems/gate-default-settings
Punt on improper global flags for now
2023-04-03 14:02:45 +02:00
John Ericson f7f44f7c96 Merge commit 'aa99005004bccc9be506a2a2f162f78bad4bcb41' into ca-drv-exotic 2023-04-01 15:15:32 -04:00
Théophane Hufschmitt 3f362f1c07 test: Make the installer work on old rhel versions
`scp -r` doesn't seem to behave properly there, so tarpipe ftw!
2023-03-31 15:29:50 +02:00
Théophane Hufschmitt 4e7a78ca97 test: Don't add a channel for the force-no-daemon installer variant
Was probably an overlook of when the tests were first added, but that
now messes-up with the `nix-channel --update` that happens down the line
(and can't access the network since we're inside a Nix build)
2023-03-31 15:27:02 +02:00
Théophane Hufschmitt e32ca3cf16
Merge pull request #8018 from tweag/ssh-password-prompt
SSH: don't erase password prompt if it is displayed
2023-03-31 12:06:10 +02:00
Eelco Dolstra 237587bc0a
Merge pull request #8084 from edolstra/store-docs
Auto-generate store documentation
2023-03-27 15:46:18 +02:00
John Ericson 1d539aa287
Merge pull request #8073 from tweag/fix-root-channels-location
Fix root channels location
2023-03-27 09:37:53 -04:00
John Ericson 7c4dea3cf3 Punt on improper global flags for now
See the note in the test.

We don't want these flags showing up for commands where they are
irrelevant.

Eventually, this needs a proper fix, but it need not be a blocker for
stabilize: for a quick-n-dirty punt, just put these flags behind the
`nix-command` unstable feature.

This is fine because they are only relevant for commands which we don't
need to stabilize for a while.
2023-03-27 09:21:50 -04:00
John Ericson 570829d67e
Merge pull request #7609 from obsidiansystems/hide-experimental-settings
Hide experimental settings
2023-03-27 09:19:29 -04:00
Robert Hensing 6b87c6577f
tests/flakes/show.sh: Broaden requirement comment 2023-03-27 05:11:23 +02:00
oxalica 2941a599fa Catch eval errors in hasContent
`legacyPackages` of nixpkgs trigger eval errors in `hasContent`, causing
the whole `legacyPackages` being skipped. We should treat it as
has-content in that case.
2023-03-26 23:05:29 +08:00
Théophane Hufschmitt 717e81df13 Test the installation of a simple package in the install tests 2023-03-23 21:58:59 +01:00
Alexander Bantyev 85a2d1d94f
Add a test for nix copy over ssh
Check that nix copy can copy stuff, refuses to copy unsigned paths by
default, and doesn't hide the ssh password prompt.
2023-03-22 09:45:08 +04:00
Eelco Dolstra 7704118d28 nix describe-stores: Remove
This command was intended for docs generation, but it was never used
for that and we don't need it.
2023-03-21 14:03:40 +01:00
John Ericson 1b6c96bbcb Write test, will fail until rest of PR 2023-03-20 09:17:33 -04:00
Linus Heckemann 3c3bd0767f Create test lockfiles in TEST_ROOT 2023-03-19 14:14:30 +01:00
Linus Heckemann ea207a2eed Add tests for alternate lockfile path functionality 2023-03-19 14:11:19 +01:00
Robert Hensing bf0c8c34e7
tests/flake-in-submodule: Remove unnecessary rm -rf 2023-03-18 14:39:38 +01:00
Josef Kemetmüller f9c24d67b9 Add a test with flake.nix in a submodule
I noticed a regression in the lazy-trees branch, which I'm trying to
capture with this test. While the tests succeeds in master, the
lazy-trees branch gives the following error message:

    error: access to path
    '/build/nix-test/tests/flakes/flake-in-submodule/rootRepo/submodule/flake.nix'
    is forbidden because it is not under Git control; maybe you should
    'git add' it to the repository
    '/build/nix-test/tests/flakes/flake-in-submodule/rootRepo'?
2023-03-17 22:10:28 +01:00
John Ericson bfb9eb87fe Cleanup test skipping
- Try not to put cryptic "99" in many places

  Factor out `exit 99` into `skipTest` function

- Alows make sure skipping a test is done with a reason

  `skipTest` takes a mandatory argument

- Separate pure conditionals vs side-effectful test skipping.

  "require daemon" already had this, but "sandbox support" did not.
2023-03-16 18:43:03 -04:00
Yueh-Shun Li c27d358abb nix-hash: support base-64 and SRI format
Add the --base64 and --sri flags for the Base64 and SRI format output.

Add the --base16 flag to explicitly specify the hexadecimal format.

Add the --to-base64 and --to-sri flag to convert a hash to the above
mentioned format.
2023-03-16 03:08:42 +08:00
Yueh-Shun Li 08510c5ed0 test/hash.sh: add to-base32 test for nix hash 2023-03-16 00:50:26 +08:00
Yueh-Shun Li 2f32303f93 tests/hash.sh: try: Use FORMAT_FLAG instead of EXTRA
Do not rely on the "multiple format flag specified" behavior.

Explicitly test without the format flag / with the --base16 flag.
2023-03-16 00:49:57 +08:00
Robert Hensing a387f46967
Merge pull request #8033 from lbodor/stop-adding-dot-to-nix-dev-env-path
`print-dev-env`: stop inadvertently adding `.` to `PATH`
2023-03-13 19:52:41 +01:00
Lazar Bodor a3a6909bc8 Use $TEST_ROOT 2023-03-14 01:48:12 +11:00
Eelco Dolstra f056468959 make clean: Delete vars-and-functions.sh 2023-03-13 13:11:03 +01:00
Lazar Bodor e210de4799 print-dev-env: test the case when PATH is empty 2023-03-13 17:50:36 +11:00
John Ericson c11836126b Harden tests' bash
Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.

 - `set -u` catches the use of undefined variables
 - `set -o pipefail` catches failures (like `set -e`) earlier in the
   pipeline.

This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.

There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.

To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.

`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.

`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.

`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-03-08 10:26:30 -05:00
Théophane Hufschmitt 2272bc6049 Fix nix-collect-garbage -d with the new profile location
Low-hanging fix for https://github.com/NixOS/nix/pull/5226#issuecomment-1454669399
2023-03-07 13:08:04 +01:00
John Ericson f9443143ae Remove needless --experimental-feature in a CA drvs test
This is already blanket enabled for these tests
2023-03-01 18:04:28 -05:00
Valentin Gagarin 306e5c5ce5
Merge pull request #7788 from bobvanderlinden/pr-improve-nix-profile-install-error
Improve error on conflict for nix profile install
2023-03-01 11:48:43 +01:00
John Ericson ea0adfc582 Get rid of .drv special-casing for store path installables
The release notes document the change in behavior, I don't include it
here so there is no risk to it getting out of sync.

> Motivation

>> Plumbing CLI should be simple

Store derivation installations are intended as "plumbing": very simple
utilities for advanced users and scripts, and not what regular users
interact with. (Similarly, regular Git users will use branch and tag
names not explicit hashes for most things.)

The plumbing CLI should prize simplicity over convenience; that is its
raison d'etre. If the user provides a path, we should treat it the same
way not caring what sort of path it is.

>> Scripting

This is especially important for the scripting use-case. when arbitrary
paths are sent to e.g. `nix copy` and the script author wants consistent
behavior regardless of what those store paths are. Otherwise the script
author needs to be careful to filter out `.drv` ones, and then run `nix
copy` again with those paths and `--derivation`. That is not good!

>> Surprisingly low impact

Only two lines in the tests need changing, showing that the impact of
this is pretty light.

Many command, like `nix log` will continue to work with just the
derivation passed as before. This because we used to:

- Special case the drv path and replace it with it's outputs (what this
  gets rid of).

- Turn those output path *back* into the original drv path.

Now we just skip that entire round trip!

> Context

Issue #7261 lays out a broader vision for getting rid of `--derivation`,
and has this as one of its dependencies. But we can do this with or
without that.

`Installable::toDerivations` is changed to handle the case of a
`DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't
need to do any extra work in that case. On this basis, commands like
`nix {show-derivation,log} /nix/store/...-foo.drv` still work as before,
as described above.

When testing older daemons, the post-build-hook will be run against the
old CLI, so we need the old version of the post-build-hook to support
that use-case.

Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-28 17:07:05 -05:00
John Ericson 5abd643c6d Merge branch 'path-info' into ca-drv-exotic 2023-02-28 12:46:00 -05:00
Bob van der Linden 0167862e8e
fixup! tests: nix-profile: test install error message upon conflicting files 2023-02-28 10:53:42 +01:00
Bob van der Linden 3efa476c54
tests: nix-profile: test install error message upon conflicting files 2023-02-28 09:28:05 +01:00
Eelco Dolstra 3d15dbadc2
Merge pull request #7911 from edolstra/revert-7689
Revert #7689
2023-02-28 08:46:55 +01:00
Théophane Hufschmitt eae89aca1b
Merge pull request #7776 from yorickvP/fix-path-escapes-7707
Properly escape local paths into URLs in fetchTree
2023-02-27 21:10:25 +01:00
Yorick ae5082bbba
tests/fetchGit: add regression test for #7707 2023-02-27 15:33:54 +01:00
Eelco Dolstra dd93c12c6a Revert "getDefaultNixPath: actually respect {restrict,pure}-eval"
This reverts commit 1cba5984a6.
2023-02-27 15:11:36 +01:00
Théophane Hufschmitt 995bfeef3b
Merge pull request #7796 from hercules-ci/fix-7263
Ensure that `self.outPath == ./.`
2023-02-27 10:26:02 +01:00
John Ericson 87da941348 Clean up daemon handling
Split `common.sh` into the vars and functions definitions vs starting
the daemon (and possibly other initialization logic). This way,
`init.sh` can just `source` the former. Trying to start the daemon
before `nix.conf` is written will fail because `nix daemon` requires
`--experimental-features 'nix-command'`.

`killDaemon` is idempotent, so it's safe to call when no daemon is
running.

`startDaemon` and `killDaemon` use the PID (which is now exported to
subshells) to decide whether there is work to be done, rather than
`NIX_REMOTE`, which might conceivably be set differently even if a
daemon is running.

`startDaemon` and `killDaemon` can save/restore the old `NIX_REMOTE` as
`NIX_REMOTE_OLD`.

`init.sh` kills daemon before deleting everything (including the daemon
socket).
2023-02-23 11:31:44 -05:00
John Ericson 5dbbf23332 Make init.sh safe to run twice
`init.sh` is tested on its own. We used to do that. I deleted it in
4720853129 but I am not sure why. Better
to just restore it; at one point working on this every other test
passed, so seems good to check whether `init.sh` can be run twice.

We don't *need* to run `init.sh` twice, but I want to try to make our
tests as robust as possible so that manual debugging (where tests for
better or worse might be run ways that we didn't expect) is less
fragile.
2023-02-23 11:31:44 -05:00
Robert Hensing 5d834c40d0 flakes: Differentiate self.outPath and self.sourceInfo.outPath
It would be incorrect to say that the `sourceInfo` has an `outPath`
that isn't the root. `sourceInfo` is about the root, whereas only
the flake may not be about the root. Thanks Eelco for pointing that
out.
2023-02-22 03:31:24 +01:00
Robert Hensing 904a107d16 flakes: Ensure that self.outPath == ./.
Users expect `self` to refer to the directory where the `flake.nix`
file resides.
2023-02-22 03:30:47 +01:00
John Ericson d7a4f08d42
Nix's own flake: Dedup and memoize more
- `nixpkgsFor` does all of native, static, cross, and the different stdenvs.

- The main Nix derivation is no longer duplicated for static.

- DRY nixpkgs.lib and lib.genAttrs calls.
2023-02-20 11:35:51 +01:00
Valentin Gagarin dda83a59c1
Merge pull request #7158 from sternenseemann/foldl-strict-accumulation-value 2023-02-19 23:54:14 +01:00
Yorick 49fd72a903
Make /etc writability conditional on uid-range feature 2023-02-14 13:55:41 +01:00
Yorick ad1f61c39b
container test: make /etc writable 2023-02-14 12:26:40 +01:00
Yorick db41f74af3
Don't allow writing to /etc 2023-02-14 12:03:34 +01:00
John Ericson 55016b6fcd Test nix build --json return output paths in floating CA case
Adding a test to ensure there is no regression.

The tests that are split out of `tests/build.sh` are ones that don't yet
work with CA derivation. I have not yet evaluated whether they should or
not.

This behavior, reported missing in issue #4661, already got fixed in
PR #4818, but didn't get a test case then.
2023-02-10 18:04:13 -05:00
Eelco Dolstra c184566046
Merge pull request #7797 from hercules-ci/tests-set-ps4
tests: Add command source locations to test log
2023-02-10 20:55:57 +01:00
Eelco Dolstra 67451d8ed7
Merge pull request #7802 from edolstra/fix-7783
Fix PID namespace support check
2023-02-10 20:41:13 +01:00
Théophane Hufschmitt 9ebbe35817
Merge pull request #5588 from tweag/balsoft/xdg
Follow XDG Base Directory standard
2023-02-10 18:05:50 +01:00
Eelco Dolstra a21405a4e8 Add regression test 2023-02-10 17:51:44 +01:00
Alexander Bantyev 2384d36083
A setting to follow XDG Base Directory standard
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:

- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels

This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:

- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels

If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
2023-02-10 20:14:06 +04:00
Robert Hensing 1a0b293eb2
Merge pull request #7775 from hercules-ci/test-authorization
Add a basic daemon authorization test
2023-02-10 13:36:45 +01:00
Robert Hensing 7908a41631
tests/authorization: Simplify assertion
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-10 13:03:24 +01:00
Robert Hensing 9813e54a74 tests: Add command source locations to test log 2023-02-09 22:14:53 +01:00
Théophane Hufschmitt 5597d68e2d
Merge pull request #7754 from obsidiansystems/narrower-scope-derivation-flag
Scope down `--derivation` to just the commands that use it
2023-02-09 19:51:43 +01:00
Eelco Dolstra 9a7dc5d718 Add some tests 2023-02-07 22:46:25 +01:00
Robert Hensing 72b18f05a2 Add a basic daemon authorization test 2023-02-07 16:43:09 +01:00
John Ericson 6352e20bc8 Remove --derivation from test
It doesn't do anything here, and in the next commit `show-derivation
will no longer accept this flag.
2023-02-04 18:30:02 -05:00
Eelco Dolstra 7a09bfbcb6
Merge pull request #7723 from yorickvP/nix-store-ping-json
nix store ping: add --json flag
2023-02-01 17:11:34 +01:00
Robert Hensing 60d48eda23 nix flake show: Ignore empty attrsets
For frameworks it's important that structures are as lazy as possible
to prevent infinite recursions, performance issues and errors that
aren't related to the thing to evaluate. As a consequence, they have
to emit more attributes than strictly (sic) necessary.
However, these attributes with empty values are not useful to the user
so we omit them.
2023-01-31 18:20:26 +01:00
Yorick 4757b3f04e
tests/store-ping: test nix store ping --json 2023-01-31 15:10:54 +01:00
Eelco Dolstra c79b1582a7
Merge pull request #5226 from NixOS/client-side-profiles
Move the default profiles to the user’s home
2023-01-30 12:21:47 +01:00
Théophane Hufschmitt 575d0aea5d
Merge pull request #6988 from max-privatevoid/pr-flake-show-foreign
nix flake show: don't evaluate derivations for foreign systems by default
2023-01-30 12:06:37 +01:00
Théophane Hufschmitt 3ac9f1658a Fix the flakes init test
Things leading to another...
2023-01-30 11:21:52 +01:00
Théophane Hufschmitt bc6e65e26f Fix the flakes/show test
Don't hardcode “x86_64-linux” as this won't work too nicely on other
platforms
2023-01-30 10:44:10 +01:00
Théophane Hufschmitt 4aaf0ee52e
Merge branch 'master' into referenceablePaths 2023-01-30 10:31:00 +01:00
Théophane Hufschmitt d70b890488
Merge pull request #7689 from ncfavier/nix-path-restrict-eval
getDefaultNixPath: actually respect `{restrict,pure}-eval`
2023-01-30 10:03:17 +01:00
Théophane Hufschmitt ccaadc9575
Merge pull request #7648 from hercules-ci/move-nixos-tests
Move nixos tests
2023-01-27 15:11:48 +01:00
Naïm Favier 1cba5984a6
getDefaultNixPath: actually respect {restrict,pure}-eval
Previously, getDefaultNixPath was called too early: at initialisation
time, before CLI and config have been processed, when `restrictEval` and
`pureEval` both have their default value `false`. Call it when
initialising the EvalState instead, and use `setDefault`.
2023-01-27 13:28:57 +01:00