Commit graph

1920 commits

Author SHA1 Message Date
John Ericson f7f37035c8 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

(cherry picked from commit 91b6833686a6a6d9eac7f3f66393ec89ef1d3b57)
(cherry picked from commit a61e42adb528b3d40ce43e07c79368d779a8b624)
2023-12-01 13:05:03 -05:00
John Ericson 30dcc19d1f Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```

(cherry picked from commit 68c81c737571794f7246db53fb4774e94fcf4b7e)
2023-12-01 12:06:43 -05:00
Valentin Gagarin 8fb477a6e7 add path based redirects
up to now, those were managed outside of this repo, which as
unsurprisingly a real hassle to deal with if one wanted to prevent URLs
from breaking when moving pages around. this change removes a large part
of the friction involved in moving content in the Nix manual.

possible next steps for further automation:
- check for content that moved and warn if it's not reachable from
  links that were valid prior to a change
- create redirect rules automatically based on this information

(cherry picked from commit 2b7016cc56d12e67de9f1f25b18311866a26a5fe)
2023-11-27 13:29:00 +00:00
Valentin Gagarin a68bf15fe6 backport fix for the --help output 2023-11-10 19:22:42 +01:00
Eelco Dolstra 44fb119218 Mark official release 2023-09-20 12:49:01 +02:00
Eelco Dolstra 10ad052f7d Release notes 2023-09-20 11:42:49 +02:00
Emil Nikolov 2cdc9c32e7
docs: fixed the default priority of nix-env --install (#8945) 2023-09-09 06:54:39 +00:00
John Ericson 7ad66cb3ef Allow dynamic derivation deps in inputDrvs
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.

The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)

`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.

As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

Apply suggestions from code review

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-07 10:39:37 -04:00
John Ericson 3a62651bd6
Merge pull request #8933 from fricklerhandwerk/option-anchors
Add anchors to option listings
2023-09-06 09:39:33 -04:00
John Ericson 216c66ddcf
Merge pull request #8932 from fricklerhandwerk/formatting
dedent common options listing; one sentence per line
2023-09-06 09:30:25 -04:00
John Ericson 754528241a
Merge pull request #8927 from obsidiansystems/test-derivation-aterm
Test and begin documentation of the ATerm format for derivations
2023-09-06 09:24:52 -04:00
John Ericson d568877eab
Retitle section as Robert suggests
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-06 08:43:16 -04:00
Valentin Gagarin 391f18063c add anchors to option listings 2023-09-06 09:52:56 +02:00
Valentin Gagarin 02c2679f0e dedent common options listing; one sentence per line
this is a pure reformatting, contents were not changed

one sentence per line makes reviewing diffs and making suggestions much
more convenient. the indentation was an artifat of the DocBook
migration.
2023-09-06 09:49:00 +02:00
Valentin Gagarin 5c23d3a90c disambiguate output from output path 2023-09-06 09:09:02 +02:00
John Ericson 880d9cabed Test and begin documentation of the ATerm format for derivations
Wanted to do this before the last dynamic derivations PR when I
introduce a variation, to make sure I wasn't changing the old version by
mistake.
2023-09-05 11:16:39 -04:00
Valentin Gagarin f34484d18a
Merge pull request #7592 from fricklerhandwerk/nix-store-realise
reword description of how realisation works
2023-09-05 11:55:01 +02:00
Eelco Dolstra 919781cacc
Merge branch 'master' into valid_deriver_2 2023-09-01 13:35:05 +02:00
Valentin Gagarin 1ac181759d revert some random change 2023-08-31 21:27:13 +02:00
Valentin Gagarin d38a539437 make description open-ended, add TODO
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-31 21:27:11 +02:00
Valentin Gagarin 894cbe43bc don't invent terms yet 2023-08-31 21:26:46 +02:00
Valentin Gagarin b951e862d0 more meaningful tagline 2023-08-31 21:26:46 +02:00
Valentin Gagarin cf4e14d58d accommodate "do nothing" branch 2023-08-31 21:26:46 +02:00
Valentin Gagarin d460dbdd30 be more precise about substituting store derivations
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-31 21:26:46 +02:00
Valentin Gagarin 6b3320ab05 mention remote builders
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-31 21:26:44 +02:00
Valentin Gagarin 0cd8f36644 add anchor to builder 2023-08-31 21:26:15 +02:00
Valentin Gagarin d50f116421 add reference link 2023-08-31 21:25:39 +02:00
Valentin Gagarin b7e9e29605 remove abstract description 2023-08-31 21:25:39 +02:00
Valentin Gagarin a57e0e8c5c reword introductory sentence 2023-08-31 21:25:39 +02:00
Valentin Gagarin 315a11bcc9 remove superfluous word 2023-08-31 21:25:39 +02:00
Valentin Gagarin 1bc9257d7c reword description of how realisation works 2023-08-31 21:25:39 +02:00
Cole Helbling 46478b44ff
docs/testing: point out the existence of GTEST_FILTER (#8883) 2023-08-29 22:13:35 +00:00
Eelco Dolstra 5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Eelco Dolstra 56763ff918 Document that redirected tarball flakerefs can specify lastModified 2023-08-29 16:08:29 +02:00
Silvan Mosberger 151120a1ae
Document nix-prefetch-url defaults (#8878) 2023-08-28 22:14:01 +02:00
tomberek b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
Guillaume Girol 925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Valentin Gagarin d5b130ef13 glossary: dedent list and do not use forced line breaks
this makes it slightly easier to work with and consistent with all the
other markdown lists in use
2023-08-24 10:00:17 +02:00
Uri Zafrir 4a435ad228
Add introductory sentence to advanced topics (#8861) 2023-08-23 15:18:25 +00:00
Alex Zero 37a509ca2d Add release notes for the previous commit 2023-08-14 18:56:02 +01:00
John Ericson 44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
John Ericson 60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Théophane Hufschmitt ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
John Ericson 9113b4252b
Merge pull request #8760 from iFreilicht/fix-json-load-assertion-errors
Fix derivation load assertion errors
2023-08-06 17:07:43 -07:00
Felix Uhl 3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
John Ericson 3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
Robert Hensing 33d58a90c2 toJSON: Add attribute path to trace 2023-07-31 13:02:54 +02:00
Alex Ameen 2d1d81114d
Add parseFlakeRef and flakeRefToString builtins (#8670)
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.

I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.

I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-25 17:43:33 +00:00
Naïm Favier 1b756e300f
doc: clarify release notes about nested attribute merges 2023-07-25 16:09:56 +02:00
Eelco Dolstra c51be0345e Release notes 2023-07-24 17:19:31 +02:00