Commit graph

1949 commits

Author SHA1 Message Date
Valentin Gagarin b951e862d0 more meaningful tagline 2023-08-31 21:26:46 +02:00
Valentin Gagarin cf4e14d58d accommodate "do nothing" branch 2023-08-31 21:26:46 +02:00
Valentin Gagarin d460dbdd30 be more precise about substituting store derivations
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-31 21:26:46 +02:00
Valentin Gagarin 6b3320ab05 mention remote builders
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-31 21:26:44 +02:00
Valentin Gagarin 0cd8f36644 add anchor to builder 2023-08-31 21:26:15 +02:00
Valentin Gagarin d50f116421 add reference link 2023-08-31 21:25:39 +02:00
Valentin Gagarin b7e9e29605 remove abstract description 2023-08-31 21:25:39 +02:00
Valentin Gagarin a57e0e8c5c reword introductory sentence 2023-08-31 21:25:39 +02:00
Valentin Gagarin 315a11bcc9 remove superfluous word 2023-08-31 21:25:39 +02:00
Valentin Gagarin 1bc9257d7c reword description of how realisation works 2023-08-31 21:25:39 +02:00
Cole Helbling 46478b44ff
docs/testing: point out the existence of GTEST_FILTER (#8883) 2023-08-29 22:13:35 +00:00
Eelco Dolstra 5b5f56a9d4
Merge pull request #8859 from edolstra/tarball-last-modified
Tarball trees: Propagate lastModified
2023-08-29 17:02:06 +02:00
Eelco Dolstra 56763ff918 Document that redirected tarball flakerefs can specify lastModified 2023-08-29 16:08:29 +02:00
Silvan Mosberger 151120a1ae
Document nix-prefetch-url defaults (#8878) 2023-08-28 22:14:01 +02:00
tomberek b563ef38cc
Merge pull request #8819 from VertexA115/fix/deep-follow-paths
Fix follow path checking at depths greater than 2
2023-08-25 10:33:05 -04:00
Guillaume Girol 925a444b92 add nix-store --query --valid-derivers command
notably useful when nix-store --query --deriver returns a non-existing
path.

Co-authored-by: Felix Uhl <iFreilicht@users.noreply.github.com>
2023-08-24 11:37:24 +02:00
Valentin Gagarin d5b130ef13 glossary: dedent list and do not use forced line breaks
this makes it slightly easier to work with and consistent with all the
other markdown lists in use
2023-08-24 10:00:17 +02:00
Uri Zafrir 4a435ad228
Add introductory sentence to advanced topics (#8861) 2023-08-23 15:18:25 +00:00
Alex Zero 37a509ca2d Add release notes for the previous commit 2023-08-14 18:56:02 +01:00
John Ericson 44c8d83831 Create outputOf primop.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.

With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!

Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.

In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!

Part of RFC 92: dynamic derivations (tracking issue #6316)

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-08-14 09:37:37 -04:00
John Ericson 60b7121d2c Make the Derived Path family of types inductive for dynamic derivations
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-08-10 00:08:32 -04:00
Théophane Hufschmitt ad410abbe0 Stabilize discard-references
It has been there for a few releases now (landed in 2.14.0), doesn't
seem to cause any major issue and is wanted in a few places
(https://github.com/NixOS/nix/pull/7087#issuecomment-1544471346).
2023-08-07 16:53:37 +02:00
John Ericson 9113b4252b
Merge pull request #8760 from iFreilicht/fix-json-load-assertion-errors
Fix derivation load assertion errors
2023-08-06 17:07:43 -07:00
Felix Uhl 3fefc2b284 Fix derivation load assertion errors
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.

Display nice error messages instead and give some indication where the
error happened.

*Before:*

```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number

$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string

```

*After:*

```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'

$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't

$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'

$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
       … while reading key 'outputs'

       error: Expected JSON value to be of type 'object' but it is of type 'string'
```
2023-08-05 01:34:30 +02:00
John Ericson 3b592c880a Add infra for experimental store implemenations
This is analogous to that for experimental settings and flags that we
have also added as of late.
2023-08-02 15:46:38 -04:00
Robert Hensing 33d58a90c2 toJSON: Add attribute path to trace 2023-07-31 13:02:54 +02:00
Alex Ameen 2d1d81114d
Add parseFlakeRef and flakeRefToString builtins (#8670)
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.

I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.

I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-25 17:43:33 +00:00
Naïm Favier 1b756e300f
doc: clarify release notes about nested attribute merges 2023-07-25 16:09:56 +02:00
Eelco Dolstra c51be0345e Release notes 2023-07-24 17:19:31 +02:00
Naïm Favier 570a1a3ad7
parser: merge nested dynamic attributes
Fixes https://github.com/NixOS/nix/issues/7115
2023-07-21 17:14:03 +02:00
Valentin Gagarin 85d0eb6316
fix broken links (#8722) 2023-07-20 17:58:14 +02:00
Valentin Gagarin 0e4f6dfcf7 revert anchor prefix for builtin constants
the original change broke many pre-existing anchor links.

also change formatting of the constants listing slightly:
- the type should not be part of the anchor
- add highlight to the "impure only" note
2023-07-20 10:27:38 +02:00
John Ericson 453c4be93c
Merge pull request #8680 from NixLayeredStore/test-groups
Introduce notion of a test group, use for CA tests
2023-07-19 11:17:57 -04:00
Valentin Gagarin b0173716f6
clarify wording on args@ default handling (#8596)
* clarify wording on args@ default handling

Most importantly use shorter sentences and emphasize the key point that defaults aren't taken into account

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-07-19 13:07:07 +00:00
Robert Hensing 32494cbb29
Merge pull request #7973 from fricklerhandwerk/remove-channels
remove the Channels section
2023-07-19 14:02:26 +02:00
Valentin Gagarin 6c3cd429a6 fix broken links 2023-07-19 11:01:48 +02:00
Valentin Gagarin 1a220bed93 do not mention output attributes
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-19 10:38:12 +02:00
Valentin Gagarin c8f04e2024 note that naming convention is from Autotools
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-07-19 10:37:40 +02:00
Valentin Gagarin 4944e37ec0 expand on the system type in hacking guide 2023-07-19 10:37:40 +02:00
Valentin Gagarin e14c8a359e list moving parts of channels 2023-07-19 10:26:25 +02:00
Valentin Gagarin 4bab5a6208 revert channel files overview 2023-07-19 09:42:53 +02:00
Valentin Gagarin cd0e39bd89 remove redundant information from channel profile description 2023-07-19 09:39:04 +02:00
Valentin Gagarin ee72ede389 remove the Channels section
this is a how-to guide which should not be in the reference manual.
it also refers to `nix-env`, which should not be the first thing readers
of the reference manual encounter, as it behaves very differently in
spirit from the rest of Nix.

slightly reword the documentation to be more concise and informative.
2023-07-19 09:39:04 +02:00
John Ericson 259e328de8 Introduce notion of a test group, use for CA tests
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.

We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.

I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-18 09:31:13 -04:00
Sinan Mohd a5c88f8609
Nix Reference Manual: keep nix expressions uptodate with nixpkgs (#8703) 2023-07-16 09:25:11 +00:00
Mathnerd314 c70484454f Expanded test suite
* Lang now verifies errors and parse output

* Some new miscellaneous tests

* Easy way to update the tests

* Document workflow in manual

* Use `!` not `~` as separater char for sed

  It is confusing to use `~` when we are talking about paths and home
  directories!

* Test test suite itself (`test/lang-test/infra.sh`)

Additionally, run shellcheck on `tests/lang.sh` to help ensure it is
correct, now that is is more complex.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-07-11 21:43:09 -04:00
Robert Hensing 40052c7613 fetchClosure: Docs and error message improvements
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-30 18:23:42 +02:00
Robert Hensing 50de11d662 doc: Improve fetchClosure documentation 2023-06-30 18:23:24 +02:00
Robert Hensing 32c69e2b17 doc: Typo 2023-06-30 18:22:47 +02:00
Eelco Dolstra a0c617348b
Merge pull request #8589 from jfroche/sign-paths-as-allowed-user
Allow to sign path as unprivileged user
2023-06-30 13:13:42 +02:00
John Ericson ca49e13414 Split testing into its own page in the contribution guide
`hacking.md` has gotten really big!
2023-06-27 18:27:49 -04:00
John Ericson 2ccc02515f Trailing commas in redirects
This avoids diff noise when more are added. Unlike with JSON, this is
allowed in JS.
2023-06-27 18:23:06 -04:00
Jean-François Roche 80c9259756 Allow to sign path as unprivileged user
User can now sign path as unprivileged/allowed user

refs #1708
2023-06-27 18:31:31 +02:00
John Ericson 22b278e011 Automatically document builtin constants
This is done in roughly the same way builtin functions are documented.

Also auto-link experimental features for primops, subsuming PR #8371.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-27 09:37:54 -04:00
John Ericson d40f0e534d Don't say this when we still pollute the global scope
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-27 09:37:31 -04:00
John Ericson 4da7c86618 Switch example to a primop this is less ill-advised
Any primop will do for this, so might as well use one that isn't impure.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-27 09:34:36 -04:00
John Ericson 97df060588 Better document build failure exit codes
- Improved API docs from comment

- Exit codes are for `nix-build`, not just `nix-store --release`

- Make note in tests so the magic numbers are not surprising

Picking up where #8387 left off.
2023-06-22 14:29:45 -04:00
John Ericson 78e1e3c753
Merge pull request #8565 from obsidiansystems/profile-delete-docs
Clarify docs on deleting generations, including fixing a mistake
2023-06-22 10:27:40 -04:00
Valentin Gagarin 0ab962d83f
Merge pull request #8556 from fricklerhandwerk/hacking-headings
hacking guide: use more self-descriptive section headings
2023-06-22 13:45:31 +02:00
John Ericson 5cc22e3370 Clarify docs on deleting generations, including fixing a mistake
Deleting store info corrected (there is a foot-gun in Nix with
`--delete-generations old`!)

Also a few things are cleaned up based on feedback.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-06-21 20:05:41 -04:00
Valentin Gagarin 5f9a921bc1
do not use "target", as it's a loaded term in the domain of compilers
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-21 14:31:09 +02:00
Valentin Gagarin 085104944b add redirects to changed anchors 2023-06-21 09:46:23 +02:00
Valentin Gagarin a78f929065 fix anchor link 2023-06-21 09:43:22 +02:00
Valentin Gagarin 71317162c5 use more self-descriptive section headings 2023-06-20 13:44:43 +02:00
Valentin Gagarin 3c618c43c6
Merge pull request #8532 from fricklerhandwerk/nix.conf-sections
split nix.conf man page into sections
2023-06-20 13:16:14 +02:00
Valentin Gagarin 68c6219343 clarify setting options on the command line 2023-06-20 12:13:32 +02:00
Valentin Gagarin bc7324e912 clarify read order for configuration settings 2023-06-20 12:13:26 +02:00
fricklerhandwerk 38bd1cc9bc split configuration file page into sections
this makes it easier to scan for specific information, such as the
format of command line flags
2023-06-20 12:11:01 +02:00
fricklerhandwerk f2b54e3b71 add links to environment variables documentation 2023-06-20 12:11:01 +02:00
Travis A. Everett 33d3889831 redirect old platform uninstall instruction links
Uninstall instructions were moved to their own page in #8267. The
overall section link was redirected in #8286, but platform-specific
links (which I give out frequently when I triage installer trouble)
weren't included.
2023-06-19 15:17:50 -05:00
Silvan Mosberger 3910430b9d
Add more links in nix-build documentation (#8545)
* Add more links in nix-build documentation


Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-19 21:00:49 +02:00
John Ericson 469d06f9bc Split out worker protocol template definitions from declarations
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.

This is now documenting in the contributing guide.

Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
2023-06-19 11:45:59 -04:00
Valentin Gagarin 7bf17f8825
Add description for file system objects (#8500)
While this is not actually a notion in the implementation, it is
explicitly described in the thesis and quite important for understanding
how the store works.

Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-18 23:45:08 -04:00
John Ericson 60d81b5163
Merge pull request #8472 from NixOS/nix-language-purpose
Refine wording on the purpose of the Nix language
2023-06-18 23:38:05 -04:00
Christina Sørensen 741f7837f8
Fix wikipedia links (#8533) 2023-06-17 09:06:17 +00:00
Eelco Dolstra e503eadafc
Merge pull request #8477 from edolstra/tarball-flake-redirects
Tarball flake improvements
2023-06-16 18:03:50 +02:00
Eelco Dolstra b1ed9b4b0c
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-06-16 16:48:37 +02:00
Eelco Dolstra cab03fb779 Add docs 2023-06-16 15:58:42 +02:00
Valentin Gagarin c3c4076342 make domain-specificity more specific
also slightly reword the purpose statement to introduce (and explain)
derivations right away.
2023-06-15 03:08:07 +02:00
John Ericson 946cd9e3f9
Merge pull request #8351 from obsidiansystems/delete-profiles-tests-docs
Expanding tests and docs relating to deleting profiles
2023-06-15 01:47:21 +02:00
John Ericson 5b7e285727 Improve nix-collect-garbage docs
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-14 19:01:08 -04:00
John Ericson b55f26c65f Improve nix-env --delete-generations docs
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-06-14 19:01:08 -04:00
Valentin Gagarin a1cf16563f
Fixup description of substituters (#8291)
Introduce what substituters "are" in the configuration option entry.
Remove arbitrary line breaks for easier editing in the future.
Link glossary some more.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-06-14 20:49:58 +00:00
John Ericson 37bffbc28f
Merge pull request #8488 from Mic92/update-requirements
nix actually needs c++20 now
2023-06-14 20:37:52 +02:00
John Ericson ff905cb796
Merge pull request #4803 from ShamrockLee/nix-channel-list-generations
Add `nix-channel --list-generations`
2023-06-14 18:30:35 +02:00
Jörg Thalheim c51f3f1eb2 nix actually needs c++20 now 2023-06-10 13:56:05 +02:00
Valentin Gagarin db680e0e57
refine wording on the purpose of the Nix language
packages and configurations are not really a concept in Nix or the Nix language. the idea of transforming files into other files clearly captures what it's all about, and the new phrasing should make the term "derivation" more obvious both in terms of meaning and origin.
2023-06-08 02:00:05 +02:00
Théophane Hufschmitt 527eb4a99a
Merge pull request #8317 from fricklerhandwerk/doc-identifier
document identifier syntax for attribute sets
2023-06-02 13:10:27 +02:00
Théophane Hufschmitt 331f0967c4 Add a release note for nix-channel --list-generations 2023-06-02 10:14:53 +02:00
Shamrock Lee b292177eec Add nix-channel --list-generations
Add support to --list-generations
as another way to say
nix-env --profile /nix/var/nix/profiles/per-user/$USER/channels --list-generations
the way we did for nix-channel --rollback [generation id]
2023-06-02 10:12:35 +02:00
Eelco Dolstra 008f89fa50 Typo 2023-05-31 14:05:00 +02:00
Eelco Dolstra afc24e6a66 Release notes 2023-05-31 12:38:05 +02:00
Eelco Dolstra 9a5b9c588f
Merge pull request #8421 from fricklerhandwerk/doc-replaceStrings
update documentation according to release notes
2023-05-31 12:10:32 +02:00
Valentin Gagarin 738c0d5064
Merge pull request #8318 from fricklerhandwerk/doc-currentTime
document `builtins.currentTime`
2023-05-31 03:15:54 +02:00
Valentin Gagarin 52004696c0
Merge pull request #8413 from doronbehar/doc/distributed-builds/clearer-warning
distributed-builds.md: Clarify warning ssh access requirements
2023-05-31 02:05:05 +02:00
Valentin Gagarin e2f33e0df1 rewrap lines to use one line per sentence
this makes future reviews easier as it reduces diff noise
2023-05-31 01:32:03 +02:00
Doron Behar 76287a9207 distributed-builds.md: Clarify warning ssh access requirements 2023-05-31 01:28:15 +02:00
Valentin Gagarin 7a5731aa07 link to mentioned builtin 2023-05-30 22:58:00 +02:00
polykernel a382919d7d
primops: lazy evaluation of replaceStrings replacements
The primop `builtins.replaceStrings` currently always strictly evaluates the
replacement strings, however time and space are wasted for their computation
if the corresponding pattern do not occur in the input string. This commit
makes the evaluation of the replacement strings lazy by deferring their
evaluation to when the corresponding pattern are matched and memoize the result
for efficient retrieval on subsequent matches.

The testcases for replaceStrings was updated to check for lazy evaluation
of the replacements. A note was also added in the release notes to
document the behavior change.
2023-05-25 18:35:23 -04:00
John Ericson 684e9be8b9
Merge pull request #8337 from fricklerhandwerk/channel-files
list files used by `nix-channel` on its own man page
2023-05-17 12:21:20 -04:00