Commit graph

14496 commits

Author SHA1 Message Date
Eelco Dolstra 880e7b8ed6 TarArchive: Remove a duplicate constant and increase the buffer size 2023-04-25 16:43:10 +02:00
Moritz Angermann 0e18254aa8
Fix shutdown behavior and resource management for recursive-nix on macOS
Previously, we relied on the `shutdown()` function to terminate `accept()`
calls on a listening socket. However, this approach did not work on macOS as
the waiting `accept()` call is not considered a connected socket, resulting in
an `ENOTCONN` error. Instead, we now close the listening socket to terminate
the `accept()` call.

Additionally, we fixed a resource management issue where we set the
`daemonSocket` variable to -1, triggering resource cleanup and causing the
`stopDaemon` function to be called twice. This resulted in errors as the socket
was already closed by the time the second `stopDaemon` call was made. Instead of
setting `daemonSocket` to -1, we now release the socket using the `release()`
method on a unique pointer. This properly transfers ownership and allows for
correct resource cleanup.

These changes ensure proper behavior and resource management for the
recursive-nix feature on macOS.
2023-04-25 09:39:05 +08:00
Eelco Dolstra 249ce28332
Merge pull request #8172 from edolstra/source-path
Backport `SourcePath` from the lazy-trees branch
2023-04-24 14:05:51 +02:00
Eelco Dolstra 5d3f6dbf59 Add some more SourcePath docs 2023-04-24 13:37:51 +02:00
Eelco Dolstra ad57cff9bc
Document tMisc
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-24 13:34:46 +02:00
Eelco Dolstra 01232358ff Merge remote-tracking branch 'origin/master' into source-path 2023-04-24 13:20:36 +02:00
Théophane Hufschmitt 7474a90db6
Merge pull request #7710 from obsidiansystems/context-not-path-set
Use `std::set<StringContextElem>` not `PathSet` for string contexts
2023-04-21 08:14:58 +02:00
John Ericson 8eeaf591db
Add more docs to TextIngestionMethod
Thanks so much!

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-04-21 01:30:55 -04:00
John Ericson 85f0cdc370 Use std::set<StringContextElem> not PathSet for string contexts
Motivation

`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.

-----

In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.

Summarizing the previous paragraph:

Old:

  - `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
  - `value.hh`: `NixStringContext Value::getContext(...)`
  - `value.hh`: `copyContext(...)`

New:

  - `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
  - `value.hh`: `copyContext(...)`
----

The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:

Less API churn:

`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.

That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.

Space usage:

Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.

----

The eval cache version is bumped.

The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.

This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.

----

To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.

Context

This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.

The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.

A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.

----

PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.

This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-04-21 01:05:49 -04:00
Ezra Singh 24005270cc
Update install-systemd-multi-user.sh 2023-04-20 18:36:01 -04:00
Robert Hensing ef432b2b15
Merge pull request #8242 from obsidiansystems/recursive.nix
Move `test/recursive.sh` nix expr to file
2023-04-20 13:22:35 +02:00
John Ericson 969def696a Fix typo in tests 2023-04-19 20:47:23 -04:00
John Ericson e26662709e Add a more interesting test
In this one, we don't just output an existing derivation as is, but
modify it first.
2023-04-19 20:36:33 -04:00
John Ericson 3eb343754e Move test/recursive.sh nix expr to file
I found it hard to read as a big string literal.
2023-04-19 19:36:05 -04:00
John Ericson f3a31b14db Make tests/dyn-drv test dir 2023-04-19 18:49:50 -04:00
John Ericson 61d3e64fd0 Require daemon version for text hashing test 2023-04-19 17:24:55 -04:00
John Ericson 7103c6da70 Remove references from fixed output derivation ab syntax
In other words, use a plain `ContentAddress` not
`ContentAddressWithReferences` for `DerivationOutput::CAFixed`.

Supporting fixed output derivations with (fixed) references would be a
cool feature, but it is out of scope at this moment.
2023-04-19 15:00:04 -04:00
John Ericson aba8a8a83a Add a few more content addressing methods
Good to round out the library interface.
2023-04-19 14:13:30 -04:00
John Ericson 20decfd302 Gate dynamic-derivations with drv fromJSON too
Don't want `nix derivation add` to be a way to sneak by experimental
feature checks!
2023-04-19 12:44:38 -04:00
John Ericson 76baaeb341 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-19 11:32:14 -04:00
Théophane Hufschmitt d3e2394e91
Merge pull request #8233 from wentasah/narinfo-corrupt-reason
Make "NAR info file is corrupt" messages more informative
2023-04-19 06:57:08 +02:00
Robert Hensing 28d7ffd448
Merge pull request #8220 from accelbread/whitelist-commit-lockfile-summary
Add commit-lockfile-summary to flake nixConfig whitelist
2023-04-18 16:49:33 +02:00
Robert Hensing 5cd9890e8a src/nix/flake.md: Itemize safe nixConfigs 2023-04-18 16:06:58 +02:00
Robert Hensing 7fcce6d48c
Merge pull request #8231 from obsidiansystems/add-derivation-xp-gates
Gate experimental features in `DerivationOutput::fromJSON`
2023-04-18 15:42:58 +02:00
Michal Sojka d30d2dc861 Make "NAR info file is corrupt" messages more informative
Recently, I encountered the "NAR info file 'xxxx' is corrupt" error
with my binary cache. The message is not helpful in determining, which
kind of corruption happened. The file, fetched with curl, looked
reasonably.

This commit adds more information to the error message, which should
allow debugging and hopefully fixing the problem.
2023-04-18 14:10:49 +02:00
Valentin Gagarin 66e28e6343
Merge pull request #8236 from theutz/master
Update installing-binary.md
2023-04-18 14:08:32 +02:00
Michael Utz 40fcb22313
Update installing-binary.md 2023-04-18 13:18:30 +03:00
Valentin Gagarin e1e6119761 Merge pull request #8235 from peeley/fix-nix-database-glossary-typo
Documentation: fix typo for `Nix database` link in manual
2023-04-18 11:05:08 +02:00
Noah Snelson 324ed0c367 Documentation: fix typo for Nix database link in manual
Fixes broken link for `Nix database` anchor in the Glossary page of the
Nix manual.
2023-04-17 20:15:08 -07:00
John Ericson 668377f217 TextHashMethod -> TextIngestionMethod, gate with XP feature
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-17 19:02:45 -04:00
John Ericson f56c4a5bdf Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 18:10:12 -04:00
John Ericson aa74c7b0bc Gate experimental features in DerivationOutput::fromJSON
This is an entry point for outside data, so we need to check enabled
experimental features here.
2023-04-17 17:36:12 -04:00
John Ericson ab5ca608bf Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 13:54:48 -04:00
John Ericson 23ee2d79a9 Use buildPathsWithResults in build-remote.cc trustless path
It handles failures more correctly; I am glad we have it now!
2023-04-17 13:49:14 -04:00
John Ericson 79ba0ba37a Improve the build remote comment. 2023-04-17 13:49:14 -04:00
Robert Hensing 64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
Robert Hensing c56705c025
Merge pull request #8194 from obsidiansystems/redisplay-feature-gated-settings
Fix some issues with experimental config settings
2023-04-17 19:25:06 +02:00
John Ericson d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00
John Ericson 2c8475600d Fix some issues with experimental config settings
Issues:

1. Features gated on disabled experimental settings should warn and be
   ignored, not silently succeed.

2. Experimental settings in the same config "batch" (file or env var)
   as the enabling of the experimental feature should work.

3. For (2), the order should not matter.

These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 12:41:04 -04:00
Robert Hensing 3f9589f17e
Merge pull request #6312 from obsidiansystems/keyed-build-result
Shuffle `BuildResult` data definition, make state machine clearer, introduce `SingleDrvOutputs`
2023-04-17 18:08:05 +02:00
John Ericson 2eb493ca51 Fix DerivationOutput::fromJSON 2023-04-17 10:28:54 -04:00
Valentin Gagarin 39e0f609cc
Merge pull request #7514 from fricklerhandwerk/opt-I
remove incorrect reference to `NIX_PATH` documentation
2023-04-17 16:27:23 +02:00
John Ericson e12efa3654 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 10:16:57 -04:00
Robert Hensing e641de085b
Merge pull request #3746 from obsidiansystems/path-info
Introduce `StoreReferences` and `ContentAddressWithReferences`
2023-04-17 15:49:48 +02:00
John Ericson e95db8f2b9 nix-testing -> daemon-trust-override
And only enable in the tests that need it. This makes it less of a
sledgehammer.
2023-04-17 09:35:43 -04:00
John Ericson b1343e8ad1 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 09:27:17 -04:00
John Ericson 537e8719f2
Explain various .self = false,
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 09:15:11 -04:00
Valentin Gagarin f4119a67cc
use @docroot@ link 2023-04-17 14:54:30 +02:00
Valentin Gagarin d0cf615cbb
add link to nix-conf setting
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-17 14:27:26 +02:00
John Ericson 72ffa7fedb
Merge pull request #7732 from hercules-ci/make-initLibStore-viable-alternative
Make `initLibStore` a viable alternative
2023-04-17 08:04:41 -04:00