Commit graph

14359 commits

Author SHA1 Message Date
Robert Hensing 64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
Robert Hensing c56705c025
Merge pull request #8194 from obsidiansystems/redisplay-feature-gated-settings
Fix some issues with experimental config settings
2023-04-17 19:25:06 +02:00
John Ericson d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00
John Ericson 2c8475600d Fix some issues with experimental config settings
Issues:

1. Features gated on disabled experimental settings should warn and be
   ignored, not silently succeed.

2. Experimental settings in the same config "batch" (file or env var)
   as the enabling of the experimental feature should work.

3. For (2), the order should not matter.

These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 12:41:04 -04:00
Robert Hensing 3f9589f17e
Merge pull request #6312 from obsidiansystems/keyed-build-result
Shuffle `BuildResult` data definition, make state machine clearer, introduce `SingleDrvOutputs`
2023-04-17 18:08:05 +02:00
John Ericson 2eb493ca51 Fix DerivationOutput::fromJSON 2023-04-17 10:28:54 -04:00
Valentin Gagarin 39e0f609cc
Merge pull request #7514 from fricklerhandwerk/opt-I
remove incorrect reference to `NIX_PATH` documentation
2023-04-17 16:27:23 +02:00
John Ericson e12efa3654 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 10:16:57 -04:00
Robert Hensing e641de085b
Merge pull request #3746 from obsidiansystems/path-info
Introduce `StoreReferences` and `ContentAddressWithReferences`
2023-04-17 15:49:48 +02:00
John Ericson e95db8f2b9 nix-testing -> daemon-trust-override
And only enable in the tests that need it. This makes it less of a
sledgehammer.
2023-04-17 09:35:43 -04:00
John Ericson b1343e8ad1 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 09:27:17 -04:00
John Ericson 537e8719f2
Explain various .self = false,
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 09:15:11 -04:00
Valentin Gagarin f4119a67cc
use @docroot@ link 2023-04-17 14:54:30 +02:00
Valentin Gagarin d0cf615cbb
add link to nix-conf setting
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-17 14:27:26 +02:00
John Ericson 72ffa7fedb
Merge pull request #7732 from hercules-ci/make-initLibStore-viable-alternative
Make `initLibStore` a viable alternative
2023-04-17 08:04:41 -04:00
Robert Hensing cb2615cf47 Merge remote-tracking branch 'upstream/master' into source-path 2023-04-17 11:41:50 +02:00
Robert Hensing 9af9c260fc
Merge pull request #8193 from hercules-ci/dry-strings
Deduplicate string literal rendering, fix 4909
2023-04-17 11:19:40 +02:00
Robert Hensing 36a473c5e8
Merge pull request #8196 from obsidiansystems/fix-8162
Mark experimental configuration settings programmatically
2023-04-17 11:18:11 +02:00
John Ericson 9800c1e807 Mark experimental configuration settings programmatically
Fix #8162

The test is changed to compare `nlohmann::json` values, not strings of dumped
JSON, which allows us to format things more nicely.
2023-04-16 10:58:04 -04:00
John Ericson ba9ae691b6 Add optionalString to manual Nix lang utilities
Use it everywhere it could be also.
2023-04-16 10:57:46 -04:00
Robert Hensing b6125772d7 libexpr: Move identifier-like printing to print.cc 2023-04-16 14:07:35 +02:00
Robert Hensing 28a5cdde02 libexpr/value/print.* -> libexpr/print.*
Generalizes the file to sensibly allow printing any part of the
language syntax.
2023-04-16 13:10:45 +02:00
Robert Hensing 1e2dd669bc printLiteral: Do not overload 2023-04-16 13:04:35 +02:00
Robert Hensing 9c74df5bb4
Format
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-15 20:56:51 +02:00
John Ericson 24866b71c4 Introduce SingleDrvOutputs
In many cases we are dealing with a collection of realisations, they are
all outputs of the same derivation. In that case, we don't need
"derivation hashes modulos" to be part of our map key, because the
output names alone will be unique. Those hashes are still part of the
realisation proper, so we aren't loosing any information, we're just
"normalizing our schema" by narrowing the "primary key".

Besides making our data model a bit "tighter" this allows us to avoid a
double `for` loop in `DerivationGoal::waiteeDone`. The inner `for` loop
was previously just to select the output we cared about without knowing
its hash. Now we can just select the output by name directly.

Note that neither protocol is changed as part of this: we are still
transferring `DrvOutputs` over the wire for `BuildResult`s. I would only
consider revising this once #6223 is merged, and we can mention protocol
versions inside factored-out serialization logic. Until then it is
better not change anything because it would come a the cost of code
reuse.
2023-04-15 12:51:19 -04:00
John Ericson 0f2b5146c7 Make restarting state machines explicit
If my memory is correct, @edolstra objected to modifying `wantedOutputs`
upon falling back to doing a build (as we did before), because we should
only modify it in response to new requests --- *actual* wants --- and
not because we are "incidentally" building all the outptus beyond what
may have been requested.

That's a fair point, and the alternative is to replace the boolean soup
with proper enums: Instead of modifying `wantedOuputs` som more, we'll
modify `needsRestart` to indicate we are passed the need.
2023-04-15 11:01:31 -04:00
John Ericson 37fca662b0 Make KeyedBuildResult, BuildResult like before, and fix bug another way
In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I
realized since derivation goals' wanted outputs can "grow" due to
overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called
by `Worker::makeDerivationGoalCommon`), the previous bug fix had an
unfortunate side effect of causing more pointless rebuilds.

In paticular, we have this situation:

1. Goal made from `DerivedPath::Built { foo, {a} }`.

2. Goal gives on on substituting, starts building.

3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just
   modified original goal.

4. Though the goal had gotten as far as building, so all outputs were
   going to be produced, `addWantedOutputs` no longer knows that and so
   the goal is flagged to be restarted.

This might sound far-fetched with input-addressed drvs, where we usually
basically have all our goals "planned out" before we start doing
anything, but with CA derivation goals and especially RFC 92, where *drv
resolution* means goals are created after some building is completed, it
is more likely to happen.

So the first thing to do was restore the clearing of `wantedOutputs` we
used to do, and then filter the outputs in `buildPathsWithResults` to
only get the ones we care about.

But fix also has its own side effect in that the `DerivedPath` in the
`BuildResult` in `DerivationGoal` cannot be trusted; it is merely the
*first* `DerivedPath` for which this goal was originally created.

To remedy this, I made `BuildResult` be like it was before, and instead
made `KeyedBuildResult` be a subclass wit the path. Only
`buildPathsWithResults` returns `KeyedBuildResult`s, everything else
just becomes like it was before, where the "key" is unambiguous from
context.

I think separating the "primary key" field(s) from the other fields is
good practical in general anyways. (I would like to do the same thing
for `ValidPathInfo`.) Among other things, it allows constructions like
`std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and
just precludes the possibility of those duplicate keys being out of
sync.

We might leverage the above someday to overload `buildPathsWithResults`
to take a *set* of return a *map* per the above.

-----

Unfortunately, we need to avoid C++20 strictness on designated
initializers.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)

No having that yet, maybe it would be better to not make
`KeyedBuildResult` a subclass to just avoid this.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-15 11:01:31 -04:00
John Ericson 9df7f3f537 Introduce Worker::makeGoal
This takes a `DerivedPath` so the caller doesn't need to care about
which sort of goal does what.
2023-04-15 11:01:31 -04:00
Robert Hensing ab228d73db
Merge pull request #8214 from raphaelr/installable-derived-path-warn
Display valid installable in `InstallableDerivedPath::parse` warning
2023-04-15 14:15:02 +02:00
John Ericson ee420ac64e Legacy vs non-legacy to_string/parse for DerivedPath
As requested by @roberth, it is good to call out the specific instances
we care about, which is `!` for the RPC protocols, and `^` for humans.

This doesn't take advantage of parametricity as much, but since the
human and computer interfaces are good to decouple anyways (we don't
care if they drift further apart over time in the slightest) some
separation and slight duplication is fine.

Also, unit test both round trips.
2023-04-14 20:45:11 -04:00
Archit Gupta bfc558c972 Whitelist commit-lockfile-summary in flake nixConfig 2023-04-14 11:33:38 -07:00
John Ericson b41f739068
Merge pull request #7864 from obsidiansystems/quickstart-long-options
Use long options instead of short ones in the "quick start"
2023-04-14 09:13:16 -04:00
John Ericson 92346d1ed4
Merge pull request #8174 from NixOS/gloss-xp-features-note-installables
Start cross-referencing experimental features
2023-04-14 08:48:10 -04:00
Robert Hensing f5ab38a688
Merge pull request #8217 from obsidiansystems/push-get-fs-accessor-unsupported-down
Push `getFSAccessor` `unsupported(...)` down `Store` class hierarchy
2023-04-14 14:23:23 +02:00
John Ericson d93e76fbb8 Start cross-referencing experimental features
- Create a glossary entry for experimental features.

- Have the man page experimental feature notice link `nix-commmand`.

  (Eventually this should be programmed, based on whether the command is
  experimental, and if so what experimental feature does it depend on.)

- Document which installables depend on which experimental features.

  I tried to use the same style (bold warning and block quote) that the
  top of the man page uses.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-14 07:45:08 -04:00
Eelco Dolstra 7eac8838df
Merge pull request #8199 from tweag/fix-sqlite-busy-reporting
Fix unnecessary reporting of SQLite busy errors
2023-04-14 10:50:50 +02:00
Eelco Dolstra 33fc09c2a5
Merge pull request #8176 from tweag/rename-confusing-write-method
Rename and protect `BufferedSink::write`
2023-04-14 10:44:36 +02:00
Raphael Robatsch 9e8f209036 Display valid installable in InstallableDerivedPath::parse warning
The warning message should produce an installable name that can be
passed to `nix build`, `nix path-info`, etc. again. Since the CLI
expects that the .drv path and the output names are separated by
a caret, the warning message must also separate the .drv path and output
names with a caret.

However, `DerivedPath::Built.to_string()` uses an exclamation point as
the separator instead. This commit adds a `separator` argument to the
to_string method.

This changes the warning message from:
If this command is now failing try again with '/nix/store/foo.drv!*'
to:
If this command is now failing try again with '/nix/store/foo.drv^*'
2023-04-14 10:32:03 +02:00
John Ericson ee97f107e8 Push getFSAccessor unsupported(...) down Store class hierarchy
More progress on issue #5729.

Instead of having it by the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.

Combined with my other recent PRs, this finally makes `Store` have no
`unsupported` calls!
2023-04-13 13:39:44 -04:00
John Ericson 615c25b0dd Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-12 00:00:12 -04:00
Eelco Dolstra ef0b48377d
Merge pull request #8206 from NixOS/bump-2.16.0
Bump version
2023-04-11 20:59:11 +02:00
Eelco Dolstra a4b6d1d9a3 Bump version 2023-04-11 20:16:37 +02:00
Eelco Dolstra 37046432b9
Merge pull request #8201 from obsidiansystems/hacky-xp-setting-fix
Do not gate or hide experimental settings
2023-04-11 17:37:09 +02:00
Ben Radford de3df3009b
Move warning timing logic into handleSQLiteBusy. 2023-04-11 16:14:16 +01:00
John Ericson 450e5ec618 Do not gate or hide experimental settings
This is somewhat hacky fix just for 2.15. I unintentionally hid them
from the manual, when no one wanted to hide them that (including
myself). I also required the experimental feature to be enabled in an
order-dependent way, which is not good.

The simplest fix for this immanent release is just to always show them,
and always allow them to be set.

Effectively undoes some changes from aa663b7e89
2023-04-11 10:56:48 -04:00
Eelco Dolstra 0a54624275
Merge pull request #8200 from edolstra/release-notes
2.15 release notes
2023-04-11 13:18:30 +02:00
Eelco Dolstra e570a91661 Release notes 2023-04-11 12:40:56 +02:00
Eelco Dolstra 162c3f5f7c
Merge pull request #8170 from tweag/fix-aarch64-build
Explicitely define `LockFile::operator!=`
2023-04-11 12:35:23 +02:00
Ben Radford da322ebda6
Revert "Add script to reproduce issue by inducing heavy load."
This reverts commit 213b838f9cfb820d2bc76d7c6edc468b27029945.
2023-04-11 11:15:34 +01:00
Ben Radford 7c56e84213
Warn after a second of being busy instead of immediately.
Getting the occasional SQLITE_BUSY is expected when the database is being
accessed concurrently. The retry will likely succeed so it is pointless to warn
immediately. Instead we track how long each retrySQLite block has been running,
and only begin warning after a second has elapsed (and then every 10 seconds
subsequently).
2023-04-11 11:15:34 +01:00