Commit graph

9077 commits

Author SHA1 Message Date
John Ericson aba8a8a83a Add a few more content addressing methods
Good to round out the library interface.
2023-04-19 14:13:30 -04:00
John Ericson 20decfd302 Gate dynamic-derivations with drv fromJSON too
Don't want `nix derivation add` to be a way to sneak by experimental
feature checks!
2023-04-19 12:44:38 -04:00
John Ericson 76baaeb341 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-19 11:32:14 -04:00
Théophane Hufschmitt d3e2394e91
Merge pull request #8233 from wentasah/narinfo-corrupt-reason
Make "NAR info file is corrupt" messages more informative
2023-04-19 06:57:08 +02:00
Robert Hensing 28d7ffd448
Merge pull request #8220 from accelbread/whitelist-commit-lockfile-summary
Add commit-lockfile-summary to flake nixConfig whitelist
2023-04-18 16:49:33 +02:00
Robert Hensing 5cd9890e8a src/nix/flake.md: Itemize safe nixConfigs 2023-04-18 16:06:58 +02:00
Michal Sojka d30d2dc861 Make "NAR info file is corrupt" messages more informative
Recently, I encountered the "NAR info file 'xxxx' is corrupt" error
with my binary cache. The message is not helpful in determining, which
kind of corruption happened. The file, fetched with curl, looked
reasonably.

This commit adds more information to the error message, which should
allow debugging and hopefully fixing the problem.
2023-04-18 14:10:49 +02:00
John Ericson 668377f217 TextHashMethod -> TextIngestionMethod, gate with XP feature
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-17 19:02:45 -04:00
John Ericson f56c4a5bdf Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 18:10:12 -04:00
John Ericson aa74c7b0bc Gate experimental features in DerivationOutput::fromJSON
This is an entry point for outside data, so we need to check enabled
experimental features here.
2023-04-17 17:36:12 -04:00
John Ericson ab5ca608bf Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 13:54:48 -04:00
John Ericson 23ee2d79a9 Use buildPathsWithResults in build-remote.cc trustless path
It handles failures more correctly; I am glad we have it now!
2023-04-17 13:49:14 -04:00
John Ericson 79ba0ba37a Improve the build remote comment. 2023-04-17 13:49:14 -04:00
Robert Hensing 64ee02890c
Merge pull request #8230 from obsidiansystems/daemon-trust-override
Experimentally allow forcing `nix-daemon` trust; use this to test
2023-04-17 19:43:41 +02:00
John Ericson d41e1bed5e Experimentally allow forcing nix-daemon trust; use this to test
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.

PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
2023-04-17 13:06:21 -04:00
John Ericson 2c8475600d Fix some issues with experimental config settings
Issues:

1. Features gated on disabled experimental settings should warn and be
   ignored, not silently succeed.

2. Experimental settings in the same config "batch" (file or env var)
   as the enabling of the experimental feature should work.

3. For (2), the order should not matter.

These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 12:41:04 -04:00
Robert Hensing 3f9589f17e
Merge pull request #6312 from obsidiansystems/keyed-build-result
Shuffle `BuildResult` data definition, make state machine clearer, introduce `SingleDrvOutputs`
2023-04-17 18:08:05 +02:00
John Ericson 2eb493ca51 Fix DerivationOutput::fromJSON 2023-04-17 10:28:54 -04:00
John Ericson e12efa3654 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2023-04-17 10:16:57 -04:00
Robert Hensing e641de085b
Merge pull request #3746 from obsidiansystems/path-info
Introduce `StoreReferences` and `ContentAddressWithReferences`
2023-04-17 15:49:48 +02:00
John Ericson e95db8f2b9 nix-testing -> daemon-trust-override
And only enable in the tests that need it. This makes it less of a
sledgehammer.
2023-04-17 09:35:43 -04:00
John Ericson b1343e8ad1 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-17 09:27:17 -04:00
John Ericson 537e8719f2
Explain various .self = false,
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-17 09:15:11 -04:00
John Ericson 72ffa7fedb
Merge pull request #7732 from hercules-ci/make-initLibStore-viable-alternative
Make `initLibStore` a viable alternative
2023-04-17 08:04:41 -04:00
Robert Hensing cb2615cf47 Merge remote-tracking branch 'upstream/master' into source-path 2023-04-17 11:41:50 +02:00
Robert Hensing 9af9c260fc
Merge pull request #8193 from hercules-ci/dry-strings
Deduplicate string literal rendering, fix 4909
2023-04-17 11:19:40 +02:00
John Ericson 9800c1e807 Mark experimental configuration settings programmatically
Fix #8162

The test is changed to compare `nlohmann::json` values, not strings of dumped
JSON, which allows us to format things more nicely.
2023-04-16 10:58:04 -04:00
Robert Hensing b6125772d7 libexpr: Move identifier-like printing to print.cc 2023-04-16 14:07:35 +02:00
Robert Hensing 28a5cdde02 libexpr/value/print.* -> libexpr/print.*
Generalizes the file to sensibly allow printing any part of the
language syntax.
2023-04-16 13:10:45 +02:00
Robert Hensing 1e2dd669bc printLiteral: Do not overload 2023-04-16 13:04:35 +02:00
Robert Hensing 9c74df5bb4
Format
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-04-15 20:56:51 +02:00
John Ericson 24866b71c4 Introduce SingleDrvOutputs
In many cases we are dealing with a collection of realisations, they are
all outputs of the same derivation. In that case, we don't need
"derivation hashes modulos" to be part of our map key, because the
output names alone will be unique. Those hashes are still part of the
realisation proper, so we aren't loosing any information, we're just
"normalizing our schema" by narrowing the "primary key".

Besides making our data model a bit "tighter" this allows us to avoid a
double `for` loop in `DerivationGoal::waiteeDone`. The inner `for` loop
was previously just to select the output we cared about without knowing
its hash. Now we can just select the output by name directly.

Note that neither protocol is changed as part of this: we are still
transferring `DrvOutputs` over the wire for `BuildResult`s. I would only
consider revising this once #6223 is merged, and we can mention protocol
versions inside factored-out serialization logic. Until then it is
better not change anything because it would come a the cost of code
reuse.
2023-04-15 12:51:19 -04:00
John Ericson 0f2b5146c7 Make restarting state machines explicit
If my memory is correct, @edolstra objected to modifying `wantedOutputs`
upon falling back to doing a build (as we did before), because we should
only modify it in response to new requests --- *actual* wants --- and
not because we are "incidentally" building all the outptus beyond what
may have been requested.

That's a fair point, and the alternative is to replace the boolean soup
with proper enums: Instead of modifying `wantedOuputs` som more, we'll
modify `needsRestart` to indicate we are passed the need.
2023-04-15 11:01:31 -04:00
John Ericson 37fca662b0 Make KeyedBuildResult, BuildResult like before, and fix bug another way
In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I
realized since derivation goals' wanted outputs can "grow" due to
overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called
by `Worker::makeDerivationGoalCommon`), the previous bug fix had an
unfortunate side effect of causing more pointless rebuilds.

In paticular, we have this situation:

1. Goal made from `DerivedPath::Built { foo, {a} }`.

2. Goal gives on on substituting, starts building.

3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just
   modified original goal.

4. Though the goal had gotten as far as building, so all outputs were
   going to be produced, `addWantedOutputs` no longer knows that and so
   the goal is flagged to be restarted.

This might sound far-fetched with input-addressed drvs, where we usually
basically have all our goals "planned out" before we start doing
anything, but with CA derivation goals and especially RFC 92, where *drv
resolution* means goals are created after some building is completed, it
is more likely to happen.

So the first thing to do was restore the clearing of `wantedOutputs` we
used to do, and then filter the outputs in `buildPathsWithResults` to
only get the ones we care about.

But fix also has its own side effect in that the `DerivedPath` in the
`BuildResult` in `DerivationGoal` cannot be trusted; it is merely the
*first* `DerivedPath` for which this goal was originally created.

To remedy this, I made `BuildResult` be like it was before, and instead
made `KeyedBuildResult` be a subclass wit the path. Only
`buildPathsWithResults` returns `KeyedBuildResult`s, everything else
just becomes like it was before, where the "key" is unambiguous from
context.

I think separating the "primary key" field(s) from the other fields is
good practical in general anyways. (I would like to do the same thing
for `ValidPathInfo`.) Among other things, it allows constructions like
`std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and
just precludes the possibility of those duplicate keys being out of
sync.

We might leverage the above someday to overload `buildPathsWithResults`
to take a *set* of return a *map* per the above.

-----

Unfortunately, we need to avoid C++20 strictness on designated
initializers.

(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)

No having that yet, maybe it would be better to not make
`KeyedBuildResult` a subclass to just avoid this.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-15 11:01:31 -04:00
John Ericson 9df7f3f537 Introduce Worker::makeGoal
This takes a `DerivedPath` so the caller doesn't need to care about
which sort of goal does what.
2023-04-15 11:01:31 -04:00
Robert Hensing ab228d73db
Merge pull request #8214 from raphaelr/installable-derived-path-warn
Display valid installable in `InstallableDerivedPath::parse` warning
2023-04-15 14:15:02 +02:00
John Ericson ee420ac64e Legacy vs non-legacy to_string/parse for DerivedPath
As requested by @roberth, it is good to call out the specific instances
we care about, which is `!` for the RPC protocols, and `^` for humans.

This doesn't take advantage of parametricity as much, but since the
human and computer interfaces are good to decouple anyways (we don't
care if they drift further apart over time in the slightest) some
separation and slight duplication is fine.

Also, unit test both round trips.
2023-04-14 20:45:11 -04:00
Archit Gupta bfc558c972 Whitelist commit-lockfile-summary in flake nixConfig 2023-04-14 11:33:38 -07:00
John Ericson 92346d1ed4
Merge pull request #8174 from NixOS/gloss-xp-features-note-installables
Start cross-referencing experimental features
2023-04-14 08:48:10 -04:00
John Ericson a6f85e052c Support repairPath on most stores.
More progress on issue #5729

The method trivially generalizes to be store-implementation-agnostic, in
fact.

However, we force it to continue to be unimplemented with `RemoteStore`
and `LegacySSHStore` because the implementation we'd get via the
generalization is probably not the one users expect. This keeps our
hands untied to do it right going forward.

For more about the tension between the scheduler logic being
store-type-agnostic and remote stores doing their own scheduling, see
issues #5025 and #5056.
2023-04-14 08:29:06 -04:00
Robert Hensing f5ab38a688
Merge pull request #8217 from obsidiansystems/push-get-fs-accessor-unsupported-down
Push `getFSAccessor` `unsupported(...)` down `Store` class hierarchy
2023-04-14 14:23:23 +02:00
John Ericson d93e76fbb8 Start cross-referencing experimental features
- Create a glossary entry for experimental features.

- Have the man page experimental feature notice link `nix-commmand`.

  (Eventually this should be programmed, based on whether the command is
  experimental, and if so what experimental feature does it depend on.)

- Document which installables depend on which experimental features.

  I tried to use the same style (bold warning and block quote) that the
  top of the man page uses.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-14 07:45:08 -04:00
Eelco Dolstra 7eac8838df
Merge pull request #8199 from tweag/fix-sqlite-busy-reporting
Fix unnecessary reporting of SQLite busy errors
2023-04-14 10:50:50 +02:00
Eelco Dolstra 33fc09c2a5
Merge pull request #8176 from tweag/rename-confusing-write-method
Rename and protect `BufferedSink::write`
2023-04-14 10:44:36 +02:00
Raphael Robatsch 9e8f209036 Display valid installable in InstallableDerivedPath::parse warning
The warning message should produce an installable name that can be
passed to `nix build`, `nix path-info`, etc. again. Since the CLI
expects that the .drv path and the output names are separated by
a caret, the warning message must also separate the .drv path and output
names with a caret.

However, `DerivedPath::Built.to_string()` uses an exclamation point as
the separator instead. This commit adds a `separator` argument to the
to_string method.

This changes the warning message from:
If this command is now failing try again with '/nix/store/foo.drv!*'
to:
If this command is now failing try again with '/nix/store/foo.drv^*'
2023-04-14 10:32:03 +02:00
John Ericson ee97f107e8 Push getFSAccessor unsupported(...) down Store class hierarchy
More progress on issue #5729.

Instead of having it by the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.

Combined with my other recent PRs, this finally makes `Store` have no
`unsupported` calls!
2023-04-13 13:39:44 -04:00
John Ericson 615c25b0dd Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-12 00:00:12 -04:00
Ben Radford de3df3009b
Move warning timing logic into handleSQLiteBusy. 2023-04-11 16:14:16 +01:00
John Ericson 450e5ec618 Do not gate or hide experimental settings
This is somewhat hacky fix just for 2.15. I unintentionally hid them
from the manual, when no one wanted to hide them that (including
myself). I also required the experimental feature to be enabled in an
order-dependent way, which is not good.

The simplest fix for this immanent release is just to always show them,
and always allow them to be set.

Effectively undoes some changes from aa663b7e89
2023-04-11 10:56:48 -04:00
Eelco Dolstra 162c3f5f7c
Merge pull request #8170 from tweag/fix-aarch64-build
Explicitely define `LockFile::operator!=`
2023-04-11 12:35:23 +02:00
Ben Radford 7c56e84213
Warn after a second of being busy instead of immediately.
Getting the occasional SQLITE_BUSY is expected when the database is being
accessed concurrently. The retry will likely succeed so it is pointless to warn
immediately. Instead we track how long each retrySQLite block has been running,
and only begin warning after a second has elapsed (and then every 10 seconds
subsequently).
2023-04-11 11:15:34 +01:00
Eelco Dolstra 60a1bf08b6
Merge pull request #7798 from peeley/list-experimental-features
Documentation: list experimental features in manual
2023-04-11 11:55:35 +02:00
Eelco Dolstra deb7f4b466
Nitpicks 2023-04-11 11:29:35 +02:00
Robert Hensing 4e0804c920 Deduplicate string literal rendering, fix 4909 2023-04-09 22:42:20 +02:00
John Ericson 51c8ffbc28 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-08 18:49:26 -04:00
John Ericson fd21f9d76e Merge remote-tracking branch 'upstream/master' into path-info 2023-04-07 20:39:04 -04:00
John Ericson 3ebd66c00e Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 19:54:34 -04:00
John Ericson 8f44edc83e Tidy up and comment daemon CLI
Some of the factoring out was taken from #7912 by @mupdt. Thanks!

No behavior should be changed in this commit.

Co-Authored-By: mupdt <25388474+mupdt@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-07 16:49:52 -04:00
Robert Hensing 4411c7d7e0
Merge pull request #8179 from tweag/disable-gc-on-coroutine
disable gc on coroutine
2023-04-07 18:21:11 +02:00
Robert Hensing ddebeb934a libstore: Remove lockCPU dead code
Left over from 9747ea84b, https://github.com/NixOS/nix/pull/5821
2023-04-07 17:50:40 +02:00
Robert Hensing 1c0b680ef9 libstore: Remove lockCPU dead code
Left over from 9747ea84b, https://github.com/NixOS/nix/pull/5821
2023-04-07 17:50:40 +02:00
Robert Hensing 2445afd92c Require openssl >= 1.1.1
Versions older this are sufficiently old that we don't want to support
them, and they require extra support code.
2023-04-07 17:50:40 +02:00
Robert Hensing 2196fd1146 libutil: Provide alternatives to startSignalHandlerThread
How signals should be handled depends on what kind of process Nix
is integrated into. The signal handler thread used by the stand-alone
Nix commands / processes may not work well in the context of other
runtime systems, such as those of Python, Perl, or Haskell.
2023-04-07 17:50:40 +02:00
John Ericson 38ae7c2891 Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 11:13:35 -04:00
John Ericson 6e1e15ffec Fix it! 2023-04-07 11:13:23 -04:00
Robert Hensing 781d3dceb3 Move initLibUtil() from initNix to initLibStore
libutil is a dependency of libstore, so it should always be
initialized as such.
libutil is also a dependency of libmain. Being explicit about this
dependency might be good, but not worth the slight code complexity
until the library structure gets more advanced.

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing 1107ea363f libmain: Clarify the lack of initLibExpr()
Quote

    Why not initLibExpr()? initGC() is essentially that, but
    detectStackOverflow is not an instance of the init function concept, as
    it may have to be invoked more than once per process.

Furthermore, renaming initGC to initLibExpr is more trouble than it's
worth at this time.
2023-04-07 16:24:18 +02:00
Robert Hensing 52d6ce6515 Move macOS TMPDIR hack from initNix to initLibStore
This code is bad. We shouldn't unset variables in programs whose
children may need them. Fixing one issue at a time, so postponing.
See https://github.com/NixOS/nix/issues/7731

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing e706ffa007 Move preloadNSS() from initNix to initLibStore
It is required for the sandbox, which is a libstore responsibility;
not just libmain.

Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing a58be39476 Move sodium_init() to initLibStore()
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
Robert Hensing 9693076715 Move initLibStore() immediately after initLibUtil()
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.

The goal of this reordering is to make initLibStore self-sufficient
in a following commit.
2023-04-07 16:24:18 +02:00
Robert Hensing a692c43729 Move loadConfFile() to initLibStore
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.

Using libstore without loading the config file is risky, as sqlite
may then be misconfigured. See https://github.com/cachix/cachix/issues/475
2023-04-07 16:24:18 +02:00
Robert Hensing 6e0b7109ab Move OpenSSL init to initLibUtil
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
2023-04-07 16:24:18 +02:00
John Ericson 0746951be1
Finish converting existing comments for internal API docs (#8146)
* Finish converting existing comments for internal API docs

99% of this was just reformatting existing comments. Only two exceptions:

- Expanded upon `BuildResult::status` compat note

- Split up file-level `symbol-table.hh` doc comments to get
  per-definition docs

Also fixed a few whitespace goofs, turning leading tabs to spaces and
removing trailing spaces.

Picking up from #8133

* Fix two things from comments

* Use triple-backtick not indent for `dumpPath`

* Convert GNU-style `\`..'` quotes to markdown style in API docs

This will render correctly.
2023-04-07 13:55:28 +00:00
John Ericson c036de086e Merge remote-tracking branch 'upstream/master' into trustless-remote-builder-simple 2023-04-07 09:40:36 -04:00
Yorick 62ddd8633c Add talkative msg for coro gc debug 2023-04-07 14:54:50 +02:00
Yorick 58d24a4cb6 Always disable GC in a coroutine unless the patch is applied 2023-04-07 14:54:38 +02:00
Yorick 00bc34430b DisableGC: replace by CoroutineContext, std::shared_ptr<void> 2023-04-07 14:53:40 +02:00
Yorick 2c53ef1bfe Disable GC inside coroutines on mac OS 2023-04-07 14:52:59 +02:00
John Ericson 59e0728714 Create nix derivation add command
Also refine `nix derivation show`'s docs very slightly.
2023-04-07 08:34:58 -04:00
John Ericson 27597f8131 Rename files to reflect new nix derivation show name
This will match the files we added for `nix add derivation` in the rest
of this PR.
2023-04-07 08:34:58 -04:00
John Ericson 2b98af2e62 nix show-derivation -> nix derivation show 2023-04-07 08:34:58 -04:00
John Ericson 4e9f32f993 Liberate checkDerivationOutputs from LocalStore
Make it instead a method on `Derivation` that can work with any store.
We will need this for a CLI command to create a derivation.
2023-04-07 08:34:58 -04:00
John Ericson b200784cec Include the name in the JSON for derivations
This is non-breaking change in the to-JSON direction. This *is* a
breaking change in the from-JSON direction, but we don't care, as that
is brand new in this PR.

`nix show-derivation --help` currently has the sole public documentation
of this format, it is updated accordingly.
2023-04-07 08:34:58 -04:00
John Ericson fe9cbe838c Create Derivation::fromJSON
And test, of course
2023-04-07 08:34:58 -04:00
Eelco Dolstra 6960de73af
Typo 2023-04-07 13:08:32 +02:00
Théophane Hufschmitt 214f1d6791 Rename and protect BufferedSink::write
The `write` name is ambiguous and could lead to some funny bugs like
https://github.com/NixOS/nix/pull/8173#issuecomment-1500009480. So
rename it to the more explicit `writeUnbuffered`.
Besides, this method shouldn't be (and isn't) used outside of the class
implementation, so mark it `protected`.

This makes it more symetrical to `BufferedSource` which uses a
`protected readUnbuffered` method.
2023-04-07 09:21:50 +02:00
matthewcroughan 9207f94582 Add Store::isTrustedClient()
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.

We include this information in `nix ping store` and `nix doctor`

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2023-04-06 19:59:57 -04:00
John Ericson bc192a95ef Describe active experimental features in the contributing guide
They are put in the manual separate pages under the new overarching
description of experimental features.

The settings page just lists the valid experimental feature names (so
people know what a valid setting entry looks like), with links to those
pages. It doesn't attempt to describe each experimental feature as that
is too much information for the configuration settings section.
2023-04-06 18:07:59 -04:00
John Ericson 6c4049b38a Link the new general documentation on xp features on the setting 2023-04-06 18:07:58 -04:00
Noah Snelson e399cb49c2
Fix typo in no-url-literals experimental feature docs
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-06 15:02:19 -07:00
Noah Snelson 8a7790f46a Expand documentation for experimental-features
Adds examples and additional information to the `impure-derivations`,
`recursive-nix`, and `no-url-literals` experimental feature
documentation.
2023-04-06 14:52:34 -07:00
Eelco Dolstra a9759407e5 Origin: Use SourcePath 2023-04-06 15:25:06 +02:00
Eelco Dolstra 94812cca98 Backport SourcePath from the lazy-trees branch
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
2023-04-06 13:15:50 +02:00
Théophane Hufschmitt faefaac875 Explicitely define LockFile::operator!=
It should be syntethised in terms of `operator==`, but the GCC version
used on aarch64-linux doesn't implement that (see
https://hydra.nixos.org/build/214848896=. So explicitely define it.

Fix https://github.com/NixOS/nix/issues/8159
2023-04-05 17:20:04 +02:00
John Ericson 53d0836347 Assemble experimental feature docs outside of Nix itself
Instead of constructing a markdown list in C++ (which involved all sorts
of nasty string literals), export some JSON and assemble it with the
manual build system.

Besides following the precedent set with other dumped data, this is a
better separate of content and presentation; if we decide for example we
want to display this information in a different way, or in a different
section of the manual, it will become much easier to do so.
2023-04-04 22:57:11 -04:00
John Ericson 3f98353f19 Merge remote-tracking branch 'upstream/master' into list-experimental-features 2023-04-04 21:34:20 -04:00
John Ericson bdeeffff96
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-04-04 19:16:10 -04:00
Théophane Hufschmitt d93c1514eb
Merge pull request #7944 from yorickvP/fix-curRepl
NixRepl::mainLoop: restore old curRepl on function exit
2023-04-04 09:44:47 +02:00
Théophane Hufschmitt 523c3f7225
Merge pull request #8157 from hercules-ci/switch-bugs
Fix current and future `switch` bugs
2023-04-03 20:43:02 +02:00
Eelco Dolstra 81491e1379
Merge pull request #8042 from lheckemann/alt-lockfiles
Allow specifying alternative paths for reading/writing flake locks
2023-04-03 19:28:09 +02:00
Robert Hensing bf2c5c3958 nix-store.cc: Refactor, remove qDefault 2023-04-03 18:45:20 +02:00
Robert Hensing fba7be80eb Enable -Werror=switch-enum
switch statements must now match all enum values or disable the
warning.

Explicit is good. This has helped us find two bugs, after solving
another one by debugging.

From now on, adding to an enum will raise errors where they are
not explicitly handled, which is good for productivity, and helps
us decide the correct behavior in all usages.

Notably still excluded from this though are the cases where the
warning is disabled by local pragmas.

fromTOML.cc did not build despite a top-level pragma, so I've had
to resort to a makefile solution for that.
2023-04-03 18:45:20 +02:00
Robert Hensing 9470ee877d Allow open switch-enum in 5 places 2023-04-03 18:45:20 +02:00
Robert Hensing 3dac4c7874 Add explicit case statements where -Wswitch-enum would report them 2023-04-03 18:17:32 +02:00
Robert Hensing ed7885017c Fix systemd logging for lvlNotice: eqv to lvlInfo, not lvlVomit 2023-04-03 18:17:32 +02:00
Robert Hensing 62cacc371f Fix BuildResult.toString() for NoSubstituters 2023-04-03 18:17:30 +02:00
John Ericson 4a0b893d5e Stuctured command stability
Prior to this, there was an ad-hoc whitelist in `main.cc`. Now, every
command states its stability.

In a future PR, we will adjust the manual to take advantage of this new
information in the JSON.
(It will be easier to do that once we have some experimental feature
docs to link too; see #5930 and #7798.)
2023-04-03 11:48:21 -04:00
Robert Hensing f3a6de6ba9
Merge pull request #8148 from hercules-ci/fix-issue-8119-printValue-tBlackhole-abort
eval: Fix crash on missing printValue tBlackhole case
2023-04-03 16:05:55 +02:00
Eelco Dolstra ac4318a1b9
Merge pull request #8152 from obsidiansystems/move-querySubstitutablePathInfos
Move `querySubstitutablePathInfos` from `LocalStore` to `Store`
2023-04-03 15:36:33 +02:00
Robert Hensing 1c55544a42 eval: Fix crash on missing printValue tBlackhole case
Fixes #8119
2023-04-03 15:32:42 +02:00
Théophane Hufschmitt 70bb7b7289
Merge pull request #7610 from obsidiansystems/gate-default-settings
Punt on improper global flags for now
2023-04-03 14:02:45 +02:00
John Ericson 9383520b75 Move querySubstitutablePathInfos from LocalStore to Store
The code is not local-store-specific, so we should share it with all
stores. More uniform behavior is better, and a less store-specific
functionality is more maintainable.

This fixes a FIXME added in f73d911628 by @edolstra himself.
2023-04-02 20:32:01 -04:00
John Ericson 32d72b1696 Add more API docs to experimental-features.hh 2023-04-02 18:57:46 -04:00
John Ericson 2585bcaa50 Rework a few things with the experimental features list
- Use struct not `std::pair`, designated initializers

- Use `constexpr` array that we can index by enum tag

- It no longer segfaults; not sure why.
2023-04-02 18:17:45 -04:00
John Ericson b2c9315bf2 Merge remote-tracking branch 'upstream/master' into list-experimental-features 2023-04-02 16:21:38 -04:00
John Ericson 5d56e2daf7 Add comparison methods for content addresses 2023-04-01 16:52:23 -04:00
John Ericson eeecfacb43 Merge branch 'path-info' into ca-drv-exotic 2023-04-01 16:40:32 -04:00
John Ericson 1f8e1edba9 Merge commit 'a6d00a7bfb18e7ec461ac1d54203cc628aca5c66' into ca-drv-exotic 2023-04-01 15:15:38 -04:00
John Ericson f7f44f7c96 Merge commit 'aa99005004bccc9be506a2a2f162f78bad4bcb41' into ca-drv-exotic 2023-04-01 15:15:32 -04:00
John Ericson 2ef99cd104
Merge pull request #8122 from bjornfor/use-nix-store-l-unless-experimental-enabled
Don't recommend 'nix log' unless experimental feature is enabled
2023-04-01 14:50:16 -04:00
John Ericson f4ab297b31 Ensure all headers have #pragma once and are in API docs
`///@file` makes them show up in the internal API dos. A tiny few were
missing `#pragma once`.
2023-03-31 23:19:44 -04:00
John Ericson abd5e7dec0 Extend internal API docs, part 2
Picking up from #8111.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-31 23:01:40 -04:00
Patrick Jackson 2b905d1d35 Replace unnecessary Sync<uint64_t> with std::atomic<uint64_t> 2023-03-31 17:56:25 -07:00
Eelco Dolstra 936e7c66eb Indentation 2023-03-31 17:15:03 +02:00
Eelco Dolstra 7ebaf0252a Add CanonPath::makeRelative() 2023-03-31 17:15:03 +02:00
Théophane Hufschmitt bb6e1244a3
Merge pull request #7406 from amesgen/gitlab-fetcher
Document `gitlab` input scheme
2023-03-31 13:21:55 +02:00
Théophane Hufschmitt e32ca3cf16
Merge pull request #8018 from tweag/ssh-password-prompt
SSH: don't erase password prompt if it is displayed
2023-03-31 12:06:10 +02:00
Patrick Jackson 804180ad52 Only lock once 2023-03-30 18:06:27 -07:00
Patrick Jackson 309753ebb5 Fix data race in copyPaths 2023-03-30 16:59:34 -07:00
John Ericson c51d554c93 Use "raw pattern" for content address types
We weren't because this ancient PR predated it!

This is actually a new version of the pattern which addresses some
issues identified in #7479.
2023-03-30 17:12:49 -04:00
John Ericson a6d00a7bfb Fix warning 2023-03-30 16:29:13 -04:00
John Ericson aa99005004 Merge remote-tracking branch 'upstream/master' into path-info
Also improve content-address.hh API docs.
2023-03-30 16:28:53 -04:00
John Ericson 06d87b95bc
Merge pull request #8111 from obsidiansystems/improve-internal-api-docs
Convert a bunch of comments in headers to Doxygen documentation
2023-03-30 09:40:22 -04:00
Valentin Gagarin 368c5b8b9a
Merge pull request #7518 from fricklerhandwerk/doc-commands
separate man pages for `nix-store` and `nix-env` subcommands
2023-03-30 15:33:08 +02:00
Eelco Dolstra 1cc5e1d5b6
Merge pull request #8123 from nomeata/import-flake-docs
Docs: Explain why `import nixpkgs` works in flakes
2023-03-30 11:20:55 +02:00
Eelco Dolstra 02050bd055
Typo 2023-03-30 10:49:21 +02:00
Joachim Breitner 1b6cfe6fa1 More precise wording 2023-03-30 10:40:04 +02:00
Alexander Bantyev 36b059748d Split nix-env and nix-store documentation per-subcommand
Documentation on "classic" commands with many sub-commands are
notoriously hard to discover due to lack of overview and anchor links.
Additionally the information on common options and environment variables
is not accessible offline in man pages, and therefore often overlooked
by readers.

With this change, each sub-command of nix-store and nix-env gets its
own page in the manual (listed in the table of contents), and each own
man page.

Also, man pages for each subcommand now (again) list common options
and environment variables. While this makes each page quite long and
some common parameters don't apply, this should still make it easier
to navigate as that additional information was not accessible on the
command line at all.

It is now possible to run 'nix-store --<subcommand> --help` to display
help pages for the given subcommand.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-30 09:46:28 +02:00
John Ericson 8cab89a94b Convert a bunch of comments in headers to Doxygen documentation
The internal API docs now contain more useful information.

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-29 23:22:07 -04:00
figsoda 081aadb76c Allow @ in git refs 2023-03-29 15:27:19 -04:00
Joachim Breitner 2e6b95dac7
Update src/libexpr/primops.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-29 13:19:51 +02:00
Vaci Koblizek a4dd87a2b3 avoid a string copy in printHash16 2023-03-29 10:54:27 +01:00
Vaci Koblizek ccf7ce26fe return string_view from printHashType rather than string 2023-03-29 10:44:22 +01:00
Bjørn Forsman 74d94b2a13 Don't recommend 'nix log' unless experimental feature is enabled
This fixes the issue that `nix-build`, without experimental feature
'nix-command' enabled, recommends the experimental CLI `nix log` to view
build logs. Now it'll recommend the stable `nix-store -l` CLI instead.

Fixes https://github.com/NixOS/nix/issues/8118
2023-03-28 21:02:08 +02:00
Vaci Koblizek 957f832074 Avoid a string copy during Hash::to_string 2023-03-28 17:20:33 +01:00
Joachim Breitner 3ce6d72b2b
Update src/libexpr/primops.cc
Co-authored-by: asymmetric <lorenzo@mailbox.org>
2023-03-28 14:38:39 +02:00
Joachim Breitner 605c8f7789 Docs: Explain why import nixpkgs works in flakes 2023-03-28 09:35:49 +02:00
Noah Snelson d1d1ae7a3b Documentation: list experimental features in manual
Lists all current experimental features in the `nix.conf` manual.
2023-03-27 17:12:49 -07:00
John Ericson 8df6f7bb0f Fix more profile dirs in docs
Picking up where #8078 left off.
2023-03-27 12:10:32 -04:00
Eelco Dolstra 5e3f855526
Merge pull request #7763 from obsidiansystems/installable-wide-info
Stratify `ExtraPathInfo` along `Installable` hierarchy
2023-03-27 17:04:08 +02:00
Eelco Dolstra 237587bc0a
Merge pull request #8084 from edolstra/store-docs
Auto-generate store documentation
2023-03-27 15:46:18 +02:00
John Ericson 1d539aa287
Merge pull request #8073 from tweag/fix-root-channels-location
Fix root channels location
2023-03-27 09:37:53 -04:00
Eelco Dolstra 5a0f5b5c34
Merge pull request #8062 from edolstra/ssl-cert-file
Add a setting for configuring the SSL certificates file
2023-03-27 15:37:15 +02:00
John Ericson 7c4dea3cf3 Punt on improper global flags for now
See the note in the test.

We don't want these flags showing up for commands where they are
irrelevant.

Eventually, this needs a proper fix, but it need not be a blocker for
stabilize: for a quick-n-dirty punt, just put these flags behind the
`nix-command` unstable feature.

This is fine because they are only relevant for commands which we don't
need to stabilize for a while.
2023-03-27 09:21:50 -04:00
John Ericson 570829d67e
Merge pull request #7609 from obsidiansystems/hide-experimental-settings
Hide experimental settings
2023-03-27 09:19:29 -04:00
Eelco Dolstra a519436e9d ssl-cert-file: Use lists 2023-03-27 14:08:39 +02:00
Eelco Dolstra 2523448bad
Merge pull request #8104 from avdv/graphml-use-long-for-narsize
nix-store: Use `long` for `narSize` in graphml output
2023-03-27 13:10:28 +02:00
Théophane Hufschmitt 128994509f Make some comments in profiles.hh doxygen-enabled
These are proper documentation of the API, so they deserve to be here
2023-03-27 10:02:10 +02:00
oxalica 2941a599fa Catch eval errors in hasContent
`legacyPackages` of nixpkgs trigger eval errors in `hasContent`, causing
the whole `legacyPackages` being skipped. We should treat it as
has-content in that case.
2023-03-26 23:05:29 +08:00
Claudio Bley e7b9dc7c89 nix-store: Use long for narSize in graphml output 2023-03-24 17:25:17 +01:00
John Ericson 256f3e3063 Stratify ExtraPathInfo along Installable hierarchy
Instead of having a bunch of optional fields, have a few subclasses
which can have mandatory fields.

Additionally, the new `getExtraPathInfo`, and `nixpkgsFlakeRef`, are
moved to `InstallableValue`.

I did these things because https://github.com/NixOS/rfcs/pull/134 ; with
these things moved to `InstallableValue`, the base `Installable` no
longer depends on libexpr! This is a major step towards that.

Also, add a bunch of doc comments for sake of the internal API docs.
2023-03-24 12:22:40 -04:00
Eelco Dolstra e00abd3f56
Merge pull request #7757 from obsidiansystems/more-installable-value
Move value-only methods to `InstallableValue`
2023-03-24 14:07:18 +01:00
Théophane Hufschmitt a4c2cd3c06
Merge pull request #8098 from hercules-ci/error-cc-Only-suggest-show-trace-when-truncated-trace-items-would-be-pri
error.cc: Only suggest `--show-trace` when relevant
2023-03-24 12:07:44 +01:00
Robert Hensing 233c4cf30f error.cc: Only suggest show-trace when truncated trace items would be printed
Otherwise, a trace consisting of

frame
frame
frame
non-frame

... would reach the non-frame and print the suggestion, even though
it would have ignored the non-frame anyway.

This resulted in a peculariar situation where --show-trace would have
no apparent effect, as the trace was actually already complete.
2023-03-23 17:29:06 +01:00
Eelco Dolstra 168b6021c5 Tweaks 2023-03-23 15:32:59 +01:00
Eelco Dolstra b79df9dedc Register LocalStore to ensure it's included in the manual 2023-03-23 15:23:13 +01:00
Eelco Dolstra 161f4b0dea Document store URLs 2023-03-23 10:38:48 +01:00
Théophane Hufschmitt fb67c1a1fb Factor out the generation of the profile/channel directory
Make sure that all the code paths use the same one, and that the
backwards-compatibility measures are probably in place when needed
2023-03-23 10:22:42 +01:00
Théophane Hufschmitt 82bd9535dd nix-channel: Restore the old root channels directory 2023-03-23 10:22:42 +01:00
Eelco Dolstra abc449bc30
Update src/libstore/local-store.md
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-03-23 10:12:57 +01:00
Eelco Dolstra 80f0b8d307 Fix SSHStore 2023-03-23 09:35:35 +01:00
Eelco Dolstra b134546f08 Fix clang build 2023-03-23 09:11:15 +01:00
John Ericson e5146a6a50
Merge pull request #8078 from obsidiansystems/profile-md
Fix `nix profile --help`
2023-03-22 19:36:12 -04:00
Eelco Dolstra 5691bac202 Improve store setting descriptions / Markdown formatting 2023-03-22 14:23:36 +01:00
Alexander Bantyev 85a2d1d94f
Add a test for nix copy over ssh
Check that nix copy can copy stuff, refuses to copy unsigned paths by
default, and doesn't hide the ssh password prompt.
2023-03-22 09:45:08 +04:00
Alexander Bantyev 5291a82cd9
SSHMaster: pause logger to show password prompt
Pause logger before starting SSH connections, and resume it after the
connection is established, so that SSH password prompts are not erased
by the logger's updates.
2023-03-22 09:45:08 +04:00
Alexander Bantyev 85df7e7ea2
Logger, ProgressBar: add a way to pause/resume
Add new virtual methods pause and resume to the Logger class, and
implement them in ProgressBar to allow to pause the bar refreshing.
2023-03-22 09:45:08 +04:00
Eelco Dolstra 16cb9b9f04 Fix rendering of fetchGit documentation
stripIndentation() doesn't support tabs, so the entire markdown ended
up indented and thus rendered as a code block.
2023-03-21 14:48:15 +01:00
Eelco Dolstra c967c29290 Add a "help" category
This makes the help commands show up prominently at the top of the
'nix' manpage.
2023-03-21 14:43:58 +01:00
Eelco Dolstra 233b063b08 Move store docs to 'nix help-stores'
Why not 'nix help stores'? Well, 'nix help <arg>' already means 'show
help on the "arg" subcommand'.
2023-03-21 14:37:09 +01:00
Eelco Dolstra 9eb53bbf17 Support per-store Markdown documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra 7704118d28 nix describe-stores: Remove
This command was intended for docs generation, but it was never used
for that and we don't need it.
2023-03-21 14:03:40 +01:00
Eelco Dolstra 8d6d59cb1b nix store --help: Include store type documentation 2023-03-21 14:03:40 +01:00
Eelco Dolstra 3ac3e31c4d Fix misrendering of 'nix store --help'
There are no categories underneath 'nix store', so having 'nix store
copy-log' in a category rendered as ':'.
2023-03-21 12:02:19 +01:00
John Ericson f87f1bbfe6 Fix nix profile --help
The documentation hadn't be updated to reflect the new default profile
location.
2023-03-20 14:22:23 -04:00
Eelco Dolstra 515662ad70 Cleanup 2023-03-20 18:06:08 +01:00
Eelco Dolstra 16db8dc96f Open slave pseudoterminal before CLONE_NEWUSER
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.

Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.
2023-03-20 17:58:36 +01:00
John Ericson 4607ac7aed Fix handling of experimental features mid-parse
If we conditionally "declare" the argument, as we did before, based upon
weather the feature is enabled, commands like

    nix --experimental-features=foo ... --thing-gated-on-foo

won't work, because the experimental feature isn't enabled until *after*
we start parsing.

Instead, allow arguments to also be associated with experimental
features (just as we did for builtins and settings), and then the
command line parser will filter out the experimental ones.

Since the effects of arguments (handler functions) are performed right
away, we get the required behavior: earlier arguments can enable later
arguments enabled!

There is just one catch: we want to keep non-positional
flags...non-positional. So if

    nix --experimental-features=foo ... --thing-gated-on-foo

works, then

    nix --thing-gated-on-foo --experimental-features=foo ...

should also work.

This is not my favorite long-term solution, but for now this is
implemented by delaying the requirement of needed experimental features
until *after* all the arguments have been parsed.
2023-03-20 11:35:34 -04:00
John Ericson aa663b7e89 Mark experimental features on settings
We hide them in various ways if the experimental feature isn't enabled.

To do this, we had to move the experimental features list out of
libnixstore, because the setting machinary itself depends on it. To do
that, we made a new `ExperimentalFeatureSettings`.
2023-03-20 11:06:40 -04:00
John Ericson 296831f641 Move enabled experimental feature to libutil struct
This is needed in subsequent commits to allow the settings and CLI args
infrastructure itself to read this setting.
2023-03-20 11:05:22 -04:00
figsoda 54b8da0b10 docs: quote URL literals 2023-03-19 11:28:10 -04:00
Linus Heckemann f1c9d83697 Only allow reference lock files when allow-dirty is set 2023-03-19 14:12:49 +01:00
Eelco Dolstra e53e5c38d4 Add a setting for configuring the SSL certificates file
This provides a platform-independent way to configure the SSL
certificates file in the Nix daemon. Previously we provided
instructions for overriding the environment variable in launchd, but
that obviously doesn't work with systemd. Now we can just tell users
to add

  ssl-cert-file = /etc/ssl/my-certificate-bundle.crt

to their nix.conf.
2023-03-17 18:32:18 +01:00
John Ericson 790dd2555b
Merge pull request #7891 from mupdt/shared-fs-out-links
no-op refactor: extract outLink generation into a function
2023-03-17 10:57:07 -04:00
John Ericson c998e0172f Move value-only methods to InstallableValue
These methods would previously fail on the other `Installable`s, so
moving them to this class is more correct as to where they actually
work.

Additionally, a `InstallableValueCommand` is created to make it easier
(or rather no worse than before) to write commands that just work on
`InstallableValue`s.

Besides being a cleanup to avoid failing default methods, this gets us
closer to https://github.com/NixOS/rfcs/pull/134.
2023-03-16 20:06:46 -04:00
Eelco Dolstra 7f46ebcf90
Merge pull request #8049 from edolstra/unexpected-eof
Fix "unexpected EOF" errors on macOS
2023-03-16 16:13:42 +01:00
Linus Heckemann af4cbdafe7 LocalDerivationGoal: set NIX_ATTRS_*_FILE correctly for sandboxed builds 2023-03-16 15:26:52 +01:00
John Ericson eb56cb7cc7
Merge pull request #7750 from obsidiansystems/no-args-prepare
Make command infra less stateful and more regular
2023-03-15 17:23:40 -04:00
John Ericson bc23a44c54 Make command infra less stateful and more regular
Already, we had classes like `BuiltPathsCommand` and `StorePathsCommand`
which provided alternative `run` virtual functions providing the
implementation with more arguments. This was a very nice and easy way to
make writing command; just fill in the virtual functions and it is
fairly clear what to do.

However, exception to this pattern were `Installable{,s}Command`. These
two classes instead just had a field where the installables would be
stored, and various side-effecting `prepare` and `load` machinery too
fill them in. Command would wish out those fields.

This isn't so clear to use.

What this commit does is make those command classes like the others,
with richer `run` functions.

Not only does this restore the pattern making commands easier to write,
it has a number of other benefits:

- `prepare` and `load` are gone entirely! One command just hands just
  hands off to the next.

- `useDefaultInstallables` because `defaultInstallables`. This takes
  over `prepare` for the one case that needs it, and provides enough
  flexiblity to handle `nix repl`'s idiosyncratic migration.

- We can use `ref` instead of `std::shared_ptr`. The former must be
  initialized (so it is like Rust's `Box` rather than `Option<Box>`,
  This expresses the invariant that the installable are in fact
  initialized much better.

  This is possible because since we just have local variables not
  fields, we can stop worrying about the not-yet-initialized case.

- Fewer lines of code! (Finally I have a large refactor that makes the
  number go down not up...)

- `nix repl` is now implemented in a clearer way.

The last item deserves further mention. `nix repl` is not like the other
installable commands because instead working from once-loaded
installables, it needs to be able to load them again and again.

To properly support this, we make a new superclass
`RawInstallablesCommand`. This class has the argument parsing and
completion logic, but does *not* hand off parsed installables but
instead just the raw string arguments.

This is exactly what `nix repl` needs, and allows us to instead of
having the logic awkwardly split between `prepare`,
`useDefaultInstallables,` and `load`, have everything right next to each
other. I think this will enable future simplifications of that argument
defaulting logic, but I am saving those for a future PR --- best to keep
code motion and more complicated boolean expression rewriting separate
steps.

The "diagnostic ignored `-Woverloaded-virtual`" pragma helps because C++
doesn't like our many `run` methods. In our case, we don't mind the
shadowing it all --- it is *intentional* that the derived class only
provides a `run` method, and doesn't call any of the overridden `run`
methods.

Helps with https://github.com/NixOS/rfcs/pull/134
2023-03-15 16:29:07 -04:00
John Ericson 0a140a92bb
Merge pull request #7690 from ShamrockLee/nix-hash-sri
nix-hash: support base-64 and SRI format
2023-03-15 16:02:46 -04:00
Yueh-Shun Li c27d358abb nix-hash: support base-64 and SRI format
Add the --base64 and --sri flags for the Base64 and SRI format output.

Add the --base16 flag to explicitly specify the hexadecimal format.

Add the --to-base64 and --to-sri flag to convert a hash to the above
mentioned format.
2023-03-16 03:08:42 +08:00
figsoda 31d588db8b respect allRefs=1 when using nix flake prefetch 2023-03-15 11:29:01 -04:00
Eelco Dolstra 5eb8bfd0f1 Remove a variable in LocalDerivationGoal 2023-03-15 10:59:10 +01:00
Eelco Dolstra 6029c763c2 Change builderOut from Pipe to AutoCloseFD 2023-03-15 10:59:10 +01:00
Eelco Dolstra c536e00c9d Move pseudoterminal slave open to child
Hopefully this fixes "unexpected EOF" failures on macOS
(#3137, #3605, #7242, #7702).

The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:

* The child finishes very quickly. That's why this bug is likely to
  trigger in nix-env tests, since that uses a builtin builder. Adding
  a short sleep before the child exits makes the problem go away.

* The parent has closed its duplicate of the slave file
  descriptor. This shouldn't matter, since the child has a duplicate
  as well, but it does. E.g. moving the close to the bottom of
  startBuilder() makes the problem go away. However, that's not a
  solution because it would make Nix hang if the child dies before
  sending the "sandbox initialized" message.

* The system is under high load. E.g. "make installcheck -j16" makes
  the issue pretty reproducible, while it's very rare under "make
  installcheck -j1".

As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.
2023-03-15 10:58:37 +01:00
Eelco Dolstra 19326ac297 Simplify commonChildInit() 2023-03-14 18:58:12 +01:00
Linus Heckemann 3a1de4c3fe Apply review suggestions
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-03-14 12:02:03 +01:00
Linus Heckemann e91596eb69 Allow specifying alternative paths for reading/writing flake locks
This allows having multiple separate lockfiles for a single
project, which can be useful for testing against different versions of
nixpkgs; it also allows tracking custom input overrides for remote
flakes without requiring local clones of these flakes.

For example, if I want to build Nix against my locally pinned nixpkgs,
and have a lock file tracking this override independently of future
updates to said nixpkgs:

nix flake lock --output-lock-file /tmp/nix-flake.lock --override-input nixpkgs flake:nixpkgs
nix build --reference-lock-file /tmp/nix-flake.lock

Co-Authored-By: Will Fancher <elvishjerricco@gmail.com>
2023-03-13 21:08:52 +01:00
Robert Hensing a387f46967
Merge pull request #8033 from lbodor/stop-adding-dot-to-nix-dev-env-path
`print-dev-env`: stop inadvertently adding `.` to `PATH`
2023-03-13 19:52:41 +01:00
Eelco Dolstra fcd0b0fbd5 Fix macOS warning 2023-03-13 05:31:03 -07:00
Lazar Bodor 526bdbda3c print-dev-env: stop inadvertently adding . to PATH 2023-03-12 22:40:47 +11:00
Théophane Hufschmitt 99af56cd0d
Merge pull request #8015 from tweag/progress-during-nix-copy
Display progress when running copyPaths (nix copy)
2023-03-10 16:23:22 +01:00
Eelco Dolstra 4f3a4b732c Make findRuntimeRoots() more resilient to disappearing processes
I saw this random failure in https://hydra.nixos.org/build/211811692:

  error: opening /proc/15307/fd: No such process

while running nix-collect-garbage in a readfile-context.sh. This is
because we're not handling ESRCH errors reading /proc/<pid>/fd. So
just move the read inside the try/catch where we do handle it.
2023-03-09 16:44:51 +01:00
Eelco Dolstra 8f96720b9f Fix another uninitialized variable
https://hydra.nixos.org/build/211811494
2023-03-09 15:30:53 +01:00
Alexander Bantyev 4bef2016a1
Display progress when running copyPaths (nix copy)
`nix copy` operations did not show progress. This is quite confusing.

Add a `progressSink` which displays the progress during `copyPaths`,
pretty much copied from `copyStorePath`.

Fixes https://github.com/NixOS/nix/issues/8000
2023-03-09 16:03:48 +04:00
Théophane Hufschmitt 39700c5cbe
Revert "Disable GC during coroutine execution + test" 2023-03-08 20:47:52 +01:00
Eelco Dolstra 1607ad5076 Fix uninitialized readFromStdIn variable
This was causing random failures in tests/ca/substitute.ca: 'nix copy
--file ./content-addressed.nix' wouldn't get the default installable
'.' applied in InstallablesCommand::load(), so it would do nothing.
2023-03-08 13:53:20 +01:00
Eelco Dolstra df48040786
Merge pull request #7994 from edolstra/fix-ca-crash
Fix crash/hang with CA derivations
2023-03-08 12:15:55 +01:00
Eelco Dolstra 6b243ef2d5
Merge pull request #7993 from tweag/fix-profile-gc
Fix `nix-collect-garbage -d` with the new profile location
2023-03-08 11:10:45 +01:00
Eelco Dolstra 7bfed34367 Fix crash/hang with CA derivations
The curl download can outlive DrvOutputSubstitutionGoal (if some other
error occurs), so at shutdown setting the promise to an exception will
fail because 'this' is no longer valid in the callback. This can
manifest itself as a segfault, "corrupted double-linked list" or hang.
2023-03-08 11:09:15 +01:00
Théophane Hufschmitt 4a6244dcf7
Merge pull request #7725 from yorickvP/check-coro-gc
Disable GC during coroutine execution + test
2023-03-08 09:51:46 +01:00
Théophane Hufschmitt 2272bc6049 Fix nix-collect-garbage -d with the new profile location
Low-hanging fix for https://github.com/NixOS/nix/pull/5226#issuecomment-1454669399
2023-03-07 13:08:04 +01:00
Théophane Hufschmitt ba0486f045
Merge pull request #7889 from sidkshatriya/sorted-fetch-paths
Print the store paths to be fetched sorted by StorePath name()
2023-03-07 11:58:10 +01:00
mupdt 762cf7f214 no-op refactor: extract outLink generation into a function 2023-03-06 10:20:21 -05:00
Eelco Dolstra 3f4dc61f7f
Merge pull request #7958 from amesgen/issue-7955
InstallableFlake: Apply nix config in `getCursors`
2023-03-06 11:08:51 +01:00
Eelco Dolstra 92771ca1be
Merge pull request #7975 from fricklerhandwerk/fix-fetchgit
remove indentation in `fetchGit` attribute listing
2023-03-06 11:00:40 +01:00
Valentin Gagarin dfeb83cac1 output paths of store derivations are now addressed explicitly 2023-03-06 03:32:14 +01:00
Valentin Gagarin f69d453c3e remove indentation in fetchGit attribute listing
also reword a confusing sentence and add links to Git terminology
2023-03-05 04:44:11 +01:00
Valentin Gagarin 2af9fd20c6 clarify definition of "installable"
the term was hard to discover, as its definition and explanation were in
a very long document lacking an overview section.
search did not help because it occurs so often.

- clarify wording in the definition
- add an overview of installable types
- add "installable" to glossary
- link to definition from occurrences of the term
- be more precise about where store derivation outputs are processed
- installable Nix expressions must evaluate to a derivation

Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
2023-03-05 01:46:17 +01:00
Valentin Gagarin 1e87d5f1ea make descriptions of each installable type an own subsection
this is easier to edit, provides anchors for free, and renders correctly
on the terminal without additional effort.
2023-03-05 01:46:17 +01:00
amesgen 77bb950566
InstallableFlake: Apply nix config in getCursors 2023-03-04 12:52:20 +01:00
Sidharth Kshatriya 427555861b Print the store paths to be fetched sorted by StorePath name() and not baseName
Presently when nix says something like:

```
these 486 paths will be fetched (511.54 MiB download, 6458.64 MiB unpacked):
 ...path1
 ...path2
 ...path3
    ...
    ...
 ...path486
```

It sorts path1, path2, path3, ..., path486 in lexicographic order of the
store path.

After this commit, nix will show path1, path2, path3, ..., path486 sorted by
StorePath name() (basically everything after the hash) rather than the store path.

This makes it easier to review what exactly is being downloaded at a glance,
especially when many paths need to be fetched.
2023-03-03 21:46:35 +05:30
Eelco Dolstra 0507462c06
Merge pull request #7918 from zimbatm/fix-empty-nix-store-env
treat empty NIX_STORE_DIR env vars as unset
2023-03-03 13:49:56 +01:00
Yorick 2683734936
Add talkative msg for coro gc debug 2023-03-03 11:43:47 +01:00
Jonas Chevalier dc8820c71f
fixup: use same style as getEnv 2023-03-03 11:34:36 +01:00
Jonas Chevalier b96d9c1687
fixup: remove warning entirely
fixes 72e1e23051 (r1124211067)
2023-03-03 11:32:06 +01:00
Eelco Dolstra f0908f592c
Merge pull request #7942 from edolstra/remove-format
Remove FormatOrString and remaining uses of format()
2023-03-03 10:22:11 +01:00
Valentin Gagarin 37c61907d7
Merge pull request #7605 from tweag/comments
add comments
2023-03-03 06:38:29 +01:00
Théophane Hufschmitt 1f394d2107
Merge branch 'master' into paths-from-stdin 2023-03-02 19:20:51 +01:00
Yorick 5bb6e3bfaf
NixRepl::mainLoop: restore old curRepl on function exit
This fixes completion callbacks after entering and leaving a nested
debugger.
2023-03-02 17:24:58 +01:00
Eelco Dolstra 687d86f8ab
Merge pull request #7940 from tweag/fix-http-errors-encoding
Log the decompressed body in case of http errors
2023-03-02 16:25:09 +01:00
Jonas Chevalier 72e1e23051
Update src/libutil/util.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-03-02 16:17:20 +01:00
Eelco Dolstra 29abc8e764 Remove FormatOrString and remaining uses of format() 2023-03-02 15:57:54 +01:00
Eelco Dolstra b69a73a230 Get rid of some unchecked calls to std::cout 2023-03-02 15:02:24 +01:00
Théophane Hufschmitt 1a825b6919 Log the decompressed body in case of http errors
Don't show the users the raw (possibly compressed) error message as
everyone isn't able to decompress brotli in their brain.
2023-03-02 14:59:15 +01:00
Eelco Dolstra 09f5975c6a Logger::cout: Use fmt()
This ensures that in cout(s), 's' does not get interpreted as a format
string.
2023-03-02 14:52:37 +01:00
Eelco Dolstra 989b823ac5 nix store cat: Use writeFull()
Fixes #7939.
2023-03-02 14:51:14 +01:00
Eelco Dolstra c13cbd20ab Logger::writeToStdout(): Use writeFull()
This ensures that write errors do not get ignored.
2023-03-02 14:46:28 +01:00
Eelco Dolstra 767974f411
Merge pull request #7924 from mkenigs/valid
Always set valid in path-info --json output
2023-03-02 09:58:20 +01:00
Félix Baylac Jacqué 25300c0ecd
Treat empty env var paths as unset
We make sure the env var paths are actually set (ie. not "") before
sending them to the canonicalization function. If we forget to do so,
the user will end up facing a puzzled failed assertion internal error.

We issue a non-failing warning as a stop-gap measure. We could want to
revisit this to issue a detailed failing error message in the future.
2023-03-01 20:50:07 +01:00
Théophane Hufschmitt 182129d985
Merge pull request #7932 from obsidiansystems/remove-sameMachine
Remove dead code `RemoteStore::sameMachine`
2023-03-01 20:05:39 +01:00
Valentin Gagarin 651dab55da
Merge pull request #7854 from aameen-tulip/patch-1 2023-03-01 18:24:53 +01:00
John Ericson b7f01a82a9 Remove dead code RemoteStore::sameMachine
It has been dead code since 9747ea84b4.
2023-03-01 11:10:30 -05:00
Yorick 176005749c
Always disable GC in a coroutine unless the patch is applied 2023-03-01 15:07:00 +01:00
Yorick 4c73eab923
DisableGC: replace by CoroutineContext, std::shared_ptr<void> 2023-03-01 13:55:41 +01:00
Yorick 53bb4a5327
tests/coro-gc: refactor and split into 3 tests 2023-03-01 13:55:41 +01:00
Yorick eaeb994d8b
Disable GC inside coroutines on mac OS 2023-03-01 13:55:41 +01:00
Yorick 0fd8f542a8
tests/coro-gc: create test for boehm stack patch
Regression test for #7679
2023-03-01 13:55:37 +01:00
Valentin Gagarin 306e5c5ce5
Merge pull request #7788 from bobvanderlinden/pr-improve-nix-profile-install-error
Improve error on conflict for nix profile install
2023-03-01 11:48:43 +01:00
Bob van der Linden 12538605fd
nix-profile: add FIXME about using C++20 std::ranges 2023-03-01 07:40:53 +01:00
Matthew Kenigsberg f86f2b973f Always set valid in path-info --json output
Currently the valid key is only present when the path is invalid, which
makes checking path validity more complex than it should be. With this
change, the valid key can always be used to check if a path is valid
2023-02-28 16:04:41 -07:00
John Ericson ea0adfc582 Get rid of .drv special-casing for store path installables
The release notes document the change in behavior, I don't include it
here so there is no risk to it getting out of sync.

> Motivation

>> Plumbing CLI should be simple

Store derivation installations are intended as "plumbing": very simple
utilities for advanced users and scripts, and not what regular users
interact with. (Similarly, regular Git users will use branch and tag
names not explicit hashes for most things.)

The plumbing CLI should prize simplicity over convenience; that is its
raison d'etre. If the user provides a path, we should treat it the same
way not caring what sort of path it is.

>> Scripting

This is especially important for the scripting use-case. when arbitrary
paths are sent to e.g. `nix copy` and the script author wants consistent
behavior regardless of what those store paths are. Otherwise the script
author needs to be careful to filter out `.drv` ones, and then run `nix
copy` again with those paths and `--derivation`. That is not good!

>> Surprisingly low impact

Only two lines in the tests need changing, showing that the impact of
this is pretty light.

Many command, like `nix log` will continue to work with just the
derivation passed as before. This because we used to:

- Special case the drv path and replace it with it's outputs (what this
  gets rid of).

- Turn those output path *back* into the original drv path.

Now we just skip that entire round trip!

> Context

Issue #7261 lays out a broader vision for getting rid of `--derivation`,
and has this as one of its dependencies. But we can do this with or
without that.

`Installable::toDerivations` is changed to handle the case of a
`DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't
need to do any extra work in that case. On this basis, commands like
`nix {show-derivation,log} /nix/store/...-foo.drv` still work as before,
as described above.

When testing older daemons, the post-build-hook will be run against the
old CLI, so we need the old version of the post-build-hook to support
that use-case.

Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-28 17:07:05 -05:00
Timothy DeHerrera df643051e2
nix-store: read paths from standard input
Resolves #7437 for new `nix-store` by adding a `--stdin` flag.
2023-02-28 12:29:16 -07:00
Timothy DeHerrera 269caa5317
feat: read installable paths from stdin
Resolves #7437 for new `nix` commands only by adding a `--stdin` flag.

If paths are also passed on the cli they will be combined with the ones
from standard input.
2023-02-28 12:29:15 -07:00
John Ericson 5abd643c6d Merge branch 'path-info' into ca-drv-exotic 2023-02-28 12:46:00 -05:00
John Ericson d381248ec0 No inheritance for TextInfo and FixedOutputInfo 2023-02-28 12:14:11 -05:00
John Ericson 85bb865d20 Revert "Remove some designated initializers"
This reverts commit ee9eb83a84.
2023-02-28 11:57:20 -05:00
John Ericson 123b11ff83 Clarify store path grammar and improve comment on makeType 2023-02-28 11:49:13 -05:00
John Ericson d12f57c2c0 Merge remote-tracking branch 'upstream/master' into path-info 2023-02-28 11:34:34 -05:00
John Ericson c36b584f8e Fix typo in the method name 2023-02-28 11:34:18 -05:00
Eelco Dolstra a4a5d828e2
Merge pull request #7793 from layus/interrupt_downloads
Check interrupts even when download stalled
2023-02-28 13:29:29 +01:00
Eelco Dolstra da1f49c4e3
Merge pull request #7904 from anatol/patch-1
Example uses gitlab.com thus clarify the comment
2023-02-28 13:29:10 +01:00
Bob van der Linden 872cdb4346
nix-profile-install: show helpful error upon package conflict
Whenever a file conflict happens during "nix profile install" an error
is shown that was previously thrown inside builtins.buildEnv.

We catch BuildProfileConflictError here so that we can provide the user
with more useful instructions on what to do next.

Most notably, we give the user concrete commands to use with all
parameters  already filled in. This avoids the need for the user to look
up these commands in manual pages.
2023-02-28 09:28:05 +01:00
Eelco Dolstra 3d15dbadc2
Merge pull request #7911 from edolstra/revert-7689
Revert #7689
2023-02-28 08:46:55 +01:00
Bob van der Linden 3113b13df9
buildenv: throw BuildEnvFileConflictError with more context
At the moment an Error is thrown that only holds an error message
regarding `nix-env` and `nix profile`. These tools make use of
builtins.buildEnv, but buildEnv is also used in other places. These
places are unrelated to Nix profiles, so the error shouldn't mention
these tools.

This generic error is now BuildEnvFileConflictError, which holds more
contextual information about the files that were conflicting while
building the environment.
2023-02-27 21:39:34 +01:00
Théophane Hufschmitt eae89aca1b
Merge pull request #7776 from yorickvP/fix-path-escapes-7707
Properly escape local paths into URLs in fetchTree
2023-02-27 21:10:25 +01:00
Eelco Dolstra f08ad5bdba
Merge pull request #7913 from fricklerhandwerk/master
add information on the `build-hook` setting
2023-02-27 17:33:25 +01:00
Valentin Gagarin fd0e21475c add information on the build-hook setting
add a warning that you probably never want to change this.
2023-02-27 16:27:56 +01:00
Yorick 2c0866fc3f
fetchTree: convert fs path to url via ParsedURL::to_string 2023-02-27 15:30:04 +01:00
Yorick 0844856c84
url: make percentEncode stricter, expose and unit test it 2023-02-27 15:30:00 +01:00
Eelco Dolstra dd93c12c6a Revert "getDefaultNixPath: actually respect {restrict,pure}-eval"
This reverts commit 1cba5984a6.
2023-02-27 15:11:36 +01:00
Eelco Dolstra e928c72cf9 Revert "Document default nix-path value"
This reverts commit dba9173a1d.
2023-02-27 14:16:49 +01:00
Théophane Hufschmitt 995bfeef3b
Merge pull request #7796 from hercules-ci/fix-7263
Ensure that `self.outPath == ./.`
2023-02-27 10:26:02 +01:00
Anatol Pomozov d731235f6b
Example uses gitlab.com thus clarify the comment 2023-02-24 07:17:47 -08:00
Théophane Hufschmitt 4a921ba43b
Merge pull request #7764 from yorickvP/build-remote-warning
build-remote: don't warn when all local build slots are taken
2023-02-22 17:43:20 +01:00
Robert Hensing 5d834c40d0 flakes: Differentiate self.outPath and self.sourceInfo.outPath
It would be incorrect to say that the `sourceInfo` has an `outPath`
that isn't the root. `sourceInfo` is about the root, whereas only
the flake may not be about the root. Thanks Eelco for pointing that
out.
2023-02-22 03:31:24 +01:00
Robert Hensing 904a107d16 flakes: Ensure that self.outPath == ./.
Users expect `self` to refer to the directory where the `flake.nix`
file resides.
2023-02-22 03:30:47 +01:00
Théophane Hufschmitt 8418d22ac1
Merge pull request #7874 from obsidiansystems/fix-no-gc-build
Fix the build without GC
2023-02-21 16:14:51 +01:00
Théophane Hufschmitt c7885ab6f2
Merge pull request #7755 from obsidiansystems/mix-read-only-mode
Make `--read-only` a separate mixin
2023-02-21 16:13:48 +01:00
John Ericson 5b0175e81d Fix the build without GC
I had given it an improper trailing comma in
1bd03ad100.
2023-02-21 09:38:46 -05:00
Théophane Hufschmitt 532c70f531
Merge pull request #7856 from yorickvP/fix-nsswitch
Wait with making /etc unwritable until after build env setup
2023-02-21 09:39:10 +01:00
John Ericson 208c8d326d Derivation::toJSON: fix bug!
When I moved this code from the binary to libnixstore #7863, I forgot to
display the environment variables!
2023-02-20 17:38:57 -05:00
Robert Hensing c7bd3a874f
Merge pull request #7863 from obsidiansystems/test-derivation-to-json
Move Derivation JSON printing logic to lib and test it
2023-02-20 23:21:50 +01:00
John Ericson 0258ac9c2a Make --read-only a separate mixin
It is independent of SourceExprCommand, which is about parsing
installables, except for the fact that parsing installables is one of
the many things influenced by read-only mode.
2023-02-20 10:39:18 -05:00
John Ericson 1bd03ad100 Split out CmdRepl and editorFor
The REPL itself and the `nix repl` CLI are conceptually different
things, and thus deserve to be in different files.
2023-02-20 09:45:29 -05:00
John Ericson 57a2e46ee0 Slight cleanup of InstallablesCommand::load 2023-02-20 09:09:11 -05:00
John Ericson fa4733fce5 Split out InstallableFlake and InstallableAttrPath 2023-02-20 09:09:11 -05:00
Valentin Gagarin dda83a59c1
Merge pull request #7158 from sternenseemann/foldl-strict-accumulation-value 2023-02-19 23:54:14 +01:00
John Ericson 7998686c00 Test toJSON of DerivationOutput and Derivation 2023-02-19 11:12:12 -05:00