Enter debugger more reliably in `let` expressions and function calls
(cherry picked from commit c4ed92fa6f836d3d8eb354a48c37a2f9eeecc3aa)
Change-Id: I16d0cad7e898feecd2399723b92ba8df67222fb4
Catch `Error`, not `BaseError` in `ValuePrinter`
BaseError includes Interrupt. We probably don't want the value printer to tell you Ctrl-C was pressed while it was printing.
(cherry picked from commit c291d2d8dda38aa88b004e2ed05b28653c07e342)
Change-Id: I70b105bfb2f52a8f345ae0281d12f022aa36b14e
`nix eval` forces values and prints derivations as attribute sets, so
commands that print derivations (e.g. `nix eval nixpkgs#bash`) will
infinitely loop and segfault.
Printing derivations as `.drv` paths makes `nix eval` complete as
expected. Further work is needed, but this is better than a segfault.
(cherry picked from commit 4910d74086a85876e093136a0e8ebc547b467af7)
Change-Id: I8e1cb39c05db812080759ec183ee7a131760e6ea
these symbols are used a *lot*, so it makes sense to cache them. this
mostly increases clarity of the code (however clear one may wish to call
the parser desugaring here), but it also provides a small performance
benefit.
(cherry picked from commit 09a1128d9e2ff0ae6176784938047350d6f8a782)
Change-Id: I73d9f66be4555168e048cb2d542277251580c2d1
there's no reason the parser itself should be doing semantic analysis
like bindVars. split this bit apart (retaining the previous name in
EvalState) and have the parser really do *only* parsing, decoupled from
EvalState.
(cherry picked from commit b596cc9e7960b9256bcd557334d81e9d555be5a2)
Change-Id: I481a7623afc783e9d28a6eb4627552cf8a780986
most EvalState and Expr members defined here could be elsewhere, where
they'd be easier to maintain (not being embedded in a file with arcane
syntax) and *somewhat* more faithfully placed according to the path of
the file they're defined in.
(cherry picked from commit e1aa585964c3d864ebff0030584f3349a539d615)
Change-Id: Ibc704567462bb40f37cda05d8fadd465519db5f5
most instances of this being used do not refer to the "current"
position, sometimes not even to one reasonably close by. it could also
be called `makePos` instead, but `at` seems clear in context.
(cherry picked from commit 835a6c7bcfd0b22acc16f31de5fc7bb650d52017)
Change-Id: I17cab8a6cc14cac5b64624431957bfcf04140809
ParserState better describes what this struct really is. the parser
really does modify its state (most notably position and symbol tables),
so calling it that rather than obliquely "data" (which implies being
input only) makes sense.
(cherry picked from commit 007605616477f4f0d8a0064c375b1d3cf6188ac5)
Change-Id: I92feaec796530e1d4d0f7d4fba924229591cea95
all of them need access to parser state in some way. make them members
to allow this without fussing so much.
(cherry picked from commit 1b09b80afac27c67157d4b315c237fa7bb9b8d08)
Change-Id: I3145c95666a5617b735eff7cb403c54c0fe86347
since nix doesn't use the bison `error` terminal anywhere any invocation
of yyerror will immediately cause a failure. since we're *already*
leaking tons of memory whatever little bit bison allocates internally
doesn't much matter any more, and we'll be replacing the parser soon anyway.
coincidentally this now also matches the error behavior of URIs when
they are disabled or ~/ paths in pure eval mode, duplicate attr
detection etc.
(cherry picked from commit e8d9de967fe47a7f9324b0022a2ef50df59f419d)
Change-Id: I560c50d11dceddc2d7cf9ed2c6c631a309ce574e
this is a proper subset of Formals anyway, so let's just use those and
avoid the extra allocations and moves.
(cherry picked from commit f07388bf985c2440413f398cf93d5f5840d1ec8c)
Change-Id: I4508c9c9c918cbaaed649dc753eb86f5cafc7ab6
Print the value in `error: cannot coerce` messages
(cherry picked from commit 5b7bfd2d6b89d7dd5f54c1ca6c8072358d31a84e)
===
test taken from 6e8d5983143ae576e3f4b1d2954a5267f2943a49; it was added
previously (and not backported because its pr was a mostly-revert), but
it's useful to have around.
Change-Id: Icbd14b55e3610ce7b774667bf14b82e6dc717982
libexpr: print value of what is attempted to be called as function
(cherry picked from commit 50e5d7b883042852538371237e32a66bb22f0485)
Change-Id: I7cb6290bd8f244e83bfce3b2eed2a4c8b4f16a83
Print the value in `value is X while a Y is expected` error
(cherry picked from commit 5f72a97092da6af28a7d2b2a50d74e9d34fae7e1)
Change-Id: Idb4bc903ae59a0f5b6fb3b1da4d47970fe0a6efe
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).
This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.
Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)
Please read the tests for example output.
Future work:
- It would be nice to provide this function as a builtin, perhaps
`builtins.toStringDebug` -- a printing function that never fails would
be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
command line, e.g. `--option to-string-max-attrs 1000`.
(cherry picked from commit 0fa08b451682fb3311fe58112ff05c4fe5bee3a4, )
===
Restore ambiguous value printer for `nix-instantiate`
The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.
(cherry picked from commit df84dd4d8dd3fd6381ac2ca3064432ab31a16b79)
Change-Id: I7cca6b4b53cd0642f2d49af657d5676a8554c9f8
Don't attempt to `git add` ignored files
(cherry picked from commit 359990dfdc713c80aabd7ea6f7e4528628fbe108)
===
also added a regression test that isn't upstream to be sure we're
actually fixing the bug.
Change-Id: I8267a3d0ece9909d8008b7435b90e7b3eee366f6
Change an `allowPath` call to take a store path again
(cherry picked from commit 7c9ed1b1a325fe64a186e9d454607eaa0a7e8951)
Change-Id: Ia5ec924315a1f2640a0438cfb4b1ee0689cd3558
Factor out `ServeProto::Serialiser<UnkeyedValidPathInfo>` and test
(cherry picked from commit 139982997eec493a0f74105c427953f6be77da6d)
Change-Id: I28e4ba5a681a90d81915a56e6dbaa5456d64f96d
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE
(cherry picked from commit 3c042f3b0b0a7ef9c47bf049f5410dbd4aac9e90)
Change-Id: I7e41838338ee1edf31fff6f9e354c3db2bba6c0e
Also be more consistent with quotes around attribute paths
(cherry picked from commit 9404ce36e4edd1df12892089bdab1ceb7d4d7a97)
Change-Id: Iaa80073b4a07a6ffef106a3c12ecd02b4f6f67aa
Restore `builtins.pathExists` behavior on broken symlinks
(cherry picked from commit d53c8901ef7f2033855dd99063522e3d56a19dab)
===
note that this variant differs markedly from the source commit because
we haven't endured quite as much lazy trees.
Change-Id: I0facf282f21fe0db4134be5c65a8368c1b3a06fc
absPath: Explicitly check if path is empty before accessing it
(cherry picked from commit 6ec08b85f607852eb6f976c1392c4917d0a53787)
Change-Id: Ieeb53fb65d0e334e6017ceb3a48b3b6ae1047843
Include phase reporting in log file for ssh-ng builds
(cherry picked from commit b1e7d7cad625095656fff05ac4aedeb12135110a)
Change-Id: I4076669b0ba160412f7c628ca9113f9abbc8c303
It is possible to exfiltrate a file descriptor out of the build sandbox
of FODs, and use it to modify the store path after it has been
registered. To avoid that issue, don't register the output of the build,
but a copy of it (that will be free of any leaked file descriptor).
Test that we can't leverage abstract unix domain sockets to leak file
descriptors out of the sandbox and modify the path after it has been
registered.
(cherry picked from commit 2dadfeb690e7f4b8f97298e29791d202fdba5ca6)
(tests cherry picked from commit c854ae5b3078ac5d99fa75fe148005044809e18c)
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Theophane Hufschmitt <theophane.hufschmitt@tweag.io>
Co-authored-by: Tom Bereknyei <tomberek@gmail.com>
Change-Id: I87cd58f1c0a4f7b7a610d354206b33301e47b1a4
Combine `AbstractPos`, `PosAdapter`, and `Pos`
(cherry picked from commit 113499d16fc87d53b73fb62fe6242154909756ed)
===
this is a bit cursed because originally it was based on InputAccessor
code that we don't have and moved/patched features we likewise don't
have (fetchToStore caching, all the individual accessors,
ContentAddressMethod). the commit is adjusted accordingly to
match (remove caching, ignore accessors, use FileIngestionMethod).
note that `state.rootPath . CanonPath == abs` and
computeStorePathForPath works relative to cwd, so the slight rewrite in
the moved fetchToStore is legal.
Change-Id: I05fd340c273f0bcc8ffabfebdc4a88b98083bce5
Increase stack size on macOS as well as Linux
(cherry picked from commit efb91d5979a625d5c50558aeabfd24e802ed9173,
4a2444b3f32a2f5d42c4d65302793b987d1ac667)
Change-Id: Ieb72283c61bb9e360683f531d6635697b293c313
Add position information to `while evaluating the attribute` errors in the debugger
(cherry picked from commit ffe67c86a8ef3695e5c8b9c9800c192ac633dded)
Change-Id: I177ea5ec60898abe09fb9d80d9602b2a32ff8f44
Fix "Failed tcsetattr(TCSADRAIN)" when `nix repl` is not a TTY
(cherry picked from commit 864fc85fc88ff092725ba99907611b2b8d2205fb)
Change-Id: I8198674b935fabd741a349cc74544e61c53ea7b3
`nix`: Fix `haveInternet` to check for proxy
(cherry picked from commit accae60e7710a18f6f2bd7d2f4cd836bcd76b684)
Change-Id: I996dafdcd266f4bc5806386c86b19040120842bf
Say how many channels were unpacked in nix-channel
(cherry picked from commit 9ae665b9e1dc64c507ab6002fc5d7824208f3777)
Change-Id: Ie0950cf32123b550c5b83981a020e513f72a9b7c
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.
For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.
There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.
I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.
Upstream-PR: https://github.com/NixOS/nix/pull/10126
Original-Change-Id: I513cceaac1371decb3d96231e6ef9181c910c218
Change-Id: I531a51f6348a746e8e41d88203b08f614898356c
Expose locals from `let` expressions to the debugger
(cherry picked from commit acef4f17a2daab4ccdf656bdf229792db2f779e4)
Change-Id: Ib3623254f67ac762f4e7230d625e9f87dff38a84
Print positions in `--debugger`, instead of pointers
(cherry picked from commit 49cf090cb2f51d6935756a6cf94d568cab063f81)
Change-Id: Ic27917b2aab3657c28d599898377bf0c14753f8a
Make `StoreConfig::getDefaultSystemFeatures` a static method
(cherry picked from commit 5a9513cdbae31ea5e6f6e7afa7b3c2e3a9a26474)
Change-Id: Ia9c0ae2b7de419bd60aea8bf905154b96c428276
Convert `Machine::speedFactor` from a non-neg int to a non-neg float
(cherry picked from commit 69d0ae27e376e7c7c4f237716b0149223b8a805a)
Change-Id: I2afb5cf9e4fe1384985c58353946135c3d102b42
Make `Machine::systemTypes` a set not vector
(cherry picked from commit f1b030415376e81c5804647c055d71eaba4aa725)
Change-Id: I6d4f5c0bfc226e9bd66c58c360cd99e3fac9a129
Fix crash when NAR is missing from binary cache
(cherry picked from commit 3b20cca9625a1701a10a883735e7315185629563)
Change-Id: I50ff18f4a6de69c323473b4a8e3e098d1f365145
Print a more helpful message if the daemon crashes
(cherry picked from commit 32706b14a7531c2c21b9f96da083a540a0031ec4)
Change-Id: Ief7c465bca7666e2b7e7c9d1dd0c01c5f9014146
Store: :buildPaths(): Fix display of store paths
(cherry picked from commit b5ed36e6633cac844fe4388dcc0cc8055a18ef9e)
Change-Id: Ic6008491088dc6febd4a1e44dc2dbb96c47661f4
Fix performance of builtins.substring for empty substrings
(cherry picked from commit b2deff1947c2fe57fdbf1a472eb9003eb407f8d3)
Change-Id: I4ddfc8d26a4781c9520fff9807849a073ee7bed8
nix shell: reflect command line order in PATH order
(cherry picked from commit b91c935c2faf08ced2c763dcd2a831f26d84fa86)
Change-Id: If16c120bb74857c2817366e74e5b0877eb997260
Improve error message for fixed-outputs with references.
(cherry picked from commit ff6de4a9ee6c3862db9ee5f09ff9c3f43ae7a088)
Change-Id: I733c49760b9a3f1b76a6bece3b250b8579cd6cac
withFramedSink(): Receive interrupts on the stderr thread
(cherry picked from commit 965cfe96886c988c3aa94bfc7fefdd37325f4536)
Change-Id: I8320a96957c01ec0e3450d1b3ae38a3baff78d49
Fix segfault on infinite recursion in some cases
(cherry picked from commit bf1b294bd81ca76c5ec9fe3ecd52196bf52a8300)
Change-Id: Id137541426ec8536567835953fccf986a3aebf16
Allow access to /dev/stderr in Darwin sandbox
(cherry picked from commit c6d7013583c568590aff285fb7414d1675a745f4)
Change-Id: I5657f6f4ee9dad8c978bad0d71f5cac51584e4f2
Fix building CA derivations with and eval store
(cherry picked from commit dfc0cee7024a082d90a4f68296f55a82dfd52126)
Change-Id: I28feb5a36d4fe75f0ed3e3e2db6eb56b67d0f371
Give `Store::queryDerivationOutputMap` and `evalStore` argument
(cherry picked from commit 8cddda4f892cb42be43e9bd87aa0111572617e78)
Change-Id: I394e7e11c3f2e0cd3dbe0f48d757c14c09835e44
libstore/daemon.cc: note trust model difference in readDerivation()s
(cherry picked from commit 5c917c32048ef185ea0eec352c3505485aa3212c)
Change-Id: I9945bc84e9529b005eafdc5c08b5bf1553335340
Give `Derivation::tryResolve` an `evalStore` argument
(cherry picked from commit 36ca6adc60511dc822870f2df43c0a578e481925)
Change-Id: If76b185a01ffa982e4c49cf333a9b5fbf9edebfe
Add option to libarchive so it behaves correctly
(cherry picked from commit c3827ff6348a4d5199eaddf8dbc2ca2e2ef46ec5)
Change-Id: Ib0f928851093f4c644bac071d1c8f8aeec803198
config: add included files into parsedContents before applying
(cherry picked from commit 82359eba6b692691ef08a71196ef25a61bc4d3d3)
Change-Id: Idde3177010fec7b8bafe6088c3c23d5caf491845
nix repl: Only hide the progress bar while waiting for user input
(cherry picked from commit 3bebaefcd0c5d650f7edcd39f397bb45c4382f41)
Change-Id: Ie7c0db46f7c2cf5f938e66bdd3c31f0b62bdb104
This allows templates such as `NLOHMANN_DEFINE_TYPE_*` templates and other generators with things like `std::vector<std::optional<T>>`.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
(cherry picked from commit 02bd821f2e71372d31bbe6700bd68086cc2ee70a)
Change-Id: I8b0ebcf2af4226610dadd565962f2d2327415a03
flakes: bare minimum fix the error message for untracked flake.nix
(cherry picked from commit 7f626dba332f320fafb9f9c749986ea523d20f42)
Change-Id: I470ac56a670a8c11e4164c6b059184a02344d491
Revert use of boost::container::small_vector in the evaluator
(cherry picked from commit 6832d18ac734f4b855f97c07b158491dd01cefcd)
Change-Id: I7f10af0c8b8a8beb4b1e36424120995f4ed82738
Unlock output paths when a derivation is already built
(cherry picked from commit 7ba4e073e8622ca86b52e03d68476e80250ab62f)
Change-Id: I9de077679290d5141a610ac43d99d3a43acff87c
* Fix boost::bad_format_string exception in builtins.addErrorContext
The message passed to addTrace was incorrectly being used as a format
string and this this would cause an exception when the string contained
a '%', which can be hit in places where arbitrary file paths are
interpolated.
* add test
(cherry picked from commit 61d6fe059e959455e156c1d57bb91155d363e983)
Change-Id: Idd671127a9c1ccc8b94e58e727632fcc064f3cbe
fix: gcc complains about if which doesn't guard the indented statement
(cherry picked from commit 8d663462938a333a4e81cce1005437f141cd11fa)
Change-Id: Ifa2e65502de4000935549dde82ab1b5867e2f0ed
Bindmount files instead of hardlinking or copying to chroot
(cherry picked from commit 622191c2b53882a1675fed5066ff8090b4f01827)
Change-Id: I278ec1baacdfa9044992b58fdec8f14d6d7d09ce
nix-shell: support single quotes in shebangs, fix whitespace parsing
(cherry picked from commit 3b99c6291377cbd22607896af9dfafa857d2f2dc)
Change-Id: I2a431b21c3467eefa1ef95d5a36d672f45b6937a
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info
(cherry picked from commit 8b68bbb77745fda0d14939b6c23d31cc89da41ce)
Change-Id: Ia3d3b9fbaf9f0ae62ab225020b7d14790e793655
Improve tests and docs prior to refactoring completions
(cherry picked from commit 5442d9b47298389918d1f38d20f768a80ffc2369)
Change-Id: Ief99ac2cd9c92981a9a522d15b9c3daf99182c9d
Factor out bits of the worker protocol to use elsewhere
(cherry picked from commit 4b1a97338f517f45e6169d3d8845c5caa5724e97)
Change-Id: If93afa0f8b1cf9b0e705b34fa71e6fd708752758
Test the rest of the worker protocol serializers
(cherry picked from commit 2f1c16dfa2378fd8616bff1b9b7cd0b4d42af69b)
Change-Id: Idfd72d32b21d14a260e02f65531d287cef7464d2
Unit test some worker protocol serializers
(cherry picked from commit c6faef61a6f31c71146aee5d88168e861df9a22a)
Change-Id: I99e36f5f17eb7642211a4e42a16b143424f164b4
do not show configuration override flags for each command
(cherry picked from commit f89b84919c1a5c796512c50311821e7779b3678b)
Change-Id: Ib98b739bd6c9a1e94f94a78a47d84d72e435e7c0
bef68e53b9 (backport of
31ebc6028b3682969d86a7b39ae87131c41cc604) accidentally broke the build
because of a change in the constructor of `SourcePath` between 2.18 and
master. Fix that.
Previously, IFDs would be built within the eval store, even though one
is typically using `--eval-store` precisely to *avoid* local builds.
Because the resulting Nix expression must be copied back to the eval
store in order to be imported, this requires the eval store to trust
the build store's signatures.
(cherry picked from commit c3942ef85ffbd83391410fbf012f1de366d2463c)
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:
- We have to be careful that wildcards don't end up with tests being
built as part of Nix proper, or test headers being installed as part
of Nix proper.
- Tests in libraries but not executables is not right:
- It means each executable runs the previous unit tests again, because
it needs the libraries.
- It doesn't work right on Windows, which doesn't want you to load a
DLL just for the side global variable . It could be made to work
with the dlopen equivalent, but that's gross!
This reorg solves these problems.
There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
(cherry picked from commit 91b6833686a6a6d9eac7f3f66393ec89ef1d3b57)
(cherry picked from commit a61e42adb528b3d40ce43e07c79368d779a8b624)
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests
- Concepts is harder to understand, the documentation makes a good
unit vs functional vs integration distinction, but when the
integration tests are just two subdirs within `tests/` this is not
clear.
- Source filtering in the `flake.nix` is more complex. We need to
filter out some of the dirs from `tests/`, rather than simply pick
the dirs we want and take all of them. This is a good sign the
structure of what we are trying to do is not matching the structure
of the files.
With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests
functional/
installer/
nixos/
```
(cherry picked from commit 68c81c737571794f7246db53fb4774e94fcf4b7e)
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).
(cherry picked from commit 46131567da96ffac298b9ec54016b37114b0dfd5)
I'm sure that we'll adjust the implementation over time, but this
at least discerns between an apple silicon bare metal machine and
a tart VM.
(cherry picked from commit 9277eb276bf0a942e88fcf499f6a6b9c262be853)
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.
Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.
Closes#9091.
Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
(cherry picked from commit 24bda0c7b381e1a017023c6f7cb9661fae8560bd)
This reverts commit 5e3986f59c. This
un-implements RFC 92 but fixes the critical bug #9052 which many people
are hitting. This is a decent stop-gap until a minimal reproduction of
that bug is found and a proper fix can be made.
Mostly fixed#9052, but I would like to leave that issue open until we
have a regression test, so I can then properly fix the bug (unbreaking
RFC 92) later.
(cherry picked from commit 8440afbed756254784d9fea3eaab06649dffd390)
The `-c` flag belongs to `sh` not `nix shell`. As it stands, the command errors with:
```
$ nix shell nixpkgs#gnumake --command sh --command "cd src && make"
sh: --command: invalid option
```
https://github.com/NixOS/nix/pull/8276 was good for readability, but it missed this since that PR used a find/replace script.
The Derivation parser and old ATerm unfortunately leaves few ways to get
nice errors when an old version of Nix encounters a new version of the
format. The most likely scenario for this to occur is with a new client
making a derivation that the old daemon it is communicating with cannot
understand.
The extensions we just created for dynamic derivation deps will add a
version field, solving the problem going forward, but there is still the
issue of what to do about old versions of Nix up to now.
The solution here is to carefully catch the bad error from the daemon
that is likely to indicate this problem, and add some extra context to
it.
There is another "Ugly backwards compatibility hack" in
`remote-store.cc` that also works by transforming an error.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We use the same nested map representation we used for goals, again in
order to save space. We might someday want to combine with `inputDrvs`,
by doing `V = bool` instead of `V = std::set<OutputName>`, but we are
not doing that yet for sake of a smaller diff.
The ATerm format for Derivations also needs to be extended, in addition
to the in-memory format. To accomodate this, we added a new basic
versioning scheme, so old versions of Nix will get nice errors. (And
going forward, if the ATerm format changes again the errors will be even
better.)
`parsedStrings`, an internal function used as part of parsing
derivations in A-Term format, used to consume the final `]` but expect
the initial `[` to already be consumed. This made for what looked like
unbalanced brackets at callsites, which was confusing. Now it consumes
both which is hopefully less confusing.
As part of testing, we also created a unit test for the A-Term format for
regular non-experimental derivations too.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Apply suggestions from code review
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
- Don't assert: Derivation ATerms are not necessarily produced by Nix,
and parsers should always throw graceful errors
- Improve error message from `static void except(..)`, shows both what
we expected and what we actually got.
The intention is that we backport it, and then hopefully a few people
might get slightly better errors if they try out new experimental drv
files (for RFC 92) with an old version of Nix.
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).
This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.
The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We're about to split up `DerivationGoal` a bit. At that point
`makeDerivationGoal` will mean something more specific than it does
today. (Perhaps a future rename will make this clearer.)
On the other hand, the more public `Worker::makeGoal` function will
continue to work exactly as before. So by moving some call sites to use
that instead, we preemptively avoid issues in the next step.
This function is now trivial enough that it doesn't need to exist.
`EvalState` can still be initialized with a custom search path, but we
don't have a need to mutate the search path after it has been
constructed, and I don't see why we would need to in the future.
Fixes#8229
Types converted:
- `NixStringContextElem`
- `OutputsSpec`
- `ExtendedOutputsSpec`
- `DerivationOutput`
- `DerivationType`
Existing ones mostly conforming the pattern cleaned up:
- `ContentAddressMethod`
- `ContentAddressWithReferences`
The `DerivationGoal::derivationType` field had a bogus initialization,
now caught, so I made it `std::optional`. I think #8829 can make it
non-optional again because it will ensure we always have the derivation
when we construct a `DerivationGoal`.
See that issue (#7479) for details on the general goal.
`git grep 'Raw::Raw'` indicates the two types I didn't yet convert
`DerivedPath` and `BuiltPath` (and their `Single` variants) . This is
because @roberth and I (can't find issue right now...) plan on reworking
them somewhat, so I didn't want to churn them more just yet.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
If you have a URL that needs to be percent-encoded, such as
`http://localhost:8181/test/+3d.tar.gz`, and try to lock that in a Nix
flake such as the following:
{
inputs.test = { url = "http://localhost:8181/test/+3d.tar.gz"; flake = false; };
outputs = { test, ... }: {
t = builtins.readFile test;
};
}
running `nix flake metadata` shows that the input URL has been
incorrectly double-encoded (despite the flake.lock being correctly
encoded only once):
[...snip...]
Inputs:
└───test: http://localhost:8181/test/%252B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D
(Notice the `%252B`? That's just `%2B` but percent-encoded again)
With this patch, the double-encoding is gone; running `nix flake
metadata` will show the proper URL:
[...snip...]
Inputs:
└───test: http://localhost:8181/test/%2B3d.tar.gz?narHash=sha256-EFUdrtf6Rn0LWIJufrmg8q99aT3jGfLvd1//zaJEufY%3D
---
As far as I can tell, this happens because Nix already percent-encodes
the URL and stores this as the value of `inputs.asdf.url`.
However, when Nix later tries to read this out of the eval state as a
string (via `getStrAttr`), it has to run it through `parseURL` again to
get the `ParsedURL` structure.
Now, this itself isn't a problem -- the true problem arises when using
`ParsedURL::to_string` later, which then _re-escapes the path_. It is
at this point that what would have been `%2B` (`+`) becomes `%252B`
(`%2B`).
Without the change build with `-D_GLIBCXX_ASSERTIONS` exposes testsuite
assertion:
$ gdb src/libexpr/tests/libnixexpr-tests
Reading symbols from src/libexpr/tests/libnixexpr-tests...
(gdb) break __glibcxx_assert_fail
(gdb) run
(gdb) bt
in std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@plt () from /mnt/archive/big/git/nix/src/libexpr/libnixexpr.so
in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=0x7fffffff56c0, __pos=4)
at /nix/store/r74fw2j8rx5idb0w8s1s6ynwwgs0qmh9-gcc-14.0.0/include/c++/14.0.0/string_view:258
in nix::SearchPath::Prefix::suffixIfPotentialMatch (this=0x7fffffff5780, path=...) at src/libexpr/search-path.cc:15
in nix::SearchPathElem_suffixIfPotentialMatch_partialPrefix_Test::TestBody (this=0x555555a17540) at src/libexpr/tests/search-path.cc:62
As string sizes are usigned types `(a - b) > 0` effectively means
`a != b`. While the intention should be `a > b`.
The change fixes test suite pass.
In the Nix language, given a drv path, we should be able to construct
another string referencing to one of its output. We can do this today
with `(import drvPath).output`, but this only works for derivations we
already have.
With dynamic derivations, however, that doesn't work well because the
`drvPath` isn't yet built: importing it like would need to trigger IFD,
when the whole point of this feature is to do "dynamic build graph"
without IFD!
Instead, what we want to do is create a placeholder value with the right
string context to refer to the output of the as-yet unbuilt derivation.
A new primop in the language, analogous to `builtins.placeholder` can be
used to create one. This will achieve all the right properties. The
placeholder machinery also will match out the `outPath` attribute for CA
derivations works.
In 60b7121d2c we added that type of
placeholder, and the derived path and string holder changes necessary to
support it. Then in the previous commit we cleaned up the code
(inspiration finally hit me!) to deduplicate the code and expose exactly
what we need. Now, we can wire up the primop trivally!
Part of RFC 92: dynamic derivations (tracking issue #6316)
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
`EvalState::mkSingleDerivedPathString` previously contained its own
inverse (printing, rather than parsing) in order to validate what was
parsed. Now that is pulled out into its own separate function:
`EvalState::coerceToSingleDerivedPath`.
In additional that pulled out logic is deduplicated with
`EvalState::mkOutputString` via `EvalState::mkOutputStringRaw`, which is
itself deduplicated (and generalized) with
`DownstreamPlaceholder::mkOutputStringRaw`.
All these changes make the unit tests simpler.
(We would ideally write more unit tests for `mkSingleDerivedPathString`
`coerceToSingleDerivedPath` directly, but we cannot yet do that because
the IO in reading the store path won't work when the dummy store cannot
hold anything. Someday we'll have a proper in-memory store which will
work for this.)
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
std::move(state->data) and data.empty() were called in a loop, and
could run with no other threads intervening. Accessing moved objects
is undefined behavior, and could cause a crash.
Virtual methods are no longer valid once the derived destructor has
run. This means the compiler is free to optimize them to be
non-virtual.
Found using clang-tidy
We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).
To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.
`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!
Important note: some JSON formats have changed.
We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This enables nix to correctly report what will be fetched in the case
that everything is a cache hit.
Note however that if an intermediate build of something which is not
cached could still cause products to end up being substituted if the
intermediate build results in a CA path which is in the cache.
Fixes#8615.
Signed-off-by: Peter Waller <p@pwaller.net>
When receiving a stream of NARs through the ssh-ng protocol, an already
existing path would cause the NAR archive to not be read in the stream,
resulting in trying to parse the NAR as a ValidPathInfo. This results in
the error message:
error: not an absolute path: 'nix-archive-1'
Fixes#6253
Usually this problem is avoided by running QueryValidPaths before
AddMultipleToStore, but can arise when two parallel nix processes gets
the same response from QueryValidPaths. This makes the problem more
prominent when running builds in parallel.
When loading a derivation from a JSON, malformed input would trigger
cryptic "assertion failed" errors. Simply replacing calls to `operator []`
with calls to `.at()` was not enough, as this would cause json.execptions
to be printed verbatim.
Display nice error messages instead and give some indication where the
error happened.
*Before:*
```
$ echo 4 | nix derivation add
error: [json.exception.type_error.305] cannot use operator[] with a string argument with number
$ nix derivation show nixpkgs#hello | nix derivation add
Assertion failed: (it != m_value.object->end()), function operator[], file /nix/store/8h9pxgq1776ns6qi5arx08ifgnhmgl22-nlohmann_json-3.11.2/include/nlohmann/json.hpp, line 2135.
$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: [json.exception.type_error.302] type must be string, but is object
$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error: [json.exception.type_error.302] type must be object, but is string
```
*After:*
```
$ echo 4 | nix derivation add
error: Expected JSON of derivation to be of type 'object', but it is of type 'number'
$ nix derivation show nixpkgs#hello | nix derivation add
error: Expected JSON object to contain key 'name' but it doesn't
$ nix derivation show nixpkgs#hello | jq '.[] | .name = 5' | nix derivation add
error: Expected JSON value to be of type 'string' but it is of type 'number'
$ nix derivation show nixpkgs#hello | jq '.[] | .outputs = { out: "/nix/store/8j3f8j-hello" }' | nix derivation add
error:
… while reading key 'outputs'
error: Expected JSON value to be of type 'object' but it is of type 'string'
```
I haven't checked when this was exactly introduced, but on Nix 2.16 I
realized that the additional lines inserted when using `--precise` are
completely separated from the tree:
nix why-depends /nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2 /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0 --precise --extra-experimental-features nix-command
/nix/store/ccgr4faaxys39s091qridxg1947lggh4-evcxr-0.14.2
→ /nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0
→ /nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0
└───bin/evcxr: …':'}.PATH=${PATH/':''/nix/store/lcf37pgp3rgww67v9x2990hbfwx96c1w-gcc-wrapper-12.2.0/bin'':'/':'}…
└───bin/cpp: …k disable=SC2193.[[ "/nix/store/b7hvml0m3qmqraz1022fwvyyg6fc1vdy-gcc-12.2.0/bin/cpp" = *++ ]] &&…
This is apparently because `std::cout` is buffered and flushed in the
end whereas the rest of the output isn't. The fix is rather simple, just
use `logger->cout` as it's already the case for the rest of the code.
This way we also don't need to insert additional newlines in the `hits`
map since that's something the logger takes care of.
Also added a small test to make sure that the layout of this is somehow
tested to reduce the risk of further regressions here.
Avoid duplicated code, and also avoid "on the fly" path construction
(which makes it harder to keep track of which paths we use).
The factored out code doesn't create the Nix state dir anymore, but this
is fine because other in nix-env and nix-channel does:
- nix-channel: Line 158 in this commit
- nix-env: Line 1407 in this commit
Special-casing the file name is rather ugly, so we shouldn't do
that. So now any {file,http,https} URL is handled by
TarballInputScheme, except for non-flake inputs (i.e. inputs that have
the attribute `flake = false`).
It was initially unclear to me which of these are temporary state for
the verify paths computation, and which of these are the results of that
computation to be used in the rest of the function. Now, it is clear,
and enforced.
We don't care about non-store-paths in there (things like `.links`, are,
in fact, allowed). So let's just skip them up front and be more strongly
typed.
Over the last year or so I've run into several use cases where I need to
parse and/or serialize URLs for use by `builtins.fetchTree` or
`builtins.getFlake`, largely in order to produce _lockfile-like_ files
for lang2nix frameworks or tools which use `nix` internally to drive
builds.
I've gone through the painstaking process of emulating
`nix::FlakeRef::fromAttrs` and `nix::parseFlakeRef` several times with
mixed success; but these are difficult to create and even harder to
maintain if I hope to stay aligned with changes to the real
parser/serializer.
I understand why adding new `builtins` isn't something we want to do
flagrantly. I'm recommending this addition simply because I keep
encountering use cases where I need to parse/serialize these URIs in
`nix` expressions, and I want a reliable solution.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Will need to do subclass-specific implementations in the next commit.
This isn't because there will be multiple variations of the daemon
protocol (whew!) but because different clients pick and choose different
parts to use.
This makes it more useful. In general, the derivation will be in one
store, and the realisation info is in another.
This also helps us avoid duplication. See how `resolveDerivedPath` is
now simpler because it uses `queryPartialDerivationOutputMap`. In #8369
we get more flavors of derived path, and need more code to resolve them
all, and this problem only gets worse.
The fact that we need a new method to deal with the multiple dispatch is
unfortunate, but this generally relates to the fact that `Store` is a
sub-par interface, too bulky/unwieldy and conflating separate concerns.
Solving that is out of scope of this PR.
This is part of the RFC 92 work. See tracking issue #6316
the original change broke many pre-existing anchor links.
also change formatting of the constants listing slightly:
- the type should not be part of the anchor
- add highlight to the "impure only" note
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.
After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.
Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
- Better types
- Own header / C++ file pair
- Test factored out methods
- Pass parsed thing around more than strings
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.
`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.
Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
'resolvedRef' was incorrect, since a resolved ref is one after
registry resolution, which may still be unlocked (e.g. 'nixpkgs' ->
'github:NixOS/nixpkgs').
If we call `adjustLoc`, the global variable `prev_yylloc` is shared
between threads and racy.
Currently, nix itself does not concurrently parsing files, but this is
helpful for libexpr users. (The parser is thread-safe except this.)
When explicitly requested by the caller, as suggested in the meeting
(https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324)
> @edolstra: { toPath } vs { fromPath } is too implicit
I've opted for the `inputAddressed = true` requirement, because it
we did not agree on renaming the path attributes.
> @roberth: more explicit
> @edolstra: except for the direction; not immediately clear in which direction the rewriting happens
This is in fact the most explicit syntax and a bit redundant, which is
good, because that redundancy lets us deliver an error message that
reminds expression authors that CA provides a better experience to
their users.
This is done in roughly the same way builtin functions are documented.
Also auto-link experimental features for primops, subsuming PR #8371.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
* nix flake check: improve error message if overlay is not a lambda
Suppose you have an overlay like this
{
inputs = { /* ... */ };
outputs = { flake-utils, ... }: flake-utils.lib.eachDefaultSystem
(system: {
overlays.default = final: prev: {
};
});
}
then `nix flake check` (correctly) fails because `overlays` are supposed
to have the structure `overlays.<name> = final: prev: exp`. However, the
error-message is a little bit counter-intuitive:
error: overlay does not take an argument named 'final'
While one might guess where the error actually comes from because the
trace above says `… while checking the overlay 'overlays.x86_64-linux'`
this is still pretty confusing because it complains about an argument
not being named `final` even though that's evidently the case.
With this change, the error-message actually makes it clear what's
wrong:
[ma27@carsten:~/Projects/nix/tmp]$ nix flake check --extra-experimental-features 'nix-command flakes' path:$(pwd)
error:
… while checking flake output 'overlays'
at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:
3| outputs = { ... }: {
4| overlays.x86_64-linux.snens = final: prev: {
| ^
5| kek = throw "snens";
… while checking the overlay 'overlays.x86_64-linux'
at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:
3| outputs = { ... }: {
4| overlays.x86_64-linux.snens = final: prev: {
| ^
5| kek = throw "snens";
error: overlay is not a lambda, but a set instead
I got very confused trying to keep all the `first` and `second` straight
reading the code, *especially* as there is also another `(boolean,
string)` pair type also being used.
Named fields is much better.
There are other cleanups that we can do (for example, the existing
TODO), but we can do them later. Doing them now would just make this
harder to review.
eldritch horrors
Rebecca Turner
jade
puck
Robert Hensing
Christina Rust
Yorick
Maximilian Bosch