Compare commits

...

192 commits

Author SHA1 Message Date
jade 71d9b64e3d wip
Change-Id: I3ddb26d2cd96670c111a4af773d19163f1f48382
2024-07-23 12:11:15 -07:00
jade eecc4ff1c0 releng: Remove workaround for skopeo feature we didn't know about
It turns out skopeo *does* support not saying the tag, but I couldn't
find it in the docs.

Asking the author in https://github.com/containers/skopeo/issues/2354
yielded that this can be requested as `@@unknown-digest@@`.

So now we have a perfectly cromulent docker upload chain, yay!!

Change-Id: I256f3cbeef4fe28b3d68d0dda57f02cdaee3996b
2024-07-23 12:11:15 -07:00
Pierre Bourdon 73c013a5df Merge "libexpr/gc-alloc: fix compilation with !HAVE_BOEHMGC" into main 2024-07-22 23:14:59 +00:00
Pierre Bourdon e76245f8e9
libexpr/gc-alloc: fix compilation with !HAVE_BOEHMGC
Fixes: 72ee25b402
Change-Id: Ib59386af1415a8ed4b53af24ec22a4ffa5e5877d
2024-07-23 00:50:09 +02:00
eldritch horrors 472ff1b833 libstore: keep Goal errors as unique_ptrs
Error is pretty large, and most goals do not fail. this alone more than
halves the size of Goal on x86_64-linux, from 720 bytes down to 344. in
derived classes the difference is not as dramatic, but even the largest
derived class (`LocalDerivationGoal`) loses almost 20% of its footprint

Change-Id: Ifda8f94c81b6566eeb3e52d55d9796ec40c7bce8
2024-07-22 19:01:40 +00:00
eldritch horrors 7bf1aff44a libstore: remove an always-defaulted argument
Change-Id: I3c7f17d5492a16bb54480fa1aa384b96fba72d61
2024-07-22 19:01:40 +00:00
eldritch horrors 58a91d70c9 libstore: use std::async instead of Goal threads
the goals are either already using std::async and merely forgot to
remove std::thread vestiges or they emulate async with threads and
promises. we can simply use async directly everywhere for clarity.

Change-Id: I3f05098310a25984f10fff1e68c573329002b500
2024-07-22 19:01:40 +00:00
eldritch horrors ad36fb43ad libstore: remove addToWeakGoals
under owner_less it's equivalent to insert(), only sometimes a little
bit faster because it does not construct a weak_ptr if the goal is in
the set already. this small difference in performance does not matter
here and c++23 will make insert transparent anyway, so we can drop it

Change-Id: I7cbd7d6e0daa95d67145ec58183162f6c4743b15
2024-07-22 19:01:40 +00:00
eldritch horrors d70e045f90 libstore: remove Goal::ecBusy
this should be an optional. "busy" is not an *exit* code!

Change-Id: Ic231cb27b022312b1a7a7b9602f32845b7a9c934
2024-07-22 19:01:40 +00:00
eldritch horrors 20f53346df libstore: remove unused Worker::waitForAnyGoal
Change-Id: Ia3ebd434b17052b6760ce74d8e20025a72148613
2024-07-22 19:01:40 +00:00
V. 85e3b9b871 De-vendor nixfmt
Change-Id: I1a051be495318a507d07f6d0a6b157616e26774c
2024-07-22 21:09:58 +04:00
eldritch horrors c74eb81356 enable -Werror=suggest-override
*accidentally* overriding a function is almost guaranteed to be an
error. overriding a function without labeling it as such is merely
bad style, but bad style that makes the code harder to understand.

Change-Id: Ic0594f3d1604ab6b3c1a75cb5facc246effe45f0
2024-07-22 16:26:55 +00:00
eldritch horrors 0463cf2aef libexpr: fix -Wunused-const-variable warning
Change-Id: Ib986ece0ab2eff83e7abd7f1f915cd8f761827ad
2024-07-22 16:26:55 +00:00
alois31 2d4aca2546
libutil/logging: fix build without precompiled header
Commit 0109368c3f missed to include a required
header, which is not noticed when the precompiled header is enabled because
it's included in that. Also include it in the file so that the build without
precompiled header works too.

Change-Id: Id7a7979684b64f937f7f8191612952d73c113015
2024-07-21 12:54:31 +02:00
alois31 94a8e5fe0d Merge "libstore/binary-cache-store: use correct buffer size for NAR decompression" into main 2024-07-21 10:42:33 +00:00
jade 4fa6961aa2 Merge "gc: refactor the gc server thread out into a class without changing it" into main 2024-07-21 10:36:10 +00:00
alois31 391088900e
libstore/binary-cache-store: use correct buffer size for NAR decompression
Due to a leftover from a previous version where the buffer was allocated on the
stack, the change introduced in commit 4ec87742a1
accidentally passes the size of a pointer as the size of the buffer to the
decompressor. Since the former is much smaller (usually 8 bytes instead of 64
kilobytes), this is safe, but leads to considerable overhead; most notably, due
to excessive progress reports, which happen for each chunk. Pass the proper
buffer size instead.

Change-Id: If4bf472d33e21587acb5235a2d99e3cb10914633
2024-07-21 11:28:23 +02:00
Winter Cute 1917e6c765 Merge "Fix namespace warning being emitted if sandbox is disabled" into main 2024-07-20 22:14:33 +00:00
Qyriad 72ee25b402 libexpr: add a strongly typed version of gcAllocBytes()
This commit adds a new helper template function to gc-alloc.hh (which is
probably where you want to look at first, O great reviewer [custom file
ordering in review diffs when]), which uses a type argument to determine
the size to allocate, rather than making the caller use sizeof().

Change-Id: Ib5d138d91a28bdda304a80db24ea9fb08669ad22
2024-07-20 20:20:01 +00:00
Qyriad e67dac1d74 libexpr: rename confusing makeImmutableString -> gcCopyStringIfNeeded
The purpose of this function has little to do with immutability. Value's
strings are never mutated, and the point of this function is to
singleton empty strings.

Change-Id: Ifd41dd952409d54e4d3de9ab59064e6928b0e480
2024-07-20 20:20:01 +00:00
Qyriad a3361557e3 libexpr: refactor gc-agnostic helpers into one place
Change-Id: Icc4b367e4f670d47256f62a3a002cd248a5c2d3b
2024-07-20 20:20:01 +00:00
eldritch horrors 0109368c3f libutil: make basic loggers thread-safe
SimpleLogger is not fully thread-safe, and all loggers that wrap it are
also not safe accordingly. this does not affect much, but in rare cases
it can cause interleaving of messages on stderr when used with the json
or raw log formats. the fix applied here is a bit of a hack, but fixing
this properly requires rearchitecting the logger infrastructure. nested
loggers are not the most natural abstraction here, and it is biting us.

Change-Id: Ifbf34fe1e85c60e73b59faee50e7411c7b5e7c12
2024-07-20 12:33:49 +00:00
eldritch horrors d8c09b5836 libutil: remove warnOnce macro
it's only used once, and even that one use is highly questionable. more
instances of warnOnce should be much more principled than this has been

Change-Id: I5856570c99cb44462e700d753d0c706a5db03c4b
2024-07-20 12:33:49 +00:00
Winter Cute 3da41fdb82 Fix namespace warning being emitted if sandbox is disabled
If useChroot = false, and user namespaces aren't available for some
reason (e.g. within a Docker container), this fixes a pointless warning
being emitted, as we would never attempt to use them even if they were
available.

Change-Id: Ibcee91c088edd2cd19e70218d5a5802bff8f537b
2024-07-19 19:14:54 -04:00
jade 77ff799cc8 gc: refactor the gc server thread out into a class without changing it
This removes a *whole load* of variables from scope and enforces thread
boundaries with the type system.

There is not much change of significance in here, so the things to watch
out for while reviewing it are primarily that the destructor ordering
may have changed inadvertently, I think.

Change-Id: I3cd87e6d5a08dfcf368637407251db22a8906316
2024-07-19 20:55:55 +00:00
jade 22252825c4 Merge changes Id8b3d289,Ib75ab5b8,I3792eeb3 into main
* changes:
  Fixup a bunch of references to nixos.org manuals
  Add release notes for removing overflow from Nix language
  expr: fix a compiler warning about different signs in comparison
2024-07-19 18:52:46 +00:00
alois31 aba5f19680 Merge changes I829581a3,I0016970d,I5dac8e77,Ib7560fe5 into main
* changes:
  doc/release-notes: add for pretty printing improvements
  libexpr/print: do not show elided nested items when there are none
  libexpr/print: never show empty attrsets or derivations as «repeated»
  libexpr/print: pretty-print idempotently
2024-07-19 06:40:13 +00:00
jade 26e56780ca Fixup a bunch of references to nixos.org manuals
(plus one reference to CppNix github)

Change-Id: Id8b3d2897f3b54e286861805cfd421adc4d5de47
2024-07-18 19:27:33 +00:00
jade 10cc3b288d Add release notes for removing overflow from Nix language
Change-Id: Ib75ab5b8b4d879035d7ee7678f9cd0c491a39c0a
2024-07-18 19:27:33 +00:00
jade 50a63f8435 expr: fix a compiler warning about different signs in comparison
We know that variable is >=0, so we can just cast it to unsigned.

Change-Id: I3792eeb3ca43e6a507cc44c1a70584d42b2acd7b
2024-07-18 19:27:33 +00:00
jade 5ee1e6ea98 Merge changes Ib20e9aa0,I178a038b,I29c7de04 into main
* changes:
  docs: document the actual comparison rules instead of lies
  daemon: remove workaround for macOS kernel bug that seems fixed
  daemon: fix a crash bug "FATAL: exception not rethrown"
2024-07-18 17:40:32 +00:00
alois31 768d1f29a2
doc/release-notes: add for pretty printing improvements
Change-Id: I829581a3f5b8b742e6c866dcdbbc635f91afceb5
2024-07-18 19:08:20 +02:00
alois31 40c39aa5d2
libexpr/print: do not show elided nested items when there are none
When the configured maximum depth has been reached, attribute sets and lists
are printed with ellipsis to indicate the elision of nested items. Previously,
this happened even in case the structure being printed is empty, so that such
items do not in fact exist. This is confusing, so stop doing it.

Change-Id: I0016970dad3e42625e085dc896e6f476b21226c9
2024-07-18 18:41:34 +02:00
alois31 b5da823138
libexpr/print: never show empty attrsets or derivations as «repeated»
The repeated value detection logic exists so that the occurrence of large
common substructures does not fill up the screen or the computer's memory.
However, empty attribute sets and derivations (when their detection is enabled)
are always cheap to print, and in practice I have observed them to make up a
significant majority of the cases where I was annoyed by the repeated value
detection kicking in. Furthermore, `nix-instantiate --eval` already disables
this logic for empty attribute sets, and empty lists are already exempted
everywhere. For these reasons, always print empty attribute sets and
derivations as what they are.

Change-Id: I5dac8e7739f9d726b76fd0521ec46f38af94463f
2024-07-18 18:41:34 +02:00
alois31 81a0624d76
libexpr/print: pretty-print idempotently
When pretty-printing is enabled, previously an unforced thunk would trigger
indentation, even when it subsequently does not evaluate to a nested structure.
The resulting output looked inconsistent, and furthermore pretty-printing was
not idempotent (since pretty-printing the same value again, which is now fully
evaluated, will not trigger indentation).
When strict evaluation is enabled, force the item before inspecting its type,
so that it is properly known whether it contains a nested structure.
Furthermore, there is no need to cause indentation for unforced thunks, since
the very next operation will be printing them as `«thunk»`.

This is mostly a port of https://github.com/NixOS/nix/pull/11100 , but we only
force the item when it's going to be forced anyway due to strict
pretty-printing, and a new test was written since the REPL testing framework in
Lix is different.

Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
Change-Id: Ib7560fe531d09e05ca6b2037a523fe21a26d9d58
2024-07-18 18:41:28 +02:00
alois31 7b1abf8107 Merge "doc/manual: clarify documentation related to the $$ parser bug" into main 2024-07-18 15:01:20 +00:00
alois31 72db9cd67b doc/release-notes: link the upcoming release notes again
The insertion marker comment broke the list into two parts, the first
containing only the link to the upcoming release notes and the second the
past releases. This confused the generator, leading to the first part being
discarded. Indent the marker comment so that it's syntactically part of the
preceding item, and in particular doesn't split the list any more.

Change-Id: I357c51bb03e4e0d79a76d30158615fd9eda95ea8
2024-07-17 22:12:41 +00:00
raito 67f62bcdb4 doc/release-notes: add date for major release
Change-Id: I93aab93c069bb3989c3f8d17e0862899e6f76865
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-07-17 22:12:41 +00:00
alois31 beb231784e doc/manual: clarify documentation related to the $$ parser bug
Due to a mistake in the grammar, a dollar character implicitly escapes a second
dollar character that immediately follows, so that it cannot start an
interpolation. Unfortunately, this behaviour has since come to be relied upon,
so it cannot be fixed. Furthermore, the documentation on regular strings did
not mention this behaviour at all, while in the case of indented strings it was
rather implicit.
Mention it explicitly in both cases, and describe how an interpolation can
follow a dollar character (namely, by escaping that). Since we have to touch
that section anyway, state that any character (other than n, r, and t; but
notably including `$` even if not succeeded by `{`) can be escaped using a
backslash in regular strings.

Change-Id: I7e5d68a9a4130eec98ce8218b485168f4b31a677
2024-07-17 22:01:48 +00:00
Max “Goldstein” Siling 68567206f2 Merge "tests/functional/repl.sh: actually fail test on wrong stdout" into main 2024-07-17 21:50:59 +00:00
Max “Goldstein” Siling 3a36c8bb90 tests/functional/repl.sh: actually fail test on wrong stdout
Previous test implementation assumed that grep supports newlines
in patterns. It doesn't, so tests spuriously passed, even though
some tests outputs were broken.

This patches output (and expected output) before grepping,
so there're no newlines in pattern.

Change-Id: Ie6561f9f2e18b83d976f162269d20136e2595141
2024-07-17 21:48:13 +00:00
eldritch horrors ef0de7c79f remove boost coroutine references
we no longer need these since sinkToSource and sourceToSink are gone.

Change-Id: Ibbf440e2cf71bf3e9f3b833af2d78a21fb1b3193
2024-07-16 01:50:16 +00:00
eldritch horrors dfedbc154f remove sourceToSink, sinkToSource, and boehm patch
Change-Id: I1379841299713175d0225b82a67f50660f9eb5e2
2024-07-16 01:50:16 +00:00
eldritch horrors d094dd0396 libstore: remove remaining sinkToSource uses
Change-Id: Id1ee0d2ad4a3774f4bbb960d76f0f76ac4f3eff9
2024-07-16 01:50:16 +00:00
eldritch horrors 6b4d46e9e0 libstore: remove WriteConn::sink fields
we no longer need these since we're no longer using sinks to serialize things.

Change-Id: Iffb1a3eab33c83f611c88fa4e8beaa8d5ffa079b
2024-07-16 00:57:42 +00:00
eldritch horrors a5d1f69841 libstore: generatorize protocol serializers
this is cursed. deeply and profoundly cursed. under NO CIRCUMSTANCES
must protocol serializer helpers be applied to temporaries! doing so
will inevitably cause dangling references and cause the entire thing
to crash. we need to do this even so to get rid of boost coroutines,
and likewise to encapsulate the serializers we suffer today at least
a little bit to allow a gradual migration to an actual IPC protocol.

(this isn't a problem that's unique to generators. c++ coroutines in
general cannot safely take references to arbitrary temporaries since
c++ does not have a lifetime system that can make this safe. -sigh-)

Change-Id: I2921ba451e04d86798752d140885d3c5cc08e146
2024-07-16 00:57:42 +00:00
eldritch horrors 5271424d14 libstore: remove a sinkToSouce from old daemon protocol
this doesn't have a test because this code path is only reached by
clients that predate 2.4, and we really should not be caring about
those any more right now. even the test suite doesn't, and the few
tests that might care are disabled because they will not even work

Change-Id: Id9eb190065138fedb2c7d90c328ff9eb9d97385b
2024-07-16 00:57:42 +00:00
eldritch horrors 4ec87742a1 libstore: rewrite the nar parser as a contents generator
this is not completely necessary at this point because the parser right
now already returns a generator to pass through all input data it read,
but the nar parser *was* very lax and would accept nars that weren't in
canonical form (defined as the form dumpPath would return). nar hashing
depends on these things, and as such rewriting the parser now allows us
to reject non-canonical nars that extract to the same store contents as
their canonical counterpart but have different nar hashes despite that.

Change-Id: Iccd319e3bd5912d8297014c84c495edc59019bb7
2024-07-16 00:57:42 +00:00
Qyriad c052716edd Merge changes I8d87c0e9,I25937702 into main
* changes:
  nix3-upgrade-nix: always use the /new/ nix-env to perform the installation
  libutil: implement a realPath() utility
2024-07-15 23:18:03 +00:00
eldritch horrors 3447dbfb2c libstore: rewrite narFromPath as generator
Change-Id: Ifa783c2c65c06ddd1d0212016d5bfd07666ea91c
2024-07-15 21:50:25 +00:00
Lunaphied 5e16b10cb1 Merge "use clangStdenv for the default devShell, so we get clangd by default" into main 2024-07-15 21:47:23 +00:00
Qyriad ae7eab49b9 nix3-upgrade-nix: always use the /new/ nix-env to perform the installation
Fixes #411.

Change-Id: I8d87c0e9295deea26ff33234e15ee33cc68ab303
2024-07-15 15:26:53 -06:00
Qyriad d9c51ec4e5 libutil: implement a realPath() utility
Just a wrapper around POSIX realpath().

Change-Id: I2593770285dbae573eace490efce5b272b00b001
2024-07-15 15:26:53 -06:00
nan-git 1eb5d22132 Merge "libexpr/eval.cc: remove unnecessary C string conversion" into main 2024-07-14 16:53:54 +00:00
nan-git 505640baec libexpr/eval.cc: remove unnecessary C string conversion
Change-Id: I5b7c21df84ff8ff64cf6a1e261fc3729a06bd4f6
2024-07-14 02:13:58 +00:00
Artemis Tosini 201e8b6994
nix-support/binary-tarball.nix: Pass through root paths
Passing through root paths allows external programs to see
which nix and cacert are in a binary tarball,
e.g. to recreate it from substituters

Change-Id: I27431134df53bbc6623484f8a0822004b51f7c87
2024-07-13 23:22:43 +00:00
jade 702f02c31f docs: document the actual comparison rules instead of lies
Although the comparison rules are ugly and we do not like various parts
of them, we must not hide them away for only catgirls to know about, so
the documentation should actually say how they work.

Change-Id: Ib20e9aa0e7b6486ade4f401035aafd85fbb08c91
2024-07-13 01:17:14 +02:00
jade 69e2ee5b25 daemon: remove workaround for macOS kernel bug that seems fixed
This was filed as https://github.com/nixos/nix/issues/7584, but as far
as I can tell, the previous solution of POLLHUP works just fine on macOS
14. I've also tested on an ancient machine with macOS 10.15.7, which
also has POLLHUP work correctly.

It's possible this might regress some older versions of macOS that have
a kernel bug, but I went looking through the history on the sources and
didn't find anything that looked terribly convincingly like a bug fix
between 2020 and today. If such a broken version exists, it seems pretty
reasonable to suggest simply updating the OS.

Change-Id: I178a038baa000f927ea2cbc4587d69d8ab786843
2024-07-13 01:17:14 +02:00
jade a8f443d960 docs: update to define integer overflow
Change-Id: Ie8a1b31035f2d27a220e5df2e9e178ec3b39ee68
2024-07-13 00:59:33 +02:00
jade b3fb8d9822 daemon: fix a crash bug "FATAL: exception not rethrown"
This is caused by pthread_cancel effectively throwing a
not-specifically-identifiable C++ exception into the targeted thread,
which, if it is not rethrown, terminates the process entirely.

This is rather "impolite" behaviour, we would say. But thread
cancellation is *always* busted, and we should simply not use it where
unnecessary. It's particularly unnecessary when what we *actually* need
it for is, err, interrupting a poll(2).

That can in turn be achieved by simply listening to more stuff in the
poll, namely, a pipe, which we send a character to when needing to
stop the thread.

While looking at this code, we also investigated whether any of the
poll() madness is required, or was even *ever* required. Curiously we
found in the XNU kernel source code that the thing about needing to
listen to POLLHUP is probably *correct*, but switching it to POLLRDNORM
should not have made any difference at all. We've left a FIXME to look
into that further because what's written here is super janky.

94d3b45284/bsd/kern/sys_generic.c (L1751-L1758)

This is the crash on some Hydra machines:

Thread 1 (Thread 0x7f56b77776c0 (LWP 955542) (Exiting)):
0  0x00007f56b8e9b7dc in __pthread_kill_implementation () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
1  0x00007f56b8e49516 in raise () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
2  0x00007f56b8e31935 in abort () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
3  0x00007f56b8e327f3 in __libc_message_impl.cold () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
4  0x00007f56b8e8e8e9 in __libc_fatal () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
5  0x00007f56b8ea23c4 in unwind_cleanup () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
6  0x00007f56b9d2a1b8 in nix::triggerInterrupt() [clone .cold] () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixutil.so
7  0x00007f56b990ac9d in std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<nix::MonitorFdHup::MonitorFdHup(int)::{lambda()#1}> > >::_M_run() () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixstore.so
8  0x00007f56b90e86d3 in execute_native_thread_routine () from /nix/store/c6r62m84hywf4i6qq1h28f13zv38yqyp-gcc-13.3.0-lib/lib/libstdc++.so.6
9  0x00007f56b8e99a42 in start_thread () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
10 0x00007f56b8f1905c in clone3 () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6

As for testing, we've started a daemon with this change and verified it
deals with HUPs correctly on x86_64-linux, but I don't think we can
easily test the destructor behaviour without whatever Hydra was
doing that broke.

Change-Id: I29c7de0425674494b6e43c075810126c3ff77363
2024-07-13 00:59:33 +02:00
jade 917c9bdee7 language: cleanly ban integer overflows
This also bans various sneaking of negative numbers from the language
into unsuspecting builtins as was exposed while auditing the
consequences of changing the Nix language integer type to a newtype.

It's unlikely that this change comprehensively ensures correctness when
passing integers out of the Nix language and we should probably add a
checked-narrowing function or something similar, but that's out of scope
for the immediate change.

During the development of this I found a few fun facts about the
language:
- You could overflow integers by converting from unsigned JSON values.
- You could overflow unsigned integers by converting negative numbers
  into them when going into Nix config, into fetchTree, and into flake
  inputs.

  The flake inputs and Nix config cannot actually be tested properly
  since they both ban thunks, however, we put in checks anyway because
  it's possible these could somehow be used to do such shenanigans some
  other way.

Note that Lix has banned Nix language integer overflows since the very
first public beta, but threw a SIGILL about them because we run with
-fsanitize=signed-overflow -fsanitize-undefined-trap-on-error in
production builds. Since the Nix language uses signed integers, overflow
was simply undefined behaviour, and since we defined that to trap, it
did.

Trapping on it was a bad UX, but we didn't even entirely notice
that we had done this at all until it was reported as a bug a couple of
months later (which is, to be fair, that flag working as intended), and
it's got enough production time that, aside from code that is IMHO buggy
(and which is, in any case, not in nixpkgs) such as
lix-project/lix#445, we don't think
anyone doing anything reasonable actually depends on wrapping overflow.

Even for weird use cases such as doing funny bit crimes, it doesn't make
sense IMO to have wrapping behaviour, since two's complement arithmetic
overflow behaviour is so *aggressively* not what you want for *any* kind
of mathematics/algorithms. The Nix language exists for package
management, a domain where bit crimes are already only dubiously in
scope to begin with, and it makes a lot more sense for that domain for
the integers to never lose precision, either by throwing errors if they
would, or by being arbitrary-precision.

This change will be ported to CppNix as well, to maintain language
consistency.

Fixes: lix-project/lix#423

Change-Id: I51f253840c4af2ea5422b8a420aa5fafbf8fae75
2024-07-13 00:59:33 +02:00
jade f9641b9efd libutil: add checked arithmetic tools
This is in preparation for adding checked arithmetic to the evaluator.

Change-Id: I6e115ce8f5411feda1706624977a4dcd5efd4d13
2024-07-13 00:56:37 +02:00
Lunaphied 0339b2fbd2 use clangStdenv for the default devShell, so we get clangd by default
The default-stdenv-devShell can always be used with `.#native-stdenvPackages`.

Change-Id: I9b3e72210ba5219b6b65c71a2818110769623904
2024-07-12 20:52:33 +00:00
jade dde51af97d Use std::strong_ordering for version comparison
The actual motive here is the avoidance of integer overflow if we were
to make these use checked NixInts and retain the subtraction.

However, the actual *intent* of this code is a three-way comparison,
which can be done with operator<=>, so we should just do *that* instead.

Change-Id: I7f9a7da1f3176424b528af6d1b4f1591e4ab26bf
2024-07-12 16:48:28 +02:00
eldritch horrors 4b109ec1a8 libstore: remove upcast_goal
upcast_goal was only ever needed to break circular includes, but the
same solution that gave us upcast_goal also lets us fully remove it:
just upcast goals without a wrapper function, but only in .cc files.

Change-Id: I9c71654b2535121459ba7dcfd6c5da5606904032
2024-07-11 21:31:52 +00:00
eldritch horrors a5d431a911 libstore: turn copyNAR into a generator
Change-Id: Id452f6a03faa1037ff13af0f63e32883966ff40d
2024-07-11 20:37:27 +00:00
eldritch horrors 03db4efab9 libstore: turn the NAR parser into a passthrough generator
this will let us turn copyNAR into a generator as well, which in turn is
necessary to turn the users of copyNAR into generators without resorting
to sinkToSource coroutines. currently this uses the SerializingTransform
in all cases, even for copyNAR where it is not necessary. should this be
a performance problem we can easily swap out the transform for one which
does not produce any bytes of its own, but that should not be necessary.

Change-Id: I7e685879318fcbb78d8b88abfddd7752360eb0ce
2024-07-11 20:37:27 +00:00
eldritch horrors 31478c810a libutil: remove makeDecompressionSink
the sole remaining user of this function can use makeDecompressionSource
instead, while making the sinkToSource in the caller unnecessary as well

Change-Id: I4258227b5dbbb735a75b477d8a57007bfca305e9
2024-07-11 11:39:18 +00:00
eldritch horrors 5587dbdcf0 libstore: make BinaryCacheStore::getFile return a source
this lets us remove the last true remaining uses of
makeDecompressionSink.

Change-Id: I146ca2bbe1a9ae9a367117a7b8a304b23a63e5e2
2024-07-11 11:39:18 +00:00
eldritch horrors df8851f286 libutil: rewrite RewritingSink as source
the rewriting sink was just broken. when given a rewrite set that
contained a key that is also a proper infix of another key it was
possible to produce an incorrectly rewritten result if the writer
used the wrong block size. fixing this duplicates rewriteStrings,
to avoid this we'll rewrite rewriteStrings to use RewritingSource
in a new mode that'll allow rewrites we had previously forbidden.

Change-Id: I57fa0a9a994e654e11d07172b8e31d15f0b7e8c0
2024-07-11 11:39:18 +00:00
Lunaphied 014410cbf0 Merge "lix-doc: update dependencies and refactor" into main 2024-07-10 17:47:27 +00:00
eldritch horrors f0c751d4d6 Merge changes I81552018,Ieb65c133 into main
* changes:
  releng: add releaseTests flake output, test script
  add aarch64-linux as a cross-build target
2024-07-10 17:45:17 +00:00
piegames 3dced96741 devShell: Fix errors with environment variables
Previous code assumed that $MANPATH always exists

Change-Id: I5a4d012045ba6ff9086373b3f46a75d82285d393
2024-07-10 14:22:47 +02:00
alois31 fa92f41a18 Merge "libmain: clear display attributes in the multiline progress bar" into main 2024-07-10 03:43:07 +00:00
Lunaphied 41963df4a5 lix-doc: update dependencies and refactor
This updates the version of rnix used and refactors the code generally
to be more precise and capable in it's identification of both lambdas
and determining which documentation comments are attached.

Change-Id: Ib0dddabd71f772c95077f9d7654023b37a7a1fd2
2024-07-10 02:25:48 +00:00
eldritch horrors f5aa5b6815 releng: add releaseTests flake output, test script
this is supposed to be a set of outputs we want to always succeed for
releases. sadly we can't add nixos installer tests using lix to these
because the nixos test framework does not allow overriding nix in the
installer test suites due to unfortunate oversights in the framework.

Change-Id: I815520181ccca70a47205d38ba27e73529347f04
2024-07-09 22:50:51 +02:00
eldritch horrors 4d8c66ec6f add aarch64-linux as a cross-build target
we want to be sure we can cross-build to aarch64 for releases, add a
target to our crossSystems list to make those cheacks easier to run.

Change-Id: Ieb65c1333a5232641ace0ba4d122fc7d528ebc04
2024-07-09 22:49:10 +02:00
Quantum Jump 6e0ca02425 Fix dry-run flag for nix-collect-garbage
`nix-collect-garbage --dry-run` previously elided the entire garbage
collection check, meaning that it would just exit the script without
printing anything.

This change makes the dry run flag instead set the GC action to
`gcReturnDead` rather than `gcDeleteDead`, and then continue with the
script. So if you set `--dry-run`, it will print the paths it *would*
have garbage collected, but not actually delete them.

I filed a bug for this: lix-project/lix#432 but then realised I could give fixing it a go myself.

Change-Id: I062dbf1a80bbab192b5fd0b3a453a0b555ad16f2
2024-07-09 13:55:05 +00:00
Qyriad accfd8aa9d libexpr: stop lying about DrvInfo's constness
DrvInfo's query methods all use mutable fields to cache, but like.
that's basically the entire interface for DrvInfo. Not only that, but
these formerly-const-marked functions can even throw due to eval errors!
Changing this only required removing some `const` markers in nix-env,
and changing a single inline `queryInstalled()` call to be an lvalue
instead.

Change-Id: I796807118f3b35b0e93668b5e28210d9e521b2ae
2024-07-08 17:52:02 +00:00
alois31 f5ff70d7f3
libmain: clear display attributes in the multiline progress bar
Activities can set display attributes in their log output using the "Select
Graphics Rendition" functionality. To prevent interfering with subsequent text
displayed, these should be reset after writing the log line. The multiline
progress bar neglected to do this, resulting for example in a colorised
"building …" header in the next line. Reset the attributes properly, like the
standard progress bar already does.

Change-Id: I1dc69f4a1d747a76b83e8721a72d9bb0e5554488
2024-07-08 19:08:23 +02:00
Artemis Tosini d461cc1d7b
libstore: make LocalDerivationGoal::needsHashRewrite virtual
This rather simple function existed just to check some flags,
but the response varies by platform. This is a perfect case for
our subclasses.

Change-Id: Ieb1732a8d024019236e0d0028ad843a24ec3dc59
2024-07-07 18:06:08 +00:00
eldritch horrors 55a32f24d3 libutil: remove RewritingSink match/size tracking
size tracking can be done with a LengthSink and a tee. match tracking
was defeated by never having done any match tracking, all users would
see the same (empty) set of matches at all times. match tracking with
bytes offsets alone would not be sufficient in the general case, only
because computeHashModulo uses a single rewrite could it have worked.

Change-Id: Idb214b5222e0ea24f450f5505712a342b63d7570
2024-07-06 12:36:37 +02:00
eldritch horrors 5af76dee37 libutil: turn HashModuloSink into a free function
Change-Id: I5878007502fa68c2816a0f4c61f7d0e60bdde702
2024-07-06 12:36:37 +02:00
eldritch horrors 4162a66cee libutil: return sources from runProgram2
this much more closely mimics what is actually happening: we're reading
data from somewhere else, actively, rather than passively waiting. with
the data flow matching the underlying system interactions better we can
remove a few sinkToSource calls that merely exists to undo the mismatch
caused by not treating subprocess output as a data source to begin with

Change-Id: If4abfc2f8398fb5e88c9b91a8bdefd5504bb2d11
2024-07-06 12:36:36 +02:00
eldritch horrors b6a08a2fed libutil: return a program handle from runProgram2
this will let us also return a source for the program output later,
which will in turn make sinkToSource unnecessary for program output
processing. this may also reopen a path for provigin program input,
but that still needs a proper async io framework to avoid problems.

Change-Id: Iaf93f47db99c38cfaf134bd60ed6a804d7ddf688
2024-07-05 22:28:16 +00:00
eldritch horrors f4f6d1d8e2 libutil: convert readFileSource to a generator
Change-Id: I5f92b15fd367d46eb047d74ab6e317b4f51a46d3
2024-07-05 22:28:16 +00:00
eldritch horrors 06220a71c1 libstore: convert dumpPath to a generator
Change-Id: Ic4cf5562504aa29130304469936f958c0426e5ef
2024-07-05 22:28:16 +00:00
eldritch horrors b51ea465de libutil: allow construction of sources from generators
Change-Id: I78ff8d0720f06bce731e26d5e1c53b1382bbd589
2024-07-05 22:28:16 +00:00
Qyriad b9f91ec3c5 mildly cleanup libexpr/eval.hh
Change-Id: I40d01a8f8b7fb101279c6f88ebdf1f0969d9d7f0
2024-07-04 17:43:03 -06:00
Qyriad 4c7165be86 distinguish between throws & errors during throw
Turns errors like this:

let
  throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"

error:
       … from call site
         at «string»:3:4:
            2|   throwMsg = a: throw (a + " invalid bar");
            3| in throwMsg "bullshit"
             |    ^

       … while calling 'throwMsg'
         at «string»:2:14:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |              ^
            3| in throwMsg "bullshit"

       … while calling the 'throw' builtin
         at «string»:2:17:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |                 ^
            3| in throwMsg "bullshit"

       error: bullshit invalid bar

into errors like this:

let
  throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"

error:
       … from call site
         at «string»:3:4:
            2|   throwMsg = a: throw (a + " invalid bar");
            3| in throwMsg "bullshit"
             |    ^

       … while calling 'throwMsg'
         at «string»:2:14:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |              ^
            3| in throwMsg "bullshit"

       … caused by explicit throw
         at «string»:2:17:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |                 ^
            3| in throwMsg "bullshit"

       error: bullshit invalid bar

Change-Id: I593688928ece20f97999d1bf03b2b46d9ac338cb
2024-07-04 17:43:03 -06:00
Qyriad 14bf54bd39 trace which part of foo.bar.baz errors
Turns errors like:

let
  somepkg.src = throw "invalid foobar";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |   ^
            3| in somepkg.src.meta

       … while calling the 'throw' builtin
         at «string»:2:17:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |                 ^
            3| in somepkg.src.meta

       error: invalid foobar

into errors like:

let
  somepkg.src = throw "invalid foobar";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |   ^
            3| in somepkg.src.meta

       … while evaluating 'somepkg.src' to select 'meta' on it
         at «string»:3:4:
            2|   somepkg.src = throw "invalid foobar";
            3| in somepkg.src.meta
             |    ^

       … while calling the 'throw' builtin
         at «string»:2:17:
            1| let
            2|   somepkg.src = throw "invalid foobar";
             |                 ^
            3| in somepkg.src.meta

       error: invalid foobar

And for type errors, from:

let
  somepkg.src = "I'm not an attrset";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = "I'm not an attrset";
             |   ^
            3| in somepkg.src.meta

       … while selecting an attribute
         at «string»:3:4:
            2|   somepkg.src = "I'm not an attrset";
            3| in somepkg.src.meta
             |    ^

       error: expected a set but found a string: "I'm not an attrset"

into:

let
  somepkg.src = "I'm not an attrset";
in somepkg.src.meta

error:
       … while evaluating the attribute 'src.meta'
         at «string»:2:3:
            1| let
            2|   somepkg.src = "I'm not an attrset";
             |   ^
            3| in somepkg.src.meta

       … while selecting 'meta' on 'somepkg.src'
         at «string»:3:4:
            2|   somepkg.src = "I'm not an attrset";
            3| in somepkg.src.meta
             |    ^

       error: expected a set but found a string: "I'm not an attrset"

For the low price of an enumerate() and a lambda you too can have the
incorrect line of code actually show up in the trace!

Change-Id: Ic1491c86e33c167891bdac9adad6224784760bd6
2024-07-04 17:42:35 -06:00
Qyriad d00edfb28d trace when the foo part of foo.bar.baz errors
Turns errors like:

let
  errpkg = throw "invalid foobar";
in errpkg.meta

error:
       … while calling the 'throw' builtin
         at «string»:2:12:
            1| let
            2|   errpkg = throw "invalid foobar";
             |            ^
            3| in errpkg.meta

       error: invalid foobar

into errors like:

let
  errpkg = throw "invalid foobar";
in errpkg.meta

error:
       … while evaluating 'errpkg' to select 'meta' on it
         at «string»:3:4:
            2|   errpkg = throw "invalid foobar";
            3| in errpkg.meta
             |    ^

       … while calling the 'throw' builtin
         at «string»:2:12:
            1| let
            2|   errpkg = throw "invalid foobar";
             |            ^
            3| in errpkg.meta

       error: invalid foobar

For the low price of one try/catch, you too can have the incorrect line
of code actually show up in the trace!

Change-Id: If8d6200ec1567706669d405c34adcd7e2d2cd29d
2024-07-04 16:33:02 -06:00
Qyriad 139cfdfb53 add an ExprPrinter class, like ValuePrinter
To be used Shortly

Change-Id: I9def7975aa55f251eb8486391677771f7352d7ce
2024-07-04 15:55:38 -06:00
Qyriad 59bf6825ef add an impl of Expr::show for ExprInheritFrom that doesn't crash
ExprVar::show() assumes it has a name. dynamic inherits do not
necessarily (ever?) have a name.

Change-Id: If10893188e307431da17f0c1bd0787adc74f7141
2024-07-04 15:55:38 -06:00
Qyriad 4f0c27abe1 give ExprInheritFrom a handle to what its standing in for
Change-Id: I12088e0b618407e5432523bbc97be63c8d6fce62
2024-07-04 15:55:38 -06:00
Artemis Tosini e040b762a4
libstore: add LocalDerivationGoal startChild hook
Add a platform-specific function for starting sandboxed child.
Generally this just means startProcess, but on Linux we use flags
for clone to start a new namespace

Change-Id: I41c8aba62676a162388bbe5ab8a7518904c7b058
2024-07-03 22:37:41 +00:00
Artemis Tosini af1dcc2d5e
libstore: Add LocalDerivationGoal prepareSandbox hook
Add a new OS-specific hook called `prepareSandbox`, run before forking
On Darwin this is empty as nothing is required,
on Linux this creates the chroot directory and adds basic files,
and on platforms using a fallback this throws an exception

Change-Id: Ie30c38c387f2e0e5844b2afa32fd4d33b1180dae
2024-07-03 22:16:03 +00:00
eldritch horrors 5eec6418de libutil: begin porting serialization to generators
generators are a better basis for serializers than streaming into sinks
as we do currently for many reasons, such as being usable as sources if
one wishes to (without requiring an intermediate sink to serialize full
data sets into memory, or boost coroutines to turn sinks into sources),
composing more naturally (as one can just yield a sub-generator instead
of being forced to wrap entire substreams into clunky functions or even
more clunky custom types to implement operator<< on), allowing wrappers
to transform data with clear ownership semantics (removing the need for
explicit memory allocations and Source wrappers), and many other things

Change-Id: I361d89ff556354f6930d9204f55117565f2f7f20
2024-07-03 11:46:53 +00:00
eldritch horrors c65f5dd18e libutil: convert drainFD to a Bytes generator
the `*Source` name is a slight misnomer since we do also have a
Source type, but we can probably live with this for time being.

Change-Id: I54eb2e59a4009014e324797f16b80b962759c7d3
2024-07-03 11:46:53 +00:00
eldritch horrors b252b3c6e3 libutil: allow draining Generator<Bytes> into sinks
Change-Id: I442d03a5399096d4baca9a2618b4c4b64db36c4b
2024-07-03 11:46:53 +00:00
eldritch horrors 4857feb910 libutil: add Bytes type
not used anywhere yet, but we'll use this a lot soon for generators that
return file contents, wire protocol fragments, or indeed any byte stream

Change-Id: I01a46f9bf9d75aaf4a5d7662773b99f498862a28
2024-07-03 11:46:53 +00:00
eldritch horrors 73ddc4540f libutil: generator type with on-yield value mapping
this will be the basis of non-boost coroutines in lix. anything that is
a boost coroutine *should* be representable with a Generator coroutine,
and many things that are not currently boost coroutines but behave much
like one (such as, notably, serializers) should be as well. this allows
us to greatly simplify many things that look like iteration but aren't.

Change-Id: I2cebcefa0148b631fb30df4c8cfa92167a407e34
2024-07-03 11:46:53 +00:00
Qyriad 45ac449d39 Merge "Revert "Revert "bump the extra --version info from info to notice, -vv -> -v""" into main 2024-07-02 22:26:34 +00:00
Qyriad 18a06aad52 Revert "Revert "bump the extra --version info from info to notice, -vv -> -v""
This reverts commit 9f16a20f3d.

Since c55e93ca2, the original reason for reverting d003dcd7f
no longer applies.

Change-Id: If88f8555a6060d0530dcfecdf55afaa40afd1ad9
2024-07-02 22:03:17 +00:00
eldritch horrors e7517419a6 libmain: better fix for #424, #425
not printing activities at all when no progress information is available
hides *all* progress information from e.g. flake show. this is not ideal
and needs to be fixed, but the fix *still* has problems with flake show:
in multiline mode we will overwrite all useful flake show output as soon
as the progress bar is redrawn. flake show output is also mangled in any
number of other situations (like -v being set), so we should probably be
not too worried about it and fix progress reporting properly another day

Change-Id: I6d39d670e261bbae00560b6a8e15dec8e16b35c4
2024-07-02 17:16:30 +02:00
alois31 24852355d8 Merge "tree-wide: unify progress bar inactive and paused states" into main 2024-07-02 14:12:07 +00:00
Delan Azabani 865a3732fa Merge "Reject fully-qualified URLs in 'from' argument of nix registry add" into main 2024-07-02 07:20:01 +00:00
alois31 0dd1d8ca1c
tree-wide: unify progress bar inactive and paused states
Previously, the progress bar had two subtly different states in which the bar
would not actually render, both with their own shortcomings: inactive (which
was irreversible) and paused (reversible, but swallowing logs). Furthermore,
there was no way of resetting the statistics, so a very bad solution was
implemented (243c0f18da) that would create a new
logger for each line of the repl, leaking the previous one and discarding the
value of printBuildLogs. Finally, if stderr was not attached to a TTY, the
update thread was started even though the logger was not active, violating the
invariant required by the destructor (which is not observed because the logger
is leaked).

In this commit, the two aforementioned states are unified into a single one,
which can be exited again, correctly upholds the invariant that the update
thread is only running while the progress bar is active, and does not swallow
logs. The latter change in behavior is not expected to be a problems in the
rare cases where the paused state was used before, since other loggers (like
the simple one) don't exhibit it anyway. The startProgressBar/stopProgressBar
API is removed due to being a footgun, and a new method for properly resetting
the progress is added.

Co-Authored-By: Qyriad <qyriad@qyriad.me>
Change-Id: I2b7c3eb17d439cd0c16f7b896cfb61239ac7ff3a
2024-07-01 18:19:34 +02:00
jade d3286d0990 Merge changes Ie29a8a89,I873eedcf into main
* changes:
  store: delete obsolete lsof-disabling code
  store: guess the URL of failing fixed-output derivations
2024-07-01 16:11:32 +00:00
eldritch horrors 010ff57ebb enable -Werror=unused-result
we do not have any of these warnings appearing at the moment, but it
seems like a good idea to enable [[nodiscard]] checking anyway. once
we start introducing more functions with must-use conditions we will
need such checking, and the rust stdlib has proven them very useful.

Change-Id: Ibb6b042ae1ec5f527f8dc2809a7816a4c1548ae2
2024-06-30 21:30:43 +00:00
alois31 a55112898e
libexpr/flake: allow automatic rejection of configuration options from flakes
The `allow-flake-configuration` option allows the user to control whether to
accept configuration options supplied by flakes. Unfortunately, setting this
to false really meant "ask each time" (with an option to remember the choice
for each specific option encountered). Let no mean no, and introduce (and
default to) a separate value for the "ask each time" behaviour.

Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Change-Id: I7ccd67a95bfc92cffc1ebdc972d243f5191cc1b4
2024-06-30 19:28:14 +02:00
Delan Azabani b2944d93a6 Reject fully-qualified URLs in 'from' argument of nix registry add
We previously allowed you to map any flake URL to any other flake URL,
including shorthand flakerefs, indirect flake URLs like `flake:nixpkgs`,
direct flake URLs like `github:NixOS/nixpkgs`, or local paths.

But flake registry entries mapping from direct flake URLs often come
from swapping the 'from' and 'to' arguments by accident, and even when
created intentionally, they may not actually work correctly.

This patch rejects those URLs (and fully-qualified flake: URLs), making
it harder to swap the arguments by accident.

Fixes #181.

Change-Id: I24713643a534166c052719b8770a4edfcfdb8cf3
2024-06-29 05:11:31 +00:00
jade d85309f7ca store: delete obsolete lsof-disabling code
Since Ifa0adda7984e, we don't use this code anymore on macOS, so we have
no reason to have a knob to disable it anymore.

Change-Id: Ie29a8a8978d9aefd4551895f4f9b3cc0827496df
2024-06-27 22:53:36 -07:00
jade d92712673b store: guess the URL of failing fixed-output derivations
This is a shameless layering violation in favour of UX. It falls back
trivially to "unknown", so it's purely a UX feature.

Diagnostic sample:

```
error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv':
        likely URL: https://meow.puppy.forge/puppy.tar.gz
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-a1Qvp3FOOkWpL9kFHgugU1ok5UtRPSu+NwCZKbbaEro=
```

Change-Id: I873eedcf7984ab23f57a6754be00232b5cb5b02c
2024-06-27 22:44:16 -07:00
jade 5dc85e8b72 Merge "packaging: make pegtl use the __forDefaults mechanism" into main 2024-06-26 22:11:52 +00:00
jade 77c5364596 Merge "doc/hacking: fix up some outdated info about cross, hydra links" into main 2024-06-26 22:11:36 +00:00
eldritch horrors 3dd7d023f4 libmain: don't print empty lines
this most notably affects `nix eval`: if there is no progress bar to be
shown and no activities going on we should not print anything at all. a
progress bar with no activities would print a bunch of terminal escapes
*and a space*, which is not helpful in simple cases like nix eval -E 1.
notably this does *not* affect nix eval called on non-terminal outputs,
but it is slightly confusing nevertheless (and not difficult to avoid).

fixes lix-project/lix#424

Change-Id: Iee793c79ba5a485d6606e0d292ed2eae6dfb7216
2024-06-26 17:44:04 +00:00
jade 9afb0fe41c Merge changes I476a2516,I8a274227 into main
* changes:
  doc/hacking: fix internal api docs section to say to enable it
  doc: Add more about the release note generator
2024-06-26 17:34:45 +00:00
alois31 30da1b17d9 Merge "libmain/progress-bar: move implementation out of the header" into main 2024-06-26 16:05:44 +00:00
jade f7d54cb6b1 packaging: make pegtl use the __forDefaults mechanism
This avoids needing to pass it in when callPackage'ing Lix from external
code.

Change-Id: Ie07e84a151e38614064609a2f6dbff165e193be7
2024-06-26 00:44:46 -07:00
jade 85c1241201 doc/hacking: fix up some outdated info about cross, hydra links
We would like to build these with Hydra but we do not currently have a
Hydra to build them with conveniently.

Change-Id: I0832a33881138dd1caab3805df7ad097db347e62
2024-06-25 21:46:26 -07:00
jade 33d53c4983 doc/hacking: fix internal api docs section to say to enable it
I filed a bug to build these in releng in the future:
lix-project/lix#422

Change-Id: I476a2516cc2be382d4b7c8529a02f9212a78fdb2
2024-06-25 21:26:18 -07:00
jade e537678f1e doc: Add more about the release note generator
Change-Id: I8a274227cb1b05d442d3f644603dd2844ecc9d05
2024-06-25 21:22:37 -07:00
jade 4ac2c496d4 Merge "change shebangs of all .sh scripts to bash" into main 2024-06-25 22:18:26 +00:00
jade aceef13682 Merge changes If0ddec6b,Iaa63ed18 into main
* changes:
  Add some release notes for things we did
  packaging: don't build internal api docs by default in dev shells
2024-06-25 22:16:04 +00:00
Lunaphied f170870ae7 Merge ".envrc: remove MAKEFLAGS and use clang environment by default" into main 2024-06-25 20:42:46 +00:00
Lunaphied 97c86908a4 .envrc: remove MAKEFLAGS and use clang environment by default
MAKEFLAGS hasn't been relevant since we switched off the Make
buildsystem and using the clang environment by default gives you clangd
by default which most developers will want.

Change-Id: I9c11d0613577047e6c908f049c1ffaca5fb5ff67
2024-06-25 12:36:18 -06:00
eldritch horrors e6cd67591b libexpr: rewrite the parser with pegtl instead of flex/bison
this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.

this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.

test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:

  - 40% faster on lix's package.nix at 10000 iterations
  - 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
  - equivalent on all of nixpkgs concatenated at 100 iterations
    (excluding invalid files, each file surrounded with parens)

more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.

Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.408s ± 0.025s
            user: 0.355s | system: 0.033s
  median:   0.389s
  range:    0.388s ... 0.442s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.332s ± 0.024s
            user: 0.279s | system: 0.033s
  median:   0.314s
  range:    0.313s ... 0.361s
  relative: 0.814

---

old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     6.133s ± 0.022s
            user: 5.395s | system: 0.437s
  median:   6.128s
  range:    6.099s ... 6.183s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     5.925s ± 0.025s
            user: 5.176s | system: 0.456s
  median:   5.934s
  range:    5.861s ... 5.943s
  relative: 0.966

---

GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.503s ± 0.027s
            user: 3.731s | system: 0.547s
  median:   4.499s
  range:    4.478s ... 4.541s
  relative: 1

GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.285s ± 0.031s
            user: 3.504s | system: 0.571s
  median:   4.281s
  range:    4.221s ... 4.328s
  relative: 0.951

---

old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     16.475s ± 0.07s
            user: 14.088s | system: 1.572s
  median:   16.495s
  range:    16.351s ... 16.536s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     15.973s ± 0.013s
            user: 13.558s | system: 1.615s
  median:   15.973s
  range:    15.946s ... 15.99s
  relative: 0.97

---

Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
2024-06-25 12:24:58 +00:00
jade c097ebe66b Merge "Revert "libfetchers: make attribute / URL query handling consistent"" into main 2024-06-25 10:19:52 +00:00
jade e19f27917c Add some release notes for things we did
Change-Id: If0ddec6b64a43c3d8f6cae39e0292863f3b49401
2024-06-24 16:26:12 -07:00
jade 1245340e44 packaging: don't build internal api docs by default in dev shells
These are totally available and you can just turn them on, but they have
very bad dependency tracking and thus bloat incremental change times,
which is not really ok.

Change-Id: Iaa63ed18a789e74fcb757248cd24c3b194afcc80
2024-06-24 15:57:38 -07:00
jade 3e151d4d77 Revert "libfetchers: make attribute / URL query handling consistent"
This reverts commit 35eec921af.

Reason for revert: Regressed nix-eval-jobs, and it appears to be this change is buggy/missing a case. It just needs another pass.

Code causing the problem in n-e-j, when invoked with `nix-eval-jobs --flake '.#hydraJobs'`:

```
n-e-j/tests/assets » ../../build/src/nix-eval-jobs --meta --workers 1 --flake .#hydraJobs
warning: unknown setting 'trusted-users'
warning: `--gc-roots-dir' not specified
error: unsupported Git input attribute 'dir'
error: worker error: error: unsupported Git input attribute 'dir'
```

```
  nix::Value *vRoot = [&]() {
        if (args.flake) {
            auto [flakeRef, fragment, outputSpec] =
                nix::parseFlakeRefWithFragmentAndExtendedOutputsSpec(
                    args.releaseExpr, nix::absPath("."));
            nix::InstallableFlake flake{
                {}, state, std::move(flakeRef), fragment, outputSpec,
                {}, {},    args.lockFlags};

            return flake.toValue(*state).first;
        } else {
            return releaseExprTopLevelValue(*state, autoArgs, args);
        }
    }();
```

Inspecting the program behaviour reveals that `dir` was in fact set in the URL going into the fetcher. This is in turn because unlike in the case changed in this commit, it was not erased before handing it to libfetchers, which is probably just a mistake.

```
(rr) up
3  0x00007ffff60262ae in nix::fetchers::Input::fromURL (url=..., requireTree=requireTree@entry=true) at src/libfetchers/fetchers.cc:39
warning: Source file is more recent than executable.
39              auto res = inputScheme->inputFromURL(url, requireTree);
(rr) p url
$1 = (const nix::ParsedURL &) @0x7fffdc874190: {url = "git+file:///home/jade/lix/nix-eval-jobs", 
  base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", authority = std::optional<std::string> = {[contained value] = ""}, 
  path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {["dir"] = "tests/assets"}, fragment = ""}
(rr) up
4  0x00007ffff789d904 in nix::parseFlakeRefWithFragment (url=".#hydraJobs", baseDir=std::optional<std::string> = {...}, 
    allowMissing=allowMissing@entry=false, isFlake=isFlake@entry=true) at src/libexpr/flake/flakeref.cc:179
warning: Source file is more recent than executable.
179                                 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
(rr) p parsedURL
$2 = {url = "git+file:///home/jade/lix/nix-eval-jobs", base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", 
  authority = std::optional<std::string> = {[contained value] = ""}, path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {
    ["dir"] = "tests/assets"}, fragment = ""}
(rr) list
174
175                             if (pathExists(flakeRoot + "/.git/shallow"))
176                                 parsedURL.query.insert_or_assign("shallow", "1");
177
178                             return std::make_pair(
179                                 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
180                                 fragment);
181                         }
```

Change-Id: Ib55a882eaeb3e59228857761dc1e3b2e366b0f5e
2024-06-24 22:49:17 +00:00
vigress8 c7af89c797 change shebangs of all .sh scripts to bash
On operating systems where /bin/sh is not Bash, some scripts are invalid
because of bashisms, and building Lix fails with errors like this:
`render-manpage.sh: 3: set: Illegal option -o pipefail`
This modifies all scripts that use a `/bin/sh` shebang to `/usr/bin/env
bash`, including currently POSIX-compliant ones, to prevent any future
confusion.

Change-Id: Ia074cc6db42d40fc59a63726f6194ea0149ea5e0
2024-06-24 14:00:43 -07:00
jade d5637ee790 devShell: guard against running from another directory
I was working on nix-eval-jobs with a dev shell with some shenanigans to
run against a locally built Lix and it was getting really annoying when
`nix develop ../lix#` was messing up my other git repo's hooks.

This is a fix via blunt force, but it is at least obvious how it works.

Change-Id: Ia29eeb5be57ab6a2c88451c00ea18a51e4dfe65e
2024-06-24 13:41:38 -07:00
jade eb5de71adc justfile: accept extra options to just setup and pass them to meson
This lets you get the default options and still be able to add more.

Change-Id: Ife32c348b1498ff2ccdddf051a5bba520cfa36f0
2024-06-24 13:37:01 -07:00
alois31 206a5dbb8f
libmain/progress-bar: move implementation out of the header
Change-Id: Ib4b42ebea290ee575294df6b2f17a38a5d850b80
2024-06-24 20:39:50 +02:00
Robert Hensing d86009bd76 Add build-dir setting, clean up default TMPDIR handling
This is a squash of upstream PRs #10303, #10312 and #10883.

fix: Treat empty TMPDIR as unset

Fixes an instance of

    nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.

... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.

(cherry picked from commit c3fb2aa1f9d1fa756dac38d3588c836c5a5395dc)

fix: Treat empty XDG_RUNTIME_DIR as unset

See preceding commit. Not observed in the wild, but is sensible
and consistent with TMPDIR behavior.

(cherry picked from commit b9e7f5aa2df3f0e223f5c44b8089cbf9b81be691)

local-derivation-goal.cc: Reuse defaultTempDir()

(cherry picked from commit fd31945742710984de22805ee8d97fbd83c3f8eb)

fix: remove usage of XDG_RUNTIME_DIR for TMP

(cherry picked from commit 1363f51bcb24ab9948b7b5093490a009947f7453)

tests/functional: Add count()

(cherry picked from commit 6221770c9de4d28137206bdcd1a67eea12e1e499)

Remove uncalled for message

(cherry picked from commit b1fe388d33530f0157dcf9f461348b61eda13228)

Add build-dir setting

(cherry picked from commit 8b16cced18925aa612049d08d5e78eccbf0530e4)
Change-Id: Ic7b75ff0b6a3b19e50a4ac8ff2d70f15c683c16a
2024-06-24 11:30:32 +03:00
Delan Azabani 4c3d93611f Merge "Fix build instructions in hacking.md and justfile" into main 2024-06-24 06:46:08 +00:00
Delan Azabani 37f276ba9d Merge "Fix compile error under gcc with -Denable-pch-std=false" into main 2024-06-24 06:45:40 +00:00
Delan Azabani 5d33e4cd59 Fix build instructions in hacking.md and justfile
The stdenv phases don’t actually do anything (at least not anymore),
and our justfile doesn’t behave the same as our docs.

This patch removes the stdenv phases from the docs, documents our
usage of just, and makes `just setup` heed `$mesonFlags`.

Fixes #413.
Fixes #414.

Change-Id: Ieb0b2a8ae420526238b5f9a73d7849ec6919995d
2024-06-24 12:55:00 +08:00
Delan Azabani 3baffbdcc5 Fix compile error under gcc with -Denable-pch-std=false
Following the latest hacking.md currently fails because of a missing
include in upstream editline. This patch fixes the build by adding
that missing include.

Fixes #410.

Change-Id: Iefd4cb687ed3da71ccda9fe9624f38e6ef4623e5
2024-06-24 12:47:04 +08:00
eldritch horrors d477b34d1d libutil: remove runProgram2 stdin functionality
this was only used in one place, and that place has been rewritten to
use a temporary file instead. keeping this around is not very helpful
at this time, and in any case we'd be better off rewriting subprocess
handling in rust where we not only have a much safer library for such
things but also async frameworks necessary for this easily available.

Change-Id: I6f8641b756857c84ae2602cdf41f74ee7a1fda02
2024-06-23 17:29:40 +00:00
eldritch horrors 2bbdaf0b19 libfetchers: write git commit message to tempfile
we want to remove runProgram's ability to provide stdin to a process
because the concurrency issues of handling both stdin and stdout are
much more pronounced once runProgram returns not is collected output
but a source. this is possible in the current c++ framework, however
it isn't necessary in almost all cases (as demonstrated by only this
single user existing) and in much better handled with a proper async
concurrency model that lets the caller handle both at the same time.

Change-Id: I29da1e1ad898d45e2e33a7320b246d5003e7712b
2024-06-23 17:29:40 +00:00
eldritch horrors 2fe9157808 flakes: add --commit-lock-file message test
we had no test to ensure that we generated a commit message at all?

Change-Id: Ic9aa8fde92b83e1ea6f61cd2a21867aa73d4e885
2024-06-23 17:29:40 +00:00
Qyriad 521e08cbde Merge changes I697f4f39,I9f25235d into main
* changes:
  doc-comment Fields for Activity and Result types
  mildly refactor the renderActivity if hell-chain
2024-06-23 16:55:36 +00:00
Maximilian Bosch 7c3b8229cc Merge "libstore: reduce loglevel of waiting for a machine to build" into main 2024-06-23 16:33:44 +00:00
Qyriad 701eb332bd doc-comment Fields for Activity and Result types
The ones we were able to figure out, at least.

Change-Id: I697f4f3942e35a7adf1b2a6cc28b3168d1de111c
2024-06-23 16:30:48 +00:00
Qyriad ce4dee0fa5 mildly refactor the renderActivity if hell-chain
This is primarily for readability, but iwrc chaining std::string's
operator+ is also pretty scuffed performance-wise, and this was doing a
lot of operator+ chainging.

Change-Id: I9f25235df153eb2bbb491f1ff7ebbeed9a8ec985
2024-06-23 16:30:38 +00:00
Maximilian Bosch 5f0062285c Merge "libfetchers: make attribute / URL query handling consistent" into main 2024-06-23 15:51:34 +00:00
eldritch horrors ce6cb14995 libutil: return Pid from startProcess, not pid_t
Change-Id: Icc8a15090c77f54ea7d9220aadedcd4a19922814
2024-06-23 11:52:49 +00:00
eldritch horrors 3d155fc509 libutil: give Pid proper resource semantics
copy-constructing or assigning from pid_t can easily lead to duplicate
Pid instances for the same process if a pid_t was used carelessly, and
Pid itself was copy-constructible. both could cause surprising results
such as killing processes twice (which could become very problemantic,
but luckily modern systems don't reuse PIDs all that quickly), or more
than one piece of the code believing it owns a process when neither do

Change-Id: Ifea7445f84200b34c1a1d0acc2cdffe0f01e20c6
2024-06-23 11:52:49 +00:00
eldritch horrors b43a2e84c4 libutil: make Pid -> pid_t operations explicit
Change-Id: I3137cc140590001fe7ba542844e735944a0a9255
2024-06-23 11:52:49 +00:00
eldritch horrors a9949f4760 libutil: add some serialize.hh serializer tests
Change-Id: I0116265a18bc44bba16c07bf419af70d5195f07d
2024-06-23 11:52:49 +00:00
eldritch horrors 39a1e248c9 libutil: remove sinkToSource eof callback
this is only used in one place, and only to set a nicer error message on
EndOfFile. the only caller that actually *catches* this exception should
provide an error message in that catch block rather than forcing support
for setting error message so deep into the stack. copyStorePath is never
called outside of PathSubstitutionGoal anyway, which catches everything.

Change-Id: Ifbae8706d781c388737706faf4c8a8b7917ca278
2024-06-23 11:52:49 +00:00
Artemis Tosini f80d95e36d Merge "libstore: Start creating LocalDerivationGoal subclasses" into main 2024-06-23 08:11:51 +00:00
Artemis Tosini 12f5d27363 libstore: Start creating LocalDerivationGoal subclasses
LocalDerivationGoal includes a large number of low-level sandboxing
primitives for Darwin and Linux, intermingled with ifdefs.
Start creating platform-specific classes to make it easier to add new
platforms and review platform-specific code.

This change only creates support infrastructure and moves two function,
more functions will be moved in future changes.

Change-Id: I9fc29fa2a7345107d4fc96c46fa90b4eabf6bb89
2024-06-23 03:33:07 +00:00
Qyriad fb8553f63c mildly cleanup ExprSelect::eval
Better variable names, some comments, and a slight logic rearrange.

Change-Id: I9685ae252f83217aa85f06432234159c9ad19d1c
2024-06-22 18:52:57 -06:00
Qyriad e09cc60df9 doc-comment ExprSelect's fields
Change-Id: I63e79991a4bab93421266785e9258e0f5bb89b8f
2024-06-22 18:52:57 -06:00
Maximilian Bosch fc6a1451af
libstore: reduce loglevel of waiting for a machine to build
This comes quite often when the available job slots on all remote
builders are exhausted and this is pretty spammy.

This isn't really an issue, but expected behavior.

A better way to display this is a nom-like approach where all scheduled
builds are shown in a tree and pending builds are being marked as such
IMHO.

Change-Id: I6bc14e6054f84e3eb0768127b490e263d8cdcf89
2024-06-22 17:38:25 +02:00
Maximilian Bosch 35eec921af
libfetchers: make attribute / URL query handling consistent
The original idea was to fix lix#174, but for a user friendly solution,
I figured that we'd need more consistency:

* Invalid query params will cause an error, just like invalid
  attributes. This has the following two consequences:

  * The `?dir=`-param from flakes will be removed before the URL to be
    fetched is passed to libfetchers.

  * The tarball fetcher doesn't allow URLs with custom query params
    anymore. I think this was questionable anyways given that an
    arbitrary set of query params was silently removed from the URL you
    wanted to fetch. The correct way is to use an attribute-set
    with a key `url` that contains the tarball URL to fetch.

  * Same for the git & mercurial fetchers: in that case it doesn't even
    matter though: both fetchers added unused query params to the URL
    that's passed from the input scheme to the fetcher (`url2` in the code).
    It turns out that this was never used since the query parameters were
    erased again in `getActualUrl`.

* Validation happens for both attributes and URLs. Previously, a lot of
  fetchers validated e.g. refs/revs only when specified in a URL and
  the validity of attribute names only in `inputFromAttrs`.

  Now, all the validation is done in `inputFromAttrs` and `inputFromURL`
  constructs attributes that will be passed to `inputFromAttrs`.

* Accept all attributes as URL query parameters. That also includes
  lesser used ones such as `narHash`.

  And "output" attributes like `lastModified`: these could be declared
  already when declaring inputs as attribute rather than URL. Now the
  behavior is at least consistent.

  Personally, I think we should differentiate in the future between
  "fetched input" (basically the attr-set that ends up in the lock-file)
  and "unfetched input" earlier: both inputFrom{Attrs,URL} entrypoints
  are probably OK for unfetched inputs, but for locked/fetched inputs
  a custom entrypoint should be used. Then, the current entrypoints
  wouldn't have to allow these attributes anymore.

Change-Id: I1be1992249f7af8287cfc37891ab505ddaa2e8cd
2024-06-22 14:42:43 +02:00
kloenk da4e46dd1f libmain: add progress bar with multiple status lines
Add the log-formats `multiline` and `multiline-with-logs` which offer
multiple current active building status lines.

Change-Id: Idd8afe62f8591b5d8b70e258c5cefa09be4cab03
2024-06-22 14:20:27 +02:00
Qyriad 21865ccce0 Merge "add a basic libmain test for the progress bar rendering" into main 2024-06-22 02:04:14 +00:00
jade 375f4c0337 Merge "libstore: remove operations that are never called by supported clients" into main 2024-06-21 20:46:18 +00:00
Qyriad fd250c51ed add a basic libmain test for the progress bar rendering
Hooray for leaky abstraction allowing us to test this particular part of
the render pipeline.

Change-Id: Ie0f251ff874f63324e6a9c6388b84ec6507eeae2
2024-06-20 13:56:53 -06:00
Qyriad e44dcd63c4 remove InstallableValueCommand class
Change-Id: Id12383f4741cba07159712700ebcfbe37e61560c
2024-06-20 17:00:06 +00:00
Qyriad 6515b1a495 de-inheritance CmdSearch for InstallableValueCommand
Change-Id: I125c8cac05c8e924b55e4eb1060496e35ea4e941
2024-06-20 17:00:06 +00:00
Qyriad 50be55ffca de-inheritance CmdEdit for InstallableValueCommand
Change-Id: If85ea78954a45470b0b25c08dc7d40bfebd53610
2024-06-20 17:00:06 +00:00
Qyriad 079eeb1de7 de-inheritance CmdRun for InstallableValueCommand
Change-Id: Ief858c1488197211e2ee8b70aa40ed6c65743558
2024-06-20 17:00:06 +00:00
Qyriad b9e9235ac0 de-inheritance CmdBundle for InstallableValueCommand
Change-Id: Icbac4ef927ddcaf0d2a74b376e5a77299529cd34
2024-06-20 17:00:06 +00:00
Qyriad 1e5f134560 de-inheritance CmdEval for InstallableValueCommand
Change-Id: I08b1702310e863d15de26dc838eb0bcb62417c10
2024-06-20 17:00:06 +00:00
Qyriad 8ba1939540 use a type alias for ProgressBar's chosen time point type
Change-Id: I621a455b1daba806fc498958aee7931fbfc55445
2024-06-20 15:24:27 +00:00
Qyriad f9594b592b extract ProgressBar declaration into its header file
Change-Id: Ica9e2ec41d99eaa196a0d535501edf45c589b2b6
2024-06-20 15:24:27 +00:00
Qyriad 3a4c21fc9e slight cleanup to ProgressBar::getStatus()
Binaries were identical before and after this commit on our machine

Change-Id: I6f8bfbe3298d6c5f43d5702c7a1e05cb180226cc
2024-06-20 15:24:27 +00:00
Ilya K 697ef65c14 Merge "BrotliDecompressionSource: don't bail out too early" into main 2024-06-20 07:06:52 +00:00
Ilya K 7d52d74bbe BrotliDecompressionSource: don't bail out too early
If we've consumed the entire input, that doesn't actually mean we're
done decompressing - there might be more output left. This worked (?)
in most cases because the input and output sizes are pretty comparable,
but sometimes they're not and then things get very funny.

Change-Id: I73435a654a911b8ce25119f713b80706c5783c1b
2024-06-20 09:21:13 +03:00
jade 6c29a2a6fc Merge "libstore: fix queryValidPaths concurrency" into main 2024-06-20 05:55:08 +00:00
jade 50472aa5be libstore: remove operations that are never called by supported clients
I did a whole bunch of `git log -S` to find out exactly when all these
things were obsoleted and found the commit in which their usage was
removed, which I have added in either the error message or a comment.

I've also made *some* of the version checks into static asserts for when
we update the minimum supported protocol version.

In the end this is not a lot of code we are deleting, but it's code that
we will never have to support into the future when we build a protocol
bridge, which is why I did it. It is not in the support baseline.

Change-Id: Iea3c80795c75ea74f328cf7ede7cbedf8c41926b
2024-06-19 19:41:04 -07:00
Eelco Dolstra fb7d315411 Merge pull request #10570 from layus/shared_caches
Share evaluation caches across installables

Before:

$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
  'nixpkgs#hello' \
  'nixpkgs#clang' \
  'nixpkgs#cargo' \
  'nixpkgs#rustup' \
  'nixpkgs#bear' \
  'nixpkgs#firefox' \
  'nixpkgs#git-revise' \
  'nixpkgs#hyperfine' \
  'nixpkgs#curlie' \
  'nixpkgs#xz' \
  'nixpkgs#ripgrep'
0:03.61

After:

$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
  'nixpkgs#hello' \
  'nixpkgs#clang' \
  'nixpkgs#cargo' \
  'nixpkgs#rustup' \
  'nixpkgs#bear' \
  'nixpkgs#firefox' \
  'nixpkgs#git-revise' \
  'nixpkgs#hyperfine' \
  'nixpkgs#curlie' \
  'nixpkgs#xz' \
  'nixpkgs#ripgrep'
0:01.46

This could probably use a more proper benchmark...

Fixes #313

(cherry picked from commit de51e5c335865e3e0a8cccd283fec1a52cce243f)
Change-Id: I9350bebd462b6af12c51db5bf432321abfe84a16
2024-06-19 18:39:11 +00:00
eldritch horrors c55dcc6c13 filetransfer: return a Source from download()
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)

Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
2024-06-19 10:50:12 +00:00
eldritch horrors 11f4a5bc7e libutil: return a source from readFile
don't consume a sink, return a source instead. the only reason to not do
this is a very slight reduction in dynamic allocations, but since we are
going to *at least* do disk io that will not be a lot of overhead anyway

Change-Id: Iae2f879ec64c3c3ac1d5310eeb6a85e696d4614a
2024-06-19 10:50:12 +00:00
eldritch horrors 67f778670c libutil: add makeDecompressionSource
Change-Id: Iac7f24d79e24417436b9b5cbefd6af051aeea0a6
2024-06-19 10:50:12 +00:00
eldritch horrors 3425e90d76 libstore: BinaryCacheStore::getFile{ -> Contents}
if we want have getFile return a source instead of consuming a sink
we'll have to disambiguate this overload another way, eg like this.

Change-Id: Ia26de2020c309a37e7ccc3775c1ad1f32e0a778b
2024-06-19 10:50:12 +00:00
alois31 fed34594d8 Merge "libfetchers: represent unfetched submodules consistently" into main 2024-06-19 07:08:19 +00:00
jade 85f282ef57 Merge changes Id0e651e4,I0ed20da8,I76bd6d22,I5d8ffb7b into main
* changes:
  store: fix null reference from DerivationGoal::waiteeDone
  libmain: fix UB in verbosity assignment
  build: make UBSan work :)
  libexpr: fix accessing uninitialized values and fix pure-eval docs
2024-06-19 03:31:47 +00:00
Qyriad b338435b75 Merge "refactor lambda formals handling" into main 2024-06-19 03:13:46 +00:00
jade c897fba787 store: fix null reference from DerivationGoal::waiteeDone
This happened during a PathSubstitutionGoal of a .drv file:

substitution of '/tmp/jade/nix-test/ca/eval-store/store/1lj7lsq5y0f25mfbnq6d3zd0bw5ay33n-dependencies-input-2.drv'

What happened here is that since PathSubstitutionGoal is not a
DerivationGoal, in production builds, the UB was not caught, since it
would early-exit from failing a dynamic_cast to DerivationGoal * on the
very next line, but before the null reference was ever used.

This was nonetheless UB. The fix should be to just rearrange the two
lines; I don't think there is a further bug there, since *substituting a
.drv* **necessarily** means you cannot have the representation of
the derivation as would be necessary for drv to not be null there.

Test failure:

++(eval-store.sh:12) _RR_TRACE_DIR=/home/jade/.local/share/rr rr record -- nix build -f dependencies.nix --eval-store /tmp/jade/nix-test/ca/eval-store/eval-store -o /tmp/jade/nix-test/ca/eval-store/result
don't know how to build these paths:
  /tmp/jade/nix-test/ca/eval-store/store/6y51mf0p57ggipgab6hdjabbvplzsicq-dependencies-top.drv
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/8027afyvqb87y1sf5xhdkqsflqn1ziy8-dependencies.builder0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/7r5pqyncvfgrryf9gzy1z56z3xigi61x-builder-dependencies-input-0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nhmgm87zlqy3ks96dxrn7l37b72azi99-builder-dependencies-input-1.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nq4qa2j6y8ajqazlfq6h46ck637my1n6-builder-dependencies-input-2.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/6vh0vna9l5afck01y7iaks3hm9ikwqyj-builder-fod-input.sh' to 'local'...
building '/tmp/jade/nix-test/ca/eval-store/store/gy91pqymf2nc5v7ld1bad94xpwxdi25s-dependencies-input-0.drv'...
building '/tmp/jade/nix-test/ca/eval-store/store/w7wlkjx97ivmnrymkac5av3nyp94hzvq-dependencies-input-1.drv'...
../src/libstore/build/derivation-goal.cc:1556:22: runtime error: reference binding to null pointer of type 'Derivation'
    0 0x734ba59a6886 in nix::DerivationGoal::waiteeDone(std::shared_ptr<nix::Goal>, nix::Goal::ExitCode) /home/jade/lix/lix2/build/src/libstore/build/derivation-goal.cc:1556:12
    1 0x734ba59c0962 in nix::Goal::amDone(nix::Goal::ExitCode, std::optional<nix::Error>) /home/jade/lix/lix2/build/src/libstore/build/goal.cc:95:25
    2 0x734ba5a1c44a in nix::PathSubstitutionGoal::done(nix::Goal::ExitCode, nix::BuildResult::Status, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>) /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:38:5
    3 0x734ba5a1b454 in nix::PathSubstitutionGoal::init() /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:56:9
    4 0x734ba5a2a6c6 in nix::Worker::run(std::set<std::shared_ptr<nix::Goal>, nix::CompareGoalPtrs, std::allocator<std::shared_ptr<nix::Goal>>> const&) /home/jade/lix/lix2/build/src/libstore/build/worker.cc:320:23
    5 0x734ba59b93d8 in nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath>> const&, nix::BuildMode, std::shared_ptr<nix::Store>) /home/jade/lix/lix2/build/src/libstore/build/entry-points.cc:60:12
    6 0x734ba663c107 in nix::Installable::build2(nix::ref<nix::Store>, nix::ref<nix::Store>, nix::Realise, std::vector<nix::ref<nix::Installable>, std::allocator<nix::ref<nix::Installable>>> const&, nix::BuildMode) /home/jade/lix/lix2/build/src/libcmd/installables.cc:637:36

Change-Id: Id0e651e480bebf6356733b01bc639e9bb59c7bd0
2024-06-18 19:26:54 -07:00
jade f2fff1faa4 libmain: fix UB in verbosity assignment
This was generating an out-of-range verbosity value. We should just
process it as an int and then convert to verbosity with a clamping
function, which trivially avoids any domain type violations.

Change-Id: I0ed20da8e1496a1225ff3008b76827d99265d404
2024-06-18 19:26:54 -07:00
jade e0a3a5f226 build: make UBSan work :)
This is really just a question of turning off the production sanitizer
configuration so we get nice diagnostics. Not much else to say.

Change-Id: I76bd6d225320056ed95bd89955f00beff2db0d2f
2024-06-18 19:25:58 -07:00
jade 1eef1927b6 libexpr: fix accessing uninitialized values and fix pure-eval docs
We got UBSan working on Lix, so we of course immediately found a bug and
some definitely nonsense behaviour.

Accessing `pureEval` or `restrictEval` from a default setting value is
nonsense, since they would never be actually set by the time that value
is set so they are not going to do anything. The configuration is not
applied in an initializer (and even if it were, it's not going to be in
the right order).

After looking into *that*, we hunted down what actually was applying
these, since clearly this code did not do anything. The EvalState
constructor should have a "search path added and removed here :)" sign
on it, because that's where it is done. We added an explicit
initialization of the optional in there because it was otherwise unclear
why pureEval also has the search path to allowed paths setup code run.

We then realized that the `pureEval` documentation was *also* bogus, and
we rewrote it. In so doing, we realized that we forgot to file a bug to
make `builtins.storePath` work in pure eval mode, so we filed one of
those: lix-project/lix#402

Yaks have been thoroughly shorn.

UBSan report:

    ../src/libexpr/eval-settings.cc:66:10: runtime error: member call on address 0x752fa9a13060 which does not point to an object of type 'nix::BaseSetting<b
    ool>'
    0x752fa9a13060: note: object has invalid vptr
     00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00
                  ^~~~~~~~~~~~~~~~~~~~~~~
                  invalid vptr
        0 0x752fa95106a6 in nix::EvalSettings::getDefaultNixPath[abi:cxx11]() /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:66:10
        1 0x752fa950e420 in nix::EvalSettings::EvalSettings() /home/jade/lix/lix2/build/src/libexpr/eval-settings.hh:36:15
        2 0x752fa9469f1f in __cxx_global_var_init.50 /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:98:14
        3 0x752fa9469f1f in _GLOBAL__sub_I_eval_settings.cc /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc
        4 0x752fabbd308d in call_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x508d) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
        5 0x752fabbd317b in _dl_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x517b) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
        6 0x752fabbe9c2f in _dl_start_user (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x1bc2f) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)

Change-Id: I5d8ffb7bfbe24b6584020ac74eed93d9f2e6d111
2024-06-18 19:25:35 -07:00
Qyriad 54d2c189ae refactor lambda formals handling
Change-Id: Iebffd5436109da270ee870670a20f5ee7db9a204
2024-06-19 00:57:56 +00:00
jade 66a9fbb7ff libstore: fix queryValidPaths concurrency
The lock usage was obviously wrong so it was entirely serialized. This
has the predicted speedups, the only question is whether it is sound
because it's exposing a bunch of new code to actual concurrency.

I did audit all the stores' queryPathInfoUncached implementations and
they all look *intended* to be thread safe, but whether that is actually
sound or not: lol lmao. I am highly confident in the s3 one because it
is calling s3 sdk methods that are thread safe and has no actual state.

Others are using Pool and look to be *supposed* to be thread safe, but
unsure if they actually are.

Change-Id: I0369152a510e878b5ac56c9ac956a98d48cd5fef
2024-06-18 23:29:08 +00:00
alois31 aa00a5a8c9 libfetchers: represent unfetched submodules consistently
Unfetched submodules are included as empty directories in archives, so they end
up as such in the store when fetched in clean mode. Make sure the same happens
in dirty mode too. Fortunately, they are already correctly represented in the
ls-files output, so we just need to make sure to include the empty directory in
our filter.

Fixes: https://github.com/NixOS/nix/issues/6247
Change-Id: I60d06ff360cfa305d081b920838c893c06da801c
2024-06-18 00:54:51 +00:00
276 changed files with 8658 additions and 4454 deletions

8
.envrc
View file

@ -1,9 +1,5 @@
# shellcheck shell=bash
source_env_if_exists .envrc.local
# TODO: `use flake .#native-clangStdenvPackages` on macOS?
use flake ".#${LIX_SHELL_VARIANT:-default}" "${LIX_SHELL_EXTRA_ARGS[@]}"
export MAKEFLAGS="$MAKEFLAGS -e"
if [[ -n "$NIX_BUILD_CORES" ]]; then
export MAKEFLAGS="$MAKEFLAGS -j $NIX_BUILD_CORES"
fi
# Use native-clangStdenvPackages to get clangd by default.
use flake ".#${LIX_SHELL_VARIANT:-native-clangStdenvPackages}" "${LIX_SHELL_EXTRA_ARGS[@]}"
export GTEST_BRIEF=1

1
.this-is-lix Normal file
View file

@ -0,0 +1 @@
This is a file used by the dev shell shellHook in package.nix to check that this is actually a Lix repo before installing git hooks. Its contents have no meaning.

View file

@ -1,54 +0,0 @@
diff --git a/pthread_stop_world.c b/pthread_stop_world.c
index 2b45489..0e6d8ef 100644
--- a/pthread_stop_world.c
+++ b/pthread_stop_world.c
@@ -776,6 +776,8 @@ STATIC void GC_restart_handler(int sig)
/* world is stopped. Should not fail if it isn't. */
GC_INNER void GC_push_all_stacks(void)
{
+ size_t stack_limit;
+ pthread_attr_t pattr;
GC_bool found_me = FALSE;
size_t nthreads = 0;
int i;
@@ -868,6 +870,40 @@ GC_INNER void GC_push_all_stacks(void)
hi = p->altstack + p->altstack_size;
# endif
/* FIXME: Need to scan the normal stack too, but how ? */
+ } else {
+ #ifdef HAVE_PTHREAD_ATTR_GET_NP
+ if (pthread_attr_init(&pattr) != 0) {
+ ABORT("GC_push_all_stacks: pthread_attr_init failed!");
+ }
+ if (pthread_attr_get_np(p->id, &pattr) != 0) {
+ ABORT("GC_push_all_stacks: pthread_attr_get_np failed!");
+ }
+ #else
+ if (pthread_getattr_np(p->id, &pattr)) {
+ ABORT("GC_push_all_stacks: pthread_getattr_np failed!");
+ }
+ #endif
+ if (pthread_attr_getstacksize(&pattr, &stack_limit)) {
+ ABORT("GC_push_all_stacks: pthread_attr_getstacksize failed!");
+ }
+ if (pthread_attr_destroy(&pattr)) {
+ ABORT("GC_push_all_stacks: pthread_attr_destroy failed!");
+ }
+ // When a thread goes into a coroutine, we lose its original sp until
+ // control flow returns to the thread.
+ // While in the coroutine, the sp points outside the thread stack,
+ // so we can detect this and push the entire thread stack instead,
+ // as an approximation.
+ // We assume that the coroutine has similarly added its entire stack.
+ // This could be made accurate by cooperating with the application
+ // via new functions and/or callbacks.
+ #ifndef STACK_GROWS_UP
+ if (lo >= hi || lo < hi - stack_limit) { // sp outside stack
+ lo = hi - stack_limit;
+ }
+ #else
+ #error "STACK_GROWS_UP not supported in boost_coroutine2 (as of june 2021), so we don't support it in Nix."
+ #endif
}
# ifdef STACKPTR_CORRECTOR_AVAILABLE
if (GC_sp_corrector != 0)

View file

@ -3,6 +3,10 @@
#
# It's used for crediting people accurately in release notes. The release notes
# script will link to forgejo, then to GitHub if forgejo is not present.
#
# When adding someone from outside the Lix project, you generally want to simply link their GitHub profile without adding a display name unless they are well-known in the community by that display name.
#
# See doc/manual/src/contributing/hacking.md for more documentation on this file's format and typical usage.
9999years:
display_name: wiggles
forgejo: rbt
@ -40,6 +44,11 @@ cole-h:
display_name: Cole Helbling
github: cole-h
delan:
display_name: delan
forgejo: delan
github: delan
edolstra:
display_name: Eelco Dolstra
github: edolstra
@ -60,6 +69,10 @@ jade:
forgejo: jade
github: lf-
kloenk:
forgejo: kloenk
github: kloenk
lovesegfault:
github: lovesegfault
@ -83,6 +96,10 @@ puck:
forgejo: puck
github: puckipedia
quantumjump:
display_name: Quantum Jump
github: QuantumBJump
r-vdp:
github: r-vdp

View file

@ -1,4 +1,4 @@
#!/bin/sh
#!/usr/bin/env bash
set -euo pipefail

View file

@ -1,4 +1,4 @@
#!/bin/sh
#!/usr/bin/env bash
set -euo pipefail

View file

@ -0,0 +1,23 @@
---
synopsis: Define integer overflow in the Nix language as an error
issues: [fj#423]
cls: [1594, 1595, 1597, 1609]
category: Fixes
credits: [jade]
---
Previously, integer overflow in the Nix language invoked C++ level signed overflow, which was undefined behaviour, but *probably* manifested as wrapping around on overflow.
Since prior to the public release of Lix, Lix had C++ signed overflow defined to crash the process and nobody noticed this having accidentally removed overflow from the Nix language for three months until it was caught by fiddling around.
Given the significant body of actual Nix code that has been evaluated by Lix in that time, it does not appear that nixpkgs or much of importance depends on integer overflow, so it is safe to turn into an error.
Some other overflows were fixed:
- `builtins.fromJSON` of values greater than the maximum representable value in a signed 64-bit integer will generate an error.
- `nixConfig` in flakes will no longer accept negative values for configuration options.
Integer overflow now looks like the following:
```
» nix eval --expr '9223372036854775807 + 1'
error: integer overflow in adding 9223372036854775807 + 1
```

View file

@ -0,0 +1,90 @@
---
synopsis: "Trace which part of a `foo.bar.baz` expression errors"
cls: 1505, 1506
credits: Qyriad
category: Improvements
---
Previously, if an attribute path selection expression like `linux_4_9.meta.description` it wouldn't show you which one of those parts in the attribute path, or even that that line of code is what caused evaluation of the failing expression.
The previous error looks like this:
```
pkgs.linuxKernel.kernels.linux_4_9.meta.description
error:
… while evaluating the attribute 'linuxKernel.kernels.linux_4_9.meta.description'
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:5:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
… while calling the 'throw' builtin
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:17:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
error: linux 4.9 was removed because it will reach its end of life within 22.11
```
Now, the error will look like this:
```
pkgs.linuxKernel.kernels.linux_4_9.meta.description
error:
… while evaluating the attribute 'linuxKernel.kernels.linux_4_9.meta.description'
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:5:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
… while evaluating 'pkgs.linuxKernel.kernels.linux_4_9' to select 'meta' on it
at «string»:1:1:
1| pkgs.linuxKernel.kernels.linux_4_9.meta.description
| ^
… caused by explicit throw
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:17:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
error: linux 4.9 was removed because it will reach its end of life within 22.11
```
Not only does the line of code that referenced the failing attribute show up in the trace, it also tells you that it was specifically the `linux_4_9` part that failed.
This includes if the failing part is a top-level binding:
```
let
inherit (pkgs.linuxKernel.kernels) linux_4_9;
in linux_4_9.meta.description
error:
… while evaluating 'linux_4_9' to select 'meta.description' on it
at «string»:3:4:
2| inherit (pkgs.linuxKernel.kernels) linux_4_9;
3| in linux_4_9.meta.description
| ^
… while evaluating the attribute 'linux_4_9'
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:5:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
… caused by explicit throw
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:17:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
error: linux 4.9 was removed because it will reach its end of life within 22.11
```

View file

@ -0,0 +1,12 @@
---
synopsis: "Add a `build-dir` setting to set the backing directory for builds"
cls: 1514
credits: [roberth, tomberek]
category: Improvements
---
`build-dir` can now be set in the Nix configuration to choose the backing directory for the build sandbox.
This can be useful on systems with `/tmp` on tmpfs, or simply to relocate large builds to another disk.
Also, `XDG_RUNTIME_DIR` is no longer considered when selecting the default temporary directory,
as it's not intended to be used for large amounts of data.

View file

@ -0,0 +1,70 @@
---
synopsis: "Distinguish between explicit throws and errors that happened while evaluating a throw"
cls: 1511
credits: Qyriad
category: Improvements
---
Previously, errors caused by an expression like `throw "invalid argument"` were treated like an error that happened simply while some builtin function was being called:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg "linuz"
error:
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg "linuz"
error: linuz isn't the right package
```
But the error didn't just happen "while" calling the `throw` builtin — it's a throw error!
Now it looks like this:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg "linuz"
error:
… caused by explicit throw
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg "linuz"
error: linuz isn't the right package
```
This also means that incorrect usage of `throw` or errors evaluating its arguments are easily distinguishable from explicit throws:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg { attrs = "error when coerced in string interpolation"; }
error:
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg { attrs = "error when coerced in string interpolation"; }
… while evaluating a path segment
at «string»:2:24:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg { attrs = "error when coerced in string interpolation"; }
error: cannot coerce a set to a string: { attrs = "error when coerced in string interpolation"; }
```
Here, instead of an actual thrown error, a type error happens first (trying to coerce an attribute set to a string), but that type error happened *while* calling `throw`.

View file

@ -0,0 +1,29 @@
---
synopsis: Fix nix-collect-garbage --dry-run
issues: [fj#432]
cls: [1566]
category: Fixes
credits: [quantumjump]
---
`nix-collect-garbage --dry-run` did not previously give any output - it simply
exited without even checking to see what paths would be deleted.
```
$ nix-collect-garbage --dry-run
$
```
We updated the behaviour of the flag such that instead it prints out how many
paths it *would* delete, but doesn't actually delete them.
```
$ nix-collect-garbage --dry-run
finding garbage collector roots...
determining live/dead paths...
...
<nix store paths>
...
2670 store paths deleted, 0.00MiB freed
$
```

View file

@ -0,0 +1,16 @@
---
synopsis: "Hash mismatch diagnostics for fixed-output derivations include the URL"
cls: [1536]
credits: [jade]
category: Improvements
---
Now, when building fixed-output derivations, Lix will guess the URL that was used in the derivation using the `url` or `urls` properties in the derivation environment.
This is a layering violation but making these diagnostics tractable when there are multiple instances of the `AAAA` hash is too significant of an improvement to pass it up.
```
error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv':
likely URL: https://meow.puppy.forge/puppy.tar.gz
specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
got: sha256-a1Qvp3FOOkWpL9kFHgugU1ok5UtRPSu+NwCZKbbaEro=
```

View file

@ -0,0 +1,14 @@
---
synopsis: Add log formats `multiline` and `multiline-with-logs`
cls: [1369]
credits: [kloenk]
category: Improvements
---
Added two new log formats (`multiline` and `multiline-with-logs`) that display
current activities below each other for better visibility.
These formats attempt to use the maximum available lines
(defaulting to 25 if unable to determine) and print up to that many lines.
The status bar is displayed as the first line, with each subsequent
activity on its own line.

View file

@ -0,0 +1,12 @@
---
synopsis: "`nix copy` is now several times faster at `querying info about /nix/store/...`"
cls: [1462]
issues: [fj#366]
credits: [jade]
category: Fixes
---
We fixed a locking bug that serialized `querying info about /nix/store/...`
onto just one thread such that it was eating `O(paths to copy * latency)` time
while setting up to copy paths to s3 and other stores. It is now `nproc` times
faster.

View file

@ -0,0 +1,21 @@
---
synopsis: "Lix no longer speaks the Nix remote-build worker protocol to clients or servers older than CppNix 2.3"
cls: [1207, 1208, 1206, 1205, 1204, 1203, 1479]
issues: [fj#325]
credits: [jade]
category: Breaking Changes
---
CppNix 2.3 was released in 2019, and is the new oldest supported version. We
will increase our support baseline in the future up to a final version of CppNix
2.18 (which may happen soon given that it is the only still-packaged and thus
still-tested >2.3 version), but this step already removes a significant amount
of dead, untested, code paths.
Lix speaks the same version of the protocol as CppNix 2.18 and that fact will
never change in the future; the Lix plans to replace the protocol for evolution
will entail a complete incompatible replacement that will be supported in
parallel with the old protocol. Lix will thus retain remote build compatibility
with CppNix as long as CppNix maintains protocol compatibility with 2.18, and
as long as Lix retains legacy protocol support (which will likely be a long
time given that we plan to convert it to a frozen-in-time shim).

View file

@ -0,0 +1,58 @@
---
synopsis: "Eliminate some pretty-printing surprises"
cls: [1616, 1617, 1618]
prs: [11100]
credits: [alois31, roberth]
category: Improvements
---
Some inconsistent and surprising behaviours have been eliminated from the pretty-printing used by the REPL and `nix eval`:
* Lists and attribute sets that contain only a single item without nested structures are no longer sometimes inappropriately indented in the REPL, depending on internal state of the evaluator.
* Empty attribute sets and derivations are no longer shown as `«repeated»`, since they are always cheap to print.
This matches the existing behaviour of `nix-instantiate` on empty attribute sets.
Empty lists were never printed as `«repeated»` already.
* The REPL by default does not print nested attribute sets and lists, and indicates elided items with an ellipsis.
Previously, the ellipsis was printed even when the structure was empty, so that such items do not in fact exist.
Since this behaviour was confusing, it does not happen any more.
Before:
```
nix-repl> :p let x = 1 + 2; in [ [ x ] [ x ] ]
[
[
3
]
[ 3 ]
]
nix-repl> let inherit (import <nixpkgs> { }) hello; in [ hello hello ]
[
«derivation /nix/store/fqs92lzychkm6p37j7fnj4d65nq9fzla-hello-2.12.1.drv»
«repeated»
]
nix-repl> let x = {}; in [ x ]
[
{ ... }
]
```
After:
```
nix-repl> :p let x = 1 + 2; in [ [ x ] [ x ] ]
[
[ 3 ]
[ 3 ]
]
nix-repl> let inherit (import <nixpkgs> { }) hello; in [ hello hello ]
[
«derivation /nix/store/fqs92lzychkm6p37j7fnj4d65nq9fzla-hello-2.12.1.drv»
«derivation /nix/store/fqs92lzychkm6p37j7fnj4d65nq9fzla-hello-2.12.1.drv»
]
nix-repl> let x = {}; in [ x ]
[
{ }
]
```

View file

@ -0,0 +1,10 @@
---
synopsis: "`nix registry add` now requires a shorthand flakeref on the 'from' side"
cls: 1494
credits: delan
category: Improvements
---
The 'from' argument must now be a shorthand flakeref like `nixpkgs` or `nixpkgs/nixos-20.03`, making it harder to accidentally swap the 'from' and 'to' arguments.
Registry entries that map from other flake URLs can still be specified in registry.json, the `nix.registry` option in NixOS, or the `--override-flake` option in the CLI, but they are not guaranteed to work correctly.

View file

@ -0,0 +1,9 @@
---
synopsis: Allow automatic rejection of configuration options from flakes
cls: [1541]
credits: [alois31]
category: Improvements
---
Setting `accept-flake-config` to `false` now respects user choice by automatically rejecting configuration options set by flakes.
The old behaviour of asking each time is still available (and default) by setting it to the special value `ask`.

View file

@ -0,0 +1,10 @@
---
synopsis: "Lix now supports building with UndefinedBehaviorSanitizer"
cls: [1483]
credits: [jade]
category: Development
---
You can now build Lix with the configuration option `-Db_sanitize=undefined` and it will both work and pass tests. AddressSanitizer support is also coming soon.
For a list of undefined behaviour fixed by sanitizer usage, see [the gerrit topic "undefined-behaviour"](https://gerrit.lix.systems/q/topic:%22undefined-behaviour%22).

View file

@ -196,8 +196,8 @@
- [C++ style guide](contributing/cxx.md)
- [Release Notes](release-notes/release-notes.md)
- [Upcoming release](release-notes/rl-next.md)
<!-- RELENG-AUTO-INSERTION-MARKER (see releng/release_notes.py) -->
- [Lix 2.90 (FIXME date)](release-notes/rl-2.90.md)
<!-- RELENG-AUTO-INSERTION-MARKER (see releng/release_notes.py) -->
- [Lix 2.90 (2024-07-10)](release-notes/rl-2.90.md)
- [Nix 2.18 (2023-09-20)](release-notes/rl-2.18.md)
- [Nix 2.17 (2023-07-24)](release-notes/rl-2.17.md)
- [Nix 2.16 (2023-05-31)](release-notes/rl-2.16.md)

View file

@ -78,6 +78,16 @@ Most commands in Lix accept the following command-line options:
Display the raw logs, with the progress bar at the bottom.
- `multiline`
Display a progress bar during the builds and in the lines below that one line per activity.
- `multiline-with-logs`
Displayes the raw logs, with a progress bar and activities each in a new line at the bottom.
- <span id="opt-no-build-output">[`--no-build-output`](#opt-no-build-output)</span> / `-Q`
By default, output written by builders to standard output and standard error is echoed to the Lix command's standard error.

View file

@ -39,17 +39,19 @@ $ nix-shell -A native-clangStdenvPackages
### Building from the development shell
As always you may run [stdenv's phases by name](https://nixos.org/manual/nixpkgs/unstable/#sec-building-stdenv-package-in-nix-shell), e.g.:
You can build and test Lix with just:
```bash
$ configurePhase
$ buildPhase
$ checkPhase
$ installPhase
$ installCheckPhase
$ just setup
$ just build
$ just test --suite=check
$ just install
$ just test --suite=installcheck
```
To build manually, however, use the following:
(Check and installcheck may both be done after install, allowing you to omit the --suite argument entirely, but this is the order package.nix runs them in.)
You can also build Lix manually:
```bash
$ meson setup ./build "--prefix=$out" $mesonFlags
@ -64,9 +66,7 @@ $ meson install -C build
$ meson test -C build --suite=installcheck
```
(Check and installcheck may both be done after install, allowing you to omit the --suite argument entirely, but this is the order package.nix runs them in.)
This will install Lix to `$PWD/outputs`, the `/bin` of which is prepended to PATH in the development shells.
In both cases, Lix will be installed to `$PWD/outputs`, the `/bin` of which is prepended to PATH in the development shells.
If the tests fail and Meson helpfully has no output for why, use the `--print-error-logs` option to `meson test`.
@ -168,8 +168,27 @@ or for Nix with the [`flakes`] and [`nix-command`] experimental features enabled
$ nix build .#packages.aarch64-linux.default
```
Cross-compiled builds are available for ARMv6 (`armv6l-linux`) and ARMv7 (`armv7l-linux`).
Add more [system types](#system-type) to `crossSystems` in `flake.nix` to bootstrap Nix on unsupported platforms.
### Cross compiling using the Lix flake
Lix can also be easily cross compiled to the following arbitrarily-chosen system doubles, which can be useful for bootstrapping Lix on new platforms.
These are specified in `crossSystems` in `flake.nix`; feel free to submit changes to add new ones if they are useful to you.
- `armv6l-linux`
- `armv7l-linux`
- `aarch64-linux`
- `riscv64-linux`
For example, to cross-compile Lix for `armv6l-linux` from another Linux, use the following:
```console
$ nix build .#nix-armv6l-linux
```
It's also possible to cross-compile a tarball of binaries suitable for the Lix installer, for example, for `riscv64-linux`:
```console
$ nix build .#nix-riscv64-linux.passthru.binaryTarball
```
### Building for multiple platforms at once
@ -282,9 +301,8 @@ Regular markdown files used for the manual have a base path of their own and the
## API documentation
Doxygen API documentation is [available
online](https://hydra.nixos.org/job/nix/master/internal-api-docs/latest/download-by-type/doc/internal-api-docs). You
can also build and view it yourself:
Doxygen API documentation will be available online in the future ([tracking issue](https://git.lix.systems/lix-project/lix/issues/422)).
You can also build and view it yourself:
```console
# nix build .#hydraJobs.internal-api-docs
@ -294,44 +312,50 @@ can also build and view it yourself:
or inside a `nix develop` shell by running:
```bash
$ meson configure build -Dinternal-api-docs=enabled
$ meson compile -C build internal-api-docs
$ xdg-open ./outputs/doc/share/doc/nix/internal-api/html/index.html
```
## Coverage analysis
A coverage analysis report is [available
online](https://hydra.nixos.org/job/nix/master/coverage/latest/download-by-type/report/coverage). You
can build it yourself:
A coverage analysis report will be available online in the future (FIXME(lix-hydra)).
You can build it yourself:
```
# nix build .#hydraJobs.coverage
# xdg-open ./result/coverage/index.html
```
Metrics about the change in line/function coverage over time are also
[available](https://hydra.nixos.org/job/nix/master/coverage#tabs-charts).
Metrics about the change in line/function coverage over time will be available in the future (FIXME(lix-hydra)).
## Add a release note
`doc/manual/rl-next` contains release notes entries for all unreleased changes.
User-visible changes should come with a release note.
Developer-facing changes should have a release note in the Development category if they are significant and if developers should know about them.
### Add an entry
Here's what a complete entry looks like. The file name is not incorporated in the document.
Here's what a complete entry looks like.
The file name is not incorporated in the final document, and is generally a super brief summary of the change synopsis.
```
```markdown
---
synopsis: Basically a title
# 1234 or gh#1234 will refer to CppNix GitHub, fj#1234 will refer to a Lix forgejo issue.
issues: [1234, fj#1234]
# Use this *only* if there is a CppNix pull request associated with this change
# Use this *only* if there is a CppNix pull request associated with this change.
prs: 1238
# List of Lix Gerrit changelist numbers; if there is an associated Lix GitHub
# PR, just put in the Gerrit CL number.
# List of Lix Gerrit changelist numbers.
# If there is an associated Lix GitHub PR, just put in the Gerrit CL number.
cls: [123]
# Heading that this release note will appear under.
category: Breaking Changes
# Add a credit mention in the bottom of the release note.
# your-name is used as a key into doc/manual/change-authors.yml for metadata
credits: [your-name]
---
Here's one or more paragraphs that describe the change.
@ -346,6 +370,31 @@ Significant changes should add the following header, which moves them to the top
significance: significant
```
The following categories of release notes are supported (see `maintainers/build-release-notes.py`):
- Breaking Changes
- Features
- Improvements
- Fixes
- Packaging
- Development
- Miscellany
The `credits` field, if present, gives credit to the author of the patch in the release notes with a message like "Many thanks to (your-name) for this" and linking to GitHub or Forgejo profiles if listed.
If you are forward-porting a change from CppNix, please credit the original author, and optionally credit yourself.
When adding credits metadata for people external to the project and deciding whether to put in a `display_name`, consider what they are generally known as in the community; even if you know their full name (e.g. from their GitHub profile), we suggest only adding it as a display name if that is what they go by in the community.
There are multiple reasons we follow this practice, but it boils down to privacy and consent: we would rather not capture full names that are not widely used in the community without the consent of the parties involved, even if they are publicly available.
As of this writing, the entries with full names as `display_name` are either members of the CppNix team or people who added them themselves.
The names specified in `credits` are used as keys to look up the authorship info in `doc/manual/change-authors.yml`.
The only mandatory part is that every key appearing in `credits` has an entry present in `change-authors.yml`.
All of the following properties are optional; you can specify `{}` as the metadata if you want a simple non-hyperlinked mention.
The following properties are supported:
- `display_name`: display name used in place of the key when showing names, if present.
- `forgejo`: Forgejo username. The name in the release notes will be a link to this, if present.
- `github`: GitHub username, used if `forgejo` is not set, again making a link.
### Build process
Releases have a precomputed `rl-MAJOR.MINOR.md`, and no `rl-next.md`.

View file

@ -247,7 +247,6 @@ To ensure that characterization testing doesn't make it harder to intentionally
The integration tests are defined in the Nix flake under the `hydraJobs.tests` attribute.
These tests include everything that needs to interact with external services or run Lix in a non-trivial distributed setup.
Because these tests are expensive and require more than what the standard github-actions setup provides, they only run on the master branch (on <https://hydra.nixos.org/jobset/nix/master>).
You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}`
@ -433,9 +432,6 @@ I grepped `src/` for `get[eE]nv\("` to find the mentions in Lix code.
- `NIX_PROFILE` - Selects which profile `nix-env` will operate on. Documented elsewhere.
- `NIX_SSHOPTS` - Options passed to `ssh(1)` when using a ssh remote store.
Incorrectly documented on `nix-copy-closure` which is *surely* not the only place they are used??
- `_NIX_TEST_NO_LSOF` - Used on non-Linux, non-macOS platforms to disable using `lsof` when finding gc roots.
Since https://git.lix.systems/lix-project/lix/issues/156 was fixed, this should probably just be removed as it was a bad workaround for a macOS issue.
- `_NIX_TEST_GC_SYNC_1` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
- `_NIX_TEST_GC_SYNC_2` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
- `_NIX_TEST_FREE_SPACE_FILE` - Path to a file containing a decimal number with the free space that the GC is to believe it has.

View file

@ -59,8 +59,10 @@ The result is a [Boolean] value.
## Arithmetic
Numbers are type-compatible:
Pure integer operations will always return integers, whereas any operation involving at least one floating point number return a floating point number.
Numbers will retain their type unless mixed with other numeric types:
Pure integer operations will always return integers, whereas any operation involving at least one floating point number returns a floating point number.
Integer overflow (of 64-bit signed integers) and division by zero are defined to throw an error.
See also [Comparison] and [Equality].
@ -143,19 +145,71 @@ All comparison operators are implemented in terms of `<`, and the following equi
| *a* `>` *b* | *b* `<` *a* |
| *a* `>=` *b* | `! (` *a* `<` *b* `)` |
Note that the above behaviour violates IEEE 754 for floating point numbers with respect to NaN, for instance.
This may be fixed in a future major language revision.
[Comparison]: #comparison-operators
## Equality
- [Attribute sets][attribute set] and [list]s are compared recursively, and therefore are fully evaluated.
- Comparison of [function]s always returns `false`.
- Numbers are type-compatible, see [arithmetic] operators.
- Floating point numbers only differ up to a limited precision.
The following equality comparison rules are followed in order:
- Comparisons are first, sometimes, performed by identity (pointer value), and whether or not this occurs varies depending on the context in which the comparison is performed; for example, through `builtins.elem`, comparison of lists, or other cases.
The exact instances in which this occurs, aside from direct list and attribute set comparisons as discussed below, are too dependent on implementation details to meaningfully document.
See [note on identity comparison](#identity-comparison) below.
- Comparisons between a combination of integers and floating point numbers are first converted to floating point then compared as floating point.
- Comparisons between values of differing types, besides the ones mentioned in the above rule, are unequal.
- Strings are compared as their string values, disregarding string contexts.
- Paths are compared as their absolute form (since they are stored as such).
- [Functions][function] are always considered unequal, including with themselves.
- The following are compared in the typical manner:
- Integers
- Floating point numbers have equality comparison per IEEE 754.
Note that this means that just like in most languages, floating point arithmetic results are not typically equality comparable, and should instead be compared by checking that the absolute difference is less than some error margin.
- Booleans
- Null
- [Attribute sets][attribute set] are compared following these rules in order:
- If both attribute sets have the same identity (via pointer equality), they are considered equal, regardless of whether the contents have reflexive equality (e.g. even if there are functions contained within).
See [note on identity comparison](#identity-comparison) below.
- If both attribute sets have `type = "derivation"` and have an attribute `outPath` that is equal, they are considered equal.
This means that two results of `builtins.derivation`, regardless of other things added to their attributes via `//` afterwards (or `passthru` in nixpkgs), will compare equal if they passed the same arguments to `builtins.derivation`.
- Otherwise, they are compared element-wise in an unspecified order.
Although this order *may* be deterministic in some cases, this is not guaranteed, and correct code must not rely on this ordering behaviour.
The order determines which elements are evaluated first and thus, if there are throwing values in the attribute set, which of those get evaluated, if any, before the comparison returns an unequal result.
- Lists are compared following these rules in order:
- If both lists have the same identity (via pointer equality), they are considered equal, regardless of whether the contents have reflexive equality (e.g. even if there are functions contained within).
See [note on identity comparison](#identity-comparison) below.
- Otherwise, they are compared element-wise in list order.
[function]: ./constructs.md#functions
[Equality]: #equality
### Identity comparison
In the current revision of the Nix language, values are first compared by identity (pointer equality).
This means that values that are not reflexively equal (that is, they do not satisfy `a == a`), such as functions, are nonetheless sometimes compared as equal with themselves if they are placed in attribute sets or lists, or are compared through other indirect means.
Whether identity comparison applies to a given usage of the language aside from direct list and attribute set comparison is strongly dependent on implementation details to the point it is not feasible to document the exact instances.
This is rather unfortunate behaviour which is regrettably load-bearing on nixpkgs (such as with the `type` attribute of NixOS options) and cannot be changed for the time being.
It may be changed in a future major language revision.
Correct code must not rely on this behaviour.
For example:
```
nix-repl> let f = x: 1; s = { func = f; }; in [ (f == f) (s == s) ]
[ false true ]
```
## Logical implication
Equivalent to `!`*b1* `||` *b2*.

View file

@ -7,13 +7,16 @@
*Strings* can be written in three ways.
The most common way is to enclose the string between double quotes,
e.g., `"foo bar"`. Strings can span multiple lines. The special
characters `"` and `\` and the character sequence `${` must be
escaped by prefixing them with a backslash (`\`). Newlines, carriage
returns and tabs can be written as `\n`, `\r` and `\t`,
respectively.
e.g., `"foo bar"`. Strings can span multiple lines. The backslash
(`\`) can be used to escape characters: newlines, carriage returns
and tabs may be written as `\n`, `\r` and `\t` respectively; any
other characters can be preceded by a backslash to remove any
special meaning they may have, like the special characters `"` and
`\` and the character sequence `${`.
You can include the results of other expressions into a string by enclosing them in `${ }`, a feature known as [string interpolation].
Due to a parser issue that has since come to be relied upon, the character sequence `$${` is interpreted literally and does not introduce an interpolation.
To express a `$` character immediately followed by an interpolation, the former must be escaped.
[string interpolation]: ./string-interpolation.md
@ -43,16 +46,16 @@
Note that the whitespace and newline following the opening `''` is
ignored if there is no non-whitespace text on the initial line.
Indented strings support [string interpolation].
Since `${` and `''` have special meaning in indented strings, you
need a way to quote them. `$` can be escaped by prefixing it with
`''` (that is, two single quotes), i.e., `''$`. `''` can be escaped
by prefixing it with `'`, i.e., `'''`. `$` removes any special
meaning from the following `$`. Linefeed, carriage-return and tab
by prefixing it with `'`, i.e., `'''`. Linefeed, carriage-return and tab
characters can be written as `''\n`, `''\r`, `''\t`, and `''\`
escapes any other character.
Indented strings support [string interpolation] using `${ }` the same way regular strings do.
`$${` is interpreted literally in indented strings as well, so the `$` character must be escaped if it is to be followed by an interpolation.
Indented strings are primarily useful in that they allow multi-line
string literals to follow the indentation of the enclosing Nix
expression, and that less escaping is typically necessary for
@ -85,6 +88,9 @@
Numbers, which can be *integers* (like `123`) or *floating point*
(like `123.43` or `.27e13`).
Integers in the Nix language are 64-bit signed integers.
Integer overflow is defined to throw an error.
See [arithmetic] and [comparison] operators for semantics.
[arithmetic]: ./operators.md#arithmetic

View file

@ -1,7 +1,7 @@
# Lix 2.90 "Vanilla Ice Cream" (FIXME date)
# Lix 2.90 "Vanilla Ice Cream" (2024-07-10)
# Lix 2.90.0 (FIXME date)
# Lix 2.90.0 (2024-07-10)
## Breaking Changes
- Deprecate the online flake registries and vendor the default registry [fj#183](https://git.lix.systems/lix-project/lix/issues/183) [fj#110](https://git.lix.systems/lix-project/lix/issues/110) [fj#116](https://git.lix.systems/lix-project/lix/issues/116) [#8953](https://github.com/NixOS/nix/issues/8953) [#9087](https://github.com/NixOS/nix/issues/9087) [cl/1127](https://gerrit.lix.systems/c/lix/+/1127)

View file

@ -84,10 +84,13 @@
];
systems = linuxSystems ++ darwinSystems;
# If you add something here, please update the list in doc/manual/src/contributing/hacking.md.
# Thanks~
crossSystems = [
"armv6l-linux"
"armv7l-linux"
"riscv64-linux"
"aarch64-linux"
# FIXME: still broken in 24.05: fails to build rustc(??) due to missing -lstdc++ dep
# "x86_64-freebsd"
# FIXME: broken dev shell due to python
@ -137,10 +140,7 @@
system = crossSystem;
}
// lib.optionalAttrs (crossSystem == "x86_64-freebsd") { useLLVM = true; };
overlays = [
(overlayFor (p: p.${stdenv}))
(final: prev: { nixfmt = final.callPackage ./nix-support/nixfmt.nix { }; })
];
overlays = [ (overlayFor (p: p.${stdenv})) ];
};
stdenvs = forAllStdenvs (make-pkgs null);
native = stdenvs.stdenvPackages;
@ -195,14 +195,19 @@
busybox-sandbox-shell = final.busybox-sandbox-shell or final.default-busybox-sandbox-shell;
};
pegtl = final.nix.passthru.pegtl;
# Export the patched version of boehmgc that Lix uses into the overlay
# for consumers of this flake.
boehmgc-nix = final.nix.boehmgc-nix;
boehmgc-nix = final.nix.passthru.boehmgc-nix;
# And same thing for our build-release-notes package.
build-release-notes = final.nix.build-release-notes;
build-release-notes = final.nix.passthru.build-release-notes;
};
in
{
# for repl debugging
inherit self;
# A Nixpkgs overlay that overrides the 'nix' and
# 'nix.perl-bindings' packages.
overlays.default = overlayFor (p: p.stdenv);
@ -321,6 +326,25 @@
pkgs = nixpkgsFor.x86_64-linux.native;
};
releaseTests = lib.foldl lib.recursiveUpdate { } [
(lib.genAttrs (linux64BitSystems ++ darwinSystems) (system: {
nativeBuild = self.packages.${system}.nix;
}))
(lib.genAttrs (linux64BitSystems) (system: {
staticBuild = self.packages.${system}.nix-static;
}))
{
x86_64-linux = {
# TODO add more cross/static release targets?
crossBuild.aarch64-linux = self.packages.x86_64-linux.nix-aarch64-linux;
# TODO wire up a nixos installer test with that lix and
# run it, once nixpkgs can actually do that (again). :/
# # nix build .#nixosTests.installer.{btrfsSimple,luksroot,lvm,simple,switchToFlake}
};
}
];
# NOTE *do not* add fresh derivations to checks, always add them to
# hydraJobs first (so CI will pick them up) and only link them here
checks = forAvailableSystems (
@ -385,7 +409,7 @@
nix = pkgs.callPackage ./package.nix {
inherit stdenv officialRelease versionSuffix;
busybox-sandbox-shell = pkgs.busybox-sandbox-shell or pkgs.default-busybox-sandbox;
internalApiDocs = true;
internalApiDocs = false;
};
pre-commit = self.hydraJobs.pre-commit.${pkgs.system} or { };
in
@ -413,7 +437,7 @@
makeShell pkgs pkgs.stdenv
))
// {
default = self.devShells.${system}.native-stdenvPackages;
default = self.devShells.${system}.native-clangStdenvPackages;
}
);
};

View file

@ -9,8 +9,8 @@ clean:
rm -rf build
# Prepare meson for building
setup:
meson setup build --prefix="$PWD/outputs/out"
setup *OPTIONS:
meson setup build --prefix="$PWD/outputs/out" $mesonFlags {{ OPTIONS }}
# Build lix
build *OPTIONS:

114
lix-doc/Cargo.lock generated
View file

@ -9,13 +9,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
[[package]]
name = "cbitset"
version = "0.2.0"
name = "countme"
version = "3.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29b6ad25ae296159fb0da12b970b2fe179b234584d7cd294c891e2bbb284466b"
dependencies = [
"num-traits",
]
checksum = "7704b5fdd17b18ae31c4c1da5a2e0305a2bf17b5249300a9ee9ed7b72114c636"
[[package]]
name = "dissimilar"
@ -33,19 +30,26 @@ dependencies = [
"once_cell",
]
[[package]]
name = "hashbrown"
version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
[[package]]
name = "lix-doc"
version = "0.0.1"
dependencies = [
"expect-test",
"rnix",
"rowan",
]
[[package]]
name = "num-traits"
version = "0.2.18"
name = "memoffset"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da0df0e5185db44f69b44f26786fe401b6c293d1907744beaa7fa62b2e5a517a"
checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a"
dependencies = [
"autocfg",
]
@ -56,44 +60,26 @@ version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
[[package]]
name = "proc-macro2"
version = "1.0.79"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e835ff2298f5721608eb1a980ecaee1aef2c132bf95ecc026a11b7bf3c01c02e"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "291ec9ab5efd934aaf503a6466c5d5251535d108ee747472c3977cc5acc868ef"
dependencies = [
"proc-macro2",
]
[[package]]
name = "rnix"
version = "0.8.1"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0a9b645f0edba447dbfc6473dd22999f46a1d00ab39e777a2713a1cf34a1597b"
checksum = "bb35cedbeb70e0ccabef2a31bcff0aebd114f19566086300b8f42c725fc2cb5f"
dependencies = [
"cbitset",
"rowan",
]
[[package]]
name = "rowan"
version = "0.9.1"
version = "0.15.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ea7cadf87a9d8432e85cb4eb86bd2e765ace60c24ef86e79084dcae5d1c5a19"
checksum = "32a58fa8a7ccff2aec4f39cc45bf5f985cec7125ab271cf681c279fd00192b49"
dependencies = [
"countme",
"hashbrown",
"memoffset",
"rustc-hash",
"smol_str",
"text_unit",
"thin-dst",
"text-size",
]
[[package]]
@ -103,59 +89,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
[[package]]
name = "serde"
version = "1.0.197"
name = "text-size"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fb1c873e1b9b056a4dc4c0c198b24c3ffa059243875552b2bd0933b1aee4ce2"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.197"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7eb0b34b42edc17f6b7cac84a52a1c5f0e1bb2227e997ca9011ea3dd34e8610b"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "smol_str"
version = "0.1.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fad6c857cbab2627dcf01ec85a623ca4e7dcb5691cbaa3d7fb7653671f0d09c9"
dependencies = [
"serde",
]
[[package]]
name = "syn"
version = "2.0.53"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7383cd0e49fff4b6b90ca5670bfd3e9d6a733b3f90c686605aa7eec8c4996032"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "text_unit"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "20431e104bfecc1a40872578dbc390e10290a0e9c35fffe3ce6f73c15a9dbfc2"
[[package]]
name = "thin-dst"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db3c46be180f1af9673ebb27bc1235396f61ef6965b3fe0dbb2e624deb604f0e"
[[package]]
name = "unicode-ident"
version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b"
checksum = "f18aa187839b2bdb1ad2fa35ead8c4c2976b64e4363c386d45ac0f7ee85c9233"

View file

@ -1,6 +1,6 @@
[package]
description = "Nix function documentation tool, stripped down into a library"
edition = "2018"
edition = "2021"
name = "lix-doc"
version = "0.0.1"
license = "BSD-2-Clause OR MIT"
@ -12,7 +12,9 @@ repository = "https://github.com/lf-/nix-doc"
crate_type = ["staticlib"]
[dependencies]
rnix = "0.8.0"
rnix = "0.11.0"
# Necessary because rnix fails to export a critical trait (Rowan's AstNode).
rowan = "0.15.0"
[dev-dependencies]
expect-test = "1.1.0"

View file

@ -1,5 +1,5 @@
// SPDX-FileCopyrightText: 2024 Jade Lovelace
//
// SPDX-FileCopyrightText: 2024 Lunaphied
// SPDX-License-Identifier: BSD-2-Clause OR MIT
//! library components of nix-doc
@ -7,13 +7,16 @@ pub mod pprint;
use crate::pprint::pprint_args;
use rnix::types::{Lambda, TypedNode};
use rnix::SyntaxKind::*;
use rnix::{NodeOrToken, SyntaxNode, TextUnit, WalkEvent};
use rnix::ast::{self, Lambda};
use rnix::{NodeOrToken, SyntaxKind};
use rnix::SyntaxNode;
// Needed because rnix fucked up and didn't reexport this, oops.
use rowan::ast::AstNode;
use std::ffi::{CStr, CString};
use std::fs;
use std::iter;
use std::os::raw::c_char;
use std::panic;
@ -23,66 +26,104 @@ use std::{fmt::Display, str};
pub type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;
const DOC_INDENT: usize = 3;
struct SearchResult {
/// Name of the function
identifier: String,
/// Dedented documentation comments
/// Dedented documentation comment
doc: String,
/// Parameter block for the function
param_block: String,
}
fn find_pos(file: &str, line: usize, col: usize) -> usize {
let mut lines = 1;
let mut line_start = 0;
let mut it = file.chars().enumerate().peekable();
while let Some((count, ch)) = it.next() {
if ch == '\n' || ch == '\r' {
lines += 1;
let addend = if ch == '\r' && it.peek().map(|x| x.1) == Some('\n') {
it.next();
1
} else {
0
};
line_start = count + addend;
}
let col_diff = ((count as i32) - (line_start as i32)).abs() as usize;
if lines == line && col_diff == col {
return count;
}
}
unreachable!();
}
impl SearchResult {
fn format<P: Display>(&self, filename: P, line: usize) -> String {
format!(
"**Synopsis:** `{}` = {}\n\n{}\n\n# {}",
self.identifier.as_str(),
self.param_block,
indented(&self.doc, DOC_INDENT),
self.doc,
format!("{}:{}", filename, line).as_str(),
)
}
}
/// Emits a string `s` indented by `indent` spaces
fn indented(s: &str, indent: usize) -> String {
let indent_s = iter::repeat(' ').take(indent).collect::<String>();
s.split('\n')
.map(|line| indent_s.clone() + line)
.collect::<Vec<_>>()
.join("\n")
/// Converts Nix compatible line endings (Nix accepts `\r`, `\n`, *and* `\r\n` as endings), to
/// standard `\n` endings for use within Rust land.
fn convert_endings(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut it = s.chars().peekable();
while let Some(ch) = it.next() {
if ch == '\n' || ch == '\r' {
out.push('\n');
if ch == '\r' && it.peek().map(|&c| c == '\n').unwrap_or(false) {
// Consume `\n` in `\r\n`.
it.next();
}
} else {
out.push(ch);
}
}
out
}
/// Converts the position information from Lix itself into an character index into the file itself.
/// Expects an input string that's already had it's line endings normalized.
///
/// Note that this returns a *byte* offset, not a character offset.
fn find_pos(s: &str, line: usize, col: usize) -> usize {
// Nix line positions are 1-indexed.
let mut lines = 1;
for (byte_pos, ch) in s.char_indices() {
// If we find a newline, increase the line count.
if ch == '\n' {
lines += 1;
}
// We've arrived at the correct line.
if lines == line {
// Column position is 1-indexed, and it's a *byte* offset, because Nix doesn't actually
// support UTF-8. Rust does though, so we need to convert to a proper byte index to
// match rnix. Lix also doesn't consider the line endings part of the column offset so
// we implicitly add one to advance to the character *after* that.
return byte_pos + col;
}
}
// If things never match that should be literally impossible.
unreachable!();
}
/// Represents a forwarded token from rnix's AST over to lix-doc.
#[derive(Debug, Clone)]
enum DocToken {
Comment(String),
Whitespace(String),
}
/// Determine if a given token string contains more than two newlines, this is used to determine when
/// we hit blank lines between comments indicating a contextually unrelated comment.
fn has_empty_line(tok: &DocToken) -> bool {
// It's either solely whitespace with two newlines inside somewhere, or it's
// contained inside a comment token and we don't want to count that as empty.
if let DocToken::Whitespace(s) = tok {
s.chars().filter(|&c| c == '\n').take(2).count() == 2
} else {
false
}
}
/// Cleans up a single line, erasing prefix single line comments but preserving indentation
fn cleanup_single_line<'a>(s: &'a str) -> &'a str {
// NOTE: We have a bit of a conflict of interest problem here due to the inconsistent format of
// doc comments. Some doc comments will use a series of single line comments that may then contain `*`
// characters to represent a list. Some will be multiline comments that don't prefix individual lines
// with `*`, only using them for lists directly, and some will prefix lines with `*` as a leading
// character to mark the block. There's no way to disambiguate all three, but we do our best to
// make the common case pretty.
fn cleanup_single_line(s: &str) -> &str {
let mut cmt_new_start = 0;
let mut iter = s.char_indices().peekable();
while let Some((idx, ch)) = iter.next() {
@ -90,7 +131,9 @@ fn cleanup_single_line<'a>(s: &'a str) -> &'a str {
let (_, next_ch) = iter.peek().unwrap_or(&(0, '\n'));
// if we find a character, save the byte position after it as our new string start
if ch == '#' || (ch == '*' && next_ch.is_whitespace()) {
// This has special handling for `>` because some Nixpkgs documentation has `*>` right
// after the start of their doc comments, and we want to strip the `*` still.
if ch == '#' || (ch == '*' && (*next_ch == '>' || next_ch.is_whitespace())) {
cmt_new_start = idx + 1;
break;
}
@ -103,15 +146,12 @@ fn cleanup_single_line<'a>(s: &'a str) -> &'a str {
&s[cmt_new_start..]
}
/// Erases indents in comments. This is *almost* a normal dedent function, but it starts by looking
/// at the second line if it can.
/// Erases indents in comments based on the indentation of the first line.
fn dedent_comment(s: &str) -> String {
let mut whitespaces = 0;
let mut lines = s.lines();
let first = lines.next();
// scan for whitespace
for line in lines.chain(first) {
for line in s.lines() {
let line_whitespace = line.chars().take_while(|ch| ch.is_whitespace()).count();
if line_whitespace != line.len() {
@ -121,16 +161,6 @@ fn dedent_comment(s: &str) -> String {
}
}
// maybe the first considered line we found was indented further, so let's look for more lines
// that might have a shorter indent. In the case of one line, do nothing.
for line in s.lines().skip(1) {
let line_whitespace = line.chars().take_while(|ch| ch.is_whitespace()).count();
if line_whitespace != line.len() {
whitespaces = line_whitespace.min(whitespaces);
}
}
// delete up to `whitespaces` whitespace characters from each line and reconstitute the string
let mut out = String::new();
for line in s.lines() {
@ -143,69 +173,163 @@ fn dedent_comment(s: &str) -> String {
out
}
/// Deletes whitespace and leading comment characters
/// Takes a series of comment and whitespace strings and output a clean single block of text to use
/// as the output documentation comment block.
///
/// Oversight we are choosing to ignore: if you put # characters at the beginning of lines in a
/// multiline comment, they will be deleted.
fn cleanup_comments<S: AsRef<str>, I: DoubleEndedIterator<Item = S>>(comment: &mut I) -> String {
/// This function expects to be given the tokens in reverse order (proceeding upwards from the
/// first comment above the definitions), this allows us to properly enforce the below conditions.
/// The output from this function will be reordered and ready for display.
///
/// The two types of documentation comments we expect are:
///
/// - A single multiline comment not whitespace separated from the start.
/// - A series of back to back single line comments not separated by whitespace.
///
/// Any other combination will be filtered out.
///
/// Once an empty line is encountered, we know no more valid documentation comments remain and stop.
fn cleanup_comments<I: Iterator<Item = DocToken>>(tokens: &mut I) -> String {
// Keep track of when we've found a single line and multiline comment, we use this to
// only process a single multiline or back to back single lines.
let mut found_single_line = false;
// Comments that have survived our filtering phase and should be cleaned up.
let mut valid = vec![];
// Filter out comments that don't meet the characteristics of documentation comments.
for tok in tokens {
if has_empty_line(&tok) {
// Take tokens until we hit whitespace containing an empty line.
break;
}
// Only care about comments from this point on.
if let DocToken::Comment(comment) = tok {
// Now determine if it's a single line comment.
let is_single_line = comment.starts_with('#');
// We've found a single line comment if we've found one before or we just found one.
found_single_line |= is_single_line;
// What we do next is only special when we hit a multiline comment.
if !is_single_line {
// If we've hit a multiline comment as our first comment, take that one alone.
if !found_single_line {
// Otherwise we've hit a multiline comment immediately and this is our
// one and only doc comment to worry about.
valid.push(comment);
}
// Otherwise we've hit a multiline comment after single line comments, in either
// case this means we're done processing comments.
break;
}
// Otherwise this is a new single line comment to push to the stack.
valid.push(comment);
}
}
// Cleanup comments for user consumption.
dedent_comment(
&comment
&valid
.into_iter()
.rev()
.map(|small_comment| {
small_comment
.as_ref()
// space before multiline start
.trim_start()
// multiline starts
// Trim off start of multiline comments.
.trim_start_matches("/*")
// trailing so we can grab multiline end
.trim_end()
// multiline ends
// Trim off end of multiline comments.
.trim_end_matches("*/")
// extra space that was in the multiline
// Trim off any internal whitespace that's trapped inside comments themselves.
.trim()
// Split comments by newlines to extract lines of multiline comments.
.split('\n')
// erase single line comments and such
// Cleanup single line comments and a few more tweaks for multiline comments.
.map(cleanup_single_line)
.collect::<Vec<_>>()
// Reconstruct the multiline comment's whitespace.
.join("\n")
})
.collect::<Vec<_>>()
.join("\n"),
// We've found that when multiple back to back single line comments are used in Nixpkgs,
// they make more sense to represent as if someone inserted line breaks into the Markdown
// properly, so we join them with linebreaks that markdown will pass through.
.join("\n\n"),
)
}
/// Get the docs for a specific function
/// Get the docs for a specific function.
// TODO: Improve error reporting?
pub fn get_function_docs(filename: &str, line: usize, col: usize) -> Option<String> {
let content = fs::read(filename).ok()?;
let decoded = str::from_utf8(&content).ok()?;
let decoded = convert_endings(str::from_utf8(&content).ok()?);
let pos = find_pos(&decoded, line, col);
let rowan_pos = TextUnit::from_usize(pos);
let tree = rnix::parse(decoded);
let rowan_pos = rnix::TextSize::from(pos as u32);
// The minimum length of a lambda is 4 characters and thus the range we're looking for must be
// at least 4 characters long `_: 3` being an example of a minimal length lambda.
let rowan_range = rnix::TextRange::at(rowan_pos, 4.into());
// Parse the file using rnix.
let root = rnix::Root::parse(&decoded).ok().ok()?;
// Extract the inner expression that represents the Root node and extract the top level expression.
let expr = root.expr()?;
// There are two cases we have to be able to handle
// 1. A straightforward definition with an attrset binding to a lambda that's defined inline.
// 2. A lambda defined in a standalone file where the attrset binding imports that file directly.
// The latter case will not be able to find the binding so we must be able to handle not finding it.
// Find the deepest node or token that covers the position given by Lix.
let covering = expr.syntax().covering_element(rowan_range);
// Climb up until we find the lambda node that contains that token.
let mut lambda = None;
for node in tree.node().preorder() {
match node {
WalkEvent::Enter(n) => {
if n.text_range().start() >= rowan_pos && n.kind() == NODE_LAMBDA {
lambda = Lambda::cast(n);
break;
}
}
WalkEvent::Leave(_) => (),
for ancestor in covering.ancestors() {
if ancestor.kind() == SyntaxKind::NODE_LAMBDA {
lambda = Some(ancestor);
break;
}
}
let lambda = lambda?;
let res = visit_lambda("func".to_string(), &lambda);
Some(res.format(filename, line))
// There is literally always a lambda or something has gone very very wrong.
let lambda =
ast::Lambda::cast(
lambda.expect("no lambda found; what.")
) .expect("not a rnix::ast::Lambda; what.");
// Search up, hopefully to find the binding so we can get the identifier name.
// TODO: Just provide this directly from the C++ code to make it possible to always have the correct identifier.
let mut binding = None;
for ancestor in lambda.syntax().ancestors() {
if ancestor.kind() == SyntaxKind::NODE_ATTRPATH_VALUE {
binding = Some(ancestor);
}
}
// Convert the binding to an identifier if it was found, otherwise use a placeholder.
let identifier;
identifier = match binding.clone() {
Some(binding) => ast::AttrpathValue::cast(binding)
.expect("not an rnix::ast::AttrpathValue; what")
.attrpath()
.expect("AttrpathValue has no attrpath; what.")
.to_string(),
_ => "<unknown binding>".to_string(),
};
// Find all the comments on the binding or the lambda if we have to fall back.
let comment_node = binding.as_ref().unwrap_or(lambda.syntax());
let comment = find_comment(comment_node).unwrap_or_else(String::new);
// And display them properly for the markdown function in Lix.
Some(visit_lambda(identifier, comment, &lambda).format(filename, line))
}
fn visit_lambda(name: String, lambda: &Lambda) -> SearchResult {
fn visit_lambda(name: String, comment: String, lambda: &Lambda) -> SearchResult {
// grab the arguments
let param_block = pprint_args(&lambda);
// find the doc comment
let comment = find_comment(lambda.node().clone()).unwrap_or_else(|| "".to_string());
let param_block = pprint_args(lambda);
SearchResult {
identifier: name,
@ -214,39 +338,47 @@ fn visit_lambda(name: String, lambda: &Lambda) -> SearchResult {
}
}
fn find_comment(node: SyntaxNode) -> Option<String> {
let mut node = NodeOrToken::Node(node);
let mut comments = Vec::new();
loop {
loop {
if let Some(new) = node.prev_sibling_or_token() {
node = new;
break;
} else {
node = NodeOrToken::Node(node.parent()?);
}
}
fn find_comment(node: &SyntaxNode) -> Option<String> {
let mut it = node
.siblings_with_tokens(rowan::Direction::Prev)
// Skip ourselves as we're always the first token returned.
.skip(1)
.peekable();
match node.kind() {
TOKEN_COMMENT => match &node {
NodeOrToken::Token(token) => comments.push(token.text().clone()),
NodeOrToken::Node(_) => unreachable!(),
},
// This stuff is found as part of `the-fn = f: ...`
// here: ^^^^^^^^
NODE_KEY | TOKEN_ASSIGN => (),
t if t.is_trivia() => (),
_ => break,
}
// Consume up to one whitespace token before the first comment. There might not always be
// whitespace such as the (rather unusual) case of `/* meow */x = a: 3`.
if matches!(it.peek(), Some(NodeOrToken::Token(token)) if token.kind() == SyntaxKind::TOKEN_WHITESPACE) {
it.next();
}
let doc = cleanup_comments(&mut comments.iter().map(|c| c.as_str()));
Some(doc).filter(|it| !it.is_empty())
let comments = it.map_while(|element| match element {
NodeOrToken::Token(token) => {
match token.kind() {
// Map the tokens we're interested in to our internal token type.
SyntaxKind::TOKEN_COMMENT => Some(DocToken::Comment(token.text().to_owned())),
SyntaxKind::TOKEN_WHITESPACE => {
Some(DocToken::Whitespace(token.text().to_owned()))
}
// If we hit a different token type, we know we've gone past relevant comments
// and should stop.
_ => None,
}
}
// If we hit a node entry we've definitely gone past comments that would be related to
// this node and we should retreat.
_ => None,
});
// For the curious, `into_iter()` here consumes the binding producing an owned value allowing us to avoid
// making the original binding mutable, we don't reuse it later so this is a cute way to handle it, though
// there's probably a better way we just can't remember.
Some(cleanup_comments(&mut comments.into_iter())).filter(|c| !c.is_empty())
}
/// Get the docs for a function in the given file path at the given file position and return it as
/// a C string pointer
#[no_mangle]
pub extern "C" fn nd_get_function_docs(
pub extern "C" fn lixdoc_get_function_docs(
filename: *const c_char,
line: usize,
col: usize,
@ -269,9 +401,9 @@ pub extern "C" fn nd_get_function_docs(
.unwrap_or(ptr::null())
}
/// Call this to free a string from nd_get_function_docs
/// Call this to free a string from `lixdoc_get_function_docs`.
#[no_mangle]
pub extern "C" fn nd_free_string(s: *const c_char) {
pub extern "C" fn lixdoc_free_string(s: *const c_char) {
unsafe {
// cast note: this cast is turning something that was cast to const
// back to mut
@ -283,35 +415,57 @@ pub extern "C" fn nd_free_string(s: *const c_char) {
mod tests {
use super::*;
#[test]
fn test_line_conversion() {
let fakefile = "abc\rdef\r\nghi";
assert_eq!(convert_endings(fakefile), "abc\ndef\nghi");
}
#[test]
fn test_bytepos() {
let fakefile = "abc\ndef\nghi";
assert_eq!(find_pos(fakefile, 2, 2), 5);
}
#[test]
fn test_bytepos_unusual() {
let fakefile = convert_endings("abc\rdef\r\nghi");
assert_eq!(find_pos(&fakefile, 2, 2), 5);
assert_eq!(find_pos(&fakefile, 3, 2), 9);
}
/// This test is to check that we correctly resolve byte positions even when inconsistent with
/// character positions.
#[test]
fn test_bytepos_cursed() {
let fakefile = "abc\rdef\r\nghi";
assert_eq!(find_pos(fakefile, 2, 2), 5);
assert_eq!(find_pos(fakefile, 3, 2), 10);
let fakefile = "hello\nwórld";
// Try to find the position of the `r` after world, which will be wrong if we don't handle
// UTF-8 properly.
let pos = find_pos(&fakefile, 2, 4);
dbg!(&fakefile[pos..]);
assert_eq!(pos, 9)
}
#[test]
fn test_comment_stripping() {
let ex1 = ["/* blah blah blah\n foooo baaar\n blah */"];
let ex1 = [DocToken::Comment(
"/* blah blah blah\n foooo baaar\n blah */".to_string(),
)];
assert_eq!(
cleanup_comments(&mut ex1.iter()),
"blah blah blah\n foooo baaar\nblah"
cleanup_comments(&mut ex1.into_iter()),
"blah blah blah\n foooo baaar\n blah"
);
let ex2 = ["# a1", "# a2", "# aa"];
assert_eq!(cleanup_comments(&mut ex2.iter()), "aa\n a2\na1");
let ex2 = ["# a1", "# a2", "# aa"]
.into_iter()
.map(|s| DocToken::Comment(s.to_string()));
assert_eq!(cleanup_comments(&mut ex2.into_iter()), "aa\n\n a2\n\na1");
}
#[test]
fn test_dedent() {
let ex1 = "a\n b\n c\n d";
assert_eq!(dedent_comment(ex1), "a\nb\nc\n d");
assert_eq!(dedent_comment(ex1), ex1);
let ex2 = "a\nb\nc";
assert_eq!(dedent_comment(ex2), ex2);
let ex3 = " a\n b\n\n c";
@ -335,4 +489,31 @@ mod tests {
let ex1 = " **Foo**:";
assert_eq!(cleanup_single_line(ex1), ex1);
}
// TODO: Next CL
//#[test]
//fn comment_test_complex() {
// let testcase = r#"
// rec {
// /*
// Hello
// 23
// This is a comment.
// this is another comment.
// and this is a third comment.
// Way
// go
// */
// meow = { g }: {a, b ? 4, ...}: g: c: 5;
// # And another comment.
// cat = 34;
// # inner layer.
// "inner-layer" = outer: meow;
// }
// "#;
// // Need to find the location of the lambda, we do a quick hack.
// let location = dbg!(testcase.find("{ g }").unwrap() as u32);
//
// //get_function_docs(filename, line, col)
//}
}

View file

@ -1,36 +1,59 @@
// SPDX-FileCopyrightText: 2024 Jade Lovelace
//
// SPDX-FileCopyrightText: 2024 Lunaphied
// SPDX-License-Identifier: BSD-2-Clause OR MIT
use rnix::types::{Lambda, TypedNode};
use rnix::SyntaxKind::*;
use rnix::ast::{Expr, Lambda};
use rowan::ast::AstNode;
/// Pretty-prints the arguments to a function
pub fn pprint_args(lambda: &Lambda) -> String {
// TODO: handle docs directly on NODE_IDENT args (uncommon case)
let mut lambda = lambda.clone();
let mut depth = 0;
let mut out = String::new();
loop {
let arg = lambda.arg().unwrap();
match arg.kind() {
NODE_IDENT => {
out += &format!("*{}*", &arg.to_string());
out.push_str(": ");
let body = lambda.body().unwrap();
if body.kind() == NODE_LAMBDA {
lambda = Lambda::cast(body).unwrap();
} else {
break;
let arg = lambda.param().unwrap();
for child in arg.syntax().children_with_tokens() {
//dbg!(child.kind());
match child {
rowan::NodeOrToken::Node(node) => {
out.push_str(&node.text().to_string());
if node.kind() == rnix::SyntaxKind::NODE_PAT_ENTRY {
out.push_str(&",\n");
}
}
rowan::NodeOrToken::Token(token) => {
use rnix::SyntaxKind::{
TOKEN_COMMENT, TOKEN_ELLIPSIS, TOKEN_L_BRACE, TOKEN_QUESTION, TOKEN_R_BRACE,
};
match token.kind() {
TOKEN_COMMENT | TOKEN_ELLIPSIS | TOKEN_QUESTION | TOKEN_L_BRACE
| TOKEN_R_BRACE => {
//dbg!(&token);
out.push_str(&token.text().to_string());
if token.kind() == TOKEN_COMMENT {
out.push('\n');
}
}
_ => {}
}
//out.push_str(&token.text().to_string());
}
}
NODE_PATTERN => {
out += &format!("*{}*", &arg.to_string());
out.push_str(": ");
break;
}
t => {
unreachable!("unhandled arg type {:?}", t);
}
out.push_str(": ");
let body = lambda.body().unwrap();
if let Expr::Lambda(inner) = body {
lambda = inner;
// If we recurse we want the next line of recursion to be indented and on a new line.
out.push('\n');
for _ in 0..=depth {
out.push('\t');
}
depth += 1;
} else {
// If we don't find an inner lambda we're done with argument handling.
break;
}
}
out.push_str("...");

View file

@ -20,6 +20,8 @@ SIGNIFICANCECES = {
# This is just hardcoded for better validation. If you think there should be
# more of them, feel free to add more.
#
# Please update doc/manual/src/contributing/hacking.md if you do. Thanks~
CATEGORIES = [
'Breaking Changes',
'Features',

View file

@ -1,4 +1,4 @@
#!/bin/sh
#!/usr/bin/env bash
# Generates a report of build time based on a meson build using -ftime-trace in
# Clang.

View file

@ -142,6 +142,16 @@ else
cpp_pch = []
endif
# gcc 12 is known to miscompile some coroutine-based code quite horribly,
# causing (among other things) copies of move-only objects and the double
# frees one would expect when the objects are unique_ptrs. these problems
# often show up as memory corruption when nesting generators (since we do
# treat generators like owned memory) and will cause inexplicable crashs.
assert(
cxx.get_id() != 'gcc' or cxx.version().version_compare('>=13'),
'GCC 12 and earlier are known to miscompile lix coroutines, use GCC 13 or clang.'
)
# Translate some historical and Mesony CPU names to Lixy CPU names.
# FIXME(Qyriad): the 32-bit x86 code is not tested right now, because cross compilation for Lix
@ -194,7 +204,7 @@ configdata += {
'HAVE_BOEHMGC': boehm.found().to_int(),
}
boost = dependency('boost', required : true, modules : ['context', 'coroutine', 'container'])
boost = dependency('boost', required : true, modules : ['container'])
# cpuid only makes sense on x86_64
cpuid_required = is_x64 ? get_option('cpuid') : false
@ -287,6 +297,14 @@ gtest = [
toml11 = dependency('toml11', version : '>=3.7.0', required : true, method : 'cmake')
pegtl = dependency(
'pegtl',
version : '>=3.2.7',
required : true,
method : 'cmake',
modules : [ 'taocpp::pegtl' ],
)
nlohmann_json = dependency('nlohmann_json', required : true)
# lix-doc is a Rust project provided via buildInputs and unfortunately doesn't have any way to be detected.
@ -335,8 +353,6 @@ endif
# that busybox sh won't run busybox applets as builtins (which would break our sandbox).
lsof = find_program('lsof', native : true)
bison = find_program('bison', native : true)
flex = find_program('flex', native : true)
# This is how Nix does generated headers...
# other instances of header generation use a very similar command.
@ -426,8 +442,10 @@ add_project_arguments(
'-Wimplicit-fallthrough',
'-Werror=switch',
'-Werror=switch-enum',
'-Werror=unused-result',
'-Wdeprecated-copy',
'-Wignored-qualifiers',
'-Werror=suggest-override',
# Enable assertions in libstdc++ by default. Harmless on libc++. Benchmarked
# at ~1% overhead in `nix search`.
#
@ -437,7 +455,9 @@ add_project_arguments(
language : 'cpp',
)
if cxx.get_id() in ['gcc', 'clang']
# We turn off the production UBSan if the slower dev UBSan is requested, to
# give better diagnostics.
if cxx.get_id() in ['gcc', 'clang'] and 'undefined' not in get_option('b_sanitize')
# 2024-03-24: jade benchmarked the default sanitize reporting in clang and got
# a regression of about 10% on hackage-packages.nix with clang. So we are trapping instead.
#
@ -452,6 +472,11 @@ if cxx.get_id() in ['gcc', 'clang']
add_project_arguments(sanitize_args, language: 'cpp')
add_project_link_arguments(sanitize_args, language: 'cpp')
endif
# Clang's default of -no-shared-libsan on Linux causes link errors; on macOS it defaults to shared.
# GCC defaults to shared libsan so is fine.
if cxx.get_id() == 'clang' and get_option('b_sanitize') != ''
add_project_link_arguments('-shared-libsan', language : 'cpp')
endif
add_project_link_arguments('-pthread', language : 'cpp')
if cxx.get_linker_id() in ['ld.bfd', 'ld.gold']

View file

@ -1,50 +0,0 @@
#!/usr/bin/env bash
# Meson will call this with an absolute path to Bash.
# The shebang is just for convenience.
# The parser and lexer tab are generated via custom Meson targets in src/libexpr/meson.build,
# but Meson doesn't support marking only part of a target for install. The generation creates
# both headers (parser-tab.hh, lexer-tab.hh) and source files (parser-tab.cc, lexer-tab.cc),
# and we definitely want the former installed, but not the latter. This script is added to
# Meson's install steps to correct this, as the logic for it is just complex enough to
# warrant separate and careful handling, because both Meson's configured include directory
# may or may not be an absolute path, and DESTDIR may or may not be set at all, but can't be
# manipulated in Meson logic.
set -euo pipefail
echo "cleanup-install: removing Meson-placed C++ sources from dest includedir"
if [[ "${1/--help/}" != "$1" ]]; then
echo "cleanup-install: this script should only be called from the Meson build system"
exit 1
fi
# Ensure the includedir was passed as the first argument
# (set -u will make this fail otherwise).
includedir="$1"
# And then ensure that first argument is a directory that exists.
if ! [[ -d "$1" ]]; then
echo "cleanup-install: this script should only be called from the Meson build system"
echo "argv[1] (${1@Q}) is not a directory"
exit 2
fi
# If DESTDIR environment variable is set, prepend it to the include dir.
# Unfortunately, we cannot do this on the Meson side. We do have an environment variable
# `MESON_INSTALL_DESTDIR_PREFIX`, but that will not refer to the include directory if
# includedir has been set separately, which Lix's split-output derivation does.
# We also cannot simply do an inline bash conditional like "${DESTDIR:=}" or similar,
# because we need to specifically *join* DESTDIR and includedir with a slash, and *not*
# have a slash if DESTDIR isn't set at all, since $includedir could be a relative directory.
# Finally, DESTDIR is only available to us as an environment variable in these install scripts,
# not in Meson logic.
# Therefore, our best option is to have Meson pass this script the configured includedir,
# and perform this dance with it and $DESTDIR.
if [[ -n "${DESTDIR:-}" ]]; then
includedir="$DESTDIR/$includedir"
fi
# Intentionally not using -f.
# If these files don't exist then our assumptions have been violated and we should fail.
rm -v "$includedir/lix/libexpr/parser-tab.cc" "$includedir/lix/libexpr/lexer-tab.cc"

23
misc/pegtl.nix Normal file
View file

@ -0,0 +1,23 @@
{
stdenv,
cmake,
ninja,
fetchFromGitHub,
}:
stdenv.mkDerivation {
pname = "pegtl";
version = "3.2.7";
src = fetchFromGitHub {
repo = "PEGTL";
owner = "taocpp";
rev = "refs/tags/3.2.7";
hash = "sha256-IV5YNGE4EWVrmg2Sia/rcU8jCuiBynQGJM6n3DCWTQU=";
};
nativeBuildInputs = [
cmake
ninja
];
}

View file

@ -106,7 +106,7 @@ pre-commit-run {
};
treefmt = {
enable = true;
settings.formatters = [ pkgs.nixfmt ];
settings.formatters = [ pkgs.nixfmt-rfc-style ];
};
};
}

View file

@ -1,6 +1,6 @@
[Unit]
Description=Nix Daemon
Documentation=man:nix-daemon https://nixos.org/manual
Documentation=man:nix-daemon https://docs.lix.systems/manual/lix/stable
RequiresMountsFor=@storedir@
RequiresMountsFor=@localstatedir@
RequiresMountsFor=@localstatedir@/nix/db

View file

@ -5,29 +5,33 @@
system,
}:
let
installerClosureInfo = buildPackages.closureInfo {
rootPaths = [
nix
cacert
];
};
rootPaths = [
nix
cacert
];
installerClosureInfo = buildPackages.closureInfo { inherit rootPaths; };
meta.description = "Distribution-independent Lix bootstrap binaries for ${system}";
in
buildPackages.runCommand "lix-binary-tarball-${nix.version}" { inherit meta; } ''
cp ${installerClosureInfo}/registration $TMPDIR/reginfo
buildPackages.runCommand "lix-binary-tarball-${nix.version}"
{
inherit meta;
passthru.rootPaths = rootPaths;
}
''
cp ${installerClosureInfo}/registration $TMPDIR/reginfo
dir=lix-${nix.version}-${system}
fn=$out/$dir.tar.xz
mkdir -p $out/nix-support
echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products
tar cvfJ $fn \
--owner=0 --group=0 --mode=u+rw,uga+r \
--mtime='1970-01-01' \
--absolute-names \
--hard-dereference \
--transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
--transform "s,$NIX_STORE,$dir/store,S" \
$TMPDIR/reginfo \
$(cat ${installerClosureInfo}/store-paths)
''
dir=lix-${nix.version}-${system}
fn=$out/$dir.tar.xz
mkdir -p $out/nix-support
echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products
tar cvfJ $fn \
--owner=0 --group=0 --mode=u+rw,uga+r \
--mtime='1970-01-01' \
--absolute-names \
--hard-dereference \
--transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
--transform "s,$NIX_STORE,$dir/store,S" \
$TMPDIR/reginfo \
$(cat ${installerClosureInfo}/store-paths)
''

View file

@ -1,65 +0,0 @@
# Copy of `nixfmt-rfc-style` vendored from `nixpkgs` master:
# https://github.com/NixOS/nixpkgs/blob/ab6071eb54cc9b66dda436111d4f569e4e56cbf4/pkgs/by-name/ni/nixfmt-rfc-style/package.nix
{
haskell,
haskellPackages,
fetchFromGitHub,
}:
let
inherit (haskell.lib.compose) justStaticExecutables;
raw-pkg = haskellPackages.callPackage (
{
mkDerivation,
base,
cmdargs,
directory,
fetchzip,
filepath,
lib,
megaparsec,
mtl,
parser-combinators,
safe-exceptions,
scientific,
text,
transformers,
unix,
}:
mkDerivation {
pname = "nixfmt";
version = "0.6.0-unstable-2024-03-14";
src = fetchFromGitHub {
owner = "serokell";
repo = "nixfmt";
rev = "8d13b593fa8d8d6e5075f541f3231222a08e84df";
hash = "sha256-HtXvzmfN4wk45qiKZ7V+/5WBV7jnTHfd7iBwF4XGl64=";
};
isLibrary = true;
isExecutable = true;
libraryHaskellDepends = [
base
megaparsec
mtl
parser-combinators
scientific
text
transformers
];
executableHaskellDepends = [
base
cmdargs
directory
filepath
safe-exceptions
text
unix
];
jailbreak = true;
homepage = "https://github.com/serokell/nixfmt";
description = "An opinionated formatter for Nix";
license = lib.licenses.mpl20;
mainProgram = "nixfmt";
}
) { };
in
justStaticExecutables raw-pkg

View file

@ -10,17 +10,16 @@
boehmgc-nix ? __forDefaults.boehmgc-nix,
boehmgc,
nlohmann_json,
bison,
build-release-notes ? __forDefaults.build-release-notes,
boost,
brotli,
bzip2,
callPackage,
cmake,
curl,
doxygen,
editline-lix ? __forDefaults.editline-lix,
editline,
flex,
git,
gtest,
jq,
@ -36,6 +35,7 @@
meson,
ninja,
openssl,
pegtl ? __forDefaults.pegtl,
pkg-config,
python3,
rapidcheck,
@ -62,22 +62,16 @@
__forDefaults ? {
canRunInstalled = stdenv.buildPlatform.canExecute stdenv.hostPlatform;
boehmgc-nix = (boehmgc.override { enableLargeConfig = true; }).overrideAttrs {
patches = [
# We do *not* include prev.patches (which doesn't exist in normal pkgs.boehmgc anyway)
# because if the caller of this package passed a patched boehm as `boehmgc` instead of
# `boehmgc-nix` then this will almost certainly have duplicate patches, which means
# the patches won't apply and we'll get a build failure.
./boehmgc-coroutine-sp-fallback.diff
];
};
boehmgc-nix = boehmgc.override { enableLargeConfig = true; };
editline-lix = editline.overrideAttrs (prev: {
configureFlags = prev.configureFlags or [ ] ++ [ (lib.enableFeature true "sigstop") ];
});
lix-doc = pkgs.callPackage ./lix-doc/package.nix { };
build-release-notes = pkgs.callPackage ./maintainers/build-release-notes.nix { };
lix-doc = callPackage ./lix-doc/package.nix { };
build-release-notes = callPackage ./maintainers/build-release-notes.nix { };
pegtl = callPackage ./misc/pegtl.nix { };
},
}:
let
@ -165,7 +159,6 @@ stdenv.mkDerivation (finalAttrs: {
functionalTestFiles
]
++ lib.optionals (!finalAttrs.dontBuild || internalApiDocs) [
./boehmgc-coroutine-sp-fallback.diff
./doc
./misc
./src
@ -210,8 +203,6 @@ stdenv.mkDerivation (finalAttrs: {
nativeBuildInputs =
[
bison
flex
python3
meson
ninja
@ -250,6 +241,7 @@ stdenv.mkDerivation (finalAttrs: {
libsodium
toml11
lix-doc
pegtl
]
++ lib.optionals hostPlatform.isLinux [
libseccomp
@ -304,6 +296,9 @@ stdenv.mkDerivation (finalAttrs: {
else
appendToVar configureFlags "--disable-tests"
fi
# Fix up /usr/bin/env shebangs relied on by the build
patchShebangs --build tests/ doc/manual/
'';
mesonBuildType = "debugoptimized";
@ -379,7 +374,12 @@ stdenv.mkDerivation (finalAttrs: {
# Export the patched version of boehmgc.
# flake.nix exports that into its overlay.
passthru = {
inherit (__forDefaults) boehmgc-nix editline-lix build-release-notes;
inherit (__forDefaults)
boehmgc-nix
editline-lix
build-release-notes
pegtl
;
inherit officialRelease;
@ -392,10 +392,11 @@ stdenv.mkDerivation (finalAttrs: {
bashInteractive,
clang-tools,
clangbuildanalyzer,
doxygen,
glibcLocales,
just,
llvmPackages,
nixfmt,
nixfmt-rfc-style,
skopeo,
xonsh,
@ -456,10 +457,22 @@ stdenv.mkDerivation (finalAttrs: {
# docker image tool
skopeo
just
nixfmt
nixfmt-rfc-style
# Included above when internalApiDocs is true, but we set that to
# false intentionally to save dev build time.
# To build them in a dev shell, you can set -Dinternal-api-docs=enabled when configuring.
doxygen
# Load-bearing order. Must come before clang-unwrapped below, but after clang_tools above.
stdenv.cc
]
++ [
pkgs.rust-analyzer
pkgs.cargo
pkgs.rustc
pkgs.rustfmt
pkgs.rustPlatform.rustLibSrc
pkgs.rustPlatform.rustcSrc
]
++ lib.optionals stdenv.cc.isClang [
# Required for clang-tidy checks.
llvmPackages.llvm
@ -476,16 +489,21 @@ stdenv.mkDerivation (finalAttrs: {
# https://git.lix.systems/lix-project/lix/src/commit/7575db522e9008685c4009423398f6900a16bcce/src/nix/develop.cc#L240-L241
# this is, of course, absurd.
if [[ $name != lix-shell-env && $name != lix-shell-env-env ]]; then
return;
return
fi
PATH=$prefix/bin:$PATH
PATH=$prefix/bin''${PATH:+:''${PATH}}
unset PYTHONPATH
export MANPATH=$out/share/man:$MANPATH
export MANPATH=$out/share/man''${MANPATH:+:''${MANPATH}}
# Make bash completion work.
XDG_DATA_DIRS+=:$out/share
if [[ ! -f ./.this-is-lix ]]; then
echo "Dev shell not started from inside a Lix repo, skipping repo setup" >&2
return
fi
${lib.optionalString (pre-commit-checks ? shellHook) pre-commit-checks.shellHook}
# Allow `touch .nocontribmsg` to turn this notice off.
if ! [[ -f .nocontribmsg ]]; then

View file

@ -65,5 +65,6 @@ if cxx.get_linker_id() in ['ld.bfd', 'ld.gold']
endif
libstore = dependency('lixstore', 'lix-store', required : true)
libutil = dependency('lixutil', 'lix-util', required : true)
subdir('lib/Nix')

View file

@ -44,23 +44,8 @@ def upload_docker_images(target: DockerTarget, paths: list[Path]):
for path in paths:
digest_file = tmp / (path.name + '.digest')
tmp_image = tmp / 'tmp-image.tar.gz'
# insecure-policy: we don't have any signature policy, we are just uploading an image
#
# Absurd: we copy it into an OCI image first so we can get the hash
# we need to upload it untagged, because skopeo has no "don't tag
# this" option.
# The reason for this is that forgejo's container registry throws
# away old versions of tags immediately, so we cannot use a temp
# tag, and it *does* reduce confusion to not upload tags that
# should not be used.
#
# Workaround for: https://github.com/containers/skopeo/issues/2354
log.info('skopeo copy to temp oci-archive %s', tmp_image)
skopeo --insecure-policy copy --format oci --all --digestfile @(digest_file) docker-archive:@(path) oci-archive:@(tmp_image)
inspection = json.loads($(skopeo inspect oci-archive:@(tmp_image)))
inspection = json.loads($(skopeo inspect docker-archive:@(path)))
docker_arch = inspection['Architecture']
docker_os = inspection['Os']
@ -68,8 +53,9 @@ def upload_docker_images(target: DockerTarget, paths: list[Path]):
log.info('Pushing image %s for %s to %s', path, docker_arch, target.registry_path)
# insecure-policy: we don't have any signature policy, we are just uploading an image
skopeo --insecure-policy copy --digestfile @(digest_file) --all docker-archive:@(path) f'docker://{target.registry_path}@@unknown-digest@@'
digest = digest_file.read_text().strip()
skopeo --insecure-policy copy --preserve-digests --all oci-archive:@(tmp_image) f'docker://{target.registry_path}@{digest}'
# skopeo doesn't give us the manifest size directly, so we just ask the registry
metadata = reg.image_info(target.registry_path, digest)

36
releng/release-tests.sh Executable file
View file

@ -0,0 +1,36 @@
#!/usr/bin/env bash
set -euo pipefail
shopt -s inherit_errexit failglob
nixpkgss=(
"$(nix eval --impure --raw --expr '(import ./flake.nix).inputs.nixpkgs.url')"
"github:NixOS/nixpkgs/nixos-unstable-small"
)
jobs=(
$(nix eval \
--json --apply '
let f = n: t:
if builtins.isAttrs t
then (if t.type or "" == "derivation"
then [ n ]
else builtins.concatMap (m: f "${n}.${m}" t.${m}) (builtins.attrNames t))
else [];
in f ".#.releaseTests"
' \
'.#.releaseTests' \
| jq -r '.[]'
)
)
for override in "${nixpkgss}"
do
(
set -x
nix build \
--log-format multiline \
--no-link \
--override-input nixpkgs "$override" \
"${jobs[@]}"
)
done

View file

@ -16,7 +16,7 @@ def add_to_summary(date: str):
if VERSION_RL.exists():
return
MARKER = '<!-- RELENG-AUTO-INSERTION-MARKER'
MARKER = ' <!-- RELENG-AUTO-INSERTION-MARKER'
new_lines = []
for line in SUMMARY.read_text().splitlines():

View file

@ -244,7 +244,7 @@ StorePath ProfileManifest::build(ref<Store> store)
/* Add the symlink tree to the store. */
StringSink sink;
dumpPath(tempDir, sink);
sink << dumpPath(tempDir);
auto narHash = hashString(htSHA256, sink.s);

View file

@ -37,7 +37,7 @@ struct ProfileElement
StorePathSet storePaths;
std::optional<ProfileElementSource> source;
bool active = true;
int priority = DEFAULT_PRIORITY;
NixInt::Inner priority = DEFAULT_PRIORITY;
std::string identifier() const;

View file

@ -1,11 +0,0 @@
#include "command-installable-value.hh"
namespace nix {
void InstallableValueCommand::run(ref<Store> store, ref<Installable> installable)
{
auto installableValue = InstallableValue::require(installable);
run(store, installableValue);
}
}

View file

@ -1,23 +0,0 @@
#pragma once
///@file
#include "installable-value.hh"
#include "command.hh"
namespace nix {
/**
* An InstallableCommand where the single positional argument must be an
* InstallableValue in particular.
*/
struct InstallableValueCommand : InstallableCommand
{
/**
* Entry point to this command
*/
virtual void run(ref<Store> store, ref<InstallableValue> installable) = 0;
void run(ref<Store> store, ref<Installable> installable) override;
};
}

View file

@ -116,12 +116,12 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
auto drvPath = attr->forceDerivation();
std::optional<NixInt> priority;
std::optional<NixInt::Inner> priority;
if (attr->maybeGetAttr(state->sOutputSpecified)) {
} else if (auto aMeta = attr->maybeGetAttr(state->sMeta)) {
if (auto aPriority = aMeta->maybeGetAttr("priority"))
priority = aPriority->getInt();
priority = aPriority->getInt().value;
}
return {{

View file

@ -40,7 +40,7 @@ struct ExtraPathInfoValue : ExtraPathInfo
/**
* An optional priority for use with "build envs". See Package
*/
std::optional<NixInt> priority;
std::optional<NixInt::Inner> priority;
/**
* The attribute path associated with this value. The idea is

View file

@ -393,13 +393,10 @@ ref<eval_cache::EvalCache> openEvalCache(
EvalState & state,
std::shared_ptr<flake::LockedFlake> lockedFlake)
{
auto fingerprint = lockedFlake->getFingerprint();
return make_ref<nix::eval_cache::EvalCache>(
evalSettings.useEvalCache && evalSettings.pureEval
? std::optional { std::cref(fingerprint) }
: std::nullopt,
state,
[&state, lockedFlake]()
auto fingerprint = evalSettings.useEvalCache && evalSettings.pureEval
? std::make_optional(lockedFlake->getFingerprint())
: std::nullopt;
auto rootLoader = [&state, lockedFlake]()
{
/* For testing whether the evaluation cache is
complete. */
@ -415,7 +412,17 @@ ref<eval_cache::EvalCache> openEvalCache(
assert(aOutputs);
return aOutputs->value;
});
};
if (fingerprint) {
auto search = state.evalCaches.find(fingerprint.value());
if (search == state.evalCaches.end()) {
search = state.evalCaches.emplace(fingerprint.value(), make_ref<nix::eval_cache::EvalCache>(fingerprint, state, rootLoader)).first;
}
return search->second;
} else {
return make_ref<nix::eval_cache::EvalCache>(std::nullopt, state, rootLoader);
}
}
Installables SourceExprCommand::parseInstallables(

View file

@ -1,6 +1,5 @@
libcmd_sources = files(
'built-path.cc',
'command-installable-value.cc',
'cmd-profiles.cc',
'command.cc',
'common-eval-args.cc',
@ -18,7 +17,6 @@ libcmd_sources = files(
libcmd_headers = files(
'built-path.hh',
'command-installable-value.hh',
'cmd-profiles.hh',
'command.hh',
'common-eval-args.hh',

View file

@ -1,3 +1,4 @@
#include <cstdio>
#include <editline.h>
#include <iostream>
#include <cstdlib>
@ -31,7 +32,6 @@
#include "local-fs-store.hh"
#include "signals.hh"
#include "print.hh"
#include "progress-bar.hh"
#include "gc-small-vector.hh"
#include "users.hh"
@ -40,24 +40,24 @@
#include <gc/gc_cpp.h>
#endif
// XXX: These are for nix-doc features and will be removed in a future rewrite where this functionality is integrated more natively.
// XXX: These are for lix-doc features and will be removed in a future rewrite where this functionality is integrated more natively.
extern "C" {
char const *nd_get_function_docs(char const *filename, size_t line, size_t col);
void nd_free_string(char const *str);
char const *lixdoc_get_function_docs(char const *filename, size_t line, size_t col);
void lixdoc_free_string(char const *str);
}
namespace nix {
/** Wrapper around std::unique_ptr with a custom deleter for strings from nix-doc **/
using NdString = std::unique_ptr<const char, decltype(&nd_free_string)>;
using NdString = std::unique_ptr<const char, decltype(&lixdoc_free_string)>;
/**
* Fetch a string representing the doc comment using nix-doc and wrap it in an RAII wrapper.
*/
NdString lambdaDocsForPos(SourcePath const path, nix::Pos const &pos) {
std::string const file = path.to_string();
return NdString{nd_get_function_docs(file.c_str(), pos.line, pos.column), &nd_free_string};
return NdString{lixdoc_get_function_docs(file.c_str(), pos.line, pos.column), &lixdoc_free_string};
}
/**
@ -242,8 +242,7 @@ NixRepl::NixRepl(const SearchPath & searchPath, nix::ref<Store> store, ref<EvalS
{
}
void runNix(Path program, const Strings & args,
const std::optional<std::string> & input = {})
void runNix(Path program, const Strings & args)
{
auto subprocessEnv = getEnv();
subprocessEnv["NIX_CONFIG"] = globalConfig.toKeyValue();
@ -252,8 +251,7 @@ void runNix(Path program, const Strings & args,
.program = settings.nixBinDir+ "/" + program,
.args = args,
.environment = subprocessEnv,
.input = input,
});
}).wait();
return;
}
@ -301,7 +299,7 @@ ReplExitStatus NixRepl::mainLoop()
/* Stop the progress bar because it interferes with the display of
the repl. */
stopProgressBar();
logger->pause();
std::string input;
@ -652,7 +650,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
// runProgram redirects stdout to a StringSink,
// using runProgram2 to allow editors to display their UI
runProgram2(RunOptions { .program = editor, .searchPath = true, .args = args });
runProgram2(RunOptions { .program = editor, .searchPath = true, .args = args }).wait();
// Reload right after exiting the editor
state->resetFileCache();
@ -685,9 +683,10 @@ ProcessLineResult NixRepl::processLine(std::string line)
// TODO: this only shows a progress bar for explicitly initiated builds,
// not eval-time fetching or builds performed for IFD.
// But we can't just show it everywhere, since that would erase partial output from evaluation.
startProgressBar();
logger->resetProgress();
logger->resume();
Finally stopLogger([&]() {
stopProgressBar();
logger->pause();
});
state->store->buildPaths({

View file

@ -1,5 +1,6 @@
#include "attr-set.hh"
#include "eval-inline.hh"
#include "eval.hh"
#include "gc-alloc.hh"
#include <algorithm>
@ -19,7 +20,7 @@ Bindings * EvalState::allocBindings(size_t capacity)
throw Error("attribute set of size %d is too big", capacity);
nrAttrsets++;
nrAttrsInAttrsets += capacity;
return new (allocBytes(sizeof(Bindings) + sizeof(Attr) * capacity)) Bindings((Bindings::size_t) capacity);
return new (gcAllocBytes(sizeof(Bindings) + sizeof(Attr) * capacity)) Bindings((Bindings::size_t) capacity);
}

View file

@ -306,7 +306,7 @@ struct AttrDb
case AttrType::Bool:
return {{rowId, queryAttribute.getInt(2) != 0}};
case AttrType::Int:
return {{rowId, int_t{queryAttribute.getInt(2)}}};
return {{rowId, int_t{NixInt{queryAttribute.getInt(2)}}}};
case AttrType::ListOfStrings:
return {{rowId, tokenizeString<std::vector<std::string>>(queryAttribute.getStr(2), "\t")}};
case AttrType::Missing:
@ -449,7 +449,7 @@ Value & AttrCursor::forceValue()
else if (v.type() == nBool)
cachedValue = {root->db->setBool(getKey(), v.boolean), v.boolean};
else if (v.type() == nInt)
cachedValue = {root->db->setInt(getKey(), v.integer), int_t{v.integer}};
cachedValue = {root->db->setInt(getKey(), v.integer.value), int_t{v.integer}};
else if (v.type() == nAttrs)
; // FIXME: do something?
else

View file

@ -4,26 +4,10 @@
#include "print.hh"
#include "eval.hh"
#include "eval-error.hh"
#include "gc-alloc.hh"
namespace nix {
/**
* Note: Various places expect the allocated memory to be zeroed.
*/
[[gnu::always_inline]]
inline void * allocBytes(size_t n)
{
void * p;
#if HAVE_BOEHMGC
p = GC_MALLOC(n);
#else
p = calloc(n, 1);
#endif
if (!p) throw std::bad_alloc();
return p;
}
[[gnu::always_inline]]
Value * EvalState::allocValue()
{
@ -43,7 +27,7 @@ Value * EvalState::allocValue()
*valueAllocCache = GC_NEXT(p);
GC_NEXT(p) = nullptr;
#else
void * p = allocBytes(sizeof(Value));
void * p = gcAllocBytes(sizeof(Value));
#endif
nrValues++;
@ -73,7 +57,7 @@ Env & EvalState::allocEnv(size_t size)
env = (Env *) p;
} else
#endif
env = (Env *) allocBytes(sizeof(Env) + size * sizeof(Value *));
env = (Env *) gcAllocBytes(sizeof(Env) + size * sizeof(Value *));
/* We assume that env->values has been cleared by the allocator; maybeThunk() and lookupVar fromWith expect this. */

View file

@ -63,11 +63,9 @@ Strings EvalSettings::getDefaultNixPath()
}
};
if (!evalSettings.restrictEval && !evalSettings.pureEval) {
add(getNixDefExpr() + "/channels");
add(rootChannelsDir() + "/nixpkgs", "nixpkgs");
add(rootChannelsDir());
}
add(getNixDefExpr() + "/channels");
add(rootChannelsDir() + "/nixpkgs", "nixpkgs");
add(rootChannelsDir());
return res;
}

View file

@ -75,8 +75,17 @@ struct EvalSettings : Config
R"(
Pure evaluation mode ensures that the result of Nix expressions is fully determined by explicitly declared inputs, and not influenced by external state:
- Restrict file system and network access to files specified by cryptographic hash
- Disable [`builtins.currentSystem`](@docroot@/language/builtin-constants.md#builtins-currentSystem) and [`builtins.currentTime`](@docroot@/language/builtin-constants.md#builtins-currentTime)
- File system and network access is restricted to accesses to immutable data only:
- Path literals relative to the home directory like `~/lix` are rejected at parse time.
- Access to absolute paths that did not result from Nix language evaluation is rejected when such paths are given as parameters to builtins like, for example, [`builtins.readFile`](@docroot@/language/builtins.md#builtins-readFile).
Access is nonetheless allowed to (absolute) paths in the Nix store that are returned by builtins like [`builtins.filterSource`](@docroot@/language/builtins.md#builtins-filterSource), [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball) and similar.
- Impure fetches such as not specifying a commit ID for `builtins.fetchGit` or not specifying a hash for `builtins.fetchTarball` are rejected.
- In flakes, access to relative paths outside of the root of the flake's source tree (often, a git repository) is rejected.
- The evaluator ignores `NIX_PATH`, `-I` and the `nix-path` setting. Thus, [`builtins.nixPath`](@docroot@/language/builtin-constants.md#builtins-nixPath) is an empty list.
- The builtins [`builtins.currentSystem`](@docroot@/language/builtin-constants.md#builtins-currentSystem) and [`builtins.currentTime`](@docroot@/language/builtin-constants.md#builtins-currentTime) are absent from `builtins`.
- [`builtins.getEnv`](@docroot@/language/builtin-constants.md#builtins-currentSystem) always returns empty string for any variable.
- [`builtins.storePath`](@docroot@/language/builtins.md#builtins-storePath) throws an error (Lix may change this, tracking issue: <https://git.lix.systems/lix-project/lix/issues/402>)
)"
};
@ -98,6 +107,7 @@ struct EvalSettings : Config
allowed to access `https://github.com/NixOS/patchelf.git`.
)"};
Setting<bool> traceFunctionCalls{this, false, "trace-function-calls",
R"(
If set to `true`, the Nix evaluator will trace every function call.

View file

@ -4,10 +4,12 @@
#include "primops.hh"
#include "print-options.hh"
#include "shared.hh"
#include "suggestions.hh"
#include "types.hh"
#include "store-api.hh"
#include "derivations.hh"
#include "downstream-placeholder.hh"
#include "gc-alloc.hh"
#include "globals.hh"
#include "eval-inline.hh"
#include "filetransfer.hh"
@ -17,7 +19,6 @@
#include "gc-small-vector.hh"
#include "fetch-to-store.hh"
#include "flake/flakeref.hh"
#include "parser-tab.hh"
#include <algorithm>
#include <chrono>
@ -42,58 +43,12 @@
#include <gc/gc.h>
#include <gc/gc_cpp.h>
#include <boost/coroutine2/coroutine.hpp>
#include <boost/coroutine2/protected_fixedsize_stack.hpp>
#include <boost/context/stack_context.hpp>
#endif
using json = nlohmann::json;
namespace nix {
static char * allocString(size_t size)
{
char * t;
#if HAVE_BOEHMGC
t = (char *) GC_MALLOC_ATOMIC(size);
#else
t = (char *) malloc(size);
#endif
if (!t) throw std::bad_alloc();
return t;
}
static char * dupString(const char * s)
{
char * t;
#if HAVE_BOEHMGC
t = GC_STRDUP(s);
#else
t = strdup(s);
#endif
if (!t) throw std::bad_alloc();
return t;
}
// When there's no need to write to the string, we can optimize away empty
// string allocations.
// This function handles makeImmutableString(std::string_view()) by returning
// the empty string.
static const char * makeImmutableString(std::string_view s)
{
const size_t size = s.size();
if (size == 0)
return "";
auto t = allocString(size + 1);
memcpy(t, s.data(), size);
t[size] = '\0';
return t;
}
RootValue allocRootValue(Value * v)
{
#if HAVE_BOEHMGC
@ -205,42 +160,6 @@ static void * oomHandler(size_t requested)
throw std::bad_alloc();
}
class BoehmGCStackAllocator : public StackAllocator {
boost::coroutines2::protected_fixedsize_stack stack {
// We allocate 8 MB, the default max stack size on NixOS.
// A smaller stack might be quicker to allocate but reduces the stack
// depth available for source filter expressions etc.
std::max(boost::context::stack_traits::default_size(), static_cast<std::size_t>(8 * 1024 * 1024))
};
// This is specific to boost::coroutines2::protected_fixedsize_stack.
// The stack protection page is included in sctx.size, so we have to
// subtract one page size from the stack size.
std::size_t pfss_usable_stack_size(boost::context::stack_context &sctx) {
return sctx.size - boost::context::stack_traits::page_size();
}
public:
boost::context::stack_context allocate() override {
auto sctx = stack.allocate();
// Stacks generally start at a high address and grow to lower addresses.
// Architectures that do the opposite are rare; in fact so rare that
// boost_routine does not implement it.
// So we subtract the stack size.
GC_add_roots(static_cast<char *>(sctx.sp) - pfss_usable_stack_size(sctx), sctx.sp);
return sctx;
}
void deallocate(boost::context::stack_context sctx) override {
GC_remove_roots(static_cast<char *>(sctx.sp) - pfss_usable_stack_size(sctx), sctx.sp);
stack.deallocate(sctx);
}
};
static BoehmGCStackAllocator boehmGCStackAllocator;
#endif
@ -256,23 +175,6 @@ static Symbol getName(const AttrName & name, EvalState & state, Env & env)
}
}
#if HAVE_BOEHMGC
/* Disable GC while this object lives. Used by CoroutineContext.
*
* Boehm keeps a count of GC_disable() and GC_enable() calls,
* and only enables GC when the count matches.
*/
class BoehmDisableGC {
public:
BoehmDisableGC() {
GC_disable();
};
~BoehmDisableGC() {
GC_enable();
};
};
#endif
static bool gcInitialised = false;
void initGC()
@ -294,17 +196,6 @@ void initGC()
GC_set_oom_fn(oomHandler);
StackAllocator::defaultAllocator = &boehmGCStackAllocator;
#if NIX_BOEHM_PATCH_VERSION != 1
printTalkative("Unpatched BoehmGC, disabling GC inside coroutines");
/* Used to disable GC when entering coroutines on macOS */
create_coro_gc_hook = []() -> std::shared_ptr<void> {
return std::make_shared<BoehmDisableGC>();
};
#endif
/* Set the initial heap size to something fairly big (25% of
physical RAM, up to a maximum of 384 MiB) so that in most cases
we don't need to garbage collect at all. (Collection has a
@ -418,7 +309,7 @@ EvalState::EvalState(
}
if (evalSettings.restrictEval || evalSettings.pureEval) {
allowedPaths = PathSet();
allowedPaths = std::optional(PathSet());
for (auto & i : searchPath.elements) {
auto r = resolveSearchPathPath(i.path);
@ -878,7 +769,7 @@ DebugTraceStacker::DebugTraceStacker(EvalState & evalState, DebugTrace t)
void Value::mkString(std::string_view s)
{
mkString(makeImmutableString(s));
mkString(gcCopyStringIfNeeded(s));
}
@ -886,10 +777,9 @@ static void copyContextToValue(Value & v, const NixStringContext & context)
{
if (!context.empty()) {
size_t n = 0;
v.string.context = (const char * *)
allocBytes((context.size() + 1) * sizeof(char *));
v.string.context = gcAllocType<char const *>(context.size() + 1);
for (auto & i : context)
v.string.context[n++] = dupString(i.to_string().c_str());
v.string.context[n++] = gcCopyStringIfNeeded(i.to_string());
v.string.context[n] = 0;
}
}
@ -909,7 +799,7 @@ void Value::mkStringMove(const char * s, const NixStringContext & context)
void Value::mkPath(const SourcePath & path)
{
mkPath(makeImmutableString(path.path.abs()));
mkPath(gcCopyStringIfNeeded(path.path.abs()));
}
@ -943,7 +833,7 @@ void EvalState::mkList(Value & v, size_t size)
{
v.mkList(size);
if (size > 2)
v.bigList.elems = (Value * *) allocBytes(size * sizeof(Value *));
v.bigList.elems = gcAllocType<Value *>(size);
nrListElems += size;
}
@ -1426,11 +1316,26 @@ static std::string showAttrPath(EvalState & state, Env & env, const AttrPath & a
void ExprSelect::eval(EvalState & state, Env & env, Value & v)
{
Value vTmp;
PosIdx pos2;
Value * vAttrs = &vTmp;
Value vFirst;
e->eval(state, env, vTmp);
// Pointer to the current attrset Value in this select chain.
Value * vCurrent = &vFirst;
// Position for the current attrset Value in this select chain.
PosIdx posCurrent;
try {
e->eval(state, env, vFirst);
} catch (Error & e) {
assert(this->e != nullptr);
state.addErrorTrace(
e,
getPos(),
"while evaluating '%s' to select '%s' on it",
ExprPrinter(state, *this->e),
showAttrPath(state.symbols, this->attrPath)
);
throw;
}
try {
auto dts = state.debugRepl
@ -1443,48 +1348,112 @@ void ExprSelect::eval(EvalState & state, Env & env, Value & v)
showAttrPath(state, env, attrPath))
: nullptr;
for (auto & i : attrPath) {
for (auto const & [partIdx, currentAttrName] : enumerate(attrPath)) {
state.nrLookups++;
Bindings::iterator j;
auto name = getName(i, state, env);
if (def) {
state.forceValue(*vAttrs, pos);
if (vAttrs->type() != nAttrs ||
(j = vAttrs->attrs->find(name)) == vAttrs->attrs->end())
{
def->eval(state, env, v);
Symbol const name = getName(currentAttrName, state, env);
// For formatting errors, which should be done only when needed.
auto partsSoFar = [&]() -> std::string {
std::stringstream ss;
// We start with the base thing this ExprSelect is selecting on.
assert(this->e != nullptr);
this->e->show(state.symbols, ss);
// Then grab each part of the attr path up to this one.
assert(partIdx < attrPath.size());
std::span<AttrName> const parts(
attrPath.begin(),
attrPath.begin() + partIdx
);
// And convert them to strings and join them.
for (auto const & part : parts) {
auto const partName = getName(part, state, env);
ss << "." << state.symbols[partName];
}
return ss.str();
};
try {
state.forceValue(*vCurrent, pos);
} catch (Error & e) {
state.addErrorTrace(
e,
getPos(),
"while evaluating '%s' to select '%s' on it",
partsSoFar(),
state.symbols[name]
);
throw;
}
if (vCurrent->type() != nAttrs) {
// If we have an `or` provided default,
// then this is allowed to not be an attrset.
if (def != nullptr) {
this->def->eval(state, env, v);
return;
}
} else {
state.forceAttrs(*vAttrs, pos, "while selecting an attribute");
if ((j = vAttrs->attrs->find(name)) == vAttrs->attrs->end()) {
std::set<std::string> allAttrNames;
for (auto & attr : *vAttrs->attrs)
allAttrNames.insert(state.symbols[attr.name]);
auto suggestions = Suggestions::bestMatches(allAttrNames, state.symbols[name]);
state.error<EvalError>("attribute '%1%' missing", state.symbols[name])
.atPos(pos).withSuggestions(suggestions).withFrame(env, *this).debugThrow();
}
// Otherwise, we must type error.
state.error<TypeError>(
"expected a set but found %s: %s",
showType(*vCurrent),
ValuePrinter(state, *vCurrent, errorPrintOptions)
).addTrace(
pos,
HintFmt("while selecting '%s' on '%s'", state.symbols[name], partsSoFar())
).debugThrow();
}
vAttrs = j->value;
pos2 = j->pos;
if (state.countCalls) state.attrSelects[pos2]++;
// Now that we know this is actually an attrset, try to find an attr
// with the selected name.
Bindings::iterator attrIt = vCurrent->attrs->find(name);
if (attrIt == vCurrent->attrs->end()) {
// If we have an `or` provided default, then we'll use that.
if (def != nullptr) {
this->def->eval(state, env, v);
return;
}
// Otherwise, missing attr error.
std::set<std::string> allAttrNames;
for (auto const & attr : *vCurrent->attrs) {
allAttrNames.insert(state.symbols[attr.name]);
}
auto suggestions = Suggestions::bestMatches(allAttrNames, state.symbols[name]);
state.error<EvalError>("attribute '%s' missing", state.symbols[name])
.atPos(pos)
.withSuggestions(suggestions)
.withFrame(env, *this)
.debugThrow();
}
// If we're here, then we successfully found the attribute.
// Set our currently operated-on attrset to this one, and keep going.
vCurrent = attrIt->value;
posCurrent = attrIt->pos;
if (state.countCalls) state.attrSelects[posCurrent]++;
}
state.forceValue(*vAttrs, (pos2 ? pos2 : this->pos ) );
state.forceValue(*vCurrent, (posCurrent ? posCurrent : this->pos));
} catch (Error & e) {
if (pos2) {
auto pos2r = state.positions[pos2];
if (posCurrent) {
auto pos2r = state.positions[posCurrent];
auto origin = std::get_if<SourcePath>(&pos2r.origin);
if (!(origin && *origin == state.derivationInternal))
state.addErrorTrace(e, pos2, "while evaluating the attribute '%1%'",
state.addErrorTrace(e, posCurrent, "while evaluating the attribute '%1%'",
showAttrPath(state, env, attrPath));
}
throw;
}
v = *vAttrs;
v = *vCurrent;
}
@ -1533,6 +1502,66 @@ public:
};
};
/** Currently these each just take one, but maybe in the future we could have diagnostics
* for all unexpected and missing arguments?
*/
struct FormalsMatch
{
std::vector<Symbol> missing;
std::vector<Symbol> unexpected;
};
/** Matchup an attribute argument set to a lambda's formal arguments,
* or return what arguments were required but not given, or given but not allowed.
* (currently returns only one, for each).
*/
FormalsMatch matchupFormals(EvalState & state, Env & env, Displacement & displ, ExprLambda const & lambda, Bindings & attrs)
{
size_t attrsUsed = 0;
for (auto const & formal : lambda.formals->formals) {
// The attribute whose name matches the name of the formal we're matching up, if it exists.
Attr const * matchingArg = attrs.get(formal.name);
if (matchingArg) {
attrsUsed += 1;
env.values[displ] = matchingArg->value;
displ += 1;
// We're done here. Move on to the next formal.
continue;
}
// The argument for this formal wasn't given.
// If the formal has a default, use it.
if (formal.def) {
env.values[displ] = formal.def->maybeThunk(state, env);
displ += 1;
} else {
// Otherwise, let our caller know what was missing.
return FormalsMatch{
.missing = {formal.name},
};
}
}
// Check for unexpected extra arguments.
if (!lambda.formals->ellipsis && attrsUsed != attrs.size()) {
// Return the first unexpected argument.
for (Attr const & attr : attrs) {
if (!lambda.formals->has(attr.name)) {
return FormalsMatch{
.unexpected = {attr.name},
};
}
}
abort(); // unreachable.
}
return FormalsMatch{};
}
void EvalState::callFunction(Value & fun, size_t nrArgs, Value * * args, Value & vRes, const PosIdx pos)
{
if (callDepth > evalSettings.maxCallDepth)
@ -1586,53 +1615,42 @@ void EvalState::callFunction(Value & fun, size_t nrArgs, Value * * args, Value &
if (lambda.arg)
env2.values[displ++] = args[0];
/* For each formal argument, get the actual argument. If
there is no matching actual argument but the formal
argument has a default, use the default. */
size_t attrsUsed = 0;
for (auto & i : lambda.formals->formals) {
auto j = args[0]->attrs->get(i.name);
if (!j) {
if (!i.def) {
error<TypeError>("function '%1%' called without required argument '%2%'",
lambda.getName(symbols),
symbols[i.name])
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withFrame(*fun.lambda.env, lambda)
.debugThrow();
}
env2.values[displ++] = i.def->maybeThunk(*this, env2);
} else {
attrsUsed++;
env2.values[displ++] = j->value;
///* For each formal argument, get the actual argument. If
// there is no matching actual argument but the formal
// argument has a default, use the default. */
auto const formalsMatch = matchupFormals(
*this,
env2,
displ,
lambda,
*args[0]->attrs
);
for (auto const & missingArg : formalsMatch.missing) {
auto const missing = symbols[missingArg];
error<TypeError>("function '%s' called without required argument '%s'", lambda.getName(symbols), missing)
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withFrame(*fun.lambda.env, lambda)
.debugThrow();
}
for (auto const & unexpectedArg : formalsMatch.unexpected) {
auto const unex = symbols[unexpectedArg];
std::set<std::string> formalNames;
for (auto const & formal : lambda.formals->formals) {
formalNames.insert(symbols[formal.name]);
}
auto sug = Suggestions::bestMatches(formalNames, unex);
error<TypeError>("function '%s' called with unexpected argument '%s'", lambda.getName(symbols), unex)
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withSuggestions(sug)
.withFrame(*fun.lambda.env, lambda)
.debugThrow();
}
/* Check that each actual argument is listed as a formal
argument (unless the attribute match specifies a `...'). */
if (!lambda.formals->ellipsis && attrsUsed != args[0]->attrs->size()) {
/* Nope, so show the first unexpected argument to the
user. */
for (auto & i : *args[0]->attrs)
if (!lambda.formals->has(i.name)) {
std::set<std::string> formalNames;
for (auto & formal : lambda.formals->formals)
formalNames.insert(symbols[formal.name]);
auto suggestions = Suggestions::bestMatches(formalNames, symbols[i.name]);
error<TypeError>("function '%1%' called with unexpected argument '%2%'",
lambda.getName(symbols),
symbols[i.name])
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withSuggestions(suggestions)
.withFrame(*fun.lambda.env, lambda)
.debugThrow();
}
abort(); // can't happen
}
}
nrFunctionCalls++;
if (countCalls) incrFunctionCall(&lambda);
@ -1679,6 +1697,15 @@ void EvalState::callFunction(Value & fun, size_t nrArgs, Value * * args, Value &
try {
fn->fun(*this, vCur.determinePos(noPos), args, vCur);
} catch (ThrownError & e) {
// Distinguish between an error that simply happened while "throw"
// was being evaluated and an explicit thrown error.
if (fn->name == "throw") {
addErrorTrace(e, pos, "caused by explicit %s", "throw");
} else {
addErrorTrace(e, pos, "while calling the '%s' builtin", fn->name);
}
throw;
} catch (Error & e) {
addErrorTrace(e, pos, "while calling the '%1%' builtin", fn->name);
throw;
@ -1845,7 +1872,7 @@ void EvalState::autoCallFunction(Bindings & args, Value & fun, Value & res)
Nix attempted to evaluate a function as a top level expression; in
this case it must have its arguments supplied either by default
values, or passed explicitly with '--arg' or '--argstr'. See
https://nixos.org/manual/nix/stable/language/constructs.html#functions.)", symbols[i.name])
https://docs.lix.systems/manual/lix/stable/language/constructs.html#functions)", symbols[i.name])
.atPos(i.pos).withFrame(*fun.lambda.env, *fun.lambda.fun).debugThrow();
}
}
@ -2004,7 +2031,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
NixStringContext context;
std::vector<BackedStringView> s;
size_t sSize = 0;
NixInt n = 0;
NixInt n{0};
NixFloat nf = 0;
bool first = !forceString;
@ -2020,7 +2047,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
Value. allocating a GC'd string directly and moving it into a
Value lets us avoid an allocation and copy. */
const auto c_str = [&] {
char * result = allocString(sSize + 1);
char * result = gcAllocString(sSize + 1);
char * tmp = result;
for (const auto & part : s) {
memcpy(tmp, part->data(), part->size());
@ -2048,17 +2075,22 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
if (firstType == nInt) {
if (vTmp.type() == nInt) {
n += vTmp.integer;
auto newN = n + vTmp.integer;
if (auto checked = newN.valueChecked(); checked.has_value()) {
n = NixInt(*checked);
} else {
state.error<EvalError>("integer overflow in adding %1% + %2%", n, vTmp.integer).atPos(i_pos).debugThrow();
}
} else if (vTmp.type() == nFloat) {
// Upgrade the type from int to float;
firstType = nFloat;
nf = n;
nf = n.value;
nf += vTmp.fpoint;
} else
state.error<EvalError>("cannot add %1% to an integer", showType(vTmp)).atPos(i_pos).withFrame(env, *this).debugThrow();
} else if (firstType == nFloat) {
if (vTmp.type() == nInt) {
nf += vTmp.integer;
nf += vTmp.integer.value;
} else if (vTmp.type() == nFloat) {
nf += vTmp.fpoint;
} else
@ -2183,7 +2215,7 @@ NixFloat EvalState::forceFloat(Value & v, const PosIdx pos, std::string_view err
try {
forceValue(v, pos);
if (v.type() == nInt)
return v.integer;
return v.integer.value;
else if (v.type() != nFloat)
error<TypeError>(
"expected a float but found %1%: %2%",
@ -2370,7 +2402,7 @@ BackedStringView EvalState::coerceToString(
shell scripting convenience, just like `null'. */
if (v.type() == nBool && v.boolean) return "1";
if (v.type() == nBool && !v.boolean) return "";
if (v.type() == nInt) return std::to_string(v.integer);
if (v.type() == nInt) return std::to_string(v.integer.value);
if (v.type() == nFloat) return std::to_string(v.fpoint);
if (v.type() == nNull) return "";
@ -2514,9 +2546,9 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
// Special case type-compatibility between float and int
if (v1.type() == nInt && v2.type() == nFloat)
return v1.integer == v2.fpoint;
return v1.integer.value == v2.fpoint;
if (v1.type() == nFloat && v2.type() == nInt)
return v1.fpoint == v2.integer;
return v1.fpoint == v2.integer.value;
// All other types are not compatible with each other.
if (v1.type() != v2.type()) return false;
@ -2879,21 +2911,6 @@ std::optional<std::string> EvalState::resolveSearchPathPath(const SearchPath::Pa
}
Expr * EvalState::parse(
char * text,
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
std::shared_ptr<StaticEnv> & staticEnv)
{
auto result = parseExprFromBuf(text, length, origin, basePath, symbols, positions, exprSymbols);
result->bindVars(*this, staticEnv);
return result;
}
std::string ExternalValueBase::coerceToString(EvalState & state, const PosIdx & pos, NixStringContext & context, bool copyMore, bool copyToStore) const
{
state.error<TypeError>(

View file

@ -3,20 +3,20 @@
#include "attr-set.hh"
#include "eval-error.hh"
#include "gc-alloc.hh"
#include "types.hh"
#include "value.hh"
#include "nixexpr.hh"
#include "symbol-table.hh"
#include "config.hh"
#include "experimental-features.hh"
#include "input-accessor.hh"
#include "search-path.hh"
#include "repl-exit-status.hh"
#include <gc/gc_allocator.h>
#include <map>
#include <optional>
#include <unordered_map>
#include <mutex>
#include <functional>
namespace nix {
@ -33,12 +33,15 @@ class EvalState;
class StorePath;
struct SingleDerivedPath;
enum RepairFlag : bool;
struct MemoryInputAccessor;
namespace eval_cache {
class EvalCache;
}
/**
* Function that implements a primop.
*/
typedef void (* PrimOpFun) (EvalState & state, const PosIdx pos, Value * * args, Value & v);
using PrimOpImpl = void(EvalState & state, PosIdx pos, Value ** args, Value & v);
/**
* Info about a primitive operation, and its implementation
@ -72,7 +75,7 @@ struct PrimOp
/**
* Implementation of the primop.
*/
std::function<std::remove_pointer<PrimOpFun>::type> fun;
std::function<PrimOpImpl> fun;
/**
* Optional experimental for this to be gated on.
@ -111,11 +114,7 @@ struct Constant
bool impureOnly = false;
};
#if HAVE_BOEHMGC
typedef std::map<std::string, Value *, std::less<std::string>, traceable_allocator<std::pair<const std::string, Value *> > > ValMap;
#else
typedef std::map<std::string, Value *> ValMap;
#endif
using ValMap = GcMap<std::string, Value *>;
struct Env
{
@ -210,7 +209,7 @@ public:
/**
* Debugger
*/
ReplExitStatus (* debugRepl)(ref<EvalState> es, const ValMap & extraEnv);
std::function<ReplExitStatus(ref<EvalState> es, ValMap const & extraEnv)> debugRepl;
bool debugStop;
bool inDebugger = false;
int trylevel;
@ -234,6 +233,11 @@ public:
return *new EvalErrorBuilder<T>(*this, args...);
}
/**
* A cache for evaluation caches, so as to reuse the same root value if possible
*/
std::map<const Hash, ref<eval_cache::EvalCache>> evalCaches;
private:
/* Cache for calls to addToStore(); maps source paths to the store
@ -243,21 +247,13 @@ private:
/**
* A cache from path names to parse trees.
*/
#if HAVE_BOEHMGC
typedef std::map<SourcePath, Expr *, std::less<SourcePath>, traceable_allocator<std::pair<const SourcePath, Expr *>>> FileParseCache;
#else
typedef std::map<SourcePath, Expr *> FileParseCache;
#endif
using FileParseCache = GcMap<SourcePath, Expr *>;
FileParseCache fileParseCache;
/**
* A cache from path names to values.
*/
#if HAVE_BOEHMGC
typedef std::map<SourcePath, Value, std::less<SourcePath>, traceable_allocator<std::pair<const SourcePath, Value>>> FileEvalCache;
#else
typedef std::map<SourcePath, Value> FileEvalCache;
#endif
using FileEvalCache = GcMap<SourcePath, Value>;
FileEvalCache fileEvalCache;
SearchPath searchPath;
@ -728,15 +724,15 @@ private:
bool countCalls;
typedef std::map<std::string, size_t> PrimOpCalls;
using PrimOpCalls = std::map<std::string, size_t>;
PrimOpCalls primOpCalls;
typedef std::map<ExprLambda *, size_t> FunctionCalls;
using FunctionCalls = std::map<ExprLambda *, size_t>;
FunctionCalls functionCalls;
void incrFunctionCall(ExprLambda * fun);
typedef std::map<PosIdx, size_t> AttrSelects;
using AttrSelects = std::map<PosIdx, size_t>;
AttrSelects attrSelects;
friend struct ExprOpUpdate;
@ -778,7 +774,7 @@ std::string showType(const Value & v);
*/
SourcePath resolveExprPath(SourcePath path);
static const std::string corepkgsPrefix{"/__corepkgs__/"};
static constexpr std::string_view corepkgsPrefix{"/__corepkgs__/"};
}

View file

@ -51,30 +51,46 @@ void ConfigFile::apply()
else
assert(false);
if (!whitelist.count(baseName) && !nix::fetchSettings.acceptFlakeConfig) {
bool trusted = false;
auto trustedList = readTrustedList();
auto tlname = get(trustedList, name);
if (auto saved = tlname ? get(*tlname, valueS) : nullptr) {
trusted = *saved;
printInfo("Using saved setting for '%s = %s' from ~/.local/share/nix/trusted-settings.json.", name, valueS);
} else {
// FIXME: filter ANSI escapes, newlines, \r, etc.
if (std::tolower(logger->ask(fmt("do you want to allow configuration setting '%s' to be set to '" ANSI_RED "%s" ANSI_NORMAL "' (y/N)?", name, valueS)).value_or('n')) == 'y') {
trusted = true;
}
if (std::tolower(logger->ask(fmt("do you want to permanently mark this value as %s (y/N)?", trusted ? "trusted": "untrusted" )).value_or('n')) == 'y') {
trustedList[name][valueS] = trusted;
writeTrustedList(trustedList);
}
bool trusted = whitelist.count(baseName);
if (!trusted) {
switch (nix::fetchSettings.acceptFlakeConfig) {
case AcceptFlakeConfig::True: {
trusted = true;
break;
}
if (!trusted) {
warn("ignoring untrusted flake configuration setting '%s'.\nPass '%s' to trust it", name, "--accept-flake-config");
continue;
case AcceptFlakeConfig::Ask: {
auto trustedList = readTrustedList();
auto tlname = get(trustedList, name);
if (auto saved = tlname ? get(*tlname, valueS) : nullptr) {
trusted = *saved;
printInfo("Using saved setting for '%s = %s' from ~/.local/share/nix/trusted-settings.json.", name, valueS);
} else {
// FIXME: filter ANSI escapes, newlines, \r, etc.
if (std::tolower(logger->ask(fmt("Do you want to allow configuration setting '%s' to be set to '" ANSI_RED "%s" ANSI_NORMAL "' (y/N)? This may allow the flake to gain root, see the nix.conf manual page.", name, valueS)).value_or('n')) == 'y') {
trusted = true;
} else {
warn("you can set '%s' to '%b' to automatically reject configuration options supplied by flakes", "accept-flake-config", false);
}
if (std::tolower(logger->ask(fmt("do you want to permanently mark this value as %s (y/N)?", trusted ? "trusted": "untrusted" )).value_or('n')) == 'y') {
trustedList[name][valueS] = trusted;
writeTrustedList(trustedList);
}
}
break;
}
case nix::AcceptFlakeConfig::False: {
trusted = false;
break;
};
}
}
globalConfig.set(name, valueS);
if (trusted) {
debug("accepting trusted flake configuration setting '%s'", name);
globalConfig.set(name, valueS);
} else {
warn("ignoring untrusted flake configuration setting '%s', pass '%s' to trust it (may allow the flake to gain root, see the nix.conf manual page)", name, "--accept-flake-config");
}
}
}

View file

@ -137,9 +137,16 @@ static FlakeInput parseFlakeInput(EvalState & state,
case nBool:
attrs.emplace(state.symbols[attr.name], Explicit<bool> { attr.value->boolean });
break;
case nInt:
attrs.emplace(state.symbols[attr.name], (long unsigned int)attr.value->integer);
case nInt: {
auto intValue = attr.value->integer.value;
if (intValue < 0) {
state.error<EvalError>("negative value given for flake input attribute %1%: %2%", state.symbols[attr.name], intValue).debugThrow();
}
uint64_t asUnsigned = intValue;
attrs.emplace(state.symbols[attr.name], asUnsigned);
break;
}
default:
state.error<TypeError>("flake input attribute '%s' is %s while a string, Boolean, or integer is expected",
state.symbols[attr.name], showType(*attr.value)).debugThrow();
@ -284,7 +291,7 @@ static Flake getFlake(
else if (setting.value->type() == nInt)
flake.config.settings.emplace(
state.symbols[setting.name],
state.forceInt(*setting.value, setting.pos, ""));
state.forceInt(*setting.value, setting.pos, "").value);
else if (setting.value->type() == nBool)
flake.config.settings.emplace(
state.symbols[setting.name],
@ -873,8 +880,14 @@ static void prim_flakeRefToString(
for (const auto & attr : *args[0]->attrs) {
auto t = attr.value->type();
if (t == nInt) {
attrs.emplace(state.symbols[attr.name],
(uint64_t) attr.value->integer);
auto intValue = attr.value->integer.value;
if (intValue < 0) {
state.error<EvalError>("negative value given for flake ref attr %1%: %2%", state.symbols[attr.name], intValue).atPos(pos).debugThrow();
}
uint64_t asUnsigned = intValue;
attrs.emplace(state.symbols[attr.name], asUnsigned);
} else if (t == nBool) {
attrs.emplace(state.symbols[attr.name],
Explicit<bool> { attr.value->boolean });

View file

@ -85,8 +85,8 @@ std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
+ "(?:#(" + queryRegex + "))?",
std::regex::ECMAScript);
static std::regex flakeRegex(
"((" + flakeIdRegexS + ")(?:/(?:" + refAndOrRevRegex + "))?)"
static std::regex flakeShorthandRegex(
flakeShorthandRegexS
+ "(?:#(" + queryRegex + "))?",
std::regex::ECMAScript);
@ -95,7 +95,7 @@ std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
/* Check if 'url' is a flake ID. This is an abbreviated syntax for
'flake:<flake-id>?ref=<ref>&rev=<rev>'. */
if (std::regex_match(url, match, flakeRegex)) {
if (std::regex_match(url, match, flakeShorthandRegex)) {
auto parsedURL = ParsedURL{
.url = url,
.base = "flake:" + match.str(1),

23
src/libexpr/gc-alloc.cc Normal file
View file

@ -0,0 +1,23 @@
#include "gc-alloc.hh"
#include <cstring>
#include <string_view>
namespace nix
{
char const * gcCopyStringIfNeeded(std::string_view toCopyFrom)
{
if (toCopyFrom.empty()) {
return "";
}
size_t const size = toCopyFrom.size();
char * cstr = gcAllocString(size + 1);
memcpy(cstr, toCopyFrom.data(), size);
cstr[size] = '\0';
return cstr;
}
}

150
src/libexpr/gc-alloc.hh Normal file
View file

@ -0,0 +1,150 @@
#pragma once
/// @file Aliases and wrapper functions that are transparently GC-enabled
/// if Lix is compiled with BoehmGC enabled.
#include <cstddef>
#include <cstring>
#include <list>
#include <map>
#include <new>
#include <string_view>
#include <vector>
#include "checked-arithmetic.hh"
#if HAVE_BOEHMGC
#include <functional> // std::less
#include <utility> // std::pair
#define GC_INCLUDE_NEW
#include <gc/gc.h>
#include <gc/gc_allocator.h>
#include <gc/gc_cpp.h>
/// calloc, transparently GC-enabled.
#define LIX_GC_CALLOC(size) GC_MALLOC(size)
/// strdup, transaprently GC-enabled.
#define LIX_GC_STRDUP(str) GC_STRDUP(str)
/// Atomic GC malloc() with GC enabled, or regular malloc() otherwise.
#define LIX_GC_MALLOC_ATOMIC(size) GC_MALLOC_ATOMIC(size)
namespace nix
{
/// Alias for std::map which uses BoehmGC's allocator conditional on this Lix
/// build having GC enabled.
template<typename KeyT, typename ValueT>
using GcMap = std::map<
KeyT,
ValueT,
std::less<KeyT>,
traceable_allocator<std::pair<KeyT const, ValueT>>
>;
/// Alias for std::vector which uses BoehmGC's allocator conditional on this Lix
/// build having GC enabled.
template<typename ItemT>
using GcVector = std::vector<ItemT, traceable_allocator<ItemT>>;
/// Alias for std::list which uses BoehmGC's allocator conditional on this Lix
/// build having GC enabled.
template<typename ItemT>
using GcList = std::list<ItemT, traceable_allocator<ItemT>>;
}
#else
#include <cstdlib>
/// calloc, transparently GC-enabled.
#define LIX_GC_CALLOC(size) calloc(size, 1)
/// strdup, transparently GC-enabled.
#define LIX_GC_STRDUP(str) strdup(str)
/// Atomic GC malloc() with GC enabled, or regular malloc() otherwise.
/// The returned memory must never contain pointers.
#define LIX_GC_MALLOC_ATOMIC(size) malloc(size)
namespace nix
{
/// Alias for std::map which uses BoehmGC's allocator conditional on this Lix
/// build having GC enabled.
template<typename KeyT, typename ValueT>
using GcMap = std::map<KeyT, ValueT>;
/// Alias for std::vector which uses BoehmGC's allocator conditional on this Lix
/// build having GC enabled.
template<typename ItemT>
using GcVector = std::vector<ItemT>;
/// Alias for std::list which uses BoehmGC's allocator conditional on this Lix
/// build having GC enabled.
template<typename ItemT>
using GcList = std::list<ItemT>;
}
#endif
namespace nix
{
[[gnu::always_inline]]
inline void * gcAllocBytes(size_t n)
{
// Note: various places expect the allocated memory to be zero.
// Hence: calloc().
void * ptr = LIX_GC_CALLOC(n);
if (ptr == nullptr) {
throw std::bad_alloc();
}
return ptr;
}
/// Typed, safe wrapper around calloc() (transparently GC-enabled). Allocates
/// enough for the requested count of the specified type. Also checks for
/// nullptr (and throws @ref std::bad_alloc), and casts the void pointer to
/// a pointer of the specified type, for type-convenient goodness.
template<typename T>
[[gnu::always_inline]]
inline T * gcAllocType(size_t howMany = 1)
{
// NOTE: size_t * size_t, which can definitely overflow.
// Unsigned integer overflow is definitely a bug, but isn't undefined
// behavior, so we can just check if we overflowed after the fact.
// However, people can and do request zero sized allocations, so we need
// to check that neither of our multiplicands were zero before complaining
// about it.
auto checkedSz = checked::Checked<size_t>(howMany) * sizeof(T);
size_t sz = checkedSz.valueWrapping();
if (checkedSz.overflowed()) {
// Congrats, you done did an overflow.
throw std::bad_alloc();
}
return static_cast<T *>(gcAllocBytes(sz));
}
/// GC-transparently allocates a buffer for a C-string of @ref size *bytes*,
/// meaning you should include the size needed by the NUL terminator in the
/// passed size. Memory allocated with this function must never contain other
/// pointers.
inline char * gcAllocString(size_t size)
{
char * cstr = static_cast<char *>(LIX_GC_MALLOC_ATOMIC(size));
if (cstr == nullptr) {
throw std::bad_alloc();
}
return cstr;
}
/// Returns a C-string copied from @ref toCopyFrom, or a single, static empty
/// string if @ref toCopyFrom is also empty.
char const * gcCopyStringIfNeeded(std::string_view toCopyFrom);
}

View file

@ -45,7 +45,7 @@ DrvInfo::DrvInfo(EvalState & state, ref<Store> store, const std::string & drvPat
}
std::string DrvInfo::queryName() const
std::string DrvInfo::queryName()
{
if (name == "" && attrs) {
auto i = attrs->find(state->sName);
@ -56,7 +56,7 @@ std::string DrvInfo::queryName() const
}
std::string DrvInfo::querySystem() const
std::string DrvInfo::querySystem()
{
if (system == "" && attrs) {
auto i = attrs->find(state->sSystem);
@ -66,7 +66,7 @@ std::string DrvInfo::querySystem() const
}
std::optional<StorePath> DrvInfo::queryDrvPath() const
std::optional<StorePath> DrvInfo::queryDrvPath()
{
if (!drvPath && attrs) {
Bindings::iterator i = attrs->find(state->sDrvPath);
@ -80,7 +80,7 @@ std::optional<StorePath> DrvInfo::queryDrvPath() const
}
StorePath DrvInfo::requireDrvPath() const
StorePath DrvInfo::requireDrvPath()
{
if (auto drvPath = queryDrvPath())
return *drvPath;
@ -88,7 +88,7 @@ StorePath DrvInfo::requireDrvPath() const
}
StorePath DrvInfo::queryOutPath() const
StorePath DrvInfo::queryOutPath()
{
if (!outPath && attrs) {
Bindings::iterator i = attrs->find(state->sOutPath);
@ -164,7 +164,7 @@ DrvInfo::Outputs DrvInfo::queryOutputs(bool withPaths, bool onlyOutputsToInstall
}
std::string DrvInfo::queryOutputName() const
std::string DrvInfo::queryOutputName()
{
if (outputName == "" && attrs) {
Bindings::iterator i = attrs->find(state->sOutputName);
@ -241,8 +241,8 @@ NixInt DrvInfo::queryMetaInt(const std::string & name, NixInt def)
if (v->type() == nString) {
/* Backwards compatibility with before we had support for
integer meta fields. */
if (auto n = string2Int<NixInt>(v->string.s))
return *n;
if (auto n = string2Int<NixInt::Inner>(v->string.s))
return NixInt{*n};
}
return def;
}

View file

@ -19,11 +19,11 @@ public:
private:
EvalState * state;
mutable std::string name;
mutable std::string system;
mutable std::optional<std::optional<StorePath>> drvPath;
mutable std::optional<StorePath> outPath;
mutable std::string outputName;
std::string name;
std::string system;
std::optional<std::optional<StorePath>> drvPath;
std::optional<StorePath> outPath;
std::string outputName;
Outputs outputs;
/**
@ -47,12 +47,12 @@ public:
DrvInfo(EvalState & state, std::string attrPath, Bindings * attrs);
DrvInfo(EvalState & state, ref<Store> store, const std::string & drvPathWithOutputs);
std::string queryName() const;
std::string querySystem() const;
std::optional<StorePath> queryDrvPath() const;
StorePath requireDrvPath() const;
StorePath queryOutPath() const;
std::string queryOutputName() const;
std::string queryName();
std::string querySystem();
std::optional<StorePath> queryDrvPath();
StorePath requireDrvPath();
StorePath queryOutPath();
std::string queryOutputName();
/**
* Return the unordered map of output names to (optional) output paths.
* The "outputs to install" are determined by `meta.outputsToInstall`.
@ -80,13 +80,7 @@ public:
bool hasFailed() { return failed; };
};
#if HAVE_BOEHMGC
typedef std::list<DrvInfo, traceable_allocator<DrvInfo>> DrvInfos;
#else
typedef std::list<DrvInfo> DrvInfos;
#endif
using DrvInfos = GcList<DrvInfo>;
/**
* If value `v` denotes a derivation, return a DrvInfo object

View file

@ -2,6 +2,7 @@
#include "value.hh"
#include "eval.hh"
#include <limits>
#include <variant>
#include <nlohmann/json.hpp>
@ -81,42 +82,46 @@ class JSONSax : nlohmann::json_sax<json> {
public:
JSONSax(EvalState & state, Value & v) : state(state), rs(new JSONState(&v)) {};
bool null()
bool null() override
{
rs->value(state).mkNull();
rs->add();
return true;
}
bool boolean(bool val)
bool boolean(bool val) override
{
rs->value(state).mkBool(val);
rs->add();
return true;
}
bool number_integer(number_integer_t val)
bool number_integer(number_integer_t val) override
{
rs->value(state).mkInt(val);
rs->add();
return true;
}
bool number_unsigned(number_unsigned_t val)
bool number_unsigned(number_unsigned_t val_) override
{
if (val_ > std::numeric_limits<NixInt::Inner>::max()) {
throw Error("unsigned json number %1% outside of Nix integer range", val_);
}
NixInt::Inner val = val_;
rs->value(state).mkInt(val);
rs->add();
return true;
}
bool number_float(number_float_t val, const string_t & s)
bool number_float(number_float_t val, const string_t & s) override
{
rs->value(state).mkFloat(val);
rs->add();
return true;
}
bool string(string_t & val)
bool string(string_t & val) override
{
rs->value(state).mkString(val);
rs->add();
@ -124,7 +129,7 @@ public:
}
#if NLOHMANN_JSON_VERSION_MAJOR >= 3 && NLOHMANN_JSON_VERSION_MINOR >= 8
bool binary(binary_t&)
bool binary(binary_t&) override
{
// This function ought to be unreachable
assert(false);
@ -132,35 +137,37 @@ public:
}
#endif
bool start_object(std::size_t len)
bool start_object(std::size_t len) override
{
rs = std::make_unique<JSONObjectState>(std::move(rs));
return true;
}
bool key(string_t & name)
bool key(string_t & name) override
{
dynamic_cast<JSONObjectState*>(rs.get())->key(name, state);
return true;
}
bool end_object() {
bool end_object() override {
rs = rs->resolve(state);
rs->add();
return true;
}
bool end_array() {
bool end_array() override {
return end_object();
}
bool start_array(size_t len) {
bool start_array(size_t len) override {
rs = std::make_unique<JSONListState>(std::move(rs),
len != std::numeric_limits<size_t>::max() ? len : 128);
return true;
}
bool parse_error(std::size_t, const std::string&, const nlohmann::detail::exception& ex) {
bool
parse_error(std::size_t, const std::string &, const nlohmann::detail::exception & ex) override
{
throw JSONParseError("%s", ex.what());
}
};

View file

@ -1,302 +0,0 @@
%option reentrant bison-bridge bison-locations
%option align
%option noyywrap
%option never-interactive
%option stack
%option nodefault
%option nounput noyy_top_state
%s DEFAULT
%x STRING
%x IND_STRING
%x INPATH
%x INPATH_SLASH
%x PATH_START
%{
#ifdef __clang__
#pragma clang diagnostic ignored "-Wunneeded-internal-declaration"
#endif
// yacc generates code that uses unannotated fallthrough.
#pragma GCC diagnostic ignored "-Wimplicit-fallthrough"
#ifdef __clang__
#pragma clang diagnostic ignored "-Wimplicit-fallthrough"
#endif
#include "nixexpr.hh"
#include "parser-tab.hh"
#include "strings.hh"
using namespace nix;
#define THROW(...) \
do { \
state->error.reset(new auto(__VA_ARGS__)); \
return YYerror; \
} while (0)
namespace nix {
#define CUR_POS state->at(*yylloc)
static void initLoc(YYLTYPE * loc)
{
loc->first_line = loc->last_line = 0;
loc->first_column = loc->last_column = 0;
}
static void adjustLoc(YYLTYPE * loc, const char * s, size_t len)
{
loc->stash();
loc->first_column = loc->last_column;
loc->last_column += len;
}
// we make use of the fact that the parser receives a private copy of the input
// string and can munge around in it.
static StringToken unescapeStr(SymbolTable & symbols, char * s, size_t length)
{
char * result = s;
char * t = s;
char c;
// the input string is terminated with *two* NULs, so we can safely take
// *one* character after the one being checked against.
while ((c = *s++)) {
if (c == '\\') {
c = *s++;
if (c == 'n') *t = '\n';
else if (c == 'r') *t = '\r';
else if (c == 't') *t = '\t';
else *t = c;
}
else if (c == '\r') {
/* Normalise CR and CR/LF into LF. */
*t = '\n';
if (*s == '\n') s++; /* cr/lf */
}
else *t = c;
t++;
}
return {result, size_t(t - result)};
}
}
#define YY_USER_INIT initLoc(yylloc)
#define YY_USER_ACTION adjustLoc(yylloc, yytext, yyleng);
#define PUSH_STATE(state) yy_push_state(state, yyscanner)
#define POP_STATE() yy_pop_state(yyscanner)
%}
ANY .|\n
ID [a-zA-Z\_][a-zA-Z0-9\_\'\-]*
INT [0-9]+
FLOAT (([1-9][0-9]*\.[0-9]*)|(0?\.[0-9]+))([Ee][+-]?[0-9]+)?
PATH_CHAR [a-zA-Z0-9\.\_\-\+]
PATH {PATH_CHAR}*(\/{PATH_CHAR}+)+\/?
PATH_SEG {PATH_CHAR}*\/
HPATH \~(\/{PATH_CHAR}+)+\/?
HPATH_START \~\/
SPATH \<{PATH_CHAR}+(\/{PATH_CHAR}+)*\>
URI [a-zA-Z][a-zA-Z0-9\+\-\.]*\:[a-zA-Z0-9\%\/\?\:\@\&\=\+\$\,\-\_\.\!\~\*\']+
%%
if { return IF; }
then { return THEN; }
else { return ELSE; }
assert { return ASSERT; }
with { return WITH; }
let { return LET; }
in { return IN; }
rec { return REC; }
inherit { return INHERIT; }
or { return OR_KW; }
\.\.\. { return ELLIPSIS; }
\=\= { return EQ; }
\!\= { return NEQ; }
\<\= { return LEQ; }
\>\= { return GEQ; }
\&\& { return AND; }
\|\| { return OR; }
\-\> { return IMPL; }
\/\/ { return UPDATE; }
\+\+ { return CONCAT; }
{ID} { yylval->id = {yytext, (size_t) yyleng}; return ID; }
{INT} { errno = 0;
std::optional<int64_t> numMay = string2Int<int64_t>(yytext);
if (numMay.has_value()) {
yylval->n = *numMay;
} else {
THROW(ParseError(ErrorInfo{
.msg = HintFmt("invalid integer '%1%'", yytext),
.pos = state->positions[CUR_POS],
}));
}
return INT;
}
{FLOAT} { errno = 0;
yylval->nf = strtod(yytext, 0);
if (errno != 0)
THROW(ParseError(ErrorInfo{
.msg = HintFmt("invalid float '%1%'", yytext),
.pos = state->positions[CUR_POS],
}));
return FLOAT;
}
\$\{ { PUSH_STATE(DEFAULT); return DOLLAR_CURLY; }
\} { /* State INITIAL only exists at the bottom of the stack and is
used as a marker. DEFAULT replaces it everywhere else.
Popping when in INITIAL state causes an empty stack exception,
so don't */
if (YYSTATE != INITIAL)
POP_STATE();
return '}';
}
\{ { PUSH_STATE(DEFAULT); return '{'; }
\" { PUSH_STATE(STRING); return '"'; }
<STRING>([^\$\"\\]|\$[^\{\"\\]|\\{ANY}|\$\\{ANY})*\$/\" |
<STRING>([^\$\"\\]|\$[^\{\"\\]|\\{ANY}|\$\\{ANY})+ {
/* It is impossible to match strings ending with '$' with one
regex because trailing contexts are only valid at the end
of a rule. (A sane but undocumented limitation.) */
yylval->str = unescapeStr(state->symbols, yytext, yyleng);
return STR;
}
<STRING>\$\{ { PUSH_STATE(DEFAULT); return DOLLAR_CURLY; }
<STRING>\" { POP_STATE(); return '"'; }
<STRING>\$|\\|\$\\ {
/* This can only occur when we reach EOF, otherwise the above
(...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
This is technically invalid, but we leave the problem to the
parser who fails with exact location. */
return EOF;
}
\'\'(\ *\n)? { PUSH_STATE(IND_STRING); return IND_STRING_OPEN; }
<IND_STRING>([^\$\']|\$[^\{\']|\'[^\'\$])+ {
yylval->str = {yytext, (size_t) yyleng, true};
return IND_STR;
}
<IND_STRING>\'\'\$ |
<IND_STRING>\$ {
yylval->str = {"$", 1};
return IND_STR;
}
<IND_STRING>\'\'\' {
yylval->str = {"''", 2};
return IND_STR;
}
<IND_STRING>\'\'\\{ANY} {
yylval->str = unescapeStr(state->symbols, yytext + 2, yyleng - 2);
return IND_STR;
}
<IND_STRING>\$\{ { PUSH_STATE(DEFAULT); return DOLLAR_CURLY; }
<IND_STRING>\'\' { POP_STATE(); return IND_STRING_CLOSE; }
<IND_STRING>\' {
yylval->str = {"'", 1};
return IND_STR;
}
{PATH_SEG}\$\{ |
{HPATH_START}\$\{ {
PUSH_STATE(PATH_START);
yyless(0);
yylloc->unstash();
}
<PATH_START>{PATH_SEG} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = {yytext, (size_t) yyleng};
return PATH;
}
<PATH_START>{HPATH_START} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = {yytext, (size_t) yyleng};
return HPATH;
}
{PATH} {
if (yytext[yyleng-1] == '/')
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = {yytext, (size_t) yyleng};
return PATH;
}
{HPATH} {
if (yytext[yyleng-1] == '/')
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = {yytext, (size_t) yyleng};
return HPATH;
}
<INPATH,INPATH_SLASH>\$\{ {
POP_STATE();
PUSH_STATE(INPATH);
PUSH_STATE(DEFAULT);
return DOLLAR_CURLY;
}
<INPATH,INPATH_SLASH>{PATH}|{PATH_SEG}|{PATH_CHAR}+ {
POP_STATE();
if (yytext[yyleng-1] == '/')
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->str = {yytext, (size_t) yyleng};
return STR;
}
<INPATH>{ANY} |
<INPATH><<EOF>> {
/* if we encounter a non-path character we inform the parser that the path has
ended with a PATH_END token and re-parse this character in the default
context (it may be ')', ';', or something of that sort) */
POP_STATE();
yyless(0);
yylloc->unstash();
return PATH_END;
}
<INPATH_SLASH>{ANY} |
<INPATH_SLASH><<EOF>> {
THROW(ParseError(ErrorInfo{
.msg = HintFmt("path has a trailing slash"),
.pos = state->positions[CUR_POS],
}));
}
{SPATH} { yylval->path = {yytext, (size_t) yyleng}; return SPATH; }
{URI} { yylval->uri = {yytext, (size_t) yyleng}; return URI; }
[ \t\r\n]+ /* eat up whitespace */
\#[^\r\n]* /* single-line comments */
\/\*([^*]|\*+[^*/])*\*+\/ /* long comments */
{ANY} {
/* Don't return a negative number, as this will cause
Bison to stop parsing without an error. */
return (unsigned char) yytext[0];
}
%%

View file

@ -1,54 +1,3 @@
parser_tab = custom_target(
input : 'parser.y',
output : [
'parser-tab.cc',
'parser-tab.hh',
],
command : [
'bison',
'-v',
'-o',
'@OUTPUT0@',
'@INPUT@',
'-d',
],
# NOTE(Qyriad): Meson doesn't support installing only part of a custom target, so we add
# an install script below which removes parser-tab.cc.
install : true,
install_dir : includedir / 'lix/libexpr',
)
lexer_tab = custom_target(
input : [
'lexer.l',
parser_tab,
],
output : [
'lexer-tab.cc',
'lexer-tab.hh',
],
command : [
'flex',
'--outfile',
'@OUTPUT0@',
'--header-file=' + '@OUTPUT1@',
'@INPUT0@',
],
# NOTE(Qyriad): Meson doesn't support installing only part of a custom target, so we add
# an install script below which removes lexer-tab.cc.
install : true,
install_dir : includedir / 'lix/libexpr',
)
# TODO(Qyriad): When the parser and lexer are rewritten this should be removed.
# NOTE(Qyriad): We do this this way instead of an inline bash or rm command
# due to subtleties in Meson. Check the comments in cleanup-install.bash for details.
meson.add_install_script(
bash,
meson.project_source_root() / 'meson/cleanup-install.bash',
'@0@'.format(includedir),
)
libexpr_generated_headers = [
gen_header.process('primops/derivation.nix', preserve_path_from : meson.current_source_dir()),
]
@ -73,8 +22,10 @@ libexpr_sources = files(
'eval.cc',
'function-trace.cc',
'get-drvs.cc',
'gc-alloc.cc',
'json-to-value.cc',
'nixexpr.cc',
'parser/parser.cc',
'paths.cc',
'primops.cc',
'print-ambiguous.cc',
@ -108,9 +59,12 @@ libexpr_headers = files(
'function-trace.hh',
'gc-small-vector.hh',
'get-drvs.hh',
'gc-alloc.hh',
'json-to-value.hh',
'nixexpr.hh',
'parser-state.hh',
'parser/change_head.hh',
'parser/grammar.hh',
'parser/state.hh',
'pos-idx.hh',
'pos-table.hh',
'primops.hh',
@ -129,8 +83,6 @@ libexpr_headers = files(
libexpr = library(
'lixexpr',
libexpr_sources,
parser_tab,
lexer_tab,
libexpr_generated_headers,
dependencies : [
liblixutil,

View file

@ -50,6 +50,13 @@ void ExprVar::show(const SymbolTable & symbols, std::ostream & str) const
str << symbols[name];
}
void ExprInheritFrom::show(SymbolTable const & symbols, std::ostream & str) const
{
str << "(/* expanded inherit (expr) */ ";
fromExpr->show(symbols, str);
str << ")";
}
void ExprSelect::show(const SymbolTable & symbols, std::ostream & str) const
{
str << "(";

View file

@ -75,6 +75,7 @@ struct ExprInt : Expr
NixInt n;
Value v;
ExprInt(NixInt n) : n(n) { v.mkInt(n); };
ExprInt(NixInt::Inner n) : n(n) { v.mkInt(n); };
Value * maybeThunk(EvalState & state, Env & env) override;
COMMON_METHODS
};
@ -144,21 +145,35 @@ struct ExprVar : Expr
*/
struct ExprInheritFrom : ExprVar
{
ExprInheritFrom(PosIdx pos, Displacement displ): ExprVar(pos, {})
ref<Expr> fromExpr;
ExprInheritFrom(PosIdx pos, Displacement displ, ref<Expr> fromExpr)
: ExprVar(pos, {}), fromExpr(fromExpr)
{
this->level = 0;
this->displ = displ;
this->fromWith = nullptr;
}
void bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env);
void show(SymbolTable const & symbols, std::ostream & str) const override;
void bindVars(EvalState & es, const std::shared_ptr<const StaticEnv> & env) override;
};
struct ExprSelect : Expr
{
PosIdx pos;
std::unique_ptr<Expr> e, def;
/** The expression attributes are being selected on. e.g. `foo` in `foo.bar.baz`. */
std::unique_ptr<Expr> e;
/** A default value specified with `or`, if the selected attr doesn't exist.
* e.g. `bix` in `foo.bar.baz or bix`
*/
std::unique_ptr<Expr> def;
/** The path of attributes being selected. e.g. `bar.baz` in `foo.bar.baz.` */
AttrPath attrPath;
ExprSelect(const PosIdx & pos, std::unique_ptr<Expr> e, AttrPath attrPath, std::unique_ptr<Expr> def) : pos(pos), e(std::move(e)), def(std::move(def)), attrPath(std::move(attrPath)) { };
ExprSelect(const PosIdx & pos, std::unique_ptr<Expr> e, Symbol name) : pos(pos), e(std::move(e)) { attrPath.push_back(AttrName(name)); };
PosIdx getPos() const override { return pos; }
@ -212,7 +227,7 @@ struct ExprAttrs : Expr
};
typedef std::map<Symbol, AttrDef> AttrDefs;
AttrDefs attrs;
std::unique_ptr<std::vector<std::unique_ptr<Expr>>> inheritFromExprs;
std::unique_ptr<std::vector<ref<Expr>>> inheritFromExprs;
struct DynamicAttrDef {
std::unique_ptr<Expr> nameExpr, valueExpr;
PosIdx pos;

View file

@ -1,503 +0,0 @@
%glr-parser
%define api.pure
%locations
%define parse.error verbose
%defines
/* %no-lines */
%parse-param { void * scanner }
%parse-param { nix::ParserState * state }
%lex-param { void * scanner }
%lex-param { nix::ParserState * state }
%expect 1
%expect-rr 1
%code requires {
#ifndef BISON_HEADER
#define BISON_HEADER
#include <variant>
#include "finally.hh"
#include "users.hh"
#include "nixexpr.hh"
#include "eval.hh"
#include "eval-settings.hh"
#include "globals.hh"
#include "parser-state.hh"
#define YYLTYPE ::nix::ParserLocation
#define YY_DECL int yylex \
(YYSTYPE * yylval_param, YYLTYPE * yylloc_param, yyscan_t yyscanner, nix::ParserState * state)
namespace nix {
Expr * parseExprFromBuf(
char * text,
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
SymbolTable & symbols,
PosTable & positions,
const Expr::AstSymbols & astSymbols);
}
#endif
}
%{
#include "parser-tab.hh"
#include "lexer-tab.hh"
YY_DECL;
using namespace nix;
#define CUR_POS state->at(*yylocp)
// otherwise destructors cause compiler errors
#pragma GCC diagnostic ignored "-Wswitch-enum"
#define THROW(err, ...) \
do { \
state->error.reset(new auto(err)); \
[](auto... d) { (delete d, ...); }(__VA_ARGS__); \
YYABORT; \
} while (0)
void yyerror(YYLTYPE * loc, yyscan_t scanner, ParserState * state, const char * error)
{
if (std::string_view(error).starts_with("syntax error, unexpected end of file")) {
loc->first_column = loc->last_column;
loc->first_line = loc->last_line;
}
throw ParseError({
.msg = HintFmt(error),
.pos = state->positions[state->at(*loc)]
});
}
template<typename T>
static std::unique_ptr<T> unp(T * e)
{
return std::unique_ptr<T>(e);
}
template<typename T = std::unique_ptr<nix::Expr>, typename... Args>
static std::vector<T> vec(Args && ... args)
{
std::vector<T> result;
result.reserve(sizeof...(Args));
(result.emplace_back(std::forward<Args>(args)), ...);
return result;
}
%}
%union {
// !!! We're probably leaking stuff here.
nix::Expr * e;
nix::ExprList * list;
nix::ExprAttrs * attrs;
nix::Formals * formals;
nix::Formal * formal;
nix::NixInt n;
nix::NixFloat nf;
nix::StringToken id; // !!! -> Symbol
nix::StringToken path;
nix::StringToken uri;
nix::StringToken str;
std::vector<nix::AttrName> * attrNames;
std::vector<std::pair<nix::AttrName, nix::PosIdx>> * inheritAttrs;
std::vector<std::pair<nix::PosIdx, std::unique_ptr<nix::Expr>>> * string_parts;
std::vector<std::pair<nix::PosIdx, std::variant<std::unique_ptr<nix::Expr>, nix::StringToken>>> * ind_string_parts;
}
%destructor { delete $$; } <e>
%destructor { delete $$; } <list>
%destructor { delete $$; } <attrs>
%destructor { delete $$; } <formals>
%destructor { delete $$; } <formal>
%destructor { delete $$; } <attrNames>
%destructor { delete $$; } <inheritAttrs>
%destructor { delete $$; } <string_parts>
%destructor { delete $$; } <ind_string_parts>
%type <e> start
%type <e> expr expr_function expr_if expr_op
%type <e> expr_select expr_simple expr_app
%type <list> expr_list
%type <attrs> binds
%type <formals> formals
%type <formal> formal
%type <attrNames> attrpath
%type <inheritAttrs> attrs
%type <string_parts> string_parts_interpolated
%type <ind_string_parts> ind_string_parts
%type <e> path_start string_parts string_attr
%type <id> attr
%token <id> ID
%token <str> STR IND_STR
%token <n> INT
%token <nf> FLOAT
%token <path> PATH HPATH SPATH PATH_END
%token <uri> URI
%token IF THEN ELSE ASSERT WITH LET IN REC INHERIT EQ NEQ AND OR IMPL OR_KW
%token DOLLAR_CURLY /* == ${ */
%token IND_STRING_OPEN IND_STRING_CLOSE
%token ELLIPSIS
%right IMPL
%left OR
%left AND
%nonassoc EQ NEQ
%nonassoc '<' '>' LEQ GEQ
%right UPDATE
%left NOT
%left '+' '-'
%left '*' '/'
%right CONCAT
%nonassoc '?'
%nonassoc NEGATE
%%
start: expr { state->result = $1; $$ = 0; };
expr: expr_function;
expr_function
: ID ':' expr_function
{ $$ = new ExprLambda(CUR_POS, state->symbols.create($1), nullptr, unp($3)); }
| '{' formals '}' ':' expr_function
{ if (auto e = state->validateFormals($2)) THROW(*e);
$$ = new ExprLambda(CUR_POS, unp($2), unp($5));
}
| '{' formals '}' '@' ID ':' expr_function
{
auto arg = state->symbols.create($5);
if (auto e = state->validateFormals($2, CUR_POS, arg)) THROW(*e, $2, $7);
$$ = new ExprLambda(CUR_POS, arg, unp($2), unp($7));
}
| ID '@' '{' formals '}' ':' expr_function
{
auto arg = state->symbols.create($1);
if (auto e = state->validateFormals($4, CUR_POS, arg)) THROW(*e, $4, $7);
$$ = new ExprLambda(CUR_POS, arg, unp($4), unp($7));
}
| ASSERT expr ';' expr_function
{ $$ = new ExprAssert(CUR_POS, unp($2), unp($4)); }
| WITH expr ';' expr_function
{ $$ = new ExprWith(CUR_POS, unp($2), unp($4)); }
| LET binds IN expr_function
{ if (!$2->dynamicAttrs.empty())
THROW(ParseError({
.msg = HintFmt("dynamic attributes not allowed in let"),
.pos = state->positions[CUR_POS]
}), $2, $4);
$$ = new ExprLet(unp($2), unp($4));
}
| expr_if
;
expr_if
: IF expr THEN expr ELSE expr { $$ = new ExprIf(CUR_POS, unp($2), unp($4), unp($6)); }
| expr_op
;
expr_op
: '!' expr_op %prec NOT { $$ = new ExprOpNot(unp($2)); }
| '-' expr_op %prec NEGATE { $$ = new ExprCall(CUR_POS, std::make_unique<ExprVar>(state->s.sub), vec(std::make_unique<ExprInt>(0), unp($2))); }
| expr_op EQ expr_op { $$ = new ExprOpEq(unp($1), unp($3)); }
| expr_op NEQ expr_op { $$ = new ExprOpNEq(unp($1), unp($3)); }
| expr_op '<' expr_op { $$ = new ExprCall(state->at(@2), std::make_unique<ExprVar>(state->s.lessThan), vec($1, $3)); }
| expr_op LEQ expr_op { $$ = new ExprOpNot(std::make_unique<ExprCall>(state->at(@2), std::make_unique<ExprVar>(state->s.lessThan), vec($3, $1))); }
| expr_op '>' expr_op { $$ = new ExprCall(state->at(@2), std::make_unique<ExprVar>(state->s.lessThan), vec($3, $1)); }
| expr_op GEQ expr_op { $$ = new ExprOpNot(std::make_unique<ExprCall>(state->at(@2), std::make_unique<ExprVar>(state->s.lessThan), vec($1, $3))); }
| expr_op AND expr_op { $$ = new ExprOpAnd(state->at(@2), unp($1), unp($3)); }
| expr_op OR expr_op { $$ = new ExprOpOr(state->at(@2), unp($1), unp($3)); }
| expr_op IMPL expr_op { $$ = new ExprOpImpl(state->at(@2), unp($1), unp($3)); }
| expr_op UPDATE expr_op { $$ = new ExprOpUpdate(state->at(@2), unp($1), unp($3)); }
| expr_op '?' attrpath { $$ = new ExprOpHasAttr(unp($1), std::move(*$3)); delete $3; }
| expr_op '+' expr_op
{ $$ = new ExprConcatStrings(state->at(@2), false, vec<std::pair<PosIdx, std::unique_ptr<Expr>>>(std::pair(state->at(@1), unp($1)), std::pair(state->at(@3), unp($3)))); }
| expr_op '-' expr_op { $$ = new ExprCall(state->at(@2), std::make_unique<ExprVar>(state->s.sub), vec($1, $3)); }
| expr_op '*' expr_op { $$ = new ExprCall(state->at(@2), std::make_unique<ExprVar>(state->s.mul), vec($1, $3)); }
| expr_op '/' expr_op { $$ = new ExprCall(state->at(@2), std::make_unique<ExprVar>(state->s.div), vec($1, $3)); }
| expr_op CONCAT expr_op { $$ = new ExprOpConcatLists(state->at(@2), unp($1), unp($3)); }
| expr_app
;
expr_app
: expr_app expr_select {
if (auto e2 = dynamic_cast<ExprCall *>($1)) {
e2->args.emplace_back($2);
$$ = $1;
} else
$$ = new ExprCall(CUR_POS, unp($1), vec(unp($2)));
}
| expr_select
;
expr_select
: expr_simple '.' attrpath
{ $$ = new ExprSelect(CUR_POS, unp($1), std::move(*$3), nullptr); delete $3; }
| expr_simple '.' attrpath OR_KW expr_select
{ $$ = new ExprSelect(CUR_POS, unp($1), std::move(*$3), unp($5)); delete $3; }
| /* Backwards compatibility: because Nixpkgs has a rarely used
function named or, allow stuff like map or [...]. */
expr_simple OR_KW
{ $$ = new ExprCall(CUR_POS, unp($1), vec(std::make_unique<ExprVar>(CUR_POS, state->s.or_))); }
| expr_simple
;
expr_simple
: ID {
std::string_view s = "__curPos";
if ($1.l == s.size() && strncmp($1.p, s.data(), s.size()) == 0)
$$ = new ExprPos(CUR_POS);
else
$$ = new ExprVar(CUR_POS, state->symbols.create($1));
}
| INT { $$ = new ExprInt($1); }
| FLOAT { $$ = new ExprFloat($1); }
| '"' string_parts '"' { $$ = $2; }
| IND_STRING_OPEN ind_string_parts IND_STRING_CLOSE {
$$ = state->stripIndentation(CUR_POS, std::move(*$2)).release();
delete $2;
}
| path_start PATH_END
| path_start string_parts_interpolated PATH_END {
$2->emplace($2->begin(), state->at(@1), $1);
$$ = new ExprConcatStrings(CUR_POS, false, std::move(*$2));
delete $2;
}
| SPATH {
std::string path($1.p + 1, $1.l - 2);
$$ = new ExprCall(CUR_POS,
std::make_unique<ExprVar>(state->s.findFile),
vec(std::make_unique<ExprVar>(state->s.nixPath),
std::make_unique<ExprString>(std::move(path))));
}
| URI {
static bool noURLLiterals = experimentalFeatureSettings.isEnabled(Xp::NoUrlLiterals);
if (noURLLiterals)
THROW(ParseError({
.msg = HintFmt("URL literals are disabled"),
.pos = state->positions[CUR_POS]
}));
$$ = new ExprString(std::string($1));
}
| '(' expr ')' { $$ = $2; }
/* Let expressions `let {..., body = ...}' are just desugared
into `(rec {..., body = ...}).body'. */
| LET '{' binds '}'
{ $3->recursive = true; $$ = new ExprSelect(noPos, unp($3), state->s.body); }
| REC '{' binds '}'
{ $3->recursive = true; $$ = $3; }
| '{' binds '}'
{ $$ = $2; }
| '[' expr_list ']' { $$ = $2; }
;
string_parts
: STR { $$ = new ExprString(std::string($1)); }
| string_parts_interpolated
{ $$ = new ExprConcatStrings(CUR_POS, true, std::move(*$1));
delete $1;
}
| { $$ = new ExprString(""); }
;
string_parts_interpolated
: string_parts_interpolated STR
{ $$ = $1; $1->emplace_back(state->at(@2), new ExprString(std::string($2))); }
| string_parts_interpolated DOLLAR_CURLY expr '}' { $$ = $1; $1->emplace_back(state->at(@2), $3); }
| DOLLAR_CURLY expr '}' { $$ = new std::vector<std::pair<PosIdx, std::unique_ptr<Expr>>>; $$->emplace_back(state->at(@1), $2); }
| STR DOLLAR_CURLY expr '}' {
$$ = new std::vector<std::pair<PosIdx, std::unique_ptr<Expr>>>;
$$->emplace_back(state->at(@1), new ExprString(std::string($1)));
$$->emplace_back(state->at(@2), $3);
}
;
path_start
: PATH {
Path path(absPath({$1.p, $1.l}, state->basePath.path.abs()));
/* add back in the trailing '/' to the first segment */
if ($1.p[$1.l-1] == '/' && $1.l > 1)
path += "/";
$$ = new ExprPath(path);
}
| HPATH {
if (evalSettings.pureEval) {
THROW(Error(
"the path '%s' can not be resolved in pure mode",
std::string_view($1.p, $1.l)
));
}
Path path(getHome() + std::string($1.p + 1, $1.l - 1));
$$ = new ExprPath(path);
}
;
ind_string_parts
: ind_string_parts IND_STR { $$ = $1; $1->emplace_back(state->at(@2), $2); }
| ind_string_parts DOLLAR_CURLY expr '}' { $$ = $1; $1->emplace_back(state->at(@2), unp($3)); }
| { $$ = new std::vector<std::pair<PosIdx, std::variant<std::unique_ptr<Expr>, StringToken>>>; }
;
binds
: binds attrpath '=' expr ';'
{ $$ = $1;
if (auto e = state->addAttr($$, std::move(*$2), unp($4), state->at(@2))) THROW(*e, $1, $2);
delete $2;
}
| binds INHERIT attrs ';'
{ $$ = $1;
for (auto & [i, iPos] : *$3) {
if ($$->attrs.find(i.symbol) != $$->attrs.end())
THROW(state->dupAttr(i.symbol, iPos, $$->attrs[i.symbol].pos), $1);
$$->attrs.emplace(
i.symbol,
ExprAttrs::AttrDef(std::make_unique<ExprVar>(iPos, i.symbol), iPos, ExprAttrs::AttrDef::Kind::Inherited));
}
delete $3;
}
| binds INHERIT '(' expr ')' attrs ';'
{ $$ = $1;
if (!$$->inheritFromExprs)
$$->inheritFromExprs = std::make_unique<std::vector<std::unique_ptr<Expr>>>();
$$->inheritFromExprs->push_back(unp($4));
for (auto & [i, iPos] : *$6) {
if ($$->attrs.find(i.symbol) != $$->attrs.end())
THROW(state->dupAttr(i.symbol, iPos, $$->attrs[i.symbol].pos), $1);
auto from = std::make_unique<nix::ExprInheritFrom>(state->at(@4), $$->inheritFromExprs->size() - 1);
$$->attrs.emplace(
i.symbol,
ExprAttrs::AttrDef(
std::make_unique<ExprSelect>(iPos, std::move(from), i.symbol),
iPos,
ExprAttrs::AttrDef::Kind::InheritedFrom));
}
delete $6;
}
| { $$ = new ExprAttrs(state->at(@0)); }
;
attrs
: attrs attr { $$ = $1; $1->emplace_back(AttrName(state->symbols.create($2)), state->at(@2)); }
| attrs string_attr
{ $$ = $1;
ExprString * str = dynamic_cast<ExprString *>($2);
if (str) {
$$->emplace_back(AttrName(state->symbols.create(str->s)), state->at(@2));
delete str;
} else
THROW(ParseError({
.msg = HintFmt("dynamic attributes not allowed in inherit"),
.pos = state->positions[state->at(@2)]
}), $1, $2);
}
| { $$ = new std::vector<std::pair<AttrName, PosIdx>>; }
;
attrpath
: attrpath '.' attr { $$ = $1; $1->push_back(AttrName(state->symbols.create($3))); }
| attrpath '.' string_attr
{ $$ = $1;
ExprString * str = dynamic_cast<ExprString *>($3);
if (str) {
$$->push_back(AttrName(state->symbols.create(str->s)));
delete str;
} else
$$->emplace_back(unp($3));
}
| attr { $$ = new std::vector<AttrName>; $$->push_back(AttrName(state->symbols.create($1))); }
| string_attr
{ $$ = new std::vector<AttrName>;
ExprString *str = dynamic_cast<ExprString *>($1);
if (str) {
$$->push_back(AttrName(state->symbols.create(str->s)));
delete str;
} else
$$->emplace_back(unp($1));
}
;
attr
: ID
| OR_KW { $$ = {"or", 2}; }
;
string_attr
: '"' string_parts '"' { $$ = $2; }
| DOLLAR_CURLY expr '}' { $$ = $2; }
;
expr_list
: expr_list expr_select { $$ = $1; $1->elems.emplace_back($2); /* !!! dangerous */ }
| { $$ = new ExprList; }
;
formals
: formal ',' formals
{ $$ = $3; $$->formals.emplace_back(std::move(*$1)); delete $1; }
| formal
{ $$ = new Formals; $$->formals.emplace_back(std::move(*$1)); $$->ellipsis = false; delete $1; }
|
{ $$ = new Formals; $$->ellipsis = false; }
| ELLIPSIS
{ $$ = new Formals; $$->ellipsis = true; }
;
formal
: ID { $$ = new Formal{CUR_POS, state->symbols.create($1), nullptr}; }
| ID '?' expr { $$ = new Formal{CUR_POS, state->symbols.create($1), unp($3)}; }
;
%%
#include "eval.hh"
namespace nix {
Expr * parseExprFromBuf(
char * text,
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
SymbolTable & symbols,
PosTable & positions,
const Expr::AstSymbols & astSymbols)
{
yyscan_t scanner;
ParserState state {
.symbols = symbols,
.positions = positions,
.basePath = basePath,
.origin = positions.addOrigin(origin, length),
.s = astSymbols,
};
yylex_init(&scanner);
Finally _destroy([&] { yylex_destroy(scanner); });
yy_scan_buffer(text, length, scanner);
yyparse(scanner, &state);
if (state.error) {
delete state.result;
throw *state.error;
}
return state.result;
}
}

View file

@ -0,0 +1,66 @@
#pragma once
///@file
#include <tao/pegtl.hpp>
namespace nix::parser {
// modified copy of change_state, as the manual suggest for more involved
// state manipulation. we want to change only the first state parameter,
// and we care about the *initial* position of a rule application (not the
// past-the-end position as pegtl change_state provides)
template<typename NewState>
struct change_head : tao::pegtl::maybe_nothing
{
template<
typename Rule,
tao::pegtl::apply_mode A,
tao::pegtl::rewind_mode M,
template<typename...> class Action,
template<typename...> class Control,
typename ParseInput,
typename State,
typename... States
>
[[nodiscard]] static bool match(ParseInput & in, State && st, States &&... sts)
{
const auto begin = in.iterator();
if constexpr (std::is_constructible_v<NewState, State, States...>) {
NewState s(st, sts...);
if (tao::pegtl::match<Rule, A, M, Action, Control>(in, s, sts...)) {
if constexpr (A == tao::pegtl::apply_mode::action) {
_success<Action<Rule>>(0, begin, in, s, st, sts...);
}
return true;
}
return false;
} else if constexpr (std::is_default_constructible_v<NewState>) {
NewState s;
if (tao::pegtl::match<Rule, A, M, Action, Control>(in, s, sts...)) {
if constexpr (A == tao::pegtl::apply_mode::action) {
_success<Action<Rule>>(0, begin, in, s, st, sts...);
}
return true;
}
return false;
} else {
static_assert(decltype(sizeof(NewState))(), "unable to instantiate new state");
}
}
template<typename Target, typename ParseInput, typename... S>
static void _success(void *, auto & begin, ParseInput & in, S & ... sts)
{
const typename ParseInput::action_t at(begin, in);
Target::success(at, sts...);
}
template<typename Target, typename... S>
static void _success(decltype(Target::success0(std::declval<S &>()...), 0), auto &, auto &, S & ... sts)
{
Target::success0(sts...);
}
};
}

View file

@ -0,0 +1,707 @@
#pragma once
///@file
#include "tao/pegtl.hpp"
#include <type_traits>
#include <variant>
#include <boost/container/small_vector.hpp>
// NOTE
// nix line endings are \n, \r\n, \r. the grammar does not use eol or
// eolf rules in favor of reproducing the old flex lexer as faithfully as
// possible, and deferring calculation of positions to downstream users.
namespace nix::parser::grammar {
using namespace tao::pegtl;
namespace p = tao::pegtl;
// character classes
namespace c {
struct path : sor<
ranges<'a', 'z', 'A', 'Z', '0', '9'>,
one<'.', '_', '-', '+'>
> {};
struct path_sep : one<'/'> {};
struct id_first : ranges<'a', 'z', 'A', 'Z', '_'> {};
struct id_rest : sor<
ranges<'a', 'z', 'A', 'Z', '0', '9'>,
one<'_', '\'', '-'>
> {};
struct uri_scheme_first : ranges<'a', 'z', 'A', 'Z'> {};
struct uri_scheme_rest : sor<
ranges<'a', 'z', 'A', 'Z', '0', '9'>,
one<'+', '-', '.'>
> {};
struct uri_sep : one<':'> {};
struct uri_rest : sor<
ranges<'a', 'z', 'A', 'Z', '0', '9'>,
one<'%', '/', '?', ':', '@', '&', '=', '+', '$', ',', '-', '_', '.', '!', '~', '*', '\''>
> {};
}
// "tokens". PEGs don't really care about tokens, we merely use them as a convenient
// way of writing down keywords and a couple complicated syntax rules.
namespace t {
struct _extend_as_path : seq<
star<c::path>,
not_at<TAO_PEGTL_STRING("/*")>,
not_at<TAO_PEGTL_STRING("//")>,
c::path_sep,
sor<c::path, TAO_PEGTL_STRING("${")>
> {};
struct _extend_as_uri : seq<
star<c::uri_scheme_rest>,
c::uri_sep,
c::uri_rest
> {};
// keywords might be extended to identifiers, paths, or uris.
// NOTE this assumes that keywords are a-zA-Z only, otherwise uri schemes would never
// match correctly.
// NOTE not a simple seq<...> because this would report incorrect positions for
// keywords used inside must<> if a prefix of the keyword matches.
template<typename S>
struct _keyword : sor<
seq<
S,
not_at<c::id_rest>,
not_at<_extend_as_path>,
not_at<_extend_as_uri>
>,
failure
> {};
struct kw_if : _keyword<TAO_PEGTL_STRING("if")> {};
struct kw_then : _keyword<TAO_PEGTL_STRING("then")> {};
struct kw_else : _keyword<TAO_PEGTL_STRING("else")> {};
struct kw_assert : _keyword<TAO_PEGTL_STRING("assert")> {};
struct kw_with : _keyword<TAO_PEGTL_STRING("with")> {};
struct kw_let : _keyword<TAO_PEGTL_STRING("let")> {};
struct kw_in : _keyword<TAO_PEGTL_STRING("in")> {};
struct kw_rec : _keyword<TAO_PEGTL_STRING("rec")> {};
struct kw_inherit : _keyword<TAO_PEGTL_STRING("inherit")> {};
struct kw_or : _keyword<TAO_PEGTL_STRING("or")> {};
// `-` can be a unary prefix op, a binary infix op, or the first character
// of a path or -> (ex 1->1--1)
// `/` can be a path leader or an operator (ex a?a /a)
struct op_minus : seq<one<'-'>, not_at<one<'>'>>, not_at<_extend_as_path>> {};
struct op_div : seq<one<'/'>, not_at<c::path>> {};
// match a rule, making sure we are not matching it where a keyword would match.
// using minus like this is a lot faster than flipping the order and using seq.
template<typename... Rules>
struct _not_at_any_keyword : minus<
seq<Rules...>,
sor<
TAO_PEGTL_STRING("inherit"),
TAO_PEGTL_STRING("assert"),
TAO_PEGTL_STRING("else"),
TAO_PEGTL_STRING("then"),
TAO_PEGTL_STRING("with"),
TAO_PEGTL_STRING("let"),
TAO_PEGTL_STRING("rec"),
TAO_PEGTL_STRING("if"),
TAO_PEGTL_STRING("in"),
TAO_PEGTL_STRING("or")
>
> {};
// identifiers are kind of horrid:
//
// - uri_scheme_first ⊂ id_first
// - uri_scheme_first ⊂ uri_scheme_rest ⊂ path
// - id_first ⊂ id_rest { ' } ⊂ path
// - id_first ∩ (path uri_scheme_first) = { _ }
// - uri_sep ∉ { id_first, id_rest, uri_scheme_first, uri_scheme_rest, path }
// - path_sep ∉ { id_first, id_rest, uri_scheme_first, uri_scheme_rest }
//
// and we want, without reading the input more than once, a string that
// matches (id_first id_rest*) and is not followed by any number of
// characters such that the extended string matches path or uri rules.
//
// since the first character must be either _ or a uri scheme character
// we can ignore path-like bits at the beginning. uri_sep cannot appear anywhere
// in an identifier, so it's only needed in lookahead checks at the uri-like
// prefix. likewise path_sep cannot appear anywhere in the idenfier, so it's
// only needed in lookahead checks in the path-like prefix.
//
// in total that gives us a decomposition of
//
// (uri-scheme-like? (?! continues-as-uri) | _)
// (path-segment-like? (?! continues-as-path))
// id_rest*
struct identifier : _not_at_any_keyword<
// we don't use (at<id_rest>, ...) matches here because identifiers are
// a really hot path and rewinding as needed by at<> isn't entirely free.
sor<
seq<
c::uri_scheme_first,
star<ranges<'a', 'z', 'A', 'Z', '0', '9', '-'>>,
not_at<_extend_as_uri>
>,
one<'_'>
>,
star<sor<ranges<'a', 'z', 'A', 'Z', '0', '9'>, one<'_', '-'>>>,
not_at<_extend_as_path>,
star<c::id_rest>
> {};
// floats may extend ints, thus these rules are very similar.
struct integer : seq<
sor<
seq<range<'1', '9'>, star<digit>, not_at<one<'.'>>>,
seq<one<'0'>, not_at<one<'.'>, digit>, star<digit>>
>,
not_at<_extend_as_path>
> {};
struct floating : seq<
sor<
seq<range<'1', '9'>, star<digit>, one<'.'>, star<digit>>,
seq<opt<one<'0'>>, one<'.'>, plus<digit>>
>,
opt<one<'E', 'e'>, opt<one<'+', '-'>>, plus<digit>>,
not_at<_extend_as_path>
> {};
struct uri : seq<
c::uri_scheme_first,
star<c::uri_scheme_rest>,
c::uri_sep,
plus<c::uri_rest>
> {};
struct sep : sor<
plus<one<' ', '\t', '\r', '\n'>>,
seq<one<'#'>, star<not_one<'\r', '\n'>>>,
seq<string<'/', '*'>, until<string<'*', '/'>>>
> {};
}
using seps = star<t::sep>;
// marker for semantic rules. not handling one of these in an action that cares about
// semantics is probably an error.
struct semantic {};
struct expr;
struct _string {
template<typename... Inner>
struct literal : semantic, seq<Inner...> {};
struct cr_lf : semantic, seq<one<'\r'>, opt<one<'\n'>>> {};
struct interpolation : semantic, seq<
p::string<'$', '{'>, seps,
must<expr>, seps,
must<one<'}'>>
> {};
struct escape : semantic, must<any> {};
};
struct string : _string, seq<
one<'"'>,
star<
sor<
_string::literal<plus<not_one<'$', '"', '\\', '\r'>>>,
_string::cr_lf,
_string::interpolation,
_string::literal<one<'$'>, opt<one<'$'>>>,
seq<one<'\\'>, _string::escape>
>
>,
must<one<'"'>>
> {};
struct _ind_string {
template<bool Indented, typename... Inner>
struct literal : semantic, seq<Inner...> {};
struct interpolation : semantic, seq<
p::string<'$', '{'>, seps,
must<expr>, seps,
must<one<'}'>>
> {};
struct escape : semantic, must<any> {};
};
struct ind_string : _ind_string, seq<
TAO_PEGTL_STRING("''"),
opt<star<one<' '>>, one<'\n'>>,
star<
sor<
_ind_string::literal<
true,
plus<
sor<
not_one<'$', '\''>,
seq<one<'$'>, not_one<'{', '\''>>,
seq<one<'\''>, not_one<'\'', '$'>>
>
>
>,
_ind_string::interpolation,
_ind_string::literal<false, one<'$'>>,
_ind_string::literal<false, one<'\''>, not_at<one<'\''>>>,
seq<one<'\''>, _ind_string::literal<false, p::string<'\'', '\''>>>,
seq<
p::string<'\'', '\''>,
sor<
_ind_string::literal<false, one<'$'>>,
seq<one<'\\'>, _ind_string::escape>
>
>
>
>,
must<TAO_PEGTL_STRING("''")>
> {};
struct _path {
// legacy lexer rules. extra l_ to avoid reserved c++ identifiers.
struct _l_PATH : seq<star<c::path>, plus<c::path_sep, plus<c::path>>, opt<c::path_sep>> {};
struct _l_PATH_SEG : seq<star<c::path>, c::path_sep> {};
struct _l_HPATH : seq<one<'~'>, plus<c::path_sep, plus<c::path>>, opt<c::path_sep>> {};
struct _l_HPATH_START : TAO_PEGTL_STRING("~/") {};
struct _path_str : sor<_l_PATH, _l_PATH_SEG, plus<c::path>> {};
// modern rules
template<typename... Inner>
struct literal : semantic, seq<Inner...> {};
struct interpolation : semantic, seq<
p::string<'$', '{'>, seps,
must<expr>, seps,
must<one<'}'>>
> {};
struct anchor : semantic, sor<
_l_PATH,
seq<_l_PATH_SEG, at<TAO_PEGTL_STRING("${")>>
> {};
struct home_anchor : semantic, sor<
_l_HPATH,
seq<_l_HPATH_START, at<TAO_PEGTL_STRING("${")>>
> {};
struct searched_path : semantic, list<plus<c::path>, c::path_sep> {};
struct forbid_prefix_triple_slash : sor<not_at<c::path_sep>, failure> {};
struct forbid_prefix_double_slash_no_interp : sor<
not_at<c::path_sep, star<c::path>, not_at<TAO_PEGTL_STRING("${")>>,
failure
> {};
// legacy parser rules
struct _str_rest : seq<
must<forbid_prefix_double_slash_no_interp>,
opt<literal<_path_str>>,
must<forbid_prefix_triple_slash>,
star<
sor<
literal<_path_str>,
interpolation
>
>
> {};
};
struct path : _path, sor<
seq<
sor<_path::anchor, _path::home_anchor>,
_path::_str_rest
>,
seq<one<'<'>, _path::searched_path, one<'>'>>
> {};
struct _formal {
struct name : semantic, t::identifier {};
struct default_value : semantic, must<expr> {};
};
struct formal : semantic, _formal, seq<
_formal::name,
opt<seps, one<'?'>, seps, _formal::default_value>
> {};
struct _formals {
struct ellipsis : semantic, p::ellipsis {};
};
struct formals : semantic, _formals, seq<
one<'{'>, seps,
// formals and attrsets share a two-token head sequence ('{' <id>).
// this rule unrolls the formals list a bit to provide better error messages than
// "expected '='" at the first ',' if formals are incorrect.
sor<
one<'}'>,
seq<_formals::ellipsis, seps, must<one<'}'>>>,
seq<
formal, seps,
if_then_else<
at<one<','>>,
seq<
star<one<','>, seps, formal, seps>,
opt<one<','>, seps, opt<_formals::ellipsis, seps>>,
must<one<'}'>>
>,
one<'}'>
>
>
>
> {};
struct _attr {
struct simple : semantic, sor<t::identifier, t::kw_or> {};
struct string : semantic, seq<grammar::string> {};
struct expr : semantic, seq<
TAO_PEGTL_STRING("${"), seps,
must<grammar::expr>, seps,
must<one<'}'>>
> {};
};
struct attr : _attr, sor<
_attr::simple,
_attr::string,
_attr::expr
> {};
struct attrpath : list<attr, one<'.'>, t::sep> {};
struct _inherit {
struct from : semantic, must<expr> {};
struct attrs : list<attr, seps> {};
};
struct inherit : _inherit, seq<
t::kw_inherit, seps,
opt<one<'('>, seps, _inherit::from, seps, must<one<')'>>, seps>,
opt<_inherit::attrs, seps>,
must<one<';'>>
> {};
struct _binding {
struct path : semantic, attrpath {};
struct equal : one<'='> {};
struct value : semantic, must<expr> {};
};
struct binding : _binding, seq<
_binding::path, seps,
must<_binding::equal>, seps,
_binding::value, seps,
must<one<';'>>
> {};
struct bindings : opt<list<sor<inherit, binding>, seps>> {};
struct op {
enum class kind {
// NOTE non-associativity is *NOT* handled in the grammar structure.
// handling it in the grammar itself instead of in semantic actions
// slows down the parser significantly and makes the rules *much*
// harder to read. maybe this will be different at some point when
// ! does not sit between two binary precedence levels.
nonAssoc,
leftAssoc,
rightAssoc,
unary,
};
template<typename Rule, unsigned Precedence, kind Kind = kind::leftAssoc>
struct _op : Rule {
static constexpr unsigned precedence = Precedence;
static constexpr op::kind kind = Kind;
};
struct unary_minus : _op<t::op_minus, 3, kind::unary> {};
// treating this like a unary postfix operator is sketchy, but that's
// the most reasonable way to implement the operator precedence set forth
// by the language way back. it'd be much better if `.` and `?` had the same
// precedence, but alas.
struct has_attr : _op<seq<one<'?'>, seps, must<attrpath>>, 4> {};
struct concat : _op<TAO_PEGTL_STRING("++"), 5, kind::rightAssoc> {};
struct mul : _op<one<'*'>, 6> {};
struct div : _op<t::op_div, 6> {};
struct plus : _op<one<'+'>, 7> {};
struct minus : _op<t::op_minus, 7> {};
struct not_ : _op<one<'!'>, 8, kind::unary> {};
struct update : _op<TAO_PEGTL_STRING("//"), 9, kind::rightAssoc> {};
struct less_eq : _op<TAO_PEGTL_STRING("<="), 10, kind::nonAssoc> {};
struct greater_eq : _op<TAO_PEGTL_STRING(">="), 10, kind::nonAssoc> {};
struct less : _op<one<'<'>, 10, kind::nonAssoc> {};
struct greater : _op<one<'>'>, 10, kind::nonAssoc> {};
struct equals : _op<TAO_PEGTL_STRING("=="), 11, kind::nonAssoc> {};
struct not_equals : _op<TAO_PEGTL_STRING("!="), 11, kind::nonAssoc> {};
struct and_ : _op<TAO_PEGTL_STRING("&&"), 12> {};
struct or_ : _op<TAO_PEGTL_STRING("||"), 13> {};
struct implies : _op<TAO_PEGTL_STRING("->"), 14, kind::rightAssoc> {};
};
struct _expr {
template<template<typename...> class OpenMod = seq, typename... Init>
struct _attrset : seq<
Init...,
OpenMod<one<'{'>>, seps,
bindings, seps,
must<one<'}'>>
> {};
struct select;
struct id : semantic, t::identifier {};
struct int_ : semantic, t::integer {};
struct float_ : semantic, t::floating {};
struct string : semantic, seq<grammar::string> {};
struct ind_string : semantic, seq<grammar::ind_string> {};
struct path : semantic, seq<grammar::path> {};
struct uri : semantic, t::uri {};
struct ancient_let : semantic, _attrset<must, t::kw_let, seps> {};
struct rec_set : semantic, _attrset<must, t::kw_rec, seps> {};
struct set : semantic, _attrset<> {};
struct _list {
struct entry : semantic, seq<select> {};
};
struct list : semantic, _list, seq<
one<'['>, seps,
opt<p::list<_list::entry, seps>, seps>,
must<one<']'>>
> {};
struct _simple : sor<
id,
int_,
float_,
string,
ind_string,
path,
uri,
seq<one<'('>, seps, must<expr>, seps, must<one<')'>>>,
ancient_let,
rec_set,
set,
list
> {};
struct _select {
struct head : _simple {};
struct attr : semantic, seq<attrpath> {};
struct attr_or : semantic, must<select> {};
struct as_app_or : semantic, t::kw_or {};
};
struct _app {
struct first_arg : semantic, seq<select> {};
struct another_arg : semantic, seq<select> {};
// can be used to stash a position of the application head node
struct select_or_fn : seq<select> {};
};
struct select : _select, seq<
_select::head, seps,
opt<
sor<
seq<
one<'.'>, seps, _select::attr,
opt<seps, t::kw_or, seps, _select::attr_or>
>,
_select::as_app_or
>
>
> {};
struct app : _app, seq<
_app::select_or_fn,
opt<seps, _app::first_arg, star<seps, _app::another_arg>>
> {};
template<typename Op>
struct operator_ : semantic, Op {};
struct unary : seq<
star<sor<operator_<op::not_>, operator_<op::unary_minus>>, seps>,
app
> {};
struct _binary_operator : sor<
operator_<op::implies>,
operator_<op::update>,
operator_<op::concat>,
operator_<op::plus>,
operator_<op::minus>,
operator_<op::mul>,
operator_<op::div>,
operator_<op::less_eq>,
operator_<op::greater_eq>,
operator_<op::less>,
operator_<op::greater>,
operator_<op::equals>,
operator_<op::not_equals>,
operator_<op::or_>,
operator_<op::and_>
> {};
struct _binop : seq<
unary,
star<
seps,
sor<
seq<_binary_operator, seps, must<unary>>,
operator_<op::has_attr>
>
>
> {};
struct _lambda {
struct arg : semantic, t::identifier {};
};
struct lambda : semantic, _lambda, sor<
seq<
_lambda::arg, seps,
sor<
seq<one<':'>, seps, must<expr>>,
seq<one<'@'>, seps, must<formals, seps, one<':'>, seps, expr>>
>
>,
seq<
formals, seps,
sor<
seq<one<':'>, seps, must<expr>>,
seq<one<'@'>, seps, must<_lambda::arg, seps, one<':'>, seps, expr>>
>
>
> {};
struct assert_ : semantic, seq<
t::kw_assert, seps,
must<expr>, seps,
must<one<';'>>, seps,
must<expr>
> {};
struct with : semantic, seq<
t::kw_with, seps,
must<expr>, seps,
must<one<';'>>, seps,
must<expr>
> {};
struct let : seq<
t::kw_let, seps,
not_at<one<'{'>>, // exclude ancient_let so we can must<kw_in>
bindings, seps,
must<t::kw_in>, seps,
must<expr>
> {};
struct if_ : semantic, seq<
t::kw_if, seps,
must<expr>, seps,
must<t::kw_then>, seps,
must<expr>, seps,
must<t::kw_else>, seps,
must<expr>
> {};
};
struct expr : semantic, _expr, sor<
_expr::lambda,
_expr::assert_,
_expr::with,
_expr::let,
_expr::if_,
_expr::_binop
> {};
// legacy support: \0 terminates input if passed from flex to bison as a token
struct eof : sor<p::eof, one<0>> {};
struct root : must<seps, expr, seps, eof> {};
template<typename Rule>
struct nothing : p::nothing<Rule> {
static_assert(!std::is_base_of_v<semantic, Rule>);
};
template<typename Self, typename OpCtx, typename AttrPathT, typename ExprT>
struct operator_semantics {
struct has_attr : grammar::op::has_attr {
AttrPathT path;
};
struct OpEntry {
OpCtx ctx;
uint8_t prec;
grammar::op::kind assoc;
std::variant<
grammar::op::not_,
grammar::op::unary_minus,
grammar::op::implies,
grammar::op::or_,
grammar::op::and_,
grammar::op::equals,
grammar::op::not_equals,
grammar::op::less_eq,
grammar::op::greater_eq,
grammar::op::update,
grammar::op::concat,
grammar::op::less,
grammar::op::greater,
grammar::op::plus,
grammar::op::minus,
grammar::op::mul,
grammar::op::div,
has_attr
> op;
};
// statistics here are taken from nixpkgs commit de502c4d0ba96261e5de803e4d1d1925afd3e22f.
// over 99.9% of contexts in nixpkgs need at most 4 slots, ~85% need only 1
boost::container::small_vector<ExprT, 4> exprs;
// over 99.9% of contexts in nixpkgs need at most 2 slots, ~85% need only 1
boost::container::small_vector<OpEntry, 2> ops;
// derived class is expected to define members:
//
// ExprT applyOp(OpCtx & pos, auto & op, auto &... args);
// [[noreturn]] static void badOperator(OpCtx & pos, auto &... args);
void reduce(uint8_t toPrecedence, auto &... args) {
while (!ops.empty()) {
auto & [ctx, precedence, kind, op] = ops.back();
// NOTE this relies on associativity not being mixed within a precedence level.
if ((precedence > toPrecedence)
|| (kind != grammar::op::kind::leftAssoc && precedence == toPrecedence))
break;
std::visit([&, ctx=std::move(ctx)] (auto & op) {
exprs.push_back(static_cast<Self &>(*this).applyOp(ctx, op, args...));
}, op);
ops.pop_back();
}
}
ExprT popExpr()
{
auto r = std::move(exprs.back());
exprs.pop_back();
return r;
}
void pushOp(OpCtx ctx, auto o, auto &... args)
{
if (o.kind != grammar::op::kind::unary)
reduce(o.precedence, args...);
if (!ops.empty() && o.kind == grammar::op::kind::nonAssoc) {
auto & [_pos, _prec, _kind, _o] = ops.back();
if (_kind == o.kind && _prec == o.precedence)
Self::badOperator(ctx, args...);
}
ops.emplace_back(ctx, o.precedence, o.kind, std::move(o));
}
ExprT finish(auto &... args)
{
reduce(255, args...);
return popExpr();
}
};
}

View file

@ -0,0 +1,866 @@
#include "attr-set.hh"
#include "error.hh"
#include "eval-settings.hh"
#include "eval.hh"
#include "finally.hh"
#include "nixexpr.hh"
#include "symbol-table.hh"
#include "users.hh"
#include "change_head.hh"
#include "grammar.hh"
#include "state.hh"
#include <charconv>
#include <clocale>
#include <memory>
// flip this define when doing parser development to enable some g checks.
#if 0
#include <tao/pegtl/contrib/analyze.hpp>
#define ANALYZE_GRAMMAR \
([] { \
const std::size_t issues = tao::pegtl::analyze<grammar::root>(); \
assert(issues == 0); \
})()
#else
#define ANALYZE_GRAMMAR ((void) 0)
#endif
namespace p = tao::pegtl;
namespace nix::parser {
namespace {
template<typename>
inline constexpr const char * error_message = nullptr;
#define error_message_for(...) \
template<> inline constexpr auto error_message<__VA_ARGS__>
error_message_for(p::one<'{'>) = "expecting '{'";
error_message_for(p::one<'}'>) = "expecting '}'";
error_message_for(p::one<'"'>) = "expecting '\"'";
error_message_for(p::one<';'>) = "expecting ';'";
error_message_for(p::one<')'>) = "expecting ')'";
error_message_for(p::one<']'>) = "expecting ']'";
error_message_for(p::one<':'>) = "expecting ':'";
error_message_for(p::string<'\'', '\''>) = "expecting \"''\"";
error_message_for(p::any) = "expecting any character";
error_message_for(grammar::eof) = "expecting end of file";
error_message_for(grammar::seps) = "expecting separators";
error_message_for(grammar::path::forbid_prefix_triple_slash) = "too many slashes in path";
error_message_for(grammar::path::forbid_prefix_double_slash_no_interp) = "path has a trailing slash";
error_message_for(grammar::expr) = "expecting expression";
error_message_for(grammar::expr::unary) = "expecting expression";
error_message_for(grammar::binding::equal) = "expecting '='";
error_message_for(grammar::expr::lambda::arg) = "expecting identifier";
error_message_for(grammar::formals) = "expecting formals";
error_message_for(grammar::attrpath) = "expecting attribute path";
error_message_for(grammar::expr::select) = "expecting selection expression";
error_message_for(grammar::t::kw_then) = "expecting 'then'";
error_message_for(grammar::t::kw_else) = "expecting 'else'";
error_message_for(grammar::t::kw_in) = "expecting 'in'";
struct SyntaxErrors
{
template<typename Rule>
static constexpr auto message = error_message<Rule>;
template<typename Rule>
static constexpr bool raise_on_failure = false;
};
template<typename Rule>
struct Control : p::must_if<SyntaxErrors>::control<Rule>
{
template<typename ParseInput, typename... States>
[[noreturn]] static void raise(const ParseInput & in, States &&... st)
{
if (in.empty()) {
std::string expected;
if constexpr (constexpr auto msg = error_message<Rule>)
expected = fmt(", %s", msg);
throw p::parse_error("unexpected end of file" + expected, in);
}
p::must_if<SyntaxErrors>::control<Rule>::raise(in, st...);
}
};
struct ExprState
: grammar::
operator_semantics<ExprState, PosIdx, AttrPath, std::pair<PosIdx, std::unique_ptr<Expr>>>
{
std::unique_ptr<Expr> popExprOnly() {
return std::move(popExpr().second);
}
template<typename Op, typename... Args>
std::unique_ptr<Expr> applyUnary(Args &&... args) {
return std::make_unique<Op>(popExprOnly(), std::forward<Args>(args)...);
}
template<typename Op>
std::unique_ptr<Expr> applyBinary(PosIdx pos) {
auto right = popExprOnly(), left = popExprOnly();
return std::make_unique<Op>(pos, std::move(left), std::move(right));
}
std::unique_ptr<Expr> call(PosIdx pos, Symbol fn, bool flip = false)
{
std::vector<std::unique_ptr<Expr>> args(2);
args[flip ? 0 : 1] = popExprOnly();
args[flip ? 1 : 0] = popExprOnly();
return std::make_unique<ExprCall>(pos, std::make_unique<ExprVar>(fn), std::move(args));
}
std::unique_ptr<Expr> order(PosIdx pos, bool less, State & state)
{
return call(pos, state.s.lessThan, !less);
}
std::unique_ptr<Expr> concatStrings(PosIdx pos)
{
std::vector<std::pair<PosIdx, std::unique_ptr<Expr>>> args(2);
args[1] = popExpr();
args[0] = popExpr();
return std::make_unique<ExprConcatStrings>(pos, false, std::move(args));
}
std::unique_ptr<Expr> negate(PosIdx pos, State & state)
{
std::vector<std::unique_ptr<Expr>> args(2);
args[0] = std::make_unique<ExprInt>(0);
args[1] = popExprOnly();
return std::make_unique<ExprCall>(pos, std::make_unique<ExprVar>(state.s.sub), std::move(args));
}
std::pair<PosIdx, std::unique_ptr<Expr>> applyOp(PosIdx pos, auto & op, State & state) {
using Op = grammar::op;
auto not_ = [] (auto e) {
return std::make_unique<ExprOpNot>(std::move(e));
};
return {
pos,
(overloaded {
[&] (Op::implies) { return applyBinary<ExprOpImpl>(pos); },
[&] (Op::or_) { return applyBinary<ExprOpOr>(pos); },
[&] (Op::and_) { return applyBinary<ExprOpAnd>(pos); },
[&] (Op::equals) { return applyBinary<ExprOpEq>(pos); },
[&] (Op::not_equals) { return applyBinary<ExprOpNEq>(pos); },
[&] (Op::less) { return order(pos, true, state); },
[&] (Op::greater_eq) { return not_(order(pos, true, state)); },
[&] (Op::greater) { return order(pos, false, state); },
[&] (Op::less_eq) { return not_(order(pos, false, state)); },
[&] (Op::update) { return applyBinary<ExprOpUpdate>(pos); },
[&] (Op::not_) { return applyUnary<ExprOpNot>(); },
[&] (Op::plus) { return concatStrings(pos); },
[&] (Op::minus) { return call(pos, state.s.sub); },
[&] (Op::mul) { return call(pos, state.s.mul); },
[&] (Op::div) { return call(pos, state.s.div); },
[&] (Op::concat) { return applyBinary<ExprOpConcatLists>(pos); },
[&] (has_attr & a) { return applyUnary<ExprOpHasAttr>(std::move(a.path)); },
[&] (Op::unary_minus) { return negate(pos, state); },
})(op)
};
}
// always_inline is needed, otherwise pushOp slows down considerably
[[noreturn, gnu::always_inline]]
static void badOperator(PosIdx pos, State & state)
{
throw ParseError({
.msg = HintFmt("syntax error, unexpected operator"),
.pos = state.positions[pos]
});
}
template<typename Expr, typename... Args>
Expr & pushExpr(PosIdx pos, Args && ... args)
{
auto p = std::make_unique<Expr>(std::forward<Args>(args)...);
auto & result = *p;
exprs.emplace_back(pos, std::move(p));
return result;
}
};
struct SubexprState {
private:
ExprState * up;
public:
explicit SubexprState(ExprState & up, auto &...) : up(&up) {}
operator ExprState &() { return *up; }
ExprState * operator->() { return up; }
};
template<typename Rule>
struct BuildAST : grammar::nothing<Rule> {};
struct LambdaState : SubexprState {
using SubexprState::SubexprState;
Symbol arg;
std::unique_ptr<Formals> formals;
};
struct FormalsState : SubexprState {
using SubexprState::SubexprState;
Formals formals{};
Formal formal{};
};
template<> struct BuildAST<grammar::formal::name> {
static void apply(const auto & in, FormalsState & s, State & ps) {
s.formal = {
.pos = ps.at(in),
.name = ps.symbols.create(in.string_view()),
};
}
};
template<> struct BuildAST<grammar::formal> {
static void apply0(FormalsState & s, State &) {
s.formals.formals.emplace_back(std::move(s.formal));
}
};
template<> struct BuildAST<grammar::formal::default_value> {
static void apply0(FormalsState & s, State & ps) {
s.formal.def = s->popExprOnly();
}
};
template<> struct BuildAST<grammar::formals::ellipsis> {
static void apply0(FormalsState & s, State &) {
s.formals.ellipsis = true;
}
};
template<> struct BuildAST<grammar::formals> : change_head<FormalsState> {
static void success0(FormalsState & f, LambdaState & s, State &) {
s.formals = std::make_unique<Formals>(std::move(f.formals));
}
};
struct AttrState : SubexprState {
using SubexprState::SubexprState;
std::vector<AttrName> attrs;
void pushAttr(auto && attr, PosIdx) { attrs.emplace_back(std::move(attr)); }
};
template<> struct BuildAST<grammar::attr::simple> {
static void apply(const auto & in, auto & s, State & ps) {
s.pushAttr(ps.symbols.create(in.string_view()), ps.at(in));
}
};
template<> struct BuildAST<grammar::attr::string> {
static void apply(const auto & in, auto & s, State & ps) {
auto e = s->popExprOnly();
if (auto str = dynamic_cast<ExprString *>(e.get()))
s.pushAttr(ps.symbols.create(str->s), ps.at(in));
else
s.pushAttr(std::move(e), ps.at(in));
}
};
template<> struct BuildAST<grammar::attr::expr> : BuildAST<grammar::attr::string> {};
struct BindingsState : SubexprState {
using SubexprState::SubexprState;
ExprAttrs attrs;
AttrPath path;
std::unique_ptr<Expr> value;
};
struct InheritState : SubexprState {
using SubexprState::SubexprState;
std::vector<std::pair<AttrName, PosIdx>> attrs;
std::unique_ptr<Expr> from;
PosIdx fromPos;
void pushAttr(auto && attr, PosIdx pos) { attrs.emplace_back(std::move(attr), pos); }
};
template<> struct BuildAST<grammar::inherit::from> {
static void apply(const auto & in, InheritState & s, State & ps) {
s.from = s->popExprOnly();
s.fromPos = ps.at(in);
}
};
template<> struct BuildAST<grammar::inherit> : change_head<InheritState> {
static void success0(InheritState & s, BindingsState & b, State & ps) {
auto & attrs = b.attrs.attrs;
// TODO this should not reuse generic attrpath rules.
for (auto & [i, iPos] : s.attrs) {
if (i.symbol)
continue;
if (auto str = dynamic_cast<ExprString *>(i.expr.get()))
i = AttrName(ps.symbols.create(str->s));
else {
throw ParseError({
.msg = HintFmt("dynamic attributes not allowed in inherit"),
.pos = ps.positions[iPos]
});
}
}
if (s.from != nullptr) {
if (!b.attrs.inheritFromExprs)
b.attrs.inheritFromExprs = std::make_unique<std::vector<ref<Expr>>>();
auto fromExpr = ref<Expr>(std::move(s.from));
b.attrs.inheritFromExprs->push_back(fromExpr);
for (auto & [i, iPos] : s.attrs) {
if (attrs.find(i.symbol) != attrs.end())
ps.dupAttr(i.symbol, iPos, attrs[i.symbol].pos);
auto inheritFrom = std::make_unique<ExprInheritFrom>(
s.fromPos,
b.attrs.inheritFromExprs->size() - 1,
fromExpr
);
attrs.emplace(
i.symbol,
ExprAttrs::AttrDef(
std::make_unique<ExprSelect>(iPos, std::move(inheritFrom), i.symbol),
iPos,
ExprAttrs::AttrDef::Kind::InheritedFrom));
}
} else {
for (auto & [i, iPos] : s.attrs) {
if (attrs.find(i.symbol) != attrs.end())
ps.dupAttr(i.symbol, iPos, attrs[i.symbol].pos);
attrs.emplace(
i.symbol,
ExprAttrs::AttrDef(
std::make_unique<ExprVar>(iPos, i.symbol),
iPos,
ExprAttrs::AttrDef::Kind::Inherited));
}
}
}
};
template<> struct BuildAST<grammar::binding::path> : change_head<AttrState> {
static void success0(AttrState & a, BindingsState & s, State & ps) {
s.path = std::move(a.attrs);
}
};
template<> struct BuildAST<grammar::binding::value> {
static void apply0(BindingsState & s, State & ps) {
s.value = s->popExprOnly();
}
};
template<> struct BuildAST<grammar::binding> {
static void apply(const auto & in, BindingsState & s, State & ps) {
ps.addAttr(&s.attrs, std::move(s.path), std::move(s.value), ps.at(in));
}
};
template<> struct BuildAST<grammar::expr::id> {
static void apply(const auto & in, ExprState & s, State & ps) {
if (in.string_view() == "__curPos")
s.pushExpr<ExprPos>(ps.at(in), ps.at(in));
else
s.pushExpr<ExprVar>(ps.at(in), ps.at(in), ps.symbols.create(in.string_view()));
}
};
template<> struct BuildAST<grammar::expr::int_> {
static void apply(const auto & in, ExprState & s, State & ps) {
int64_t v;
if (std::from_chars(in.begin(), in.end(), v).ec != std::errc{}) {
throw ParseError({
.msg = HintFmt("invalid integer '%1%'", in.string_view()),
.pos = ps.positions[ps.at(in)],
});
}
s.pushExpr<ExprInt>(noPos, v);
}
};
template<> struct BuildAST<grammar::expr::float_> {
static void apply(const auto & in, ExprState & s, State & ps) {
// copy the input into a temporary string so we can call stod.
// can't use from_chars because libc++ (thus darwin) does not have it,
// and floats are not performance-sensitive anyway. if they were you'd
// be in much bigger trouble than this.
//
// we also get to do a locale-save dance because stod is locale-aware and
// something (a plugin?) may have called setlocale or uselocale.
static struct locale_hack {
locale_t posix;
locale_hack(): posix(newlocale(LC_ALL_MASK, "POSIX", 0))
{
if (posix == 0)
throw SysError("could not get POSIX locale");
}
} locale;
auto tmp = in.string();
double v = [&] {
auto oldLocale = uselocale(locale.posix);
Finally resetLocale([=] { uselocale(oldLocale); });
try {
return std::stod(tmp);
} catch (...) {
throw ParseError({
.msg = HintFmt("invalid float '%1%'", in.string_view()),
.pos = ps.positions[ps.at(in)],
});
}
}();
s.pushExpr<ExprFloat>(noPos, v);
}
};
struct StringState : SubexprState {
using SubexprState::SubexprState;
std::string currentLiteral;
PosIdx currentPos;
std::vector<std::pair<nix::PosIdx, std::unique_ptr<Expr>>> parts;
void append(PosIdx pos, std::string_view s)
{
if (currentLiteral.empty())
currentPos = pos;
currentLiteral += s;
}
// FIXME this truncates strings on NUL for compat with the old parser. ideally
// we should use the decomposition the g gives us instead of iterating over
// the entire string again.
static void unescapeStr(std::string & str)
{
char * s = str.data();
char * t = s;
char c;
while ((c = *s++)) {
if (c == '\\') {
c = *s++;
if (c == 'n') *t = '\n';
else if (c == 'r') *t = '\r';
else if (c == 't') *t = '\t';
else *t = c;
}
else if (c == '\r') {
/* Normalise CR and CR/LF into LF. */
*t = '\n';
if (*s == '\n') s++; /* cr/lf */
}
else *t = c;
t++;
}
str.resize(t - str.data());
}
void endLiteral()
{
if (!currentLiteral.empty()) {
unescapeStr(currentLiteral);
parts.emplace_back(currentPos, std::make_unique<ExprString>(std::move(currentLiteral)));
}
}
std::unique_ptr<Expr> finish()
{
if (parts.empty()) {
unescapeStr(currentLiteral);
return std::make_unique<ExprString>(std::move(currentLiteral));
} else {
endLiteral();
auto pos = parts[0].first;
return std::make_unique<ExprConcatStrings>(pos, true, std::move(parts));
}
}
};
template<typename... Content> struct BuildAST<grammar::string::literal<Content...>> {
static void apply(const auto & in, StringState & s, State & ps) {
s.append(ps.at(in), in.string_view());
}
};
template<> struct BuildAST<grammar::string::cr_lf> {
static void apply(const auto & in, StringState & s, State & ps) {
s.append(ps.at(in), in.string_view()); // FIXME compat with old parser
}
};
template<> struct BuildAST<grammar::string::interpolation> {
static void apply(const auto & in, StringState & s, State & ps) {
s.endLiteral();
s.parts.emplace_back(ps.at(in), s->popExprOnly());
}
};
template<> struct BuildAST<grammar::string::escape> {
static void apply(const auto & in, StringState & s, State & ps) {
s.append(ps.at(in), "\\"); // FIXME compat with old parser
s.append(ps.at(in), in.string_view());
}
};
template<> struct BuildAST<grammar::string> : change_head<StringState> {
static void success0(StringState & s, ExprState & e, State &) {
e.exprs.emplace_back(noPos, s.finish());
}
};
struct IndStringState : SubexprState {
using SubexprState::SubexprState;
std::vector<std::pair<PosIdx, std::variant<std::unique_ptr<Expr>, StringToken>>> parts;
};
template<bool Indented, typename... Content>
struct BuildAST<grammar::ind_string::literal<Indented, Content...>> {
static void apply(const auto & in, IndStringState & s, State & ps) {
s.parts.emplace_back(ps.at(in), StringToken{in.string_view(), Indented});
}
};
template<> struct BuildAST<grammar::ind_string::interpolation> {
static void apply(const auto & in, IndStringState & s, State & ps) {
s.parts.emplace_back(ps.at(in), s->popExprOnly());
}
};
template<> struct BuildAST<grammar::ind_string::escape> {
static void apply(const auto & in, IndStringState & s, State & ps) {
switch (*in.begin()) {
case 'n': s.parts.emplace_back(ps.at(in), StringToken{"\n"}); break;
case 'r': s.parts.emplace_back(ps.at(in), StringToken{"\r"}); break;
case 't': s.parts.emplace_back(ps.at(in), StringToken{"\t"}); break;
default: s.parts.emplace_back(ps.at(in), StringToken{in.string_view()}); break;
}
}
};
template<> struct BuildAST<grammar::ind_string> : change_head<IndStringState> {
static void success(const auto & in, IndStringState & s, ExprState & e, State & ps) {
e.exprs.emplace_back(noPos, ps.stripIndentation(ps.at(in), std::move(s.parts)));
}
};
template<typename... Content> struct BuildAST<grammar::path::literal<Content...>> {
static void apply(const auto & in, StringState & s, State & ps) {
s.append(ps.at(in), in.string_view());
s.endLiteral();
}
};
template<> struct BuildAST<grammar::path::interpolation> : BuildAST<grammar::string::interpolation> {};
template<> struct BuildAST<grammar::path::anchor> {
static void apply(const auto & in, StringState & s, State & ps) {
Path path(absPath(in.string(), ps.basePath.path.abs()));
/* add back in the trailing '/' to the first segment */
if (in.string_view().ends_with('/') && in.size() > 1)
path += "/";
s.parts.emplace_back(ps.at(in), new ExprPath(std::move(path)));
}
};
template<> struct BuildAST<grammar::path::home_anchor> {
static void apply(const auto & in, StringState & s, State & ps) {
if (evalSettings.pureEval)
throw Error("the path '%s' can not be resolved in pure mode", in.string_view());
Path path(getHome() + in.string_view().substr(1));
s.parts.emplace_back(ps.at(in), new ExprPath(std::move(path)));
}
};
template<> struct BuildAST<grammar::path::searched_path> {
static void apply(const auto & in, StringState & s, State & ps) {
std::vector<std::unique_ptr<Expr>> args{2};
args[0] = std::make_unique<ExprVar>(ps.s.nixPath);
args[1] = std::make_unique<ExprString>(in.string());
s.parts.emplace_back(
ps.at(in),
std::make_unique<ExprCall>(
ps.at(in),
std::make_unique<ExprVar>(ps.s.findFile),
std::move(args)));
}
};
template<> struct BuildAST<grammar::path> : change_head<StringState> {
template<typename E>
static void check_slash(PosIdx end, StringState & s, State & ps) {
auto e = dynamic_cast<E *>(s.parts.back().second.get());
if (!e || !e->s.ends_with('/'))
return;
if (s.parts.size() > 1 || e->s != "/")
throw ParseError({
.msg = HintFmt("path has a trailing slash"),
.pos = ps.positions[end],
});
}
static void success(const auto & in, StringState & s, ExprState & e, State & ps) {
s.endLiteral();
check_slash<ExprPath>(ps.atEnd(in), s, ps);
check_slash<ExprString>(ps.atEnd(in), s, ps);
if (s.parts.size() == 1) {
e.exprs.emplace_back(noPos, std::move(s.parts.back().second));
} else {
e.pushExpr<ExprConcatStrings>(ps.at(in), ps.at(in), false, std::move(s.parts));
}
}
};
// strings and paths sare handled fully by the grammar-level rule for now
template<> struct BuildAST<grammar::expr::string> : p::maybe_nothing {};
template<> struct BuildAST<grammar::expr::ind_string> : p::maybe_nothing {};
template<> struct BuildAST<grammar::expr::path> : p::maybe_nothing {};
template<> struct BuildAST<grammar::expr::uri> {
static void apply(const auto & in, ExprState & s, State & ps) {
static bool noURLLiterals = experimentalFeatureSettings.isEnabled(Xp::NoUrlLiterals);
if (noURLLiterals)
throw ParseError({
.msg = HintFmt("URL literals are disabled"),
.pos = ps.positions[ps.at(in)]
});
s.pushExpr<ExprString>(ps.at(in), in.string());
}
};
template<> struct BuildAST<grammar::expr::ancient_let> : change_head<BindingsState> {
static void success(const auto & in, BindingsState & b, ExprState & s, State & ps) {
b.attrs.pos = ps.at(in);
b.attrs.recursive = true;
s.pushExpr<ExprSelect>(b.attrs.pos, b.attrs.pos, std::make_unique<ExprAttrs>(std::move(b.attrs)), ps.s.body);
}
};
template<> struct BuildAST<grammar::expr::rec_set> : change_head<BindingsState> {
static void success(const auto & in, BindingsState & b, ExprState & s, State & ps) {
b.attrs.pos = ps.at(in);
b.attrs.recursive = true;
s.pushExpr<ExprAttrs>(b.attrs.pos, std::move(b.attrs));
}
};
template<> struct BuildAST<grammar::expr::set> : change_head<BindingsState> {
static void success(const auto & in, BindingsState & b, ExprState & s, State & ps) {
b.attrs.pos = ps.at(in);
s.pushExpr<ExprAttrs>(b.attrs.pos, std::move(b.attrs));
}
};
using ListState = std::vector<std::unique_ptr<Expr>>;
template<> struct BuildAST<grammar::expr::list> : change_head<ListState> {
static void success(const auto & in, ListState & ls, ExprState & s, State & ps) {
auto e = std::make_unique<ExprList>();
e->elems = std::move(ls);
s.exprs.emplace_back(ps.at(in), std::move(e));
}
};
template<> struct BuildAST<grammar::expr::list::entry> : change_head<ExprState> {
static void success0(ExprState & e, ListState & s, State & ps) {
s.emplace_back(e.finish(ps).second);
}
};
struct SelectState : SubexprState {
using SubexprState::SubexprState;
PosIdx pos;
ExprSelect * e = nullptr;
};
template<> struct BuildAST<grammar::expr::select::head> {
static void apply(const auto & in, SelectState & s, State & ps) {
s.pos = ps.at(in);
}
};
template<> struct BuildAST<grammar::expr::select::attr> : change_head<AttrState> {
static void success0(AttrState & a, SelectState & s, State &) {
s.e = &s->pushExpr<ExprSelect>(s.pos, s.pos, s->popExprOnly(), std::move(a.attrs), nullptr);
}
};
template<> struct BuildAST<grammar::expr::select::attr_or> {
static void apply0(SelectState & s, State &) {
s.e->def = s->popExprOnly();
}
};
template<> struct BuildAST<grammar::expr::select::as_app_or> {
static void apply(const auto & in, SelectState & s, State & ps) {
std::vector<std::unique_ptr<Expr>> args(1);
args[0] = std::make_unique<ExprVar>(ps.at(in), ps.s.or_);
s->pushExpr<ExprCall>(s.pos, s.pos, s->popExprOnly(), std::move(args));
}
};
template<> struct BuildAST<grammar::expr::select> : change_head<SelectState> {
static void success0(const auto &...) {}
};
struct AppState : SubexprState {
using SubexprState::SubexprState;
PosIdx pos;
ExprCall * e = nullptr;
};
template<> struct BuildAST<grammar::expr::app::select_or_fn> {
static void apply(const auto & in, AppState & s, State & ps) {
s.pos = ps.at(in);
}
};
template<> struct BuildAST<grammar::expr::app::first_arg> {
static void apply(auto & in, AppState & s, State & ps) {
auto arg = s->popExprOnly(), fn = s->popExprOnly();
if ((s.e = dynamic_cast<ExprCall *>(fn.get()))) {
// TODO remove.
// AST compat with old parser, semantics are the same.
// this can happen on occasions such as `<p> <p>` or `a or b or`,
// neither of which are super worth optimizing.
s.e->args.push_back(std::move(arg));
s->exprs.emplace_back(noPos, std::move(fn));
} else {
std::vector<std::unique_ptr<Expr>> args{1};
args[0] = std::move(arg);
s.e = &s->pushExpr<ExprCall>(s.pos, s.pos, std::move(fn), std::move(args));
}
}
};
template<> struct BuildAST<grammar::expr::app::another_arg> {
static void apply0(AppState & s, State & ps) {
s.e->args.push_back(s->popExprOnly());
}
};
template<> struct BuildAST<grammar::expr::app> : change_head<AppState> {
static void success0(const auto &...) {}
};
template<typename Op> struct BuildAST<grammar::expr::operator_<Op>> {
static void apply(const auto & in, ExprState & s, State & ps) {
s.pushOp(ps.at(in), Op{}, ps);
}
};
template<> struct BuildAST<grammar::expr::operator_<grammar::op::has_attr>> : change_head<AttrState> {
static void success(const auto & in, AttrState & a, ExprState & s, State & ps) {
s.pushOp(ps.at(in), ExprState::has_attr{{}, std::move(a.attrs)}, ps);
}
};
template<> struct BuildAST<grammar::expr::lambda::arg> {
static void apply(const auto & in, LambdaState & s, State & ps) {
s.arg = ps.symbols.create(in.string_view());
}
};
template<> struct BuildAST<grammar::expr::lambda> : change_head<LambdaState> {
static void success(const auto & in, LambdaState & l, ExprState & s, State & ps) {
if (l.formals)
l.formals = ps.validateFormals(std::move(l.formals), ps.at(in), l.arg);
s.pushExpr<ExprLambda>(ps.at(in), ps.at(in), l.arg, std::move(l.formals), l->popExprOnly());
}
};
template<> struct BuildAST<grammar::expr::assert_> {
static void apply(const auto & in, ExprState & s, State & ps) {
auto body = s.popExprOnly(), cond = s.popExprOnly();
s.pushExpr<ExprAssert>(ps.at(in), ps.at(in), std::move(cond), std::move(body));
}
};
template<> struct BuildAST<grammar::expr::with> {
static void apply(const auto & in, ExprState & s, State & ps) {
auto body = s.popExprOnly(), scope = s.popExprOnly();
s.pushExpr<ExprWith>(ps.at(in), ps.at(in), std::move(scope), std::move(body));
}
};
template<> struct BuildAST<grammar::expr::let> : change_head<BindingsState> {
static void success(const auto & in, BindingsState & b, ExprState & s, State & ps) {
if (!b.attrs.dynamicAttrs.empty())
throw ParseError({
.msg = HintFmt("dynamic attributes not allowed in let"),
.pos = ps.positions[ps.at(in)]
});
s.pushExpr<ExprLet>(ps.at(in), std::make_unique<ExprAttrs>(std::move(b.attrs)), b->popExprOnly());
}
};
template<> struct BuildAST<grammar::expr::if_> {
static void apply(const auto & in, ExprState & s, State & ps) {
auto else_ = s.popExprOnly(), then = s.popExprOnly(), cond = s.popExprOnly();
s.pushExpr<ExprIf>(ps.at(in), ps.at(in), std::move(cond), std::move(then), std::move(else_));
}
};
template<> struct BuildAST<grammar::expr> : change_head<ExprState> {
static void success0(ExprState & inner, ExprState & outer, State & ps) {
outer.exprs.push_back(inner.finish(ps));
}
};
}
}
namespace nix {
Expr * EvalState::parse(
char * text,
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
std::shared_ptr<StaticEnv> & staticEnv)
{
parser::State s = {
symbols,
positions,
basePath,
positions.addOrigin(origin, length),
exprSymbols,
};
parser::ExprState x;
assert(length >= 2);
assert(text[length - 1] == 0);
assert(text[length - 2] == 0);
length -= 2;
p::string_input<p::tracking_mode::lazy> inp{std::string_view{text, length}, "input"};
try {
p::parse<parser::grammar::root, parser::BuildAST, parser::Control>(inp, x, s);
} catch (p::parse_error & e) {
auto pos = e.positions().back();
throw ParseError({
.msg = HintFmt("syntax error, %s", e.message()),
.pos = positions[s.positions.add(s.origin, pos.byte)]
});
}
auto [_pos, result] = x.finish(s);
result->bindVars(*this, staticEnv);
return result.release();
}
}

View file

@ -3,77 +3,61 @@
#include "eval.hh"
namespace nix {
namespace nix::parser {
/**
* @note Storing a C-style `char *` and `size_t` allows us to avoid
* having to define the special members that using string_view here
* would implicitly delete.
*/
struct StringToken
{
const char * p;
size_t l;
std::string_view s;
bool hasIndentation;
operator std::string_view() const { return {p, l}; }
operator std::string_view() const { return s; }
};
struct ParserLocation
{
int first_line, first_column;
int last_line, last_column;
// backup to recover from yyless(0)
int stashed_first_column, stashed_last_column;
void stash() {
stashed_first_column = first_column;
stashed_last_column = last_column;
}
void unstash() {
first_column = stashed_first_column;
last_column = stashed_last_column;
}
};
struct ParserState
struct State
{
SymbolTable & symbols;
PosTable & positions;
Expr * result;
SourcePath basePath;
PosTable::Origin origin;
const Expr::AstSymbols & s;
std::unique_ptr<Error> error;
[[nodiscard]] ParseError dupAttr(const AttrPath & attrPath, const PosIdx pos, const PosIdx prevPos);
[[nodiscard]] ParseError dupAttr(Symbol attr, const PosIdx pos, const PosIdx prevPos);
[[nodiscard]] std::optional<ParseError> addAttr(ExprAttrs * attrs, AttrPath && attrPath, std::unique_ptr<Expr> e, const PosIdx pos);
[[nodiscard]] std::optional<ParseError> validateFormals(Formals * formals, PosIdx pos = noPos, Symbol arg = {});
void dupAttr(const AttrPath & attrPath, const PosIdx pos, const PosIdx prevPos);
void dupAttr(Symbol attr, const PosIdx pos, const PosIdx prevPos);
void addAttr(ExprAttrs * attrs, AttrPath && attrPath, std::unique_ptr<Expr> e, const PosIdx pos);
std::unique_ptr<Formals> validateFormals(std::unique_ptr<Formals> formals, PosIdx pos = noPos, Symbol arg = {});
std::unique_ptr<Expr> stripIndentation(const PosIdx pos,
std::vector<std::pair<PosIdx, std::variant<std::unique_ptr<Expr>, StringToken>>> && es);
PosIdx at(const ParserLocation & loc);
// lazy positioning means we don't get byte offsets directly, in.position() would work
// but also requires line and column (which is expensive)
PosIdx at(const auto & in)
{
return positions.add(origin, in.begin() - in.input().begin());
}
PosIdx atEnd(const auto & in)
{
return positions.add(origin, in.end() - in.input().begin());
}
};
inline ParseError ParserState::dupAttr(const AttrPath & attrPath, const PosIdx pos, const PosIdx prevPos)
inline void State::dupAttr(const AttrPath & attrPath, const PosIdx pos, const PosIdx prevPos)
{
return ParseError({
throw ParseError({
.msg = HintFmt("attribute '%1%' already defined at %2%",
showAttrPath(symbols, attrPath), positions[prevPos]),
.pos = positions[pos]
});
}
inline ParseError ParserState::dupAttr(Symbol attr, const PosIdx pos, const PosIdx prevPos)
inline void State::dupAttr(Symbol attr, const PosIdx pos, const PosIdx prevPos)
{
return ParseError({
throw ParseError({
.msg = HintFmt("attribute '%1%' already defined at %2%", symbols[attr], positions[prevPos]),
.pos = positions[pos]
});
}
inline std::optional<ParseError> ParserState::addAttr(ExprAttrs * attrs, AttrPath && attrPath, std::unique_ptr<Expr> e, const PosIdx pos)
inline void State::addAttr(ExprAttrs * attrs, AttrPath && attrPath, std::unique_ptr<Expr> e, const PosIdx pos)
{
AttrPath::iterator i;
// All attrpaths have at least one attr
@ -88,12 +72,12 @@ inline std::optional<ParseError> ParserState::addAttr(ExprAttrs * attrs, AttrPat
ExprAttrs * attrs2 = dynamic_cast<ExprAttrs *>(j->second.e.get());
if (!attrs2) {
attrPath.erase(i + 1, attrPath.end());
return dupAttr(attrPath, pos, j->second.pos);
dupAttr(attrPath, pos, j->second.pos);
}
attrs = attrs2;
} else {
attrPath.erase(i + 1, attrPath.end());
return dupAttr(attrPath, pos, j->second.pos);
dupAttr(attrPath, pos, j->second.pos);
}
} else {
auto next = attrs->attrs.emplace(std::piecewise_construct,
@ -119,7 +103,7 @@ inline std::optional<ParseError> ParserState::addAttr(ExprAttrs * attrs, AttrPat
auto * jAttrs = dynamic_cast<ExprAttrs *>(j->second.e.get());
if (jAttrs && ae) {
if (ae->inheritFromExprs && !jAttrs->inheritFromExprs)
jAttrs->inheritFromExprs = std::make_unique<std::vector<std::unique_ptr<Expr>>>();
jAttrs->inheritFromExprs = std::make_unique<std::vector<ref<Expr>>>();
for (auto & ad : ae->attrs) {
auto j2 = jAttrs->attrs.find(ad.first);
if (j2 != jAttrs->attrs.end()) // Attr already defined in iAttrs, error.
@ -135,7 +119,7 @@ inline std::optional<ParseError> ParserState::addAttr(ExprAttrs * attrs, AttrPat
if (ae->inheritFromExprs)
std::ranges::move(*ae->inheritFromExprs, std::back_inserter(*jAttrs->inheritFromExprs));
} else {
return dupAttr(attrPath, pos, j->second.pos);
dupAttr(attrPath, pos, j->second.pos);
}
} else {
// This attr path is not defined. Let's create it.
@ -147,11 +131,9 @@ inline std::optional<ParseError> ParserState::addAttr(ExprAttrs * attrs, AttrPat
} else {
attrs->dynamicAttrs.emplace_back(std::move(i->expr), std::move(e), pos);
}
return {};
}
inline std::optional<ParseError> ParserState::validateFormals(Formals * formals, PosIdx pos, Symbol arg)
inline std::unique_ptr<Formals> State::validateFormals(std::unique_ptr<Formals> formals, PosIdx pos, Symbol arg)
{
std::sort(formals->formals.begin(), formals->formals.end(),
[] (const auto & a, const auto & b) {
@ -166,21 +148,21 @@ inline std::optional<ParseError> ParserState::validateFormals(Formals * formals,
duplicate = std::min(thisDup, duplicate.value_or(thisDup));
}
if (duplicate)
return ParseError({
throw ParseError({
.msg = HintFmt("duplicate formal function argument '%1%'", symbols[duplicate->first]),
.pos = positions[duplicate->second]
});
if (arg && formals->has(arg))
return ParseError({
throw ParseError({
.msg = HintFmt("duplicate formal function argument '%1%'", symbols[arg]),
.pos = positions[pos]
});
return {};
return formals;
}
inline std::unique_ptr<Expr> ParserState::stripIndentation(const PosIdx pos,
inline std::unique_ptr<Expr> State::stripIndentation(const PosIdx pos,
std::vector<std::pair<PosIdx, std::variant<std::unique_ptr<Expr>, StringToken>>> && es)
{
if (es.empty()) return std::make_unique<ExprString>("");
@ -201,11 +183,11 @@ inline std::unique_ptr<Expr> ParserState::stripIndentation(const PosIdx pos,
}
continue;
}
for (size_t j = 0; j < str->l; ++j) {
for (size_t j = 0; j < str->s.size(); ++j) {
if (atStartOfLine) {
if (str->p[j] == ' ')
if (str->s[j] == ' ')
curIndent++;
else if (str->p[j] == '\n') {
else if (str->s[j] == '\n') {
/* Empty line, doesn't influence minimum
indentation. */
curIndent = 0;
@ -213,7 +195,7 @@ inline std::unique_ptr<Expr> ParserState::stripIndentation(const PosIdx pos,
atStartOfLine = false;
if (curIndent < minIndent) minIndent = curIndent;
}
} else if (str->p[j] == '\n') {
} else if (str->s[j] == '\n') {
atStartOfLine = true;
curIndent = 0;
}
@ -233,23 +215,23 @@ inline std::unique_ptr<Expr> ParserState::stripIndentation(const PosIdx pos,
};
const auto trimString = [&] (const StringToken t) {
std::string s2;
for (size_t j = 0; j < t.l; ++j) {
for (size_t j = 0; j < t.s.size(); ++j) {
if (atStartOfLine) {
if (t.p[j] == ' ') {
if (t.s[j] == ' ') {
if (curDropped++ >= minIndent)
s2 += t.p[j];
s2 += t.s[j];
}
else if (t.p[j] == '\n') {
else if (t.s[j] == '\n') {
curDropped = 0;
s2 += t.p[j];
s2 += t.s[j];
} else {
atStartOfLine = false;
curDropped = 0;
s2 += t.p[j];
s2 += t.s[j];
}
} else {
s2 += t.p[j];
if (t.p[j] == '\n') atStartOfLine = true;
s2 += t.s[j];
if (t.s[j] == '\n') atStartOfLine = true;
}
}
@ -274,9 +256,4 @@ inline std::unique_ptr<Expr> ParserState::stripIndentation(const PosIdx pos,
return std::make_unique<ExprConcatStrings>(pos, true, std::move(es2));
}
inline PosIdx ParserState::at(const ParserLocation & loc)
{
return positions.add(origin, loc.first_column);
}
}

View file

@ -582,9 +582,9 @@ struct CompareValues
{
try {
if (v1->type() == nFloat && v2->type() == nInt)
return v1->fpoint < v2->integer;
return v1->fpoint < v2->integer.value;
if (v1->type() == nInt && v2->type() == nFloat)
return v1->integer < v2->fpoint;
return v1->integer.value < v2->fpoint;
if (v1->type() != v2->type())
state.error<EvalError>("cannot compare %s with %s", showType(*v1), showType(*v2)).debugThrow();
// Allow selecting a subset of enum values
@ -622,14 +622,13 @@ struct CompareValues
}
};
/// NOTE: this type must NEVER be outside of GC-scanned memory.
#if HAVE_BOEHMGC
typedef std::list<Value *, gc_allocator<Value *>> ValueList;
using UnsafeValueList = std::list<Value *, gc_allocator<Value *>>;
#else
typedef std::list<Value *> ValueList;
using UnsafeValueList = std::list<Value *>;
#endif
static Bindings::iterator getAttr(
EvalState & state,
Symbol attrSym,
@ -652,7 +651,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a
state.forceList(*startSet->value, noPos, "while evaluating the 'startSet' attribute passed as argument to builtins.genericClosure");
ValueList workSet;
UnsafeValueList workSet;
for (auto elem : startSet->value->listItems())
workSet.push_back(elem);
@ -668,7 +667,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a
/* Construct the closure by applying the operator to elements of
`workSet', adding the result to `workSet', continuing until
no new elements are found. */
ValueList res;
UnsafeValueList res;
// `doneKeys' doesn't need to be a GC root, because its values are
// reachable from res.
auto cmp = CompareValues(state, noPos, "while comparing the `key` attributes of two genericClosure elements");
@ -866,7 +865,7 @@ static void prim_tryEval(EvalState & state, const PosIdx pos, Value * * args, Va
/* increment state.trylevel, and decrement it when this function returns. */
MaintainCount trylevel(state.trylevel);
ReplExitStatus (* savedDebugRepl)(ref<EvalState> es, const ValMap & extraEnv) = nullptr;
std::function<ReplExitStatus(ref<EvalState> es, ValMap const & extraEnv)> savedDebugRepl;
if (state.debugRepl && evalSettings.ignoreExceptionsDuringTry)
{
/* to prevent starting the repl from exceptions withing a tryEval, null it. */
@ -923,14 +922,15 @@ static RegisterPrimOp primop_getEnv({
.args = {"s"},
.doc = R"(
`getEnv` returns the value of the environment variable *s*, or an
empty string if the variable doesnt exist. This function should be
empty string if the variable doesn't exist. This function should be
used with care, as it can introduce all sorts of nasty environment
dependencies in your Nix expression.
`getEnv` is used in Nix Packages to locate the file
`~/.nixpkgs/config.nix`, which contains user-local settings for Nix
Packages. (That is, it does a `getEnv "HOME"` to locate the users
home directory.)
`getEnv` is used in nixpkgs for evil impurities such as locating the file
`~/.config/nixpkgs/config.nix` which contains user-local settings for nixpkgs.
(That is, it does a `getEnv "HOME"` to locate the user's home directory.)
When in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval), this function always returns an empty string.
)",
.fun = prim_getEnv,
});
@ -1506,6 +1506,7 @@ static RegisterPrimOp primop_storePath({
in a new path (e.g. `/nix/store/ld01dnzc-source-source`).
Not available in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval).
Lix may change this, tracking issue: <https://git.lix.systems/lix-project/lix/issues/402>
See also [`builtins.fetchClosure`](#builtins-fetchClosure).
)",
@ -2512,13 +2513,13 @@ static struct LazyPosAcessors {
PrimOp primop_lineOfPos{
.arity = 1,
.fun = [] (EvalState & state, PosIdx pos, Value * * args, Value & v) {
v.mkInt(state.positions[PosIdx(args[0]->integer)].line);
v.mkInt(state.positions[PosIdx(args[0]->integer.value)].line);
}
};
PrimOp primop_columnOfPos{
.arity = 1,
.fun = [] (EvalState & state, PosIdx pos, Value * * args, Value & v) {
v.mkInt(state.positions[PosIdx(args[0]->integer)].column);
v.mkInt(state.positions[PosIdx(args[0]->integer.value)].column);
}
};
@ -2988,7 +2989,8 @@ static void elemAt(EvalState & state, const PosIdx pos, Value & list, int n, Val
/* Return the n-1'th element of a list. */
static void prim_elemAt(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
elemAt(state, pos, *args[0], state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.elemAt"), v);
NixInt::Inner elem = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.elemAt").value;
elemAt(state, pos, *args[0], elem, v);
}
static RegisterPrimOp primop_elemAt({
@ -3272,17 +3274,19 @@ static RegisterPrimOp primop_all({
static void prim_genList(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
auto len = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.genList");
auto len_ = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.genList").value;
if (len < 0)
state.error<EvalError>("cannot create list of size %1%", len).atPos(pos).debugThrow();
if (len_ < 0)
state.error<EvalError>("cannot create list of size %1%", len_).atPos(pos).debugThrow();
size_t len = len_;
// More strict than striclty (!) necessary, but acceptable
// as evaluating map without accessing any values makes little sense.
state.forceFunction(*args[0], noPos, "while evaluating the first argument passed to builtins.genList");
state.mkList(v, len);
for (unsigned int n = 0; n < (unsigned int) len; ++n) {
for (size_t n = 0; n < len; ++n) {
auto arg = state.allocValue();
arg->mkInt(n);
(v.listElems()[n] = state.allocValue())->mkApp(args[0], arg);
@ -3532,9 +3536,17 @@ static void prim_add(EvalState & state, const PosIdx pos, Value * * args, Value
if (args[0]->type() == nFloat || args[1]->type() == nFloat)
v.mkFloat(state.forceFloat(*args[0], pos, "while evaluating the first argument of the addition")
+ state.forceFloat(*args[1], pos, "while evaluating the second argument of the addition"));
else
v.mkInt( state.forceInt(*args[0], pos, "while evaluating the first argument of the addition")
+ state.forceInt(*args[1], pos, "while evaluating the second argument of the addition"));
else {
auto i1 = state.forceInt(*args[0], pos, "while evaluating the first argument of the addition");
auto i2 = state.forceInt(*args[1], pos, "while evaluating the second argument of the addition");
auto result_ = i1 + i2;
if (auto result = result_.valueChecked(); result.has_value()) {
v.mkInt(*result);
} else {
state.error<EvalError>("integer overflow in adding %1% + %2%", i1, i2).atPos(pos).debugThrow();
}
}
}
static RegisterPrimOp primop_add({
@ -3553,9 +3565,18 @@ static void prim_sub(EvalState & state, const PosIdx pos, Value * * args, Value
if (args[0]->type() == nFloat || args[1]->type() == nFloat)
v.mkFloat(state.forceFloat(*args[0], pos, "while evaluating the first argument of the subtraction")
- state.forceFloat(*args[1], pos, "while evaluating the second argument of the subtraction"));
else
v.mkInt( state.forceInt(*args[0], pos, "while evaluating the first argument of the subtraction")
- state.forceInt(*args[1], pos, "while evaluating the second argument of the subtraction"));
else {
auto i1 = state.forceInt(*args[0], pos, "while evaluating the first argument of the subtraction");
auto i2 = state.forceInt(*args[1], pos, "while evaluating the second argument of the subtraction");
auto result_ = i1 - i2;
if (auto result = result_.valueChecked(); result.has_value()) {
v.mkInt(*result);
} else {
state.error<EvalError>("integer overflow in subtracting %1% - %2%", i1, i2).atPos(pos).debugThrow();
}
}
}
static RegisterPrimOp primop_sub({
@ -3574,9 +3595,18 @@ static void prim_mul(EvalState & state, const PosIdx pos, Value * * args, Value
if (args[0]->type() == nFloat || args[1]->type() == nFloat)
v.mkFloat(state.forceFloat(*args[0], pos, "while evaluating the first of the multiplication")
* state.forceFloat(*args[1], pos, "while evaluating the second argument of the multiplication"));
else
v.mkInt( state.forceInt(*args[0], pos, "while evaluating the first argument of the multiplication")
* state.forceInt(*args[1], pos, "while evaluating the second argument of the multiplication"));
else {
auto i1 = state.forceInt(*args[0], pos, "while evaluating the first argument of the multiplication");
auto i2 = state.forceInt(*args[1], pos, "while evaluating the second argument of the multiplication");
auto result_ = i1 * i2;
if (auto result = result_.valueChecked(); result.has_value()) {
v.mkInt(*result);
} else {
state.error<EvalError>("integer overflow in multiplying %1% * %2%", i1, i2).atPos(pos).debugThrow();
}
}
}
static RegisterPrimOp primop_mul({
@ -3603,10 +3633,12 @@ static void prim_div(EvalState & state, const PosIdx pos, Value * * args, Value
NixInt i1 = state.forceInt(*args[0], pos, "while evaluating the first operand of the division");
NixInt i2 = state.forceInt(*args[1], pos, "while evaluating the second operand of the division");
/* Avoid division overflow as it might raise SIGFPE. */
if (i1 == std::numeric_limits<NixInt>::min() && i2 == -1)
state.error<EvalError>("overflow in integer division").atPos(pos).debugThrow();
v.mkInt(i1 / i2);
auto result_ = i1 / i2;
if (auto result = result_.valueChecked(); result.has_value()) {
v.mkInt(*result);
} else {
state.error<EvalError>("integer overflow in dividing %1% / %2%", i1, i2).atPos(pos).debugThrow();
}
}
}
@ -3621,8 +3653,9 @@ static RegisterPrimOp primop_div({
static void prim_bitAnd(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
v.mkInt(state.forceInt(*args[0], pos, "while evaluating the first argument passed to builtins.bitAnd")
& state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.bitAnd"));
auto i1 = state.forceInt(*args[0], pos, "while evaluating the first argument passed to builtins.bitAnd");
auto i2 = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.bitAnd");
v.mkInt(i1.value & i2.value);
}
static RegisterPrimOp primop_bitAnd({
@ -3636,8 +3669,10 @@ static RegisterPrimOp primop_bitAnd({
static void prim_bitOr(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
v.mkInt(state.forceInt(*args[0], pos, "while evaluating the first argument passed to builtins.bitOr")
| state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.bitOr"));
auto i1 = state.forceInt(*args[0], pos, "while evaluating the first argument passed to builtins.bitOr");
auto i2 = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.bitOr");
v.mkInt(i1.value | i2.value);
}
static RegisterPrimOp primop_bitOr({
@ -3651,8 +3686,10 @@ static RegisterPrimOp primop_bitOr({
static void prim_bitXor(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
v.mkInt(state.forceInt(*args[0], pos, "while evaluating the first argument passed to builtins.bitXor")
^ state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.bitXor"));
auto i1 = state.forceInt(*args[0], pos, "while evaluating the first argument passed to builtins.bitXor");
auto i2 = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.bitXor");
v.mkInt(i1.value ^ i2.value);
}
static RegisterPrimOp primop_bitXor({
@ -3732,13 +3769,19 @@ static RegisterPrimOp primop_toString({
non-negative. */
static void prim_substring(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
int start = state.forceInt(*args[0], pos, "while evaluating the first argument (the start offset) passed to builtins.substring");
NixInt::Inner start = state.forceInt(*args[0], pos, "while evaluating the first argument (the start offset) passed to builtins.substring").value;
if (start < 0)
state.error<EvalError>("negative start position in 'substring'").atPos(pos).debugThrow();
int len = state.forceInt(*args[1], pos, "while evaluating the second argument (the substring length) passed to builtins.substring");
NixInt::Inner len = state.forceInt(*args[1], pos, "while evaluating the second argument (the substring length) passed to builtins.substring").value;
// Negative length may be idiomatically passed to builtins.substring to get
// the tail of the string.
if (len < 0) {
len = std::numeric_limits<NixInt::Inner>::max();
}
// Special-case on empty substring to avoid O(n) strlen
// This allows for the use of empty substrings to efficently capture string context
@ -3780,7 +3823,7 @@ static void prim_stringLength(EvalState & state, const PosIdx pos, Value * * arg
{
NixStringContext context;
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.stringLength");
v.mkInt(s->size());
v.mkInt(NixInt::Inner(s->size()));
}
static RegisterPrimOp primop_stringLength({
@ -4160,7 +4203,8 @@ static void prim_compareVersions(EvalState & state, const PosIdx pos, Value * *
{
auto version1 = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.compareVersions");
auto version2 = state.forceStringNoCtx(*args[1], pos, "while evaluating the second argument passed to builtins.compareVersions");
v.mkInt(compareVersions(version1, version2));
auto result = compareVersions(version1, version2);
v.mkInt(result < 0 ? -1 : result > 0 ? 1 : 0);
}
static RegisterPrimOp primop_compareVersions({

View file

@ -148,9 +148,16 @@ static void fetchTree(
}
else if (attr.value->type() == nBool)
attrs.emplace(state.symbols[attr.name], Explicit<bool>{attr.value->boolean});
else if (attr.value->type() == nInt)
attrs.emplace(state.symbols[attr.name], uint64_t(attr.value->integer));
else
else if (attr.value->type() == nInt) {
auto intValue = attr.value->integer.value;
if (intValue < 0) {
state.error<EvalError>("negative value given for fetchTree attr %1%: %2%", state.symbols[attr.name], intValue).atPos(pos).debugThrow();
}
unsigned long asUnsigned = intValue;
attrs.emplace(state.symbols[attr.name], asUnsigned);
} else
state.error<TypeError>("fetchTree argument '%s' is %s while a string, Boolean or integer is expected",
state.symbols[attr.name], showType(*attr.value)).debugThrow();
}

View file

@ -264,22 +264,24 @@ private:
return true;
}
if (options.force) {
// The item is going to be forced during printing anyway, but we need its type now.
state.forceValue(*item, item->determinePos(noPos));
}
// Pretty-print single-item attrsets only if they contain nested
// structures.
auto itemType = item->type();
return itemType == nList || itemType == nAttrs || itemType == nThunk;
return itemType == nList || itemType == nAttrs;
}
void printAttrs(Value & v, size_t depth)
{
if (seen && !seen->insert(v.attrs).second) {
printRepeated();
return;
}
if (options.force && options.derivationPaths && state.isDerivation(v)) {
printDerivation(v);
} else if (depth < options.maxDepth) {
} else if (seen && !v.attrs->empty() && !seen->insert(v.attrs).second) {
printRepeated();
} else if (depth < options.maxDepth || v.attrs->empty()) {
increaseIndent();
output << "{";
@ -335,10 +337,15 @@ private:
return true;
}
if (options.force) {
// The item is going to be forced during printing anyway, but we need its type now.
state.forceValue(*item, item->determinePos(noPos));
}
// Pretty-print single-item lists only if they contain nested
// structures.
auto itemType = item->type();
return itemType == nList || itemType == nAttrs || itemType == nThunk;
return itemType == nList || itemType == nAttrs;
}
void printList(Value & v, size_t depth)
@ -348,7 +355,7 @@ private:
return;
}
if (depth < options.maxDepth) {
if (depth < options.maxDepth || v.listSize() == 0) {
increaseIndent();
output << "[";
auto listItems = v.listItems();
@ -574,4 +581,10 @@ fmt_internal::HintFmt & fmt_internal::HintFmt::operator%(const ValuePrinter & va
return *this;
}
std::ostream & operator<<(std::ostream & output, ExprPrinter const & printer)
{
printer.expr.show(printer.state.symbols, output);
return output;
}
}

View file

@ -15,6 +15,8 @@
namespace nix {
struct Expr;
class EvalState;
struct Value;
@ -50,6 +52,12 @@ void printValue(EvalState & state, std::ostream & str, Value & v, PrintOptions o
/**
* A partially-applied form of `printValue` which can be formatted using `<<`
* without allocating an intermediate string.
* This class should not outlive the eval state or it will UAF.
* FIXME: This should take `nix::ref`s, to avoid that, but our eval methods all have
* EvalState &, not ref<EvalState>, and constructing a new shared_ptr to data that
* already has a shared_ptr is a much bigger footgun. In the current architecture of
* libexpr, using a ValuePrinter after an EvalState has been destroyed would be
* pretty hard.
*/
class ValuePrinter {
friend std::ostream & operator << (std::ostream & output, const ValuePrinter & printer);
@ -73,4 +81,26 @@ std::ostream & operator<<(std::ostream & output, const ValuePrinter & printer);
template<>
fmt_internal::HintFmt & fmt_internal::HintFmt::operator%(const ValuePrinter & value);
/**
* A partially-applied form of Expr::show(), which can be formatted using `<<`
* without allocating an intermediate string.
* This class should not outlive the eval state or it will UAF.
* FIXME: This should take `nix::ref`s, to avoid that, but our eval methods all have
* EvalState &, not ref<EvalState>, and constructing a new shared_ptr to data that
* already has a shared_ptr is a much bigger footgun. In the current architecture of
* libexpr, using an ExprPrinter after an EvalState has been destroyed would be
* pretty hard.
*/
class ExprPrinter
{
/** The eval state used to get symbols. */
EvalState const & state;
/** The expression to print. */
Expr const & expr;
public:
ExprPrinter(EvalState const & state, Expr const & expr) : state(state), expr(expr) { }
friend std::ostream & operator << (std::ostream & output, ExprPrinter const & printer);
};
}

View file

@ -22,7 +22,7 @@ json printValueAsJSON(EvalState & state, bool strict,
switch (v.type()) {
case nInt:
out = v.integer;
out = v.integer.value;
break;
case nBool:

View file

@ -4,15 +4,14 @@
#include <cassert>
#include <climits>
#include "gc-alloc.hh"
#include "symbol-table.hh"
#include "value/context.hh"
#include "input-accessor.hh"
#include "source-path.hh"
#include "print-options.hh"
#include "checked-arithmetic.hh"
#if HAVE_BOEHMGC
#include <gc/gc_allocator.h>
#endif
#include <nlohmann/json_fwd.hpp>
namespace nix {
@ -73,8 +72,8 @@ class EvalState;
class XMLWriter;
class Printer;
typedef int64_t NixInt;
typedef double NixFloat;
using NixInt = checked::Checked<int64_t>;
using NixFloat = double;
/**
* External values must descend from ExternalValueBase, so that
@ -254,6 +253,11 @@ public:
app.left = app.right = 0;
}
inline void mkInt(NixInt::Inner n)
{
mkInt(NixInt{n});
}
inline void mkInt(NixInt n)
{
clearValue();
@ -458,17 +462,9 @@ void Value::mkBlackhole()
thunk.expr = (Expr*) &eBlackHole;
}
#if HAVE_BOEHMGC
typedef std::vector<Value *, traceable_allocator<Value *>> ValueVector;
typedef std::map<Symbol, Value *, std::less<Symbol>, traceable_allocator<std::pair<const Symbol, Value *>>> ValueMap;
typedef std::map<Symbol, ValueVector, std::less<Symbol>, traceable_allocator<std::pair<const Symbol, ValueVector>>> ValueVectorMap;
#else
typedef std::vector<Value *> ValueVector;
typedef std::map<Symbol, Value *> ValueMap;
typedef std::map<Symbol, ValueVector> ValueVectorMap;
#endif
using ValueVector = GcVector<Value *>;
using ValueMap = GcMap<Symbol, Value *>;
using ValueVectorMap = std::map<Symbol, ValueVector>;
/**
* A value allocated in traceable memory.

View file

@ -1,7 +1,50 @@
#include "abstract-setting-to-json.hh"
#include "args.hh"
#include "config-impl.hh"
#include "fetch-settings.hh"
#include <nlohmann/json.hpp>
namespace nix {
template<> AcceptFlakeConfig BaseSetting<AcceptFlakeConfig>::parse(const std::string & str) const
{
if (str == "true") return AcceptFlakeConfig::True;
else if (str == "ask") return AcceptFlakeConfig::Ask;
else if (str == "false") return AcceptFlakeConfig::False;
else throw UsageError("option '%s' has invalid value '%s'", name, str);
}
template<> std::string BaseSetting<AcceptFlakeConfig>::to_string() const
{
if (value == AcceptFlakeConfig::True) return "true";
else if (value == AcceptFlakeConfig::Ask) return "ask";
else if (value == AcceptFlakeConfig::False) return "false";
else abort();
}
template<> void BaseSetting<AcceptFlakeConfig>::convertToArg(Args & args, const std::string & category)
{
args.addFlag({
.longName = name,
.description = "Accept Lix configuration options from flakes without confirmation. This allows flakes to gain root access to your machine if you are a trusted user; see the nix.conf manual page for more details.",
.category = category,
.handler = {[this]() { override(AcceptFlakeConfig::True); }}
});
args.addFlag({
.longName = "ask-" + name,
.description = "Ask whether to accept Lix configuration options from flakes.",
.category = category,
.handler = {[this]() { override(AcceptFlakeConfig::Ask); }}
});
args.addFlag({
.longName = "no-" + name,
.description = "Reject Lix configuration options from flakes.",
.category = category,
.handler = {[this]() { override(AcceptFlakeConfig::False); }}
});
}
FetchSettings::FetchSettings()
{
}

View file

@ -11,6 +11,8 @@
namespace nix {
enum class AcceptFlakeConfig { False, Ask, True };
struct FetchSettings : public Config
{
FetchSettings();
@ -86,15 +88,21 @@ struct FetchSettings : public Config
"Whether to use flake registries to resolve flake references.",
{}, true, Xp::Flakes};
Setting<bool> acceptFlakeConfig{this, false, "accept-flake-config",
Setting<AcceptFlakeConfig> acceptFlakeConfig{
this, AcceptFlakeConfig::Ask, "accept-flake-config",
R"(
Whether to accept Lix configuration from the `nixConfig` attribute of
a flake without prompting. This is almost always a very bad idea.
Setting this setting as a trusted user allows Nix flakes to gain root
a flake. Doing so as a trusted user allows Nix flakes to gain root
access on your machine if they set one of the several
trusted-user-only settings that execute commands as root.
If set to `true`, such configuration will be accepted without asking;
this is almost always a very bad idea. Setting this to `ask` will
prompt the user each time whether to allow a certain configuration
option set this way, and offer to optionally remember their choice.
When set to `false`, the configuration will be automatically
declined.
See [multi-user installations](@docroot@/installation/multi-user.md)
for more details on the Lix security model.
)",

View file

@ -232,7 +232,7 @@ std::pair<StorePath, Input> fetchFromWorkdir(ref<Store> store, Input & input, co
if (S_ISDIR(st.st_mode)) {
auto prefix = file + "/";
auto i = files.lower_bound(prefix);
return i != files.end() && (*i).starts_with(prefix);
return (i != files.end() && (*i).starts_with(prefix)) || files.count(file);
}
return files.count(file);
@ -361,7 +361,7 @@ struct GitInputScheme : InputScheme
args.push_back(destDir);
runProgram("git", true, args, {}, true);
runProgram("git", true, args, true);
}
std::optional<Path> getSourcePath(const Input & input) const override
@ -399,12 +399,12 @@ struct GitInputScheme : InputScheme
if (commitMsg) {
// Pause the logger to allow for user input (such as a gpg passphrase) in `git commit`
logger->pause();
Finally restoreLogger([]() { logger->resume(); });
auto [_fd, msgPath] = createTempFile("nix-msg");
AutoDelete const _delete{msgPath};
writeFile(msgPath, *commitMsg);
runProgram("git", true,
{ "-C", *root, "--git-dir", gitDir, "commit", std::string(path.rel()), "-F", "-" },
*commitMsg);
{ "-C", *root, "--git-dir", gitDir, "commit", std::string(path.rel()), "-F", msgPath }, true);
}
}
}
@ -589,7 +589,7 @@ struct GitInputScheme : InputScheme
: ref == "HEAD"
? *ref
: "refs/heads/" + *ref;
runProgram("git", true, { "-C", repoDir, "--git-dir", gitDir, "fetch", "--quiet", "--force", "--", actualUrl, fmt("%s:%s", fetchRef, fetchRef) }, {}, true);
runProgram("git", true, { "-C", repoDir, "--git-dir", gitDir, "fetch", "--quiet", "--force", "--", actualUrl, fmt("%s:%s", fetchRef, fetchRef) }, true);
} catch (Error & e) {
if (!pathExists(localRefFile)) throw;
warn("could not update local clone of Git repository '%s'; continuing with the most recent version", actualUrl);
@ -656,7 +656,7 @@ struct GitInputScheme : InputScheme
// everything to ensure we get the rev.
Activity act(*logger, lvlTalkative, actUnknown, fmt("making temporary clone of '%s'", repoDir));
runProgram("git", true, { "-C", tmpDir, "fetch", "--quiet", "--force",
"--update-head-ok", "--", repoDir, "refs/*:refs/*" }, {}, true);
"--update-head-ok", "--", repoDir, "refs/*:refs/*" }, true);
}
runProgram("git", true, { "-C", tmpDir, "checkout", "--quiet", input.getRev()->gitRev() });
@ -683,22 +683,19 @@ struct GitInputScheme : InputScheme
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching submodules of '%s'", actualUrl));
runProgram("git", true, { "-C", tmpDir, "submodule", "--quiet", "update", "--init", "--recursive" }, {}, true);
runProgram("git", true, { "-C", tmpDir, "submodule", "--quiet", "update", "--init", "--recursive" }, true);
}
filter = isNotDotGitDirectory;
} else {
// FIXME: should pipe this, or find some better way to extract a
// revision.
auto source = sinkToSource([&](Sink & sink) {
runProgram2({
.program = "git",
.args = { "-C", repoDir, "--git-dir", gitDir, "archive", input.getRev()->gitRev() },
.standardOut = &sink
});
auto proc = runProgram2({
.program = "git",
.args = { "-C", repoDir, "--git-dir", gitDir, "archive", input.getRev()->gitRev() },
.captureStdout = true,
});
Finally const _wait([&] { proc.wait(); });
unpackTarfile(*source, tmpDir);
unpackTarfile(*proc.stdout(), tmpDir);
}
auto storePath = store->addToStore(name, tmpDir, FileIngestionMethod::Recursive, htSHA256, filter);

View file

@ -28,10 +28,9 @@ static RunOptions hgOptions(const Strings & args)
}
// runProgram wrapper that uses hgOptions instead of stock RunOptions.
static std::string runHg(const Strings & args, const std::optional<std::string> & input = {})
static std::string runHg(const Strings & args)
{
RunOptions opts = hgOptions(args);
opts.input = input;
auto res = runProgram(std::move(opts));

View file

@ -130,10 +130,8 @@ struct PathInputScheme : InputScheme
time_t mtime = 0;
if (!storePath || storePath->name() != "source" || !store->isValidPath(*storePath)) {
// FIXME: try to substitute storePath.
auto src = sinkToSource([&](Sink & sink) {
mtime = dumpPathAndGetMtime(absPath, sink, defaultPathFilter);
});
storePath = store->addToStoreFromDump(*src, "source");
auto src = GeneratorSource{dumpPathAndGetMtime(absPath, mtime, defaultPathFilter)};
storePath = store->addToStoreFromDump(src, "source");
}
input.attrs.insert_or_assign("lastModified", uint64_t(mtime));

View file

@ -71,7 +71,7 @@ DownloadFileResult downloadFile(
storePath = std::move(cached->storePath);
} else {
StringSink sink;
dumpString(res.data, sink);
sink << dumpString(res.data);
auto hash = hashString(htSHA256, res.data);
ValidPathInfo info {
*store,

View file

@ -1,5 +1,6 @@
#include "common-args.hh"
#include "args/root.hh"
#include "error.hh"
#include "globals.hh"
#include "loggers.hh"
#include "logging.hh"
@ -14,14 +15,14 @@ MixCommonArgs::MixCommonArgs(const std::string & programName)
.shortName = 'v',
.description = "Increase the logging verbosity level.",
.category = loggingCategory,
.handler = {[]() { verbosity = (Verbosity) (verbosity + 1); }},
.handler = {[]() { verbosity = verbosityFromIntClamped(int(verbosity) + 1); }},
});
addFlag({
.longName = "quiet",
.description = "Decrease the logging verbosity level.",
.category = loggingCategory,
.handler = {[]() { verbosity = verbosity > lvlError ? (Verbosity) (verbosity - 1) : lvlError; }},
.handler = {[]() { verbosity = verbosityFromIntClamped(int(verbosity) - 1); }},
});
addFlag({
@ -57,7 +58,7 @@ MixCommonArgs::MixCommonArgs(const std::string & programName)
addFlag({
.longName = "log-format",
.description = "Set the format of log output; one of `raw`, `internal-json`, `bar` or `bar-with-logs`.",
.description = "Set the format of log output; one of `raw`, `internal-json`, `bar`, `bar-with-logs`, `multiline` or `multiline-with-logs`.",
.category = loggingCategory,
.labels = {"format"},
.handler = {[](std::string format) { setLogFormat(format); }},

View file

@ -17,6 +17,10 @@ LogFormat parseLogFormat(const std::string & logFormatStr) {
return LogFormat::bar;
else if (logFormatStr == "bar-with-logs")
return LogFormat::barWithLogs;
else if (logFormatStr == "multiline")
return LogFormat::multiline;
else if (logFormatStr == "multiline-with-logs")
return LogFormat::multilineWithLogs;
throw Error("option 'log-format' has an invalid value '%s'", logFormatStr);
}
@ -35,6 +39,17 @@ Logger * makeDefaultLogger() {
logger->setPrintBuildLogs(true);
return logger;
}
case LogFormat::multiline: {
auto logger = makeProgressBar();
logger->setPrintMultiline(true);
return logger;
}
case LogFormat::multilineWithLogs: {
auto logger = makeProgressBar();
logger->setPrintMultiline(true);
logger->setPrintBuildLogs(true);
return logger;
}
default:
abort();
}

View file

@ -11,6 +11,8 @@ enum class LogFormat {
internalJSON,
bar,
barWithLogs,
multiline,
multilineWithLogs,
};
void setLogFormat(const std::string & logFormatStr);

File diff suppressed because it is too large Load diff

View file

@ -1,14 +1,118 @@
#pragma once
///@file
#include <chrono>
#include "logging.hh"
#include "sync.hh"
namespace nix {
struct ProgressBar : public Logger
{
struct ActInfo
{
using TimePoint = std::chrono::time_point<std::chrono::steady_clock>;
std::string s, lastLine, phase;
ActivityType type = actUnknown;
uint64_t done = 0;
uint64_t expected = 0;
uint64_t running = 0;
uint64_t failed = 0;
std::map<ActivityType, uint64_t> expectedByType;
bool visible = true;
ActivityId parent;
std::optional<std::string> name;
TimePoint startTime;
};
struct ActivitiesByType
{
std::map<ActivityId, std::list<ActInfo>::iterator> its;
uint64_t done = 0;
uint64_t expected = 0;
uint64_t failed = 0;
};
struct State
{
std::list<ActInfo> activities;
std::map<ActivityId, std::list<ActInfo>::iterator> its;
std::map<ActivityType, ActivitiesByType> activitiesByType;
int lastLines = 0;
uint64_t filesLinked = 0, bytesLinked = 0;
uint64_t corruptedPaths = 0, untrustedPaths = 0;
uint32_t paused = 1;
bool haveUpdate = false;
};
Sync<State> state_;
std::thread updateThread;
std::condition_variable quitCV, updateCV;
bool printBuildLogs = false;
bool printMultiline = false;
bool isTTY;
ProgressBar(bool isTTY);
~ProgressBar();
void pause() override;
void resetProgress() override;
void resume() override;
bool isVerbose() override;
void log(Verbosity lvl, std::string_view s) override;
void logEI(const ErrorInfo & ei) override;
void log(State & state, Verbosity lvl, std::string_view s);
void startActivity(
ActivityId act,
Verbosity lvl,
ActivityType type,
const std::string & s,
const Fields & fields,
ActivityId parent
) override;
bool hasAncestor(State & state, ActivityType type, ActivityId act);
void stopActivity(ActivityId act) override;
void result(ActivityId act, ResultType type, const std::vector<Field> & fields) override;
void update(State & state);
std::chrono::milliseconds draw(State & state, const std::optional<std::string_view> & s);
std::string getStatus(State & state);
void writeToStdout(std::string_view s) override;
std::optional<char> ask(std::string_view msg) override;
void setPrintBuildLogs(bool printBuildLogs) override;
void setPrintMultiline(bool printMultiline) override;
private:
void eraseProgressDisplay(State & state);
};
Logger * makeProgressBar();
void startProgressBar();
void stopProgressBar();
}

View file

@ -4,7 +4,6 @@
#include "gc-store.hh"
#include "signals.hh"
#include "loggers.hh"
#include "progress-bar.hh"
#include "current-process.hh"
#include <algorithm>
@ -34,9 +33,13 @@ void printGCWarning()
{
if (!gcWarning) return;
static bool haveWarned = false;
warnOnce(haveWarned,
"you did not specify '--add-root'; "
"the result might be removed by the garbage collector");
if (!haveWarned) {
haveWarned = true;
warn(
"you did not specify '--add-root'; "
"the result might be removed by the garbage collector"
);
}
}
@ -282,7 +285,7 @@ void parseCmdLine(const std::string & programName, const Strings & args,
void printVersion(const std::string & programName)
{
std::cout << fmt("%1% (Lix, like Nix) %2%", programName, nixVersion) << std::endl;
if (verbosity > lvlInfo) {
if (verbosity > lvlNotice) {
Strings cfg;
#if HAVE_BOEHMGC
cfg.push_back("gc");
@ -349,7 +352,7 @@ RunPager::RunPager()
if (!pager) pager = getenv("PAGER");
if (pager && ((std::string) pager == "" || (std::string) pager == "cat")) return;
stopProgressBar();
logger->pause();
Pipe toPager;
toPager.create();
@ -378,7 +381,7 @@ RunPager::RunPager()
RunPager::~RunPager()
{
try {
if (pid != -1) {
if (pid) {
std::cout.flush();
dup2(std_out, STDOUT_FILENO);
pid.wait();

View file

@ -38,7 +38,7 @@ void BinaryCacheStore::init()
{
std::string cacheInfoFile = "nix-cache-info";
auto cacheInfo = getFile(cacheInfoFile);
auto cacheInfo = getFileContents(cacheInfoFile);
if (!cacheInfo) {
upsertFile(cacheInfoFile, "StoreDir: " + storeDir + "\n", "text/x-nix-cache-info");
} else {
@ -67,16 +67,18 @@ void BinaryCacheStore::upsertFile(const std::string & path,
upsertFile(path, std::make_shared<std::stringstream>(std::move(data)), mimeType);
}
void BinaryCacheStore::getFile(const std::string & path, Sink & sink)
box_ptr<Source> BinaryCacheStore::getFile(const std::string & path)
{
sink(*getFile(path));
return make_box_ptr<GeneratorSource>([](std::string data) -> Generator<Bytes> {
co_yield std::span{data.data(), data.size()};
}(std::move(*getFileContents(path))));
}
std::optional<std::string> BinaryCacheStore::getFile(const std::string & path)
std::optional<std::string> BinaryCacheStore::getFileContents(const std::string & path)
{
StringSink sink;
try {
getFile(path, sink);
return getFile(path)->drain();
} catch (NoSuchBinaryCacheFile &) {
return std::nullopt;
}
@ -327,26 +329,33 @@ std::optional<StorePath> BinaryCacheStore::queryPathFromHashPart(const std::stri
}
}
void BinaryCacheStore::narFromPath(const StorePath & storePath, Sink & sink)
WireFormatGenerator BinaryCacheStore::narFromPath(const StorePath & storePath)
{
auto info = queryPathInfo(storePath).cast<const NarInfo>();
LengthSink narSize;
TeeSink tee { sink, narSize };
auto decompressor = makeDecompressionSink(info->compression, tee);
try {
getFile(info->url, *decompressor);
auto file = getFile(info->url);
return [](auto info, auto file, auto & stats) -> WireFormatGenerator {
constexpr size_t buflen = 65536;
auto buf = std::make_unique<char []>(buflen);
size_t total = 0;
auto decompressor = makeDecompressionSource(info->compression, *file);
try {
while (true) {
const auto len = decompressor->read(buf.get(), buflen);
co_yield std::span{buf.get(), len};
total += len;
}
} catch (EndOfFile &) {
}
stats.narRead++;
//stats.narReadCompressedBytes += nar->size(); // FIXME
stats.narReadBytes += total;
}(std::move(info), std::move(file), stats);
} catch (NoSuchBinaryCacheFile & e) {
throw SubstituteGone(std::move(e.info()));
}
decompressor->finish();
stats.narRead++;
//stats.narReadCompressedBytes += nar->size(); // FIXME
stats.narReadBytes += narSize.length;
}
std::shared_ptr<const ValidPathInfo> BinaryCacheStore::queryPathInfoUncached(const StorePath & storePath)
@ -359,7 +368,7 @@ std::shared_ptr<const ValidPathInfo> BinaryCacheStore::queryPathInfoUncached(con
auto narInfoFile = narInfoFileFor(storePath);
auto data = getFile(narInfoFile);
auto data = getFileContents(narInfoFile);
if (!data) return nullptr;
@ -383,16 +392,14 @@ StorePath BinaryCacheStore::addToStore(
HashSink sink { hashAlgo };
if (method == FileIngestionMethod::Recursive) {
dumpPath(srcPath, sink, filter);
sink << dumpPath(srcPath, filter);
} else {
readFile(srcPath, sink);
sink << readFileSource(srcPath);
}
auto h = sink.finish().first;
auto source = sinkToSource([&](Sink & sink) {
dumpPath(srcPath, sink, filter);
});
return addToStoreCommon(*source, repair, CheckSigs, [&](HashResult nar) {
auto source = GeneratorSource{dumpPath(srcPath, filter)};
return addToStoreCommon(source, repair, CheckSigs, [&](HashResult nar) {
ValidPathInfo info {
*this,
name,
@ -425,7 +432,7 @@ StorePath BinaryCacheStore::addTextToStore(
return path;
StringSink sink;
dumpString(s, sink);
sink << dumpString(s);
StringSource source(sink.s);
return addToStoreCommon(source, repair, CheckSigs, [&](HashResult nar) {
ValidPathInfo info {
@ -446,7 +453,7 @@ std::shared_ptr<const Realisation> BinaryCacheStore::queryRealisationUncached(co
{
auto outputInfoFilePath = realisationsPrefix + "/" + id.to_string() + ".doi";
auto data = getFile(outputInfoFilePath);
auto data = getFileContents(outputInfoFilePath);
if (!data) return {};
auto realisation = Realisation::fromJSON(
@ -486,7 +493,7 @@ std::optional<std::string> BinaryCacheStore::getBuildLogExact(const StorePath &
debug("fetching build log from binary cache '%s/%s'", getUri(), logPath);
return getFile(logPath);
return getFileContents(logPath);
}
void BinaryCacheStore::addBuildLog(const StorePath & drvPath, std::string_view log)

View file

@ -83,9 +83,9 @@ public:
/**
* Dump the contents of the specified file to a sink.
*/
virtual void getFile(const std::string & path, Sink & sink);
virtual box_ptr<Source> getFile(const std::string & path);
virtual std::optional<std::string> getFile(const std::string & path);
virtual std::optional<std::string> getFileContents(const std::string & path);
public:
@ -136,7 +136,7 @@ public:
std::shared_ptr<const Realisation> queryRealisationUncached(const DrvOutput &) override;
void narFromPath(const StorePath & path, Sink & sink) override;
WireFormatGenerator narFromPath(const StorePath & path) override;
ref<FSAccessor> getFSAccessor() override;

View file

@ -12,6 +12,8 @@
#include "topo-sort.hh"
#include "local-store.hh" // TODO remove, along with remaining downcasts
#include "logging-json.hh"
#include "substitution-goal.hh"
#include "drv-output-substitution-goal.hh"
#include <regex>
#include <queue>
@ -175,7 +177,7 @@ void DerivationGoal::getDerivation()
return;
}
addWaitee(upcast_goal(worker.makePathSubstitutionGoal(drvPath)));
addWaitee(worker.makePathSubstitutionGoal(drvPath));
state = &DerivationGoal::loadDerivation;
}
@ -276,19 +278,17 @@ void DerivationGoal::haveDerivation()
if (!status.wanted) continue;
if (!status.known)
addWaitee(
upcast_goal(
worker.makeDrvOutputSubstitutionGoal(
DrvOutput{status.outputHash, outputName},
buildMode == bmRepair ? Repair : NoRepair
)
worker.makeDrvOutputSubstitutionGoal(
DrvOutput{status.outputHash, outputName},
buildMode == bmRepair ? Repair : NoRepair
)
);
else {
auto * cap = getDerivationCA(*drv);
addWaitee(upcast_goal(worker.makePathSubstitutionGoal(
addWaitee(worker.makePathSubstitutionGoal(
status.known->path,
buildMode == bmRepair ? Repair : NoRepair,
cap ? std::optional { *cap } : std::nullopt)));
cap ? std::optional { *cap } : std::nullopt));
}
}
@ -426,7 +426,7 @@ void DerivationGoal::gaveUpOnSubstitution()
if (!settings.useSubstitutes)
throw Error("dependency '%s' of '%s' does not exist, and substitution is disabled",
worker.store.printStorePath(i), worker.store.printStorePath(drvPath));
addWaitee(upcast_goal(worker.makePathSubstitutionGoal(i)));
addWaitee(worker.makePathSubstitutionGoal(i));
}
if (waitees.empty()) /* to prevent hang (no wake-up event) */
@ -479,7 +479,7 @@ void DerivationGoal::repairClosure()
worker.store.printStorePath(i), worker.store.printStorePath(drvPath));
auto drvPath2 = outputsToDrv.find(i);
if (drvPath2 == outputsToDrv.end())
addWaitee(upcast_goal(worker.makePathSubstitutionGoal(i, Repair)));
addWaitee(worker.makePathSubstitutionGoal(i, Repair));
else
addWaitee(worker.makeGoal(
DerivedPath::Built {
@ -760,7 +760,7 @@ void DerivationGoal::tryToBuild()
/* Not now; wait until at least one child finishes or
the wake-up timeout expires. */
if (!actLock)
actLock = std::make_unique<Activity>(*logger, lvlWarn, actBuildWaiting,
actLock = std::make_unique<Activity>(*logger, lvlTalkative, actBuildWaiting,
fmt("waiting for a machine to build '%s'", Magenta(worker.store.printStorePath(drvPath))));
worker.waitForAWhile(shared_from_this());
outputLocks.unlock();
@ -781,7 +781,7 @@ void DerivationGoal::tryLocalBuild() {
throw Error(
"unable to build with a primary store that isn't a local store; "
"either pass a different '--store' or enable remote builds."
"\nhttps://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html");
"\nhttps://docs.lix.systems/manual/lix/stable/advanced-topics/distributed-builds.html");
}
@ -923,12 +923,16 @@ void runPostBuildHook(
};
LogSink sink(act);
runProgram2({
auto proc = runProgram2({
.program = settings.postBuildHook,
.environment = hookEnvironment,
.standardOut = &sink,
.captureStdout = true,
.mergeStderrToStdout = true,
});
Finally const _wait([&] { proc.wait(); });
// FIXME just process the data, without a wrapper sink class
proc.stdout()->drainInto(sink);
}
void DerivationGoal::buildDone()
@ -1162,7 +1166,7 @@ HookReply DerivationGoal::tryBuildHook()
}
else {
s += "\n";
writeToStderr(s);
writeLogsToStderr(s);
}
}
@ -1200,11 +1204,9 @@ HookReply DerivationGoal::tryBuildHook()
throw;
}
CommonProto::WriteConn conn { hook->sink };
/* Tell the hook all the inputs that have to be copied to the
remote system. */
CommonProto::write(worker.store, conn, inputPaths);
hook->sink << CommonProto::write(worker.store, {}, inputPaths);
/* Tell the hooks the missing outputs that have to be copied back
from the remote system. */
@ -1215,7 +1217,7 @@ HookReply DerivationGoal::tryBuildHook()
if (buildMode != bmCheck && status.known && status.known->isValid()) continue;
missingOutputs.insert(outputName);
}
CommonProto::write(worker.store, conn, missingOutputs);
hook->sink << CommonProto::write(worker.store, {}, missingOutputs);
}
hook->sink = FdSink();
@ -1553,11 +1555,12 @@ void DerivationGoal::waiteeDone(GoalPtr waitee, ExitCode result)
Goal::waiteeDone(waitee, result);
if (!useDerivation) return;
auto & fullDrv = *dynamic_cast<Derivation *>(drv.get());
auto * dg = dynamic_cast<DerivationGoal *>(&*waitee);
if (!dg) return;
auto & fullDrv = *dynamic_cast<Derivation *>(drv.get());
auto * nodeP = fullDrv.inputDrvs.findSlot(DerivedPath::Opaque { .path = dg->drvPath });
if (!nodeP) return;
auto & outputs = nodeP->value;

Some files were not shown because too many files have changed in this diff Show more