Commit graph

15982 commits

Author SHA1 Message Date
jade 10cc3b288d Add release notes for removing overflow from Nix language
Change-Id: Ib75ab5b8b4d879035d7ee7678f9cd0c491a39c0a
2024-07-18 19:27:33 +00:00
jade 50a63f8435 expr: fix a compiler warning about different signs in comparison
We know that variable is >=0, so we can just cast it to unsigned.

Change-Id: I3792eeb3ca43e6a507cc44c1a70584d42b2acd7b
2024-07-18 19:27:33 +00:00
jade 5ee1e6ea98 Merge changes Ib20e9aa0,I178a038b,I29c7de04 into main
* changes:
  docs: document the actual comparison rules instead of lies
  daemon: remove workaround for macOS kernel bug that seems fixed
  daemon: fix a crash bug "FATAL: exception not rethrown"
2024-07-18 17:40:32 +00:00
alois31 7b1abf8107 Merge "doc/manual: clarify documentation related to the $$ parser bug" into main 2024-07-18 15:01:20 +00:00
alois31 72db9cd67b doc/release-notes: link the upcoming release notes again
The insertion marker comment broke the list into two parts, the first
containing only the link to the upcoming release notes and the second the
past releases. This confused the generator, leading to the first part being
discarded. Indent the marker comment so that it's syntactically part of the
preceding item, and in particular doesn't split the list any more.

Change-Id: I357c51bb03e4e0d79a76d30158615fd9eda95ea8
2024-07-17 22:12:41 +00:00
raito 67f62bcdb4 doc/release-notes: add date for major release
Change-Id: I93aab93c069bb3989c3f8d17e0862899e6f76865
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-07-17 22:12:41 +00:00
alois31 beb231784e doc/manual: clarify documentation related to the $$ parser bug
Due to a mistake in the grammar, a dollar character implicitly escapes a second
dollar character that immediately follows, so that it cannot start an
interpolation. Unfortunately, this behaviour has since come to be relied upon,
so it cannot be fixed. Furthermore, the documentation on regular strings did
not mention this behaviour at all, while in the case of indented strings it was
rather implicit.
Mention it explicitly in both cases, and describe how an interpolation can
follow a dollar character (namely, by escaping that). Since we have to touch
that section anyway, state that any character (other than n, r, and t; but
notably including `$` even if not succeeded by `{`) can be escaped using a
backslash in regular strings.

Change-Id: I7e5d68a9a4130eec98ce8218b485168f4b31a677
2024-07-17 22:01:48 +00:00
Max “Goldstein” Siling 68567206f2 Merge "tests/functional/repl.sh: actually fail test on wrong stdout" into main 2024-07-17 21:50:59 +00:00
Max “Goldstein” Siling 3a36c8bb90 tests/functional/repl.sh: actually fail test on wrong stdout
Previous test implementation assumed that grep supports newlines
in patterns. It doesn't, so tests spuriously passed, even though
some tests outputs were broken.

This patches output (and expected output) before grepping,
so there're no newlines in pattern.

Change-Id: Ie6561f9f2e18b83d976f162269d20136e2595141
2024-07-17 21:48:13 +00:00
eldritch horrors ef0de7c79f remove boost coroutine references
we no longer need these since sinkToSource and sourceToSink are gone.

Change-Id: Ibbf440e2cf71bf3e9f3b833af2d78a21fb1b3193
2024-07-16 01:50:16 +00:00
eldritch horrors dfedbc154f remove sourceToSink, sinkToSource, and boehm patch
Change-Id: I1379841299713175d0225b82a67f50660f9eb5e2
2024-07-16 01:50:16 +00:00
eldritch horrors d094dd0396 libstore: remove remaining sinkToSource uses
Change-Id: Id1ee0d2ad4a3774f4bbb960d76f0f76ac4f3eff9
2024-07-16 01:50:16 +00:00
eldritch horrors 6b4d46e9e0 libstore: remove WriteConn::sink fields
we no longer need these since we're no longer using sinks to serialize things.

Change-Id: Iffb1a3eab33c83f611c88fa4e8beaa8d5ffa079b
2024-07-16 00:57:42 +00:00
eldritch horrors a5d1f69841 libstore: generatorize protocol serializers
this is cursed. deeply and profoundly cursed. under NO CIRCUMSTANCES
must protocol serializer helpers be applied to temporaries! doing so
will inevitably cause dangling references and cause the entire thing
to crash. we need to do this even so to get rid of boost coroutines,
and likewise to encapsulate the serializers we suffer today at least
a little bit to allow a gradual migration to an actual IPC protocol.

(this isn't a problem that's unique to generators. c++ coroutines in
general cannot safely take references to arbitrary temporaries since
c++ does not have a lifetime system that can make this safe. -sigh-)

Change-Id: I2921ba451e04d86798752d140885d3c5cc08e146
2024-07-16 00:57:42 +00:00
eldritch horrors 5271424d14 libstore: remove a sinkToSouce from old daemon protocol
this doesn't have a test because this code path is only reached by
clients that predate 2.4, and we really should not be caring about
those any more right now. even the test suite doesn't, and the few
tests that might care are disabled because they will not even work

Change-Id: Id9eb190065138fedb2c7d90c328ff9eb9d97385b
2024-07-16 00:57:42 +00:00
eldritch horrors 4ec87742a1 libstore: rewrite the nar parser as a contents generator
this is not completely necessary at this point because the parser right
now already returns a generator to pass through all input data it read,
but the nar parser *was* very lax and would accept nars that weren't in
canonical form (defined as the form dumpPath would return). nar hashing
depends on these things, and as such rewriting the parser now allows us
to reject non-canonical nars that extract to the same store contents as
their canonical counterpart but have different nar hashes despite that.

Change-Id: Iccd319e3bd5912d8297014c84c495edc59019bb7
2024-07-16 00:57:42 +00:00
Qyriad c052716edd Merge changes I8d87c0e9,I25937702 into main
* changes:
  nix3-upgrade-nix: always use the /new/ nix-env to perform the installation
  libutil: implement a realPath() utility
2024-07-15 23:18:03 +00:00
eldritch horrors 3447dbfb2c libstore: rewrite narFromPath as generator
Change-Id: Ifa783c2c65c06ddd1d0212016d5bfd07666ea91c
2024-07-15 21:50:25 +00:00
Lunaphied 5e16b10cb1 Merge "use clangStdenv for the default devShell, so we get clangd by default" into main 2024-07-15 21:47:23 +00:00
Qyriad ae7eab49b9 nix3-upgrade-nix: always use the /new/ nix-env to perform the installation
Fixes #411.

Change-Id: I8d87c0e9295deea26ff33234e15ee33cc68ab303
2024-07-15 15:26:53 -06:00
Qyriad d9c51ec4e5 libutil: implement a realPath() utility
Just a wrapper around POSIX realpath().

Change-Id: I2593770285dbae573eace490efce5b272b00b001
2024-07-15 15:26:53 -06:00
nan-git 1eb5d22132 Merge "libexpr/eval.cc: remove unnecessary C string conversion" into main 2024-07-14 16:53:54 +00:00
nan-git 505640baec libexpr/eval.cc: remove unnecessary C string conversion
Change-Id: I5b7c21df84ff8ff64cf6a1e261fc3729a06bd4f6
2024-07-14 02:13:58 +00:00
Artemis Tosini 201e8b6994
nix-support/binary-tarball.nix: Pass through root paths
Passing through root paths allows external programs to see
which nix and cacert are in a binary tarball,
e.g. to recreate it from substituters

Change-Id: I27431134df53bbc6623484f8a0822004b51f7c87
2024-07-13 23:22:43 +00:00
jade 702f02c31f docs: document the actual comparison rules instead of lies
Although the comparison rules are ugly and we do not like various parts
of them, we must not hide them away for only catgirls to know about, so
the documentation should actually say how they work.

Change-Id: Ib20e9aa0e7b6486ade4f401035aafd85fbb08c91
2024-07-13 01:17:14 +02:00
jade 69e2ee5b25 daemon: remove workaround for macOS kernel bug that seems fixed
This was filed as https://github.com/nixos/nix/issues/7584, but as far
as I can tell, the previous solution of POLLHUP works just fine on macOS
14. I've also tested on an ancient machine with macOS 10.15.7, which
also has POLLHUP work correctly.

It's possible this might regress some older versions of macOS that have
a kernel bug, but I went looking through the history on the sources and
didn't find anything that looked terribly convincingly like a bug fix
between 2020 and today. If such a broken version exists, it seems pretty
reasonable to suggest simply updating the OS.

Change-Id: I178a038baa000f927ea2cbc4587d69d8ab786843
2024-07-13 01:17:14 +02:00
jade a8f443d960 docs: update to define integer overflow
Change-Id: Ie8a1b31035f2d27a220e5df2e9e178ec3b39ee68
2024-07-13 00:59:33 +02:00
jade b3fb8d9822 daemon: fix a crash bug "FATAL: exception not rethrown"
This is caused by pthread_cancel effectively throwing a
not-specifically-identifiable C++ exception into the targeted thread,
which, if it is not rethrown, terminates the process entirely.

This is rather "impolite" behaviour, we would say. But thread
cancellation is *always* busted, and we should simply not use it where
unnecessary. It's particularly unnecessary when what we *actually* need
it for is, err, interrupting a poll(2).

That can in turn be achieved by simply listening to more stuff in the
poll, namely, a pipe, which we send a character to when needing to
stop the thread.

While looking at this code, we also investigated whether any of the
poll() madness is required, or was even *ever* required. Curiously we
found in the XNU kernel source code that the thing about needing to
listen to POLLHUP is probably *correct*, but switching it to POLLRDNORM
should not have made any difference at all. We've left a FIXME to look
into that further because what's written here is super janky.

94d3b45284/bsd/kern/sys_generic.c (L1751-L1758)

This is the crash on some Hydra machines:

Thread 1 (Thread 0x7f56b77776c0 (LWP 955542) (Exiting)):
0  0x00007f56b8e9b7dc in __pthread_kill_implementation () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
1  0x00007f56b8e49516 in raise () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
2  0x00007f56b8e31935 in abort () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
3  0x00007f56b8e327f3 in __libc_message_impl.cold () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
4  0x00007f56b8e8e8e9 in __libc_fatal () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
5  0x00007f56b8ea23c4 in unwind_cleanup () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
6  0x00007f56b9d2a1b8 in nix::triggerInterrupt() [clone .cold] () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixutil.so
7  0x00007f56b990ac9d in std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<nix::MonitorFdHup::MonitorFdHup(int)::{lambda()#1}> > >::_M_run() () from /nix/store/sahgw550p621m9dy1pd7whl9c5g1g0p7-lix-2.90.0-rc1/lib/liblixstore.so
8  0x00007f56b90e86d3 in execute_native_thread_routine () from /nix/store/c6r62m84hywf4i6qq1h28f13zv38yqyp-gcc-13.3.0-lib/lib/libstdc++.so.6
9  0x00007f56b8e99a42 in start_thread () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6
10 0x00007f56b8f1905c in clone3 () from /nix/store/m71p7f0nymb19yn1dascklyya2i96jfw-glibc-2.39-52/lib/libc.so.6

As for testing, we've started a daemon with this change and verified it
deals with HUPs correctly on x86_64-linux, but I don't think we can
easily test the destructor behaviour without whatever Hydra was
doing that broke.

Change-Id: I29c7de0425674494b6e43c075810126c3ff77363
2024-07-13 00:59:33 +02:00
jade 917c9bdee7 language: cleanly ban integer overflows
This also bans various sneaking of negative numbers from the language
into unsuspecting builtins as was exposed while auditing the
consequences of changing the Nix language integer type to a newtype.

It's unlikely that this change comprehensively ensures correctness when
passing integers out of the Nix language and we should probably add a
checked-narrowing function or something similar, but that's out of scope
for the immediate change.

During the development of this I found a few fun facts about the
language:
- You could overflow integers by converting from unsigned JSON values.
- You could overflow unsigned integers by converting negative numbers
  into them when going into Nix config, into fetchTree, and into flake
  inputs.

  The flake inputs and Nix config cannot actually be tested properly
  since they both ban thunks, however, we put in checks anyway because
  it's possible these could somehow be used to do such shenanigans some
  other way.

Note that Lix has banned Nix language integer overflows since the very
first public beta, but threw a SIGILL about them because we run with
-fsanitize=signed-overflow -fsanitize-undefined-trap-on-error in
production builds. Since the Nix language uses signed integers, overflow
was simply undefined behaviour, and since we defined that to trap, it
did.

Trapping on it was a bad UX, but we didn't even entirely notice
that we had done this at all until it was reported as a bug a couple of
months later (which is, to be fair, that flag working as intended), and
it's got enough production time that, aside from code that is IMHO buggy
(and which is, in any case, not in nixpkgs) such as
lix-project/lix#445, we don't think
anyone doing anything reasonable actually depends on wrapping overflow.

Even for weird use cases such as doing funny bit crimes, it doesn't make
sense IMO to have wrapping behaviour, since two's complement arithmetic
overflow behaviour is so *aggressively* not what you want for *any* kind
of mathematics/algorithms. The Nix language exists for package
management, a domain where bit crimes are already only dubiously in
scope to begin with, and it makes a lot more sense for that domain for
the integers to never lose precision, either by throwing errors if they
would, or by being arbitrary-precision.

This change will be ported to CppNix as well, to maintain language
consistency.

Fixes: lix-project/lix#423

Change-Id: I51f253840c4af2ea5422b8a420aa5fafbf8fae75
2024-07-13 00:59:33 +02:00
jade f9641b9efd libutil: add checked arithmetic tools
This is in preparation for adding checked arithmetic to the evaluator.

Change-Id: I6e115ce8f5411feda1706624977a4dcd5efd4d13
2024-07-13 00:56:37 +02:00
Lunaphied 0339b2fbd2 use clangStdenv for the default devShell, so we get clangd by default
The default-stdenv-devShell can always be used with `.#native-stdenvPackages`.

Change-Id: I9b3e72210ba5219b6b65c71a2818110769623904
2024-07-12 20:52:33 +00:00
jade dde51af97d Use std::strong_ordering for version comparison
The actual motive here is the avoidance of integer overflow if we were
to make these use checked NixInts and retain the subtraction.

However, the actual *intent* of this code is a three-way comparison,
which can be done with operator<=>, so we should just do *that* instead.

Change-Id: I7f9a7da1f3176424b528af6d1b4f1591e4ab26bf
2024-07-12 16:48:28 +02:00
eldritch horrors 4b109ec1a8 libstore: remove upcast_goal
upcast_goal was only ever needed to break circular includes, but the
same solution that gave us upcast_goal also lets us fully remove it:
just upcast goals without a wrapper function, but only in .cc files.

Change-Id: I9c71654b2535121459ba7dcfd6c5da5606904032
2024-07-11 21:31:52 +00:00
eldritch horrors a5d431a911 libstore: turn copyNAR into a generator
Change-Id: Id452f6a03faa1037ff13af0f63e32883966ff40d
2024-07-11 20:37:27 +00:00
eldritch horrors 03db4efab9 libstore: turn the NAR parser into a passthrough generator
this will let us turn copyNAR into a generator as well, which in turn is
necessary to turn the users of copyNAR into generators without resorting
to sinkToSource coroutines. currently this uses the SerializingTransform
in all cases, even for copyNAR where it is not necessary. should this be
a performance problem we can easily swap out the transform for one which
does not produce any bytes of its own, but that should not be necessary.

Change-Id: I7e685879318fcbb78d8b88abfddd7752360eb0ce
2024-07-11 20:37:27 +00:00
eldritch horrors 31478c810a libutil: remove makeDecompressionSink
the sole remaining user of this function can use makeDecompressionSource
instead, while making the sinkToSource in the caller unnecessary as well

Change-Id: I4258227b5dbbb735a75b477d8a57007bfca305e9
2024-07-11 11:39:18 +00:00
eldritch horrors 5587dbdcf0 libstore: make BinaryCacheStore::getFile return a source
this lets us remove the last true remaining uses of
makeDecompressionSink.

Change-Id: I146ca2bbe1a9ae9a367117a7b8a304b23a63e5e2
2024-07-11 11:39:18 +00:00
eldritch horrors df8851f286 libutil: rewrite RewritingSink as source
the rewriting sink was just broken. when given a rewrite set that
contained a key that is also a proper infix of another key it was
possible to produce an incorrectly rewritten result if the writer
used the wrong block size. fixing this duplicates rewriteStrings,
to avoid this we'll rewrite rewriteStrings to use RewritingSource
in a new mode that'll allow rewrites we had previously forbidden.

Change-Id: I57fa0a9a994e654e11d07172b8e31d15f0b7e8c0
2024-07-11 11:39:18 +00:00
Lunaphied 014410cbf0 Merge "lix-doc: update dependencies and refactor" into main 2024-07-10 17:47:27 +00:00
eldritch horrors f0c751d4d6 Merge changes I81552018,Ieb65c133 into main
* changes:
  releng: add releaseTests flake output, test script
  add aarch64-linux as a cross-build target
2024-07-10 17:45:17 +00:00
piegames 3dced96741 devShell: Fix errors with environment variables
Previous code assumed that $MANPATH always exists

Change-Id: I5a4d012045ba6ff9086373b3f46a75d82285d393
2024-07-10 14:22:47 +02:00
alois31 fa92f41a18 Merge "libmain: clear display attributes in the multiline progress bar" into main 2024-07-10 03:43:07 +00:00
Lunaphied 41963df4a5 lix-doc: update dependencies and refactor
This updates the version of rnix used and refactors the code generally
to be more precise and capable in it's identification of both lambdas
and determining which documentation comments are attached.

Change-Id: Ib0dddabd71f772c95077f9d7654023b37a7a1fd2
2024-07-10 02:25:48 +00:00
eldritch horrors f5aa5b6815 releng: add releaseTests flake output, test script
this is supposed to be a set of outputs we want to always succeed for
releases. sadly we can't add nixos installer tests using lix to these
because the nixos test framework does not allow overriding nix in the
installer test suites due to unfortunate oversights in the framework.

Change-Id: I815520181ccca70a47205d38ba27e73529347f04
2024-07-09 22:50:51 +02:00
eldritch horrors 4d8c66ec6f add aarch64-linux as a cross-build target
we want to be sure we can cross-build to aarch64 for releases, add a
target to our crossSystems list to make those cheacks easier to run.

Change-Id: Ieb65c1333a5232641ace0ba4d122fc7d528ebc04
2024-07-09 22:49:10 +02:00
Quantum Jump 6e0ca02425 Fix dry-run flag for nix-collect-garbage
`nix-collect-garbage --dry-run` previously elided the entire garbage
collection check, meaning that it would just exit the script without
printing anything.

This change makes the dry run flag instead set the GC action to
`gcReturnDead` rather than `gcDeleteDead`, and then continue with the
script. So if you set `--dry-run`, it will print the paths it *would*
have garbage collected, but not actually delete them.

I filed a bug for this: lix-project/lix#432 but then realised I could give fixing it a go myself.

Change-Id: I062dbf1a80bbab192b5fd0b3a453a0b555ad16f2
2024-07-09 13:55:05 +00:00
Qyriad accfd8aa9d libexpr: stop lying about DrvInfo's constness
DrvInfo's query methods all use mutable fields to cache, but like.
that's basically the entire interface for DrvInfo. Not only that, but
these formerly-const-marked functions can even throw due to eval errors!
Changing this only required removing some `const` markers in nix-env,
and changing a single inline `queryInstalled()` call to be an lvalue
instead.

Change-Id: I796807118f3b35b0e93668b5e28210d9e521b2ae
2024-07-08 17:52:02 +00:00
alois31 f5ff70d7f3
libmain: clear display attributes in the multiline progress bar
Activities can set display attributes in their log output using the "Select
Graphics Rendition" functionality. To prevent interfering with subsequent text
displayed, these should be reset after writing the log line. The multiline
progress bar neglected to do this, resulting for example in a colorised
"building …" header in the next line. Reset the attributes properly, like the
standard progress bar already does.

Change-Id: I1dc69f4a1d747a76b83e8721a72d9bb0e5554488
2024-07-08 19:08:23 +02:00
Artemis Tosini d461cc1d7b
libstore: make LocalDerivationGoal::needsHashRewrite virtual
This rather simple function existed just to check some flags,
but the response varies by platform. This is a perfect case for
our subclasses.

Change-Id: Ieb1732a8d024019236e0d0028ad843a24ec3dc59
2024-07-07 18:06:08 +00:00
eldritch horrors 55a32f24d3 libutil: remove RewritingSink match/size tracking
size tracking can be done with a LengthSink and a tee. match tracking
was defeated by never having done any match tracking, all users would
see the same (empty) set of matches at all times. match tracking with
bytes offsets alone would not be sufficient in the general case, only
because computeHashModulo uses a single rewrite could it have worked.

Change-Id: Idb214b5222e0ea24f450f5505712a342b63d7570
2024-07-06 12:36:37 +02:00