Also fixed up the precedence of ? by making it a special case of ?:. or
is kept as part of the language for now but will be removed in a future
commit.
Open TODOs:
- Fix the semantics around non-attrset null values
- Changelog
(and, as usual, tests)
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Iaac19101805c3e09fd98322ad1f7b35ae9a0225d
Same as list unpacking, but for sets. Semantics still need to be
fine-tuned, especially w.r.t. interactions with recursion
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I60a4d7c74767135552826f75e9c9fc1946120f56
Allows inlining lists within list elements by prefixing them with a `*`.
The main motivation for this is to make common concatenation chains
involving lib.optionals prettier and faster.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ia1b4036f7f6d984f27b28ce84779b088ca422f0a
This allows creating subsets of attrsets with `set.{identifiers...}`,
and also lists of attribute values with `set.[identifiers...]`.
The latter is a first-class replacements for anti-patterns like
`with pkgs;`.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Iff104eb344992b41c03840ff330224c630e5cacf
It's finally time!
Mixing space indentation and tab indentation within a single string is
not allowed. Apart from that, it's a rather trivial extension of the
syntax.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I738acfa4f09ac3593056c612668a0658fa604af3
Copied over the escape rules improvements to indented strings (with
appropriate adjustments). Indented strings now have their line endings
properly normalized. Also the first line is not stripped anymore, and
mixing first line and multiline indented strings is now forbidden to
minimize the risk of confusion.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: If9e2240a04627cb43aa25c76db021a5acf7c58fa
A complete overhaul of the escape semantics, which are just horrible and
mostly useless. The design is inspired by JSON and Rust. Indented
strings are not affected
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Id89679d4115d59869090bdbb5d9b305f374447fb
CRLF line endings remain allowed for Windows support. This commit is
also preparation for proper line ending normalization in strings, to
ensure that derivations build the same on all platforms.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: Ib4946c9975804e02ea7584e0c7373c82ed4961f9
Strongly restricted which identifiers are allowed in let bindings.
Also removed some silly parser optimizations which were bleeding into
the semantics, once again.
Change-Id: I89470e0a81274c280179f72fa1f5718576d08afb
This was never a feature to begin with, just a leaky parser abstraction.
(Point in case: The overrides never worked for builtins.)
This also removes the weird syntax sugar expansion for those operators,
and uses dedicated AST nodes instead.
Change-Id: Ic3e06c4f9b7181d62357e144140b985105f5d824
This is done because the one in nixpkgs has several problems and we
don't want to conflict with them fixing those problems:
- not building shared libs
- not building debuginfo (more critical for us due to
lix-project/lix#549)
- not setting the correct cmake build type
- not setting the correct cxxflags to build the coroutine library
- not building with clang (required for the coroutine library since gcc
is known to miscompile coroutines *in kj*).
CC: lix-project/lix#551
Fixes: lix-project/lix#550
Change-Id: Ia5b78dc1809963fdd1a8203b127a216cb575d751
This workaround was for the stdenv not being set when callPackage'ing
package.nix for some of the stranger CI outputs.
Change-Id: I2acdd6efa721b90dd3cb04358544d25d591ff084
The old behavior results in lots of concatenations happening for no good
reason and is an artifact of the technical limitations of the old parser
(combined with some lack of care for such details).
Change-Id: I0d78d6220ca6aeaa10bc437e48e08bf7922e0bb3
This is only a minor semantical distinction, but we should be able to
properly test it, and the parser tests rely on show for that.
Change-Id: I25e868cf9544e30cdff17deb5fd50a434e0f367e
This commit should faithfully reproduce the old behavior down to the
bugs. The new code is a lot more readable, all quirks are well
documented, and it is overall much more maintainable.
Change-Id: I629585918e4f2b7d296b6b8330235cdc90b7bade
This failed due to https://github.com/NixOS/nixpkgs/pull/346945, which
makes a second lowdown-unsandboxed that works in nix builds, and the
regular lowdown has executables that fail closed when the sandbox setup
fails.
The actual failure here is only visible on nixos-unstable at the moment,
not 24.05, but this commit should fix it up for all versions.
Fixes: lix-project/lix#547
Change-Id: I50c0ecb59518ef01a7c0181114c1b4c5a7c6b78b
A better fix than in 104448e75d, hence a
revert + the fix.
It turns out that this commit has the side-effect that when having e.g.
`StrictHostKeyChecking=accept-new` for a remote builder, the warnings à la
Warning: Permanently added 'builder' (ED25519) to the list of known hosts.
actually end up in the derivation's log whereas hostkey verification
errors don't, but only in the stderr of the `nix-build` invocation
(which was the motivation for the patch).
This change writes the stderr from the build-hook to
* the daemon's stderr, so that the SSH errors appear in the journal
(which was the case before 104448e75d)
* the client's stderr, as a log message
* NOT to the drv log (this is handled via `handleJSONLogMessage`)
I tried to fix the issue for legacy-ssh as well, but failed and
ultimately decided to not bother.
I know that we'll sooner or later replace the entire component, however
this is the part of the patch I have working for a while, so I figured I
might still submit it for the time being.
Change-Id: I21ca1aa0d8ae281d2eacddf26e0aa825272707e5
When the git default branch is not set to master the installcheck
test suite fails. This patch adjusts the test setup scripts to
ignore the system and user git config files.
GIT_CONFIG_SYSTEM is set to /dev/null to ignore /etc/gitconfig
GIT_CONFIG_GLOBAL is not set because the global config files
are loaded from $HOME or $XDG_CONFIG_HOME which we already
reset.
git documentation: https://git-scm.com/docs/git#Documentation/git.txt-codeGITCONFIGGLOBALcode
Change-Id: Ie73bbed1db9419c9885b9d57e4edb7a4047d5cce
While debugging something else I observed that latest `main` ignores
`Control-C` on `sudo nix-build`.
After reading through the capnproto docs, it seems as if the promise
must be fulfilled to actually terminate the `promise.wait()` below.
This also applies to scenarios such as stopping the client
(`nix-build`), but the builders on the daemon-side are still running,
i.e. closes#540
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I9634d14df4909fc1b65d05654aad0309bcca8a0a
So we received a report that the thread pool crashed due to an
Interrupted exception.
Relevant log tail:
copying path '/nix/store/0kal2k73inviikxv9f1ciaj39lkl9a87-etc-os-release' to 'ssh://192.168.0.27'...
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:
error (ignored): error: interrupted by the user
Exception: nix::Interrupted: error: interrupted by the user
Relevant stack trace:
4# __cxa_rethrow in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
5# nix::ignoreExceptionExceptInterrupt(nix::Verbosity) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
6# nix::ThreadPool::doWork(bool) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
7# 0x00007FA7A00E86D3 in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
8# 0x00007FA79FE99A42 in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
9# 0x00007FA79FF1905C in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
Notably, this is *not* in the main thread, so this implies that the
thread didn't get joined properly before their destructors got called.
That, in turn, should have only possibly happened because join() threw
on a previous iteration of the loop joining threads, I think. Or if it
threw while in the ThreadPool destructor. Either way we had better stop
letting Interrupted fall out of our child threads!
If:
- Interrupted was thrown inside the action in the main thread: it would
have fallen out of doWork if state->exception was already set and got
caught by ThreadPool::process, calling shutdown() and the join loop
which would crash the process entirely.
- Interrupted was thrown inside the action on a secondary thread: it
would have been caught and put into the exception field and then
possibly rethrown to fall out of the thread (since it was previously
ignoreExceptionExceptInterrupt).
The one possible hole in this hypothesis is that there is an "error
(ignored)" line in there implying that at least one Interrupted got
eaten by an ignoreExceptionInDestructor. It's also unclear whether this
got reordered because of stderr buffering.
Fixes: lix-project/lix#542
Change-Id: I322cf050da660af78f5cb0e08ec6e6d27d09ac76
This is capped at 12 because 3.7 seconds of startup is painful enough
and 5.5 seconds with 24 was more annoying.
Change-Id: I327db40fd98deaa5330cd9cf6de99fb07b2c1cb0
I am tired of bad shell scripts, let me write bad python quickly
instead. It's definitely, $100%, better.
This is not planned as an immediate replacement of the old test suite,
but we::jade would not oppose tests getting ported.
What is here is a mere starting point and there is a lot more
functionality that we need.
Fixes: lix-project/lix#488
Change-Id: If762efce69030bb667491b263b874c36024bf7b6
I followed @pennae's advice and moved the constructor definition of
`AttrName` from the header file `nixexpr.hh` to `nixexpr.cc`.
Change-Id: I733f56c25635b366b11ba332ccec38dd7444e793
The approach that was taken here was to add default values to the type
definitions rather than specify them whenever they are missing.
Now the only remaining warning is '-Wunused-parameter' which @jade said
is usually counterproductive and that we can just disable it:
lix-project/lix#456 (comment)
So this change adds the flags '-Wall', '-Wextra' and
'-Wno-unused-parameter', so that all warnings are enabled except for
'-Wunused-parameter'.
Change-Id: Ic223a964d67ab429e8da804c0721ba5e25d53012
There was a bug report about a potential call to `memcpy` with a null
pointer which is not reproducible:
lix-project/lix#492
This occurred in `src/libstore/filetransfer.cc` in `InnerSource::read`.
To ensure that this doesn't happen, an early return is added before
calling `memcpy` if the length of the data to be copied is 0.
This change also adds a test that ensures that when `InnerSource::read`
is called with an empty file, it throws an `EndOfFile` exception.
Change-Id: Ia18149bee9a3488576c864f28475a3a0c9eadfbb
these two functions are now nearly trivial and much better inline into
makeGoalCommon. keeping them separate also separates information about
goal completion flows and how failure information ends up in `Worker`.
Change-Id: I6af86996e4a2346583371186595e3013c88fb082
we can use our newfound powers of Goal::work Is A Real Promise to remove
completed goals from continuation promises. apart from being much easier
to follow it's also a lot more efficient because we have the iterator to
the item we are trying to remove, skipping a linear search of the cache.
Change-Id: Ie0190d051c5f4b81304d98db478348b20c209df5
Goal::work() is a fully usable promise that does not rely on the worker
to report completion conditions. as such we no longer need the `notify`
field that enabled this interplay. we do have to clear goal caches when
destroying the worker though, otherwise goal promises may (incorrectly)
keep goals alive due to strong shared pointers created by childStarted.
Change-Id: Ie607209aafec064dbdf3464fe207d70ba9ee158a
derivation goals still hold a BuildResult member variable since parts of
these results of accumulated in different places, but the Goal class now
no longer has such a field. substitution goals don't need it at all, and
derivation goals should also be refactored to not drop their buildResult
Change-Id: Ic6d3d471cdbe790a6e09a43445e25bedec6ed446
the field is simply duplicated between the two, and now that we can
return WorkResults from Worker::run we no longer need both of them.
Change-Id: I82fc47d050b39b7bb7d1656445630d271f6c9830
this will be needed to move all interesting result fields out of Goal
proper and into WorkResult. once that is done we can treat goals as a
totally internal construct of the worker mechanism, which also allows
us to fully stop exposing unclear intermediate state to Worker users.
Change-Id: I98d7778a4b5b2590b7b070bdfc164a22a0ef7190
since we now propagate goal exceptions properly we no longer need to
check topGoals for a reason to abort early. any early abort reasons,
whether by exception or a clean top goal failure, can now be handled
by inspecting the goal result in the main loop. this greatly reduces
goal-to-goal interactions that do not happen at the main loop level.
since the underscore-free name is now available for use as variables
we'll migrate to that where we currently use `_topGoals` for locals.
Change-Id: I5727c5ea7799647c0a69ab76975b1a03a6558aa6
drop childException since it's no longer needed. also makes
waitForInput, childFinished, and childTerminated redundant.
Change-Id: I05d88ffd323c5b5c909ac21056162f69ffb0eb9f
there's no reason to have the worker set information on goals that the
goals themselves return from their entry point. doing this in the goal
`work()` function is much cleaner, and a prerequisite to removing more
implicit strong shared references to goals that are currently running.
Change-Id: Ibb3e953ab8482a6a21ce2ed659d5023a991e7923
this simplifies the worker loop, and lets us remove it entirely later.
note that ideally only one promise waiting for interrupts should exist
in the entire system. not one per event loop, one per *process*. extra
interrupt waiters make interrupt response nondeterministic and as such
aren't great for user experience. if anything wants to react to aborts
caused by explicit interruptions, or anything else, those things would
be much better served using RAII guards such as Finally (or KJ_DEFER).
Change-Id: I41d035ff40172d536e098153c7375b0972110d51
this was a triumph. i'm making a note here: huge success. it's hard to
overstate my satisfaction! i'm not even angry. i'm being so sincere ri
actually, no. we *are* angry. this was one dumbass odyssey. nobody has
asked for this. but not doing it would have locked us into old, broken
protocols forever or (possibly worse) forced us to write our own async
framework building on the old did-you-mean-continuations in Worker. if
we had done that we'd be locked into ever more, and ever more complex,
manual state management all over the place. this just could not stand.
Change-Id: I43a6de1035febff59d2eff83be9ad52af4659871
with async runtime scheduling we can no longer guarantee exact error
counts for builds that do not set keepGoing. the old behavior can be
recovered with a number of hacks that affect scheduling, but none of
those are very easy to follow now advisable. exact error counts will
like not be needed for almost all uses except tests, and *those* had
better check the actual messages rather than how many they got. more
messages can even help to avoid unnecessary rebuilds for most users.
Change-Id: I1c9aa7a401227dcaf2e19975b8cb83c5d4f85d64
In 0e6b3435a1, installation of the HTML manual
was accidentally dropped: setting install_dir on a custom_target only sets the
directory where something is going to be installed if it is installed at all,
but does not itself trigger installation. The latter has to be explicitly
requested, which is just what we do here to get the manual back.
Change-Id: Iff8b791de7e7cb4c8d747c2a9b1154b5fcc32fe0
The test suite can load the global configuration files under certain
circumstances, and, though we would really rather it didn't ever do that
at all, we should at least break the mechanism.
Fixes: lix-project/lix#474
Change-Id: Ib27cb43dd5dfaa70ac491c395b5ba308fd7bd289
due to event loop scheduling behavior it's possible for a derivation
goal to fully finish (having seen all paths it was asked to create),
but to not notify the worker of this in time to prevent another goal
asking the recently-finished goal for more outputs. if this happened
the finished goal would ignore the request for more outputs since it
considered itself fully done, and the delayed result reporting would
cause the requesting goal to assume its request had been honored. if
the requested goal had finished *properly* the worker would recreate
it instead of asking for more outputs, and this would succeed. it is
thus safe to always recreate goals once they are done, so we now do.
Change-Id: Ifedd69ca153372c623abe9a9b49cd1523588814f
This moves the "legacy"/"nix2" commands under a new `src/legacy/`
directory, instead of being scattered around in a bunch of different
directories.
A new `liblegacy` build target is defined, and the `nix` binary is
linked against it.
Then, `RegisterLegacyCommand` is replaced with `LegacyCommand::add`
calls in functions like `registerNixCollectGarbage()`. These
registration functions are called explicitly in `src/nix/main.cc`.
See: lix-project/lix#359
Change-Id: Id450ffc3f793374907599cfcc121863b792aac1a
we'll now loop to update displayed statistics, and use this loop to
limit the update rate to 50 times per second. we could have updated
much more frequently before this (once per iteration of `runImpl`),
much faster than would ever be useful in practice. aggressive stats
updates can even impede progress due to terminal or network delays.
Change-Id: Ifba755a2569f73c919b1fbb06a142c0951395d6d
Worker::run() is now entirely based on the kj event loop and promises,
so we need not handle awakeness of goals manually any more. every goal
can instead, once it has finished a partial work call, defer itself to
being called again in the next iteration of the loop. same end effect.
Change-Id: I320eee2fa60bcebaabd74d1323fa96d1402c1d15
notably we will check whether we want to do GC at all only once during
startup, and we'll only attempt GC every ten seconds rather than every
time a goal has finished a partial work call. this shouldn't cause any
problems in practice since relying on auto-gc is not deterministic and
stores in which builds can fill all remaining free space in merely ten
seconds are severely troubled even when gargage collection runs a lot.
Change-Id: I1175a56bf7f4e531f8be90157ad88750ff2ddec4
Revert submission 1946
Reason for revert: regression in building (found via bisection)
Reported by users:
> error: path '/nix/store/04ca5xwvasz6s3jg0k7njz6rzi0d225w-jq-1.7.1-dev' does not exist in the store
Reverted changes: /q/submissionid:1946
Change-Id: I6f1a4b2f7d7ef5ca430e477fc32bca62fd97036b
nothing needs to signal being still active but not actively pollable,
only that immediate polling for the next goal work phase is in order.
Change-Id: Ia43c1015e94ba4f5f6b9cb92943da608c4a01555
this was immensely inefficient on large caches, as can exist when many
derivations are buildable simultaneously. since we have smart pointers
to goals we can do cache maintenance in goal deleters instead, and use
the exact iterators instead of doing a linear search. this *does* rely
on goals being deleted to remove them from the cache, which isn't true
for toplevel goals. those would have previously been removed when done
in all cases, removing the cache entry when keep-going is set. this is
arguably incorrect since it might result in those goals being retried,
although that could only happen with dynamic derivations or the likes.
(luckily dynamic derivations not complete enough to allow this at all)
Change-Id: I8e750b868393588c33e4829333d370f2c509ce99
makeDerivationGoalCommon had the right idea, but it didn't quite go far
enough. let's do the rest and remove the remaining factory duplication.
Change-Id: I1fe32446bdfb501e81df56226fd962f85720725b
this was a debugging aid from day one that should not have any impact on
build semantics, and if it *does* have an impact on build semantics then
build semantics are seriously broken. keeping the order imposed by these
keys will be impossible once we let a real event loop schedule our jobs.
Change-Id: I5c313324e1f213ab6453d82f41ae5e59de809a5b
without circular references we do not need weak goal pointers except for
caches, which should not prevent goal destructors running. caches though
cannot create circular references even when they keep strong references.
if we removed goals from caches when their work() is fully finished, not
when their destructors are run, we could keep strong pointers in caches.
since we do not gain much from this we keep those pointers weak for now.
Change-Id: I1d4a6850ff5e264443c90eb4531da89f5e97a3a0
have DerivationGoal and its subclasses produce a wrapper promise for
their intermediate results instead, and return this wrapper promise.
Worker already handles promises that do not complete immediately, so
we do not have to duplicate this into an entire result type variant.
Change-Id: Iae8dbf63cfc742afda4d415922a29ac5a3f39348
the new event loop could very occasionally notice that a dependency of
some goal has failed, process the failure, cause the depending goal to
fail accordingly, and in the doing of the latter two steps let further
dependencies that previously have not been reported as failed do their
reporting anyway. in such cases a goal could fail with "1 dependencies
failed", but more than one dependency failure message was shown. we'll
now report the correct number of failed dependency goals in all cases.
Change-Id: I5aa95dcb2db4de4fd5fee8acbf5db833531d81a8
these can be unique rather than shared because shared_ptr has a
converting constructor. preparatory refactor for something else
and not necessary on its own, and the extra allocations we must
do for shared_ptr control blocks isn't usually relevant anyway.
Change-Id: I5391715545240c6ec8e83a031206edafdfc6462f
Since fb38459d6e, each `ref` is appended
with `refs/heads` unless it starts with `refs/` already. This regressed
two use-cases that worked fine before:
* Specifying a commit hash as `ref`: now, if `ref` looks like a commit
hash it will be directly passed to `git fetch`.
* Specifying a tag without `refs/tags` as prefix: now, the fetcher prepends
`refs/*` to a ref that doesn't start with `refs/` and doesn't look
like a commit hash. That way, both a branch and a tag specified in
`ref` can be fetched.
The order of preference in git is
* file in `refs/` (e.g. `HEAD`)
* file in `refs/tags/`
* file in `refs/heads` (i.e. a branch)
After fetching `refs/*`, ref is resolved the same way as git does.
Change-Id: Idd49b97cbdc8c6fdc8faa5a48bef3dec25e4ccc3
This was already the de facto requirement, we use the method `full_path`
on a file object (introduced in Meson 1.4.0) in the functional test
suite's build.
This version of Meson is in NixOS 24.05, so there should be no
compatibility issues should this make it into a backported release of
Lix.
CC: lix-project/lix#247
Change-Id: I5c640824807353b6eb4287e7ed09c4e89a4bdde2
Using `configure_file` to copy files has been deprecated since Meson 0.64.0.
The intended replacement is the `fs.copyfile` method.
This removes the following deprecation warning that arises when a minimum
Meson version is specified:
``
Project [...] uses feature deprecated since '0.64.0': copy arg in configure_file. Use fs.copyfile instead
``
Change-Id: I09ffc92e96311ef9ed594343a0a16d51e74b114a
In Meson, `install_subdir` is meant to be used with directories in the source
directory. When using it to install the HTML manual, we provide it with a path
under the build directory.
We should instead specify an install directory for the HTML manual as part of
the custom target that builds it.
What we do currently isn't broken, just semantically incorrect. Changing it does
get rid of the following deprecation warning, though:
``
Project [...] uses feature deprecated since '0.60.0': install_subdir with empty directory. It worked by accident and is buggy. Use install_emptydir instead.
``
Change-Id: I259583b7bdff8ecbb3b342653d70dc5f034c7fad
also gets rid of explicit strong references to dependencies of any goal,
and weak references to dependers as well. those are now only held within
promises representing goal completion and thus independent of the goal's
relation to each other. the weak references to dependers was only needed
for notifications, and that's much better handled entirely by kj itself.
Change-Id: I00d06df9090f8d6336ee4bb0c1313a7052fb016b
now that we have an event loop in the worker we can use it and its
magical execution suspending properties to replace the slot counts
we managed explicitly with semaphores and raii tokens. technically
this would not have needed an event loop base to be doable, but it
is a whole lot easier to wait for a token to be available if there
is a callback mechanism ready for use that doesn't require a whole
damn dedicated abstract method in Goal to work, and specific calls
to that dedicated method strewn all over the worker implementation
Change-Id: I1da7cf386d94e2bbf2dba9b53ff51dbce6a0cff7
with waitForAWhile turned into promised the core functionality of
waitForInput is now merely to let gc run every so often if needed
Change-Id: I68da342bbc1d67653901cf4502dabfa5bc947628
this simplifies waitForInput quite a lot, and at the same time makes
polling less thundering-herd-y. it even fixes early polling wakeups!
Change-Id: I6dfa62ce91729b8880342117d71af5ae33366414
this removes the rather janky did-you-mean-async poll loop we had so
far. sadly kj does not play well with pty file descriptors, so we do
have to add our own async input stream that does not eat pty EIO and
turns it into an exception. that's still a *lot* better than the old
code, and using a real even loop makes everything else easier later.
Change-Id: Idd7e0428c59758602cc530bcad224cd2fed4c15e
When `nix fmt` is called without an argument, Nix appends the "." argument before calling the formatter. The comment in the code is:
> Format the current flake out of the box
This also happens when formatting sub-folders.
This means that the formatter is now unable to distinguish, as an interface, whether the "." argument is coming from the flake or the user's intent to format the current folder. This decision should be up to the formatter.
Treefmt, for example, will automatically look up the project's root and format all the files. This is the desired behaviour. But because the "." argument is passed, it cannot function as expected.
Upstream-PR: https://github.com/nixos/nix/pull/11438
Change-Id: I60fb6b3ed4ec1b24f81b5f0d76c0be98470817ce
like kj::joinPromisesFailFast this allows waiting for the results of
multiple promises at once, but unlike it not all input promises must
be complete (or any of them failed) for results to become available.
Change-Id: I0e4a37e7bd90651d56b33d0bc5afbadc56cde70c
like a normal semaphore, but with awaitable acquire actions. this is
primarily intended as an intermediate concurrency limiting device in
the Worker code, but it may find other uses over time. we do not use
std::counting_semaphore as a base because the counter of that is not
inspectable as will be needed for Worker. we also do not need atomic
operations for cross-thread consistency since we don't have multiple
threads (thanks to kj event loops being confined to a single thread)
Change-Id: Ie2bcb107f3a2c0185138330f7cbba4cec6cbdd95
Without this, verifying TLS certificates would fail on macOS, as well
as any system that doesn't have a certificate file at /etc/ssl/certs/ca-certificates.crt,
which includes e.g. Fedora.
Change-Id: Iaa2e0e9db3747645b5482c82e3e0e4e8f229f5f9
This is better for privacy and to avoid leaking netrc credentials in a
MITM attack, but also the assumption that we check the hash no longer
holds in some cases (in particular for impure derivations).
Partially reverts 5db358d4d7.
(cherry picked from commit c04bc17a5a0fdcb725a11ef6541f94730112e7b6)
(cherry picked from commit f2f47fa725fc87bfb536de171a2ea81f2789c9fb)
(cherry picked from commit 7b39cd631e0d3c3d238015c6f450c59bbc9cbc5b)
Upstream-PR: https://github.com/NixOS/nix/pull/11585
Change-Id: Ia973420f6098113da05a594d48394ce1fe41fbb9
These stack traces kind of suck for the reasons mentioned on the
CppTrace page here (no symbols for inline functions is a major one):
https://github.com/jeremy-rifkin/cpptrace
I would consider using CppTrace if it were packaged, but to be honest, I
think that the more reasonable option is actually to move entirely to
out-of-process crash handling and symbolization.
The reason for this is that if you want to generate anything of
substance on SIGSEGV or really any deadly signal, you are stuck in
async-signal-safe land, which is not a place to be trying to run a
symbolizer. LLVM does it anyway, probably carefully, and chromium *can*
do it on debug builds but in general uses crashpad:
https://source.chromium.org/chromium/chromium/src/+/main:base/debug/stack_trace_posix.cc;l=974;drc=82dff63dbf9db05e9274e11d9128af7b9f51ceaa;bpv=1;bpt=1
However, some stack traces are better than *no* stack traces when we get
mystery exceptions falling out the bottom of the program. I've also
promoted the path for "mystery exceptions falling out the bottom of the
program" to hard crash and generate a core dump because although there's
been some months since the last one of these, these are nonetheless
always *atrociously* diagnosed.
We can't improve the crash handling further until either we use Crashpad
(which involves more C++ deps, no thanks) or we put in the ostensibly
work in progress Rust minidump infrastructure, in which case we need to
finish full support for Rust in libutil first.
Sample report:
Lix crashed. This is a bug. We would appreciate if you report it at https://git.lix.systems/lix-project/lix/issues with the following information included:
Exception: std::runtime_error: lol
Stack trace:
0# nix::printStackTrace() in /home/jade/lix/lix3/build/src/nix/../libutil/liblixutil.so
1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
5# 0x00005CF65B6B048B in /home/jade/lix/lix3/build/src/nix/nix
6# 0x000073C985C8810E in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
7# __libc_start_main in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
8# 0x00005CF65B610335 in /home/jade/lix/lix3/build/src/nix/nix
Change-Id: I1a9f6d349b617fd7145a37159b78ecb9382cb4e9
We don't support GCC anymore for building, so the overlay currently
fails to evaluate with
error: assertion '((stdenv).cc.isClang || lintInsteadOfBuild)' failed
`clangStdenv` seems like a reasonable default now.
Noticed while upgrading Lix for our Hydra fork.
Change-Id: I948a7c03b3e5648fc7c596f96e1b8053a9e7f92f
Previously, Doxygen needed to be ran from the project's source root dir
due to the relative paths in the config's `INPUT` tag. We now preprocess
the relative paths by prefixing them with the absolute path of the
project's source root dir. The HTML output remains unchanged.
Fixes: lix-project/lix#240
Change-Id: I85f099c22bfc5fdbf26be27c2db7dcbc8155c8b2
This caused an infinite loop before since it would just keep asking the
underlying source for more data.
In practice this happened because an HTTP server served a
response to a HEAD request (for which curl will not retrieve any body or
call our write callback function) with Content-Encoding: br, leading to
decompressing nothing at all and going into an infinite loop.
This adds a test to make sure none of our compression methods do that
again, as well as just patching the HTTP client to never feed empty data
into a compression algorithm (since they absolutely have the right to
throw CompressionError on unexpectedly-short streams!).
Reported on Matrix: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$8BWQR_zKxCQDJ40C5NnDo4bQPId3pZ_aoDj2ANP7Itc?via=lix.systems&via=matrix.org&via=tchncs.de
Change-Id: I027566e280f0f569fdb8df40e5ecbf46c211dad1
The legitimate output of `nix path-info` may visually interfere with the
progress bar, by appending to stale progress output before the latter has been
erased. Conveniently, all expensive operations (evaluation or building) have
already been performed before, so we can simply wipe the progress bar at this
point to fix the issue.
Fixes: lix-project/lix#343
Change-Id: Id9a807a5c882295b3e6fbf841f9c15dc96f67f6e
This test suite was in desperate need of using the parameterization
available with gtest, and was a bunch of useless duplicated code. At
least now it's not duplicated code, though it still probably should be
more full of property tests.
Change-Id: Ia8ccee7ef4f02b2fa40417b79aa8c8f0626ea479
See lix-project/lix#496.
The core idea is to be able to do e.g.
nix-instantiate -A some-nonfree-thing --arg config.allowUnfree true
which is currently not possible since `config.allowUnfree` is
interpreted as attribute name with a dot in it.
In order to change that (probably), Jade suggested to find out if there
are any folks out there relying on this behavior.
For such a use-case, it may still be possible to accept strings, i.e.
`--arg '"config.allowUnfree"'.
Change-Id: I986c73619fbd87a95b55e2f0ac03feaed3de2d2d
Apparently forgejo has a more creative interpretation of \(\) than I was
hoping in their markdown parser and thought it was maths. I have no idea
then how you put a link in parens next to another square-bracket link,
but I am not going to worry about it.
There were several more typos, which I also fixed.
Fixes: lix-project/lix#517
Change-Id: I6b144c6881f92ca60ba72a304ce7a0bcb9c6659a
* Move the extended attribute deletion after the hardlink sanity check. We
shouldn't be removing extended attributes on random files.
* Make the entity owner-writable before attempting to remove extended
attributes, since this operation usually requires write access on the file,
and we shouldn't fail xattr deletion on a file that has been made unwritable
by the builder or a previous canonicalisation pass.
Fixes: lix-project/lix#507
Change-Id: I7e6ccb71649185764cd5210f4a4794ee174afea6
Remove the mutable state stuff that assumes that one file is being
written a time. It's true that we don't write multiple files
interleaved, but that mutable state is evil.
Change-Id: Ia1481da48255d901e4b09a9b783e7af44fae8cff
When generating shell completions, no logging output should be visible because
it would destroy the shell prompt. Originally this was attempted to be done by
simply disabling the progress bar (ca946860ce),
since the situation is particularly bad there (the screen clearing required for
the rendering ends up erasing the shell prompt). Due to overlooking the
implementation of this hack, it was accidentally undone during a later change
(0dd1d8ca1c).
Since even with the hack correctly in place, it is still possible to mess up
the prompt by logging output (for example warnings for disabled experimental
features, or messages generated by `builtins.trace`), simply send it to the bit
bucket where it belongs. This was already done for bash and zsh
(9d840758a8), and it seems that fish was simply
missed at that time. The last trace of the no-longer-working and obsolete hack
is deleted too.
Fixes: lix-project/lix#513
Change-Id: I59f1ebf90903034e2059298fa8d76bf970bc3315
- Rename the listener to not be called a "sink". If it were a "sink" it
would be eating bytes and conform with any of the Nix sink stuff
(maybe FileHandle should be a Sink itself! but that's a later CL's
problem). This is a parser listener.
- Move the RetrieveRegularNARSink thing into store-api.cc, which is its
only usage, and fix it to actually do what it is stated to do: crash
if its invariants are violated.
It's, of course, used to erm, unpack single-file NAR files, generated
via a horrible contraption of sources and sinks that looks like a
plumbing blueprint. Refactoring that is a future task.
- Add a description of the invariants of NARParseVisitor in preparation
of refactoring it.
Change-Id: Ifca1d74d2947204a1f66349772e54dad0743e944
while gcc 12 and older miscompile our generators, gcc 13 and older
outright crash on kj coroutines. (newer gcc versions may fix this)
Change-Id: I19f12c8c147239680eb0fa5a84ef5c7de38c9263
using a proper event loop basis we no longer have to worry about most of
the intricacies of poll(), or platform-dependent replacements for it. we
may even be able to use the event loop and its promise system for all of
our scheduling in the future. we don't do any real async processing yet,
this is just preparation to separate the first such change from the huge
api design difference with the async framework we chose (kj from capnp):
kj::Promise, unlike std::future, doesn't return exceptions unmangled. it
instead wraps any non-kj exception into a kj exception, erasing all type
information and preserving mostly the what() string in the process. this
makes sense in the capnp rpc use case where unrestricted exception types
can't be transferred, and since it moves error handling styles closer to
a world we'd actually like there's no harm in doing it only here for now
Change-Id: I20f888de74d525fb2db36ca30ebba4bcfe9cc838
we're using boost::outcome rather than leaf or stl types because stl
types are not available everywhere and leaf does not provide its own
storage for error values, relying on thread-locals and the stack. if
we want to use promises we won't have a stack and would have to wrap
everything into leaf-specific allocating wrappers, so outcome it is.
Change-Id: I35111a1f9ed517e7f12a839e2162b1ba6a993f8f
When the multi-line log format is enabled, the progress bar usually occupies
multiple lines on the screen. When stopping the progress bar, only the last
line was wiped, leaving all others visible on the screen. Erase all lines
belonging to the progress bar to prevent these leftovers.
Asking the user for input is theoretically affected by a similar issue, but
this is not observed in practice since the only place where the user is asked
(whether configuration options coming from flakes should be accepted) does not
actually have multiple lines on the progress bar. However, there is no real
reason to not fix this either, so let's do it anyway.
Change-Id: Iaa5a701874fca32e6f06d85912835d86b8fa7a16
The AcceptFlakeConfig type used was missing its JSON serialisation definition,
so it was incorrectly serialised as an integer, ending up that way for example
in the nix.conf manual page. Declare a proper serialisation.
Change-Id: If8ec210f9d4dd42fe480c4e97d0a4920eb66a01e
The JSON serialisation should be declared in the header so that all translation
units can see it when needed, even though it seems that it has not been used
anywhere else so far. Unfortunately, this means we cannot use the
NLOHMANN_JSON_SERIALIZE_ENUM convenience macro, since it uses a slightly
different signature, but the code is not too bad either.
Change-Id: I6e2851b250e0b53114d2fecb8011ff1ea9379d0f
This is the repl overlay from my dotfiles, which I think provides a
reasonable and ergonomic set of variables. We can iterate on this over
time, or (perhaps?) provide a sentinel value like `repl-overlays =
<DEFAULT>` to include a "suggested default" overlay like this one.
Change-Id: I8eba3934c50fbac8367111103e66c7375b8d134e
it just makes sense to have it too, rather than just the pass/fail
information we keep so far. once we turn goals into something more
promise-shaped it'll also help detangle the current data flow mess
Change-Id: I915cf04d177cad849ea7a5833215d795326f1946
it doesn't have a purpose except cache priming, which is largely
irrelevant by default (since another code path already runs this
exact query). our store implementations do not benefit that much
from this either, and the more bursty load may indeed harm them.
Change-Id: I1cc12f8c21cede42524317736d5987f1e43fc9c9
updating statistics *immediately* when any counter changes declutters
things somewhat and makes useful status reports less dependent on the
current worker main loop. using callbacks will make it easier to move
the worker loop into kj entirely, using only promises for scheduling.
Change-Id: I695dfa83111b1ec09b1a54cff268f3c1d7743ed6
there's no reason to go through the event loop in these cases. returning
ContinueImmediately here is just a very convoluted way of jumping to the
state we've just set after unwinding one frame of the stack, which never
matters in the cases changed here because there are no live RAII guards.
Change-Id: I7c00948c22e3caf35e934c1a14ffd2d40efc5547
this is not ideal, but it's better than having this stuck in the worker
loop itself. setting ex on all failing goals is not problematic because
only toplevel goals can ever be observable, all the others are ignored.
notably only derivation goals ever set `ex`, substitution goals do not.
Change-Id: I02e2164487b2955df053fef3c8e774d557aa638a
this doesn't serve a great purpose yet except to confine construction of
goals to the stack frame of Worker::run() and its child frames. we don't
need this yet (and the goal constructors remain fully visible), but in a
future change that fully removes the current worker loop we'll need some
way of knowing which goals are top-level goals without passing the goals
themselves around. once that's possible we can remove visible goals as a
concept and rely on build result futures and a scheduler built upon them
Change-Id: Ia73cdeffcfb9ba1ce9d69b702dc0bc637a4c4ce6
whether goal errors are reported via the `ex` member or just printed to
the log depends on whether the goal is a toplevel goal or a dependency.
if goals are aware of this themselves we can move error printing out of
the worker loop, and since a running worker can only be used by running
goals it's totally sufficient to keep a `Worker::running` flag for this
Change-Id: I6b5cbe6eccee1afa5fde80653c4b968554ddd16f
(but only if it is set to relaxed. no security hole here.)
Thanks to lilyball for pointing out this omission in the docs.
Change-Id: I2408a943bfe817fe660fe1c8fefef898aaf5f7e9
Apparently the fmt contraption has some extremely popular overloads, and
the boost stuff in there gets built approximately infinite times in
every compilation unit.
Change-Id: Ideba2db7d6bf8559e4d91974bab636f5ed106198
Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
also do memory corruption with them with references.
- Remove casts to Expr * where ExprBlackHole is an incomplete type by
introducing an explicitly-cast eBlackHoleAddr as Expr *.
- An incredibly illegal cast of the text bytes of the StorePath hash
into a size_t directly. You can't DO THAT.
Replaced with actually parsing the hash so we get 100% of the bits
being entropy, then memcpying the start of the hash. If this shows
up in a profile we should just make the hash parser faster with a
lookup table or something sensible like that.
- This horrendous bit of UB which I thankfully slapped a deprecation
warning on, built, and it didn't trigger anywhere so it was dead
code and I just deleted it. But holy crap you *cannot* do that.
inline void mkString(const Symbol & s)
{
mkString(((const std::string &) s).c_str());
}
- Some wrong lints. Lots of wrong macro lints, one wrong
suspicious-sizeof lint triggered by the template being instantiated
with only pointers, but the calculation being correct for both
pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
exceptions that might actually happen rather than all the exceptions
imaginable. We can let the runtime hard-kill it on other exceptions
imo.
Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
It's nice for this to be a separate function and not just inline in
`absPath`.
Prepared as part of cl/1865, though I don't think I actually ended up
using it there.
Change-Id: I24d9d4a984cee0af587010baf04b3939a1c147ec
the current test relies on derivation build order being deterministic,
which will not be a reasonable expectation for all that long any more.
Change-Id: I9be44a7725185f614a9a4c724045b8b1e6962c03
this makes WorkResult copyable, and just all around easier to deal with.
in the future we'll need this to let Goal::work() return a promise for a
WorkResult (or even just a Finished) that can be awaited by other goals.
Change-Id: Ic5a1ce04c5a0f8e683bd00a2ed2b77a2e28989c1
this should be done where we're actually trying to build something, not
in the main worker loop that shouldn't have to be aware of such details
Change-Id: I07276740c0e2e5591a8ce4828a4bfc705396527e
This caused an absolute saga which I would not like anyone else to have
to experience. Let's put in a laser targeted error message that
diagnoses this exact problem.
Fixes: lix-project/lix#484
Change-Id: I2a79f04aeb4a1b67c10115e5e39501d958836298
This took parsing time from 1421s or so to 1060s or so. The reason is
entirely nlohmann. All of the stuff below is just Obliterated because it's
built in the PCH instead:
**** Templates that took longest to instantiate:
219051 ms: nlohmann::basic_json<>::parse<const char *> (276 times, avg 793 ms)
169675 ms: nlohmann::basic_json<>::basic_json (1127 times, avg 150 ms)
129416 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (276 times, avg 468 ms)
98155 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (276 times, avg 355 ms)
81322 ms: nlohmann::basic_json<>::json_value::json_value (1405 times, avg 57 ms)
53531 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (276 times, avg 193 ms)
clang-only. This brings the clang build time to not far from *half* of
the gcc build time.
Also, clang does not enjoy so much to miscompile coroutines. Maybe we
should just be clang-only.
Change-Id: Id3135db0094e4560830674090e32e6da2c22fcc6
This avoids C++'s standard library regexes, which aren't the same
across platforms, and have many other issues, like using stack
so much that they stack overflow when processing a lot of data.
To avoid backwards and forward compatibility issues, regexes are
processed using a function converting libstdc++ regexes into Boost
regexes, escaping characters that Boost needs to have escaped, and
rejecting features that Boost has and libstdc++ doesn't.
Related context:
- Original failed attempt to use `boost::regex` in CppNix, failed due to
boost icu dependency being large (disabling ICU is no longer necessary
because linking ICU requires using a different header file,
`boost/regex/icu.hpp`): https://github.com/NixOS/nix/pull/3826
- An attempt to use PCRE, rejected due to providing less backwards
compatibility with `std::regex` than `boost::regex`:
https://github.com/NixOS/nix/pull/7336
- Second attempt to use `boost::regex`, failed due to `}` regex failing
to compile (dealt with by writing a wrapper that parses a regular
expression and escapes `}` characters):
https://github.com/NixOS/nix/pull/7762Closes#34. Closes#476.
Change-Id: Ieb0eb9e270a93e4c7eed412ba4f9f96cb00a5fa4
nixpkgs delivered us the untimely gift of a meson 1.5 upgrade, which
*does* make our lives easier by allowing us to delete wrap generation
code, but it does so at the cost of renaming all rust crates in such a
way that the wrap logic cannot tolerate the new names on the old meson
version 😭.
It also means that support burden for this is going to be atrocious
until we either give in and vendor meson 1.5 or we make a CI target for
it. Neither seems appealing, though the latter is not super absurd for
ensuring we don't break nixpkgs unstable.
This commit causes meson 1.5 to ignore the .wrap files in subprojects/
entirely (since they have the wrong names lol) and instead use
Cargo.lock, so it now hard-depends on our workspace reshuffling
improvement.
It also deletes the hack that we were using to get the sources of Cargo
deps into meson by using a feature that went unnoticed when this code
was originally written: MESON_PACKAGE_CACHE_DIR:
8a202de6ec/mesonbuild/wrap/wrap.py (L490-L502)
Change-Id: I7a28f12fc2812c6ed7537b60bc3025c141a05874
This is purely to let Cargo's dependency resolver do stuff for us, we do
not actually intend to build this stuff with Cargo to begin with.
Change-Id: I4c08d55595c7c27b7096375022581e1e34308a87
There have been multiple setting types for paths that are supposed to be
canonicalised, depending on whether zero or one, one, or any number of paths is
to be specified. Naturally, they behaved in slightly different ways in the
code. Simplify things by unifying them and removing special behaviour (mainly
the "multiple paths type can coerce to boolean" thing).
Change-Id: I7c1ce95e9c8e1829a866fb37d679e167811e9705
Commit 0dd1d8ca1c included an accidental revert
of 1461e6cdda (actually slightly worse), leading
to the progress bar not being stopped properly when a legacy command was
invoked with `--log-format bar` (or similar options that show a progress bar).
Move the progress bar stopping code to its proper place again to fix this
regression.
Change-Id: I676333da096d5990b717a387924bb988c9b73fab
lix-doc is now built with Meson, with lix-doc's dependencies built as
Meson subprojects, either fetched on demand with .wrap files, or fetched
in advance by Nix with importCargoLock. It even builds statically.
Fixes#256.
Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Co-authored-by: Jade Lovelace <lix@jade.fyi>
Change-Id: I3a4731ff13278e7117e0316bc0d7169e85f5eb0c
The <() process substitution syntax doesn't work for this one testcase
in bash for FreeBSD. The exact reason for this is unknown, possibly to
do with pipe vs file vs fifo EOF behavior. The prior behavior was this
test hanging forever, with no children of the bash process.
Change-Id: I71822a4b9dea6059b34300568256c5b7848109ac
Closes#460
I managed to trigger the issue by having the following inputs (shortened):
authentik-nix.url = "github:nix-community/authentik-nix";
authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";
When evaluating this using
nix-eval-jobs --flake .#hydraJobs
I got the following error:
error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode
The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).
The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.
Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
this can be a proper WorkResult now. childTerminated is unfortunately a
lot more stubborn and won't be made private for quite a while yet. once
we can get rid of the Worker poll loop that *should* be possible though
Change-Id: I2218df202da5cb84e852f6a37e4c20367495b617
we'll need this once we want to pass extra information out of accepting
replies, such as fd sets or possibly even async output reader promises.
Change-Id: I5e2f18cdb80b0d2faf3067703cc18bd263329b3f
don't keep fds open we're not using. currently this does not cause any
problems, but it does increase the size of our fd table needlessly and
in the future, when we have proper async processing, having builderOut
open in the daemon once the hook has been fully started is problematic
Change-Id: I6e7fb773b280b042873103638d3e04272ca1e4fc
this is useless to do on the face of it, but it'll make it easier to
convert the entire output handling to use async io and promises soon
Change-Id: I2d1eb62c4bbf8f57bd558b9599c08710a389b1a8
only DerivationGoal can set the hook to anything at all. it always sets
buildOutFD to something that is not related to fromHook in any way, and
mixing the two would have rather dire consequences for log consistency.
Change-Id: Ida86727fd1cd5e1ecd78f07f3bde330a346658a8
This is incredibly haunted, but it can happen that you change libutil,
breaking the generation of the .json files, which then does not rebuild
the files. I don't expect they are slow to build, so it does not seem so
bad to just rebuild them every time instead of extracting a list of all
the possible deps.
We want to delete this nonsense anyway and replace it with generated
code.
Change-Id: Ia576d1a3bdee48fbaefbb5ac194354428d179a84
all derivation goals need a log fd of some description. let's save this
single fd in a dedicated pointer field for all subclasses so that later
we have just the one spot to change if we turn this into async promises
Change-Id: If223adf90909247363fb823d751cae34d25d0c0b
we don't need to expose information about how busy a Worker is if the
worker can instead tell its work items whether they are in a slot. in
the future we might use this to not start items waiting for a slot if
no slots are currently available, but that requires more preparation.
Change-Id: Ibe01ac536da7e6d6f80520164117c43e772f9bd9
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437
Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
The target_machine variable is meant for the target
of cross compilers. We are not a cross compiler, so
instead reuse our host_machine based checks.
Fixes Linux→FreeBSD cross, since Meson can't figure
out `target_machine.kernel()` in that case.
Fixes: lix-project/lix#469
Change-Id: Ia46a64c8d507c3b08987a1de1eda171ff5e50df4
This merge commit returns to the previous state prior to the release but leaves the tag in the branch history.
Release created with releng/create_release.xsh
Change-Id: I8fc975f856631dec7fb3314abd436675adabb59c
For years both the documentation and nixpkgs have said that CppNix is
LGPL-2.1-or-later, not LGPL-2.1-only as is somewhat implied by the
README. We are choosing to update the README to match the rest of the
references.
Related: https://github.com/NixOS/nix/pull/5218
Change-Id: I6a765ae7857a2f84872f80a25983c4c4b2b3b1c1
Seems a little bit Rich that musl does not implement close_range because
they suspect that the system call itself is a bad idea, so they uhhhh
are considering not implementing a wrapper. Let's just fix the problem
at hand by writing our own wrapper.
Change-Id: I1f8e5858e4561d58a5450503d9c4585aded2b216
Turns out strings do not like being resized to -4.
This was discovered while messing with the tests to remove unbuffer and
trying stdbuf instead. Turns out that was not the right approach.
This basically rewrites the handling of this case to be much more
correct, and fixes a bug where with small window sizes where it would
ALSO truncate the attr names in addition to the optional descriptions.
Change-Id: Ifd1beeaffdb47cbb5f4a462b183fcb6c0ff6c524
I was packaging Lix 2.91 for nixpkgs and was annoyed at the expect
dependency. Turns out that you can replace unbuffer with a pretty-short
Python script.
It became less short after I found out that Linux was converting \n to
\r\n in the terminal subsystem, which was not very funny, but is at
least solved by twiddling termios bits.
Change-Id: I8a2700abcbbf6a9902e01b05b40fa9340c0ab90c
This will stop printing stuff to dumb terminals that they don't support.
I've overall audited usage of isatty and replaced the ones with intent
to mean "is a Real terminal" with checking for that. I've also caught a
case of carelessly assuming "is a tty" means "should be colour" in
nix-env.
Change-Id: I6d83725d9a2d932ac94ff2294f92c0a1100d23c9
I noticed there was some stuff setting configureFlags that definitely do
not do anything with meson, so let's rip them out.
As for the empty file, it was added when I was thinking I needed a fake
C++ target to convince meson to create the necessary dependencies. That
was not in fact possible so it should have never been committed.
Change-Id: Ied4723d8a5d21aed85f352c48b080ab2c977a496
We're going for Dragon's Breath because horrors called dibs on it.
It's fine to merge this a little before the final release, since all the
dev versions have -pre in them anyway.
Change-Id: I763acb2fc1bf76030f7feaed983addf6ae2fdd53
this is only used to close non-stdio files in derivation sandboxes. we
may as well encode that in its name, drop the unnecessary integer set,
and use close_range to deal with the actual closing of files. not only
is this clearer, it also makes sandbox setup on linux fast by 1ms each
Change-Id: Id90e259a49c7bc896189e76bfbbf6ef2c0bcd3b2
implementing a build hook is pretty much impossible without either being
a nix, or blindly forwarding the important bits of all build requests to
some kind of nix. we've found no uses of build-hook in the wild, and the
build-hook protocol (apart from being entirely undocumented) is not able
to convey any kind of versioning information between hook and daemon. if
we want to upgrade this infrastructure (which we do), this must not stay
Change-Id: I1ec4976a35adf8105b8ca9240b7984f8b91e147e
this is a bit of a hack, but it's apparently the cleanest way of doing
this in the absence of any kind of priority/provenance information for
values of some given setting. we'll need this to deprecate build-hook.
Change-Id: I03644a9c3f17681c052ecdc610b4f1301266ab9e
sure, linux has been providing argv[0] by default for a while now. other
OSes may not be as forthcoming though, and relying on the OS to create a
world in which we can just make assumptions we could test for instead is
unnecessarily lazy. we *could* default argv0, but that's a little silly.
notably we abort instead of returning normally to avoid confusions where
a caller interprets our exit status like a Worker build results bitmask.
Change-Id: Id73f8cd0a630293b789c59a8c4b0c4a2b936b505
* changes:
sqlite: add a Use::fromStrNullable
util: implement charptr_cast
tree-wide: fix a pile of lints
refactor: make HashType and Base enum classes for type safety
build: integrate clang-tidy into CI
This lets us ensure that nobody is putting in new reinterpret_cast
instances where they could safely use charptr_cast instead.
Change-Id: I6358a3934c8133c7150042635843bdbb6b9218d4
There were several usages of the raw sqlite primitives along with C
style casts, seemingly because nobody thought to use an optional for
getting a string or NULL.
Let's fix this API given we already *have* a wrapper.
Change-Id: I526cceedc2e356209d8fb62e11b3572282c314e8
I don't like having so many reinterpret_cast statements that have to
actually be looked at to determine if they are UB. A huge number of the
reinterpret_cast instances in Lix are actually casting to some pointer
of some character type, which is always valid no matter the source type.
However, it is also worth looking at if it is not casting both *from* a
character type and also *to* a character type, since IMO splatting a
struct into a character array should be a very deliberate action instead
of just being about dealing with bad APIs.
So let's write a template that encapsulates this invariant so we can
not worry about the trivially safe reinterpret_cast invocations.
Change-Id: Ia4e2f1fa0c567123a96604ddadb3bdd7449660a4
This:
- Converts a bunch of C style casts into C++ casts.
- Removes some very silly pointer subtraction code (which is no more or
less busted on i686 than it began)
- Fixes some "technically UB" that never had to be UB in the first
place.
- Makes finally follow the noexcept status of the inner function. Maybe
in the future we should ban the function from not being noexcept, but
that is not today.
- Makes various locally-used exceptions inherit from std::exception.
Change-Id: I22e66972602604989b5e494fd940b93e0e6e9297
This still has utterly unacceptably bad output format design that I
would not inflict on anyone I like, but it *does* now exist, and you
*can* find the errors in the log.
Future work would obviously be to fix that and integrate the actual
errors into Gerrit using codechecker or so.
Followup issue: lix-project/lix#457
Fixes: lix-project/lix#147
Change-Id: Ifca22e443d357762125f4ad6bc4f568af3a26c62
The |> operator is a reverse function operator with low binding strength
to replace lib.pipe. Implements RFC 148, see the RFC text for more
details. Closes#438.
Change-Id: I21df66e8014e0d4dd9753dd038560a2b0b7fd805
This is not a proper fix for the confusion that can happen about how the
tags are supposed to be used.
For a proper fix, we need to do
lix-project/lix#439 and implement
worktrees such that the user never sees the git state anymore.
Change-Id: I7b543967f522cede486e42684b48cad47da95429
This was broken because Nix language's version comparison does not know
how to deal with versions like -rc1 and considers them newer, which is
in this case not desirable.
That in turn led to not tagging 2.90.0 docker images as "latest" since
the heuristic was wrong.
This commit also adds some more cross-checking and failsafes in case the
person running releng does not have a local main branch that is up to
date.
Fixes: lix-project/lix#443
Change-Id: I537103ebab58ae978c00e06972abe14432dd9c80
Currently, the parser relies on the global experimental feature flags.
In order to properly test conditional language features, we instead need
to pass it around in the parser::State.
This means that the parser cannot cache the result of isEnabled anymore,
which wouldn't necessarily hurt performance if the function didn't
perform a linear search on the list of enabled features on every single
call. While we could simply evaluate once at the start of parsing and
cache the result in the parser state, the more sustainable solution
would be to fix `isEnabled` such that all callers may profit from the
performance improvement.
Change-Id: Ic9b9c5d882b6270e1114988b63e6064d36c25cf2
This adds a second form to the `:log` command: it now can accept a
derivation path in addition to a derivation expression. As derivation
store paths start with `/nix/store`, this is not ambiguous.
Resolves: lix-project/lix#51
Change-Id: Iebc7b011537e7012fae8faed4024ea1b8fdc81c3
I definitely don't think we were using this, and it is probably an
omission in the original autoconf deletion more than anything.
Change-Id: Ib7c8082685e550575bca5af06f0e93adf982bd7c
I don't know why but I was getting a spurious -Werror=switch-enum inside
toml11. It does not make sense why it did not occur before, but it
should be stopped.
This was not done at an earlier stage to better match the legacy make
build system, but we don't use it anyway.
Change-Id: I636f8a71e8a0ba5e0feb80b435ae24c3af995c5d
This has been causing various seemingly spurious CI failures as well as
some failures on people running tests on beta builds.
lix> ++(nix-collect-garbage-dry-run.sh:20) nix-store --gc --print-dead
lix> ++(nix-collect-garbage-dry-run.sh:20) wc -l
lix> finding garbage collector roots...
lix> error: Listing pid 87261 file descriptors: Undefined error: 0
There is no real way to write a proper test for this, other than to
start a process like the following:
int main(void) {
for (int i = 0; i < 1000; ++i) {
close(i);
}
sleep(10000);
}
and then let Lix's gc look at it.
I have a relatively high confidence this *will* fix the problem since I
have manually confirmed the behaviour of the libproc call is
as-unexpected, and it would perfectly explain the observed symptom.
Fixes: lix-project/lix#446
Change-Id: I67669b98377af17895644b3bafdf42fc33abd076
* changes:
tree-wide: fix various lint warnings
flake & doxygen: update tagline
nix flake metadata: print modified dates for input flakes
cli: eat terminal codes from stdout also
Implement forcing CLI colour on, and document it better
manual: fix a syntax error in redirects.js that made it not do anything
misc docs/meson tidying
build: implement clang-tidy using our plugin
The growth of the seccomp filter in 127ee1a101
made its compilation time significant (roughly 10 milliseconds have been
measured on one machine). For this reason, it is now precompiled and cached in
the parent process so that this overhead is not hit for every single build. It
is still not optimal when going through the daemon, because compilation still
happens once per client, but it's better than before and doing it only once for
the entire daemon requires excessive crimes with the current architecture.
Fixes: lix-project/lix#461
Change-Id: I2277eaaf6bab9bd74bbbfd9861e52392a54b61a3
This is a preparation for precompiling the filter, which is done separately.
The behaviour should be unchanged for now.
Change-Id: I899aa7242962615949208597aca88913feba1cb8
The seccomp setup code was a huge chunk of conditionally compiled
platform-specific code. For this reason, it is appropriate to move it to the
platform-specific implementation file. Ideally its setup could be moved a bit
to make it happen at the same place as the Darwin restrictions, but that change
is going to be less mechanical.
Change-Id: I496aa3c4fabf34656aba1e32b0089044ab5b99f8
When MANPATH is unset or contains an empty component, a reasonable default is
used. Previously (after 3dced96741), when MANPATH
was unset, the shell hook would only place a location containing the Lix manual
pages there, and system-wide manual pages would become unavailable in the
development shell, which is undesired. Fix the issue by including an empty
component in this case.
Change-Id: Ib3c67a831d709fe2a87520e15917eebb59397bd1
This was always in the lock file and we can simply actually print it.
The test for this is a little bit silly but it should correctly
control for my daring to exercise timezone code *and* locale code in a
test, which I strongly suspect nobody dared do before.
Sample (abridged):
```
Path: /nix/store/gaxb42z68bcr8lch467shvmnhjjzgd8b-source
Last modified: 1970-01-01 00:16:40
Inputs:
├───flake-compat: github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33
│ Last modified: 2023-10-04 13:37:54
├───flake-utils: github:numtide/flake-utils/b1d9ab70662946ef0850d488da1c9019f3a9752a
│ Last modified: 2024-03-11 08:33:50
│ └───systems: github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e
│ Last modified: 2023-04-09 08:27:08
```
Change-Id: I355f82cb4b633974295375ebad646fb6e2107f9b
This *should* be sound, plus or minus the amount that the terminal code
eating code is messed up already.
This is useful for testing CLI output because it will strip the escapes
enough to just shove the expected output in a file.
Change-Id: I8a9b58fafb918466ac76e9ab585fc32fb9294819
The docs page has an incorrect escape that leads to a backslash
appearing in output. Meson stuff is self-explanatory, just shortens and
simplifies a bit.
Change-Id: Ib63adf934efd3caeb82ca82988f230e8858a79f9
The principle of this is that you can either externally build it with
Nix (actual implementation will be in a future commit), or it can be
built with meson if the Nix one is not passed in.
The idea I have is that dev shells don't receive the one from Nix to
avoid having to build it, but CI can use the one from Nix and save some
gratuitous rebuilds.
The design of this is that you can run `ninja -C build clang-tidy` and
it will simply correctly clang-tidy the codebase in spite of PCH
bullshit caused by the cc-wrapper.
This is a truly horrendous number of hacks in a ball, caused by bugs in
several pieces of software, and I am not even getting started.
I don't consider this to fix the clang-tidy issue filing, since we still
have a fair number of issues to fix even on the existing minimal
configuration, and I have not yet implemented it in CI. Realistically we
will need to do something like https://github.com/Ericsson/codechecker
to be able to silence warnings without physically touching the code, or
at least *diff* reports between versions.
Also, the run-clang-tidy output design is rather atrocious and must
not be inflicted upon anyone I have respect for, since it buries the
diagnostics in a pile of invocation logs. We would do really well to
integrate with the Gerrit SARIF stuff so we can dump the reports on
people in a user-friendly manner.
Related: lix-project/lix#147
Change-Id: Ifefe533f3b56874795de231667046b2da6ff2461
Expose an option for disabling the BDW-GC build dependency entirely. Fix the
place where one of its headers was included (unnecessarily) without proper
guarding. Finally, use this machinery to exclude BDW-GC from the ASAN builds
entirely (its usage has already been disabled due to compatibility issues
anyway), to ensure this configuration is not regressed again.
Change-Id: I2ebe8094abf67e7d1e99eed971de3e99d071c10b
this begins a long and arduous journey to remove all result state from
Goal, to eventually drop the std::enable_shared_from_this base, and to
completely eliminate all unsynchronized modification of states of both
Goal and Worker. by the end of this we will hopefully be able to start
and reap multiple derivation builds in parallel, which should speed up
the process quite a bit (at least for short local builds, others might
not notice a large difference. the build hooks will remain a problem.)
Change-Id: I57dcd9b2cab4636ed4aa24cdec67124fef883345
In the SSH code, the logger was conditionally paused, but unconditionally
resumed. This was fine as long as resuming the logger was idempotent. Starting
with 0dd1d8ca1c, it isn't any more, and the
behaviour of the code in question was missed. Consequently, an assertion
failure is triggered for example when performing builds against an "SSH" store
on localhost. Fix the issue by only resuming the logger when it has actually
been paused.
Fixes: lix-project/lix#458
Change-Id: Ib1e4d047744a129f15730b7216f9c9368c2f4211
we still mutate goal state to store the results of any given goal run,
but now we also have that information in Worker and could in theory do
something else with it. we could return a map of goal to goal results,
which would also let us better diagnose failures of subgoals (at all).
Change-Id: I1df956bbd9fa8cc9485fb6df32918d68dda3ff48
this is the first step towards removing all result-related mutation of
Goal state from goal implementations themselves, and into Worker state
instead. once that is done we can treat all non-const Goal fields like
private state of the goal itself, and make threading of goals possible
Change-Id: I69ff7d02a6fd91a65887c6640bfc4f5fb785b45c
once goals run on multiple threads these fields must by synchronized as
one, or we try to run build hooks to often (or worse, not often enough)
Change-Id: I47860e46fe5c6db41755b2a3a1d9dbb5701c4ca4
this will usually be used either directly (which is always fine) or in
Finally blocks (where it must never throw execptions). make sure that,
exceptions being handled or not, the calling wait() in Finally doesn't
cause crashes due to the Finally no-nested-exceptions-thrown assertion
Change-Id: Ib83a5d9483b1fe83b9a957dcefeefce5d088f06d
The original attempt at this introduced a regression; this commit
reverts the revert and fixes the regression.
This reverts commit 3e151d4d77.
Fix to the regression:
flakeref: fix handling of `?dir=` param for flakes in subdirs
As reported in #419[1], accessing a flake in a subdir of a Git
repository fails with the previous commit[2] applied with the error
error: unsupported Git input attribute 'dir'
The problem is that the `dir`-param is inserted into the parsed URL if a
flake is fetched from the subdir of a Git repository. However, for the
fetching part this isn't even needed. The fix is to just pass `subdir`
as second argument to `FlakeRef` (which needs a `basedir` that can be
empty) and leave the parsedURL as-is.
Added a regression test to make sure we don't run into this again.
[1] lix-project/lix#419
[2] e22172aaf6b6a366cecd3c025590e68fa2b91bcc,
originally 3e151d4d77
Change-Id: I2c72d5a32e406a7ca308e271730bd0af01c5d18b
I have added an option to turn off this build input because I'm much
more comfortable when I don't have that type of thing on my computer.
Its default value is true in order to avoid impacting anyone who depends
on AWS features.
Change-Id: Ic57f3c9b9468f422e9fbdcf3ba0fe96177631067
* changes:
remove unused headers in installable-attr-path
libexpr: include the type of the non-derivation value in the type error
libexpr: mild cleanup to getDerivations
libexpr: DrvInfo: remove unused bad-citizen constructor
cleanup and slightly refactor DrvInfo::queryOutputs
I was reminded by various evil things puck did to the evaluator
involving null bytes that you can get funny bytes by abusing JSON
parsing. It's neater than putting binary in the source file, so let's do
it.
Change-Id: I1ff2e0d829eb303fbed81fa2ebb3a39412e89ff1
* changes:
releng: move officialRelease to version.json
Add -Werror CI job
ci: add a asan+ubsan test run on x86_64-linux
tree-wide: add support for asan!
Shuffled the logic around a bit so the shorter code paths are early
returns, added comments, etc.
Should be NFC.
Change-Id: Ie3ddb3d0eddd614d6f8c37bf9a4d5a50282084ea
DrvInfo's constructor that only takes `EvalState` leaves everything else
empty; a DrvInfo which has no iota of information about the derivation
it represents is not useful, and was not used anywhere.
Change-Id: Ic4d93a08cb2748b8cef9a61e41e70404834b23f9
This was causing a few bits of suffering downstream, in particular, in
the NixOS module, which, after this change, can have the
`officialRelease` stuff in *it* completely deleted since we now have
correct defaulting in package.nix for it.
It also eliminates some automated editing of Nix files, which is
certainly always welcome to eliminate.
Fixes: lix-project/lix#406
Change-Id: Id12f3018cff4633e379dbfcbe26b7bc84922bdaf
We should cause CLs that introduce compiler warnings to fail CI. Sadly
this will only cover Clang, but it will cover Clang for free, so it's
truly impossible to say if it's bad or not.
Change-Id: I45ca20d77251af9671d5cbe0d29cb08c5f1d03c2
This should at least catch out blatantly bad patches that don't pass the
test suite with ASan. We don't do this to the integration tests since
they run on relatively limited-memory VMs and so it may not be super
safe to run an evaluator with leak driven garbage collection for them.
Fixes: lix-project/lix#403
Fixes: lix-project/lix#319
Change-Id: I5267b02626866fd33e8b4d8794344531af679f78
What if you could find memory bugs in Lix without really trying very
hard? I've had variously scuffed patches to do this, but this is
blocked on boost coroutines removal at this point tbh.
Change-Id: Id762af076aa06ad51e77a6c17ed10275929ed578
there are no other uses for this yet, but asking for just a subset of
outputs does seem at least somewhat useful to have as a generic thing
Change-Id: I30ff5055a666c351b1b086b8d05b9d7c9fb1c77a
If `:edit`ing a store path, don't reload repl afterwards
to avoid losing local variables: store is immutable,
so "editing" a store path is always just viewing it.
Resolves: lix-project/lix#341
Change-Id: I3747f75ce26e0595e953069c39ddc3ee80699718
limiting CA substitutions was a rather recent addition, and it used a
dedicated counter to not interfere with regular substitutions. though
this works fine it somewhat contradicts the documentation; job limits
should apply to all kinds of substitutions, or be one limit for each.
Change-Id: I1505105b14260ecc1784039b2cc4b7afcf9115c8
all goals do this. it makes no sense to not notify a goal of EOF
conditions because this is the universal signal for "child done"
Change-Id: Ic3980de312547e616739c57c6248a8e81308b5ee
just update progress every time a goal has returned from work(). there
seem to be no performance penalties, and the code is much simpler now.
Change-Id: I288ee568b764ee61f40a498d986afda49987cb50
bindPath/doBind is a useful function in build that is used in several
parts of LocalDerivationGoal. Moving this function makes it easier to
split LocalDerivationGoal implementation between several files.
Change-Id: Ic5a0768479c153c1aa3ed425f12604b20bbf0f42
Unfortunately, io_uring is totally opaque to seccomp, and while currently there
are no dangerous operations implemented, there is no guarantee that it remains
this way. This means that io_uring should be blocked entirely to ensure that
the sandbox is future-proof. This has not been observed to cause issues in
practice.
Change-Id: I45d3895f95abe1bc103a63969f444c334dbbf50d
Previously, system call filtering (to prevent builders from storing files with
setuid/setgid permission bits or extended attributes) was performed using a
blocklist. While this looks simple at first, it actually carries significant
security and maintainability risks: after all, the kernel may add new syscalls
to achieve the same functionality one is trying to block, and it can even be
hard to actually add the syscall to the blocklist when building against a C
library that doesn't know about it yet. For a recent demonstration of this
happening in practice to Nix, see the introduction of fchmodat2 [0] [1].
The allowlist approach does not share the same drawback. While it does require
a rather large list of harmless syscalls to be maintained in the codebase,
failing to update this list (and roll out the update to all users) in time has
rather benign effects; at worst, very recent programs that already rely on new
syscalls will fail with an error the same way they would on a slightly older
kernel that doesn't support them yet. Most importantly, no unintended new ways
of performing dangerous operations will be silently allowed.
Another possible drawback is reduced system call performance due to the larger
filter created by the allowlist requiring more computation [2]. However, this
issue has not convincingly been demonstrated yet in practice, for example in
systemd or various browsers. To the contrary, it has been measured that the the
actual filter constructed here has approximately the same overhead as a very
simple filter blocking only one system call.
This commit tries to keep the behavior as close to unchanged as possible. The
system call list is in line with libseccomp 2.5.5 and glibc 2.39, which are the
latest versions at the point of writing. Since libseccomp 2.5.5 is already a
requirement and the distributions shipping this together with older versions of
glibc are mostly not a thing any more, this should not lead to more build
failures any more.
[0] https://github.com/NixOS/nixpkgs/issues/300635
[1] https://github.com/NixOS/nix/issues/10424
[2] https://github.com/flatpak/flatpak/pull/4462#issuecomment-1061690607
Change-Id: I541be3ea9b249bcceddfed6a5a13ac10b11e16ad
In f047e4357b, I missed the behavior that if
building without a dedicated build user (i.e. in single-user setups), seccomp
setup failures are silently ignored. This was introduced without explanation 7
years ago (ff6becafa8). Hopefully the only
use-case nowadays is causing spurious test suite successes when messing up the
seccomp filter during development. Let's try removing it.
Change-Id: Ibe51416d9c7a6dd635c2282990224861adf1ceab
(cherry picked from commit 8cd1d02f90eb9915e640c5d370d919fad9833c65)
nix flake show: Only print up to the first new line if it exists.
(cherry picked from commit 5281a44927bdb51bfe6e5de12262d815c98f6fe7)
add tests
(cherry picked from commit 74ae0fbdc70a5079a527fe143c4832d1357011f7)
Handle long strings, embedded new lines and empty descriptions
(cherry picked from commit 2ca7b3afdbbd983173a17fa0a822cf7623601367)
Account for total length of 80
(cherry picked from commit 1cc808c18cbaaf26aaae42bb1d7f7223f25dd364)
docs: add nix flake show description release note
fix: remove white space
nix flake show: trim length based on terminal size
test: account for terminal size
docs(flake-description): before and after commands; add myself to credits
Upstream-PR: https://github.com/NixOS/nix/pull/10980
Change-Id: Ie1c667dc816b3dd81e65a1f5395e57ea48ee0362
2024-07-23 13:21:15 +01:00
580 changed files with 14510 additions and 5571 deletions
synopsis: "Build failures caused by `allowSubstitutes = false` while being the wrong system now produce a decent error"
issues: [fj#484]
cls: [1841]
category: Fixes
credits: jade
---
Nix allows derivations to set `allowSubstitutes = false` in order to force them to be built locally without querying substituters for them.
This is useful for derivations that are very fast to build (especially if they produce large output).
However, this can shoot you in the foot if the derivation *has* to be substituted such as if the derivation is for another architecture, which is what `--always-allow-substitutes` is for.
Perhaps such derivations that are known to be impossible to build locally should ignore `allowSubstitutes` (irrespective of remote builders) in the future, but this at least reports the failure and solution directly.
```
$ nix build -f fail.nix
error: a 'unicornsandrainbows-linux' with features {} is required to build '/nix/store/...-meow.drv', but I am a 'x86_64-linux' with features {...}
Hint: the failing derivation has allowSubstitutes set to false, forcing it to be built rather than substituted.
Passing --always-allow-substitutes to force substitution may resolve this failure if the path is available in a substituter.
synopsis: Define integer overflow in the Nix language as an error
issues: [fj#423]
cls: [1594, 1595, 1597, 1609]
category: Fixes
credits: [jade]
---
Previously, integer overflow in the Nix language invoked C++ level signed overflow, which was undefined behaviour, but *probably* manifested as wrapping around on overflow.
Since prior to the public release of Lix, Lix had C++ signed overflow defined to crash the process and nobody noticed this having accidentally removed overflow from the Nix language for three months until it was caught by fiddling around.
Given the significant body of actual Nix code that has been evaluated by Lix in that time, it does not appear that nixpkgs or much of importance depends on integer overflow, so it is safe to turn into an error.
Some other overflows were fixed:
- `builtins.fromJSON` of values greater than the maximum representable value in a signed 64-bit integer will generate an error.
- `nixConfig` in flakes will no longer accept negative values for configuration options.
Integer overflow now looks like the following:
```
» nix eval --expr '9223372036854775807 + 1'
error: integer overflow in adding 9223372036854775807 + 1
synopsis: "Trace which part of a `foo.bar.baz` expression errors"
cls: 1505, 1506
credits: Qyriad
category: Improvements
---
Previously, if an attribute path selection expression like `linux_4_9.meta.description` it wouldn't show you which one of those parts in the attribute path, or even that that line of code is what caused evaluation of the failing expression.
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:17:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
error: linux 4.9 was removed because it will reach its end of life within 22.11
```
Not only does the line of code that referenced the failing attribute show up in the trace, it also tells you that it was specifically the `linux_4_9` part that failed.
This includes if the failing part is a top-level binding:
```
let
inherit (pkgs.linuxKernel.kernels) linux_4_9;
in linux_4_9.meta.description
error:
… while evaluating 'linux_4_9' to select 'meta.description' on it
at «string»:3:4:
2| inherit (pkgs.linuxKernel.kernels) linux_4_9;
3| in linux_4_9.meta.description
| ^
… while evaluating the attribute 'linux_4_9'
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:5:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
… caused by explicit throw
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:17:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
error: linux 4.9 was removed because it will reach its end of life within 22.11
synopsis: "Distinguish between explicit throws and errors that happened while evaluating a throw"
cls: 1511
credits: Qyriad
category: Improvements
---
Previously, errors caused by an expression like `throw "invalid argument"` were treated like an error that happened simply while some builtin function was being called:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg "linuz"
error:
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg "linuz"
error: linuz isn't the right package
```
But the error didn't just happen "while" calling the `throw` builtin — it's a throw error!
Now it looks like this:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg "linuz"
error:
… caused by explicit throw
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg "linuz"
error: linuz isn't the right package
```
This also means that incorrect usage of `throw` or errors evaluating its arguments are easily distinguishable from explicit throws:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg { attrs = "error when coerced in string interpolation"; }
error:
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg { attrs = "error when coerced in string interpolation"; }
… while evaluating a path segment
at «string»:2:24:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg { attrs = "error when coerced in string interpolation"; }
error: cannot coerce a set to a string: { attrs = "error when coerced in string interpolation"; }
```
Here, instead of an actual thrown error, a type error happens first (trying to coerce an attribute set to a string), but that type error happened *while* calling `throw`.
synopsis: "Hash mismatch diagnostics for fixed-output derivations include the URL"
cls: [1536]
credits: [jade]
category: Improvements
---
Now, when building fixed-output derivations, Lix will guess the URL that was used in the derivation using the `url` or `urls` properties in the derivation environment.
This is a layering violation but making these diagnostics tractable when there are multiple instances of the `AAAA` hash is too significant of an improvement to pass it up.
```
error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv':
synopsis: "Eliminate some pretty-printing surprises"
cls: [1616, 1617, 1618]
prs: [11100]
credits: [alois31, roberth]
category: Improvements
---
Some inconsistent and surprising behaviours have been eliminated from the pretty-printing used by the REPL and `nix eval`:
* Lists and attribute sets that contain only a single item without nested structures are no longer sometimes inappropriately indented in the REPL, depending on internal state of the evaluator.
* Empty attribute sets and derivations are no longer shown as `«repeated»`, since they are always cheap to print.
This matches the existing behaviour of `nix-instantiate` on empty attribute sets.
Empty lists were never printed as `«repeated»` already.
* The REPL by default does not print nested attribute sets and lists, and indicates elided items with an ellipsis.
Previously, the ellipsis was printed even when the structure was empty, so that such items do not in fact exist.
Since this behaviour was confusing, it does not happen any more.
Before:
```
nix-repl> :p let x = 1 + 2; in [ [ x ] [ x ] ]
[
[
3
]
[ 3 ]
]
nix-repl> let inherit (import <nixpkgs> { }) hello; in [ hello hello ]
synopsis: "The beginnings of a new pytest-based functional test suite"
category: Development
cls: [2036, 2037]
credits: jade
---
The existing integration/functional test suite is based on a large volume of shell scripts.
This often makes it somewhat challenging to debug at the best of times.
The goal of the pytest test suite is to make tests have more obvious dependencies on files and to make tests more concise and easier to write, as well as making new testing methods like snapshot testing easy.
synopsis: "`nix registry add` now requires a shorthand flakeref on the 'from' side"
cls: 1494
credits: delan
category: Improvements
---
The 'from' argument must now be a shorthand flakeref like `nixpkgs` or `nixpkgs/nixos-20.03`, making it harder to accidentally swap the 'from' and 'to' arguments.
Registry entries that map from other flake URLs can still be specified in registry.json, the `nix.registry` option in NixOS, or the `--override-flake` option in the CLI, but they are not guaranteed to work correctly.
synopsis: "`nix repl` now allows tab-completing the special repl :colon commands"
cls: 1367
credits: Qyriad
category: Improvements
---
The REPL (`nix repl`) supports pressing `<TAB>` to complete a partial expression, but now also supports completing the special :colon commands as well (`:b`, `:edit`, `:doc`, etc), if the line starts with a colon.
synopsis: "Lix now supports building with UndefinedBehaviorSanitizer"
cls: [1483]
credits: [jade]
category: Development
---
You can now build Lix with the configuration option `-Db_sanitize=undefined` and it will both work and pass tests. AddressSanitizer support is also coming soon.
For a list of undefined behaviour fixed by sanitizer usage, see [the gerrit topic "undefined-behaviour"](https://gerrit.lix.systems/q/topic:%22undefined-behaviour%22).
synopsis: "Some Lix crashes now produce reporting instructions and a stack trace, then abort"
cls: [1854]
category: Improvements
credits: jade
---
Lix, being a C++ program, can crash in a few kinds of ways.
It can obviously do a memory access violation, which will generate a core dump and thus be relatively debuggable.
But, worse, it could throw an unhandled exception, and, in the past, we would just show the message but not where it comes from, in spite of this always being a bug, since we expect all such errors to be translated to a Lix specific error.
Now the latter kind of bug should print reporting instructions, a rudimentary stack trace and (depending on system configuration) generate a core dump.
Sample output:
```
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:
Exception: std::runtime_error: test exception
Stack trace:
0# nix::printStackTrace() in /home/jade/lix/lix3/build/src/nix/../libutil/liblixutil.so
1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
4# nix::handleExceptions(std::__cxx11::basic_string<char,std::char_traits<char>, std::allocator<char> > const&, std::function<void()>) in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
synopsis: "`<nix/fetchurl.nix>` now uses TLS verification"
category: Fixes
prs: [11585]
credits: edolstra
---
Previously `<nix/fetchurl.nix>` did not do TLS verification. This was because the Nix sandbox in the past did not have access to TLS certificates, and Nix checks the hash of the fetched file anyway. However, this can expose authentication data from `netrc` and URLs to man-in-the-middle attackers. In addition, Nix now in some cases (such as when using impure derivations) does *not* check the hash. Therefore we have now enabled TLS verification. This means that downloads by `<nix/fetchurl.nix>` will now fail if you're fetching from a HTTPS server that does not have a valid certificate.
`<nix/fetchurl.nix>` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue.
This section describes the notion of *deprecated features*, and how it fits into the big picture of the development of Lix.
# What are deprecated features?
Deprecated features are disabled by default, with the intent to eventually remove them.
Users must explicitly enable them to keep using them, by toggling the associated [deprecated feature flags](@docroot@/command-ref/conf-file.md#conf-deprecated-features).
This allows backwards compatibility and a graceful transition away from undesired features.
# Which features can be deprecated?
Undesired features should be soft-deprecated by yielding a warning when used for a significant amount of time before the can be deprecated.
Legacy obsolete feature with little to no usage may go through this process faster.
Deprecated features should have a migration path to a preferred alternative.
# Lifecycle of a deprecated feature
This description is not normative, but a feature removal may roughly happen like this:
1. Add a warning when the feature is being used.
2. Disable the feature by default, putting it behind a deprecated feature flag.
- If disabling the feature started out as an opt-in experimental feature, turn that experimental flag into a no-op or remove it entirely.
For example, `--extra-experimental-features=no-url-literals` becomes `--extra-deprecated-features=url-literals`.
3. Decide on a time frame for how long that feature will still be supported for backwards compatibility, and clearly communicate that in the error messages.
- Sometimes, automatic migration to alternatives is possible, and such should be provided if possible
- At least one NixOS release cycle should be the minimum
4. Finally remove the feature entirely, only keeping the error message for those still using it.
In an ideal world, we'd have SemVer controls over the language and its features, cleanly allowing us to make breaking changes.
See https://wiki.lix.systems/books/lix-contributors/page/language-versioning and [RFC 137](https://github.com/nixos/rfcs/pull/137) for efforts on that.
However, we do not live in such an ideal world, and currently this goal is so far away, that "just disable it with some back-compat for a couple of years" is the most realistic solution, especially for comparatively minor changes.
@ -36,7 +36,10 @@ All users of the Lix daemon may do the following to bring things into the Nix st
- Input-addressed, so they are run in the sandbox with no network access, with the following exceptions:
- The (poorly named, since it is not *just* about chroot) property `__noChroot` is set on the derivation and `sandbox` is set to `relaxed`.
- On macOS, the derivation property `__darwinAllowLocalNetworking` allows network access to localhost from input-addressed derivations regardless of the `sandbox` setting value. This property exists with such semantics because macOS has no network namespace equivalent to isolate individual processes' localhost networking.
- On macOS, the derivation property `__darwinAllowLocalNetworking` allows network access to localhost from input-addressed derivations regardless of the `sandbox` setting value.
This property exists with such semantics because macOS has no network namespace equivalent to isolate individual processes' localhost networking.
- On macOS, the derivation property `__sandboxProfile` accepts extra sandbox profile S-expressions, allowing derivations to bypass arbitrary parts of the sandbox without altogether disabling it.
This is only permitted when `sandbox` is set to `relaxed`.
- Output-addressed, so they are run with network access but their result must match an expected hash.
Trusted users may set any setting, including `sandbox = false`, so the sandbox state can be different at runtime from what is described in `nix.conf` for builds invoked with such settings.
@ -292,6 +292,12 @@ Derivations can declare some infrequently used optional attributes.
(associative) arrays. For example, the attribute `hardening.format = true`
ends up as the Bash associative array element `${hardening[format]}`.
> **Warning**
>
> If set to `true`, other advanced attributes such as [`allowedReferences`](#adv-attr-allowedReferences), [`allowedReferences`](#adv-attr-allowedReferences), [`allowedRequisites`](#adv-attr-allowedRequisites),
[`disallowedReferences`](#adv-attr-disallowedReferences) and [`disallowedRequisites`](#adv-attr-disallowedRequisites), maxSize, and maxClosureSize.
will have no effect.
- [`outputChecks`]{#adv-attr-outputChecks}\
When using [structured attributes](#adv-attr-structuredAttrs), the `outputChecks`
attribute allows defining checks per-output.
@ -326,7 +332,6 @@ Derivations can declare some infrequently used optional attributes.
- Block io_uring in the Linux sandbox [cl/1611](https://gerrit.lix.systems/c/lix/+/1611)
The io\_uring API has the unfortunate property that it is not possible to selectively decide which operations should be allowed.
This, together with the fact that new operations are routinely added, makes it a hazard to the proper function of the sandbox.
Therefore, any access to io\_uring has been made unavailable inside the sandbox.
As such, attempts to execute any system calls forming part of this API will fail with the error `ENOSYS`, as if io\_uring support had not been configured into the kernel.
Many thanks to [alois31](https://git.lix.systems/alois31) for this.
- The `build-hook` setting is now deprecated
Build hooks communicate with the daemon using a custom, internal, undocumented protocol that is entirely unversioned and cannot be changed.
Since we intend to change it anyway we must unfortunately deprecate the current build hook infrastructure.
We do not expect this to impact most users—we have not found any uses of `build-hook` in the wild—but if this does affect you, we'd like to hear from you!
- Lix no longer speaks the Nix remote-build worker protocol to clients or servers older than CppNix 2.3 [fj#325](https://git.lix.systems/lix-project/lix/issues/325) [cl/1207](https://gerrit.lix.systems/c/lix/+/1207) [cl/1208](https://gerrit.lix.systems/c/lix/+/1208) [cl/1206](https://gerrit.lix.systems/c/lix/+/1206) [cl/1205](https://gerrit.lix.systems/c/lix/+/1205) [cl/1204](https://gerrit.lix.systems/c/lix/+/1204) [cl/1203](https://gerrit.lix.systems/c/lix/+/1203) [cl/1479](https://gerrit.lix.systems/c/lix/+/1479)
CppNix 2.3 was released in 2019, and is the new oldest supported version. We
will increase our support baseline in the future up to a final version of CppNix
2.18 (which may happen soon given that it is the only still-packaged and thus
still-tested >2.3 version), but this step already removes a significant amount
of dead, untested, code paths.
Lix speaks the same version of the protocol as CppNix 2.18 and that fact will
never change in the future; the Lix plans to replace the protocol for evolution
will entail a complete incompatible replacement that will be supported in
parallel with the old protocol. Lix will thus retain remote build compatibility
with CppNix as long as CppNix maintains protocol compatibility with 2.18, and
as long as Lix retains legacy protocol support (which will likely be a long
time given that we plan to convert it to a frozen-in-time shim).
Many thanks to [jade](https://git.lix.systems/jade) for this.
Implementation of the pipe operator (`|>`) in the language as described in [RFC 148](https://github.com/NixOS/rfcs/pull/148).
The feature is still marked experimental, enable `--extra-experimental-features pipe-operator` to use it.
Many thanks to [piegames](https://git.lix.systems/piegames) and [eldritch horrors](https://git.lix.systems/pennae) for this.
## Improvements
- Trace which part of a `foo.bar.baz` expression errors [cl/1505](https://gerrit.lix.systems/c/lix/+/1505) [cl/1506](https://gerrit.lix.systems/c/lix/+/1506)
Previously, if an attribute path selection expression like `linux_4_9.meta.description` it wouldn't show you which one of those parts in the attribute path, or even that that line of code is what caused evaluation of the failing expression.
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:17:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
error: linux 4.9 was removed because it will reach its end of life within 22.11
```
Not only does the line of code that referenced the failing attribute show up in the trace, it also tells you that it was specifically the `linux_4_9` part that failed.
This includes if the failing part is a top-level binding:
```
let
inherit (pkgs.linuxKernel.kernels) linux_4_9;
in linux_4_9.meta.description
error:
… while evaluating 'linux_4_9' to select 'meta.description' on it
at «string»:3:4:
2| inherit (pkgs.linuxKernel.kernels) linux_4_9;
3| in linux_4_9.meta.description
| ^
… while evaluating the attribute 'linux_4_9'
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:5:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
… caused by explicit throw
at /nix/store/dk2rpyb6ndvfbf19bkb2plcz5y3k8i5v-source/pkgs/top-level/linux-kernels.nix:278:17:
277| } // lib.optionalAttrs config.allowAliases {
278| linux_4_9 = throw "linux 4.9 was removed because it will reach its end of life within 22.11";
| ^
279| linux_4_14 = throw "linux 4.14 was removed because it will reach its end of life within 23.11";
error: linux 4.9 was removed because it will reach its end of life within 22.11
```
Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
- Confusing 'invalid path' errors are now 'path does not exist' [cl/1161](https://gerrit.lix.systems/c/lix/+/1161) [cl/1160](https://gerrit.lix.systems/c/lix/+/1160) [cl/1159](https://gerrit.lix.systems/c/lix/+/1159)
Previously, if a path did not exist in a Nix store, it was referred to as the internal name "path is invalid".
This is, however, very confusing, and there were numerous such errors that were exactly the same, making it hard to debug.
These errors are now more specific and refer to the path not existing in the store.
Many thanks to [julia](https://git.lix.systems/midnightveil) for this.
- Add a `build-dir` setting to set the backing directory for builds [gh#10303](https://github.com/NixOS/nix/pull/10303) [gh#10312](https://github.com/NixOS/nix/pull/10312) [gh#10883](https://github.com/NixOS/nix/pull/10883) [cl/1514](https://gerrit.lix.systems/c/lix/+/1514)
`build-dir` can now be set in the Nix configuration to choose the backing directory for the build sandbox.
This can be useful on systems with `/tmp` on tmpfs, or simply to relocate large builds to another disk.
Also, `XDG_RUNTIME_DIR` is no longer considered when selecting the default temporary directory,
as it's not intended to be used for large amounts of data.
Many thanks to [Robert Hensing](https://github.com/roberth) and [Tom Bereknyei](https://github.com/tomberek) for this.
- Better usage of colour control environment variables [cl/1699](https://gerrit.lix.systems/c/lix/+/1699) [cl/1702](https://gerrit.lix.systems/c/lix/+/1702)
Lix now heeds `NO_COLOR`/`NOCOLOR` for more output types, such as that used in `nix search`, `nix flake metadata` and similar.
It also now supports `CLICOLOR_FORCE`/`FORCE_COLOR` to force colours regardless of whether there is a terminal on the other side.
It now follows rules compatible with those described on <https://bixense.com/clicolors/> with `CLICOLOR` defaulted to enabled.
That is to say, the following procedure is followed in order:
- NO_COLOR or NOCOLOR set
Always disable colour
- CLICOLOR_FORCE or FORCE_COLOR set
Enable colour
- The output is a tty; TERM != "dumb"
Enable colour
- Otherwise
Disable colour
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Distinguish between explicit throws and errors that happened while evaluating a throw [cl/1511](https://gerrit.lix.systems/c/lix/+/1511)
Previously, errors caused by an expression like `throw "invalid argument"` were treated like an error that happened simply while some builtin function was being called:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg "linuz"
error:
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg "linuz"
error: linuz isn't the right package
```
But the error didn't just happen "while" calling the `throw` builtin — it's a throw error!
Now it looks like this:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg "linuz"
error:
… caused by explicit throw
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg "linuz"
error: linuz isn't the right package
```
This also means that incorrect usage of `throw` or errors evaluating its arguments are easily distinguishable from explicit throws:
```
let
throwMsg = p: throw "${p} isn't the right package";
in throwMsg { attrs = "error when coerced in string interpolation"; }
error:
… while calling the 'throw' builtin
at «string»:2:17:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg { attrs = "error when coerced in string interpolation"; }
… while evaluating a path segment
at «string»:2:24:
1| let
2| throwMsg = p: throw "${p} isn't the right package";
| ^
3| in throwMsg { attrs = "error when coerced in string interpolation"; }
error: cannot coerce a set to a string: { attrs = "error when coerced in string interpolation"; }
```
Here, instead of an actual thrown error, a type error happens first (trying to coerce an attribute set to a string), but that type error happened *while* calling `throw`.
Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
- `nix flake metadata` prints modified date [cl/1700](https://gerrit.lix.systems/c/lix/+/1700)
Ever wonder "gee, when *did* I update nixpkgs"?
Wonder no more, because `nix flake metadata` now simply tells you the times every locked flake input was updated:
```
<...>
Description: The purely functional package manager
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Hash mismatch diagnostics for fixed-output derivations include the URL [cl/1536](https://gerrit.lix.systems/c/lix/+/1536)
Now, when building fixed-output derivations, Lix will guess the URL that was used in the derivation using the `url` or `urls` properties in the derivation environment.
This is a layering violation but making these diagnostics tractable when there are multiple instances of the `AAAA` hash is too significant of an improvement to pass it up.
```
error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv':
Many thanks to [kjeremy](https://github.com/kjeremy) and [isabelroses](https://git.lix.systems/isabelroses) for this.
- Eliminate some pretty-printing surprises [#11100](https://github.com/NixOS/nix/pull/11100) [cl/1616](https://gerrit.lix.systems/c/lix/+/1616) [cl/1617](https://gerrit.lix.systems/c/lix/+/1617) [cl/1618](https://gerrit.lix.systems/c/lix/+/1618)
Some inconsistent and surprising behaviours have been eliminated from the pretty-printing used by the REPL and `nix eval`:
* Lists and attribute sets that contain only a single item without nested structures are no longer sometimes inappropriately indented in the REPL, depending on internal state of the evaluator.
* Empty attribute sets and derivations are no longer shown as `«repeated»`, since they are always cheap to print.
This matches the existing behaviour of `nix-instantiate` on empty attribute sets.
Empty lists were never printed as `«repeated»` already.
* The REPL by default does not print nested attribute sets and lists, and indicates elided items with an ellipsis.
Previously, the ellipsis was printed even when the structure was empty, so that such items do not in fact exist.
Since this behaviour was confusing, it does not happen any more.
Before:
```
nix-repl> :p let x = 1 + 2; in [ [ x ] [ x ] ]
[
[
3
]
[ 3 ]
]
nix-repl> let inherit (import <nixpkgs> { }) hello; in [ hello hello ]
Many thanks to [alois31](https://git.lix.systems/alois31) and [Robert Hensing](https://github.com/roberth) for this.
- `nix registry add` now requires a shorthand flakeref on the 'from' side [cl/1494](https://gerrit.lix.systems/c/lix/+/1494)
The 'from' argument must now be a shorthand flakeref like `nixpkgs` or `nixpkgs/nixos-20.03`, making it harder to accidentally swap the 'from' and 'to' arguments.
Registry entries that map from other flake URLs can still be specified in registry.json, the `nix.registry` option in NixOS, or the `--override-flake` option in the CLI, but they are not guaranteed to work correctly.
Many thanks to [delan](https://git.lix.systems/delan) for this.
- Allow automatic rejection of configuration options from flakes [cl/1541](https://gerrit.lix.systems/c/lix/+/1541)
Setting `accept-flake-config` to `false` now respects user choice by automatically rejecting configuration options set by flakes.
The old behaviour of asking each time is still available (and default) by setting it to the special value `ask`.
Many thanks to [alois31](https://git.lix.systems/alois31) for this.
- `nix repl` now allows tab-completing the special repl :colon commands [cl/1367](https://gerrit.lix.systems/c/lix/+/1367)
The REPL (`nix repl`) supports pressing `<TAB>` to complete a partial expression, but now also supports completing the special :colon commands as well (`:b`, `:edit`, `:doc`, etc), if the line starts with a colon.
Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
- `:edit`ing a file in Nix store no longer reloads the repl [fj#341](https://git.lix.systems/lix-project/lix/issues/341) [cl/1620](https://gerrit.lix.systems/c/lix/+/1620)
Calling `:edit` from the repl now only reloads if the file being edited was outside of Nix store.
That means that all the local variables are now preserved across `:edit`s of store paths.
This is always safe because the store is read-only.
Many thanks to [goldstein](https://git.lix.systems/goldstein) for this.
- `:log` in repl now works on derivation paths [fj#51](https://git.lix.systems/lix-project/lix/issues/51) [cl/1716](https://gerrit.lix.systems/c/lix/+/1716)
`:log` can now accept store derivation paths in addition to derivation expressions.
Many thanks to [goldstein](https://git.lix.systems/goldstein) for this.
## Fixes
- Define integer overflow in the Nix language as an error [fj#423](https://git.lix.systems/lix-project/lix/issues/423) [cl/1594](https://gerrit.lix.systems/c/lix/+/1594) [cl/1595](https://gerrit.lix.systems/c/lix/+/1595) [cl/1597](https://gerrit.lix.systems/c/lix/+/1597) [cl/1609](https://gerrit.lix.systems/c/lix/+/1609)
Previously, integer overflow in the Nix language invoked C++ level signed overflow, which was undefined behaviour, but *probably* manifested as wrapping around on overflow.
Since prior to the public release of Lix, Lix had C++ signed overflow defined to crash the process and nobody noticed this having accidentally removed overflow from the Nix language for three months until it was caught by fiddling around.
Given the significant body of actual Nix code that has been evaluated by Lix in that time, it does not appear that nixpkgs or much of importance depends on integer overflow, so it is safe to turn into an error.
Some other overflows were fixed:
- `builtins.fromJSON` of values greater than the maximum representable value in a signed 64-bit integer will generate an error.
- `nixConfig` in flakes will no longer accept negative values for configuration options.
Integer overflow now looks like the following:
```
» nix eval --expr '9223372036854775807 + 1'
error: integer overflow in adding 9223372036854775807 + 1
```
Many thanks to [jade](https://git.lix.systems/jade) for this.
`nix-collect-garbage --dry-run` did not previously give any output - it simply
exited without even checking to see what paths would be deleted.
```
$ nix-collect-garbage --dry-run
$
```
We updated the behaviour of the flag such that instead it prints out how many
paths it *would* delete, but doesn't actually delete them.
```
$ nix-collect-garbage --dry-run
finding garbage collector roots...
determining live/dead paths...
...
<nixstorepaths>
...
2670 store paths deleted, 0.00MiB freed
$
```
Many thanks to [Quantum Jump](https://github.com/QuantumBJump) for this.
- Fix unexpectedly-successful GC failures on macOS [fj#446](https://git.lix.systems/lix-project/lix/issues/446) [cl/1723](https://gerrit.lix.systems/c/lix/+/1723)
Has the following happened to you on macOS? This failure has been successfully eliminated, thanks to our successful deployment of advanced successful-failure detection technology (it's just `if (failed && errno == 0)`. Patent pending<sup>not really</sup>):
Many thanks to [jade](https://git.lix.systems/jade) for this.
- `nix copy` is now several times faster at `querying info about /nix/store/...` [fj#366](https://git.lix.systems/lix-project/lix/issues/366) [cl/1462](https://gerrit.lix.systems/c/lix/+/1462)
We fixed a locking bug that serialized `querying info about /nix/store/...`
onto just one thread such that it was eating `O(paths to copy * latency)` time
while setting up to copy paths to s3 and other stores. It is now `nproc` times
faster.
Many thanks to [jade](https://git.lix.systems/jade) for this.
## Development
- clang-tidy support [fj#147](https://git.lix.systems/lix-project/lix/issues/147) [cl/1697](https://gerrit.lix.systems/c/lix/+/1697)
`clang-tidy` can be used to lint Lix with a limited set of lints using `ninja -C build clang-tidy` and `ninja -C build clang-tidy-fix`.
In practice, this fixes the built-in meson rule that was used the same as above being broken ever since precompiled headers were introduced.
Many thanks to [jade](https://git.lix.systems/jade) for this.
- Lix now supports building with UndefinedBehaviorSanitizer [cl/1483](https://gerrit.lix.systems/c/lix/+/1483) [cl/1481](https://gerrit.lix.systems/c/lix/+/1481) [cl/1669](https://gerrit.lix.systems/c/lix/+/1669)
You can now build Lix with the configuration option `-Db_sanitize=undefined,address` and it will both work and pass tests with both AddressSanitizer and UndefinedBehaviorSanitizer enabled.
To use ASan specifically, you have to set `-Dgc=disabled`, which an error message will tell you to do if necessary anyhow.
Furthermore, tests passing with Clang ASan+UBSan is checked on every change in CI.
For a list of undefined behaviour found by tooling usage, see [the gerrit topic "undefined-behaviour"](https://gerrit.lix.systems/q/topic:%22undefined-behaviour%22).
Many thanks to [jade](https://git.lix.systems/jade) for this.
The installer is cross-built to several systems from a Mac using
`build-all.xsh` and `upload-to-lix.xsh` in the installer repo (FIXME: currently
at least; maybe this should be moved here?) .
The installer is cross-built to several systems from a Mac using `build-all.xsh` and `upload-to-lix.xsh` in the installer repo (FIXME: currently at least; maybe this should be moved here?).
It installs a binary tarball (FIXME: [it should be taught to substitute from
cache instead][installer-substitute])
from some URL; this is the `hydraJobs.binaryTarball`. The default URLs differ
by architecture and are [configured here][tarball-urls].
It installs a binary tarball (FIXME: [it should be taught to substitute from cache instead][installer-substitute]) from some URL; this is the `hydraJobs.binaryTarball`.
The default URLs differ by architecture and are [configured here][tarball-urls].
To automatically do the file changes for a new version, run `python3 set_version.py NEW_VERSION`, and submit the result for review.
The website has various release-version dependent pieces.
You can update them with `python3 update_version.py NEW_VERSION`, which will regenerate the affected page sources.
These need the release to have been done first as they need hashes for tarballs and such.
### NixOS module
The NixOS module has underdeveloped releng in it.
Currently you have to do the whole branch-off dance manually to a `release-VERSION` branch and update the tarball URLs to point to the release versions manually.
FIXME: this should be unified with the `set_version.py` work in `lix-installer` and probably all the releng kept in here, or kept elsewhere.
piegames
jade
V.
Maximilian Bosch
Rebecca Turner
Lulu
eldritch horrors
alois31
Olivia Crain
Robert Hensing
raito
Jonas Chevalier
puck
Pierre Bourdon
sugar🍬🍭🏳️⚧️
Audrey Dutcher
Qyriad
Artemis Tosini
Max “Goldstein” Siling
Jeremy List
Isabel