Commit graph

16282 commits

Author SHA1 Message Date
eldritch horrors 1a52e4f755 libstore: fix build tests
the new event loop could very occasionally notice that a dependency of
some goal has failed, process the failure, cause the depending goal to
fail accordingly, and in the doing of the latter two steps let further
dependencies that previously have not been reported as failed do their
reporting anyway. in such cases a goal could fail with "1 dependencies
failed", but more than one dependency failure message was shown. we'll
now report the correct number of failed dependency goals in all cases.

Change-Id: I5aa95dcb2db4de4fd5fee8acbf5db833531d81a8
2024-09-29 13:17:15 +00:00
eldritch horrors 3f7519526f libstore: have makeLocalDerivationGoal return unique_ptrs
these can be unique rather than shared because shared_ptr has a
converting constructor. preparatory refactor for something else
and not necessary on its own, and the extra allocations we must
do for shared_ptr control blocks isn't usually relevant anyway.

Change-Id: I5391715545240c6ec8e83a031206edafdfc6462f
2024-09-29 12:09:24 +00:00
Maximilian Bosch 289e7a6b5a Merge "libfetchers/git: restore compat with builtins.fetchGit from 2.3" into main 2024-09-29 08:56:16 +00:00
Olivia Crain f12b60273b Merge changes I5c640824,I09ffc92e,I259583b7 into main
* changes:
  build: require meson 1.4.0 or newer
  build: fix deprecated uses of configure_file
  build: install html manual without using install_subdir
2024-09-28 23:41:30 +00:00
Maximilian Bosch 04daff94e3
libfetchers/git: restore compat with builtins.fetchGit from 2.3
Since fb38459d6e, each `ref` is appended
with `refs/heads` unless it starts with `refs/` already. This regressed
two use-cases that worked fine before:

* Specifying a commit hash as `ref`: now, if `ref` looks like a commit
  hash it will be directly passed to `git fetch`.

* Specifying a tag without `refs/tags` as prefix: now, the fetcher prepends
  `refs/*` to a ref that doesn't start with `refs/` and doesn't look
  like a commit hash. That way, both a branch and a tag specified in
  `ref` can be fetched.

  The order of preference in git is

  * file in `refs/` (e.g. `HEAD`)
  * file in `refs/tags/`
  * file in `refs/heads` (i.e. a branch)

  After fetching `refs/*`, ref is resolved the same way as git does.

Change-Id: Idd49b97cbdc8c6fdc8faa5a48bef3dec25e4ccc3
2024-09-28 14:52:06 +02:00
Olivia Crain b86863d935
build: require meson 1.4.0 or newer
This was already the de facto requirement, we use the method `full_path`
on a file object (introduced in Meson 1.4.0) in the functional test
suite's build.

This version of Meson is in NixOS 24.05, so there should be no
compatibility issues should this make it into a backported release of
Lix.

CC: #247
Change-Id: I5c640824807353b6eb4287e7ed09c4e89a4bdde2
2024-09-27 11:57:53 -05:00
Olivia Crain 624f44bf25
build: fix deprecated uses of configure_file
Using `configure_file` to copy files has been deprecated since Meson 0.64.0.
The intended replacement is the `fs.copyfile` method.

This removes the following deprecation warning that arises when a minimum
Meson version is specified:

``
Project [...] uses feature deprecated since '0.64.0': copy arg in configure_file. Use fs.copyfile instead
``

Change-Id: I09ffc92e96311ef9ed594343a0a16d51e74b114a
2024-09-27 11:55:32 -05:00
Olivia Crain 0e6b3435a1
build: install html manual without using install_subdir
In Meson, `install_subdir` is meant to be used with directories in the source
directory. When using it to install the HTML manual, we provide it with a path
under the build directory.

We should instead specify an install directory for the HTML manual as part of
the custom target that builds it.

What we do currently isn't broken, just semantically incorrect. Changing it does
get rid of the following deprecation warning, though:

``
Project [...] uses feature deprecated since '0.60.0': install_subdir with empty directory. It worked by accident and is buggy. Use install_emptydir instead.
``

Change-Id: I259583b7bdff8ecbb3b342653d70dc5f034c7fad
2024-09-27 11:55:28 -05:00
eldritch horrors ae5d8dae1b libstore: turn Goal::WaitForGoals into a promise
also gets rid of explicit strong references to dependencies of any goal,
and weak references to dependers as well. those are now only held within
promises representing goal completion and thus independent of the goal's
relation to each other. the weak references to dependers was only needed
for notifications, and that's much better handled entirely by kj itself.

Change-Id: I00d06df9090f8d6336ee4bb0c1313a7052fb016b
2024-09-27 16:40:27 +02:00
eldritch horrors 852da07b67 libstore: replace Goal::WaitForSlot with semaphores
now that we have an event loop in the worker we can use it and its
magical execution suspending properties to replace the slot counts
we managed explicitly with semaphores and raii tokens. technically
this would not have needed an event loop base to be doable, but it
is a whole lot easier to wait for a token to be available if there
is a callback mechanism ready for use that doesn't require a whole
damn dedicated abstract method in Goal to work, and specific calls
to that dedicated method strewn all over the worker implementation

Change-Id: I1da7cf386d94e2bbf2dba9b53ff51dbce6a0cff7
2024-09-27 16:40:27 +02:00
eldritch horrors bf32085d63 libstore: simplify Worker::waitForInput
with waitForAWhile turned into promised the core functionality of
waitForInput is now merely to let gc run every so often if needed

Change-Id: I68da342bbc1d67653901cf4502dabfa5bc947628
2024-09-27 16:40:26 +02:00
eldritch horrors cd1ceffb0e libstore: make waiting for a while a promise
this simplifies waitForInput quite a lot, and at the same time makes
polling less thundering-herd-y. it even fixes early polling wakeups!

Change-Id: I6dfa62ce91729b8880342117d71af5ae33366414
2024-09-27 16:39:33 +02:00
eldritch horrors 0478949c72 libstore: turn builder output processing into event loop
this removes the rather janky did-you-mean-async poll loop we had so
far. sadly kj does not play well with pty file descriptors, so we do
have to add our own async input stream that does not eat pty EIO and
turns it into an exception. that's still a *lot* better than the old
code, and using a real even loop makes everything else easier later.

Change-Id: Idd7e0428c59758602cc530bcad224cd2fed4c15e
2024-09-27 16:38:16 +02:00
jade 14dc84ed03 Merge changes Iaa2e0e9d,Ia973420f into main
* changes:
  Fix passing custom CA files into the builtin:fetchurl sandbox
  [security] builtin:fetchurl: Enable TLS verification
2024-09-26 20:53:46 +00:00
eldritch horrors 619a93bd54 Merge "libutil: add async collection mechanism" into main 2024-09-26 17:23:52 +00:00
jade 5dc7671d81 Merge "fmt: fail hard on bad format strings going into nix::fmt too" into main 2024-09-26 17:07:29 +00:00
jade b6038e988d Merge "main: log stack traces for std::terminate" into main 2024-09-26 17:06:01 +00:00
eldritch horrors 531d040e8c libutil: add async collection mechanism
like kj::joinPromisesFailFast this allows waiting for the results of
multiple promises at once, but unlike it not all input promises must
be complete (or any of them failed) for results to become available.

Change-Id: I0e4a37e7bd90651d56b33d0bc5afbadc56cde70c
2024-09-26 16:56:08 +00:00
eldritch horrors ca9256a789 libutil: add an async semaphore implementation
like a normal semaphore, but with awaitable acquire actions. this is
primarily intended as an intermediate concurrency limiting device in
the Worker code, but it may find other uses over time. we do not use
std::counting_semaphore as a base because the counter of that is not
inspectable as will be needed for Worker. we also do not need atomic
operations for cross-thread consistency since we don't have multiple
threads (thanks to kj event loops being confined to a single thread)

Change-Id: Ie2bcb107f3a2c0185138330f7cbba4cec6cbdd95
2024-09-26 16:32:02 +00:00
eldritch horrors 4b66e1e24f fix internal-api-docs build
this one is also run from a gcc stdenv.

Change-Id: I91ff6915c6689ece15224f348f54367cff5d2b5a
2024-09-26 16:11:43 +00:00
puck 37b22dae04 Fix passing custom CA files into the builtin:fetchurl sandbox
Without this, verifying TLS certificates would fail on macOS, as well
as any system that doesn't have a certificate file at /etc/ssl/certs/ca-certificates.crt,
which includes e.g. Fedora.

Change-Id: Iaa2e0e9db3747645b5482c82e3e0e4e8f229f5f9
2024-09-26 15:25:28 +00:00
Maximilian Bosch 31954b5136 Merge "flake: use clangStdenv for overlays.default" into main 2024-09-26 07:13:41 +00:00
Eelco Dolstra c1631b0a39 [security] builtin:fetchurl: Enable TLS verification
This is better for privacy and to avoid leaking netrc credentials in a
MITM attack, but also the assumption that we check the hash no longer
holds in some cases (in particular for impure derivations).

Partially reverts 5db358d4d7.

(cherry picked from commit c04bc17a5a0fdcb725a11ef6541f94730112e7b6)
(cherry picked from commit f2f47fa725fc87bfb536de171a2ea81f2789c9fb)
(cherry picked from commit 7b39cd631e0d3c3d238015c6f450c59bbc9cbc5b)

Upstream-PR: https://github.com/NixOS/nix/pull/11585

Change-Id: Ia973420f6098113da05a594d48394ce1fe41fbb9
2024-09-25 18:40:58 -07:00
jade aca19187d0 fmt: fail hard on bad format strings going into nix::fmt too
Previously we would only crash the program for bad HintFmt calls.
nix::fmt should also crash.

Change-Id: I4ba0abeb8557b208bd9c0be624c022a60446ef7e
2024-09-25 15:20:48 -07:00
jade 19e0ce2c03 main: log stack traces for std::terminate
These stack traces kind of suck for the reasons mentioned on the
CppTrace page here (no symbols for inline functions is a major one):
https://github.com/jeremy-rifkin/cpptrace

I would consider using CppTrace if it were packaged, but to be honest, I
think that the more reasonable option is actually to move entirely to
out-of-process crash handling and symbolization.

The reason for this is that if you want to generate anything of
substance on SIGSEGV or really any deadly signal, you are stuck in
async-signal-safe land, which is not a place to be trying to run a
symbolizer. LLVM does it anyway, probably carefully, and chromium *can*
do it on debug builds but in general uses crashpad:
https://source.chromium.org/chromium/chromium/src/+/main:base/debug/stack_trace_posix.cc;l=974;drc=82dff63dbf9db05e9274e11d9128af7b9f51ceaa;bpv=1;bpt=1

However, some stack traces are better than *no* stack traces when we get
mystery exceptions falling out the bottom of the program. I've also
promoted the path for "mystery exceptions falling out the bottom of the
program" to hard crash and generate a core dump because although there's
been some months since the last one of these, these are nonetheless
always *atrociously* diagnosed.

We can't improve the crash handling further until either we use Crashpad
(which involves more C++ deps, no thanks) or we put in the ostensibly
work in progress Rust minidump infrastructure, in which case we need to
finish full support for Rust in libutil first.

Sample report:

Lix crashed. This is a bug. We would appreciate if you report it at https://git.lix.systems/lix-project/lix/issues with the following information included:

Exception: std::runtime_error: lol
Stack trace:
 0# nix::printStackTrace() in /home/jade/lix/lix3/build/src/nix/../libutil/liblixutil.so
 1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
 2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
 3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
 4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
 5# 0x00005CF65B6B048B in /home/jade/lix/lix3/build/src/nix/nix
 6# 0x000073C985C8810E in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
 7# __libc_start_main in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
 8# 0x00005CF65B610335 in /home/jade/lix/lix3/build/src/nix/nix

Change-Id: I1a9f6d349b617fd7145a37159b78ecb9382cb4e9
2024-09-25 14:03:45 -07:00
jade 8a6b84df14 Merge "package.nix: fix cross for editline" into main 2024-09-25 20:23:39 +00:00
Maximilian Bosch eccbe9586a
flake: use clangStdenv for overlays.default
We don't support GCC anymore for building, so the overlay currently
fails to evaluate with

    error: assertion '((stdenv).cc.isClang || lintInsteadOfBuild)' failed

`clangStdenv` seems like a reasonable default now.

Noticed while upgrading Lix for our Hydra fork.

Change-Id: I948a7c03b3e5648fc7c596f96e1b8053a9e7f92f
2024-09-25 18:31:34 +02:00
alois31 5f298f74c9 Merge "local-store: make extended attribute handling more robust" into main 2024-09-21 07:55:13 +00:00
jade 79246a3733 Merge "util: fix brotli decompression of empty input" into main 2024-09-18 23:36:25 +00:00
jade 789b19a0cf util: fix brotli decompression of empty input
This caused an infinite loop before since it would just keep asking the
underlying source for more data.

In practice this happened because an HTTP server served a
response to a HEAD request (for which curl will not retrieve any body or
call our write callback function) with Content-Encoding: br, leading to
decompressing nothing at all and going into an infinite loop.

This adds a test to make sure none of our compression methods do that
again, as well as just patching the HTTP client to never feed empty data
into a compression algorithm (since they absolutely have the right to
throw CompressionError on unexpectedly-short streams!).

Reported on Matrix: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$8BWQR_zKxCQDJ40C5NnDo4bQPId3pZ_aoDj2ANP7Itc?via=lix.systems&via=matrix.org&via=tchncs.de

Change-Id: I027566e280f0f569fdb8df40e5ecbf46c211dad1
2024-09-18 15:37:29 -07:00
Rebecca Turner 0943b214c9 Merge "tests/compression: rewrite" into main 2024-09-18 20:38:18 +00:00
alois31 2afdf1ed66
path-info: wipe the progress bar before printing
The legitimate output of `nix path-info` may visually interfere with the
progress bar, by appending to stale progress output before the latter has been
erased. Conveniently, all expensive operations (evaluation or building) have
already been performed before, so we can simply wipe the progress bar at this
point to fix the issue.

Fixes: #343
Change-Id: Id9a807a5c882295b3e6fbf841f9c15dc96f67f6e
2024-09-18 19:26:40 +02:00
jade ed381cd58a package.nix: fix cross for editline
editline's dep on ncurses is a runtime one, so it should be in
buildInputs, not nativeBuildInputs.

CC: #527
Change-Id: I631c192a55677b0cc77faa7511986f1fa2205e91
2024-09-17 20:43:21 -07:00
jade 4046e019ca tests/compression: rewrite
This test suite was in desperate need of using the parameterization
available with gtest, and was a bunch of useless duplicated code. At
least now it's not duplicated code, though it still probably should be
more full of property tests.

Change-Id: Ia8ccee7ef4f02b2fa40417b79aa8c8f0626ea479
2024-09-17 19:07:48 -07:00
Rebecca Turner 8ab5743904 Merge "Remove readline support" into main 2024-09-17 16:36:34 +00:00
Rebecca Turner 7ae0409989
Remove readline support
Lix cannot be built with GNU readline, and we would "rather not" be GPL.

Change-Id: I0e86f0f10dab966ab1d1d467fb61fd2de50c00de
2024-09-16 10:48:20 -07:00
Maximilian Bosch 80202e3ca3
common-eval-args: raise warning if --arg isn't a valid Nix identifier
See #496.

The core idea is to be able to do e.g.

    nix-instantiate -A some-nonfree-thing --arg config.allowUnfree true

which is currently not possible since `config.allowUnfree` is
interpreted as attribute name with a dot in it.

In order to change that (probably), Jade suggested to find out if there
are any folks out there relying on this behavior.

For such a use-case, it may still be possible to accept strings, i.e.
`--arg '"config.allowUnfree"'.

Change-Id: I986c73619fbd87a95b55e2f0ac03feaed3de2d2d
2024-09-15 16:52:30 +02:00
jade 727258241f fix: docs issue template was busted
Apparently forgejo has a more creative interpretation of \(\) than I was
hoping in their markdown parser and thought it was maths. I have no idea
then how you put a link in parens next to another square-bracket link,
but I am not going to worry about it.

There were several more typos, which I also fixed.

Fixes: #517
Change-Id: I6b144c6881f92ca60ba72a304ce7a0bcb9c6659a
2024-09-14 19:28:46 +00:00
jade 5246cea6c8 Merge "store: add a hint on how to fix Lix installs broken by macOS Sequoia" into main 2024-09-14 19:28:24 +00:00
jade 8f88590d13 Merge changes Ia1481da4,Ifca1d74d into main
* changes:
  archive: refactor bad mutable-state API in the NAR parse listener
  archive: rename ParseSink to NARParseVisitor
2024-09-14 19:26:08 +00:00
alois31 3f07c65510
local-store: make extended attribute handling more robust
* Move the extended attribute deletion after the hardlink sanity check. We
  shouldn't be removing extended attributes on random files.
* Make the entity owner-writable before attempting to remove extended
  attributes, since this operation usually requires write access on the file,
  and we shouldn't fail xattr deletion on a file that has been made unwritable
  by the builder or a previous canonicalisation pass.

Fixes: #507
Change-Id: I7e6ccb71649185764cd5210f4a4794ee174afea6
2024-09-14 10:36:22 +02:00
jade b7fc37b015 store: add a hint on how to fix Lix installs broken by macOS Sequoia
This is not a detailed diagnosis, and it's not worth writing one, tbh.
This error basically never happens in normal operation, so diagnosing it
by changing the error on macOS is good enough.

Relevant: lix-project/lix-installer#24
Relevant: lix-project/lix-installer#18
Relevant: #521

Change-Id: I03701f917d116575c72a97502b8e1617679447f2
2024-09-14 07:31:30 +00:00
jade ca1dc3f70b archive: refactor bad mutable-state API in the NAR parse listener
Remove the mutable state stuff that assumes that one file is being
written a time. It's true that we don't write multiple files
interleaved, but that mutable state is evil.

Change-Id: Ia1481da48255d901e4b09a9b783e7af44fae8cff
2024-09-13 17:11:43 -07:00
alois31 b2fc007811 Merge "fish-completion: leave the shell prompt intact" into main 2024-09-12 06:14:22 +00:00
alois31 82aa1ccab4
fish-completion: leave the shell prompt intact
When generating shell completions, no logging output should be visible because
it would destroy the shell prompt. Originally this was attempted to be done by
simply disabling the progress bar (ca946860ce),
since the situation is particularly bad there (the screen clearing required for
the rendering ends up erasing the shell prompt). Due to overlooking the
implementation of this hack, it was accidentally undone during a later change
(0dd1d8ca1c).
Since even with the hack correctly in place, it is still possible to mess up
the prompt by logging output (for example warnings for disabled experimental
features, or messages generated by `builtins.trace`), simply send it to the bit
bucket where it belongs. This was already done for bash and zsh
(9d840758a8), and it seems that fish was simply
missed at that time. The last trace of the no-longer-working and obsolete hack
is deleted too.

Fixes: #513
Change-Id: I59f1ebf90903034e2059298fa8d76bf970bc3315
2024-09-11 19:03:11 +02:00
Rebecca Turner df0137226d
editline: Vendor cl/1883 patch to recognize Alt+Left/Alt+Right
This vendors the patch added in cl/1883 to avoid GitHub
garbage-collecting the commits we're referring to.

As @emilazy pointed out on GitHub:

> GitHub can garbage‐collect unmerged PR commits if they are later
> force‐pushed, which means that code review in upstreams can cause
> Nixpkgs builds to fail to reproduce in future.

See: https://github.com/NixOS/nixpkgs/pull/341131#discussion_r1753046220
See: https://github.com/troglobit/editline/pull/70
See: https://gerrit.lix.systems/c/lix/+/1883

Change-Id: Ifff522f7f23310d6dbe9efc72fd40be5500ae872
2024-09-11 09:35:00 -07:00
jade 81c2e0ac8e archive: rename ParseSink to NARParseVisitor
- Rename the listener to not be called a "sink". If it were a "sink" it
  would be eating bytes and conform with any of the Nix sink stuff
  (maybe FileHandle should be a Sink itself! but that's a later CL's
  problem). This is a parser listener.
- Move the RetrieveRegularNARSink thing into store-api.cc, which is its
  only usage, and fix it to actually do what it is stated to do: crash
  if its invariants are violated.

  It's, of course, used to erm, unpack single-file NAR files, generated
  via a horrible contraption of sources and sinks that looks like a
  plumbing blueprint. Refactoring that is a future task.
- Add a description of the invariants of NARParseVisitor in preparation
  of refactoring it.

Change-Id: Ifca1d74d2947204a1f66349772e54dad0743e944
2024-09-11 01:10:49 -07:00
Rebecca Turner 24db81eaf2 Merge "repl: Patch editline to recognize Meta-Left & Meta-Right" into main 2024-09-11 01:02:27 +00:00
Rebecca Turner cc183fdbc1 Merge "repl-overlays: Provide an elaborate example" into main 2024-09-10 00:17:01 +00:00
Rebecca Turner f5ae72d445 Merge "Add getCwd" into main 2024-09-10 00:10:40 +00:00