Commit graph

16261 commits

Author SHA1 Message Date
eldritch horrors 531d040e8c libutil: add async collection mechanism
like kj::joinPromisesFailFast this allows waiting for the results of
multiple promises at once, but unlike it not all input promises must
be complete (or any of them failed) for results to become available.

Change-Id: I0e4a37e7bd90651d56b33d0bc5afbadc56cde70c
2024-09-26 16:56:08 +00:00
eldritch horrors ca9256a789 libutil: add an async semaphore implementation
like a normal semaphore, but with awaitable acquire actions. this is
primarily intended as an intermediate concurrency limiting device in
the Worker code, but it may find other uses over time. we do not use
std::counting_semaphore as a base because the counter of that is not
inspectable as will be needed for Worker. we also do not need atomic
operations for cross-thread consistency since we don't have multiple
threads (thanks to kj event loops being confined to a single thread)

Change-Id: Ie2bcb107f3a2c0185138330f7cbba4cec6cbdd95
2024-09-26 16:32:02 +00:00
eldritch horrors 4b66e1e24f fix internal-api-docs build
this one is also run from a gcc stdenv.

Change-Id: I91ff6915c6689ece15224f348f54367cff5d2b5a
2024-09-26 16:11:43 +00:00
Maximilian Bosch 31954b5136 Merge "flake: use clangStdenv for overlays.default" into main 2024-09-26 07:13:41 +00:00
jade 8a6b84df14 Merge "package.nix: fix cross for editline" into main 2024-09-25 20:23:39 +00:00
Maximilian Bosch eccbe9586a
flake: use clangStdenv for overlays.default
We don't support GCC anymore for building, so the overlay currently
fails to evaluate with

    error: assertion '((stdenv).cc.isClang || lintInsteadOfBuild)' failed

`clangStdenv` seems like a reasonable default now.

Noticed while upgrading Lix for our Hydra fork.

Change-Id: I948a7c03b3e5648fc7c596f96e1b8053a9e7f92f
2024-09-25 18:31:34 +02:00
alois31 5f298f74c9 Merge "local-store: make extended attribute handling more robust" into main 2024-09-21 07:55:13 +00:00
jade 79246a3733 Merge "util: fix brotli decompression of empty input" into main 2024-09-18 23:36:25 +00:00
jade 789b19a0cf util: fix brotli decompression of empty input
This caused an infinite loop before since it would just keep asking the
underlying source for more data.

In practice this happened because an HTTP server served a
response to a HEAD request (for which curl will not retrieve any body or
call our write callback function) with Content-Encoding: br, leading to
decompressing nothing at all and going into an infinite loop.

This adds a test to make sure none of our compression methods do that
again, as well as just patching the HTTP client to never feed empty data
into a compression algorithm (since they absolutely have the right to
throw CompressionError on unexpectedly-short streams!).

Reported on Matrix: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$8BWQR_zKxCQDJ40C5NnDo4bQPId3pZ_aoDj2ANP7Itc?via=lix.systems&via=matrix.org&via=tchncs.de

Change-Id: I027566e280f0f569fdb8df40e5ecbf46c211dad1
2024-09-18 15:37:29 -07:00
Rebecca Turner 0943b214c9 Merge "tests/compression: rewrite" into main 2024-09-18 20:38:18 +00:00
alois31 2afdf1ed66
path-info: wipe the progress bar before printing
The legitimate output of `nix path-info` may visually interfere with the
progress bar, by appending to stale progress output before the latter has been
erased. Conveniently, all expensive operations (evaluation or building) have
already been performed before, so we can simply wipe the progress bar at this
point to fix the issue.

Fixes: #343
Change-Id: Id9a807a5c882295b3e6fbf841f9c15dc96f67f6e
2024-09-18 19:26:40 +02:00
jade ed381cd58a package.nix: fix cross for editline
editline's dep on ncurses is a runtime one, so it should be in
buildInputs, not nativeBuildInputs.

CC: #527
Change-Id: I631c192a55677b0cc77faa7511986f1fa2205e91
2024-09-17 20:43:21 -07:00
jade 4046e019ca tests/compression: rewrite
This test suite was in desperate need of using the parameterization
available with gtest, and was a bunch of useless duplicated code. At
least now it's not duplicated code, though it still probably should be
more full of property tests.

Change-Id: Ia8ccee7ef4f02b2fa40417b79aa8c8f0626ea479
2024-09-17 19:07:48 -07:00
Rebecca Turner 8ab5743904 Merge "Remove readline support" into main 2024-09-17 16:36:34 +00:00
Rebecca Turner 7ae0409989
Remove readline support
Lix cannot be built with GNU readline, and we would "rather not" be GPL.

Change-Id: I0e86f0f10dab966ab1d1d467fb61fd2de50c00de
2024-09-16 10:48:20 -07:00
Maximilian Bosch 80202e3ca3
common-eval-args: raise warning if --arg isn't a valid Nix identifier
See #496.

The core idea is to be able to do e.g.

    nix-instantiate -A some-nonfree-thing --arg config.allowUnfree true

which is currently not possible since `config.allowUnfree` is
interpreted as attribute name with a dot in it.

In order to change that (probably), Jade suggested to find out if there
are any folks out there relying on this behavior.

For such a use-case, it may still be possible to accept strings, i.e.
`--arg '"config.allowUnfree"'.

Change-Id: I986c73619fbd87a95b55e2f0ac03feaed3de2d2d
2024-09-15 16:52:30 +02:00
jade 727258241f fix: docs issue template was busted
Apparently forgejo has a more creative interpretation of \(\) than I was
hoping in their markdown parser and thought it was maths. I have no idea
then how you put a link in parens next to another square-bracket link,
but I am not going to worry about it.

There were several more typos, which I also fixed.

Fixes: #517
Change-Id: I6b144c6881f92ca60ba72a304ce7a0bcb9c6659a
2024-09-14 19:28:46 +00:00
jade 5246cea6c8 Merge "store: add a hint on how to fix Lix installs broken by macOS Sequoia" into main 2024-09-14 19:28:24 +00:00
jade 8f88590d13 Merge changes Ia1481da4,Ifca1d74d into main
* changes:
  archive: refactor bad mutable-state API in the NAR parse listener
  archive: rename ParseSink to NARParseVisitor
2024-09-14 19:26:08 +00:00
alois31 3f07c65510
local-store: make extended attribute handling more robust
* Move the extended attribute deletion after the hardlink sanity check. We
  shouldn't be removing extended attributes on random files.
* Make the entity owner-writable before attempting to remove extended
  attributes, since this operation usually requires write access on the file,
  and we shouldn't fail xattr deletion on a file that has been made unwritable
  by the builder or a previous canonicalisation pass.

Fixes: #507
Change-Id: I7e6ccb71649185764cd5210f4a4794ee174afea6
2024-09-14 10:36:22 +02:00
jade b7fc37b015 store: add a hint on how to fix Lix installs broken by macOS Sequoia
This is not a detailed diagnosis, and it's not worth writing one, tbh.
This error basically never happens in normal operation, so diagnosing it
by changing the error on macOS is good enough.

Relevant: lix-project/lix-installer#24
Relevant: lix-project/lix-installer#18
Relevant: #521

Change-Id: I03701f917d116575c72a97502b8e1617679447f2
2024-09-14 07:31:30 +00:00
jade ca1dc3f70b archive: refactor bad mutable-state API in the NAR parse listener
Remove the mutable state stuff that assumes that one file is being
written a time. It's true that we don't write multiple files
interleaved, but that mutable state is evil.

Change-Id: Ia1481da48255d901e4b09a9b783e7af44fae8cff
2024-09-13 17:11:43 -07:00
alois31 b2fc007811 Merge "fish-completion: leave the shell prompt intact" into main 2024-09-12 06:14:22 +00:00
alois31 82aa1ccab4
fish-completion: leave the shell prompt intact
When generating shell completions, no logging output should be visible because
it would destroy the shell prompt. Originally this was attempted to be done by
simply disabling the progress bar (ca946860ce),
since the situation is particularly bad there (the screen clearing required for
the rendering ends up erasing the shell prompt). Due to overlooking the
implementation of this hack, it was accidentally undone during a later change
(0dd1d8ca1c).
Since even with the hack correctly in place, it is still possible to mess up
the prompt by logging output (for example warnings for disabled experimental
features, or messages generated by `builtins.trace`), simply send it to the bit
bucket where it belongs. This was already done for bash and zsh
(9d840758a8), and it seems that fish was simply
missed at that time. The last trace of the no-longer-working and obsolete hack
is deleted too.

Fixes: #513
Change-Id: I59f1ebf90903034e2059298fa8d76bf970bc3315
2024-09-11 19:03:11 +02:00
Rebecca Turner df0137226d
editline: Vendor cl/1883 patch to recognize Alt+Left/Alt+Right
This vendors the patch added in cl/1883 to avoid GitHub
garbage-collecting the commits we're referring to.

As @emilazy pointed out on GitHub:

> GitHub can garbage‐collect unmerged PR commits if they are later
> force‐pushed, which means that code review in upstreams can cause
> Nixpkgs builds to fail to reproduce in future.

See: https://github.com/NixOS/nixpkgs/pull/341131#discussion_r1753046220
See: https://github.com/troglobit/editline/pull/70
See: https://gerrit.lix.systems/c/lix/+/1883

Change-Id: Ifff522f7f23310d6dbe9efc72fd40be5500ae872
2024-09-11 09:35:00 -07:00
jade 81c2e0ac8e archive: rename ParseSink to NARParseVisitor
- Rename the listener to not be called a "sink". If it were a "sink" it
  would be eating bytes and conform with any of the Nix sink stuff
  (maybe FileHandle should be a Sink itself! but that's a later CL's
  problem). This is a parser listener.
- Move the RetrieveRegularNARSink thing into store-api.cc, which is its
  only usage, and fix it to actually do what it is stated to do: crash
  if its invariants are violated.

  It's, of course, used to erm, unpack single-file NAR files, generated
  via a horrible contraption of sources and sinks that looks like a
  plumbing blueprint. Refactoring that is a future task.
- Add a description of the invariants of NARParseVisitor in preparation
  of refactoring it.

Change-Id: Ifca1d74d2947204a1f66349772e54dad0743e944
2024-09-11 01:10:49 -07:00
Rebecca Turner 24db81eaf2 Merge "repl: Patch editline to recognize Meta-Left & Meta-Right" into main 2024-09-11 01:02:27 +00:00
Rebecca Turner cc183fdbc1 Merge "repl-overlays: Provide an elaborate example" into main 2024-09-10 00:17:01 +00:00
Rebecca Turner f5ae72d445 Merge "Add getCwd" into main 2024-09-10 00:10:40 +00:00
Rebecca Turner 6de6cae3e7
repl: Patch editline to recognize Meta-Left & Meta-Right
This applies https://github.com/troglobit/editline/pull/70 to our build
of editline, which translates `meta-left` and `meta-right` into
`fd_word` and `bk_word`. This makes `nix repl` soooo much nicer to use!

Note: My terminal renders `meta-left` as `\e\e[C` and `meta-right` as
`\e\e[D`.

Closes #501

Change-Id: I048b10cf17231bbf4e6bf38e1d1d8572cedaa194
2024-09-09 15:34:50 -07:00
alois31 8f7ab26f96 Merge changes If8ec210f,I6e2851b2 into main
* changes:
  libfetchers: serialise accept-flake-config properly
  libstore: declare SandboxMode JSON serialisation in the header
2024-09-09 16:14:23 +00:00
eldritch horrors c14486ae8d forbid gcc for compilation, only allow clang
while gcc 12 and older miscompile our generators, gcc 13 and older
outright crash on kj coroutines. (newer gcc versions may fix this)

Change-Id: I19f12c8c147239680eb0fa5a84ef5c7de38c9263
2024-09-09 01:48:20 +00:00
alois31 e9505dcc5a Merge "libmain/progress-bar: erase all lines of the multi-line format" into main 2024-09-08 13:42:17 +00:00
eldritch horrors f2a49032a6 libstore: turn Worker in a kj event loop user
using a proper event loop basis we no longer have to worry about most of
the intricacies of poll(), or platform-dependent replacements for it. we
may even be able to use the event loop and its promise system for all of
our scheduling in the future. we don't do any real async processing yet,
this is just preparation to separate the first such change from the huge
api design difference with the async framework we chose (kj from capnp):

kj::Promise, unlike std::future, doesn't return exceptions unmangled. it
instead wraps any non-kj exception into a kj exception, erasing all type
information and preserving mostly the what() string in the process. this
makes sense in the capnp rpc use case where unrestricted exception types
can't be transferred, and since it moves error handling styles closer to
a world we'd actually like there's no harm in doing it only here for now

Change-Id: I20f888de74d525fb2db36ca30ebba4bcfe9cc838
2024-09-08 01:57:48 +00:00
eldritch horrors 92eccfbd68 libutil: add a result type using boost outcome
we're using boost::outcome rather than leaf or stl types because stl
types are not available everywhere and leaf does not provide its own
storage for error values, relying on thread-locals and the stack. if
we want to use promises we won't have a stack and would have to wrap
everything into leaf-specific allocating wrappers, so outcome it is.

Change-Id: I35111a1f9ed517e7f12a839e2162b1ba6a993f8f
2024-09-08 01:57:48 +00:00
alois31 4715d557ef
libmain/progress-bar: erase all lines of the multi-line format
When the multi-line log format is enabled, the progress bar usually occupies
multiple lines on the screen. When stopping the progress bar, only the last
line was wiped, leaving all others visible on the screen. Erase all lines
belonging to the progress bar to prevent these leftovers.
Asking the user for input is theoretically affected by a similar issue, but
this is not observed in practice since the only place where the user is asked
(whether configuration options coming from flakes should be accepted) does not
actually have multiple lines on the progress bar. However, there is no real
reason to not fix this either, so let's do it anyway.

Change-Id: Iaa5a701874fca32e6f06d85912835d86b8fa7a16
2024-09-07 10:37:12 +02:00
alois31 991d8ce275 Merge "Stop the logger in legacy commands again" into main 2024-09-06 17:07:16 +00:00
Rebecca Turner 72589e7032 Merge "Test including relative paths in configuration" into main 2024-09-03 18:48:47 +00:00
Rebecca Turner 644176a631 Merge "Expand comment on std::string operator+" into main 2024-09-03 18:48:37 +00:00
alois31 63ee2cdda3
libfetchers: serialise accept-flake-config properly
The AcceptFlakeConfig type used was missing its JSON serialisation definition,
so it was incorrectly serialised as an integer, ending up that way for example
in the nix.conf manual page. Declare a proper serialisation.

Change-Id: If8ec210f9d4dd42fe480c4e97d0a4920eb66a01e
2024-09-02 18:50:15 +02:00
alois31 d7c37324bb
libstore: declare SandboxMode JSON serialisation in the header
The JSON serialisation should be declared in the header so that all translation
units can see it when needed, even though it seems that it has not been used
anywhere else so far. Unfortunately, this means we cannot use the
NLOHMANN_JSON_SERIALIZE_ENUM convenience macro, since it uses a slightly
different signature, but the code is not too bad either.

Change-Id: I6e2851b250e0b53114d2fecb8011ff1ea9379d0f
2024-09-02 18:50:14 +02:00
Rebecca Turner 75c0de3e3c
Test including relative paths in configuration
Change-Id: If6c69a5e16d1ccd223fba392890f08f0032fb754
2024-09-01 15:52:48 -07:00
Rebecca Turner fc4a160878
repl-overlays: Provide an elaborate example
This is the repl overlay from my dotfiles, which I think provides a
reasonable and ergonomic set of variables. We can iterate on this over
time, or (perhaps?) provide a sentinel value like `repl-overlays =
<DEFAULT>` to include a "suggested default" overlay like this one.

Change-Id: I8eba3934c50fbac8367111103e66c7375b8d134e
2024-09-01 15:30:58 -07:00
Rebecca Turner b7b1b9723f
Clarify that diff-hook no longer needs to be an absolute path
See: https://gerrit.lix.systems/c/lix/+/1864
Change-Id: Ic70bfe42b261a83f2cb68b8f102833b739b8e03a
2024-09-01 15:20:09 -07:00
Rebecca Turner 9d8f433246
Expand comment on std::string operator+
Nuts!

Change-Id: Ib5bc0606d7c86e57ef76dd7bcc89dce91bd3d50a
2024-09-01 15:10:31 -07:00
Rebecca Turner 02eb07cfd5 Merge changes I5566a985,I88cf53d3 into main
* changes:
  Support relative and `~/` paths in config settings
  Thread `ApplyConfigOptions` through config parsing
2024-09-01 22:06:36 +00:00
eldritch horrors d75df91f74 libstore: add build result to Goal::Finished
it just makes sense to have it too, rather than just the pass/fail
information we keep so far. once we turn goals into something more
promise-shaped it'll also help detangle the current data flow mess

Change-Id: I915cf04d177cad849ea7a5833215d795326f1946
2024-08-30 19:01:30 +02:00
eldritch horrors a385c5935a libstore: rename Goal::Finished::result to exitCode
the more useful type for `result` is BuildResult.

Change-Id: If93d9384e8d686eb63b33320f1d565f9b9afbf3a
2024-08-30 19:01:30 +02:00
eldritch horrors dc0cace604 libstore: remove queryMissing call from Worker
it doesn't have a purpose except cache priming, which is largely
irrelevant by default (since another code path already runs this
exact query). our store implementations do not benefit that much
from this either, and the more bursty load may indeed harm them.

Change-Id: I1cc12f8c21cede42524317736d5987f1e43fc9c9
2024-08-30 19:01:30 +02:00
eldritch horrors e0fd0ba211 libstore: use notifications for stats counters
updating statistics *immediately* when any counter changes declutters
things somewhat and makes useful status reports less dependent on the
current worker main loop. using callbacks will make it easier to move
the worker loop into kj entirely, using only promises for scheduling.

Change-Id: I695dfa83111b1ec09b1a54cff268f3c1d7743ed6
2024-08-30 19:01:30 +02:00