zebreus/lix
0
0
Fork 0
forked from lix-project/lix
Code Pull requests 1 Activity

Fix tests on systems with a non-master git defaultBranch #1

Open
zebreus wants to merge 140 commits from fix-tests-without-master into main
pull from: fix-tests-without-master
merge into: zebreus:main
Conversation 0 Commits 140 Files changed 216 +4044 -2090
zebreus commented 2024-10-09 20:32:06 +00:00
Owner
Copy link

Summary

When the git default branch is not set to master the installcheck test suite fails. This patch adjusts the tests to use the current default branch instead of master.

Reproduction

Setup

  1. Set init.defaultBranch to main
  2. Run just test --suite=installcheck

Expected result

All tests pass

Actual result

Some tests fail

Alternative fixes

We could also make sure to explicitly name the default branch master when running git init in the tests.

# Summary When the git default branch is not set to master the `installcheck` test suite fails. This patch adjusts the tests to use the current default branch instead of master. ## Reproduction ### Setup 1. Set `init.defaultBranch` to `main` 2. Run `just test --suite=installcheck` ### Expected result All tests pass ### Actual result Some tests fail ## Alternative fixes We could also make sure to explicitly name the default branch `master` when running `git init` in the tests.
zebreus added 140 commits 2024-10-09 20:32:11 +00:00 
Commit 0dd1d8ca1c included an accidental revert
of 1461e6cdda (actually slightly worse), leading
to the progress bar not being stopped properly when a legacy command was
invoked with `--log-format bar` (or similar options that show a progress bar).
Move the progress bar stopping code to its proper place again to fix this
regression.

Change-Id: I676333da096d5990b717a387924bb988c9b73fab
The logic in the comment is the opposite of the truth.

Change-Id: I64add84539209782ffa46431f3db1fb306d90b3f
This makes no changes to logic but makes the `ApplyConfigOptions` value
available to consumers.

Change-Id: I88cf53d38faac8472c556aee55c13d0acbd1e5db
Change-Id: I5566a9858ba255f4ac5051d1368c7dfb24460f0a
It's nice for this to be a separate function and not just inline in
`absPath`.

Prepared as part of cl/1865, though I don't think I actually ended up
using it there.

Change-Id: I24d9d4a984cee0af587010baf04b3939a1c147ec
This fixes an ambiguous pointer comparison warning.

See: https://github.com/rust-analyzer/rowan/pull/162
Change-Id: Iaac2c8cab0051eb97211893ad547d8dfa8fda560
Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
  extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
  also do memory corruption with them with references.
  - Remove casts to Expr * where ExprBlackHole is an incomplete type by
    introducing an explicitly-cast eBlackHoleAddr as Expr *.
  - An incredibly illegal cast of the text bytes of the StorePath hash
    into a size_t directly. You can't DO THAT.

    Replaced with actually parsing the hash so we get 100% of the bits
    being entropy, then memcpying the start of the hash. If this shows
    up in a profile we should just make the hash parser faster with a
    lookup table or something sensible like that.
  - This horrendous bit of UB which I thankfully slapped a deprecation
    warning on, built, and it didn't trigger anywhere so it was dead
    code and I just deleted it. But holy crap you *cannot* do that.

    inline void mkString(const Symbol & s)
    {
        mkString(((const std::string &) s).c_str());
    }
- Some wrong lints. Lots of wrong macro lints, one wrong
  suspicious-sizeof lint triggered by the template being instantiated
  with only pointers, but the calculation being correct for both
  pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
  exceptions that might actually happen rather than all the exceptions
  imaginable. We can let the runtime hard-kill it on other exceptions
  imo.

Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
Ref #359.

Change-Id: Ia45530ddee25fa9fc399ff10738bb0d8bbc8b221
Apparently the fmt contraption has some extremely popular overloads, and
the boost stuff in there gets built approximately infinite times in
every compilation unit.

Change-Id: Ideba2db7d6bf8559e4d91974bab636f5ed106198
This, however, took fixing a pile of lints that we predictably missed
because of this bug.

Change-Id: I92c36feb4a03f62bc594c2051c7bd7418d13fb08
This didn't really feel so worth it afterwards, but I did untangle a
bunch of stuff that should not have been tangled.

The general gist of this change is that variant bullshit was causing a
bunch of compile time, and it seems like the only way to deal with
variant induced compile time is to keep variant types out of headers.
Explicit template instantiation seems to do nothing for them.

I also seem to have gotten some back-end time improvement from
explicitly instantiating regex, but I don't know why. There is no
corresponding front-end time improvement from it: regex is still at the
top of the sinners list.

**** Templates that took longest to instantiate:
 15231 ms: std::basic_regex<char>::_M_compile (28 times, avg 543 ms)
 15066 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 538 ms)
 12571 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 448 ms)
 12454 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 444 ms)
 12225 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 436 ms)
 11363 ms: nlohmann::basic_json<>::parse<const char *> (21 times, avg 541 ms)
 10628 ms: nlohmann::basic_json<>::basic_json (109 times, avg 97 ms)
 10134 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 361 ms)

Back-end time before messing with the regex:
**** Function sets that took longest to compile / optimize:
  8076 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4382 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1247 times, avg 3 ms)
  3137 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 22 ms)
  2896 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 16 ms)
  2304 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (210 times, avg 10 ms)
  2116 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 18 ms)
  2051 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 8 ms)
  2037 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 21 ms)
  1928 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 68 ms)
  1859 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (41 times, avg 45 ms)
  1824 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (973 times, avg 1 ms)
  1810 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 16 ms)
  1793 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1793 ms)
  1759 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (281 times, avg 6 ms)
  1722 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 90 ms)
  1677 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1674 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (249 times, avg 6 ms)
  1660 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (304 times, avg 5 ms)
  1599 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 84 ms)
  1568 ms: void std::__detail::_Compiler<$>::_M_insert_bracket_matcher<$>(bool) (112 times, avg 14 ms)
  1541 ms: std::__shared_ptr<$>::~__shared_ptr() (531 times, avg 2 ms)
  1539 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (41 times, avg 37 ms)
  1471 ms: void std::__detail::_Compiler<$>::_M_insert_character_class_matcher<... (112 times, avg 13 ms)

After messing with the regex (notice std::__detail::_Compiler vanishes
here, but I don't know why):

**** Function sets that took longest to compile / optimize:
  8054 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4313 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1217 times, avg 3 ms)
  3259 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 23 ms)
  3045 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 17 ms)
  2314 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (207 times, avg 11 ms)
  1923 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (216 times, avg 8 ms)
  1817 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 100 ms)
  1816 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms)
  1788 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (40 times, avg 44 ms)
  1749 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (278 times, avg 6 ms)
  1724 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (248 times, avg 6 ms)
  1697 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1684 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1684 ms)
  1680 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (303 times, avg 5 ms)
  1589 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 88 ms)
  1483 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (181 times, avg 8 ms)
  1447 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (40 times, avg 36 ms)
  1441 ms: std::__shared_ptr<$>::~__shared_ptr() (496 times, avg 2 ms)
  1420 ms: boost::stacktrace::basic_stacktrace<$>::init(unsigned long, unsigned... (137 times, avg 10 ms)
  1396 ms: boost::basic_format<$>::~basic_format() (194 times, avg 7 ms)
  1290 ms: std::__cxx11::basic_string<$>::_M_replace_cold(char*, unsigned long,... (231 times, avg 5 ms)
  1258 ms: std::vector<$>::~vector() (354 times, avg 3 ms)
  1222 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (231 times, avg 5 ms)
  1194 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (49 times, avg 24 ms)
  1186 ms: bool tao::pegtl::internal::sor<$>::match<$>(std::integer_sequence<$>... (1 times, avg 1186 ms)
  1149 ms: std::__detail::_Executor<$>::_M_dfs(std::__detail::_Executor<$>::_Ma... (70 times, avg 16 ms)
  1123 ms: toml::detail::sequence<$>::invoke(toml::detail::location&) (69 times, avg 16 ms)
  1110 ms: nlohmann::json_abi_v3_11_3::basic_json<$>::json_value::destroy(nlohm... (55 times, avg 20 ms)
  1079 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (541 times, avg 1 ms)
  1033 ms: nlohmann::json_abi_v3_11_3::detail::lexer<$>::scan_number() (20 times, avg 51 ms)

Change-Id: I10af282bcd4fc39c2d3caae3453e599e4639c70b
Saves about 16s of CPU time. Not a lot but not nothing. Feels more like
the principle of the thing.

Change-Id: I0992d4024317c20d6985a7977d5649edfb9f46bb
(but only if it is set to relaxed. no security hole here.)

Thanks to lilyball for pointing out this omission in the docs.

Change-Id: I2408a943bfe817fe660fe1c8fefef898aaf5f7e9
whether goal errors are reported via the `ex` member or just printed to
the log depends on whether the goal is a toplevel goal or a dependency.
if goals are aware of this themselves we can move error printing out of
the worker loop, and since a running worker can only be used by running
goals it's totally sufficient to keep a `Worker::running` flag for this

Change-Id: I6b5cbe6eccee1afa5fde80653c4b968554ddd16f
this doesn't serve a great purpose yet except to confine construction of
goals to the stack frame of Worker::run() and its child frames. we don't
need this yet (and the goal constructors remain fully visible), but in a
future change that fully removes the current worker loop we'll need some
way of knowing which goals are top-level goals without passing the goals
themselves around. once that's possible we can remove visible goals as a
concept and rely on build result futures and a scheduler built upon them

Change-Id: Ia73cdeffcfb9ba1ce9d69b702dc0bc637a4c4ce6
this is not ideal, but it's better than having this stuck in the worker
loop itself. setting ex on all failing goals is not problematic because
only toplevel goals can ever be observable, all the others are ignored.
notably only derivation goals ever set `ex`, substitution goals do not.

Change-Id: I02e2164487b2955df053fef3c8e774d557aa638a
there's no reason to go through the event loop in these cases. returning
ContinueImmediately here is just a very convoluted way of jumping to the
state we've just set after unwinding one frame of the stack, which never
matters in the cases changed here because there are no live RAII guards.

Change-Id: I7c00948c22e3caf35e934c1a14ffd2d40efc5547
updating statistics *immediately* when any counter changes declutters
things somewhat and makes useful status reports less dependent on the
current worker main loop. using callbacks will make it easier to move
the worker loop into kj entirely, using only promises for scheduling.

Change-Id: I695dfa83111b1ec09b1a54cff268f3c1d7743ed6
it doesn't have a purpose except cache priming, which is largely
irrelevant by default (since another code path already runs this
exact query). our store implementations do not benefit that much
from this either, and the more bursty load may indeed harm them.

Change-Id: I1cc12f8c21cede42524317736d5987f1e43fc9c9
the more useful type for `result` is BuildResult.

Change-Id: If93d9384e8d686eb63b33320f1d565f9b9afbf3a
it just makes sense to have it too, rather than just the pass/fail
information we keep so far. once we turn goals into something more
promise-shaped it'll also help detangle the current data flow mess

Change-Id: I915cf04d177cad849ea7a5833215d795326f1946
* changes:
  Support relative and `~/` paths in config settings
  Thread `ApplyConfigOptions` through config parsing
Nuts!

Change-Id: Ib5bc0606d7c86e57ef76dd7bcc89dce91bd3d50a
See: https://gerrit.lix.systems/c/lix/+/1864
Change-Id: Ic70bfe42b261a83f2cb68b8f102833b739b8e03a
This is the repl overlay from my dotfiles, which I think provides a
reasonable and ergonomic set of variables. We can iterate on this over
time, or (perhaps?) provide a sentinel value like `repl-overlays =
<DEFAULT>` to include a "suggested default" overlay like this one.

Change-Id: I8eba3934c50fbac8367111103e66c7375b8d134e
Change-Id: If6c69a5e16d1ccd223fba392890f08f0032fb754
The JSON serialisation should be declared in the header so that all translation
units can see it when needed, even though it seems that it has not been used
anywhere else so far. Unfortunately, this means we cannot use the
NLOHMANN_JSON_SERIALIZE_ENUM convenience macro, since it uses a slightly
different signature, but the code is not too bad either.

Change-Id: I6e2851b250e0b53114d2fecb8011ff1ea9379d0f
The AcceptFlakeConfig type used was missing its JSON serialisation definition,
so it was incorrectly serialised as an integer, ending up that way for example
in the nix.conf manual page. Declare a proper serialisation.

Change-Id: If8ec210f9d4dd42fe480c4e97d0a4920eb66a01e
When the multi-line log format is enabled, the progress bar usually occupies
multiple lines on the screen. When stopping the progress bar, only the last
line was wiped, leaving all others visible on the screen. Erase all lines
belonging to the progress bar to prevent these leftovers.
Asking the user for input is theoretically affected by a similar issue, but
this is not observed in practice since the only place where the user is asked
(whether configuration options coming from flakes should be accepted) does not
actually have multiple lines on the progress bar. However, there is no real
reason to not fix this either, so let's do it anyway.

Change-Id: Iaa5a701874fca32e6f06d85912835d86b8fa7a16
we're using boost::outcome rather than leaf or stl types because stl
types are not available everywhere and leaf does not provide its own
storage for error values, relying on thread-locals and the stack. if
we want to use promises we won't have a stack and would have to wrap
everything into leaf-specific allocating wrappers, so outcome it is.

Change-Id: I35111a1f9ed517e7f12a839e2162b1ba6a993f8f
using a proper event loop basis we no longer have to worry about most of
the intricacies of poll(), or platform-dependent replacements for it. we
may even be able to use the event loop and its promise system for all of
our scheduling in the future. we don't do any real async processing yet,
this is just preparation to separate the first such change from the huge
api design difference with the async framework we chose (kj from capnp):

kj::Promise, unlike std::future, doesn't return exceptions unmangled. it
instead wraps any non-kj exception into a kj exception, erasing all type
information and preserving mostly the what() string in the process. this
makes sense in the capnp rpc use case where unrestricted exception types
can't be transferred, and since it moves error handling styles closer to
a world we'd actually like there's no harm in doing it only here for now

Change-Id: I20f888de74d525fb2db36ca30ebba4bcfe9cc838
while gcc 12 and older miscompile our generators, gcc 13 and older
outright crash on kj coroutines. (newer gcc versions may fix this)

Change-Id: I19f12c8c147239680eb0fa5a84ef5c7de38c9263
* changes:
  libfetchers: serialise accept-flake-config properly
  libstore: declare SandboxMode JSON serialisation in the header
This applies https://github.com/troglobit/editline/pull/70 to our build
of editline, which translates `meta-left` and `meta-right` into
`fd_word` and `bk_word`. This makes `nix repl` soooo much nicer to use!

Note: My terminal renders `meta-left` as `\e\e[C` and `meta-right` as
`\e\e[D`.

Closes lix-project/lix#501

Change-Id: I048b10cf17231bbf4e6bf38e1d1d8572cedaa194
- Rename the listener to not be called a "sink". If it were a "sink" it
  would be eating bytes and conform with any of the Nix sink stuff
  (maybe FileHandle should be a Sink itself! but that's a later CL's
  problem). This is a parser listener.
- Move the RetrieveRegularNARSink thing into store-api.cc, which is its
  only usage, and fix it to actually do what it is stated to do: crash
  if its invariants are violated.

  It's, of course, used to erm, unpack single-file NAR files, generated
  via a horrible contraption of sources and sinks that looks like a
  plumbing blueprint. Refactoring that is a future task.
- Add a description of the invariants of NARParseVisitor in preparation
  of refactoring it.

Change-Id: Ifca1d74d2947204a1f66349772e54dad0743e944
This vendors the patch added in cl/1883 to avoid GitHub
garbage-collecting the commits we're referring to.

As @emilazy pointed out on GitHub:

> GitHub can garbage‐collect unmerged PR commits if they are later
> force‐pushed, which means that code review in upstreams can cause
> Nixpkgs builds to fail to reproduce in future.

See: https://github.com/NixOS/nixpkgs/pull/341131#discussion_r1753046220
See: https://github.com/troglobit/editline/pull/70
See: https://gerrit.lix.systems/c/lix/+/1883

Change-Id: Ifff522f7f23310d6dbe9efc72fd40be5500ae872
When generating shell completions, no logging output should be visible because
it would destroy the shell prompt. Originally this was attempted to be done by
simply disabling the progress bar (ca946860ce),
since the situation is particularly bad there (the screen clearing required for
the rendering ends up erasing the shell prompt). Due to overlooking the
implementation of this hack, it was accidentally undone during a later change
(0dd1d8ca1c).
Since even with the hack correctly in place, it is still possible to mess up
the prompt by logging output (for example warnings for disabled experimental
features, or messages generated by `builtins.trace`), simply send it to the bit
bucket where it belongs. This was already done for bash and zsh
(9d840758a8), and it seems that fish was simply
missed at that time. The last trace of the no-longer-working and obsolete hack
is deleted too.

Fixes: lix-project/lix#513
Change-Id: I59f1ebf90903034e2059298fa8d76bf970bc3315
Remove the mutable state stuff that assumes that one file is being
written a time. It's true that we don't write multiple files
interleaved, but that mutable state is evil.

Change-Id: Ia1481da48255d901e4b09a9b783e7af44fae8cff
This is not a detailed diagnosis, and it's not worth writing one, tbh.
This error basically never happens in normal operation, so diagnosing it
by changing the error on macOS is good enough.

Relevant: lix-project/lix-installer#24
Relevant: lix-project/lix-installer#18
Relevant: lix-project/lix#521

Change-Id: I03701f917d116575c72a97502b8e1617679447f2
* Move the extended attribute deletion after the hardlink sanity check. We
  shouldn't be removing extended attributes on random files.
* Make the entity owner-writable before attempting to remove extended
  attributes, since this operation usually requires write access on the file,
  and we shouldn't fail xattr deletion on a file that has been made unwritable
  by the builder or a previous canonicalisation pass.

Fixes: lix-project/lix#507
Change-Id: I7e6ccb71649185764cd5210f4a4794ee174afea6
* changes:
  archive: refactor bad mutable-state API in the NAR parse listener
  archive: rename ParseSink to NARParseVisitor
Apparently forgejo has a more creative interpretation of \(\) than I was
hoping in their markdown parser and thought it was maths. I have no idea
then how you put a link in parens next to another square-bracket link,
but I am not going to worry about it.

There were several more typos, which I also fixed.

Fixes: lix-project/lix#517
Change-Id: I6b144c6881f92ca60ba72a304ce7a0bcb9c6659a
See lix-project/lix#496.

The core idea is to be able to do e.g.

    nix-instantiate -A some-nonfree-thing --arg config.allowUnfree true

which is currently not possible since `config.allowUnfree` is
interpreted as attribute name with a dot in it.

In order to change that (probably), Jade suggested to find out if there
are any folks out there relying on this behavior.

For such a use-case, it may still be possible to accept strings, i.e.
`--arg '"config.allowUnfree"'.

Change-Id: I986c73619fbd87a95b55e2f0ac03feaed3de2d2d
Lix cannot be built with GNU readline, and we would "rather not" be GPL.

Change-Id: I0e86f0f10dab966ab1d1d467fb61fd2de50c00de
This test suite was in desperate need of using the parameterization
available with gtest, and was a bunch of useless duplicated code. At
least now it's not duplicated code, though it still probably should be
more full of property tests.

Change-Id: Ia8ccee7ef4f02b2fa40417b79aa8c8f0626ea479
editline's dep on ncurses is a runtime one, so it should be in
buildInputs, not nativeBuildInputs.

CC: lix-project/lix#527
Change-Id: I631c192a55677b0cc77faa7511986f1fa2205e91
The legitimate output of `nix path-info` may visually interfere with the
progress bar, by appending to stale progress output before the latter has been
erased. Conveniently, all expensive operations (evaluation or building) have
already been performed before, so we can simply wipe the progress bar at this
point to fix the issue.

Fixes: lix-project/lix#343
Change-Id: Id9a807a5c882295b3e6fbf841f9c15dc96f67f6e
This caused an infinite loop before since it would just keep asking the
underlying source for more data.

In practice this happened because an HTTP server served a
response to a HEAD request (for which curl will not retrieve any body or
call our write callback function) with Content-Encoding: br, leading to
decompressing nothing at all and going into an infinite loop.

This adds a test to make sure none of our compression methods do that
again, as well as just patching the HTTP client to never feed empty data
into a compression algorithm (since they absolutely have the right to
throw CompressionError on unexpectedly-short streams!).

Reported on Matrix: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$8BWQR_zKxCQDJ40C5NnDo4bQPId3pZ_aoDj2ANP7Itc?via=lix.systems&via=matrix.org&via=tchncs.de

Change-Id: I027566e280f0f569fdb8df40e5ecbf46c211dad1
Previously, Doxygen needed to be ran from the project's source root dir
due to the relative paths in the config's `INPUT` tag. We now preprocess
the relative paths by prefixing them with the absolute path of the
project's source root dir. The HTML output remains unchanged.

Fixes: lix-project/lix#240
Change-Id: I85f099c22bfc5fdbf26be27c2db7dcbc8155c8b2
We don't support GCC anymore for building, so the overlay currently
fails to evaluate with

    error: assertion '((stdenv).cc.isClang || lintInsteadOfBuild)' failed

`clangStdenv` seems like a reasonable default now.

Noticed while upgrading Lix for our Hydra fork.

Change-Id: I948a7c03b3e5648fc7c596f96e1b8053a9e7f92f
These stack traces kind of suck for the reasons mentioned on the
CppTrace page here (no symbols for inline functions is a major one):
https://github.com/jeremy-rifkin/cpptrace

I would consider using CppTrace if it were packaged, but to be honest, I
think that the more reasonable option is actually to move entirely to
out-of-process crash handling and symbolization.

The reason for this is that if you want to generate anything of
substance on SIGSEGV or really any deadly signal, you are stuck in
async-signal-safe land, which is not a place to be trying to run a
symbolizer. LLVM does it anyway, probably carefully, and chromium *can*
do it on debug builds but in general uses crashpad:
https://source.chromium.org/chromium/chromium/src/+/main:base/debug/stack_trace_posix.cc;l=974;drc=82dff63dbf9db05e9274e11d9128af7b9f51ceaa;bpv=1;bpt=1

However, some stack traces are better than *no* stack traces when we get
mystery exceptions falling out the bottom of the program. I've also
promoted the path for "mystery exceptions falling out the bottom of the
program" to hard crash and generate a core dump because although there's
been some months since the last one of these, these are nonetheless
always *atrociously* diagnosed.

We can't improve the crash handling further until either we use Crashpad
(which involves more C++ deps, no thanks) or we put in the ostensibly
work in progress Rust minidump infrastructure, in which case we need to
finish full support for Rust in libutil first.

Sample report:

Lix crashed. This is a bug. We would appreciate if you report it at https://git.lix.systems/lix-project/lix/issues with the following information included:

Exception: std::runtime_error: lol
Stack trace:
 0# nix::printStackTrace() in /home/jade/lix/lix3/build/src/nix/../libutil/liblixutil.so
 1# 0x000073C9862331F2 in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
 2# 0x000073C985F2E21A in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
 3# 0x000073C985F2E285 in /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
 4# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/jade/lix/lix3/build/src/nix/../libmain/liblixmain.so
 5# 0x00005CF65B6B048B in /home/jade/lix/lix3/build/src/nix/nix
 6# 0x000073C985C8810E in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
 7# __libc_start_main in /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
 8# 0x00005CF65B610335 in /home/jade/lix/lix3/build/src/nix/nix

Change-Id: I1a9f6d349b617fd7145a37159b78ecb9382cb4e9
Previously we would only crash the program for bad HintFmt calls.
nix::fmt should also crash.

Change-Id: I4ba0abeb8557b208bd9c0be624c022a60446ef7e
This is better for privacy and to avoid leaking netrc credentials in a
MITM attack, but also the assumption that we check the hash no longer
holds in some cases (in particular for impure derivations).

Partially reverts 5db358d4d7.

(cherry picked from commit c04bc17a5a0fdcb725a11ef6541f94730112e7b6)
(cherry picked from commit f2f47fa725fc87bfb536de171a2ea81f2789c9fb)
(cherry picked from commit 7b39cd631e0d3c3d238015c6f450c59bbc9cbc5b)

Upstream-PR: https://github.com/NixOS/nix/pull/11585

Change-Id: Ia973420f6098113da05a594d48394ce1fe41fbb9
Without this, verifying TLS certificates would fail on macOS, as well
as any system that doesn't have a certificate file at /etc/ssl/certs/ca-certificates.crt,
which includes e.g. Fedora.

Change-Id: Iaa2e0e9db3747645b5482c82e3e0e4e8f229f5f9
this one is also run from a gcc stdenv.

Change-Id: I91ff6915c6689ece15224f348f54367cff5d2b5a
like a normal semaphore, but with awaitable acquire actions. this is
primarily intended as an intermediate concurrency limiting device in
the Worker code, but it may find other uses over time. we do not use
std::counting_semaphore as a base because the counter of that is not
inspectable as will be needed for Worker. we also do not need atomic
operations for cross-thread consistency since we don't have multiple
threads (thanks to kj event loops being confined to a single thread)

Change-Id: Ie2bcb107f3a2c0185138330f7cbba4cec6cbdd95
like kj::joinPromisesFailFast this allows waiting for the results of
multiple promises at once, but unlike it not all input promises must
be complete (or any of them failed) for results to become available.

Change-Id: I0e4a37e7bd90651d56b33d0bc5afbadc56cde70c
* changes:
  Fix passing custom CA files into the builtin:fetchurl sandbox
  [security] builtin:fetchurl: Enable TLS verification
When `nix fmt` is called without an argument, Nix appends the "." argument before calling the formatter. The comment in the code is:
> Format the current flake out of the box

This also happens when formatting sub-folders.

This means that the formatter is now unable to distinguish, as an interface, whether the "." argument is coming from the flake or the user's intent to format the current folder. This decision should be up to the formatter.

Treefmt, for example, will automatically look up the project's root and format all the files. This is the desired behaviour. But because the "." argument is passed, it cannot function as expected.

Upstream-PR: https://github.com/nixos/nix/pull/11438

Change-Id: I60fb6b3ed4ec1b24f81b5f0d76c0be98470817ce
this removes the rather janky did-you-mean-async poll loop we had so
far. sadly kj does not play well with pty file descriptors, so we do
have to add our own async input stream that does not eat pty EIO and
turns it into an exception. that's still a *lot* better than the old
code, and using a real even loop makes everything else easier later.

Change-Id: Idd7e0428c59758602cc530bcad224cd2fed4c15e
this simplifies waitForInput quite a lot, and at the same time makes
polling less thundering-herd-y. it even fixes early polling wakeups!

Change-Id: I6dfa62ce91729b8880342117d71af5ae33366414
with waitForAWhile turned into promised the core functionality of
waitForInput is now merely to let gc run every so often if needed

Change-Id: I68da342bbc1d67653901cf4502dabfa5bc947628
now that we have an event loop in the worker we can use it and its
magical execution suspending properties to replace the slot counts
we managed explicitly with semaphores and raii tokens. technically
this would not have needed an event loop base to be doable, but it
is a whole lot easier to wait for a token to be available if there
is a callback mechanism ready for use that doesn't require a whole
damn dedicated abstract method in Goal to work, and specific calls
to that dedicated method strewn all over the worker implementation

Change-Id: I1da7cf386d94e2bbf2dba9b53ff51dbce6a0cff7
also gets rid of explicit strong references to dependencies of any goal,
and weak references to dependers as well. those are now only held within
promises representing goal completion and thus independent of the goal's
relation to each other. the weak references to dependers was only needed
for notifications, and that's much better handled entirely by kj itself.

Change-Id: I00d06df9090f8d6336ee4bb0c1313a7052fb016b
In Meson, `install_subdir` is meant to be used with directories in the source
directory. When using it to install the HTML manual, we provide it with a path
under the build directory.

We should instead specify an install directory for the HTML manual as part of
the custom target that builds it.

What we do currently isn't broken, just semantically incorrect. Changing it does
get rid of the following deprecation warning, though:

``
Project [...] uses feature deprecated since '0.60.0': install_subdir with empty directory. It worked by accident and is buggy. Use install_emptydir instead.
``

Change-Id: I259583b7bdff8ecbb3b342653d70dc5f034c7fad
Using `configure_file` to copy files has been deprecated since Meson 0.64.0.
The intended replacement is the `fs.copyfile` method.

This removes the following deprecation warning that arises when a minimum
Meson version is specified:

``
Project [...] uses feature deprecated since '0.64.0': copy arg in configure_file. Use fs.copyfile instead
``

Change-Id: I09ffc92e96311ef9ed594343a0a16d51e74b114a
This was already the de facto requirement, we use the method `full_path`
on a file object (introduced in Meson 1.4.0) in the functional test
suite's build.

This version of Meson is in NixOS 24.05, so there should be no
compatibility issues should this make it into a backported release of
Lix.

CC: lix-project/lix#247
Change-Id: I5c640824807353b6eb4287e7ed09c4e89a4bdde2
We have manually enabled libstdc++ assertions since cl/797. Meson 1.4.0
(the minimum version we mandate) enables this by default, so we can
remove the enabling compiler flag from the list of project arguments.

With this patch, `-D_GLIBCXX_ASSERTIONS=1` is still present in the
compile command logs when building with both gccStdenv and clangStdenv.

See: https://gerrit.lix.systems/c/lix/+/797
See: https://mesonbuild.com/Release-notes-for-1-4-0.html#ndebug-setting-now-controls-c-stdlib-assertions
Change-Id: I53483fadfe5cbd11ba35544b437d3a9ee8031631
Since fb38459d6e, each `ref` is appended
with `refs/heads` unless it starts with `refs/` already. This regressed
two use-cases that worked fine before:

* Specifying a commit hash as `ref`: now, if `ref` looks like a commit
  hash it will be directly passed to `git fetch`.

* Specifying a tag without `refs/tags` as prefix: now, the fetcher prepends
  `refs/*` to a ref that doesn't start with `refs/` and doesn't look
  like a commit hash. That way, both a branch and a tag specified in
  `ref` can be fetched.

  The order of preference in git is

  * file in `refs/` (e.g. `HEAD`)
  * file in `refs/tags/`
  * file in `refs/heads` (i.e. a branch)

  After fetching `refs/*`, ref is resolved the same way as git does.

Change-Id: Idd49b97cbdc8c6fdc8faa5a48bef3dec25e4ccc3
* changes:
  build: require meson 1.4.0 or newer
  build: fix deprecated uses of configure_file
  build: install html manual without using install_subdir
these can be unique rather than shared because shared_ptr has a
converting constructor. preparatory refactor for something else
and not necessary on its own, and the extra allocations we must
do for shared_ptr control blocks isn't usually relevant anyway.

Change-Id: I5391715545240c6ec8e83a031206edafdfc6462f
the new event loop could very occasionally notice that a dependency of
some goal has failed, process the failure, cause the depending goal to
fail accordingly, and in the doing of the latter two steps let further
dependencies that previously have not been reported as failed do their
reporting anyway. in such cases a goal could fail with "1 dependencies
failed", but more than one dependency failure message was shown. we'll
now report the correct number of failed dependency goals in all cases.

Change-Id: I5aa95dcb2db4de4fd5fee8acbf5db833531d81a8
have DerivationGoal and its subclasses produce a wrapper promise for
their intermediate results instead, and return this wrapper promise.
Worker already handles promises that do not complete immediately, so
we do not have to duplicate this into an entire result type variant.

Change-Id: Iae8dbf63cfc742afda4d415922a29ac5a3f39348
without circular references we do not need weak goal pointers except for
caches, which should not prevent goal destructors running. caches though
cannot create circular references even when they keep strong references.
if we removed goals from caches when their work() is fully finished, not
when their destructors are run, we could keep strong pointers in caches.
since we do not gain much from this we keep those pointers weak for now.

Change-Id: I1d4a6850ff5e264443c90eb4531da89f5e97a3a0
this was a debugging aid from day one that should not have any impact on
build semantics, and if it *does* have an impact on build semantics then
build semantics are seriously broken. keeping the order imposed by these
keys will be impossible once we let a real event loop schedule our jobs.

Change-Id: I5c313324e1f213ab6453d82f41ae5e59de809a5b
makeDerivationGoalCommon had the right idea, but it didn't quite go far
enough. let's do the rest and remove the remaining factory duplication.

Change-Id: I1fe32446bdfb501e81df56226fd962f85720725b
this was immensely inefficient on large caches, as can exist when many
derivations are buildable simultaneously. since we have smart pointers
to goals we can do cache maintenance in goal deleters instead, and use
the exact iterators instead of doing a linear search. this *does* rely
on goals being deleted to remove them from the cache, which isn't true
for toplevel goals. those would have previously been removed when done
in all cases, removing the cache entry when keep-going is set. this is
arguably incorrect since it might result in those goals being retried,
although that could only happen with dynamic derivations or the likes.
(luckily dynamic derivations not complete enough to allow this at all)

Change-Id: I8e750b868393588c33e4829333d370f2c509ce99
nothing needs to signal being still active but not actively pollable,
only that immediate polling for the next goal work phase is in order.

Change-Id: Ia43c1015e94ba4f5f6b9cb92943da608c4a01555
Revert submission 1946

Reason for revert: regression in building (found via bisection)

Reported by users:
> error: path '/nix/store/04ca5xwvasz6s3jg0k7njz6rzi0d225w-jq-1.7.1-dev' does not exist in the store

Reverted changes: /q/submissionid:1946

Change-Id: I6f1a4b2f7d7ef5ca430e477fc32bca62fd97036b
Change-Id: I8355d8d3f6c43a812990c1912b048e5735b07f7b
Change-Id: Ib112ea9a3e67d5cb3d7d0ded30bbd25c96262470
notably we will check whether we want to do GC at all only once during
startup, and we'll only attempt GC every ten seconds rather than every
time a goal has finished a partial work call. this shouldn't cause any
problems in practice since relying on auto-gc is not deterministic and
stores in which builds can fill all remaining free space in merely ten
seconds are severely troubled even when gargage collection runs a lot.

Change-Id: I1175a56bf7f4e531f8be90157ad88750ff2ddec4
Worker::run() is now entirely based on the kj event loop and promises,
so we need not handle awakeness of goals manually any more. every goal
can instead, once it has finished a partial work call, defer itself to
being called again in the next iteration of the loop. same end effect.

Change-Id: I320eee2fa60bcebaabd74d1323fa96d1402c1d15
we'll now loop to update displayed statistics, and use this loop to
limit the update rate to 50 times per second. we could have updated
much more frequently before this (once per iteration of `runImpl`),
much faster than would ever be useful in practice. aggressive stats
updates can even impede progress due to terminal or network delays.

Change-Id: Ifba755a2569f73c919b1fbb06a142c0951395d6d
Change-Id: I38cfe8c7059251b581f1013c4213804f36b985ea
Change-Id: I2d4dcedff0a278d2d8f3d264a9186dfb399275e2
Change-Id: Ic2f7bc2bd6a1879ad614e4be81a7214f64eb0e85
This splits `ignoreException` into `ignoreExceptionExceptInterrupt`
(which ignores all exceptions except `Interrupt`, which indicates a
SIGINT/CTRL-C) and `ignoreExceptionInDestructor` (which ignores all
exceptions, so that destructors do not throw exceptions).

This prevents many cases where Nix ignores CTRL-C entirely.
See: https://github.com/NixOS/nix/issues/7245

Upstream-PR: https://github.com/NixOS/nix/pull/11618
Change-Id: Ie7d2467eedbe840d1b9fa2e88a4e88e4ab26a87b
Change-Id: Ibb59ddc21f5d3ef7fb4c900e3413e426c201334d
due to event loop scheduling behavior it's possible for a derivation
goal to fully finish (having seen all paths it was asked to create),
but to not notify the worker of this in time to prevent another goal
asking the recently-finished goal for more outputs. if this happened
the finished goal would ignore the request for more outputs since it
considered itself fully done, and the delayed result reporting would
cause the requesting goal to assume its request had been honored. if
the requested goal had finished *properly* the worker would recreate
it instead of asking for more outputs, and this would succeed. it is
thus safe to always recreate goals once they are done, so we now do.

Change-Id: Ifedd69ca153372c623abe9a9b49cd1523588814f
In 0e6b3435a1, installation of the HTML manual
was accidentally dropped: setting install_dir on a custom_target only sets the
directory where something is going to be installed if it is installed at all,
but does not itself trigger installation. The latter has to be explicitly
requested, which is just what we do here to get the manual back.

Change-Id: Iff8b791de7e7cb4c8d747c2a9b1154b5fcc32fe0
with async runtime scheduling we can no longer guarantee exact error
counts for builds that do not set keepGoing. the old behavior can be
recovered with a number of hacks that affect scheduling, but none of
those are very easy to follow now advisable. exact error counts will
like not be needed for almost all uses except tests, and *those* had
better check the actual messages rather than how many they got. more
messages can even help to avoid unnecessary rebuilds for most users.

Change-Id: I1c9aa7a401227dcaf2e19975b8cb83c5d4f85d64
I'm very excited for cl/2016, so others will probably be excited also!
Let's add a release note.

Change-Id: Ic84a4444241aafce4cb6d5a6d1dddb47e7a7dd7b
this was a triumph. i'm making a note here: huge success. it's hard to
overstate my satisfaction! i'm not even angry. i'm being so sincere ri

actually, no. we *are* angry. this was one dumbass odyssey. nobody has
asked for this. but not doing it would have locked us into old, broken
protocols forever or (possibly worse) forced us to write our own async
framework building on the old did-you-mean-continuations in Worker. if
we had done that we'd be locked into ever more, and ever more complex,
manual state management all over the place. this just could not stand.

Change-Id: I43a6de1035febff59d2eff83be9ad52af4659871
this simplifies the worker loop, and lets us remove it entirely later.
note that ideally only one promise waiting for interrupts should exist
in the entire system. not one per event loop, one per *process*. extra
interrupt waiters make interrupt response nondeterministic and as such
aren't great for user experience. if anything wants to react to aborts
caused by explicit interruptions, or anything else, those things would
be much better served using RAII guards such as Finally (or KJ_DEFER).

Change-Id: I41d035ff40172d536e098153c7375b0972110d51
there's no reason to have the worker set information on goals that the
goals themselves return from their entry point. doing this in the goal
`work()` function is much cleaner, and a prerequisite to removing more
implicit strong shared references to goals that are currently running.

Change-Id: Ibb3e953ab8482a6a21ce2ed659d5023a991e7923
Change-Id: Idd218ec1572eda84dc47accc0dcd8a954d36f098
drop childException since it's no longer needed. also makes
waitForInput, childFinished, and childTerminated redundant.

Change-Id: I05d88ffd323c5b5c909ac21056162f69ffb0eb9f
since we now propagate goal exceptions properly we no longer need to
check topGoals for a reason to abort early. any early abort reasons,
whether by exception or a clean top goal failure, can now be handled
by inspecting the goal result in the main loop. this greatly reduces
goal-to-goal interactions that do not happen at the main loop level.

since the underscore-free name is now available for use as variables
we'll migrate to that where we currently use `_topGoals` for locals.

Change-Id: I5727c5ea7799647c0a69ab76975b1a03a6558aa6
this will be needed to move all interesting result fields out of Goal
proper and into WorkResult. once that is done we can treat goals as a
totally internal construct of the worker mechanism, which also allows
us to fully stop exposing unclear intermediate state to Worker users.

Change-Id: I98d7778a4b5b2590b7b070bdfc164a22a0ef7190
the field is simply duplicated between the two, and now that we can
return WorkResults from Worker::run we no longer need both of them.

Change-Id: I82fc47d050b39b7bb7d1656445630d271f6c9830
derivation goals still hold a BuildResult member variable since parts of
these results of accumulated in different places, but the Goal class now
no longer has such a field. substitution goals don't need it at all, and
derivation goals should also be refactored to not drop their buildResult

Change-Id: Ic6d3d471cdbe790a6e09a43445e25bedec6ed446
yet another duplicated field. it's the last one though.

Change-Id: I352df8d306794d262d8c9066f3be78acd40e82cf
Goal::work() is a fully usable promise that does not rely on the worker
to report completion conditions. as such we no longer need the `notify`
field that enabled this interplay. we do have to clear goal caches when
destroying the worker though, otherwise goal promises may (incorrectly)
keep goals alive due to strong shared pointers created by childStarted.

Change-Id: Ie607209aafec064dbdf3464fe207d70ba9ee158a
we can use our newfound powers of Goal::work Is A Real Promise to remove
completed goals from continuation promises. apart from being much easier
to follow it's also a lot more efficient because we have the iterator to
the item we are trying to remove, skipping a linear search of the cache.

Change-Id: Ie0190d051c5f4b81304d98db478348b20c209df5
these two functions are now nearly trivial and much better inline into
makeGoalCommon. keeping them separate also separates information about
goal completion flows and how failure information ends up in `Worker`.

Change-Id: I6af86996e4a2346583371186595e3013c88fb082
There was a bug report about a potential call to `memcpy` with a null
pointer which is not reproducible:
lix-project/lix#492

This occurred in `src/libstore/filetransfer.cc` in `InnerSource::read`.

To ensure that this doesn't happen, an early return is added before
calling `memcpy` if the length of the data to be copied is 0.

This change also adds a test that ensures that when `InnerSource::read`
is called with an empty file, it throws an `EndOfFile` exception.

Change-Id: Ia18149bee9a3488576c864f28475a3a0c9eadfbb
Add the compile flag '-Wsign-compare' and adapt the code to fix all
cases of this warning.

Change-Id: I26b08fa5a03e4ac294daf697d32cf9140d84350d
The approach that was taken here was to add default values to the type
definitions rather than specify them whenever they are missing.

Now the only remaining warning is '-Wunused-parameter' which @jade said
is usually counterproductive and that we can just disable it:
lix-project/lix#456 (comment)

So this change adds the flags '-Wall', '-Wextra' and
'-Wno-unused-parameter', so that all warnings are enabled except for
'-Wunused-parameter'.

Change-Id: Ic223a964d67ab429e8da804c0721ba5e25d53012
I followed @pennae's advice and moved the constructor definition of
`AttrName` from the header file `nixexpr.hh` to `nixexpr.cc`.

Change-Id: I733f56c25635b366b11ba332ccec38dd7444e793
Change-Id: Ie73bbed1db9419c9885b9d57e4edb7a4047d5cce
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix-tests-without-master:fix-tests-without-master
git checkout fix-tests-without-master

Merge

Merge the changes and update on Forgejo.
git checkout main
git merge --no-ff fix-tests-without-master
git checkout main
git merge --ff-only fix-tests-without-master
git checkout fix-tests-without-master
git rebase main
git checkout main
git merge --no-ff fix-tests-without-master
git checkout main
git merge --squash fix-tests-without-master
git checkout main
git merge --ff-only fix-tests-without-master
git checkout main
git merge fix-tests-without-master
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: zebreus/lix#1
No description provided.
Delete branch "fix-tests-without-master"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?