Commit graph

2220 commits

Author SHA1 Message Date
jade 789b19a0cf util: fix brotli decompression of empty input
This caused an infinite loop before since it would just keep asking the
underlying source for more data.

In practice this happened because an HTTP server served a
response to a HEAD request (for which curl will not retrieve any body or
call our write callback function) with Content-Encoding: br, leading to
decompressing nothing at all and going into an infinite loop.

This adds a test to make sure none of our compression methods do that
again, as well as just patching the HTTP client to never feed empty data
into a compression algorithm (since they absolutely have the right to
throw CompressionError on unexpectedly-short streams!).

Reported on Matrix: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$8BWQR_zKxCQDJ40C5NnDo4bQPId3pZ_aoDj2ANP7Itc?via=lix.systems&via=matrix.org&via=tchncs.de

Change-Id: I027566e280f0f569fdb8df40e5ecbf46c211dad1
2024-09-18 15:37:29 -07:00
jade 4046e019ca tests/compression: rewrite
This test suite was in desperate need of using the parameterization
available with gtest, and was a bunch of useless duplicated code. At
least now it's not duplicated code, though it still probably should be
more full of property tests.

Change-Id: Ia8ccee7ef4f02b2fa40417b79aa8c8f0626ea479
2024-09-17 19:07:48 -07:00
Rebecca Turner 75c0de3e3c
Test including relative paths in configuration
Change-Id: If6c69a5e16d1ccd223fba392890f08f0032fb754
2024-09-01 15:52:48 -07:00
Rebecca Turner 02eb07cfd5 Merge changes I5566a985,I88cf53d3 into main
* changes:
  Support relative and `~/` paths in config settings
  Thread `ApplyConfigOptions` through config parsing
2024-09-01 22:06:36 +00:00
jade 04f8a14833
tree-wide: shuffle headers around for about 30s compile time
This didn't really feel so worth it afterwards, but I did untangle a
bunch of stuff that should not have been tangled.

The general gist of this change is that variant bullshit was causing a
bunch of compile time, and it seems like the only way to deal with
variant induced compile time is to keep variant types out of headers.
Explicit template instantiation seems to do nothing for them.

I also seem to have gotten some back-end time improvement from
explicitly instantiating regex, but I don't know why. There is no
corresponding front-end time improvement from it: regex is still at the
top of the sinners list.

**** Templates that took longest to instantiate:
 15231 ms: std::basic_regex<char>::_M_compile (28 times, avg 543 ms)
 15066 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 538 ms)
 12571 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 448 ms)
 12454 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 444 ms)
 12225 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 436 ms)
 11363 ms: nlohmann::basic_json<>::parse<const char *> (21 times, avg 541 ms)
 10628 ms: nlohmann::basic_json<>::basic_json (109 times, avg 97 ms)
 10134 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 361 ms)

Back-end time before messing with the regex:
**** Function sets that took longest to compile / optimize:
  8076 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4382 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1247 times, avg 3 ms)
  3137 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 22 ms)
  2896 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 16 ms)
  2304 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (210 times, avg 10 ms)
  2116 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 18 ms)
  2051 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 8 ms)
  2037 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 21 ms)
  1928 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 68 ms)
  1859 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (41 times, avg 45 ms)
  1824 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (973 times, avg 1 ms)
  1810 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 16 ms)
  1793 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1793 ms)
  1759 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (281 times, avg 6 ms)
  1722 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 90 ms)
  1677 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1674 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (249 times, avg 6 ms)
  1660 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (304 times, avg 5 ms)
  1599 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 84 ms)
  1568 ms: void std::__detail::_Compiler<$>::_M_insert_bracket_matcher<$>(bool) (112 times, avg 14 ms)
  1541 ms: std::__shared_ptr<$>::~__shared_ptr() (531 times, avg 2 ms)
  1539 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (41 times, avg 37 ms)
  1471 ms: void std::__detail::_Compiler<$>::_M_insert_character_class_matcher<... (112 times, avg 13 ms)

After messing with the regex (notice std::__detail::_Compiler vanishes
here, but I don't know why):

**** Function sets that took longest to compile / optimize:
  8054 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms)
  4313 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1217 times, avg 3 ms)
  3259 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 23 ms)
  3045 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 17 ms)
  2314 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (207 times, avg 11 ms)
  1923 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (216 times, avg 8 ms)
  1817 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 100 ms)
  1816 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms)
  1788 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (40 times, avg 44 ms)
  1749 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (278 times, avg 6 ms)
  1724 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (248 times, avg 6 ms)
  1697 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms)
  1684 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1684 ms)
  1680 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (303 times, avg 5 ms)
  1589 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 88 ms)
  1483 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (181 times, avg 8 ms)
  1447 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (40 times, avg 36 ms)
  1441 ms: std::__shared_ptr<$>::~__shared_ptr() (496 times, avg 2 ms)
  1420 ms: boost::stacktrace::basic_stacktrace<$>::init(unsigned long, unsigned... (137 times, avg 10 ms)
  1396 ms: boost::basic_format<$>::~basic_format() (194 times, avg 7 ms)
  1290 ms: std::__cxx11::basic_string<$>::_M_replace_cold(char*, unsigned long,... (231 times, avg 5 ms)
  1258 ms: std::vector<$>::~vector() (354 times, avg 3 ms)
  1222 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (231 times, avg 5 ms)
  1194 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (49 times, avg 24 ms)
  1186 ms: bool tao::pegtl::internal::sor<$>::match<$>(std::integer_sequence<$>... (1 times, avg 1186 ms)
  1149 ms: std::__detail::_Executor<$>::_M_dfs(std::__detail::_Executor<$>::_Ma... (70 times, avg 16 ms)
  1123 ms: toml::detail::sequence<$>::invoke(toml::detail::location&) (69 times, avg 16 ms)
  1110 ms: nlohmann::json_abi_v3_11_3::basic_json<$>::json_value::destroy(nlohm... (55 times, avg 20 ms)
  1079 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (541 times, avg 1 ms)
  1033 ms: nlohmann::json_abi_v3_11_3::detail::lexer<$>::scan_number() (20 times, avg 51 ms)

Change-Id: I10af282bcd4fc39c2d3caae3453e599e4639c70b
2024-08-28 09:55:05 -07:00
jade 0cc285f87b
treewide: fix a bunch of lints
Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
  extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
  also do memory corruption with them with references.
  - Remove casts to Expr * where ExprBlackHole is an incomplete type by
    introducing an explicitly-cast eBlackHoleAddr as Expr *.
  - An incredibly illegal cast of the text bytes of the StorePath hash
    into a size_t directly. You can't DO THAT.

    Replaced with actually parsing the hash so we get 100% of the bits
    being entropy, then memcpying the start of the hash. If this shows
    up in a profile we should just make the hash parser faster with a
    lookup table or something sensible like that.
  - This horrendous bit of UB which I thankfully slapped a deprecation
    warning on, built, and it didn't trigger anywhere so it was dead
    code and I just deleted it. But holy crap you *cannot* do that.

    inline void mkString(const Symbol & s)
    {
        mkString(((const std::string &) s).c_str());
    }
- Some wrong lints. Lots of wrong macro lints, one wrong
  suspicious-sizeof lint triggered by the template being instantiated
  with only pointers, but the calculation being correct for both
  pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
  exceptions that might actually happen rather than all the exceptions
  imaginable. We can let the runtime hard-kill it on other exceptions
  imo.

Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
2024-08-26 16:13:03 -07:00
Rebecca Turner 690f07272e
Support relative and ~/ paths in config settings
Change-Id: I5566a9858ba255f4ac5051d1368c7dfb24460f0a
2024-08-25 15:54:22 -07:00
Rebecca Turner 5fc6fcb310
Thread ApplyConfigOptions through config parsing
This makes no changes to logic but makes the `ApplyConfigOptions` value
available to consumers.

Change-Id: I88cf53d38faac8472c556aee55c13d0acbd1e5db
2024-08-25 15:54:22 -07:00
eldritch horrors b6884388a1 add dedicated test for hash mismatch url reporting
the current test relies on derivation build order being deterministic,
which will not be a reasonable expectation for all that long any more.

Change-Id: I9be44a7725185f614a9a4c724045b8b1e6962c03
2024-08-25 22:21:32 +00:00
jade 72f91767a8 Merge "fix: good errors for failures caused by allowSubstitutes" into main 2024-08-25 20:00:58 +00:00
eldritch horrors cae260a158 libstore: diagnose local build failure in goal
this should be done where we're actually trying to build something, not
in the main worker loop that shouldn't have to be aware of such details

Change-Id: I07276740c0e2e5591a8ce4828a4bfc705396527e
2024-08-25 19:55:47 +02:00
jade 686120ee4a fix: good errors for failures caused by allowSubstitutes
This caused an absolute saga which I would not like anyone else to have
to experience. Let's put in a laser targeted error message that
diagnoses this exact problem.

Fixes: #484
Change-Id: I2a79f04aeb4a1b67c10115e5e39501d958836298
2024-08-23 17:49:15 -07:00
alois31 e3c289dbe9
libutil/config: unify path setting types
There have been multiple setting types for paths that are supposed to be
canonicalised, depending on whether zero or one, one, or any number of paths is
to be specified. Naturally, they behaved in slightly different ways in the
code. Simplify things by unifying them and removing special behaviour (mainly
the "multiple paths type can coerce to boolean" thing).

Change-Id: I7c1ce95e9c8e1829a866fb37d679e167811e9705
2024-08-21 17:57:23 +02:00
piegames 0edfea450b libexpr: Soft-deprecate ancient let syntax
Change-Id: I6802b26f038578870ea1fa1ed298f0c4b1f29c4a
2024-08-21 12:59:03 +02:00
piegames 0a8888d1c7 treewide: Stop using ancient let syntax
Shows for how long these tests have not been touched by anyone …

Change-Id: I3d0c1209a86283ddb012db4e7d45073264fdd0eb
2024-08-21 06:55:52 +00:00
piegames 7210ed1b87 libexpr: Soft-deprecate __overrides
Change-Id: I787e69e1dad6edc5ccdb747b74a9ccd6e8e13bb3
2024-08-21 06:55:52 +00:00
Audrey Dutcher ac6974777e Merge "tests/functional/restricted: Don't use a process substitution" into main 2024-08-20 22:51:38 +00:00
Audrey Dutcher ae628d4af2 tests/functional/restricted: Don't use a process substitution
The <() process substitution syntax doesn't work for this one testcase
in bash for FreeBSD. The exact reason for this is unknown, possibly to
do with pipe vs file vs fifo EOF behavior. The prior behavior was this
test hanging forever, with no children of the bash process.

Change-Id: I71822a4b9dea6059b34300568256c5b7848109ac
2024-08-19 20:37:51 -07:00
Maximilian Bosch 040e783232 flake: don't refetch unmodified inputs by recursive follows
Closes #460

I managed to trigger the issue by having the following inputs (shortened):

    authentik-nix.url = "github:nix-community/authentik-nix";
    authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";

When evaluating this using

    nix-eval-jobs --flake .#hydraJobs

I got the following error:

    error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode

The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).

The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.

Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
2024-08-19 19:57:12 +00:00
piegames 278fddc317 libexpr: Deprecate URL literals
Closes #437.

Change-Id: I9f67fc965bb4a7e7fd849e5067ac1cb3bab064cd
2024-08-17 20:31:57 +02:00
piegames 49d61b2e4b libexpr: Introduce Deprecated features
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437

Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
2024-08-17 19:47:51 +02:00
piegames 1c080a8239 treewide: Stop using URL literals
They must die

Change-Id: Ibe2b1818b21d98ec1a68836d01d5dad729b8c501
2024-08-17 15:48:10 +00:00
jade b15d5cc6ee nix: remove explosions if you have a window size less than four
Turns out strings do not like being resized to -4.

This was discovered while messing with the tests to remove unbuffer and
trying stdbuf instead. Turns out that was not the right approach.

This basically rewrites the handling of this case to be much more
correct, and fixes a bug where with small window sizes where it would
ALSO truncate the attr names in addition to the optional descriptions.

Change-Id: Ifd1beeaffdb47cbb5f4a462b183fcb6c0ff6c524
2024-08-10 16:11:58 -07:00
jade 0c76195351 build: remove expect as a dependency
I was packaging Lix 2.91 for nixpkgs and was annoyed at the expect
dependency. Turns out that you can replace unbuffer with a pretty-short
Python script.

It became less short after I found out that Linux was converting \n to
\r\n in the terminal subsystem, which was not very funny, but is at
least solved by twiddling termios bits.

Change-Id: I8a2700abcbbf6a9902e01b05b40fa9340c0ab90c
2024-08-10 16:10:16 -07:00
jade 757041c3e7 Merge changes I526cceed,Ia4e2f1fa,I22e66972,I9fbd55a9,Ifca22e44 into main
* changes:
  sqlite: add a Use::fromStrNullable
  util: implement charptr_cast
  tree-wide: fix a pile of lints
  refactor: make HashType and Base enum classes for type safety
  build: integrate clang-tidy into CI
2024-08-08 22:43:10 +00:00
jade e34833c025 tree-wide: fix a pile of lints
This:
- Converts a bunch of C style casts into C++ casts.
- Removes some very silly pointer subtraction code (which is no more or
  less busted on i686 than it began)
- Fixes some "technically UB" that never had to be UB in the first
  place.
- Makes finally follow the noexcept status of the inner function. Maybe
  in the future we should ban the function from not being noexcept, but
  that is not today.
- Makes various locally-used exceptions inherit from std::exception.

Change-Id: I22e66972602604989b5e494fd940b93e0e6e9297
2024-08-08 14:53:17 -07:00
jade 370ac940dd refactor: make HashType and Base enum classes for type safety
Change-Id: I9fbd55a9d50464a56fe11cb42a06a206914150d8
2024-08-08 14:53:17 -07:00
piegames 28ae24f3f7 libexpr: Add experimental pipe operator
The |> operator is a reverse function operator with low binding strength
to replace lib.pipe. Implements RFC 148, see the RFC text for more
details. Closes #438.

Change-Id: I21df66e8014e0d4dd9753dd038560a2b0b7fd805
2024-08-08 11:13:53 +02:00
Max “Goldstein” Siling 6fdb47f0b2 Merge "src/libcmd/repl.cc: allow :log /path/to/store.drv" into main 2024-08-07 21:48:01 +00:00
piegames ec7552ff74 libexpr/parser: Test experimental features
Currently, the parser relies on the global experimental feature flags.
In order to properly test conditional language features, we instead need
to pass it around in the parser::State.

This means that the parser cannot cache the result of isEnabled anymore,
which wouldn't necessarily hurt performance if the function didn't
perform a linear search on the list of enabled features on every single
call. While we could simply evaluate once at the start of parsing and
cache the result in the parser state, the more sustainable solution
would be to fix `isEnabled` such that all callers may profit from the
performance improvement.

Change-Id: Ic9b9c5d882b6270e1114988b63e6064d36c25cf2
2024-08-07 13:07:50 +00:00
Max “Goldstein” Siling 9adfd9b8ad
src/libcmd/repl.cc: allow :log /path/to/store.drv
This adds a second form to the `:log` command: it now can accept a
derivation path in addition to a derivation expression. As derivation
store paths start with `/nix/store`, this is not ambiguous.

Resolves: #51
Change-Id: Iebc7b011537e7012fae8faed4024ea1b8fdc81c3
2024-08-07 15:58:44 +03:00
Maximilian Bosch 27a63db710 Merge "fix: warn and document when advanced attributes will have no impact due to __structuredAttrs" into main 2024-08-07 10:38:39 +00:00
jade ca9d3e6e00 tree-wide: fix various lint warnings
Change-Id: I0fc80718eb7e02d84cc4b5d5deec4c0f41116134
2024-08-04 20:55:45 -07:00
jade bd1344ec54 nix flake metadata: print modified dates for input flakes
This was always in the lock file and we can simply actually print it.

The test for this is a little bit silly but it should correctly
control for my daring to exercise timezone code *and* locale code in a
test, which I strongly suspect nobody dared do before.

Sample (abridged):
```
Path:          /nix/store/gaxb42z68bcr8lch467shvmnhjjzgd8b-source
Last modified: 1970-01-01 00:16:40
Inputs:
├───flake-compat: github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33
│   Last modified: 2023-10-04 13:37:54
├───flake-utils: github:numtide/flake-utils/b1d9ab70662946ef0850d488da1c9019f3a9752a
│   Last modified: 2024-03-11 08:33:50
│   └───systems: github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e
│       Last modified: 2023-04-09 08:27:08
```

Change-Id: I355f82cb4b633974295375ebad646fb6e2107f9b
2024-08-04 20:41:19 -07:00
jade 5f0ef50077 cli: eat terminal codes from stdout also
This *should* be sound, plus or minus the amount that the terminal code
eating code is messed up already.

This is useful for testing CLI output because it will strip the escapes
enough to just shove the expected output in a file.

Change-Id: I8a9b58fafb918466ac76e9ab585fc32fb9294819
2024-08-04 20:41:19 -07:00
Tom Bereknyei 7fc481396c
fix: warn and document when advanced attributes will have no impact due to __structuredAttrs
Backport of https://github.com/NixOS/nix/pull/10884.

Change-Id: I82cc2794730ae9f4a9b7df0185ed0aea83efb65a
2024-08-03 13:32:51 +02:00
Isabel 9eb374dc6d Merge "nix flake show: add the description if it exists" into main 2024-08-02 07:56:06 +00:00
Maximilian Bosch 3bb8c627ae Merge "Reapply "libfetchers: make attribute / URL query handling consistent"" into main 2024-08-02 04:50:25 +00:00
Maximilian Bosch 87fd1f024c Reapply "libfetchers: make attribute / URL query handling consistent"
The original attempt at this introduced a regression; this commit
reverts the revert and fixes the regression.

This reverts commit 3e151d4d77.

Fix to the regression:

flakeref: fix handling of `?dir=` param for flakes in subdirs

As reported in #419[1], accessing a flake in a subdir of a Git
repository fails with the previous commit[2] applied with the error

    error: unsupported Git input attribute 'dir'

The problem is that the `dir`-param is inserted into the parsed URL if a
flake is fetched from the subdir of a Git repository. However, for the
fetching part this isn't even needed. The fix is to just pass `subdir`
as second argument to `FlakeRef` (which needs a `basedir` that can be
empty) and leave the parsedURL as-is.

Added a regression test to make sure we don't run into this again.

[1] #419
[2] e22172aaf6b6a366cecd3c025590e68fa2b91bcc,
    originally 3e151d4d77

Change-Id: I2c72d5a32e406a7ca308e271730bd0af01c5d18b
2024-08-01 15:41:30 -07:00
jade 19ae87e5ce tree-wide: add support for asan!
What if you could find memory bugs in Lix without really trying very
hard? I've had variously scuffed patches to do this, but this is
blocked on boost coroutines removal at this point tbh.

Change-Id: Id762af076aa06ad51e77a6c17ed10275929ed578
2024-07-31 14:13:39 -07:00
Max “Goldstein” Siling 1a6d7a3af4 src/libcmd/repl.cc: avoid unneeded reload after :e
If `:edit`ing a store path, don't reload repl afterwards
to avoid losing local variables: store is immutable,
so "editing" a store path is always just viewing it.

Resolves: #341
Change-Id: I3747f75ce26e0595e953069c39ddc3ee80699718
2024-07-30 16:08:26 +00:00
alois31 e7188e211a
libstore/build: block io_uring
Unfortunately, io_uring is totally opaque to seccomp, and while currently there
are no dangerous operations implemented, there is no guarantee that it remains
this way. This means that io_uring should be blocked entirely to ensure that
the sandbox is future-proof. This has not been observed to cause issues in
practice.

Change-Id: I45d3895f95abe1bc103a63969f444c334dbbf50d
2024-07-25 18:24:45 +02:00
alois31 127ee1a101
libstore/build: use an allowlist approach to syscall filtering
Previously, system call filtering (to prevent builders from storing files with
setuid/setgid permission bits or extended attributes) was performed using a
blocklist. While this looks simple at first, it actually carries significant
security and maintainability risks: after all, the kernel may add new syscalls
to achieve the same functionality one is trying to block, and it can even be
hard to actually add the syscall to the blocklist when building against a C
library that doesn't know about it yet. For a recent demonstration of this
happening in practice to Nix, see the introduction of fchmodat2 [0] [1].

The allowlist approach does not share the same drawback. While it does require
a rather large list of harmless syscalls to be maintained in the codebase,
failing to update this list (and roll out the update to all users) in time has
rather benign effects; at worst, very recent programs that already rely on new
syscalls will fail with an error the same way they would on a slightly older
kernel that doesn't support them yet. Most importantly, no unintended new ways
of performing dangerous operations will be silently allowed.

Another possible drawback is reduced system call performance due to the larger
filter created by the allowlist requiring more computation [2]. However, this
issue has not convincingly been demonstrated yet in practice, for example in
systemd or various browsers. To the contrary, it has been measured that the the
actual filter constructed here has approximately the same overhead as a very
simple filter blocking only one system call.

This commit tries to keep the behavior as close to unchanged as possible. The
system call list is in line with libseccomp 2.5.5 and glibc 2.39, which are the
latest versions at the point of writing. Since libseccomp 2.5.5 is already a
requirement and the distributions shipping this together with older versions of
glibc are mostly not a thing any more, this should not lead to more build
failures any more.

[0] https://github.com/NixOS/nixpkgs/issues/300635
[1] https://github.com/NixOS/nix/issues/10424
[2] https://github.com/flatpak/flatpak/pull/4462#issuecomment-1061690607

Change-Id: I541be3ea9b249bcceddfed6a5a13ac10b11e16ad
2024-07-25 18:24:40 +02:00
Isabel d2422771eb
nix flake show: add the description if it exists
(cherry picked from commit 8cd1d02f90eb9915e640c5d370d919fad9833c65)

nix flake show: Only print up to the first new line if it exists.

(cherry picked from commit 5281a44927bdb51bfe6e5de12262d815c98f6fe7)

add tests

(cherry picked from commit 74ae0fbdc70a5079a527fe143c4832d1357011f7)

Handle long strings, embedded new lines and empty descriptions

(cherry picked from commit 2ca7b3afdbbd983173a17fa0a822cf7623601367)

Account for total length of 80

(cherry picked from commit 1cc808c18cbaaf26aaae42bb1d7f7223f25dd364)

docs: add nix flake show description release note

fix: remove white space

nix flake show: trim length based on terminal size

test: account for terminal size

docs(flake-description): before and after commands; add myself to credits

Upstream-PR: https://github.com/NixOS/nix/pull/10980
Change-Id: Ie1c667dc816b3dd81e65a1f5395e57ea48ee0362
2024-07-23 13:21:15 +01:00
jade 77ff799cc8 gc: refactor the gc server thread out into a class without changing it
This removes a *whole load* of variables from scope and enforces thread
boundaries with the type system.

There is not much change of significance in here, so the things to watch
out for while reviewing it are primarily that the destructor ordering
may have changed inadvertently, I think.

Change-Id: I3cd87e6d5a08dfcf368637407251db22a8906316
2024-07-19 20:55:55 +00:00
alois31 40c39aa5d2
libexpr/print: do not show elided nested items when there are none
When the configured maximum depth has been reached, attribute sets and lists
are printed with ellipsis to indicate the elision of nested items. Previously,
this happened even in case the structure being printed is empty, so that such
items do not in fact exist. This is confusing, so stop doing it.

Change-Id: I0016970dad3e42625e085dc896e6f476b21226c9
2024-07-18 18:41:34 +02:00
alois31 b5da823138
libexpr/print: never show empty attrsets or derivations as «repeated»
The repeated value detection logic exists so that the occurrence of large
common substructures does not fill up the screen or the computer's memory.
However, empty attribute sets and derivations (when their detection is enabled)
are always cheap to print, and in practice I have observed them to make up a
significant majority of the cases where I was annoyed by the repeated value
detection kicking in. Furthermore, `nix-instantiate --eval` already disables
this logic for empty attribute sets, and empty lists are already exempted
everywhere. For these reasons, always print empty attribute sets and
derivations as what they are.

Change-Id: I5dac8e7739f9d726b76fd0521ec46f38af94463f
2024-07-18 18:41:34 +02:00
alois31 81a0624d76
libexpr/print: pretty-print idempotently
When pretty-printing is enabled, previously an unforced thunk would trigger
indentation, even when it subsequently does not evaluate to a nested structure.
The resulting output looked inconsistent, and furthermore pretty-printing was
not idempotent (since pretty-printing the same value again, which is now fully
evaluated, will not trigger indentation).
When strict evaluation is enabled, force the item before inspecting its type,
so that it is properly known whether it contains a nested structure.
Furthermore, there is no need to cause indentation for unforced thunks, since
the very next operation will be printing them as `«thunk»`.

This is mostly a port of https://github.com/NixOS/nix/pull/11100 , but we only
force the item when it's going to be forced anyway due to strict
pretty-printing, and a new test was written since the REPL testing framework in
Lix is different.

Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
Change-Id: Ib7560fe531d09e05ca6b2037a523fe21a26d9d58
2024-07-18 18:41:28 +02:00
Max “Goldstein” Siling 68567206f2 Merge "tests/functional/repl.sh: actually fail test on wrong stdout" into main 2024-07-17 21:50:59 +00:00
Max “Goldstein” Siling 3a36c8bb90 tests/functional/repl.sh: actually fail test on wrong stdout
Previous test implementation assumed that grep supports newlines
in patterns. It doesn't, so tests spuriously passed, even though
some tests outputs were broken.

This patches output (and expected output) before grepping,
so there're no newlines in pattern.

Change-Id: Ie6561f9f2e18b83d976f162269d20136e2595141
2024-07-17 21:48:13 +00:00
eldritch horrors 6b4d46e9e0 libstore: remove WriteConn::sink fields
we no longer need these since we're no longer using sinks to serialize things.

Change-Id: Iffb1a3eab33c83f611c88fa4e8beaa8d5ffa079b
2024-07-16 00:57:42 +00:00
eldritch horrors a5d1f69841 libstore: generatorize protocol serializers
this is cursed. deeply and profoundly cursed. under NO CIRCUMSTANCES
must protocol serializer helpers be applied to temporaries! doing so
will inevitably cause dangling references and cause the entire thing
to crash. we need to do this even so to get rid of boost coroutines,
and likewise to encapsulate the serializers we suffer today at least
a little bit to allow a gradual migration to an actual IPC protocol.

(this isn't a problem that's unique to generators. c++ coroutines in
general cannot safely take references to arbitrary temporaries since
c++ does not have a lifetime system that can make this safe. -sigh-)

Change-Id: I2921ba451e04d86798752d140885d3c5cc08e146
2024-07-16 00:57:42 +00:00
Qyriad ae7eab49b9 nix3-upgrade-nix: always use the /new/ nix-env to perform the installation
Fixes #411.

Change-Id: I8d87c0e9295deea26ff33234e15ee33cc68ab303
2024-07-15 15:26:53 -06:00
jade 917c9bdee7 language: cleanly ban integer overflows
This also bans various sneaking of negative numbers from the language
into unsuspecting builtins as was exposed while auditing the
consequences of changing the Nix language integer type to a newtype.

It's unlikely that this change comprehensively ensures correctness when
passing integers out of the Nix language and we should probably add a
checked-narrowing function or something similar, but that's out of scope
for the immediate change.

During the development of this I found a few fun facts about the
language:
- You could overflow integers by converting from unsigned JSON values.
- You could overflow unsigned integers by converting negative numbers
  into them when going into Nix config, into fetchTree, and into flake
  inputs.

  The flake inputs and Nix config cannot actually be tested properly
  since they both ban thunks, however, we put in checks anyway because
  it's possible these could somehow be used to do such shenanigans some
  other way.

Note that Lix has banned Nix language integer overflows since the very
first public beta, but threw a SIGILL about them because we run with
-fsanitize=signed-overflow -fsanitize-undefined-trap-on-error in
production builds. Since the Nix language uses signed integers, overflow
was simply undefined behaviour, and since we defined that to trap, it
did.

Trapping on it was a bad UX, but we didn't even entirely notice
that we had done this at all until it was reported as a bug a couple of
months later (which is, to be fair, that flag working as intended), and
it's got enough production time that, aside from code that is IMHO buggy
(and which is, in any case, not in nixpkgs) such as
#445, we don't think
anyone doing anything reasonable actually depends on wrapping overflow.

Even for weird use cases such as doing funny bit crimes, it doesn't make
sense IMO to have wrapping behaviour, since two's complement arithmetic
overflow behaviour is so *aggressively* not what you want for *any* kind
of mathematics/algorithms. The Nix language exists for package
management, a domain where bit crimes are already only dubiously in
scope to begin with, and it makes a lot more sense for that domain for
the integers to never lose precision, either by throwing errors if they
would, or by being arbitrary-precision.

This change will be ported to CppNix as well, to maintain language
consistency.

Fixes: #423

Change-Id: I51f253840c4af2ea5422b8a420aa5fafbf8fae75
2024-07-13 00:59:33 +02:00
jade f9641b9efd libutil: add checked arithmetic tools
This is in preparation for adding checked arithmetic to the evaluator.

Change-Id: I6e115ce8f5411feda1706624977a4dcd5efd4d13
2024-07-13 00:56:37 +02:00
eldritch horrors df8851f286 libutil: rewrite RewritingSink as source
the rewriting sink was just broken. when given a rewrite set that
contained a key that is also a proper infix of another key it was
possible to produce an incorrectly rewritten result if the writer
used the wrong block size. fixing this duplicates rewriteStrings,
to avoid this we'll rewrite rewriteStrings to use RewritingSource
in a new mode that'll allow rewrites we had previously forbidden.

Change-Id: I57fa0a9a994e654e11d07172b8e31d15f0b7e8c0
2024-07-11 11:39:18 +00:00
Quantum Jump 6e0ca02425 Fix dry-run flag for nix-collect-garbage
`nix-collect-garbage --dry-run` previously elided the entire garbage
collection check, meaning that it would just exit the script without
printing anything.

This change makes the dry run flag instead set the GC action to
`gcReturnDead` rather than `gcDeleteDead`, and then continue with the
script. So if you set `--dry-run`, it will print the paths it *would*
have garbage collected, but not actually delete them.

I filed a bug for this: #432 but then realised I could give fixing it a go myself.

Change-Id: I062dbf1a80bbab192b5fd0b3a453a0b555ad16f2
2024-07-09 13:55:05 +00:00
eldritch horrors b51ea465de libutil: allow construction of sources from generators
Change-Id: I78ff8d0720f06bce731e26d5e1c53b1382bbd589
2024-07-05 22:28:16 +00:00
Qyriad 4c7165be86 distinguish between throws & errors during throw
Turns errors like this:

let
  throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"

error:
       … from call site
         at «string»:3:4:
            2|   throwMsg = a: throw (a + " invalid bar");
            3| in throwMsg "bullshit"
             |    ^

       … while calling 'throwMsg'
         at «string»:2:14:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |              ^
            3| in throwMsg "bullshit"

       … while calling the 'throw' builtin
         at «string»:2:17:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |                 ^
            3| in throwMsg "bullshit"

       error: bullshit invalid bar

into errors like this:

let
  throwMsg = a: throw (a + " invalid bar");
in throwMsg "bullshit"

error:
       … from call site
         at «string»:3:4:
            2|   throwMsg = a: throw (a + " invalid bar");
            3| in throwMsg "bullshit"
             |    ^

       … while calling 'throwMsg'
         at «string»:2:14:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |              ^
            3| in throwMsg "bullshit"

       … caused by explicit throw
         at «string»:2:17:
            1| let
            2|   throwMsg = a: throw (a + " invalid bar");
             |                 ^
            3| in throwMsg "bullshit"

       error: bullshit invalid bar

Change-Id: I593688928ece20f97999d1bf03b2b46d9ac338cb
2024-07-04 17:43:03 -06:00
Qyriad d00edfb28d trace when the foo part of foo.bar.baz errors
Turns errors like:

let
  errpkg = throw "invalid foobar";
in errpkg.meta

error:
       … while calling the 'throw' builtin
         at «string»:2:12:
            1| let
            2|   errpkg = throw "invalid foobar";
             |            ^
            3| in errpkg.meta

       error: invalid foobar

into errors like:

let
  errpkg = throw "invalid foobar";
in errpkg.meta

error:
       … while evaluating 'errpkg' to select 'meta' on it
         at «string»:3:4:
            2|   errpkg = throw "invalid foobar";
            3| in errpkg.meta
             |    ^

       … while calling the 'throw' builtin
         at «string»:2:12:
            1| let
            2|   errpkg = throw "invalid foobar";
             |            ^
            3| in errpkg.meta

       error: invalid foobar

For the low price of one try/catch, you too can have the incorrect line
of code actually show up in the trace!

Change-Id: If8d6200ec1567706669d405c34adcd7e2d2cd29d
2024-07-04 16:33:02 -06:00
Qyriad 59bf6825ef add an impl of Expr::show for ExprInheritFrom that doesn't crash
ExprVar::show() assumes it has a name. dynamic inherits do not
necessarily (ever?) have a name.

Change-Id: If10893188e307431da17f0c1bd0787adc74f7141
2024-07-04 15:55:38 -06:00
eldritch horrors 5eec6418de libutil: begin porting serialization to generators
generators are a better basis for serializers than streaming into sinks
as we do currently for many reasons, such as being usable as sources if
one wishes to (without requiring an intermediate sink to serialize full
data sets into memory, or boost coroutines to turn sinks into sources),
composing more naturally (as one can just yield a sub-generator instead
of being forced to wrap entire substreams into clunky functions or even
more clunky custom types to implement operator<< on), allowing wrappers
to transform data with clear ownership semantics (removing the need for
explicit memory allocations and Source wrappers), and many other things

Change-Id: I361d89ff556354f6930d9204f55117565f2f7f20
2024-07-03 11:46:53 +00:00
eldritch horrors 73ddc4540f libutil: generator type with on-yield value mapping
this will be the basis of non-boost coroutines in lix. anything that is
a boost coroutine *should* be representable with a Generator coroutine,
and many things that are not currently boost coroutines but behave much
like one (such as, notably, serializers) should be as well. this allows
us to greatly simplify many things that look like iteration but aren't.

Change-Id: I2cebcefa0148b631fb30df4c8cfa92167a407e34
2024-07-03 11:46:53 +00:00
alois31 24852355d8 Merge "tree-wide: unify progress bar inactive and paused states" into main 2024-07-02 14:12:07 +00:00
Delan Azabani 865a3732fa Merge "Reject fully-qualified URLs in 'from' argument of nix registry add" into main 2024-07-02 07:20:01 +00:00
alois31 0dd1d8ca1c
tree-wide: unify progress bar inactive and paused states
Previously, the progress bar had two subtly different states in which the bar
would not actually render, both with their own shortcomings: inactive (which
was irreversible) and paused (reversible, but swallowing logs). Furthermore,
there was no way of resetting the statistics, so a very bad solution was
implemented (243c0f18da) that would create a new
logger for each line of the repl, leaking the previous one and discarding the
value of printBuildLogs. Finally, if stderr was not attached to a TTY, the
update thread was started even though the logger was not active, violating the
invariant required by the destructor (which is not observed because the logger
is leaked).

In this commit, the two aforementioned states are unified into a single one,
which can be exited again, correctly upholds the invariant that the update
thread is only running while the progress bar is active, and does not swallow
logs. The latter change in behavior is not expected to be a problems in the
rare cases where the paused state was used before, since other loggers (like
the simple one) don't exhibit it anyway. The startProgressBar/stopProgressBar
API is removed due to being a footgun, and a new method for properly resetting
the progress is added.

Co-Authored-By: Qyriad <qyriad@qyriad.me>
Change-Id: I2b7c3eb17d439cd0c16f7b896cfb61239ac7ff3a
2024-07-01 18:19:34 +02:00
jade d3286d0990 Merge changes Ie29a8a89,I873eedcf into main
* changes:
  store: delete obsolete lsof-disabling code
  store: guess the URL of failing fixed-output derivations
2024-07-01 16:11:32 +00:00
alois31 a55112898e
libexpr/flake: allow automatic rejection of configuration options from flakes
The `allow-flake-configuration` option allows the user to control whether to
accept configuration options supplied by flakes. Unfortunately, setting this
to false really meant "ask each time" (with an option to remember the choice
for each specific option encountered). Let no mean no, and introduce (and
default to) a separate value for the "ask each time" behaviour.

Co-Authored-By: Jade Lovelace <lix@jade.fyi>
Change-Id: I7ccd67a95bfc92cffc1ebdc972d243f5191cc1b4
2024-06-30 19:28:14 +02:00
Delan Azabani b2944d93a6 Reject fully-qualified URLs in 'from' argument of nix registry add
We previously allowed you to map any flake URL to any other flake URL,
including shorthand flakerefs, indirect flake URLs like `flake:nixpkgs`,
direct flake URLs like `github:NixOS/nixpkgs`, or local paths.

But flake registry entries mapping from direct flake URLs often come
from swapping the 'from' and 'to' arguments by accident, and even when
created intentionally, they may not actually work correctly.

This patch rejects those URLs (and fully-qualified flake: URLs), making
it harder to swap the arguments by accident.

Fixes #181.

Change-Id: I24713643a534166c052719b8770a4edfcfdb8cf3
2024-06-29 05:11:31 +00:00
jade d92712673b store: guess the URL of failing fixed-output derivations
This is a shameless layering violation in favour of UX. It falls back
trivially to "unknown", so it's purely a UX feature.

Diagnostic sample:

```
error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv':
        likely URL: https://meow.puppy.forge/puppy.tar.gz
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-a1Qvp3FOOkWpL9kFHgugU1ok5UtRPSu+NwCZKbbaEro=
```

Change-Id: I873eedcf7984ab23f57a6754be00232b5cb5b02c
2024-06-27 22:44:16 -07:00
jade 4ac2c496d4 Merge "change shebangs of all .sh scripts to bash" into main 2024-06-25 22:18:26 +00:00
eldritch horrors e6cd67591b libexpr: rewrite the parser with pegtl instead of flex/bison
this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.

this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.

test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:

  - 40% faster on lix's package.nix at 10000 iterations
  - 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
  - equivalent on all of nixpkgs concatenated at 100 iterations
    (excluding invalid files, each file surrounded with parens)

more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.

Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.408s ± 0.025s
            user: 0.355s | system: 0.033s
  median:   0.389s
  range:    0.388s ... 0.442s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.332s ± 0.024s
            user: 0.279s | system: 0.033s
  median:   0.314s
  range:    0.313s ... 0.361s
  relative: 0.814

---

old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     6.133s ± 0.022s
            user: 5.395s | system: 0.437s
  median:   6.128s
  range:    6.099s ... 6.183s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     5.925s ± 0.025s
            user: 5.176s | system: 0.456s
  median:   5.934s
  range:    5.861s ... 5.943s
  relative: 0.966

---

GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.503s ± 0.027s
            user: 3.731s | system: 0.547s
  median:   4.499s
  range:    4.478s ... 4.541s
  relative: 1

GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.285s ± 0.031s
            user: 3.504s | system: 0.571s
  median:   4.281s
  range:    4.221s ... 4.328s
  relative: 0.951

---

old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     16.475s ± 0.07s
            user: 14.088s | system: 1.572s
  median:   16.495s
  range:    16.351s ... 16.536s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     15.973s ± 0.013s
            user: 13.558s | system: 1.615s
  median:   15.973s
  range:    15.946s ... 15.99s
  relative: 0.97

---

Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
2024-06-25 12:24:58 +00:00
jade c097ebe66b Merge "Revert "libfetchers: make attribute / URL query handling consistent"" into main 2024-06-25 10:19:52 +00:00
jade 3e151d4d77 Revert "libfetchers: make attribute / URL query handling consistent"
This reverts commit 35eec921af.

Reason for revert: Regressed nix-eval-jobs, and it appears to be this change is buggy/missing a case. It just needs another pass.

Code causing the problem in n-e-j, when invoked with `nix-eval-jobs --flake '.#hydraJobs'`:

```
n-e-j/tests/assets » ../../build/src/nix-eval-jobs --meta --workers 1 --flake .#hydraJobs
warning: unknown setting 'trusted-users'
warning: `--gc-roots-dir' not specified
error: unsupported Git input attribute 'dir'
error: worker error: error: unsupported Git input attribute 'dir'
```

```
  nix::Value *vRoot = [&]() {
        if (args.flake) {
            auto [flakeRef, fragment, outputSpec] =
                nix::parseFlakeRefWithFragmentAndExtendedOutputsSpec(
                    args.releaseExpr, nix::absPath("."));
            nix::InstallableFlake flake{
                {}, state, std::move(flakeRef), fragment, outputSpec,
                {}, {},    args.lockFlags};

            return flake.toValue(*state).first;
        } else {
            return releaseExprTopLevelValue(*state, autoArgs, args);
        }
    }();
```

Inspecting the program behaviour reveals that `dir` was in fact set in the URL going into the fetcher. This is in turn because unlike in the case changed in this commit, it was not erased before handing it to libfetchers, which is probably just a mistake.

```
(rr) up
3  0x00007ffff60262ae in nix::fetchers::Input::fromURL (url=..., requireTree=requireTree@entry=true) at src/libfetchers/fetchers.cc:39
warning: Source file is more recent than executable.
39              auto res = inputScheme->inputFromURL(url, requireTree);
(rr) p url
$1 = (const nix::ParsedURL &) @0x7fffdc874190: {url = "git+file:///home/jade/lix/nix-eval-jobs", 
  base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", authority = std::optional<std::string> = {[contained value] = ""}, 
  path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {["dir"] = "tests/assets"}, fragment = ""}
(rr) up
4  0x00007ffff789d904 in nix::parseFlakeRefWithFragment (url=".#hydraJobs", baseDir=std::optional<std::string> = {...}, 
    allowMissing=allowMissing@entry=false, isFlake=isFlake@entry=true) at src/libexpr/flake/flakeref.cc:179
warning: Source file is more recent than executable.
179                                 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
(rr) p parsedURL
$2 = {url = "git+file:///home/jade/lix/nix-eval-jobs", base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", 
  authority = std::optional<std::string> = {[contained value] = ""}, path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {
    ["dir"] = "tests/assets"}, fragment = ""}
(rr) list
174
175                             if (pathExists(flakeRoot + "/.git/shallow"))
176                                 parsedURL.query.insert_or_assign("shallow", "1");
177
178                             return std::make_pair(
179                                 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")),
180                                 fragment);
181                         }
```

Change-Id: Ib55a882eaeb3e59228857761dc1e3b2e366b0f5e
2024-06-24 22:49:17 +00:00
vigress8 c7af89c797 change shebangs of all .sh scripts to bash
On operating systems where /bin/sh is not Bash, some scripts are invalid
because of bashisms, and building Lix fails with errors like this:
`render-manpage.sh: 3: set: Illegal option -o pipefail`
This modifies all scripts that use a `/bin/sh` shebang to `/usr/bin/env
bash`, including currently POSIX-compliant ones, to prevent any future
confusion.

Change-Id: Ia074cc6db42d40fc59a63726f6194ea0149ea5e0
2024-06-24 14:00:43 -07:00
Robert Hensing d86009bd76 Add build-dir setting, clean up default TMPDIR handling
This is a squash of upstream PRs #10303, #10312 and #10883.

fix: Treat empty TMPDIR as unset

Fixes an instance of

    nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.

... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.

(cherry picked from commit c3fb2aa1f9d1fa756dac38d3588c836c5a5395dc)

fix: Treat empty XDG_RUNTIME_DIR as unset

See preceding commit. Not observed in the wild, but is sensible
and consistent with TMPDIR behavior.

(cherry picked from commit b9e7f5aa2df3f0e223f5c44b8089cbf9b81be691)

local-derivation-goal.cc: Reuse defaultTempDir()

(cherry picked from commit fd31945742710984de22805ee8d97fbd83c3f8eb)

fix: remove usage of XDG_RUNTIME_DIR for TMP

(cherry picked from commit 1363f51bcb24ab9948b7b5093490a009947f7453)

tests/functional: Add count()

(cherry picked from commit 6221770c9de4d28137206bdcd1a67eea12e1e499)

Remove uncalled for message

(cherry picked from commit b1fe388d33530f0157dcf9f461348b61eda13228)

Add build-dir setting

(cherry picked from commit 8b16cced18925aa612049d08d5e78eccbf0530e4)
Change-Id: Ic7b75ff0b6a3b19e50a4ac8ff2d70f15c683c16a
2024-06-24 11:30:32 +03:00
eldritch horrors 2fe9157808 flakes: add --commit-lock-file message test
we had no test to ensure that we generated a commit message at all?

Change-Id: Ic9aa8fde92b83e1ea6f61cd2a21867aa73d4e885
2024-06-23 17:29:40 +00:00
Maximilian Bosch 5f0062285c Merge "libfetchers: make attribute / URL query handling consistent" into main 2024-06-23 15:51:34 +00:00
eldritch horrors ce6cb14995 libutil: return Pid from startProcess, not pid_t
Change-Id: Icc8a15090c77f54ea7d9220aadedcd4a19922814
2024-06-23 11:52:49 +00:00
eldritch horrors a9949f4760 libutil: add some serialize.hh serializer tests
Change-Id: I0116265a18bc44bba16c07bf419af70d5195f07d
2024-06-23 11:52:49 +00:00
eldritch horrors 39a1e248c9 libutil: remove sinkToSource eof callback
this is only used in one place, and only to set a nicer error message on
EndOfFile. the only caller that actually *catches* this exception should
provide an error message in that catch block rather than forcing support
for setting error message so deep into the stack. copyStorePath is never
called outside of PathSubstitutionGoal anyway, which catches everything.

Change-Id: Ifbae8706d781c388737706faf4c8a8b7917ca278
2024-06-23 11:52:49 +00:00
Maximilian Bosch 35eec921af
libfetchers: make attribute / URL query handling consistent
The original idea was to fix lix#174, but for a user friendly solution,
I figured that we'd need more consistency:

* Invalid query params will cause an error, just like invalid
  attributes. This has the following two consequences:

  * The `?dir=`-param from flakes will be removed before the URL to be
    fetched is passed to libfetchers.

  * The tarball fetcher doesn't allow URLs with custom query params
    anymore. I think this was questionable anyways given that an
    arbitrary set of query params was silently removed from the URL you
    wanted to fetch. The correct way is to use an attribute-set
    with a key `url` that contains the tarball URL to fetch.

  * Same for the git & mercurial fetchers: in that case it doesn't even
    matter though: both fetchers added unused query params to the URL
    that's passed from the input scheme to the fetcher (`url2` in the code).
    It turns out that this was never used since the query parameters were
    erased again in `getActualUrl`.

* Validation happens for both attributes and URLs. Previously, a lot of
  fetchers validated e.g. refs/revs only when specified in a URL and
  the validity of attribute names only in `inputFromAttrs`.

  Now, all the validation is done in `inputFromAttrs` and `inputFromURL`
  constructs attributes that will be passed to `inputFromAttrs`.

* Accept all attributes as URL query parameters. That also includes
  lesser used ones such as `narHash`.

  And "output" attributes like `lastModified`: these could be declared
  already when declaring inputs as attribute rather than URL. Now the
  behavior is at least consistent.

  Personally, I think we should differentiate in the future between
  "fetched input" (basically the attr-set that ends up in the lock-file)
  and "unfetched input" earlier: both inputFrom{Attrs,URL} entrypoints
  are probably OK for unfetched inputs, but for locked/fetched inputs
  a custom entrypoint should be used. Then, the current entrypoints
  wouldn't have to allow these attributes anymore.

Change-Id: I1be1992249f7af8287cfc37891ab505ddaa2e8cd
2024-06-22 14:42:43 +02:00
Qyriad fd250c51ed add a basic libmain test for the progress bar rendering
Hooray for leaky abstraction allowing us to test this particular part of
the render pipeline.

Change-Id: Ie0f251ff874f63324e6a9c6388b84ec6507eeae2
2024-06-20 13:56:53 -06:00
Ilya K 7d52d74bbe BrotliDecompressionSource: don't bail out too early
If we've consumed the entire input, that doesn't actually mean we're
done decompressing - there might be more output left. This worked (?)
in most cases because the input and output sizes are pretty comparable,
but sometimes they're not and then things get very funny.

Change-Id: I73435a654a911b8ce25119f713b80706c5783c1b
2024-06-20 09:21:13 +03:00
eldritch horrors c55dcc6c13 filetransfer: return a Source from download()
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)

Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
2024-06-19 10:50:12 +00:00
eldritch horrors 67f778670c libutil: add makeDecompressionSource
Change-Id: Iac7f24d79e24417436b9b5cbefd6af051aeea0a6
2024-06-19 10:50:12 +00:00
alois31 fed34594d8 Merge "libfetchers: represent unfetched submodules consistently" into main 2024-06-19 07:08:19 +00:00
eldritch horrors 0b9a72524a filetransfer: {up,down}load -> transfer
even the transfer function is not all that necessary since there aren't
that many users, but we'll keep it for now. we could've kept both names
but we also kind of want to use `download` for something else very soon

Change-Id: I005e403ee59de433e139e37aa2045c26a523ccbf
2024-06-18 23:58:25 +00:00
alois31 aa00a5a8c9 libfetchers: represent unfetched submodules consistently
Unfetched submodules are included as empty directories in archives, so they end
up as such in the store when fetched in clean mode. Make sure the same happens
in dirty mode too. Fortunately, they are already correctly represented in the
ls-files output, so we just need to make sure to include the empty directory in
our filter.

Fixes: https://github.com/NixOS/nix/issues/6247
Change-Id: I60d06ff360cfa305d081b920838c893c06da801c
2024-06-18 00:54:51 +00:00
jade ce2b48aa41 Merge changes from topic "protocol" into main
* changes:
  libstore client: remove remaining dead code
  libstore: refuse to serialise ancient protocols
  libstore client: remove support for <2.3 clients
  libstore daemon: remove very old protocol support (<2.3)
  Delete old ValidPathInfo test, fix UnkeyedValidPathInfo
  Set up minimum protocol version
2024-06-17 22:08:48 +00:00
eldritch horrors bcb774688f libexpr: add expr memory management
with the prepatory work done this mostly means turning plain pointers
into unique_ptrs, with all the associated churn that necessitates. we
might want to change some of these to box_ptrs at some point as well,
but that would be a semantic change that isn't fully appropriate yet.

Change-Id: I0c238c118617420650432f4ed45569baa3e3f413
2024-06-17 19:46:44 +00:00
eldritch horrors ad5366c2ad libexpr: pass Exprs as references, not pointers
almost all places where Exprs are passed as pointers expect the pointers
to be non-null. pass them as references to encode this constraint in the
type system as well (and also communicate that Exprs must not be freed).

Change-Id: Ia98f166fec3c23151f906e13acb4a0954a5980a2
2024-06-17 19:46:44 +00:00
jade c22a7f50cb libstore: refuse to serialise ancient protocols
We don't want to deal with these at all, let's stop doing so.

(marking this one as the fix commit since its immediate predecessors
aren't the complete fix)
Fixes: #325

Change-Id: Ieea1b0b8ac0f903d1e24e5b3e63cfe12eeec119d
2024-06-16 19:15:08 -07:00
jade 24255748b4 Delete old ValidPathInfo test, fix UnkeyedValidPathInfo
The UnkeyedValidPathInfo test was testing an ancient version but not the
current version. Doesn't make much sense to me.

Change-Id: Ib476a4297d9075f2dcd31a073b3e7b149b2189af
2024-06-16 19:13:51 -07:00
Robert Hensing b588a761fe Merge pull request #10799 from hercules-ci/safer-tab-completion
Add repl completion test

(cherry picked from commit 1e2b26734b4da101247678aec405c9dcfdc33f98)

Change-Id: Ic3de39e71960a05a8676190b1ec9a7f0bb6057f5
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
2024-06-16 04:03:32 +00:00
alois31 1d6fd94cf9 Merge "tests/libcmd: set HOME to a temporary directory" into main 2024-06-13 05:27:55 +00:00
jade 73898cad0e tests/flake-registry: Fix occasional deadlocks
This seems to have been caused by having the wrong PID. I don't know why
it worked before in the sandbox, but the code was definitely wrong
before, so let's just fix it.

Change-Id: I556580bdf614c716566310e975a36daa6d6c9a91
2024-06-12 15:34:23 -07:00
jade d9345d8836 tests: verify that NIX_DAEMON_PACKAGE's version looks like a version
Followup to https://gerrit.lix.systems/c/lix/+/1417 to ensure that this
parser will never take something that doesn't look like a version.

It turns out this problem is less alarming than initially thought
because it only applies to the testsuite in a non-default mode.

Change-Id: I26aba24aaf0215f2b782966314b94784db766266
2024-06-12 15:34:23 -07:00
Pierre Bourdon f7b6552699 [resubmit] flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
-- message from cl/1418 --

The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

-- jade resubmit changes --

This is a resubmission of https://gerrit.lix.systems/c/lix/+/1418, which
was reverted in https://gerrit.lix.systems/c/lix/+/1432 for breaking CI
evaluation without being detected.

I have run `nix flake check -Lv` on this one before submission and it
passes on my machine and crucially without eval errors, so the CI result
should be accurate.

It seems like someone renamed forbiddenDependenciesRegex to
forbiddenDependenciesRegexes in nixpkgs and also changed the type
incompatibly. That's pretty silly, but at least it's just an eval error.

Also, `xonsh` regressed the availability of `xonsh-unwrapped`, but it
was fixed by us in https://github.com/NixOS/nixpkgs/pull/317636, which
is now in our channel, so we update nixpkgs compared to the original
iteration of this to simply get that.

We originally had a regression related to some reorganization of the
nixpkgs lib test suite in which there was broken parameter passing.
This, too, we got quickfixed in nixpkgs, so we don't need any changes
for it: https://github.com/NixOS/nixpkgs/pull/317772

Related: https://gerrit.lix.systems/c/lix/+/1428
Fixes: #385

Change-Id: I26d41ea826fec900ebcad0f82a727feb6bcd28f3
2024-06-12 15:34:22 -07:00
alois31 3c0434999e tests/libcmd: set HOME to a temporary directory
The libcmd unit test creates files (more specifically, the fetcher cache) in
its home directory. In the single-user sandbox, this leads to the creation of
/homeless-shelter, since this is the default HOME and the root is writable.
Unfortunately, this conflicts with the assumption of the functional tests that
this directory does not exist. Use a different home directory to prevent these
test failures, and thus restore the ability to build inside the single-user
sandbox.

Fixes: #365
Change-Id: I4df8c53d043234b95a7c0ac45fc5ee89e8d46aff
2024-06-12 22:13:55 +00:00
jade f432e464dd Merge "tests: fix daemon version in isDaemonNewer function" into main 2024-06-10 23:22:05 +00:00
Pierre Bourdon 9281a12532
tests/nixos/nix-copy: fix NixOS >= 24.05 compatibility
4b128008c5d9fde881ce1b0a25e60ae0415a14d5 in nixpkgs introduced a default
hashedPasswordFile for root in NixOS tests, which takes precedence over
the password option set in the nix-copy test.

Change-Id: Iffaebec5992e50614b854033f0d14312c8d275b5
2024-06-08 17:59:08 +02:00
Mario Rodas a05de58ebd tests: fix daemon version in isDaemonNewer function
Since ad8a4b380e, the version printer returns "nix (Lix, like Nix) 2.x",
hence the `daemonVersion` was being set to the string "like".

Using `compareVersions` with a letter compares them lexicographically:

   builtins.compareVersions "like" "2.12pre20230103"  // => -1
   builtins.compareVersions "like" "2.16.0"           // => -1

This caused that `isDaemonNewer` always returned 1, falsy in Bash terms.
Therefore, the test suite skipped those tests where they use it.

Fixes #324

Change-Id: If6682515bf0bf8b8add641af9a4e98b50a9acb51
2024-06-08 04:20:00 +00:00
Qyriad 06e65e537b build: expose option to enable or disable precompiled std headers
They are enabled by default, and Meson will also prints whether or not
they're enabled at the bottom at the end of configuration.

Change-Id: I48db238510bf9e74340b86f243f4bbe360794281
2024-06-06 12:46:26 -06:00
Pierre Bourdon 6e59b4b407 Merge "tests/nixos: make the tarball-flakes test better reflect real use cases" into main 2024-06-04 15:14:02 +00:00
Pierre Bourdon a3256a9375
tests/nixos: make the tarball-flakes test better reflect real use cases
In most real world cases, the Link header is set on the redirect, not on
the final file. This regressed in Lix earlier and while new unit tests
were added to cover it, this integration test should probably have also
caught it.

Change-Id: I2a9d8d952fff36f2c22cfd751451c2b523f7045c
2024-06-04 08:12:59 +02:00
Olmo Kramer 72d85acba4 nix flake update: add test for multiple inputs from nix#10073
Upstream-PR: https://github.com/NixOS/nix/pull/10073
Change-Id: I53fcb43b387e55439e062e208877afeb88493bb4
2024-06-03 21:50:33 +00:00
Qyriad d374a9908f Merge "build: fix static linking with a hack" into main 2024-06-01 19:17:13 +00:00
raito b8cb7abcf0 chore: rebrand Nix to Lix when it makes sense
Here's my guide so far:

$ rg '((?!(recursive).*) Nix
(?!(daemon|store|expression|Rocks!|Packages|language|derivation|archive|account|user|sandbox|flake).*))'
-g '!doc/' --pcre2

All items from this query have been tackled. For the documentation side:
that's for #162.

Additionally, all remaining references to github.com/NixOS/nix which
were not relevant were also replaced.

Fixes: #148.
Fixes: #162.
Change-Id: Ib3451fae5cb8ab8cd9ac9e4e4551284ee6794545
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-06-01 20:31:24 +02:00
Qyriad e54d4c9381 build: fix static linking with a hack
This causes libstore, libexpr, libfetchers, and libutil to be linked
with -Wl,--whole-archive to executables, when building statically.

libstore for the store backends, libexpr for the primops, libfetchers
for the fetcher backends I assume(?), and libutil for the nix::logger
initializer (which notably shows in pre-main constructors when HOME is
not owned by the user. cursed.).

This workaround should be removed when #359 is fixed.

Fixes #306.

Change-Id: Ie9ef0154e09a6ed97920ee8ab23810ca5e2de84c
2024-05-31 21:47:16 -06:00
jade adedac70fa Merge changes Ifcb0d310,I664366b8,Ibe7cf546 into main
* changes:
  gitignore: delete 90% of it
  build-time: remove 20% more by PCH'ing C++ stdlib
  shellHook: make it actually run
2024-05-31 19:19:29 +00:00
alois31 cf756fdf3c libstore/build: copy ca-certificates too
In b469c6509b, the ca-certificates file was
missed. It should be copied too so that we don't end up bind-mounting a broken
symlink.

Change-Id: Ic9b292d602eb94b0e78f77f2a27a19d24665783c
2024-05-31 07:54:18 +00:00
jade 0f99ed43f1 build-time: remove 20% more by PCH'ing C++ stdlib
It seems like someone implemented precompiled headers a long time ago
and then it never got ported to meson or maybe didn't work at all.

This is, however, blessedly easy to simply implement. I went looking for
`#define` that could affect the result of precompiling the headers, and
as far as I can tell we aren't doing any of that, so this should truly
just be free build time savings.

Previous state:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

New state:
**** Time summary:
Compilation (567 times):
  Parsing (frontend):         1123.0 s
  Codegen & opts (backend):   1078.1 s

I wonder if the "regression" in codegen time is just doing the PCH
operation a few times, because meson does it per-target.

Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30 21:54:21 +00:00
jade 2f104bbe3b Merge "Revert "tests/filetransfer: reënable on Darwin"" into main 2024-05-30 21:53:38 +00:00
jade 533d469875 Revert "tests/filetransfer: reënable on Darwin"
This reverts commit 285bc67318.

Reason for revert: #364

For some reason this broke `main` even though the change we are reverting passed CI! Mysterious, haunted, etc. Needs more debugging, let's turn it off for now.

Change-Id: Ica4819d61cd35b83eb52985bfcb657e858f025a9
2024-05-30 21:38:32 +00:00
jade c161687b5f Merge "tests/filetransfer: reënable on Darwin" into main 2024-05-30 14:58:23 +00:00
jade 285bc67318 tests/filetransfer: reënable on Darwin
Since we put __darwinAllowLocalNetworking in our derivation in
I752b81c85ebeaab4e582ac01c239d69d65580f37, this stuff will just work
fine. I checked our derivation works on the darwin community builder.

Change-Id: I40e3a801d6bb38efede79af4aded65c1e1f57cec
2024-05-29 20:41:22 -07:00
Cole Helbling 031d924116 libutil/args: warn on unknown settings after parsing all flags
Upstream change: https://github.com/NixOS/nix/pull/10701

Change-Id: Icf271df57ec529dd8c64667d1ef9f6dbf02d33d3
2024-05-30 03:07:21 +00:00
jade 562ff516ab Merge changes from topic "libutil-split" into main
* changes:
  util.hh: Delete remaining file and clean up headers
  util.hh: Move nativeSystem to local-derivation-goal.cc
  util.hh: Move stuff to types.hh
  util.cc: Delete remaining file
  util.{hh,cc}: Move ignoreException to error.{hh,cc}
  util.{hh,cc}: Split out namespaces.{hh,cc}
  util.{hh,cc}: Split out users.{hh,cc}
  util.{hh,cc}: Split out strings.{hh,cc}
  util.{hh,cc}: Split out unix-domain-socket.{hh,cc}
  util.{hh,cc}: Split out child.{hh,cc}
  util.{hh,cc}: Split out current-process.{hh,cc}
  util.{hh,cc}: Split out processes.{hh,cc}
  util.{hh,cc}: Split out file-descriptor.{hh,cc}
  util.{hh,cc}: Split out file-system.{hh,cc}
  util.{hh,cc}: Split out terminal.{hh,cc}
  util.{hh,cc}: Split out environment-variables.{hh,cc}
2024-05-30 02:33:05 +00:00
Mario Rodas c71f21da3a Merge "tests: fix functional-timeout" into main 2024-05-29 23:57:49 +00:00
eldritch horrors dd4a2c1759 libstore: fix http abuses no longer working
while refactoring the curl wrapper we inadvertently broken the immutable
flake protocol, because the immutable flake protocol accumulates headers
across the entire redirect chain instead of using only the headers given
in the final response of the chain. this is a problem because Some Known
Providers Of Flake Infrastructure set rel=immutable link headers only in
the penultimate entry of the redirect chain, and curl does not regard it
as worth returning to us via its response header enumeration mechanisms.

fixes #358

Change-Id: I645c3932b465cde848bd6a3565925a1e3cbcdda0
2024-05-29 22:35:29 +00:00
Tom Hubrecht a39ba22ff7 util.hh: Delete remaining file and clean up headers
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29 12:38:51 +02:00
Tom Hubrecht b910551120 util.{hh,cc}: Split out strings.{hh,cc}
Change-Id: I4f642d1046d56b5db26f1b0296ee16a0e02d444a
2024-05-29 11:01:34 +02:00
Tom Hubrecht 9a52e4688c util.{hh,cc}: Split out processes.{hh,cc}
Change-Id: I39280dc40ca3f7f9007bc6c898ffcf760e2238b7
2024-05-29 11:01:34 +02:00
Tom Hubrecht 8cd9aa24a8 util.{hh,cc}: Split out file-descriptor.{hh,cc}
Change-Id: I0dd0f9a9c2003fb887e076127e7f825fd3289c76
2024-05-29 09:54:47 +02:00
Tom Hubrecht 6b5078c815 util.{hh,cc}: Split out file-system.{hh,cc}
Change-Id: Ifa89a529e7e34e7291eca87d802d2f569cf2493e
2024-05-29 09:54:47 +02:00
Tom Hubrecht 81bdf8d2d6 util.{hh,cc}: Split out terminal.{hh,cc}
Change-Id: I9de2296b4012d50f540124001d54d6ca3be4c6da
2024-05-29 09:54:47 +02:00
Tom Hubrecht 6fd6795bc4 util.{hh,cc}: Split out environment-variables.{hh,cc}
Change-Id: Icff0aa33fda5147bd5dbe256a0b9d6a6c8a2c3f6
2024-05-28 11:29:29 +02:00
Mario Rodas ec5f025ec2 tests: fix functional-timeout
grepQuietInvert is a typo introduced by c11836126b.

The test functional-timeout was failing silently because Bash considered
the command-not-found error as truthy.

Change-Id: Ic13829d02ec55d6ecd63a0f4d34ec0d32379609f
2024-05-28 04:20:00 +00:00
alois31 ddfe379a6b Merge "libstore/build: always enable seccomp filtering and no-new-privileges" into main 2024-05-25 04:21:53 +00:00
alois31 f047e4357b libstore/build: always enable seccomp filtering and no-new-privileges
Seccomp filtering and the no-new-privileges functionality improve the security
of the sandbox, and have been enabled by default for a long time. In
#265 it was decided that they
should be enabled unconditionally. Accordingly, remove the allow-new-privileges
(which had weird behavior anyway) and filter-syscall settings, and force the
security features on. Syscall filtering can still be enabled at build time to
support building on architectures libseccomp doesn't support.

Change-Id: Iedbfa18d720ae557dee07a24f69b2520f30119cb
2024-05-24 21:19:29 +00:00
Qyriad 8c06b7b431 libfetchers: log fetches by URL just before they happen
Addresses but does not close #305, as we still need an indicator for
frozen fetches.

Change-Id: Iba34ad42dc1c8772f7da249b90fe794b041bbf73
2024-05-24 15:15:42 -06:00
Pierre Bourdon 6ade981476 Merge "nixos/tests: enable remoteBuilds tests against Nix 2.18" into main 2024-05-24 00:55:25 +00:00
Qyriad 00bf2b105d Merge changes I462a8cf0,I3b0bcea3,I2acd56e7,Ifc149764,I9e2ef170 into main
* changes:
  docs: linkify nix3-build mention in nix-build.md
  build: make internal-api-docs PHONY
  cleanup lookupFileArg
  add docstring to lookupFileArg
  add libcmd test for lookupFileArg
2024-05-23 23:28:42 +00:00
jade c97e17144e packaging: rename nixexpr -> lixexpr and so on
This breaks downstreams linking to us on purpose to make sure that if
someone is linking to Lix they're doing it on purpose and crucially not
mixing up Nix and Lix versions in compatibility code.

We still need to fix the internal includes to follow the same schema so
we can drop the single-level include system entirely. However, this
requires a little more effort.

This adds pkg-config for libfetchers and config.h.

Migration path:
expr.hh      -> lix/libexpr/expr.hh
nix/config.h -> lix/config.h

To apply this migration automatically, remove all `<nix/>` from
includes, so: `#include <nix/expr.hh>` -> `#include <expr.hh>`. Then,
the correct paths will be resolved from the tangled mess, and the
clang-tidy automated fix will work.

Then run the following for out of tree projects:

```
lix_root=$HOME/lix
(cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build')
run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -fix src
```

Related: lix-project/nix-eval-jobs#5
Fixes: #279
Change-Id: I7498e903afa6850a731ef8ce77a70da6b2b46966
2024-05-23 16:45:23 -06:00
Qyriad 0565f97e78 add libcmd test for lookupFileArg
Change-Id: I9e2ef170ffe916f902daec8b5630d29434c5d5f2
2024-05-23 14:11:10 -06:00
Pierre Bourdon 6260563bed
nixos/tests: enable remoteBuilds tests against Nix 2.18
Fixes #321.

Change-Id: I60812aec9f9b68ab742413835c581d3b53432b9b
2024-05-23 21:16:40 +02:00
Pierre Bourdon d1c8fd3b09 Merge "derived-path: refuse built derived path with a non-derivation base" into main 2024-05-18 07:26:26 +00:00
julia 7a3745b076
Deprecate the online flake registries and vendor the default registry
Fixes #183, #110, #116.

The default flake-registry option becomes 'vendored', and refers
to a vendored flake-registry.json file in the install path.

Vendored copy of the flake-registry is from github:NixOS/flake-registry
at commit 9c69f7bd2363e71fe5cd7f608113290c7614dcdd.

Change-Id: I752b81c85ebeaab4e582ac01c239d69d65580f37
2024-05-18 12:27:23 +10:00
Pierre Bourdon 5a1824ebe1
derived-path: refuse built derived path with a non-derivation base
Example: /nix/store/dr53sp25hyfsnzjpm8mh3r3y36vrw3ng-neovim-0.9.5^out

This is nonsensical since selecting outputs can only be done for a
buildable derivation, not for a realised store path. The build worker
side of things ends up crashing with an assertion when trying to handle
such malformed paths.

Change-Id: Ia3587c71fe3da5bea45d4e506e1be4dd62291ddf
2024-05-17 02:16:15 +02:00
julia 5b7dcb3005 Allow enabling core dumps from builds for nix & child processes
Fixes #268

Change-Id: I3f1b0ddf064f891cca8b53229c5c31c74cea3d9f
2024-05-16 17:11:21 -07:00
Maximilian Bosch aabc902040 Merge "nix3-build: show all FOD errors with --keep-going" into main 2024-05-15 18:46:06 +00:00
Maximilian Bosch e146393183
nix3-build: show all FOD errors with --keep-going
Basically I'd expect the same behavior as with `nix-build`, i.e.
with `--keep-going` the hash-mismatch error of each failing
fixed-output derivation is shown.

The approach is derived from `Store::buildPaths` (`entry-point.cc`):
instead of throwing the first build-result, check if there are any build
errors and if so, display all of them and throw after that.

Unfortunately, the BuildResult struct doesn't have an `ErrorInfo`
(there's a FIXME for that at least), so I have to construct my own here.
This is a rather cheap bugfix and I decided against touching too many
parts of libstore for that (also I don't know if that's in line with the
ongoing refactoring work).

Closes #302

Change-Id: I378ab984fa271e6808c6897c45e0f070eb4c6fac
2024-05-15 15:35:18 +02:00
Pierre Bourdon e9ca5c92d2
tests: don't build test plugin shared libs on static builds
This changes the way plugins.sh is excluded to remove the need for
BUILD_SHARED_LIBS along the way.

Change-Id: I19fe55b4a37c49a11fffa61c8a3be7e8d1a51b4d
2024-05-12 23:04:21 +02:00
Artemis Tosini 4b35e6a75e Merge "libstore: Fix sandbox=relaxed" into main 2024-05-12 03:51:19 +00:00
Pierre Bourdon a30c567336
filetransfer: unit test content-encoding handling
Very basic behavior test to ensure that gzip data gets internally
decompressed by the file transfer pipeline.

Change a std::string_view return value in the test harness to
std::string. I wouldn't call myself a C++ beginner and I still managed
to shoot myself in the foot like three times with the lifetime
managements there (e.g. [&] { return an_std_string; } ends up with a
dangling string_view!).

Change-Id: I1360750d4181ce1ca2a3aa4dc0e97e131351c469
2024-05-11 15:58:32 +02:00
eldritch horrors b66451ae7f libstore: de-callback-ify FileTransfer
also add a few more tests for exception propagation behavior. using
packaged_tasks and futures (which only allow a single call to a few
of their methods) introduces error paths that weren't there before.

Change-Id: I42ca5236f156fefec17df972f6e9be45989cf805
2024-05-10 02:21:11 +02:00
Qyriad 54322f09d3 nix3-eval: don't elide top-level errors
Fixes #276.

Change-Id: I83e71beb5c35d6f3b10a4186caa5e52a2f95b510
2024-05-08 13:37:20 -06:00
Théophane Hufschmitt adea821d87
libstore: Fix sandbox=relaxed
The fix for the Darwin vulnerability in ecdbc3b207
also broke setting `__sandboxProfile` when `sandbox=relaxed` or
`sandbox=false`. This cppnix change fixes `sandbox=relaxed` and
adds a suitable test.

Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I40190f44f3e1d61846df1c7b89677c20a1488522
2024-05-08 19:31:43 +00:00
Qyriad b9be46fb31 remove the autoconf+Make buildsystem
We're not using it anymore. Any leftover bugs in the Meson buildsystem
are now just bugs.

Closes #249.

Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-05-07 17:04:30 -06:00