Commit graph

9950 commits

Author SHA1 Message Date
jade 72f91767a8 Merge "fix: good errors for failures caused by allowSubstitutes" into main 2024-08-25 20:00:58 +00:00
jade 3bf8819fa2 Merge changes Ief8e8ebc,Id3135db0,If1e76169 into main
* changes:
  libutil: delete unused boost context cruft
  build: remove approximately 400 seconds of CPU time (30%)
  fix: use http proxy for s3 access
2024-08-25 19:59:46 +00:00
eldritch horrors cae260a158 libstore: diagnose local build failure in goal
this should be done where we're actually trying to build something, not
in the main worker loop that shouldn't have to be aware of such details

Change-Id: I07276740c0e2e5591a8ce4828a4bfc705396527e
2024-08-25 19:55:47 +02:00
jade 686120ee4a fix: good errors for failures caused by allowSubstitutes
This caused an absolute saga which I would not like anyone else to have
to experience. Let's put in a laser targeted error message that
diagnoses this exact problem.

Fixes: lix-project/lix#484
Change-Id: I2a79f04aeb4a1b67c10115e5e39501d958836298
2024-08-23 17:49:15 -07:00
Rebecca Turner c5949bfe31 Merge "libutil/config: unify path setting types" into main 2024-08-23 22:09:11 +00:00
jade 7e677d15a4 libutil: delete unused boost context cruft
This was from before we got rid of the boost coroutines. Now we don't
need any of this code.

Change-Id: Ief8e8ebc184f02f48e30cb253a66b540faa56329
2024-08-23 13:23:33 -07:00
jade 9aacf425dc fix: use http proxy for s3 access
I don't know why the AWS sdk disabled it by default. It would be nice
to have test coverage of the s3 store or proxies, but neither currently
exist.

Fixes: lix-project/lix#433
Change-Id: If1e76169a3d66dbec2e926af0d0d0eccf983b97b
2024-08-23 13:23:33 -07:00
jade 87fd6e0095 Merge "Revert "libexpr: Replace regex engine with boost::regex"" into main 2024-08-22 22:34:10 +00:00
jade 9896d309cb Revert "libexpr: Replace regex engine with boost::regex"
This reverts commit 447212fa65.

Reason for revert: Regression in eval behaviour bug-compatibility.

Expected behaviour (Nix 2.18.5, macOS and Linux [libstdc++/libc++]):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ "keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[ "." "keep" ]
```

Actual behaviour (boost::regex):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ ".keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[
  "."
  "keep"
]
```

Bug: lix-project/lix#483
Change-Id: Id462eb8586dcd54856cf095f09b3e3a216955b60
2024-08-22 18:35:11 +00:00
sugar🍬🍭🏳️‍⚧️ f2e7f8bab8 Merge "libexpr: Replace regex engine with boost::regex" into main 2024-08-22 07:20:00 +00:00
sugar🍬🍭🏳️‍⚧️ 447212fa65 libexpr: Replace regex engine with boost::regex
This avoids C++'s standard library regexes, which aren't the same
across platforms, and have many other issues, like using stack
so much that they stack overflow when processing a lot of data.

To avoid backwards and forward compatibility issues, regexes are
processed using a function converting libstdc++ regexes into Boost
regexes, escaping characters that Boost needs to have escaped, and
rejecting features that Boost has and libstdc++ doesn't.

Related context:

- Original failed attempt to use `boost::regex` in CppNix, failed due to
  boost icu dependency being large (disabling ICU is no longer necessary
  because linking ICU requires using a different header file,
  `boost/regex/icu.hpp`): https://github.com/NixOS/nix/pull/3826

- An attempt to use PCRE, rejected due to providing less backwards
  compatibility with `std::regex` than `boost::regex`:
  https://github.com/NixOS/nix/pull/7336

- Second attempt to use `boost::regex`, failed due to `}` regex failing
  to compile (dealt with by writing a wrapper that parses a regular
  expression and escapes `}` characters):
  https://github.com/NixOS/nix/pull/7762

Closes #34. Closes #476.

Change-Id: Ieb0eb9e270a93e4c7eed412ba4f9f96cb00a5fa4
2024-08-22 03:17:55 +02:00
jade 651cc0e5b4 fix: build with meson 1.5 also
nixpkgs delivered us the untimely gift of a meson 1.5 upgrade, which
*does* make our lives easier by allowing us to delete wrap generation
code, but it does so at the cost of renaming all rust crates in such a
way that the wrap logic cannot tolerate the new names on the old meson
version 😭.

It also means that support burden for this is going to be atrocious
until we either give in and vendor meson 1.5 or we make a CI target for
it. Neither seems appealing, though the latter is not super absurd for
ensuring we don't break nixpkgs unstable.

This commit causes meson 1.5 to ignore the .wrap files in subprojects/
entirely (since they have the wrong names lol) and instead use
Cargo.lock, so it now hard-depends on our workspace reshuffling
improvement.

It also deletes the hack that we were using to get the sources of Cargo
deps into meson by using a feature that went unnoticed when this code
was originally written: MESON_PACKAGE_CACHE_DIR:
8a202de6ec/mesonbuild/wrap/wrap.py (L490-L502)

Change-Id: I7a28f12fc2812c6ed7537b60bc3025c141a05874
2024-08-21 17:09:10 +00:00
jade dba615098d build: move to a Cargo workspace
This is purely to let Cargo's dependency resolver do stuff for us, we do
not actually intend to build this stuff with Cargo to begin with.

Change-Id: I4c08d55595c7c27b7096375022581e1e34308a87
2024-08-21 17:09:10 +00:00
alois31 e3c289dbe9
libutil/config: unify path setting types
There have been multiple setting types for paths that are supposed to be
canonicalised, depending on whether zero or one, one, or any number of paths is
to be specified. Naturally, they behaved in slightly different ways in the
code. Simplify things by unifying them and removing special behaviour (mainly
the "multiple paths type can coerce to boolean" thing).

Change-Id: I7c1ce95e9c8e1829a866fb37d679e167811e9705
2024-08-21 17:57:23 +02:00
piegames e38410799b Merge "libexpr: Soft-deprecate ancient let syntax" into main 2024-08-21 11:22:48 +00:00
piegames 0edfea450b libexpr: Soft-deprecate ancient let syntax
Change-Id: I6802b26f038578870ea1fa1ed298f0c4b1f29c4a
2024-08-21 12:59:03 +02:00
jade 3cbbe22fab Merge "flake: fix compiler warning" into main 2024-08-21 10:44:01 +00:00
piegames 7210ed1b87 libexpr: Soft-deprecate __overrides
Change-Id: I787e69e1dad6edc5ccdb747b74a9ccd6e8e13bb3
2024-08-21 06:55:52 +00:00
jade c25c43d8c8 flake: fix compiler warning
GCC was complaining, rightfully, about mixed-sign comparisons in there.
I removed some extra sign mixing too.

Change-Id: I949a618c7405c23d4dc3fd17440ea2d7b5c22c9d
2024-08-20 16:13:17 -07:00
jade 736b5d5913 lix-doc: move under src/
This is required to make more meson stuff easier/possible, and honestly
it *is* now Lix sources anyhow.

Change-Id: Ia6c38fabce9aa5c53768745ee38c5cf344f5c226
2024-08-20 13:38:46 -06:00
Qyriad 95863b258b build: build lix-doc with Meson! 🎉
lix-doc is now built with Meson, with lix-doc's dependencies built as
Meson subprojects, either fetched on demand with .wrap files, or fetched
in advance by Nix with importCargoLock. It even builds statically.

Fixes #256.

Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Co-authored-by: Jade Lovelace <lix@jade.fyi>

Change-Id: I3a4731ff13278e7117e0316bc0d7169e85f5eb0c
2024-08-20 17:21:13 +00:00
Yureka f1533160aa Merge "libutil: fix conditional for close_range availability" into main 2024-08-20 07:55:01 +00:00
Yureka df49d37b71 libutil: fix conditional for close_range availability
This check is wrong and would cause the close_range() function being called even when it's not available

Change-Id: Ide65b36830e705fe772196c37349873353622761
2024-08-20 08:58:25 +02:00
Maximilian Bosch 040e783232 flake: don't refetch unmodified inputs by recursive follows
Closes #460

I managed to trigger the issue by having the following inputs (shortened):

    authentik-nix.url = "github:nix-community/authentik-nix";
    authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";

When evaluating this using

    nix-eval-jobs --flake .#hydraJobs

I got the following error:

    error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode

The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).

The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.

Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
2024-08-19 19:57:12 +00:00
eldritch horrors e727dbc3a3 libstore: un-enable_shared_from_this Goal
it's no longer needed for anything, and not even a great idea.

Change-Id: Ia7a59e1e3f9d8f4ad2ac3b054e38485157c210a6
2024-08-19 09:13:44 +00:00
eldritch horrors b40369942c libstore: make Worker::childStarted private
this can be a proper WorkResult now. childTerminated is unfortunately a
lot more stubborn and won't be made private for quite a while yet. once
we can get rid of the Worker poll loop that *should* be possible though

Change-Id: I2218df202da5cb84e852f6a37e4c20367495b617
2024-08-19 09:13:44 +00:00
eldritch horrors fca523d661 libstore: turn HookReply into a variant type
we'll need this once we want to pass extra information out of accepting
replies, such as fd sets or possibly even async output reader promises.

Change-Id: I5e2f18cdb80b0d2faf3067703cc18bd263329b3f
2024-08-19 09:13:44 +00:00
eldritch horrors 5e9db09761 libstore: downsize hook pipes
don't keep fds open we're not using. currently this does not cause any
problems, but it does increase the size of our fd table needlessly and
in the future, when we have proper async processing, having builderOut
open in the daemon once the hook has been fully started is problematic

Change-Id: I6e7fb773b280b042873103638d3e04272ca1e4fc
2024-08-19 09:13:44 +00:00
eldritch horrors e513cd2beb libstore: run childStarted as late as possible
otherwise we *technically* give away the output fds before we've read them.

Change-Id: I6ad0d6a1bb553ecfcdd7708f50d34142a425374d
2024-08-19 09:13:44 +00:00
eldritch horrors fb8eb539fc libstore: move respect-timeoutiness to goal method
this is useless to do on the face of it, but it'll make it easier to
convert the entire output handling to use async io and promises soon

Change-Id: I2d1eb62c4bbf8f57bd558b9599c08710a389b1a8
2024-08-19 09:13:44 +00:00
eldritch horrors 5cbca85535 libstore: clarify that build log fd and hook log fd are different
only DerivationGoal can set the hook to anything at all. it always sets
buildOutFD to something that is not related to fromHook in any way, and
mixing the two would have rather dire consequences for log consistency.

Change-Id: Ida86727fd1cd5e1ecd78f07f3bde330a346658a8
2024-08-18 22:44:11 +00:00
eldritch horrors e2d330aeed libstore: remove DerivationGoal::isReadDesc
all derivation goals need a log fd of some description. let's save this
single fd in a dedicated pointer field for all subclasses so that later
we have just the one spot to change if we turn this into async promises

Change-Id: If223adf90909247363fb823d751cae34d25d0c0b
2024-08-18 22:04:06 +00:00
piegames 007211e7a2 libutil: Optimize feature checks
Instead of doing a linear search on an std::set, we use a bitset enum.

Change-Id: Ide537f6cffdd16d06e59aaeb2e4ac0acb6493421
2024-08-18 16:56:49 +00:00
eldritch horrors 7506d680ac libstore: don't ignore max-build-log-size for ssh-ng
Change-Id: Ieab14662bea6e6f5533325f0e945147be998f9a2
2024-08-18 09:10:05 +00:00
eldritch horrors 38f550708d libstore: add explicit in-build-slot-ness to goals
we don't need to expose information about how busy a Worker is if the
worker can instead tell its work items whether they are in a slot. in
the future we might use this to not start items waiting for a slot if
no slots are currently available, but that requires more preparation.

Change-Id: Ibe01ac536da7e6d6f80520164117c43e772f9bd9
2024-08-18 09:10:05 +00:00
eldritch horrors 176e1058f1 libstore: remove method without definition
Change-Id: I676411752a4b1777045d7211ac1176693f1a3d7d
2024-08-18 09:10:05 +00:00
eldritch horrors 91a74ba82a libstore: remove unused includes in worker code
Change-Id: I6c7fccc4e710e23a22faae2669cb75f2f6da27b4
2024-08-18 09:10:05 +00:00
eldritch horrors b66fd9ff4b libstore: make Worker::removeGoal private
Change-Id: I8583d9ff752f702a10ec52b0330b0d4d4d2614fa
2024-08-18 09:10:05 +00:00
piegames 278fddc317 libexpr: Deprecate URL literals
Closes #437.

Change-Id: I9f67fc965bb4a7e7fd849e5067ac1cb3bab064cd
2024-08-17 20:31:57 +02:00
piegames 49d61b2e4b libexpr: Introduce Deprecated features
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437

Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
2024-08-17 19:47:51 +02:00
Artemis Tosini b016eb0895 Merge "libutil: Add bindPath function from libstore" into main 2024-08-13 19:39:10 +00:00
jade b9ed79c99a libutil: deal with Linux systems that do not implement close_range
Seems a little bit Rich that musl does not implement close_range because
they suspect that the system call itself is a bad idea, so they uhhhh
are considering not implementing a wrapper. Let's just fix the problem
at hand by writing our own wrapper.

Change-Id: I1f8e5858e4561d58a5450503d9c4585aded2b216
2024-08-10 16:11:58 -07:00
jade b15d5cc6ee nix: remove explosions if you have a window size less than four
Turns out strings do not like being resized to -4.

This was discovered while messing with the tests to remove unbuffer and
trying stdbuf instead. Turns out that was not the right approach.

This basically rewrites the handling of this case to be much more
correct, and fixes a bug where with small window sizes where it would
ALSO truncate the attr names in addition to the optional descriptions.

Change-Id: Ifd1beeaffdb47cbb5f4a462b183fcb6c0ff6c524
2024-08-10 16:11:58 -07:00
jade 292567e0b0 fix: check if it is a Real terminal, not just if it is a terminal
This will stop printing stuff to dumb terminals that they don't support.

I've overall audited usage of isatty and replaced the ones with intent
to mean "is a Real terminal" with checking for that. I've also caught a
case of carelessly assuming "is a tty" means "should be colour" in
nix-env.

Change-Id: I6d83725d9a2d932ac94ff2294f92c0a1100d23c9
2024-08-10 16:07:21 -07:00
eldritch horrors c7d97802e4 libutil: rename and optimize closeMostFDs
this is only used to close non-stdio files in derivation sandboxes. we
may as well encode that in its name, drop the unnecessary integer set,
and use close_range to deal with the actual closing of files. not only
is this clearer, it also makes sandbox setup on linux fast by 1ms each

Change-Id: Id90e259a49c7bc896189e76bfbbf6ef2c0bcd3b2
2024-08-09 19:59:17 +00:00
eldritch horrors 35a2f28a46 libstore: deprecate the build-hook setting
implementing a build hook is pretty much impossible without either being
a nix, or blindly forwarding the important bits of all build requests to
some kind of nix. we've found no uses of build-hook in the wild, and the
build-hook protocol (apart from being entirely undocumented) is not able
to convey any kind of versioning information between hook and daemon. if
we want to upgrade this infrastructure (which we do), this must not stay

Change-Id: I1ec4976a35adf8105b8ca9240b7984f8b91e147e
2024-08-09 19:30:45 +00:00
jade 790d1079e1 Merge changes Ib7c80826,I636f8a71,I67669b98 into main
* changes:
  perl: un-autos your conf
  build: declare all the deps as -isystem
  darwin: workaround PROC_PIDLISTFDS on processes with no fds
2024-08-09 19:24:29 +00:00
Qyriad 346e340cbf Merge "libexpr: move Value implementations out of eval.cc" into main 2024-08-09 14:25:13 +00:00
eldritch horrors 5d4686bcd5 libutil: allow marking settings as deprecated
this is a bit of a hack, but it's apparently the cleanest way of doing
this in the absence of any kind of priority/provenance information for
values of some given setting. we'll need this to deprecate build-hook.

Change-Id: I03644a9c3f17681c052ecdc610b4f1301266ab9e
2024-08-09 11:33:09 +00:00
eldritch horrors baa4fda340 main: require argv[0]
sure, linux has been providing argv[0] by default for a while now. other
OSes may not be as forthcoming though, and relying on the OS to create a
world in which we can just make assumptions we could test for instead is
unnecessarily lazy. we *could* default argv0, but that's a little silly.

notably we abort instead of returning normally to avoid confusions where
a caller interprets our exit status like a Worker build results bitmask.

Change-Id: Id73f8cd0a630293b789c59a8c4b0c4a2b936b505
2024-08-09 11:33:09 +00:00