Commit graph

9959 commits

Author SHA1 Message Date
jade 5d31e889d7 Merge "treewide: fix a bunch of lints" into main 2024-08-28 03:40:27 +00:00
Rebecca Turner 0dc486a5bf Merge "Fix comment in getHome" into main 2024-08-26 23:17:04 +00:00
jade 0cc285f87b
treewide: fix a bunch of lints
Fixes:
- Identifiers starting with _ are prohibited
- Some driveby header dependency cleaning which wound up with doing some
  extra fixups.
- Fucking C style casts, man. C++ made these 1000% worse by letting you
  also do memory corruption with them with references.
  - Remove casts to Expr * where ExprBlackHole is an incomplete type by
    introducing an explicitly-cast eBlackHoleAddr as Expr *.
  - An incredibly illegal cast of the text bytes of the StorePath hash
    into a size_t directly. You can't DO THAT.

    Replaced with actually parsing the hash so we get 100% of the bits
    being entropy, then memcpying the start of the hash. If this shows
    up in a profile we should just make the hash parser faster with a
    lookup table or something sensible like that.
  - This horrendous bit of UB which I thankfully slapped a deprecation
    warning on, built, and it didn't trigger anywhere so it was dead
    code and I just deleted it. But holy crap you *cannot* do that.

    inline void mkString(const Symbol & s)
    {
        mkString(((const std::string &) s).c_str());
    }
- Some wrong lints. Lots of wrong macro lints, one wrong
  suspicious-sizeof lint triggered by the template being instantiated
  with only pointers, but the calculation being correct for both
  pointers and not-pointers.
- Exceptions in destructors strike again. I tried to catch the
  exceptions that might actually happen rather than all the exceptions
  imaginable. We can let the runtime hard-kill it on other exceptions
  imo.

Change-Id: I71761620846cba64d66ee7ca231b20c061e69710
2024-08-26 16:13:03 -07:00
Rebecca Turner ca08f1217d
rowan: 0.15.15 -> 0.15.16
This fixes an ambiguous pointer comparison warning.

See: https://github.com/rust-analyzer/rowan/pull/162
Change-Id: Iaac2c8cab0051eb97211893ad547d8dfa8fda560
2024-08-26 11:34:43 -07:00
Rebecca Turner 0582999bd1 Merge "Add ApplyConfigOptions" into main 2024-08-25 22:06:45 +00:00
eldritch horrors 398894b856 libstore: make Goal::ex a shared_ptr
this makes WorkResult copyable, and just all around easier to deal with.
in the future we'll need this to let Goal::work() return a promise for a
WorkResult (or even just a Finished) that can be awaited by other goals.

Change-Id: Ic5a1ce04c5a0f8e683bd00a2ed2b77a2e28989c1
2024-08-25 21:21:55 +00:00
eldritch horrors 30a87b4cd5 libstore: remove unused Goal ctor parameter
Change-Id: I9345fe272d6df5bd592621ce2da369fc1cd36d6d
2024-08-25 20:40:19 +00:00
jade 72f91767a8 Merge "fix: good errors for failures caused by allowSubstitutes" into main 2024-08-25 20:00:58 +00:00
jade 3bf8819fa2 Merge changes Ief8e8ebc,Id3135db0,If1e76169 into main
* changes:
  libutil: delete unused boost context cruft
  build: remove approximately 400 seconds of CPU time (30%)
  fix: use http proxy for s3 access
2024-08-25 19:59:46 +00:00
Rebecca Turner c300efc0e1
Add ApplyConfigOptions
Change-Id: Ic876bcabd0b68e579bbd30ca1755919df43d4813
2024-08-25 12:18:20 -07:00
eldritch horrors cae260a158 libstore: diagnose local build failure in goal
this should be done where we're actually trying to build something, not
in the main worker loop that shouldn't have to be aware of such details

Change-Id: I07276740c0e2e5591a8ce4828a4bfc705396527e
2024-08-25 19:55:47 +02:00
jade 686120ee4a fix: good errors for failures caused by allowSubstitutes
This caused an absolute saga which I would not like anyone else to have
to experience. Let's put in a laser targeted error message that
diagnoses this exact problem.

Fixes: lix-project/lix#484
Change-Id: I2a79f04aeb4a1b67c10115e5e39501d958836298
2024-08-23 17:49:15 -07:00
Rebecca Turner fabc9f29b8
Fix comment in getHome
The logic in the comment is the opposite of the truth.

Change-Id: I64add84539209782ffa46431f3db1fb306d90b3f
2024-08-23 15:15:21 -07:00
Rebecca Turner c5949bfe31 Merge "libutil/config: unify path setting types" into main 2024-08-23 22:09:11 +00:00
jade 7e677d15a4 libutil: delete unused boost context cruft
This was from before we got rid of the boost coroutines. Now we don't
need any of this code.

Change-Id: Ief8e8ebc184f02f48e30cb253a66b540faa56329
2024-08-23 13:23:33 -07:00
jade 9aacf425dc fix: use http proxy for s3 access
I don't know why the AWS sdk disabled it by default. It would be nice
to have test coverage of the s3 store or proxies, but neither currently
exist.

Fixes: lix-project/lix#433
Change-Id: If1e76169a3d66dbec2e926af0d0d0eccf983b97b
2024-08-23 13:23:33 -07:00
jade 87fd6e0095 Merge "Revert "libexpr: Replace regex engine with boost::regex"" into main 2024-08-22 22:34:10 +00:00
jade 9896d309cb Revert "libexpr: Replace regex engine with boost::regex"
This reverts commit 447212fa65.

Reason for revert: Regression in eval behaviour bug-compatibility.

Expected behaviour (Nix 2.18.5, macOS and Linux [libstdc++/libc++]):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ "keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[ "." "keep" ]
```

Actual behaviour (boost::regex):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ ".keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[
  "."
  "keep"
]
```

Bug: lix-project/lix#483
Change-Id: Id462eb8586dcd54856cf095f09b3e3a216955b60
2024-08-22 18:35:11 +00:00
sugar🍬🍭🏳️‍⚧️ f2e7f8bab8 Merge "libexpr: Replace regex engine with boost::regex" into main 2024-08-22 07:20:00 +00:00
sugar🍬🍭🏳️‍⚧️ 447212fa65 libexpr: Replace regex engine with boost::regex
This avoids C++'s standard library regexes, which aren't the same
across platforms, and have many other issues, like using stack
so much that they stack overflow when processing a lot of data.

To avoid backwards and forward compatibility issues, regexes are
processed using a function converting libstdc++ regexes into Boost
regexes, escaping characters that Boost needs to have escaped, and
rejecting features that Boost has and libstdc++ doesn't.

Related context:

- Original failed attempt to use `boost::regex` in CppNix, failed due to
  boost icu dependency being large (disabling ICU is no longer necessary
  because linking ICU requires using a different header file,
  `boost/regex/icu.hpp`): https://github.com/NixOS/nix/pull/3826

- An attempt to use PCRE, rejected due to providing less backwards
  compatibility with `std::regex` than `boost::regex`:
  https://github.com/NixOS/nix/pull/7336

- Second attempt to use `boost::regex`, failed due to `}` regex failing
  to compile (dealt with by writing a wrapper that parses a regular
  expression and escapes `}` characters):
  https://github.com/NixOS/nix/pull/7762

Closes #34. Closes #476.

Change-Id: Ieb0eb9e270a93e4c7eed412ba4f9f96cb00a5fa4
2024-08-22 03:17:55 +02:00
jade 651cc0e5b4 fix: build with meson 1.5 also
nixpkgs delivered us the untimely gift of a meson 1.5 upgrade, which
*does* make our lives easier by allowing us to delete wrap generation
code, but it does so at the cost of renaming all rust crates in such a
way that the wrap logic cannot tolerate the new names on the old meson
version 😭.

It also means that support burden for this is going to be atrocious
until we either give in and vendor meson 1.5 or we make a CI target for
it. Neither seems appealing, though the latter is not super absurd for
ensuring we don't break nixpkgs unstable.

This commit causes meson 1.5 to ignore the .wrap files in subprojects/
entirely (since they have the wrong names lol) and instead use
Cargo.lock, so it now hard-depends on our workspace reshuffling
improvement.

It also deletes the hack that we were using to get the sources of Cargo
deps into meson by using a feature that went unnoticed when this code
was originally written: MESON_PACKAGE_CACHE_DIR:
8a202de6ec/mesonbuild/wrap/wrap.py (L490-L502)

Change-Id: I7a28f12fc2812c6ed7537b60bc3025c141a05874
2024-08-21 17:09:10 +00:00
jade dba615098d build: move to a Cargo workspace
This is purely to let Cargo's dependency resolver do stuff for us, we do
not actually intend to build this stuff with Cargo to begin with.

Change-Id: I4c08d55595c7c27b7096375022581e1e34308a87
2024-08-21 17:09:10 +00:00
alois31 e3c289dbe9
libutil/config: unify path setting types
There have been multiple setting types for paths that are supposed to be
canonicalised, depending on whether zero or one, one, or any number of paths is
to be specified. Naturally, they behaved in slightly different ways in the
code. Simplify things by unifying them and removing special behaviour (mainly
the "multiple paths type can coerce to boolean" thing).

Change-Id: I7c1ce95e9c8e1829a866fb37d679e167811e9705
2024-08-21 17:57:23 +02:00
piegames e38410799b Merge "libexpr: Soft-deprecate ancient let syntax" into main 2024-08-21 11:22:48 +00:00
piegames 0edfea450b libexpr: Soft-deprecate ancient let syntax
Change-Id: I6802b26f038578870ea1fa1ed298f0c4b1f29c4a
2024-08-21 12:59:03 +02:00
jade 3cbbe22fab Merge "flake: fix compiler warning" into main 2024-08-21 10:44:01 +00:00
piegames 7210ed1b87 libexpr: Soft-deprecate __overrides
Change-Id: I787e69e1dad6edc5ccdb747b74a9ccd6e8e13bb3
2024-08-21 06:55:52 +00:00
jade c25c43d8c8 flake: fix compiler warning
GCC was complaining, rightfully, about mixed-sign comparisons in there.
I removed some extra sign mixing too.

Change-Id: I949a618c7405c23d4dc3fd17440ea2d7b5c22c9d
2024-08-20 16:13:17 -07:00
jade 736b5d5913 lix-doc: move under src/
This is required to make more meson stuff easier/possible, and honestly
it *is* now Lix sources anyhow.

Change-Id: Ia6c38fabce9aa5c53768745ee38c5cf344f5c226
2024-08-20 13:38:46 -06:00
Qyriad 95863b258b build: build lix-doc with Meson! 🎉
lix-doc is now built with Meson, with lix-doc's dependencies built as
Meson subprojects, either fetched on demand with .wrap files, or fetched
in advance by Nix with importCargoLock. It even builds statically.

Fixes #256.

Co-authored-by: Lunaphied <lunaphied@lunaphied.me>
Co-authored-by: Jade Lovelace <lix@jade.fyi>

Change-Id: I3a4731ff13278e7117e0316bc0d7169e85f5eb0c
2024-08-20 17:21:13 +00:00
Yureka f1533160aa Merge "libutil: fix conditional for close_range availability" into main 2024-08-20 07:55:01 +00:00
Yureka df49d37b71 libutil: fix conditional for close_range availability
This check is wrong and would cause the close_range() function being called even when it's not available

Change-Id: Ide65b36830e705fe772196c37349873353622761
2024-08-20 08:58:25 +02:00
Maximilian Bosch 040e783232 flake: don't refetch unmodified inputs by recursive follows
Closes #460

I managed to trigger the issue by having the following inputs (shortened):

    authentik-nix.url = "github:nix-community/authentik-nix";
    authentik-nix.inputs.poetry2nix.inputs.nixpkgs.follows = "nixpkgs";

When evaluating this using

    nix-eval-jobs --flake .#hydraJobs

I got the following error:

    error: cannot update unlocked flake input 'authentik-nix/poetry2nix' in pure mode

The issue we have here is that `authentik-nix/poetry2nix` was written
into the `overrideMap` which caused Nix to assume it's a new input and
tried to refetch it (#460) or errored out in pure mode
(nix-eval-jobs / Hydra).

The testcase unfortunately only involves checking for the output log
and makes sure that something *is* logged on the first fetch so that
the test doesn't rot when the logging changes since I didn't
manage to trigger the error above with the reproducer from #460. In
fact, I only managed to trigger the `cannot update unlocked flake input`
error in this context with `nix-eval-jobs`.

Change-Id: Ifd00091eec9a0067ed4bb3e5765a15d027328807
2024-08-19 19:57:12 +00:00
eldritch horrors e727dbc3a3 libstore: un-enable_shared_from_this Goal
it's no longer needed for anything, and not even a great idea.

Change-Id: Ia7a59e1e3f9d8f4ad2ac3b054e38485157c210a6
2024-08-19 09:13:44 +00:00
eldritch horrors b40369942c libstore: make Worker::childStarted private
this can be a proper WorkResult now. childTerminated is unfortunately a
lot more stubborn and won't be made private for quite a while yet. once
we can get rid of the Worker poll loop that *should* be possible though

Change-Id: I2218df202da5cb84e852f6a37e4c20367495b617
2024-08-19 09:13:44 +00:00
eldritch horrors fca523d661 libstore: turn HookReply into a variant type
we'll need this once we want to pass extra information out of accepting
replies, such as fd sets or possibly even async output reader promises.

Change-Id: I5e2f18cdb80b0d2faf3067703cc18bd263329b3f
2024-08-19 09:13:44 +00:00
eldritch horrors 5e9db09761 libstore: downsize hook pipes
don't keep fds open we're not using. currently this does not cause any
problems, but it does increase the size of our fd table needlessly and
in the future, when we have proper async processing, having builderOut
open in the daemon once the hook has been fully started is problematic

Change-Id: I6e7fb773b280b042873103638d3e04272ca1e4fc
2024-08-19 09:13:44 +00:00
eldritch horrors e513cd2beb libstore: run childStarted as late as possible
otherwise we *technically* give away the output fds before we've read them.

Change-Id: I6ad0d6a1bb553ecfcdd7708f50d34142a425374d
2024-08-19 09:13:44 +00:00
eldritch horrors fb8eb539fc libstore: move respect-timeoutiness to goal method
this is useless to do on the face of it, but it'll make it easier to
convert the entire output handling to use async io and promises soon

Change-Id: I2d1eb62c4bbf8f57bd558b9599c08710a389b1a8
2024-08-19 09:13:44 +00:00
eldritch horrors 5cbca85535 libstore: clarify that build log fd and hook log fd are different
only DerivationGoal can set the hook to anything at all. it always sets
buildOutFD to something that is not related to fromHook in any way, and
mixing the two would have rather dire consequences for log consistency.

Change-Id: Ida86727fd1cd5e1ecd78f07f3bde330a346658a8
2024-08-18 22:44:11 +00:00
eldritch horrors e2d330aeed libstore: remove DerivationGoal::isReadDesc
all derivation goals need a log fd of some description. let's save this
single fd in a dedicated pointer field for all subclasses so that later
we have just the one spot to change if we turn this into async promises

Change-Id: If223adf90909247363fb823d751cae34d25d0c0b
2024-08-18 22:04:06 +00:00
piegames 007211e7a2 libutil: Optimize feature checks
Instead of doing a linear search on an std::set, we use a bitset enum.

Change-Id: Ide537f6cffdd16d06e59aaeb2e4ac0acb6493421
2024-08-18 16:56:49 +00:00
eldritch horrors 7506d680ac libstore: don't ignore max-build-log-size for ssh-ng
Change-Id: Ieab14662bea6e6f5533325f0e945147be998f9a2
2024-08-18 09:10:05 +00:00
eldritch horrors 38f550708d libstore: add explicit in-build-slot-ness to goals
we don't need to expose information about how busy a Worker is if the
worker can instead tell its work items whether they are in a slot. in
the future we might use this to not start items waiting for a slot if
no slots are currently available, but that requires more preparation.

Change-Id: Ibe01ac536da7e6d6f80520164117c43e772f9bd9
2024-08-18 09:10:05 +00:00
eldritch horrors 176e1058f1 libstore: remove method without definition
Change-Id: I676411752a4b1777045d7211ac1176693f1a3d7d
2024-08-18 09:10:05 +00:00
eldritch horrors 91a74ba82a libstore: remove unused includes in worker code
Change-Id: I6c7fccc4e710e23a22faae2669cb75f2f6da27b4
2024-08-18 09:10:05 +00:00
eldritch horrors b66fd9ff4b libstore: make Worker::removeGoal private
Change-Id: I8583d9ff752f702a10ec52b0330b0d4d4d2614fa
2024-08-18 09:10:05 +00:00
piegames 278fddc317 libexpr: Deprecate URL literals
Closes #437.

Change-Id: I9f67fc965bb4a7e7fd849e5067ac1cb3bab064cd
2024-08-17 20:31:57 +02:00
piegames 49d61b2e4b libexpr: Introduce Deprecated features
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437

Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
2024-08-17 19:47:51 +02:00
Artemis Tosini b016eb0895 Merge "libutil: Add bindPath function from libstore" into main 2024-08-13 19:39:10 +00:00