Commit graph

16078 commits

Author SHA1 Message Date
jade 1437d3df15 darwin: workaround PROC_PIDLISTFDS on processes with no fds
This has been causing various seemingly spurious CI failures as well as
some failures on people running tests on beta builds.

lix> ++(nix-collect-garbage-dry-run.sh:20) nix-store --gc --print-dead
lix> ++(nix-collect-garbage-dry-run.sh:20) wc -l
lix> finding garbage collector roots...
lix> error: Listing pid 87261 file descriptors: Undefined error: 0

There is no real way to write a proper test for this, other than to
start a process like the following:

int main(void) {
    for (int i = 0; i < 1000; ++i) {
        close(i);
    }
    sleep(10000);
}

and then let Lix's gc look at it.

I have a relatively high confidence this *will* fix the problem since I
have manually confirmed the behaviour of the libproc call is
as-unexpected, and it would perfectly explain the observed symptom.

Fixes: lix-project/lix#446
Change-Id: I67669b98377af17895644b3bafdf42fc33abd076
2024-08-07 02:52:00 -07:00
jade 529eed74c4 Merge changes I0fc80718,Ia182b86f,I355f82cb,I8a9b58fa,Id89f8a1f, ... into main
* changes:
  tree-wide: fix various lint warnings
  flake & doxygen: update tagline
  nix flake metadata: print modified dates for input flakes
  cli: eat terminal codes from stdout also
  Implement forcing CLI colour on, and document it better
  manual: fix a syntax error in redirects.js that made it not do anything
  misc docs/meson tidying
  build: implement clang-tidy using our plugin
2024-08-07 00:50:30 +00:00
alois31 2c48460850
libstore/linux: precompile and cache the seccomp BPF
The growth of the seccomp filter in 127ee1a101
made its compilation time significant (roughly 10 milliseconds have been
measured on one machine). For this reason, it is now precompiled and cached in
the parent process so that this overhead is not hit for every single build. It
is still not optimal when going through the daemon, because compilation still
happens once per client, but it's better than before and doing it only once for
the entire daemon requires excessive crimes with the current architecture.

Fixes: lix-project/lix#461
Change-Id: I2277eaaf6bab9bd74bbbfd9861e52392a54b61a3
2024-08-06 19:10:33 +02:00
alois31 403fa9e2b6
libstore/linux: compile the seccomp BPF explicitly
This is a preparation for precompiling the filter, which is done separately.
The behaviour should be unchanged for now.

Change-Id: I899aa7242962615949208597aca88913feba1cb8
2024-08-06 18:31:40 +02:00
alois31 741d3b441c
libstore: add LocalDerivationGoal setupSyscallFilter hook
The seccomp setup code was a huge chunk of conditionally compiled
platform-specific code. For this reason, it is appropriate to move it to the
platform-specific implementation file. Ideally its setup could be moved a bit
to make it happen at the same place as the Darwin restrictions, but that change
is going to be less mechanical.

Change-Id: I496aa3c4fabf34656aba1e32b0089044ab5b99f8
2024-08-06 18:27:09 +02:00
alois31 f84997cbef
package: don't hide system-wide manual pages
When MANPATH is unset or contains an empty component, a reasonable default is
used. Previously (after 3dced96741), when MANPATH
was unset, the shell hook would only place a location containing the Lix manual
pages there, and system-wide manual pages would become unavailable in the
development shell, which is undesired. Fix the issue by including an empty
component in this case.

Change-Id: Ib3c67a831d709fe2a87520e15917eebb59397bd1
2024-08-06 17:18:05 +02:00
jade ca9d3e6e00 tree-wide: fix various lint warnings
Change-Id: I0fc80718eb7e02d84cc4b5d5deec4c0f41116134
2024-08-04 20:55:45 -07:00
jade 9238e62ae6 flake & doxygen: update tagline
This tagline was left over from CppNix and we should make it tastier.

Change-Id: Ia182b86f6e751591be71a50521992ad73c7b38b5
2024-08-04 20:41:19 -07:00
jade bd1344ec54 nix flake metadata: print modified dates for input flakes
This was always in the lock file and we can simply actually print it.

The test for this is a little bit silly but it should correctly
control for my daring to exercise timezone code *and* locale code in a
test, which I strongly suspect nobody dared do before.

Sample (abridged):
```
Path:          /nix/store/gaxb42z68bcr8lch467shvmnhjjzgd8b-source
Last modified: 1970-01-01 00:16:40
Inputs:
├───flake-compat: github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33
│   Last modified: 2023-10-04 13:37:54
├───flake-utils: github:numtide/flake-utils/b1d9ab70662946ef0850d488da1c9019f3a9752a
│   Last modified: 2024-03-11 08:33:50
│   └───systems: github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e
│       Last modified: 2023-04-09 08:27:08
```

Change-Id: I355f82cb4b633974295375ebad646fb6e2107f9b
2024-08-04 20:41:19 -07:00
jade 5f0ef50077 cli: eat terminal codes from stdout also
This *should* be sound, plus or minus the amount that the terminal code
eating code is messed up already.

This is useful for testing CLI output because it will strip the escapes
enough to just shove the expected output in a file.

Change-Id: I8a9b58fafb918466ac76e9ab585fc32fb9294819
2024-08-04 20:41:19 -07:00
jade 378ec5fb06 Implement forcing CLI colour on, and document it better
This is necessary to make some old tests work when testing colour
against non-interactive outputs.

Change-Id: Id89f8a1f45c587fede35a69db85f7a52f2c0a981
2024-08-04 20:41:19 -07:00
jade 700762d8b2 manual: fix a syntax error in redirects.js that made it not do anything
lol lmao

Let's put in a syntax checker in CI so we do not have to deal with this
nonsense ever again.

Change-Id: I0fe875e0cfc59ab1783087762e5bb07e09ded105
2024-08-04 20:41:19 -07:00
jade 0f998056fa misc docs/meson tidying
The docs page has an incorrect escape that leads to a backslash
appearing in output. Meson stuff is self-explanatory, just shortens and
simplifies a bit.

Change-Id: Ib63adf934efd3caeb82ca82988f230e8858a79f9
2024-08-04 20:41:19 -07:00
jade 3daeeaefb1 build: implement clang-tidy using our plugin
The principle of this is that you can either externally build it with
Nix (actual implementation will be in a future commit), or it can be
built with meson if the Nix one is not passed in.

The idea I have is that dev shells don't receive the one from Nix to
avoid having to build it, but CI can use the one from Nix and save some
gratuitous rebuilds.

The design of this is that you can run `ninja -C build clang-tidy` and
it will simply correctly clang-tidy the codebase in spite of PCH
bullshit caused by the cc-wrapper.

This is a truly horrendous number of hacks in a ball, caused by bugs in
several pieces of software, and I am not even getting started.

I don't consider this to fix the clang-tidy issue filing, since we still
have a fair number of issues to fix even on the existing minimal
configuration, and I have not yet implemented it in CI. Realistically we
will need to do something like https://github.com/Ericsson/codechecker
to be able to silence warnings without physically touching the code, or
at least *diff* reports between versions.

Also, the run-clang-tidy output design is rather atrocious and must
not be inflicted upon anyone I have respect for, since it buries the
diagnostics in a pile of invocation logs. We would do really well to
integrate with the Gerrit SARIF stuff so we can dump the reports on
people in a user-friendly manner.

Related: lix-project/lix#147

Change-Id: Ifefe533f3b56874795de231667046b2da6ff2461
2024-08-04 20:41:19 -07:00
eldritch horrors 66469fc281 libstore: move Goal::waiteeDone into Worker::goalFinished
this begins a long and arduous journey to remove all result state from
Goal, to eventually drop the std::enable_shared_from_this base, and to
completely eliminate all unsynchronized modification of states of both
Goal and Worker. by the end of this we will hopefully be able to start
and reap multiple derivation builds in parallel, which should speed up
the process quite a bit (at least for short local builds, others might
not notice a large difference. the build hooks will remain a problem.)

Change-Id: I57dcd9b2cab4636ed4aa24cdec67124fef883345
2024-08-03 00:08:44 +00:00
alois31 32ca194ebf Merge "libstore/ssh: only resume the logger when we paused it" into main 2024-08-02 16:59:44 +00:00
alois31 a93dade821
libstore/ssh: only resume the logger when we paused it
In the SSH code, the logger was conditionally paused, but unconditionally
resumed. This was fine as long as resuming the logger was idempotent. Starting
with 0dd1d8ca1c, it isn't any more, and the
behaviour of the code in question was missed. Consequently, an assertion
failure is triggered for example when performing builds against an "SSH" store
on localhost. Fix the issue by only resuming the logger when it has actually
been paused.

Fixes: lix-project/lix#458
Change-Id: Ib1e4d047744a129f15730b7216f9c9368c2f4211
2024-08-02 18:38:14 +02:00
eldritch horrors e5177dddff libstore: move Goal::amDone to Worker
we still mutate goal state to store the results of any given goal run,
but now we also have that information in Worker and could in theory do
something else with it. we could return a map of goal to goal results,
which would also let us better diagnose failures of subgoals (at all).

Change-Id: I1df956bbd9fa8cc9485fb6df32918d68dda3ff48
2024-08-02 13:52:15 +00:00
eldritch horrors dfcab1c3f0 libstore: return finishedness from Goal methods
this is the first step towards removing all result-related mutation of
Goal state from goal implementations themselves, and into Worker state
instead. once that is done we can treat all non-const Goal fields like
private state of the goal itself, and make threading of goals possible

Change-Id: I69ff7d02a6fd91a65887c6640bfc4f5fb785b45c
2024-08-02 13:52:15 +00:00
eldritch horrors 724b345eb9 libstore: encapsulate worker build hook state
once goals run on multiple threads these fields must by synchronized as
one, or we try to run build hooks to often (or worse, not often enough)

Change-Id: I47860e46fe5c6db41755b2a3a1d9dbb5701c4ca4
2024-08-02 13:52:15 +00:00
eldritch horrors 868eb5ecde libutil: make RunningProgram::wait more resilient
this will usually be used either directly (which is always fine) or in
Finally blocks (where it must never throw execptions). make sure that,
exceptions being handled or not, the calling wait() in Finally doesn't
cause crashes due to the Finally no-nested-exceptions-thrown assertion

Change-Id: Ib83a5d9483b1fe83b9a957dcefeefce5d088f06d
2024-08-02 13:12:44 +00:00
Jeremy List c907d805bf Merge "package: make aws-sdk-cpp build input optional" into main 2024-08-02 11:42:13 +00:00
Isabel 9eb374dc6d Merge "nix flake show: add the description if it exists" into main 2024-08-02 07:56:06 +00:00
Maximilian Bosch 3bb8c627ae Merge "Reapply "libfetchers: make attribute / URL query handling consistent"" into main 2024-08-02 04:50:25 +00:00
Maximilian Bosch 87fd1f024c Reapply "libfetchers: make attribute / URL query handling consistent"
The original attempt at this introduced a regression; this commit
reverts the revert and fixes the regression.

This reverts commit 3e151d4d77.

Fix to the regression:

flakeref: fix handling of `?dir=` param for flakes in subdirs

As reported in #419[1], accessing a flake in a subdir of a Git
repository fails with the previous commit[2] applied with the error

    error: unsupported Git input attribute 'dir'

The problem is that the `dir`-param is inserted into the parsed URL if a
flake is fetched from the subdir of a Git repository. However, for the
fetching part this isn't even needed. The fix is to just pass `subdir`
as second argument to `FlakeRef` (which needs a `basedir` that can be
empty) and leave the parsedURL as-is.

Added a regression test to make sure we don't run into this again.

[1] lix-project/lix#419
[2] e22172aaf6b6a366cecd3c025590e68fa2b91bcc,
    originally 3e151d4d77

Change-Id: I2c72d5a32e406a7ca308e271730bd0af01c5d18b
2024-08-01 15:41:30 -07:00
jade 8b69d13368 Merge "flake: remove control character from file by using fromJSON" into main 2024-08-01 22:25:36 +00:00
Jeremy List f41190552f
package: make aws-sdk-cpp build input optional
I have added an option to turn off this build input because I'm much
more comfortable when I don't have that type of thing on my computer.
Its default value is true in order to avoid impacting anyone who depends
on AWS features.

Change-Id: Ic57f3c9b9468f422e9fbdcf3ba0fe96177631067
2024-08-02 09:14:48 +12:00
Qyriad 61a93d5308 Merge changes Icc4747ae,Id4717b5b,Ie3ddb3d0,Ic4d93a08,I00d9ff70 into main
* changes:
  remove unused headers in installable-attr-path
  libexpr: include the type of the non-derivation value in the type error
  libexpr: mild cleanup to getDerivations
  libexpr: DrvInfo: remove unused bad-citizen constructor
  cleanup and slightly refactor DrvInfo::queryOutputs
2024-08-01 16:25:43 +00:00
jade e6fc3e9227 flake: remove control character from file by using fromJSON
I was reminded by various evil things puck did to the evaluator
involving null bytes that you can get funny bytes by abusing JSON
parsing. It's neater than putting binary in the source file, so let's do
it.

Change-Id: I1ff2e0d829eb303fbed81fa2ebb3a39412e89ff1
2024-07-31 23:23:42 -07:00
jade a3ab2cc78a Merge changes from topic "undefined-behaviour" into main
* changes:
  releng: move officialRelease to version.json
  Add -Werror CI job
  ci: add a asan+ubsan test run on x86_64-linux
  tree-wide: add support for asan!
2024-08-01 04:01:34 +00:00
Qyriad 17d7e88707 remove unused headers in installable-attr-path
Change-Id: Icc4747aed195e3855b128c73df82e202405af6a8
2024-08-01 00:37:13 +00:00
Qyriad 4f6a3d7e9e libexpr: include the type of the non-derivation value in the type error
Change-Id: Id4717b5b0df7c09b0dbf17e642d8713a0a3efbae
2024-08-01 00:37:03 +00:00
Qyriad 5ffed6d06a libexpr: mild cleanup to getDerivations
Shuffled the logic around a bit so the shorter code paths are early
returns, added comments, etc.

Should be NFC.

Change-Id: Ie3ddb3d0eddd614d6f8c37bf9a4d5a50282084ea
2024-08-01 00:36:55 +00:00
Qyriad 6a30ea0cc4 libexpr: DrvInfo: remove unused bad-citizen constructor
DrvInfo's constructor that only takes `EvalState` leaves everything else
empty; a DrvInfo which has no iota of information about the derivation
it represents is not useful, and was not used anywhere.

Change-Id: Ic4d93a08cb2748b8cef9a61e41e70404834b23f9
2024-08-01 00:36:41 +00:00
Qyriad eb18dcb0ea cleanup and slightly refactor DrvInfo::queryOutputs
Change-Id: I00d9ff707fe61995737b86af6d2eaa1e4d8116ff
2024-08-01 00:36:27 +00:00
jade 5eecdd3ae9 releng: move officialRelease to version.json
This was causing a few bits of suffering downstream, in particular, in
the NixOS module, which, after this change, can have the
`officialRelease` stuff in *it* completely deleted since we now have
correct defaulting in package.nix for it.

It also eliminates some automated editing of Nix files, which is
certainly always welcome to eliminate.

Fixes: lix-project/lix#406
Change-Id: Id12f3018cff4633e379dbfcbe26b7bc84922bdaf
2024-07-31 14:13:39 -07:00
jade b5c6ce7a53 Add -Werror CI job
We should cause CLs that introduce compiler warnings to fail CI. Sadly
this will only cover Clang, but it will cover Clang for free, so it's
truly impossible to say if it's bad or not.

Change-Id: I45ca20d77251af9671d5cbe0d29cb08c5f1d03c2
2024-07-31 14:13:39 -07:00
jade e51263057f ci: add a asan+ubsan test run on x86_64-linux
This should at least catch out blatantly bad patches that don't pass the
test suite with ASan. We don't do this to the integration tests since
they run on relatively limited-memory VMs and so it may not be super
safe to run an evaluator with leak driven garbage collection for them.

Fixes: lix-project/lix#403
Fixes: lix-project/lix#319
Change-Id: I5267b02626866fd33e8b4d8794344531af679f78
2024-07-31 14:13:39 -07:00
jade 19ae87e5ce tree-wide: add support for asan!
What if you could find memory bugs in Lix without really trying very
hard? I've had variously scuffed patches to do this, but this is
blocked on boost coroutines removal at this point tbh.

Change-Id: Id762af076aa06ad51e77a6c17ed10275929ed578
2024-07-31 14:13:39 -07:00
Qyriad ddfca6e81b libexpr: implement actual constructors for nix::Value
Change-Id: Iebc2bb4e4ea5e93045afe47677df756de4ec4d05
2024-07-31 15:38:37 +02:00
V. c347d3df8f Merge changes I609a5898,I7afb53c9 into main
* changes:
  devendor pegtl
  update flake.lock
2024-07-30 20:32:52 +00:00
eldritch horrors 97a389b0be libstore: move Goal::getBuildResult to BuildResult
there are no other uses for this yet, but asking for just a subset of
outputs does seem at least somewhat useful to have as a generic thing

Change-Id: I30ff5055a666c351b1b086b8d05b9d7c9fb1c77a
2024-07-30 16:37:13 +00:00
Max “Goldstein” Siling 53bfcf2586 Merge "src/libcmd/repl.cc: avoid unneeded reload after :e" into main 2024-07-30 16:32:56 +00:00
Max “Goldstein” Siling 1a6d7a3af4 src/libcmd/repl.cc: avoid unneeded reload after :e
If `:edit`ing a store path, don't reload repl afterwards
to avoid losing local variables: store is immutable,
so "editing" a store path is always just viewing it.

Resolves: lix-project/lix#341
Change-Id: I3747f75ce26e0595e953069c39ddc3ee80699718
2024-07-30 16:08:26 +00:00
eldritch horrors d265dd5993 libstore: count all substitutions toward the same limit
limiting CA substitutions was a rather recent addition, and it used a
dedicated counter to not interfere with regular substitutions. though
this works fine it somewhat contradicts the documentation; job limits
should apply to all kinds of substitutions, or be one limit for each.

Change-Id: I1505105b14260ecc1784039b2cc4b7afcf9115c8
2024-07-30 15:37:27 +00:00
eldritch horrors d9af753a7f libstore: always wake up goals on EOF
all goals do this. it makes no sense to not notify a goal of EOF
conditions because this is the universal signal for "child done"

Change-Id: Ic3980de312547e616739c57c6248a8e81308b5ee
2024-07-30 15:37:27 +00:00
eldritch horrors 6c0dcd1220 libstore: simplify substitution handleEOF
both substitution goals add only this single fd to their wait set.

Change-Id: Ibf921f5bb3919106208a0871523b32c8f67fb3d3
2024-07-30 15:37:27 +00:00
eldritch horrors 548c973e82 libstore: remove Worker::updateProgress
just update progress every time a goal has returned from work(). there
seem to be no performance penalties, and the code is much simpler now.

Change-Id: I288ee568b764ee61f40a498d986afda49987cb50
2024-07-29 22:16:11 +00:00
jade 6abad7cb23 Merge "releng: Remove workaround for skopeo feature we didn't know about" into main 2024-07-26 11:09:17 +00:00
V. a98dce2a1f devendor pegtl
Change-Id: I609a58985fc5210806d0959049a48976ae079c30
2024-07-26 11:22:34 +04:00