Commit graph

16369 commits

Author SHA1 Message Date
Lily Ballard 5ce1d8463a Merge "libstore: add missing #include on darwin" into main 2024-10-22 02:05:44 +00:00
Lily Ballard 4308ec1ae4 gitignore: ignore *.pyc files
Running our installcheck test suite creates these files.

Change-Id: I97ac8f1aa165a491c55dff6b48486db17b75443b
2024-10-20 22:08:35 -07:00
Lily Ballard 65551175e3 libstore: add missing #include on darwin
optimise-store.cc used std::regex on darwin, but forgot to include the
header. This probably compiled due to the precompiled headers file, but
it caused errors in the editor.

Change-Id: I23297c08cb66d44e4d4f303560f46e4adc7d5a43
2024-10-20 22:08:35 -07:00
Lily Ballard 068f4b147d libstore: fix sign comparison warnings in darwin platform
It's not clear to me if `proc_pidinfo()` or `proc_pidfdinfo()` can
actually return negative values, the syscall wrappers convert `-1` into
zero and the semantics suggest that negative values don't make sense,
but just to be safe we'll preserve the int type until we've checked that
it's a positive value.

Fixes: #548
Change-Id: If575aec6b1e27dba63091c7a0316c7b3788747cd
2024-10-20 13:13:11 -07:00
eldritch horrors 0ff8f91325 libutil: disallow AsyncCollect relocations
some promises capture `this`.  we could also allocate a shared state,
but this thing doesn't really need to ever be moved anyway. so there.

Change-Id: I50b5c44684a8ab4e984b1323de21f97ace4a864a
2024-10-19 19:47:46 +00:00
eldritch horrors b0e619b8bd Merge "libstore: always release build/substitution slot tokens" into main 2024-10-19 18:32:30 +00:00
eldritch horrors 564d931134 libstore: always release build/substitution slot tokens
not doing this can freeze slots until the goal that occupied them is
freed (rather than simply complete), and then can freeze the system.

fixes #549

Change-Id: I042df04222f8ffbaa18ef4a4eae6cbd6f89b679e
2024-10-19 16:17:58 +02:00
jade 60578b4d7d Merge "flake.lock: update everything" into main 2024-10-19 00:03:42 +00:00
jade eadce58a90 Merge "testsuite: Fix tests on systems with a non-master defaultBranch" into main 2024-10-19 00:00:30 +00:00
jade ca55060ac6 packaging: use in-tree capnproto derivation
This is done because the one in nixpkgs has several problems and we
don't want to conflict with them fixing those problems:
- not building shared libs
- not building debuginfo (more critical for us due to
  #549)
- not setting the correct cmake build type
- not setting the correct cxxflags to build the coroutine library
- not building with clang (required for the coroutine library since gcc
  is known to miscompile coroutines *in kj*).

CC: #551

Fixes: #550
Change-Id: Ia5b78dc1809963fdd1a8203b127a216cb575d751
2024-10-18 15:40:22 -07:00
jade 77b6f6734f packaging: remove workaround for clang stdenv asserts
This workaround was for the stdenv not being set when callPackage'ing
package.nix for some of the stranger CI outputs.

Change-Id: I2acdd6efa721b90dd3cb04358544d25d591ff084
2024-10-18 15:40:22 -07:00
piegames e2d00ac3a8 libexpr: Fix typo in error message
Closes #523

Change-Id: Ib5705e405b74d07a8fcf0163847405e9c791c3e3
2024-10-18 19:37:23 +02:00
piegames 3ba5ef91bc benchmarks: Fix purity
Some stuff wasn't working on my machine because it depended on the host
environment in some ways, fixed those.

Change-Id: Iff4931a9a26c6827978f1ee6434710f406d67a96
2024-10-18 19:37:23 +02:00
piegames e5de1d13c4 libexpr: Optimize complex indented strings
The old behavior results in lots of concatenations happening for no good
reason and is an artifact of the technical limitations of the old parser
(combined with some lack of care for such details).

Change-Id: I0d78d6220ca6aeaa10bc437e48e08bf7922e0bb3
2024-10-18 19:37:23 +02:00
piegames 878e181882 libexpr: Print interpolations more accurately in show
This is only a minor semantical distinction, but we should be able to
properly test it, and the parser tests rely on show for that.

Change-Id: I25e868cf9544e30cdff17deb5fd50a434e0f367e
2024-10-18 11:40:04 +00:00
piegames c852ae60da libexpr: Rewrite stripIndentation for indented strings
This commit should faithfully reproduce the old behavior down to the
bugs. The new code is a lot more readable, all quirks are well
documented, and it is overall much more maintainable.

Change-Id: I629585918e4f2b7d296b6b8330235cdc90b7bade
2024-10-18 11:40:04 +00:00
piegames 765771a355 tests/functional-lang: Add test cases for indented strings
Accidentally came across them elsewhere, wanted them covered in the
ind-strings test as well.

Change-Id: Iba418a687388ba85516d13d4c9d815744297dc5c
2024-10-18 11:40:04 +00:00
piegames 14291856e4 tests/functional-lang: Test indented string parsing
There is already an eval test for it, but it doesn't test the parser's
output

Change-Id: I94809080f90ace8aa3f610bcd315e9df0cf5d12f
2024-10-18 11:40:04 +00:00
piegames e7d6212f77 libexpr: move parser semantics into separate file
Another preparation for forking off and versioning the parser

Change-Id: I7b1225a44a3b81486414c1d37bd3e76a3ab307f9
2024-10-18 11:40:04 +00:00
piegames f98ee07573 libexpr: rename grammar to grammar::v1
Let's make some space in the namespace for a v2

https://wiki.lix.systems/books/lix-contributors/page/nix-lang-v2
Change-Id: If56e6dbf680d931233aa822ef91c8832464471e4
2024-10-18 11:40:04 +00:00
piegames 580df9bdb3 tests/functional-lang: Small cleanup of flagfiles handling
Flag files now properly work for all tests.

Change-Id: If652235960e8cf4120270a1b974c17586a31a157
2024-10-18 11:40:04 +00:00
jade 52b64662a4 Merge "fix: macOS build broken by fatal lowdown CLI sandbox setup" into main 2024-10-17 21:17:49 +00:00
jade f6077314fa Merge "Fix std::terminate call in thread pool" into main 2024-10-15 00:11:59 +00:00
jade a020f5f6cb fix: macOS build broken by fatal lowdown CLI sandbox setup
This failed due to https://github.com/NixOS/nixpkgs/pull/346945, which
makes a second lowdown-unsandboxed that works in nix builds, and the
regular lowdown has executables that fail closed when the sandbox setup
fails.

The actual failure here is only visible on nixos-unstable at the moment,
not 24.05, but this commit should fix it up for all versions.

Fixes: #547
Change-Id: I50c0ecb59518ef01a7c0181114c1b4c5a7c6b78b
2024-10-14 16:49:16 -07:00
V. fbf7a8b440 Merge "Remove ancient let from 2 test files" into main 2024-10-14 22:06:48 +00:00
V. 31ff77b3f9 Remove ancient let from 2 test files
Change-Id: I992bc7f9e1cfcb1e4038fbe6ee04178bbf938556
2024-10-14 17:19:22 +04:00
Maximilian Bosch 4682e40183 ssh-ng: better way to keep SSH errors visible
A better fix than in 104448e75d, hence a
revert + the fix.

It turns out that this commit has the side-effect that when having e.g.
`StrictHostKeyChecking=accept-new` for a remote builder, the warnings à la

    Warning: Permanently added 'builder' (ED25519) to the list of known hosts.

actually end up in the derivation's log whereas hostkey verification
errors don't, but only in the stderr of the `nix-build` invocation
(which was the motivation for the patch).

This change writes the stderr from the build-hook to

* the daemon's stderr, so that the SSH errors appear in the journal
  (which was the case before 104448e75d)
* the client's stderr, as a log message
* NOT to the drv log (this is handled via `handleJSONLogMessage`)

I tried to fix the issue for legacy-ssh as well, but failed and
ultimately decided to not bother.

I know that we'll sooner or later replace the entire component, however
this is the part of the patch I have working for a while, so I figured I
might still submit it for the time being.

Change-Id: I21ca1aa0d8ae281d2eacddf26e0aa825272707e5
2024-10-14 06:01:18 +00:00
Zebreus d726236e27 testsuite: Fix tests on systems with a non-master defaultBranch
When the git default branch is not set to master the installcheck
test suite fails. This patch adjusts the test setup scripts to
ignore the system and user git config files.

GIT_CONFIG_SYSTEM is set to /dev/null to ignore /etc/gitconfig

GIT_CONFIG_GLOBAL is not set because the global config files
are loaded from $HOME or $XDG_CONFIG_HOME which we already
reset.

git documentation: https://git-scm.com/docs/git#Documentation/git.txt-codeGITCONFIGGLOBALcode

Change-Id: Ie73bbed1db9419c9885b9d57e4edb7a4047d5cce
2024-10-14 05:56:02 +00:00
jade 326cbecb61 Merge changes I327db40f,If762efce into main
* changes:
  testsuite: use xdist for parallel test running
  testsuite: add a functional2 test suite based on pytest
2024-10-14 05:53:44 +00:00
Maximilian Bosch a322fcea4a
worker: respect C-c on sudo nix-build
While debugging something else I observed that latest `main` ignores
`Control-C` on `sudo nix-build`.

After reading through the capnproto docs, it seems as if the promise
must be fulfilled to actually terminate the `promise.wait()` below.

This also applies to scenarios such as stopping the client
(`nix-build`), but the builders on the daemon-side are still running,
i.e. closes #540

Co-authored-by: eldritch horrors <pennae@lix.systems>

Change-Id: I9634d14df4909fc1b65d05654aad0309bcca8a0a
2024-10-12 21:16:30 +02:00
jade a0fb52c0af Fix std::terminate call in thread pool
So we received a report that the thread pool crashed due to an
Interrupted exception.

Relevant log tail:

copying path '/nix/store/0kal2k73inviikxv9f1ciaj39lkl9a87-etc-os-release' to 'ssh://192.168.0.27'...
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:

error (ignored): error: interrupted by the user
Exception: nix::Interrupted: error: interrupted by the user

Relevant stack trace:

 4# __cxa_rethrow in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
 5# nix::ignoreExceptionExceptInterrupt(nix::Verbosity) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
 6# nix::ThreadPool::doWork(bool) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
 7# 0x00007FA7A00E86D3 in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
 8# 0x00007FA79FE99A42 in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
 9# 0x00007FA79FF1905C in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6

Notably, this is *not* in the main thread, so this implies that the
thread didn't get joined properly before their destructors got called.
That, in turn, should have only possibly happened because join() threw
on a previous iteration of the loop joining threads, I think. Or if it
threw while in the ThreadPool destructor. Either way we had better stop
letting Interrupted fall out of our child threads!

If:
- Interrupted was thrown inside the action in the main thread: it would
  have fallen out of doWork if state->exception was already set and got
  caught by ThreadPool::process, calling shutdown() and the join loop
  which would crash the process entirely.
- Interrupted was thrown inside the action on a secondary thread: it
  would have been caught and put into the exception field and then
  possibly rethrown to fall out of the thread (since it was previously
  ignoreExceptionExceptInterrupt).

The one possible hole in this hypothesis is that there is an "error
(ignored)" line in there implying that at least one Interrupted got
eaten by an ignoreExceptionInDestructor. It's also unclear whether this
got reordered because of stderr buffering.

Fixes: #542
Change-Id: I322cf050da660af78f5cb0e08ec6e6d27d09ac76
2024-10-09 15:38:40 -07:00
jade 4180b84a67 testsuite: use xdist for parallel test running
This is capped at 12 because 3.7 seconds of startup is painful enough
and 5.5 seconds with 24 was more annoying.

Change-Id: I327db40fd98deaa5330cd9cf6de99fb07b2c1cb0
2024-10-09 14:47:39 -07:00
jade 3caf3e1e08 testsuite: add a functional2 test suite based on pytest
I am tired of bad shell scripts, let me write bad python quickly
instead. It's definitely, $100%, better.

This is not planned as an immediate replacement of the old test suite,
but we::jade would not oppose tests getting ported.

What is here is a mere starting point and there is a lot more
functionality that we need.

Fixes: #488

Change-Id: If762efce69030bb667491b263b874c36024bf7b6
2024-10-09 14:47:39 -07:00
jade 9865ebaaa6 Merge "Remove static initializers for RegisterLegacyCommand" into main 2024-10-09 20:37:58 +00:00
jade 7f7a38f278 Merge changes Ib27cb43d,I03687b8b into main
* changes:
  testsuite: override NIX_CONF_DIR and NIX_USER_CONF_FILES
  Remove some outdated `make test` invocation suggestions
2024-10-09 20:37:16 +00:00
Rebecca Turner 0012887310 Merge "Add release note for CTRL-C improvements" into main 2024-10-08 22:15:56 +00:00
Lulu 4ea8c9d643 Set c++ version to c++23
I followed @pennae's advice and moved the constructor definition of
`AttrName` from the header file `nixexpr.hh` to `nixexpr.cc`.

Change-Id: I733f56c25635b366b11ba332ccec38dd7444e793
2024-10-08 20:05:28 +02:00
Lulu 43e79f4434 Fix gcc warning -Wmissing-field-initializers
The approach that was taken here was to add default values to the type
definitions rather than specify them whenever they are missing.

Now the only remaining warning is '-Wunused-parameter' which @jade said
is usually counterproductive and that we can just disable it:
#456 (comment)

So this change adds the flags '-Wall', '-Wextra' and
'-Wno-unused-parameter', so that all warnings are enabled except for
'-Wunused-parameter'.

Change-Id: Ic223a964d67ab429e8da804c0721ba5e25d53012
2024-10-08 01:44:38 +00:00
Lulu 299813f324 Merge "Avoid calling memcpy when len == 0 in filetransfer.cc" into main 2024-10-08 01:41:41 +00:00
Lulu d6e1b11d3e Fix gcc warning -Wsign-compare
Add the compile flag '-Wsign-compare' and adapt the code to fix all
cases of this warning.

Change-Id: I26b08fa5a03e4ac294daf697d32cf9140d84350d
2024-10-08 01:32:12 +02:00
Lulu 51a5025913 Avoid calling memcpy when len == 0 in filetransfer.cc
There was a bug report about a potential call to `memcpy` with a null
pointer which is not reproducible:
#492

This occurred in `src/libstore/filetransfer.cc` in `InnerSource::read`.

To ensure that this doesn't happen, an early return is added before
calling `memcpy` if the length of the data to be copied is 0.

This change also adds a test that ensures that when `InnerSource::read`
is called with an empty file, it throws an `EndOfFile` exception.

Change-Id: Ia18149bee9a3488576c864f28475a3a0c9eadfbb
2024-10-08 01:26:30 +02:00
eldritch horrors ed9b7f4f84 libstore: remove Worker::{childStarted, goalFinished}
these two functions are now nearly trivial and much better inline into
makeGoalCommon. keeping them separate also separates information about
goal completion flows and how failure information ends up in `Worker`.

Change-Id: I6af86996e4a2346583371186595e3013c88fb082
2024-10-05 21:19:51 +00:00
eldritch horrors 649d8cd08f libstore: remove Worker::removeGoal
we can use our newfound powers of Goal::work Is A Real Promise to remove
completed goals from continuation promises. apart from being much easier
to follow it's also a lot more efficient because we have the iterator to
the item we are trying to remove, skipping a linear search of the cache.

Change-Id: Ie0190d051c5f4b81304d98db478348b20c209df5
2024-10-05 21:19:51 +00:00
eldritch horrors 9adf6f4568 libstore: remove Goal::notify
Goal::work() is a fully usable promise that does not rely on the worker
to report completion conditions. as such we no longer need the `notify`
field that enabled this interplay. we do have to clear goal caches when
destroying the worker though, otherwise goal promises may (incorrectly)
keep goals alive due to strong shared pointers created by childStarted.

Change-Id: Ie607209aafec064dbdf3464fe207d70ba9ee158a
2024-10-05 21:19:51 +00:00
eldritch horrors 03cbc0ecb9 libstore: move Goal::ex to WorkResult
yet another duplicated field. it's the last one though.

Change-Id: I352df8d306794d262d8c9066f3be78acd40e82cf
2024-10-05 21:19:51 +00:00
eldritch horrors 1caf2afb1d libstore: move Goal::buildResult to WorkResult
derivation goals still hold a BuildResult member variable since parts of
these results of accumulated in different places, but the Goal class now
no longer has such a field. substitution goals don't need it at all, and
derivation goals should also be refactored to not drop their buildResult

Change-Id: Ic6d3d471cdbe790a6e09a43445e25bedec6ed446
2024-10-05 20:53:39 +00:00
eldritch horrors 7ff60b7445 libstore: move Goal::exitCode to WorkResult
the field is simply duplicated between the two, and now that we can
return WorkResults from Worker::run we no longer need both of them.

Change-Id: I82fc47d050b39b7bb7d1656445630d271f6c9830
2024-10-05 20:17:20 +00:00
eldritch horrors fc6291e46d libstore: return goal results from Worker::run()
this will be needed to move all interesting result fields out of Goal
proper and into WorkResult. once that is done we can treat goals as a
totally internal construct of the worker mechanism, which also allows
us to fully stop exposing unclear intermediate state to Worker users.

Change-Id: I98d7778a4b5b2590b7b070bdfc164a22a0ef7190
2024-10-05 20:12:13 +00:00
eldritch horrors 40f154c0ed libstore: remove Worker::topGoals
since we now propagate goal exceptions properly we no longer need to
check topGoals for a reason to abort early. any early abort reasons,
whether by exception or a clean top goal failure, can now be handled
by inspecting the goal result in the main loop. this greatly reduces
goal-to-goal interactions that do not happen at the main loop level.

since the underscore-free name is now available for use as variables
we'll migrate to that where we currently use `_topGoals` for locals.

Change-Id: I5727c5ea7799647c0a69ab76975b1a03a6558aa6
2024-10-05 19:53:30 +00:00
eldritch horrors f389a54079 libstore: propagate goal exceptions using promises
drop childException since it's no longer needed. also makes
waitForInput, childFinished, and childTerminated redundant.

Change-Id: I05d88ffd323c5b5c909ac21056162f69ffb0eb9f
2024-10-05 19:44:47 +00:00