random() is not thread-safe, it relies on global state, and calling it
from worker threads can result in multiple threads producing the same
value. It also doesn't guarantee unique values even in single-threaded
use.
Use an atomic counter for the use-case of generating temporary paths,
and switch to a thread-local RNG for the one remaining call.
This will probably fix https://github.com/NixOS/nix/issues/7273 though
I'm not willing to risk corrupting my store to find out.
Change-Id: I4c4c4c9796613573ffefd29cc8efe3d07839facc
Also tweak `pathAccessible` to ignore other relevant errors too. It was
documented as ignoring permission errors but it was only ignoring
`EPERM`, which comes from the darwin sandbox, and not ignoring `EACCESS`
which is the real permission error. I figured it also makes sense to
ignore `ELOOP`.
Fixes: #560
Change-Id: Ibb849b68d07386eb80afb52b57f7d12b3a48a202
curl handles timeouts internally when it acts as the main event loop. we
only need to wake up to add new transfers to the multi handle or restart
an existing transfer that has passed its restart wait time. periodically
waking up is not required for curl, and we don't need it any more either
Change-Id: Ic774e1d9519f807cda1a89694bc3ede75216f329
don't pause the entire curl thread. we have multiple consumer threads
after all, not just one, so stalling all of them is likely not great.
note that libcurl advises against using transfer pauses if compressed
encodings are allowed and automatically decoded. this should not lead
to problems in practice because our data is usually not compressed to
such a degree that curl buffering *uncompressed* data matters. should
this cause problems we can reintroduce the whole-thread pause, but we
will probably get away with this until the entire file transfer class
is made kj::Promise-using async (and *then* curl can be hardpaused if
it cannot get rid of its data, solving the problem once and for all).
Change-Id: I218e41bfa5a27c7454eafb0bdb54f2a29a7f6493
Sets the `X-GitHub-Api-Version` header to `2022-11-28` for calls to the
GitHub API.
This follows the later version as per
https://docs.github.com/en/rest/about-the-rest-api/api-versions?apiVersion=2022-11-28.
This affected the check on whether to use the API versus unauthenticated
calls as well, given the headers would no longer be empty if the
authentication token were missing.
The workaround used here is to use a check similar to an existing
check for the token.
In the current implementation, headers are (still) similarly sent to
non-authenticated as well as GitHub on-prem calls.
For what it's worth, manual curl calls with such a header seemed to
break nor unauthenticated calls nor ones to the github.com API.
Change-Id: I6e10839e6b99cb65eb451e923b2a64f5d3c0f578
Before:
error: derivation '/nix/store/4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
After:
error: derivation '4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: outputs differ
output differs: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
output differs: output '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0' differs from '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0.check'
Change-Id: Ib2871fa602bf1fa9c00e2565b3a2e1b26f908152
curl can't pause downloads of file:// urls, which is very much in the
way of making the curl wrapper fully asynchronous. we can emulate all
the things curl does, but unfortunately curl is *rather extensive* in
its support of frankly weird shit. hopefully this subset of features,
which notably does not include curl readdir support, is enough for us
Change-Id: I5f67768c4b512565655b94b0421270c7dbbd8d11
with the api cleaned up we can suddenly reunify uploads, downloads, and
existence checks through curl in the same wrapper function. uploads and
existence checks simply don't use the result source, and given that all
transfers (or at least *most* transfers to date) go through the network
the few extra allocations do not hurt us at all. even for file:// calls
the overhead won't much matter as going to disk and back *is* expensive
Change-Id: I4f9ca6681a8fc303377b4cf4c63e3363ae32c18b
it's no longer needed. `download` can do everything `enqueueDownload`
did, and a lot more. e.g. not block the calling thread, for instance.
Change-Id: I4b36235ed707c92d117b4c33efa3db50d26f9a84
this will let us return metadata from FileTransfer::download, which in
turn is necessary to remove enqueueDownload. it also opens avenues for
streaming downloads that keep download metadata instead of dropping it
Change-Id: If0fc6af5eb2aeb689fc866c345c9d7bce4d59f2d
let's use the automatic decoding functions curl provides instead of
implementing them ourselves for the dubious ability to support both
xz and bzip2 encodings as well, neither of which anything will send
Change-Id: I3edfebeb596a0e9d5c986efca9270501c996f2dd
We were calling `lstat()` twice on the link path, and a third time if it
existed. We only need to call it once.
Change-Id: Ifadc2f24681648d0ec7e4bad5c6d2dcb2e560e7b
etag changing implies with high probability that the content of the
resource changed. immutable url changing implies that the immutable
url we got previously was wrong, which is probably a server bug. if
the encoding changes our decoding will break completely, so that is
also very illegal. one notable change we still allow is etags going
away completely, mostly since this does not imply any data changes.
Change-Id: I0220ceddc3fd732cd1b3bb39b40021cc631baadc
do not retread the entire redirection path if we've seen the end of the
road. this avoids silently downloading wrong data, and notifies us when
a url we've received data from turns into a redirect when retrying. for
reasons of simplicity we don't turn of libcurl redirects on retries. if
we did that we'd have to conditionally process http status codes, which
sounds annoying and would make the header callback even more of a mess.
Change-Id: Ide0c0512ef9b2579350101246d654a2375541a39
this will let us return metadata for a transfer without having to wait
for the entire transfer to complete. more importantly for current uses
though is that we could now send retries to the effective url directly
instead of retreading the entire redirect path. this improves latency,
and in such cases where redirects change while we're downloading it'll
also improve correctness (previously we'd silently download bad data).
Change-Id: I6fcd920eb96fbdb2e960b73773c0b854e0300e99
it's always legal to call curl_multi_remove_handle on a valid pair of
multi and easy handles. removing an easy handle that is not currently
attached to a multi handle is a no-op, and removing an easy handle of
a different multi handle is something we can't reasonably trigger. if
we *did* ever manage it would result in an error we'd ignore, and the
handles in question would not be changed at all. this is just simpler
Change-Id: I85ec62ff89385981ca49d243376b9c32586bd128
return it as a separate item in a pair instead. this will let us remove
enqueueDownload() in favor of returning metadata from download() itself
Change-Id: I74fad2ca15f920da1eefabc950c2baa2c360f2ba
it's just a uri and some headers now. those can be function arguments
with no loss of clarity. *actual* additional arguments, for example a
TLS context with additional certificates, could be added on a new and
improved FileTransfer class that carries not just a backend reference
but some real, visible context for its transfers. curl not being very
multi-threading-friendly when using multi handles will make sharing a
bit hard anyway once we drop the single global download worker thread
Change-Id: Id2112c95cbd118c6d920488f38d272d7da926460
we don't even need this outside of tests. maybe we should not do
automatic retries at this level at all and use retrying wrappers
instead? at some point we may have to do this, but not just yet.
Change-Id: If0088aa55215be81f1770c25b3bb1b5268c65cf8
never set explicitly, and transfers are never instantiated with one
current activity but submitted with a *different* current activity.
Change-Id: I1a3ec57c02013565aeb9e9398ea42d0c4279095e
it's only used by HttpBinaryCacheStore, and even there used in only on
place. this one place can set the header explicitly, which it now does
Change-Id: Id89228150669e25e7f59a3d6bd939e46059ce29e
it only sets the one field anyway (and the parent activity as a side
effect that does not depend on the exact location of the constructor
call). when FileTransferRequest goes away we would need this anyway.
Change-Id: I35cf2ed3533239181449a62cf34cd282b395e5db
* changes:
treewide: make more settings conditionally available
libstore/build: only send overridden settings to the build hook
treewide: consistently mark overridden settings as such
a first little step into pushing the event loops up, up and away.
eventually we will want them to be instantiated only at the roots
of every thread (since kj binds loops to threads), but not today.
Change-Id: Ic97f1debba382a5a3f46daeaf2d6d434ee42569f
goals should be considered internal to the worker architecture due to
the tight coupling of the two, and we can finally do that. doing this
is also a prerequisite for turning Worker::run() into a real promise.
Change-Id: I7cf273d4a6fdb75b8d192fce1af07c6265ff6980
previously it was possible to fetchurl a dict server, or an ldap server,
or an imap server. this is a bit of a problem, both because rare schemes
may not be available on all systems, and because some schemes (e.g. scp)
are inherently insecure in potentially surprising ways we needn't allow.
Change-Id: I18fc567c6f58c3221b5ea8ce927f4da780057828
we already have a results type for the entire worker call, we may as
well put this bit of info in there. we do keep the assumption of the
old code that the field will only be read if some goals have failed;
fixing that is a very different mess, and not immediately necessary.
Change-Id: If3fc32649dcd88e1987cdd1758c6c5743e3b35ac
without this derivations do not show as completely processed in the
internal-json logs (or the newer multiline output). the former also
breaks external tools like nix-output-monitor which, like multiline
output, grow vertically until at least some goals are finally freed
Change-Id: I55758daf526ba29ae15fb82e0d88da8afb45bf5c
optimise-store.cc used std::regex on darwin, but forgot to include the
header. This probably compiled due to the precompiled headers file, but
it caused errors in the editor.
Change-Id: I23297c08cb66d44e4d4f303560f46e4adc7d5a43
It's not clear to me if `proc_pidinfo()` or `proc_pidfdinfo()` can
actually return negative values, the syscall wrappers convert `-1` into
zero and the semantics suggest that negative values don't make sense,
but just to be safe we'll preserve the int type until we've checked that
it's a positive value.
Fixes: #548
Change-Id: If575aec6b1e27dba63091c7a0316c7b3788747cd
some promises capture `this`. we could also allocate a shared state,
but this thing doesn't really need to ever be moved anyway. so there.
Change-Id: I50b5c44684a8ab4e984b1323de21f97ace4a864a
not doing this can freeze slots until the goal that occupied them is
freed (rather than simply complete), and then can freeze the system.
fixes#549
Change-Id: I042df04222f8ffbaa18ef4a4eae6cbd6f89b679e
Daemon client handler processes are forked off of the main nix process
and thus will not have a signal handler thread anymore. This leads to a
high likelihood of bustage, since the Worker infrastructure expects the
interrupt infrastructure to actually, you know, work, to be able to get
interrupted.
The expected behaviour after fork is either:
- Start a signal handler thread if you expect to do complicated things
that need ReceiveInterrupts.
- Call restoreProcessContext and don't handle signals specially
otherwise.
Change-Id: I73d36b5bbf96dddd21d5e1c3bd0484d715c00e8b
this was missed in 3f07c65510. if mode is
not restored, the tree will have mutability where it shouldn't.
fixes test `functional-repair`, which fails from the output path
directory being unnecessarily writable:
```
++(repair.sh:12) nix-hash /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2
+(repair.sh:12) hash=d790f49fc89cb6f384b6dbe450790d07
+(repair.sh:15) chmod u+w /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2
+(repair.sh:16) touch /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2/bad
+(repair.sh:18) nix-store --verify --check-contents -v
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/20z19rjkwmwpb2ba4x29kac6xnslai38-dependencies-top.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/3mmawi9lj33xz96cf6kw9989xc8v5i96-fod-input'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/5fp4r2kh1fcv6mv9dv6rywhhr1am9hhm-builder-dependencies-input-0.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/9kryn4ihv6b7bjswv2rsjq4533n2w5zk-fod-input.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/ap8s0fim8s3ilzj8aqwlwk7gv1crjp4j-dependencies-top'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i2ipd9p282zkqr1zb4glqiqigv8ybsyk-dependencies.builder0.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i3xyw46h0lsx7ad6bczwi9sqjjx5f0j0-dependencies-input-0'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2'
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' was modified! expected hash 'sha256:1rhaylnjs5lbp089lnk7qvsypqmbm5vvyvxvv7i68b4x33pncqgs', got 'sha256:0nkjrmdc6ixf935chj3zhpqph5i15p306ffdsa850qh8mncpnsmc'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/pfaxqiw8zf3bw0w8w8gazswh76d729yy-builder-dependencies-input-2.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/pzzms3k5wl8d3wszv3maw29zylfkiiw0-dependencies-input-1'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/riwx84p57ckfvgli9nwhx88z6zh1c8ss-builder-fod-input.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/rjsphx7xvnqh2qafdrr7fiyxqc1rljhw-dependencies-input-2.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/scywzq87dvm0c5f1h16hww6mvzhcsx3f-builder-dependencies-input-1.sh'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/v65j4w033i05nyfd2h0g8h41ijmfglp7-dependencies-input-0.drv'
checking contents of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/ykwibh62xyp81k75vdw9c7y21kg4ibzf-dependencies-input-1.drv'
warning: not all store errors were fixed
+(repair.sh:21) nix-store --verify --check-contents --repair
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' was modified! expected hash 'sha256:1rhaylnjs5lbp089lnk7qvsypqmbm5vvyvxvv7i68b4x33pncqgs', got 'sha256:0nkjrmdc6ixf935chj3zhpqph5i15p306ffdsa850qh8mncpnsmc'
checking path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2'...
path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2' is corrupted or missing!
checking path '/tmp/nix-shell.IIUJAq/nix-test/repair/store/i3xyw46h0lsx7ad6bczwi9sqjjx5f0j0-dependencies-input-0'...
repairing outputs of '/tmp/nix-shell.IIUJAq/nix-test/repair/store/rjsphx7xvnqh2qafdrr7fiyxqc1rljhw-dependencies-input-2.drv'...
+(repair.sh:23) '[' -e /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2/bad ']'
+(repair.sh:24) '[' -w /tmp/nix-shell.IIUJAq/nix-test/repair/store/j3kmp9zhc5y7gqs591l0nrscm5hw4145-dependencies-input-2 ']'
++(repair.sh:24) onError
++(/var/home/bb010g/Sources/Nix/lix/build/tests/functional/common/vars-and-functions.sh:244) set +x
repair.sh: test failed at:
main in repair.sh:24
```
fixes test `functional-simple`, which fails from the output path file
being unnecessarily writable:
```
+(simple.sh:11) echo 'output path is /tmp/nix-shell.IIUJAq/nix-test/simple/store/9fqn0rs99ymn1r09yip7bsifcdh3ra0y-simple'
+(simple.sh:13) '[' -w /tmp/nix-shell.IIUJAq/nix-test/simple/store/9fqn0rs99ymn1r09yip7bsifcdh3ra0y-simple ']'
++(simple.sh:13) onError
++(/var/home/bb010g/Sources/Nix/lix/build/tests/functional/common/vars-and-functions.sh:244) set +x
simple.sh: test failed at:
main in simple.sh:13
```
fixes test `functional-optimise-store`, which fails from the unnecessary
mutability making Lix avoid hard linking:
```
++(optimise-store.sh:5) nix-build - --no-out-link --auto-optimise-store
this derivation will be built:
/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/2ql4kjxhnzdard8d6n3h9hc1m3lawr2b-foo1.drv
building '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/2ql4kjxhnzdard8d6n3h9hc1m3lawr2b-foo1.drv'...
warning: skipping suspicious writable file '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1/foo'
+(optimise-store.sh:5) outPath1=/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1
++(optimise-store.sh:6) echo 'with import ./config.nix; mkDerivation { name = "foo2"; builder = builtins.toFile "builder" "mkdir $out; echo hello > $out/foo"; }'
++(optimise-store.sh:6) nix-build - --no-out-link --auto-optimise-store
this derivation will be built:
/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/wjgvfhfsp14w06im8bbp1kqzz7smdkcy-foo2.drv
building '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/wjgvfhfsp14w06im8bbp1kqzz7smdkcy-foo2.drv'...
warning: skipping suspicious writable file '/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2/foo'
+(optimise-store.sh:6) outPath2=/tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2
++(optimise-store.sh:8) stat --format=%i /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/h7i1pp848f9a8452y0s18kgsnis77vjn-foo1/foo
+(optimise-store.sh:8) inode1=328316
++(optimise-store.sh:9) stat --format=%i /tmp/nix-shell.IIUJAq/nix-test/optimise-store/store/gjg6gayj2f6x3h53sp5i21nbnsd7b4i3-foo2/foo
+(optimise-store.sh:9) inode2=328404
+(optimise-store.sh:10) '[' 328316 '!=' 328404 ']'
+(optimise-store.sh:11) echo 'inodes do not match'
+(optimise-store.sh:12) exit 1
```
Signed-off-by: Dusk Banks <me@bb010g.com>
Change-Id: I87eeb74e718746a587be2ac52bcc9b5b1e5529db
The old behavior results in lots of concatenations happening for no good
reason and is an artifact of the technical limitations of the old parser
(combined with some lack of care for such details).
Change-Id: I0d78d6220ca6aeaa10bc437e48e08bf7922e0bb3
This is only a minor semantical distinction, but we should be able to
properly test it, and the parser tests rely on show for that.
Change-Id: I25e868cf9544e30cdff17deb5fd50a434e0f367e
This commit should faithfully reproduce the old behavior down to the
bugs. The new code is a lot more readable, all quirks are well
documented, and it is overall much more maintainable.
Change-Id: I629585918e4f2b7d296b6b8330235cdc90b7bade
Some settings only make sense on particular platforms, or only when a certain
experimental feature is enabled. Several of those were already conditionally
available. Do the same for a bunch more instead of silently ignoring them.
Exceptionally, the use-case-hack setting is not made conditional because it is
included in the test suite.
Change-Id: I29e66ad8ee6178a7c0eff9efb55c3410fae32514
The build hook is still running locally, so it will run with the same default
settings. Hence, just as with the daemon, it is enough to send it only the
overridden settings. This will prevent warnings like
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
when the user didn't actually set those settings.
This is inspired by and an alternative to [0].
[0] https://github.com/NixOS/nix/pull/10049
Change-Id: I77ea62cd017614b16b55979dd30e75f09f860d21
Only overridden settings are sent to the daemon, and we're going to do the same
for the build hook to. It needs to be ensured that overridden settings are in
fact consistently marked as such, so that they actually get sent.
Change-Id: I7cd58d925702f86cf2c35ad121eb191ceb62a355
Before this change, expressions like:
with import <nixpkgs> {};
runCommand "foo" {} ''
echo '@nix {}' >&$NIX_LOG_FD
''
would result in Lix crashing, because accessing nonexistent fields of
a JSON object throws an exception.
Rather than handling each field individually, we just catch JSON
exceptions wholesale. Since these log messages are an unusual
circumstance, log a warning when this happens.
Fixes#544.
Change-Id: Idc2d8acf6e37046b3ec212f42e29269163dca893
By implementing the `PathSet` specialization for `PathsSetting`, we'll
be able to use `PathsSetting` for the `sandboxPaths` setting in
`src/libstore/globals.hh`.
Fixes: #498
Change-Id: I8bf7dfff98609d1774fdb36d63e57d787bcc829f
This was always a terrible idea independently of whether it crashes.
Stop doing it!
This commit was verified by running nix-shell on a trivial derivation
with --debug --verbose to get the vomit-level output of the shell rc
file and then diffing it before/after this change. I have reasonable
confidence it did not regress anything, though this code is genuinely
really hard to follow (which is a second reason that I split it into two
fmt calls).
Fixes: #533
Change-Id: I8e11ddbece2b12749fda13efe0b587a71b00bfe5
A better fix than in 104448e75d, hence a
revert + the fix.
It turns out that this commit has the side-effect that when having e.g.
`StrictHostKeyChecking=accept-new` for a remote builder, the warnings à la
Warning: Permanently added 'builder' (ED25519) to the list of known hosts.
actually end up in the derivation's log whereas hostkey verification
errors don't, but only in the stderr of the `nix-build` invocation
(which was the motivation for the patch).
This change writes the stderr from the build-hook to
* the daemon's stderr, so that the SSH errors appear in the journal
(which was the case before 104448e75d)
* the client's stderr, as a log message
* NOT to the drv log (this is handled via `handleJSONLogMessage`)
I tried to fix the issue for legacy-ssh as well, but failed and
ultimately decided to not bother.
I know that we'll sooner or later replace the entire component, however
this is the part of the patch I have working for a while, so I figured I
might still submit it for the time being.
Change-Id: I21ca1aa0d8ae281d2eacddf26e0aa825272707e5
While debugging something else I observed that latest `main` ignores
`Control-C` on `sudo nix-build`.
After reading through the capnproto docs, it seems as if the promise
must be fulfilled to actually terminate the `promise.wait()` below.
This also applies to scenarios such as stopping the client
(`nix-build`), but the builders on the daemon-side are still running,
i.e. closes#540
Co-authored-by: eldritch horrors <pennae@lix.systems>
Change-Id: I9634d14df4909fc1b65d05654aad0309bcca8a0a
So we received a report that the thread pool crashed due to an
Interrupted exception.
Relevant log tail:
copying path '/nix/store/0kal2k73inviikxv9f1ciaj39lkl9a87-etc-os-release' to 'ssh://192.168.0.27'...
Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:
error (ignored): error: interrupted by the user
Exception: nix::Interrupted: error: interrupted by the user
Relevant stack trace:
4# __cxa_rethrow in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
5# nix::ignoreExceptionExceptInterrupt(nix::Verbosity) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
6# nix::ThreadPool::doWork(bool) in /nix/store/ghxr2ykqc3rrfcy8rzdys0rzx9ah5fqj-lix-2.92.0-dev-pre20241005-ed9b7f4/lib/liblixutil.so
7# 0x00007FA7A00E86D3 in /nix/store/22nxhmsfcv2q2rpkmfvzwg2w5z1l231z-gcc-13.3.0-lib/lib/libstdc++.so.6
8# 0x00007FA79FE99A42 in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
9# 0x00007FA79FF1905C in /nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6
Notably, this is *not* in the main thread, so this implies that the
thread didn't get joined properly before their destructors got called.
That, in turn, should have only possibly happened because join() threw
on a previous iteration of the loop joining threads, I think. Or if it
threw while in the ThreadPool destructor. Either way we had better stop
letting Interrupted fall out of our child threads!
If:
- Interrupted was thrown inside the action in the main thread: it would
have fallen out of doWork if state->exception was already set and got
caught by ThreadPool::process, calling shutdown() and the join loop
which would crash the process entirely.
- Interrupted was thrown inside the action on a secondary thread: it
would have been caught and put into the exception field and then
possibly rethrown to fall out of the thread (since it was previously
ignoreExceptionExceptInterrupt).
The one possible hole in this hypothesis is that there is an "error
(ignored)" line in there implying that at least one Interrupted got
eaten by an ignoreExceptionInDestructor. It's also unclear whether this
got reordered because of stderr buffering.
Fixes: #542
Change-Id: I322cf050da660af78f5cb0e08ec6e6d27d09ac76
There is absolutely no good reason these should show up in NARs besides
misconfigured systems and as long as the case hack exists, unpacking
such a NAR will cause its repacking to be wrong on systems with case
hack enabled.
This should not have any security impact on Lix to fix, but it was one
of the vectors for CVE-2024-45593:
https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493
Change-Id: I85b6075aacc069ee7039240b0f525804a2d8edcb
I followed @pennae's advice and moved the constructor definition of
`AttrName` from the header file `nixexpr.hh` to `nixexpr.cc`.
Change-Id: I733f56c25635b366b11ba332ccec38dd7444e793
The approach that was taken here was to add default values to the type
definitions rather than specify them whenever they are missing.
Now the only remaining warning is '-Wunused-parameter' which @jade said
is usually counterproductive and that we can just disable it:
#456 (comment)
So this change adds the flags '-Wall', '-Wextra' and
'-Wno-unused-parameter', so that all warnings are enabled except for
'-Wunused-parameter'.
Change-Id: Ic223a964d67ab429e8da804c0721ba5e25d53012
There was a bug report about a potential call to `memcpy` with a null
pointer which is not reproducible:
#492
This occurred in `src/libstore/filetransfer.cc` in `InnerSource::read`.
To ensure that this doesn't happen, an early return is added before
calling `memcpy` if the length of the data to be copied is 0.
This change also adds a test that ensures that when `InnerSource::read`
is called with an empty file, it throws an `EndOfFile` exception.
Change-Id: Ia18149bee9a3488576c864f28475a3a0c9eadfbb
these two functions are now nearly trivial and much better inline into
makeGoalCommon. keeping them separate also separates information about
goal completion flows and how failure information ends up in `Worker`.
Change-Id: I6af86996e4a2346583371186595e3013c88fb082
we can use our newfound powers of Goal::work Is A Real Promise to remove
completed goals from continuation promises. apart from being much easier
to follow it's also a lot more efficient because we have the iterator to
the item we are trying to remove, skipping a linear search of the cache.
Change-Id: Ie0190d051c5f4b81304d98db478348b20c209df5
Goal::work() is a fully usable promise that does not rely on the worker
to report completion conditions. as such we no longer need the `notify`
field that enabled this interplay. we do have to clear goal caches when
destroying the worker though, otherwise goal promises may (incorrectly)
keep goals alive due to strong shared pointers created by childStarted.
Change-Id: Ie607209aafec064dbdf3464fe207d70ba9ee158a
derivation goals still hold a BuildResult member variable since parts of
these results of accumulated in different places, but the Goal class now
no longer has such a field. substitution goals don't need it at all, and
derivation goals should also be refactored to not drop their buildResult
Change-Id: Ic6d3d471cdbe790a6e09a43445e25bedec6ed446
the field is simply duplicated between the two, and now that we can
return WorkResults from Worker::run we no longer need both of them.
Change-Id: I82fc47d050b39b7bb7d1656445630d271f6c9830
this will be needed to move all interesting result fields out of Goal
proper and into WorkResult. once that is done we can treat goals as a
totally internal construct of the worker mechanism, which also allows
us to fully stop exposing unclear intermediate state to Worker users.
Change-Id: I98d7778a4b5b2590b7b070bdfc164a22a0ef7190
since we now propagate goal exceptions properly we no longer need to
check topGoals for a reason to abort early. any early abort reasons,
whether by exception or a clean top goal failure, can now be handled
by inspecting the goal result in the main loop. this greatly reduces
goal-to-goal interactions that do not happen at the main loop level.
since the underscore-free name is now available for use as variables
we'll migrate to that where we currently use `_topGoals` for locals.
Change-Id: I5727c5ea7799647c0a69ab76975b1a03a6558aa6
drop childException since it's no longer needed. also makes
waitForInput, childFinished, and childTerminated redundant.
Change-Id: I05d88ffd323c5b5c909ac21056162f69ffb0eb9f
there's no reason to have the worker set information on goals that the
goals themselves return from their entry point. doing this in the goal
`work()` function is much cleaner, and a prerequisite to removing more
implicit strong shared references to goals that are currently running.
Change-Id: Ibb3e953ab8482a6a21ce2ed659d5023a991e7923
this simplifies the worker loop, and lets us remove it entirely later.
note that ideally only one promise waiting for interrupts should exist
in the entire system. not one per event loop, one per *process*. extra
interrupt waiters make interrupt response nondeterministic and as such
aren't great for user experience. if anything wants to react to aborts
caused by explicit interruptions, or anything else, those things would
be much better served using RAII guards such as Finally (or KJ_DEFER).
Change-Id: I41d035ff40172d536e098153c7375b0972110d51
this was a triumph. i'm making a note here: huge success. it's hard to
overstate my satisfaction! i'm not even angry. i'm being so sincere ri
actually, no. we *are* angry. this was one dumbass odyssey. nobody has
asked for this. but not doing it would have locked us into old, broken
protocols forever or (possibly worse) forced us to write our own async
framework building on the old did-you-mean-continuations in Worker. if
we had done that we'd be locked into ever more, and ever more complex,
manual state management all over the place. this just could not stand.
Change-Id: I43a6de1035febff59d2eff83be9ad52af4659871
Aria
Lily Ballard
eldritch horrors
Linus Heckemann
Kiara Grouwstra
Dusk Banks
Justin !
raito
alois31
jade
piegames
Maximilian Bosch
Lulu
Rebecca Turner