Commit graph

16456 commits

Author SHA1 Message Date
Maximilian Bosch 116895acb1 Merge "libexpr/flake: (opinionated) changes to interactive flake config" into main 2024-11-08 17:35:38 +00:00
Maximilian Bosch 86eddb9e27
libexpr/flake: (opinionated) changes to interactive flake config
So I recently saw it the first time in the wild, I liked that you get
interactively asked about the nix.conf settings from the flake, but
there were a few minor things that I'd like to see changed:

* The `(y/N)` was somewhere in the middle of the line. Moved it to
  the end. At first I assumed it was a bug because another thread into
  my terminal while I was answering the question.

* I had to say no four times for a single flake with two options. So if
  you already know you don't want any of the config for _this_ flake, I
  found a `No to all` switch that ignores the rest of the nix.conf
  settings a little more ergonomic than having to stop the invocation,
  looking up the exact wording of `--no-accept-flake-config` and
  restarting it. Hence, I added it.

* Added a note where the choices which settings to trust are persisted.
  My initial assumption was that this went into `nix.conf` which is not
  writable on NixOS, so I said no there as well.

Change-Id: I0a0d9c403f0662df4707697a77f08e6cd003ec6f
2024-11-07 21:47:26 +01:00
Lily Ballard 72cce7be3f Merge "libstore: replace random() calls with atomic counter" into main 2024-11-07 00:15:23 +00:00
V. 1ecfff9c37 Merge "fix: make static build work again" into main 2024-11-06 22:49:56 +00:00
Kiara Grouwstra 72292671a9 Merge "fix(libfetchers): set GitHub API version header, closes #255" into main 2024-11-05 22:44:29 +00:00
eldritch horrors 1c28270c9d libstore: don't hold state lock while unpausing transfers
libcurl may call callbacks during unpause. if libcurl calls these
callbacks often enough they may find that they've exhausted their
allotted buffer space, at which point they will call dataCallback
as provided in enqueueFileTransfer. download() provides callbacks
that have their own state locks and may call unpause on transfers
with *their* state locks they share with curlFileTransfer. it was
possible to cause a deadlock between these two if the curl worker
thread tried to provide some data to a callback with the transfer
state lock held and the user transfer simultaneously unpaused its
associated curl transfer, with its own state lock held. obviously
not a great situation, but avoidable by not holding any lock when
we unpause transfers and execute their callbacks, as is otherwise
the case when a transfer runs normally and is never paused at all

Change-Id: I58556d292adaf7dfb14001d3a6c5c38fa71994da
2024-11-04 23:54:41 +00:00
kloenk 6b7076f81c Merge "add .mailmap" into main 2024-11-04 13:16:16 +00:00
kloenk d73211356a add .mailmap
Add .mailmap file to allow display/email changes in the git log
without rewriting the history

Change-Id: Ie507aba563cd4fa8ba24e65269aefc647c6376ed
2024-11-04 13:46:10 +01:00
Lily Ballard b1a0e3c002 Merge "libstore: remove unnecessary lstats in optimisePath" into main 2024-11-03 05:15:40 +00:00
raito 763a61bb7c Merge "Revert "Reject weak hash algorithms as SRIs, and warn in any other cases"" into main 2024-11-02 19:50:52 +00:00
Artemis Tosini 7df8b15b39 Merge "libstore: Fix FreeBSD build" into main 2024-11-02 17:20:46 +00:00
Artemis Tosini 3aad088cf1
libstore: Fix FreeBSD build
FreeBSD was left out of a few refactors over the last few months.

Add a header and register the store implementation so it's back
to working as well as it was before.

Change-Id: I6f7b2ceb557c290f2d9e0d7f207b3fea87b353ed
2024-11-02 07:23:55 +00:00
Artemis Tosini add8a4df9f
package.nix: Fix cross devShell
The devShell relied on several packages directly from `pkgs`
or used with a non-splice-aware functions.
These would be built for the host system, making them useless
in a devShell for the build system.

Make sure that all packages are for the build system when needed.

Some other minor changes also required:
 * Make devShells use `clangStdenv` because GCC is currently broken
 * Disable rr when making a cross stdenv

Change-Id: Iee5f8a1a0c594139a50f2261b203491bd1644866
2024-11-02 06:32:20 +00:00
Artemis Tosini 9903bed3f4
flake.nix: Fix cross build
The lix package currently fails unless it's using a clang stdenv.
However, the flake's cross build outputs (e.g. `packages.x86_64-linux.nix-armv7l-linux`)
used the default stdenv, normally gcc.
Replace this with clang to fix package build.

Also take this opportunity to remove the no longer necessary `useLLVM = true`
override on FreeBSD. Since 24.05, nixpkgs always sets `useLLVM = true`
on FreeBSD in `lib.systems.elaborate`.

Change-Id: I939302e4f6385291fa9e582d38d908c42f6db89a
2024-11-02 00:57:09 +00:00
V. 486d1a1437 fix: make static build work again
I copied the workaround from here:
a2de0eff59
Properly fixing the issue upstream will be much more difficult.

Closes: lix-project/lix#527
Change-Id: I967d53fa9e80510b620df485af448f76bd9aa52a
2024-11-01 16:55:47 +04:00
raito 6e2349d2e1 Revert "Reject weak hash algorithms as SRIs, and warn in any other cases"
This reverts commit 02c35ea9df.

Reason for revert: this code path is also used for `Input::getRev()`, i.e. flakes VCS revision validation, which, in the case of Git, are using SHA1.
As a result, this cause too much noise due to SHA1 revisions in Flakes.

Change-Id: I8064c1ebc26e4e83b627f0803a7a9ba56cfe1f37
2024-11-01 11:59:59 +00:00
Lily Ballard 834450e237 Merge changes Ibb849b68,I501397c8 into main
* changes:
  libstore: ignore broken symlinks in ssl-cert-file default
  change-authors: add lilyball
2024-11-01 03:51:21 +00:00
Aria 8005d17365 Merge "Reject weak hash algorithms as SRIs, and warn in any other cases" into main 2024-10-31 22:13:27 +00:00
Lily Ballard 69957a971e libstore: replace random() calls with atomic counter
random() is not thread-safe, it relies on global state, and calling it
from worker threads can result in multiple threads producing the same
value. It also doesn't guarantee unique values even in single-threaded
use.

Use an atomic counter for the use-case of generating temporary paths,
and switch to a thread-local RNG for the one remaining call.

This will probably fix https://github.com/NixOS/nix/issues/7273 though
I'm not willing to risk corrupting my store to find out.

Change-Id: I4c4c4c9796613573ffefd29cc8efe3d07839facc
2024-10-30 19:54:43 -07:00
Lily Ballard 684f93e783 libstore: ignore broken symlinks in ssl-cert-file default
Also tweak `pathAccessible` to ignore other relevant errors too. It was
documented as ignoring permission errors but it was only ignoring
`EPERM`, which comes from the darwin sandbox, and not ignoring `EACCESS`
which is the real permission error. I figured it also makes sense to
ignore `ELOOP`.

Fixes: lix-project/lix#560
Change-Id: Ibb849b68d07386eb80afb52b57f7d12b3a48a202
2024-10-30 19:50:38 -07:00
Lily Ballard 11950a0a79 change-authors: add lilyball
Change-Id: I501397c8e3a215a2ccb6074100e2508bae98d1a9
2024-10-30 19:50:38 -07:00
eldritch horrors beb193d1e2 libstore: remove our custom early timeout handling
curl handles timeouts internally when it acts as the main event loop. we
only need to wake up to add new transfers to the multi handle or restart
an existing transfer that has passed its restart wait time. periodically
waking up is not required for curl, and we don't need it any more either

Change-Id: Ic774e1d9519f807cda1a89694bc3ede75216f329
2024-10-30 22:52:19 +00:00
eldritch horrors 4ae6fb5a8f libstore: pause only stalling transfers
don't pause the entire curl thread. we have multiple consumer threads
after all, not just one, so stalling all of them is likely not great.

note that libcurl advises against using transfer pauses if compressed
encodings are allowed and automatically decoded. this should not lead
to problems in practice because our data is usually not compressed to
such a degree that curl buffering *uncompressed* data matters. should
this cause problems we can reintroduce the whole-thread pause, but we
will probably get away with this until the entire file transfer class
is made kj::Promise-using async (and *then* curl can be hardpaused if
it cannot get rid of its data, solving the problem once and for all).

Change-Id: I218e41bfa5a27c7454eafb0bdb54f2a29a7f6493
2024-10-30 22:52:19 +00:00
Linus Heckemann c95b73d8a1 Merge "libstore: report all differing outputs rather than just the first" into main 2024-10-30 19:04:57 +00:00
Kiara Grouwstra a778b0f85a fix(libfetchers): set GitHub API version header, closes #255
Sets the `X-GitHub-Api-Version` header to `2022-11-28` for calls to the
GitHub API.
This follows the later version as per
https://docs.github.com/en/rest/about-the-rest-api/api-versions?apiVersion=2022-11-28.

This affected the check on whether to use the API versus unauthenticated
calls as well, given the headers would no longer be empty if the
authentication token were missing.
The workaround used here is to use a check similar to an existing
check for the token.

In the current implementation, headers are (still) similarly sent to
non-authenticated as well as GitHub on-prem calls.
For what it's worth, manual curl calls with such a header seemed to
break nor unauthenticated calls nor ones to the github.com API.

Change-Id: I6e10839e6b99cb65eb451e923b2a64f5d3c0f578
2024-10-30 18:40:13 +01:00
V. 56ead73fda Merge "chore: remove monolithic coreutils requirement" into main 2024-10-30 16:53:43 +00:00
Dusk Banks 22eb47f0fd tests/functional/flakes: test with UTF-8 bullets
using UTF-8 bullets in the sample avoids locale confusion where Bash
doesn't know to treat `•` as a single character.

Signed-off-by: Dusk Banks <me@bb010g.com>
Change-Id: I829019b66e93e6d33ac3a6641df07d0dd2332a5a
2024-10-30 08:21:58 -07:00
Dusk Banks 8b2f8d538b Merge "libstore: restore mode after changing xattrs" into main 2024-10-30 14:56:43 +00:00
V. fb1b211037 chore: remove monolithic coreutils requirement
It's only used in a couple of tests, and only in such a way that
replacing it with a random command suffices.
I also removed a few pointless uses of the variable.

Fixes: lix-project/lix#376
Change-Id: I90aedb61d64b02f7c9b007e72f9d614cc1b37a2e
2024-10-30 15:12:35 +04:00
Linus Heckemann 8b0ac51f12 libstore: report all differing outputs rather than just the first
Before:

error: derivation '/nix/store/4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'

After:

error: derivation '4spy3nz1661zm15gkybsy1h5f36aliwx-python3.11-test-1.0.0.drv' may not be deterministic: outputs differ
         output differs: output '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist' differs from '/nix/store/ccqcp01zg18wp9iadzmzimqzdi3ll08d-python3.11-test-1.0.0-dist.check'
         output differs: output '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0' differs from '/nix/store/yl59v08356i841c560alb0zmk7q16klb-python3.11-test-1.0.0.check'

Change-Id: Ib2871fa602bf1fa9c00e2565b3a2e1b26f908152
2024-10-29 18:34:54 +01:00
eldritch horrors 9c22a4d31b libstore: don't use curl for file:// downloads
curl can't pause downloads of file:// urls, which is very much in the
way of making the curl wrapper fully asynchronous. we can emulate all
the things curl does, but unfortunately curl is *rather extensive* in
its support of frankly weird shit. hopefully this subset of features,
which notably does not include curl readdir support, is enough for us

Change-Id: I5f67768c4b512565655b94b0421270c7dbbd8d11
2024-10-28 18:52:49 +00:00
eldritch horrors c83b13eafd libstore: reunify all file transfer methods again
with the api cleaned up we can suddenly reunify uploads, downloads, and
existence checks through curl in the same wrapper function. uploads and
existence checks simply don't use the result source, and given that all
transfers (or at least *most* transfers to date) go through the network
the few extra allocations do not hurt us at all. even for file:// calls
the overhead won't much matter as going to disk and back *is* expensive

Change-Id: I4f9ca6681a8fc303377b4cf4c63e3363ae32c18b
2024-10-28 18:52:49 +00:00
eldritch horrors d65838a900 libstore: remove FileTransfer::enqueueDownload
it's no longer needed. `download` can do everything `enqueueDownload`
did, and a lot more. e.g. not block the calling thread, for instance.

Change-Id: I4b36235ed707c92d117b4c33efa3db50d26f9a84
2024-10-28 18:52:49 +00:00
eldritch horrors c68f0cdf00 libstore: return transfer metadata from download
as promised earlier. nothing uses it yet, but just you wait.

Change-Id: I77d185578d96c2134b756d20f2fcf1c02de0da6f
2024-10-28 18:52:49 +00:00
eldritch horrors 14eff10fe4 libstore: split callback into metadata and finished parts
this will let us return metadata from FileTransfer::download, which in
turn is necessary to remove enqueueDownload. it also opens avenues for
streaming downloads that keep download metadata instead of dropping it

Change-Id: If0fc6af5eb2aeb689fc866c345c9d7bce4d59f2d
2024-10-28 18:52:49 +00:00
eldritch horrors 923abe347c libstore: use data callback for simple downloads too
this is an intermediate step towards removing enqueueDownload entirely.

Change-Id: I05ec0c7f4a234fdc966e5005308b37f6f905d433
2024-10-28 18:52:49 +00:00
eldritch horrors 64864c3730 libstore: pass only data to TransferItem data callback
with encoding being handled by curl the reference is no longer needed.

Change-Id: Ibfaf5f55e5314e81ce45ba4523b960c401dd2e1c
2024-10-28 18:52:49 +00:00
eldritch horrors 10488f7431 libstore: use curl content-encoding support, not our own
let's use the automatic decoding functions curl provides instead of
implementing them ourselves for the dubious ability to support both
xz and bzip2 encodings as well, neither of which anything will send

Change-Id: I3edfebeb596a0e9d5c986efca9270501c996f2dd
2024-10-28 18:52:49 +00:00
Justin ! 8c567c0424 Merge "libutil: implement PathsSetting<PathSet>" into main 2024-10-28 14:22:59 +00:00
Lily Ballard a042bbde56 libstore: remove unnecessary lstats in optimisePath
We were calling `lstat()` twice on the link path, and a third time if it
existed. We only need to call it once.

Change-Id: Ifadc2f24681648d0ec7e4bad5c6d2dcb2e560e7b
2024-10-27 20:03:29 -07:00
eldritch horrors 61146c73ce Merge changes I0220cedd,Ide0c0512,I6fcd920e,I85ec62ff,I35853a91, ... into main
* changes:
  libstore: check that transfer headers don't change during retries
  libstore: use effective transfer url for retries
  libstore: collect effective url and cachedness earlier
  libstore: remove TransferItem::active
  libstore: always allocate TransferItem::req
  libstore: remove FileTransferResult::data
  libstore: de-future-ize FileTransfer::enqueueUpload
  libstore: remove FileTransferRequest
  libstore: remove FileTransferRequest::expectedETag
  libstore: remove FileTransferResult::bodySize
  libstore: remove FileTransferRequest::verifyTLS
  libstore: remove FiletransferRequest::head
2024-10-28 01:36:45 +00:00
Linus Heckemann f55ed83991 Merge "libutil: handle json builder log messages with unexpected format" into main 2024-10-27 22:54:11 +00:00
eldritch horrors 212a14bb1f libstore: check that transfer headers don't change during retries
etag changing implies with high probability that the content of the
resource changed. immutable url changing implies that the immutable
url we got previously was wrong, which is probably a server bug. if
the encoding changes our decoding will break completely, so that is
also very illegal. one notable change we still allow is etags going
away completely, mostly since this does not imply any data changes.

Change-Id: I0220ceddc3fd732cd1b3bb39b40021cc631baadc
2024-10-27 21:44:38 +00:00
eldritch horrors 7c716b9716 libstore: use effective transfer url for retries
do not retread the entire redirection path if we've seen the end of the
road. this avoids silently downloading wrong data, and notifies us when
a url we've received data from turns into a redirect when retrying. for
reasons of simplicity we don't turn of libcurl redirects on retries. if
we did that we'd have to conditionally process http status codes, which
sounds annoying and would make the header callback even more of a mess.

Change-Id: Ide0c0512ef9b2579350101246d654a2375541a39
2024-10-27 21:44:38 +00:00
eldritch horrors 2b3bdda027 libstore: collect effective url and cachedness earlier
this will let us return metadata for a transfer without having to wait
for the entire transfer to complete. more importantly for current uses
though is that we could now send retries to the effective url directly
instead of retreading the entire redirect path. this improves latency,
and in such cases where redirects change while we're downloading it'll
also improve correctness (previously we'd silently download bad data).

Change-Id: I6fcd920eb96fbdb2e960b73773c0b854e0300e99
2024-10-27 21:44:38 +00:00
eldritch horrors 97c76c4655 libstore: remove TransferItem::active
it's always legal to call curl_multi_remove_handle on a valid pair of
multi and easy handles. removing an easy handle that is not currently
attached to a multi handle is a no-op, and removing an easy handle of
a different multi handle is something we can't reasonably trigger. if
we *did* ever manage it would result in an error we'd ignore, and the
handles in question would not be changed at all. this is just simpler

Change-Id: I85ec62ff89385981ca49d243376b9c32586bd128
2024-10-27 21:44:38 +00:00
eldritch horrors c82407fc1e libstore: always allocate TransferItem::req
there's no reason not to do this. also improve error handling a bit.

Change-Id: I35853a919fa58a9a34ad47ffab6de77ba6f7fb86
2024-10-27 21:44:38 +00:00
eldritch horrors 982d049d3b libstore: remove FileTransferResult::data
return it as a separate item in a pair instead. this will let us remove
enqueueDownload() in favor of returning metadata from download() itself

Change-Id: I74fad2ca15f920da1eefabc950c2baa2c360f2ba
2024-10-27 21:44:38 +00:00
eldritch horrors 5cd7055044 libstore: de-future-ize FileTransfer::enqueueUpload
it's only used once, and synchronously at that.

Change-Id: Ife9db15dd97bc0de8de59a25d27f3f7afeb8791b
2024-10-27 21:44:38 +00:00
eldritch horrors 6f18e1ebde libstore: remove FileTransferRequest
it's just a uri and some headers now. those can be function arguments
with no loss of clarity. *actual* additional arguments, for example a
TLS context with additional certificates, could be added on a new and
improved FileTransfer class that carries not just a backend reference
but some real, visible context for its transfers. curl not being very
multi-threading-friendly when using multi handles will make sharing a
bit hard anyway once we drop the single global download worker thread

Change-Id: Id2112c95cbd118c6d920488f38d272d7da926460
2024-10-27 21:44:38 +00:00