Commit graph

16490 commits

Author SHA1 Message Date
jade fbdb177b31 manual: Make link to install page clickable
Closes: https://github.com/lix-project/lix/pull/11

Thanks to xsova for suggesting this.

Change-Id: I5958fa4438f4b83d928e8f309aab9a33fb4cbedd
2024-11-18 16:32:20 -08:00
eldritch horrors c859d03013 libfetchers: remove scheme static initializers
provide a initLibFetchers function instead that runs them explicitly.

Change-Id: Ida532e6ff18b72fdc21c9f98df7aceda6713e5df
2024-11-15 16:15:11 +01:00
eldritch horrors 8088927b90 libexpr: initGC -> initLibExpr
we'll put more stuff in there shortly. unfortunately.

Change-Id: I74be957469c3dcd5157f999e2cf23b1c668426d8
2024-11-15 16:15:11 +01:00
eldritch horrors 3dba1d341a libstore: remove TransferItem::phase
transferComplete has outlived its usefulness, and the other two phases
can be a boolean flag telling us whether we're still receiving headers

Change-Id: Ia943f95dcf0ce8ee2cc6d0f44ddf13151aef4346
2024-11-15 16:15:11 +01:00
eldritch horrors 3b1298efce libstore: simplify TransferSource::read
things can be much simpler if the read loop knows whether the transfer
is done or not. this information is available, let's pass it back down

Change-Id: Idb99afeb06a0d7a0999c3f4a1c6ce5adeab5f054
2024-11-15 16:15:11 +01:00
eldritch horrors b22f2bc8e9 libstore: use http for exceptionAbortsDownload tests
file:// urls are no longer handled by curl itself, and this is supposed
to test the curl wrapper. use http to force the wrapper to be involved.

Change-Id: Ib13087db07b3b2f1ae44ce8e3ec7a96d935b1bab
2024-11-15 16:15:11 +01:00
eldritch horrors 986a98d32f libstore: remove curlFileTransfer reference from transfers
they don't need these any more if we consider unpausing and cancelling
of transfers a cooperative affair between transfers and the curl_multi
handler. we can also remove an enable_shared_from_this base class now.

Change-Id: Iefa380df60c0cada53719cdaa500bab953892319
2024-11-15 16:15:11 +01:00
eldritch horrors 4ba19f68fb libstore: improve curl wrapper resource handling
wrap all curl pointers in unique pointers for move correctness. the
destructor of TransferItem can go away completely since it does not
do anything special any more: memory bookkeeping is done using raii
wrappers, and its effects on the transfer state are not observable.
only TransferSource inspects the result, and even it does not check
for transfer results in its own destructor (only during transfers).

Change-Id: Ibb58484703854d5d5425e7c63cf2985d93920e97
2024-11-15 16:15:11 +01:00
eldritch horrors 43777939eb libstore: simplify download buffer handling
keep the download buffer in TransferItem. keep only *one* download
buffer (not one for success returns and one for failures). we also
remove the asynchronous callbacks since they are no longer needed.

Change-Id: I9b1c1e4c59d5333602aea1c36e477c481eaaf98e
2024-11-12 14:01:57 +01:00
eldritch horrors 7d7e16ecec libstore: use curl-builtin read callback
when no custom READFUNCTION is set curl will fall back to using its
READDATA as a FILE pointer and calling fread on it to retrieve data
to send to the peer. we can wrap our upload data in a FILE pointer.
doing so also allows for upload retries, but we don't do that here.

Change-Id: I0509f936229dd978e89135862dacc8a3937cd4de
2024-11-12 14:01:57 +01:00
eldritch horrors 5be7956592 libstore: move remaining retry state out of TransferItem
Change-Id: Ie2c22722bdda18027017e12d10e16931a694743e
2024-11-12 14:01:57 +01:00
Emilia Bopp 8a9094303b Merge "libutil: fix non-ASCII chars in URL encoding" into main 2024-11-11 14:08:37 +00:00
Emilia Bopp a378c61948 libutil: fix non-ASCII chars in URL encoding
Due to the cast to (unsigned int), the encoding appended broken
bytes padding. This is fixed here with a bitmask.

Fixes: lix-project/lix#562
Change-Id: I0c93bd2b8c2f82df208d4693b7254544e3121dc3
2024-11-11 14:26:03 +01:00
alois31 d0bcdfa6c4
libmain/progress-bar: actually don't show short activities
The progress bar is not supposed to show very short activities (<10ms) in order
to prevent excessive flickering. This behaviour was inadvertedly disabled in
commit da4e46dd1f due to not actually skipping
the short activity after calculating that it should have been. Fix this
oversight.

Fixes: lix-project/lix#561
Change-Id: Ibee256a7018cdc431662cd8ea9f6c14ef1706972
2024-11-10 20:00:42 +01:00
alois31 d1d96cc4c8
libexpr: fix install_dir typo
Change-Id: I29f123e29115954d0395c1822128731117fba961
2024-11-10 11:44:48 +01:00
alois31 27af6989bb Merge changes I7b4379a9,If048b73a,I4c802052 into main
* changes:
  libexpr: generate builtins from data
  treewide: generate global settings from data
  libutil: generate experimental and deprecated features from data
2024-11-10 10:19:05 +00:00
eldritch horrors efb04754be libstore: inline TransferItem::init into constructor
this function is called only once per object lifetime, and it's no
longer written such that it must be called by the curl thread. all
curl options set here are independent of the multi handle, there's
no good reason any more to split the constructor in two like that.

Change-Id: Ibfa9075bc1cfdbab979f995c5a942b47add9789d
2024-11-09 20:08:48 +00:00
eldritch horrors 001a078b96 libstore: move file transfer retry handling out of TransferItem
this simplifies the immediate curl wrappers significantly and clarifies
control flow for retries. we can almost see the promise land from here!

Change-Id: Idc66b744631eec9c1ad5a2be388beb942a04f1f9
2024-11-09 20:08:48 +00:00
eldritch horrors d75c399b29 libstore: delocalize TransferSource
future changes will need to add template functions to this class, which
cannot be done for classes declared locally in unctions. otherwise this
is mostly code motion to clean up enqueueFileTransfer a little further.

Change-Id: If9a9d9eb47ceadfa75a4eebd54e2db39f2305643
2024-11-09 20:08:48 +00:00
eldritch horrors 1338e93bc9 libstore: remove curlFileTransfer::enqueueItem return type
it's used only once, and the transfer is used for something else
afterwards as well. returning it hides that the same transfer is
being returned unchanged, and pointlessly extends the signature.

Change-Id: Idbd0af00f9aebd3606f6b537b9b7cc9263a45cf7
2024-11-09 20:08:48 +00:00
eldritch horrors 60e984f0cf libstore: allow explicit cancellation of transfers
we only wait for the worker thread to remove the handle from its multi
and then signal us back. after that's done the transfer can be cleaned
up independently of the multi handle and no transfer progress is made.

we also no longer remove transfer items from multi handles in the item
destructor; only the worker thread can add a transfer and as such only
the worker thread should be responsible for removing them again later.

Change-Id: I6ff57bb5e7c1e91faf8657b257e73d6a72aa928e
2024-11-09 20:08:48 +00:00
eldritch horrors 40cf413a48 libstore: make acceptRanges a function
we don't need to keep this as state. it's only used for retries anyway.

Change-Id: I807f161a516a226567972757ec07ff91b3cf0899
2024-11-09 20:08:48 +00:00
eldritch horrors 90536e27e1 libstore: simplify TransferSource::awaitData
don't pass in a lock that's only used inside this function.

Change-Id: I15c01e9cfe343cf13828ae3b3742c36ae697291f
2024-11-09 20:08:48 +00:00
eldritch horrors 5d02800e57 libstore: do not retry FileTransfer uploads
this only ever worked for empty uploads, and there it worked only by
complete accident: curl was asked to send more data than the wrapper
would provide, which curl would not like and report as an error. the
error would cause a retry with even less data to send, until finally
failing by running into the retry limit. let's just forbid all this.

Change-Id: I229a94b3b8b33e2c6cdb8ea19edd57cd6740e6c6
2024-11-09 20:08:48 +00:00
eldritch horrors 40be91afbf libstore: add filetranfer retry handling tests
we did not have any, despite retry handling being somewhat complex.

Change-Id: I5051a1c0a3861849ff67f512b33f6d3dda12cc95
2024-11-09 20:08:48 +00:00
eldritch horrors fb85228755 libstore: extract eager transfers into new method
we'll need this shared code for kjified transfers that don't use
sources. it's a while out, but we can clean this up now already.

Change-Id: Ife8c160e6ab379761362d6c54aba05093deee99e
2024-11-09 20:08:48 +00:00
eldritch horrors 12156d3beb libstore: fix download thread notifications
since 4ae6fb5a8f dropping a source of a
download might not properly cancel the associated curl transfer after
the transfer was paused. we have also not unpaused the transfer often
enough, only if the transfer buffer had been drained in its entirety.

Change-Id: Ic9298d9df71daa0f3d1c3fd718ed441edae9e863
2024-11-09 20:08:48 +00:00
alois31 c1746300a9
libexpr: generate builtins from data
Most builtins are now generated from data too, with two exceptions:
* Undocumented builtins, since supporting them would add complexity to the
  generator, the harms of the current implementation mostly don't apply, and
  the proper fix is to document them.
* `derivation` is somewhat magic (it is a function, but defined in the code as
  a constant), so the current treatment of having it separately documented is
  kept (for now, at least).
Since it is slightly easier to do and probably a good idea anyway, the builtin
function registrations generated this way are now processed directly in code
and don't go through global variables any more.
Unfortunately, a slight breaking change is introduced because the order of the
builtins' names in the symbol table changes. Hopefully, this will turn out to
not matter in practice.

Change-Id: I7b4379a93ae380b6524e41a916a21c5c6f70555e
2024-11-09 16:17:26 +01:00
alois31 2e0c1a5ea9
treewide: generate global settings from data
Change-Id: If048b73a55d42522827eb9c140a066ba061e957c
2024-11-09 16:17:14 +01:00
alois31 21fc0ddce5
libutil: generate experimental and deprecated features from data
Currently, a bunch of documentation is generated by embedding parts of it in
the nix executable, getting it out again by running it, and then postprocessing
the output. This is bad, since it creates a pointless dependency of the
documentation on the executable, and also makes documentation generation
impossible when cross-compiling.
Instead, both the code and the documentation should be generated from data, see
lix-project/lix#292 . Here we start applying
this approach to the experimental and deprecated features, which are done in
one go since the technical implementation is very similar.
Of course, the actual benefits are not realised yet, since the offending
pattern is used in several more places. These will be fixed later.

Change-Id: I4c802052cc7e865c61119a34b8f1063c4decc9cb
2024-11-09 16:05:12 +01:00
alois31 b967f1d5fe Merge "libutil/config: fix appendable options not getting marked as overridden" into main 2024-11-09 15:01:53 +00:00
alois31 08a362a540
libutil/config: fix appendable options not getting marked as overridden
Commit 4dbbd721eb intended to mark all settings
as overridden when they are. Unfortunately, due to an oversight, this marking
was accidentally performed in the implementation details of non-appendable
options. Move it to the common codepath so that it works for appendable options
too.

Fixes: lix-project/lix#573
Change-Id: Idc3402bac48b19d832acd9b553e16e5791470c26
2024-11-09 12:08:24 +01:00
Lily Ballard d0e0969810 Merge "feat: Add temp-dir setting" into main 2024-11-09 08:44:58 +00:00
Maximilian Bosch 116895acb1 Merge "libexpr/flake: (opinionated) changes to interactive flake config" into main 2024-11-08 17:35:38 +00:00
Maximilian Bosch 86eddb9e27
libexpr/flake: (opinionated) changes to interactive flake config
So I recently saw it the first time in the wild, I liked that you get
interactively asked about the nix.conf settings from the flake, but
there were a few minor things that I'd like to see changed:

* The `(y/N)` was somewhere in the middle of the line. Moved it to
  the end. At first I assumed it was a bug because another thread into
  my terminal while I was answering the question.

* I had to say no four times for a single flake with two options. So if
  you already know you don't want any of the config for _this_ flake, I
  found a `No to all` switch that ignores the rest of the nix.conf
  settings a little more ergonomic than having to stop the invocation,
  looking up the exact wording of `--no-accept-flake-config` and
  restarting it. Hence, I added it.

* Added a note where the choices which settings to trust are persisted.
  My initial assumption was that this went into `nix.conf` which is not
  writable on NixOS, so I said no there as well.

Change-Id: I0a0d9c403f0662df4707697a77f08e6cd003ec6f
2024-11-07 21:47:26 +01:00
Lily Ballard 3c8096e5cb feat: Add temp-dir setting
This adds a new temp-dir setting for controlling the temporary directory
without having to change the TMPDIR env var. This can be used to e.g.
use a path on a case-sensitive store on macOS for temporary files
without changing the TMPDIR var used by interactive shells or commands
invoked with `nix run`.

This also stops unsetting `TMPDIR` on darwin when the env var value
starts with `/var/folders/`, preferring instead to just do the check
when reading `TMPDIR`. This way the inherited `TMPDIR` env var is
preserved for child processes (such as interactive shells).

As a side effect this changes the behavior of `nix-build -o ''` to act
like `nix-build --no-out-link` instead of failing with an error caused
by trying to create a symlink at the cwd.

Fixes: lix-project/lix#253
Fixes: lix-project/lix#112
Change-Id: I9ee826323f2deca62854715a77ca7a373a948a29
2024-11-06 18:11:47 -08:00
Lily Ballard 72cce7be3f Merge "libstore: replace random() calls with atomic counter" into main 2024-11-07 00:15:23 +00:00
V. 1ecfff9c37 Merge "fix: make static build work again" into main 2024-11-06 22:49:56 +00:00
Kiara Grouwstra 72292671a9 Merge "fix(libfetchers): set GitHub API version header, closes #255" into main 2024-11-05 22:44:29 +00:00
eldritch horrors 1c28270c9d libstore: don't hold state lock while unpausing transfers
libcurl may call callbacks during unpause. if libcurl calls these
callbacks often enough they may find that they've exhausted their
allotted buffer space, at which point they will call dataCallback
as provided in enqueueFileTransfer. download() provides callbacks
that have their own state locks and may call unpause on transfers
with *their* state locks they share with curlFileTransfer. it was
possible to cause a deadlock between these two if the curl worker
thread tried to provide some data to a callback with the transfer
state lock held and the user transfer simultaneously unpaused its
associated curl transfer, with its own state lock held. obviously
not a great situation, but avoidable by not holding any lock when
we unpause transfers and execute their callbacks, as is otherwise
the case when a transfer runs normally and is never paused at all

Change-Id: I58556d292adaf7dfb14001d3a6c5c38fa71994da
2024-11-04 23:54:41 +00:00
kloenk 6b7076f81c Merge "add .mailmap" into main 2024-11-04 13:16:16 +00:00
kloenk d73211356a add .mailmap
Add .mailmap file to allow display/email changes in the git log
without rewriting the history

Change-Id: Ie507aba563cd4fa8ba24e65269aefc647c6376ed
2024-11-04 13:46:10 +01:00
Lily Ballard b1a0e3c002 Merge "libstore: remove unnecessary lstats in optimisePath" into main 2024-11-03 05:15:40 +00:00
raito 763a61bb7c Merge "Revert "Reject weak hash algorithms as SRIs, and warn in any other cases"" into main 2024-11-02 19:50:52 +00:00
Artemis Tosini 7df8b15b39 Merge "libstore: Fix FreeBSD build" into main 2024-11-02 17:20:46 +00:00
Artemis Tosini 3aad088cf1
libstore: Fix FreeBSD build
FreeBSD was left out of a few refactors over the last few months.

Add a header and register the store implementation so it's back
to working as well as it was before.

Change-Id: I6f7b2ceb557c290f2d9e0d7f207b3fea87b353ed
2024-11-02 07:23:55 +00:00
Artemis Tosini add8a4df9f
package.nix: Fix cross devShell
The devShell relied on several packages directly from `pkgs`
or used with a non-splice-aware functions.
These would be built for the host system, making them useless
in a devShell for the build system.

Make sure that all packages are for the build system when needed.

Some other minor changes also required:
 * Make devShells use `clangStdenv` because GCC is currently broken
 * Disable rr when making a cross stdenv

Change-Id: Iee5f8a1a0c594139a50f2261b203491bd1644866
2024-11-02 06:32:20 +00:00
Artemis Tosini 9903bed3f4
flake.nix: Fix cross build
The lix package currently fails unless it's using a clang stdenv.
However, the flake's cross build outputs (e.g. `packages.x86_64-linux.nix-armv7l-linux`)
used the default stdenv, normally gcc.
Replace this with clang to fix package build.

Also take this opportunity to remove the no longer necessary `useLLVM = true`
override on FreeBSD. Since 24.05, nixpkgs always sets `useLLVM = true`
on FreeBSD in `lib.systems.elaborate`.

Change-Id: I939302e4f6385291fa9e582d38d908c42f6db89a
2024-11-02 00:57:09 +00:00
V. 486d1a1437 fix: make static build work again
I copied the workaround from here:
a2de0eff59
Properly fixing the issue upstream will be much more difficult.

Closes: lix-project/lix#527
Change-Id: I967d53fa9e80510b620df485af448f76bd9aa52a
2024-11-01 16:55:47 +04:00
raito 6e2349d2e1 Revert "Reject weak hash algorithms as SRIs, and warn in any other cases"
This reverts commit 02c35ea9df.

Reason for revert: this code path is also used for `Input::getRev()`, i.e. flakes VCS revision validation, which, in the case of Git, are using SHA1.
As a result, this cause too much noise due to SHA1 revisions in Flakes.

Change-Id: I8064c1ebc26e4e83b627f0803a7a9ba56cfe1f37
2024-11-01 11:59:59 +00:00