Commit graph

16512 commits

Author SHA1 Message Date
eldritch horrors a911b44a31 libstore: break up QueryMissingContext::doPath
std::visit makes coroutines nearly impossible to use correctly.

Change-Id: I085c08b595a5d4a43208eb97f023aa00b48bff45
2024-11-20 14:17:02 +00:00
eldritch horrors 74d820e5e8 libstore: move Store::queryMissing into a fresh struct
we want to get rid of all these local lambdas, they make coroutines impossible.

Change-Id: I94336f0844e9cdb0aa27788d0ee5f99925f24e0c
2024-11-20 14:17:02 +00:00
eldritch horrors 9cf91b7385 cli infra: modernize legacy command interface
give legacy commands their program name and c++-converted argv directly
instead of passing on the (argc, argv) pair untouched. all are required
to process these things, and all of them do it in exactly the same way.
we can also remove a few old helpers only used to make this less awful.

Change-Id: I4ecd02343bca0cf85faf6fe043031d4f64c5f29c
2024-11-20 14:17:02 +00:00
eldritch horrors c4a077d0b8 libcmd: drop non-default ctors virtual inheritance
it's a trap. it's always a trap. non-default ctors of virtual base
classes must be called with the same parameters every time or very
surprising things will happen. good luck if you virtually derive a
class more than once with different ctor arguments. that'll break.

Change-Id: I1345963e69e98f37c3cbd070bae164d89eade455
2024-11-20 14:17:02 +00:00
eldritch horrors ac74dc3a2b treewide: drop trivial leaf virtual inheritance
this does nothing on classes that are never used as base classes and
have only a single base class of their own. all of these classes are
also made final to prove that they are indeed in this category. once
we need to derive any of them we can erase their `final` specifiers.

Change-Id: I4bd8e50a58815227392b5a5a762fda4542d15bae
2024-11-20 14:17:02 +00:00
eldritch horrors 34e592ea6a libcmd, nix: drop NixMultiCommand
there are no uses of plain MultiCommand as a base class *except* in
NixArgs, which is the only one that does not implement run(). there
is not much of a reason not to implement a run member there though,
so let's just do that and get rid of this weird intermediate class.

Change-Id: Ie84e3acd071b43bc186a2bac87646cbfb3aff845
2024-11-20 14:17:02 +00:00
eldritch horrors 95a9a4cece libstore: don't derive store classes from their configs
the very slight speedup in config setting access is not worth the
maintenance overhead of conflating concers like this. the virtual
inheritance scheme used for configs requires too much duplication
of base class constructor arguments to be worth doing. perhaps we
should get rid of all virtual inheritance of data-membered bases?

Change-Id: I4acf5ceaedb4ed7476efe1114c2e065ec72d2c6d
2024-11-20 14:59:40 +01:00
eldritch horrors b0d7a81613 fix tooling after include reorganization
clangd broke because it can't look through symlinks. compile_commands
manipulation does not fix it, clangd configuration does not fix it, a
vfs overlay does not fix it, and while a combination of those can fix
it with a bind mount in place that's just too cursed to even consider

clangd bug: https://github.com/llvm/llvm-project/issues/116877

Change-Id: I8e3e8489548eb3a7aa65ac9d12a5ec8abf814aec
2024-11-19 22:55:32 +00:00
jade f116608a20 Merge "libstore: abort all curl transfers on interrupt" into main 2024-11-19 05:03:47 +00:00
jade 0bec915522 Merge "unnamed threads: Obliterate" into main 2024-11-19 05:02:48 +00:00
jade 519957bd59 unnamed threads: Obliterate
Ever read gdb output and you just kinda get a headache because you have
to infer what a thread is by reading the stack trace? It's not hard, but
we could also just never have to do that again, which is also not hard.

Sample:

(gdb) info thr
  Id   Target Id                    Frame
* 1    LWP 3719283 "nix-daemon"     0x00007e558587da0f in accept ()
   from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6
  2    LWP 3719284 "signal handler" 0x00007e55857b2bea in sigtimedwait ()
   from target:/nix/store/c10zhkbp6jmyh0xc5kd123ga8yy2p4hk-glibc-2.39-52/lib/libc.so.6

The API design for this is forced by the macOS pthread_setname_np only
being able to change the current thread's name, but if we just conform
everything to that, it works everywhere.

Change-Id: I2b1d6ed41e3c94170cb0b4e73ad66f239ebd9c88
2024-11-18 18:53:40 -08:00
jade 950c213ddf Merge "feat: better warning for common SSL errors" into main 2024-11-19 02:48:24 +00:00
eldritch horrors 16bed313c6 libstore: abort all curl transfers on interrupt
give the transfer progress callback a chance to register all transfers
as aborted when an interrupt event occurs. checkInterrupt() will throw
an Interrupted exception if an interrupt signal was caught, which will
immediately break out of the curl loop. it isn't even necessary within
the curl main loop as curl guarantees to call progress callbacks about
once per second (or more often), and we already abort transfers from a
progress update if a signal was caught. this may delay shutdown a bit,
but "about one second" as should not be noticeable in most situations.

fixes #577

Change-Id: Ida4e72732562bdf6560e4f182ecdcdb663c6dda5
2024-11-19 02:49:41 +01:00
jade 33e832ba5a Merge "manual: Make link to install page clickable" into main 2024-11-19 01:47:03 +00:00
Tom Bereknyei 6334e91b2f feat: better warning for common SSL errors
(cherry picked from commit 3e5bf903413f420c1f997e4b55140761172b8434)
Upstream-PR: https://github.com/NixOS/nix/pull/11246
Change-Id: Iccd92721bbcd92604a4b0e5c845242bdf3dd0f66
2024-11-18 16:49:35 -08:00
jade fbdb177b31 manual: Make link to install page clickable
Closes: https://github.com/lix-project/lix/pull/11

Thanks to xsova for suggesting this.

Change-Id: I5958fa4438f4b83d928e8f309aab9a33fb4cbedd
2024-11-18 16:32:20 -08:00
jade 6a9b66357e pkg-config: add the base lix include-dir as an include-dir
This allows <lix/*> includes to work from external projects not taking
Lix as a buildInput with Nix, but just using pkg-config.

Change-Id: I5b24fd0981344b013beaf406c16988dad17f0edb
2024-11-18 12:31:41 -08:00
jade 13f54451e6 meson: remove the ability to use unqualified includes from inside Lix
These are still supported externally with pkg-config, but this commit
ensures they no longer build internally.

Change-Id: I2fa38f85e43b18f1f1dbcfdb0367de8f192aca2c
2024-11-18 12:31:41 -08:00
jade 103cd44a04 Include rearrangement: rest
Change-Id: If9b5502ff348d358d7007b885e26e98a0d228f39
2024-11-18 12:31:19 -08:00
jade eeaa72b62b Include rearrangement: libexpr
Change-Id: Ib230a3021d544d3ab1376542b73ea1df1a220a3c
2024-11-18 12:22:39 -08:00
jade 94ca95714e Include rearrangement: libstore
Change-Id: I27c83510d9053bb8bdd7eda9ad3a4b77d633056a
2024-11-18 12:22:39 -08:00
jade df0e1993b7 Include rearrangement: libutil
Change-Id: Ibf4ee57e430ce1cd175f994d1db7964c8407ffa7
2024-11-18 12:22:39 -08:00
jade b62cc7b361 meson: prepare for include rearrangement
Context: we have include paths that are "types.hh" and similarly common
names. We currently have these compatibly available as
"lix/libutil/types.hh" externally but *not yet internally*. This is
because we don't have any way for the src directory to appear as
`"lix/"` from inside of Lix: the lix/ include directory is created by
the install process.

The goal of this whole thing is to make it clearer which component of
Lix that files are a part of, which should hopefully help at least a
little bit to new developers. One disadvantage of un-mixing these is
that it will cause some API changes if we ever move a file between
libraries, but that is not very common, and we don't care that much
about external API users.

This was planned for a while and is why we have a FixIncludes check to
begin with.

Personally I don't see a great benefit in rearranging our source code,
and in fact, it would probably be counterproductive:
- Moving the includes into a separate `include/` directory would just
  make developers have to deal with more directories, when we can
  already generate the desired layout through the build process.
- This would also decouple the .cc and .hh files which currently
  conventionally have each others' definitions and declarations
  respectively, right next to each other, making it easier for them to
  feel decoupled and diverge.

Content: Add ../include as an include directory so that lix/ in include
paths will resolve to src/ within Lix itself, just as it does externally
today. This prepares for a further series of commits applying the actual
change to each library one-by-one by accepting both include versions at
once.

This could have been done with ../ and a symlink called lix, but we
would like to not accept libexpr/foo.hh internally for it would be
broken externally, so we need an otherwise empty directory for the
include.

Change-Id: Ideac17faadae2bcea2dffbab34eb27c582ede399
2024-11-18 12:22:39 -08:00
eldritch horrors c859d03013 libfetchers: remove scheme static initializers
provide a initLibFetchers function instead that runs them explicitly.

Change-Id: Ida532e6ff18b72fdc21c9f98df7aceda6713e5df
2024-11-15 16:15:11 +01:00
eldritch horrors 8088927b90 libexpr: initGC -> initLibExpr
we'll put more stuff in there shortly. unfortunately.

Change-Id: I74be957469c3dcd5157f999e2cf23b1c668426d8
2024-11-15 16:15:11 +01:00
eldritch horrors 3dba1d341a libstore: remove TransferItem::phase
transferComplete has outlived its usefulness, and the other two phases
can be a boolean flag telling us whether we're still receiving headers

Change-Id: Ia943f95dcf0ce8ee2cc6d0f44ddf13151aef4346
2024-11-15 16:15:11 +01:00
eldritch horrors 3b1298efce libstore: simplify TransferSource::read
things can be much simpler if the read loop knows whether the transfer
is done or not. this information is available, let's pass it back down

Change-Id: Idb99afeb06a0d7a0999c3f4a1c6ce5adeab5f054
2024-11-15 16:15:11 +01:00
eldritch horrors b22f2bc8e9 libstore: use http for exceptionAbortsDownload tests
file:// urls are no longer handled by curl itself, and this is supposed
to test the curl wrapper. use http to force the wrapper to be involved.

Change-Id: Ib13087db07b3b2f1ae44ce8e3ec7a96d935b1bab
2024-11-15 16:15:11 +01:00
eldritch horrors 986a98d32f libstore: remove curlFileTransfer reference from transfers
they don't need these any more if we consider unpausing and cancelling
of transfers a cooperative affair between transfers and the curl_multi
handler. we can also remove an enable_shared_from_this base class now.

Change-Id: Iefa380df60c0cada53719cdaa500bab953892319
2024-11-15 16:15:11 +01:00
eldritch horrors 4ba19f68fb libstore: improve curl wrapper resource handling
wrap all curl pointers in unique pointers for move correctness. the
destructor of TransferItem can go away completely since it does not
do anything special any more: memory bookkeeping is done using raii
wrappers, and its effects on the transfer state are not observable.
only TransferSource inspects the result, and even it does not check
for transfer results in its own destructor (only during transfers).

Change-Id: Ibb58484703854d5d5425e7c63cf2985d93920e97
2024-11-15 16:15:11 +01:00
eldritch horrors 43777939eb libstore: simplify download buffer handling
keep the download buffer in TransferItem. keep only *one* download
buffer (not one for success returns and one for failures). we also
remove the asynchronous callbacks since they are no longer needed.

Change-Id: I9b1c1e4c59d5333602aea1c36e477c481eaaf98e
2024-11-12 14:01:57 +01:00
eldritch horrors 7d7e16ecec libstore: use curl-builtin read callback
when no custom READFUNCTION is set curl will fall back to using its
READDATA as a FILE pointer and calling fread on it to retrieve data
to send to the peer. we can wrap our upload data in a FILE pointer.
doing so also allows for upload retries, but we don't do that here.

Change-Id: I0509f936229dd978e89135862dacc8a3937cd4de
2024-11-12 14:01:57 +01:00
eldritch horrors 5be7956592 libstore: move remaining retry state out of TransferItem
Change-Id: Ie2c22722bdda18027017e12d10e16931a694743e
2024-11-12 14:01:57 +01:00
Emilia Bopp 8a9094303b Merge "libutil: fix non-ASCII chars in URL encoding" into main 2024-11-11 14:08:37 +00:00
Emilia Bopp a378c61948 libutil: fix non-ASCII chars in URL encoding
Due to the cast to (unsigned int), the encoding appended broken
bytes padding. This is fixed here with a bitmask.

Fixes: #562
Change-Id: I0c93bd2b8c2f82df208d4693b7254544e3121dc3
2024-11-11 14:26:03 +01:00
alois31 d0bcdfa6c4
libmain/progress-bar: actually don't show short activities
The progress bar is not supposed to show very short activities (<10ms) in order
to prevent excessive flickering. This behaviour was inadvertedly disabled in
commit da4e46dd1f due to not actually skipping
the short activity after calculating that it should have been. Fix this
oversight.

Fixes: #561
Change-Id: Ibee256a7018cdc431662cd8ea9f6c14ef1706972
2024-11-10 20:00:42 +01:00
alois31 d1d96cc4c8
libexpr: fix install_dir typo
Change-Id: I29f123e29115954d0395c1822128731117fba961
2024-11-10 11:44:48 +01:00
alois31 27af6989bb Merge changes I7b4379a9,If048b73a,I4c802052 into main
* changes:
  libexpr: generate builtins from data
  treewide: generate global settings from data
  libutil: generate experimental and deprecated features from data
2024-11-10 10:19:05 +00:00
eldritch horrors efb04754be libstore: inline TransferItem::init into constructor
this function is called only once per object lifetime, and it's no
longer written such that it must be called by the curl thread. all
curl options set here are independent of the multi handle, there's
no good reason any more to split the constructor in two like that.

Change-Id: Ibfa9075bc1cfdbab979f995c5a942b47add9789d
2024-11-09 20:08:48 +00:00
eldritch horrors 001a078b96 libstore: move file transfer retry handling out of TransferItem
this simplifies the immediate curl wrappers significantly and clarifies
control flow for retries. we can almost see the promise land from here!

Change-Id: Idc66b744631eec9c1ad5a2be388beb942a04f1f9
2024-11-09 20:08:48 +00:00
eldritch horrors d75c399b29 libstore: delocalize TransferSource
future changes will need to add template functions to this class, which
cannot be done for classes declared locally in unctions. otherwise this
is mostly code motion to clean up enqueueFileTransfer a little further.

Change-Id: If9a9d9eb47ceadfa75a4eebd54e2db39f2305643
2024-11-09 20:08:48 +00:00
eldritch horrors 1338e93bc9 libstore: remove curlFileTransfer::enqueueItem return type
it's used only once, and the transfer is used for something else
afterwards as well. returning it hides that the same transfer is
being returned unchanged, and pointlessly extends the signature.

Change-Id: Idbd0af00f9aebd3606f6b537b9b7cc9263a45cf7
2024-11-09 20:08:48 +00:00
eldritch horrors 60e984f0cf libstore: allow explicit cancellation of transfers
we only wait for the worker thread to remove the handle from its multi
and then signal us back. after that's done the transfer can be cleaned
up independently of the multi handle and no transfer progress is made.

we also no longer remove transfer items from multi handles in the item
destructor; only the worker thread can add a transfer and as such only
the worker thread should be responsible for removing them again later.

Change-Id: I6ff57bb5e7c1e91faf8657b257e73d6a72aa928e
2024-11-09 20:08:48 +00:00
eldritch horrors 40cf413a48 libstore: make acceptRanges a function
we don't need to keep this as state. it's only used for retries anyway.

Change-Id: I807f161a516a226567972757ec07ff91b3cf0899
2024-11-09 20:08:48 +00:00
eldritch horrors 90536e27e1 libstore: simplify TransferSource::awaitData
don't pass in a lock that's only used inside this function.

Change-Id: I15c01e9cfe343cf13828ae3b3742c36ae697291f
2024-11-09 20:08:48 +00:00
eldritch horrors 5d02800e57 libstore: do not retry FileTransfer uploads
this only ever worked for empty uploads, and there it worked only by
complete accident: curl was asked to send more data than the wrapper
would provide, which curl would not like and report as an error. the
error would cause a retry with even less data to send, until finally
failing by running into the retry limit. let's just forbid all this.

Change-Id: I229a94b3b8b33e2c6cdb8ea19edd57cd6740e6c6
2024-11-09 20:08:48 +00:00
eldritch horrors 40be91afbf libstore: add filetranfer retry handling tests
we did not have any, despite retry handling being somewhat complex.

Change-Id: I5051a1c0a3861849ff67f512b33f6d3dda12cc95
2024-11-09 20:08:48 +00:00
eldritch horrors fb85228755 libstore: extract eager transfers into new method
we'll need this shared code for kjified transfers that don't use
sources. it's a while out, but we can clean this up now already.

Change-Id: Ife8c160e6ab379761362d6c54aba05093deee99e
2024-11-09 20:08:48 +00:00
eldritch horrors 12156d3beb libstore: fix download thread notifications
since 4ae6fb5a8f dropping a source of a
download might not properly cancel the associated curl transfer after
the transfer was paused. we have also not unpaused the transfer often
enough, only if the transfer buffer had been drained in its entirety.

Change-Id: Ic9298d9df71daa0f3d1c3fd718ed441edae9e863
2024-11-09 20:08:48 +00:00
alois31 c1746300a9
libexpr: generate builtins from data
Most builtins are now generated from data too, with two exceptions:
* Undocumented builtins, since supporting them would add complexity to the
  generator, the harms of the current implementation mostly don't apply, and
  the proper fix is to document them.
* `derivation` is somewhat magic (it is a function, but defined in the code as
  a constant), so the current treatment of having it separately documented is
  kept (for now, at least).
Since it is slightly easier to do and probably a good idea anyway, the builtin
function registrations generated this way are now processed directly in code
and don't go through global variables any more.
Unfortunately, a slight breaking change is introduced because the order of the
builtins' names in the symbol table changes. Hopefully, this will turn out to
not matter in practice.

Change-Id: I7b4379a93ae380b6524e41a916a21c5c6f70555e
2024-11-09 16:17:26 +01:00