modernize curl usage #246

New issue

Open

opened 2024-04-22 03:21:06 +00:00 by pennae · 3 comments

pennae commented

2024-04-22 03:21:06 +00:00

Owner

there's a bunch of ancient curl stuff that needs changed:

checks for features that were added over a decade ago (https://gerrit.lix.systems/c/lix/+/853)
usage of pipes and curl_multi_wait instead of curl_multi_poll (https://gerrit.lix.systems/c/lix/+/854)
questionable header parsing #245 (https://gerrit.lix.systems/c/lix/+/856)
hol blocking when a consumer of downloaded data is slow
the whole checkInterrupted mess
content decompression inline into curl wrapper
highly questionable/absent raii
s3 support is hacked into curl for some reason?
~~maybe drop the curl_multi pretense and do curl_share in threads?~~ (nope, won't share connections between threads)

there's a bunch of ancient curl stuff that needs changed: 1. [x] checks for features that were added over a decade ago (https://gerrit.lix.systems/c/lix/+/853) 1. [x] usage of pipes and `curl_multi_wait` instead of `curl_multi_poll` (https://gerrit.lix.systems/c/lix/+/854) 1. [x] questionable header parsing #245 (https://gerrit.lix.systems/c/lix/+/856) 1. [ ] hol blocking when a consumer of downloaded data is slow 1. [ ] the whole checkInterrupted mess 1. [ ] content decompression inline into curl wrapper 1. [ ] highly questionable/absent raii 1. [ ] s3 support is hacked into curl for some reason? 1. [x] ~~maybe drop the curl_multi pretense and do curl_share in threads?~~ (nope, won't share connections between threads)

jade added the

stability

devx

E/help wanted

labels

2024-04-22 03:39:18 +00:00

puck commented

2024-04-24 23:53:49 +00:00

Owner

We probably want to replace any of the existing S3 support with CURLOPT_AWS_SIGV4 - sending a PUT to an S3 endpoint with the encryption flags set is (iirc?) exactly what Nix does already.

Curl 7.75.0 has been released since February of 2021, which should be more than modern enough.

(It'd be nice if the HTTP binary cache stuff would also let us set authorization headers more arbitrarily; for e.g. Basic auth support.)

We probably want to replace any of the existing S3 support with [CURLOPT_AWS_SIGV4](https://curl.se/libcurl/c/CURLOPT_AWS_SIGV4.html) - sending a `PUT` to an S3 endpoint with the encryption flags set is (iirc?) exactly what Nix does already. Curl 7.75.0 has been released since February of 2021, which should be more than modern enough. (It'd be nice if the HTTP binary cache stuff would also let us set authorization headers more arbitrarily; for e.g. Basic auth support.)

jade referenced this issue

2024-05-06 05:09:39 +00:00

kill aws-sdk-cpp with fire #272

delroth referenced this issue

2024-05-20 11:05:15 +00:00

kill aws-sdk-cpp with fire #272

jade added the

Area/fetching

label

2024-10-20 01:04:57 +00:00

dwt commented

2025-03-14 09:49:08 +00:00

Would it be in scope for this issue to also get progress reports in the CLI for large downloads, or should I open a different issue? It may be more UI related.

Right now it looks like stuff just hangs in curl.

Would it be in scope for this issue to also get progress reports in the CLI for large downloads, or should I open a different issue? It may be more UI related. Right now it looks like stuff just hangs in curl.

pennae commented

2025-03-14 09:55:26 +00:00

Author

Owner

please open a new issue; iwrc we have all the infra needed to give progress reports fort individual downloads but just hide all but the global progress meter for some reason

👍 1