modernize curl usage #246

New issue

Open

opened 2024-04-22 03:21:06 +00:00 by pennae · 1 comment

pennae commented

2024-04-22 03:21:06 +00:00

Owner

there's a bunch of ancient curl stuff that needs changed:

checks for features that were added over a decade ago (https://gerrit.lix.systems/c/lix/+/853)
usage of pipes and curl_multi_wait instead of curl_multi_poll (https://gerrit.lix.systems/c/lix/+/854)
questionable header parsing #245 (https://gerrit.lix.systems/c/lix/+/856)
hol blocking when a consumer of downloaded data is slow
the whole checkInterrupted mess
content decompression inline into curl wrapper
highly questionable/absent raii
s3 support is hacked into curl for some reason?
~~maybe drop the curl_multi pretense and do curl_share in threads?~~ (nope, won't share connections between threads)

there's a bunch of ancient curl stuff that needs changed: 1. [x] checks for features that were added over a decade ago (https://gerrit.lix.systems/c/lix/+/853) 1. [x] usage of pipes and `curl_multi_wait` instead of `curl_multi_poll` (https://gerrit.lix.systems/c/lix/+/854) 1. [x] questionable header parsing #245 (https://gerrit.lix.systems/c/lix/+/856) 1. [ ] hol blocking when a consumer of downloaded data is slow 1. [ ] the whole checkInterrupted mess 1. [ ] content decompression inline into curl wrapper 1. [ ] highly questionable/absent raii 1. [ ] s3 support is hacked into curl for some reason? 1. [x] ~~maybe drop the curl_multi pretense and do curl_share in threads?~~ (nope, won't share connections between threads)

jade added the

stability

devx

E/help wanted

labels 2024-04-22 03:39:34 +00:00

puck commented

2024-04-24 23:53:49 +00:00

Owner

We probably want to replace any of the existing S3 support with CURLOPT_AWS_SIGV4 - sending a PUT to an S3 endpoint with the encryption flags set is (iirc?) exactly what Nix does already.

Curl 7.75.0 has been released since February of 2021, which should be more than modern enough.

(It'd be nice if the HTTP binary cache stuff would also let us set authorization headers more arbitrarily; for e.g. Basic auth support.)

We probably want to replace any of the existing S3 support with [CURLOPT_AWS_SIGV4](https://curl.se/libcurl/c/CURLOPT_AWS_SIGV4.html) - sending a `PUT` to an S3 endpoint with the encryption flags set is (iirc?) exactly what Nix does already. Curl 7.75.0 has been released since February of 2021, which should be more than modern enough. (It'd be nice if the HTTP binary cache stuff would also let us set authorization headers more arbitrarily; for e.g. Basic auth support.)

jade referenced this issue

2024-05-06 05:09:39 +00:00

kill aws-sdk-cpp with fire #272