lix-project/lix

Fork 56

Investigate http3 support #1033

New issue

Closed

opened 2025-11-14 04:09:37 +00:00 by mweinelt · 7 comments

mweinelt commented

2025-11-14 04:09:37 +00:00

Member

Since a few minutes cache.nixos.org supports http3, but lix does not.

Describe the solution you'd like

Investigate enabling http3 support in curl and see how well it works.

Describe alternatives you've considered

None, this is purely informational.

Additional context

It does not currently advertise support via Alt-Svc, which may hamper support.

Nothing we can do about that until we have widespread HTTPS resource record support.

## Is your feature request related to a problem? Please describe. Since a few minutes cache.nixos.org supports http3, but lix does not. ## Describe the solution you'd like Investigate enabling http3 support in curl and see how well it works. ## Describe alternatives you've considered None, this is purely informational. ## Additional context It does not currently advertise support via Alt-Svc, which may hamper support. Nothing we can do about that until we have widespread HTTPS resource record support.

alois31 commented

2025-11-15 12:11:09 +00:00

Member

If HTTP/3 is something we want to support, this can be split into two questions.

How to enable HTTP/3?

Curl has three relevant knobs here.

By setting CURLOPT_HTTP_VERSION to CURL_HTTP_VERSION_3, it will try to open QUIC and TLS connections simultaneously, and use HTTP/3 if it works (otherwise it will fall back to the TLS connection and use whichever of 1.2 and 2 comes out of ALPN).
The Alt-Svc mechanism is supported if setting CURLOPT_ALTSVC.
There is experimental support for HTTPS resource record using a compile-time option.

Given that HTTPS resource record is experimental in curl, and neither Alt-Svc nor HTTPS record can be observed on cache.nixos.org, the first option seems the best.

When to enable HTTP/3

The three basic options here are never, together with HTTP/2, or as a separate level of support. Given some information from curl it is plausible that "never" may be the best option (as I expect bulk transfer speed and CPU usage to be more relevant than handshake latency in the Lix use case), but without an experiment there is no way to really know. Separate level of support, but enabled by default early in the 2.95 cycle, looks most reasonable to me here. This only applies to builds against nixpkgs ≥25.11, on ≤25.05 I would leave it disabled due to quictls usage in the curlHTTP3 package.

If HTTP/3 is something we want to support, this can be split into two questions. # How to enable HTTP/3? Curl has three relevant knobs here. * By setting [CURLOPT_HTTP_VERSION](https://curl.se/libcurl/c/CURLOPT_HTTP_VERSION.html) to `CURL_HTTP_VERSION_3`, it will try to open QUIC and TLS connections simultaneously, and use HTTP/3 if it works (otherwise it will fall back to the TLS connection and use whichever of 1.2 and 2 comes out of ALPN). * The Alt-Svc mechanism is supported if setting [CURLOPT_ALTSVC](https://curl.se/libcurl/c/CURLOPT_ALTSVC.html). * There is [experimental](https://daniel.haxx.se/blog/2025/03/31/https-rr-in-curl/) support for HTTPS resource record using a compile-time option. Given that HTTPS resource record is experimental in curl, and neither Alt-Svc nor HTTPS record can be observed on cache.nixos.org, the first option seems the best. # When to enable HTTP/3 The three basic options here are never, together with HTTP/2, or as a separate level of support. Given some [information](https://daniel.haxx.se/blog/2024/06/10/http-3-in-curl-mid-2024/) from curl it is plausible that "never" may be the best option (as I expect bulk transfer speed and CPU usage to be more relevant than handshake latency in the Lix use case), but without an experiment there is no way to really know. Separate level of support, but enabled by default early in the 2.95 cycle, looks most reasonable to me here. This only applies to builds against nixpkgs ≥25.11, on ≤25.05 I would leave it disabled due to quictls usage in the `curlHTTP3` package.

pennae commented

2025-11-15 14:23:50 +00:00

Owner

considering how stable http 2 is in curl we should probably hold off on turning http 3 on for a very long time of interested parties testing it in their own time. we tried testing h3 with plain curl and it does not work for shit.

considering how stable http **2** is in curl we should probably hold off on turning http 3 on for a very long time of interested parties testing it in their own time. we tried testing h3 with *plain* curl and it does not work for shit.

alois31 commented

2025-11-16 14:47:23 +00:00

Member

Scenario	HTTP/1.1	HTTP/2	HTTP/3
Machine 1 (query)	5.090 s ± 0.070 s [User: 2.074 s, System: 0.912 s]	2.219 s ± 0.097 s [User: 2.304 s, System: 0.794 s]	2.081 s ± 0.068 s [User: 2.184 s, System: 0.783 s]
Machine 1 (query, shit internet)	10.636 s ± 0.916 s [User: 3.168 s, System: 0.837 s]	6.611 s ± 0.410 s [User: 3.703 s, System: 1.157 s]	5.141 s ± 0.300 s [User: 3.433 s, System: 1.169 s]
Machine 2 (query)	3.022 s ± 0.112 s [User: 3.224 s, System: 0.857 s]	2.367 s ± 0.092 s [User: 2.803 s, System: 0.644 s]	2.264 s ± 0.071 s [User: 2.763 s, System: 0.597 s]
Machine 2 (substitute)	60.722 s ± 0.865 s [User: 238.933 s, System: 37.095 s]	58.269 s ± 0.710 s [User: 240.887 s, System: 37.953 s]	56.669 s ± 1.028 s [User: 249.107 s, System: 31.472 s]

Not visible from the table is the fact that the HTTP/2 and HTTP/3 substitute tests had to be retried once each, as the first attempt failed with an error. Unfortunately hyperfine ate the error message, so that I can't say for sure where the fault lies.

I would summarize the results as follows:

HTTP/2 or HTTP/3 speeds up queries quite a bit, but the advantage mostly disappears when considering the entire substitution (pretty much as expected).
The CPU usage picture is more mixed, but counter to my expectation after reading the curl article HTTP/3 is at least not wildly worse.
HTTP/3 does not look unusably buggy, at least not a lot worse than HTTP/2.

However in total, also taking into account the HTTP/2 bugs encountered earlier (and I have no reason to assume HTTP/3 is going to be better on that front), the case for HTTP/2 or HTTP/3 in Lix does not look that great. Maybe we should actually disable HTTP/2 instead of further adding HTTP/3.

I did some measurements with a build of lightly patched Lix main against nixos-unstable (I do not expect the patches to be relevant here, except for the HTTP/3 enablement of course). Machine 1 is running at home, machine 2 in a data center. "Shit internet" has added 5% packet loss and 200 ms delay via tc. "Query" is `nix-build --dry-run` of my main system configuration with 20 runs, "substitute" is `nix-build` of `nixosTests.allDrivers.plasma6` from nixos-unstable with 10 runs. All tests were run on a fresh store and empty `XDG_CACHE_DIR` both on tmpfs, except that the drv files were copied in to run `nix-build` directly against them (to avoid including the evaluation in the measurements, which would only add noise). | Scenario | HTTP/1.1 | HTTP/2 | HTTP/3 | |-|-|-|-| | Machine 1 (query) | 5.090 s ± 0.070 s [User: 2.074 s, System: 0.912 s] | 2.219 s ± 0.097 s [User: 2.304 s, System: 0.794 s] | 2.081 s ± 0.068 s [User: 2.184 s, System: 0.783 s] | | Machine 1 (query, shit internet) | 10.636 s ± 0.916 s [User: 3.168 s, System: 0.837 s] | 6.611 s ± 0.410 s [User: 3.703 s, System: 1.157 s] | 5.141 s ± 0.300 s [User: 3.433 s, System: 1.169 s] | | Machine 2 (query) | 3.022 s ± 0.112 s [User: 3.224 s, System: 0.857 s] | 2.367 s ± 0.092 s [User: 2.803 s, System: 0.644 s] | 2.264 s ± 0.071 s [User: 2.763 s, System: 0.597 s] | | Machine 2 (substitute) | 60.722 s ± 0.865 s [User: 238.933 s, System: 37.095 s] | 58.269 s ± 0.710 s [User: 240.887 s, System: 37.953 s] | 56.669 s ± 1.028 s [User: 249.107 s, System: 31.472 s] | Not visible from the table is the fact that the HTTP/2 and HTTP/3 substitute tests had to be retried once each, as the first attempt failed with an error. Unfortunately hyperfine ate the error message, so that I can't say for sure where the fault lies. I would summarize the results as follows: * HTTP/2 or HTTP/3 speeds up queries quite a bit, but the advantage mostly disappears when considering the entire substitution (pretty much as expected). * The CPU usage picture is more mixed, but counter to my expectation after reading the curl article HTTP/3 is at least not wildly worse. * HTTP/3 does not look unusably buggy, at least not a lot worse than HTTP/2. However in total, also taking into account the HTTP/2 bugs encountered earlier (and I have no reason to assume HTTP/3 is going to be better on that front), the case for HTTP/2 or HTTP/3 in Lix does not look that great. Maybe we should actually disable HTTP/2 instead of further adding HTTP/3.

👀 1

raito commented

2025-11-16 14:58:51 +00:00

Owner

Did you test with https://github.com/NixOS/nixpkgs/pull/458142 applied or not? It is in master but no idea if it has reached one of the channel.

I did some benchmarks on throughput and H2 was able to saturate properly gigabit pipes (once the bug is fixed, of course.), I think it makes sense to keep H2 as long as we can have it reasonably work.

You tested substitution using a xz-enabled cache, what about a zstd-enabled cache? xz destroys performance unnecessarily IME.

Did you test with https://github.com/NixOS/nixpkgs/pull/458142 applied or not? It is in master but no idea if it has reached one of the channel. I did some benchmarks on throughput and H2 was able to saturate properly gigabit pipes (once the bug is fixed, of course.), I think it makes sense to keep H2 as long as we can have it reasonably work. You tested substitution using a xz-enabled cache, what about a zstd-enabled cache? xz destroys performance unnecessarily IME.

alois31 commented

2025-11-16 15:04:07 +00:00

Member

That is in nixos-unstable so it was applied (also, otherwise I would expect the HTTP/2 substitute test to fail a lot more often). I did not try with a zstd-enabled cache. In general, I would be surprised if HTTP/2 was actually meaningfully slower in any scenario, but also if it was meaningfully faster during bulk transfers (merely that it is too buggy). Do you have any data that suggests the contrary?