lix-project/lix

Fork 72

DNS resolution issues with curl 8.20 and systemd-resolved #1234

New issue

Closed

opened 2026-06-24 12:23:24 +00:00 by gepbird · 5 comments

gepbird commented

2026-06-24 12:23:24 +00:00

Describe the bug

Using Lix from nixpkgs after curl was upgraded from 8.19 to 8.20 while having some DNS settings with systemd-resolved with DNSOverTLS enabled, Lix usually hangs or times out during DNS resolution.

Steps To Reproduce

Make sure Lix is using curl 8.20 (default in nixpkgs-unstable)
Use this NixOS config:

{
  services.resolved = {
    enable = true;
    settings.Resolve = {
      DNSOverTLS = "true";
      DNS = "1.1.1.1";
    };
  };
}

Clear nix cache and query something: rm -rf ~/.cache/nix; nix flake show github:NixOS/nixpkgs -vvvvv
a) Observe the output for a minute and notice that Lix hasn't gotten past this:

fetching github input 'github:NixOS/nixpkgs'
did not find cache entry for '{"ingestionMethod":0,"name":"source","type":"file","url":"https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD"}'
downloading 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD'...
downloading 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD

b) Or observe the output and notice the ~5s timeout (which is a soft error, the operation will finish successfully just a little slower):

fetching github input 'github:NixOS/nixpkgs'
did not find cache entry for '{"ingestionMethod":0,"name":"source","type":"file","url":"https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD"}'
downloading 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD'...
starting download of https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD
curl: Could not find host api.github.com in the /etc/nix/netrc file; using defaults
curl: Resolving timed out after 5006 milliseconds
curl: closing connection #0
finished download of 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD'; curl status = 28, HTTP status = 0, body = 0 bytes
warning: error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD': Resolving timed out after 5006 milliseconds (curl error code=28); retrying in 532 ms (attempt 1/5)
<many lines omitted>

Expected behavior

It should work as before, just like when disabling systemd-resolved or using curl 8.19. The download progress should be counting up in the first second and there should be no errors.

`nix --version` output

nix (Lix, like Nix) 2.95.2
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/gep/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/gep/.local/share/flatpak/exports/etc/xdg/nix/nix.conf:/var/lib/flatpak/exports/etc/xdg/nix/nix.conf:/home/gep/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/gep/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/gep/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/h1vl2jz6v2yr2p0qd6kp2nbr0wibflfd-lix-2.95.2/share

Additional context

Curl 8.20 has this change that sounds related:

now uses a thread pool and queue for resolving

I can reproduce the issues when Lix is using curl 8.20.0, 8.21.0-rc3, but not with 8.19.0. I can reproduce this with Lix 2.94 and 2.95, but not with upstream Nix, nor nslookup or using the curl CLI, maybe I haven't tried hard enough with the latter.

I've tried to keep this issue narrow, but with this curl and DNS setup I've also had other problems:

soft warning time out for the first time for nix flake update --commit-lock-file (but works after the 2/5 retry) with a cold cache:

warning: error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/nixpkgs-unstable': Resolving timed out after 5005 milliseconds (curl error code=28); retrying in 451 ms (attempt 1/5)
- update-yes-dns-pass
a hard error after GitHub returning HTTP 304 (with 0 bytes in the body) with a warm ~/.nix/cache:

error: JSON processing error: [json.exception.parse_error.101] parse error at line 1, column 1: attempting to parse an empty input; check that your input string or stream contains the expected JSON
- update-yes-dns-fail
hard error including "changed final destionation":

warning: error: unable to download 'https://github.com/ymstnt/nur-packages/archive/80211247799da4562d034a76dd3bf3323fef9269.zip': Resolving timed out after 5004 milliseconds (curl error code=28); retrying in 571 ms (attempt 1/5)
warning: uri https://github.com/ymstnt/nur-packages/archive/80211247799da4562d034a76dd3bf3323fef9269.zip changed final destination from https://github.com/ymstnt/nur-packages/archive/80211247799da4562d034a76dd3bf3323fef9269.zip to https://codeload.github.com/ymstnt/nur-packages/zip/80211247799da4562d034a76dd3bf3323fef9269 during transfer
hang while querying missing paths

querying info about missing paths
querying activate on https://cache.nixos.org

Here's a good log without DNS and using the default 8.20.0 curl: update-no-dns. I made some modifications to the logs for easier diffing between the logs (changing the exact >5000ms timing, changing the randomly generated paths and commits hashes, changing remaining rate limits...). I also used a different DNS resolver in the logs, not 1.1.1.1.

## Describe the bug Using Lix from nixpkgs after curl was upgraded from 8.19 to 8.20 while having some DNS settings with systemd-resolved with DNSOverTLS enabled, Lix usually hangs or times out during DNS resolution. ## Steps To Reproduce 1. Make sure Lix is using curl 8.20 (default in nixpkgs-unstable) 1. Use this NixOS config: ```nix { services.resolved = { enable = true; settings.Resolve = { DNSOverTLS = "true"; DNS = "1.1.1.1"; }; }; } ``` 2. Clear nix cache and query something: `rm -rf ~/.cache/nix; nix flake show github:NixOS/nixpkgs -vvvvv` 3. a) Observe the output for a minute and notice that Lix hasn't gotten past this: ```console fetching github input 'github:NixOS/nixpkgs' did not find cache entry for '{"ingestionMethod":0,"name":"source","type":"file","url":"https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD"}' downloading 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD'... downloading 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD ``` 3. b) Or observe the output and notice the ~5s timeout (which is a soft error, the operation will finish successfully just a little slower): ```console fetching github input 'github:NixOS/nixpkgs' did not find cache entry for '{"ingestionMethod":0,"name":"source","type":"file","url":"https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD"}' downloading 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD'... starting download of https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD curl: Could not find host api.github.com in the /etc/nix/netrc file; using defaults curl: Resolving timed out after 5006 milliseconds curl: closing connection #0 finished download of 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD'; curl status = 28, HTTP status = 0, body = 0 bytes warning: error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/HEAD': Resolving timed out after 5006 milliseconds (curl error code=28); retrying in 532 ms (attempt 1/5) <many lines omitted> ``` ## Expected behavior It should work as before, just like when disabling systemd-resolved or using curl 8.19. The download progress should be counting up in the first second and there should be no errors. ## `nix --version` output ```console nix (Lix, like Nix) 2.95.2 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/gep/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/gep/.local/share/flatpak/exports/etc/xdg/nix/nix.conf:/var/lib/flatpak/exports/etc/xdg/nix/nix.conf:/home/gep/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/gep/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/gep/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/h1vl2jz6v2yr2p0qd6kp2nbr0wibflfd-lix-2.95.2/share ``` ## Additional context [Curl 8.20](https://daniel.haxx.se/blog/2026/04/29/curl-8-20-0/) has this change that sounds related: > now uses a thread pool and queue for resolving I can reproduce the issues when Lix is using curl 8.20.0, 8.21.0-rc3, but not with 8.19.0. I can reproduce this with Lix 2.94 and 2.95, but not with upstream Nix, nor nslookup or using the curl CLI, maybe I haven't tried hard enough with the latter. I've tried to keep this issue narrow, but with this curl and DNS setup I've also had other problems: - soft warning time out for the first time for `nix flake update --commit-lock-file` (but works after the 2/5 retry) with a cold cache: > warning: error: unable to download 'https://api.github.com/repos/NixOS/nixpkgs/commits/nixpkgs-unstable': Resolving timed out after 5005 milliseconds (curl error code=28); retrying in 451 ms (attempt 1/5) - [update-yes-dns-pass](/attachments/02287376-eb21-411a-8c1e-973cbcb475d3) - a hard error after GitHub returning HTTP 304 (with 0 bytes in the body) with a warm ~/.nix/cache: > error: JSON processing error: [json.exception.parse_error.101] parse error at line 1, column 1: attempting to parse an empty input; check that your input string or stream contains the expected JSON - [update-yes-dns-fail](/attachments/17ca030f-0135-4672-8c1c-258668d8b9e0) - hard error including "changed final destionation": > warning: error: unable to download 'https://github.com/ymstnt/nur-packages/archive/80211247799da4562d034a76dd3bf3323fef9269.zip': Resolving timed out after 5004 milliseconds (curl error code=28); retrying in 571 ms (attempt 1/5) > warning: uri https://github.com/ymstnt/nur-packages/archive/80211247799da4562d034a76dd3bf3323fef9269.zip changed final destination from https://github.com/ymstnt/nur-packages/archive/80211247799da4562d034a76dd3bf3323fef9269.zip to https://codeload.github.com/ymstnt/nur-packages/zip/80211247799da4562d034a76dd3bf3323fef9269 during transfer - hang while querying missing paths > querying info about missing paths > querying activate on https://cache.nixos.org Here's a good log without DNS and using the default 8.20.0 curl: [update-no-dns](/attachments/37add42c-5744-4786-a142-cd9fc3df7798). I made some modifications to the logs for easier diffing between the logs (changing the exact >5000ms timing, changing the randomly generated paths and commits hashes, changing remaining rate limits...). I also used a different DNS resolver in the logs, not 1.1.1.1.

update-yes-dns-fail.log

9.8 KiB

update-yes-dns-pass.log

24 KiB

update-no-dns.log

24 KiB

gepbird added the

bug

label

2026-06-24 12:23:24 +00:00

qyriad added the

labels

2026-06-24 15:07:52 +00:00

qyriad was assigned by raito

2026-06-24 15:32:05 +00:00

qyriad commented

2026-06-24 15:42:18 +00:00

Member

Thank you for the detailed logs!

We have a workaround in main (cl/5705, 071b810f95), and Nixpkgs also has a workaround (https://github.com/NixOS/nixpkgs/pull/534757), but it has not hit master or unstable yet.

Short-term, you can use HEAD Lix, or revert to older Lix until Nixpkgs' PR hits. Long-term, we are working on tracking down this bug to hopefully report it clearly upstream, and hopefully reduce our dependence on Curl in general

Thank you for the detailed logs! We have a workaround in main (cl/5705, 071b810f952aad61b3ffc85379877b6ea8ff6b88), and Nixpkgs also has a workaround (https://github.com/NixOS/nixpkgs/pull/534757), but it [has not hit master or unstable yet](https://nixpk.gs/pr-tracker.html?pr=534757). Short-term, you can use HEAD Lix, or revert to older Lix until Nixpkgs' PR hits. Long-term, we are working on tracking down this bug to hopefully report it clearly upstream, and hopefully reduce our dependence on Curl in general

👍 1

qyriad pinned this

2026-06-25 09:36:17 +00:00

gepbird commented

2026-06-28 16:54:07 +00:00

Author

Removing the fix-wakeup-consumption.patch didn't help in this case, I'm still getting "Resolving timed out after 5005 milliseconds".

I'll stick using curl 8.19 for Lix, but I'm happy to test any other solutions/workarounds :)

Removing the `fix-wakeup-consumption.patch` didn't help in this case, I'm still getting "Resolving timed out after 5005 milliseconds". I'll stick using curl 8.19 for Lix, but I'm happy to test any other solutions/workarounds :)

gepbird commented

2026-06-29 12:13:22 +00:00

Author

Another potentially related soft error I just got:

error: unable to download 'https://nix-cache.tchfoo.com/nar/1vs4z18nfdad5gnzd0s681gl3jzll670mz8vvyj8qpklq6ggprx2.nar': HTTP error 200 () (curl error code=23: Failed writing received data to disk/application)

This Lix client isn't using systemd-resolved or any custom DNS settings and It's trying to download a NAR from my ncps cache server (that uses systemd-resolved). Could be an ncps issue as I had quite a few with them too.

Another potentially related soft error I just got: ```console error: unable to download 'https://nix-cache.tchfoo.com/nar/1vs4z18nfdad5gnzd0s681gl3jzll670mz8vvyj8qpklq6ggprx2.nar': HTTP error 200 () (curl error code=23: Failed writing received data to disk/application) ``` This Lix client isn't using systemd-resolved or any custom DNS settings and It's trying to download a NAR from my ncps cache server (that uses systemd-resolved). Could be an ncps issue as I had quite a few with them too.

pennae commented

2026-06-29 12:36:20 +00:00

Owner

that's probably the same as #1222, which (as far as we can tell so far) is a different issue. curl is having some problems :(

that's probably the same as https://git.lix.systems/lix-project/lix/issues/1222, which (as far as we can tell so far) is a *different* issue. curl is having some *problems* :(

👍 1

lix-bot commented

2026-07-07 19:58:04 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/5842 ("libstore: Fix libcurl thread wakeup with curl >= 8.21")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/5842](https://gerrit.lix.systems/c/lix/+/5842) ("libstore: Fix libcurl thread wakeup with curl >= 8.21")