lix-project/lix

Fork 57

Various IO, networking or curl errors causing builds to fail #1026

New issue

Open

opened 2025-11-04 13:13:20 +00:00 by gepbird · 3 comments

gepbird commented

2025-11-04 13:13:20 +00:00

Describe the bug

Sometimes NixOS configurations, packages or just directly downloading a nix store path fails.
After restarting the build/download (sometimes even multiple times), the issue is gone.

Steps To Reproduce

This is not consistently reproducible, if you're lucky this could work:

Run the following command:

nix-store -r /nix/store/99112ckjfsyxqqxwkfsidfwklfg2kh75-n8n-1.117.3 \
  --option binary-caches 'https://cache.nixos.org/ https://nixpkgs-update-cache.nix-community.org/' \
  --option trusted-public-keys '
  nixpkgs-update-cache.nix-community.org-1:U8d6wiQecHUPJFSqHN9GSSmNkmdiFW7GW7WNAnHW0SM=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

See any of the following errors:

curl error code=23: Failed writing received data to disk/application
curl error code=56: Recv failure: Connection reset by peer
failed to read compressed data (truncated input)
failed due to signal 5 (Trace/breakpoint trap), previous message: malloc_consolidate(): unaligned fastbin chunk detected; error: reading a line: Input/output error
HTTP error 200 () (curl error code=55: OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0)
HTTP error 200 () (curl error code=56: OpenSSL SSL_read: OpenSSL/3.6.0: error:0A000126:SSL routines::unexpected eof while reading, errno 0)

Expected behavior

Don't fail, or retry it under the hood and display a warning.

`nix --version` output

nix (Lix, like Nix) 2.93.3
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/gep/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/gep/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/gep/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/gep/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/ialf5cfp9vj84jkm2irqw2gyipkm0xpp-lix-2.93.3/share

Additional context

This is more likely for bigger downloads and possibly with WiFi connections, but it's not uncommon with a wired connection either.
We experienced this on all our machines: 3 x86_64-linux and 2 aarch64-linux multiple.

Screenshots

## Describe the bug Sometimes NixOS configurations, packages or just directly downloading a nix store path fails. After restarting the build/download (sometimes even multiple times), the issue is gone. ## Steps To Reproduce This is not consistently reproducible, if you're lucky this could work: 1. Run the following command: ```bash nix-store -r /nix/store/99112ckjfsyxqqxwkfsidfwklfg2kh75-n8n-1.117.3 \ --option binary-caches 'https://cache.nixos.org/ https://nixpkgs-update-cache.nix-community.org/' \ --option trusted-public-keys ' nixpkgs-update-cache.nix-community.org-1:U8d6wiQecHUPJFSqHN9GSSmNkmdiFW7GW7WNAnHW0SM= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ' ``` 1. See any of the following errors: - > curl error code=23: Failed writing received data to disk/application - > curl error code=56: Recv failure: Connection reset by peer - > failed to read compressed data (truncated input) - > failed due to signal 5 (Trace/breakpoint trap), previous message: malloc_consolidate(): unaligned fastbin chunk detected; error: reading a line: Input/output error - > HTTP error 200 () (curl error code=55: OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0) - > HTTP error 200 () (curl error code=56: OpenSSL SSL_read: OpenSSL/3.6.0: error:0A000126:SSL routines::unexpected eof while reading, errno 0) ## Expected behavior Don't fail, or retry it under the hood and display a warning. ## `nix --version` output ```console nix (Lix, like Nix) 2.93.3 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux, x86_64-v4-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/gep/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/gep/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/gep/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/gep/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/ialf5cfp9vj84jkm2irqw2gyipkm0xpp-lix-2.93.3/share ``` ## Additional context This is more likely for bigger downloads and possibly with WiFi connections, but it's not uncommon with a wired connection either. We experienced this on all our machines: 3 x86_64-linux and 2 aarch64-linux multiple. ## Screenshots ![image](/attachments/214b38a0-bf78-48ed-a1d0-c5877574251f) ![image](/attachments/5eac3a99-9f1c-4cee-86c3-391d30845c22) ![image](/attachments/61fbc64c-a6bd-4e37-a93b-a7ed4b63e054) ![image](/attachments/3a9141aa-d1e9-4311-9526-37695f367159) ![image](/attachments/a554d3bd-39ef-4309-b849-334f9d68a4cc)

image.png

79 KiB

image.png

31 KiB

image.png

200 KiB

image.png

265 KiB

image.png

347 KiB

gepbird added the

bug

label

2025-11-04 13:13:20 +00:00

pennae commented

2025-11-04 13:21:17 +00:00

Owner

code 23 is a curl bug, truncated input is very likely to be that as well. #1009 (comment)

code 56 sounds most like a connection being torn down by the remote server for any reason than a lix bug.

code 55 is something we haven't seen ever before the curl update that caused #1009, which sounds very suspicious. we should keep an eye on that, but if anything it's another curl bug.

the signal 5 thing though, that is very interesting. can you reproduce this?

code 23 is a curl bug, truncated input is very likely to be that as well. https://git.lix.systems/lix-project/lix/issues/1009#issuecomment-15331 code 56 sounds most like a connection being torn down by the remote server for any reason than a lix bug. code 55 is something we haven't seen *ever* before the curl update that caused #1009, which sounds very suspicious. we should keep an eye on that, but if anything it's another curl bug. the signal 5 thing though, that is very interesting. can you reproduce this?

👍 1

gepbird commented

2025-11-04 14:30:58 +00:00

Author

Thanks for the quick response!

I'll post here if I encounter any of the errors that could be related to lix. Unfortunately for each case (except maybe 1, I'll confirm this later) rerunning the failing command will make it pass. However after they pass it's harder to reproduce since it's already downloaded to the nix store thus no new download will be started, is there a better way than hunting down the store paths and running nix-store --delete /nix/store/...?

Thanks for the quick response! I'll post here if I encounter any of the errors that could be related to lix. Unfortunately for each case (except maybe 1, I'll confirm this later) rerunning the failing command will make it pass. However after they pass it's harder to reproduce since it's already downloaded to the nix store thus no new download will be started, is there a better way than hunting down the store paths and running `nix-store --delete /nix/store/...`?

pennae commented

2025-11-04 15:12:07 +00:00

Owner

the download problems honestly aren't that interesting. if you do want to repro them we'd suggest building the derivation you're interested in into a fresh store (ie nix build --store $some_new_directory ...) since you can then discard that entire store in one piece without disturbing the rest of your system.

if you have the coredump or stack traces of the process that failed during build env setup that would also be very helpful.

the download problems honestly aren't that interesting. if you do want to repro them we'd suggest building the derivation you're interested in into a fresh store (ie `nix build --store $some_new_directory ...`) since you can then discard that entire store in one piece without disturbing the rest of your system. if you have the coredump or stack traces of the process that failed during build env setup that would also be very helpful.

❤️ 1