lix-project/lix

Fork 55

Decompression of LZMA streams may fail if the input stream is too slow #1009

New issue

Open

opened 2025-10-11 14:25:49 +00:00 by raito · 6 comments

raito commented

2025-10-11 14:25:49 +00:00

Owner

If the input LZMA stream is split / arriving too slowly, the libarchive processing machinery might receive from the LZMA code an LZMA_BUF_ERROR:

LZMA_BUF_ERROR 	

No progress is possible.

This error code is returned when the coder cannot consume any new input and produce any new output. The most common reason for this error is that the input stream being decoded is truncated or corrupt.

This error is not fatal. Coding can be continued normally by providing more input and/or more output space, if possible.

libarchive code does not recover from this error and seems to be considering it fatal, bubbling into Lix as "failed to read compression data: No progress can be made".

Solutions includes:

create a test case to reproduce this out of "sheer luck" (which seems to be related
patching libarchive
using Rust-ified archived libraries as long as the integration work is not too expensive

(Credits where it's due: all the investigative work has been done by @pennae)

If the input LZMA stream is split / arriving too slowly, the libarchive processing machinery might receive from the LZMA code an `LZMA_BUF_ERROR`: ``` LZMA_BUF_ERROR No progress is possible. This error code is returned when the coder cannot consume any new input and produce any new output. The most common reason for this error is that the input stream being decoded is truncated or corrupt. This error is not fatal. Coding can be continued normally by providing more input and/or more output space, if possible. ``` [libarchive code does not recover from this error and seems to be considering it fatal](https://github.com/libarchive/libarchive/blob/master/libarchive/archive_read_support_filter_xz.c#L704), bubbling into Lix as "failed to read compression data: No progress can be made". Solutions includes: - create a test case to reproduce this out of "sheer luck" (which seems to be related - patching libarchive - using Rust-ified archived libraries as long as the integration work is not too expensive (Credits where it's due: all the investigative work has been done by @pennae)

raito added the

labels

2025-10-11 14:26:16 +00:00

raito added the

release-blocker

label

2025-10-13 10:35:46 +00:00

raito commented

2025-10-13 12:12:58 +00:00

Author

Owner

Work done by @yu-re-ka seems to indicate this is caused by a bump in our dependencies:

commit 15ed8f7638116135ac9d3bd4353c482e7c539e0f
Merge: ee5dae584d75 17eefed8a027
Author: Vladimír Čunát <v@cunat.cz>
Date:   Sun Oct 5 13:28:34 2025 +0000

    staging-next 2025-09-21 (#444862)

Work done by @yu-re-ka seems to indicate this is caused by a bump in our dependencies: ``` commit 15ed8f7638116135ac9d3bd4353c482e7c539e0f Merge: ee5dae584d75 17eefed8a027 Author: Vladimír Čunát <v@cunat.cz> Date: Sun Oct 5 13:28:34 2025 +0000 staging-next 2025-09-21 (#444862) ```

raito commented

2025-10-14 20:09:35 +00:00

Author

Owner

This was confirmed to be the curl bump from 8.14.1 to 8.16.0.

helle self-assigned this

2025-10-21 19:21:44 +00:00

helle commented

2025-10-21 19:34:30 +00:00

Member

dealing with the test case for this currently

will release it when this exists as we may not know how to actually fix it (though we have some ideas)

dealing with the test case for this currently will release it when this exists as we may not know how to actually fix it (though we have some ideas)

raito pinned this

2025-10-22 21:34:03 +00:00

raito commented

2025-10-28 14:37:13 +00:00

Author

Owner

More messages as per @pennae demand:

[ERROR]   stderr) copying path '/nix/store/0ygil6x4hrqdxxh70fcs6kw9gkjq5ns4-python3.13-pyparsing-2.4.7' from 'https://cache.nixos.org'...
[ERROR]   stderr) copying path '/nix/store/hxsgwpkq43mz1850j5gjmlc2msajxq7q-perl5.40.0-IPC-Run-20231003.0' from 'https://cache.nixos.org'...
[ERROR]   stderr) building '/nix/store/7204dp1bag17zsqvvw6ir6qf7cna4j9r-rspamd-locals-milter_headers.conf.drv'...
[ERROR]   stderr) building '/nix/store/w4092xxikhbhbdkyaxysdxkfh1sggc75-acme-postrun.drv'...
[ERROR]   stderr) building '/nix/store/g629a8cvxdr7wmjb7id0fc148jn8pz80-nixos-tmpfiles.d.drv'...
[ERROR]   stderr) building '/nix/store/svrvm09fskw4d4xbkms61mi3ji9q75h8-rspamd-exporter-config.yml.drv'...
[ERROR]   stderr) copying path '/nix/store/780yhckz1mqa5d0jav6y7zg3fpsrcmlv-tor-client-auth-gen' from 'https://cache.nixos.org'...
[ERROR]   stderr) building '/nix/store/60ysidngn3jf7wczn3fvh59sqm665pn2-unit-wireguard-wgv6delivery-key.service.drv'...
[ERROR]   stderr) error: failed to read compressed data (Lzma library error:  No progress is possible)
[ERROR]   stderr) error: some substitutes for the outputs of derivation '/nix/store/9hhaykm6r1gwf0jflfr9ndh3ri3696bd-linux-6.12.55.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
[ERROR]   stderr) error: 1 dependencies of derivation '/nix/store/df2dcxdwwcdqyl4b9f40ifz2s1zc814g-boot.json.drv' failed to build
[ERROR]   stderr) error: 1 dependencies of derivation '/nix/store/wqf40ya88lbx2p28r0619qhwx4kngy9v-linux-6.12.55-modules.drv' failed to build

More messages as per @pennae demand: ``` [ERROR] stderr) copying path '/nix/store/0ygil6x4hrqdxxh70fcs6kw9gkjq5ns4-python3.13-pyparsing-2.4.7' from 'https://cache.nixos.org'... [ERROR] stderr) copying path '/nix/store/hxsgwpkq43mz1850j5gjmlc2msajxq7q-perl5.40.0-IPC-Run-20231003.0' from 'https://cache.nixos.org'... [ERROR] stderr) building '/nix/store/7204dp1bag17zsqvvw6ir6qf7cna4j9r-rspamd-locals-milter_headers.conf.drv'... [ERROR] stderr) building '/nix/store/w4092xxikhbhbdkyaxysdxkfh1sggc75-acme-postrun.drv'... [ERROR] stderr) building '/nix/store/g629a8cvxdr7wmjb7id0fc148jn8pz80-nixos-tmpfiles.d.drv'... [ERROR] stderr) building '/nix/store/svrvm09fskw4d4xbkms61mi3ji9q75h8-rspamd-exporter-config.yml.drv'... [ERROR] stderr) copying path '/nix/store/780yhckz1mqa5d0jav6y7zg3fpsrcmlv-tor-client-auth-gen' from 'https://cache.nixos.org'... [ERROR] stderr) building '/nix/store/60ysidngn3jf7wczn3fvh59sqm665pn2-unit-wireguard-wgv6delivery-key.service.drv'... [ERROR] stderr) error: failed to read compressed data (Lzma library error: No progress is possible) [ERROR] stderr) error: some substitutes for the outputs of derivation '/nix/store/9hhaykm6r1gwf0jflfr9ndh3ri3696bd-linux-6.12.55.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source [ERROR] stderr) error: 1 dependencies of derivation '/nix/store/df2dcxdwwcdqyl4b9f40ifz2s1zc814g-boot.json.drv' failed to build [ERROR] stderr) error: 1 dependencies of derivation '/nix/store/wqf40ya88lbx2p28r0619qhwx4kngy9v-linux-6.12.55-modules.drv' failed to build ```

raito commented

2025-10-30 00:28:43 +00:00

Author

Owner

Discovered:

oct. 29 15:07:55 Thors nix-daemon[1538273]: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gcc-14.3.0/include/c++/14.3.0/bits/regex_executor.tcc:538: void std::__detail::_Executor<__gnu_cxx::__normal_iterator<const >
oct. 29 15:07:55 Thors systemd-coredump[1538407]: [🡕] Process 1538273 (nix) of user 0 dumped core.
                                                  
                                                  Module /nix/store/cqmswwgzcjyv0f8vm6700z6hr264dl2l-lix-2.94.020251022054146-raito-edition/bin/nix without build-id.
                                                  Module libkeyutils.so.1 without build-id.
                                                  Module libkrb5support.so.0 without build-id.
                                                  Module libcom_err.so.3 without build-id.
                                                  Module libk5crypto.so.3 without build-id.
                                                  Module libkrb5.so.3 without build-id.
                                                  Module libunistring.so.5 without build-id.
                                                  Module libattr.so.1 without build-id.
                                                  Module libncursesw.so.6 without build-id.
                                                  Module libgssapi_krb5.so.2 without build-id.
                                                  Module libpsl.so.5 without build-id.
                                                  Module libssh2.so.1 without build-id.
                                                  Module libidn2.so.0 without build-id.
                                                  Module libnghttp2.so.14 without build-id.
                                                  Module libngtcp2.so.16 without build-id.
                                                  Module libngtcp2_crypto_ossl.so.0 without build-id.
                                                  Module libnghttp3.so.9 without build-id.
                                                  Module libaws-c-common.so.1 without build-id.
                                                  Module libaws-c-sdkutils.so.1.0.0 without build-id.
                                                  Module libaws-c-compression.so.1.0.0 without build-id.
                                                  Module libaws-checksums.so.1.0.0 without build-id.
                                                  Module libaws-c-cal.so.1.0.0 without build-id.
                                                  Module libs2n.so.1 without build-id.
                                                  Module libaws-c-io.so.1.0.0 without build-id.
                                                  Module libaws-c-http.so.1.0.0 without build-id.
                                                  Module libaws-c-auth.so.1.0.0 without build-id.
                                                  Module libaws-c-s3.so.0unstable without build-id.
                                                  Module libaws-c-event-stream.so.1.0.0 without build-id.
                                                  Module libaws-c-mqtt.so.1.0.0 without build-id.
                                                  Module libaws-crt-cpp.so without build-id.
                                                  Module libxml2.so.16 without build-id.
                                                  Module liblzma.so.5 without build-id.
                                                  Module libz.so.1 without build-id.
                                                  Module libbz2.so.1 without build-id.
                                                  Module libzstd.so.1 without build-id.
                                                  Module libacl.so.1 without build-id.
                                                  Module libbrotlicommon.so.1 without build-id.
                                                  Module liblowdown.so.2 without build-id.
                                                  Module libeditline.so.1 without build-id.
                                                  Module libseccomp.so.2 without build-id.
                                                  Module libaws-cpp-sdk-transfer.so without build-id.
                                                  Module libaws-cpp-sdk-core.so without build-id.
                                                  Module libaws-cpp-sdk-s3.so without build-id.
                                                  Module libarchive.so.13 without build-id.
                                                  Module libcpuid.so.17 without build-id.
                                                  Module libbrotlienc.so.1 without build-id.
                                                  Module libbrotlidec.so.1 without build-id.
                                                  Module libgcc_s.so.1 without build-id.
                                                  Module libstdc++.so.6 without build-id.
                                                  Module liblixcmd.so without build-id.
                                                  Module liblixmain.so without build-id.
                                                  Module liblixexpr.so without build-id.
                                                  Module liblixfetchers.so without build-id.
                                                  Module liblixstore.so without build-id.
                                                  Module liblixutil.so without build-id.
                                                  Stack trace of thread 1538403:
                                                  #0  0x00007f0319e9caac __pthread_kill_implementation (libc.so.6 + 0x9caac)
                                                  #1  0x00007f0319e4190e raise (libc.so.6 + 0x4190e)
                                                  #2  0x00007f0319e28942 abort (libc.so.6 + 0x28942)
                                                  #3  0x00007f031a2e115e _ZSt21__glibcxx_assert_failPKciS0_S0_ (libstdc++.so.6 + 0xe115e)
                                                  #4  0x000055ffdaad6692 _ZNSt8__detail9_ExecutorIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS5_9sub_matchISB_EEENS>
                                                  #5  0x000055ffdaad40ca _ZNSt8__detail17__regex_algo_implIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS5_9sub_match>
                                                  #6  0x00007f031b67c7ee _ZN3nix16curlFileTransfer12TransferItem14headerCallbackEPvmm (liblixstore.so + 0x47c7ee)
                                                  #7  0x00007f0319d4449e cw_out_cb_write (libcurl.so.4 + 0x3649e)
                                                  #8  0x00007f0319d447a3 cw_out_ptr_flush (libcurl.so.4 + 0x367a3)
                                                  #9  0x00007f0319d44dd5 cw_out_do_write (libcurl.so.4 + 0x36dd5)
                                                  #10 0x00007f0319d45436 cw_pause_write (libcurl.so.4 + 0x37436)
                                                  #11 0x00007f0319d8db6c cw_download_write (libcurl.so.4 + 0x7fb6c)
                                                  #12 0x00007f0319d8f05b Curl_client_write (libcurl.so.4 + 0x8105b)
                                                  #13 0x00007f0319d600a6 http_rw_hd (libcurl.so.4 + 0x520a6)
                                                  #14 0x00007f0319d61ee7 Curl_http_write_resp_hd (libcurl.so.4 + 0x53ee7)
                                                  #15 0x00007f0319d65ab2 h2_xfer_write_resp_hd.part.0 (libcurl.so.4 + 0x57ab2)
                                                  #16 0x00007f0319d68f56 on_header (libcurl.so.4 + 0x5af56)
                                                  #17 0x00007f03188f7a22 nghttp2_session_mem_recv2 (libnghttp2.so.14 + 0x12a22)
                                                  #18 0x00007f0319d63a57 h2_process_pending_input (libcurl.so.4 + 0x55a57)
                                                  #19 0x00007f0319d66b41 h2_progress_ingress (libcurl.so.4 + 0x58b41)
                                                  #20 0x00007f0319d66f02 cf_h2_recv (libcurl.so.4 + 0x58f02)
                                                  #21 0x00007f0319da1fa4 Curl_sendrecv (libcurl.so.4 + 0x93fa4)
                                                  #22 0x00007f0319d81ac4 multi_runsingle (libcurl.so.4 + 0x73ac4)
                                                  #23 0x00007f0319d82c3c curl_multi_perform (libcurl.so.4 + 0x74c3c)
                                                  #24 0x00007f031b660f48 _ZN3nix16curlFileTransfer16workerThreadMainEv (liblixstore.so + 0x460f48)
                                                  #25 0x00007f031b660889 _ZN3nix16curlFileTransfer17workerThreadEntryEv (liblixstore.so + 0x460889)
                                                  #26 0x00007f031a2ed064 execute_native_thread_routine (libstdc++.so.6 + 0xed064)
                                                  #27 0x00007f0319e9a97a start_thread (libc.so.6 + 0x9a97a)
                                                  #28 0x00007f0319f22d2c __clone3 (libc.so.6 + 0x122d2c)
                                                  ```

in my daemon traces

Discovered: ``` oct. 29 15:07:55 Thors nix-daemon[1538273]: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gcc-14.3.0/include/c++/14.3.0/bits/regex_executor.tcc:538: void std::__detail::_Executor<__gnu_cxx::__normal_iterator<const > oct. 29 15:07:55 Thors systemd-coredump[1538407]: [🡕] Process 1538273 (nix) of user 0 dumped core. Module /nix/store/cqmswwgzcjyv0f8vm6700z6hr264dl2l-lix-2.94.020251022054146-raito-edition/bin/nix without build-id. Module libkeyutils.so.1 without build-id. Module libkrb5support.so.0 without build-id. Module libcom_err.so.3 without build-id. Module libk5crypto.so.3 without build-id. Module libkrb5.so.3 without build-id. Module libunistring.so.5 without build-id. Module libattr.so.1 without build-id. Module libncursesw.so.6 without build-id. Module libgssapi_krb5.so.2 without build-id. Module libpsl.so.5 without build-id. Module libssh2.so.1 without build-id. Module libidn2.so.0 without build-id. Module libnghttp2.so.14 without build-id. Module libngtcp2.so.16 without build-id. Module libngtcp2_crypto_ossl.so.0 without build-id. Module libnghttp3.so.9 without build-id. Module libaws-c-common.so.1 without build-id. Module libaws-c-sdkutils.so.1.0.0 without build-id. Module libaws-c-compression.so.1.0.0 without build-id. Module libaws-checksums.so.1.0.0 without build-id. Module libaws-c-cal.so.1.0.0 without build-id. Module libs2n.so.1 without build-id. Module libaws-c-io.so.1.0.0 without build-id. Module libaws-c-http.so.1.0.0 without build-id. Module libaws-c-auth.so.1.0.0 without build-id. Module libaws-c-s3.so.0unstable without build-id. Module libaws-c-event-stream.so.1.0.0 without build-id. Module libaws-c-mqtt.so.1.0.0 without build-id. Module libaws-crt-cpp.so without build-id. Module libxml2.so.16 without build-id. Module liblzma.so.5 without build-id. Module libz.so.1 without build-id. Module libbz2.so.1 without build-id. Module libzstd.so.1 without build-id. Module libacl.so.1 without build-id. Module libbrotlicommon.so.1 without build-id. Module liblowdown.so.2 without build-id. Module libeditline.so.1 without build-id. Module libseccomp.so.2 without build-id. Module libaws-cpp-sdk-transfer.so without build-id. Module libaws-cpp-sdk-core.so without build-id. Module libaws-cpp-sdk-s3.so without build-id. Module libarchive.so.13 without build-id. Module libcpuid.so.17 without build-id. Module libbrotlienc.so.1 without build-id. Module libbrotlidec.so.1 without build-id. Module libgcc_s.so.1 without build-id. Module libstdc++.so.6 without build-id. Module liblixcmd.so without build-id. Module liblixmain.so without build-id. Module liblixexpr.so without build-id. Module liblixfetchers.so without build-id. Module liblixstore.so without build-id. Module liblixutil.so without build-id. Stack trace of thread 1538403: #0 0x00007f0319e9caac __pthread_kill_implementation (libc.so.6 + 0x9caac) #1 0x00007f0319e4190e raise (libc.so.6 + 0x4190e) #2 0x00007f0319e28942 abort (libc.so.6 + 0x28942) #3 0x00007f031a2e115e _ZSt21__glibcxx_assert_failPKciS0_S0_ (libstdc++.so.6 + 0xe115e) #4 0x000055ffdaad6692 _ZNSt8__detail9_ExecutorIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS5_9sub_matchISB_EEENS> #5 0x000055ffdaad40ca _ZNSt8__detail17__regex_algo_implIN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEESaINS5_9sub_match> #6 0x00007f031b67c7ee _ZN3nix16curlFileTransfer12TransferItem14headerCallbackEPvmm (liblixstore.so + 0x47c7ee) #7 0x00007f0319d4449e cw_out_cb_write (libcurl.so.4 + 0x3649e) #8 0x00007f0319d447a3 cw_out_ptr_flush (libcurl.so.4 + 0x367a3) #9 0x00007f0319d44dd5 cw_out_do_write (libcurl.so.4 + 0x36dd5) #10 0x00007f0319d45436 cw_pause_write (libcurl.so.4 + 0x37436) #11 0x00007f0319d8db6c cw_download_write (libcurl.so.4 + 0x7fb6c) #12 0x00007f0319d8f05b Curl_client_write (libcurl.so.4 + 0x8105b) #13 0x00007f0319d600a6 http_rw_hd (libcurl.so.4 + 0x520a6) #14 0x00007f0319d61ee7 Curl_http_write_resp_hd (libcurl.so.4 + 0x53ee7) #15 0x00007f0319d65ab2 h2_xfer_write_resp_hd.part.0 (libcurl.so.4 + 0x57ab2) #16 0x00007f0319d68f56 on_header (libcurl.so.4 + 0x5af56) #17 0x00007f03188f7a22 nghttp2_session_mem_recv2 (libnghttp2.so.14 + 0x12a22) #18 0x00007f0319d63a57 h2_process_pending_input (libcurl.so.4 + 0x55a57) #19 0x00007f0319d66b41 h2_progress_ingress (libcurl.so.4 + 0x58b41) #20 0x00007f0319d66f02 cf_h2_recv (libcurl.so.4 + 0x58f02) #21 0x00007f0319da1fa4 Curl_sendrecv (libcurl.so.4 + 0x93fa4) #22 0x00007f0319d81ac4 multi_runsingle (libcurl.so.4 + 0x73ac4) #23 0x00007f0319d82c3c curl_multi_perform (libcurl.so.4 + 0x74c3c) #24 0x00007f031b660f48 _ZN3nix16curlFileTransfer16workerThreadMainEv (liblixstore.so + 0x460f48) #25 0x00007f031b660889 _ZN3nix16curlFileTransfer17workerThreadEntryEv (liblixstore.so + 0x460889) #26 0x00007f031a2ed064 execute_native_thread_routine (libstdc++.so.6 + 0xed064) #27 0x00007f0319e9a97a start_thread (libc.so.6 + 0x9a97a) #28 0x00007f0319f22d2c __clone3 (libc.so.6 + 0x122d2c) ``` in my daemon traces

pennae commented

2025-11-03 03:15:26 +00:00

Owner

raito put together a reproducer that was kind of unreliable, running it in rr chaos mode made it reliable enough to debug and bisect. https://github.com/curl/curl/pull/18428 introduces the failure by subtly changing how paused streams are handled, with the effect that pausing a stream immediately trashes it and aborts that transfer with an error.