Deadlock on daemon interruption during path substitution #577

New issue

Closed

opened 2024-11-18 23:57:33 +00:00 by jade · 2 comments

jade commented 2024-11-18 23:57:33 +00:00

Owner

Copy link

Synopsis

A Lix daemon, version c859d03013, got stuck after its corresponding client got Ctrl-C'd (I believe that was proximate to the sadness, at least).
This stuck state leads to further clients trying to build the paths getting stuck on path locks, for which we have poor diagnostics (#535), which is somewhat confusing.

Diagnosis

Involved stack traces:

PathSubstitutionGoal threads:

Thread "puppy":

Thread 5 (Thread 0x754feb1aa6c0 (LWP 3558701) "nix-daemon"):
#0  0x000075500696637e in __futex_abstimed_wait_common () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#1  0x0000755006968ed0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#2  0x00007550074d201d in nix::Sync<nix::curlFileTransfer::TransferItem::DownloadState, std::mutex>::Lock::wait (this=0x754feb1a8268, cv=...) at src/libutil/sync.hh:97
#3  nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}::operator()() const (this=this@entry=0x754feb1a82c8) at src/libstore/filetransfer.cc:873
#4  0x00007550074d1d3e in nix::curlFileTransfer::TransferSource::withRetries<nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}>(nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}) (this=this@entry=0x754fc801b940, fn=...) at src/libstore/filetransfer.cc:785
#5  0x00007550074cb0ab in nix::curlFileTransfer::TransferSource::awaitData (this=0x754fc801b940) at src/libstore/filetransfer.cc:856
#6  nix::curlFileTransfer::TransferSource::read (this=0x754fc801b940, data=0x754fc802e360 "\3757zXZ", len=65536) at src/libstore/filetransfer.cc:884
#7  0x0000755007894c52 in nix::callback_read (archive=0x754fc8008550, _self=<optimized out>, buffer=<optimized out>) at src/libutil/tarfile.cc:23
#8  0x00007550067a5016 in __archive_read_filter_ahead () from target:/nix/store/fnjlamc0x1hx245px1s3rbazxfngffi0-libarchive-3.7.7-lib/lib/libarchive.so.13
#9  0x00007550067ac094 in bzip2_reader_bid () from target:/nix/store/fnjlamc0x1hx245px1s3rbazxfngffi0-libarchive-3.7.7-lib/lib/libarchive.so.13
#10 0x00007550067a58bd in archive_read_open1 () from target:/nix/store/fnjlamc0x1hx245px1s3rbazxfngffi0-libarchive-3.7.7-lib/lib/libarchive.so.13
#11 0x0000755007894b49 in nix::TarArchive::TarArchive (this=0x754fc8001bc0, source=..., raw=true) at src/libutil/tarfile.cc:59
#12 0x00007550078082c3 in std::make_unique<nix::TarArchive, nix::Source&, bool> (__args=<optimized out>, __args=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/unique_ptr.h:1070
#13 nix::ArchiveDecompressionSource::read (this=0x754fc8011e70, data=0x754fc801e350 "", len=65536) at src/libutil/compression.cc:47
#14 0x0000755007456f40 in nix::BinaryCacheStore::narFromPath(nix::StorePath const&)::$_0::operator()<nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats>(nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats&) const (this=<optimized out>, info=..., file=..., stats=...) at src/libstore/binary-cache-store.cc:345
#15 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754fc800ccf8) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#16 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=0x754fc800ccf0) at src/libutil/generator.hh:150
#17 0x00007550075bdbac in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform>::next (this=0x754fc80018e0) at src/libutil/generator.hh:266
#18 nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag)::$_0::operator()<nix::Activity, nix::ref<nix::ValidPathInfo const>, nix::Store, nix::StorePath const>(nix::Activity&, nix::ref<nix::ValidPathInfo const>&, nix::Store&, nix::StorePath const&) const (this=<optimized out>, act=..., info=..., srcStore=..., storePath=...) at src/libstore/store-api.cc:1108
#19 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb1a8c00) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#20 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=this@entry=0x754feb1a8bf8) at src/libutil/generator.hh:150
#21 0x00005aa91d813ccb in nix::Generator<std::span<char const, 18446744073709551615ul>, void>::next (this=0x754feb1a8bf8) at src/libutil/generator.hh:266
#22 nix::GeneratorSource::read (this=0x754feb1a8bf0, data=0x754feb1a86d0 ".", len=8) at src/libutil/serialise.hh:352
#23 0x000075500745d305 in nix::TeeSource::read (this=0x754feb1a8948, data=0x189 <error: Cannot access memory at address 0x189>, len=0) at src/libutil/serialise.hh:238
#24 0x000075500788794f in nix::Source::operator() (data=0x754feb1a86d0 ".", len=8, this=<optimized out>) at src/libutil/serialise.cc:111
#25 nix::readNum<unsigned long> (source=...) at src/libutil/serialise.cc:31
#26 nix::readString[abi:cxx11](nix::Source&, unsigned long) (source=..., max=max@entry=13) at src/libutil/serialise.cc:267
#27 0x00007550077e3e0c in nix::nar::parse (source=...) at src/libutil/archive.cc:320
#28 0x00007550077e4762 in std::__n4861::coroutine_handle<void>::resume (this=0x754fc80023c0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#29 nix::_generator::GeneratorBase<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next (this=0x754fc80023c0) at src/libutil/generator.hh:150
#30 nix::Generator<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, void>::next (this=0x754fc80023c0) at src/libutil/generator.hh:266
#31 nix::restore (sink=..., nar=...) at src/libutil/archive.cc:336
#32 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb1a8870) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#33 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=0x754feb1a8868) at src/libutil/generator.hh:150
#34 0x00007550077df26b in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform>::next (this=0x754feb1a8868) at src/libutil/generator.hh:266
#35 nix::parseDump (sink=..., source=...) at src/libutil/archive.cc:382
#36 0x00007550077df359 in nix::restorePath (path=..., source=...) at src/libutil/archive.cc:508
#37 0x000075500751fe21 in nix::LocalStore::addToStore (this=0x5aa93ba6a1b0, info=..., source=..., repair=nix::NoRepair, checkSigs=<optimized out>) at src/libstore/local-store.cc:1263
#38 0x00007550075b6278 in nix::copyStorePath (srcStore=..., dstStore=..., storePath=..., repair=nix::NoRepair, checkSigs=nix::CheckSigs) at src/libstore/store-api.cc:1116
#39 0x0000755007655126 in nix::PathSubstitutionGoal::tryToRun()::$_0::operator()() const (this=<optimized out>) at src/libstore/build/substitution-goal.cc:222
#40 std::__invoke_impl<void, nix::PathSubstitutionGoal::tryToRun()::$_0>(std::__invoke_other, nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61
#41 std::__invoke<nix::PathSubstitutionGoal::tryToRun()::$_0>(nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:96
#42 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::_M_invoke<0ul> (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:292
#43 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::operator() (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:299
#44 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::operator() (this=0x754feb1a8ff0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1432
#45 std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61
#46 std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:114
#47 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void> >::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290
#48 0x00007550074c1d8a in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>::operator() (this=0x189) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591
#49 std::__future_base::_State_baseV2::_M_do_set (this=0x5aa93d4baf40, __f=0x189, __did_set=0x0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:589
#50 0x000075500696ee2f in __pthread_once_slow () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#51 0x00007550074c1ca0 in __gthread_once (__once=0x5aa93d4baf58, __func=0x189) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700
#52 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*> (__once=..., __f=@0x754feb1a8f70: (void (std::__future_base::_State_baseV2::*)(class std::__future_base::_State_baseV2 * const, class std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x7550074c1d60 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __args=@0x754feb1a8f58: 0x754feb1a8f87, __args=@0x754feb1a8f58: 0x754feb1a8f87, __args=@0x754feb1a8f58: 0x754feb1a8f87) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/mutex:907
#53 std::__future_base::_State_baseV2::_M_set_result (this=0x5aa93d4baf40, __res=..., __ignore_failure=false) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:428
#54 0x0000755007654bd2 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::_M_run (this=0x754fc8007c28) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1774
#55 0x0000755006be16d3 in execute_native_thread_routine () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6
#56 0x0000755006969d02 in start_thread () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#57 0x00007550069e93ac in __clone3 () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6

Thread "kitty":

Thread 3 (Thread 0x754feb9ab6c0 (LWP 3558703) "nix-daemon"):
#0  0x00007550069e713d in syscall () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#1  0x0000755006bdec69 in std::__atomic_futex_unsigned_base::_M_futex_wait_until(unsigned int*, unsigned int, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6
#2  0x00007550074d02b1 in std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until (__assumed=<optimized out>, this=<optimized out>, __operand=<optimized out>, __equal=<optimized out>, __mo=<optimized out>, __has_timeout=<optimized out>, __s=..., __ns=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/atomic_futex.h:109
#3  std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test (this=<optimized out>, __assumed=<optimized out>, __operand=<optimized out>, __equal=<optimized out>, __mo=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/atomic_futex.h:158
#4  std::__atomic_futex_unsigned<2147483648u>::_M_load_when_equal (this=0x754fd8005710, __val=1, __mo=std::memory_order::acquire) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/atomic_futex.h:212
#5  std::__future_base::_State_baseV2::wait (this=0x754fd8005700) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:353
#6  std::__basic_future<nix::FileTransferResult>::_M_get_result (this=this@entry=0x754feb9a90e8) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:741
#7  0x00007550074ccd9d in std::future<nix::FileTransferResult>::get (this=0x0, this@entry=0x754feb9a90e8) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:827
#8  0x00007550074cc18f in nix::curlFileTransfer::TransferSource::startTransfer (this=this@entry=0x754fd80103d0, uri=..., offset=offset@entry=0) at src/libstore/filetransfer.cc:807
#9  0x00007550074caad8 in nix::curlFileTransfer::TransferSource::TransferSource(nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool)::{lambda()#1}::operator()() const (this=<optimized out>) at src/libstore/filetransfer.cc:762
#10 nix::curlFileTransfer::TransferSource::withRetries<nix::curlFileTransfer::TransferSource::withRetries(nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool)::{lambda()#1}>(nix::curlFileTransfer::TransferSource::withRetries(nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool)::{lambda()#1}) (this=this@entry=0x754fd80103d0, fn=...) at src/libstore/filetransfer.cc:785
#11 0x00007550074ca83f in nix::curlFileTransfer::TransferSource::TransferSource (this=this@entry=0x754fd80103d0, parent=..., uri=..., headers=..., data=..., noBody=false) at src/libstore/filetransfer.cc:762
#12 0x00007550074ca599 in std::make_unique<nix::curlFileTransfer::TransferSource, nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool&> (__args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/unique_ptr.h:1070
#13 0x00007550074c6126 in nix::make_box_ptr<nix::curlFileTransfer::TransferSource, nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool&> (args=@0x754feb9a9327: false, args=@0x754feb9a9327: false, args=@0x754feb9a9327: false, args=@0x754feb9a9327: false, args=@0x754feb9a9327: false) at src/libutil/box_ptr.hh:119
#14 nix::curlFileTransfer::enqueueFileTransfer (this=0x754fa003a5c0, uri=..., headers=..., data=..., noBody=false) at src/libstore/filetransfer.cc:726
#15 0x00007550074bf436 in nix::curlFileTransfer::download (this=0x0, uri=..., headers=...) at src/libstore/filetransfer.cc:916
#16 0x0000755007501060 in nix::HttpBinaryCacheStore::getFile (this=0x754fa0006c80, path=...) at src/libstore/http-binary-cache-store.cc:150
#17 0x0000755007501830 in virtual thunk to nix::HttpBinaryCacheStore::getFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from target:/nix/store/6516nyk2cr6qr5b1nzfzgfnq79ksdlbn-lix-2.92.0-dev-pre20241115-c859d03/lib/liblixstore.so
#18 0x00007550074523f8 in nix::BinaryCacheStore::narFromPath (this=0x754fa0007638, storePath=...) at src/libstore/binary-cache-store.cc:337
#19 0x0000755007452743 in virtual thunk to nix::BinaryCacheStore::narFromPath(nix::StorePath const&) () at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:1349
#20 0x00007550075bdb96 in nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag)::$_0::operator()<nix::Activity, nix::ref<nix::ValidPathInfo const>, nix::Store, nix::StorePath const>(nix::Activity&, nix::ref<nix::ValidPathInfo const>&, nix::Store&, nix::StorePath const&) const (this=<optimized out>, act=..., info=..., srcStore=..., storePath=...) at src/libstore/store-api.cc:1106
#21 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb9a9c00) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#22 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=this@entry=0x754feb9a9bf8) at src/libutil/generator.hh:150
#23 0x00005aa91d813ccb in nix::Generator<std::span<char const, 18446744073709551615ul>, void>::next (this=0x754feb9a9bf8) at src/libutil/generator.hh:266
#24 nix::GeneratorSource::read (this=0x754feb9a9bf0, data=0x754feb9a96d0 "/", len=8) at src/libutil/serialise.hh:352
#25 0x000075500745d305 in nix::TeeSource::read (this=0x754feb9a9948, data=0x0, len=2147483648) at src/libutil/serialise.hh:238
#26 0x000075500788794f in nix::Source::operator() (data=0x754feb9a96d0 "/", len=8, this=<optimized out>) at src/libutil/serialise.cc:111
#27 nix::readNum<unsigned long> (source=...) at src/libutil/serialise.cc:31
#28 nix::readString[abi:cxx11](nix::Source&, unsigned long) (source=..., max=max@entry=13) at src/libutil/serialise.cc:267
#29 0x00007550077e3e0c in nix::nar::parse (source=...) at src/libutil/archive.cc:320
#30 0x00007550077e4762 in std::__n4861::coroutine_handle<void>::resume (this=0x754fd8002b50) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#31 nix::_generator::GeneratorBase<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next (this=0x754fd8002b50) at src/libutil/generator.hh:150
#32 nix::Generator<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, void>::next (this=0x754fd8002b50) at src/libutil/generator.hh:266
#33 nix::restore (sink=..., nar=...) at src/libutil/archive.cc:336
#34 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb9a9870) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#35 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=0x754feb9a9868) at src/libutil/generator.hh:150
#36 0x00007550077df26b in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform>::next (this=0x754feb9a9868) at src/libutil/generator.hh:266
#37 nix::parseDump (sink=..., source=...) at src/libutil/archive.cc:382
#38 0x00007550077df359 in nix::restorePath (path=..., source=...) at src/libutil/archive.cc:508
#39 0x000075500751fe21 in nix::LocalStore::addToStore (this=0x5aa93ba6a1b0, info=..., source=..., repair=nix::NoRepair, checkSigs=<optimized out>) at src/libstore/local-store.cc:1263
#40 0x00007550075b6278 in nix::copyStorePath (srcStore=..., dstStore=..., storePath=..., repair=nix::NoRepair, checkSigs=nix::CheckSigs) at src/libstore/store-api.cc:1116
#41 0x0000755007655126 in nix::PathSubstitutionGoal::tryToRun()::$_0::operator()() const (this=<optimized out>) at src/libstore/build/substitution-goal.cc:222
#42 std::__invoke_impl<void, nix::PathSubstitutionGoal::tryToRun()::$_0>(std::__invoke_other, nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61
#43 std::__invoke<nix::PathSubstitutionGoal::tryToRun()::$_0>(nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:96
#44 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::_M_invoke<0ul> (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:292
#45 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::operator() (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:299
#46 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::operator() (this=0x754feb9a9ff0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1432
#47 std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61
#48 std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:114
#49 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void> >::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290
#50 0x00007550074c1d8a in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>::operator() (this=0x0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591
#51 std::__future_base::_State_baseV2::_M_do_set (this=0x5aa93c23ac90, __f=0x0, __did_set=0x80000000) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:589
#52 0x000075500696ee2f in __pthread_once_slow () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#53 0x00007550074c1ca0 in __gthread_once (__once=0x5aa93c23aca8, __func=0x0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700
#54 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*> (__once=..., __f=@0x754feb9a9f70: (void (std::__future_base::_State_baseV2::*)(class std::__future_base::_State_baseV2 * const, class std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x7550074c1d60 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __args=@0x754feb9a9f58: 0x754feb9a9f87, __args=@0x754feb9a9f58: 0x754feb9a9f87, __args=@0x754feb9a9f58: 0x754feb9a9f87) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/mutex:907
#55 std::__future_base::_State_baseV2::_M_set_result (this=0x5aa93c23ac90, __res=..., __ignore_failure=false) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:428
#56 0x0000755007654bd2 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::_M_run (this=0x754fd8005710) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1774
#57 0x0000755006be16d3 in execute_native_thread_routine () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6
#58 0x0000755006969d02 in start_thread () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#59 0x00007550069e93ac in __clone3 () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6

Main thread:

Thread 1 (Thread 0x755004b5e5c0 (LWP 3541973) "nix-daemon"):
#0  0x000075500696637e in __futex_abstimed_wait_common () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#1  0x000075500696b803 in __pthread_clockjoin_ex () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#2  0x0000755006be1747 in std::thread::join() () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6
#3  0x000075500696ee2f in __pthread_once_slow () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6
#4  0x0000755007609f71 in __gthread_once (__once=0x5aa93d245248, __func=0x109) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700
#5  std::call_once<void (std::thread::*)(), std::thread*> (__once=..., __f=@0x7ffdfcf10858: (void (std::thread::*)(class std::thread * const)) 0x755006be1730 <std::thread::join()>, __args=@0x7ffdfcf10850: 0x5aa93d245240) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/mutex:907
#6  std::__future_base::_Async_state_commonV2::_M_join (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1737
#7  std::__future_base::_Async_state_commonV2::_M_complete_async (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1735
#8  0x00007550074d1b3a in std::__future_base::_State_baseV2::wait (this=0x5aa93d245220) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:350
#9  std::__basic_future<void>::_M_get_result (this=this@entry=0x5aa93d29c9c0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:741
#10 0x00007550074d1382 in std::future<void>::get (this=0x754fd27fc990, this@entry=0x5aa93d29c9c0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:913
#11 0x000075500764ed65 in nix::PathSubstitutionGoal::cleanup (this=0x5aa93d29c8d0) at src/libstore/build/substitution-goal.cc:294
#12 nix::PathSubstitutionGoal::~PathSubstitutionGoal (this=0x5aa93d29c8d0) at src/libstore/build/substitution-goal.cc:32
#13 0x000075500764f29c in nix::PathSubstitutionGoal::~PathSubstitutionGoal (this=0x754fd27fc990) at src/libstore/build/substitution-goal.cc:31
#14 0x000075500766c0b0 in std::default_delete<nix::PathSubstitutionGoal>::operator() (__ptr=0x754fd27fc990, this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/unique_ptr.h:99
#15 std::_Sp_counted_deleter<nix::PathSubstitutionGoal*, std::default_delete<nix::PathSubstitutionGoal>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:527
#16 0x0000755007669a0a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5aa93d27f190) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:346
#17 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x5aa93d29cf08) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:1071
#18 std::__shared_ptr<nix::PathSubstitutionGoal, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:1524
#19 nix::Worker::CachedGoal<nix::PathSubstitutionGoal>::~CachedGoal (this=<optimized out>) at src/libstore/build/worker.hh:128
#20 std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >::~pair (this=this@entry=0x5aa93d29cee0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_pair.h:187
#21 0x0000755007669989 in std::destroy_at<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > (__location=0x754fd27fc990, __location@entry=0x5aa93bebe8a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_construct.h:88
#22 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > > >::destroy<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > (__p=0x754fd27fc990, __p@entry=0x5aa93bebe8a0, __a=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/alloc_traits.h:560
#23 std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_destroy_node (this=0x5aa93bebe8a0, __p=0x5aa93d29cec0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:625
#24 std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_drop_node (this=0x5aa93bebe8a0, __p=0x5aa93d29cec0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:633
#25 std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93d29cec0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1938
#26 0x000075500766997c in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93c377c60) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1936
#27 0x000075500766997c in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93c2b73a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1936
#28 0x000075500766997c in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93c29e450) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1936
#29 0x0000755007659262 in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::clear (this=0x5aa93bebe8a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1255
#30 std::map<nix::StorePath, nix::Worker::CachedGoal<nix::PathSubstitutionGoal>, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::clear (this=0x5aa93bebe8a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_map.h:1184
#31 nix::Worker::~Worker (this=0x5aa93bebe860) at src/libstore/build/worker.cc:53
#32 0x00007550076101b0 in nix::processGoals<nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0>(nix::Store&, nix::Store&, kj::AsyncIoContext&, nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0&&) (store=..., evalStore=..., aio=..., mkGoals=...) at src/libstore/build/worker.hh:302
#33 0x00007550058bdb92 in std::__n4861::coroutine_handle<void>::resume (this=0x149) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135
#34 kj::_::CoroutineBase::fire (this=0xf9) at /build/source/c++/src/kj/async.c++:3031
#35 0x00007550058bdb92 in non-virtual thunk to kj::_::CoroutineBase::fire() () from target:/nix/store/0vpa01040n1wr9dw13ffy7svgl13hng2-capnproto-1.0.2/lib/libkj-async.so.1.0.2
#36 0x00007550058beb52 in kj::EventLoop::turn (this=0x754fb4014a50) at /build/source/c++/src/kj/async.c++:1806
#37 kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const (this=this@entry=0x7ffdfcf10b30) at /build/source/c++/src/kj/async.c++:1970
#38 0x00007550058b7d58 in kj::WaitScope::runOnStackPool<kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2>(kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2&&) (this=0x754fb4014aa8, func=...) at /build/source/c++/src/kj/async.h:1368
#39 kj::_::waitImpl (node=..., result=..., waitScope=..., location=...) at /build/source/c++/src/kj/async.c++:1967
#40 0x0000755007612012 in kj::Promise<boost::outcome_v2::basic_result<nix::Worker::Results, std::__exception_ptr::exception_ptr, boost::outcome_v2::policy::exception_ptr_rethrow<nix::Worker::Results, std::__exception_ptr::exception_ptr, void> > >::wait (this=0xfffffffffffffe00, this@entry=0x7ffdfcf10e98, waitScope=..., location=...) at /nix/store/0vpa01040n1wr9dw13ffy7svgl13hng2-capnproto-1.0.2/include/kj/async-inl.h:1357
#41 0x000075500760b6de in nix::Store::buildPathsWithResults (this=0x5aa93ba6acc0, reqs=..., buildMode=nix::bmNormal, evalStore=...) at src/libstore/build/entry-points.cc:57
#42 0x0000755007483dfc in nix::daemon::performOp (logger=logger@entry=0x5aa93ba69270, store=..., trusted=<optimized out>, recursive=<optimized out>, clientVersion=291, from=..., to=..., op=nix::WorkerProto::Op::BuildPathsWithResults) at src/libstore/daemon.cc:572
#43 0x00007550074806b5 in nix::daemon::processConnection (store=..., from=..., to=..., trusted=<optimized out>, recursive=<optimized out>) at src/libstore/daemon.cc:1076
#44 0x00005aa91d79df95 in nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0::operator()() const (this=0x5aa93ba6b960) at src/nix/daemon.cc:343
#45 std::__invoke_impl<void, nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&>(std::__invoke_other, nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61
#46 std::__invoke_r<void, nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&>(nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:111
#47 std::_Function_handler<void(), nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0>::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290
#48 0x000075500786f12a in std::function<void()>::operator() (this=0x754fd27fc990) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591
#49 nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0::operator()() const (this=0x7ffdfcf11a20) at src/libutil/processes.cc:194
#50 std::__invoke_impl<void, nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&>(std::__invoke_other, nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61
#51 std::__invoke_r<void, nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&>(nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:111
#52 std::_Function_handler<void(), nix::startProcess(std::function<void()>, nix::ProcessOptions const&)::$_0>::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290
#53 0x000075500786de4e in nix::doFork (fun=...) at src/libutil/processes.cc:171
#54 nix::startProcess (fun=..., options=...) at src/libutil/processes.cc:225
#55 0x00005aa91d79c1fb in nix::daemonLoop (forceTrustClientOpt=...) at src/nix/daemon.cc:320
#56 nix::runDaemon (stdio=<optimized out>, forceTrustClientOpt=...) at src/nix/daemon.cc:436
#57 0x00005aa91d79b1f6 in nix::main_nix_daemon (argc=<optimized out>, argv=<optimized out>) at src/nix/daemon.cc:467
#58 0x00005aa91d81ad25 in std::function<void(int, char**)>::operator() (this=0x7ffdfcf12200, __args=0x7ffdfcf12f28, __args=0x7ffdfcf12f28) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591
#59 nix::mainWrapped (argc=2, argv=0x7ffdfcf12f28) at src/nix/main.cc:389
#60 0x0000755006eca421 in std::function<void()>::operator() (this=0x7ffdfcf12da0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591
#61 nix::handleExceptions (programName=..., fun=...) at src/libmain/shared.cc:325
#62 0x00005aa91d81f08b in main (argc=2, argv=0x7ffdfcf12f28) at src/nix/main.cc:547

Notably, there is no thread with workerThreadEntry in the call stack. This is almost certainly related.

More detailed execution state description:

The main thread is waiting for PathSubstitutionGoal::thr.get():

void PathSubstitutionGoal::cleanup()
{
    try {
        if (thr.valid()) {
            // FIXME: signal worker thread to quit.
            thr.get();
        }
    } catch (...) {
        ignoreExceptionInDestructor();
    }
}

where thr is one of the PathSubstitutionGoal threads mentioned above. It's waiting on joining that thread.

Those threads are either "puppy" or "kitty". Puppy threads are waiting on transfer->downloadEvent:

        bool awaitData()
        {
            return withRetries([&] {
                /* Grab data if available, otherwise wait for the download
                   thread to wake us up. */
                while (buffered.empty()) {
                    auto state(transfer->downloadState.lock());

                    if (!state->data.empty()) {
                        chunk = std::move(state->data);
                        buffered = chunk;
                        totalReceived += chunk.size();
                        parent.unpause(transfer);
                    } else if (state->exc) {
                        std::rethrow_exception(state->exc);
                    } else if (state->done) {
                        return false;
                    } else {
                        parent.unpause(transfer);
                        state.wait(transfer->downloadEvent); // <--------- HERE
                    }
                }

                return true;
            });

Kitty threads are waiting on transfer->metadataPromise.get_future().get():

        FileTransferResult startTransfer(const std::string & uri, curl_off_t offset = 0)
        {
            attempt += 1;
            auto uploadData = data ? std::optional(std::string_view(*data)) : std::nullopt;
            transfer =
                std::make_shared<TransferItem>(uri, headers, parentAct, uploadData, noBody, offset);
            parent.enqueueItem(transfer);
            return transfer->metadataPromise.get_future().get(); // <------------ HERE
        }

In both instances, these are states that are progressed by the download thread.
For puppy threads, it would be progressed if the download got the data callback called for more data.
For kitty threads, it would be progressed by receiving headers (or finishing the request with headers?).

The main thread is stuck because it is waiting on either a puppy or kitty thread.

So, it appears that the root cause is that the download thread can exit without fulfilling all its expectations or notifying them that it cannot complete them.

How did this happen?

Well, and this is the question. I have no idea the correct order of operations to be able to cleanly tear down the download thread while rejecting all its expectations.

Effectively the waiting on synchronization primitives needs to have that channel multiplexed with a possible error that probably gets rethrown.

I also don't know why this used to work. At one point, the answer to this is that the daemon crashed if the client hangs up due to buggy signal handling.
Now that the signal handling is a bit less buggy, it seems like the problem is that the async runtime is being bypassed while waiting for a future on the main thread, and so it cannot switch from the task and is stuck in blocking code.

The puppy/kitty worker threads getting abandoned appears to be intentional based on the TODO above, and this is not itself a big problem. The issue is that the main thread is getting stuck waiting on one of them in a way that does not allow the async runtime to do other things and thus not being able to tear the whole thing down properly.

## Synopsis A Lix daemon, version c859d03013, got stuck after its corresponding client got Ctrl-C'd (I believe that was proximate to the sadness, at least). This stuck state leads to further clients trying to build the paths getting stuck on path locks, for which we have poor diagnostics (https://git.lix.systems/lix-project/lix/issues/535), which is somewhat confusing. ## Diagnosis Involved stack traces: PathSubstitutionGoal threads: Thread "puppy": ``` Thread 5 (Thread 0x754feb1aa6c0 (LWP 3558701) "nix-daemon"): #0 0x000075500696637e in __futex_abstimed_wait_common () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #1 0x0000755006968ed0 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #2 0x00007550074d201d in nix::Sync<nix::curlFileTransfer::TransferItem::DownloadState, std::mutex>::Lock::wait (this=0x754feb1a8268, cv=...) at src/libutil/sync.hh:97 #3 nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}::operator()() const (this=this@entry=0x754feb1a82c8) at src/libstore/filetransfer.cc:873 #4 0x00007550074d1d3e in nix::curlFileTransfer::TransferSource::withRetries<nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}>(nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}) (this=this@entry=0x754fc801b940, fn=...) at src/libstore/filetransfer.cc:785 #5 0x00007550074cb0ab in nix::curlFileTransfer::TransferSource::awaitData (this=0x754fc801b940) at src/libstore/filetransfer.cc:856 #6 nix::curlFileTransfer::TransferSource::read (this=0x754fc801b940, data=0x754fc802e360 "\3757zXZ", len=65536) at src/libstore/filetransfer.cc:884 #7 0x0000755007894c52 in nix::callback_read (archive=0x754fc8008550, _self=<optimized out>, buffer=<optimized out>) at src/libutil/tarfile.cc:23 #8 0x00007550067a5016 in __archive_read_filter_ahead () from target:/nix/store/fnjlamc0x1hx245px1s3rbazxfngffi0-libarchive-3.7.7-lib/lib/libarchive.so.13 #9 0x00007550067ac094 in bzip2_reader_bid () from target:/nix/store/fnjlamc0x1hx245px1s3rbazxfngffi0-libarchive-3.7.7-lib/lib/libarchive.so.13 #10 0x00007550067a58bd in archive_read_open1 () from target:/nix/store/fnjlamc0x1hx245px1s3rbazxfngffi0-libarchive-3.7.7-lib/lib/libarchive.so.13 #11 0x0000755007894b49 in nix::TarArchive::TarArchive (this=0x754fc8001bc0, source=..., raw=true) at src/libutil/tarfile.cc:59 #12 0x00007550078082c3 in std::make_unique<nix::TarArchive, nix::Source&, bool> (__args=<optimized out>, __args=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/unique_ptr.h:1070 #13 nix::ArchiveDecompressionSource::read (this=0x754fc8011e70, data=0x754fc801e350 "", len=65536) at src/libutil/compression.cc:47 #14 0x0000755007456f40 in nix::BinaryCacheStore::narFromPath(nix::StorePath const&)::$_0::operator()<nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats>(nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats&) const (this=<optimized out>, info=..., file=..., stats=...) at src/libstore/binary-cache-store.cc:345 #15 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754fc800ccf8) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #16 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=0x754fc800ccf0) at src/libutil/generator.hh:150 #17 0x00007550075bdbac in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform>::next (this=0x754fc80018e0) at src/libutil/generator.hh:266 #18 nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag)::$_0::operator()<nix::Activity, nix::ref<nix::ValidPathInfo const>, nix::Store, nix::StorePath const>(nix::Activity&, nix::ref<nix::ValidPathInfo const>&, nix::Store&, nix::StorePath const&) const (this=<optimized out>, act=..., info=..., srcStore=..., storePath=...) at src/libstore/store-api.cc:1108 #19 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb1a8c00) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #20 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=this@entry=0x754feb1a8bf8) at src/libutil/generator.hh:150 #21 0x00005aa91d813ccb in nix::Generator<std::span<char const, 18446744073709551615ul>, void>::next (this=0x754feb1a8bf8) at src/libutil/generator.hh:266 #22 nix::GeneratorSource::read (this=0x754feb1a8bf0, data=0x754feb1a86d0 ".", len=8) at src/libutil/serialise.hh:352 #23 0x000075500745d305 in nix::TeeSource::read (this=0x754feb1a8948, data=0x189 <error: Cannot access memory at address 0x189>, len=0) at src/libutil/serialise.hh:238 #24 0x000075500788794f in nix::Source::operator() (data=0x754feb1a86d0 ".", len=8, this=<optimized out>) at src/libutil/serialise.cc:111 #25 nix::readNum<unsigned long> (source=...) at src/libutil/serialise.cc:31 #26 nix::readString[abi:cxx11](nix::Source&, unsigned long) (source=..., max=max@entry=13) at src/libutil/serialise.cc:267 #27 0x00007550077e3e0c in nix::nar::parse (source=...) at src/libutil/archive.cc:320 #28 0x00007550077e4762 in std::__n4861::coroutine_handle<void>::resume (this=0x754fc80023c0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #29 nix::_generator::GeneratorBase<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next (this=0x754fc80023c0) at src/libutil/generator.hh:150 #30 nix::Generator<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, void>::next (this=0x754fc80023c0) at src/libutil/generator.hh:266 #31 nix::restore (sink=..., nar=...) at src/libutil/archive.cc:336 #32 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb1a8870) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #33 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=0x754feb1a8868) at src/libutil/generator.hh:150 #34 0x00007550077df26b in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform>::next (this=0x754feb1a8868) at src/libutil/generator.hh:266 #35 nix::parseDump (sink=..., source=...) at src/libutil/archive.cc:382 #36 0x00007550077df359 in nix::restorePath (path=..., source=...) at src/libutil/archive.cc:508 #37 0x000075500751fe21 in nix::LocalStore::addToStore (this=0x5aa93ba6a1b0, info=..., source=..., repair=nix::NoRepair, checkSigs=<optimized out>) at src/libstore/local-store.cc:1263 #38 0x00007550075b6278 in nix::copyStorePath (srcStore=..., dstStore=..., storePath=..., repair=nix::NoRepair, checkSigs=nix::CheckSigs) at src/libstore/store-api.cc:1116 #39 0x0000755007655126 in nix::PathSubstitutionGoal::tryToRun()::$_0::operator()() const (this=<optimized out>) at src/libstore/build/substitution-goal.cc:222 #40 std::__invoke_impl<void, nix::PathSubstitutionGoal::tryToRun()::$_0>(std::__invoke_other, nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61 #41 std::__invoke<nix::PathSubstitutionGoal::tryToRun()::$_0>(nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:96 #42 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::_M_invoke<0ul> (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:292 #43 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::operator() (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:299 #44 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::operator() (this=0x754feb1a8ff0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1432 #45 std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61 #46 std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:114 #47 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void> >::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290 #48 0x00007550074c1d8a in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>::operator() (this=0x189) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591 #49 std::__future_base::_State_baseV2::_M_do_set (this=0x5aa93d4baf40, __f=0x189, __did_set=0x0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:589 #50 0x000075500696ee2f in __pthread_once_slow () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #51 0x00007550074c1ca0 in __gthread_once (__once=0x5aa93d4baf58, __func=0x189) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700 #52 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*> (__once=..., __f=@0x754feb1a8f70: (void (std::__future_base::_State_baseV2::*)(class std::__future_base::_State_baseV2 * const, class std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x7550074c1d60 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __args=@0x754feb1a8f58: 0x754feb1a8f87, __args=@0x754feb1a8f58: 0x754feb1a8f87, __args=@0x754feb1a8f58: 0x754feb1a8f87) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/mutex:907 #53 std::__future_base::_State_baseV2::_M_set_result (this=0x5aa93d4baf40, __res=..., __ignore_failure=false) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:428 #54 0x0000755007654bd2 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::_M_run (this=0x754fc8007c28) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1774 #55 0x0000755006be16d3 in execute_native_thread_routine () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6 #56 0x0000755006969d02 in start_thread () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #57 0x00007550069e93ac in __clone3 () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 ``` Thread "kitty": ``` Thread 3 (Thread 0x754feb9ab6c0 (LWP 3558703) "nix-daemon"): #0 0x00007550069e713d in syscall () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #1 0x0000755006bdec69 in std::__atomic_futex_unsigned_base::_M_futex_wait_until(unsigned int*, unsigned int, bool, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6 #2 0x00007550074d02b1 in std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until (__assumed=<optimized out>, this=<optimized out>, __operand=<optimized out>, __equal=<optimized out>, __mo=<optimized out>, __has_timeout=<optimized out>, __s=..., __ns=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/atomic_futex.h:109 #3 std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test (this=<optimized out>, __assumed=<optimized out>, __operand=<optimized out>, __equal=<optimized out>, __mo=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/atomic_futex.h:158 #4 std::__atomic_futex_unsigned<2147483648u>::_M_load_when_equal (this=0x754fd8005710, __val=1, __mo=std::memory_order::acquire) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/atomic_futex.h:212 #5 std::__future_base::_State_baseV2::wait (this=0x754fd8005700) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:353 #6 std::__basic_future<nix::FileTransferResult>::_M_get_result (this=this@entry=0x754feb9a90e8) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:741 #7 0x00007550074ccd9d in std::future<nix::FileTransferResult>::get (this=0x0, this@entry=0x754feb9a90e8) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:827 #8 0x00007550074cc18f in nix::curlFileTransfer::TransferSource::startTransfer (this=this@entry=0x754fd80103d0, uri=..., offset=offset@entry=0) at src/libstore/filetransfer.cc:807 #9 0x00007550074caad8 in nix::curlFileTransfer::TransferSource::TransferSource(nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool)::{lambda()#1}::operator()() const (this=<optimized out>) at src/libstore/filetransfer.cc:762 #10 nix::curlFileTransfer::TransferSource::withRetries<nix::curlFileTransfer::TransferSource::withRetries(nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool)::{lambda()#1}>(nix::curlFileTransfer::TransferSource::withRetries(nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool)::{lambda()#1}) (this=this@entry=0x754fd80103d0, fn=...) at src/libstore/filetransfer.cc:785 #11 0x00007550074ca83f in nix::curlFileTransfer::TransferSource::TransferSource (this=this@entry=0x754fd80103d0, parent=..., uri=..., headers=..., data=..., noBody=false) at src/libstore/filetransfer.cc:762 #12 0x00007550074ca599 in std::make_unique<nix::curlFileTransfer::TransferSource, nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool&> (__args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false, __args=@0x754feb9a9327: false) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/unique_ptr.h:1070 #13 0x00007550074c6126 in nix::make_box_ptr<nix::curlFileTransfer::TransferSource, nix::curlFileTransfer&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, bool&> (args=@0x754feb9a9327: false, args=@0x754feb9a9327: false, args=@0x754feb9a9327: false, args=@0x754feb9a9327: false, args=@0x754feb9a9327: false) at src/libutil/box_ptr.hh:119 #14 nix::curlFileTransfer::enqueueFileTransfer (this=0x754fa003a5c0, uri=..., headers=..., data=..., noBody=false) at src/libstore/filetransfer.cc:726 #15 0x00007550074bf436 in nix::curlFileTransfer::download (this=0x0, uri=..., headers=...) at src/libstore/filetransfer.cc:916 #16 0x0000755007501060 in nix::HttpBinaryCacheStore::getFile (this=0x754fa0006c80, path=...) at src/libstore/http-binary-cache-store.cc:150 #17 0x0000755007501830 in virtual thunk to nix::HttpBinaryCacheStore::getFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from target:/nix/store/6516nyk2cr6qr5b1nzfzgfnq79ksdlbn-lix-2.92.0-dev-pre20241115-c859d03/lib/liblixstore.so #18 0x00007550074523f8 in nix::BinaryCacheStore::narFromPath (this=0x754fa0007638, storePath=...) at src/libstore/binary-cache-store.cc:337 #19 0x0000755007452743 in virtual thunk to nix::BinaryCacheStore::narFromPath(nix::StorePath const&) () at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:1349 #20 0x00007550075bdb96 in nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag)::$_0::operator()<nix::Activity, nix::ref<nix::ValidPathInfo const>, nix::Store, nix::StorePath const>(nix::Activity&, nix::ref<nix::ValidPathInfo const>&, nix::Store&, nix::StorePath const&) const (this=<optimized out>, act=..., info=..., srcStore=..., storePath=...) at src/libstore/store-api.cc:1106 #21 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb9a9c00) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #22 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=this@entry=0x754feb9a9bf8) at src/libutil/generator.hh:150 #23 0x00005aa91d813ccb in nix::Generator<std::span<char const, 18446744073709551615ul>, void>::next (this=0x754feb9a9bf8) at src/libutil/generator.hh:266 #24 nix::GeneratorSource::read (this=0x754feb9a9bf0, data=0x754feb9a96d0 "/", len=8) at src/libutil/serialise.hh:352 #25 0x000075500745d305 in nix::TeeSource::read (this=0x754feb9a9948, data=0x0, len=2147483648) at src/libutil/serialise.hh:238 #26 0x000075500788794f in nix::Source::operator() (data=0x754feb9a96d0 "/", len=8, this=<optimized out>) at src/libutil/serialise.cc:111 #27 nix::readNum<unsigned long> (source=...) at src/libutil/serialise.cc:31 #28 nix::readString[abi:cxx11](nix::Source&, unsigned long) (source=..., max=max@entry=13) at src/libutil/serialise.cc:267 #29 0x00007550077e3e0c in nix::nar::parse (source=...) at src/libutil/archive.cc:320 #30 0x00007550077e4762 in std::__n4861::coroutine_handle<void>::resume (this=0x754fd8002b50) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #31 nix::_generator::GeneratorBase<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next (this=0x754fd8002b50) at src/libutil/generator.hh:150 #32 nix::Generator<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, void>::next (this=0x754fd8002b50) at src/libutil/generator.hh:266 #33 nix::restore (sink=..., nar=...) at src/libutil/archive.cc:336 #34 0x00005aa91d779fa7 in std::__n4861::coroutine_handle<void>::resume (this=0x754feb9a9870) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #35 nix::_generator::GeneratorBase<std::span<char const, 18446744073709551615ul> >::next (this=0x754feb9a9868) at src/libutil/generator.hh:150 #36 0x00007550077df26b in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform>::next (this=0x754feb9a9868) at src/libutil/generator.hh:266 #37 nix::parseDump (sink=..., source=...) at src/libutil/archive.cc:382 #38 0x00007550077df359 in nix::restorePath (path=..., source=...) at src/libutil/archive.cc:508 #39 0x000075500751fe21 in nix::LocalStore::addToStore (this=0x5aa93ba6a1b0, info=..., source=..., repair=nix::NoRepair, checkSigs=<optimized out>) at src/libstore/local-store.cc:1263 #40 0x00007550075b6278 in nix::copyStorePath (srcStore=..., dstStore=..., storePath=..., repair=nix::NoRepair, checkSigs=nix::CheckSigs) at src/libstore/store-api.cc:1116 #41 0x0000755007655126 in nix::PathSubstitutionGoal::tryToRun()::$_0::operator()() const (this=<optimized out>) at src/libstore/build/substitution-goal.cc:222 #42 std::__invoke_impl<void, nix::PathSubstitutionGoal::tryToRun()::$_0>(std::__invoke_other, nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61 #43 std::__invoke<nix::PathSubstitutionGoal::tryToRun()::$_0>(nix::PathSubstitutionGoal::tryToRun()::$_0&&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:96 #44 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::_M_invoke<0ul> (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:292 #45 std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >::operator() (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_thread.h:299 #46 std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::operator() (this=0x754feb9a9ff0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1432 #47 std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61 #48 std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>&> (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:114 #49 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void> >::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290 #50 0x00007550074c1d8a in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>::operator() (this=0x0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591 #51 std::__future_base::_State_baseV2::_M_do_set (this=0x5aa93c23ac90, __f=0x0, __did_set=0x80000000) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:589 #52 0x000075500696ee2f in __pthread_once_slow () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #53 0x00007550074c1ca0 in __gthread_once (__once=0x5aa93c23aca8, __func=0x0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700 #54 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()>*, bool*> (__once=..., __f=@0x754feb9a9f70: (void (std::__future_base::_State_baseV2::*)(class std::__future_base::_State_baseV2 * const, class std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x7550074c1d60 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __args=@0x754feb9a9f58: 0x754feb9a9f87, __args=@0x754feb9a9f58: 0x754feb9a9f87, __args=@0x754feb9a9f58: 0x754feb9a9f87) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/mutex:907 #55 std::__future_base::_State_baseV2::_M_set_result (this=0x5aa93c23ac90, __res=..., __ignore_failure=false) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:428 #56 0x0000755007654bd2 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::_M_run (this=0x754fd8005710) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1774 #57 0x0000755006be16d3 in execute_native_thread_routine () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6 #58 0x0000755006969d02 in start_thread () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #59 0x00007550069e93ac in __clone3 () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 ``` Main thread: ``` Thread 1 (Thread 0x755004b5e5c0 (LWP 3541973) "nix-daemon"): #0 0x000075500696637e in __futex_abstimed_wait_common () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #1 0x000075500696b803 in __pthread_clockjoin_ex () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #2 0x0000755006be1747 in std::thread::join() () from target:/nix/store/97f3gw9vpyxvwjv2i673isvg92q65mwn-gcc-13.3.0-lib/lib/libstdc++.so.6 #3 0x000075500696ee2f in __pthread_once_slow () from target:/nix/store/pacbfvpzqz2mksby36awvbcn051zcji3-glibc-2.40-36/lib/libc.so.6 #4 0x0000755007609f71 in __gthread_once (__once=0x5aa93d245248, __func=0x109) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:700 #5 std::call_once<void (std::thread::*)(), std::thread*> (__once=..., __f=@0x7ffdfcf10858: (void (std::thread::*)(class std::thread * const)) 0x755006be1730 <std::thread::join()>, __args=@0x7ffdfcf10850: 0x5aa93d245240) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/mutex:907 #6 std::__future_base::_Async_state_commonV2::_M_join (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1737 #7 std::__future_base::_Async_state_commonV2::_M_complete_async (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:1735 #8 0x00007550074d1b3a in std::__future_base::_State_baseV2::wait (this=0x5aa93d245220) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:350 #9 std::__basic_future<void>::_M_get_result (this=this@entry=0x5aa93d29c9c0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:741 #10 0x00007550074d1382 in std::future<void>::get (this=0x754fd27fc990, this@entry=0x5aa93d29c9c0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/future:913 #11 0x000075500764ed65 in nix::PathSubstitutionGoal::cleanup (this=0x5aa93d29c8d0) at src/libstore/build/substitution-goal.cc:294 #12 nix::PathSubstitutionGoal::~PathSubstitutionGoal (this=0x5aa93d29c8d0) at src/libstore/build/substitution-goal.cc:32 #13 0x000075500764f29c in nix::PathSubstitutionGoal::~PathSubstitutionGoal (this=0x754fd27fc990) at src/libstore/build/substitution-goal.cc:31 #14 0x000075500766c0b0 in std::default_delete<nix::PathSubstitutionGoal>::operator() (__ptr=0x754fd27fc990, this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/unique_ptr.h:99 #15 std::_Sp_counted_deleter<nix::PathSubstitutionGoal*, std::default_delete<nix::PathSubstitutionGoal>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:527 #16 0x0000755007669a0a in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5aa93d27f190) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:346 #17 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x5aa93d29cf08) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:1071 #18 std::__shared_ptr<nix::PathSubstitutionGoal, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:1524 #19 nix::Worker::CachedGoal<nix::PathSubstitutionGoal>::~CachedGoal (this=<optimized out>) at src/libstore/build/worker.hh:128 #20 std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >::~pair (this=this@entry=0x5aa93d29cee0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_pair.h:187 #21 0x0000755007669989 in std::destroy_at<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > (__location=0x754fd27fc990, __location@entry=0x5aa93bebe8a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_construct.h:88 #22 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > > >::destroy<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > (__p=0x754fd27fc990, __p@entry=0x5aa93bebe8a0, __a=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/alloc_traits.h:560 #23 std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_destroy_node (this=0x5aa93bebe8a0, __p=0x5aa93d29cec0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:625 #24 std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_drop_node (this=0x5aa93bebe8a0, __p=0x5aa93d29cec0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:633 #25 std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93d29cec0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1938 #26 0x000075500766997c in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93c377c60) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1936 #27 0x000075500766997c in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93c2b73a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1936 #28 0x000075500766997c in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase (this=this@entry=0x5aa93bebe8a0, __x=0x5aa93c29e450) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1936 #29 0x0000755007659262 in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::clear (this=0x5aa93bebe8a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_tree.h:1255 #30 std::map<nix::StorePath, nix::Worker::CachedGoal<nix::PathSubstitutionGoal>, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::clear (this=0x5aa93bebe8a0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/stl_map.h:1184 #31 nix::Worker::~Worker (this=0x5aa93bebe860) at src/libstore/build/worker.cc:53 #32 0x00007550076101b0 in nix::processGoals<nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0>(nix::Store&, nix::Store&, kj::AsyncIoContext&, nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0&&) (store=..., evalStore=..., aio=..., mkGoals=...) at src/libstore/build/worker.hh:302 #33 0x00007550058bdb92 in std::__n4861::coroutine_handle<void>::resume (this=0x149) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/coroutine:135 #34 kj::_::CoroutineBase::fire (this=0xf9) at /build/source/c++/src/kj/async.c++:3031 #35 0x00007550058bdb92 in non-virtual thunk to kj::_::CoroutineBase::fire() () from target:/nix/store/0vpa01040n1wr9dw13ffy7svgl13hng2-capnproto-1.0.2/lib/libkj-async.so.1.0.2 #36 0x00007550058beb52 in kj::EventLoop::turn (this=0x754fb4014a50) at /build/source/c++/src/kj/async.c++:1806 #37 kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const (this=this@entry=0x7ffdfcf10b30) at /build/source/c++/src/kj/async.c++:1970 #38 0x00007550058b7d58 in kj::WaitScope::runOnStackPool<kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2>(kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2&&) (this=0x754fb4014aa8, func=...) at /build/source/c++/src/kj/async.h:1368 #39 kj::_::waitImpl (node=..., result=..., waitScope=..., location=...) at /build/source/c++/src/kj/async.c++:1967 #40 0x0000755007612012 in kj::Promise<boost::outcome_v2::basic_result<nix::Worker::Results, std::__exception_ptr::exception_ptr, boost::outcome_v2::policy::exception_ptr_rethrow<nix::Worker::Results, std::__exception_ptr::exception_ptr, void> > >::wait (this=0xfffffffffffffe00, this@entry=0x7ffdfcf10e98, waitScope=..., location=...) at /nix/store/0vpa01040n1wr9dw13ffy7svgl13hng2-capnproto-1.0.2/include/kj/async-inl.h:1357 #41 0x000075500760b6de in nix::Store::buildPathsWithResults (this=0x5aa93ba6acc0, reqs=..., buildMode=nix::bmNormal, evalStore=...) at src/libstore/build/entry-points.cc:57 #42 0x0000755007483dfc in nix::daemon::performOp (logger=logger@entry=0x5aa93ba69270, store=..., trusted=<optimized out>, recursive=<optimized out>, clientVersion=291, from=..., to=..., op=nix::WorkerProto::Op::BuildPathsWithResults) at src/libstore/daemon.cc:572 #43 0x00007550074806b5 in nix::daemon::processConnection (store=..., from=..., to=..., trusted=<optimized out>, recursive=<optimized out>) at src/libstore/daemon.cc:1076 #44 0x00005aa91d79df95 in nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0::operator()() const (this=0x5aa93ba6b960) at src/nix/daemon.cc:343 #45 std::__invoke_impl<void, nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&>(std::__invoke_other, nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61 #46 std::__invoke_r<void, nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&>(nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:111 #47 std::_Function_handler<void(), nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0>::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290 #48 0x000075500786f12a in std::function<void()>::operator() (this=0x754fd27fc990) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591 #49 nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0::operator()() const (this=0x7ffdfcf11a20) at src/libutil/processes.cc:194 #50 std::__invoke_impl<void, nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&>(std::__invoke_other, nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&) (__f=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:61 #51 std::__invoke_r<void, nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&>(nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0&) (__fn=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:111 #52 std::_Function_handler<void(), nix::startProcess(std::function<void()>, nix::ProcessOptions const&)::$_0>::_M_invoke (__functor=...) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:290 #53 0x000075500786de4e in nix::doFork (fun=...) at src/libutil/processes.cc:171 #54 nix::startProcess (fun=..., options=...) at src/libutil/processes.cc:225 #55 0x00005aa91d79c1fb in nix::daemonLoop (forceTrustClientOpt=...) at src/nix/daemon.cc:320 #56 nix::runDaemon (stdio=<optimized out>, forceTrustClientOpt=...) at src/nix/daemon.cc:436 #57 0x00005aa91d79b1f6 in nix::main_nix_daemon (argc=<optimized out>, argv=<optimized out>) at src/nix/daemon.cc:467 #58 0x00005aa91d81ad25 in std::function<void(int, char**)>::operator() (this=0x7ffdfcf12200, __args=0x7ffdfcf12f28, __args=0x7ffdfcf12f28) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591 #59 nix::mainWrapped (argc=2, argv=0x7ffdfcf12f28) at src/nix/main.cc:389 #60 0x0000755006eca421 in std::function<void()>::operator() (this=0x7ffdfcf12da0) at /nix/store/xzfmarrq8x8s4ivpya24rrndqsq2ndiz-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591 #61 nix::handleExceptions (programName=..., fun=...) at src/libmain/shared.cc:325 #62 0x00005aa91d81f08b in main (argc=2, argv=0x7ffdfcf12f28) at src/nix/main.cc:547 ``` Notably, there is *no* thread with `workerThreadEntry` in the call stack. This is almost certainly related. More detailed execution state description: The main thread is waiting for `PathSubstitutionGoal::thr.get()`: ```c++ void PathSubstitutionGoal::cleanup() { try { if (thr.valid()) { // FIXME: signal worker thread to quit. thr.get(); } } catch (...) { ignoreExceptionInDestructor(); } } ``` where `thr` is one of the PathSubstitutionGoal threads mentioned above. It's waiting on joining that thread. Those threads are either "puppy" or "kitty". Puppy threads are waiting on `transfer->downloadEvent`: ```c++ bool awaitData() { return withRetries([&] { /* Grab data if available, otherwise wait for the download thread to wake us up. */ while (buffered.empty()) { auto state(transfer->downloadState.lock()); if (!state->data.empty()) { chunk = std::move(state->data); buffered = chunk; totalReceived += chunk.size(); parent.unpause(transfer); } else if (state->exc) { std::rethrow_exception(state->exc); } else if (state->done) { return false; } else { parent.unpause(transfer); state.wait(transfer->downloadEvent); // <--------- HERE } } return true; }); ``` Kitty threads are waiting on `transfer->metadataPromise.get_future().get()`: ```c++ FileTransferResult startTransfer(const std::string & uri, curl_off_t offset = 0) { attempt += 1; auto uploadData = data ? std::optional(std::string_view(*data)) : std::nullopt; transfer = std::make_shared<TransferItem>(uri, headers, parentAct, uploadData, noBody, offset); parent.enqueueItem(transfer); return transfer->metadataPromise.get_future().get(); // <------------ HERE } ``` In both instances, these are states that are progressed by the download thread. For puppy threads, it would be progressed if the download got the data callback called for more data. For kitty threads, it would be progressed by receiving headers (or finishing the request with headers?). The main thread is stuck because it is waiting on either a puppy or kitty thread. So, it appears that the root cause is that the download thread can exit without fulfilling all its expectations or notifying them that it cannot complete them. ## How did this happen? Well, and this is the question. I have no idea the correct order of operations to be able to cleanly tear down the download thread while rejecting all its expectations. Effectively the waiting on synchronization primitives needs to have that channel multiplexed with a possible error that probably gets rethrown. I also don't know why this used to work. At one point, the answer to this is that the daemon crashed if the client hangs up due to buggy signal handling. Now that the signal handling is a bit less buggy, it seems like the problem is that the async runtime is being bypassed while waiting for a future on the main thread, and so it cannot switch from the task and is stuck in blocking code. The puppy/kitty worker threads getting abandoned appears to be intentional based on the TODO above, and this is not itself a big problem. The issue is that the main thread is getting stuck waiting on one of them in a way that does not allow the async runtime to do other things and thus not being able to tear the whole thing down properly.

jade added the

bug

label 2024-11-18 23:57:33 +00:00

pennae was assigned by jade 2024-11-19 00:39:40 +00:00

jade commented 2024-11-19 00:39:54 +00:00

Author

Owner

Copy link

(assigning pennae per discussions since they are fixing it at the moment)

(assigning pennae per discussions since they are fixing it at the moment)

lix-bot commented 2024-11-19 02:01:21 +00:00

Member

Copy link

This issue was mentioned on Gerrit on the following CLs:

• commit message in cl/2209 ("libstore: abort all curl transfers on interrupt")

<!-- GERRIT_LINKBOT: {"cls": [{"backlink": "https://gerrit.lix.systems/c/lix/+/2209", "number": 2209, "kind": "commit message"}], "cl_meta": {"2209": {"change_title": "libstore: abort all curl transfers on interrupt"}}} --> This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/2209](https://gerrit.lix.systems/c/lix/+/2209) ("libstore: abort all curl transfers on interrupt")

lix-project referenced this issue from a commit 2024-11-19 05:04:15 +00:00

libstore: abort all curl transfers on interrupt

lix-project closed this issue 2024-11-19 05:04:15 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

release-2.91

sb/raito/phantom-referrers-gc

repl-overlay-example

reuse-compliance

release-2.90

git-series/activity-cleanup

sb/tom-hubrecht/primops.cc-v2

sb/bacchanalia/cache-strlen%topic=substr-perf

sb/arcuru/commit-msg

sb/qyriad/beta-1

sb/qyriad/2.90-beta.0

rbt/pre-commit-update

sb/rbt/test-branch

sb/rbt/justfile

sb/rbt/repl-overlays

sb/fuck/what

sb/rbt/pre-commit

sb/rbt/makeflags

sb/pennae/io-rewrite

sb/qyriad/maint-meson

sb/qyriad/maint/meson

sb/puck/meson-c-api

sb/pennae/parser-rewrite

sb/jade/burn-it-all

dependabot/github_actions/zeebe-io/backport-action-2.4.1

dependabot/github_actions/cachix/install-nix-action-25

dependabot/github_actions/actions/labeler-5

dependabot/github_actions/cachix/cachix-action-14

2.91.1

2.91.0

2.90.0

2.90.0-rc1

2.90-beta.1

2.90-beta.0

Labels

Clear labels   

Area/build-packaging

regarding the building or packaging of Lix itself

  

Area/cli

The command line interface of Lix

  

Area/evaluator

libexpr and anything else related to the evaluation of Nix expressions

  

Area/fetching

Fetching sources or other materials from the World Wide Web in Lix

  

Area/flakes

  

Area/language

language design, features, extensions, etc

  

Area/profiles

profiles: nix-env, nix profile, and installer flavors all alike

  

Area/protocol

The protocol, used for communicating between the client and the daemon

  

Area/releng

Release engineering

  

Area/remote-builds

remote builds

  

Area/repl

nix repl

  

Area/store

libstore and anything else relating to the Nix store

  

bug

it's a bug

  

crash 💥

Lix crashes (possibly bad error handling)

  

Cross Compilation

  

devx

development experience issues

  

docs

requires or is about documentation changes

  

Downstream Dependents

regarding an external tool's usage with Lix (e.g. nix-eval-jobs)

  

E/easy

bug is relatively simple and could simply be fixed with some minor effort

  

E/hard

this issue should probably be solved with close involvement of an expert

  

E/help wanted

maintainers may not get to this soon, but we want it fixed

  

E/reproducible

This bug is reproducible and obvious.

  

E/requires rearchitecture

this bug cannot be fixed properly without fixing a known architectural flaw

  

imported

Issue imported from upstream nix (please do not remove this label, it will cause dupes if import is rerun)

  

Needs Langver

We can't get to this until we implement language versioning

  

OS/Linux

Affects specifically Lix on Linux

  

OS/macOS

Affects specifically Lix on macOS

  

performance

make lix go faster

  

regression

This is a regression since some previous version of Lix

  

release-blocker

Blocks public release

  

RFD

Request for discussion

  

stability

Fixing this makes the software more stable

  

Status

blocked

Blocked on something

  

Status

invalid

not our bug, duplicate, similar

  

Status

postponed

closed but should maybe look at later (generally with S-wontfix)

  

Status

wontfix

We chose not to change this (see issue comments for more detail).

  

testing

Tests that should be added

  

testing/flakey

Tests that are unreliable/flakey (not to be confused with flakes that test)

  

ux

makes the user experience better :)

No labels

Area/build-packaging

Area/cli

Area/evaluator

Area/fetching

Area/flakes

Area/language

Area/profiles

Area/protocol

Area/releng

Area/remote-builds

Area/repl

Area/store

bug

crash 💥

Cross Compilation

devx

docs

Downstream Dependents

E/easy

E/hard

E/help wanted

E/reproducible

E/requires rearchitecture

imported

Needs Langver

OS/Linux

OS/macOS

performance

regression

release-blocker

RFD

stability

Status

blocked

Status

invalid

Status

postponed

Status

wontfix

testing

testing/flakey

ux

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

pennae

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#577

No description provided.