Deadlocked path due to deadlocked download event #613

New issue

Closed

opened 2025-01-04 18:04:38 +00:00 by raito · 1 comment

raito commented

2025-01-04 18:04:38 +00:00

Owner

Describe the bug

On Dec 2024 Lix HEADs, it is possible to get into a state where a path is deadlocked (i.e. a .lock file is held by a nix-daemon) and if you capture all stack traces, you can see that there's a nix-daemon thread invoking a PathSubstitutionGoal destructor, waiting for a worker thread to stop and another nix-daemon thread waiting on a condition_variable to be woken up to get data from a FileTransfer.

The program is effectively deadlocked at this point as there's no thread to wake up the FileTransfer and it's likely that the destructor is waiting for the FileTransfer thread to terminate as well.

Steps To Reproduce

There's no easy recipe to reproduce, it happens randomly probably when lock contention is high.

Expected behavior

No deadlocks and graceful shutdowns.

`nix --version` output

Lix 2024-12-11 @ 92ed9fe.

Additional context

Here's a backtrace:

(gdb) thread apply all bt

Thread 3 (Thread 0x7fc0983716c0 (LWP 348575) "signal handler"):
#0  0x00007fc099e4180a in sigtimedwait () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#1  0x00007fc099e40e3c in sigwait () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#2  0x00007fc09b19c748 in nix::signalHandlerThread(__sigset_t) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#3  0x00007fc09b19cd04 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(__sigset_t), __sigset_t> > >::_M_run() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#4  0x00007fc09a2ed0a4 in execute_native_thread_routine () from target:/nix/store/4jnr44snx7hi1z3qsahlnpwmng81lflr-gcc-14-20241116-lib/lib/libstdc++.so.6
#5  0x00007fc099e96a72 in start_thread () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#6  0x00007fc099f1a23c in __clone3 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6

Thread 2 (Thread 0x7fc045ffb6c0 (LWP 360686) "nix-daemon"):
#0  0x00007fc099e92fde in __futex_abstimed_wait_common () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#1  0x00007fc099e95c50 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#2  0x00007fc09ada5021 in nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}::operator()() const () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#3  0x00007fc09ada4d4f in decltype ({parm#1}()) nix::curlFileTransfer::TransferSource::withRetries<nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}>(nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#4  0x00007fc09ad9dd9b in nix::curlFileTransfer::TransferSource::read(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#5  0x00007fc09b1a59d2 in nix::callback_read(archive*, void*, void const**) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#6  0x00007fc09a4ddc56 in __archive_read_filter_ahead () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13
#7  0x00007fc09a4e8917 in xz_filter_read () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13
#8  0x00007fc09a4ddc56 in __archive_read_filter_ahead () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13
#9  0x00007fc09a50e260 in archive_read_format_raw_read_data () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13
#10 0x00007fc09a4dd4c4 in archive_read_data () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13
#11 0x00007fc09b1160d4 in nix::ArchiveDecompressionSource::read(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#12 0x00007fc09ad27a70 in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform> nix::BinaryCacheStore::narFromPath(nix::StorePath const&)::$_0::operator()<nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats>(nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#13 0x00007fc09ae988f2 in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform> nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag)::$_0::operator()<nix::Activity, nix::ref<nix::ValidPathInfo const>, nix::Store, nix::StorePath const>(nix::Activity&, nix::ref<nix::ValidPathInfo const>&, nix::Store&, nix::StorePath const&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#14 0x00005625ee3c148f in nix::GeneratorSource::read(char*, unsigned long) ()
#15 0x00007fc09ad2dfe5 in nix::TeeSource::read(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#16 0x00007fc09b1981ff in nix::Source::operator()(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#17 0x00007fc09b0eeb83 in nix::nar::parseObject(nix::Source&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::$_0::operator()(nix::Source&, unsigned long&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#18 0x00007fc09b0f1f12 in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform> nix::restore(nix::NARParseVisitor&, nix::Generator<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, void>)::$_3::operator()(nix::nar::File) const::{lambda(auto:1, auto:2&)#1}::operator()<nix::nar::File, nix::NARParseVisitor>(nix::nar::File, nix::NARParseVisitor&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#19 0x00007fc09b0ec87c in nix::parseDump(nix::NARParseVisitor&, nix::Source&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#20 0x00007fc09b0ec989 in nix::restorePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Source&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#21 0x00007fc09adf75ae in nix::LocalStore::addToStore(nix::ValidPathInfo const&, nix::Source&, nix::RepairFlag, nix::CheckSigsFlag) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#22 0x00007fc09ae9096f in nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#23 0x00007fc09af33187 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void> >::_M_invoke(std::_Any_data const&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#24 0x00007fc09ad9470a in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#25 0x00007fc099e9bd57 in __pthread_once_slow.isra.0 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#26 0x00007fc099e9bdd1 in pthread_once@GLIBC_2.2.5 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#27 0x00007fc09ad94620 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#28 0x00007fc09af32be2 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::_M_run() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#29 0x00007fc09a2ed0a4 in execute_native_thread_routine () from target:/nix/store/4jnr44snx7hi1z3qsahlnpwmng81lflr-gcc-14-20241116-lib/lib/libstdc++.so.6
#30 0x00007fc099e96a72 in start_thread () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#31 0x00007fc099f1a23c in __clone3 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6

Thread 1 (Thread 0x7fc09837a5c0 (LWP 348574) "nix-daemon"):
#0  0x00007fc099e92fde in __futex_abstimed_wait_common () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#1  0x00007fc099e98653 in __pthread_clockjoin_ex () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#2  0x00007fc09a2ed11b in std::thread::join() () from target:/nix/store/4jnr44snx7hi1z3qsahlnpwmng81lflr-gcc-14-20241116-lib/lib/libstdc++.so.6
#3  0x00007fc099e9bd57 in __pthread_once_slow.isra.0 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#4  0x00007fc099e9bdd1 in pthread_once@GLIBC_2.2.5 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6
#5  0x00007fc09aee61d1 in std::__future_base::_Async_state_commonV2::_M_complete_async() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#6  0x00007fc09ada4b4a in std::__basic_future<void>::_M_get_result() const () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#7  0x00007fc09ada44a2 in std::future<void>::get() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#8  0x00007fc09af2cad5 in nix::PathSubstitutionGoal::~PathSubstitutionGoal() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#9  0x00007fc09af2d00c in nix::PathSubstitutionGoal::~PathSubstitutionGoal() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#10 0x00007fc09af4a590 in std::_Sp_counted_deleter<nix::PathSubstitutionGoal*, std::default_delete<nix::PathSubstitutionGoal>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#11 0x00007fc09af47dea in std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >::~pair() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#12 0x00007fc09af47d59 in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase(std::_Rb_tree_node<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >*) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#13 0x00007fc09af37462 in nix::Worker::~Worker() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#14 0x00007fc09aeec555 in kj::Promise<boost::outcome_v2::basic_result<nix::Worker::Results, std::__exception_ptr::exception_ptr, boost::outcome_v2::policy::exception_ptr_rethrow<nix::Worker::Results, std::__exception_ptr::exception_ptr, void> > > nix::processGoals<nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0>(nix::Store&, nix::Store&, kj::AsyncIoContext&, nix::Store::buildPathsWithResults(--Type <RET> for more, q to quit, c to continue without paging--c
std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0&&) [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#15 0x00007fc09934dc82 in non-virtual thunk to kj::_::CoroutineBase::fire() () from target:/nix/store/kci1bwby2gfrpz17s49rxj8aqgw39qly-capnproto-1.0.2/lib/libkj-async.so.1.0.2
#16 0x00007fc09934ecb2 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const () from target:/nix/store/kci1bwby2gfrpz17s49rxj8aqgw39qly-capnproto-1.0.2/lib/libkj-async.so.1.0.2
#17 0x00007fc099347d68 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) () from target:/nix/store/kci1bwby2gfrpz17s49rxj8aqgw39qly-capnproto-1.0.2/lib/libkj-async.so.1.0.2
#18 0x00007fc09aeee3b2 in kj::Promise<boost::outcome_v2::basic_result<nix::Worker::Results, std::__exception_ptr::exception_ptr, boost::outcome_v2::policy::exception_ptr_rethrow<nix::Worker::Results, std::__exception_ptr::exception_ptr, void> > >::wait(kj::WaitScope&, kj::SourceLocation) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#19 0x00007fc09aee7988 in nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#20 0x00007fc09ad5466b in nix::daemon::performOp(nix::daemon::TunnelLogger*, nix::ref<nix::Store>, nix::TrustedFlag, nix::daemon::RecursiveFlag, unsigned int, nix::Source&, nix::BufferedSink&, nix::WorkerProto::Op) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#21 0x00007fc09ad50d07 in nix::daemon::processConnection(nix::ref<nix::Store>, nix::FdSource&, nix::FdSink&, nix::TrustedFlag, nix::daemon::RecursiveFlag) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so
#22 0x00005625ee345cd4 in std::_Function_handler<void (), nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0>::_M_invoke(std::_Any_data const&) ()
#23 0x00007fc09b17f35a in std::_Function_handler<void (), nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0>::_M_invoke(std::_Any_data const&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#24 0x00007fc09b17e05e in nix::startProcess(std::function<void ()>, nix::ProcessOptions const&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so
#25 0x00005625ee3443d6 in nix::runDaemon(bool, std::optional<nix::TrustedFlag>) ()
#26 0x00005625ee342b83 in nix::main_nix_daemon(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) ()
#27 0x00005625ee34aa02 in int std::__invoke_impl<int, int (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >(std::__invoke_other, int (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&) ()
#28 0x00005625ee3c8b32 in nix::mainWrapped(int, char**) ()
#29 0x00007fc09abeca9b in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixmain.so
#30 0x00005625ee3cc84b in main ()

## Describe the bug On Dec 2024 Lix HEADs, it is possible to get into a state where a path is deadlocked (i.e. a .lock file is held by a nix-daemon) and if you capture all stack traces, you can see that there's a nix-daemon thread invoking a PathSubstitutionGoal destructor, waiting for a worker thread to stop and another nix-daemon thread waiting on a condition_variable to be woken up to get data from a FileTransfer. The program is effectively deadlocked at this point as there's no thread to wake up the FileTransfer and it's likely that the destructor is waiting for the FileTransfer thread to terminate as well. ## Steps To Reproduce There's no easy recipe to reproduce, it happens randomly probably when lock contention is high. ## Expected behavior No deadlocks and graceful shutdowns. ## `nix --version` output Lix 2024-12-11 @ 92ed9fe. ## Additional context Here's a backtrace: ``` (gdb) thread apply all bt Thread 3 (Thread 0x7fc0983716c0 (LWP 348575) "signal handler"): #0 0x00007fc099e4180a in sigtimedwait () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #1 0x00007fc099e40e3c in sigwait () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #2 0x00007fc09b19c748 in nix::signalHandlerThread(__sigset_t) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #3 0x00007fc09b19cd04 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(__sigset_t), __sigset_t> > >::_M_run() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #4 0x00007fc09a2ed0a4 in execute_native_thread_routine () from target:/nix/store/4jnr44snx7hi1z3qsahlnpwmng81lflr-gcc-14-20241116-lib/lib/libstdc++.so.6 #5 0x00007fc099e96a72 in start_thread () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #6 0x00007fc099f1a23c in __clone3 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 Thread 2 (Thread 0x7fc045ffb6c0 (LWP 360686) "nix-daemon"): #0 0x00007fc099e92fde in __futex_abstimed_wait_common () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #1 0x00007fc099e95c50 in pthread_cond_wait@@GLIBC_2.3.2 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #2 0x00007fc09ada5021 in nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}::operator()() const () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #3 0x00007fc09ada4d4f in decltype ({parm#1}()) nix::curlFileTransfer::TransferSource::withRetries<nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}>(nix::curlFileTransfer::TransferSource::awaitData()::{lambda()#1}) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #4 0x00007fc09ad9dd9b in nix::curlFileTransfer::TransferSource::read(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #5 0x00007fc09b1a59d2 in nix::callback_read(archive*, void*, void const**) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #6 0x00007fc09a4ddc56 in __archive_read_filter_ahead () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13 #7 0x00007fc09a4e8917 in xz_filter_read () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13 #8 0x00007fc09a4ddc56 in __archive_read_filter_ahead () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13 #9 0x00007fc09a50e260 in archive_read_format_raw_read_data () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13 #10 0x00007fc09a4dd4c4 in archive_read_data () from target:/nix/store/nas3rdxgi28misnbc8sxlcpr1gz70p6q-libarchive-3.7.7-lib/lib/libarchive.so.13 #11 0x00007fc09b1160d4 in nix::ArchiveDecompressionSource::read(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #12 0x00007fc09ad27a70 in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform> nix::BinaryCacheStore::narFromPath(nix::StorePath const&)::$_0::operator()<nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats>(nix::ref<nix::NarInfo const>, nix::box_ptr<nix::Source>, nix::Store::Stats&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #13 0x00007fc09ae988f2 in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform> nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag)::$_0::operator()<nix::Activity, nix::ref<nix::ValidPathInfo const>, nix::Store, nix::StorePath const>(nix::Activity&, nix::ref<nix::ValidPathInfo const>&, nix::Store&, nix::StorePath const&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #14 0x00005625ee3c148f in nix::GeneratorSource::read(char*, unsigned long) () #15 0x00007fc09ad2dfe5 in nix::TeeSource::read(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #16 0x00007fc09b1981ff in nix::Source::operator()(char*, unsigned long) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #17 0x00007fc09b0eeb83 in nix::nar::parseObject(nix::Source&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::$_0::operator()(nix::Source&, unsigned long&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #18 0x00007fc09b0f1f12 in nix::Generator<std::span<char const, 18446744073709551615ul>, nix::SerializingTransform> nix::restore(nix::NARParseVisitor&, nix::Generator<std::variant<nix::nar::MetadataString, nix::nar::MetadataRaw, nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, void>)::$_3::operator()(nix::nar::File) const::{lambda(auto:1, auto:2&)#1}::operator()<nix::nar::File, nix::NARParseVisitor>(nix::nar::File, nix::NARParseVisitor&) const [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #19 0x00007fc09b0ec87c in nix::parseDump(nix::NARParseVisitor&, nix::Source&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #20 0x00007fc09b0ec989 in nix::restorePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Source&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #21 0x00007fc09adf75ae in nix::LocalStore::addToStore(nix::ValidPathInfo const&, nix::Source&, nix::RepairFlag, nix::CheckSigsFlag) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #22 0x00007fc09ae9096f in nix::copyStorePath(nix::Store&, nix::Store&, nix::StorePath const&, nix::RepairFlag, nix::CheckSigsFlag) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #23 0x00007fc09af33187 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void> >::_M_invoke(std::_Any_data const&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #24 0x00007fc09ad9470a in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #25 0x00007fc099e9bd57 in __pthread_once_slow.isra.0 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #26 0x00007fc099e9bdd1 in pthread_once@GLIBC_2.2.5 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #27 0x00007fc09ad94620 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #28 0x00007fc09af32be2 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<nix::PathSubstitutionGoal::tryToRun()::$_0> >, void>::_M_run() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #29 0x00007fc09a2ed0a4 in execute_native_thread_routine () from target:/nix/store/4jnr44snx7hi1z3qsahlnpwmng81lflr-gcc-14-20241116-lib/lib/libstdc++.so.6 #30 0x00007fc099e96a72 in start_thread () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #31 0x00007fc099f1a23c in __clone3 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 Thread 1 (Thread 0x7fc09837a5c0 (LWP 348574) "nix-daemon"): #0 0x00007fc099e92fde in __futex_abstimed_wait_common () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #1 0x00007fc099e98653 in __pthread_clockjoin_ex () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #2 0x00007fc09a2ed11b in std::thread::join() () from target:/nix/store/4jnr44snx7hi1z3qsahlnpwmng81lflr-gcc-14-20241116-lib/lib/libstdc++.so.6 #3 0x00007fc099e9bd57 in __pthread_once_slow.isra.0 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #4 0x00007fc099e9bdd1 in pthread_once@GLIBC_2.2.5 () from target:/nix/store/c58w17bgzg5nw0c03pl8223qdvrv31i5-glibc-2.40-36/lib/libc.so.6 #5 0x00007fc09aee61d1 in std::__future_base::_Async_state_commonV2::_M_complete_async() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #6 0x00007fc09ada4b4a in std::__basic_future<void>::_M_get_result() const () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #7 0x00007fc09ada44a2 in std::future<void>::get() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #8 0x00007fc09af2cad5 in nix::PathSubstitutionGoal::~PathSubstitutionGoal() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #9 0x00007fc09af2d00c in nix::PathSubstitutionGoal::~PathSubstitutionGoal() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #10 0x00007fc09af4a590 in std::_Sp_counted_deleter<nix::PathSubstitutionGoal*, std::default_delete<nix::PathSubstitutionGoal>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #11 0x00007fc09af47dea in std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >::~pair() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #12 0x00007fc09af47d59 in std::_Rb_tree<nix::StorePath, std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> >, std::_Select1st<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > > >::_M_erase(std::_Rb_tree_node<std::pair<nix::StorePath const, nix::Worker::CachedGoal<nix::PathSubstitutionGoal> > >*) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #13 0x00007fc09af37462 in nix::Worker::~Worker() () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #14 0x00007fc09aeec555 in kj::Promise<boost::outcome_v2::basic_result<nix::Worker::Results, std::__exception_ptr::exception_ptr, boost::outcome_v2::policy::exception_ptr_rethrow<nix::Worker::Results, std::__exception_ptr::exception_ptr, void> > > nix::processGoals<nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0>(nix::Store&, nix::Store&, kj::AsyncIoContext&, nix::Store::buildPathsWithResults(--Type <RET> for more, q to quit, c to continue without paging--c std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>)::$_0&&) [clone .resume] () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #15 0x00007fc09934dc82 in non-virtual thunk to kj::_::CoroutineBase::fire() () from target:/nix/store/kci1bwby2gfrpz17s49rxj8aqgw39qly-capnproto-1.0.2/lib/libkj-async.so.1.0.2 #16 0x00007fc09934ecb2 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation)::$_2::operator()() const () from target:/nix/store/kci1bwby2gfrpz17s49rxj8aqgw39qly-capnproto-1.0.2/lib/libkj-async.so.1.0.2 #17 0x00007fc099347d68 in kj::_::waitImpl(kj::Own<kj::_::PromiseNode, kj::_::PromiseDisposer>&&, kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) () from target:/nix/store/kci1bwby2gfrpz17s49rxj8aqgw39qly-capnproto-1.0.2/lib/libkj-async.so.1.0.2 #18 0x00007fc09aeee3b2 in kj::Promise<boost::outcome_v2::basic_result<nix::Worker::Results, std::__exception_ptr::exception_ptr, boost::outcome_v2::policy::exception_ptr_rethrow<nix::Worker::Results, std::__exception_ptr::exception_ptr, void> > >::wait(kj::WaitScope&, kj::SourceLocation) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #19 0x00007fc09aee7988 in nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath> > const&, nix::BuildMode, std::shared_ptr<nix::Store>) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #20 0x00007fc09ad5466b in nix::daemon::performOp(nix::daemon::TunnelLogger*, nix::ref<nix::Store>, nix::TrustedFlag, nix::daemon::RecursiveFlag, unsigned int, nix::Source&, nix::BufferedSink&, nix::WorkerProto::Op) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #21 0x00007fc09ad50d07 in nix::daemon::processConnection(nix::ref<nix::Store>, nix::FdSource&, nix::FdSink&, nix::TrustedFlag, nix::daemon::RecursiveFlag) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixstore.so #22 0x00005625ee345cd4 in std::_Function_handler<void (), nix::daemonLoop(std::optional<nix::TrustedFlag>)::$_0>::_M_invoke(std::_Any_data const&) () #23 0x00007fc09b17f35a in std::_Function_handler<void (), nix::startProcess(std::function<void ()>, nix::ProcessOptions const&)::$_0>::_M_invoke(std::_Any_data const&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #24 0x00007fc09b17e05e in nix::startProcess(std::function<void ()>, nix::ProcessOptions const&) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixutil.so #25 0x00005625ee3443d6 in nix::runDaemon(bool, std::optional<nix::TrustedFlag>) () #26 0x00005625ee342b83 in nix::main_nix_daemon(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) () #27 0x00005625ee34aa02 in int std::__invoke_impl<int, int (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >(std::__invoke_other, int (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&) () #28 0x00005625ee3c8b32 in nix::mainWrapped(int, char**) () #29 0x00007fc09abeca9b in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) () from target:/nix/store/r36d73af7p2vfmihian744gfpk12hxgp-lix-2.92.0-dev-pre20241211-92ed9fe/lib/liblixmain.so #30 0x00005625ee3cc84b in main () ```

raito added the

labels

2025-01-04 18:04:38 +00:00

lix-bot commented

2025-01-05 00:26:58 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/2350 ("libstore: fix download deadlock on interruption")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/2350](https://gerrit.lix.systems/c/lix/+/2350) ("libstore: fix download deadlock on interruption")