Ctrl-Cing at the wrong moment a S3 transfer can SEGSIGV #1023

New issue

Open

opened 2025-11-01 14:08:02 +00:00 by raito · 3 comments

raito commented

2025-11-01 14:08:02 +00:00

Owner

Describe the bug

Ctrl-Cing at the wrong moment a S3 transfer can lead to segmentation faults.

Steps To Reproduce

Perform a nix copy
Ctrl-C twice "at the wrong moment"
See a segfault

Expected behavior

No segfaults.

`nix --version` output

HEAD as of 91867941fa + cl/4514 + cl/4513.

Additional context

Core was generated by `./outputs/out/bin/nix copy --to s3://lix-perf-tests\?region=garage\&endpoint=s3.dc1.infra.lahfa.xyz\&multipart-upload=true /run/current-system'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f266ebab92f in Aws::Utils::Crypto::CreateSha256Implementation () at /build/source/src/aws-cpp-sdk-core/source/utils/crypto/factory/Factories.cpp:804

warning: 804	/build/source/src/aws-cpp-sdk-core/source/utils/crypto/factory/Factories.cpp: No such file or directory
[Current thread is 1 (Thread 0x7f2550b406c0 (LWP 2958020))]
gdb$ bt
#0  0x00007f266ebab92f in Aws::Utils::Crypto::CreateSha256Implementation () at /build/source/src/aws-cpp-sdk-core/source/utils/crypto/factory/Factories.cpp:804
#1  0x00007f266ebab27a in Aws::Utils::Crypto::Sha256::Sha256 (this=this@entry=0x7f2550b3bde0) at /build/source/src/aws-cpp-sdk-core/source/utils/crypto/Sha256.cpp:14
#2  0x00007f266eb9e8c7 in Aws::Utils::HashingUtils::CalculateSHA256 (str="PUT\n/lix-perf-tests/nar/184rfncm8as6fvdmagzmcwhx4zmr9y69g9a623gnmr29cwih7qhc.nar.zst\npartNumber=2&uploadId=fbd49f245f4b867e75156d0125b626245295cdd57d0052009da32ee36dc2b11c\namz-sdk-invocation-id:8AEBBD"...) at /build/source/src/aws-cpp-sdk-core/source/utils/HashingUtils.cpp:49
#3  0x00007f266eb3aa67 in Aws::Client::AWSAuthV4Signer::SignRequestWithCreds (this=this@entry=0x5621e74b0880, request=..., credentials=..., region=region@entry=0x7f2550b3db48 "garage", serviceName=serviceName@entry=0x7f2550b3db20 "s3", signBody=<optimized out>, signBody@entry=0x1) at /build/source/src/aws-cpp-sdk-core/source/auth/signer/AWSAuthV4Signer.cpp:326
#4  0x00007f266eb3b50e in Aws::Client::AWSAuthV4Signer::SignRequest (this=0x5621e74b0880, request=..., region=0x7f2550b3db48 "garage", serviceName=0x7f2550b3db20 "s3", signBody=0x1) at /build/source/src/aws-cpp-sdk-core/source/auth/signer/AWSAuthV4Signer.cpp:354
#5  0x00007f266eb51da5 in std::function<bool()>::operator() (this=0x7f2550b3cac0) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:591
#6  smithy::components::tracing::TracingUtils::MakeCallWithTiming<bool> (func=..., metricName="smithy.client.auth.signing_duration", meter=..., attributes=..., description="") at /build/source/src/aws-cpp-sdk-core/include/smithy/tracing/TracingUtils.h:66
#7  0x00007f266eb2fef7 in Aws::Client::AWSClient::AttemptOneRequest (this=this@entry=0x5621e73de750, httpRequest=std::shared_ptr<Aws::Http::HttpRequest> (use count 2, weak count 0) = {...}, request=..., signerName=signerName@entry=0x7f2550b3db00 "SignatureV4", signerRegionOverride=<optimized out>, signerRegionOverride@entry=0x7f2550b3db48 "garage", signerServiceNameOverride=<optimized out>) at /build/source/src/aws-cpp-sdk-core/source/client/AWSClient.cpp:562
#8  0x00007f266eb30d0c in Aws::Client::AWSClient::AttemptExhaustively (this=this@entry=0x5621e73de750, uri=..., request=..., method=method@entry=Aws::Http::HttpMethod::HTTP_PUT, signerName=signerName@entry=0x7f2550b3db00 "SignatureV4", signerRegionOverride=0x7f2550b3db48 "garage", signerServiceNameOverride=0x7f2550b3db20 "s3") at /build/source/src/aws-cpp-sdk-core/source/client/AWSClient.cpp:297
#9  0x00007f266eb339f9 in Aws::Client::AWSXMLClient::MakeRequest (this=this@entry=0x5621e73de750, uri=..., request=..., method=method@entry=Aws::Http::HttpMethod::HTTP_PUT, signerName=signerName@entry=0x7f2550b3db00 "SignatureV4", signerRegionOverride=0x7f2550b3db48 "garage", signerServiceNameOverride=0x7f2550b3db20 "s3") at /build/source/src/aws-cpp-sdk-core/source/client/AWSXmlClient.cpp:101
#10 0x00007f266eb3436d in Aws::Client::AWSXMLClient::MakeRequest (this=0x5621e73de750, request=..., endpoint=..., method=method@entry=Aws::Http::HttpMethod::HTTP_PUT, signerName=0x7f2550b3db00 "SignatureV4", signerRegionOverride=0x7f2550b3db48 "garage", signerServiceNameOverride=0x7f2550b3db20 "s3") at /build/source/src/aws-cpp-sdk-core/source/client/AWSXmlClient.cpp:68
#11 0x00007f266eddf57c in operator() (__closure=0x7f252c0ef340) at /build/source/src/aws-cpp-sdk-core/include/aws/core/utils/Outcome.h:136
#12 0x00007f266eddf79e in std::__invoke_impl<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>, Aws::S3::S3Client::UploadPart(const Aws::S3::Model::UploadPartRequest&) const::<lambda()>&> (__f=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:60
#13 std::__invoke_r<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>, Aws::S3::S3Client::UploadPart(const Aws::S3::Model::UploadPartRequest&) const::<lambda()>&> (__fn=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:139
#14 std::_Function_handler<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>(), Aws::S3::S3Client::UploadPart(const Aws::S3::Model::UploadPartRequest&) const::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:291
#15 0x00007f266ed80f5e in std::function<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>()>::operator() (this=0x7f2550b3de70) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:591
#16 smithy::components::tracing::TracingUtils::MakeCallWithTiming<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> > (func=..., metricName="smithy.client.duration", meter=..., attributes=..., description="") at /build/source/src/aws-cpp-sdk-core/include/smithy/tracing/TracingUtils.h:66
#17 Aws::S3::S3Client::UploadPart (this=0x5621e73de750, request=...) at /build/source/generated/src/aws-cpp-sdk-s3/source/S3Client.cpp:4474
#18 0x00007f2671480e30 in Aws::Client::MakeAsyncOperation<Aws::S3::S3Client, Aws::S3::Model::UploadPartRequest, std::function<void (Aws::S3::S3Client const*, Aws::S3::Model::UploadPartRequest const&, Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> const&, std::shared_ptr<Aws::Client::AsyncCallerContext const> const&)>, std::shared_ptr<Aws::Client::AsyncCallerContext const>, Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> (Aws::S3::S3Client::*&)(Aws::S3::Model::UploadPartRequest const&) const, Aws::Utils::Threading::Executor>(Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> (Aws::S3::S3Client::*&)(Aws::S3::Model::UploadPartRequest const&) const, Aws::S3::S3Client const*, Aws::S3::Model::UploadPartRequest const&, std::function<void (Aws::S3::S3Client const*, Aws::S3::Model::UploadPartRequest const&, Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> const&, std::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::shared_ptr<Aws::Client::AsyncCallerContext const> const&, Aws::Utils::Threading::Executor*)::{lambda()#1}::operator()() const (__closure=0x7f25b800a520) at /build/source/src/aws-cpp-sdk-core/include/aws/core/client/AWSAsyncOperationTemplate.h:40
#19 0x00007f266eb363f6 in std::function<void()>::operator() (this=0x7f25b80011f8) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:591
#20 operator() (__closure=0x7f25b80011f0, storedFx=...) at /build/source/src/aws-cpp-sdk-core/source/utils/threading/DefaultExecutor.cpp:19
#21 std::__invoke_impl<void, Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>&, std::function<void()>&> (__f=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61
#22 std::__invoke<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>&, std::function<void()>&> (__fn=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96
#23 std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>::__call<void, 0> (this=0x7f25b80011f0, __args=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/functional:513
#24 std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>::operator()<> (this=0x7f25b80011f0) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/functional:598
#25 std::__invoke_impl<void, std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>&> (__f=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61
#26 std::__invoke_r<void, std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>&> (__fn=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:150
#27 std::_Function_handler<void(), std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:290
#28 0x00007f266faed064 in execute_native_thread_routine () from /nix/store/dj06r96j515npcqi9d8af1d1c60bx2vn-gcc-14.3.0-lib/lib/libstdc++.so.6
#29 0x00007f266f6978ee in start_thread (arg=<optimized out>) at pthread_create.c:447
#30 0x00007f266f71b9bc in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
gdb$

Here's the stacktrace, it seems plausible to me that we are shutting down some global facility that AWS uses but we are unable to "cancel" the work being done, so when things are continuing their life, they are exploding.

An minor issue.

## Describe the bug Ctrl-Cing at the wrong moment a S3 transfer can lead to segmentation faults. ## Steps To Reproduce 1. Perform a `nix copy` 2. Ctrl-C twice "at the wrong moment" 3. See a segfault ## Expected behavior No segfaults. ## `nix --version` output HEAD as of 91867941fa73afea7869b7c71ede82e5ef8927da + cl/4514 + cl/4513. ## Additional context ``` Core was generated by `./outputs/out/bin/nix copy --to s3://lix-perf-tests\?region=garage\&endpoint=s3.dc1.infra.lahfa.xyz\&multipart-upload=true /run/current-system'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f266ebab92f in Aws::Utils::Crypto::CreateSha256Implementation () at /build/source/src/aws-cpp-sdk-core/source/utils/crypto/factory/Factories.cpp:804 warning: 804 /build/source/src/aws-cpp-sdk-core/source/utils/crypto/factory/Factories.cpp: No such file or directory [Current thread is 1 (Thread 0x7f2550b406c0 (LWP 2958020))] gdb$ bt #0 0x00007f266ebab92f in Aws::Utils::Crypto::CreateSha256Implementation () at /build/source/src/aws-cpp-sdk-core/source/utils/crypto/factory/Factories.cpp:804 #1 0x00007f266ebab27a in Aws::Utils::Crypto::Sha256::Sha256 (this=this@entry=0x7f2550b3bde0) at /build/source/src/aws-cpp-sdk-core/source/utils/crypto/Sha256.cpp:14 #2 0x00007f266eb9e8c7 in Aws::Utils::HashingUtils::CalculateSHA256 (str="PUT\n/lix-perf-tests/nar/184rfncm8as6fvdmagzmcwhx4zmr9y69g9a623gnmr29cwih7qhc.nar.zst\npartNumber=2&uploadId=fbd49f245f4b867e75156d0125b626245295cdd57d0052009da32ee36dc2b11c\namz-sdk-invocation-id:8AEBBD"...) at /build/source/src/aws-cpp-sdk-core/source/utils/HashingUtils.cpp:49 #3 0x00007f266eb3aa67 in Aws::Client::AWSAuthV4Signer::SignRequestWithCreds (this=this@entry=0x5621e74b0880, request=..., credentials=..., region=region@entry=0x7f2550b3db48 "garage", serviceName=serviceName@entry=0x7f2550b3db20 "s3", signBody=<optimized out>, signBody@entry=0x1) at /build/source/src/aws-cpp-sdk-core/source/auth/signer/AWSAuthV4Signer.cpp:326 #4 0x00007f266eb3b50e in Aws::Client::AWSAuthV4Signer::SignRequest (this=0x5621e74b0880, request=..., region=0x7f2550b3db48 "garage", serviceName=0x7f2550b3db20 "s3", signBody=0x1) at /build/source/src/aws-cpp-sdk-core/source/auth/signer/AWSAuthV4Signer.cpp:354 #5 0x00007f266eb51da5 in std::function<bool()>::operator() (this=0x7f2550b3cac0) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:591 #6 smithy::components::tracing::TracingUtils::MakeCallWithTiming<bool> (func=..., metricName="smithy.client.auth.signing_duration", meter=..., attributes=..., description="") at /build/source/src/aws-cpp-sdk-core/include/smithy/tracing/TracingUtils.h:66 #7 0x00007f266eb2fef7 in Aws::Client::AWSClient::AttemptOneRequest (this=this@entry=0x5621e73de750, httpRequest=std::shared_ptr<Aws::Http::HttpRequest> (use count 2, weak count 0) = {...}, request=..., signerName=signerName@entry=0x7f2550b3db00 "SignatureV4", signerRegionOverride=<optimized out>, signerRegionOverride@entry=0x7f2550b3db48 "garage", signerServiceNameOverride=<optimized out>) at /build/source/src/aws-cpp-sdk-core/source/client/AWSClient.cpp:562 #8 0x00007f266eb30d0c in Aws::Client::AWSClient::AttemptExhaustively (this=this@entry=0x5621e73de750, uri=..., request=..., method=method@entry=Aws::Http::HttpMethod::HTTP_PUT, signerName=signerName@entry=0x7f2550b3db00 "SignatureV4", signerRegionOverride=0x7f2550b3db48 "garage", signerServiceNameOverride=0x7f2550b3db20 "s3") at /build/source/src/aws-cpp-sdk-core/source/client/AWSClient.cpp:297 #9 0x00007f266eb339f9 in Aws::Client::AWSXMLClient::MakeRequest (this=this@entry=0x5621e73de750, uri=..., request=..., method=method@entry=Aws::Http::HttpMethod::HTTP_PUT, signerName=signerName@entry=0x7f2550b3db00 "SignatureV4", signerRegionOverride=0x7f2550b3db48 "garage", signerServiceNameOverride=0x7f2550b3db20 "s3") at /build/source/src/aws-cpp-sdk-core/source/client/AWSXmlClient.cpp:101 #10 0x00007f266eb3436d in Aws::Client::AWSXMLClient::MakeRequest (this=0x5621e73de750, request=..., endpoint=..., method=method@entry=Aws::Http::HttpMethod::HTTP_PUT, signerName=0x7f2550b3db00 "SignatureV4", signerRegionOverride=0x7f2550b3db48 "garage", signerServiceNameOverride=0x7f2550b3db20 "s3") at /build/source/src/aws-cpp-sdk-core/source/client/AWSXmlClient.cpp:68 #11 0x00007f266eddf57c in operator() (__closure=0x7f252c0ef340) at /build/source/src/aws-cpp-sdk-core/include/aws/core/utils/Outcome.h:136 #12 0x00007f266eddf79e in std::__invoke_impl<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>, Aws::S3::S3Client::UploadPart(const Aws::S3::Model::UploadPartRequest&) const::<lambda()>&> (__f=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:60 #13 std::__invoke_r<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>, Aws::S3::S3Client::UploadPart(const Aws::S3::Model::UploadPartRequest&) const::<lambda()>&> (__fn=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:139 #14 std::_Function_handler<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>(), Aws::S3::S3Client::UploadPart(const Aws::S3::Model::UploadPartRequest&) const::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:291 #15 0x00007f266ed80f5e in std::function<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error>()>::operator() (this=0x7f2550b3de70) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:591 #16 smithy::components::tracing::TracingUtils::MakeCallWithTiming<Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> > (func=..., metricName="smithy.client.duration", meter=..., attributes=..., description="") at /build/source/src/aws-cpp-sdk-core/include/smithy/tracing/TracingUtils.h:66 #17 Aws::S3::S3Client::UploadPart (this=0x5621e73de750, request=...) at /build/source/generated/src/aws-cpp-sdk-s3/source/S3Client.cpp:4474 #18 0x00007f2671480e30 in Aws::Client::MakeAsyncOperation<Aws::S3::S3Client, Aws::S3::Model::UploadPartRequest, std::function<void (Aws::S3::S3Client const*, Aws::S3::Model::UploadPartRequest const&, Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> const&, std::shared_ptr<Aws::Client::AsyncCallerContext const> const&)>, std::shared_ptr<Aws::Client::AsyncCallerContext const>, Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> (Aws::S3::S3Client::*&)(Aws::S3::Model::UploadPartRequest const&) const, Aws::Utils::Threading::Executor>(Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> (Aws::S3::S3Client::*&)(Aws::S3::Model::UploadPartRequest const&) const, Aws::S3::S3Client const*, Aws::S3::Model::UploadPartRequest const&, std::function<void (Aws::S3::S3Client const*, Aws::S3::Model::UploadPartRequest const&, Aws::Utils::Outcome<Aws::S3::Model::UploadPartResult, Aws::S3::S3Error> const&, std::shared_ptr<Aws::Client::AsyncCallerContext const> const&)> const&, std::shared_ptr<Aws::Client::AsyncCallerContext const> const&, Aws::Utils::Threading::Executor*)::{lambda()#1}::operator()() const (__closure=0x7f25b800a520) at /build/source/src/aws-cpp-sdk-core/include/aws/core/client/AWSAsyncOperationTemplate.h:40 #19 0x00007f266eb363f6 in std::function<void()>::operator() (this=0x7f25b80011f8) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:591 #20 operator() (__closure=0x7f25b80011f0, storedFx=...) at /build/source/src/aws-cpp-sdk-core/source/utils/threading/DefaultExecutor.cpp:19 #21 std::__invoke_impl<void, Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>&, std::function<void()>&> (__f=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61 #22 std::__invoke<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>&, std::function<void()>&> (__fn=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96 #23 std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>::__call<void, 0> (this=0x7f25b80011f0, __args=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/functional:513 #24 std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>::operator()<> (this=0x7f25b80011f0) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/functional:598 #25 std::__invoke_impl<void, std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>&> (__f=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61 #26 std::__invoke_r<void, std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)>&> (__fn=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:150 #27 std::_Function_handler<void(), std::_Bind<Aws::Utils::Threading::DefaultExecutor::SubmitToThread(std::function<void()>&&)::<lambda(std::function<void()>&)>(std::function<void()>)> >::_M_invoke(const std::_Any_data &) (__functor=...) at /nix/store/8adzgnxs3s0pbj22qhk9zjxi1fqmz3xv-gcc-14.3.0/include/c++/14.3.0/bits/std_function.h:290 #28 0x00007f266faed064 in execute_native_thread_routine () from /nix/store/dj06r96j515npcqi9d8af1d1c60bx2vn-gcc-14.3.0-lib/lib/libstdc++.so.6 #29 0x00007f266f6978ee in start_thread (arg=<optimized out>) at pthread_create.c:447 #30 0x00007f266f71b9bc in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 gdb$ ``` Here's the stacktrace, it seems plausible to me that we are shutting down some global facility that AWS uses but we are unable to "cancel" the work being done, so when things are continuing their life, they are exploding. An minor issue.

raito added the

bug

label

2025-11-01 14:08:02 +00:00

pennae commented

2025-11-02 17:25:10 +00:00

Owner

since there is no way to cancel single async operations of an s3 client this may be unfixable in principle. the best workaround we can hope for may be to keep the s3 client alive while operations are running by shoving it into a shared_ptr and capturing a copy of that into operation callbacks, but that makes it impossible to stop operations since the only avenue for this offered by the sdk is to pause the queue, keeping the client alive forever.

tl;dr the aws sdk is ... very badly suited for our use case and we should use something else

since there is no way to cancel single async operations of an s3 client this may be unfixable in principle. the best workaround we can hope for may be to keep the s3 client alive while operations are running by shoving it into a `shared_ptr` and capturing a copy of that into operation callbacks, but that makes it impossible to *stop* operations since the only avenue for this offered by the sdk is to *pause the queue*, keeping the client alive forever. tl;dr the aws sdk is ... ***very badly** suited* for our use case and we should use something else

jade added the

Feature/S3

label

2025-11-02 17:45:40 +00:00

jade commented

2025-11-02 17:46:24 +00:00

Owner

well, fortunately we want to get rid of it. so i guess we can disregard this bug and consider it blocked on someone funding getting rid of the problem library.