Resource temporarily unavailable and stack trace on running `nix copy` to S3 bucket #1168

New issue

Open

opened 2026-03-27 15:38:03 +00:00 by benaryorg · 1 comment

benaryorg commented

2026-03-27 15:38:03 +00:00

Describe the bug

While trying to copy data to my S3 bucket (nc.benary.org) Lix as described below.
I'm specifically running this here (to be consistent with what my Hydra most likely uses):

# sudo stuff since the key is not readable for my user
sudo --preserve-env=AWS_ACCESS_KEY_ID,AWS_ENDPOINT_URL,AWS_SECRET_ACCESS_KEY,AWS_REGION nix copy --to 's3://nc.benary.org?secret-key=/run/agenix/hydraSigningKey&endpoint=s3.ovh.xn--idk5byd.net&write-nar-listing=1&want-mass-query=1' /nix/store/b0adabqbdip29dh3clcnvjdzx0v43qjj-unit-container-mds-2.service.drv

This happened many times before with different derivations copied at different times in the copy process, and it is usually enough to run it in a until timeout -k 16 64 $command;do sleep 4;done loop.
I have recently messed up my S3 bucket while testing the new Ceph release, so there could be some malformed data on the other side, or there could be unexpected errors, however neither of those should lead to Lix giving me a full on stack trace.

I am still trying to reproduce the issue while running with -vvvvv, but no luck so far.
Opening this issue ahead of time for discoverability in case someone else is having similar issues, but also for the off chance that someone might already figure out something from the stack trace.

full exception/stack trace

It fails basically in the middle of it, with the progress being stuck on the start of the line and the rest being the first line of this text:

This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:

Exception: std::__exception_ptr::exception_ptr: Resource temporarily unavailable
Stack trace:
 0# nix::getStackTrace[abi:cxx11]() in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
 1# nix::logException(std::basic_string_view<char, std::char_traits<char> >, std::exception const&) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
 2# 0x00007F11A7ED8558 in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
 3# 0x00007F11A68BF1AA in /nix/store/alrbhz7im0w0jdwcfdgcfk7pxhkl1fzj-gcc-14.3.0-lib/lib/libstdc++.so.6
 4# std::unexpected() in /nix/store/alrbhz7im0w0jdwcfdgcfk7pxhkl1fzj-gcc-14.3.0-lib/lib/libstdc++.so.6
 5# 0x00007F11A68AD800 in /nix/store/alrbhz7im0w0jdwcfdgcfk7pxhkl1fzj-gcc-14.3.0-lib/lib/libstdc++.so.6
 6# 0x000055A3DED8BF91 in nix
 7# 0x000055A3DED8B0A9 in nix
 8# nix::BuiltPathsCommand::run(nix::ref<nix::Store>, std::vector<nix::ref<nix::Installable>, std::allocator<nix::ref<nix::Installable> > >&&) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
 9# nix::InstallablesCommand::run(nix::ref<nix::Store>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
10# nix::RawInstallablesCommand::run(nix::ref<nix::Store>) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
11# nix::StoreCommand::run() in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
12# 0x000055A3DEE44E22 in nix
13# 0x000055A3DEE474D5 in nix
14# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<int ()>) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so
15# 0x000055A3DEE46CA5 in nix
16# 0x00007F11A642A4D8 in /nix/store/km4g87jxsqxvcq344ncyb8h1i6f3cqxh-glibc-2.40-218/lib/libc.so.6
17# __libc_start_main in /nix/store/km4g87jxsqxvcq344ncyb8h1i6f3cqxh-glibc-2.40-218/lib/libc.so.6
18# 0x000055A3DED53535 in nix

Async task trace (probably incomplete):
#0: virtual kj::Promise<Result<box_ptr<AsyncInputStream>>> nix::S3BinaryCacheStoreImpl::getFile(const std::string &, const Activity *) (lix/libstore/s3-binary-cache-store.cc:620:66)
#1: virtual kj::Promise<Result<std::optional<std::string>>> nix::BinaryCacheStore::getFileContents(const std::string &, const Activity *) (lix/libstore/binary-cache-store.cc:90:71)
#2: virtual kj::Promise<Result<std::shared_ptr<const ValidPathInfo>>> nix::BinaryCacheStore::queryPathInfoUncached(const StorePath &, const Activity *) (lix/libstore/binary-cache-store.cc:472:61)
#3: kj::Promise<Result<ref<const ValidPathInfo>>> nix::Store::queryPathInfo(const StorePath &, const Activity *) (lix/libstore/store-api.cc:726:68)
#4: auto nix::Store::queryValidPaths(const StorePathSet &, SubstituteFlag)::(anonymous class)::operator()(const StorePath &) const (lix/libstore/store-api.cc:787:42)
#5: virtual kj::Promise<Result<StorePathSet>> nix::Store::queryValidPaths(const StorePathSet &, SubstituteFlag) (lix/libstore/store-api.cc:796:42)
#6: kj::Promise<Result<std::map<StorePath, StorePath>>> nix::copyPaths(Store &, Store &, const StorePathSet &, RepairFlag, CheckSigsFlag, SubstituteFlag) (lix/libstore/store-api.cc:1125:76)
#7: kj::Promise<Result<std::map<StorePath, StorePath>>> nix::copyPaths(Store &, Store &, const RealisedPath::Set &, RepairFlag, CheckSigsFlag, SubstituteFlag) (lix/libstore/store-api.cc:1112:97)
#8: virtual void nix::CmdCopy::run(ref<Store>, BuiltPaths &&) (lix/nix/copy.cc:58:15)

Steps To Reproduce

Unable to reproduce reliably.
Basically every once in a while my Hydra fails due some nar files missing from the bucket (which is what I'm trying to fix using the copy).
Upon then running the copy it tends (not always) to err out with the stack trace, however upon rerunning it continues to copy some files before failing again, until it successfully completes.

Expected behavior

Lix should either give me a tangible error which could help me fix whatever issue it is encountering, or succeed to begin with.
It most certainly should not print a stack trace (although to be fair that's still better than failing silently).

`nix --version` output

lix-2.96.0-dev-35b7765 (35b7765 on top of NixOS 25.11; I can switch to any git commit or add patches for testing)

% nix --version
nix (Lix, like Nix) 2.96.0-dev-35b7765
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/benaryorg/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/benaryorg/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/benaryorg/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/benaryorg/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/share

Additional context

The closest issue I found was a really old CppNix issue with different semantics and no stack trace, and the CppNix issue about the fork limit not being high enough but since mine is set to 1048576 this should not be related.

If you want/need a bucket for testing I can provide an HDD backed bucket with virtually no space limit running on the exact same Ceph cluster/radosgw instance.

## Describe the bug While trying to copy data to my S3 bucket (`nc.benary.org`) Lix as described below. I'm specifically running this here (to be consistent with what my *Hydra* most likely uses): ```bash # sudo stuff since the key is not readable for my user sudo --preserve-env=AWS_ACCESS_KEY_ID,AWS_ENDPOINT_URL,AWS_SECRET_ACCESS_KEY,AWS_REGION nix copy --to 's3://nc.benary.org?secret-key=/run/agenix/hydraSigningKey&endpoint=s3.ovh.xn--idk5byd.net&write-nar-listing=1&want-mass-query=1' /nix/store/b0adabqbdip29dh3clcnvjdzx0v43qjj-unit-container-mds-2.service.drv ``` This happened many times before with different derivations copied at different times in the copy process, and it is usually enough to run it in a `until timeout -k 16 64 $command;do sleep 4;done` loop. I have recently messed up my S3 bucket while testing the new Ceph release, so there could be some malformed data on the other side, or there could be unexpected errors, however neither of those should lead to Lix giving me a full on stack trace. I am still trying to reproduce the issue while running with `-vvvvv`, but no luck so far. Opening this issue ahead of time for discoverability in case someone else is having similar issues, but also for the off chance that someone might already figure out something from the stack trace. <details><summary>full exception/stack trace</summary> It fails basically in the middle of it, with the progress being stuck on the start of the line and the rest being the first line of this text: ```text This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included: Exception: std::__exception_ptr::exception_ptr: Resource temporarily unavailable Stack trace: 0# nix::getStackTrace[abi:cxx11]() in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 1# nix::logException(std::basic_string_view<char, std::char_traits<char> >, std::exception const&) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 2# 0x00007F11A7ED8558 in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 3# 0x00007F11A68BF1AA in /nix/store/alrbhz7im0w0jdwcfdgcfk7pxhkl1fzj-gcc-14.3.0-lib/lib/libstdc++.so.6 4# std::unexpected() in /nix/store/alrbhz7im0w0jdwcfdgcfk7pxhkl1fzj-gcc-14.3.0-lib/lib/libstdc++.so.6 5# 0x00007F11A68AD800 in /nix/store/alrbhz7im0w0jdwcfdgcfk7pxhkl1fzj-gcc-14.3.0-lib/lib/libstdc++.so.6 6# 0x000055A3DED8BF91 in nix 7# 0x000055A3DED8B0A9 in nix 8# nix::BuiltPathsCommand::run(nix::ref<nix::Store>, std::vector<nix::ref<nix::Installable>, std::allocator<nix::ref<nix::Installable> > >&&) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 9# nix::InstallablesCommand::run(nix::ref<nix::Store>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 10# nix::RawInstallablesCommand::run(nix::ref<nix::Store>) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 11# nix::StoreCommand::run() in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 12# 0x000055A3DEE44E22 in nix 13# 0x000055A3DEE474D5 in nix 14# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<int ()>) in /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/lib/liblix.so 15# 0x000055A3DEE46CA5 in nix 16# 0x00007F11A642A4D8 in /nix/store/km4g87jxsqxvcq344ncyb8h1i6f3cqxh-glibc-2.40-218/lib/libc.so.6 17# __libc_start_main in /nix/store/km4g87jxsqxvcq344ncyb8h1i6f3cqxh-glibc-2.40-218/lib/libc.so.6 18# 0x000055A3DED53535 in nix Async task trace (probably incomplete): #0: virtual kj::Promise<Result<box_ptr<AsyncInputStream>>> nix::S3BinaryCacheStoreImpl::getFile(const std::string &, const Activity *) (lix/libstore/s3-binary-cache-store.cc:620:66) #1: virtual kj::Promise<Result<std::optional<std::string>>> nix::BinaryCacheStore::getFileContents(const std::string &, const Activity *) (lix/libstore/binary-cache-store.cc:90:71) #2: virtual kj::Promise<Result<std::shared_ptr<const ValidPathInfo>>> nix::BinaryCacheStore::queryPathInfoUncached(const StorePath &, const Activity *) (lix/libstore/binary-cache-store.cc:472:61) #3: kj::Promise<Result<ref<const ValidPathInfo>>> nix::Store::queryPathInfo(const StorePath &, const Activity *) (lix/libstore/store-api.cc:726:68) #4: auto nix::Store::queryValidPaths(const StorePathSet &, SubstituteFlag)::(anonymous class)::operator()(const StorePath &) const (lix/libstore/store-api.cc:787:42) #5: virtual kj::Promise<Result<StorePathSet>> nix::Store::queryValidPaths(const StorePathSet &, SubstituteFlag) (lix/libstore/store-api.cc:796:42) #6: kj::Promise<Result<std::map<StorePath, StorePath>>> nix::copyPaths(Store &, Store &, const StorePathSet &, RepairFlag, CheckSigsFlag, SubstituteFlag) (lix/libstore/store-api.cc:1125:76) #7: kj::Promise<Result<std::map<StorePath, StorePath>>> nix::copyPaths(Store &, Store &, const RealisedPath::Set &, RepairFlag, CheckSigsFlag, SubstituteFlag) (lix/libstore/store-api.cc:1112:97) #8: virtual void nix::CmdCopy::run(ref<Store>, BuiltPaths &&) (lix/nix/copy.cc:58:15) ``` </details> ## Steps To Reproduce Unable to reproduce reliably. Basically every once in a while my Hydra fails due some nar files missing from the bucket (which is what I'm trying to fix using the copy). Upon then running the copy it tends (not always) to err out with the stack trace, however upon rerunning it continues to copy some files before failing again, until it successfully completes. ## Expected behavior Lix should either give me a tangible error which could help me fix whatever issue it is encountering, or succeed to begin with. It most certainly should not print a stack trace (although to be fair that's still better than failing silently). ## `nix --version` output lix-2.96.0-dev-35b7765 (35b7765 on top of NixOS 25.11; I can switch to any git commit or add patches for testing) ```console % nix --version nix (Lix, like Nix) 2.96.0-dev-35b7765 System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/benaryorg/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/benaryorg/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/benaryorg/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/benaryorg/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/lzxq6f0bvq9i2xkfqpz9arc7zfhbavby-lix-2.96.0-dev-35b7765/share ``` ## Additional context The closest issue I found was [a really old CppNix issue with different semantics and no stack trace](https://github.com/NixOS/nix/issues/2560), and the [CppNix issue about the fork limit not being high enough](https://github.com/NixOS/nix/issues/3717) but since mine is set to 1048576 this should not be related. If you want/need a bucket for testing I can provide an HDD backed bucket with virtually no space limit running on the exact same Ceph cluster/radosgw instance.

benaryorg added the

bug

label

2026-03-27 15:38:03 +00:00

benaryorg commented

2026-03-28 04:03:46 +00:00

Author

I've just encountered a similar error which may be related, but I'm not entirely sure, since that one required a Ctrl+C specifically (thus triggering #1023), which I am fairly (but not 100%) certain that this issue did not trigger.
Anyway at some point during a different copy, the progress just hung seemingly forever with the -vvvvv output showing a semblance of the following in a loop:

AWS: [TRACE] 2026-03-28 02:57:21.062 event-loop [140673924576960] id=0x5617805ab200: wake up with 0 events to process.
AWS: [TRACE] 2026-03-28 02:57:21.063 event-loop [140673924576960] id=0x5617805ab200: running scheduled tasks.
AWS: [TRACE] 2026-03-28 02:57:21.063 event-loop [140673924576960] id=0x5617805ab200: no more scheduled tasks using default timeout.
AWS: [TRACE] 2026-03-28 02:57:21.063 event-loop [140673924576960] id=0x5617805ab200: waiting for a maximum of 100000 ms

However I can imagine something else (I dunno, SIGWINCH or whatever) to trigger the same thing in my earlier issue.
Still waiting for the stars to align to allow me to reproduce the issue without Ctrl+C with -vvvvv.

I've just encountered a similar error which *may* be related, but I'm not entirely sure, since that one required a Ctrl+C specifically (thus triggering #1023), which I am fairly (but not 100%) certain that this issue did not trigger. Anyway at some point during a different copy, the progress just hung seemingly forever with the `-vvvvv` output showing a semblance of the following in a loop: ```text AWS: [TRACE] 2026-03-28 02:57:21.062 event-loop [140673924576960] id=0x5617805ab200: wake up with 0 events to process. AWS: [TRACE] 2026-03-28 02:57:21.063 event-loop [140673924576960] id=0x5617805ab200: running scheduled tasks. AWS: [TRACE] 2026-03-28 02:57:21.063 event-loop [140673924576960] id=0x5617805ab200: no more scheduled tasks using default timeout. AWS: [TRACE] 2026-03-28 02:57:21.063 event-loop [140673924576960] id=0x5617805ab200: waiting for a maximum of 100000 ms ``` However I can imagine something else (I dunno, *SIGWINCH* or whatever) to trigger the same thing in my earlier issue. Still waiting for the stars to align to allow me to reproduce the issue without Ctrl+C with `-vvvvv`.