nix nar ls on a zstd compressed file crashes nix #993

New issue

Closed

opened 2025-09-12 17:10:00 +00:00 by jmjl · 3 comments

jmjl commented

2025-09-12 17:10:00 +00:00

Describe the bug

I tried to download a NAR file to understand how it works, but the remote store (attic) seems to give users compressed NAR files (which I didn't know until i asked someone more knowledgeable)

Steps To Reproduce

Download a nar file, or make one with nix nar dump-path <path> | zstd >test.nar
Run nix nar ls test.nar /
See error

Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included:

Exception: std::length_error: vector::_M_default_append
Stack trace:
 0# nix::getStackTrace[abi:cxx11]() in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so
 1# nix::logException(std::basic_string_view<char, std::char_traits<char> >, std::exception const&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so
 2# 0x00007FB16FDD6A54 in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixmain.so
 3# 0x00007FB16E4BF1CA in /nix/store/7c0v0kbrrdc2cqgisi78jdqxn73n3401-gcc-14.2.1.20250322-lib/lib/libstdc++.so.6
 4# std::unexpected() in /nix/store/7c0v0kbrrdc2cqgisi78jdqxn73n3401-gcc-14.2.1.20250322-lib/lib/libstdc++.so.6
 5# 0x00007FB16E4BF1BE in /nix/store/7c0v0kbrrdc2cqgisi78jdqxn73n3401-gcc-14.2.1.20250322-lib/lib/libstdc++.so.6
 6# decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<std::optional<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> > >, nix::overloaded<nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::finished)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::failure&)#1}>, std::variant<nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >, std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, nix::_generator::failure, nix::_generator::finished>&>(nix::overloaded<nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::finished)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::failure&)#1}>&&, std::variant<nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >, std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, nix::_generator::failure, nix::_generator::finished>&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so
 7# nix::parseDump(nix::NARParseVisitor&, nix::Source&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so
 8# nix::nar_index::create(nix::Source&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so
 9# nix::NarAccessor::NarAccessor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixstore.so
10# nix::makeNarAccessor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixstore.so
11# 0x0000563D76EE0394 in nix
12# 0x0000563D76EE2DD5 in nix
13# 0x0000563D76EE56BB in nix
14# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixmain.so
15# 0x0000563D76EE4F6B in nix
16# 0x00007FB16E02A47E in /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66/lib/libc.so.6
17# __libc_start_main in /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66/lib/libc.so.6
18# 0x0000563D76E03845 in nix

zsh: abort (core dumped)  nix nar ls ./testt.nar /

Expected behavior

nix nar ls work with all compressed files

`nix --version` output

nix (Lix, like Nix) 2.94.0-devpre20250829_f4bdddf
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/user/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/user/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/user/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/user/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/share

Additional context

I think you might need to get low memory or smth, because I was unable to reproduce this when making my own compressed nar files, i get the error out of memory instead of the crash

The file testt.nar is f7pd1mr3fwd02n5l7wrmq8k31phxwv8p.nar, from a private cache, and since it doesn't contain any PII, i'm willing to share it if needed (it's 11M)

## Describe the bug I tried to download a NAR file to understand how it works, but the remote store (attic) seems to give users compressed NAR files (which I didn't know until i asked someone more knowledgeable) ## Steps To Reproduce 1. Download a nar file, or make one with `nix nar dump-path <path> | zstd >test.nar` 2. Run `nix nar ls test.nar /` 3. See error ``` Lix crashed. This is a bug. We would appreciate if you report it along with what caused it at https://git.lix.systems/lix-project/lix/issues with the following information included: Exception: std::length_error: vector::_M_default_append Stack trace: 0# nix::getStackTrace[abi:cxx11]() in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so 1# nix::logException(std::basic_string_view<char, std::char_traits<char> >, std::exception const&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so 2# 0x00007FB16FDD6A54 in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixmain.so 3# 0x00007FB16E4BF1CA in /nix/store/7c0v0kbrrdc2cqgisi78jdqxn73n3401-gcc-14.2.1.20250322-lib/lib/libstdc++.so.6 4# std::unexpected() in /nix/store/7c0v0kbrrdc2cqgisi78jdqxn73n3401-gcc-14.2.1.20250322-lib/lib/libstdc++.so.6 5# 0x00007FB16E4BF1BE in /nix/store/7c0v0kbrrdc2cqgisi78jdqxn73n3401-gcc-14.2.1.20250322-lib/lib/libstdc++.so.6 6# decltype(auto) std::__do_visit<std::__detail::__variant::__deduce_visit_result<std::optional<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> > >, nix::overloaded<nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::finished)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::failure&)#1}>, std::variant<nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >, std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, nix::_generator::failure, nix::_generator::finished>&>(nix::overloaded<nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::finished)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>&)#1}, nix::_generator::GeneratorBase<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >::next()::{lambda(nix::_generator::failure&)#1}>&&, std::variant<nix::_generator::link<std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory> >, std::variant<nix::nar::File, nix::nar::Symlink, nix::nar::Directory>, nix::_generator::failure, nix::_generator::finished>&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so 7# nix::parseDump(nix::NARParseVisitor&, nix::Source&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so 8# nix::nar_index::create(nix::Source&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixutil.so 9# nix::NarAccessor::NarAccessor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixstore.so 10# nix::makeNarAccessor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixstore.so 11# 0x0000563D76EE0394 in nix 12# 0x0000563D76EE2DD5 in nix 13# 0x0000563D76EE56BB in nix 14# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/lib/liblixmain.so 15# 0x0000563D76EE4F6B in nix 16# 0x00007FB16E02A47E in /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66/lib/libc.so.6 17# __libc_start_main in /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66/lib/libc.so.6 18# 0x0000563D76E03845 in nix zsh: abort (core dumped) nix nar ls ./testt.nar / ``` ## Expected behavior nix nar ls work with all compressed files ## `nix --version` output ``` nix (Lix, like Nix) 2.94.0-devpre20250829_f4bdddf System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/user/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/user/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/user/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/user/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/aqyzczs8jcfw32cl9v7im609707m3dl2-lix-2.94.0-devpre20250829_f4bdddf/share ``` ## Additional context I think you might need to get low memory or smth, because I was unable to reproduce this when making my own compressed nar files, i get the error out of memory instead of the crash The file testt.nar is f7pd1mr3fwd02n5l7wrmq8k31phxwv8p.nar, from a private cache, and since it doesn't contain any PII, i'm willing to share it if needed (it's 11M)

jmjl added the

bug

label

2025-09-12 17:10:00 +00:00

pennae commented

2025-09-12 17:59:44 +00:00

Owner

the cli does not support compressed nar files in this manner. we definitely should not be crashing like this and show an actionable error message instead (e.g. "input is not a correct NAR file. if it is compressed it must be decompressed first." idk).

lix-bot commented

2025-09-12 21:22:31 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/4162 ("libutil: reject malformed nars harder with better errors")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/4162](https://gerrit.lix.systems/c/lix/+/4162) ("libutil: reject malformed nars harder with better errors")

lix-project referenced this issue from a commit

2025-09-12 22:18:55 +00:00

libutil: reject malformed nars harder with better errors

lix-project closed this issue

2025-09-12 22:18:55 +00:00

jmjl commented

2025-09-13 10:14:16 +00:00

Author

@pennae wrote in #993 (comment):

the cli does not support compressed nar files in this manner. we definitely should not be crashing like this and show an actionable error message instead (e.g. "input is not a correct NAR file. if it is compressed it must be decompressed first." idk).

Oh yeah sorry, i thought it did because it was sometimes failing with "Out of memory" messages, which led to me to incorrectly assume that it was taking the little memory available to try to decompress it, and couldn't.

Thanks!

@pennae wrote in https://git.lix.systems/lix-project/lix/issues/993#issuecomment-14916: > the cli does not support compressed nar files in this manner. we definitely should not be crashing like this and show an actionable error message instead (e.g. "input is not a correct NAR file. if it is compressed it must be decompressed first." idk). Oh yeah sorry, i thought it did because it was sometimes failing with "Out of memory" messages, which led to me to incorrectly assume that it was taking the little memory available to try to decompress it, and couldn't. Thanks!