Erroneous download size for compressed nars #967

New issue

Open

opened 2025-08-19 11:29:28 +00:00 by tom-hubrecht · 3 comments

tom-hubrecht commented

2025-08-19 11:29:28 +00:00

Member

Describe the bug

The download progress for compressed nars is erroneous when the nar is compressed on the fly.

Steps To Reproduce

Use snix-cache
Build a cached derivation that is quite large (e.g. a nixpkgs revision)
Fetch it

Then, the download size is shown to be the same as the unpacked size, even though the nar is zstd-compressed on the fly by the cache.

C.f.

(This does not happen if the nars are pre-compressed and the downloaded files are ...nar.zstd for example)

Expected behavior

Show the correct download size (e.g. the size of the compressed file)

`nix --version` output

nix (Lix, like Nix) 2.94.0-dev-8bbd5e1d0df9c31b4d86ba07bc85beb952e42ccb
System type: x86_64-linux
Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/thubrecht/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/thubrecht/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/thubrecht/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/thubrecht/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/store/wzvcw93ncl4cnw7062mn77q4ym2hmxb1-gnome-settings-daemon-48.1/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/9zi2cvapl6v92dbk4czfwqfq5h68y6d9-lix-2.94.0-dev-8bbd5e1d0df9c31b4d86ba07bc85beb952e42ccb/share

Additional context

Add any other context about the problem here.

## Describe the bug The download progress for compressed nars is erroneous when the nar is compressed on the fly. ## Steps To Reproduce - Use [snix-cache](https://git.dgnum.eu/DGNum/snix-cache) - Build a cached derivation that is quite large (e.g. a nixpkgs revision) - Fetch it Then, the download size is shown to be the same as the unpacked size, even though the nar is zstd-compressed on the fly by the cache. C.f. ![image](/attachments/10162da9-f49b-497d-bfc3-863c93f38d31) (This does not happen if the nars are pre-compressed and the downloaded files are ...nar.zstd for example) ## Expected behavior Show the correct download size (e.g. the size of the compressed file) ## `nix --version` output nix (Lix, like Nix) 2.94.0-dev-8bbd5e1d0df9c31b4d86ba07bc85beb952e42ccb System type: x86_64-linux Additional system types: i686-linux, x86_64-v1-linux, x86_64-v2-linux, x86_64-v3-linux Features: gc, signed-caches System configuration file: /etc/nix/nix.conf User configuration files: /home/thubrecht/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/thubrecht/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/thubrecht/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/thubrecht/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf:/nix/store/wzvcw93ncl4cnw7062mn77q4ym2hmxb1-gnome-settings-daemon-48.1/etc/xdg/nix/nix.conf Store directory: /nix/store State directory: /nix/var/nix Data directory: /nix/store/9zi2cvapl6v92dbk4czfwqfq5h68y6d9-lix-2.94.0-dev-8bbd5e1d0df9c31b4d86ba07bc85beb952e42ccb/share ## Additional context Add any other context about the problem here.

image.png

15 KiB

tom-hubrecht added the

bug

label

2025-08-19 11:29:28 +00:00

raito commented

2025-08-19 11:41:54 +00:00

Owner

I think this is a snix cache bug IIRC cc @flokli as snix doesn't communicate the right sizes in its narinfo or its responses.

flokli commented

2025-08-19 11:56:17 +00:00

We communicate the right sizes. The zstd compression is only happening via accept-encoding on the transport level, content-length and FileSize are equal to the uncompressed NAR size.

👍 1

pennae commented

2025-08-19 18:57:26 +00:00

Owner

we'd just delete the "downloaded" progress meters entirely. they're pretty much meaningless since curl decompresses some things but not others, and in general they're not indicative of the overall progress of the operation anyway. a meter for current overall transfer rate might be a lot more useful (but entirely impossible because the wire protocol is complete garbage and cannot be changed).

we'd just delete the "downloaded" progress meters entirely. they're pretty much meaningless since curl decompresses some things but not others, and in general they're not indicative of the overall progress of the operation *anyway*. a meter for current overall transfer rate might be a lot more useful (but entirely impossible because the wire protocol is complete garbage and cannot be changed).

👍 1