[Nix#4533] Nix cached truncated files downloaded #140

New issue

Open

opened 2024-03-16 06:45:08 +00:00 by lix-bot · 0 comments

lix-bot commented

2024-03-16 06:45:08 +00:00

Member

Upstream-Issue: NixOS/nix#4533

Describe the bug

In cases of some network error or proxy server error, downloading is ended too early and gives a truncated file. But the truncated file is still cached by nix, leading to immediate truncated gzip input every next time without refetching.

This issue happens because:

The response of github tarball url does NOT contain Content-Length, which makes curl not possible to validate the output length.
Some network error like proxy server being killed, will not RESET the connection. This makes curl returns zero but gives truncated file.
Nix caches the output after curl returns zero, but before extracting. It results in cached truncated files.

Steps To Reproduce

Setup a proxy server.
Run nix-build -A osu-lazer https://github.com/r-ryantm/nixpkgs/archive/cc77f910fc1a06b7cb7eb43639c5904540483c70.tar.gz or with other github archive URL.
During the download, kill the proxy server. nix will fail with truncated gzip input
Run the command again, it immediately fails with truncated gzip input again without network access.

Expected behavior
Nix caches the downloaded file only if it can be unpacked successfully.
So it can re-download the file and re-build in the next time.

nix-env --version output
nix-env (Nix) 2.4pre20201205_a5d85d0

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/4533 **Describe the bug** In cases of some network error or proxy server error, downloading is ended too early and gives a truncated file. But the truncated file is still cached by nix, leading to immediate `truncated gzip input` every next time without refetching. This issue happens because: 1. The response of github tarball url does **NOT** contain `Content-Length`, which makes `curl` not possible to validate the output length. 2. Some network error like proxy server being killed, will not RESET the connection. This makes `curl` returns zero but gives truncated file. 2. Nix caches the output after `curl` returns zero, but before extracting. It results in cached truncated files. **Steps To Reproduce** 1. Setup a proxy server. 2. Run `nix-build -A osu-lazer https://github.com/r-ryantm/nixpkgs/archive/cc77f910fc1a06b7cb7eb43639c5904540483c70.tar.gz` or with other github archive URL. 3. During the download, kill the proxy server. `nix` will fail with `truncated gzip input` 4. Run the command again, it immediately fails with `truncated gzip input` again without network access. <img width="482" alt="image" src="https://user-images.githubusercontent.com/14816024/107239711-19f9b080-6a64-11eb-8111-226d7dc41247.png"> **Expected behavior** Nix caches the downloaded file only if it can be unpacked successfully. So it can re-download the file and re-build in the next time. **`nix-env --version` output** `nix-env (Nix) 2.4pre20201205_a5d85d0`