Find a file
Jörg Thalheim f20d3726dd advertise transport encoding in http transfers to
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-03 12:33:38 +00:00
.github Merge pull request #7648 from hercules-ci/move-nixos-tests 2023-01-27 15:11:48 +01:00
config Run autoupdate 2021-06-01 11:42:38 +02:00
contrib function-trace: always show the trace 2019-09-18 23:23:21 +02:00
doc/manual Update doc/manual/src/release-notes/rl-2.13.md 2023-01-25 08:58:41 -05:00
m4 autoconf: Fix C++17 detection not working on Ubuntu 16.04. 2019-07-03 04:32:25 +02:00
maintainers state priorities in triaging and discussion process 2023-01-12 19:33:41 +01:00
misc Merge pull request #6645 from Artturin/limitinfinity 2022-12-09 13:36:13 +01:00
mk Make ./mk/run-test.sh work by itself; add mk/debug-test.sh 2022-12-21 02:28:33 -05:00
perl Remove std::string alias (for real this time) 2022-02-25 16:13:02 +01:00
scripts Add escape for systemd service in installer script 2023-01-14 03:46:11 +08:00
src advertise transport encoding in http transfers to 2023-02-03 12:33:38 +00:00
tests Merge pull request #7648 from hercules-ci/move-nixos-tests 2023-01-27 15:11:48 +01:00
.dir-locals.el .dir-locals.el: Set c-block-comment-prefix 2020-07-10 11:21:06 +02:00
.editorconfig Add .editorconfig 2017-06-05 22:57:28 +01:00
.gitignore Add fish suport to installer 2022-09-13 12:56:16 -04:00
.version Bump version 2023-01-17 22:08:36 +01:00
boehmgc-coroutine-sp-fallback.diff Update boehmgc-coroutine-sp-fallback.diff 2023-01-24 15:11:55 +01:00
bootstrap.sh bootstrap: Simplify & make more robust. 2011-09-06 12:11:05 +00:00
configure.ac Add rapidcheck dependency for testing 2023-01-23 07:05:50 -05:00
COPYING * Change this to LGPL to keep the government happy. 2006-04-25 16:41:06 +00:00
default.nix Remove url literals 2022-01-24 13:28:21 +01:00
docker.nix Add nobody user/group to Nix docker image 2022-11-29 10:01:46 +00:00
flake.lock flake.lock: Update 2022-12-08 17:09:50 +01:00
flake.nix Merge pull request #7648 from hercules-ci/move-nixos-tests 2023-01-27 15:11:48 +01:00
local.mk Remove 'dist' target 2020-12-03 16:17:58 +01:00
Makefile Merge branch 'master' into lto 2022-05-25 11:55:13 +00:00
Makefile.config.in Embed the sandbox shell into the statically linked 'nix' binary 2022-06-23 04:08:28 +02:00
precompiled-headers.h Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
README.md Fix link to hacking doc 2022-07-30 09:12:50 -07:00
shell.nix Remove url literals 2022-01-24 13:28:21 +01:00

Nix

Open Collective supporters Test

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. Please refer to the Nix manual for more details.

Installation

On Linux and macOS the easiest way to install Nix is to run the following shell command (as a user other than root):

$ curl -L https://nixos.org/nix/install | sh

Information on additional installation methods is available on the Nix download page.

Building And Developing

See our Hacking guide in our manual for instruction on how to build nix from source with nix-build or how to get a development environment.

Additional Resources

License

Nix is released under the LGPL v2.1.