Commit graph

13660 commits

Author SHA1 Message Date
Jörg Thalheim f20d3726dd advertise transport encoding in http transfers to
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-03 12:33:38 +00:00
Eelco Dolstra dbe0748f97
Merge pull request #7739 from obsidiansystems/user-settings
Move `trustedUsers` and `allowedUsers` to separate config struct
2023-02-03 11:55:37 +01:00
Eelco Dolstra 1647354d6b
Merge pull request #7738 from obsidiansystems/delete-authHook
Get rid of the `authHook` parameter on `processConnection`
2023-02-03 11:54:05 +01:00
John Ericson a47e055e09 Move trustedUsers and allowedUsers to separate config struct
These settings are not needed for libstore at all, they are just used by
the nix daemon *command* for authorization on unix domain sockets. My
moving them to a new configuration struct just in that file, we avoid
them leaking anywhere else.

Also, it is good to break up the mammoth `Settings` struct in general.
Issue #5638 tracks this.

The message is not changed because I do not want to regress in
convenience to the user. Just saying "this connection is not trusted"
doesn't tell them out to fix the issue. The ideal thing to do would be
to somehow parameterize `processCommand` on how the error should be
displayed, so different sorts of connections can display different
information to the user based on how authentication is performed for the
connection in question. This, however, is a good bit more work, so it is
left for the future.

This came up with me thinking about the tcp:// store (#5265). The larger
project is not TCP *per se*, but the idea that it should be possible for
something else to manage access control to services like the Nix Daemon,
and those services simply trust or trust the incoming connection as they
are told. This is a more capability-oriented way of thinking about trust
than "every server implements its own auth separately" as we are used to today.

Its very great that libstore itself already implements just this model,
and so via this refactor I basically want to "enshrine" that so it
continues to be the case.
2023-02-02 14:17:24 -05:00
John Ericson 479c011784 Get rid of the authHook parameter on processConnection
This is (morally) dead code.

As @edolstra pointed out in
https://github.com/NixOS/nix/pull/5226#discussion_r1073470813, this is
no longer needed.

I created this in 8d4162ff9e, so it is
fitting that I now destroy it :).
2023-02-02 12:02:03 -05:00
Eelco Dolstra 5cbeff64f2
Merge pull request #7607 from Hoverbear/installation-expand
Expand installation.md
2023-02-02 17:05:54 +01:00
Maximilian Bosch 51013da921
perl: run initLibStore() on openStore()
Since #7478 it's mandatory that `initLibStore()` is called for store
operations. However that's not the case when running `openStore()` in
Perl using the perl-bindings. That breaks e.g. `hydra-eval-jobset` when
built against Nix 2.13 which uses small portions of the store API.
2023-02-02 15:56:00 +01:00
Eelco Dolstra b574c70ccb
Merge pull request #7736 from shlevy/plugin-files-daemon
Don't send plugin-files to the daemon.
2023-02-02 12:35:51 +01:00
Shea Levy 895c525d04
daemon: Warn on old clients passing unexpected plugin-files.
The setting itself was already ignored due to exception trying to set pluginFiles.
2023-02-02 06:03:45 -05:00
Théophane Hufschmitt cde2e2a404
Merge pull request #7734 from Sciencentistguy/root-red-ps1
nix-shell: Colour the prompt red if the user is root
2023-02-02 10:54:29 +01:00
Shea Levy 92edc38369
Don't send plugin-files to the daemon.
This is radically unsafe and the daemon has already loaded its plugins
anyway.

Fixes cachix/devenv#276
2023-02-01 20:05:56 -05:00
Jamie Quigley 32ca59649b
nix-shell: Colour the prompt red if the user is root
This matches the nixos prompt colours - green for standard user, red for
root
2023-02-01 20:50:44 +00:00
Eelco Dolstra 119ba50eb8
Merge pull request #7733 from edolstra/clang-c++20
Fix C++20 compilation with clang
2023-02-01 21:15:34 +01:00
Eelco Dolstra e8ca49f6ef Fix clang compilation 2023-02-01 20:34:44 +01:00
Eelco Dolstra 57a4258426 Remove an unused capture 2023-02-01 20:27:35 +01:00
Eelco Dolstra 7a09bfbcb6
Merge pull request #7723 from yorickvP/nix-store-ping-json
nix store ping: add --json flag
2023-02-01 17:11:34 +01:00
Théophane Hufschmitt 518da6c6a3
Merge pull request #7716 from obsidiansystems/small-storePath-cleanups
Separate `path.hh` from `content-address.hh`
2023-02-01 16:00:28 +01:00
Eelco Dolstra b55a946d8d
Merge pull request #7717 from obsidiansystems/delete-dead-code
Delete dead code
2023-02-01 15:57:04 +01:00
Eelco Dolstra ed9d7c83db
Merge pull request #7718 from obsidiansystems/fix-rapidcheck-autoconf
Fix configure.ac rapidcheck tests
2023-02-01 15:56:35 +01:00
Théophane Hufschmitt e32c5c2c77
Merge pull request #7667 from dramforever/flake-search-attr
Better error message for nix search when attr is not found
2023-02-01 15:56:22 +01:00
Eelco Dolstra 6f0436a337
Merge pull request #7719 from andersk/manual-timestamp
manual: Document that the store timestamp is now 1, not 0
2023-02-01 15:50:36 +01:00
Eelco Dolstra 845b7f067d
Merge pull request #7726 from hercules-ci/flake-show-when-empty-hide
nix flake show: Ignore empty attrsets
2023-02-01 15:49:13 +01:00
Eelco Dolstra 14b0b9ea5a
Merge pull request #7203 from graham33/feature/cpp20
Proposal: Use C++20
2023-02-01 15:41:04 +01:00
Robert Hensing 60d48eda23 nix flake show: Ignore empty attrsets
For frameworks it's important that structures are as lazy as possible
to prevent infinite recursions, performance issues and errors that
aren't related to the thing to evaluate. As a consequence, they have
to emit more attributes than strictly (sic) necessary.
However, these attributes with empty values are not useful to the user
so we omit them.
2023-01-31 18:20:26 +01:00
Yorick 4757b3f04e
tests/store-ping: test nix store ping --json 2023-01-31 15:10:54 +01:00
Yorick 4f4a6074e4
nix store ping: try to print json if connect() fails aswell 2023-01-31 15:10:39 +01:00
Théophane Hufschmitt 0079d29437
Merge pull request #7714 from ncfavier/doc-structuredAttrs
doc: add `__structuredAttrs`, `outputChecks`, `unsafeDiscardReferences`
2023-01-31 14:43:58 +01:00
Yorick 28648ed784
nix store ping: add --json flag 2023-01-31 13:24:23 +01:00
Anders Kaseorg c5d4c50aba manual: Document that the store timestamp is now 1, not 0
Commit 14bc3ce3d6 (0.13~43) changed the
timestamps in the Nix store from 0 to 1.  Update the nix-store man
page to match.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2023-01-30 23:51:35 -08:00
Robert Hensing c9b9260f34
Merge pull request #7713 from obsidiansystems/more-rapid-check
Add more property tests
2023-01-30 18:54:53 +01:00
John Ericson 16fa8c2e0e Fix configure.ac rapidcheck tests
- `AC_LANG_PUSH(C++)` is needed for the header check

- The library check is hopeless (without lots of third-party macros I
  don't feel like getting) because name mangling

Pkg-config would make all this easier. I previously opened
https://github.com/emil-e/rapidcheck/issues/302, I should write a PR
too.
2023-01-30 12:08:57 -05:00
John Ericson e21aa43212 Delete dead code
The references set seems to have been unused since `LegacySSHStore`
references were first created in
caa5793b4a.

The method decls never were upstream, and accidentally added by me in
062533f7cd (probably due to `git rerere`).
Sorry!

This reduces the diff from #3746.
2023-01-30 11:29:01 -05:00
dramforever 6b779e4b07 Fix extra "." in CmdSearch::getDefaultFlakeAttrPaths
No other getDefaultFlakeAttrPaths implementation has this trailing dot,
and the dot can show up in error messages like:

  error: flake '...' does not provide attribute 'packages.x86_64-linux.', ...
2023-01-31 00:04:05 +08:00
dramforever 377d5eb388 Installable::getCursors: Cleanup
- Clarify doc comments, Installables::getCursors returns non-empty
  vector
- Use vector::at in Installable::getCursor instead of checking for empty
  vector and throwing an exception with error message.
2023-01-31 00:04:05 +08:00
dramforever b26562c629 InstallableFlake: Handle missing attr in getCursors
Handle the case where none of getActualAttrPaths() actually exists,
in which case instead of returning an empty vector.

This fixes the case where the user misspells the attribute name in nix
search. Instead of getting no search results, now it shows an error with
suggestions.

Also remove InstallableFlake::getCursor() override since it's now
equivalent to the base class version.
2023-01-30 23:59:09 +08:00
John Ericson 560142fec0 Make per-variant Arbitrary impls too
This is a nice idea that @roberth requested. If we could factor our a
generic `std::variant` impl as a follow-up it would be even better!
2023-01-30 10:56:00 -05:00
John Ericson 02e745ba5b Separate path.hh from content-address.hh
It is good to separate concerns; `StorePath` (in general) has nothing to
do with `ContentAddress` anyways.

This reduces the diff from #3746.
2023-01-30 10:14:03 -05:00
Robert Hensing a31d7d4e5e
Merge pull request #7715 from obsidiansystems/small-storePath-cleanups
Avoid some `StorePath` <-> `Path` round trips
2023-01-30 16:12:19 +01:00
John Ericson f3e272ba02 Avoid some StorePath <-> Path round trips
Avoid needless work and throwing away invariants.

These conversions date back to when `StorePath` was in Rust and there
were issues with it missing utility methods.
2023-01-30 09:37:57 -05:00
Naïm Favier 0b3464a107
doc: add __structuredAttrs, outputChecks, unsafeDiscardReferences 2023-01-30 14:49:45 +01:00
Eelco Dolstra c79b1582a7
Merge pull request #5226 from NixOS/client-side-profiles
Move the default profiles to the user’s home
2023-01-30 12:21:47 +01:00
Théophane Hufschmitt 575d0aea5d
Merge pull request #6988 from max-privatevoid/pr-flake-show-foreign
nix flake show: don't evaluate derivations for foreign systems by default
2023-01-30 12:06:37 +01:00
Théophane Hufschmitt 3ac9f1658a Fix the flakes init test
Things leading to another...
2023-01-30 11:21:52 +01:00
Théophane Hufschmitt de1b593233
Merge pull request #7087 from ncfavier/referenceablePaths
Self-contained outputs
2023-01-30 11:06:54 +01:00
Théophane Hufschmitt bc6e65e26f Fix the flakes/show test
Don't hardcode “x86_64-linux” as this won't work too nicely on other
platforms
2023-01-30 10:44:10 +01:00
Théophane Hufschmitt 7cd08ae379
Merge pull request #7645 from typetetris/fix-url-parsing-file-as-application-scheme
Fix url parsing for urls using `file+`
2023-01-30 10:42:03 +01:00
Théophane Hufschmitt 7408776b00 Fix the release-notes
Slightly butchered during the merge
2023-01-30 10:32:23 +01:00
Théophane Hufschmitt 4aaf0ee52e
Merge branch 'master' into referenceablePaths 2023-01-30 10:31:00 +01:00
Théophane Hufschmitt d70b890488
Merge pull request #7689 from ncfavier/nix-path-restrict-eval
getDefaultNixPath: actually respect `{restrict,pure}-eval`
2023-01-30 10:03:17 +01:00
John Ericson ecd3e4ebd7 More property tests
Also put proper comparison methods on `DerivedPath` and
`NixStringContextElem`, which is needed for the tests but good in
general.
2023-01-29 17:09:59 -05:00