Commit graph

13597 commits

Author SHA1 Message Date
Théophane Hufschmitt 5597d68e2d
Merge pull request #7754 from obsidiansystems/narrower-scope-derivation-flag
Scope down `--derivation` to just the commands that use it
2023-02-09 19:51:43 +01:00
Eelco Dolstra 0a7071ed33
Merge pull request #7774 from edolstra/submodule-fixes
Git submodule fixes
2023-02-09 17:19:48 +01:00
Eelco Dolstra e46429f674
Merge pull request #7712 from Mic92/advertise-compressions
advertise transport encoding in http transfers to
2023-02-09 17:15:25 +01:00
Eelco Dolstra 862e56c23d
Improve comment
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-02-09 16:42:45 +01:00
Eelco Dolstra 15313bfdb7
Fix activity message
Co-authored-by: Josef Kemetmüller <josef.kemetmueller@gmail.com>
2023-02-09 16:42:14 +01:00
Joachim Breitner e4a2a5c074
Documentation: builtins.fetchGit when used on a local path (#7706)
* Documentation:  builtins.fetchGit when used on a local path

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-08 11:53:28 +00:00
Valentin Gagarin ec788964e9
Merge pull request #7626 from hercules-ci/issue-7623 2023-02-08 01:22:23 +01:00
Robert Hensing 40d8955b89
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-08 00:24:28 +01:00
Valentin Gagarin c1921514d4
Merge pull request #7700 from iFreilicht/patch-1
docs: Fix small formatting errors
2023-02-07 23:37:42 +01:00
Eelco Dolstra ae6de012ee
Merge pull request #7692 from edolstra/fix-docker-auto-uid-allocation
Fix auto-uid-allocation in Docker containers
2023-02-07 23:35:39 +01:00
David Dunn 9aeaf98c4b
Make install command in documentation compatible with fish shell (#7474)
Use a pipe for all install commands

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-07 22:32:27 +00:00
Eelco Dolstra fd01b8fc1e
Merge pull request #7780 from edolstra/pr-template
PR template: Ask release notes for incompatible changes, not bug fixes
2023-02-07 23:17:58 +01:00
Felix Uhl d910dfe978 docs: Fix formatting of || operator
This is a workaround for
[mdBook#2000](https://github.com/rust-lang/mdBook/issues/2000)
2023-02-07 23:06:14 +01:00
Felix Uhl 32db5e6349 docs: Fix broken anchor link 2023-02-07 23:06:14 +01:00
Eelco Dolstra 0a70b411e1 Print debug message if a namespace test fails 2023-02-07 23:01:39 +01:00
Eelco Dolstra c5c0617d6f Mention --no-sandbox if sandboxing is unsupported 2023-02-07 22:59:46 +01:00
Eelco Dolstra 4e61877b5c More #ifdef 2023-02-07 22:51:53 +01:00
Eelco Dolstra d834de2894 Fix macOS build 2023-02-07 22:51:53 +01:00
Eelco Dolstra bc1d9fd8b5 Check whether we can use PID namespaces
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
2023-02-07 22:51:53 +01:00
Eelco Dolstra fb2f7f5dcc Fix auto-uid-allocation in Docker containers
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.

So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.

This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
2023-02-07 22:51:53 +01:00
Eelco Dolstra 0a82d2ca49 Ask release notes for incompatible changes, not bug fixes 2023-02-07 22:48:31 +01:00
Eelco Dolstra 9a7dc5d718 Add some tests 2023-02-07 22:46:25 +01:00
Eelco Dolstra 1ba13b17db
Merge pull request #7758 from peterbecich/clang11-devshell-documentation
fix documentation for `clang11Stdenv` dev shell
2023-02-07 22:35:31 +01:00
Eelco Dolstra 7a6daf61e8 Fix activity message 2023-02-07 22:22:50 +01:00
Valentin Gagarin 8ab9fdac01
Merge pull request #7768 from ncfavier/fixup-7714 2023-02-07 16:24:19 +01:00
Eelco Dolstra a8fe0dc16c Speed up fetching submodules
Previously we would completely refetch the submodules from the
network, even though the repo might already have them. Now we copy the
.git/modules directory from the repo as an optimisation. This speeds
up evaluating

  builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }

(where /path/to/blender already has the needed submodules) from 121s
to 57s.

This is still pretty inefficient and a hack, but a better solution is
best done on the lazy-trees branch.

This change also help in the case where the repo already has the
submodules but the origin is unfetchable for whatever reason
(e.g. there have been cases where Nix in a GitHub action doesn't have
the right authentication set up).
2023-02-07 16:01:36 +01:00
Eelco Dolstra 2edd5cf618 Fix the origin URL used for fetching submodules
We cannot use 'actualUrl', because for file:// repos that's not the
original URL that the repo was fetched from. This is a problem since
submodules may be relative to the original URL.

Fixes e.g.

  nix eval --impure --json --expr 'builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }'

where /path/to/blender is a clone of
https://github.com/blender/blender.git (which has several relative
submodules like '../blender-addons.git').
2023-02-07 16:01:36 +01:00
Eelco Dolstra 81e75e4bf6 Add some progress indication when fetching submodules 2023-02-07 16:01:36 +01:00
Eelco Dolstra ffa9feee88
Merge pull request #7766 from NixOS/dependabot/github_actions/cachix/install-nix-action-19
Bump cachix/install-nix-action from 18 to 19
2023-02-07 14:00:16 +01:00
Eelco Dolstra 16929b0c73
Merge pull request #7767 from NixOS/backport-pr-name
Backport PR title: Bot-based -> Automatic
2023-02-07 13:59:22 +01:00
Valentin Gagarin 0d73313c55
Merge pull request #7318 from fricklerhandwerk/doc-instantiate
add "instantiate" to glossary
2023-02-07 12:46:11 +01:00
Valentin Gagarin c20394245d add "instantiate" to glossary 2023-02-07 12:09:26 +01:00
Naïm Favier 2915db7b28
doc: fixup 7714 2023-02-07 11:38:09 +01:00
Valentin Gagarin 9670cac1ab
Backport PR title: Bot-based -> Automatic
In hope to reduce friction in contributor-facing communication, use a more common word. "bot" needless jargon for "something the machine does itself".
2023-02-07 11:11:50 +01:00
dependabot[bot] 6fdce7a9df
Bump cachix/install-nix-action from 18 to 19
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 18 to 19.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Commits](https://github.com/cachix/install-nix-action/compare/v18...v19)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 22:00:49 +00:00
Peter Becich b8e96351f7
fix clang11Stdenv dev shell documentation
`clang11StdenvPackages` does not exist

```
│   └───x86_64-linux
│       ├───ccacheStdenv: development environment 'nix'
│       ├───clang11Stdenv: development environment 'nix'
│       ├───clangStdenv: development environment 'nix'
│       ├───default: development environment 'nix'
│       ├───gccStdenv: development environment 'nix'
│       ├───libcxxStdenv: development environment 'nix'
│       └───stdenv: development environment 'nix'
```
2023-02-05 12:00:48 -08:00
John Ericson 44bea52ae3 Scope down --derivation to just the commands that use it
Per the old FIXME, this flag was on too many commands, and mostly
ignored. Now it is just on the commands where it actually has an effect.

Per https://github.com/NixOS/nix/issues/7261, I would still like to get
rid of it entirely, but that is a separate project. This change should
be good with or without doing that.
2023-02-04 18:30:02 -05:00
John Ericson 6352e20bc8 Remove --derivation from test
It doesn't do anything here, and in the next commit `show-derivation
will no longer accept this flag.
2023-02-04 18:30:02 -05:00
Robert Hensing 895dfc656a
Merge pull request #7705 from Ma27/fix-initNix-in-perl-bindings
perl: run `initLibStore()` on `openStore()`
2023-02-03 23:33:41 +01:00
Jörg Thalheim f20d3726dd advertise transport encoding in http transfers to
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-03 12:33:38 +00:00
Eelco Dolstra dbe0748f97
Merge pull request #7739 from obsidiansystems/user-settings
Move `trustedUsers` and `allowedUsers` to separate config struct
2023-02-03 11:55:37 +01:00
Eelco Dolstra 1647354d6b
Merge pull request #7738 from obsidiansystems/delete-authHook
Get rid of the `authHook` parameter on `processConnection`
2023-02-03 11:54:05 +01:00
John Ericson a47e055e09 Move trustedUsers and allowedUsers to separate config struct
These settings are not needed for libstore at all, they are just used by
the nix daemon *command* for authorization on unix domain sockets. My
moving them to a new configuration struct just in that file, we avoid
them leaking anywhere else.

Also, it is good to break up the mammoth `Settings` struct in general.
Issue #5638 tracks this.

The message is not changed because I do not want to regress in
convenience to the user. Just saying "this connection is not trusted"
doesn't tell them out to fix the issue. The ideal thing to do would be
to somehow parameterize `processCommand` on how the error should be
displayed, so different sorts of connections can display different
information to the user based on how authentication is performed for the
connection in question. This, however, is a good bit more work, so it is
left for the future.

This came up with me thinking about the tcp:// store (#5265). The larger
project is not TCP *per se*, but the idea that it should be possible for
something else to manage access control to services like the Nix Daemon,
and those services simply trust or trust the incoming connection as they
are told. This is a more capability-oriented way of thinking about trust
than "every server implements its own auth separately" as we are used to today.

Its very great that libstore itself already implements just this model,
and so via this refactor I basically want to "enshrine" that so it
continues to be the case.
2023-02-02 14:17:24 -05:00
John Ericson 479c011784 Get rid of the authHook parameter on processConnection
This is (morally) dead code.

As @edolstra pointed out in
https://github.com/NixOS/nix/pull/5226#discussion_r1073470813, this is
no longer needed.

I created this in 8d4162ff9e, so it is
fitting that I now destroy it :).
2023-02-02 12:02:03 -05:00
Eelco Dolstra 5cbeff64f2
Merge pull request #7607 from Hoverbear/installation-expand
Expand installation.md
2023-02-02 17:05:54 +01:00
Maximilian Bosch 51013da921
perl: run initLibStore() on openStore()
Since #7478 it's mandatory that `initLibStore()` is called for store
operations. However that's not the case when running `openStore()` in
Perl using the perl-bindings. That breaks e.g. `hydra-eval-jobset` when
built against Nix 2.13 which uses small portions of the store API.
2023-02-02 15:56:00 +01:00
Eelco Dolstra b574c70ccb
Merge pull request #7736 from shlevy/plugin-files-daemon
Don't send plugin-files to the daemon.
2023-02-02 12:35:51 +01:00
Shea Levy 895c525d04
daemon: Warn on old clients passing unexpected plugin-files.
The setting itself was already ignored due to exception trying to set pluginFiles.
2023-02-02 06:03:45 -05:00
Théophane Hufschmitt cde2e2a404
Merge pull request #7734 from Sciencentistguy/root-red-ps1
nix-shell: Colour the prompt red if the user is root
2023-02-02 10:54:29 +01:00
Shea Levy 92edc38369
Don't send plugin-files to the daemon.
This is radically unsafe and the daemon has already loaded its plugins
anyway.

Fixes cachix/devenv#276
2023-02-01 20:05:56 -05:00