Commit graph

13871 commits

Author SHA1 Message Date
Joachim Breitner e4a2a5c074
Documentation: builtins.fetchGit when used on a local path (#7706)
* Documentation:  builtins.fetchGit when used on a local path

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-08 11:53:28 +00:00
Valentin Gagarin ec788964e9
Merge pull request #7626 from hercules-ci/issue-7623 2023-02-08 01:22:23 +01:00
Robert Hensing 40d8955b89
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-08 00:24:28 +01:00
Valentin Gagarin c1921514d4
Merge pull request #7700 from iFreilicht/patch-1
docs: Fix small formatting errors
2023-02-07 23:37:42 +01:00
Eelco Dolstra ae6de012ee
Merge pull request #7692 from edolstra/fix-docker-auto-uid-allocation
Fix auto-uid-allocation in Docker containers
2023-02-07 23:35:39 +01:00
Robert Hensing 19b495a48a NarInfoDiskCache: Also test id consistency with updated fields
And clarify test
2023-02-07 23:34:36 +01:00
Robert Hensing fb94d5cabd NarInfoDiskCache: Keep BinaryCache.id stable and improve test
Fixes #3898

The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.

Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.
2023-02-07 23:34:36 +01:00
Robert Hensing 2ceece3ef3 NarInfoDiskCache: Prepare reproducer for #3898 2023-02-07 23:34:36 +01:00
Robert Hensing 79f62d2dda NarInfoDiskCacheImpl: Make dbPath a parameter
This allows testing with a clean database.
2023-02-07 23:34:36 +01:00
Robert Hensing 29f0b196f4 NarInfoDiskCache: Rename cacheExists -> upToDateCacheExists
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.
2023-02-07 23:34:36 +01:00
Robert Hensing 8a0ef5d58e sqlite.cc: Add SQL tracing
Set environment variable NIX_DEBUG_SQLITE_TRACES=1 to log all sql statements.
2023-02-07 23:34:36 +01:00
Robert Hensing 1a86d3e98e local.mk: Don't log docroot comments
These were accidentally logged and do not need to appear in make's
log output.
2023-02-07 23:34:36 +01:00
David Dunn 9aeaf98c4b
Make install command in documentation compatible with fish shell (#7474)
Use a pipe for all install commands

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2023-02-07 22:32:27 +00:00
Eelco Dolstra fd01b8fc1e
Merge pull request #7780 from edolstra/pr-template
PR template: Ask release notes for incompatible changes, not bug fixes
2023-02-07 23:17:58 +01:00
Felix Uhl d910dfe978 docs: Fix formatting of || operator
This is a workaround for
[mdBook#2000](https://github.com/rust-lang/mdBook/issues/2000)
2023-02-07 23:06:14 +01:00
Felix Uhl 32db5e6349 docs: Fix broken anchor link 2023-02-07 23:06:14 +01:00
Eelco Dolstra 0a70b411e1 Print debug message if a namespace test fails 2023-02-07 23:01:39 +01:00
Eelco Dolstra c5c0617d6f Mention --no-sandbox if sandboxing is unsupported 2023-02-07 22:59:46 +01:00
Eelco Dolstra 4e61877b5c More #ifdef 2023-02-07 22:51:53 +01:00
Eelco Dolstra d834de2894 Fix macOS build 2023-02-07 22:51:53 +01:00
Eelco Dolstra bc1d9fd8b5 Check whether we can use PID namespaces
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
2023-02-07 22:51:53 +01:00
Eelco Dolstra fb2f7f5dcc Fix auto-uid-allocation in Docker containers
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.

So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.

This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
2023-02-07 22:51:53 +01:00
Eelco Dolstra 0a82d2ca49 Ask release notes for incompatible changes, not bug fixes 2023-02-07 22:48:31 +01:00
Eelco Dolstra 9a7dc5d718 Add some tests 2023-02-07 22:46:25 +01:00
Eelco Dolstra 1ba13b17db
Merge pull request #7758 from peterbecich/clang11-devshell-documentation
fix documentation for `clang11Stdenv` dev shell
2023-02-07 22:35:31 +01:00
Eelco Dolstra 7a6daf61e8 Fix activity message 2023-02-07 22:22:50 +01:00
Valentin Gagarin f370a5a26c
Update doc/manual/src/glossary.md
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-02-07 20:40:29 +01:00
Robert Hensing 72b18f05a2 Add a basic daemon authorization test 2023-02-07 16:43:09 +01:00
Valentin Gagarin 8ab9fdac01
Merge pull request #7768 from ncfavier/fixup-7714 2023-02-07 16:24:19 +01:00
Eelco Dolstra a8fe0dc16c Speed up fetching submodules
Previously we would completely refetch the submodules from the
network, even though the repo might already have them. Now we copy the
.git/modules directory from the repo as an optimisation. This speeds
up evaluating

  builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }

(where /path/to/blender already has the needed submodules) from 121s
to 57s.

This is still pretty inefficient and a hack, but a better solution is
best done on the lazy-trees branch.

This change also help in the case where the repo already has the
submodules but the origin is unfetchable for whatever reason
(e.g. there have been cases where Nix in a GitHub action doesn't have
the right authentication set up).
2023-02-07 16:01:36 +01:00
Eelco Dolstra 2edd5cf618 Fix the origin URL used for fetching submodules
We cannot use 'actualUrl', because for file:// repos that's not the
original URL that the repo was fetched from. This is a problem since
submodules may be relative to the original URL.

Fixes e.g.

  nix eval --impure --json --expr 'builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }'

where /path/to/blender is a clone of
https://github.com/blender/blender.git (which has several relative
submodules like '../blender-addons.git').
2023-02-07 16:01:36 +01:00
Eelco Dolstra 81e75e4bf6 Add some progress indication when fetching submodules 2023-02-07 16:01:36 +01:00
Eelco Dolstra ffa9feee88
Merge pull request #7766 from NixOS/dependabot/github_actions/cachix/install-nix-action-19
Bump cachix/install-nix-action from 18 to 19
2023-02-07 14:00:16 +01:00
Eelco Dolstra 16929b0c73
Merge pull request #7767 from NixOS/backport-pr-name
Backport PR title: Bot-based -> Automatic
2023-02-07 13:59:22 +01:00
Valentin Gagarin 0d73313c55
Merge pull request #7318 from fricklerhandwerk/doc-instantiate
add "instantiate" to glossary
2023-02-07 12:46:11 +01:00
Valentin Gagarin c20394245d add "instantiate" to glossary 2023-02-07 12:09:26 +01:00
Yorick 631ba6442a
build-remote: store maxBuildJobs before forcing it to 1 2023-02-07 12:08:00 +01:00
Valentin Gagarin e4f12ff482
store paths are not uniquely determined 2023-02-07 11:59:18 +01:00
Naïm Favier 2915db7b28
doc: fixup 7714 2023-02-07 11:38:09 +01:00
Valentin Gagarin 9670cac1ab
Backport PR title: Bot-based -> Automatic
In hope to reduce friction in contributor-facing communication, use a more common word. "bot" needless jargon for "something the machine does itself".
2023-02-07 11:11:50 +01:00
dependabot[bot] 6fdce7a9df
Bump cachix/install-nix-action from 18 to 19
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 18 to 19.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Commits](https://github.com/cachix/install-nix-action/compare/v18...v19)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-06 22:00:49 +00:00
Yorick 3050005211
build-remote: don't warn when all local build slots are taken
Previously, build-remote would show a warning if all build slots were
taken, even if they would open up later. This caused a lot of spam in
the logs. Disable this warning when maxJobs > 0.

See #6263
2023-02-06 17:53:03 +01:00
Peter Becich b8e96351f7
fix clang11Stdenv dev shell documentation
`clang11StdenvPackages` does not exist

```
│   └───x86_64-linux
│       ├───ccacheStdenv: development environment 'nix'
│       ├───clang11Stdenv: development environment 'nix'
│       ├───clangStdenv: development environment 'nix'
│       ├───default: development environment 'nix'
│       ├───gccStdenv: development environment 'nix'
│       ├───libcxxStdenv: development environment 'nix'
│       └───stdenv: development environment 'nix'
```
2023-02-05 12:00:48 -08:00
John Ericson 44bea52ae3 Scope down --derivation to just the commands that use it
Per the old FIXME, this flag was on too many commands, and mostly
ignored. Now it is just on the commands where it actually has an effect.

Per https://github.com/NixOS/nix/issues/7261, I would still like to get
rid of it entirely, but that is a separate project. This change should
be good with or without doing that.
2023-02-04 18:30:02 -05:00
John Ericson 6352e20bc8 Remove --derivation from test
It doesn't do anything here, and in the next commit `show-derivation
will no longer accept this flag.
2023-02-04 18:30:02 -05:00
Robert Hensing 895dfc656a
Merge pull request #7705 from Ma27/fix-initNix-in-perl-bindings
perl: run `initLibStore()` on `openStore()`
2023-02-03 23:33:41 +01:00
John Ericson 45fa297e40 Factor out InstallableStorePath to its own file, dedup
`nix app` had something called `InstallableDerivedPath` which is
actually the same thing. We go with the later's name because it has
become more correct.

I originally did this change (more hurriedly) as part of #6225 --- a
mini store-only Nix and a full Nix need to share this code. In the first
RFC meeting for https://github.com/NixOS/rfcs/pull/134 we discussed how
some splitting of the massive `installables.cc` could begin prior, as
that is a good thing anyways. (@edolstra's words, not mine!) This would
be one such PR.
2023-02-03 11:26:39 -05:00
Jörg Thalheim f20d3726dd advertise transport encoding in http transfers to
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.

Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.

This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.

It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.

I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.

The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)

```nix
{
  services.nginx.virtualHosts."cache.yourhost.com" = {
    locations."/".extraConfig = ''
      proxy_pass http://127.0.0.1:5000;
      proxy_set_header Host $host;
      proxy_redirect http:// https://;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

      zstd on;
      zstd_types application/x-nix-archive;
    '';
  };
}
```

For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.

Before:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty

real    0m18,375s
user    0m2,889s
sys     0m1,558s
```

After:

```console
$ nix build && rm -rf /tmp/hello  && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'

real    0m11,884s
user    0m4,130s
sys     0m1,439s
```

Signed-off-by: Jörg Thalheim <joerg@thalheim.io>

Update src/libstore/filetransfer.cc

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2023-02-03 12:33:38 +00:00
Eelco Dolstra dbe0748f97
Merge pull request #7739 from obsidiansystems/user-settings
Move `trustedUsers` and `allowedUsers` to separate config struct
2023-02-03 11:55:37 +01:00
Eelco Dolstra 1647354d6b
Merge pull request #7738 from obsidiansystems/delete-authHook
Get rid of the `authHook` parameter on `processConnection`
2023-02-03 11:54:05 +01:00