Commit graph

3100 commits

Author SHA1 Message Date
Eelco Dolstra
ef34fd0656 scanForReferences(): Use a StorePathSet 2021-10-04 13:47:38 +02:00
621aa65325 local-derivation-goal.cc: downgrade "warning: rewriting hashes in..." down to debug
Before the changes when building the whole system with
`contentAddressedByDefault = true;` we get many noninformative messages:

    $ nix build -f nixos system --keep-going
    ...
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-python'; cross fingers
    error: 2 dependencies of derivation '/nix/store/...-hub-2.14.2.drv' failed to build
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-man'; cross fingers
    ...

Let's downgrade these messages down to debug().
2021-10-01 17:41:44 +01:00
Eelco Dolstra
7cc220825d
Merge pull request #5167 from Ma27/keep-failed-on-ssh-remote-build
nix-store --serve: pass on `settings.keepFailed` from SSH store
2021-10-01 16:35:02 +02:00
John Ericson
242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Eelco Dolstra
6b5efeafd1
Merge pull request #5295 from Ma27/bmcheck-fod-mismatch
build: also throw hash-mismatch errors if `buildMode == bmCheck`
2021-09-29 10:13:45 +02:00
bb1a851bcf
build: also throw hash-mismatch errors if buildMode == bmCheck
This actually bit me quite recently in `nixpkgs` because I assumed that
`nix-build --check` would also error out if hashes don't match anymore[1]
and so I wrongly assumed that I couldn't reproduce the mismatch error.

The fix is rather simple, during the output registration a so-called
`delayedException` is instantiated e.g. if a FOD hash-mismatch occurs.
However, in case of `nix-build --check` (or `--rebuild` in case of `nix
build`), the code-path where this exception is thrown will never be
reached.

By adding that check to the if-clause that causes an early exit in case
of `bmCheck`, the issue is gone. Also added a (previously failing)
test-case to demonstrate the problem.

[1] https://github.com/NixOS/nixpkgs/pull/139238, the underlying issue
    was that `nix-prefetch-git` returns different hashes than `fetchgit`
    because the latter one fetches submodules by default.
2021-09-27 15:44:39 +02:00
Eelco Dolstra
8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Eelco Dolstra
362d8f925e
Merge pull request #5253 from edolstra/flake-ifd
Don't allow IFD in flakes by default
2021-09-24 10:48:23 +02:00
Eelco Dolstra
ea9df6fe51 Shut down write side before draining the read side
This is important if the remote side *does* execute
nix-store/nix-daemon successfully, but stdout is polluted
(e.g. because the remote user's bashrc script prints something to
stdout). In that case we have to shutdown the write side to force the
remote nix process to exit.
2021-09-23 18:01:04 +02:00
Eelco Dolstra
994348e9e0 SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin"
Instead of

  error: serialised integer 7161674624452356180 is too large for type 'j'

we now get

  error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.'

Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 17:50:29 +02:00
Eelco Dolstra
60642aa5e2 Remove risky char * 2021-09-23 12:07:50 +02:00
Eelco Dolstra
198fa786a1 Add some more instrumentation 2021-09-23 11:42:33 +02:00
Eelco Dolstra
d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra
bcd73ebf60 Add missing #include
Fixes #5282.
2021-09-22 12:14:50 +02:00
Gregor Kleen
fd67a0f927 Fix scheme argument to s3 cache 2021-09-17 10:45:19 +02:00
Emily
56025ad3b1 sandbox: allow Rosetta 2 on Darwin
This allows sandboxed x86_64-darwin builds on aarch64-darwin.
2021-09-15 02:02:03 +01:00
Eelco Dolstra
4b738fc7a9 Add 'nix profile wipe-history' command 2021-09-14 20:35:41 +02:00
Eelco Dolstra
f359b9981b Generations -> profile versions 2021-09-14 19:57:45 +02:00
Eelco Dolstra
817562e694 Add "nix profile rollback" command 2021-09-14 19:32:33 +02:00
Eelco Dolstra
d589a6aa8a
Merge pull request #5247 from edolstra/stablize-ca-references
Remove the 'ca-references' feature check
2021-09-14 15:42:50 +02:00
Eelco Dolstra
01a4fa3b6e Advise using --extra-experimental-features instead of --experimental-features
Fixes #3737.
2021-09-14 13:56:36 +02:00
Eelco Dolstra
761ac9d584 Remove the 'ca-references' feature check
Fixes #3422.
Fixes #4425.
2021-09-14 13:53:20 +02:00
Eelco Dolstra
76e368a3b4 Fix macOS build 2021-09-14 10:07:31 +02:00
Eelco Dolstra
4ed66735b6 RunOptions: Use designated initializers
Also get rid of _killStderr because it wasn't actually checked
anywhere.
2021-09-13 23:31:04 +02:00
Eelco Dolstra
ff09acd1f9 Remove tabs 2021-09-10 11:00:50 +02:00
Eelco Dolstra
1e43bc6dc0
Merge pull request #5224 from baloo/baloo/5089/force-nss_dns-load
preloadNSS: fixup nss_dns load
2021-09-09 10:37:41 +02:00
Arthur Gautier
3b72741f23 preloadNSS: load NSS before threads are started
preloadNSS is not thread-safe, this commit moves it before we start the
first thread.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-08 18:29:31 +00:00
Arthur Gautier
0b42a0f781 preloadNSS: fixup nss_dns load
Before this commit, the dns lookup in preloadNSS would still go through
nscd. This did not have the effect of loading the nss_dns.so as expected
(nss_dns.so being out of reach from within the sandbox).

Should LOCALDOMAIN environment variable be defined, nss will completely
avoid nscd and will do its dns resolution on its own.

By temporarly setting LOCALDOMAIN variable before calling in NSS, we can
force NSS to load the shared libraries as expected.

Fixes #5089

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-06 23:13:48 +00:00
regnat
497225b07d Don’t create lockfiles with an invalid path name
Store paths are only allowed to contain a limited subset of the
alphabet, which doesn’t include `!`. So don’t create lockfiles that
contain this `!` character as that would otherwise confuse (and break)
the gc.

Fix #5176
2021-09-02 09:57:41 +02:00
Eelco Dolstra
f73d911628 Style 2021-09-01 21:46:25 +02:00
Eelco Dolstra
2cb7a502a4 Merge branch 'short-circuit-subst' of https://github.com/Kha/nix 2021-09-01 21:45:04 +02:00
50edbc4ddf
nix-store --serve: pass on settings.keepFailed from SSH store
When doing e.g.

    nix-build -A package --keep-failed --option \
      builders \
      'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel'

this doesn't work properly because this build-setting is ignored.

I changed this behavior by passing the `settings.keepFailed` through the
serve-protocol to remote machines to make sure that I can introspect the
build-directory (which is particularly helpful when I have to look at a
`config.log` from a failed build for instance).
2021-08-31 13:11:46 +02:00
Eelco Dolstra
f3b8b4040e
Merge pull request #5071 from NixOS/dont-send-xp-features-to-the-daemon
Don’t send the experimental-features to the daemon
2021-08-31 11:55:39 +02:00
Eelco Dolstra
eda0fee160
Merge pull request #5175 from Pamplemousse/make
Don't overwrite user provided `lib*_LDFLAGS`
2021-08-30 12:44:29 +02:00
regnat
b2bce915ad Don’t accept experimental features from the client
If the client sends an “experimental features” setting, just ignore it
2021-08-30 10:48:05 +02:00
Pamplemousse
a4c6d319a8 Don't overwrite user provided lib*_LDFLAGS
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-25 08:59:19 -07:00
Piotr Szubiakowski
9f13cb31e8
Install pkg-config files in the correct location
Use `$(libdir)` while installing .pc files looks like a more generic
solution. For example, it will work for distributions like RHEL or
Fedora where .pc files are installed in `/usr/lib64/pkgconfig`.
2021-08-13 21:08:58 +00:00
Pamplemousse
0da416f820 15f4d4f follow up
* libstore: `bz2` should not be linked
  * libutil: `zlib.h` should not be included

Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-10 16:04:53 -07:00
Eelco Dolstra
8943e3176d
Merge pull request #5111 from Pamplemousse/clean
Minor maintenance cleaning
2021-08-09 20:05:03 +02:00
Sebastian Ullrich
a0f97b1f54 Expand docstring of Store::querySubstitutablePathInfos 2021-08-09 15:58:50 +02:00
Eelco Dolstra
4c5ce2a345
Merge pull request #5104 from andir/refscan-race
Fix potential race-condition in reference scanning code
2021-08-09 14:49:24 +02:00
Andreas Rammhold
a9cb1ca32c libstore: use set instead of list for waiter list
This replaces the O(n) search complexity in our insert code with a
lookup of O(log n). It also makes removing waitees easier as we can use
the extract method provided by the set class.
2021-08-08 14:05:38 +02:00
Sebastian Ullrich
12f50b6510 Short-circuit querying substituters on success 2021-08-08 13:24:13 +02:00
Andreas Rammhold
b2d3976163 Fix potential race-condition in reference scanning code
Previously the code ensures that the isBase32 array would only be
initialised once in a single-threaded context. If two threads happen to
call the function before the initialisation was completed both of them
would have completed the initialization step. This allowed for a
race-condition where one thread might be done with the initialization
but the other thread sets all the fields to false again. For a brief
moment the base32 detection would then produce false-negatives.
2021-08-07 19:10:25 +02:00
Pamplemousse
d64cb33e90 Remove unused struct in libstore
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-30 10:34:28 -07:00
regnat
d9ba3385a9 Don’t send the experimental-features to the daemon
The experimental features are, well, experimental, and shouldn’t be
carelessly and transparently enabled.
Besides, some (`ca-derivations` at least) need to be enabled at startup
in order to work properly.
So it’s better to just require that daemon be started with the right
`experimental-features` option.

Fix #5017
2021-07-30 10:34:50 +02:00
Eelco Dolstra
72c5bac39d Revert no longer necessary change 2021-07-26 13:50:18 +02:00
Eelco Dolstra
fe1f34fa60 Low-latency closure copy
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.

For example, on a connection with an 50 ms delay:

Before:

  $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
    /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
    --derivation --no-check-sigs
  real    0m57.868s
  user    0m0.103s
  sys     0m0.056s

After:

  real    0m0.690s
  user    0m0.017s
  sys     0m0.011s
2021-07-26 13:31:09 +02:00
Eelco Dolstra
9957315ce0 RemoteStore::buildPaths(): Handle evalStore 2021-07-22 22:50:48 +02:00
Eelco Dolstra
a7b7fcfb16 Remove redundant RealisedPath::closure() call 2021-07-22 22:43:18 +02:00