Commit graph

3613 commits

Author SHA1 Message Date
Eelco Dolstra 89ca75c9f9
Merge pull request #7080 from squalus/nar-close-file
archive: check close errors when extracting nars
2022-09-30 12:58:28 +02:00
Théophane Hufschmitt db29ddd113
Merge pull request #7078 from obsidiansystems/trustworthy-signature
"valid signature" -> "signature by a trusted key"
2022-09-24 12:52:35 +02:00
Théophane Hufschmitt 74276cb354
Merge pull request #7079 from matthewbauer/allow-untrusted-settings
Allow pass max-silent-time and build-poll-interval to daemon untrusted
2022-09-24 12:47:49 +02:00
Eelco Dolstra c13007f012
Merge pull request #7059 from NixOS/remove-useless-ca-file-message
Remove a useless debug message in filetransfer.cc
2022-09-24 10:39:23 +02:00
John Ericson 60e23c8bae
Apply suggestions from code review
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Rune K. Svendsen <runesvend@gmail.com>
2022-09-23 13:57:57 -04:00
squalus 223f8dace0 archive: check close errors when extracting nars 2022-09-22 12:50:32 -07:00
Matthew Bauer 6e049ae607 Allow pass max-silent-time and build-poll-interval to daemon untrusted
These settings seem harmless, they control the same polling
functionality that timeout does, but with different behavior. Should
be safe for untrusted users to pass in.
2022-09-22 13:59:16 -05:00
John Ericson a2a8cb10ac Dodge "trusted" vs "trustworthy" by being explicit
Hopefully this is best!
2022-09-22 14:37:52 -04:00
John Ericson 752f967c0f "valid signature" -> "trustworthy signature"
I just had a colleague get confused by the previous phrase for good
reason. "valid" sounds like an *objective* criterion, e.g. and *invalid
signature* would be one that would be trusted by no one, e.g. because it
misformatted or something.

What is actually going is that there might be a signature which is
perfectly valid to *someone else*, but not to the user, because they
don't trust the corresponding public key. This is a *subjective*
criterion, because it depends on the arbitrary and personal choice of
which public keys to trust.

I therefore think "trustworthy" is a better adjective to use. Whether
something is worthy of trust is clearly subjective, and then "trust"
within that word nicely evokes `trusted-public-keys` and friends.
2022-09-22 10:49:31 -04:00
squalus 1b595026e1 Improve durability of schema version file writes
- call close explicitly in writeFile to prevent the close exception
  from being ignored
- fsync after writing schema file to flush data to disk
- fsync schema file parent to flush metadata to disk

https://github.com/NixOS/nix/issues/7064
2022-09-19 20:13:30 -07:00
Théophane Hufschmitt 0f977bf91e
Remove a useless debug message in filetransfer.cc
Remove the `verify TLS: Nix CA file = 'blah'` message that Nix used to print when fetching anything as it's both useless (`libcurl` prints the same info in its logs) and misleading (gives the impression that a new TLS connection is being established which might not be the case because of multiplexing. See #7011 )
2022-09-19 08:42:43 +02:00
Théophane Hufschmitt 0f64bf445a
Merge pull request #6994 from agbrooks/master
Prevent tempdir from being GC-ed before addToStoreFromDump completes
2022-09-13 09:23:16 +02:00
Andrew Brooks 565d888e0f Address PR feedback on #6694 2022-09-12 11:33:23 -05:00
Andrew Brooks 84fe75a12a Keep created temp dirs inside store, but protect from GC
Implements the approach suggested by feedback on PR #6994, where
tempdir paths are created in the store (now with an exclusive lock).

As part of this work, the currently-broken and unused
`createTempDirInStore` function is updated to create an exclusive lock
on the temp directory in the store.

The GC now makes a non-blocking attempt to lock any store directories
that "look like" the temp directories created by this function, and if
it can't acquire one, ignores the directory.
2022-09-06 17:48:00 -05:00
Andrew Brooks 1f041ac54f Prevent tempdir from being GC-ed before addToStoreFromDump has renamed it
This fixes issue 6823 by placing the tempdir used in LocalStore::addToStoreFromDump
outside the Nix store, where automatic GC is no longer a concern.
2022-09-02 18:32:35 -05:00
Matthew Bauer 4894e567fb Don’t readDerivation if impure derivations feature is disabled
readDerivation is pretty slow, and while it may not be significant for
some use cases, on things like ghc-nix where we have thousands of
derivations is really slows things down.

So, this just doesn’t do the impure derivation check if the impure
derivation experimental feature is disabled. Perhaps we could cache
the result of isPure() and keep the check, but this is a quick fix to
for the slowdown introduced with impure derivations features in 2.8.0.
2022-09-02 11:46:34 -05:00
Adam Joseph 1ab913467e linkify mention of other options 2022-09-01 18:03:35 -07:00
Adam Joseph 59dc8346ca move substituter signature-checking conditions to configuration file documentation 2022-09-01 17:51:56 -07:00
Eelco Dolstra 7918adbb62
Merge pull request #6954 from winterqt/darwin-sandbox-trustd
fix(libstore): allow access to trustd on macOS
2022-08-26 11:45:00 +02:00
Winter Cute 8e5659423e fix(libstore): allow access to trustd on macOS 2022-08-24 13:09:44 -04:00
Eelco Dolstra 56d97d4b4d Remove redundant Finally 2022-08-24 14:49:58 +02:00
Eelco Dolstra 8d906b1f3b Fix macOS build 2022-08-24 14:11:03 +02:00
Eelco Dolstra f0358ed465 Fix a hang in nix-copy-ssh.sh
This hang for some reason didn't trigger in the Nix build, but did
running 'make installcheck' interactively. What happened:

* Store::addMultipleToStore() calls a SinkToSource object to copy a
  path, which in turn calls LegacySSHStore::narFromPath(), which
  acquires a connection.

* The SinkToSource object is not destroyed after the last bytes has
  been read, so the coroutine's stack is still alive and its
  destructors are not run. So the connection is not released.

* Then when the next path is copied, because max-connections = 1,
  LegacySSHStore::narFromPath() hangs forever waiting for a connection
  to be released.

The fix is to make sure that the source object is destroyed when we're
done with it.
2022-08-23 14:19:53 +02:00
Eelco Dolstra f865048332 Indentation 2022-08-22 15:32:53 +02:00
Eelco Dolstra 4c96761c2b Merge remote-tracking branch 'origin/master' into parallel-nix-copy 2022-08-22 15:29:10 +02:00
Eelco Dolstra 8d84634e26
Merge pull request #6926 from rapenne-s/download_limit
add a nix.conf option to set a download speed limit
2022-08-22 14:57:19 +02:00
Eelco Dolstra c21b1a7e67
Spelling 2022-08-22 14:14:14 +02:00
Solène Rapenne caad87e6db
Better documentation wording
Co-authored-by: Anderson Torres <torres.anderson.85@protonmail.com>
2022-08-20 18:21:36 +02:00
Solene Rapenne 0d2bf7acf9 add a nix.conf option to set a download speed limit 2022-08-19 12:40:22 +02:00
pennae 7d934f7880 don't read outputs into memory for output rewriting
RewritingSink can handle being fed input where a reference crosses a
chunk boundary. we don't need to load the whole source into memory, and
in fact *not* loading the whole source lets nix build FODs that do not
fit into memory (eg fetchurl'ing data files larger than system memory).
2022-08-19 11:26:26 +02:00
Eelco Dolstra 53e7b7e8ac Remove warnLargeDump()
This message was unhelpful (#1184) and probably misleading since
memory is O(1) in most cases now.
2022-08-17 11:32:01 +02:00
Théophane Hufschmitt 8f3fdef1e0
Merge pull request #6850 from NinjaTrappeur/nin/build-check
Fix Nix build --check flag
2022-08-10 17:44:06 +02:00
Théophane Hufschmitt 90f9680733 Only use renameFile where needed
In most places the fallback to copying isn’t needed and can actually be
bad, so we’d rather not transparently fallback
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt d71d9e9fbf moveFile -> renameFile
`move` tends to have this `mv` connotation of “I will copy it for you if
needs be”
2022-08-03 10:27:25 +02:00
Théophane Hufschmitt c2de0a232c Create a wrapper around stdlib’s rename
Directly takes some c++ strings, and gently throws an exception on error
(rather than having to inline this logic everywhere)
2022-08-03 10:27:25 +02:00
Félix Baylac-Jacqué 1467a98d4c
derivation-goal.cc: remove bmCheck custom return branch on buildDone
Once a derivation goal has been completed, we check whether or not
this goal was meant to be repeated to check its output.

An early return branch was preventing the worker to reach that repeat
code branch, hence breaking the --check command (#2619).

It seems like this early return branch is an artifact of a passed
refactoring. As far as I can tell, buildDone's main branch also
cleanup the tmp directory before returning.
2022-08-01 11:39:19 +02:00
Théophane Hufschmitt 2805439335
Merge pull request #6814 from amjoseph-nixpkgs/pr/sandbox-error-messages
local-derivation-goal.cc: improve error messages when sandboxing fails
2022-07-22 13:27:52 +02:00
Théophane Hufschmitt 7ed91d6c6a
Merge branch 'master' into parallel-nix-copy 2022-07-20 10:05:34 +02:00
Alex Wied 722de8ddcc libstore/globals.cc: Move cgroup detection to libutil 2022-07-19 16:25:53 -04:00
Alex Wied 1af5d798a4 libstore/globals.cc: Automatically set cores based on cgroup CPU limit
By default, Nix sets the "cores" setting to the number of CPUs which are
physically present on the machine. If cgroups are used to limit the CPU
and memory consumption of a large Nix build, the OOM killer may be
invoked.

For example, consider a GitLab CI pipeline which builds a large software
package. The GitLab runner spawns a container whose CPU is limited to 4
cores and whose memory is limited to 16 GiB. If the underlying machine
has 64 cores, Nix will invoke the build with -j64. In many cases, that
level of parallelism will invoke the OOM killer and the build will
completely fail.

This change sets the default value of "cores" to be
ceil(cpu_quota / cpu_period), with a fallback to
std:🧵:hardware_concurrency() if cgroups v2 is not detected.
2022-07-19 16:03:58 -04:00
Théophane Hufschmitt 56f6f3725f Don't ultimately trust the signed paths
Like the old implem did (and like you'd want it to be anyways)
2022-07-19 19:46:00 +02:00
Adam Joseph 36e1383b6b local-derivation-goal.cc: save global errno to the stack before performing tests which might clobber it 2022-07-19 03:53:20 -07:00
Adam Joseph 99fcc91f67 as requested by @thufschmitt https://github.com/NixOS/nix/pull/6814#discussion_r924275777 2022-07-19 03:33:12 -07:00
Adam Joseph 5f51539f88 change warn() to notice() 2022-07-19 03:30:52 -07:00
Adam Joseph c8c6203c2c local-derivation-goal.cc: detect unprivileged_userns_clone failure mode
The workaround for "Some distros patch Linux" mentioned in
local-derivation-goal.cc will not help in the `--option
sandbox-fallback false` case.  To provide the user more helpful
guidance on how to get the sandbox working, let's check to see if the
`/proc` node created by the aforementioned patch is present and
configured in a way that will cause us problems.  If so, give the user
a suggestion for how to troubleshoot the problem.
2022-07-17 01:27:22 -07:00
Adam Joseph 6fc56318bf local-derivation-goal.cc: add comment re: CLONE_NEWUSER
local-derivation-goal.cc contains a comment stating that "Some distros
patch Linux to not allow unprivileged user namespaces."  Let's give a
pointer to a common version of this patch for those who want more
details about this failure mode.
2022-07-17 01:23:32 -07:00
Adam Joseph 8d35f387dc local-derivation-goal.cc: warn if failing and /proc/self/ns/user missing
This commit causes nix to `warn()` if sandbox setup has failed and
`/proc/self/ns/user` does not exist.  This is usually a sign that the
kernel was compiled without `CONFIG_USER_NS=y`, which is required for
sandboxing.
2022-07-16 19:37:27 -07:00
Adam Joseph 90830b1074 local-derivation-goal.cc: warn if failing due to max_user_namespaces==0
This commit uses `warn()` to notify the user if sandbox setup fails
with errno==EPERM and /proc/sys/user/max_user_namespaces is missing or
zero, since that is at least part of the reason why sandbox setup
failed.

Note that `echo -n 0 > /proc/sys/user/max_user_namespaces` or
equivalent at boot time has been the recommended mitigation for
several Linux LPE vulnerabilities over the past few years.  Many users
have applied this mitigation and then forgotten that they have done
so.
2022-07-16 19:30:53 -07:00
Adam Joseph 8ea3a911aa local-derivation-goal.cc: improve error messages when sandboxing fails
The failure modes for nix's sandboxing setup are pretty complicated.
When nix is unable to set up the sandbox, let's provide more detail
about what went wrong.  Specifically:

* Make sure the error message includes the word "sandbox" so the user
  knows that the failure was related to sandboxing.

* If `--option sandbox-fallback false` was provided, and removing it
  would have allowed further attempts to make progress, let the user
  know.
2022-07-16 14:56:24 -07:00
Alex Wied b88fb50e21 fix(libstore): allow Nix to access all Rosetta 2 paths on MacOS
Fixes: #5884
2022-07-15 12:10:56 -07:00
Eelco Dolstra 3bcd7a5474 Disable auto-chroot if $NIX_STATE_DIR is set
Issue #6732.
2022-07-15 12:32:29 +02:00
John Ericson 8735f55dec Fix bug, test more, document more 2022-07-14 20:23:43 -04:00
John Ericson 6cafe308c9 Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-07-14 16:15:37 -04:00
Eelco Dolstra ff49c75502 Disable auto-chroot if $NIX_STORE_DIR is set
Fixes #6732.
2022-07-14 17:47:09 +02:00
Eelco Dolstra 6cab528461 Don't fail if we can't create ~/.local/share/nix/root
https://hydra.nixos.org/build/182135943
2022-06-29 12:16:51 +02:00
Théophane Hufschmitt 56cf96a1b9
Merge pull request #6706 from lheckemann/cache-info-cache-invalidation
nar-info-disk-cache: refresh nix-cache-info weekly
2022-06-29 07:56:27 +02:00
Eelco Dolstra 586fa707fc
Merge pull request #6714 from edolstra/auto-chroot-store
Automatically use a chroot store if /nix doesn't exist
2022-06-25 00:03:35 +02:00
Eelco Dolstra 30d4aa5dd6 Only do the auto chroot store on Linux 2022-06-24 23:35:21 +02:00
Linus Heckemann 8cf26385cd [fixup] handle cache expiration in sqlite query 2022-06-23 14:54:25 -04:00
Cole Helbling 561a258f1d libstore/nar-info: drop unused system field
This was unused everywhere (and even the official NixOS binary cache
did not produce .narinfo files containing a "System:" field).
2022-06-23 14:25:10 -04:00
Eelco Dolstra 1cb376d60e
Fix typo
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2022-06-23 17:18:22 +02:00
Eelco Dolstra 2a9fddc0b1 Automatically use a chroot store if /nix doesn't exist
Specifically, if we're not root and the daemon socket does not exist,
then we use ~/.local/share/nix/root as a chroot store. This enables
non-root users to download nix-static and have it work out of the box,
e.g.

  ubuntu@ip-10-13-1-146:~$ ~/nix run nixpkgs#hello
  warning: '/nix' does not exists, so Nix will use '/home/ubuntu/.local/share/nix/root' as a chroot store
  Hello, world!
2022-06-23 16:29:50 +02:00
Eelco Dolstra 3c57db1a0f
Merge pull request #6710 from edolstra/embedded-sandbox-shell
Embed the sandbox shell into the statically linked 'nix' binary
2022-06-23 15:34:16 +02:00
Eelco Dolstra 0b2ea0023c Fix typo 2022-06-23 14:22:11 +02:00
Théophane Hufschmitt 027f6a735f
Merge pull request #6673 from asymmetric/warn
libstore: improve warning message on missing sig
2022-06-23 06:42:48 +02:00
Eelco Dolstra 925b975224 Embed the sandbox shell into the statically linked 'nix' binary
With this, Nix will write a copy of the sandbox shell to /bin/sh in
the sandbox rather than bind-mounting it from the host filesystem.
This makes /bin/sh work out of the box with nix-static, i.e. you no
longer get

  /nix/store/qa36xhc5gpf42l3z1a8m1lysi40l9p7s-bootstrap-stage4-stdenv-linux/setup: ./configure: /bin/sh: bad interpreter: No such file or directory
2022-06-23 04:08:28 +02:00
Eelco Dolstra 184f4e40de Remove NIX_LIBEXEC_DIR 2022-06-23 01:32:46 +02:00
Eelco Dolstra d3176ce076 Fix build-remote in nix-static
'build-remote' is now executed via /proc/self/exe so it always works.
2022-06-23 01:32:46 +02:00
Linus Heckemann d533a88546 nar-info-disk-cache: refresh nix-cache-info weekly
This allows changes to nix-cache-info to be picked up by existing
clients. Previously, the only way for this to happen would be for
clients to delete binary-cache-v6.sqlite, which is quite awkward for
users.

On the other hand, updates to nix-cache-info should be pretty rare,
hence the choice of a fairly long TTL. Configurability is probably not
useful enough to warrant implementing it.
2022-06-22 10:49:18 -04:00
Lorenzo Manacorda 475249db8a libstore: improve warning message on missing sig
Clarifies that the substitute will be ignored/skipped.
2022-06-15 17:34:28 +02:00
Eelco Dolstra 9f58df4c91
Merge pull request #6619 from Jonpez2/patch-1
Add security.csm to ignored-acls
2022-06-13 16:23:13 +02:00
Eelco Dolstra 2fef24f528
Merge pull request #6634 from lovesegfault/fix-getgrouplist
fix(libstore/lock): support users that belong to more than 10 groups
2022-06-13 15:45:09 +02:00
Anders Kaseorg 754cd53faf Add missing rethrows in conditional exception handlers
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2022-06-10 10:52:41 -07:00
Bernardo Meurer 931930feb1 fix(libstore/lock): support users that belong to more than 10 groups
The manpage for `getgrouplist` says:

> If the number of groups of which user is a member is less than or
> equal to *ngroups, then the value *ngroups is returned.
>
> If the user is a member of more than *ngroups groups, then
> getgrouplist() returns -1.  In this case, the value returned in
> *ngroups can be used to resize the buffer passed to a further
> call getgrouplist().

In our original code, however, we allocated a list of size `10` and, if
`getgrouplist` returned `-1` threw an exception. In practice, this
caused the code to fail for any user belonging to more than 10 groups.

While unusual for single-user systems, large companies commonly have a
huge number of POSIX groups users belong to, causing this issue to crop
up and make multi-user Nix unusable in such settings.

The fix is relatively simple, when `getgrouplist` fails, it stores the
real number of GIDs in `ngroups`, so we must resize our list and retry.
Only then, if it errors once more, we can raise an exception.

This should be backported to, at least, 2.9.x.
2022-06-08 13:45:41 -04:00
Théophane Hufschmitt 480c2b6699 Rewrite the CA paths when moving them between store
Bring back the possibility to copy CA paths with no reference (like the
outputs of FO derivations or stuff imported at eval time) between stores
that have a different prefix.
2022-06-08 15:13:11 +02:00
Théophane Hufschmitt cb0553ecd0 Restore the "low-latency" ssh copying 2022-06-08 14:03:46 +02:00
Jonpez2 a7d25d339d
Add security.csm to the default ignore list 2022-06-08 09:32:14 +01:00
Théophane Hufschmitt 95f47c28fb Make nix copy parallel again
FILLME
2022-06-03 17:01:16 +02:00
Eelco Dolstra 2097c30b08
Merge pull request #6600 from asymmetric/hash-comment
schema.sql: add comment about hash being in base16
2022-06-03 08:51:07 +02:00
John Ericson b585548dfe Merge remote-tracking branch 'upstream/master' into indexed-store-path-outputs 2022-06-02 19:53:11 +00:00
Théophane Hufschmitt a5f7b934bd
Merge branch 'master' into lto 2022-06-01 17:55:03 +02:00
Lorenzo Manacorda a9358a6097 schema.sql: add comment about hash being in base16 2022-06-01 14:59:57 +02:00
Eelco Dolstra 6378f0bb32
RemoteStore::queryRealisationUncached(): Fix potential noexcept violation 2022-05-30 13:27:13 +02:00
Eelco Dolstra b8faa83742
HttpBinaryCacheStore::getFile(): Don't throw an exception
This violates the noexcept specification.

Fixes #6445.
2022-05-30 13:24:04 +02:00
pennae b092afe77d
Merge branch 'master' into lto 2022-05-25 11:55:13 +00:00
Eli Kogan-Wang e53349dd6e change priority conflict message 2022-05-16 16:16:06 +02:00
Eli Kogan-Wang be2b19041e Integrate review changes 2022-05-13 22:02:28 +02:00
John Ericson 49ad315c03 Use ^ not ! in indexed store derivations installable syntax
Match the other syntax that was recently added
2022-05-12 20:10:02 +00:00
Eli Kogan-Wang aefc6c4f41 Add priority for nix profile install 2022-05-11 12:16:35 +02:00
Eelco Dolstra 107613ad2b Fix compiler warning 2022-05-04 11:31:39 +02:00
Eelco Dolstra 3e87c8e62b Move json stuff out of util.cc 2022-05-04 11:22:06 +02:00
Alain Zscheile 1385b20078
Get rid of most .at calls (#6393)
Use one of `get` or `getOr` instead which will either return a null-pointer (with a nicer error message) or a default value when the key is missing.
2022-05-04 07:44:32 +02:00
Eelco Dolstra a3c6c5b1c7 nix profile: Support overriding outputs 2022-05-03 15:00:34 +02:00
Eelco Dolstra 4a79cba511 Allow selecting derivation outputs using 'installable!outputs'
E.g. 'nixpkgs#glibc^dev,static' or 'nixpkgs#glibc^*'.
2022-05-03 13:43:52 +02:00
Eelco Dolstra 70a30dbc00 Fix libcxx build
Fixes #6458.
2022-04-28 14:37:05 +02:00
Eelco Dolstra 4a9623b129 Fix passing $OUT_PATHS to the post-build hook
Fixes #6446.
2022-04-28 13:36:01 +02:00
Adam Joseph 49119072e7 local-derivation-goal.cc: seccomp filters for MIPS secondary arch/abi
A mips64el Linux MIPS kernel can execute userspace code using any of
three ABIs:

  mips64el-linux-*abin64
  mips64el-linux-*abin32
    mipsel-linux-*

The first of these is the native 64-bit ABI, and the only ABI with
64-bit pointers; this is sometimes called "n64".  The last of these is
the old legacy 32-bit ABI, whose binaries can execute natively on
32-bit MIPS hardware; this is sometimes called "o32".

The second ABI, "n32" is essentially the 64-bit ABI with 32-bit
pointers and address space.  Hardware 64-bit integer/floating
arithmetic is still allowed, as well as the much larger mips64
register set and more-efficient calling convention.

Let's enable seccomp filters for all of these.  Likewise for big
endian (mips64-linux-*).
2022-04-27 21:39:26 -07:00
pennae d6d6bbd9ef
Merge branch 'master' into lto 2022-04-25 14:02:37 +00:00
Eelco Dolstra ebad9213f7
Merge pull request #6431 from NixOS/unbreak-my-build
Make the default SQLiteError constructor public
2022-04-21 10:41:07 +02:00
Sergei Trofimovich 975b0b52e7 ca: add sqlite index on RealisationsRefs(realisationReference)
Without the change any CA deletion triggers linear scan on large
RealisationsRefs table:

    sqlite>.eqp full
    sqlite> delete from RealisationsRefs where realisationReference IN ( select id from Realisations where outputPath = 1234567890 );
    QUERY PLAN
    |--SCAN RealisationsRefs
    `--LIST SUBQUERY 1
       `--SEARCH Realisations USING COVERING INDEX IndexRealisationsRefsOnOutputPath (outputPath=?)

With the change it gets turned into a lookup:

    sqlite> CREATE INDEX IndexRealisationsRefsRealisationReference on RealisationsRefs(realisationReference);
    sqlite> delete from RealisationsRefs where realisationReference IN ( select id from Realisations where outputPath = 1234567890 );
    QUERY PLAN
    |--SEARCH RealisationsRefs USING INDEX IndexRealisationsRefsRealisationReference (realisationReference=?)
    `--LIST SUBQUERY 1
       `--SEARCH Realisations USING COVERING INDEX IndexRealisationsRefsOnOutputPath (outputPath=?)
2022-04-21 10:06:39 +02:00
regnat 86d7a11c6b Make sure to delete all the realisation refs
Deleting just one will only work in the test cases where I didn’t bother
creating too many of them :p
2022-04-21 10:06:39 +02:00
regnat 92656da0b9 Fix the gc with indirect self-references via the realisations
If the derivation `foo` depends on `bar`, and they both have the same
output path (because they are CA derivations), then this output path
will depend both on the realisation of `foo` and of `bar`, which
themselves depend on each other.
This confuses SQLite which isn’t able to automatically solve this
diamond dependency scheme.

Help it by adding a trigger to delete all the references between the
relevant realisations.

Fix #5320
2022-04-21 10:06:39 +02:00
Théophane Hufschmitt e7d79c7861 Make the default SQLiteError constructor public
Otherwise the clang builds fail because the constructor of `SQLiteBusy`
inherits it, `SQLiteError::_throw` tries to call it, which fails.

Strangely, gcc works fine with it. Not sure what the correct behavior is
and who is buggy here, but either way, making it public is at the worst
a reasonable workaround
2022-04-21 09:40:55 +02:00
John Ericson f63b0f4540 Actually, solve this in a lighter-weight way
The templating is very superficial
2022-04-20 17:37:59 +00:00
John Ericson 05ec0beb40 Move templated functions to sqlite-impl.hh
This ensures that use-sites properly trigger new monomorphisations on
one hand, and on the other hand keeps the main `sqlite.hh` clean and
interface-only. I think that is good practice in general, but in this
situation in particular we do indeed have `sqlite.hh` users that don't
need the `throw_` function.
2022-04-20 16:57:06 +00:00
John Ericson 3c220442ff Merge remote-tracking branch 'upstream/master' into fix-url-format 2022-04-20 16:53:16 +00:00
John Ericson 75b62e5260 Avoid fmt when constructor already does it
There is a correctnes issue here, but #3724 will fix that. This is just
a cleanup for brevity's sake.
2022-04-19 01:44:11 +00:00
Alex Ameen e5c934cd48
doc: rephrase connect-timeout help message
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2022-04-17 18:17:37 -05:00
Alex Ameen 25c85f5a0e
doc: document nix.conf connect-timeout default 2022-04-17 17:14:38 -05:00
Sebastian Blunt 63d9a81819 Log builder args and environment variables
Previously it only logged the builder's path, this changes it to log the
arguments at the same log level, and the environment variables at the
vomit level.

This helped me debug https://github.com/svanderburg/node2nix/issues/75
2022-04-10 21:10:37 -07:00
Eelco Dolstra c68963eaea Remove duplicate "error:" 2022-04-08 11:48:30 +02:00
Eelco Dolstra 8bd9ebf52c Error: Remove unused sname() method 2022-04-08 11:31:51 +02:00
Eelco Dolstra 168ef9f3ab Remove unused Error.name field 2022-04-08 11:31:51 +02:00
Eelco Dolstra 318936366d Fix empty 'nix copy' error message
This was caused by SubstitutionGoal not setting the errorMsg field in
its BuildResult. We now get a more descriptive message than in 2.7.0, e.g.

  error: path '/nix/store/13mh...' is required, but there is no substituter that can build it

instead of the misleading (since there was no build)

  error: build of '/nix/store/13mh...' failed

Fixes #6295.
2022-04-06 12:43:53 +02:00
Eelco Dolstra a4a1de69dc Add missing #include 2022-04-04 16:49:39 +02:00
Eelco Dolstra a99af85a77 Fix macOS build 2022-03-31 16:39:18 +02:00
Eelco Dolstra 6051cc954b Rename 'pure' -> 'sandboxed' for consistency 2022-03-31 16:12:25 +02:00
Eelco Dolstra e279fbb16a needsNetworkAccess() -> isSandboxed() 2022-03-31 16:06:40 +02:00
Eelco Dolstra 4e043c2f32 Document isPure() 2022-03-31 16:01:50 +02:00
Eelco Dolstra d7fc33c842 Fix macOS build 2022-03-31 15:59:14 +02:00
Eelco Dolstra 18935e8b9f Support fixed-output derivations depending on impure derivations 2022-03-31 13:43:20 +02:00
Eelco Dolstra 5cd72598fe Add support for impure derivations
Impure derivations are derivations that can produce a different result
every time they're built. Example:

  stdenv.mkDerivation {
    name = "impure";
    __impure = true; # marks this derivation as impure
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    buildCommand = "date > $out";
  };

Some important characteristics:

* This requires the 'impure-derivations' experimental feature.

* Impure derivations are not "cached". Thus, running "nix-build" on
  the example above multiple times will cause a rebuild every time.

* They are implemented similar to CA derivations, i.e. the output is
  moved to a content-addressed path in the store. The difference is
  that we don't register a realisation in the Nix database.

* Pure derivations are not allowed to depend on impure derivations. In
  the future fixed-output derivations will be allowed to depend on
  impure derivations, thus forming an "impurity barrier" in the
  dependency graph.

* When sandboxing is enabled, impure derivations can access the
  network in the same way as fixed-output derivations. In relaxed
  sandboxing mode, they can access the local filesystem.
2022-03-31 13:43:20 +02:00
Théophane Hufschmitt 390269ed87 Simplify the handling of the hash modulo
Rather than having four different but very similar types of hashes, make
only one, with a tag indicating whether it corresponds to a regular of
deferred derivation.

This implies a slight logical change: The original Nix+multiple-outputs
model assumed only one hash-modulo per derivation. Adding
multiple-outputs CA derivations changed this as these have one
hash-modulo per output. This change is now treating each derivation as
having one hash modulo per output.
This obviously means that we internally loose the guaranty that
all the outputs of input-addressed derivations have the same hash
modulo. But it turns out that it doesn’t matter because there’s nothing
in the code taking advantage of that fact (and it probably shouldn’t
anyways).

The upside is that it is now much easier to work with these hashes, and
we can get rid of a lot of useless `std::visit{ overloaded`.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2022-03-29 18:17:35 +02:00
Eelco Dolstra 057f9ee190 nix profile install: Don't use queryDerivationOutputMap()
Instead get the outputs from Installable::build(). This will also
allow 'nix profile install' to support impure derivations.

Fixes #6286.
2022-03-28 14:23:39 +02:00
Eelco Dolstra 50c229ad9a
Use wantOutput
Co-authored-by: John Ericson <git@JohnEricson.me>
2022-03-25 08:02:49 +01:00
Eelco Dolstra 540d7e33d8 Retry substitution after an incomplete closure only once
This avoids an infinite loop in the final test in
tests/binary-cache.sh. I think this was only not triggered previously
by accident (because we were clearing wantedOutputs in between).
2022-03-24 23:25:12 +01:00
Eelco Dolstra fe5509df9a Only return wanted outputs 2022-03-24 23:24:48 +01:00
Eelco Dolstra 09796c0263 Random cleanup 2022-03-24 23:24:10 +01:00
Eelco Dolstra 175c78591b Random cleanup 2022-03-24 23:09:43 +01:00
Eelco Dolstra 28186b7044 Add a test for fetchClosure and 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra f18607549c Fix makeContentAddressed() on self-references
LocalStore::addToStore() since
79ae9e4558 expects a regular NAR hash,
rather than a NAR hash modulo self-references. Fixes #6300.

Also, makeContentAddressed() now rewrites the entire closure (so 'nix
store make-content-addressable' no longer needs '-r'). See #6301.
2022-03-24 21:33:33 +01:00
Eelco Dolstra 545c2d0d8c fetchClosure: Allow a path to be rewritten to CA on the fly
The advantage is that the resulting closure doesn't need to be signed,
so you don't need to configure any binary cache keys on the client.
2022-03-24 21:33:33 +01:00
Eelco Dolstra 0b42afe027 buildProfile(): Ignore manifest.{nix,json}
If a package installs a file named manifest.json, it caused nix-env to
consider the profile a new-style profile created by 'nix
profile'. Fixes #6032.
2022-03-22 10:48:02 +01:00
John Ericson a544ed7684 Generalize DerivationType in preparation for impure derivations 2022-03-18 14:59:56 +00:00
John Ericson 197feed51d Clean up DerivationOutput, and headers
1. `DerivationOutput` now as the `std::variant` as a base class. And the
   variants are given hierarchical names under `DerivationOutput`.

   In 8e0d0689be @matthewbauer and I
   didn't know a better idiom, and so we made it a field. But this sort
   of "newtype" is anoying for literals downstream.

   Since then we leaned the base class, inherit the constructors trick,
   e.g. used in `DerivedPath`. Switching to use that makes this more
   ergonomic, and consistent.

2. `store-api.hh` and `derivations.hh` are now independent.

   In bcde5456cc I swapped the dependency,
   but I now know it is better to just keep on using incomplete types as
   much as possible for faster compilation and good separation of
   concerns.
2022-03-17 22:35:53 +00:00
Eelco Dolstra 6afc361798
Merge pull request #6277 from thufschmitt/ca/nix-build-dry-run
Fix `nix build --dry-run` with CA derivations
2022-03-17 22:23:15 +01:00
Sergei Trofimovich d58453f72e gc: don't visit implicit referrers on garbage collection
Before the change garbage collector was not considering
`.drv` and outputs as alive even if configuration says otherwise.
As a result `nix store gc --dry-run` could visit (and parse)
`.drv` files multiple times (worst case it's quadratic).

It happens because `alive` set was populating only runtime closure
without regard for actual configuration. The change fixes it.

Benchmark: my system has about 139MB, 40K `.drv` files.

Performance before the change:

    $ time nix store gc --dry-run
    real    4m22,148s

Performance after the change:

    $ time nix store gc --dry-run
    real    0m14,178s
2022-03-17 18:47:29 +00:00
Théophane Hufschmitt 3fc4c612fb Fix nix build --dry-run with CA derivations
Don’t try and assume that we know the output paths when we’ve just built
with `--dry-run`. Instead make `--dry-run` follow a different code path
that won’t assume the knowledge of the output paths at all.

Fix #6275
2022-03-17 11:55:19 +01:00
Eelco Dolstra a53c1dc96d
Merge pull request #6245 from trofi/fix-nix-store-gc-limit-auto-optimise-store
nix store gc: account for auto-optimised store
2022-03-16 09:19:57 +01:00
Théophane Hufschmitt 516a7ac4de
Merge pull request #6229 from obsidiansystems/refactor-hash-modulo
Overhaul derivation hash modulo somewhat
2022-03-15 21:23:44 +01:00
Sergei Trofimovich 6b1872312f nix store gc: account for auto-optimised store
Before the change on a system with `auto-optimise-store = true`:

    $ nix store gc --verbose --max 1

deleted all the paths instead of one path (we requested 1 byte limit).

It happens because every file in `auto-optimise-store = true` has at
least 2 links: file itself and a link in /nix/store/.links/ directory.

The change conservatively assumes that any file that has one (as before)
or two links (assume auto-potimise mode) will free space.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2022-03-13 07:24:48 +00:00
John Ericson 0948b8e94d Reduce variants for derivation hash modulo
This changes was taken from dynamic derivation (#4628). It` somewhat
undoes the refactors I first did for floating CA derivations, as the
benefit of hindsight + requirements of dynamic derivations made me
reconsider some things.

They aren't to consequential, but I figured they might be good to land
first, before the more profound changes @thufschmitt has in the works.
2022-03-11 21:20:37 +00:00
John Ericson a03b1fd7f6 Deduplicate the Store downcasting with a template 2022-03-11 13:32:33 +00:00
John Ericson 678d1c2aa0 Factor out a LogStore interface
Continue progress on #5729.

Just as I hoped, this uncovered an issue: the daemon protocol is missing
a way to query build logs. This doesn't effect `unix://`, but does
effect `ssh://`. A FIXME is left for this, so we come back to it later.
2022-03-11 13:32:16 +00:00
John Ericson 89effe9d4a GcStore::resolve should print the URI 2022-03-11 13:27:38 +00:00
John Ericson 073e134de6 Rename requireGcStore to GcStore::require
I should have done this to begin with. This will be nicer once more
Store sub-interfaces exist too, to illustrate the pattern.
2022-03-11 13:27:38 +00:00
pennae f2603e9c92
Merge branch 'master' into lto 2022-03-10 00:32:34 +00:00
Eelco Dolstra 4d98143914 BuildResult: Remove unused drvPath field 2022-03-09 20:31:50 +01:00
Eelco Dolstra 761242afa0 BuildResult: Use DerivedPath 2022-03-09 12:25:35 +01:00
Eelco Dolstra a4604f1928 Add Store::buildPathsWithResults()
This function is like buildPaths(), except that it returns a vector of
BuildResults containing the exact statuses and output paths of each
derivation / substitution. This is convenient for functions like
Installable::build(), because they then don't need to do another
series of calls to get the outputs of CA derivations. It's also a
precondition to impure derivations, where we *can't* query the output
of those derivations since they're not stored in the Nix database.

Note that PathSubstitutionGoal can now also return a BuildStatus.
2022-03-08 19:56:34 +01:00
Théophane Hufschmitt a2ace54fe4
Merge pull request #6029 from Ma27/nix-log-ssh-ng
ssh-ng: also store build logs to make them accessible by `nix log`
2022-03-07 09:51:40 +01:00
John Ericson 6636202356 Factor out a GcStore interface
Starts progress on #5729.

The idea is that we should not have these default methods throwing
"unimplemented". This is a small step in that direction.

I kept `addTempRoot` because it is a no-op, rather than failure. Also,
as a practical matter, it is called all over the place, while doing
other tasks, so the downcasting would be annoying.

Maybe in the future I could move the "real" `addTempRoot` to `GcStore`,
and the existing usecases use a `tryAddTempRoot` wrapper to downcast or
do nothing, but I wasn't sure whether that was a good idea so with a
bias to less churn I didn't do it yet.
2022-03-03 19:01:25 +00:00
regnat 417aaf4ff7 Correctly hijack the file:// uri scheme with _NIX_FORCE_HTTP
Setting the `_NIX_FORCE_HTTP` environment variable is supposed to force `file://` store urls to use the `HttpBinaryCacheStore` implementation rather than the `LocalBinaryCacheStore` one (very useful for testing).
However because of a name mismatch, the `LocalBinaryCacheStore` was still registering the `file` scheme when this variable was set, meaning that the actual store implementation picked up on `file://` uris was dependent on the registration order of the stores (itself dependent on the link order of the object files).

Fix this by making the `LocalBinaryCacheStore` gracefully not register the `file` uri scheme when the variable is set.
2022-03-03 17:47:50 +01:00
Eelco Dolstra 6097790863 Fix segfault in headerCallback()
https://hydra.nixos.org/build/168594664
2022-03-03 11:11:16 +01:00
John Ericson e862833ec6 Move BuildResult defintion to its own header
Just like we did for `ValidPathInfo` in
d92d4f85a5.
2022-03-01 19:43:07 +00:00
John Ericson ea71da395f Move some stuff from Settings to a new FetchSettings.
Starting work on #5638

The exact boundary between `FetchSettings` and `EvalSettings` is not
clear to me, but that's fine. First lets clean out `libstore`, and then
worry about what, if anything, should be the separation between those
two.
2022-03-01 01:39:25 +00:00
John Ericson dc92b01885 Merge remote-tracking branch 'upstream/master' into auto-uid-allocation 2022-02-28 23:54:20 +00:00
Maximilian Bosch 102cb39086
libstore/build: add a few explanatory comments; simplify 2022-02-28 17:27:52 +01:00
Maximilian Bosch cd92ea5885
libstore/derivation-goal: avoid double-parsing of JSON messages
To avoid that JSON messages are parsed twice in case of
remote builds with `ssh-ng://`, I split up the original
`handleJSONLogMessage` into three parts:

* `parseJSONMessage(const std::string&)` checks if it's a message in the
  form of `@nix {...}` and tries to parse it (and prints an error if the
  parsing fails).
* `handleJSONLogMessage(nlohmann::json&, ...)` reads the fields from the
  message and passes them to the logger.
* `handleJSONLogMessage(const std::string&, ...)` behaves as before, but
  uses the two functions mentioned above as implementation.

In case of `ssh-ng://`-logs the first two methods are invoked manually.
2022-02-28 17:27:52 +01:00
Maximilian Bosch 7a04839ea5
ssh-ng: also store build logs to make them accessible by nix log
Right now when building a derivation remotely via

    $ nix build -j0 -f . hello -L --builders 'ssh://builder'

it's possible later to read through the entire build-log by running
`nix log -f . hello`. This isn't possible however when using `ssh-ng`
rather than `ssh`.

The reason for that is that there are two different ways to transfer
logs in Nix through e.g. an SSH tunnel (that are used by `ssh`/`ssh-ng`
respectively):

* `ssh://` receives its logs from the fd pointing to `builderOut`. This
  is directly passed to the "log-sink" (and to the logger on each `\n`),
  hence `nix log` works here.
* `ssh-ng://` however expects JSON-like messages (i.e. `@nix {log data
  in here}`) and passes it directly to the logger without doing anything
  with the `logSink`. However it's certainly possible to extract
  log-lines from this format as these have their own message-type in the
  JSON payload (i.e. `resBuildLogLine`).

  This is basically what I changed in this patch: if the code-path for
  `builderOut` is not reached and a `logSink` is initialized, the
  message was successfully processed by the JSON logger (i.e. it's in
  the expected format) and the line is of the expected type (i.e.
  `resBuildLogLine`), the line will be written to the log-sink as well.

Closes #5079
2022-02-28 17:27:52 +01:00
Eelco Dolstra b91500a14e Fix clang warning 2022-02-28 15:48:26 +01:00
Eelco Dolstra a949673a5b Fix Darwin build
Fixes #6169
2022-02-28 15:48:26 +01:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Kevin Amado b312d4d096
refactor: remove verbose-build from docs
- From what I see it is an implementation detail
  but is no longer configurable from the settings
2022-02-25 09:16:42 -05:00
Eelco Dolstra 36c7b12f33 Remove std::string alias 2022-02-21 16:37:25 +01:00
Eelco Dolstra fe9afb65bb Remove std::set alias 2022-02-21 16:28:23 +01:00
Eelco Dolstra afcdc7606c Remove std::list alias 2022-02-21 16:25:12 +01:00
Linus Heckemann dbdc63bc41 path-info: use full store paths when we have them
Fixes #5645
2022-02-19 00:32:05 +01:00
Eelco Dolstra 2d6d9a28eb addToStoreFromDump(): Take std::string_view 2022-02-16 11:02:35 +01:00
Sebastian Ullrich c437e1326d Fix using sandbox without user namespaces 2022-02-12 16:28:36 +01:00
Eelco Dolstra 97e02c23bd Fix 'basic_string::_M_construct null not valid' in interrupted download
Fixes #5985.
2022-02-07 10:53:47 +01:00
Eelco Dolstra cd35bbbeef Merge branch 'more-stringviews' of https://github.com/pennae/nix 2022-02-02 12:38:37 +01:00
Thomas Koch 85b1427662 fix spelling mistakes reported by Debian's lintian tool 2022-01-30 10:51:39 +02:00
pennae 41d70a2fc8 return string_views from forceString*
once a string has been forced we already have dynamic storage allocated for it,
so we can easily reuse that storage instead of copying.
2022-01-27 17:15:43 +01:00
pennae 0d7fae6a57 convert a for more utilities to string_view 2022-01-27 17:15:43 +01:00
Eelco Dolstra 35dbdbedd4 nix store ping: Report Nix daemon version
Fixes #5952.
2022-01-25 21:15:58 +01:00
Eelco Dolstra 076945c808
Merge pull request #5875 from hercules-ci/fix-large-drv-field-stack-overflow
Fix segfault or stack overflow caused by large derivation fields
2022-01-24 12:05:30 +01:00
Robert Hensing 0407436b0f
derivations.cc: Use larger buffer in printString
If we want to be careful about hitting the stack protector page, we should use `-fstack-check` instead.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-01-21 17:25:37 +01:00
Eelco Dolstra e85cf34ea3
Merge pull request #5949 from obsidiansystems/old-copyClosure
Add back `copyClosure` for plain `StorePath`s
2022-01-21 11:33:32 +01:00
John Ericson 5ee937523d Add back copyClosure for plain StorePaths
This was removed in 2e199673a5 when
`copyPath` transitioned to use `RealisedPath`. But then in
e9848beca7 we added it back just for
`realisedPath`.

I think it is a good utility function --- one can easily imagine it
becoming optimized in the future, and copying paths *violating* the
closure is a very niche feature.

So if we have `copyPaths` for both sorts of paths, I think we should
have `copyClosure` for both sorts too.
2022-01-20 20:57:44 +00:00
Robert Hensing dec7748119 Replace withBuffer by boost small_vector
Although this will leave gaps in the stack, the performance impact
of those should be insignificant and we get a simpler solution
this way.
2022-01-19 15:21:56 +01:00
Robert Hensing 6dd271b7b4 withBuffer: avoid allocating a std::function 2022-01-19 15:21:56 +01:00
Robert Hensing 55c58580be Add withBuffer
... to avoid non-standard, unidiomatic alloca.
2022-01-19 15:21:56 +01:00
Robert Hensing d038a67bd3 Fix segfault or stack overflow caused by large derivation fields
This removes a dynamic stack allocation, making the derivation
unparsing logic robust against overflows when large strings are
added to a derivation.
Overflow behavior depends on the platform and stack configuration.

For instance, x86_64-linux/glibc behaves as (somewhat) expected:

$ (ulimit -s 20000; nix-instantiate tests/lang/eval-okay-big-derivation-attr.nix)
error: stack overflow (possible infinite recursion)

$ (ulimit -s 40000; nix-instantiate tests/lang/eval-okay-big-derivation-attr.nix)
error: expression does not evaluate to a derivation (or a set or list of those)

However, on aarch64-darwin:

$ nix-instantiate big-attr.nix                                                                                                                                                                                                                                                       ~
zsh: segmentation fault  nix-instantiate big-attr.nix

This indicates a slight flaw in the single stack protection page
approach that is not encountered with normal stack frames.
2022-01-19 15:21:56 +01:00
regnat e36add56cf Fix the build with nlohmann/json 3.10.4+ 2022-01-19 14:37:54 +01:00
lincoln auster [they/them] 3e5a9ad7ff allow modifying lockfile commit msg with nix config option
This allows setting the commit-lockfile-summary option to a non-empty
string to override the commit summary while leaving the body unchanged.
2022-01-18 16:10:27 -07:00
Eelco Dolstra 5b243a2b4b BinaryCacheStore: Implement addBuildLog() 2022-01-18 16:14:01 +01:00
Eelco Dolstra 4dda1f92aa Add command 'nix store copy-log'
Fixes #5222.
2022-01-18 14:08:49 +01:00
Eelco Dolstra f6f0bcf11f
Doh 2022-01-18 14:06:51 +01:00
Eelco Dolstra 50be51d9a8 Doh 2022-01-18 13:50:25 +01:00
Eelco Dolstra d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string>
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
2022-01-18 11:12:30 +01:00
Eelco Dolstra 776eb97a43 serialise.hh: Use std::string_view 2022-01-17 22:20:23 +01:00
Eelco Dolstra 8a446aff75
Merge pull request #5898 from layus/repair-path-links
Make --repair-path also repair corrupt optimised links
2022-01-11 14:14:44 +01:00
Guillaume Maudoux 9f9f39a24b Prefer RepairFlag over bool when applicable 2022-01-11 13:38:28 +01:00
Guillaume Maudoux e9a4abdb5d Make --repair-path also repair corrupt optimised links
There already existed a smoke test for the link content length,
but it appears that there exists some corruptions pernicious enough
to replace the file content with zeros, and keeping the same length.

--repair-path now goes as far as checking the content of the link,
making it true to its name and actually repairing the path for such
coruption cases.
2022-01-11 11:57:45 +01:00
Eelco Dolstra 19fd6e585d 'target' points to 'source' 2022-01-10 16:52:25 +01:00
Eelco Dolstra df11e75d0e Merge branch 'recursive-nix-fix' of https://github.com/L-as/nix 2022-01-10 16:51:59 +01:00
Eelco Dolstra be64fb9b51 DerivationGoal::loadDerivation(): Don't use derivationFromPath()
This causes a recursive call to ensurePath(), which is not a good
idea.
2022-01-07 13:22:34 +01:00
Domen Kožar bdc577936f
Merge pull request #5749 from GavinRay97/patch-1
Better diagnostics if no valid signature found
2022-01-06 08:39:58 +01:00
Eelco Dolstra 9747ea84b4 Remove CPU locking
This was already accidentally disabled in ba87b08. It also no longer
appears to be beneficial, and in fact slow things down, e.g. when
evaluating a NixOS system configuration:

  elapsed time:       median =      3.8170  mean =      3.8202  stddev =      0.0195  min =      3.7894  max =      3.8600  [rejected, p=0.00000, Δ=0.36929±0.02513]
2021-12-22 15:56:25 +01:00
Eelco Dolstra ec8f24ed3a Ignore EPERM when unsharing FS state
On Docker (but not podman), unshare(CLONE_FS) fails with EPERM. So
let's ignore it and hope nothing bad happens.

Attempted fix for #5777.
2021-12-16 22:02:50 +01:00
Eelco Dolstra 18e4851752
Merge pull request #5769 from NixOS/ca/get-build-stats
Add a crude tracing mechansim for the build results
2021-12-14 10:53:51 +01:00
Théophane Hufschmitt 6234e1c811
Merge pull request #5702 from baloo/baloo/hide-non-reproducible-settings
reproducibility: hide non-reproducible settings from manual
2021-12-14 09:14:25 +01:00
Théophane Hufschmitt 3fc8042f77
Merge pull request #5366 from trofi/speedup-ca-query
ca-specific-schema.sql: add index on RealisationsRefs(referrer)
2021-12-14 09:08:17 +01:00
Eelco Dolstra 95bd5da341 queryMissing(): Use a bigger thread pool
This is a temporary hack until we make queryMissing() async.

Closes #5324.
2021-12-13 21:11:37 +01:00
Eelco Dolstra 46d86e06ba Simplify 2021-12-13 20:28:53 +01:00
Eelco Dolstra 9529de2eed Merge branch 'arm32-personality' of https://github.com/cleverca22/nix 2021-12-13 19:58:40 +01:00
Eelco Dolstra d17d46cfc2 More debug info 2021-12-13 19:03:08 +01:00
Eelco Dolstra 05fcc681ac Retry on ECONNREFUSED
https://hydra.nixos.org/build/161439235
2021-12-13 19:00:19 +01:00
Eelco Dolstra 6d30f9e6fe Explicitly make GC roots client sockets blocking
On macOS / BSD, these sockets inherit the non-blocking flag of the
server soocket, which is not what we want.

https://github.com/dotnet/runtime/issues/25069
https://bugs.python.org/issue7995

https://hydra.nixos.org/build/161439304
2021-12-13 18:59:50 +01:00
regnat 2eec2f765a Add a crude tracing mechansim for the build results
Add a `_NIX_TRACE_BUILT_OUTPUTS` environment variable that can be set to
a filename in which the result of each build will be logged.

This is intentionally crude and undocumented as it’s only meant to be a
temporary thing to assess the usefulness of CA derivations.
Any other use would need a cleaner re-implementation first.
2021-12-13 17:02:14 +01:00
regnat 55dbb7f1cc More properly track the status of CA builds
Make the build of unresolved derivations return the same status as the
resolved one, except in the case of an `AlreadyValid` in which case it
will return `ResolvesToAlreadyValid` to mean that the outputs of the unresolved
derivation weren’t known, but the resolved one is.
2021-12-13 17:02:13 +01:00
Gavin Ray 3542d4fe16
Incorporate suggestions from @edolstra 2021-12-10 19:02:22 -05:00
Eelco Dolstra 20b1290103 Ignore system.nfs4_acl
Closes #1584.
2021-12-10 15:12:55 +01:00
Eelco Dolstra f9437b4f18 Merge branch 'master' of https://github.com/MagicRB/nix 2021-12-10 14:49:25 +01:00
Eelco Dolstra bc57b3854e
Merge pull request #5695 from obsidiansystems/tidy-logging
Tidy up the logging
2021-12-09 15:31:08 +01:00
Gavin Ray 971382cab0
Better diagnostics if no valid signature found
I downloaded Nix tonight, and immediately broke it by accidentally removing the default binary caching.
After figuring this out, I also failed to fix it properly, due to using the wrong key for Nix's default binary cache

If the diagnostic message would have been clearer about what/where a "signature" for a "substituter" is + comes from, it probably would have saved me a few hours.
Maybe we can save other noobs the same pain?
2021-12-08 19:55:34 -05:00
Eelco Dolstra 92ff5b4254 Tweak warning 2021-12-07 22:22:02 +01:00
Eelco Dolstra 21206caf13 Merge remote-tracking branch 'origin/5712-ca-drvs-with-old-daemon' 2021-12-07 22:06:22 +01:00
Jan Tojnar ae21aab456 Update manual links
Fixes: https://github.com/NixOS/nixos-homepage/issues/762
2021-12-06 16:42:57 +01:00
regnat fa58bff1ab Fallback when the daemon is too old for CA derivations
Fix #5712
2021-12-06 11:45:18 +01:00
Eelco Dolstra 782837d934 Merge branch 'nix-env-install-suggestions' of https://github.com/tweag/nix 2021-12-01 21:47:28 +01:00
Arthur Gautier 21520297da reproducibility: hide non-reproducible settings from manual
Because the manual is generated from default values which are themselves
generated from various sources (cpuid, bios settings (kvm), number of
cores). This commit hides non-reproducible settings from the manual
output.
2021-12-01 17:25:58 +01:00
John Ericson 2f5c913d4a Fix #5299
No matter what, we need to resize the buffer to not have any scratch
space after we do the `read`. In the end of file case, `got` will be 0
from it's initial value.

Before, we forgot to resize in the EOF case with the break. Yes, we know
we didn't recieve any data in that case, but we still have the scatch
space to undo.

Co-Authored-By: Will Fancher <Will.Fancher@Obsidian.Systems>
2021-11-30 22:11:46 +00:00
John Ericson 95157b4e66 Push wopAddToStore old style stream adapters into smaller scopes
This doesn't fix the bug, but makes the code less difficult to read.
Also improve the comments, now that it is clear what part is needed in
each code path.
2021-11-30 22:10:58 +00:00
John Ericson 70a717f7a8 Nix daemon stream old wopAddToStore
No more buffering in string.
2021-11-30 21:55:53 +00:00
John Ericson 1f15441103 Tidy up the logging
Use the macros more, so we properly skip work when the log level
excludes. Also log the daemon operation number on the daemon side.
2021-11-30 20:23:13 +00:00
Eelco Dolstra d5d0621250
Merge pull request #5603 from obsidiansystems/recursive-nix-system-feature
"recursive-nix" system feature only with experimental feature
2021-11-25 17:20:58 +01:00
Las Safin 8388d2c7c6
Make recursive-nix work even when not privileged
Before this, `setns` would fail when switching to the mount namespace,
since we did not have the privileges to do so when not root.

Closes #5360
2021-11-25 11:15:11 +00:00
John Ericson 286eb81143 "recursive-nix" system feature only with experimental feature 2021-11-23 20:19:04 +00:00
Alex Shabalin b26cb0c9ac Fix use after free in content-address.cc
Inspired by https://github.com/NixOS/nix/pull/5599
2021-11-23 16:15:34 +01:00
Eelco Dolstra dbfcaa607a
Merge pull request #5628 from matthewbauer/rosetta2-new-path
Set new rosetta 2 path in sandbox
2021-11-23 10:46:50 +01:00
Matthew Bauer 08b1ac3e38 Set new rosetta 2 path in sandbox
see:

https://github.com/NixOS/nix/pull/5388

and

https://github.com/NixOS/nix/pull/5251
2021-11-22 14:42:31 -06:00
Alex Shabalin db2e4489a5 Unify #if linux 2021-11-22 13:57:56 +01:00
Alex Shabalin a18d9269a5 Fix build warnings on MacOS 2021-11-19 15:22:31 +01:00
Alex Shabalin 4b28798bfc Offer suggestions for nix-env -i
Closes https://github.com/NixOS/nix/issues/972
2021-11-18 15:34:57 +01:00
Eelco Dolstra 0961c1068a Merge branch 'accept_conf' of https://github.com/tomberek/nix 2021-11-18 12:56:09 +01:00
Eelco Dolstra 262a3c7ce3 Simplify 2021-11-18 12:12:31 +01:00
Farid Zakaria bc14465e08 Fix stack buffer overflow
Fix a stack buffer overflow found by running MemorySanitizer.
2021-11-18 04:05:25 +00:00
Eelco Dolstra 6463eaca14
Merge pull request #5472 from NixOS/async-realisation-substitution
async realisation substitution
2021-11-16 12:54:20 +01:00
Tom Bereknyei 30496af598 Adds an accept-flake-config flag 2021-11-12 09:50:07 -05:00
Eelco Dolstra bceda30498 Typo 2021-11-12 13:41:15 +01:00
Sergei Trofimovich edfc5b2f12 ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.

RealisationsRefs(referrer)
--------------------------

Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:

    $ nix edit -f . bash # add unused environment variable, like FOO="1"
    # populate RealisationsRefs, build fresh system
    $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
    $ nix edit -f . bash # add unused environment variable, like FOO="2"
    $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'

In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).

Before the change:

    $ time nix build -f nixos system ... # step 4 above
    real    34m3,613s
    user    0m5,232s
    sys     0m0,758s

Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.

After the change:

    $ time nix build -f nixos system ... # step 4 above
    real    4m50,061s
    user    0m5,038s
    sys     0m0,677s

Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.

Realisations(outputPath)
------------------------

After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.

The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.

Before the change:
    $ time nix-collect-garbage -d --max-freed 100G
    <interrupted before finish, took too long>
    real    23m32.411s
    user    17m49.679s
    sys     4m50.609s

Most of time was spent in re-scanning Realisations table on each path deletion.

After the change:
    $ time nix-collect-garbage -d --max-freed 100G

    real    8m43.226s
    user    6m16.317s
    sys     1m40.188s

Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-11-10 08:32:05 +00:00
Eelco Dolstra 67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev 0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra ff2af4d64e Unshare mount namespace in main()
Doing it as a side-effect of calling LocalStore::makeStoreWritable()
is very ugly.

Also, make sure that stopping the progress bar joins the update
thread, otherwise that thread should be unshared as well.
2021-11-08 22:00:45 +01:00
Eelco Dolstra 7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Alexander Bantyev 9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
regnat 1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
Doron Behar 14fcf17277 libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-03 14:19:11 +02:00