Commit graph

6286 commits

Author SHA1 Message Date
regnat e9b39f6004 Restrict the operations on drv outputs in recursive Nix
There's currently no way to properly filter them, so disallow them
altogether instead.
2020-12-11 21:17:25 +01:00
regnat bab1cda0e6 Use the hash modulo in the derivation outputs
Rather than storing the derivation outputs as `drvPath!outputName` internally,
store them as `drvHashModulo!outputName` (or `outputHash!outputName` for
fixed-output derivations).

This makes the storage slightly more opaque, but enables an earlier
cutoff in cases where a fixed-output dependency changes (but keeps the
same output hash) − same as what we already do for input-addressed
derivations.
2020-12-11 21:17:23 +01:00
regnat 8914e01e37 Store the realisations as JSON in the binary cache
Fix #4332
2020-12-11 21:05:09 +01:00
regnat 3ac9d74eb1 Rework the db schema for derivation outputs
Add a new table for tracking the derivation output mappings.

We used to hijack the `DerivationOutputs` table for that, but (despite its
name), it isn't a really good fit:

- Its entries depend on the drv being a valid path, making it play badly with
  garbage collection and preventing us to copy a drv output without copying
  the whole drv closure too;
- It dosen't guaranty that the output path exists;

By using a different table, we can experiment with a different schema better
suited for tracking the output mappings of CA derivations.
(incidentally, this also fixes #4138)
2020-12-11 20:41:32 +01:00
regnat 58cdab64ac Store metadata about drv outputs realisations
For each known realisation, store:
- its output
- its output path

This comes with a set of needed changes:

- New `realisations` module declaring the types needed for describing
  these mappings
- New `Store::registerDrvOutput` method registering all the needed informations
  about a derivation output (also replaces `LocalStore::linkDeriverToPath`)
- new `Store::queryRealisation` method to retrieve the informations for a
  derivations

This introcudes some redundancy on the remote-store side between
`wopQueryDerivationOutputMap` and `wopQueryRealisation`.
However we might need to keep both (regardless of backwards compat)
because we sometimes need to get some infos for all the outputs of a
derivation (where `wopQueryDerivationOutputMap` is handy), but all the
stores can't implement it − because listing all the outputs of a
derivation isn't really possible for binary caches where the server
doesn't allow to list a directory.
2020-12-11 20:41:32 +01:00
Michael Bishop 63b3536f50
treat s3 permission errors as file-not-found
Signed-off-by: Jonathan Ringer <jonringer117@gmail.com>
2020-12-11 09:49:24 -08:00
Eelco Dolstra 9c143c411b
Merge pull request #4350 from NixOS/ca/fix-build-with-nix-command
Fix the `nix` command with CA derivations
2020-12-11 15:01:49 +01:00
regnat eb45308109 Fix the nix command with CA derivations
Prevents a crash because most `nix` subcommands assumed that derivations
know their output path, which isn't the case for CA derivations
2020-12-11 10:28:09 +01:00
Eelco Dolstra c6a1bcd0ec nix store make-content-addressable: Show rewritten path 2020-12-10 17:11:56 +01:00
Eelco Dolstra a8f533b664 Add lvlNotice log level
This is like syslog's LOG_NOTICE: "normal, but significant,
condition".
2020-12-10 16:41:24 +01:00
Eelco Dolstra 253571e4ec
Merge pull request #4342 from tweag/fix-remote-build-hook
fix remote build hook
2020-12-09 12:40:00 +01:00
Maximilian Bosch 93a8a005de
libstore/openStore: fix stores with IPv6 addresses
In `nixStable` (2.3.7 to be precise) it's possible to connect to stores
using an IPv6 address:

  nix ping-store --store ssh://root@2001:db8::1

This is also useful for `nixops(1)` where you could specify an IPv6
address in `deployment.targetHost`.

However, this behavior is broken on `nixUnstable` and fails with the
following error:

  $ nix store ping --store ssh://root@2001:db8::1
  don't know how to open Nix store 'ssh://root@2001:db8::1'

This happened because `openStore` from `libstore` uses the `parseURL`
function from `libfetchers` which expects a valid URL as defined in
RFC2732. However, this is unsupported by `ssh(1)`:

  $ nix store ping --store 'ssh://root@[2001:db8::1]'
  cannot connect to 'root@[2001:db8::1]'

This patch now allows both ways of specifying a store (`root@2001:db8::1`) and
also `root@[2001:db8::1]` since the latter one is useful to pass query
parameters to the remote store.

In order to achieve this, the following changes were made:

* The URL regex from `url-parts.hh` now allows an IPv6 address in the
  form `2001:db8::1` and also `[2001:db8::1]`.

* In `libstore`, a new function named `extractConnStr` ensures that a
  proper URL is passed to e.g. `ssh(1)`:

  * If a URL looks like either `[2001:db8::1]` or `root@[2001:db8::1]`,
    the brackets will be removed using a regex. No additional validation
    is done here as only strings parsed by `parseURL` are expected.

  * In any other case, the string will be left untouched.

* The rules above only apply for `LegacySSHStore` and `SSHStore` (a.k.a
  `ssh://` and `ssh-ng://`).

Unresolved questions:

* I'm not really sure whether we want to allow both variants of IPv6
  addresses in the URL parser. However it should be noted that both seem
  to be possible according to RFC2732:

  > This document incudes an update to the generic syntax for Uniform
  > Resource Identifiers defined in RFC 2396 [URL].  It defines a syntax
  > for IPv6 addresses and allows the use of "[" and "]" within a URI
  > explicitly for this reserved purpose.

* Currently, it's not supported to specify a port number behind the
  hostname, however it seems as this is not really supported by the URL
  parser. Hence, this is probably out of scope here.
2020-12-09 12:23:29 +01:00
regnat c87267c2a4 Store the final drv outputs in memory when building remotely
The `DerivationGoal` has a variable storing the “final” derivation
output paths that is used (amongst other things) to fill the environment
for the post build hook. However this variable wasn't set when the
build-hook is used, causing a crash when both hooks are used together.

Fix this by setting this variable (from the informations in the db) after a run
of the post build hook.
2020-12-09 10:45:12 +01:00
regnat 6758e65612 Revert "Re-query for the derivation outputs in the post-build-hook"
This reverts commit 1b1e076033.

Using `queryPartialDerivationOutputMap` assumes that the derivation
exists locally which isn't the case for remote builders.
2020-12-09 09:44:07 +01:00
Eelco Dolstra 82e5511594
Merge pull request #4325 from tweag/hide-local-store-sql-statements
Hide the sqlite statements declarations for the local store
2020-12-08 14:04:16 +01:00
regnat c0f21f08f8 Hide the sqlite statements declarations for the local store
These have no need to be in the public interface and it causes spurious
rebuilds each time one wants to add or remove a new statement.
2020-12-08 13:29:13 +01:00
regnat 1b1e076033 Re-query for the derivation outputs in the post-build-hook
We can't assume that the runtime state knows about them as they might have
been built remotely, in which case we must query the db again to get
them.
2020-12-08 11:11:02 +01:00
Matthew Bauer aa07502009 Always default to cache.nixos.org even when different nix store dir
Since 0744f7f, it is now useful to have cache.nixos.org in substituers
even if /nix/store is not the Nix Store Dir. This can always be
overridden via configuration, though.
2020-12-06 23:04:42 -06:00
Eelco Dolstra 1d1a85eb0a
Merge pull request #4318 from matthewbauer/add-slash-to-trusted-binary-cache
Canonicalize binary caches with ‘/’ when one is missing
2020-12-05 10:17:17 +01:00
Matthew Bauer b9a00fd15b Canonicalize binary caches with ‘/’ when one is missing
This checks if there is a trusted substituter with a slash, so
trusting https://cache.nixos.org also implies https://cache.nixos.org/
is trusted.
2020-12-04 22:17:19 -06:00
Matthew Bauer 3c9b7029ba Use com.apple.oahd.plist for rosetta 2 detection 2020-12-04 13:26:53 -06:00
Stéphan Kochen e20a3ec756 Fix compatibility with newer AWS SDKs
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-04 19:36:09 +01:00
Eelco Dolstra f337aa7099 Split 'nix store add-to-store' into 'add-path' and 'add-file'
This makes it consistent with 'nix hash <path|file>'.
2020-12-04 00:59:24 +01:00
Eelco Dolstra fa8dad10ed Typo 2020-12-03 23:26:23 +01:00
Eelco Dolstra ea2062a2d9 Move most store-related commands to 'nix store' 2020-12-03 23:22:22 +01:00
Eelco Dolstra a1cd805cba Add 'nix nar dump-path'
This only differs from 'nix store dump-path' in that the path doesn't
need to be a store path.
2020-12-03 22:52:01 +01:00
Eelco Dolstra af373c2ece Add deprecated aliases for renamed commands 2020-12-03 22:45:44 +01:00
Matthew Bauer 4b9acf4e21 Use posix_spawn_setbinpref_np to advise which architecture to run
When running universal binaries like /bin/bash, Darwin XNU will choose
which architecture of the binary to use based on "binary preferences".
This change sets that to the current platform for aarch64 and x86_64
builds. In addition it now uses posix_spawn instead of the usual
execve. Note, that this does not prevent the other architecture from
being run, just advises which to use.

Unfortunately, posix_spawnattr_setbinpref_np does not appear to be
inherited by child processes in x86_64 Rosetta 2 translations, meaning
that this will not always work as expected.

For example:

  {
    arm = derivation {
      name = "test";
      system = "aarch64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = arm64 ]
        /usr/bin/touch $out
      '') ];
    };
    rosetta = derivation {
      name = "test";
      system = "x86_64-darwin";
      builder = "/bin/bash";
      args = [ "-e" (builtins.toFile "test" ''
        set -x
        /usr/sbin/sysctl sysctl.proc_translated
        /usr/sbin/sysctl sysctl.proc_native
        [ "$(/usr/bin/arch)" = i386 ]
        echo It works!
        /usr/bin/touch $out
      '') ];
    };
  }

`arm' fails on x86_64-compiled Nix, but `arm' and `rosetta' succeed on
aarch64-compiled Nix. I suspect there is a way to fix this since:

  $ /usr/bin/arch -arch x86_64 /bin/bash \
    -c '/usr/bin/arch -arch arm64e /bin/bash -c /usr/bin/arch'
  arm64

seems to work correctly. We may need to wait for Apple to update
system_cmds in opensource.apple.com to find out how though.
2020-12-03 15:41:59 -06:00
Matthew Bauer 9b1824ecbd Add extraPlatforms for Rosetta 2 macOS
macOS systems with ARM64 can utilize a translation layer at
/Library/Apple/usr/libexec/oah to run x86_64 binaries. This change
makes Nix recognize that and it to "extra-platforms". Note that there
are two cases here since Nix could be built for either x86_64 or
aarch64. In either case, we can switch to the other architecture.
Unfortunately there is not a good way to prevent aarch64 binaries from
being run in x86_64 contexts or vice versa - programs can always
execute programs for the other architecture.
2020-12-03 15:41:43 -06:00
Eelco Dolstra 0c15ae5d4b Add FIXME 2020-12-03 20:31:45 +01:00
Eelco Dolstra ef583303f0 Move NAR-related commands to 'nix nar' 2020-12-03 18:09:02 +01:00
Eelco Dolstra 79c1967ded Introduce 'nix store' command 2020-12-03 18:07:13 +01:00
Eelco Dolstra 5781f45c46 Allow registering subcommands of subcommands 2020-12-03 17:55:55 +01:00
Eelco Dolstra b2d6c6161e Move 'nix hash-*' and 'nix to-*' to 'nix hash'
From the 'nix' UX review.
2020-12-03 17:55:55 +01:00
Eelco Dolstra 8ad2c9c4b9 Remove 'dist' target
We're not producing source tarballs anymore so this has been
bitrotting.
2020-12-03 16:17:58 +01:00
Eelco Dolstra 1b0ca3866b nix add-to-store: Move markdown docs into a separate file 2020-12-03 16:17:58 +01:00
Eelco Dolstra c3c858ac6d Make doc() return arbitrary Markdown rather than the contents of the "Description" section
Thus we can return the examples section (and any other sections) from
doc() and don't need examples() anymore.
2020-12-03 16:17:58 +01:00
Eelco Dolstra 0bd060f23a
Merge pull request #4308 from tweag/properly-test-early-cutoff
Properly test the early cutoff for CA derivations
2020-12-03 14:45:29 +01:00
Eelco Dolstra 4f25644a13
Merge pull request #4304 from NixOS/separate-manpages
Separate manpages for 'nix' subcommands
2020-12-03 13:38:29 +01:00
regnat 0afab668fa Don't fail early when -j0 is passed
If the build closure contains some CA derivations, then we can't know
ahead-of-time that we won't build anything as early-cutoff might come-in
at a laster stage
2020-12-03 13:24:36 +01:00
Eelco Dolstra df552a2645 nix eval: Add option to write a directory
This is useful for generating the nix manpages, but it may have other
applications (like generating configuration files without a Nix store).
2020-12-02 23:23:23 +01:00
Eelco Dolstra 148608ba6d Add 'nix help' 2020-12-02 23:23:23 +01:00
Greg Hale d8fc1bb7b0 fix tokens documentation 2020-12-02 10:15:18 -05:00
regnat 0d9e1af695 Remove an unknown pragma gcc warning 2020-12-02 14:33:20 +01:00
regnat a8a96dbaf8 Add forgotten override annotation 2020-12-02 14:23:38 +01:00
Eelco Dolstra 1b79b5b983 read(): Use char * instead of unsigned char *
This gets rid of some pointless casts.
2020-12-02 14:17:27 +01:00
Eelco Dolstra faa31f4084 Sink: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra aa68486112 writeFull/writeFile: Use std::string_view 2020-12-02 14:17:27 +01:00
Eelco Dolstra e5cf501c77
Merge pull request #4284 from tweag/fixed-output-depending-on-ca
Allow fixed-output derivations to depend on (floating) content-addressed ones
2020-12-01 20:25:41 +01:00
regnat 438977731c shut up clang warnings
- Fix some class/struct discrepancies
- Explicit the overloading of `run` in the `Cmd*` classes
- Ignore a warning in the generated lexer
2020-12-01 15:04:03 +01:00
Eelco Dolstra 88798613ee replaceStrings(): Use std::string_view 2020-12-01 13:45:43 +01:00
Eelco Dolstra c0d1354b7d Macro hygiene 2020-12-01 13:45:06 +01:00
Eelco Dolstra 3b7e00ce22 Move primeCache() to Worker::run()
We need the missing path info to communicate the worker's remaining
goals to the progress bar.
2020-12-01 13:44:48 +01:00
Eelco Dolstra 5927624473 Lower verbosity for 'Failed to find a machine' message 2020-12-01 13:43:36 +01:00
Eelco Dolstra e224c16d28 Macro hygiene 2020-12-01 13:43:33 +01:00
regnat 9bd8184f1f Allow fixed-output derivations to depend on (floating) content-addressed ones
Fix an overlook of https://github.com/NixOS/nix/pull/4056
2020-11-27 15:39:24 +01:00
Eelco Dolstra 05d9442f68 builtins.fetchGit: Fix shortRev attribute for dirty trees 2020-11-26 21:45:28 +01:00
Dominique Martinet 1fd13d67e8 archive: disable preallocate-contents by default
using fallocate() to preallocate files space does more harm than good:
 - breaks compression on btrfs
 - has been called "not the right thing to do" by xfs developers
(because delayed allocation that most filesystems implement leads to smarter
allocation than what the filesystem needs to do if we upfront fallocate files)
2020-11-26 14:26:57 +01:00
Eelco Dolstra 8252a44e96 Move to separate file 2020-11-26 13:16:36 +01:00
Eelco Dolstra 9a586e34ac Record trusted/untrusted settings in ~/.local/share/nix 2020-11-26 13:11:07 +01:00
Eelco Dolstra 0287f83057 Ask for confirmation before allowing flake Nix configuration settings 2020-11-26 12:37:23 +01:00
regnat 13c557fe82 fix the hash rewriting for ca-derivations 2020-11-25 11:33:00 +01:00
Lily Ballard 437189e446 Escape filename given to nix-shell in shebang mode
This prevents spaces or other metacharacters from causing nix-shell to
execute the wrong path.

Fixes #4229.
2020-11-24 15:08:37 -08:00
Luke Granger-Brown 226116f482 fetchMercurial: set HGPLAIN when invoking hg
Without setting HGPLAIN, the user's environment leaks into
hg invocations, which means that the output may not be in the
expected format.

HGPLAIN is the Mercurial-recommended solution for this in that
it's intended for uses by scripts and programs which are looking
to parse Mercurial's output in a consistent manner.
2020-11-23 16:12:33 +00:00
Wil Taylor 07603890d2 Removed reference check from bundler command 2020-11-23 21:19:40 +10:00
Wil Taylor c3bad73e27 Added switch 2020-11-21 14:28:49 +10:00
Eelco Dolstra 4dcb183af3 AttrCursor::getStringWithContext(): Force re-evaluation if the cached context is not valid
Fixes #4236.
2020-11-19 20:59:36 +01:00
Eelco Dolstra 0327580e54 Fix assertion failure in LockFile::LockFile()
Fixes #4241.
2020-11-19 20:31:30 +01:00
John Ericson 2113ae2d85 Make drv hash modulo memo table thread-safe
Let's get one step closer to the daemon not needing to fork.
2020-11-19 16:50:06 +00:00
Eelco Dolstra 4dbd05e933
Merge pull request #4189 from edolstra/flake-config
Allow nix.conf options to be set in flake.nix
2020-11-17 15:39:14 +01:00
Eelco Dolstra f4e790cc85
Merge pull request #4182 from mkenigs/fix-1930
Print built derivations as json for build
2020-11-17 14:59:49 +01:00
Eelco Dolstra df5c69a94e
Merge pull request #4180 from Ma27/ssh-ng-substitute
Allow substituting paths when building remotely using `ssh-ng://`
2020-11-17 14:01:04 +01:00
Eelco Dolstra e6b7c7b79c Cleanup 2020-11-17 13:58:55 +01:00
Eelco Dolstra bccff827dc Fix deadlock in IFD through the daemon
Fixes #4235.
2020-11-17 13:50:36 +01:00
regnat 7de21f6664 Make the sql debug statements more useful
Print the expanded sql query (with the variables bound to their value) rather
than the original one in case of error
2020-11-17 10:05:45 +01:00
Eelco Dolstra ef84c780bb
filterANSIEscapes(): Handle UTF-8 characters 2020-11-16 16:41:53 +01:00
Eelco Dolstra 0d6419ad87
Merge pull request #4262 from jbaum98/master
Fix deadlock in nix-store when max-connections=1
2020-11-16 10:20:36 +01:00
Jake Waksbaum 01db455733 Fix deadlock in nix-store when max-connections=1
This fixes a bug I encountered where `nix-store -qR` will deadlock when
the `--include-outputs` flag is passed and `max-connections=1`.

The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes
the only connection from the connection pool and uses it to check the
daemon version. If the version is new enough, it calls
`Store::queryDerivationOutputs`, which eventually calls
`RemoteStore::queryPartialDerivationOutputMap`, where we take another
connection from the connection pool to check the version again. Because
we still haven't released the connection from the caller, this waits for
a connection to be available, causing a deadlock.

This diff solves the issue by using `getProtocol` to check the protocol
version in the caller `RemoteStore::queryDerivationOutputs`, which
immediately frees the connection back to the pool before returning the
protocol version. That way we've already freed the connection by the
time we call `RemoteStore::queryPartialDerivationOutputMap`.
2020-11-16 02:35:50 -05:00
Robert Hensing ac5081d280 nix-build: Fix #4197 output order regression 2020-11-13 17:49:27 +01:00
Christian Höppner c4c3c15c19 Fix default nix-path
The default nix-path values for nixpkgs and root channels were
incorrect.
2020-11-12 15:46:08 +00:00
Ben Burdette b327de9c2d change message 2020-11-11 11:09:59 -07:00
Ben Burdette 8895fa70a4 pare down the error message 2020-11-11 11:05:21 -07:00
Ben Burdette 3edfe6090e missing argument error 2020-11-11 09:29:32 -07:00
Matthew Kenigsberg 8abb80a478 Print built derivations as json for build
Add --json option to nix build to allow machine readable output on
stdout with all built derivations

Fixes #1930
2020-11-11 10:27:02 -06:00
Ben Burdette 7d9037035e usage example location 2020-11-11 09:21:26 -07:00
Eelco Dolstra 4badb6943f Fix use of dirty Git/Mercurial inputs with chroot stores
Fixes:

  $ nix build --store /tmp/nix /home/eelco/Dev/patchelf#hydraJobs.build.x86_64-linux
  warning: Git tree '/home/eelco/Dev/patchelf' is dirty
  error: --- RestrictedPathError ------------------------------------------------------------------------------------------- nix
  access to path '/tmp/nix/nix/store/xmkvfmffk7xfnazykb5kx999aika8an4-source/flake.nix' is forbidden in restricted mode
  (use '--show-trace' to show detailed location information)
2020-11-10 23:22:45 +01:00
Eelco Dolstra cdc840d60b
Merge pull request #4242 from wizeman/fix-stack-overflow
Fix stack overflow introduced in #4206
2020-11-10 11:51:00 +01:00
Eelco Dolstra 3f680c1dcc
Merge pull request #4233 from Kha/master
nix develop: Preserve stdin with `-c`
2020-11-10 10:51:48 +01:00
Ricardo M. Correia 108a2dab7e Fix stack overflow introduced in #4206 2020-11-10 04:25:24 +01:00
Ben Burdette d8ef423a18 error message formatting 2020-11-09 19:16:50 -07:00
Ben Burdette 9f2b25ce55 remove unused ftn; reformat line breaks 2020-11-09 17:17:47 -07:00
Ben Burdette 6c2933a8d7 add position 2020-11-09 17:04:52 -07:00
Ben Burdette 107c91f5fe auto-call error 2020-11-09 16:48:35 -07:00
Sebastian Ullrich fb7735e4cf nix develop: Preserve stdin with -c 2020-11-09 22:43:14 +01:00
Eelco Dolstra b87f84cf55
Fix appending to Setting<StringSet>
Fixes: warning: unknown setting 'extra-sandbox-paths'
2020-11-09 15:04:34 +01:00
Maximilian Bosch 3a63fc6cd5
Allow substituting paths when building remotely using ssh-ng://
Until now, it was not possible to substitute missing paths from e.g.
`https://cache.nixos.org` on a remote server when building on it using
the new `ssh-ng` protocol.

This is because every store implementation except legacy `ssh://`
ignores the substitution flag passed to `Store::queryValidPaths` while
the `legacy-ssh-store` substitutes the remote store using
`cmdQueryValidPaths` when the remote store is opened with `nix-store
--serve`.

This patch slightly modifies the daemon protocol to allow passing an
integer value suggesting whether to substitute missing paths during
`wopQueryValidPaths`. To implement this on the daemon-side, the
substitution logic from `nix-store --serve` has been moved into a
protected method named `Store::substitutePaths` which gets currently
called from `LocalStore::queryValidPaths` and `Store::queryValidPaths`
if `maybeSubstitute` is `true`.

Fixes #2770
2020-11-05 20:12:37 +01:00
Eelco Dolstra 387f824cab
Merge pull request #4206 from hercules-ci/fix-coroutine-gc
Fix memory corruption caused by GC-invisible coroutine stacks
2020-11-05 10:18:31 +01:00
Eelco Dolstra 5e6eabe155 Fix error message 'assertion failed at' 2020-11-03 14:45:24 +01:00
Eelco Dolstra e8c379555f LocalStore: Get rid of recursive_mutex 2020-11-03 14:45:24 +01:00
Eelco Dolstra 797a52e31d Add FIXME 2020-11-03 14:45:24 +01:00
Eelco Dolstra 550e11f077 nix repl: Fix handling of multi-line expressions 2020-11-02 19:07:37 +01:00
Eelco Dolstra 7cf874c17d Don't use readDerivation() in addValidPath()
readDerivation() requires a valid path.

Fixes #4210.
2020-11-02 18:46:44 +01:00
Eelco Dolstra ab2ef851b6
Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlock
Fix RemoteStore pool deadlock in filterSource etc
2020-11-02 14:46:10 +01:00
Eelco Dolstra db5424bf09 Don't send eval-related settings to the daemon 2020-11-02 13:57:58 +01:00
Robert Hensing e8a45d07bc Restore RestrictedStore.addToStoreFromDump implementation
It was accidentally removed in commit ca30abb3fb
2020-10-31 23:56:03 +01:00
Robert Hensing b43c13a916 BoehmGCStackAllocator: increase stack size to 8MB
The default stack size was not based on the normal stack size and
was too small.
2020-10-30 23:18:26 +01:00
Robert Hensing 2192cac634 Fix RemoteStore pool deadlock in filterSource etc 2020-10-30 21:47:34 +01:00
Robert Hensing c4d903ddb0 Fix memory corruption caused by GC-invisible coroutine stacks
Crucially this introduces BoehmGCStackAllocator, but it also
adds a bunch of wiring to avoid making libutil depend on bdw-gc.

Part of the solutions for #4178, #4200
2020-10-30 21:21:59 +01:00
Eelco Dolstra dc5696b84f Fix test 2020-10-30 12:00:53 +01:00
Eelco Dolstra 7f56cf67ba Fix assertion failure in tab completion for --option 2020-10-29 18:26:35 +01:00
Eelco Dolstra ff4dea63c9 Generalize extra-* settings
This removes the extra-substituters and extra-sandbox-paths settings
and instead makes every array setting extensible by setting
"extra-<name> = <value>" in the configuration file or passing
"--<name> <value>" on the command line.
2020-10-29 18:17:39 +01:00
Eelco Dolstra bb8e837e4c
Merge pull request #4199 from stefanjaax/replaceWantedBySpecified
Alter "wanted:" to "specified:" in hash mismatch output
2020-10-29 07:38:38 +01:00
Eelco Dolstra 662e67f8de
Merge pull request #4198 from mkenigs/capitalize-JSON
Capitalize JSON for consistency
2020-10-29 07:34:34 +01:00
stev 869c0321ff Alter "wanted:" to "specified:" in hash mismatch output
This makes it even clearer which of the two hashes was specified in the
nix files. Some may think that "wanted" and "got" is obvious, but:
"got" could mean "got in nix file" and "wanted" could mean "want to see in nix file".
2020-10-29 00:33:14 +01:00
Matthew Kenigsberg 6a4bf535d8 Capitalize JSON for consistency 2020-10-28 17:54:28 -05:00
Eelco Dolstra a5019f0508 Consistency 2020-10-28 20:45:57 +01:00
Eelco Dolstra 02a1facbdc
Merge pull request #4056 from tweag/non-ca-depending-on-ca
Allow non-CA derivations to depend on CA ones
2020-10-27 17:38:29 +01:00
regnat bc081bcd81 Inline unkownHashes
See https://github.com/NixOS/nix/pull/4056#discussion_r493661632
2020-10-27 07:29:25 +01:00
regnat c092fa4702 Allow non-CA derivations to depend on CA derivations 2020-10-27 07:29:23 +01:00
Eelco Dolstra 343239fc8a Allow nix.conf options to be set in flake.nix
This makes it possible to have per-project configuration in flake.nix,
e.g. binary caches and other stuff:

  nixConfig.bash-prompt-suffix = "ngi# ";
  nixConfig.substituters = [ "https://cache.ngi0.nixos.org/" ];
2020-10-26 20:45:39 +01:00
Eelco Dolstra 731edf0d9b isTrivial(): Support trivial lists 2020-10-26 20:37:11 +01:00
Eelco Dolstra 14aecbb288 BaseSetting<StringMap>::set(): Don't append to previous value 2020-10-26 20:36:46 +01:00
Eelco Dolstra b875b8f45c Remove edition field 2020-10-26 17:59:36 +01:00
Eelco Dolstra 9d5e9ef0da Move Explicit 2020-10-26 17:01:20 +01:00
Eelco Dolstra dc7d1322ef Make the prompt used in development shells configurable 2020-10-26 14:24:25 +01:00
Eelco Dolstra 750ce500c2 Fix clang build 2020-10-22 13:40:30 +02:00
Eelco Dolstra f9438fb64a nix develop: Add --redirect flag to redirect dependencies
This is primarily useful if you're hacking simultaneously on a package
and one of its dependencies. E.g. if you're hacking on Hydra and Nix,
you would start a dev shell for Nix, and then a dev shell for Hydra as
follows:

  $ nix develop \
    --redirect .#hydraJobs.build.x86_64-linux.nix ~/Dev/nix/outputs/out \
    --redirect .#hydraJobs.build.x86_64-linux.nix.dev ~/Dev/nix/outputs/dev

(This assumes hydraJobs.build.x86_64-linux has a passthru.nix
attribute. You can also use a store path.)

This causes all references in the environment to those store paths to
be rewritten to ~/Dev/nix/outputs/{out,dev}. Note: unfortunately, you
may need to set LD_LIBRARY_PATH=~/Dev/nix/outputs/out/lib because
Nixpkgs' ld-wrapper only adds -rpath entries for -L flags that point
to the Nix store.
2020-10-22 13:40:30 +02:00
Eelco Dolstra e556a1beb7 nix develop: Handle 'declare -ax' in bash output
Fixes 'nix develop nixpkgs#qpdfview'.
2020-10-21 17:54:21 +02:00
Christian Kampka 461cf2b856
Add NIX_CONFIG env var for applying nix.conf overrides 2020-10-21 13:41:26 +02:00
Matthew Kenigsberg f6aaac2b59 Make bash non-interactive for nix develop --phase
Fix #3975: Currently if Ctrl-C is pressed during a phase, the interactive subshell
is not exited. Removing --rcfile when --phase is present makes bash
non-interactive
2020-10-20 12:00:30 -05:00
Eelco Dolstra 9635fb77bd
Merge pull request #4080 from kquick/kwq/flake-int-doc
Add some internal documentation for flake support objects.
2020-10-19 11:29:12 +02:00
Eelco Dolstra fda835b231
Merge pull request #4143 from obsidiansystems/typed-goal-maps
Properly type the derivation and substitution goal maps
2020-10-18 18:12:21 +02:00
Robert Hensing bd9eb5c743 DerivationGoal: only retry if output closure incomplete is only problem 2020-10-18 14:26:37 +02:00
Robert Hensing 94f1e4a441 Typo 2020-10-18 14:26:37 +02:00
John Ericson 7ed46c1574 Explain that upcast_goal is still a static cast 2020-10-17 21:50:12 +00:00
John Ericson 57d0432b39 Just use auto in two places. 2020-10-17 21:47:52 +00:00
John Ericson e6f8ae56d8 tab -> space 2020-10-17 21:45:31 +00:00
John Ericson 619d262c97 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-17 21:44:27 +00:00
John Ericson 64be1c15c2 Add missing include for MAX_PATH
And remove one that we didn't actually need to add
2020-10-15 19:05:17 +00:00
John Ericson fccef6a7fa Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-15 18:55:03 +00:00
John Ericson 0fefc2a439 Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-14 20:49:01 +00:00
Eelco Dolstra 11882d7c7c Create /etc/passwd *after* figuring out the sandbox uid/gid
Fixes build failures like

  # nix log /nix/store/gjaa0psfcmqvw7ivggsncx9w364p3s8s-sshd.conf-validated.drv
  No user exists for uid 30012
2020-10-14 12:20:58 +02:00
John Ericson 55592b253f Add some more docs 2020-10-13 18:04:24 +00:00
John Ericson 13804f126e Merge remote-tracking branch 'upstream/master' into typed-goal-maps 2020-10-13 18:02:32 +00:00
Eelco Dolstra 2653801939 Merge branch 'split_build_cc' of https://github.com/obsidiansystems/nix 2020-10-13 15:36:55 +02:00
John Ericson 1b8ebe92dc Merge remote-tracking branch 'obsidian/split_build_cc' into typed-goal-maps 2020-10-12 20:47:22 +00:00
John Ericson a73959e6be Merge remote-tracking branch 'upstream/master' into split_build_cc 2020-10-12 17:20:46 +00:00
John Ericson d334fd4882 Merge branches 'split_build_hh-0', 'split_build_hh-1', 'split_build_hh-2', 'split_build_hh-3' and 'split_build_hh-4' into split_build_cc 2020-10-12 17:20:12 +00:00
John Ericson 542972f029 Trim worker.hh 2020-10-12 17:16:48 +00:00
John Ericson 0e2306204a Rename to hand-hold git (worker.hh) 2020-10-12 17:16:48 +00:00
John Ericson 3ffa3546bd Trim substitution-goal.hh 2020-10-12 17:16:25 +00:00
John Ericson e77a2344d5 Rename to hand-hold git (substitution-goal.hh) 2020-10-12 17:16:25 +00:00
John Ericson d585b4c54f Trim hook-instance.hh 2020-10-12 17:16:13 +00:00
John Ericson 10b749a156 Rename to hand-hold git (hook-instance.hh) 2020-10-12 17:16:13 +00:00
John Ericson 8067d32f2a Trim goal.hh 2020-10-12 17:16:00 +00:00
John Ericson 0d0e345cdc Rename to hand-hold git (goal.hh) 2020-10-12 17:16:00 +00:00
John Ericson 2ce726947a Trim derivation-goal.hh 2020-10-12 17:15:32 +00:00
John Ericson 4eb8c69853 Rename to hand-hold git (derivation-goal.hh) 2020-10-12 17:15:32 +00:00
John Ericson f7099965bf Change .cc files to use split build headers 2020-10-12 17:08:52 +00:00
John Ericson 5a97621d6d Prepare for build/*.hh headers 2020-10-12 17:07:51 +00:00
volth eee18f88dd
Handle amount of disk space saved by hard linking being negative
Fixes bogus messages like "currently hard linking saves 17592186044416.00 MiB".
2020-10-12 16:06:38 +00:00
John Ericson 15fdb7cc6b Split out uds-remote-store.{cc.hh} 2020-10-11 17:37:05 +00:00
John Ericson 38e3897162 Copy {uds-,}remote-store.{cc,hh}
This prepares for the splitting that happens in the next commit.
2020-10-11 17:18:26 +00:00
John Ericson 5c74a6147b Properly type the derivation and substitution goal maps
As a bonus, Worker::removeGoal is less inefficient.
2020-10-11 17:07:14 +00:00
Eelco Dolstra b0fbf3a653
Merge pull request #4135 from obsidiansystems/split-local-fs-store-header
Split out `local-fs-store.hh`
2020-10-11 18:49:40 +02:00
John Ericson 8cc510fb79 Merge branches 'build-1', 'build-2', 'build-3', 'build-4', 'build-5', 'build-6', 'build-7' and 'build-8' into split_build_cc 2020-10-11 16:44:34 +00:00
John Ericson bcb67e1ed8 Trim lock.cc 2020-10-11 16:44:19 +00:00
John Ericson e0be04129b Rename to hand-hold git (lock.cc) 2020-10-11 16:44:14 +00:00
John Ericson dbc588651c Trim lock.hh 2020-10-11 16:44:07 +00:00
John Ericson d0004bfcab Rename to hand-hold git (lock.hh) 2020-10-11 16:43:59 +00:00
John Ericson eed53ed87a Trim build/worker.cc 2020-10-11 16:43:51 +00:00
John Ericson 904e315dae Rename to hand-hold git (build/worker.cc) 2020-10-11 16:43:42 +00:00
John Ericson d24ffe0eb1 Trim build/substitution-goal.cc 2020-10-11 16:43:24 +00:00
John Ericson 3633b3572b Rename to hand-hold git (build/substitution-goal.cc) 2020-10-11 16:43:18 +00:00
John Ericson 4bdff7d1b0 Trim build/local-store-build.cc 2020-10-11 16:43:12 +00:00
John Ericson dc5225cde5 Rename to hand-hold git (build/local-store-build.cc) 2020-10-11 16:43:05 +00:00
John Ericson 159054f730 Trim build/hook-instance.cc 2020-10-11 16:42:35 +00:00
John Ericson f0b8987299 Rename to hand-hold git (build/hook-instance.cc) 2020-10-11 16:42:08 +00:00
John Ericson 819fe848ac Trim build/goal.cc 2020-10-11 16:41:58 +00:00
John Ericson 184bfc301e Rename to hand-hold git (build/goal.cc) 2020-10-11 16:41:18 +00:00
John Ericson 3bab1c5bb0 Trim build/derivation-goal.cc 2020-10-11 16:41:11 +00:00
John Ericson 9629290eda Rename to hand-hold git (build/derivation-goal.cc) 2020-10-11 16:40:52 +00:00
John Ericson a4f0fecb03 Trim build.hh 2020-10-11 16:40:34 +00:00
John Ericson fc72cb0760 Rename to hand-hold git (build.hh) 2020-10-11 16:40:14 +00:00
John Ericson 428536fd75 Prepare for build/* files 2020-10-11 16:39:08 +00:00
John Ericson aef44cbaa9 Split out commonChildInit 2020-10-11 16:38:46 +00:00
John Ericson 6cc1541782 Split out local-fs-store.hh
This matches the already-existing `local-fs-store.cc`.
2020-10-09 20:18:08 +00:00
Eelco Dolstra 725488b892 nix develop: Unset $HOSTNAME
This is set to "localhost" by stdenv which is probably not what you
want.
2020-10-09 22:03:18 +02:00
Eelco Dolstra 44349064f7 nix develop: Source ~/.bashrc
Fixes #4104.
2020-10-09 22:02:00 +02:00
Eelco Dolstra 59bd6e87a4 Completions::add(): Guard against newlines 2020-10-09 21:55:59 +02:00
Eelco Dolstra ea4b2b985f
Merge pull request #4128 from tweag/extended-completions
Add a zsh completion script
2020-10-09 21:53:12 +02:00
John Ericson 39de73550d Merge remote-tracking branch 'upstream/master' into fix-and-ci-static-builds 2020-10-09 18:26:47 +00:00
Eelco Dolstra e845d19ae3 Remove Lazy
This fixes a crash during startup when compiling Nix as a single
compilation unit.
2020-10-09 17:54:59 +02:00
Eelco Dolstra 87157b2bd3 writeFile(): Add error context to writeFull() failure
Issue #4092.
2020-10-09 16:02:53 +02:00
Eelco Dolstra 636ec17139
Remove stray DerivationOutputsAndPaths type 2020-10-09 15:41:24 +02:00
regnat 04e5d0e704 Add a description in the completion outputs
Make nix output completions in the form `completion\tdescription`.
This can't be used by bash (afaik), but other shells like zsh or fish
can display it along the completion choices
2020-10-09 09:39:51 +02:00
Eelco Dolstra 97ffc1e013 Dynamically disable user namespaces if CLONE_NEWUSER fails
This makes builds work inside nixos-enter.

Fixes #3145.
2020-10-07 22:46:01 +02:00
Eelco Dolstra 6aa64627c8 Support user namespaces being disabled
If max_user_namespaces is set to 0, then don't run the build in a user
namespace.

Fixes #4092.
2020-10-07 22:02:36 +02:00
Eelco Dolstra f66bbd8c7b Doh 2020-10-07 21:25:06 +02:00
Eelco Dolstra e705c24294 Tweak error messages 2020-10-07 17:28:43 +02:00