Commit graph

7127 commits

Author SHA1 Message Date
Eelco Dolstra ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Chuck 3e2c77d001 Check for and repair bad .links entries
A corrupt entry in .links prevents adding a fixed version of that file
to the store in any path.  The user experience is that corruption
present in the store 'spreads' to new paths added to the store:

(With store optimisation enabled)

1. A file in the store gets corrupted somehow (eg: filesystem bug).
2. The user tries to add a thing to the store which contains a good copy
   of the corrupted file.
3. The file being added to the store is hashed, found to match the bad
   .links entry, and is replaced by a link to the bad .links entry.
   (The .links entry's hash is not verified during add -- this would
   impose a substantial performance burden.)
4. The user observes that the thing in the store that is supposed to be
   a copy of what they were trying to add is not a correct copy -- some
   files have different contents!  Running "nix-store --verify
   --check-contents --repair" does not fix the problem.

This change makes "nix-store --verify --check-contents --repair" fix
this problem.  Bad .links entries are simply removed, allowing future
attempts to insert a good copy of the file to succeed.
2019-11-15 11:55:36 -08:00
Eelco Dolstra fd900c45b5
Merge pull request #3220 from nh2/manual-nix-shell-p-expr
manual: nix-shell: Elaborate on using `-p` with expressions
2019-11-14 11:25:47 +01:00
Eelco Dolstra 0352c1a4f8
Typo 2019-11-13 17:18:17 +01:00
Eelco Dolstra 804910fb0e
Merge pull request #3213 from singron/fetchurl_test
Replace $TMPDIR with $TEST_ROOT in tests/fetchurl.sh
2019-11-11 12:15:59 +01:00
Eelco Dolstra 5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson 8669db1dcc Clean up semicolon and comma
Thanks @bhipple for catching!
2019-11-10 16:21:59 -05:00
Niklas Hambüchen 07294e988c manual: nix-shell: Elaborate on using -p with expressions.
This documents the outcome of the change in
https://github.com/NixOS/nix/issues/454:

> We can also automatically add parentheses in the generated
> `buildInputs`, so you can type `nix-shell -p "expr"`
> instead of `"(expr").
2019-11-10 17:29:13 +01:00
John Ericson 4c34054673 Remove unneeded semicolons 2019-11-10 11:24:47 -05:00
John Ericson 96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Domen Kožar 1f174226d1
Merge pull request #3218 from kolloch/patch-1
De-duplicate struct PrimOp forward declaration
2019-11-10 15:28:18 +01:00
Peter Kolloch 2ba9f22715
De-duplicate struct PrimOp forward declaration 2019-11-10 10:02:22 +01:00
Eric Culp 6c041e8413 Replace $TMPDIR with $TEST_ROOT in tests/fetchurl.sh
$TMPDIR isn't necessarily set and would cause this test to fail.
2019-11-08 12:08:10 -08:00
Eelco Dolstra d1db7fa952
Merge pull request #3211 from zimbatm/gitignore-precompiled-headers
gitignore /precompiled-headers.h.gch
2019-11-08 16:23:57 +01:00
Jonas Chevalier a08f353922
gitignore /precompiled-headers.h.?ch 2019-11-08 14:48:52 +00:00
Eelco Dolstra 0d6774468c
Move editorFor srom libutil to nix
libutil should not depend on libexpr.
2019-11-08 15:13:32 +01:00
Eelco Dolstra 48f0a76372
Fix installerScript job
https://hydra.nixos.org/build/105961653
2019-11-07 18:31:16 +01:00
Eelco Dolstra 4145cd2da0
Use upstream nlohmann_json 2019-11-07 18:23:17 +01:00
Eelco Dolstra e5bf81256c
Fix Perl bindings 2019-11-07 12:18:37 +01:00
Eelco Dolstra 6d2605500f
Fix macOS build 2019-11-07 11:53:28 +01:00
Eelco Dolstra 99af822004
Disable the evalNixOS test
It also OOMs.

https://hydra.nixos.org/build/105942679
2019-11-07 10:14:00 +01:00
Eelco Dolstra 04bf9acd22
Remove #include 2019-11-07 10:12:35 +01:00
Eelco Dolstra f5b7991e59
Revert "autoconf: Allow overriding CFLAGS/CXXFLAGS from outside."
This reverts commit 717e821b99. It's
much more convenient to do 'make OPTIMIZE=0'.
2019-11-07 10:12:35 +01:00
Eelco Dolstra 5ff4d77f55
Precompile headers
This cuts 'make install -j6' on my laptop from 170s to 134s.
2019-11-07 10:12:35 +01:00
Maximilian Bosch 52ffe2797a
doc: Document --dry-run option for nix-build 2019-11-07 00:11:57 +01:00
Eelco Dolstra 39a2e166dd
Cleanup 2019-11-06 16:53:02 +01:00
Eelco Dolstra 35732a95bc
Disable the evalNixpkgs test
It constantly OOMs.

https://hydra.nixos.org/build/105784912
2019-11-06 10:36:06 +01:00
Eelco Dolstra 7614a127a0
Fix binaryTarball test 2019-11-06 10:35:50 +01:00
Eelco Dolstra 69326f3637
Recursive Nix: Handle concurrent client connections 2019-11-06 00:55:03 +01:00
Eelco Dolstra c119ab9db0
Enable recursive Nix using a feature
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
2019-11-06 00:55:03 +01:00
Eelco Dolstra 2af9561316
Add a test for recursive Nix 2019-11-06 00:55:03 +01:00
Eelco Dolstra c921074c19
RestrictedStore: Implement addToStore() 2019-11-06 00:55:03 +01:00
Eelco Dolstra c4d7c76b64
Recursive Nix support
This allows Nix builders to call Nix to build derivations, with some
limitations.

Example:

  let nixpkgs = fetchTarball channel:nixos-18.03; in

  with import <nixpkgs> {};

  runCommand "foo"
    {
      buildInputs = [ nix jq ];
      NIX_PATH = "nixpkgs=${nixpkgs}";
    }
    ''
      hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')

      $hello/bin/hello

      mkdir -p $out/bin
      ln -s $hello/bin/hello $out/bin/hello

      nix path-info -r --json $hello | jq .
    ''

This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.

  # ll ./result/bin/
  lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello

  # nix-store -qR ./result
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
  /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

* Before running the outer builder, Nix creates a Unix domain socket
  '.nix-socket' in the builder's temporary directory and sets
  $NIX_REMOTE to point to it. It starts a thread to process
  connections to this socket. (Thus you don't need to have nix-daemon
  running.)

* The daemon thread uses a wrapper store (RestrictedStore) to keep
  track of paths added through recursive Nix calls, to implement some
  restrictions (see below), and to do some censorship (e.g. for
  purity, queryPathInfo() won't return impure information such as
  signatures and timestamps).

* After the build finishes, the output paths are scanned for
  references to the paths added through recursive Nix calls (in
  addition to the inputs closure). Thus, in the example above, $out
  has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing

  nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.

Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
2019-11-06 00:52:38 +01:00
Eelco Dolstra b874272f7a
Make --enable-gc the default 2019-11-06 00:46:37 +01:00
Eelco Dolstra d823381c0a
Merge branch 'fix/nix-doctor-output' of https://github.com/bhipple/nix 2019-11-05 16:04:40 +01:00
Eelco Dolstra b4e260d887
Disable shellcheck
It's broken at the moment: https://hydra.nixos.org/build/105746055

Also it pulls in GHC which is a pretty big dependency.
2019-11-05 16:00:30 +01:00
Eelco Dolstra 81a9b93689
Fix manual build 2019-11-05 11:21:32 +01:00
Eelco Dolstra 852554bb16
Merge branch 'nix-repl-e' of https://github.com/zimbatm/nix 2019-11-05 11:20:53 +01:00
Eelco Dolstra 7876027071
Merge pull request #3193 from matthewbauer/patch-11
Update man to show that nix-shell allows --arg
2019-11-05 11:18:24 +01:00
Eelco Dolstra 78b8203e50
Merge pull request #3180 from kevinastock/patch-1
docs: fix upper bound on number of consumed cores
2019-11-05 11:17:02 +01:00
Eelco Dolstra 376802c9b8
Merge pull request #3199 from kevinastock/patch-2
docs: xref doesn't render in title
2019-11-05 11:16:15 +01:00
Eelco Dolstra e1725ba946
Fix VM tests 2019-11-05 11:12:25 +01:00
Eelco Dolstra 6b708711f5
Merge branch 'switch-to-19.09' of https://github.com/Ericson2314/nix 2019-11-05 10:32:11 +01:00
Eelco Dolstra 1b600ecd14
Don't use SOCK_CLOEXEC on macOS
https://hydra.nixos.org/build/105428308
2019-11-05 10:25:09 +01:00
Eelco Dolstra 3770f5c944
Merge pull request #3206 from kevinastock/patch-3
docs: correct default location of log directory
2019-11-04 22:30:07 +01:00
Kevin Stock cea05e5ee7
docs: correct default location of log directory 2019-11-04 16:23:03 -05:00
Eelco Dolstra f5a46ef0b1
Merge pull request #3202 from kraem/master
Update nix eval --help msg to not include deprecated command
2019-11-04 09:34:30 +01:00
Eelco Dolstra 8ec1b1e7b8
Merge pull request #3203 from hvdijk/prefetch-progress
Fix progress bar when nix-prefetch-url is piped.
2019-11-04 09:28:17 +01:00
Harald van Dijk c935ad3f02
Fix progress bar when nix-prefetch-url is piped.
The intent of the code was that if the window size cannot be determined,
it would be treated as having the maximum possible size. Because of a
missing assignment, it was actually treated as having a width of 0.

The reason the width could not be determined was because it was obtained
from stdout, not stderr, even though the printing was done to stderr.

This commit addresses both issues.
2019-11-03 21:46:59 +00:00
kraem dcd7a26063
Update nix eval --help msg to not include deprecated command 2019-11-03 18:47:28 +01:00