Commit graph

749 commits

Author SHA1 Message Date
edef c65a6fa86a passAsFile: leave out the hash prefix
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path 
format in the tests to prevent regressions.
2020-01-02 23:56:06 +00:00
Puck Meerburg 515c0a263e passAsFile: hash the attribute name instead of numbering sequentially
This makes the paths consistent without relying on ordering.

Co-authored-by: edef <edef@edef.eu>
2020-01-02 22:56:03 +00:00
Graham Christensen c502831a1d
exportReferencesGraph: support working
Before, we would get:

    [deploy@bastion:~]$ nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    these derivations will be built:
      /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
      /nix/store/ssxwmll7v21did1c8j027q0m8w6pg41i-unit-prometheus-alertmanager-irc-notifier.service.drv
      /nix/store/mvyvkj46ay7pp7b1znqbkck2mq98k0qd-unit-script-network-local-commands-start.drv
      /nix/store/vsl1y9mz38qfk6pyirjwnfzfggz5akg6-unit-network-local-commands.service.drv
      /nix/store/wi5ighfwwb83fdmav6z6n2fw6npm9ffl-unit-prometheus-hydra-exporter.service.drv
      /nix/store/x0qkv535n75pbl3xn6nn1w7qkrg9wwyg-unit-prometheus-packet-sd.service.drv
      /nix/store/lv491znsjxdf51xnfxh9ld7r1zg14d52-unit-script-packet-sd-env-key-pre-start.drv
      /nix/store/nw4nzlca49agsajvpibx7zg5b873gk9f-unit-script-packet-sd-env-key-start.drv
      /nix/store/x674wwabdwjrkhnykair4c8mpxa9532w-unit-packet-sd-env-key.service.drv
      /nix/store/ywivz64ilb1ywlv652pkixw3vxzfvgv8-unit-wireguard-wg0.service.drv
      /nix/store/v3b648293g3zl8pnn0m1345nvmyd8dwb-unit-script-acme-selfsigned-status.nixos.org-start.drv
      /nix/store/zci5d3zvr6fgdicz6k7jjka6lmx0v3g4-unit-acme-selfsigned-status.nixos.org.service.drv
      /nix/store/f6pwvnm63d0kw5df0v7sipd1rkhqxk5g-system-units.drv
      /nix/store/iax8071knxk9c7krpm9jqg0lcrawf4lc-etc.drv
      /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    error: invalid file name 'closure-init-0' in 'exportReferencesGraph'

This was tough to debug, I didn't figure out which one was broken until I did:

    nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv 2>&1 | grep  nix/store | xargs -n1 nix-store -r

and then looking at the remaining build graph:

    $ nix-store -r /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    these derivations will be built:
      /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
      /nix/store/grfnl76cahwls0igd2by2pqv0dimi8h2-nixos-system-eris-19.09.20191213.03f3def.drv
    error: invalid file name 'closure-init-0' in 'exportReferencesGraph'

and knowing the initrd build is before the system, then:

    $ nix show-derivation /nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv
    {
      "/nix/store/3ka4ihvwh6wsyhpd2qa9f59506mnxvx1-initrd-linux-4.19.88.drv": {
        [...]
        "exportReferencesGraph": "closure-init-0 /nix/store/...-stage-1-init.sh closure-mdadm.conf-1 /nix/store/...-mdadm.conf closure-ubuntu.conf-2 ...",
        [...]
      }
    }

I then searched the repo for "in 'exportReferencesGraph'", found this
recently updated regex, and realized it was missing a "-".
2020-01-01 20:50:40 -05:00
Eelco Dolstra bbe97dff8b Make the Store API more type-safe
Most functions now take a StorePath argument rather than a Path (which
is just an alias for std::string). The StorePath constructor ensures
that the path is syntactically correct (i.e. it looks like
<store-dir>/<base32-hash>-<name>). Similarly, functions like
buildPaths() now take a StorePathWithOutputs, rather than abusing Path
by adding a '!<outputs>' suffix.

Note that the StorePath type is implemented in Rust. This involves
some hackery to allow Rust values to be used directly in C++, via a
helper type whose destructor calls the Rust type's drop()
function. The main issue is the dynamic nature of C++ move semantics:
after we have moved a Rust value, we should not call the drop function
on the original value. So when we move a value, we set the original
value to bitwise zero, and the destructor only calls drop() if the
value is not bitwise zero. This should be sufficient for most types.

Also lots of minor cleanups to the C++ API to make it more modern
(e.g. using std::optional and std::string_view in some places).
2019-12-10 22:06:05 +01:00
Eelco Dolstra 61cc9f34d2 Remove UserLock self-lock check
This is no longer needed since we're not using POSIX locks anymore.
2019-12-09 23:57:33 +01:00
Eelco Dolstra 47a937d512 Show hash mismatch warnings in SRI format
(cherry picked from commit 63c5c91cc0)
2019-12-05 20:32:25 +01:00
Eelco Dolstra ac2bc721d8 Merge remote-tracking branch 'origin/recursive-nix' 2019-12-02 12:34:46 +01:00
Eelco Dolstra 045708db43 Make <nix/unpack-channel.nix> a builtin builder
This was the last function using a shell script, so this allows us to
get rid of tar, coreutils, bash etc.
2019-11-26 22:07:28 +01:00
Eelco Dolstra 7c8d7c17f8
Merge pull request #3144 from matthewbauer/fix-sandbox-fallback
Fix sandbox fallback settings
2019-11-26 20:51:52 +01:00
Eelco Dolstra ba87b08f85
getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.
2019-11-22 16:18:13 +01:00
Eelco Dolstra 5ee23c35b9
Merge pull request #3219 from Ericson2314/semicolons
Fix extra semicolons warnings
2019-11-11 12:13:51 +01:00
John Ericson 96e6e680c1 Fix extra ; warnings involving MakeError 2019-11-10 11:24:47 -05:00
Eelco Dolstra 39a2e166dd
Cleanup 2019-11-06 16:53:02 +01:00
Eelco Dolstra 69326f3637
Recursive Nix: Handle concurrent client connections 2019-11-06 00:55:03 +01:00
Eelco Dolstra c119ab9db0
Enable recursive Nix using a feature
Derivations that want to use recursion should now set

  requiredSystemFeatures = [ "recursive-nix" ];

to make the daemon socket appear.

Also, Nix should be configured with "experimental-features =
recursive-nix".
2019-11-06 00:55:03 +01:00
Eelco Dolstra c921074c19
RestrictedStore: Implement addToStore() 2019-11-06 00:55:03 +01:00
Eelco Dolstra c4d7c76b64
Recursive Nix support
This allows Nix builders to call Nix to build derivations, with some
limitations.

Example:

  let nixpkgs = fetchTarball channel:nixos-18.03; in

  with import <nixpkgs> {};

  runCommand "foo"
    {
      buildInputs = [ nix jq ];
      NIX_PATH = "nixpkgs=${nixpkgs}";
    }
    ''
      hello=$(nix-build -E '(import <nixpkgs> {}).hello.overrideDerivation (args: { name = "hello-3.5"; })')

      $hello/bin/hello

      mkdir -p $out/bin
      ln -s $hello/bin/hello $out/bin/hello

      nix path-info -r --json $hello | jq .
    ''

This derivation makes a recursive Nix call to build GNU Hello and
symlinks it from its $out, i.e.

  # ll ./result/bin/
  lrwxrwxrwx 1 root root 63 Jan  1  1970 hello -> /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5/bin/hello

  # nix-store -qR ./result
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/s0awxrs71gickhaqdwxl506hzccb30y5-hello-3.5
  /nix/store/sgmvvyw8vhfqdqb619bxkcpfn9lvd8ss-foo

This is implemented as follows:

* Before running the outer builder, Nix creates a Unix domain socket
  '.nix-socket' in the builder's temporary directory and sets
  $NIX_REMOTE to point to it. It starts a thread to process
  connections to this socket. (Thus you don't need to have nix-daemon
  running.)

* The daemon thread uses a wrapper store (RestrictedStore) to keep
  track of paths added through recursive Nix calls, to implement some
  restrictions (see below), and to do some censorship (e.g. for
  purity, queryPathInfo() won't return impure information such as
  signatures and timestamps).

* After the build finishes, the output paths are scanned for
  references to the paths added through recursive Nix calls (in
  addition to the inputs closure). Thus, in the example above, $out
  has a reference to $hello.

The main restriction on recursive Nix calls is that they cannot do
arbitrary substitutions. For example, doing

  nix-store -r /nix/store/kmwd1hq55akdb9sc7l3finr175dajlby-hello-2.10

is forbidden unless /nix/store/kmwd... is in the inputs closure or
previously built by a recursive Nix call. This is to prevent
irreproducible derivations that have hidden dependencies on
substituters or the current store contents. Building a derivation is
fine, however, and Nix will use substitutes if available. In other
words, the builder has to present proof that it knows how to build a
desired store path from scratch by constructing a derivation graph for
that path.

Probably we should also disallow instantiating/building fixed-output
derivations (specifically, those that access the network, but
currently we have no way to mark fixed-output derivations that don't
access the network). Otherwise sandboxed derivations can bypass
sandbox restrictions and access the network.

When sandboxing is enabled, we make paths appear in the sandbox of the
builder by entering the mount namespace of the builder and
bind-mounting each path. This is tricky because we do a pivot_root()
in the builder to change the root directory of its mount namespace,
and thus the host /nix/store is not visible in the mount namespace of
the builder. To get around this, just before doing pivot_root(), we
branch a second mount namespace that shares its /nix/store mountpoint
with the parent.

Recursive Nix currently doesn't work on macOS in sandboxed mode
(because we can't change the sandbox policy of a running build) and in
non-root mode (because setns() barfs).
2019-11-06 00:52:38 +01:00
Benjamin Hipple 80d5ec6ff4 Minor updates to inline comments
Add missing docstring on InstallableCommand. Also, some of these were wrapped
when they're right next to a line longer than the unwrapped line, so we can just
unwrap them to save vertical space.
2019-10-31 05:56:37 -04:00
Eelco Dolstra 95c727caef
Remove the check against concurrent builds in the same process 2019-10-29 12:43:20 +01:00
Eelco Dolstra f7ce80f90a
Factor out linkOrCopy() 2019-10-27 18:19:13 +01:00
Eelco Dolstra 0abb3ad537 Allow content-addressable paths to have references
This adds a command 'nix make-content-addressable' that rewrites the
specified store paths into content-addressable paths. The advantage of
such paths is that 1) they can be imported without signatures; 2) they
can enable deduplication in cases where derivation changes do not
cause output changes (apart from store path hashes).

For example,

  $ nix make-content-addressable -r nixpkgs.cowsay
  rewrote '/nix/store/g1g31ah55xdia1jdqabv1imf6mcw0nb1-glibc-2.25-49' to '/nix/store/48jfj7bg78a8n4f2nhg269rgw1936vj4-glibc-2.25-49'
  ...
  rewrote '/nix/store/qbi6rzpk0bxjw8lw6azn2mc7ynnn455q-cowsay-3.03+dfsg1-16' to '/nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16'

We can then copy the resulting closure to another store without
signatures:

  $ nix copy --trusted-public-keys '' ---to ~/my-nix /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16

In order to support self-references in content-addressable paths,
these paths are hashed "modulo" self-references, meaning that
self-references are zeroed out during hashing. Somewhat annoyingly,
this means that the NAR hash stored in the Nix database is no longer
necessarily equal to the output of "nix hash-path"; for
content-addressable paths, you need to pass the --modulo flag:

  $ nix path-info --json /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16  | jq -r .[].narHash
  sha256:0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16
  1ggznh07khq0hz6id09pqws3a8q9pn03ya3c03nwck1kwq8rclzs

  $ nix hash-path --type sha256 --base32 /nix/store/iq6g2x4q62xp7y7493bibx0qn5w7xz67-cowsay-3.03+dfsg1-16 --modulo iq6g2x4q62xp7y7493bibx0qn5w7xz67
  0ri611gdilz2c9rsibqhsipbfs9vwcqvs811a52i2bnkhv7w9mgw
2019-10-21 17:47:24 +02:00
Matthew Bauer 96c84937c4 Move tmpDirInSandbox to initTmpDir 2019-10-13 16:41:49 -04:00
Matthew Bauer 499b038875 Fix sandbox fallback settings
The tmpDirInSandbox is different when in sandboxed vs. non-sandboxed.
Since we don’t know ahead of time here whether sandboxing is enabled,
we need to reset all of the env vars we’ve set previously. This fixes
the issue encountered in https://github.com/NixOS/nixpkgs/issues/70856.
2019-10-12 19:22:13 -04:00
Eelco Dolstra 99b73fb507
OCD performance fix: {find,count}+insert => insert 2019-10-09 16:06:29 +02:00
Benjamin Hipple c6a542f22a Fix spelling in comment 2019-09-21 18:53:15 -04:00
Eelco Dolstra e07ec8d27e
Support allowSubstitutes attribute in structured attribute derivations
Hopefully fixes #3081 (didn't test).
2019-09-03 16:03:49 +02:00
Eelco Dolstra e5b397b2c7 Merge branch 'test-sandboxing' of https://github.com/matthewbauer/nix 2019-08-27 20:58:47 +02:00
Matthew Bauer 5c06a8d328 Reset tmpDirInSandbox for unsandboxed 2019-08-23 20:24:39 -04:00
Eelco Dolstra 56df30cd3f
Merge pull request #2995 from tweag/post-build-hook
Add a post build hook
2019-08-07 15:02:29 +02:00
Eelco Dolstra e349f2c0a3
Use BSD instead of POSIX file locks
POSIX file locks are essentially incompatible with multithreading. BSD
locks have much saner semantics. We need this now that there can be
multiple concurrent LocalStore::buildPaths() invocations.
2019-08-02 18:39:16 +02:00
regnat 7c5596734f
Add a post-build-hook
Passing `--post-build-hook /foo/bar` to a nix-* command will cause
`/foo/bar` to be executed after each build with the following
environment variables set:

    DRV_PATH=/nix/store/drv-that-has-been-built.drv
    OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev

This can be useful in particular to upload all the builded artifacts to
the cache (including the ones that don't appear in the runtime closure
of the final derivation or are built because of IFD).

This new feature prints the stderr/stdout output to the `nix-build`
and `nix build` client, and the output is printed in a Nix 2
compatible format:

    [nix]$ ./inst/bin/nix-build ./test.nix
    these derivations will be built:
      /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv
    building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'...
    hello!
    bye!
    running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation
    post-build-hook: + sleep 1
    post-build-hook: + printf 'very important stuff'
    /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation

    [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix
    my-example-derivation> hello!
    my-example-derivation> bye!
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation
    my-example-derivation (post)> + sleep 1
    my-example-derivation (post)> + printf 'very important stuff'
    [1 built, 0.0 MiB DL]

Co-authored-by: Graham Christensen <graham@grahamc.com>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2019-08-02 10:48:15 -04:00
Matthew Bauer 9a0855bbb6 Don’t rely on EPERM
startProcess does not appear to send the exit code to the helper
correctly. Not sure why this is, but it is probably safe to just
fallback on all sandbox errors.
2019-07-30 17:53:37 -04:00
Matthew Bauer 11d8534629 Use sandbox fallback when cloning fails in builder
When sandbox-fallback = true (the default), the Nix builder will fall
back to disabled sandbox mode when the kernel doesn’t allow users to
set it up. This prevents hard errors from occuring in tricky places,
especially the initial installer. To restore the previous behavior,
users can set:

  sandbox-fallback = false

in their /etc/nix/nix.conf configuration.
2019-07-25 14:42:30 -04:00
Matthew Bauer d171090530 Disable CLONE_NEWUSER when it’s unavailable
Some kernels disable "unpriveleged user namespaces". This is
unfortunate, but we can still use mount namespaces. Anyway, since each
builder has its own nixbld user, we already have most of the benefits
of user namespaces.
2019-07-25 14:42:25 -04:00
Eelco Dolstra 2f853b20df
Merge pull request #2975 from matthewbauer/fix-nsswitch-issue
Don’t use entire /etc/nsswitch.conf file
2019-07-13 17:08:02 +02:00
Daiderd Jordan a52c331edb
build: replace 100 offset for build exit codes 2019-07-02 00:12:38 +02:00
Daiderd Jordan cbf84bcce7
build: use binary mask for build status flags
If multiple builds with fail with different errors it will be reflected
in the status code.

eg.

	103 => timeout + hash mismatch
	105 => timeout + check mismatch
	106 => hash mismatch + check mismatch
	107 => timeout + hash mismatch + check mismatch
2019-07-02 00:12:38 +02:00
Daiderd Jordan 97baf32fbc
build: add exit code for hash and check mismatches
Makes it easier to identify the failure reason in other tooling, eg.
differentiate between a non-deterministic --check vs a failed build.

	$ nix-build '<nix/fetchurl.nix>' --argstr url http://example.org --argstr sha256 0000000000000000000000000000000000000000000000000000
	hash mismatch in fixed-output derivation '/nix/store/nzi9ck45rwlxzcwr25is7qlf3hs5xl83-example.org':
	  wanted: sha256:0000000000000000000000000000000000000000000000000000
	  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm
	$ echo $?
	102

	$ nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "date +%s > $out"' --check
	warning: rewriting hashes in '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo'; cross fingers
	error: derivation '/nix/store/mggc8dz13ackb49qca6m23zq4fpq132q-foo.drv' may not be deterministic: output '/nix/store/g3k47g0399fvjmbm0p0mnad74k4w8vkz-foo' differs
	$ echo $?
	104
2019-07-02 00:12:34 +02:00
Matthew Bauer ec0087df0a Don’t use entire /etc/nsswitch.conf file
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).

/cc @Ericson2314
2019-06-27 14:31:29 -04:00
Eelco Dolstra 3cc1125595
Another attempt at getting pseudoterminals to work on macOS 2019-06-17 08:08:04 +02:00
Eelco Dolstra 2743bf0bb1 Hopefully fix macOS tests 2019-06-16 20:02:40 +02:00
Eelco Dolstra 82ca6ef390 Set $TERM 2019-06-16 17:36:30 +02:00
Eelco Dolstra e84c265645 Run builds in a pseudo-terminal
This allows many programs (e.g. gcc, clang, cmake) to print colorized
log output (assuming $TERM is set to a value like "xterm").

There are other ways to get colors, in particular setting
CLICOLOR_FORCE, but they're less widely supported and can break
programs that parse tool output.
2019-06-16 17:36:30 +02:00
Eelco Dolstra 5f6840fbb4
Merge pull request #2802 from LnL7/fix-needs-hashrewrite
build: move needsHashRewrite initialization to startBuilder
2019-05-13 08:53:48 +02:00
Daiderd Jordan ce02fc74b2
build: make needsHashRewrite a method 2019-05-12 22:56:27 +02:00
Graham Christensen 73b797c207
handleDiffHook: stop passing allowVfork 2019-05-12 13:44:22 -04:00
Graham Christensen b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen dde8eeb39a
chdir, setgroups 2019-05-12 13:17:26 -04:00
Graham Christensen 6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00
Graham Christensen c78686e411
build: run diff-hook under --check and document diff-hook 2019-05-12 13:17:24 -04:00
Daiderd Jordan ff6867ab94
build: move needsHashRewrite initialization to startBuilder
The value of useChroot is not set yet in the constructor, resulting in
hash rewriting being enabled in certain cases where it should not be.

Fixes #2801
2019-05-12 15:53:40 +02:00
Eelco Dolstra f9a2ea4486
Fix "Bad system call" running i686-linux binaries on x86_64-linux
To determine which seccomp filters to install, we were incorrectly
using settings.thisSystem, which doesn't denote the actual system when
--system is used.

Fixes #2791.
2019-05-03 10:48:33 +02:00
Daiderd Jordan b614e0e53d
build: only skip hash rewriting for sandboxing on linux
The sandbox on darwin, and possibly other future platforms, doesn't have
enough isolation to redirect outputs without hash rewriting.
2019-04-17 23:41:23 +02:00
Eelco Dolstra 41ba5135e0 primeCache(): Barf if builds are needed but not allowed
Fixes #2753.
2019-04-01 21:09:49 +02:00
Eelco Dolstra ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra 01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
Jonas Chevalier 5e6fa9092f
libstore: improve hash mismatch error messages
Use the same output ordering and format everywhere.

This is such a common issue that we trade the single-line error message for
more readability.

Old message:
```
fixed-output derivation produced path '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com' with sha256 hash '08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm' instead of the expected hash '1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m'
```

New message:
```
hash mismatch in fixed-output derivation '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com':
  wanted: sha256:1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m
  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm

```
2018-12-12 17:42:00 +01:00
Eelco Dolstra a32ff2573b
Fix 'Read-only file system' when building a derivation 2018-11-15 13:20:39 +01:00
Eelco Dolstra 18b4c53f71
Restore old (dis)allowedRequisites behaviour for self-references
stdenv relies on this. So ignore self-references (but only in legacy non-structured attributes mode).
2018-10-27 15:41:53 +02:00
Eelco Dolstra 3cd15c5b1f
Per-output reference and closure size checks
In structured-attributes derivations, you can now specify per-output
checks such as:

  outputChecks."out" = {
    # The closure of 'out' must not be larger than 256 MiB.
    maxClosureSize = 256 * 1024 * 1024;

    # It must not refer to C compiler or to the 'dev' output.
    disallowedRequisites = [ stdenv.cc "dev" ];
  };

  outputChecks."dev" = {
    # The 'dev' output must not be larger than 128 KiB.
    maxSize = 128 * 1024;
  };

Also fixed a bug in allowedRequisites that caused it to ignore
self-references.
2018-10-23 01:29:16 +02:00
Eelco Dolstra 1e7b8deea7
Check requiredSystemFeatures for local builds
For example, this prevents a "kvm" build on machines that don't have
KVM.

Fixes #2012.
2018-09-28 16:10:27 +02:00
Eelco Dolstra 7ae7a38c9a
Move structured attrs handling into a separate class
This is primarily because Derivation::{can,will}BuildLocally() depends
on attributes like preferLocalBuild and requiredSystemFeatures, but it
can't handle them properly because it doesn't have access to the
structured attributes.
2018-09-28 14:32:46 +02:00
Eelco Dolstra 99d4bb2d4c
Remove obsolete check on system features 2018-09-28 13:28:15 +02:00
Eelco Dolstra c9ba33870e
Support special attributes in structured attributes derivations
E.g. __noChroot and allowedReferences now work correctly. We also now
check that the attribute type is correct. For instance, instead of

  allowedReferences = "out";

you have to write

  allowedReferences = [ "out" ];

Fixes #2453.
2018-09-28 12:43:01 +02:00
Eelco Dolstra bba3f0a308
If fallback is enabled, temporarily disable substituters after a failure
Otherwise, we just keep asking the substituter for other .narinfo
files, which can take a very long time due to retries/timeouts.
2018-09-07 17:08:43 +02:00
Eelco Dolstra 33c3f91885
Handle queryPathInfo() failure from substituters when fallback is enabled
Fixes #1990.
2018-09-07 16:35:48 +02:00
Eelco Dolstra c0c31b58a4
Add localhost alias for ::1 to the sandbox 2018-08-31 11:12:51 +02:00
Yorick 72a78beb34
Fix #2162: use getaddrinfo instead of curl to preload NSS 2018-06-12 13:05:14 +02:00
Eelco Dolstra 4ac4f675df
Don't require --fallback to recover from disappeared binary cache NARs 2018-06-05 16:04:41 +02:00
Eelco Dolstra 737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Eelco Dolstra c717d8e3bd
Don't set a CA assertion for paths with references
Really fixes #2133.
2018-05-03 14:49:56 +02:00
Eelco Dolstra 53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Shea Levy 8e6108ff71
Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix
Support extra compatible architectures (#1916)
2018-04-23 08:48:22 -04:00
Will Dietz 6d9129014d build.cc: fix bind-mount of /dev/{pts,ptmx} fallback
Don't bind-mount these to themselves,
mount them into the chroot directory.

Fixes pty issues when using sandbox on CentOS 7.4.
(build of perlPackages.IOTty fails before this change)
2018-04-20 12:24:23 -05:00
Eelco Dolstra c74f838620
exportReferences: Check whether the path is in the input closure
Fixes #2082.
2018-04-17 12:03:27 +02:00
Eelco Dolstra d34fa2bcc3
Fix #1921 2018-04-13 15:42:35 +02:00
Eelco Dolstra f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Eelco Dolstra 668ac3ea2c
Make <nix/buildenv.nix> a builtin builder
This avoids sandbox annoyances.
2018-03-20 17:28:09 +01:00
Eelco Dolstra 1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Eelco Dolstra ca14b14200
Use boost::format from the boost package
Note that this only requires headers from boost so it doesn't add a
runtime dependency.

Also, use Nixpkgs 18.03.
2018-03-14 19:24:04 +01:00
Eelco Dolstra 6e60141a80
Typo 2018-03-13 17:48:09 +01:00
Will Dietz 56253bb08f rand() -> random(), since we use srandom().
rand() requires we call srand() instead,
but might as well use random().
2018-03-06 17:44:25 -06:00
Will Dietz 6b9a03f5d8 hoist vector out of loop just in case 2018-03-02 10:52:05 -06:00
Will Dietz c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Linus Heckemann 919c3c20b3 seccomp: add 32-bit ARM on aarch64-linux
This allows building armv[67]l-linux derivations on compatible aarch64
machines. Failure to add the architecture may result from missing
hardware support, in which case we can't run 32-bit binaries and don't
need to restrict them with seccomp anyway,
2018-02-27 00:32:15 +00:00
Eelco Dolstra d4e93532e2
Fix incorrect (and unnecessary) format string
https://hydra.nixos.org/eval/1434547#tabs-now-fail
2018-02-19 20:46:39 +01:00
Shea Levy e59a8a63e1
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
Running Nix with build users without seccomp on Linux is dangerous,
and administrators should very explicitly opt-in to it.
2018-02-19 09:56:24 -05:00
Shea Levy 690ac7c90b
configure: Add a flag to disable seccomp.
This is needed for new arches where libseccomp support doesn't exist
yet.

Fixes #1878.
2018-02-18 02:35:01 -05:00
Eelco Dolstra 4f09ce7940
Fix 'deadlock: trying to re-acquire self-held lock'
This was caused by derivations with 'allowSubstitutes = false'. Such
derivations will be built locally. However, if there is another
SubstitionGoal that has the output of the first derivation in its
closure, then the path will be simultaneously built and substituted.

There was a check to catch this situation (via pathIsLockedByMe()),
but it no longer worked reliably because substitutions are now done in
another thread. (Thus the comment 'It can't happen between here and
the lockPaths() call below because we're not allowing multi-threading'
was no longer valid.)

The fix is to handle the path already being locked in both
SubstitutionGoal and DerivationGoal.
2018-02-12 17:06:06 +01:00
Eelco Dolstra 960e9c560e
nix: Ensure that the user sees errors from substituters 2018-02-09 15:06:47 +01:00
Eelco Dolstra 84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra bb1d046f5c
Allow substituters to be marked as trusted
This is needed by nixos-install, which uses the Nix store on the
installation CD as a substituter. We don't want to disable signature
checking entirely because substitutes from cache.nixos.org should
still be checked. So now we can pas "local?trusted=1" to mark only the
Nix store in /nix as not requiring signatures.

Fixes #1819.
2018-02-05 18:08:30 +01:00
Eelco Dolstra 2175eee9fe
Fix segfault using non-binary cache stores as substituters 2018-02-05 17:46:43 +01:00
Eelco Dolstra 84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Dan Peebles d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Eelco Dolstra 3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Will Dietz 00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Eelco Dolstra b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Eelco Dolstra 197922ea4e
Merge pull request #1646 from copumpkin/optional-sandbox-local-network
Allow optional localhost network access to sandboxed derivations
2017-10-30 18:54:40 +01:00