Commit graph

711 commits

Author SHA1 Message Date
Matthew Bauer ec0087df0a Don’t use entire /etc/nsswitch.conf file
The default nsswitch.conf(5) file in most distros can handle many
different things including host name, user names, groups, etc. In Nix,
we want to limit the amount of impurities that come from these things.
As a result, we should only allow nss to be used for gethostbyname(3)
and getservent(3).

/cc @Ericson2314
2019-06-27 14:31:29 -04:00
Eelco Dolstra 3cc1125595
Another attempt at getting pseudoterminals to work on macOS 2019-06-17 08:08:04 +02:00
Eelco Dolstra 2743bf0bb1 Hopefully fix macOS tests 2019-06-16 20:02:40 +02:00
Eelco Dolstra 82ca6ef390 Set $TERM 2019-06-16 17:36:30 +02:00
Eelco Dolstra e84c265645 Run builds in a pseudo-terminal
This allows many programs (e.g. gcc, clang, cmake) to print colorized
log output (assuming $TERM is set to a value like "xterm").

There are other ways to get colors, in particular setting
CLICOLOR_FORCE, but they're less widely supported and can break
programs that parse tool output.
2019-06-16 17:36:30 +02:00
Eelco Dolstra 5f6840fbb4
Merge pull request #2802 from LnL7/fix-needs-hashrewrite
build: move needsHashRewrite initialization to startBuilder
2019-05-13 08:53:48 +02:00
Daiderd Jordan ce02fc74b2
build: make needsHashRewrite a method 2019-05-12 22:56:27 +02:00
Graham Christensen 73b797c207
handleDiffHook: stop passing allowVfork 2019-05-12 13:44:22 -04:00
Graham Christensen b4a05edbfe
runProgram: support gid, uid, chdir 2019-05-12 13:17:27 -04:00
Graham Christensen dde8eeb39a
chdir, setgroups 2019-05-12 13:17:26 -04:00
Graham Christensen 6df61db060
diff hook: execute as the build user, and pass the temp dir 2019-05-12 13:17:26 -04:00
Graham Christensen c78686e411
build: run diff-hook under --check and document diff-hook 2019-05-12 13:17:24 -04:00
Daiderd Jordan ff6867ab94
build: move needsHashRewrite initialization to startBuilder
The value of useChroot is not set yet in the constructor, resulting in
hash rewriting being enabled in certain cases where it should not be.

Fixes #2801
2019-05-12 15:53:40 +02:00
Eelco Dolstra f9a2ea4486
Fix "Bad system call" running i686-linux binaries on x86_64-linux
To determine which seccomp filters to install, we were incorrectly
using settings.thisSystem, which doesn't denote the actual system when
--system is used.

Fixes #2791.
2019-05-03 10:48:33 +02:00
Daiderd Jordan b614e0e53d
build: only skip hash rewriting for sandboxing on linux
The sandbox on darwin, and possibly other future platforms, doesn't have
enough isolation to redirect outputs without hash rewriting.
2019-04-17 23:41:23 +02:00
Eelco Dolstra 41ba5135e0 primeCache(): Barf if builds are needed but not allowed
Fixes #2753.
2019-04-01 21:09:49 +02:00
Eelco Dolstra ef52ccf035
experimental/optional -> optional 2019-03-14 14:10:52 +01:00
Eelco Dolstra 01d07b1e92
Revert "Restore parent mount namespace before executing a child process"
This reverts commit a0ef21262f. This
doesn't work in 'nix run' and nix-shell because setns() fails in
multithreaded programs, and Boehm GC mark threads are uncancellable.

Fixes #2646.
2019-02-05 10:49:19 +01:00
Jonas Chevalier 5e6fa9092f
libstore: improve hash mismatch error messages
Use the same output ordering and format everywhere.

This is such a common issue that we trade the single-line error message for
more readability.

Old message:
```
fixed-output derivation produced path '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com' with sha256 hash '08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm' instead of the expected hash '1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m'
```

New message:
```
hash mismatch in fixed-output derivation '/nix/store/d4nw9x2sy9q3r32f3g5l5h1k833c01vq-example.com':
  wanted: sha256:1xzwnipjd54wl8g93vpw6hxnpmdabq0wqywriiwmh7x8k0lvpq5m
  got:    sha256:08y4734bm2zahw75b16bcmcg587vvyvh0n11gwiyir70divwp1rm

```
2018-12-12 17:42:00 +01:00
Eelco Dolstra a32ff2573b
Fix 'Read-only file system' when building a derivation 2018-11-15 13:20:39 +01:00
Eelco Dolstra 18b4c53f71
Restore old (dis)allowedRequisites behaviour for self-references
stdenv relies on this. So ignore self-references (but only in legacy non-structured attributes mode).
2018-10-27 15:41:53 +02:00
Eelco Dolstra 3cd15c5b1f
Per-output reference and closure size checks
In structured-attributes derivations, you can now specify per-output
checks such as:

  outputChecks."out" = {
    # The closure of 'out' must not be larger than 256 MiB.
    maxClosureSize = 256 * 1024 * 1024;

    # It must not refer to C compiler or to the 'dev' output.
    disallowedRequisites = [ stdenv.cc "dev" ];
  };

  outputChecks."dev" = {
    # The 'dev' output must not be larger than 128 KiB.
    maxSize = 128 * 1024;
  };

Also fixed a bug in allowedRequisites that caused it to ignore
self-references.
2018-10-23 01:29:16 +02:00
Eelco Dolstra 1e7b8deea7
Check requiredSystemFeatures for local builds
For example, this prevents a "kvm" build on machines that don't have
KVM.

Fixes #2012.
2018-09-28 16:10:27 +02:00
Eelco Dolstra 7ae7a38c9a
Move structured attrs handling into a separate class
This is primarily because Derivation::{can,will}BuildLocally() depends
on attributes like preferLocalBuild and requiredSystemFeatures, but it
can't handle them properly because it doesn't have access to the
structured attributes.
2018-09-28 14:32:46 +02:00
Eelco Dolstra 99d4bb2d4c
Remove obsolete check on system features 2018-09-28 13:28:15 +02:00
Eelco Dolstra c9ba33870e
Support special attributes in structured attributes derivations
E.g. __noChroot and allowedReferences now work correctly. We also now
check that the attribute type is correct. For instance, instead of

  allowedReferences = "out";

you have to write

  allowedReferences = [ "out" ];

Fixes #2453.
2018-09-28 12:43:01 +02:00
Eelco Dolstra bba3f0a308
If fallback is enabled, temporarily disable substituters after a failure
Otherwise, we just keep asking the substituter for other .narinfo
files, which can take a very long time due to retries/timeouts.
2018-09-07 17:08:43 +02:00
Eelco Dolstra 33c3f91885
Handle queryPathInfo() failure from substituters when fallback is enabled
Fixes #1990.
2018-09-07 16:35:48 +02:00
Eelco Dolstra c0c31b58a4
Add localhost alias for ::1 to the sandbox 2018-08-31 11:12:51 +02:00
Yorick 72a78beb34
Fix #2162: use getaddrinfo instead of curl to preload NSS 2018-06-12 13:05:14 +02:00
Eelco Dolstra 4ac4f675df
Don't require --fallback to recover from disappeared binary cache NARs 2018-06-05 16:04:41 +02:00
Eelco Dolstra 737ed88f35
Modularize config settings
Allow global config settings to be defined in multiple Config
classes. For example, this means that libutil can have settings and
evaluator settings can be moved out of libstore. The Config classes
are registered in a new GlobalConfig class to which config files
etc. are applied.

Relevant to https://github.com/NixOS/nix/issues/2009 in that it
removes the need for ad hoc handling of useCaseHack, which was the
underlying cause of that issue.
2018-05-30 13:28:01 +02:00
Eelco Dolstra c717d8e3bd
Don't set a CA assertion for paths with references
Really fixes #2133.
2018-05-03 14:49:56 +02:00
Eelco Dolstra 53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Shea Levy 8e6108ff71
Merge branch 'aarch64-armv7' of git://github.com/lheckemann/nix
Support extra compatible architectures (#1916)
2018-04-23 08:48:22 -04:00
Will Dietz 6d9129014d build.cc: fix bind-mount of /dev/{pts,ptmx} fallback
Don't bind-mount these to themselves,
mount them into the chroot directory.

Fixes pty issues when using sandbox on CentOS 7.4.
(build of perlPackages.IOTty fails before this change)
2018-04-20 12:24:23 -05:00
Eelco Dolstra c74f838620
exportReferences: Check whether the path is in the input closure
Fixes #2082.
2018-04-17 12:03:27 +02:00
Eelco Dolstra d34fa2bcc3
Fix #1921 2018-04-13 15:42:35 +02:00
Eelco Dolstra f87e286e82
Merge branch 'fix/avoid-large-stack-buffers' of https://github.com/dtzWill/nix 2018-03-22 13:19:25 +01:00
Eelco Dolstra 668ac3ea2c
Make <nix/buildenv.nix> a builtin builder
This avoids sandbox annoyances.
2018-03-20 17:28:09 +01:00
Eelco Dolstra 1aca195e52
Hack to get SSH error messages from build-remote
E.g.

  cannot build on 'ssh://mac1': cannot connect to 'mac1': bash: nix-store: command not found
  cannot build on 'ssh://mac2': cannot connect to 'mac2': Host key verification failed.
  cannot build on 'ssh://mac3': cannot connect to 'mac3': Received disconnect from 213... port 6001:2: Too many authentication failures
  Authentication failed.
2018-03-20 15:25:26 +01:00
Eelco Dolstra ca14b14200
Use boost::format from the boost package
Note that this only requires headers from boost so it doesn't add a
runtime dependency.

Also, use Nixpkgs 18.03.
2018-03-14 19:24:04 +01:00
Eelco Dolstra 6e60141a80
Typo 2018-03-13 17:48:09 +01:00
Will Dietz 56253bb08f rand() -> random(), since we use srandom().
rand() requires we call srand() instead,
but might as well use random().
2018-03-06 17:44:25 -06:00
Will Dietz 6b9a03f5d8 hoist vector out of loop just in case 2018-03-02 10:52:05 -06:00
Will Dietz c89a3d5368 don't allocate large buffers on the stack 2018-03-02 10:52:04 -06:00
Linus Heckemann 919c3c20b3 seccomp: add 32-bit ARM on aarch64-linux
This allows building armv[67]l-linux derivations on compatible aarch64
machines. Failure to add the architecture may result from missing
hardware support, in which case we can't run 32-bit binaries and don't
need to restrict them with seccomp anyway,
2018-02-27 00:32:15 +00:00
Eelco Dolstra d4e93532e2
Fix incorrect (and unnecessary) format string
https://hydra.nixos.org/eval/1434547#tabs-now-fail
2018-02-19 20:46:39 +01:00
Shea Levy e59a8a63e1
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
Running Nix with build users without seccomp on Linux is dangerous,
and administrators should very explicitly opt-in to it.
2018-02-19 09:56:24 -05:00
Shea Levy 690ac7c90b
configure: Add a flag to disable seccomp.
This is needed for new arches where libseccomp support doesn't exist
yet.

Fixes #1878.
2018-02-18 02:35:01 -05:00
Eelco Dolstra 4f09ce7940
Fix 'deadlock: trying to re-acquire self-held lock'
This was caused by derivations with 'allowSubstitutes = false'. Such
derivations will be built locally. However, if there is another
SubstitionGoal that has the output of the first derivation in its
closure, then the path will be simultaneously built and substituted.

There was a check to catch this situation (via pathIsLockedByMe()),
but it no longer worked reliably because substitutions are now done in
another thread. (Thus the comment 'It can't happen between here and
the lockPaths() call below because we're not allowing multi-threading'
was no longer valid.)

The fix is to handle the path already being locked in both
SubstitutionGoal and DerivationGoal.
2018-02-12 17:06:06 +01:00
Eelco Dolstra 960e9c560e
nix: Ensure that the user sees errors from substituters 2018-02-09 15:06:47 +01:00
Eelco Dolstra 84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra bb1d046f5c
Allow substituters to be marked as trusted
This is needed by nixos-install, which uses the Nix store on the
installation CD as a substituter. We don't want to disable signature
checking entirely because substitutes from cache.nixos.org should
still be checked. So now we can pas "local?trusted=1" to mark only the
Nix store in /nix as not requiring signatures.

Fixes #1819.
2018-02-05 18:08:30 +01:00
Eelco Dolstra 2175eee9fe
Fix segfault using non-binary cache stores as substituters 2018-02-05 17:46:43 +01:00
Eelco Dolstra 84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Dan Peebles d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Eelco Dolstra 3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Will Dietz 00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Eelco Dolstra b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Eelco Dolstra 197922ea4e
Merge pull request #1646 from copumpkin/optional-sandbox-local-network
Allow optional localhost network access to sandboxed derivations
2017-10-30 18:54:40 +01:00
Dan Peebles 4a4a009f78 Allow optional localhost network access to sandboxed derivations
This will allow bind and connect to 127.0.0.1, which can reduce purity/
security (if you're running a vulnerable service on localhost) but is
also needed for a ton of test suites, so I'm leaving it turned off by
default but allowing certain derivations to turn it on as needed.

It also allows DNS resolution of arbitrary hostnames but I haven't found
a way to avoid that. In principle I'd just want to allow resolving
localhost but that doesn't seem to be possible.

I don't think this belongs under `build-use-sandbox = relaxed` because we
want it on Hydra and I don't think it's the end of the world.
2017-10-30 17:59:12 +01:00
Eelco Dolstra 9971d875a4
Fix building on clang
https://hydra.nixos.org/build/62945761
2017-10-25 17:20:47 +02:00
Eelco Dolstra 82327e3cc4
exportReferencesGraph: Allow exporting a list of store paths 2017-10-25 15:18:49 +02:00
Eelco Dolstra 3395e3bbc4
Fix exportReferencesGraph in the structured attrs case 2017-10-25 14:08:29 +02:00
Eelco Dolstra 2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra 3460e4cf00
More progress indicator improvements
In particular, don't show superfluous "fetching path" and "building
path(s)" messages, and show the current round (with --repeat).
2017-10-24 15:32:38 +02:00
Eelco Dolstra 96051dd057
More progress indicator improvements
Fixes #1599.
2017-10-24 14:47:23 +02:00
Eelco Dolstra be220702a7
Progress indicator: Show on what machine we're building
E.g.

  $ nix build nixpkgs.hello --builders 'root@wendy'
  [1/0/1 built] building hello-2.10 on ssh://root@wendy: checking for minix/config.h... no
2017-10-24 14:24:57 +02:00
Eelco Dolstra fe9d2f974d
Handle log messages from build-remote
This makes the progress indicator show statuses like "connecting to
'root@machine'".
2017-10-24 14:05:22 +02:00
Eelco Dolstra 25f32625e2
Remove the remote-builds option
This is superfluous since you can now just set "builders" to empty,
e.g. "--builders ''".
2017-10-24 11:00:16 +02:00
Eelco Dolstra 37fbfffd8e
Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Dan Peebles 6e5165b773 Shift Darwin sandbox to separate installed files
This makes it slightly more manageable to see at a glance what in a
build's sandbox profile is unique to the build and what is standard. Also
a first step to factoring more of our Darwin logic into scheme functions
that will allow us a bit more flexibility. And of course less of that
nasty codegen in C++! 😀
2017-10-17 19:15:51 -04:00
Eelco Dolstra 1dd29d7aeb
Add option to disable the seccomp filter
I needed this to test ACL/xattr removal in
canonicalisePathMetaData(). Might also be useful if you need to build
old Nixpkgs that doesn't have the required patches to remove
setuid/setgid creation.
2017-10-12 18:21:55 +02:00
Eelco Dolstra 88e6bb76de Communicate the structured log FD to builders
Since we may use a dedicated file descriptor in the future, this
allows us to change it. So builders can do

  if [[ -n $NIX_LOG_FD ]]; then
    echo "@nix { message... }" >&$NIX_LOG_FD
  fi
2017-09-08 11:27:10 +02:00
Eelco Dolstra 0b606aad46
Add automatic garbage collection
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.

Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.

Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
2017-09-05 20:43:42 +02:00
Eelco Dolstra bbdf08bc0f
Call queryMissing() prior to building
Without this, substitute info is fetched sequentially, which is
superslow. In the old UI (e.g. nix-build), we call printMissing(),
which calls queryMissing(), thereby preheating the binary cache
cache. But the new UI doesn't do that.
2017-08-31 16:05:07 +02:00
Eelco Dolstra c2154d4c84
Rename a few configuration options
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say

  nix build --no-sandbox

instead of

  nix build --no-build-use-sandbox
2017-08-31 14:28:25 +02:00
Eelco Dolstra 2cc345b95f
Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra 94a0548dc4
Simplify 2017-08-25 21:26:37 +02:00
Eelco Dolstra 0ac35b67b8
Allow derivations to update the build phase
So the progress bar can show

  [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25 18:04:05 +02:00
Eelco Dolstra c137c0a5eb
Allow activities to be nested
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Eelco Dolstra 0e9ddcc306
Restore activity metadata
This allows the progress bar to display "building perl-5.22.3" instead
of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-25 15:58:35 +02:00
Eelco Dolstra 1f56235438
Clean up JSON construction 2017-08-21 12:18:46 +02:00
Eelco Dolstra 4af2611bd1
Allow builders to create activities
Actually, currently they can only create download activities. Thus,
downloads by builtins.fetchurl show up in the progress bar.
2017-08-21 12:18:46 +02:00
Eelco Dolstra b4ed97e3a3
nix optimise-store: Show how much space has been freed 2017-08-16 20:56:03 +02:00
Eelco Dolstra 40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra dff12b38f9
Progress indicator: More improvements 2017-08-16 20:56:03 +02:00
Eelco Dolstra bf1f123b09
Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra 0e0dcf2c7e
Progress indicator: Unify "copying" and "substituting"
They're the same thing after all.

Example:

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16 20:56:03 +02:00
Eelco Dolstra c36467ad2e
Improve substitution progress indicator
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16 20:56:02 +02:00
Eelco Dolstra af765a8eab
Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra fc3568e263
FD_SETSIZE check: BuildError -> Error
BuildError denotes a permanent build failure, which is not the case
here.
2017-07-20 13:33:13 +02:00
Dmitry Kalinkin d5e1bffd2a
Do not try to fill fd_set with fd>=FD_SETSIZE
This is UB and causes buffer overflow and crash on linux.
2017-07-18 17:51:50 -04:00
Eelco Dolstra a3dc1e65ab
Add X32 to the seccomp filter
Fixes #1432.
2017-07-04 19:00:51 +02:00
Eelco Dolstra 6cf23c3e8f
Add allow-new-privileges option
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.

Example:

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
    cannot raise the capability into the Ambient set
    : Operation not permitted

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms

Fixes #1429.
2017-07-04 15:48:25 +02:00
Eelco Dolstra c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00