Commit graph

763 commits

Author SHA1 Message Date
Shea Levy e59a8a63e1
Don't silently succeed seccomp setup when !HAVE_SECCOMP.
Running Nix with build users without seccomp on Linux is dangerous,
and administrators should very explicitly opt-in to it.
2018-02-19 09:56:24 -05:00
Shea Levy 690ac7c90b
configure: Add a flag to disable seccomp.
This is needed for new arches where libseccomp support doesn't exist
yet.

Fixes #1878.
2018-02-18 02:35:01 -05:00
Eelco Dolstra 4f09ce7940
Fix 'deadlock: trying to re-acquire self-held lock'
This was caused by derivations with 'allowSubstitutes = false'. Such
derivations will be built locally. However, if there is another
SubstitionGoal that has the output of the first derivation in its
closure, then the path will be simultaneously built and substituted.

There was a check to catch this situation (via pathIsLockedByMe()),
but it no longer worked reliably because substitutions are now done in
another thread. (Thus the comment 'It can't happen between here and
the lockPaths() call below because we're not allowing multi-threading'
was no longer valid.)

The fix is to handle the path already being locked in both
SubstitutionGoal and DerivationGoal.
2018-02-12 17:06:06 +01:00
Eelco Dolstra 960e9c560e
nix: Ensure that the user sees errors from substituters 2018-02-09 15:06:47 +01:00
Eelco Dolstra 84989d3af2
Improve filtering of ANSI escape sequences in build logs
All ANSI sequences except color setting are now filtered out. In
particular, terminal resets (such as from NixOS VM tests) are filtered
out.

Also, fix the completely broken tab character handling.
2018-02-07 15:23:57 +01:00
Eelco Dolstra bb1d046f5c
Allow substituters to be marked as trusted
This is needed by nixos-install, which uses the Nix store on the
installation CD as a substituter. We don't want to disable signature
checking entirely because substitutes from cache.nixos.org should
still be checked. So now we can pas "local?trusted=1" to mark only the
Nix store in /nix as not requiring signatures.

Fixes #1819.
2018-02-05 18:08:30 +01:00
Eelco Dolstra 2175eee9fe
Fix segfault using non-binary cache stores as substituters 2018-02-05 17:46:43 +01:00
Eelco Dolstra 84722d67d2
Remove nix-build --hash
Instead, if a fixed-output derivation produces has an incorrect output
hash, we now unconditionally move the outputs to the path
corresponding with the actual hash and register it as valid. Thus,
after correcting the hash in the Nix expression (e.g. in a fetchurl
call), the fixed-output derivation doesn't have to be built again.

It would still be good to have a command for reporting the actual hash
of a fixed-output derivation (instead of throwing an error), but
"nix-build --hash" didn't do that.
2018-02-03 10:08:05 +01:00
Dan Peebles d43a8b25f0 Fix obscure corner case in name resolution for builtin:fetchurl in sandboxed environments 2018-01-23 14:45:50 -05:00
Eelco Dolstra 3c4c30eadd
Rewrite builtin derivation environment
Also add a test.

Fixes #1803.
Closes #1805.
2018-01-19 15:00:38 +01:00
Eelco Dolstra ba75c69e00
Barf when using a diverted store on macOS
Fixes #1792.
2018-01-15 12:14:43 +01:00
Will Dietz 00e0c416ff Mark comparison call operator as const 2017-12-11 12:08:40 -06:00
Eelco Dolstra b52846ab5b
Show log tail when a remote build fails 2017-11-28 15:36:28 +01:00
Eelco Dolstra 197922ea4e
Merge pull request #1646 from copumpkin/optional-sandbox-local-network
Allow optional localhost network access to sandboxed derivations
2017-10-30 18:54:40 +01:00
Dan Peebles 4a4a009f78 Allow optional localhost network access to sandboxed derivations
This will allow bind and connect to 127.0.0.1, which can reduce purity/
security (if you're running a vulnerable service on localhost) but is
also needed for a ton of test suites, so I'm leaving it turned off by
default but allowing certain derivations to turn it on as needed.

It also allows DNS resolution of arbitrary hostnames but I haven't found
a way to avoid that. In principle I'd just want to allow resolving
localhost but that doesn't seem to be possible.

I don't think this belongs under `build-use-sandbox = relaxed` because we
want it on Hydra and I don't think it's the end of the world.
2017-10-30 17:59:12 +01:00
Eelco Dolstra 9971d875a4
Fix building on clang
https://hydra.nixos.org/build/62945761
2017-10-25 17:20:47 +02:00
Eelco Dolstra 82327e3cc4
exportReferencesGraph: Allow exporting a list of store paths 2017-10-25 15:18:49 +02:00
Eelco Dolstra 3395e3bbc4
Fix exportReferencesGraph in the structured attrs case 2017-10-25 14:08:29 +02:00
Eelco Dolstra 2d5b1b24bf
Pass lists/attrsets to bash as (associative) arrays 2017-10-25 13:01:50 +02:00
Eelco Dolstra 3460e4cf00
More progress indicator improvements
In particular, don't show superfluous "fetching path" and "building
path(s)" messages, and show the current round (with --repeat).
2017-10-24 15:32:38 +02:00
Eelco Dolstra 96051dd057
More progress indicator improvements
Fixes #1599.
2017-10-24 14:47:23 +02:00
Eelco Dolstra be220702a7
Progress indicator: Show on what machine we're building
E.g.

  $ nix build nixpkgs.hello --builders 'root@wendy'
  [1/0/1 built] building hello-2.10 on ssh://root@wendy: checking for minix/config.h... no
2017-10-24 14:24:57 +02:00
Eelco Dolstra fe9d2f974d
Handle log messages from build-remote
This makes the progress indicator show statuses like "connecting to
'root@machine'".
2017-10-24 14:05:22 +02:00
Eelco Dolstra 25f32625e2
Remove the remote-builds option
This is superfluous since you can now just set "builders" to empty,
e.g. "--builders ''".
2017-10-24 11:00:16 +02:00
Eelco Dolstra 37fbfffd8e
Pass all settings to build-remote
This ensures that command line flags such as --builders get passed
correctly.
2017-10-23 20:50:28 +02:00
Dan Peebles 6e5165b773 Shift Darwin sandbox to separate installed files
This makes it slightly more manageable to see at a glance what in a
build's sandbox profile is unique to the build and what is standard. Also
a first step to factoring more of our Darwin logic into scheme functions
that will allow us a bit more flexibility. And of course less of that
nasty codegen in C++! 😀
2017-10-17 19:15:51 -04:00
Eelco Dolstra 1dd29d7aeb
Add option to disable the seccomp filter
I needed this to test ACL/xattr removal in
canonicalisePathMetaData(). Might also be useful if you need to build
old Nixpkgs that doesn't have the required patches to remove
setuid/setgid creation.
2017-10-12 18:21:55 +02:00
Eelco Dolstra 88e6bb76de Communicate the structured log FD to builders
Since we may use a dedicated file descriptor in the future, this
allows us to change it. So builders can do

  if [[ -n $NIX_LOG_FD ]]; then
    echo "@nix { message... }" >&$NIX_LOG_FD
  fi
2017-09-08 11:27:10 +02:00
Eelco Dolstra 0b606aad46
Add automatic garbage collection
Nix can now automatically run the garbage collector during builds or
while adding paths to the store. The option "min-free = <bytes>"
specifies that Nix should run the garbage collector whenever free
space in the Nix store drops below <bytes>. It will then delete
garbage until "max-free" bytes are available.

Garbage collection during builds is asynchronous; running builds are
not paused and new builds are not blocked. However, there also is a
synchronous GC run prior to the first build/substitution.

Currently, no old GC roots are deleted (as in "nix-collect-garbage
-d").
2017-09-05 20:43:42 +02:00
Eelco Dolstra bbdf08bc0f
Call queryMissing() prior to building
Without this, substitute info is fetched sequentially, which is
superslow. In the old UI (e.g. nix-build), we call printMissing(),
which calls queryMissing(), thereby preheating the binary cache
cache. But the new UI doesn't do that.
2017-08-31 16:05:07 +02:00
Eelco Dolstra c2154d4c84
Rename a few configuration options
In particular, drop the "build-" and "gc-" prefixes which are
pointless. So now you can say

  nix build --no-sandbox

instead of

  nix build --no-build-use-sandbox
2017-08-31 14:28:25 +02:00
Eelco Dolstra 2cc345b95f
Give activities a verbosity level again
And print them (separately from the progress bar) given sufficient -v
flags.
2017-08-28 19:13:24 +02:00
Eelco Dolstra 94a0548dc4
Simplify 2017-08-25 21:26:37 +02:00
Eelco Dolstra 0ac35b67b8
Allow derivations to update the build phase
So the progress bar can show

  [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25 18:04:05 +02:00
Eelco Dolstra c137c0a5eb
Allow activities to be nested
In particular, this allows more relevant activities ("substituting X")
to supersede inferior ones ("downloading X").
2017-08-25 17:49:40 +02:00
Eelco Dolstra 0e9ddcc306
Restore activity metadata
This allows the progress bar to display "building perl-5.22.3" instead
of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-25 15:58:35 +02:00
Eelco Dolstra 1f56235438
Clean up JSON construction 2017-08-21 12:18:46 +02:00
Eelco Dolstra 4af2611bd1
Allow builders to create activities
Actually, currently they can only create download activities. Thus,
downloads by builtins.fetchurl show up in the progress bar.
2017-08-21 12:18:46 +02:00
Eelco Dolstra b4ed97e3a3
nix optimise-store: Show how much space has been freed 2017-08-16 20:56:03 +02:00
Eelco Dolstra 40bffe0a43
Progress indicator: Cleanup 2017-08-16 20:56:03 +02:00
Eelco Dolstra dff12b38f9
Progress indicator: More improvements 2017-08-16 20:56:03 +02:00
Eelco Dolstra bf1f123b09
Progress indicator: Show number of active items 2017-08-16 20:56:03 +02:00
Eelco Dolstra 0e0dcf2c7e
Progress indicator: Unify "copying" and "substituting"
They're the same thing after all.

Example:

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16 20:56:03 +02:00
Eelco Dolstra c36467ad2e
Improve substitution progress indicator
E.g.

  $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped
  [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16 20:56:02 +02:00
Eelco Dolstra af765a8eab
Use /proc/self/fd to efficiently close all FDs on Linux
Issue #1506.
2017-08-09 16:22:05 +02:00
Jörg Thalheim 2fd8f8bb99 Replace Unicode quotes in user-facing strings by ASCII
Relevant RFC: NixOS/rfcs#4

$ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-07-30 12:32:45 +01:00
Eelco Dolstra fc3568e263
FD_SETSIZE check: BuildError -> Error
BuildError denotes a permanent build failure, which is not the case
here.
2017-07-20 13:33:13 +02:00
Dmitry Kalinkin d5e1bffd2a
Do not try to fill fd_set with fd>=FD_SETSIZE
This is UB and causes buffer overflow and crash on linux.
2017-07-18 17:51:50 -04:00
Eelco Dolstra a3dc1e65ab
Add X32 to the seccomp filter
Fixes #1432.
2017-07-04 19:00:51 +02:00
Eelco Dolstra 6cf23c3e8f
Add allow-new-privileges option
This allows builds to call setuid binaries. This was previously
possible until we started using seccomp. Turns out that seccomp by
default disallows processes from acquiring new privileges. Generally,
any use of setuid binaries (except those created by the builder
itself) is by definition impure, but some people were relying on this
ability for certain tests.

Example:

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --no-allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 2 log lines:
    cannot raise the capability into the Ambient set
    : Operation not permitted

  $ nix build '(with import <nixpkgs> {}; runCommand "foo" {} "/run/wrappers/bin/ping -c 1 8.8.8.8; exit 1")' --allow-new-privileges
  builder for ‘/nix/store/j0nd8kv85hd6r4kxgnwzvr0k65ykf6fv-foo.drv’ failed with exit code 1; last 6 log lines:
    PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
    64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=15.2 ms

Fixes #1429.
2017-07-04 15:48:25 +02:00
Eelco Dolstra c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra 1888f7889b
macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra 7f5b750b40
Don't run pre-build-hook if we don't have a derivation
This fixes a build failure on OS X when using Hydra or Nix 1.12's
build-remote (since they don't copy the derivation to the build
machine).
2017-06-12 16:07:34 +02:00
Eelco Dolstra c8cc50d46e
Disable the build user mechanism on all platforms except Linux and OS X 2017-06-06 18:52:15 +02:00
Eelco Dolstra 85e93d7b87
Always use the Darwin sandbox
Even with "build-use-sandbox = false", we now use sandboxing with a
permissive profile that allows everything except the creation of
setuid/setgid binaries.
2017-06-06 18:44:49 +02:00
Eelco Dolstra c96e8cd097
OS X sandbox: Improve builtin sandbox profile
Also, add rules to allow fixed-output derivations to access the
network.

These rules are sufficient to build stdenvDarwin without any
__sandboxProfile magic.
2017-05-31 17:25:51 +02:00
Eelco Dolstra 44f3f8048f OS X sandbox: Don't use a deterministic $TMPDIR
This doesn't work because the OS X sandbox cannot bind-mount
path to a different location.
2017-05-31 14:09:00 +02:00
Eelco Dolstra c740c3ce50 OS X sandbox: Store .sb file in $TMPDIR rather than the Nix store
The filename used was not unique and owned by the build user, so
builds could fail with

error: while setting up the build environment: cannot unlink ‘/nix/store/99i210ihnsjacajaw8r33fmgjvzpg6nr-bison-3.0.4.drv.sb’: Permission denied
2017-05-31 14:09:00 +02:00
Eelco Dolstra acc889c821
Darwin sandbox: Use sandbox-defaults.sb
Issue #759.

Also, remove nix.conf from the sandbox since I don't really see a
legitimate reason for builders to access the Nix configuration.
2017-05-30 17:40:12 +02:00
Eelco Dolstra 53a1644187
Darwin sandbox: Disallow creating setuid/setgid binaries
Suggested by Daiderd Jordan.
2017-05-30 17:17:17 +02:00
Eelco Dolstra fe08d17934
Fix seccomp build failure on clang
Fixes

  src/libstore/build.cc:2321:45: error: non-constant-expression cannot be narrowed from type 'int' to 'scmp_datum_t' (aka 'unsigned long') in initializer list [-Wc++11-narrowing]
2017-05-30 14:37:24 +02:00
Eelco Dolstra d552d38758
Shut up some clang warnings 2017-05-30 14:35:50 +02:00
Eelco Dolstra 2ac99a32da
Add a seccomp rule to disallow setxattr() 2017-05-30 13:59:24 +02:00
Eelco Dolstra ff6becafa8
Require seccomp only in multi-user setups 2017-05-30 12:37:04 +02:00
Eelco Dolstra cf93397d3f
Fix seccomp initialisation on i686-linux 2017-05-29 16:14:10 +02:00
Eelco Dolstra 6cc6c15a2d
Add a seccomp filter to prevent creating setuid/setgid binaries
This prevents builders from setting the S_ISUID or S_ISGID bits,
preventing users from using a nixbld* user to create a setuid/setgid
binary to interfere with subsequent builds under the same nixbld* uid.

This is based on aszlig's seccomp code
(47f587700d).

Reported by Linus Heckemann.
2017-05-29 16:14:10 +02:00
Eelco Dolstra b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Eelco Dolstra e80257f122
Simplify fixed-output check 2017-05-15 18:50:54 +02:00
Eelco Dolstra b30f5784d0
Linux sandbox: Don't barf on invalid paths
This is useful when we're using a diverted store (e.g. "--store
local?root=/tmp/nix") in conjunction with a statically-linked sh from
the host store (e.g. "sandbox-paths =/bin/sh=/nix/store/.../bin/busybox").
2017-05-15 17:36:32 +02:00
Eelco Dolstra 00b286275c
Linux sandbox: Fix compatibility with older kernels 2017-05-08 15:42:59 +02:00
Eelco Dolstra 0a97eb6bd7
Remove superfluous #ifdef 2017-05-08 11:27:20 +02:00
Eelco Dolstra bb50c89319
Make the location of the build directory in the sandbox configurable
This is mostly for use in the sandbox tests, since if the Nix store is
under /build, then we can't use /build as the build directory.
2017-05-05 17:45:22 +02:00
Eelco Dolstra eba840c8a1
Linux sandbox: Use /build instead of /tmp as $TMPDIR
There is a security issue when a build accidentally stores its $TMPDIR
in some critical place, such as an RPATH. If
TMPDIR=/tmp/nix-build-..., then any user on the system can recreate
that directory and inject libraries into the RPATH of programs
executed by other users. Since /build probably doesn't exist (or isn't
world-writable), this mitigates the issue.
2017-05-04 16:57:03 +02:00
Eelco Dolstra 1a68710d4d
Add an option for specifying remote builders
This is useful for one-off situations where you want to specify a
builder on the command line instead of having to mess with
nix.machines. E.g.

  $ nix-build -A hello --argstr system x86_64-darwin \
    --option builders 'root@macstadium1 x86_64-darwin'

will perform the specified build on "macstadium1".

It also removes the need for a separate nix.machines file since you
can specify builders in nix.conf directly. (In fact nix.machines is
yet another hack that predates the general nix.conf configuration
file, IIRC.)

Note: this option is supported by the daemon for trusted users. The
fact that this allows trusted users to specify paths to SSH keys to
which they don't normally have access is maybe a bit too much trust...
2017-05-02 15:42:43 +02:00
Eelco Dolstra 174b68a2a2
build-hook: If there are no machines defined, quit permanently 2017-05-02 12:16:29 +02:00
Eelco Dolstra 031d70e500
Support arbitrary store URIs in nix.machines
For backwards compatibility, if the URI is just a hostname, ssh://
(i.e. LegacySSHStore) is prepended automatically.

Also, all fields except the URI are now optional. For example, this is
a valid nix.machines file:

  local?root=/tmp/nix

This is useful for testing the remote build machinery since you don't
have to mess around with ssh.
2017-05-01 17:35:30 +02:00
Eelco Dolstra 3f5b98e65a
Chomp log output from the build hook 2017-05-01 17:30:16 +02:00
Eelco Dolstra d7653dfc6d
Remove $NIX_BUILD_HOOK and $NIX_CURRENT_LOAD
This is to simplify remote build configuration. These environment
variables predate nix.conf.

The build hook now has a sensible default (namely build-remote).

The current load is kept in the Nix state directory now.
2017-05-01 17:30:16 +02:00
Eelco Dolstra ca9f589a93
build-remote: Don't copy the .drv closure
Since build-remote uses buildDerivation() now, we don't need to copy
the .drv file anymore. This greatly reduces the set of input paths
copied to the remote side (e.g. from 392 to 51 store paths for GNU
hello on x86_64-darwin).
2017-05-01 17:30:16 +02:00
Eelco Dolstra b986c7f8b1
Pass verbosity level to build hook 2017-05-01 14:43:14 +02:00
Eelco Dolstra a1a5e63e14
Fix brainfart 2017-04-28 16:21:54 +02:00
Eelco Dolstra f8a2e8a552
Shut up some warnings 2017-04-14 14:42:20 +02:00
Eelco Dolstra ba9ad29fdb
Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra 1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra 2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra 8decb07c31
Allow default sandbox paths to be overriden
E.g. you can now redirect /etc/resolv.conf to a different file.
2017-04-04 17:54:16 +02:00
Eelco Dolstra 488792a87d
Make /var/run/nscd/socket optional
Not every distribution uses nscd.
2017-04-04 17:40:50 +02:00
Eelco Dolstra 29d35805c6
Sandbox: Fix /dev/ptmx on recent kernels
This fixes "No such file or directory" when opening /dev/ptmx
(e.g. http://hydra.nixos.org/build/51094249).

The reason appears to be some changes to /dev/ptmx / /dev/pts handling
between Linux 4.4 and 4.9. See
https://patchwork.kernel.org/patch/7832531/.

The fix is to go back to mounting a proper /dev/pts instance inside
the sandbox. Happily, this now works inside user namespaces, even for
unprivileged users. So

  NIX_REMOTE=local?root=/tmp/nix nix-build \
    '<nixpkgs/nixos/tests/misc.nix>' -A test

works for non-root users.

The downside is that the fix breaks sandbox builds on older kernels
(probably pre-4.6), since mounting a devpts fails inside user
namespaces for some reason I've never been able to figure out. Builds
on those systems will fail with

  error: while setting up the build environment: mounting /dev/pts: Invalid argument

Ah well.
2017-03-31 18:20:19 +02:00
Eelco Dolstra 0ec7f47b00
Remove "killing process <pid>" messages
They convey no useful information.
2017-03-16 10:52:28 +01:00
Eelco Dolstra 0afeb7f51e
Store: Add a method for getting build logs
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.

Also, remove the log-servers option since we can use substituters for
this.
2017-03-15 16:48:29 +01:00
Eelco Dolstra 2f992692e2
Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing 2017-03-03 15:40:06 +01:00
Eelco Dolstra e76df9bd52
Register content-addressability assertion for fixed outputs 2017-02-24 16:39:53 +01:00
Dan Peebles e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Eelco Dolstra 302386f775
Support netrc in <nix/fetchurl.nix>
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial
repositories, e.g.

  import <nix/fetchurl.nix> {
    url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2;
    sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7";
  }

where /etc/nix/netrc contains:

  machine bitbucket.org
  login edolstra
  password blabla...

This works even when sandboxing is enabled.

To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16 15:51:50 +01:00
Tuomas Tynkkynen 2cd468874f Include config.h implicitly with '-include config.h' in CFLAGS
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not
every compilation unit includes config.h, we currently compile half of
Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS
unset. This causes major havoc with the Settings class on e.g. 32-bit ARM,
where different compilation units disagree with the struct layout.

E.g.:

diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc
@@ -166,6 +166,8 @@ void Settings::update()
     _get(useSubstitutes, "build-use-substitutes");
+    fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     _get(buildUsersGroup, "build-users-group");
diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn)
 void RemoteStore::setOptions(Connection & conn)
 {
+    fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes);
     conn.to << wopSetOptions

Gave me:

at Settings::update(): &useSubstitutes = 0xb6e5c5cb
at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7

That was not a fun one to debug!
2017-02-08 21:51:02 +02:00
Eelco Dolstra 1351b0df87
exportReferencesGraph: Only export in JSON format when in structured mode
This prevents breaking compatibility with builders that read
"closure.*", since they would accidentally pick up the new JSON files.
2017-02-02 12:20:28 +01:00
Eelco Dolstra 7a65b2470e
Restore default signal handling in child processes
In particular, this fixes Ctrl-C in nix-shell sessions.
2017-02-01 13:00:21 +01:00
Eelco Dolstra c2b0d8749f
exportReferencesGraph: Export more complete info in JSON format
This writes info about every path in the closure in the same format as
‘nix path-info --json’. Thus it also includes NAR hashes and sizes.

Example:

  [
    {
      "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl",
      "narSize": 197648,
      "references": [
        "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20939776
    },
    {
      "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24",
      "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3",
      "narSize": 20742128,
      "references": [
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20742128
    }
  ]

Fixes #1134.
2017-01-26 20:41:08 +01:00
Eelco Dolstra 6de33a9c67
Add support for passing structured data to builders
Previously, all derivation attributes had to be coerced into strings
so that they could be passed via the environment. This is lossy
(e.g. lists get flattened, necessitating configureFlags
vs. configureFlagsArray, of which the latter cannot be specified as an
attribute), doesn't support attribute sets at all, and has size
limitations (necessitating hacks like passAsFile).

This patch adds a new mode for passing attributes to builders, namely
encoded as a JSON file ".attrs.json" in the current directory of the
builder. This mode is activated via the special attribute

  __structuredAttrs = true;

(The idea is that one day we can set this in stdenv.mkDerivation.)

For example,

  stdenv.mkDerivation {
    __structuredAttrs = true;
    name = "foo";
    buildInputs = [ pkgs.hello pkgs.cowsay ];
    doCheck = true;
    hardening.format = false;
  }

results in a ".attrs.json" file containing (sans the indentation):

  {
    "buildInputs": [],
    "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash",
    "configureFlags": [
      "--with-foo",
      "--with-bar=1 2"
    ],
    "doCheck": true,
    "hardening": {
      "format": false
    },
    "name": "foo",
    "nativeBuildInputs": [
      "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03"
    ],
    "propagatedBuildInputs": [],
    "propagatedNativeBuildInputs": [],
    "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv",
    "system": "x86_64-linux"
  }

"passAsFile" is ignored in this mode because it's not needed - large
strings are included directly in the JSON representation.

It is up to the builder to do something with the JSON
representation. For example, in bash-based builders, lists/attrsets of
string values could be mapped to bash (associative) arrays.
2017-01-26 20:40:33 +01:00
Eelco Dolstra 951357e5fb
UserLock: Fix multi-threaded access to a global variable 2017-01-26 20:40:33 +01:00
Eelco Dolstra c0f2f4eeef
UserLock: Make more RAII-ish 2017-01-26 20:40:33 +01:00
Eelco Dolstra a529c740d2
Moving more code out of DerivationGoal::startBuilder() 2017-01-26 20:40:33 +01:00
Eelco Dolstra 4425a5c547
Move exportReferencesGraph into a separate method
startBuilder() is getting rather obese.
2017-01-26 20:40:32 +01:00
Eelco Dolstra 21948deed9
Kill builds when we get EOF on the log FD
This closes a long-time bug that allowed builds to hang Nix
indefinitely (regardless of timeouts) simply by doing

  exec > /dev/null 2>&1; while true; do true; done

Now, on EOF, we just send SIGKILL to the child to make sure it's
really gone.
2017-01-19 17:16:14 +01:00
Eelco Dolstra 2579e32c2b
Use std::unique_ptr for HookInstance 2017-01-19 17:06:04 +01:00
Eelco Dolstra 7adb986e35 Merge pull request #1139 from Mic92/master
Simplify remouting with MS_PRIVATE in sandbox build
2017-01-09 16:46:59 +01:00
Eelco Dolstra b6b142b4b1
Provide /var/run/nscd/socket in the sandbox
Otherwise sandbox builds can fail, e.g.

  $ NIX_REMOTE=local?root=/tmp/nix nix-build '<nixpkgs>' -A hello --option build-use-substitutes false
  ...
  downloading ‘http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047’...
  error: unable to download ‘http://ftpmirror.gnu.org/bash/bash-4.3-patches/bash43-047’: Couldn't resolve host name (6)
2017-01-02 14:46:37 +01:00
Eelco Dolstra 786ee585b8
Add comment 2016-12-22 17:39:49 +01:00
Eelco Dolstra 3a4bd320c2
Revert "Merge branch 'seccomp' of https://github.com/aszlig/nix"
This reverts commit 9f3f2e21ed, reversing
changes made to 47f587700d.
2016-12-19 11:52:57 +01:00
Eelco Dolstra 05862209de
Revert "Give root a valid home directory"
This reverts commit ec7d498b72.
2016-12-19 11:49:03 +01:00
Eelco Dolstra ec7d498b72
Give root a valid home directory
Some programs barf if the current user has a non-writable home
directory, e.g. http://hydra.nixos.org/build/44818144.
2016-12-15 15:56:08 +01:00
Eelco Dolstra 9f3f2e21ed
Merge branch 'seccomp' of https://github.com/aszlig/nix 2016-12-15 12:04:45 +01:00
Eelco Dolstra b30d1e7ada
Don't delete .check directories of running builds
We need to keep them around for diffoscope.
2016-12-08 21:38:58 +01:00
Eelco Dolstra e629a17cc1
Fix build
http://hydra.nixos.org/build/44628517
2016-12-08 20:36:14 +01:00
Eelco Dolstra 9a313469a4
Add a hook to run diffoscope when non-determinism is detected 2016-12-07 17:57:35 +01:00
Eelco Dolstra b07060688a
Keep track of the exact build start/stop times 2016-12-07 16:09:38 +01:00
Eelco Dolstra dadfddfa7c
Bail out early when non-determinism is detected 2016-12-07 15:31:18 +01:00
Eelco Dolstra 8bdf83f936
Add an option to make non-determinism non-fatal
That is, when build-repeat > 0, and the output of two rounds differ,
then print a warning rather than fail the build. This is primarily to
let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Eelco Dolstra ceeedb58d2
Use a steady clock for timeouts
Fixes #1146.
2016-12-06 21:58:04 +01:00
Eelco Dolstra 7a3e7d0e61
nix-store --serve: Suppress log output on stderr when repeating a build 2016-12-06 17:43:39 +01:00
Eelco Dolstra 215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Jörg Thalheim 855abd85d8
Simplify remouting with MS_PRIVATE in sandbox build
also fix race condition if mounts are added after mountinfo is read.
2016-11-25 00:15:39 +01:00
Shea Levy 8bf378e999 Update darwin build for optional sandbox paths
Fixes #1132
2016-11-17 08:06:32 -05:00
aszlig 4e1a2cd537
seccomp: Forge return values for *chown32
These syscalls are only available in 32bit architectures, but libseccomp
should handle them correctly even if we're on native architectures that
do not have these syscalls.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:36 +01:00
aszlig ed64976cec
seccomp: Forge return codes for POSIX ACL syscalls
Commands such as "cp -p" also use fsetxattr() in addition to fchown(),
so we need to make sure these syscalls always return successful as well
in order to avoid nasty "Invalid value" errors.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:14 +01:00
aszlig b90a435332
libstore/build: Forge chown() to return success
What we basically want is a seccomp mode 2 BPF program like this but for
every architecture:

  BPF_STMT(BPF_LD+BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_chown, 4, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchown, 3, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchownat, 2, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_lchown, 1, 0),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO)

However, on 32 bit architectures we do have chown32, lchown32 and
fchown32, so we'd need to add all the architecture blurb which
libseccomp handles for us.

So we only need to make sure that we add the 32bit seccomp arch while
we're on x86_64 and otherwise we just stay at the native architecture
which was set during seccomp_init(), which more or less replicates
setting 32bit personality during runChild().

The FORCE_SUCCESS() macro here could be a bit less ugly but I think
repeating the seccomp_rule_add() all over the place is way uglier.

Another way would have been to create a vector of syscalls to iterate
over, but that would make error messages uglier because we can either
only print the (libseccomp-internal) syscall number or use
seccomp_syscall_resolve_num_arch() to get the name or even make the
vector a pair number/name, essentially duplicating everything again.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:30 +01:00
aszlig e8838713df
Run builds as root in user namespace again
This reverts commit ff0c0b645c.

We're going to use seccomp to allow "cp -p" and force chown-related
syscalls to always return 0.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:08 +01:00
Eelco Dolstra 10ae8fabf1 buildPaths(): Handle ecIncompleteClosure
buildPaths() on a non-derivation would incorrectly not throw an error
if the path didn't have a substitute.
2016-11-14 15:00:17 +01:00
Eelco Dolstra dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Eelco Dolstra a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra 18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Eelco Dolstra 5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00
Eelco Dolstra ae8884b949 Fix /dev/ptmx in sandboxes
This was broken since ff0c0b645c. Since
I can't figure out how to mount a devpts instance in the sandbox,
let's just bind-mount the host devpts.
2016-10-14 15:42:13 +02:00
Eelco Dolstra 818ab58cc6 Add sandbox-dev-shm-size option
Fixes #1069.
2016-09-21 16:54:53 +02:00
Eelco Dolstra c55bf085eb printMsg(lvlError, ...) -> printError(...) etc. 2016-09-21 16:54:53 +02:00
Eelco Dolstra 4036185cb4 Some notational convenience for formatting strings
We can now write

  throw Error("file '%s' not found", path);

instead of

  throw Error(format("file '%s' not found") % path);

and similarly

  printError("file '%s' not found", path);

instead of

  printMsg(lvlError, format("file '%s' not found") % path);
2016-09-21 16:54:53 +02:00
Eelco Dolstra ff0c0b645c Don't run builds as root in the user namespace
This largely reverts c68e5913c7. Running
builds as root breaks "cp -p", since when running as root, "cp -p"
assumes that it can succesfully chown() files. But that's not actually
the case since the user namespace doesn't provide a complete uid
mapping. So it barfs with a fatal error message ("cp: failed to
preserve ownership for 'foo': Invalid argument").
2016-09-08 18:16:23 +02:00
Eelco Dolstra e4bdd49022 Give a better error message if a path in build-sandbox-paths is invalid 2016-09-08 17:30:25 +02:00
Eelco Dolstra 0babc4ff31 Fix assertion failure after early build setup failure 2016-09-08 17:29:50 +02:00
Eelco Dolstra 97b1af1cbe Fix --fallback
This fixes an assertion failure in "assert(goal);" in
Worker::waitForInput() after a substitution goal is cancelled by the
termination of another goal. The problem was the line

  //worker.childTerminated(shared_from_this()); // FIXME

in the SubstitutionGoal destructor. This was disabled because
shared_from_this() obviously doesn't work from a destructor. So we now
use a real pointer for object identity.
2016-08-30 15:45:39 +02:00
Eelco Dolstra 22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Eelco Dolstra d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo
This allows such paths to be imported without signatures.
2016-08-10 18:05:35 +02:00
Eelco Dolstra 3eb6217508 Add a "root" parameter to local stores
This makes it easier to create a diverted store, i.e.

  NIX_REMOTE="local?root=/tmp/root"

instead of

  NIX_REMOTE="local?real=/tmp/root/nix/store&state=/tmp/root/nix/var/nix" NIX_LOG_DIR=/tmp/root/nix/var/log
2016-08-10 18:05:35 +02:00
Eelco Dolstra ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Shea Levy 18b0808475 Respect --keep-going when a substituter fails.
Fixes #977
2016-07-23 09:16:08 -04:00
Shea Levy cb5e7254b6 Modernize AutoCloseFD 2016-07-11 15:44:44 -04:00
Eelco Dolstra c68e5913c7 Run builds in a user namespace
This way, all builds appear to have a uid/gid of 0 inside the
chroot. In the future, this may allow using programs like
systemd-nspawn inside builds, but that will require assigning a larger
UID/GID map to the build.

Issue #625.
2016-06-09 18:30:39 +02:00
Eelco Dolstra 202683a4fc Use O_CLOEXEC in most places 2016-06-09 16:37:08 +02:00
Eelco Dolstra 5e51ffb1c2 Support sandbox builds by non-root users
This allows an unprivileged user to perform builds on a diverted store
(i.e. where the physical store location differs from the logical
location).

Example:

  $ NIX_LOG_DIR=/tmp/log NIX_REMOTE="local?real=/tmp/store&state=/tmp/var" nix-build -E \
    'with import <nixpkgs> {}; runCommand "foo" { buildInputs = [procps nettools]; } "id; ps; ifconfig; echo $out > $out"'

will do a build in the Nix store physically in /tmp/store but
logically in /nix/store (and thus using substituters for the latter).
2016-06-03 15:59:28 +02:00
Eelco Dolstra 080d607975 Don't call shared_from_this() from a destructor 2016-06-03 14:02:44 +02:00
Eelco Dolstra 4494000e04 LocalStore: Allow the physical and logical store directories to differ
This is primarily to subsume the functionality of the
copy-from-other-stores substituter. For example, in the NixOS
installer, we can now do (assuming we're in the target chroot, and the
Nix store of the installation CD is bind-mounted on /tmp/nix):

  $ nix-build ... --option substituters 'local?state=/tmp/nix/var&real=/tmp/nix/store'

However, unlike copy-from-other-stores, this also allows write access
to such a store. One application might be fetching substitutes for
/nix/store in a situation where the user doesn't have sufficient
privileges to create /nix, e.g.:

  $ NIX_REMOTE="local?state=/home/alice/nix/var&real=/home/alice/nix/store" nix-build ...
2016-06-02 16:02:48 +02:00
Eelco Dolstra 064816ab98 Respect build-use-substitutes 2016-06-02 16:02:48 +02:00
Eelco Dolstra e24e2caaaf Skip substituters with an incompatible store directory 2016-06-01 16:40:49 +02:00
Eelco Dolstra 7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra 57d33013ce Check signatures before downloading the substitute 2016-05-30 15:18:12 +02:00
Eelco Dolstra b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra 6c75cf69c3 Cleanup: Remove singleton() 2016-05-04 16:16:53 +02:00
Eelco Dolstra 0d4a10e910 Do compression in a sink 2016-05-04 16:16:53 +02:00
Eelco Dolstra dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra aa3bc3d5dc Eliminate the substituter mechanism
Substitution is now simply a Store -> Store copy operation, most
typically from BinaryCacheStore to LocalStore.
2016-04-29 13:57:08 +02:00
Eelco Dolstra 21e9d183cc Really handle carriage return 2016-04-28 14:27:00 +02:00
Eelco Dolstra cc804d0dc6 Handle carriage return 2016-04-25 19:18:45 +02:00
Eelco Dolstra 5761827d5b Show the log tail when a build fails
If --no-build-output is given (which will become the default for the
"nix" command at least), show the last 10 lines of the build output if
the build fails.
2016-04-25 19:18:45 +02:00
Eelco Dolstra 6e1b099279 Remove --print-build-trace
This was added to support Hydra, but Hydra no longer uses it.
2016-04-25 19:18:45 +02:00
Eelco Dolstra 41633f9f73 Improved logging abstraction
This also gets rid of --log-type, since the nested log type isn't
useful in a multi-threaded situation, and nobody cares about the
"pretty" log type.
2016-04-25 19:18:45 +02:00
Eelco Dolstra e0204f8d46 Move path info caching from BinaryCacheStore to Store
Caching path info is generally useful. For instance, it speeds up "nix
path-info -rS /run/current-system" (i.e. showing the closure sizes of
all paths in the closure of the current system) from 5.6s to 0.15s.

This also eliminates some APIs like Store::queryDeriver() and
Store::queryReferences().
2016-04-19 18:52:53 +02:00
Dan Peebles 608b0265e1 Print out all bad references/requisites at once
Also updates tests to check for new information. Fixes #799
2016-04-16 19:58:26 -04:00
Eelco Dolstra 3c1c6b8f00 Set RLIMIT_CORE to 0, infinity in builders
This prevents the builder from being affected by whatever the host
system limits happen to be.
2016-04-14 13:39:14 +02:00
Eelco Dolstra 8cffec8485 Remove failed build caching
This feature was implemented for Hydra, but Hydra no longer uses it.
2016-04-08 18:19:04 +02:00
Eelco Dolstra f398949b40 Make LocalStore thread-safe
Necessary for multi-threaded commands like "nix verify-paths".
2016-04-08 18:07:13 +02:00
Eelco Dolstra e39999ed48 Sign locally-built paths
Locally-built paths are now signed automatically using the secret keys
specified by the ‘secret-key-files’ option.
2016-04-07 15:16:57 +02:00
Eelco Dolstra 9cee600c88 LocalStore: Keep track of ultimately trusted paths
These are content-addressed paths or outputs of locally performed
builds. They are trusted even if they don't have signatures, so "nix
verify-paths" won't complain about them.
2016-03-30 17:35:48 +02:00
Eelco Dolstra 42bc395b63 Eliminate some large string copying 2016-03-04 16:49:56 +01:00
Nathan Zadoks fe2be8f016 build.cc: fs.h doesn't appear to be necessary anymore 2016-03-03 14:11:00 +01:00
Eelco Dolstra 152b1d6bf9 deletePath(): Succeed if path doesn't exist
Also makes it robust against concurrent deletions.
2016-02-24 17:44:55 +01:00
Eelco Dolstra 5ac27053e9 Rename ValidPathInfo::hash -> narHash for consistency 2016-02-16 11:49:12 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra d367b8e787 Add build mode to compute fixed-output derivation hashes
For example,

  $ nix-build --hash -A nix-repl.src

will build the fixed-output derivation nix-repl.src (a fetchFromGitHub
call), but instead of *verifying* the hash given in the Nix
expression, it prints out the resulting hash, and then moves the
result to its content-addressed location in the Nix store. E.g

  build produced path ‘/nix/store/504a4k6zi69dq0yjc0bm12pa65bccxam-nix-repl-8a2f5f0607540ffe56b56d52db544373e1efb980-src’ with sha256 hash ‘0cjablz01i0g9smnavhf86imwx1f9mnh5flax75i615ml71gsr88’

The goal of this is to make all nix-prefetch-* scripts unnecessary: we
can just let Nix run the real thing (i.e., the corresponding fetch*
derivation).

Another example:

  $ nix-build --hash -E 'with import <nixpkgs> {}; fetchgit { url = "https://github.com/NixOS/nix.git"; sha256 = "ffffffffffffffffffffffffffffffffffffffffffffffffffff"; }'
  ...
  git revision is 9e7c1a4bbd
  ...
  build produced path ‘/nix/store/gmsnh9i7x4mb7pyd2ns7n3c9l90jfsi1-nix’ with sha256 hash ‘1188xb621diw89n25rifqg9lxnzpz7nj5bfh4i1y3dnis0dmc0zp’

(Having to specify a fake sha256 hash is a bit annoying...)
2016-01-31 12:18:09 +01:00
Eelco Dolstra 786046cf13 --option build-repeat: Keep the differing output if -K is given
Similar to 00903fa799. Regardless of -K,
we now also print which output differs.
2016-01-12 18:26:24 +01:00
Eelco Dolstra 00903fa799 --check: Keep the differing output if -K is given
This makes it easier to investigate the non-determinism, e.g.

  $ nix-build pkgs/stdenv/linux -A stage1.pkgs.zlib --check -K
  error: derivation ‘/nix/store/l54i8wlw22656i4pk05c52ngv9rpl39q-zlib-1.2.8.drv’ may not be deterministic: output ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8’ differs from ‘/nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check’

  $ diffoscope /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8 /nix/store/11a27shh6n2ivi4a7s964i65ql80cf27-zlib-1.2.8-check
  ...
  ├── lib/libz.a
  │   ├── metadata
  │   │ @@ -1,15 +1,15 @@
  │   │ -rw-r--r-- 30001/30000   3096 Jan 12 15:20 2016 adler32.o
  ...
  │   │ +rw-r--r-- 30001/30000   3096 Jan 12 15:28 2016 adler32.o
  ...
2016-01-12 16:44:26 +01:00
Eelco Dolstra 0cad1f8049 --check: Fix "failed to produce output path"
This occured when sandbox building is disabled, at least one output
exists, and at least one other output does not.
2016-01-12 15:13:47 +01:00
Eelco Dolstra 1c57ab8b31 --check: Fix assertion failure when some outputs are missing
E.g.

  $ nix-build pkgs/stdenv/linux/ -A stage1.pkgs.perl --check
  nix-store: src/libstore/build.cc:1323: void nix::DerivationGoal::tryToBuild(): Assertion `buildMode != bmCheck || validPaths.size() == drv->outputs.size()' failed.

when perl.out exists but perl.man doesn't. The fix is to only check
the outputs that exist. Note that "nix-build -A stage1.pkgs.all
--check" will still give a (proper) error in this case.
2016-01-12 14:54:39 +01:00
Eelco Dolstra 458711e4ee Fix "Bad address" executing build hook
This was observed in the deb_debian7x86_64 build:
http://hydra.nixos.org/build/29973215

Calling c_str() on a temporary should be fine because the temporary
shouldn't be destroyed until after the execl() call, but who knows...
2016-01-07 15:10:14 +01:00
Eelco Dolstra 9aac1861f7 Fix some signedness warnings 2016-01-07 14:37:39 +01:00
Eelco Dolstra 02a66b3fd7 nix-store --repair-path: Rebuild if there is no substituter 2016-01-06 22:07:59 +01:00
Eelco Dolstra caaaff3954 Fix --repair failure on multiple-output derivations
If repair found a corrupted/missing path that depended on a
multiple-output derivation, and some of the outputs of the latter were
not present, it failed with a message like

  error: path ‘/nix/store/cnfn9d5fjys1y93cz9shld2xwaibd7nn-bash-4.3-p42-doc’ is not valid
2016-01-06 21:49:32 +01:00
Eelco Dolstra 743e310046 Fix non-Darwin build 2016-01-05 13:31:15 +01:00
Tuomas Tynkkynen f770b9e6c5 libstore: mmap() returns MAP_FAILED, not NULL on failure 2016-01-05 13:26:35 +01:00
Eelco Dolstra 71a93a5f0e Don't allow sandbox profile except in relaxed mode
This makes Darwin consistent with Linux: Nix expressions can't break
out of the sandbox unless relaxed sandbox mode is enabled.

For the normal sandbox mode this will require fixing #759 however.
2016-01-04 20:01:13 +01:00
Eelco Dolstra b8258a4475 Fix regression in passAsFile
Caused by 8063fc497a. If tmpDir !=
tmpDirInSandbox (typically when there are multiple concurrent builds
with the same name), the *Path attribute would not point to an
existing file. This caused Nixpkgs' writeTextFile to write an empty
file. In particular this showed up as hanging VM builds (because it
would run an empty run-nixos-vm script and then wait for it to finish
booting).
2015-12-29 15:28:20 +01:00
Eelco Dolstra 52120123a5 Handle /tmp being a symlink
Hopefully fixes Darwin sandbox regression introduced in
8063fc497a.
2015-12-22 17:16:17 +01:00
Eelco Dolstra f696af0fab Fix bad error message in Darwin chroots 2015-12-22 17:05:29 +01:00
Eelco Dolstra 8f67325a7c Build sandbox support etc. unconditionally on Linux
Also, use "#if __APPLE__" instead of "#if SANDBOX_ENABLED" to prevent
ambiguity.
2015-12-10 11:47:17 +01:00
Bjørn Forsman 65bd82d42a Clarify error message for hash mismatches (again)
This is arguably nitpicky, but I think this new formulation is even
clearer. My thinking is that it's easier to comprehend when the
calculated hash value is displayed close to the output path. (I think it
is somewhat similar to eliminating double negatives in logic
statements.)

The formulation is inspired / copied from the OpenEmbedded build tool,
bitbake.
2015-12-08 19:50:25 +01:00
Eelco Dolstra 8063fc497a Use deterministic $TMPDIR in sandbox
Rather than using $<host-TMPDIR>/nix-build-<drvname>-<number>, the
temporary directory is now always /tmp/nix-build-<drvname>-0. This
improves bitwise-exact reproducibility for builds that store $TMPDIR
in their build output. (Of course, those should still be fixed...)
2015-12-02 15:04:00 +01:00
Eelco Dolstra 0ab4d905e7 Merge branch 'p/sandbox-rename-minimal' of https://github.com/vcunat/nix 2015-11-25 14:53:42 +01:00
Eelco Dolstra a0f0733413 Fix build failure introduced by #704
Also, make the FreeBSD checks conditional on FreeBSD.
2015-11-25 14:41:19 +01:00
Eelco Dolstra cad40adce5 Merge pull request #704 from ysangkok/freebsd-support
FreeBSD support with knowledge about Linux emulation
2015-11-24 19:24:21 +01:00