Commit graph

1491 commits

Author SHA1 Message Date
Eelco Dolstra c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes #1437.
2017-07-04 15:07:41 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
David McFarland 596b0e0a04 Call SetDllDirectory("") after sqlite3 init on cygwin
Cygwin sqlite3 is patched to call SetDllDirectory("/usr/bin") on init, which
affects the current process and is inherited by child processes.  It causes
DLLs to be loaded from /usr/bin/ before $PATH, which breaks all sorts of
things.  A typical failures would be header/lib version mismatches (e.g.
openssl when running checkPhase on openssh).  We'll just set it back to the
default value.

Note that this is a problem with the cygwin version of sqlite3 (currently
3.18.0).  nixpkgs doesn't have the problematic patch.
2017-06-20 10:59:13 -03:00
Eelco Dolstra a1355917ec
Disable use of virtual hosting in aws-sdk-cpp
Recently aws-sdk-cpp quietly switched to using S3 virtual host URIs
(https://github.com/aws/aws-sdk-cpp/commit/69d9c53882), i.e. it sends
requests to http://<bucket>.<region>.s3.amazonaws.com rather than
http://<region>.s3.amazonaws.com/<bucket>. However this interacts
badly with curl connection reuse. For example, if we do the following:

1) Check whether a bucket exists using GetBucketLocation.
2) If it doesn't, create it using CreateBucket.
3) Do operations on the bucket.

then 3) will fail for a minute or so with a NoSuchBucket exception,
presumably because the server being hit is a fallback for cases when
buckets don't exist.

Disabling the use of virtual hosts ensures that 3) succeeds
immediately. (I don't know what S3's consistency guarantees are for
bucket creation, but in practice buckets appear to be available
immediately.)
2017-06-19 18:51:33 +02:00
Eelco Dolstra 82a0d614cf
Support creating S3 caches in other regions than us-east-1 2017-06-19 18:51:33 +02:00
Eelco Dolstra b33621d425
Handle S3Errors::RESOURCE_NOT_FOUND from aws-sdk-cpp
This is returned by recent versions. Also handle NO_SUCH_KEY even
though the library doesn't actually return that at the moment.
2017-06-19 18:51:32 +02:00
Eelco Dolstra 1c969611ba
Suppress "will retry in N ms" for non-retriable errors
Newer versions of aws-sdk-cpp call CalculateDelayBeforeNextRetry()
even for non-retriable errors (like NoSuchKey) whih causes log spam in
hydra-queue-runner.
2017-06-19 18:51:32 +02:00
Eelco Dolstra 00aa7c6705
Show aws-sdk-cpp log messages 2017-06-19 18:51:32 +02:00
Eelco Dolstra 1888f7889b
macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra b5bdfdef73
macOS: Remove flags
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the
path to be garbage-collected or optimised.

See https://github.com/NixOS/nixpkgs/issues/25819.
+       the file from being garbage-collected.
2017-06-19 14:19:21 +02:00
Eelco Dolstra 38b7d55af1
Remove redundant debug line 2017-06-14 13:45:38 +02:00
Eelco Dolstra 88b291ffc4
canonicalisePathMetaData(): Ignore security.selinux attribute
Untested, hopefully fixes #1406.
2017-06-14 11:41:03 +02:00
Eelco Dolstra 847f19a5f7
Provide a builtin default for $NIX_SSL_CERT_FILE
This is mostly to ensure that when Nix is started on macOS via a
launchd service or sshd (for a remote build), it gets a certificate
bundle.
2017-06-12 16:44:43 +02:00
Eelco Dolstra 7f5b750b40
Don't run pre-build-hook if we don't have a derivation
This fixes a build failure on OS X when using Hydra or Nix 1.12's
build-remote (since they don't copy the derivation to the build
machine).
2017-06-12 16:07:34 +02:00
Eelco Dolstra 186571965d
Don't show flags from config settings in "nix --help" 2017-06-07 18:41:20 +02:00
Eelco Dolstra b8283773bd
nix: Make all options available as flags
Thus, instead of ‘--option <name> <value>’, you can write ‘--<name>
<value>’. So

  --option http-connections 100

becomes

  --http-connections 100

Apart from brevity, the difference is that it's not an error to set a
non-existent option via --option, but unrecognized arguments are
fatal.

Boolean options have special treatment: they're mapped to the
argument-less flags ‘--<name>’ and ‘--no-<name>’. E.g.

  --option auto-optimise-store false

becomes

  --no-auto-optimise-store
2017-06-07 16:17:17 +02:00
Eelco Dolstra c8cc50d46e
Disable the build user mechanism on all platforms except Linux and OS X 2017-06-06 18:52:15 +02:00
Eelco Dolstra 85e93d7b87
Always use the Darwin sandbox
Even with "build-use-sandbox = false", we now use sandboxing with a
permissive profile that allows everything except the creation of
setuid/setgid binaries.
2017-06-06 18:44:49 +02:00
Eelco Dolstra 52fec8dde8
Remove listxattr assertion
It appears that sometimes, listxattr() returns a different value for
the query case (i.e. when the buffer size is 0).
2017-05-31 20:43:47 +02:00
Eelco Dolstra c96e8cd097
OS X sandbox: Improve builtin sandbox profile
Also, add rules to allow fixed-output derivations to access the
network.

These rules are sufficient to build stdenvDarwin without any
__sandboxProfile magic.
2017-05-31 17:25:51 +02:00
Eelco Dolstra 44f3f8048f OS X sandbox: Don't use a deterministic $TMPDIR
This doesn't work because the OS X sandbox cannot bind-mount
path to a different location.
2017-05-31 14:09:00 +02:00
Eelco Dolstra c740c3ce50 OS X sandbox: Store .sb file in $TMPDIR rather than the Nix store
The filename used was not unique and owned by the build user, so
builds could fail with

error: while setting up the build environment: cannot unlink ‘/nix/store/99i210ihnsjacajaw8r33fmgjvzpg6nr-bison-3.0.4.drv.sb’: Permission denied
2017-05-31 14:09:00 +02:00
Eelco Dolstra acc889c821
Darwin sandbox: Use sandbox-defaults.sb
Issue #759.

Also, remove nix.conf from the sandbox since I don't really see a
legitimate reason for builders to access the Nix configuration.
2017-05-30 17:40:12 +02:00
Eelco Dolstra 53a1644187
Darwin sandbox: Disallow creating setuid/setgid binaries
Suggested by Daiderd Jordan.
2017-05-30 17:17:17 +02:00
Eelco Dolstra fe08d17934
Fix seccomp build failure on clang
Fixes

  src/libstore/build.cc:2321:45: error: non-constant-expression cannot be narrowed from type 'int' to 'scmp_datum_t' (aka 'unsigned long') in initializer list [-Wc++11-narrowing]
2017-05-30 14:37:24 +02:00
Eelco Dolstra d552d38758
Shut up some clang warnings 2017-05-30 14:35:50 +02:00
Eelco Dolstra 2ac99a32da
Add a seccomp rule to disallow setxattr() 2017-05-30 13:59:24 +02:00
Eelco Dolstra d798349ede
canonicalisePathMetaData(): Remove extended attributes / ACLs
EAs/ACLs are not part of the NAR canonicalisation. Worse, setting an
ACL allows a builder to create writable files in the Nix store. So get
rid of them.

Closes #185.
2017-05-30 13:47:41 +02:00
Eelco Dolstra ff6becafa8
Require seccomp only in multi-user setups 2017-05-30 12:37:04 +02:00
Eelco Dolstra cf93397d3f
Fix seccomp initialisation on i686-linux 2017-05-29 16:14:10 +02:00
Eelco Dolstra 6cc6c15a2d
Add a seccomp filter to prevent creating setuid/setgid binaries
This prevents builders from setting the S_ISUID or S_ISGID bits,
preventing users from using a nixbld* user to create a setuid/setgid
binary to interfere with subsequent builds under the same nixbld* uid.

This is based on aszlig's seccomp code
(47f587700d).

Reported by Linus Heckemann.
2017-05-29 16:14:10 +02:00
Eelco Dolstra 6e01ecd112
Fix nix-copy-closure test
Fixes

  client# error: size mismatch importing path ‘/nix/store/ywf5fihjlxwijm6ygh6s0a353b5yvq4d-libidn2-0.16’; expected 0, got 120264

This is mostly an artifact of the NixOS VM test environment, where the
Nix database doesn't contain hashes/sizes.

http://hydra.nixos.org/build/53537471
2017-05-29 16:08:56 +02:00
Eelco Dolstra 63145be2a5
Fix typo 2017-05-29 15:52:36 +02:00
Eelco Dolstra edbb105e98
Merge branch 'nar-accessor-tree' of https://github.com/bennofs/nix 2017-05-24 11:04:43 +02:00
Eelco Dolstra b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Benno Fünfstück a1f428b13b nar-accessor.cc: remove unused member NarIndexer::currentName 2017-05-15 19:41:59 +02:00
Benno Fünfstück 5ee06e612a nar-accessor: non-recursive NarMember::find
This avoids a possible stack overflow if directories are very deeply nested.
2017-05-15 19:34:18 +02:00
Eelco Dolstra e80257f122
Simplify fixed-output check 2017-05-15 18:50:54 +02:00
Eelco Dolstra a2d92bb20e
Add --with-sandbox-shell configure flag
And add a 116 KiB ash shell from busybox to the release build. This
helps to make sandbox builds work out of the box on non-NixOS systems
and with diverted stores.
2017-05-15 17:36:32 +02:00
Eelco Dolstra b30f5784d0
Linux sandbox: Don't barf on invalid paths
This is useful when we're using a diverted store (e.g. "--store
local?root=/tmp/nix") in conjunction with a statically-linked sh from
the host store (e.g. "sandbox-paths =/bin/sh=/nix/store/.../bin/busybox").
2017-05-15 17:36:32 +02:00
Benno Fünfstück 75a1d9849d nar-accessor: use tree, fixes readDirectory missing children
Previously, if a directory `foo` existed and a file `foo-` (where `-` is any character that is sorted before `/`), then  `readDirectory` would return an empty list.

To fix this, we now use a tree where we can just access the children of the node, and do not need to rely on sorting behavior to list the contents of a directory.
2017-05-15 10:23:16 +02:00
Eelco Dolstra 510bc1735b
Add an option for extending the user agent header
This is useful e.g. for distinguishing traffic to a binary cache
(e.g. certain machines can use a different tag in the user agent).
2017-05-11 15:10:22 +02:00
Eelco Dolstra 62d476c7ee
Fix typo 2017-05-11 14:02:03 +02:00
Eelco Dolstra ea65ae0f9c
Tweak error message 2017-05-11 13:59:47 +02:00
Eelco Dolstra 6f245bf24a
Change the meaning of info.ultimate
It now means "paths that were built locally". It no longer includes
paths that were added locally. For those we don't need info.ultimate,
since we have the content-addressability assertion (info.ca).
2017-05-11 13:31:23 +02:00
Eelco Dolstra 45d7b1a9e9
LocalStore::addToStore(): Check info.narSize
It allowed the client to specify bogus narSize values. In particular,
Downloader::downloadCached wasn't setting narSize at all.
2017-05-11 13:26:03 +02:00
Eelco Dolstra 7689181e4f
Minor cleanup 2017-05-08 15:56:52 +02:00
Eelco Dolstra 00b286275c
Linux sandbox: Fix compatibility with older kernels 2017-05-08 15:42:59 +02:00
Eelco Dolstra 0a97eb6bd7
Remove superfluous #ifdef 2017-05-08 11:27:20 +02:00
Eelco Dolstra bb50c89319
Make the location of the build directory in the sandbox configurable
This is mostly for use in the sandbox tests, since if the Nix store is
under /build, then we can't use /build as the build directory.
2017-05-05 17:45:22 +02:00