Commit graph

7740 commits

Author SHA1 Message Date
Eelco Dolstra 8bdf83f936
Add an option to make non-determinism non-fatal
That is, when build-repeat > 0, and the output of two rounds differ,
then print a warning rather than fail the build. This is primarily to
let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Eelco Dolstra ceeedb58d2
Use a steady clock for timeouts
Fixes #1146.
2016-12-06 21:58:04 +01:00
Eelco Dolstra 7a3e7d0e61
nix-store --serve: Suppress log output on stderr when repeating a build 2016-12-06 17:43:39 +01:00
Eelco Dolstra 6069b946ad
nix-store --serve: Support setting build-repeat
This allows Hydra to test whether builds are reproducible.
2016-12-06 17:19:22 +01:00
Eelco Dolstra 603f08506e
Tweak error message 2016-12-06 17:18:40 +01:00
Eelco Dolstra 09191caea8
Add shell.nix 2016-12-06 17:17:29 +01:00
Eelco Dolstra d1da6967b8
Drop unused WWW::Curl dependency 2016-12-06 17:17:29 +01:00
Eelco Dolstra ae71895f55 Merge pull request #947 from layus/fix-path-slash
Improve error message on trailing path slashes
2016-12-06 15:21:17 +01:00
Guillaume Maudoux e4b82af387 Improve error message on trailing path slashes 2016-11-27 17:48:46 +01:00
Eelco Dolstra 215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Jörg Thalheim 855abd85d8
Simplify remouting with MS_PRIVATE in sandbox build
also fix race condition if mounts are added after mountinfo is read.
2016-11-25 00:15:39 +01:00
Eelco Dolstra 7ee43df862
nix-channel: Fix --update <CHANNELS>
This unbreaks "nixos-rebuild --upgrade".
2016-11-21 15:54:19 +01:00
Eelco Dolstra 349e988903
Typo 2016-11-17 17:55:41 +01:00
Shea Levy 00b8bce4d0 Fix binary-cache-store build 2016-11-17 11:48:10 -05:00
Shea Levy 0d2ebb4373 istringstream_nocopy: Implement in a standards-compliant way
Fixes #1135.
2016-11-17 10:10:32 -05:00
Shea Levy 3f4d3f8a1a nix-daemon: Fix splice faking on non-Linux 2016-11-17 08:10:12 -05:00
Shea Levy 8bf378e999 Update darwin build for optional sandbox paths
Fixes #1132
2016-11-17 08:06:32 -05:00
aszlig 4e1a2cd537
seccomp: Forge return values for *chown32
These syscalls are only available in 32bit architectures, but libseccomp
should handle them correctly even if we're on native architectures that
do not have these syscalls.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:36 +01:00
aszlig ed64976cec
seccomp: Forge return codes for POSIX ACL syscalls
Commands such as "cp -p" also use fsetxattr() in addition to fchown(),
so we need to make sure these syscalls always return successful as well
in order to avoid nasty "Invalid value" errors.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 17:29:14 +01:00
aszlig 651a18dd24
release.nix: Add a test for sandboxing
Right now it only tests whether seccomp correctly forges the return
value of chown, but the long-term goal is to test the full sandboxing
functionality at some point in the future.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:34 +01:00
aszlig b90a435332
libstore/build: Forge chown() to return success
What we basically want is a seccomp mode 2 BPF program like this but for
every architecture:

  BPF_STMT(BPF_LD+BPF_W+BPF_ABS, offsetof(struct seccomp_data, nr)),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_chown, 4, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchown, 3, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_fchownat, 2, 0),
  BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, __NR_lchown, 1, 0),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ALLOW),
  BPF_STMT(BPF_RET+BPF_K, SECCOMP_RET_ERRNO)

However, on 32 bit architectures we do have chown32, lchown32 and
fchown32, so we'd need to add all the architecture blurb which
libseccomp handles for us.

So we only need to make sure that we add the 32bit seccomp arch while
we're on x86_64 and otherwise we just stay at the native architecture
which was set during seccomp_init(), which more or less replicates
setting 32bit personality during runChild().

The FORCE_SUCCESS() macro here could be a bit less ugly but I think
repeating the seccomp_rule_add() all over the place is way uglier.

Another way would have been to create a vector of syscalls to iterate
over, but that would make error messages uglier because we can either
only print the (libseccomp-internal) syscall number or use
seccomp_syscall_resolve_num_arch() to get the name or even make the
vector a pair number/name, essentially duplicating everything again.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:30 +01:00
aszlig 1c52e344c4
Add build dependency for libseccomp
We're going to use libseccomp instead of creating the raw BPF program,
because we have different syscall numbers on different architectures.

Although our initial seccomp rules will be quite small it really doesn't
make sense to generate the raw BPF program because we need to duplicate
it and/or make branches on every single architecture we want to suuport.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:26 +01:00
aszlig e8838713df
Run builds as root in user namespace again
This reverts commit ff0c0b645c.

We're going to use seccomp to allow "cp -p" and force chown-related
syscalls to always return 0.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-11-16 16:48:08 +01:00
Eelco Dolstra 4be4f6de56
S3BinaryCacheStore:: Eliminate a string copy while uploading
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-16 16:21:30 +01:00
Eelco Dolstra 10ae8fabf1 buildPaths(): Handle ecIncompleteClosure
buildPaths() on a non-derivation would incorrectly not throw an error
if the path didn't have a substitute.
2016-11-14 15:00:17 +01:00
Eelco Dolstra b77fb8acb5 Don't rely on %m 2016-11-14 13:37:16 +01:00
Ludovic Courtès ccb1022022 daemon: Do not error out when deduplication fails due to ENOSPC.
This solves a problem whereby if /gnu/store/.links had enough entries,
ext4's directory index would be full, leading to link(2) returning
ENOSPC.

* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
ENOSPC from link(2), print a message and return instead of throwing a
'SysError'.
2016-11-14 13:35:03 +01:00
Eelco Dolstra b8d9616af1 Merge pull request #1129 from layus/fix-comments
Fix comments parsing
2016-11-14 11:06:30 +01:00
Guillaume Maudoux a5e761dddb Fix comments parsing
Fixed the parsing of multiline strings ending with an even number of
stars, like /** this **/.
Added test cases for comments.
2016-11-13 17:20:34 +01:00
Eelco Dolstra dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Shea Levy bff3ad767e build-remote: replace strtoull with stoull to take advantage of C++ error handling 2016-11-10 11:09:15 -05:00
Shea Levy 167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Eelco Dolstra 2af5d35fdc Merge branch 'master' of github.com:NixOS/nix 2016-11-09 19:09:44 +01:00
Eelco Dolstra e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra 4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra 21c55ab3b5 Implement backwards-compatible RemoteStore::addToStore()
The SSHStore PR adds this functionality to the daemon, but we have to
handle the case where the Nix daemon is 1.11.

Also, don't require signatures for trusted users. This restores 1.11
behaviour.

Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-09 18:45:06 +01:00
Eelco Dolstra a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra b3ba762dbf Add missing #include 2016-11-07 14:35:47 +01:00
Vladimír Čunát 818aad3ec4
Detect and disallow base32 hash overflow
Example (before this commit):
$ nix-hash --type sha256 --to-base16 4n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0 \
    | xargs nix-hash --type sha256 --to-base32
0n0igfxbd3kqvvj2k2xgysrp63l4v2gd110fwkk4apfpm0hvzwh0

It's a real-life example:
https://github.com/NixOS/nixpkgs/pull/20208/files#r86695567
2016-11-06 22:17:37 +01:00
Manav Rathi eec5409a69 installation: allow profile modification to be skipped (#1072)
The current behaviour modifies the first writeable file from amongst
.bash_profile, .bash_login and .profile.  So .bash_profile (if it is
writable) would be modified even if a user has already sourced nix.sh
in, say, .profile.

This commit introduces a new environment variable,
NIX_INSTALLER_NO_MODIFY_PROFILE.  If this is set during installation,
then the modifications are unconditionally skipped.

This is useful for users who have a manually curated set of dotfiles
that they are porting to a new machine. In such scenarios, nix.sh is
already sourced at a place where the user prefers.  Without this
change, the nix installer would insist on modifying .bash_profile if
it exists.

This commit also add documentations for both the current behaviour and
the new override.
2016-11-03 18:02:29 +01:00
Eelco Dolstra 18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Shea Levy c4969aebaf Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:49:34 -04:00
Eelco Dolstra e02a1352c1 Merge pull request #1108 from dezgeg/fix-sigfpe
Fix SIGFPE from integer overflow during division
2016-10-26 17:56:14 +02:00
Tuomas Tynkkynen f0fc3dd88b Fix SIGFPE from integer overflow during division
On some architectures (like x86_64 or i686, but not ARM for example)
overflow during integer division causes a crash due to SIGFPE.
Reproduces on a 64-bit system with:

    nix-instantiate --eval -E '(-9223372036854775807 - 1) / -1'

The only way this can happen is when the smallest possible integer is
divided by -1, so just special-case that.
2016-10-26 18:41:59 +03:00
Eelco Dolstra 5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00
Eelco Dolstra d2c58ba605 Restore the CachedFailure status code
The removal of CachedFailure caused the value of TimedOut to change,
which broke timed-out handling in Hydra (so timed-out builds would
show up as "aborted" and would be retried, e.g. at
http://hydra.nixos.org/build/42537427).
2016-10-26 15:02:38 +02:00
Eelco Dolstra fdbbcc4492 Remove addPathToAccessor 2016-10-21 18:09:30 +02:00
Eelco Dolstra 542ae5c8f8 BinaryCacheStore: Optionally write a NAR listing
The store parameter "write-nar-listing=1" will cause BinaryCacheStore
to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’
added to the binary cache. This file contains an XZ-compressed JSON
file describing the contents of the NAR, excluding the contents of
regular files.

E.g.

  {
    "version": 1,
    "root": {
      "type": "directory",
      "entries": {
        "lib": {
          "type": "directory",
          "entries": {
            "Mcrt1.o": {
              "type": "regular",
              "size": 1288
            },
            "Scrt1.o": {
              "type": "regular",
              "size": 3920
            },
          }
        }
      }
      ...
    }
  }

(The actual file has no indentation.)

This is intended to speed up the NixOS channels programs index
generator [1], since fetching gazillions of large NARs from
cache.nixos.org is currently a bottleneck for updating the regular
(non-small) channel.

[1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-21 16:50:28 +02:00
Eelco Dolstra 307cc8c33d Merge branch 'priorityqueue' of https://github.com/groxxda/nix 2016-10-19 16:37:04 +02:00