Commit graph

1293 commits

Author SHA1 Message Date
Eelco Dolstra 47f587700d
Probably fix a segfault in PathLocks 2016-12-09 13:26:43 +01:00
Eelco Dolstra b30d1e7ada
Don't delete .check directories of running builds
We need to keep them around for diffoscope.
2016-12-08 21:38:58 +01:00
Eelco Dolstra 88ef77226e
Fix warning on 32-bit systems
http://hydra.nixos.org/build/44628517
2016-12-08 20:37:58 +01:00
Eelco Dolstra e629a17cc1
Fix build
http://hydra.nixos.org/build/44628517
2016-12-08 20:36:14 +01:00
Eelco Dolstra fe1162a805
S3BinaryCacheStore: Ensure it only builds on Linux 2016-12-08 15:35:46 +01:00
Eelco Dolstra e6a61b8da7
Fix S3BinaryCacheStore
It failed with

   AWS error uploading ‘6gaxphsyhg66mz0a00qghf9nqf7majs2.ls.xz’: Unable to parse ExceptionName: MissingContentLength Message: You must provide the Content-Length HTTP header.

possibly because the istringstream_nocopy introduced in
0d2ebb4373 doesn't supply the seek
method that the AWS library expects. So bring back the old version,
but only for S3BinaryCacheStore.
2016-12-08 15:31:27 +01:00
Eelco Dolstra 9a313469a4
Add a hook to run diffoscope when non-determinism is detected 2016-12-07 17:57:35 +01:00
Eelco Dolstra b07060688a
Keep track of the exact build start/stop times 2016-12-07 16:09:38 +01:00
Eelco Dolstra dadfddfa7c
Bail out early when non-determinism is detected 2016-12-07 15:31:18 +01:00
Eelco Dolstra 8bdf83f936
Add an option to make non-determinism non-fatal
That is, when build-repeat > 0, and the output of two rounds differ,
then print a warning rather than fail the build. This is primarily to
let Hydra check reproducibility of all packages.
2016-12-07 13:16:06 +01:00
Eelco Dolstra ceeedb58d2
Use a steady clock for timeouts
Fixes #1146.
2016-12-06 21:58:04 +01:00
Eelco Dolstra 7a3e7d0e61
nix-store --serve: Suppress log output on stderr when repeating a build 2016-12-06 17:43:39 +01:00
Eelco Dolstra 215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Shea Levy 8bf378e999 Update darwin build for optional sandbox paths
Fixes #1132
2016-11-17 08:06:32 -05:00
Eelco Dolstra 4be4f6de56
S3BinaryCacheStore:: Eliminate a string copy while uploading
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-16 16:21:30 +01:00
Eelco Dolstra 10ae8fabf1 buildPaths(): Handle ecIncompleteClosure
buildPaths() on a non-derivation would incorrectly not throw an error
if the path didn't have a substitute.
2016-11-14 15:00:17 +01:00
Eelco Dolstra b77fb8acb5 Don't rely on %m 2016-11-14 13:37:16 +01:00
Ludovic Courtès ccb1022022 daemon: Do not error out when deduplication fails due to ENOSPC.
This solves a problem whereby if /gnu/store/.links had enough entries,
ext4's directory index would be full, leading to link(2) returning
ENOSPC.

* nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon
ENOSPC from link(2), print a message and return instead of throwing a
'SysError'.
2016-11-14 13:35:03 +01:00
Eelco Dolstra dd77f7d593 Store::computeFSClosure(): Support a set of paths
This way, callers can exploits the parallelism of computeFSClosure()
when they have multiple paths that they need the (combined) closure of.
2016-11-10 17:45:04 +01:00
Eelco Dolstra e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra 4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra 21c55ab3b5 Implement backwards-compatible RemoteStore::addToStore()
The SSHStore PR adds this functionality to the daemon, but we have to
handle the case where the Nix daemon is 1.11.

Also, don't require signatures for trusted users. This restores 1.11
behaviour.

Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-09 18:45:06 +01:00
Eelco Dolstra a83b10f84c Typo 2016-11-08 20:19:02 +01:00
Eelco Dolstra 18b7363a69 Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.
2016-10-31 17:09:52 +01:00
Shea Levy c4969aebaf Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:49:34 -04:00
Eelco Dolstra 5c0bd51d49 Fix build log output in nix-store --serve 2016-10-26 17:13:38 +02:00
Eelco Dolstra d2c58ba605 Restore the CachedFailure status code
The removal of CachedFailure caused the value of TimedOut to change,
which broke timed-out handling in Hydra (so timed-out builds would
show up as "aborted" and would be retried, e.g. at
http://hydra.nixos.org/build/42537427).
2016-10-26 15:02:38 +02:00
Eelco Dolstra fdbbcc4492 Remove addPathToAccessor 2016-10-21 18:09:30 +02:00
Eelco Dolstra 542ae5c8f8 BinaryCacheStore: Optionally write a NAR listing
The store parameter "write-nar-listing=1" will cause BinaryCacheStore
to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’
added to the binary cache. This file contains an XZ-compressed JSON
file describing the contents of the NAR, excluding the contents of
regular files.

E.g.

  {
    "version": 1,
    "root": {
      "type": "directory",
      "entries": {
        "lib": {
          "type": "directory",
          "entries": {
            "Mcrt1.o": {
              "type": "regular",
              "size": 1288
            },
            "Scrt1.o": {
              "type": "regular",
              "size": 3920
            },
          }
        }
      }
      ...
    }
  }

(The actual file has no indentation.)

This is intended to speed up the NixOS channels programs index
generator [1], since fetching gazillions of large NARs from
cache.nixos.org is currently a bottleneck for updating the regular
(non-small) channel.

[1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-21 16:50:28 +02:00
Eelco Dolstra 307cc8c33d Merge branch 'priorityqueue' of https://github.com/groxxda/nix 2016-10-19 16:37:04 +02:00
Eelco Dolstra 29727ff944 Shut up clang warning 2016-10-19 15:02:48 +02:00
Eelco Dolstra e5346c54df Fix uninitialised variable 2016-10-19 15:02:38 +02:00
Alexander Ried efb938468c downloader: use priority_queue 2016-10-18 15:45:51 +02:00
Eelco Dolstra ae8884b949 Fix /dev/ptmx in sandboxes
This was broken since ff0c0b645c. Since
I can't figure out how to mount a devpts instance in the sandbox,
let's just bind-mount the host devpts.
2016-10-14 15:42:13 +02:00
Eelco Dolstra 2f9d60ed7f CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUM
http://hydra.nixos.org/build/42025230
2016-10-14 14:44:15 +02:00
Eelco Dolstra 5e61b422c5 Fix HTTP/2 support
Commit 86e8c67efc broke it, because
CURL_* are not actually #defines.
2016-10-13 17:25:42 +02:00
Eelco Dolstra fb2dd32100 SSL_CERT_FILE -> NIX_SSL_CERT_FILE
This prevents collisions with the "native" OpenSSL, in particular on
OS X.

Fixes #921.
2016-10-13 17:09:10 +02:00
Eelco Dolstra 816d3e5724 Shut up some warnings 2016-10-12 15:53:38 +02:00
Eelco Dolstra edf9eb8181 querySubstitutablePaths(): Don't query paths for which we already have a substituter 2016-10-07 19:57:47 +02:00
Eelco Dolstra 0c85ef7090 LocalStore::querySubstitutablePaths(): Implement using queryValidPaths() 2016-10-07 19:46:56 +02:00
Eelco Dolstra 844219f364 Store::queryValidPaths(): Use async queryPathInfo()
This allows the binary cache substituter to pipeline requests.
2016-10-07 19:43:36 +02:00
Eelco Dolstra c663b84573 Implement generic Store::queryValidPaths() 2016-10-07 19:20:47 +02:00
Eelco Dolstra 35db4f65a0 Add copyClosure utility function for Hydra 2016-10-07 19:15:25 +02:00
Eelco Dolstra 629ab80022 importPaths(): Fix accessor support for Hydra 2016-10-07 18:13:40 +02:00
Eelco Dolstra ae522f930d Fix getS3Stats() 2016-10-06 17:00:52 +02:00
Eelco Dolstra c6a929986a Use std::random_device 2016-10-05 21:29:18 +02:00
Eelco Dolstra 86e8c67efc Fix build with older versions of libcurl 2016-10-04 14:43:23 +02:00
Eelco Dolstra 4bd51d74af Handle the case where signed-binary-caches consists of whitespace 2016-09-22 15:59:25 +02:00
Eelco Dolstra 818ab58cc6 Add sandbox-dev-shm-size option
Fixes #1069.
2016-09-21 16:54:53 +02:00