Commit graph

2945 commits

Author SHA1 Message Date
Eelco Dolstra bf0dde9597 * Always print hook output on stderr, even if --no-build-output is
set.
* In the build hook, print a trace message to allow Hydra to pick up
  the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
Eelco Dolstra e2e168f7c2 `nix-store --verify' improvements:
* If a path has disappeared, check its referrers first, and don't try
  to invalidate paths that have valid referrers.  Otherwise we get a
  foreign key constraint violation.
* Read the whole Nix store directory instead of statting each valid
  path, which is slower.
* Acquire the global GC lock.
2010-08-31 11:47:31 +00:00
Eelco Dolstra 80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Eelco Dolstra 20acd43c25 * Disable the GC reachability check for now (when creating new roots),
as it's hopelessly inefficient.
2010-08-30 14:11:57 +00:00
Eelco Dolstra 766f708418 * Experimental feature: allow a derivation to tell the build hook that
it requires a certain feature on the build machine, e.g.

    requiredSystemFeatures = [ "kvm" ];

  We need this in Hydra to make sure that builds that require KVM
  support are forwarded to machines that have KVM support.  Probably
  this should also be enforced for local builds.
2010-08-27 13:18:13 +00:00
Eelco Dolstra df50916e46 * Oops - "null" was displayed as "true". 2010-08-27 12:10:56 +00:00
Eelco Dolstra e41ecbf730 2010-08-27 11:09:04 +00:00
Eelco Dolstra e437b08250 * Made the build hook mechanism more efficient. Rather than starting
the hook every time we want to ask whether we can run a remote build
  (which can be very often), we now reuse a hook process for answering
  those queries until it accepts a build.  So if there are N
  derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
Eelco Dolstra 1a396f3789 * Don't call "cat". 2010-08-25 12:19:30 +00:00
Eelco Dolstra 95deba581d * In the build hook, temporarily register the derivation and its
output as GC roots.  This prevents a race if the garbage collector
  is running during the build.
2010-08-25 11:54:11 +00:00
Eelco Dolstra 1e5f5ea2e9 * Correctly handle SSH failing to establish a connection. Because
this didn't work right, the build hook wouldn't fall back to using
  other machines of the desired type.
2010-08-24 14:27:07 +00:00
Eelco Dolstra 034f608e00 * Don't complain if the stored hash of a path is zero (unknown). 2010-08-24 14:25:33 +00:00
Eelco Dolstra f58f51f380 * Handle the unlikely case where a derivation has no dependencies at
all.
2010-08-24 11:45:44 +00:00
Eelco Dolstra 955d11aae7 * Sync with the trunk. 2010-08-24 09:56:28 +00:00
Eelco Dolstra c67eccc26d * nix-build: pass --cores. 2010-08-24 09:21:59 +00:00
Eelco Dolstra 2c8e070e5d * Bump the version number. 2010-08-17 15:39:35 +00:00
Eelco Dolstra ed133e6e64 * rpmBuild already includes the disk image in the output name. 2010-08-17 14:08:44 +00:00
Eelco Dolstra 2de17f4edc * Update date. 2010-08-17 10:06:26 +00:00
Eelco Dolstra 86f65edf4e * Document --cores in the manual. 2010-08-17 07:22:05 +00:00
Eelco Dolstra b75e1043a3 * Typo. 2010-08-16 13:23:45 +00:00
Eelco Dolstra 8ec6594d6d * Remove the "tarball" jobset argument. 2010-08-16 13:01:31 +00:00
Eelco Dolstra 12721a3a9a * Nix 0.16 release notes. 2010-08-16 12:38:32 +00:00
Eelco Dolstra 5fb824e896 * Urgh, this was supposed to go in the trunk... 2010-08-12 13:36:56 +00:00
Eelco Dolstra 5c5ab2bc12 * Don't link against pthreads. This was added way back in r211
because Berkeley DB needed it on some platforms, but we don't use
  BDB anymore.

  On FreeBSD, if you link against pthreads, then the main thread gets
  a 2 MB stack which cannot be overriden (it ignores "ulimit -s"):

    http://www.mail-archive.com/freebsd-hackers@freebsd.org/msg62445.html
  
  This is not enough for Nix.  For instance, the garbage collector can
  fail if there is a pathologically deep chain of references
  (http://hydra.nixos.org/build/556199).  2 MB is also not enough for
  many Nix expressions.
  
  Arguably the garbage collector shouldn't use recursion, because in
  NixOS unprivileged users can DOS the garbage collector by creating a
  sufficiently deeply nested chain of references.  But getting rid of
  recursion is a bit harder.
2010-08-12 13:34:34 +00:00
Eelco Dolstra 6846ed8b44 * Make --cores work when building through the Nix daemon. 2010-08-12 09:21:50 +00:00
Eelco Dolstra 5f9aad44ca * nix-build: recognise --cores. 2010-08-11 15:28:02 +00:00
Eelco Dolstra d7875d1648 * Use SQLite 3.7.0.1. 2010-08-06 07:51:27 +00:00
Eelco Dolstra 587dc8aa00 * Sync with the trunk. 2010-08-04 17:48:29 +00:00
Eelco Dolstra fd9c77dfc7 * Use SQLite 3.7.0's write-ahead logging (WAL mode). This is a lot
faster than the old mode when fsyncs are enabled, because it only
  performs an fsync() when doing a checkpoint, rather than at every
  commit.  Some timings for doing a "nix-instantiate /etc/nixos/nixos
  -A system" after modifying the stdenv setup script:

    42.5s - SQLite 3.6.23 with truncate mode and fsync
     3.4s - SQLite 3.6.23 with truncate mode and no fsync
    32.1s - SQLite 3.7.0 with truncate mode and fsync
    16.8s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
            every 1000 pages
     8.3s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
            every 8192 pages
     1.7s - SQLite 3.7.0 with WAL mode and no fsync

  The default is now to use WAL mode with fsyncs.  Because WAL doesn't
  work on remote filesystems such as NFS (as it uses shared memory),
  truncate mode can be re-enabled by setting the "use-sqlite-wal"
  option to false.
2010-08-04 17:35:59 +00:00
Eelco Dolstra 750be19ae8 * Remove "auto" and "guess" as synonyms for 0 in the handling of
build-cores and --cores.  They're superfluous and just complicate
  the parsing.
2010-08-04 12:23:59 +00:00
Eelco Dolstra 7f893b7a43 * Allow derivations to hint that they should not be built remotely
using the build hook mechanism, by setting the derivation attribute
  "preferLocalBuild" to true.  This has a few use cases:

  - The user environment builder.  Since it just creates a bunch of
    symlinks without much computation, there is no reason to do it
    remotely.  In fact, doing it remotely requires the entire closure
    of the user environment to be copied to the remote machine, which
    is extremely wasteful.

  - `fetchurl'.  Performing the download on a remote machine and then
    copying it to the local machine involves twice as much network
    traffic as performing the download locally, and doesn't save any
    CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
Eelco Dolstra 315d8fbd75 * Set the default system filter to "*". This ensures that (for
instance) "nix-env -i wine" works on x86_64-linux, even though Wine
  is built on i686-linux.  In the event that there are multiple
  matching derivations, prefer those built for the current system.
2010-08-04 09:32:42 +00:00
Eelco Dolstra 6d6200f37a * Optimisation in the // operator: if one of the sets is empty, return
the other set.
2010-08-02 16:31:05 +00:00
Eelco Dolstra 7af6a2fd71 * intersectAttrs: optimise for the case where the second set is larger
than the first set.  (That's usually the case with callPackage.)
2010-08-02 11:54:44 +00:00
Eelco Dolstra 532d766c27 * Don't barf if the source NAR for a patch has disappeared. 2010-07-21 11:30:23 +00:00
Peter Simons 7e043d28a6 src/bsdiff-4.3/Makefile.am: include the 'compat-include' directory in distribution tarballs 2010-07-15 14:35:20 +00:00
Peter Simons 60b632b173 tests/build-hook.hook.sh: prefer more portable ... syntax over $(...) for running sub-shells
The /bin/sh interpreter on Solaris doesn't understand $(...) syntax for running
sub-shells. Consequently, this test fails on Solaris. To remedy the situation,
the script either needs to be run by /bin/bash -- which is non-standard --, or
it needs to use the ancient but portable `...` syntax.
2010-06-25 14:05:37 +00:00
Peter Simons a0d29040f7 Revert "configure.ac: make flex and bison required programs"
This reverts commit 22405. Apparently, these programs aren't necessarily
required when building from a release archive.
2010-06-24 22:22:24 +00:00
Peter Simons af09fe12dd Fix build of bsdiff-4.3 on machines that don't have <err.h>, such as Solaris. 2010-06-24 17:51:31 +00:00
Peter Simons d63375d529 configure.ac: Incredibly enough, tr(1) on Solaris doesn't understand A-Z syntax for ranges. 2010-06-24 17:51:24 +00:00
Peter Simons 4c21c016c5 configure.ac: make flex and bison required programs
The build fails if these tools aren't available.
2010-06-24 17:51:19 +00:00
Peter Simons bcec46057c src/libutil/util.cc: include <limit.h> to ensure that PATH_MAX is defined 2010-06-24 17:51:13 +00:00
Peter Simons a17071fef1 Include <cstring> to ensure that strcpy(), strlen(), and memset() are declared.
An "using namespace std" was added locally in those functions that refer to
names from <cstring>. That is not pretty, but it's a very portable solution,
because strcpy() and friends will be found in both the 'std' and in the global
namespace.
2010-06-24 17:51:04 +00:00
Eelco Dolstra 560ab22f7d * Ignore packages that don't have a version.
* Work on a manifest instead of a channel directory.
2010-06-23 21:11:33 +00:00
Peter Simons 8b7f8b56f1 Added support for passing an (impure) NIX_BUILD_CORES variable to build expressions.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.

The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
2010-06-23 14:34:08 +00:00
Eelco Dolstra 87ef5907e9 * Sync. 2010-06-22 14:42:34 +00:00
Eelco Dolstra 819548d92f * Pass `--fallback' to the remote build to ignore failing
substituters.
2010-06-22 14:41:22 +00:00
Eelco Dolstra 3e5e0faf9c * Okay, putting a lock on the temporary directory used by importPath()
doesn't work because the garbage collector doesn't actually look at
  locks.  So r22253 was stupid.  Use addTempRoot() instead.  Also,
  locking the temporary directory in exportPath() was silly because it
  isn't even in the store.
2010-06-21 11:08:09 +00:00
Eelco Dolstra bf87cc44b4 * Sync with the trunk. 2010-06-21 07:55:38 +00:00
Eelco Dolstra b57189174f * In importPath() and exportPath(), lock the temporary directory to
prevent it from being deleted by the garbage collector.
2010-06-14 08:34:48 +00:00