Commit graph

447 commits

Author SHA1 Message Date
Eelco Dolstra 0243eea4b9 * Create a symlink to /nix/var/nix/manifests in /nix/var/nix/gcroots
if it doesn't exist.
2011-07-20 11:47:00 +00:00
Eelco Dolstra e649f3168b * Fix concurrency issues in download-using-manifests' handling of the
SQLite manifest cache.  The DBI AutoCommit feature caused every
  process to have an active transaction at all times, which could
  indefinitely block processes wanting to update the manifest cache.

* Disable fsync() in the manifest cache because we don't need
  integrity (the cache can always be recreated if it gets corrupted).
2011-07-13 14:05:54 +00:00
Eelco Dolstra d94cb02bfe * nix-install-package: unset NIX_REMOTE because $NIX_MANIFESTS_DIR
doesn't work when building through the Nix daemon.  This also
  ensures an error message when the user doesn't have sufficient
  privileges to do nix-pull.
2011-04-19 10:52:35 +00:00
Eelco Dolstra 412914d004 * Read manifests directly into the database, rather than first reading
them into memory.  This brings memory use down to (more or less)
  O(1).  For instance, on my test case, the maximum resident size of
  download-using-manifests while filling the DB went from 142 MiB to
  11 MiB.
2011-04-11 13:16:54 +00:00
Eelco Dolstra 08c8971498 * Lock the database during updates. 2011-04-11 12:40:13 +00:00
Eelco Dolstra 3cbf680f5d * configure: detect whether DBD::SQLite is present. If necessary the
location to DBI and DBD::SQLite can be passed with --with-dbi and
  --with-dbd-sqlite.
2011-04-11 10:13:53 +00:00
Eelco Dolstra 83252b4ca9 * Subtle bug: if you import File::stat in one module, it affects other
modules as well.  So use symbolic field names everywhere (which is
  nicer anyway).
2011-04-11 08:21:30 +00:00
Eelco Dolstra b1882c3ef7 * Create $manifestDir if it doesn't exist. 2011-04-11 08:06:14 +00:00
Eelco Dolstra 5591fcc529 * Cache the manifests in /nix/var/nix/manifests in a SQLite database.
This significantly speeds up the download-using-manifests
  substituter, especially if manifests are very large.  For instance,
  one "nix-build -A geeqie" operation that updated four packages using
  binary patches went from 18.5s to 1.6s.  It also significantly
  reduces memory use.

  The cache is kept in /nix/var/nix/manifests/cache.sqlite.  It's
  updated automatically when manifests are added to or removed from
  /nix/var/nix/manifests.  It might be interesting to have nix-pull
  store manifests directly in the DB, rather than storing them as
  separate flat files, but then we would need a command line interface
  to delete manifests from the DB.
2011-04-10 23:22:46 +00:00
Eelco Dolstra 1e7e4f21ba * Remove the localPaths feature in manifests since it's no longer used
and redundant anyway.
2011-04-06 09:16:22 +00:00
Eelco Dolstra 0423d0692a * Print a better error message. 2011-03-16 15:55:57 +00:00
Eelco Dolstra 802c2651c8 2011-02-17 15:08:33 +00:00
Eelco Dolstra e945b52a31 * nix-push: handle the case where the hash is not set in the DB. 2011-02-17 13:26:33 +00:00
Eelco Dolstra d6c8b995c5 * In Hydra manifests the Size field is missing, so don't rely on it.
This caused a lot of "Use of uninitialized value" warnings from
  Perl.
2010-12-22 10:32:32 +00:00
Eelco Dolstra a0be433fec * Disable X11 forwarding, it's not needed. 2010-12-15 14:25:54 +00:00
Eelco Dolstra f1a6b97639 * nix-copy-closure: make sure that the shell doesn't do globbing of
`+' and `?' in filenames.  This is very slow if /nix/store is very
  large.  (This is a quick hack - a cleaner solution would be to
  bypass the shell entirely.)
2010-12-15 08:39:37 +00:00
Eelco Dolstra d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra 542fc69062 * When doing a query (e.g. `nix-store -r --dry-run'), don't make a lot
of expensive calls to `nix-store --check-validity'.
2010-12-13 08:39:10 +00:00
Eelco Dolstra 7d0444e244 * Bad things happen when a filehandle has the same name as a module. 2010-12-07 12:33:42 +00:00
Eelco Dolstra 365f3028dd * Use CamelCase for the Perl modules. 2010-12-05 17:50:29 +00:00
Eelco Dolstra f42a505ab7 * Add a script `nix-generate-patches'.
* Fix the binary patching test.
2010-12-05 17:36:02 +00:00
Eelco Dolstra d3bba0c2d8 * Move the patch generator into a module. 2010-11-29 15:26:28 +00:00
Eelco Dolstra 9737a7eba0 * Don't generate patches if the system type differs. 2010-11-29 09:26:05 +00:00
Eelco Dolstra bd48fd97f6 * Impose a configurable time limit on patch generation. This is
necessary because bsdiff has some pathological worst-case behaviour,
  e.g. O(n^2) on files consisting only of 0s.
2010-11-29 09:15:45 +00:00
Eelco Dolstra 1a211d812f * Oops. 2010-11-17 17:54:49 +00:00
Eelco Dolstra a4f0365b2d * When checking whether a patch is applicable, for efficiency, use
`nix-store -q --hash' to get the hash of the base path rather than
  `nix-hash'.  However, only do this for estimating the size of a
  download, not for the actual substitution, because sometimes the
  contents of store paths are modified (which they shouldn't, of
  course).
2010-11-17 17:41:59 +00:00
Eelco Dolstra 3d38a49840 * In the download size indication, take binary patches into account.
Hopefully this doesn't slow things down too much.
2010-11-17 17:32:25 +00:00
Eelco Dolstra bdf089f463 * Before a build, show the disk space that the downloaded store paths
will approximately require.
2010-11-17 14:31:42 +00:00
Eelco Dolstra 06699d4219 * Store the NAR size in the manifest. 2010-11-17 12:57:52 +00:00
Eelco Dolstra 5693b8a7e2 * nix-push: no need to compute the NAR hash, since the Nix database
already has it (`nix-store -q --hash').
2010-11-17 12:51:54 +00:00
Eelco Dolstra bfa6ee7d91 * Don't use SSH's `-tt' flag because it doesn't seem to work
on OpenSolaris when using connection sharing.  Instead have
  the remote side check for disconnection and kill the process 
  group when that happens.
2010-10-04 12:30:46 +00:00
Eelco Dolstra bf0dde9597 * Always print hook output on stderr, even if --no-build-output is
set.
* In the build hook, print a trace message to allow Hydra to pick up
  the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
Eelco Dolstra 80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Eelco Dolstra 766f708418 * Experimental feature: allow a derivation to tell the build hook that
it requires a certain feature on the build machine, e.g.

    requiredSystemFeatures = [ "kvm" ];

  We need this in Hydra to make sure that builds that require KVM
  support are forwarded to machines that have KVM support.  Probably
  this should also be enforced for local builds.
2010-08-27 13:18:13 +00:00
Eelco Dolstra e437b08250 * Made the build hook mechanism more efficient. Rather than starting
the hook every time we want to ask whether we can run a remote build
  (which can be very often), we now reuse a hook process for answering
  those queries until it accepts a build.  So if there are N
  derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
Eelco Dolstra 1a396f3789 * Don't call "cat". 2010-08-25 12:19:30 +00:00
Eelco Dolstra 95deba581d * In the build hook, temporarily register the derivation and its
output as GC roots.  This prevents a race if the garbage collector
  is running during the build.
2010-08-25 11:54:11 +00:00
Eelco Dolstra 1e5f5ea2e9 * Correctly handle SSH failing to establish a connection. Because
this didn't work right, the build hook wouldn't fall back to using
  other machines of the desired type.
2010-08-24 14:27:07 +00:00
Eelco Dolstra 5f9aad44ca * nix-build: recognise --cores. 2010-08-11 15:28:02 +00:00
Eelco Dolstra 587dc8aa00 * Sync with the trunk. 2010-08-04 17:48:29 +00:00
Eelco Dolstra 532d766c27 * Don't barf if the source NAR for a patch has disappeared. 2010-07-21 11:30:23 +00:00
Eelco Dolstra 560ab22f7d * Ignore packages that don't have a version.
* Work on a manifest instead of a channel directory.
2010-06-23 21:11:33 +00:00
Eelco Dolstra 87ef5907e9 * Sync. 2010-06-22 14:42:34 +00:00
Eelco Dolstra 819548d92f * Pass `--fallback' to the remote build to ignore failing
substituters.
2010-06-22 14:41:22 +00:00
Eelco Dolstra c778ed1768 * Fix the copy-from-other-stores substituter. 2010-04-26 13:39:55 +00:00
Eelco Dolstra 05fbf61f0e * nix-prefetch-url depends on sed (see
http://hydra.nixos.org/build/311170).
2010-03-05 18:26:47 +00:00
Eelco Dolstra 68e55cd9da * Since Hydra now escapes +', =' and `?' to %NN in URIs,
nix-prefetch-url should unescape them, because `%' is not a valid
  character in store path names.
2010-02-16 00:10:39 +00:00
Eelco Dolstra 719cebcac2 * "Fix" incorrect help message. 2010-02-04 09:38:09 +00:00
Eelco Dolstra 443673620d * Don't use ssh's -f flag since it leads to lots of lingering ssh
processes.
2010-02-04 02:38:40 +00:00
Eelco Dolstra 7ec5a65925 * Doh! The scope of $slotLock should extend to the end of the script,
because otherwise the lock will be released at the end of the while
  loop.
2010-02-04 02:18:29 +00:00
Eelco Dolstra c9b2d80bcd * Typo. 2010-02-04 02:05:34 +00:00
Eelco Dolstra 3cfe65e516 * Doh! Calling `system' in an END block causes the exit status in $?
to be changed to 0.
2010-02-04 02:05:22 +00:00
Eelco Dolstra e51a276907 * Remove the `cat' calls when not using --gzip. 2010-02-04 01:39:23 +00:00
Eelco Dolstra 408913bbaf * Revert r19796 for now. 2010-02-04 00:12:57 +00:00
Eelco Dolstra c45de33c67 * Respect @sysconfdir@. 2010-02-03 21:13:37 +00:00
Eelco Dolstra d0c32dc135 * In the build hook, if connecting to a machine fails, try the other
machines of the right type (if available).  This makes the build
  farm more robust to failures.
2010-02-03 20:35:37 +00:00
Eelco Dolstra f56a039775 * Use SSH connection sharing in the remote build script.
* Removed the Cygwin password hack since the problem is apparently
  fixed in Visual Studio.
2010-02-03 20:12:18 +00:00
Eelco Dolstra bc1e478db1 * nix-copy-closure: start only one SSH connection to the server, or
recycle an already existing connection (using  OpenSSH's connection
  sharing feature).
2010-02-03 15:34:52 +00:00
Eelco Dolstra e839802720 * Don't do a chdir to $tmpDir. It's not necessary, and Windows doesn't
support deleting the current directory.
2010-02-02 15:29:18 +00:00
Eelco Dolstra 50e34891f0 * Disable gzip compression in build-remote.pl because it puts too much
load on the Hydra build farm (where it's unnecessary anyway because
  it has a fast connection to the build machines).  In any case,
  compression can be enabled by using the `-C' option to ssh.
2010-01-25 16:14:45 +00:00
Eelco Dolstra 7ca9972636 * When doing a nix-pull, remove old manifests downloaded from the same
URL.  This prevents lots of old cruft accumulating in
  /nix/var/nix/manifests.
2009-12-09 19:36:54 +00:00
Eelco Dolstra c4c84d1edb * nix-build: be less verbose. 2009-12-09 18:08:28 +00:00
Eelco Dolstra 997db91e07 * Don't pass -K. It should really inherit the setting of the calling
Nix though.
2009-11-17 16:22:39 +00:00
Eelco Dolstra 3392d32e8b * In nix-pull/nix-channel, create the manifests directory if it
doesn't exist.  The Debian packages don't include the manifests
  directory, so nix-channel would silently skip doing a nix-pull,
  resulting in everything being built from source.  Thanks to Juan
  Pedro Bolívar Puente.
2009-11-13 10:08:31 +00:00
Eelco Dolstra e1df4ef73c 2009-09-17 17:02:14 +00:00
Eelco Dolstra 86408b3f47 * build-remote.pl: Pick machines in a round-robin order, rather than
giving jobs to the first machine until it hits its job limit, then
  the second machine and so on.  This should improve utilisation of
  the Hydra build farm a lot.  Also take an optional speed factor
  into account to cause fast machines to be preferred over slower
  machines with a similar load.
2009-09-17 15:48:17 +00:00
Eelco Dolstra 57e0d73c77 * build-remote.pl: allow the system type to be a comma-separated list
of system types.  Don't treat the x86_64-linux system type
  specially.
2009-09-17 13:51:04 +00:00
Eelco Dolstra 20b6f94b65 * nix-build: pass the --show-trace flag. 2009-07-15 09:10:38 +00:00
Eelco Dolstra f24cf5d303 * nix-build: if nix-instantiate or nix-store fails due to a signal
(e.g. out of memory or a segfault), print an error message.
  Otherwise the user doesn't see anything.
2009-06-10 11:30:34 +00:00
Eelco Dolstra c710fe540e * Machines of type x86_64-linux can do i686-linux builds. 2009-04-21 12:42:47 +00:00
Eelco Dolstra 0460ea4c39 * Cleanup. 2009-04-16 09:27:33 +00:00
Eelco Dolstra 096affb55b * Update the mtime on the lock file to make it easy to see when a slot
was last used.
2009-03-29 18:40:44 +00:00
Eelco Dolstra 737423a89c * Use polling to wait for a remote build slot when using a build hook
(that is, call the build hook with a certain interval until it
  accepts the build).
* build-remote.pl was totally broken: for all system types other than
  the local system type, it would send all builds to the *first*
  machine of the appropriate type.
2009-03-29 18:00:45 +00:00
Eelco Dolstra 47706e3924 * nix-copy-closure: compute the closure in one call to nix-store,
which is much faster.
2009-03-28 21:10:29 +00:00
Eelco Dolstra 3a2bbe7f8a * Simplify communication with the hook a bit (don't use file
descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
Eelco Dolstra 7fb548aa26 * Clean up the output a bit. 2009-03-28 16:12:46 +00:00
Eelco Dolstra 7e05b8b75e * Future proofing: assume we can read manifests up to version 10
(which should therefore be backwards compatible).
2009-03-19 10:02:02 +00:00
Eelco Dolstra b88460bcbc * Disregard the Hash field in manifests. 2009-03-19 09:47:34 +00:00
Eelco Dolstra 2897286487 * Unify exportReferencesGraph and exportBuildReferencesGraph, and make
sure that it works as expected when you pass it a derivation.  That
  is, we have to make sure that all build-time dependencies are built,
  and that they are all in the input closure (otherwise remote builds
  might fail, for example).  This is ensured at instantiation time by
  adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
Eelco Dolstra 1dcf208f56 * Clean up some tests (use nix-build where appropriate). 2009-03-18 13:15:55 +00:00
Eelco Dolstra 1273d355ac * nix-install-package: don't pollute /nix/var/nix/manifests. 2009-02-27 14:06:38 +00:00
Eelco Dolstra d4ca5c3952 * Use the regular progress bar; the hash bar isn't very useful when
the size of the download isn't known in advance.
2009-02-27 13:55:11 +00:00
Eelco Dolstra fd2e14b3c8 * nix-build: support --option. 2009-02-27 12:09:30 +00:00
Eelco Dolstra 8ab6bc5a49 * nix-channel: use nix-build. 2009-02-27 11:01:03 +00:00
Eelco Dolstra f052c10eed * Check the manifest version. 2009-02-27 09:53:58 +00:00
Eelco Dolstra a7cee528c5 * Handle base-16 hashes in manifests. 2009-02-26 21:12:35 +00:00
Eelco Dolstra 041717eda3 * download-using-manifests: don't check the cryptographic hash of
downloaded files; rather, check the hash of the unpacked store
  path.

  When the server produces bzipped NAR archives on demand (like Hydra
  does), the hash of the file is not known in advance; it's streamed
  from the server.  Thus the manifest doesn't contain a hash for the
  bzipped NAR archive.  However, the server does know the hash of the
  *uncompressed* NAR archive (the "NarHash" field), since it's stored
  in the Nix database (nix-store -q --hash /nix/store/bla).  So we use
  that instead for checking the integrity of the download.
2009-02-19 23:46:37 +00:00
Michael Raskin 621093cb1c Replace wrong (w.r.t. PATH) sed call with in-shell substitution 2009-02-19 20:46:45 +00:00
Eelco Dolstra 1bb0f1e84b * Build hook: compress the transferred data. 2009-02-03 10:34:15 +00:00
Eelco Dolstra b682fae9d9 * Build hooks: use nix-store --import. This prevents a redundant
scan for runtime dependencies (i.e. the local machine shouldn't do a
  scan that the remote machine has already done).  Also pipe directly
  into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
Eelco Dolstra 019176137f * When using a build hook, distinguish between transient failures
(e.g. an SSH connection problem) and permanent failures (i.e. the
  builder failed).  This matters to Hydra (it wants to know whether it
  makes sense to retry a build).
2009-01-13 11:39:09 +00:00
Eelco Dolstra 4ce692df88 2009-01-13 10:42:13 +00:00
Eelco Dolstra d91dc086bb 2008-12-04 16:51:36 +00:00
Eelco Dolstra 9ac3f5df9c * Propagate --max-silent-time to remote machines. 2008-12-04 16:51:16 +00:00
Eelco Dolstra bcfe98acff * Prefer building on a remote machine over a local machine. This
makes more sense for the build farm, otherwise every nix-store
  invocation will lead to at least one local build.  Will come up with
  a better solution later...
2008-12-04 16:35:47 +00:00
Eelco Dolstra 909fbb9de1 * When using build hooks, for any nix-store -r build operation, it is
necessary that at least one build hook doesn't return "postpone",
  otherwise nix-store will barf ("waiting for a build slot, yet there
  are no running children").  So inform the build hook when this is
  the case, so that it can start a build even when that would exceed
  the maximum load on a machine.
2008-12-04 14:29:41 +00:00
Eelco Dolstra 5dfba0b4db * Force allocation of a pseudo-terminal to clean up the remote
nix-store process when the connection is interrupted.
2008-12-04 13:36:52 +00:00
Eelco Dolstra 9fd9c4c635 * Support multiple system types per remote machine, e.g. a machine
list like

    root@example.org x86_64-linux /root/.ssh/id_buildfarm 1
    root@example.org i686-darwin  /root/.ssh/id_buildfarm 1

  This is possible when the Nix installation on example.org itself has
  remote builds enabled.
2008-12-04 12:20:06 +00:00
Eelco Dolstra f8713e1287 * Dirty hack to make nix-push work properly on derivations: the
derivation should be a source rather than a derivation dependency of
  the call to the NAR derivation.  Otherwise the derivation (and all
  its dependencies) will be built as a side-effect, which may not even
  succeed.
2008-12-04 10:40:41 +00:00
Eelco Dolstra 6bbff48079 2008-11-20 17:22:42 +00:00