Commit graph

1637 commits

Author SHA1 Message Date
Eelco Dolstra 155d7c8dfa * Substitutes should occupy a build slot. 2004-06-22 09:00:31 +00:00
Eelco Dolstra c4cb6ea2bc * Refactoring. 2004-06-22 08:50:25 +00:00
Eelco Dolstra 88fb4f6e53 * Missing files added to `make dist'. 2004-06-21 12:20:47 +00:00
Eelco Dolstra 2db9748221 * Remove debug output. 2004-06-21 10:01:17 +00:00
Eelco Dolstra 37ee6cef99 * Adapted nix-pull to use the new substitute mechanism. 2004-06-21 09:51:23 +00:00
Eelco Dolstra 3f3a3ae87b * Acquire a lock on the output path when running a substitute. Also
delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
Eelco Dolstra 72c857f0eb * Ugh, nasty Heisenbug due to an uninitialiased variable. The bug
only caused a crash if the program was *not* invoked with a high
  verbosity level.
2004-06-21 08:51:55 +00:00
Eelco Dolstra be1a917beb * Remove obstructing invalid store paths add[Text]ToStore(). 2004-06-21 07:46:02 +00:00
Eelco Dolstra daf0a923c7 * Wrap calls to registerSubstitute() in a single transaction to
improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
  Theta(N^2) time and log file consumption when adding N substitutes.
  Maybe we can do without it.
2004-06-21 07:38:17 +00:00
Eelco Dolstra 15c60ca1b6 * Disable calls to fsync() since Berkeley DB's DB_TXN_WRITE_NOSYNC
flag doesn't seem to work as advertised.
2004-06-21 07:36:01 +00:00
Eelco Dolstra 112ee89501 * Re-enable support for substitutes in the normaliser.
* A better substitute mechanism.

  Instead of generating a store expression for each store path for
  which we have a substitute, we can have a single store expression
  that builds a generic program that is invoked to build the desired
  store path, which is passed as an argument.

  This means that operations like `nix-pull' only produce O(1) files
  instead of O(N) files in the store when registering N substitutes.
  (It consumes O(N) database storage, of course, but that's not a
  performance problem).

* Added a test for the substitute mechanism.
  
* `nix-store --substitute' reads the substitutes from standard input,
  instead of from the command line.  This prevents us from running
  into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
Eelco Dolstra bafb2357d1 * README for the Emacs mode (written a while ago, but forgot to commit). 2004-06-20 19:08:59 +00:00
Eelco Dolstra 85ae781765 * Refactoring. 2004-06-20 13:37:51 +00:00
Eelco Dolstra 23bb902d1f * Re-enable build hooks. 2004-06-19 21:45:04 +00:00
Eelco Dolstra 41ec982f31 * Big refactoring. Move to a much more explicitly state machine based
approach.  This makes it much easier to add extra complexity in the
  normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
Eelco Dolstra 3454c685ee * This is also useful. 2004-06-18 16:52:31 +00:00
Eelco Dolstra 6ba26f27c3 * Shared (garbage collecting) pointers. Copied from Boost. 2004-06-18 09:20:51 +00:00
Eelco Dolstra 0b70231b9d * Refactoring. 2004-06-15 13:49:42 +00:00
Eelco Dolstra 1bc6afefac * Cleanup. 2004-06-08 13:21:03 +00:00
Eelco Dolstra e8411948ff * A Nix mode for Emacs. 2004-06-04 14:31:57 +00:00
Eelco Dolstra 5e4a2272bf * Drain the output of the build hook to show error messages. Ugly
hack.
2004-05-18 14:52:35 +00:00
Eelco Dolstra 19479899fb * Don't set the rpath here --- it's not portable. 2004-05-18 12:57:26 +00:00
Eelco Dolstra 1d08093b48 * Go back to the old way of generating the system name, and allow it
to be specified in configure (using `--with-system=SYSTEM').
2004-05-18 09:45:46 +00:00
Eelco Dolstra 8e9fd57ef9 * setpgrp() is not POSIX (and on Mac OS X it's different than on
Linux), so use setpgid().
2004-05-18 09:45:18 +00:00
Eelco Dolstra ace8872706 * execl() requires a terminating 0.
* When a fast build wakes up a goal, try to start that goal in the
  same iteration of the startBuild() loop of run().  Otherwise no job
  might be started until the next job terminates.
2004-05-14 12:24:29 +00:00
Eelco Dolstra 4fc00cbec1 * Distributed builds and load balancing now seem to work pretty well.
(Though the `build-remote.pl' script has a gigantic race condition).
2004-05-13 22:52:37 +00:00
Eelco Dolstra 2fa3304933 * Set the executable bit. 2004-05-13 20:28:20 +00:00
Eelco Dolstra 25db622454 * Load balancing. `build-remote.pl' will only execute up to a
per-machine maximum number of parallel jobs on a remote machine.
2004-05-13 19:35:46 +00:00
Eelco Dolstra 3426d19547 * Perform all tests. 2004-05-13 19:16:48 +00:00
Eelco Dolstra a8306cb98f * The build hooks used to implement distributed builds can now be run
in parallel.  Hooks are more efficient: locks on output paths are
  only acquired when the hook says that it is willing to accept a
  build job.  Hooks now work in two phases.  First, they should first
  tell Nix whether they are willing to accept a job.  Nix guarantuees
  that no two hooks will ever be in the first phase at the same time
  (this simplifies the implementation of hooks, since they don't have
  to perform locking (?)).  Second, if they accept a job, they are
  then responsible for building it (on the remote system), and copying
  the result back.  These can be run in parallel with other hooks and
  locally executed jobs.

  The implementation is a bit messy right now, though.  

* The directory `distributed' shows a (hacky) example of a hook that
  distributes build jobs over a set of machines listed in a
  configuration file.
2004-05-13 19:14:49 +00:00
Eelco Dolstra 5087c8f645 * Use `-j0'. 2004-05-12 14:30:57 +00:00
Eelco Dolstra efa5fa1a91 * A switch `-j NUMBER' to set the maximum number of parallel jobs (0 =
no limit).
* Add missing file to distribution.
2004-05-12 14:20:32 +00:00
Eelco Dolstra 1f48aa0be7 * Broken test. 2004-05-12 13:49:10 +00:00
Eelco Dolstra aa5a5084e4 * Pass to the build hook all sorts of information useful for
distributing a build action to another machine.  In particular, the
  paths in the input closures, the output paths, and successor mapping
  for sub-derivations.
2004-05-12 13:32:26 +00:00
Eelco Dolstra 8c0b42f857 * An quick and dirty hack to support distributed builds. 2004-05-12 09:35:51 +00:00
Eelco Dolstra c8d3882cdc * True parallel builds. Nix can now run as many build jobs in
parallel as possible (similar to GNU Make's `-j' switch).  This is
  useful on SMP systems, but it is especially useful for doing builds
  on multiple machines.  The idea is that a large derivation is
  initiated on one master machine, which then distributes
  sub-derivations to any number of slave machines.  This should not
  happen synchronously or in lock-step, so the master must be capable
  of dealing with multiple parallel build jobs.  We now have the
  infrastructure to support this.

  TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
Eelco Dolstra aea436503e * Ignore interrupt signals while handling an exception.
* Ignore EINTR in reads and writes.
2004-05-11 13:48:25 +00:00
Eelco Dolstra a9858c9f26 * A test to verify that Nix executes build jobs in parallel, if
possible.

  This test fails right now because this hasn't been implemented right
  now.  Yes, I'm doing Test-Driven Development! ;-)
2004-05-04 17:04:17 +00:00
Eelco Dolstra f044ccf702 * 1000th revision!
* A test to verify that locking of output paths (caused by concurrent
  invocations of Nix) works correctly.
2004-05-04 13:56:30 +00:00
Eelco Dolstra a7bbe73971 * Another test. 2004-05-04 13:22:33 +00:00
Eelco Dolstra ef093aac8f * Grrr. TESTS are not included in EXTRA_DIST. 2004-05-04 12:45:04 +00:00
Eelco Dolstra 256eeab711 * Allow the location of the store etc. to be specified using
environment variables.
* Started adding some automatic tests.
* Do a `make check' when building RPMs.
2004-05-04 12:15:30 +00:00
Eelco Dolstra fd927c5d25 * Bump the version number. 2004-04-26 13:44:26 +00:00
Eelco Dolstra 22371cbd3f * Fixed URL. 2004-04-26 09:54:37 +00:00
Eelco Dolstra bcce9c1ff5 * Only add `-preRELEASE' to the version if STABLE != 1.
* Documented release procedures.
2004-04-26 09:52:06 +00:00
Eelco Dolstra d4779abc04 * Pass SYSTEM through config.h, and allow spaces. 2004-04-23 15:16:08 +00:00
Eelco Dolstra 759c953196 * Look for GC roots in @localstatedir@/nix/gcroots. 2004-04-22 14:17:57 +00:00
Eelco Dolstra d7238bc84e * Don't create $(localstatedir)/nix/profiles if --disable-init-state
is specified.
2004-04-22 07:47:41 +00:00
Eelco Dolstra b6df68c942 * Dist error. 2004-04-21 14:57:46 +00:00
Eelco Dolstra 21655a70f5 * Channels. These allow you to stay current with an evolving set of
Nix expressions.

  To subscribe to a channel (needs to be done only once):

    nix-channel --add \
      http://catamaran.labs.cs.uu.nl/dist/nix/channels/nixpkgs-unstable

  This just adds the given URL to ~/.nix-channels (which can also be
  edited manually).

  To update from all channels:

    nix-channel --update

  This fetches the latest expressions and pulls cache manifests.  The
  default Nix expression (~/.nix-defexpr) is made to point to the
  conjunction of the expressions downloaded from all channels.

  So to update all installed derivations in the current user
  environment:

    nix-channel --update
    nix-env --upgrade '*'

  If you are really courageous, you can put this in a cronjob or
  something.

  You can subscribe to multiple channels.  It is not entirely clear
  what happens when there are name clashes between derivations from
  different channels.  From nix-env/main.cc it appears that the one
  with the lowest (highest?) hash will be used, which is pretty
  meaningless.
2004-04-21 14:54:05 +00:00