Commit graph

1898 commits

Author SHA1 Message Date
Eelco Dolstra 3f3a3ae87b * Acquire a lock on the output path when running a substitute. Also
delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
Eelco Dolstra 72c857f0eb * Ugh, nasty Heisenbug due to an uninitialiased variable. The bug
only caused a crash if the program was *not* invoked with a high
  verbosity level.
2004-06-21 08:51:55 +00:00
Eelco Dolstra be1a917beb * Remove obstructing invalid store paths add[Text]ToStore(). 2004-06-21 07:46:02 +00:00
Eelco Dolstra daf0a923c7 * Wrap calls to registerSubstitute() in a single transaction to
improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
  Theta(N^2) time and log file consumption when adding N substitutes.
  Maybe we can do without it.
2004-06-21 07:38:17 +00:00
Eelco Dolstra 15c60ca1b6 * Disable calls to fsync() since Berkeley DB's DB_TXN_WRITE_NOSYNC
flag doesn't seem to work as advertised.
2004-06-21 07:36:01 +00:00
Eelco Dolstra 112ee89501 * Re-enable support for substitutes in the normaliser.
* A better substitute mechanism.

  Instead of generating a store expression for each store path for
  which we have a substitute, we can have a single store expression
  that builds a generic program that is invoked to build the desired
  store path, which is passed as an argument.

  This means that operations like `nix-pull' only produce O(1) files
  instead of O(N) files in the store when registering N substitutes.
  (It consumes O(N) database storage, of course, but that's not a
  performance problem).

* Added a test for the substitute mechanism.
  
* `nix-store --substitute' reads the substitutes from standard input,
  instead of from the command line.  This prevents us from running
  into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
Eelco Dolstra 85ae781765 * Refactoring. 2004-06-20 13:37:51 +00:00
Eelco Dolstra 23bb902d1f * Re-enable build hooks. 2004-06-19 21:45:04 +00:00
Eelco Dolstra 41ec982f31 * Big refactoring. Move to a much more explicitly state machine based
approach.  This makes it much easier to add extra complexity in the
  normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
Eelco Dolstra 0b70231b9d * Refactoring. 2004-06-15 13:49:42 +00:00
Eelco Dolstra 1bc6afefac * Cleanup. 2004-06-08 13:21:03 +00:00
Eelco Dolstra 5e4a2272bf * Drain the output of the build hook to show error messages. Ugly
hack.
2004-05-18 14:52:35 +00:00
Eelco Dolstra 8e9fd57ef9 * setpgrp() is not POSIX (and on Mac OS X it's different than on
Linux), so use setpgid().
2004-05-18 09:45:18 +00:00
Eelco Dolstra ace8872706 * execl() requires a terminating 0.
* When a fast build wakes up a goal, try to start that goal in the
  same iteration of the startBuild() loop of run().  Otherwise no job
  might be started until the next job terminates.
2004-05-14 12:24:29 +00:00
Eelco Dolstra 4fc00cbec1 * Distributed builds and load balancing now seem to work pretty well.
(Though the `build-remote.pl' script has a gigantic race condition).
2004-05-13 22:52:37 +00:00
Eelco Dolstra 25db622454 * Load balancing. `build-remote.pl' will only execute up to a
per-machine maximum number of parallel jobs on a remote machine.
2004-05-13 19:35:46 +00:00
Eelco Dolstra a8306cb98f * The build hooks used to implement distributed builds can now be run
in parallel.  Hooks are more efficient: locks on output paths are
  only acquired when the hook says that it is willing to accept a
  build job.  Hooks now work in two phases.  First, they should first
  tell Nix whether they are willing to accept a job.  Nix guarantuees
  that no two hooks will ever be in the first phase at the same time
  (this simplifies the implementation of hooks, since they don't have
  to perform locking (?)).  Second, if they accept a job, they are
  then responsible for building it (on the remote system), and copying
  the result back.  These can be run in parallel with other hooks and
  locally executed jobs.

  The implementation is a bit messy right now, though.  

* The directory `distributed' shows a (hacky) example of a hook that
  distributes build jobs over a set of machines listed in a
  configuration file.
2004-05-13 19:14:49 +00:00
Eelco Dolstra efa5fa1a91 * A switch `-j NUMBER' to set the maximum number of parallel jobs (0 =
no limit).
* Add missing file to distribution.
2004-05-12 14:20:32 +00:00
Eelco Dolstra aa5a5084e4 * Pass to the build hook all sorts of information useful for
distributing a build action to another machine.  In particular, the
  paths in the input closures, the output paths, and successor mapping
  for sub-derivations.
2004-05-12 13:32:26 +00:00
Eelco Dolstra 8c0b42f857 * An quick and dirty hack to support distributed builds. 2004-05-12 09:35:51 +00:00
Eelco Dolstra c8d3882cdc * True parallel builds. Nix can now run as many build jobs in
parallel as possible (similar to GNU Make's `-j' switch).  This is
  useful on SMP systems, but it is especially useful for doing builds
  on multiple machines.  The idea is that a large derivation is
  initiated on one master machine, which then distributes
  sub-derivations to any number of slave machines.  This should not
  happen synchronously or in lock-step, so the master must be capable
  of dealing with multiple parallel build jobs.  We now have the
  infrastructure to support this.

  TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
Eelco Dolstra a4d2b22c8c * Be stricter in verifying store paths. 2004-04-14 08:08:55 +00:00
Eelco Dolstra 7823db2137 * Some more nesting. 2004-03-22 21:42:28 +00:00
Eelco Dolstra a5619f1dff * Set the NIX_STORE and NIX_BUILD_TOP environment variables in
builders to point to the store and the temporary build directory,
  respectively.  Useful for purity checking.
* Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make
  sure that tools in the builder store temporary files in the right
  location.
2004-03-12 10:45:08 +00:00
Eelco Dolstra 0dfdafdf6d * Allow linking against an external Berkeley DB / ATerm library. 2004-02-16 16:37:16 +00:00
Eelco Dolstra 76c0e85929 * The environment variable NIX_ROOT can now be set to execute Nix in a
chroot() environment.
* A operation `--validpath' to register path validity.  Useful for
  bootstrapping in a pure Nix environment.
* Safety checks: ensure that files involved in store operations are in
  the store.
2004-02-14 21:44:18 +00:00
Eelco Dolstra 6f5a5ea5ea * Regression fix: realise substitutes and detect cycles. 2004-02-13 10:45:09 +00:00
Eelco Dolstra 447089a5f6 * Catch SIGINT to terminate cleanly when the user tries to interrupt
Nix.  This is to prevent Berkeley DB from becoming wedged.

  Unfortunately it is not possible to throw C++ exceptions from a
  signal handler.  In fact, you can't do much of anything except
  change variables of type `volatile sig_atomic_t'.  So we set an
  interrupt flag in the signal handler and check it at various
  strategic locations in the code (by calling checkInterrupt()).
  Since this is unlikely to cover all cases (e.g., (semi-)infinite
  loops), sometimes SIGTERM may now be required to kill Nix.
2004-01-15 20:23:55 +00:00
Eelco Dolstra ff9af107d3 * Option `-B' to always show the output of builders, regardless of
verbosity level.
2004-01-13 16:35:43 +00:00
Eelco Dolstra 3495d153b3 * Periodically checkpoint the log. 2004-01-13 13:37:25 +00:00
Eelco Dolstra 698e880c9f * Tricky: make sure that the accessor count is not reset to 0 if
recovery fails.
2004-01-13 12:36:43 +00:00
Eelco Dolstra 23fbc72f5d * Print error messages, not debug messages. 2004-01-13 11:53:12 +00:00
Eelco Dolstra 4c4fe7a114 * Changed the extension for store expressions from ".nix" to ".store"
(following the Usenix paper).
2004-01-12 10:44:48 +00:00
Eelco Dolstra 7959354379 * Upgraded to Berkeley DB 4.2.52. The main advantage of 4.2 is that
it automatically removes log files when they are no longer needed.

  *** IMPORTANT ***

  If you have an existing Nix installation, you must checkpoint the
  Nix database to prevent recent transactions from being undone.  Do
  the following:

  - optional: make a backup of $prefix/var/nix/db.

  - run `db_checkpoint' from Berkeley DB 4.1:
  
    $ db_checkpoint -h $prefix/var/nix/db -1

  - optional (?): run `db_recover' from Berkeley DB 4.1:

    $ db_recover -h $prefix/var/nix/db

  - remove $prefix/var/nix/db/log* and $prefix/var/nix/db/__db*
2004-01-07 15:53:14 +00:00
Eelco Dolstra 4a373a3e9a * Implemented Eelco V.'s `nix-env -I' command to specify the default
path of the Nix expression to be used with the import, upgrade, and
  query commands.  For instance,

  $ nix-env -I ~/nixpkgs/pkgs/system/i686-linux.nix

  $ nix-env --query --available   [aka -qa]
  sylpheed-0.9.7
  bison-1.875
  pango-1.2.5
  subversion-0.35.1
  ...

  $ nix-env -i sylpheed

  $ nix-env -u subversion

  There can be only one default at a time.

* If the path to a Nix expression is a symlink, follow the symlink
  prior to resolving relative path references in the expression.
2004-01-05 16:26:43 +00:00
Eelco Dolstra 833f2fc92d * GCC 2.95 compatibility. 2003-12-22 16:40:46 +00:00
Eelco Dolstra 528f1d1867 * Bug fix: parallel builds of the same derivation failed due to lock file removal. 2003-12-21 17:09:16 +00:00
Eelco Dolstra cff6fd22eb * Allow successors that don't exist but have a substitute.
* Integrity: check in successor / substitute registration whether
  the target path exists or has a substitute.
2003-12-05 11:05:19 +00:00
Eelco Dolstra 5d2b424804 * Use a system name that does not include the OS manufacturer (i.e.,
"i686-linux" instead of "i686-suse-linux").
2003-12-01 14:36:50 +00:00
Eelco Dolstra 66c115ef5f * More `make dist' fixes. 2003-11-25 13:01:21 +00:00
Eelco Dolstra c3ee8c9166 * `make dist' fix. 2003-11-25 12:35:52 +00:00
Eelco Dolstra c9cb1fa21f * Bug fix in path invalidation.
* More consistency checks.
2003-11-24 09:24:52 +00:00
Eelco Dolstra ab0bc4999a * Maintain integrity of the substitute and successor mappings when
deleting a path in the store.
* Allow absolute paths in Nix expressions.
* Get nix-prefetch-url to work again.
* Various other fixes.
2003-11-22 18:45:56 +00:00
Eelco Dolstra 40d9eb14df * Fix the garbage collector. 2003-11-22 15:58:34 +00:00
Eelco Dolstra 7a02d95418 * Remove lock files after building. 2003-11-21 16:05:19 +00:00
Eelco Dolstra 9898746ef3 * nix-env: a tool to manage user environments.
* Replace all directory reading code by a generic readDirectory()
  function.
2003-11-19 17:27:16 +00:00
Eelco Dolstra ce92d1bf14 * "Nix expression" -> "store expression".
* More refactoring.
2003-11-18 11:22:29 +00:00
Eelco Dolstra 9f0f020929 * libnix -> libstore. 2003-11-18 10:55:27 +00:00