Commit graph

1429 commits

Author SHA1 Message Date
Eelco Dolstra 462bd50aef * Use normal (rather than full) synchronous mode, which I gather from
the description at http://www.sqlite.org/atomiccommit.html should be
  safe enough.
2010-02-24 10:57:57 +00:00
Eelco Dolstra 63b09c5e41 2010-02-23 22:31:38 +00:00
Eelco Dolstra b4e6d98fc3 * configure: flag --with-sqlite. 2010-02-23 22:12:46 +00:00
Eelco Dolstra 2b20318b0e 2010-02-22 14:24:37 +00:00
Eelco Dolstra 9cda616949 * The database needs a trigger to get rid of self-references to
prevent a foreign key constraint violation on the Refs table when
  deleting a path.
2010-02-22 14:18:55 +00:00
Eelco Dolstra c4d388add4 * Get derivation outputs from the database instead of the .drv file,
which requires more I/O.
2010-02-22 12:44:36 +00:00
Eelco Dolstra 103cfee056 * Revert r19650 (implement gc-keep-outputs by looking for derivations
with the same name as the output) and instead use the
  DerivationOutputs table in the database, which is the correct way to
  to do things.
2010-02-22 11:44:17 +00:00
Eelco Dolstra 299ff64812 * Put the derivation outputs in the database. This is useful for the
garbage collector.
2010-02-22 11:15:50 +00:00
Eelco Dolstra 1930570ad9 * Foreign key support in SQLite is not a persistent setting, so enable
it at startup.
* Implement negative caching.  Now `make check' passes.
2010-02-19 17:15:22 +00:00
Eelco Dolstra 9c9a88e9e2 * Implement more stuff. 2010-02-19 16:43:25 +00:00
Eelco Dolstra 762cee72cc * Implement registerValidPath(). 2010-02-19 16:04:51 +00:00
Eelco Dolstra 268f9aaf28 * Implemented queryValidPaths() and verifyStore(). 2010-02-18 16:51:27 +00:00
Eelco Dolstra 836e5b6f57 * Implemented queryReferrers(). 2010-02-18 16:21:59 +00:00
Eelco Dolstra 77cb9e3fb1 * Implement queryPathInfo(). 2010-02-18 15:52:57 +00:00
Eelco Dolstra 885e22b16e * Implement isValidPath(). 2010-02-18 15:11:08 +00:00
Eelco Dolstra cfb09e0fad * Automatically abort transactions if they go out of scope without
committing.
2010-02-18 14:40:07 +00:00
Eelco Dolstra e0305bb7a8 * Some wrapper objects to ensure that SQLite objects are properly
destroyed.
2010-02-18 14:30:42 +00:00
Eelco Dolstra a053d2d8e5 * Add the deriver to the ValidPaths table. In principle we could now
store all the derivers of a path efficiently.  But that opens a big
  can of worms with respect to garbage collector semantics.
2010-02-18 13:48:18 +00:00
Eelco Dolstra dbddac0fe9 * Assign an integer id to every row in the ValidPaths table in order
to make the Refs table more space-efficient.  For instance, this
  reduces the size of the database on my laptop from 93 MiB to 18
  MiB.  (It was 72 MiB with the old schema on an ext3 disk with a 1
  KiB block size.)
2010-02-18 13:40:46 +00:00
Eelco Dolstra c1a07f9445 * Convert the Nix database to SQLite. 2010-02-18 13:16:59 +00:00
Eelco Dolstra 4e17be7981 * Revert r19797, and use a simpler solution: just don't monitor build
hooks for silence.  It's unnecessary because the remote nix-store
  command is already monitoring the real build.
2010-02-03 21:38:41 +00:00
Eelco Dolstra f859a8d3c3 * While waiting for a lock, print a sign of life every 5 minutes.
This prevents remote builders from being killed by the
  `max-silent-time' inactivity monitor while they are waiting for a
  long garbage collection to finish.  This happens fairly often in the
  Hydra build farm.
2010-02-03 21:22:57 +00:00
Eelco Dolstra 4bbbe25802 * Remove most Cygwin-specific code. Cygwin 1.7 implements advisory
POSIX locks, and simulates Unix-style file deletion semantics
  sufficiently.  Note that this means that Nix won't work on Cygwin
  1.5 anymore.
2010-02-02 15:28:36 +00:00
Eelco Dolstra 07ffdc2862 * Added an option "fsync-metadata" to fsync() changes to
/nix/var/nix/db.
* Removed the function writeStringToFile since it does (almost) the
  same thing as writeFile.
2010-01-29 12:22:58 +00:00
Eelco Dolstra ad529fb89f * Don't consider a store path valid if its info file exists but is
zero bytes long.  That makes Nix more robust in case of crashes
  (especially on ext4).
2010-01-29 11:53:58 +00:00
Eelco Dolstra fdcaf37361 * Made `nix-store -qR --include-outputs' much faster if there are
multiple paths specified on the command line (from O(n * m) to O(n +
  m), where n is the number of arguments and m is the size of the
  closure).
2010-01-25 17:18:44 +00:00
Eelco Dolstra 5388944e8d * Make the garbage collector do the right thing when `gc-keep-outputs'
is enabled by not depending on the deriver.
2010-01-25 16:04:32 +00:00
Eelco Dolstra ef92a14bfe * Include config.h before the C library headers, because it defines
_FILE_OFFSET_BITS=64.  Without it, functions like stat() fail on
  large file sizes.  This happened with a Nix store on squashfs:

  $ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
  error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type

  $ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
  File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
  Size: 0               Blocks: 36028797018963968 IO Block: 1024   regular empty file

  (This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
  to fail.)
2009-12-17 14:12:44 +00:00
Eelco Dolstra 945d8218fb * Build correctly against newer ATerm releases. Fixes "error: 'union'
tag used in naming 'struct _ATerm'".
2009-12-16 15:29:50 +00:00
Eelco Dolstra bcd6cdf0d8 * Give a better error message when trying to build something and
readOnlyMode is set.
2009-12-09 17:45:22 +00:00
Eelco Dolstra f9e766db98 * Randomise the order in which we delete entries to make the collector
less biased towards deleting paths that come alphabetically first
  (e.g. /nix/store/000...).  This matters when using --max-freed etc.
2009-11-24 09:53:18 +00:00
Eelco Dolstra ca50c83fbb 2009-11-23 21:21:29 +00:00
Eelco Dolstra ae6bf87273 * `nix-store --gc --print-roots': also print the path of the actual
root symlink, not just its target.  E.g.:

  /nix/var/nix/profiles/system-99-link -> /nix/store/76kwf88657nq7wgk1hx3l1z5q91zb9wd-system
2009-11-23 17:23:12 +00:00
Eelco Dolstra c364d5d1e3 * Made the garbage collector a lot faster. It no longer computes the
complete set of live and dead paths before starting the actual
  deletion, but determines liveness on demand.  I.e. for any path in
  the store, it first tries to delete all the referrers, and then the
  path itself.  This means that the collector can start deleting paths
  almost immediately.
2009-11-23 16:34:24 +00:00
Eelco Dolstra 8824d60fe5 * Remove the --use-atime / --max-atime garbage collector flags. Many
(Linux) machines no longer maintain the atime because it's too
  expensive, and on the machines where --use-atime is useful (like the
  buildfarm), reading the atimes on the entire Nix store takes way too
  much time to make it practical.
2009-11-20 17:12:38 +00:00
Eelco Dolstra 327a232c85 * Remove support for old (before Nix 0.12pre12020) databases. 2009-11-06 01:15:44 +00:00
Eelco Dolstra 8520542071 * When building in a chroot, make a copy of a file if hard-linking
fails.  This is likely to happen after a `nix-store --optimise',
  because some files may have 32000 links (NIX-111).
2009-10-22 08:28:33 +00:00
Eelco Dolstra 6b9f6b0222 * Remove a prototype for a function that no longer exists. 2009-10-22 08:12:38 +00:00
Sander van der Burg 53a4981fa2 Added optional parameter which adds -lnsl -lsocket to make the Nix package manager work on OpenSolaris 2009-10-08 14:50:37 +00:00
Eelco Dolstra c7057fc1f2 * And some more. 2009-09-24 07:39:55 +00:00
Eelco Dolstra 193f59e077 * Fix a build failure on Fedora 11. rename() needs <stdio.h>. 2009-09-24 07:21:29 +00:00
Eelco Dolstra 676e07902e * Darwin hack. 2009-09-23 18:04:55 +00:00
Eelco Dolstra 64e89980e8 * Create some state directories automatically as a convenience. 2009-09-23 17:05:51 +00:00
Rob Vermaas 48b58617e9 * include wait.h for WEXITSTATUS 2009-09-23 12:57:15 +00:00
Eelco Dolstra 14bc3ce3d6 * Canonicalise timestamps in the Nix store to 1 (1970-01-01 00:00:01
UTC) rather than 0 (00:00:00).  1 is a better choice because some
  programs use 0 as a special value.  For instance, the Template
  Toolkit uses a timestamp of 0 to denote the non-existence of a file,
  so it barfs on files in the Nix store (see
  template-toolkit-nix-store.patch in Nixpkgs).  Similarly, Maya 2008
  fails to load script directories with a timestamp of 0 and can't be
  patched because it's closed source.

  This will also shut up those "implausibly old time stamp" GNU tar
  warnings.
2009-06-13 16:30:58 +00:00
Eelco Dolstra d5eab2fc82 * Use foreach in a lot of places. 2009-04-21 11:52:16 +00:00
Eelco Dolstra 8f1bf28505 * nix-store --verify: don't bail out if a referenced path is missing.
(It can't fix it though.)
2009-04-21 11:06:27 +00:00
Eelco Dolstra 4e646b0ddb * Fix a few "comparison is always false/true due to limited range of
data type" warnings on 64-bit platforms.  The one in parser.y is
  likely to be a real bug.
2009-04-16 12:03:17 +00:00
Eelco Dolstra dfb863f333 * Don't cache transient build hook problems. 2009-04-15 06:25:02 +00:00
Eelco Dolstra 351bf658f9 * Do a substitution even if --max-jobs == 0. 2009-03-31 21:14:07 +00:00
Eelco Dolstra cff2b2a13a * Doh! 2009-03-31 15:50:03 +00:00
Eelco Dolstra 160a60d663 2009-03-30 19:35:55 +00:00
Eelco Dolstra 805144b705 * Make the poll interval configurable. 2009-03-30 11:34:03 +00:00
Eelco Dolstra 7377195297 * With `--max-jobs 0', print a nicer error message than "Assertion
`!awake.empty()' failed."
2009-03-29 18:06:00 +00:00
Eelco Dolstra 737423a89c * Use polling to wait for a remote build slot when using a build hook
(that is, call the build hook with a certain interval until it
  accepts the build).
* build-remote.pl was totally broken: for all system types other than
  the local system type, it would send all builds to the *first*
  machine of the appropriate type.
2009-03-29 18:00:45 +00:00
Eelco Dolstra 6e946c8e72 * Scan for references and compute the SHA-256 hash of the output in
one pass.  This halves the amount of I/O.
2009-03-28 20:51:33 +00:00
Eelco Dolstra c7152c8f97 * Don't use the non-standard __gnu_cxx::stdio_filebuf class. 2009-03-28 19:41:53 +00:00
Eelco Dolstra 3a2bbe7f8a * Simplify communication with the hook a bit (don't use file
descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
Eelco Dolstra f54e800366 * The `fixedOutput' variable didn't get initialised when using a build
hook, causing negative caching to fail randomly.
2009-03-28 16:12:27 +00:00
Eelco Dolstra cb85bc396e * Create /nix/store if it doesn't exist. 2009-03-27 14:19:04 +00:00
Eelco Dolstra 5dd8fb2069 * Don't use ULLONG_MAX in maxFreed - use 0 to mean "no limit".
18446744073709551615ULL breaks on GCC 3.3.6 (`integer constant is
  too large for "long" type').
2009-03-26 11:02:07 +00:00
Eelco Dolstra 92f525ecf4 * Negative caching, i.e. caching of build failures. Disabled by
default.  This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
Eelco Dolstra a046858a22 * Doh. 2009-03-24 23:40:24 +00:00
Eelco Dolstra 12c8c64efa 2009-03-24 14:07:37 +00:00
Eelco Dolstra cacff1be88 * No longer block while waiting for a lock on a store path. Instead
poll for it (i.e. if we can't acquire the lock, then let the main
  select() loop wait for at most a few seconds and then try again).
  This improves parallelism: if two nix-store processes are both
  trying to build a path at the same time, the second one shouldn't
  block; it should first see if it can build other goals.  Also, it
  prevents the deadlocks that have been occuring in Hydra lately,
  where a process waits for a lock held by another process that's
  waiting for a lock held by the first.

  The downside is that polling isn't really elegant, but POSIX doesn't
  provide a way to wait for locks in a select() loop.  The only
  solution would be to spawn a thread for each lock to do a blocking
  fcntl() and then signal the main thread, but that would require
  pthreads.
2009-03-23 01:05:54 +00:00
Eelco Dolstra 58969fa2bf * Refactoring. 2009-03-22 23:53:05 +00:00
Eelco Dolstra 77d272623f * NAR archives: handle files larger than 2^32 bytes. Previously it
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
  isn't a NAR archive.
2009-03-22 17:36:43 +00:00
Eelco Dolstra 2897286487 * Unify exportReferencesGraph and exportBuildReferencesGraph, and make
sure that it works as expected when you pass it a derivation.  That
  is, we have to make sure that all build-time dependencies are built,
  and that they are all in the input closure (otherwise remote builds
  might fail, for example).  This is ensured at instantiation time by
  adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
Eelco Dolstra c183ee5c79 * Acquire the locks on the output paths before trying to run the build
hook.  This fixes a problem with log files being partially or
  completely filled with 0's because another nix-store process
  truncates the log file.  It should also be more efficient.
2009-03-18 14:48:42 +00:00
Eelco Dolstra 33ecb42991 * Cleanup. 2009-03-17 11:42:55 +00:00
Eelco Dolstra 60cb7de336 * Allow options from the Nix config file to be overriden from the
command line (e.g. "--option build-use-chroot true").
2009-02-27 11:04:41 +00:00
Eelco Dolstra 824b154ce8 * Release output locks as soon as possible, not when the destructor of
the DerivationGoal runs.  Otherwise, if a goal is a top-level goal,
  then the lock won't be released until nix-store finishes.  With
  --keep-going and lots of top-level goals, it's possible to run out
  of file descriptors (this happened sometimes in the build farm for
  Nixpkgs).  Also, for failed derivation, it won't be possible to
  build it again  until the lock is released.
  
* Idem for locks on build users: these weren't released in a timely
  manner for failed top-level derivation goals.  So if there were more
  than (say) 10 such failed builds, you would get an error about
  having run out of build users.
2009-02-16 09:24:20 +00:00
Eelco Dolstra b682fae9d9 * Build hooks: use nix-store --import. This prevents a redundant
scan for runtime dependencies (i.e. the local machine shouldn't do a
  scan that the remote machine has already done).  Also pipe directly
  into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
Eelco Dolstra 061141e632 * Make it compile on Debian 4.0 (which doesn't define PER_LINUX32_3GB
in sys/personality.h).
2009-01-27 13:36:59 +00:00
Eelco Dolstra 019176137f * When using a build hook, distinguish between transient failures
(e.g. an SSH connection problem) and permanent failures (i.e. the
  builder failed).  This matters to Hydra (it wants to know whether it
  makes sense to retry a build).
2009-01-13 11:39:09 +00:00
Eelco Dolstra c504d90c11 * Support i686-linux builds directly on x86_64-linux Nix
installations.  This is implemented using the personality() syscall,
  which causes uname to return "i686" in child processes.
2009-01-12 16:30:32 +00:00
Eelco Dolstra 8fce03e0ad * nix-store --verify: repair bad hash fields in the metadata file. 2008-12-16 13:28:18 +00:00
Eelco Dolstra 60ec75048a * Pass --use-atime / --max-atime to the daemon. 2008-12-16 12:23:35 +00:00
Eelco Dolstra 6f6bb1fdea * Delete the chroot if it already exists (e.g. left over from an
interrupted build).
2008-12-15 23:55:11 +00:00
Eelco Dolstra 92cb7c4dfe * Put chroots in the Nix store. This ensures that we can create hard
links to the inputs.
2008-12-12 17:14:57 +00:00
Eelco Dolstra 0008b0006d * Simplify deleting .lock files in /nix/store: just don't delete them
if they belong a path that's currently being built.  This gets rid
  of some Cygwin-specific code.
2008-12-12 17:03:18 +00:00
Eelco Dolstra 2b70a8e7c9 * Detect whether unshare() is available. 2008-12-12 13:41:36 +00:00
Eelco Dolstra 6776a52bb3 * Use a PathSet for the chroot directories so that we don't
accidentally bind-mount a directory twice.
2008-12-12 11:49:42 +00:00
Eelco Dolstra d86bd22d24 * Define _GNU_SOURCE. Hopefully this fixes the build on Debian 4.0
(http://hydra.nixos.org/nixlog/384/1).
2008-12-12 10:20:19 +00:00
Eelco Dolstra ac5478eb52 * Don't provide the whole Nix store in the chroot, but only the
closure of the inputs.  This really enforces that there can't be any
  undeclared dependencies on paths in the store.  This is done by
  creating a fake Nix store and creating bind-mounts or hard-links in
  the fake store for all paths in the closure.  After the build, the
  build output is moved from the fake store to the real store.  TODO:
  the chroot has to be on the same filesystem as the Nix store for
  this to work, but this isn't enforced yet.  (I.e. it only works
  currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
Eelco Dolstra 652817046b * Revert r13150: now that we use private namespaces for the chroot, we
don't have to put the chroot in /nix/var/nix/chroots anymore.
  They're back in /tmp now.
2008-12-11 17:52:34 +00:00
Eelco Dolstra 5a569509b4 * Provide a minimal /etc/passwd in the chroot to keep some builders
happy.
2008-12-11 17:44:02 +00:00
Eelco Dolstra 7c54f1603f * Do chroot builds in a private namespace. This means that all the
bind-mounts we do are only visible to the builder process and its
  children.  So accidentally doing "rm -rf" on the chroot directory
  won't wipe out /nix/store and other bind-mounted directories
  anymore.  Also, the bind-mounts in the private namespace disappear
  automatically when the builder exits.
2008-12-11 17:00:12 +00:00
Eelco Dolstra 07cdfb09fb * Open the connection to the daemon lazily (on demand) so that
read-only operations (like nix-env -qa) work properly when the
  daemon isn't running.
2008-12-11 14:30:25 +00:00
Eelco Dolstra a0766eca27 * Build on Debian 4.0 (GCC 4.1.1). 2008-12-04 21:07:29 +00:00
Eelco Dolstra 9ac3f5df9c * Propagate --max-silent-time to remote machines. 2008-12-04 16:51:16 +00:00
Eelco Dolstra 909fbb9de1 * When using build hooks, for any nix-store -r build operation, it is
necessary that at least one build hook doesn't return "postpone",
  otherwise nix-store will barf ("waiting for a build slot, yet there
  are no running children").  So inform the build hook when this is
  the case, so that it can start a build even when that would exceed
  the maximum load on a machine.
2008-12-04 14:29:41 +00:00
Eelco Dolstra 82ae85de27 * addToStore() in nix-worker: don't write the NAR dump received from
the client to a temporary directory, as that is highly inefficient.
2008-12-03 18:05:14 +00:00
Eelco Dolstra ff762fb499 * Pass HashType values instead of strings. 2008-12-03 16:10:17 +00:00
Eelco Dolstra 1307b22223 * Made addToStore() a lot more efficient: it no longer reads the path
being copied 3 times in the worst case.  It doesn't run in constant space,
  but it didn't do that anyway.
2008-12-03 15:51:17 +00:00
Eelco Dolstra 64519cfd65 * Unify the treatment of sources copied to the store, and recursive
SHA-256 outputs of fixed-output derivations.  I.e. they now produce
  the same store path:

  $ nix-store --add x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  $ nix-store --add-fixed --recursive sha256 x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  the latter being the same as the path that a derivation

    derivation {
      name = "x";
      outputHashAlgo = "sha256";
      outputHashMode = "recursive";
      outputHash = "...";
      ...
    };

  produces.

  This does change the output path for such fixed-output derivations.
  Fortunately they are quite rare.  The most common use is fetchsvn
  calls with SHA-256 hashes.  (There are a handful of those is
  Nixpkgs, mostly unstable development packages.)
  
* Documented the computation of store paths (in store-api.cc).
2008-12-03 15:06:30 +00:00
Eelco Dolstra a55113411f * Nix daemon: reload the configuration file after forking (NIX-100). 2008-11-20 12:25:11 +00:00
Eelco Dolstra e13da525a7 * Files in the info directory starting with "." are temporary files
and don't indicate path validity.
2008-11-19 16:27:07 +00:00
Eelco Dolstra 6fedb7aa0f * Restore SIGPIPE to SIG_DFL when running the builder. This prevents
subtle and often hard-to-reproduce bugs where programs in pipes
  either barf with a "Broken pipe" message or not, depending on the
  exact timing conditions.  This particularly happened in GNU M4 (and
  Bison, which uses M4).
2008-11-14 15:46:45 +00:00
Eelco Dolstra a519bb0635 * Some somewhat ad hoc mechanism to allow the build farm to monitor
build progress.
2008-11-12 11:08:27 +00:00
Eelco Dolstra 96598e7b06 * Pass the --no-build-output flag to the daemon. 2008-11-11 15:11:10 +00:00
Eelco Dolstra 4166b11a53 * Add /dev/pts to the default nix.conf. 2008-11-11 14:59:20 +00:00
Eelco Dolstra 709b55ee02 * Put the chroots under /nix/var/nix/chroots to reduce the risk of
disasters involving `rm -rf' on bind mounts.  Will try the
  definitive fix (per-process mounts, apparently possible via the
  CLONE_NEWNS flag in clone()) some other time.
2008-10-29 15:34:48 +00:00
Ludovic Courtès c98ea254dc libstore: Always mount `/dev/pts' individually.
This fixes problems such as Tcl's PTY handling:

  ERROR: The system has no more ptys.  Ask your system administrator to
  create more.
2008-10-16 21:04:32 +00:00
Eelco Dolstra 9d6d50269b * Bug fix for building on some old installations (contributed by Pjotr). 2008-10-16 14:16:03 +00:00
Eelco Dolstra f32fef1b07 * GC option `--max-atime' that specifies an upper limit to the last
accessed time of paths that may be deleted.  Anything more recently
  used won't be deleted.  The time is specified in time_t,
  e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
  convert to time_t from the command line. 

  Example: to delete everything that hasn't been used in the last two
  months:

  $ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
2008-09-17 14:52:35 +00:00
Eelco Dolstra 4af2fdba6d * Typo. 2008-09-17 13:00:55 +00:00
Eelco Dolstra c987061aa4 * Some refactoring. Better output with `-v' for --use-atime. 2008-09-17 12:54:07 +00:00
Eelco Dolstra 7ab68961e4 * Garbage collector: added an option `--use-atime' to delete paths in
order of ascending last access time.  This is useful in conjunction
  with --max-freed or --max-links to prefer deleting non-recently used
  garbage, which is good (especially in the build farm) since garbage
  may become live again.

  The code could easily be modified to accept other criteria for
  ordering garbage by changing the comparison operator used by the
  priority queue in collectGarbage().
2008-09-17 10:02:55 +00:00
Eelco Dolstra 7718b19389 * Explicitly set PWD to prevent problems with chroot builds. In
particular, dietlibc cannot figure out the cwd because the inode of
  the current directory doesn't appear in .. (because getdents returns
  the inode of the mount point).
2008-08-27 17:20:25 +00:00
Eelco Dolstra 9cc0da8453 * Create a /tmp with 1777 permission in the chroot. Some builders
need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 16:03:03 +00:00
Eelco Dolstra d06be428f6 * Disable chroot builds for fixed-output derivations so that we don't
need /etc in the chroot (in particular, /etc/resolv.conf for
  fetchurl).  Not having /etc/resolv.conf in the chroot is a good
  thing, since we don't want normal derivations to download files.
2008-08-25 15:49:22 +00:00
Eelco Dolstra b428adc267 * Strip off the `.nix' suffix from the attribute name for files in
~/.nix-defexpr,  otherwise the attribute cannot be selected with the
  `-A' option.  Useful if you want to stick a Nix expression directly
  in ~/.nix-defexpr.
2008-08-25 13:31:57 +00:00
Eelco Dolstra db4f4a8425 * Backward compatibility check to prevent nixos-rebuild from barfing
when upgrading Nix.
2008-08-14 13:02:19 +00:00
Eelco Dolstra c03b729319 * Increase the sleep periods a bit to make the test less likely to
fail on slow machines.  Of course it would be better if this test
  wasn't timing dependent...
2008-08-14 09:26:30 +00:00
Eelco Dolstra 98b07466fb * Better error checking of the data from the substituters. 2008-08-05 10:57:53 +00:00
Eelco Dolstra 5adbb0aabe * build.cc: only use a substituter if it returns info for a path. 2008-08-04 13:15:35 +00:00
Eelco Dolstra c4f98941ed * nix-env --dry-run: show the total size of the substituter
downloads.
2008-08-04 12:29:04 +00:00
Eelco Dolstra 03427e76f1 * querySubstitutablePathInfo: work properly when run via the daemon.
* --dry-run: print the paths that we don't know how to build/substitute.
2008-08-04 11:44:50 +00:00
Eelco Dolstra 3c92ea399d * Make nix-env --dry-run print the paths to be substituted correctly
again.  (After the previous substituter mechanism refactoring I
  didn't update the code that obtains the references of substitutable
  paths.)  This required some refactoring: the substituter programs
  are now kept running and receive/respond to info requests via
  stdin/stdout.
2008-08-02 12:54:35 +00:00
Eelco Dolstra 989176c56e * Allow read-only access to the store (e.g., non-root users on NixOS
can do operations like "nix-store -qR <path>" even without the Nix
  daemon).
2008-07-18 15:34:46 +00:00
Eelco Dolstra 6c8641a542 2008-06-18 19:17:05 +00:00
Eelco Dolstra d3aa183beb * Garbage collector: option `--max-freed' to stop after at least N
bytes have been freed, `--max-links' to stop when the Nix store
  directory has fewer than N hard links (the latter being important
  for very large Nix stores on filesystems with a 32000 subdirectories
  limit).
2008-06-18 14:20:16 +00:00
Eelco Dolstra a8f3b02092 * `nix-store --optimise': handle files with >= 32000 hard links.
(There can easily be more than 32000 occurrences of the empty file.)
2008-06-18 14:13:00 +00:00
Eelco Dolstra a72709afd8 * Some refactoring: put the GC options / results in separate structs.
* The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
Eelco Dolstra 94fd46fa1c * Note. 2008-06-14 16:02:31 +00:00
Eelco Dolstra 826b271d9a * Garbage collector: don't do a complete topological sort of the Nix
store under the reference relation, since that means that the
  garbage collector will need a long time to start deleting paths.
  Instead just delete the referrers of a path first.
2008-06-13 18:25:24 +00:00
Eelco Dolstra 30c9f909b2 * Print some progress info during the early GC stages. 2008-06-13 17:21:20 +00:00
Eelco Dolstra b0e92f6d47 * Merged the no-bdb branch (-r10900:HEAD
https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Eelco Dolstra bd955e15e1 * GCC 4.3.0 (Fedora 9) compatibility fixes. Reported by Gour and
Armijn Hemel.
2008-05-21 11:17:31 +00:00
Eelco Dolstra 72034ab35d * sockaddr_un doesn't allow path names of more than 108 characters.
This isn't usually a problem, except that it causes tests to fail
  when performed in a directory with a very long path name.  So chdir
  to the socket directory and use a relative path name.
2008-04-09 05:57:01 +00:00
Eelco Dolstra 329025253d * Use /tmp/nix-build-<drvpath>-<counter> instead of
/tmp/nix-<pid>-<counter> for temporary build directories.  This
  increases purity a bit: many packages store the temporary build path
  in their output, causing (generally unimportant) binary differences.
2008-03-27 13:45:17 +00:00
Eelco Dolstra f789ea1d09 * A transaction is unnecessary since the path cannot be
garbage-collected (it's a temporary root).
2008-03-01 21:05:33 +00:00
Eelco Dolstra 66c51dc215 * nix-store --dump-db / --load-db to dump/load the Nix DB.
* nix-store --register-validity: option to supply the content hash of
  each path.
* Removed compatibility with Nix <= 0.7 stores.
2008-01-29 18:17:36 +00:00
Michael Raskin 5eb5c23447 Fixed exportBuildReferenceGraph 2008-01-15 04:32:08 +00:00
Michael Raskin 2196cb67c5 Added a way to include derivation with its outputs and dependencies - exportBuildReferenceGraph 2007-12-30 09:30:56 +00:00
Eelco Dolstra 633518628f * nix-env -e: support uninstalling by path, so that one can say
$ nix-env -e $(which firefox)

  or

    $ nix-env -e /nix/store/nywzlygrkfcgz7dfmhm5xixlx1l0m60v-pan-0.132

* nix-env -i: if an argument contains a slash anywhere, treat it as a
  path and follow it through symlinks into the Nix store.  This allows
  things like

    $ nix-build -A firefox
    $ nix-env -i ./result

* nix-env -q/-i/-e: don't complain when the `*' selector doesn't match
  anything.  In particular, `nix-env -q \*' doesn't fail anymore on an
  empty profile.
2007-11-29 16:18:24 +00:00
Eelco Dolstra c370755583 * Flag `--no-build-hook' to disable distributed builds.
* queryDeriver in daemon mode: don't barf if the other side returns an
  empty string (which means there is no deriver).
2007-11-16 16:15:26 +00:00
Eelco Dolstra 149e273023 * Don't need gc.hh. 2007-11-15 15:07:27 +00:00
Michael Raskin 6fc60e2060 Added possibility to specify garbage collection levels for store paths; so packages can share intermediate results of compilation and GC will collect it automatically while never touching tarballs, for example. 2007-11-15 03:47:12 +00:00
Eelco Dolstra 42d80d1170 * On FreeBSD, sys/mount.h needs sys/param.h. 2007-10-29 10:03:07 +00:00
Eelco Dolstra d91cd30563 * Detect whether chroot / bind-mount support is available. 2007-10-27 16:51:55 +00:00
Eelco Dolstra dc6f373842 * Delete the chroot directory automatically.
* Removed some debug messages.
2007-10-27 16:06:38 +00:00
Eelco Dolstra 9397cd30c8 * Support for doing builds in a chroot under Linux. The builder is
executed in a chroot that contains just the Nix store, the temporary
  build directory, and a configurable set of additional directories
  (/dev and /proc by default).  This allows a bit more purity
  enforcement: hidden build-time dependencies on directories such as
  /usr or /nix/var/nix/profiles are no longer possible.  As an added
  benefit, accidental network downloads (cf. NIXPKGS-52) are prevented
  as well (because files such as /etc/resolv.conf are not available in
  the chroot).

  However the usefulness of chroots is diminished by the fact that
  many builders depend on /bin/sh, so you need /bin in the list of
  additional directories.  (And then on non-NixOS you need /lib as
  well...)
2007-10-27 00:46:59 +00:00
Eelco Dolstra d04291cfab * Doh! Don't change the permissions on /nix/store. 2007-10-10 13:52:38 +00:00
Eelco Dolstra 315183f194 * nix-store --optimise: flag "--dry-run" to just query what the disk
savings would be.
2007-10-10 13:43:04 +00:00
Eelco Dolstra a8629de827 * New command `nix-store --optimise' to reduce Nix store disk space
usage by finding identical files in the store and hard-linking them
  to each other.  It typically reduces the size of the store by
  something like 25-35%.  This is what the optimise-store.pl script
  did, but the new command is faster and more correct (it's safe wrt
  garbage collection and concurrent builds).
2007-10-09 22:14:27 +00:00
Eelco Dolstra f3441e6122 * Pass various options to the worker so that flags like -K or -j work
in multi-user Nix (NIX-72).
* Client/worker: exchange a protocol version number for future
  compatibility.
2007-09-18 09:11:20 +00:00
Eelco Dolstra 0d65fc08e2 * Create the Nix daemon socket in a separate directory
(/nix/var/nix/daemon-socket).  This allows access to the Nix daemon
  to be restricted by setting the mode/ownership on that directory as
  desired, e.g.

    $ chmod 770 /nix/var/nix/daemon-socket
    $ chown root.wheel /nix/var/nix/daemon-socket

  to allow only users in the wheel group to use Nix.

  Setting the ownership on a socket is much trickier, since the socket
  must be deleted and recreated every time the daemon is started
  (which would require additional Nix configuration file directives to
  specify the mode/ownership, and wouldn't support arbitrary ACLs),
  some BSD variants appear to ignore permissions on sockets, and it's
  not clear whether the umask is respected on every platform when
  creating sockets.
2007-08-30 09:50:44 +00:00
Eelco Dolstra cb1c1004cd * When there are multiple substituters, make sure to release the
lock on the output path after trying each.  Otherwise the
  pathIsLockedByMe() test gets confused.
2007-08-28 16:22:08 +00:00
Eelco Dolstra c970b28ba0 * Fix a race condition with parallel builds where multiple
fixed-output derivations or substitutions try to build the same
  store path at the same time.  Locking generally catches this, but
  not between multiple goals in the same process.  This happened
  especially often (actually, only) in the build farm with fetchurl
  downloads of the same file being executed on multiple machines and
  then copied back to the main machine where they would clobber each
  other (NIXBF-13).

  Solution: if a goal notices that the output path is already locked,
  then go to sleep until another goal finishes (hopefully the one
  locking the path) and try again.
2007-08-28 11:36:17 +00:00
Eelco Dolstra e0d7e47862 * PathLocks::lockPaths: don't allow reacquiring a lock we already
hold.
2007-08-28 09:39:03 +00:00
Eelco Dolstra 3757ee589f * Bump the Nix database schema version number; delete the substitutes
table.
2007-08-13 11:37:39 +00:00
Eelco Dolstra 59afc1a15c * Backwards compatibility. 2007-08-13 11:15:13 +00:00
Eelco Dolstra 9e975458b4 * Get rid of the substitutes database table (NIX-47). Instead, if we
need any info on substitutable paths, we just call the substituters
  (such as download-using-manifests.pl) directly.  This means that
  it's no longer necessary for nix-pull to register substitutes or for
  nix-channel to clear them, which makes those operations much faster
  (NIX-95).  Also, we don't have to worry about keeping nix-pull
  manifests (in /nix/var/nix/manifests) and the database in sync with
  each other.

  The downside is that there is some overhead in calling an external
  program to get the substitutes info.  For instance, "nix-env -qas"
  takes a bit longer.

  Abolishing the substitutes table also makes the logic in
  local-store.cc simpler, as we don't need to store info for invalid
  paths.  On the downside, you cannot do things like "nix-store -qR"
  on a substitutable but invalid path (but nobody did that anyway).

* Never catch interrupts (the Interrupted exception).
2007-08-12 00:29:28 +00:00
Eelco Dolstra 6d1a1191b0 * Support queryDeriver() in multi-user installations. 2007-06-12 16:53:44 +00:00
Eelco Dolstra 9bff7ad728 * Check against creation of GC roots in the store. Those roots don't
work, because findRoots() stops when it encounters a symlink to the
  store.  And of course the store is supposed to be read-only.
2007-06-11 11:36:22 +00:00
Eelco Dolstra 7046c35843 * Typo (reported by Marc Weber). 2007-05-14 12:16:41 +00:00
Eelco Dolstra e5da9c8803 * Create the database directory if it doesn't exist. 2007-05-07 16:59:19 +00:00
Eelco Dolstra cbfac2fdcc * Set a terminate() handler to ensure that we leave the BDB
environment cleanly even when an exception is thrown from a
  destructor.  We still crash, but we don't take all other Nix
  processes with us.
2007-05-01 15:16:17 +00:00
Eelco Dolstra 644946feed * Make --verify more interruptable. 2007-05-01 13:21:05 +00:00
Eelco Dolstra ae7990cc88 * Work around a bug in Apple's GCC preprocessor. 2007-03-30 13:24:35 +00:00
Eelco Dolstra 803cb6e3b9 * Override the setuid helper using NIX_SETUID_HELPER. 2007-03-20 22:04:25 +00:00
Eelco Dolstra 8ab229ddf2 * Terminate build hooks and substitutes with a TERM signal, not a KILL
signal.  This is necessary because those processes may have joined
  the BDB environment, so they have to be given a chance to clean up.
  (NIX-85)
2007-03-19 12:48:45 +00:00
Eelco Dolstra b2b6cf3fc8 * Undocumented option `gc-check-reachability' to allow reachability
checking to be turned off on machines with way too many roots.
2007-03-19 09:16:47 +00:00
Eelco Dolstra 917e06bf63 * Delete the output paths before invoking the build hook. 2007-03-07 15:53:11 +00:00
Eelco Dolstra db1973d012 * Look for the openssl program at compile time. If not found, call
openssl through $PATH at runtime.
2007-03-01 13:30:46 +00:00
Eelco Dolstra b4a040e52b * Don't check the signature unless we have to. 2007-03-01 12:30:24 +00:00
Eelco Dolstra 2ea3bebc23 * Doh! The deriver can be empty. 2007-02-27 23:18:57 +00:00
Eelco Dolstra 9da367b7d5 * `nix-store -qR' and friends: print the paths sorted topologically
under the references relation.  This is useful for commands that
  want to copy paths to another Nix store in the right order.
2007-02-21 22:45:10 +00:00
Eelco Dolstra 65f195f4c7 * Check that the file containing the secret key is secret. 2007-02-21 17:51:10 +00:00
Eelco Dolstra bdadb98de8 * `nix-store --import' now also works in remote mode. The worker
always requires a signature on the archive.  This is to ensure that
  unprivileged users cannot add Trojan horses to the Nix store.
2007-02-21 17:34:02 +00:00
Eelco Dolstra 0f5da8a83c * Support exportPath() in remote mode. 2007-02-21 16:34:00 +00:00
Eelco Dolstra dc7d594776 * importPath(): set the deriver.
* exportPath(): lock the path, use a transaction.
2007-02-21 16:23:25 +00:00
Eelco Dolstra 43c4d18c6a * nix-store --import': import an archive created by nix-store
--export' into the Nix store, and optionally check the cryptographic
  signatures against /nix/etc/nix/signing-key.pub.  (TODO: verify
  against a set of public keys.)
2007-02-21 15:45:32 +00:00
Eelco Dolstra 46e0919ced * `nix-store --export --sign': sign the Nix archive using the RSA key
in /nix/etc/nix/signing-key.sec
2007-02-21 14:31:42 +00:00
Eelco Dolstra b824a1daee * Start of `nix-store --export' operation for serialising a store
path.  This is like `nix-store --dump', only it also dumps the
  meta-information of the store path (references, deriver).  Will add
  a `--sign' flag later to add a cryptographic signature, which we
  will use for exchanging store paths between build farm machines in a
  secure manner.
2007-02-20 23:17:20 +00:00
Eelco Dolstra 46605fb4f5 * Fix 64-bit compiler warnings. 2007-02-06 20:03:53 +00:00
Eelco Dolstra f52de527c7 * Doh! 2007-01-29 15:55:49 +00:00
Eelco Dolstra b618fa6eb6 * computeStorePathForText: take the references into account when
computing the store path (NIX-77).  This is an important security
  property in multi-user Nix stores.

  Note that this changes the store paths of derivations (since the
  derivation aterms are added using addTextToStore), but not most
  outputs (unless they use builtins.toFile).
2007-01-29 15:51:37 +00:00
Eelco Dolstra fac63d6416 * exportReferencesGraph: work on paths within store paths as well. 2007-01-23 16:57:43 +00:00
Eelco Dolstra 63f3ce6d9a * `nix-store --verify': revive checking the referrers table. This is
important to get garbage collection to work if there is any
  inconsistency in the database (because the referrer table is used to
  determine whether it is safe to delete a path).
* `nix-store --verify': show some progress.
2007-01-14 17:28:30 +00:00
Eelco Dolstra 8f67b35886 * Make the garbage collector more resilient to certain consistency
errors: in-use paths now cause a warning, not a fatal error.
2007-01-14 16:24:49 +00:00
Eelco Dolstra a3e6415ba8 * New primop builtins.filterSource, which can be used to filter files
from a source directory.  All files for which a predicate function
  returns true are copied to the store.  Typical example is to leave
  out the .svn directory:

    stdenv.mkDerivation {
      ...
      src = builtins.filterSource
        (path: baseNameOf (toString path) != ".svn")
        ./source-dir;
      # as opposed to
      #   src = ./source-dir;
    }

  This is important because the .svn directory influences the hash in
  a rather unpredictable and variable way.
2006-12-12 23:05:01 +00:00
Eelco Dolstra b17677462c * Use lchown() instead of chown() in canonicalisePathMetaData(). This
matters when running as root, since then we don't use the setuid
  helper (which already used lchown()).
  
* Also check for an obscure security problem on platforms that don't
  have lchown.  Then we can't change the ownership of symlinks, which
  doesn't matter *except* when the containing directory is writable by
  the owner (which is the case with the top-level Nix store directory).
2006-12-09 20:02:27 +00:00
Eelco Dolstra 5f681988f2 * Use deletePathWrapped() in more places. 2006-12-09 00:26:24 +00:00
Eelco Dolstra fa33303146 * Goal cancellation inside the waitForInput() loop needs to be handled
very carefully, since it can invalidate iterators into the
  `children' map.
2006-12-08 18:41:48 +00:00
Eelco Dolstra 06c4929958 * Some refactoring.
* Throw more exceptions as BuildErrors instead of Errors.  This
  matters when --keep-going is turned on.  (A BuildError is caught
  and terminates the goal in question, an Error terminates the
  program.)
2006-12-08 17:26:21 +00:00
Eelco Dolstra 9dbfe242e3 * Kill a build if it has gone for more than a certain number of
seconds without producing output on stdout or stderr (NIX-65).  This
  timeout can be specified using the `--max-silent-time' option or the
  `build-max-silent-time' configuration setting.  The default is
  infinity (0).

* Fix a tricky race condition: if we kill the build user before the
  child has done its setuid() to the build user uid, then it won't be
  killed, and we'll potentially lock up in pid.wait().  So also send a
  conventional kill to the child.
2006-12-08 15:44:00 +00:00
Eelco Dolstra d3fe6ab024 * Also for convenience, change the ownership of the build output even
in case of failure.
2006-12-08 00:19:50 +00:00
Eelco Dolstra 096194ab29 * Remove ancient terminology. 2006-12-07 23:58:36 +00:00
Eelco Dolstra 6833e8bbe8 * When keeping the temporary build directory (-K), change the owner
back to the Nix account.
2006-12-07 23:27:40 +00:00
Eelco Dolstra 2819eb36a4 * Be less verbose. 2006-12-07 21:43:35 +00:00
Eelco Dolstra c3286ec020 * Don't count on the Pid deconstructor to kill the child process,
since if we're running a build user in non-root mode, we can't.  Let
  the setuid helper do it.
2006-12-07 17:52:58 +00:00
Eelco Dolstra f76fdb6d42 * If not running as root, let the setuid helper kill the build user's
processes before and after the build.
2006-12-07 16:33:31 +00:00
Eelco Dolstra ec23ecc64d * In the garbage collector, if deleting a path fails, try to fix its
ownership, then try again.
2006-12-07 15:54:52 +00:00
Eelco Dolstra a0a43c3206 * When not running as root, call the setuid helper to change the
ownership of the build result after the build.
2006-12-07 15:18:14 +00:00
Eelco Dolstra 6a07ff1ec0 * Change the ownership of store paths to the Nix account before
deleting them using the setuid helper.
2006-12-07 14:14:35 +00:00
Eelco Dolstra 7d8cf316ee * Pass the actual build user to the setuid helper. 2006-12-07 11:27:32 +00:00
Eelco Dolstra a45c498e4e * If Nix is not running as root, call the setuid helper to start the
builder under the desired build user.
2006-12-07 00:42:30 +00:00
Eelco Dolstra 813a7c65c9 * Sanity check. 2006-12-07 00:19:27 +00:00
Eelco Dolstra 6a8e60913a * Move killUser() to libutil so that the setuid helper can use it. 2006-12-07 00:16:07 +00:00
Eelco Dolstra 6e5ec1029a * Get rid of `build-users'. We'll just take all the members of
`build-users-group'.  This makes configuration easier: you can just
  add users in /etc/group.
2006-12-06 20:00:15 +00:00
Eelco Dolstra 9f0efa6611 * Start of the setuid helper (the program that performs the operations
that have to be done as root: running builders under different uids,
  changing ownership of build results, and deleting paths in the store
  with the wrong ownership).
2006-12-06 01:24:02 +00:00
Eelco Dolstra 2b558843a2 * Be less chatty. 2006-12-05 19:01:19 +00:00
Eelco Dolstra 44cad9630f * Urgh. Do setgid() before setuid(), because the semantics of setgid()
changes completely depending on whether you're root...
2006-12-05 18:28:15 +00:00
Eelco Dolstra 4c1c37d0b6 * FreeBSD returns ESRCH when there are no processes to kill. 2006-12-05 18:07:46 +00:00
Eelco Dolstra 62b0497c0f * Better message. 2006-12-05 16:17:01 +00:00
Eelco Dolstra a9c4f66cfb * Allow unprivileged users to run the garbage collector and to do
`nix-store --delete'.  But unprivileged users are not allowed to
  ignore liveness.
* `nix-store --delete --ignore-liveness': ignore the runtime roots as
  well.
2006-12-05 02:18:46 +00:00
Eelco Dolstra 29cf434a35 * The determination of the root set should be made by the privileged
process, so forward the operation.
* Spam the user about GC misconfigurations (NIX-71).
* findRoots: skip all roots that are unreadable - the warnings with
  which we spam the user should be enough.
2006-12-05 01:31:45 +00:00
Eelco Dolstra 8623256f48 * findRoots: return a map from the symlink (outside of the store) to
the store path (inside the store).
2006-12-05 00:48:36 +00:00
Eelco Dolstra d27a73b1a9 * In addPermRoot, check that the root that we just registered can be
found by the garbage collector.  This addresses NIX-71 and is a
  particular concern in multi-user stores.
2006-12-05 00:34:42 +00:00
Eelco Dolstra 74033a844f * Add indirect root registration to the protocol so that unprivileged
processes can register indirect roots.  Of course, there is still
  the problem that the garbage collector can only read the targets of
  the indirect roots when it's running as root...
2006-12-04 23:29:16 +00:00
Eelco Dolstra 7751160e9f * Don't redirect stderr. 2006-12-04 19:10:23 +00:00
Eelco Dolstra 40c3529909 * Handle exceptions and stderr for all protocol functions.
* SIGIO -> SIGPOLL (POSIX calls it that).
* Use sigaction instead of signal to register the SIGPOLL handler.
  Sigaction is better defined, and a handler registered with signal
  appears not to interrupt fcntl(..., F_SETLKW, ...), which is bad.
2006-12-04 17:55:14 +00:00
Eelco Dolstra 0130ef88ea * Daemon mode (`nix-worker --daemon'). Clients connect to the server
via the Unix domain socket in /nix/var/nix/daemon.socket.  The
  server forks a worker process per connection.
* readString(): use the heap, not the stack.
* Some protocol fixes.
2006-12-04 17:17:13 +00:00
Eelco Dolstra 4740baf3a6 * When NIX_REMOTE=daemon, connect to /nix/var/nix/daemon.socket
instead of forking a worker.
2006-12-04 14:21:39 +00:00
Eelco Dolstra f5f0cf423f * Refactoring. 2006-12-04 13:28:14 +00:00
Eelco Dolstra 052b6fb149 * Pass the verbosity level to the worker. 2006-12-04 13:15:29 +00:00
Eelco Dolstra 1e16d20655 * Install the worker in bindir, not libexecdir.
* Allow the worker path to be overriden through the NIX_WORKER
  environment variable.
2006-12-04 13:09:16 +00:00
Eelco Dolstra f4279bcde0 * Don't run setuid root when build-users is empty.
* Send startup errors to the client.
2006-12-03 16:25:19 +00:00
Eelco Dolstra 35247c4c9f * Removed `build-allow-root'.
* Added `build-users-group', the group under which builds are to be
  performed.
* Check that /nix/store has 1775 permission and is owner by the
  build-users-group.
2006-12-03 15:32:38 +00:00
Eelco Dolstra 4251f94b32 * Use a Unix domain socket instead of pipes. 2006-12-03 02:36:44 +00:00
Eelco Dolstra 8c76df93e6 * Better error message if the worker doesn't start. 2006-12-03 02:22:04 +00:00
Eelco Dolstra 7951c3c546 * Some hackery to propagate the worker's stderr and exceptions to the
client.
2006-12-03 02:08:13 +00:00
Eelco Dolstra e25fad691a * Move addTempRoot() to the store API, and add another function
syncWithGC() to allow clients to register GC roots without needing
  write access to the global roots directory or the GC lock.
2006-12-02 16:41:36 +00:00
Eelco Dolstra 9c9cdb06d0 * Remove SwitchToOriginalUser, we're not going to need it anymore. 2006-12-02 14:34:14 +00:00
Eelco Dolstra 8ba5d32769 * Remove queryPathHash().
* Help for nix-worker.
2006-12-02 14:27:24 +00:00
Eelco Dolstra fcd9900d74 * Replace read-only calls to addTextToStore. 2006-12-01 21:00:39 +00:00
Eelco Dolstra a824d58b56 * Merge addToStore and addToStoreFixed.
* addToStore now adds unconditionally, it doesn't use readOnlyMode.
  Read-only operation is up to the caller (who can call
  computeStorePathForPath).
2006-12-01 20:51:18 +00:00
Eelco Dolstra b0d8e05be1 * More operations.
* addToStore() and friends: don't do a round-trip to the worker if
  we're only interested in the path (i.e., in read-only mode).
2006-12-01 18:00:01 +00:00
Eelco Dolstra 0565b5f2b3 * More remote operations.
* Added new operation hasSubstitutes(), which is more efficient than
  querySubstitutes().size() > 0.
2006-11-30 22:43:55 +00:00
Eelco Dolstra aac547a8b3 * Doh. 2006-11-30 21:32:46 +00:00
Eelco Dolstra 0263279071 * More operations. 2006-11-30 20:45:20 +00:00
Eelco Dolstra a711689368 * First remote operation: isValidPath(). 2006-11-30 20:13:59 +00:00
Eelco Dolstra 765bdfe542 * When NIX_REMOTE is set to "slave", fork off nix-worker in slave
mode.  Presumably nix-worker would be setuid to the Nix store user.
  The worker performs all operations on the Nix store and database, so
  the caller can be completely unprivileged.

  This is already much more secure than the old setuid scheme, since
  the worker doesn't need to do Nix expression evaluation and so on.
  Most importantly, this means that it doesn't need to access any user
  files, with all resulting security risks; it only performs pure
  store operations.

  Once this works, it is easy to move to a daemon model that forks off
  a worker for connections established through a Unix domain socket.
  That would be even more secure.
2006-11-30 19:54:43 +00:00
Eelco Dolstra 40b3f64b55 * Skeleton of the privileged worker program.
* Some refactoring: put the NAR archive integer/string serialisation
  code in a separate file so it can be reused by the worker protocol
  implementation.
2006-11-30 19:19:59 +00:00
Eelco Dolstra 9adc074dc3 * Oops. 2006-11-30 18:35:50 +00:00
Eelco Dolstra 9cf1948993 * Skeleton of remote store implementation. 2006-11-30 18:35:36 +00:00
Eelco Dolstra 6ecb840fd1 * Put building in the store API. 2006-11-30 18:02:04 +00:00
Eelco Dolstra e2ef5e07fd * Refactoring. There is now an abstract interface class StoreAPI
containing functions that operate on the Nix store.  One
  implementation is LocalStore, which operates on the Nix store
  directly.  The next step, to enable secure multi-user Nix, is to
  create a different implementation RemoteStore that talks to a
  privileged daemon process that uses LocalStore to perform the actual
  operations.
2006-11-30 17:43:04 +00:00
Eelco Dolstra c6a97e3b74 * Doh! Path sizes need to be computed recursively of course.
(NIX-70)
2006-11-24 20:24:14 +00:00
Eelco Dolstra a76efaeb3f * Dead files. 2006-11-24 20:07:30 +00:00
Eelco Dolstra f459a5bb3a * Remove the undocumented `noscan' feature. It's no longer necessary
now that reference scanning is sufficiently streamy.
2006-11-13 18:19:05 +00:00
Eelco Dolstra e2a70b7ec0 * Magic attribute `exportReferencesGraph' that allows the references
graph to be passed to a builder.  This attribute should be a list of
  pairs [name1 path1 name2 path2 ...].  The references graph of each
  `pathN' will be stored in a text file `nameN' in the temporary build
  directory.  The text files have the format used by `nix-store
  --register-validity'.  However, the deriver fields are left empty.

  `exportReferencesGraph' is useful for builders that want to do
  something with the closure of a store path.  Examples: the builders
  that make initrds and ISO images for NixOS.

  `exportReferencesGraph' is entirely pure.  It's necessary because
  otherwise the only way for a builder to get this information would
  be to call `nix-store' directly, which is not allowed (though
  unfortunately possible).
2006-11-13 18:18:13 +00:00
Eelco Dolstra ae6fb27f18 * `nix-store --read-log / -l PATH' shows the build log of PATH, if
available.  For instance,

    $ nix-store -l $(which svn) | less

  lets you read the build log of the Subversion instance in your
  profile.

* `nix-store -qb': if applied to a non-derivation, take the deriver.
2006-10-28 16:33:54 +00:00
Eelco Dolstra 6a67556f71 * Special derivation attribute `allowedReferences' that causes Nix to
check that the references of the output of a derivation are in the
  specified set.  For instance,

    allowedReferences = [];

  specifies that the output cannot have any references.  (This is
  useful, for instance, for the generation of bootstrap binaries for
  stdenv-linux, which must not have any references for purity).  It
  could also be used to guard against undesired runtime dependencies,
  e.g.,

    {gcc, dynlib}: derivation {
      ...
      allowedReferences = [dynlib];
    }

  says that the output can refer to the path of `dynlib' but not
  `gcc'.  A `forbiddedReferences' attribute would be more useful for
  this, though.
2006-10-19 16:09:24 +00:00
Eelco Dolstra d7efd76394 * Big cleanup of the semantics of paths, strings, contexts, string
concatenation and string coercion.  This was a big mess (see
  e.g. NIX-67).  Contexts are now folded into strings, so that they
  don't cause evaluation errors when they're not expected.  The
  semantics of paths has been clarified (see nixexpr-ast.def).
  toString() and coerceToString() have been merged.

  Semantic change: paths are now copied to the store when they're in a
  concatenation (and in most other situations - that's the
  formalisation of the meaning of a path).  So

    "foo " + ./bla

  evaluates to "foo /nix/store/hash...-bla", not "foo
  /path/to/current-dir/bla".  This prevents accidental impurities, and
  is more consistent with the treatment of derivation outputs, e.g.,
  `"foo " + bla' where `bla' is a derivation.  (Here `bla' would be
  replaced by the output path of `bla'.)
2006-10-16 15:55:34 +00:00
Eelco Dolstra 37c8a664f3 * A helpful message. 2006-10-13 11:49:55 +00:00
Eelco Dolstra e4af398681 * Don't crash when upgrading the Berkeley DB environment. 2006-10-13 11:15:53 +00:00
Eelco Dolstra 666babbbfa * Use a bounded amount of memory in scanForReferences() by not reading
regular files into memory all at once.
2006-09-22 13:10:30 +00:00
Eelco Dolstra d22d7565f3 * Don't allocate the buffer twice. 2006-09-22 11:28:23 +00:00
Eelco Dolstra b43aeadbc9 * Don't allocate more than SIZE_MAX bytes. 2006-09-22 11:13:35 +00:00
Eelco Dolstra 5c38c863bd * Fix a huge gaping hole in nix-env w.r.t. the garbage collector.
Nix-env failed to call addPermRoot(), which is necessary to safely
  add a new root.  So if nix-env started after and finished before the
  garbage collector, the user environment (plus all other new stuff)
  it built might be garbage collected, leading to a dangling symlink
  chain in ~/.nix-profile...

* Be more explicit if we block on the GC lock ("waiting for the big
  garbage collector lock...").

* Don't loop trying to create a new generation.  It's not necessary
  anymore since profiles are locked nowadays.
2006-09-14 22:30:33 +00:00
Eelco Dolstra e5a6c09b12 * Install header files in /nix/include/nix. 2006-09-04 22:41:36 +00:00
Eelco Dolstra 4be5443882 * Remove unnecessary inclusions of aterm2.h. 2006-09-04 22:08:40 +00:00
Eelco Dolstra 75068e7d75 * Use a proper namespace.
* Optimise header file usage a bit.
* Compile the parser as C++.
2006-09-04 21:06:23 +00:00
Eelco Dolstra aab8812732 * Store the Nix libraries in ${libdir}/nix instead of ${libdir}. 2006-09-04 15:12:24 +00:00
Martin Bravenboer 3e5b68068b On cygwin, disable the check that the output is not group or world
writable. File permissions on Cygwin are rather complex, and in this
case this check introduced a problem with build jobs invoke from
outside of Cygwin (MSYS). It seemed almost impossible to fix the
permissions of the directory, so for now this safety check is disabled
on Cygwin.
2006-08-15 21:37:48 +00:00
Eelco Dolstra 4250b641d8 * `nix-store --gc --print-dead': print the total size of the store
objects that would be freed.
2006-08-11 20:26:49 +00:00
Eelco Dolstra 92f7dfa5b7 * Don't assume that paths returned by the runtime root finder are
valid.
2006-08-11 20:15:20 +00:00
Eelco Dolstra 6ac237e242 * Show some progress. 2006-08-01 09:43:41 +00:00
Eelco Dolstra a4273156c4 * Use $(libexecdir) to find find-runtime-roots.pl. 2006-07-20 13:21:37 +00:00
Eelco Dolstra ee2cf45d76 * Use debug(). 2006-07-20 12:19:55 +00:00
Eelco Dolstra c15f544356 * Call find-runtime-roots.pl from the garbage collector to prevent
running applications etc. from being garbage collected.
2006-07-20 12:17:25 +00:00
Eelco Dolstra d51aede4af * Allow the canonical system name to be specified at runtime in the
Nix config file.
2006-07-06 15:30:37 +00:00
Eelco Dolstra dbf6d7e783 * Concurrent GC on Cygwin. 2006-06-20 17:48:10 +00:00
Eelco Dolstra cc51f9c539 * Oops. 2006-06-19 16:35:35 +00:00
Eelco Dolstra 5bb3444032 * _exit() doesn't seem to work right on Cygwin. 2006-06-19 16:24:15 +00:00
Eelco Dolstra b35735d8b2 * On Windows we cannot delete open (lock) files, so we delete lock
files after we've closed them.  Since this only succeeds if the lock
  is no longer opened by any process, the token trick used on Unix is
  not necessary.
2006-06-19 14:43:13 +00:00
Eelco Dolstra c937b73622 * Show when we're blocked waiting for a lock. 2006-06-16 10:13:03 +00:00
Eelco Dolstra 588cb0eade * In `nix-env -i|-u|-e', lock the profile to prevent races between
concurrent nix-env operations on the same profile.  Fixes NIX-7.
2006-06-15 11:56:49 +00:00
Eelco Dolstra 23960e92df * Minor cleanup. 2006-06-01 18:13:33 +00:00
Eelco Dolstra 50fe85f016 * For fixed-output derivations, pass the environment variables listed
in the attribute variable `impureEnvVars' from the caller to the
  builder.
2006-05-31 09:51:45 +00:00
Eelco Dolstra 1390ce4142 * Not all platforms have sys/select.h. 2006-05-30 11:37:21 +00:00
Eelco Dolstra 58b4198ed8 * Disable the concurrent garbage collector on Cygwin for now. 2006-05-29 20:46:51 +00:00
Eelco Dolstra d764409d97 * Some Cygwin fixes. 2006-05-24 13:23:20 +00:00
Eelco Dolstra b5988004d6 * Support for srcdir != builddir (NIX-41). 2006-05-12 11:47:45 +00:00
Eelco Dolstra 9d72bf8835 * 64-bit compatibility fixes (for problems revealed by building on an Athlon
64 running 64-bit SUSE).  A patched ATerm library is required to run Nix
  succesfully.
2006-05-11 02:19:43 +00:00
Eelco Dolstra c791e94aee * Removed a bunch of ATreverses. 2006-05-02 17:51:50 +00:00
Eelco Dolstra fdea084c36 * Allow `make check' to work in directories that have symlink
components.
2006-03-10 22:27:26 +00:00
Eelco Dolstra c8bfb11b34 * `nix-env (-i|-u) --dry-run' now shows exactly which missing paths
will be built or substituted.
2006-03-06 11:21:15 +00:00
Eelco Dolstra 74166f2f44 * db.hh shouldn't depend on the Berkeley DB headers. 2006-03-01 17:44:28 +00:00
Eelco Dolstra d822bf32e4 * Close the database before the destructor runs. 2006-03-01 16:36:35 +00:00
Eelco Dolstra b602d2dfdf * Wrong delete. Thanks valgrind. 2006-03-01 14:39:10 +00:00
Eelco Dolstra 6b8bb8d74a * Remove dead code. 2006-03-01 12:51:39 +00:00
Eelco Dolstra 345a95afe9 * Allow the size of the GC reserved file to be specified in nix.conf
through the new `gc-reserved-space' option.
2006-02-16 13:58:10 +00:00
Eelco Dolstra 651ab439cf * A simple hack to fix NIX-18: the garbage collector cannot run when
the disk is full (because to delete something from the Nix store, we
  need a Berkeley DB transaction, which takes up disk space).  Under
  normal operation, we make sure that there exists a file
  /nix/var/nix/db/reserved of 1 MB.  When running the garbage
  collector, we delete that file before we open the Berkeley DB
  environment.
2006-02-16 13:19:15 +00:00
Eelco Dolstra 81de538e46 * Use setsid instead of setpgrp in child processes. This not only
creates a new process group but also a new session.  New sessions
  have no controlling tty, so child processes like ssh cannot open
  /dev/tty (which is bad).
2006-02-03 14:20:59 +00:00
Eelco Dolstra b90daaaf6c * When killing a build hook, kill the entire process group (as
intended).  This ensures that any ssh child processes to remote
  machines are also killed, and thus the Nix process on the remote
  machine also exits.  Without this, the remote Nix process will
  continue until it exists or until its stdout buffer gets full and it
  locks up.  (Partially fixes NIX-35.)
2006-02-02 16:27:31 +00:00
Eelco Dolstra 0f8d3c871b * More GCC 2.95 compatibility. 2005-12-25 11:29:12 +00:00
Eelco Dolstra 52d20ef124 * Hack around a GCC 2.95 bug. 2005-12-25 02:02:29 +00:00
Eelco Dolstra 1440419b45 * GCC 2.95 compatibility. 2005-12-24 23:32:59 +00:00
Eelco Dolstra f96d2dea26 * Added a flag --ignore-liveness' to nix-store --delete'. It
deletes a path even if it is reachable from a root.  However, it
  won't delete a path that still has referrers (since that would
  violate store invariants).

  Don't try this at home.  It's a useful hack for recovering from
  certain situations in a somewhat clean way (e.g., holes in closures
  due to disk corruption).
2005-12-23 21:36:44 +00:00
Eelco Dolstra 4b9e7f59ca * Revived the old "nix-store --delete" operation that deletes the
specified paths from the Nix store.  However, this operation is
  safe: it refuses to delete anything that the garbage collector
  wouldn't delete.
2005-12-23 21:08:42 +00:00
Eelco Dolstra 530b27df1e * `nix-store --gc' prints out the number of bytes freed on stdout
(even when it is interrupted by a signal).
2005-12-15 21:11:39 +00:00
Eelco Dolstra b1eed6b586 * Split the database upgrade into multiple transactions to prevent
Berkeley DB from running out of locks.
2005-12-15 16:53:21 +00:00
Eelco Dolstra ab5c6bb3a3 * Change referer' to referrer' throughout. In particular, the
nix-store query options `--referer' and `--referer-closure' have
  been changed to `--referrer' and `--referrer-closure' (but the old
  ones are still accepted for compatibility).
2005-12-13 21:04:48 +00:00
Eelco Dolstra d87549c1c7 * Automatically delete the old referers table. 2005-12-12 19:14:38 +00:00
Eelco Dolstra 8463f27d8c * Fix NIX-23: quadratic complexity in maintaining the referers
mapping.  The referer table is replaced by a referrer table (note
  spelling fix) that stores each referrer separately.  That is,
  instead of having

    referer[P] = {Q_1, Q_2, Q_3, ...}

  we store

    referer[(P, Q_1)] = ""
    referer[(P, Q_2)] = ""
    referer[(P, Q_3)] = ""
    ...

  To find the referrers of P, we enumerate over the keys with a value
  lexicographically greater than P.  This requires the referrer table
  to be stored as a B-Tree rather than a hash table.

  (The tuples (P, Q) are stored as P + null-byte + Q.)

  Old Nix databases are upgraded automatically to the new schema.
2005-12-12 18:24:42 +00:00
Eelco Dolstra a33b561a6b * Use Berkeley DB 4.4's process registry feature to recover from
crashed Nix instances, and toss out our own recovery code.
2005-12-09 22:55:07 +00:00
Eelco Dolstra 760264bffe * Require Berkeley DB 4.4.
* Checkpoint after an upgrade.
2005-12-06 15:00:04 +00:00
Eelco Dolstra b7f008fc35 * Did something useful while waiting at IAD: reference scanning is now
much faster.
2005-11-16 08:27:06 +00:00
Rob Vermaas f0856fd905 * Repair the referers table from the references table. 2005-10-29 15:44:02 +00:00
Eelco Dolstra 92d599c6a7 * Prevent uids from being used for more than one build
simultaneously.  We do this using exclusive locks on uid files in
  /nix/var/nix/userpool, e.g., /nix/var/nix/userpool/123 for uid 123.
2005-10-20 16:58:34 +00:00
Eelco Dolstra e932c40f8e * Oops. Fixed-output derivations were broken. 2005-10-19 14:27:44 +00:00
Eelco Dolstra 13b089c890 * Also kill all processes of the build user after the build. This is
critical to prevent certain kinds of 0wnage.
2005-10-17 17:43:21 +00:00
Eelco Dolstra f1b3a418fa * Before starting a build under some uid, kill all current processes
running under that uid.
2005-10-17 17:35:37 +00:00
Eelco Dolstra 439823ae80 * Check that the build result is owned by the build user, and that
nobody else has write permission to the build result.  This catches
  most hack attempts.
2005-10-17 16:59:25 +00:00
Eelco Dolstra 7ef574e5d0 * Don't use FIFOs to make Nix create the output path on behalf of the
builder.  Instead, require that the Nix store has sticky permission
  (S_ISVTX); everyone can created files in the Nix store, but they
  cannot delete, rename or modify files created by others.
2005-10-17 16:52:29 +00:00
Eelco Dolstra 32282abcea * Beginning of secure multi-user Nix stores. If Nix is started as
root (or setuid root), then builds will be performed under one of
  the users listed in the `build-users' configuration variables.  This
  is to make it impossible to influence build results externally,
  allowing locally built derivations to be shared safely between
  users (see ASE-2005 paper).

  To do: only one builder should be active per build user.
2005-10-17 15:33:24 +00:00
Eelco Dolstra ebfe57166d * Lets not go wild with templates. 2005-09-22 17:23:43 +00:00
Eelco Dolstra 4578a490ce * Parse multi-valued options. 2005-09-22 15:43:22 +00:00
Eelco Dolstra 8d06842a76 * Configuration options for trusted local builds. 2005-09-21 12:19:39 +00:00
Eelco Dolstra ef9e2c8e73 * Typo. 2005-09-20 16:14:00 +00:00
Eelco Dolstra 4c20a08293 * Build dynamic libraries. 2005-07-22 14:52:45 +00:00
Eelco Dolstra 109cde6706 * Ignore (with a warning) invalid garbage collector roots. 2005-05-10 14:56:10 +00:00
Eelco Dolstra 456f3251d2 * Typo. 2005-05-10 14:21:46 +00:00
Eelco Dolstra 8f57634c14 * Automatically upgrade the Berkeley DB environment if necessary. 2005-05-09 15:25:47 +00:00
Eelco Dolstra 6057b51835 * Don't try to register GC roots in read-only mode. 2005-05-07 21:33:31 +00:00
Eelco Dolstra 5dea0622d1 * Idem (constness fix).
* `compare' in GCC 2.95 is broken.
2005-05-04 16:31:49 +00:00
Eelco Dolstra 4a266e35d4 * GCC 2.95 compatibility fix in constness; strangely, I think this
should not have worked at all.
2005-05-04 16:31:24 +00:00
Eelco Dolstra f913283570 * Remove redundant message. 2005-04-13 09:20:27 +00:00
Eelco Dolstra f3660b1c8c * Garbage collector fix: allow deletion of paths that have invalid
(but substitutable) referers.
2005-04-12 10:51:00 +00:00
Eelco Dolstra cab7816b56 * Slightly nicer message. 2005-04-11 08:07:41 +00:00
Eelco Dolstra c815aff21b * `nix-store --add-fixed' to preload the outputs of fixed-output
derivations.  This is mostly to simplify the implementation of
  nix-prefetch-{url, svn}, which now work properly in setuid
  installations.

* Enforce valid store names in `nix-store --add / --add-fixed'.
2005-04-07 14:01:51 +00:00
Eelco Dolstra 298dd487bb * When finding live paths, the deriver need not be valid. 2005-03-25 14:31:12 +00:00
Eelco Dolstra 7eaf038763 * `nix-store --verify': repair bad referer mappings. 2005-03-25 14:21:49 +00:00
Eelco Dolstra c6178f0b03 * Create missing log and temproots directories automatically (reported
by Rob).
2005-03-24 17:46:38 +00:00
Eelco Dolstra 0df9f08078 * Export the references graph to the build hook. 2005-03-23 13:16:36 +00:00
Eelco Dolstra 3f236f01ae * `nix-store --register-validity': allow a path to refer to a path
listed later in the list of new valid paths.
2005-03-23 13:07:28 +00:00
Eelco Dolstra f20f081560 * nix-store: --isvalid' -> --check-validity', `--validpath' ->
`--register-validity'.
* `nix-store --register-validity': read arguments from stdin, and
  allow the references and deriver to be set.
2005-03-23 11:25:20 +00:00
Eelco Dolstra c757d16c8c * Bug in clearSubstitutes(). 2005-03-14 18:54:40 +00:00
Eelco Dolstra 4bbdcfbb45 * Don't use fork() in copyPath(), but a string buffer. 2005-03-03 13:58:02 +00:00
Eelco Dolstra 86cb3cc554 * Increase Berkeley DB limits a bit more.
* Maintain the cleanup invariant in clearSubstitutes().
2005-03-03 13:10:44 +00:00
Eelco Dolstra 07b4399fb6 * `nix-store -q --hash' to quickly query the hash of the contents of a
store path (which is stored in the database).
2005-03-02 15:57:06 +00:00
Eelco Dolstra bfaf83a0fd * When multiple derivations are specified in `nix-store -r', don't
continue building when one fails unless `--keep-going' is
  specified.
* When `--keep-going' is specified, print out the set of failing
  derivations at the end (otherwise it can be hard to find out which
  failed).
2005-02-23 11:19:27 +00:00
Eelco Dolstra 3a2c3f0cf2 * Support for fixed-output hashes over directory trees (i.e., over the
NAR dump of the path).
2005-02-22 21:14:41 +00:00
Eelco Dolstra eda2c3c253 * Compatibility hack so that Nixpkgs can continue to do hash checking
in `fetchurl' in Nix <= 0.7, but doesn't in Nix 0.8.
2005-02-22 15:23:24 +00:00
Eelco Dolstra 3c1630131e * Subtle bug in the builder: if a subgoal that is instantiated
multiple times is also a top-level goal, then the second and later
  instantiations would never be created because there would be a
  stable pointer to the first one that would keep it alive in the
  WeakGoalMap.
* Some tracing code for debugging this kind of problem.
2005-02-18 09:50:20 +00:00
Eelco Dolstra 8a3a96dd5b * Switch to the calling user context for some more operations in a
setuid installation.
2005-02-17 13:55:18 +00:00
Eelco Dolstra 88273f9574 * Put build logs in $prefix/var/nix/log/drvs/. 2005-02-17 13:54:45 +00:00
Eelco Dolstra 8992fce3da * It is now possible to add store derivations or paths directly to a
user environment, e.g.,

    $ nix-env -i /nix/store/z58v41v21xd3ywrqk1vmvdwlagjx7f10-aterm-2.3.1.drv

  or 

    $ nix-env -i /nix/store/hsyj5pbn0d9iz7q0aj0fga7cpaadvp1l-aterm-2.3.1

  This is useful because it allows Nix expressions to be bypassed
  entirely.  For instance, if only a nix-pull manifest is provided,
  plus the top-level path of some component, it can be installed
  without having to supply the Nix expression (e.g., for obfuscation,
  or to be independent of Nix expression language changes or context
  dependencies).
2005-02-14 17:35:10 +00:00
Eelco Dolstra a04a5de8f7 * Implement the `gc-keep-derivations' global configuretion flag. 2005-02-14 14:16:56 +00:00
Eelco Dolstra 6a8ef36fe6 * Global configuration option `env-keep-derivations' to store pointer
to derivations in user environments.  Nice for developers (since it
  prevents build-time-only dependencies from being GC'ed, in
  conjunction with `gc-keep-outputs').  Turned off by default.
2005-02-14 13:07:09 +00:00
Eelco Dolstra 32429142cd * Type error in constructor call (caught by GCC 3.3, but not 3.4!). 2005-02-14 09:53:11 +00:00
Eelco Dolstra 80870d9291 * Input sources should be in the set of all referenceable paths too. 2005-02-11 16:03:47 +00:00
Eelco Dolstra 3a99616968 * Commit more often to prevent out-of-memory errors. 2005-02-09 14:37:24 +00:00
Eelco Dolstra 98df735b51 * Propagate the deriver of a path through the substitute mechanism.
* Removed some dead code (successor stuff) from nix-push.
* Updated terminology in the tests (store expr -> drv path).
* Check that the deriver is set properly in the tests.
2005-02-09 12:57:13 +00:00
Eelco Dolstra 582e01c06f * Automatically upgrade <= 0.7 Nix stores to the new schema (so that
existing user environments continue to work).
* `nix-store --verify': detect incomplete closures.
2005-02-09 09:50:29 +00:00
Eelco Dolstra c547439843 * Subflag in --verify': nix-store --verify --check-contents' checks
that the contents of store paths has not changed by comparing hashes
  of their current contents to the hashes stored in the database.
2005-02-08 13:48:53 +00:00
Eelco Dolstra 3d74274b37 * Updated `nix-store --verify' to the new schema. 2005-02-08 13:23:55 +00:00
Eelco Dolstra fbc434ee4c * `nix-store -qb' to query derivation environment bindings. Useful
for finding build-time dependencies (possibly after a build).  E.g.,

    $ nix-store -qb aterm $(nix-store -qd $(which strc))
    /nix/store/jw7c7s65n1gwhxpn35j9rgcci6ilzxym-aterm-2.3.1

* Arguments to nix-store can be files within store objects, e.g.,
  /nix/store/jw7c...-aterm-2.3.1/bin/baffle.

* Idem for garbage collector roots.
2005-02-07 14:32:44 +00:00
Eelco Dolstra 450c358e20 * Maintain a database table (`derivers') that maps output paths to the
derivation that produced them.
* `nix-store -qd PATH' prints out the derivation that produced a path.
2005-02-07 13:40:40 +00:00
Eelco Dolstra a37338815d * A GC setting `gc-keep-outputs' to specify whether output paths of
derivations should be kept.
2005-02-01 22:07:48 +00:00
Eelco Dolstra 65b6c8ab4c * Move root finding from nix-collect-garbage' to nix-store --gc'.
This was necessary becase root finding must be done after
  acquisition of the global GC lock.

  This makes `nix-collect-garbage' obsolete; it is now just a wrapper
  around `nix-store --gc'.

* Automatically remove stale GC roots (i.e., indirect GC roots that
  point to non-existent paths).
2005-02-01 15:05:32 +00:00
Eelco Dolstra 630ae0c9d7 * nix-build: use an indirection scheme to make it easier for users to
get rid of GC roots.  Nix-build places a symlink `result' in the
  current directory.  Previously, removing that symlink would not
  remove the store path being linked to as a GC root.  Now, the GC
  root created by nix-build is actually a symlink in
  `/nix/var/nix/gcroots/auto' to `result'.  So if that symlink is
  removed the GC root automatically becomes invalid (since it can no
  longer be resolved).  The root itself is not automatically removed -
  the garbage collector should delete dangling roots.
2005-02-01 13:48:46 +00:00
Eelco Dolstra dcc37c236c * nix-store, nix-instantiate: added an option `--add-root' to
immediately add the result as a permanent GC root.  This is the only
  way to prevent a race with the garbage collector.  For instance, the
  old style

    ln -s $(nix-store -r $(nix-instantiate foo.nix)) \
      /nix/var/nix/gcroots/result

  has two time windows in which the garbage collector can interfere
  (by GC'ing the derivation and the output, respectively).  On the
  other hand,

    nix-store --add-root /nix/var/nix/gcroots/result -r \
      $(nix-instantiate --add-root /nix/var/nix/gcroots/drv \
        foo.nix)

  is safe.

* nix-build: use `--add-root' to prevent GC races.
2005-02-01 12:36:25 +00:00
Eelco Dolstra 06b4424286 * Add missing files to dist.
* Fix GC and substitute bugs related to self-references.  Add a
  regression test.
2005-02-01 09:23:38 +00:00
Eelco Dolstra 32fa82a56a * Acquire a global GC lock to prevent new temporary root files from
being created after the garbage collector has read the temproots
  directory.  This blocks the creation of new processes, but the
  garbage collector could periodically release the GC lock to allow
  them to run.
2005-01-31 22:23:49 +00:00
Eelco Dolstra 89c9bc11ab * Add a test for a more subtle race: a process starting after the
temporary root files have been read but creating outputs before the
  store directory has been read.
2005-01-31 22:01:55 +00:00
Eelco Dolstra 207bdcbe86 * Automatically remove temporary root files. 2005-01-31 21:20:59 +00:00
Eelco Dolstra 252c9c91ab * Topologically sort paths under the references relation to ensure
that they are deleted in an order that maintains the closure
  invariant.
* Presence of a path in a temporary roots file does not imply that all
  paths in its closure are also present, so add the closure.
2005-01-31 14:00:43 +00:00
Eelco Dolstra 33c5d23b81 * Don't delete active lock files. 2005-01-31 12:19:53 +00:00
Eelco Dolstra 1328aa3307 * Start of concurrent garbage collection. Processes write temporary
roots to a per-process temporary file in /nix/var/nix/temproots
  while holding a write lock on that file.  The garbage collector
  acquires read locks on all those files, thus blocking further
  progress in other Nix processes, and reads the sets of temporary
  roots.
2005-01-31 10:27:25 +00:00
Eelco Dolstra 0ea8b6993a * Only invalidate paths when they are in fact valid. 2005-01-28 11:05:46 +00:00
Eelco Dolstra 8a3eef22e3 * Fix deadlock. 2005-01-27 17:48:14 +00:00
Eelco Dolstra c60a4943ba * Update referers mappings when updating/clearing the references
mapping.
* Do things in the right order in invalidatePath().
2005-01-27 16:18:39 +00:00
Eelco Dolstra c505702265 * Fix and simplify the garbage collector (it's still not concurrent,
though).  In particular it's now much easier to register a GC root.
  Just place a symlink to whatever store path it is that you want to
  keep in /nix/var/nix/gcroots.
2005-01-27 15:21:29 +00:00
Eelco Dolstra 59682e6188 * Make lock removal safe by signalling to blocked processes that the
lock they are waiting on has become stale (we do this by writing a
  meaningless token to the unlinked file).
2005-01-27 12:19:25 +00:00
Eelco Dolstra a24b78e9f1 * Maintain the references/referers relation also for derivations.
This simplifies garbage collection and `nix-store --query
  --requisites' since we no longer need to treat derivations
  specially.

* Better maintaining of the invariants, e.g., setReferences() can only
  be called on a valid/substitutable path.
2005-01-25 21:28:25 +00:00
Eelco Dolstra 2a2756b856 * Simplification: registerSubstitutes -> registerSubstitute. We no
longer need the former since there we no longer have the
  substitutes-rev table (which triggered a O(n^2) cost in updating
  them).
2005-01-25 20:27:40 +00:00
Eelco Dolstra 066da4ab85 * Really fix the substitute mechanism, i.e., ensure the closure
invariant by registering references through the manifest.
* Added a test for nix-pull.
2005-01-25 17:08:52 +00:00
Eelco Dolstra c6290e42bc * Fix the `--fallback' switch.
* Fix the substitutes tests.
2005-01-25 13:00:12 +00:00
Eelco Dolstra 581fc47783 * Fix the build hook mechanism; pass the pointer graph to the hook. 2005-01-25 11:55:43 +00:00
Eelco Dolstra 52bf9b86bb * In nix-store: added query `--referers-closure' that returns the
closure of the referers relation rather than the references
  relation, i.e., the set of all paths that directly or indirectly
  refer to the given path.  Note that contrary to the references
  closure this set is not fixed; it can change as paths are added to
  or removed from the store.
2005-01-25 11:18:03 +00:00
Eelco Dolstra 80faa2f98a * In nix-store: change --build' back to --realise'. Also brought
back the query flag `--force-realise'.
* Fixed some of the tests.
2005-01-25 10:55:33 +00:00
Eelco Dolstra 6a0a2d5593 * Terminology fixes. 2005-01-20 16:01:07 +00:00
Eelco Dolstra 05f0430de1 * Another change to low-level derivations. The last one this year, I
promise :-) This allows derivations to specify on *what* output
  paths of input derivations they are dependent.  This helps to
  prevent unnecessary downloads.  For instance, a build might be
  dependent on the `devel' and `lib' outputs of some library
  component, but not the `docs' output.
2005-01-20 14:10:19 +00:00
Eelco Dolstra e0f4e587c3 * Nix-store queries --references' and referers' to query the pointer
graph.  That is, `nix-store --query --references PATH' shows the set
  of paths referenced by PATH, and `nix-store --query --referers PATH'
  shows the set of paths referencing PATH.
2005-01-19 16:59:56 +00:00
Eelco Dolstra 96de272b48 * Renamed normalise.cc' -> build.cc', `storeexprs.cc' ->
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
  they have been built/added.  This is so that we can check whether
  the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
  holds.
2005-01-19 16:39:47 +00:00
Eelco Dolstra ef5f254a55 * `nix-store --build' now builds its arguments in parallel instead of
sequentially (within the limits set by `--jobs').  This should
  greatly improve the utilisation of the build farm when doing Nixpkgs
  builds.
2005-01-19 15:02:02 +00:00
Eelco Dolstra 06c77bf7a8 * Change extension .store' to .drv'.
* Re-enable `nix-store --query --requisites'.
2005-01-19 14:36:00 +00:00
Eelco Dolstra 863dcff6c5 * Started removing closure store expressions, i.e., the explicit
representation of closures as ATerms in the Nix store.  Instead, the
  file system pointer graph is now stored in the Nix database.  This
  has many advantages:

  - It greatly simplifies the implementation (we can drop the notion
    of `successors', and so on).

  - It makes registering roots for the garbage collector much easier.
    Instead of specifying the closure expression as a root, you can
    simply specify the store path that must be retained as a root.
    This could not be done previously, since there was no way to find
    the closure store expression containing a given store path.
    
  - Better traceability: it is now possible to query what paths are
    referenced by a path, and what paths refer to a path.
2005-01-19 11:16:11 +00:00
Eelco Dolstra 32aac8748a * Actually check that the result of fixed-output derivations matches
the specified hash.
2005-01-17 19:01:48 +00:00
Eelco Dolstra f3dc231250 * Removed the `id' attribute hack.
* Formalise the notion of fixed-output derivations, i.e., derivations
  for which a cryptographic hash of the output is known in advance.
  Changes to such derivations should not propagate upwards through the
  dependency graph.  Previously this was done by specifying the hash
  component of the output path through the `id' attribute, but this is
  insecure since you can lie about it (i.e., you can specify any hash
  and then produce a completely different output).  Now the
  responsibility for checking the output is moved from the builder to
  Nix itself.

  A fixed-output derivation can be created by specifying the
  `outputHash' and `outputHashAlgo' attributes, the latter taking
  values `md5', `sha1', and `sha256', and the former specifying the
  actual hash in hexadecimal or in base-32 (auto-detected by looking
  at the length of the attribute value).  MD5 is included for
  compatibility but should be considered deprecated.

* Removed the `drvPath' pseudo-attribute in derivation results.  It's
  no longer necessary.

* Cleaned up the support for multiple output paths in derivation store
  expressions.  Each output now has a unique identifier (e.g., `out',
  `devel', `docs').  Previously there was no way to tell output paths
  apart at the store expression level.

* `nix-hash' now has a flag `--base32' to specify that the hash should
  be printed in base-32 notation.

* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
  `md5'.

* `nix-prefetch-url' now prints out a SHA-1 hash in base-32.  (TODO: a
  flag to specify the hash.)
2005-01-17 16:55:19 +00:00
Eelco Dolstra d58a11e019 * Shorten SHA-256 hashes used in store path name generation to 160
bits, then encode them in a radix-32 representation (using digits
  and letters except e, o, u, and t).  This produces store paths like
  /nix/store/4i0zb0z7f88mwghjirkz702a71dcfivn-aterm-2.3.1.  The nice
  thing about this is that the hash part of the file name is still 32
  characters, as before with MD5.

  (Of course, shortening SHA-256 to 160 bits makes it no better than
  SHA-160 in theory, but hopefully it's a bit more resistant to
  attacks; it's certainly a lot slower.)
2005-01-14 16:04:03 +00:00
Eelco Dolstra 9530cc3170 * Start move towards SHA-256 hashes instead of MD5.
* Start cleaning up unique store path generation (they weren't always
  unique; in particular the suffix ("-aterm-2.2", "-builder.sh") was
  not part of the hash, therefore changes to the suffix would cause
  multiple store objects with the same hash).
2005-01-14 13:51:38 +00:00
Eelco Dolstra 7e8961f720 * Added SHA-1 support. nix-hash' now has an option --type sha1' to
select SHA-1 hashing.
2005-01-13 17:39:26 +00:00
Eelco Dolstra 73992371a3 * Refactoring to support SHA-1. 2005-01-13 15:44:44 +00:00
Eelco Dolstra a03397be4c * Cygwin compatibility. 2005-01-04 17:38:26 +00:00
Eelco Dolstra 8b9697e575 * An operation `nix-store --clear-substitutes' to remove all
registered substitute mappings.
2004-12-20 14:16:55 +00:00
Eelco Dolstra fa9259f5f5 * Simplify the substitute mechanism:
- Drop the store expression.  So now a substitute is just a
    command-line invocation (a program name + arguments).  If you
    register a substitute you are responsible for registering the
    expression that built it (if any) as a root of the garbage
    collector.
  - Drop the substitutes-rev DB table.
2004-12-20 13:43:32 +00:00
Eelco Dolstra 13f77276d1 * utime() follows symlinks, so don't change the mtime if the file is a
symlink.
2004-11-29 19:22:16 +00:00
Eelco Dolstra f17553a212 * Remove debug statement. 2004-11-29 15:30:44 +00:00
Eelco Dolstra 4115d8d8ce * Canonicalise metadata of all files in store paths (set the mtime to
0, set the mode to either 444 or 555, set the group to the default).
2004-11-29 15:09:29 +00:00
Eelco Dolstra cb7ccb528b * string2ATerm -> overloaded toATerm. 2004-11-03 18:12:03 +00:00
Eelco Dolstra a69534fc21 * Drop ATmake / ATMatcher also in handling store expressions. 2004-10-29 11:22:49 +00:00
Eelco Dolstra f4d44a0026 * Allow certain operations to succeed even if we don't have write
permission to the Nix store or database.  E.g., `nix-env -qa' will
  work, but `nix-env -qas' won't (the latter needs DB access).  The
  option `--readonly-mode' forces this mode; otherwise, it's only
  activated when the database cannot be opened.
2004-10-25 14:38:23 +00:00
Eelco Dolstra 1317242780 * Make store objects created by substitutes read-only. 2004-09-22 12:15:04 +00:00
Eelco Dolstra b357284a32 * Fallback didn't work for subderivations of an unnormalised the main
derivation, since NormalisationGoal would first run a
  NormalisationGoal on the subderivation (a no-op, since in a
  situation where we need fallback the successor is known), and then
  runs a RealisationGoal on the normal form, which then cannot do a
  fallback because it doesn't know the derivation expression for which
  it is a normal form.

  Tossed out the 2-phase normalisation/realisation in
  NormalisationGoal and SubstitutionGoal since it's no longer needed -
  a RealisationGoal will run a NormalisationGoal if necessary.
2004-09-12 19:08:57 +00:00
Eelco Dolstra c16be6ac92 * Remove write permission from store objects after they have been
added to the store.  Bug reported by Martin.
2004-09-09 21:19:20 +00:00
Eelco Dolstra 47f87072ad * A very dirty hack to make setuid installations a bit nicer to use.
Previously there was the problem that all files read by nix-env
  etc. should be reachable and readable by the Nix user.  So for
  instance building a Nix expression in your home directory meant that
  the home directory should have at least g+x or o+x permission so
  that the Nix user could reach the Nix expression.  Now we just
  switch back to the original user just prior to reading sources and
  the like.  The places where this happens are somewhat arbitrary,
  however.  Any scope that has a live SwitchToOriginalUser object in
  it is executed as the original user.

* Back out r1385.  setreuid() sets the saved uid to the new
  real/effective uid, which prevents us from switching back to the
  original uid.  setresuid() doesn't have this problem (although the
  manpage has a bug: specifying -1 for the saved uid doesn't leave it
  unchanged; an explicit value must be specified).
2004-09-09 21:12:53 +00:00
Eelco Dolstra 5c443b6550 * Main the `substitutes-rev' table again, but now in a way that
doesn't take \Theta(n^2) space/time complexity.
2004-08-31 16:13:10 +00:00
Eelco Dolstra c25f2883b1 * Quadruple the Berkeley DB locking limits to get rid of out of memory
errors while running `nix-store --verify'.
2004-08-31 10:50:08 +00:00
Eelco Dolstra fe122c5a15 * Removed nrWaitees field. It was redundant with waitees.size() and
could get out of sync if multiple input derivations mapped to the
  same closure expression (since waitees is a set).
2004-08-30 11:51:36 +00:00
Eelco Dolstra eb233e728f * `--min-age' flag in nix-store and nix-collect-garbage to only delete
unreachable paths that haven't been used for N hours.  For instance,
  `nix-collect-garbage --min-age 168' only deletes paths that haven't
  been accessed in the last week.

  This is useful for instance in the build farm where many derivations
  can be shared between consecutive builds, and we wouldn't want a
  garbage collect to throw them all away.  We could of course register
  them as roots, but then we'd to unregister them at some point, which
  would be a pain to manage.  The `--min-age' flag gives us a sort of
  MRU caching scheme.

  BUG: this really shouldn't be in gc.cc since that violates
  mechanism/policy separation.
2004-08-25 16:54:08 +00:00
Eelco Dolstra 818047881e * Put the garbage collector in nix-store: operation `--gc',
suboperations `--print-live', `--print-dead', and `--delete'.  The
  roots are not determined by nix-store; they are read from standard
  input.  This is to make it easy to customise what the roots are.

  The collector now no longer fails when store expressions are missing
  (which legally happens when using substitutes).  It never tries to
  fetch paths through substitutes.

  TODO: acquire a global lock on the store while garbage collecting.
  
* Removed `nix-store --delete'.
2004-08-25 11:43:49 +00:00
Eelco Dolstra 1eddee59f2 * The default verbosity level of all Nix commands is now lvlInfo.
* Builder output is written to standard error by default.
  * The option `-B' is gone.
  * The option `-Q' suppresses builder output.

The result of this is that most Nix invocations shouldn't need any
flags w.r.t. logging.
2004-08-18 12:19:06 +00:00
Eelco Dolstra 62fe5c4a22 * The predecessor of a successor need not be present. This in
particular happens on distributed builds or when using push/pull.
2004-08-11 19:03:13 +00:00
Eelco Dolstra ae1a1efa41 * Clean up the temporary directory for hook communication (and don't
print out incorrect "build failed" messages).
2004-08-05 14:53:27 +00:00
Eelco Dolstra e3a50f7e25 * Creating a file nix-support/no-scan in the output path of a
derivation disables scanning for dependencies.  Use at your own
  risk.  This is a quick hack to speed up UML image generation (image
  are very big, say 1 GB).

  It would be better if the scanner were faster, and didn't read the
  whole file into memory.
2004-08-04 09:25:21 +00:00
Eelco Dolstra c1a18f543e * Fixed format string error. 2004-07-06 11:21:34 +00:00
Eelco Dolstra 056cd1d3b7 * Don't go into a (sometimes infinite) loop calling the build hook. 2004-07-01 16:24:35 +00:00
Eelco Dolstra b584253af4 * Include some missing headers. 2004-07-01 11:11:16 +00:00
Eelco Dolstra f5d5ffe536 * Write build logs to disk again. 2004-06-29 09:41:50 +00:00
Eelco Dolstra 4d2946c516 * In a realisation goal, check the result of the corresponding
normalisation goal.
2004-06-28 13:51:24 +00:00
Eelco Dolstra 91dc023665 * Added a switch `--fallback'. From the manual:
Whenever Nix attempts to realise a derivation for which a closure is
  already known, but this closure cannot be realised, fall back on
  normalising the derivation.

  The most common scenario in which this is useful is when we have
  registered substitutes in order to perform binary distribution from,
  say, a network repository.  If the repository is down, the
  realisation of the derivation will fail.  When this option is
  specified, Nix will build the derivation instead.  Thus, binary
  installation falls back on a source installation.  This option is
  not the default since it is generally not desirable for a transient
  failure in obtaining the substitutes to lead to a full build from
  source (with the related consumption of resources).
2004-06-28 10:42:57 +00:00
Eelco Dolstra b113edeab7 * A flag `--keep-going / -k' to keep building goals if one fails, as
much as possible.  (This is similar to GNU Make's `-k' flag.)

* Refactoring to implement this: previously we just bombed out when
  a build failed, but now we have to clean up.  In particular this
  means that goals must be freed quickly --- they shouldn't hang
  around until the worker exits.  So the worker now maintains weak
  pointers in order not to prevent garbage collection.

* Documented the `-k' and `-j' flags.
2004-06-25 15:36:09 +00:00
Eelco Dolstra e4883211f9 * Don't throw an exception when a build fails. Just terminate the
goal and allow the problem to be handled elsewhere (e.g., at
  top-level).
2004-06-25 10:21:44 +00:00
Eelco Dolstra 795d9f8b08 * Obsolete. 2004-06-24 14:36:50 +00:00
Eelco Dolstra ec32627621 * Multiple and/or failing substitutes now work. 2004-06-24 13:40:38 +00:00
Eelco Dolstra 66c7f34759 * Arghhhhhh 2004-06-22 17:07:32 +00:00
Eelco Dolstra 05a5362d63 * Some more diagnostics changes. 2004-06-22 17:04:10 +00:00
Eelco Dolstra 83ae1723da * Well, it's better than printf. 2004-06-22 15:01:01 +00:00
Eelco Dolstra 72bc9a522f * Started making Nix's diagnostic messages a bit more useful. 2004-06-22 14:48:59 +00:00
Eelco Dolstra 5e2cf44a4d * Put WEXITSTATUS stuff somewhere else. 2004-06-22 11:03:41 +00:00
Eelco Dolstra 84007a0958 * Reduce gratuitous cut & pasting. 2004-06-22 10:21:44 +00:00
Eelco Dolstra c9fbd2dfd5 * Wrapper class around pids. 2004-06-22 09:51:44 +00:00
Eelco Dolstra 155d7c8dfa * Substitutes should occupy a build slot. 2004-06-22 09:00:31 +00:00
Eelco Dolstra c4cb6ea2bc * Refactoring. 2004-06-22 08:50:25 +00:00
Eelco Dolstra 3f3a3ae87b * Acquire a lock on the output path when running a substitute. Also
delete obstructing invalid paths.
2004-06-21 09:35:50 +00:00
Eelco Dolstra 72c857f0eb * Ugh, nasty Heisenbug due to an uninitialiased variable. The bug
only caused a crash if the program was *not* invoked with a high
  verbosity level.
2004-06-21 08:51:55 +00:00
Eelco Dolstra be1a917beb * Remove obstructing invalid store paths add[Text]ToStore(). 2004-06-21 07:46:02 +00:00
Eelco Dolstra daf0a923c7 * Wrap calls to registerSubstitute() in a single transaction to
improve throughput.
* Don't build the `substitute-rev' table for now, since it caused
  Theta(N^2) time and log file consumption when adding N substitutes.
  Maybe we can do without it.
2004-06-21 07:38:17 +00:00
Eelco Dolstra 15c60ca1b6 * Disable calls to fsync() since Berkeley DB's DB_TXN_WRITE_NOSYNC
flag doesn't seem to work as advertised.
2004-06-21 07:36:01 +00:00
Eelco Dolstra 112ee89501 * Re-enable support for substitutes in the normaliser.
* A better substitute mechanism.

  Instead of generating a store expression for each store path for
  which we have a substitute, we can have a single store expression
  that builds a generic program that is invoked to build the desired
  store path, which is passed as an argument.

  This means that operations like `nix-pull' only produce O(1) files
  instead of O(N) files in the store when registering N substitutes.
  (It consumes O(N) database storage, of course, but that's not a
  performance problem).

* Added a test for the substitute mechanism.
  
* `nix-store --substitute' reads the substitutes from standard input,
  instead of from the command line.  This prevents us from running
  into the kernel's limit on command line length.
2004-06-20 19:17:54 +00:00
Eelco Dolstra 85ae781765 * Refactoring. 2004-06-20 13:37:51 +00:00
Eelco Dolstra 23bb902d1f * Re-enable build hooks. 2004-06-19 21:45:04 +00:00
Eelco Dolstra 41ec982f31 * Big refactoring. Move to a much more explicitly state machine based
approach.  This makes it much easier to add extra complexity in the
  normaliser / realiser (e.g., build hooks, substitutes).
2004-06-18 18:09:32 +00:00
Eelco Dolstra 0b70231b9d * Refactoring. 2004-06-15 13:49:42 +00:00
Eelco Dolstra 1bc6afefac * Cleanup. 2004-06-08 13:21:03 +00:00
Eelco Dolstra 5e4a2272bf * Drain the output of the build hook to show error messages. Ugly
hack.
2004-05-18 14:52:35 +00:00
Eelco Dolstra 8e9fd57ef9 * setpgrp() is not POSIX (and on Mac OS X it's different than on
Linux), so use setpgid().
2004-05-18 09:45:18 +00:00
Eelco Dolstra ace8872706 * execl() requires a terminating 0.
* When a fast build wakes up a goal, try to start that goal in the
  same iteration of the startBuild() loop of run().  Otherwise no job
  might be started until the next job terminates.
2004-05-14 12:24:29 +00:00
Eelco Dolstra 4fc00cbec1 * Distributed builds and load balancing now seem to work pretty well.
(Though the `build-remote.pl' script has a gigantic race condition).
2004-05-13 22:52:37 +00:00
Eelco Dolstra 25db622454 * Load balancing. `build-remote.pl' will only execute up to a
per-machine maximum number of parallel jobs on a remote machine.
2004-05-13 19:35:46 +00:00
Eelco Dolstra a8306cb98f * The build hooks used to implement distributed builds can now be run
in parallel.  Hooks are more efficient: locks on output paths are
  only acquired when the hook says that it is willing to accept a
  build job.  Hooks now work in two phases.  First, they should first
  tell Nix whether they are willing to accept a job.  Nix guarantuees
  that no two hooks will ever be in the first phase at the same time
  (this simplifies the implementation of hooks, since they don't have
  to perform locking (?)).  Second, if they accept a job, they are
  then responsible for building it (on the remote system), and copying
  the result back.  These can be run in parallel with other hooks and
  locally executed jobs.

  The implementation is a bit messy right now, though.  

* The directory `distributed' shows a (hacky) example of a hook that
  distributes build jobs over a set of machines listed in a
  configuration file.
2004-05-13 19:14:49 +00:00
Eelco Dolstra efa5fa1a91 * A switch `-j NUMBER' to set the maximum number of parallel jobs (0 =
no limit).
* Add missing file to distribution.
2004-05-12 14:20:32 +00:00
Eelco Dolstra aa5a5084e4 * Pass to the build hook all sorts of information useful for
distributing a build action to another machine.  In particular, the
  paths in the input closures, the output paths, and successor mapping
  for sub-derivations.
2004-05-12 13:32:26 +00:00
Eelco Dolstra 8c0b42f857 * An quick and dirty hack to support distributed builds. 2004-05-12 09:35:51 +00:00
Eelco Dolstra c8d3882cdc * True parallel builds. Nix can now run as many build jobs in
parallel as possible (similar to GNU Make's `-j' switch).  This is
  useful on SMP systems, but it is especially useful for doing builds
  on multiple machines.  The idea is that a large derivation is
  initiated on one master machine, which then distributes
  sub-derivations to any number of slave machines.  This should not
  happen synchronously or in lock-step, so the master must be capable
  of dealing with multiple parallel build jobs.  We now have the
  infrastructure to support this.

  TODO: substitutes are currently broken.
2004-05-11 18:05:44 +00:00
Eelco Dolstra a4d2b22c8c * Be stricter in verifying store paths. 2004-04-14 08:08:55 +00:00
Eelco Dolstra 7823db2137 * Some more nesting. 2004-03-22 21:42:28 +00:00
Eelco Dolstra a5619f1dff * Set the NIX_STORE and NIX_BUILD_TOP environment variables in
builders to point to the store and the temporary build directory,
  respectively.  Useful for purity checking.
* Also set TEMPDIR, TMPDIR, TEMP, and TEMP to NIX_BUILD_TOP to make
  sure that tools in the builder store temporary files in the right
  location.
2004-03-12 10:45:08 +00:00
Eelco Dolstra 0dfdafdf6d * Allow linking against an external Berkeley DB / ATerm library. 2004-02-16 16:37:16 +00:00
Eelco Dolstra 76c0e85929 * The environment variable NIX_ROOT can now be set to execute Nix in a
chroot() environment.
* A operation `--validpath' to register path validity.  Useful for
  bootstrapping in a pure Nix environment.
* Safety checks: ensure that files involved in store operations are in
  the store.
2004-02-14 21:44:18 +00:00
Eelco Dolstra 6f5a5ea5ea * Regression fix: realise substitutes and detect cycles. 2004-02-13 10:45:09 +00:00
Eelco Dolstra 447089a5f6 * Catch SIGINT to terminate cleanly when the user tries to interrupt
Nix.  This is to prevent Berkeley DB from becoming wedged.

  Unfortunately it is not possible to throw C++ exceptions from a
  signal handler.  In fact, you can't do much of anything except
  change variables of type `volatile sig_atomic_t'.  So we set an
  interrupt flag in the signal handler and check it at various
  strategic locations in the code (by calling checkInterrupt()).
  Since this is unlikely to cover all cases (e.g., (semi-)infinite
  loops), sometimes SIGTERM may now be required to kill Nix.
2004-01-15 20:23:55 +00:00
Eelco Dolstra ff9af107d3 * Option `-B' to always show the output of builders, regardless of
verbosity level.
2004-01-13 16:35:43 +00:00
Eelco Dolstra 3495d153b3 * Periodically checkpoint the log. 2004-01-13 13:37:25 +00:00
Eelco Dolstra 698e880c9f * Tricky: make sure that the accessor count is not reset to 0 if
recovery fails.
2004-01-13 12:36:43 +00:00
Eelco Dolstra 23fbc72f5d * Print error messages, not debug messages. 2004-01-13 11:53:12 +00:00
Eelco Dolstra 4c4fe7a114 * Changed the extension for store expressions from ".nix" to ".store"
(following the Usenix paper).
2004-01-12 10:44:48 +00:00
Eelco Dolstra 7959354379 * Upgraded to Berkeley DB 4.2.52. The main advantage of 4.2 is that
it automatically removes log files when they are no longer needed.

  *** IMPORTANT ***

  If you have an existing Nix installation, you must checkpoint the
  Nix database to prevent recent transactions from being undone.  Do
  the following:

  - optional: make a backup of $prefix/var/nix/db.

  - run `db_checkpoint' from Berkeley DB 4.1:
  
    $ db_checkpoint -h $prefix/var/nix/db -1

  - optional (?): run `db_recover' from Berkeley DB 4.1:

    $ db_recover -h $prefix/var/nix/db

  - remove $prefix/var/nix/db/log* and $prefix/var/nix/db/__db*
2004-01-07 15:53:14 +00:00
Eelco Dolstra 4a373a3e9a * Implemented Eelco V.'s `nix-env -I' command to specify the default
path of the Nix expression to be used with the import, upgrade, and
  query commands.  For instance,

  $ nix-env -I ~/nixpkgs/pkgs/system/i686-linux.nix

  $ nix-env --query --available   [aka -qa]
  sylpheed-0.9.7
  bison-1.875
  pango-1.2.5
  subversion-0.35.1
  ...

  $ nix-env -i sylpheed

  $ nix-env -u subversion

  There can be only one default at a time.

* If the path to a Nix expression is a symlink, follow the symlink
  prior to resolving relative path references in the expression.
2004-01-05 16:26:43 +00:00
Eelco Dolstra 833f2fc92d * GCC 2.95 compatibility. 2003-12-22 16:40:46 +00:00
Eelco Dolstra 528f1d1867 * Bug fix: parallel builds of the same derivation failed due to lock file removal. 2003-12-21 17:09:16 +00:00
Eelco Dolstra cff6fd22eb * Allow successors that don't exist but have a substitute.
* Integrity: check in successor / substitute registration whether
  the target path exists or has a substitute.
2003-12-05 11:05:19 +00:00
Eelco Dolstra 5d2b424804 * Use a system name that does not include the OS manufacturer (i.e.,
"i686-linux" instead of "i686-suse-linux").
2003-12-01 14:36:50 +00:00
Eelco Dolstra 66c115ef5f * More `make dist' fixes. 2003-11-25 13:01:21 +00:00
Eelco Dolstra c3ee8c9166 * `make dist' fix. 2003-11-25 12:35:52 +00:00
Eelco Dolstra c9cb1fa21f * Bug fix in path invalidation.
* More consistency checks.
2003-11-24 09:24:52 +00:00
Eelco Dolstra ab0bc4999a * Maintain integrity of the substitute and successor mappings when
deleting a path in the store.
* Allow absolute paths in Nix expressions.
* Get nix-prefetch-url to work again.
* Various other fixes.
2003-11-22 18:45:56 +00:00
Eelco Dolstra 40d9eb14df * Fix the garbage collector. 2003-11-22 15:58:34 +00:00
Eelco Dolstra 7a02d95418 * Remove lock files after building. 2003-11-21 16:05:19 +00:00
Eelco Dolstra 9898746ef3 * nix-env: a tool to manage user environments.
* Replace all directory reading code by a generic readDirectory()
  function.
2003-11-19 17:27:16 +00:00
Eelco Dolstra ce92d1bf14 * "Nix expression" -> "store expression".
* More refactoring.
2003-11-18 11:22:29 +00:00
Eelco Dolstra 9f0f020929 * libnix -> libstore. 2003-11-18 10:55:27 +00:00