Commit graph

579 commits

Author SHA1 Message Date
Eelco Dolstra 6f5e3326ce * Move topoSortPaths() out of gc.cc. 2011-12-30 15:02:50 +00:00
Eelco Dolstra b1004f40f7 * Reject a build if there is a cycle among the outputs. This is
necessary because existing code assumes that the references graph is
  acyclic.
2011-12-30 14:47:14 +00:00
Eelco Dolstra 254b3399ba * Sync with the trunk. 2011-12-30 13:08:14 +00:00
Eelco Dolstra 8c42a8c8ff * Make sure that lock files are cleaned up properly when building
through the build hook.
2011-12-25 16:38:37 +00:00
Eelco Dolstra 524fa8a4f1 * Oops. 2011-12-22 16:27:03 +00:00
Eelco Dolstra b33da599c5 * In the garbage collector, delete invalid paths before deleting
unreachable paths.  This matters when using --max-freed etc.:
  unreachable paths could become reachable again, so it's nicer to
  keep them if there is "real" garbage to be deleted.  Also, don't use
  readDirectory() but read the Nix store and delete invalid paths in
  parallel.  This reduces GC latency on very large Nix stores.
2011-12-22 15:55:53 +00:00
Eelco Dolstra 58d974336c * Drop unnecessary call to canonPath() (nixStore is already canonical). 2011-12-22 14:33:34 +00:00
Eelco Dolstra 2aac7cd021 * Another case of lock file permissions being too liberal. 2011-12-21 19:17:45 +00:00
Shea Levy b4cee3f816 Revert previous commit
It doesn't detect indirect references
2011-12-21 17:31:34 +00:00
Shea Levy f3c88f297d Detect and reject mutually-referential outputs
There is probably a more efficient way to do this.
2011-12-21 17:14:28 +00:00
Eelco Dolstra b5363810bb * Fix the build. 2011-12-20 16:37:01 +00:00
Eelco Dolstra 194d21f9f6 * Sync with the trunk. 2011-12-16 23:33:01 +00:00
Eelco Dolstra 273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Eelco Dolstra e0bd307802 * Make the import operation through the daemon much more efficient
(way fewer roundtrips) by allowing the client to send data in bigger
  chunks.
* Some refactoring.
2011-12-16 19:44:13 +00:00
Eelco Dolstra 5a1b9ed0aa * Refactoring: move sink/source buffering into separate classes.
* Buffer the HashSink.  This speeds up hashing a bit because it
  prevents lots of calls to the hash update functions (e.g. nix-hash
  went from 9.3s to 8.7s of user time on the closure of my
  /var/run/current-system).
2011-12-15 16:19:53 +00:00
Eelco Dolstra 3a48282b06 * Buffer writes in FdSink. This significantly reduces the number of
system calls / context switches when dumping a NAR and in the worker
  protocol.
2011-12-14 23:30:06 +00:00
Eelco Dolstra c8c0380744 * Remove unnecessary quotes. showPaths() already adds quotes. 2011-12-05 21:04:20 +00:00
Eelco Dolstra 000160f5b9 * In ‘nix-store --verify --check-contents’, repair missing hashes
rather than complain about them.
2011-12-02 17:52:18 +00:00
Eelco Dolstra 24f863d86b * When doing "nix-store --add-fixed" without "--recursive" via the Nix
daemon (which is an error), print a nicer error message than
  "Connection reset by peer" or "broken pipe".
* In the daemon, log errors that occur during request parameter
  processing.
2011-12-01 13:48:48 +00:00
Eelco Dolstra b1eb8f4249 * Get rid of some superfluous error messages if a substituter fails.
* Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
Eelco Dolstra 5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra 993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Rob Vermaas 4e1ea17052 nix: add /etc/hosts with localhost entry to chroot builds. 2011-11-21 15:19:51 +00:00
Eelco Dolstra c0b706213d * Boldly make SQLite WAL mode the default again. Hopefully the
intermittent problems are gone by now.  WAL mode is preferrable
  because it does way fewer fsyncs.
2011-11-07 21:11:59 +00:00
Shea Levy af2e53fd48 Include all outputs of derivations in the closure of explicitly-passed derivation paths
This required adding a queryOutputDerivationNames function in the store API
2011-11-06 06:28:20 +00:00
Eelco Dolstra 13114daa3e * Ouch. A store upgrade could cause a substituter to be triggered,
causing a deadlock.
2011-09-12 09:07:43 +00:00
Eelco Dolstra 93227ff65c * Eliminate all uses of the global variable ‘store’ from libstore.
This should also fix:

    nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.

  which was caused by hashDerivationModulo() calling the ‘store’
  object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Eelco Dolstra b2027f70d9 * Fix a huuuuge security hole in the Nix daemon. It didn't check that
derivations added to the store by clients have "correct" output
  paths (meaning that the output paths are computed by hashing the
  derivation according to a certain algorithm).  This means that a
  malicious user could craft a special .drv file to build *any*
  desired path in the store with any desired contents (so long as the
  path doesn't already exist).  Then the attacker just needs to wait
  for a victim to come along and install the compromised path.

  For instance, if Alice (the attacker) knows that the latest Firefox
  derivation in Nixpkgs produces the path

    /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1

  then (provided this path doesn't already exist) she can craft a .drv
  file that creates that path (i.e., has it as one of its outputs),
  add it to the store using "nix-store --add", and build it with
  "nix-store -r".  So the fake .drv could write a Trojan to the
  Firefox path.  Then, if user Bob (the victim) comes along and does

    $ nix-env -i firefox
    $ firefox

  he executes the Trojan injected by Alice.

  The fix is to have the Nix daemon verify that derivation outputs are
  correct (in addValidPath()).  This required some refactoring to move
  the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
Ludovic Courtès 5c9e9f732d Add support for the build-timeout' and --timeout' options. 2011-06-30 15:19:13 +00:00
Eelco Dolstra 8c69dac8a1 * Handle error messages from the Nix worker containing the `%'
character.  (Nix/216)
2011-04-19 10:44:44 +00:00
Eelco Dolstra c931a7aec5 * Do a short sleep after SQLITE_BUSY. 2010-12-17 17:23:15 +00:00
Eelco Dolstra b1eb252172 * Propagate the "100" exit status for failed builds through the Nix
daemon.
2010-12-17 11:28:26 +00:00
Eelco Dolstra 3dd02580e3 * I forgot to catch SQLiteBusy in registerValidPaths(). So
registerValidPaths() now handles busy errors and registerValidPath()
  is simply a wrapper around it.
2010-12-14 13:25:20 +00:00
Eelco Dolstra d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra 5833243c92 * Create /nix/var/nix/db if it's missing. 2010-12-13 13:42:34 +00:00
Eelco Dolstra d7ca6f44eb * Update some comments. 2010-12-13 13:19:46 +00:00
Eelco Dolstra e4720b1a79 * Ignore the result of sqlite3_reset(). 2010-12-08 18:19:15 +00:00
Eelco Dolstra 8062d3af30 * `nix-store --verify --check-contents': don't hold the global GC lock
while checking the contents, since this operation can take a very
  long time to finish.  Also, fill in missing narSize fields in the DB
  while doing this.
2010-12-06 15:29:38 +00:00
Eelco Dolstra de79d23f76 * Retry a transaction if SQLite returns SQLITE_BUSY. This can happen
even with a very long busy timeout, because SQLITE_BUSY is also
  returned to resolve deadlocks.  This should get rid of random
  "database is locked" errors.  This is kind of hard to test though.
* Fix a horrible bug in deleteFromStore(): deletePathWrapped() should
  be called after committing the transaction, not before, because the
  commit might not succeed.
2010-12-05 18:23:19 +00:00
Eelco Dolstra bdf089f463 * Before a build, show the disk space that the downloaded store paths
will approximately require.
2010-11-17 14:31:42 +00:00
Eelco Dolstra e60c962fb8 * Add an operation `nix-store -q --size'. 2010-11-17 12:40:52 +00:00
Eelco Dolstra 1db6259076 * Implement RemoteStore::queryPathInfo(). 2010-11-17 12:08:01 +00:00
Eelco Dolstra a3883cbd28 * Store the size of a store path in the database (to be precise, the
size of the NAR serialisation of the path, i.e., `nix-store --dump
  PATH').  This is useful for Hydra.
2010-11-16 17:11:46 +00:00
Eelco Dolstra fb9368b5a0 * Sync with the trunk. 2010-11-16 12:49:47 +00:00
Eelco Dolstra 64fd29855a * Wrap deleteFromStore() in a transaction. Otherwise there might be a
race with other processes that add new referrers to a path,
  resulting in the garbage collector crashing with "foreign key
  constraint failed".  (Nix/4)
* Make --gc --print-dead etc. interruptible.
2010-10-14 15:55:51 +00:00
Eelco Dolstra 705868a8a9 * Make sure that config.h is included before the system headers,
because it defines _FILE_OFFSET_BITS.  Without this, on 
  OpenSolaris the system headers define it to be 32, and then 
  the 32-bit stat() ends up being called with a 64-bit "struct 
  stat", or vice versa.

  This also ensures that we get 64-bit file sizes everywhere.

* Remove the redundant call to stat() in parseExprFromFile().
  The file cannot be a symlink because that's the exit condition
  of the loop before.
2010-10-04 17:55:38 +00:00
Eelco Dolstra e4907411c2 * Only do "pragma journal_mode = ..." if the current journal mode
differs from the desired mode.  There is an open SQLite ticket
  `Executing "PRAGMA journal_mode" may delete journal file while it is
  in use.'
2010-09-01 11:36:22 +00:00
Eelco Dolstra bf0dde9597 * Always print hook output on stderr, even if --no-build-output is
set.
* In the build hook, print a trace message to allow Hydra to pick up
  the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
Eelco Dolstra e2e168f7c2 `nix-store --verify' improvements:
* If a path has disappeared, check its referrers first, and don't try
  to invalidate paths that have valid referrers.  Otherwise we get a
  foreign key constraint violation.
* Read the whole Nix store directory instead of statting each valid
  path, which is slower.
* Acquire the global GC lock.
2010-08-31 11:47:31 +00:00
Eelco Dolstra 80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00