Commit graph

1562 commits

Author SHA1 Message Date
Eelco Dolstra e9fc91df45 Fix error message
This fixes the error message

  error: file `' was not found in the Nix search path (add it using $NIX_PATH or -I)
2012-02-09 18:56:48 +01:00
Eelco Dolstra d5a5a83ad4 Use data() instead of c_str() where appropriate 2012-02-09 18:27:45 +01:00
Eelco Dolstra f491ae97d4 * Inline some functions and get rid of the indirection through
EvalState::eval().  This gives a 12% speedup on ‘nix-instantiate
  /etc/nixos/nixos/ -A system --readonly-mode’ (from 1.01s to 0.89s).
2012-02-04 13:50:25 +00:00
Eelco Dolstra 2bda12ef3b * Print elapsed evaluation time. 2012-02-04 13:27:11 +00:00
Eelco Dolstra baa0501cc1 * Fix importing a derivation. This gave a segfault. 2012-01-26 13:13:00 +00:00
Eelco Dolstra 330df4b4db * Allow comparisons between derivations by comparing the outPath
attributes.
2012-01-19 23:08:47 +00:00
Eelco Dolstra 18047d4625 * Add some debug output to print the derivation name once it's known.
This makes it easier to pinpoint the source of a crash.
2012-01-19 23:07:42 +00:00
Eelco Dolstra 4e624849b6 * Added a command ‘nix-store --print-env $drvpath’ that prints out the
environment of the given derivation in a format that can be sourced
  by the shell, e.g.

  $ eval "$(nix-store --print-env $(nix-instantiate /etc/nixos/nixpkgs -A pkg))"
  $ NIX_BUILD_TOP=/tmp
  $ source $stdenv/setup

  This is especially useful to reproduce the environment used to build
  a package outside of its builder for development purposes.

  TODO: add a nix-build option to do the above and fetch the
  dependencies of the derivation as well.
2012-01-17 23:07:22 +00:00
Eelco Dolstra 9fe24c5a0d * Don't create thunks for simple constants (integers, strings, paths)
and allocate them only once.
* Move Value and related functions into value.hh.
2012-01-07 17:26:33 +00:00
Eelco Dolstra b52966e821 * Remove dead code. 2012-01-04 21:47:48 +00:00
Eelco Dolstra 35f2a6ba82 * Don't use dynamic_cast, it's very slow. "nix-instantiate
/etc/nixos/nixos -A system" spent about 10% of its time in
  dynamic_cast.
2012-01-04 21:24:11 +00:00
Eelco Dolstra 71f3c46cf6 * Drop the inefficient "Path" suffix in output attribute names. 2012-01-03 15:27:18 +00:00
Eelco Dolstra 921111d197 * Move the implementation of the ‘derivation’ primop into a separate
file.
2012-01-03 14:01:47 +00:00
Eelco Dolstra 6c31232e14 * Sync with the trunk. 2012-01-03 12:59:31 +00:00
Eelco Dolstra 48cea0d01e * Refactoring: Get rid of a few subdirectories in corepkgs/, and some
other simplifications.
* Use <nix/...> to locate the corepkgs.  This allows them to be
  overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
  to create a temporary file.
2012-01-03 00:16:29 +00:00
Eelco Dolstra 93b56acb2d * Support multiple outputs in nix-store (specifically the ‘--query’
and ‘--realise’ actions).
2011-12-30 17:25:19 +00:00
Eelco Dolstra a71d02440b * Oops. 2011-12-30 17:13:25 +00:00
Eelco Dolstra 6f5e3326ce * Move topoSortPaths() out of gc.cc. 2011-12-30 15:02:50 +00:00
Eelco Dolstra b1004f40f7 * Reject a build if there is a cycle among the outputs. This is
necessary because existing code assumes that the references graph is
  acyclic.
2011-12-30 14:47:14 +00:00
Eelco Dolstra 254b3399ba * Sync with the trunk. 2011-12-30 13:08:14 +00:00
Eelco Dolstra 8c42a8c8ff * Make sure that lock files are cleaned up properly when building
through the build hook.
2011-12-25 16:38:37 +00:00
Eelco Dolstra 524fa8a4f1 * Oops. 2011-12-22 16:27:03 +00:00
Eelco Dolstra b33da599c5 * In the garbage collector, delete invalid paths before deleting
unreachable paths.  This matters when using --max-freed etc.:
  unreachable paths could become reachable again, so it's nicer to
  keep them if there is "real" garbage to be deleted.  Also, don't use
  readDirectory() but read the Nix store and delete invalid paths in
  parallel.  This reduces GC latency on very large Nix stores.
2011-12-22 15:55:53 +00:00
Eelco Dolstra 58d974336c * Drop unnecessary call to canonPath() (nixStore is already canonical). 2011-12-22 14:33:34 +00:00
Eelco Dolstra 2aac7cd021 * Another case of lock file permissions being too liberal. 2011-12-21 19:17:45 +00:00
Eelco Dolstra 5679041132 * The ‘foo.drvPath’ feature was already broken in read-only mode.
Since it's rarely used and fixing it is too much work right now,
  just document it.
2011-12-21 18:19:05 +00:00
Shea Levy b4cee3f816 Revert previous commit
It doesn't detect indirect references
2011-12-21 17:31:34 +00:00
Shea Levy f3c88f297d Detect and reject mutually-referential outputs
There is probably a more efficient way to do this.
2011-12-21 17:14:28 +00:00
Eelco Dolstra b19a0f63db * Simplify the context handling logic. 2011-12-21 15:33:30 +00:00
Eelco Dolstra b5363810bb * Fix the build. 2011-12-20 16:37:01 +00:00
Eelco Dolstra 194d21f9f6 * Sync with the trunk. 2011-12-16 23:33:01 +00:00
Eelco Dolstra 273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Eelco Dolstra 8d3dfa2c17 * Avoid expensive conversions from char arrays to STL strings. 2011-12-16 21:29:46 +00:00
Eelco Dolstra e0bd307802 * Make the import operation through the daemon much more efficient
(way fewer roundtrips) by allowing the client to send data in bigger
  chunks.
* Some refactoring.
2011-12-16 19:44:13 +00:00
Eelco Dolstra 78598d06f0 * Clean up exception handling. 2011-12-16 15:45:42 +00:00
Eelco Dolstra 5a1b9ed0aa * Refactoring: move sink/source buffering into separate classes.
* Buffer the HashSink.  This speeds up hashing a bit because it
  prevents lots of calls to the hash update functions (e.g. nix-hash
  went from 9.3s to 8.7s of user time on the closure of my
  /var/run/current-system).
2011-12-15 16:19:53 +00:00
Eelco Dolstra a3e0656cbb * Buffer reads in FdSource. Together with write buffering, this
significantly cuts down the number of syscalls (e.g., for "nix-store
  -qR /var/run/current-system" via the daemon, it reduced the number
  of syscalls in the client from 29134 to 4766 and in the daemon from
  44266 to 20666).
2011-12-15 12:32:08 +00:00
Eelco Dolstra 3a48282b06 * Buffer writes in FdSink. This significantly reduces the number of
system calls / context switches when dumping a NAR and in the worker
  protocol.
2011-12-14 23:30:06 +00:00
Eelco Dolstra 893cac1402 * Remove the terminate handler, which was only really needed because
of Berkeley DB (see r8632).
2011-12-14 22:41:10 +00:00
Eelco Dolstra c8c0380744 * Remove unnecessary quotes. showPaths() already adds quotes. 2011-12-05 21:04:20 +00:00
Eelco Dolstra 000160f5b9 * In ‘nix-store --verify --check-contents’, repair missing hashes
rather than complain about them.
2011-12-02 17:52:18 +00:00
Eelco Dolstra 49f59dceca * Move parseHash16or32 into libutil, and use in nix-hash. 2011-12-02 11:47:06 +00:00
Eelco Dolstra b12b21825c * Allow '<nixexpr>' syntax to be used in nix-instantiate, nix-build
and nix-env, e.g.,

  $ nix-env -f '<nixpkgs>' -i patchelf

  or

  $ nix-build '<nixos/tests>' -A login.test
2011-12-01 16:41:43 +00:00
Eelco Dolstra 23c38a04cc * Slight improvement. 2011-12-01 13:51:05 +00:00
Eelco Dolstra 24f863d86b * When doing "nix-store --add-fixed" without "--recursive" via the Nix
daemon (which is an error), print a nicer error message than
  "Connection reset by peer" or "broken pipe".
* In the daemon, log errors that occur during request parameter
  processing.
2011-12-01 13:48:48 +00:00
Eelco Dolstra b1eb8f4249 * Get rid of some superfluous error messages if a substituter fails.
* Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
Eelco Dolstra 5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra 993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Rob Vermaas 4e1ea17052 nix: add /etc/hosts with localhost entry to chroot builds. 2011-11-21 15:19:51 +00:00
Eelco Dolstra c0b706213d * Boldly make SQLite WAL mode the default again. Hopefully the
intermittent problems are gone by now.  WAL mode is preferrable
  because it does way fewer fsyncs.
2011-11-07 21:11:59 +00:00
Shea Levy 3c3107da86 There's no need to mess with drvPath at all 2011-11-06 07:18:19 +00:00
Shea Levy 2ab29be70c Fix faulty reversion of my changes to unsafeDiscardOutputDependency 2011-11-06 07:03:14 +00:00
Shea Levy ca0d47a70c Respect all outputs passed to the derivation, not just the last one 2011-11-06 06:54:05 +00:00
Shea Levy 24b65937e1 Remove the unused sCurrentOutput symbol 2011-11-06 06:28:34 +00:00
Shea Levy 3522730316 Embed output name into the context of the *OutPath attributes and extract it for input derivations
Multiple outputs test passes!
2011-11-06 06:28:30 +00:00
Shea Levy af2e53fd48 Include all outputs of derivations in the closure of explicitly-passed derivation paths
This required adding a queryOutputDerivationNames function in the store API
2011-11-06 06:28:20 +00:00
Shea Levy 981edeab7b The 'insert output between = signs' approach was not helpful 2011-11-06 06:28:14 +00:00
Shea Levy 2721e9f56f Merge from trunk 2011-11-06 00:13:09 +00:00
Eelco Dolstra a12095d3be * In printValueAsXML, handle the case where a "type" attribute is not
a string.  This happens in the NixOS option system.
* Remove a bogus comparison of a unsigned integer with -1.
2011-10-27 19:06:23 +00:00
Shea Levy e81c09edbf Remove the current output metadata from the string for unsaveDiscardOutputDependency 2011-09-16 11:31:00 +00:00
Shea Levy bf50d6ad32 Add information about which output is active to drvPath's context
This will break things that depend on being able to just strip away an equals sign, so those have to be updated next
2011-09-16 11:30:52 +00:00
Shea Levy ffa038f66d Add an sCurrentOutput member to EvalState 2011-09-16 11:30:44 +00:00
Shea Levy f3e410d4bf Add a currentOutput attribute to derivations keep track of which output is active 2011-09-16 11:30:03 +00:00
Shea Levy c172d16b00 First attempt at the output-as-derivation semantics
For each output, this adds a corresponding attribute to the derivation that is
the same as the derivation except for outPath, which is set to the path specific
to that output. Additionally, an "all" attribute is added that is a list of all
of the output derivations. This has to be done outside of derivationStrict as
each output is itself a derivation that contains itself (and all other outputs)
as an attribute. The derivation itself is equivalent to the first output in the
outputs list (or "out" if that list isn't set).
2011-09-14 05:59:17 +00:00
Eelco Dolstra 13114daa3e * Ouch. A store upgrade could cause a substituter to be triggered,
causing a deadlock.
2011-09-12 09:07:43 +00:00
Eelco Dolstra e6cb3d0a0d * Added a command ‘nix-store --verify-paths PATHS’ to check whether
the contents of any of the given store paths have been modified.
  E.g.

    $ nix-store --verify-path $(nix-store -qR /var/run/current-system)
    path `/nix/store/m2smyiwbxidlprfxfz4rjlvz2c3mg58y-etc' was modified! expected hash `fc87e271c5fdf179b47939b08ad13440493805584b35e3014109d04d8436e7b8', got `20f1a47281b3c0cbe299ce47ad5ca7340b20ab34246426915fce0ee9116483aa'

  All paths are checked; the exit code is 1 if any path has been
  modified, 0 otherwise.
2011-09-06 12:06:30 +00:00
Eelco Dolstra 93227ff65c * Eliminate all uses of the global variable ‘store’ from libstore.
This should also fix:

    nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.

  which was caused by hashDerivationModulo() calling the ‘store’
  object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Eelco Dolstra 2d663b502d * Cache the result of file evaluation (i.e, memoize evalFile()). This
prevents files from being evaluated and stored as values multiple
  times.  For instance, evaluation of the ‘system’ attribute in NixOS
  causes ‘nixpkgs/pkgs/lib/lists.nix’ to be evaluated 2019 times.

  Caching gives a modest speedup and a decent memory footprint
  reduction (e.g., from 1.44s to 1.28s, and from 81 MiB to 59 MiB with
  GC_INITIAL_HEAP_SIZE=100000 on my system).
2011-08-06 19:45:43 +00:00
Eelco Dolstra 9d091ee99a * Handle the case where the search path element is a regular file. 2011-08-06 18:45:28 +00:00
Eelco Dolstra 00a724ebc6 * Remove a debug statement. 2011-08-06 18:24:43 +00:00
Eelco Dolstra 07340b8be7 * Add the Nix corepkgs to the end of the search path. This makes it
possible for other Nix expressions to use corepkgs (mostly useful
  for the buildenv function).
2011-08-06 18:23:38 +00:00
Eelco Dolstra c7101dac0b * Allow redirections in search path entries. E.g. if you have a
directory

    /home/eelco/src/stdenv-updates

  that you want to use as the directory for import such as

    with (import <nixpkgs> { });

  then you can say

    $ nix-build -I nixpkgs=/home/eelco/src/stdenv-updates
2011-08-06 17:48:57 +00:00
Eelco Dolstra 1ecc97b6bd * Add a Nix expression search path feature. Paths between angle
brackets, e.g.

    import <nixpkgs/pkgs/lib>

  are resolved by looking them up relative to the elements listed in
  the search path.  This allows us to get rid of hacks like

    import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"

  The search path can be specified through the ‘-I’ command-line flag
  and through the colon-separated ‘NIX_PATH’ environment variable,
  e.g.,

    $ nix-build -I /etc/nixos ...

  If a file is not found in the search path, an error message is
  lazily thrown.
2011-08-06 16:05:24 +00:00
Eelco Dolstra 54945a2950 * Refactoring: move parseExprFromFile() and parseExprFromString() into
the EvalState class.
2011-08-06 13:02:55 +00:00
Eelco Dolstra c8606664ab * Don't allow derivations with fixed and non-fixed outputs. 2011-07-20 18:26:00 +00:00
Eelco Dolstra b2027f70d9 * Fix a huuuuge security hole in the Nix daemon. It didn't check that
derivations added to the store by clients have "correct" output
  paths (meaning that the output paths are computed by hashing the
  derivation according to a certain algorithm).  This means that a
  malicious user could craft a special .drv file to build *any*
  desired path in the store with any desired contents (so long as the
  path doesn't already exist).  Then the attacker just needs to wait
  for a victim to come along and install the compromised path.

  For instance, if Alice (the attacker) knows that the latest Firefox
  derivation in Nixpkgs produces the path

    /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1

  then (provided this path doesn't already exist) she can craft a .drv
  file that creates that path (i.e., has it as one of its outputs),
  add it to the store using "nix-store --add", and build it with
  "nix-store -r".  So the fake .drv could write a Trojan to the
  Firefox path.  Then, if user Bob (the victim) comes along and does

    $ nix-env -i firefox
    $ firefox

  he executes the Trojan injected by Alice.

  The fix is to have the Nix daemon verify that derivation outputs are
  correct (in addValidPath()).  This required some refactoring to move
  the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
Eelco Dolstra d329c3ea9d * Support multiple outputs. A derivation can declare multiple outputs
by setting the ‘outputs’ attribute.  For example:

    stdenv.mkDerivation {
      name = "aterm-2.5";

      src = ...;

      outputs = [ "out" "tools" "dev" ];

      configureFlags = "--bindir=$(tools)/bin --includedir=$(dev)/include";
    }

  This derivation creates three outputs, named like this:

    /nix/store/gcnqgllbh01p3d448q8q6pzn2nc2gpyl-aterm-2.5
    /nix/store/gjf1sgirwfnrlr0bdxyrwzpw2r304j02-aterm-2.5-tools
    /nix/store/hp6108bqfgxvza25nnxfs7kj88xi2vdx-aterm-2.5-dev

  That is, the symbolic name of the output is suffixed to the store
  path (except for the ‘out’ output).  Each path is passed to the
  builder through the corresponding environment variable, e.g.,
  ${tools}.

  The main reason for multiple outputs is to allow parts of a package
  to be distributed and garbage-collected separately.  For instance,
  most packages depend on Glibc for its libraries, but don't need its
  header files.  If these are separated into different store paths,
  then a package that depends on the Glibc libraries only causes the
  libraries and not the headers to be downloaded.

  The main problem with multiple outputs is that if one output exists
  while the others have been garbage-collected (or never downloaded in
  the first place), and we want to rebuild the other outputs, then
  this isn't possible because we can't clobber a valid output (it
  might be in active use).  This currently gives an error message
  like:

    error: derivation `/nix/store/1s9zw4c8qydpjyrayxamx2z7zzp5pcgh-aterm-2.5.drv' is blocked by its output paths

  There are two solutions: 1) Do the build in a chroot.  Then we don't
  need to overwrite the existing path.  2) Use hash rewriting (see the
  ASE-2005 paper).  Scary but it should work.

  This is not finished yet.  There is not yet an easy way to refer to
  non-default outputs in Nix expressions.  Also, mutually recursive
  outputs aren't detected yet and cause the garbage collector to
  crash.
2011-07-18 23:31:03 +00:00
Eelco Dolstra d5d4dcd4c9 * Allow attribute names to be strings. Based on the
allow-arbitrary-strinsg-in-names patch by Marc Weber.
2011-07-13 15:53:24 +00:00
Eelco Dolstra 0a623a10c7 * Allow a default value in attribute selection by writing
x.y.z or default

  (as originally proposed in
  https://mail.cs.uu.nl/pipermail/nix-dev/2009-September/002989.html).

  For instance, an expression like

    stdenv.lib.attrByPath ["features" "ckSched"] false args

  can now be written as

    args.features.ckSched or false
2011-07-13 12:19:57 +00:00
Eelco Dolstra 2b9e29b1c8 * Change the right-hand side of the ‘.’ operator from an attribute to
an attribute path.  This is a refactoring to support default values.
2011-07-06 12:28:57 +00:00
Eelco Dolstra 5637037802 * In the ‘?’ operator, allow attribute paths. For instance, you can
write ‘attrs ? a.b’ to test whether ‘attrs’ has an attribute ‘a’
  containing an attribute ‘b’.  This is more convenient than ‘attrs ?
  a && attrs.a ? b’.

  Slight change in the semantics: it's no longer an error if the
  left-hand side of ‘?’ is not an attribute set.  In that case it just
  returns false.  So, ‘null ? foo’ no longer throws an error.
2011-07-06 10:58:17 +00:00
Ludovic Courtès 5c9e9f732d Add support for the build-timeout' and --timeout' options. 2011-06-30 15:19:13 +00:00
Eelco Dolstra 8c69dac8a1 * Handle error messages from the Nix worker containing the `%'
character.  (Nix/216)
2011-04-19 10:44:44 +00:00
Eelco Dolstra 4ba6afaf48 * nix-env -ub' (--prebuilt-only') didn't really work because it
checked too soon whether substitutes are available.  That is, it did
  so for every available package, rather than those matching installed
  packages.  This was very slow and subject to assertion failures.  So
  do the check much later.  Idem for `nix-env -qab' and `nix-env -ib'.
2011-04-11 16:27:05 +00:00
Eelco Dolstra 538b7caab0 * Don't allocate a big initial GC address space on machines with
little RAM.  Even if the memory isn't actually used, it can cause
  problems with the overcommit heuristics in the kernel.  So use a VM
  space of 25% of RAM, up to 384 MB.
2011-02-10 14:31:04 +00:00
Eelco Dolstra 5a6b039802 * Don't call GC_expand_hp unless we're actually using the garbage
collector.
2011-02-09 22:59:50 +00:00
Ludovic Courtès f1462c208b Use $BDW_GC_LIBS instead of a custom variable. 2011-02-09 21:23:13 +00:00
Eelco Dolstra 543988572e * Sync with the trunk. 2011-02-09 12:25:53 +00:00
Eelco Dolstra aeae0beba4 * Clang compatibility fix. Clang actually checks template definitions
when they are defined
2011-02-05 16:29:10 +00:00
Ludovic Courtès 3854fc9b42 Remove useless <config.h> inclusion from public header. 2011-01-14 13:55:58 +00:00
Eelco Dolstra 9db190eb31 * builtins.substring: if "start" is beyond the end of the string,
return the empty string.
2011-01-14 12:47:10 +00:00
Eelco Dolstra c931a7aec5 * Do a short sleep after SQLITE_BUSY. 2010-12-17 17:23:15 +00:00
Eelco Dolstra b1eb252172 * Propagate the "100" exit status for failed builds through the Nix
daemon.
2010-12-17 11:28:26 +00:00
Eelco Dolstra 3dd02580e3 * I forgot to catch SQLiteBusy in registerValidPaths(). So
registerValidPaths() now handles busy errors and registerValidPath()
  is simply a wrapper around it.
2010-12-14 13:25:20 +00:00
Eelco Dolstra d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra 5833243c92 * Create /nix/var/nix/db if it's missing. 2010-12-13 13:42:34 +00:00
Eelco Dolstra 100becf8d1 * createDirs(path): if path already exists, make sure it's a
directory.
* Provide a C++ wrapper around lstat().
2010-12-13 13:32:58 +00:00
Eelco Dolstra d7ca6f44eb * Update some comments. 2010-12-13 13:19:46 +00:00
Eelco Dolstra e4720b1a79 * Ignore the result of sqlite3_reset(). 2010-12-08 18:19:15 +00:00
Eelco Dolstra 8062d3af30 * `nix-store --verify --check-contents': don't hold the global GC lock
while checking the contents, since this operation can take a very
  long time to finish.  Also, fill in missing narSize fields in the DB
  while doing this.
2010-12-06 15:29:38 +00:00