Commit graph

38 commits

Author SHA1 Message Date
Guillaume Maudoux f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Eelco Dolstra 4be4f6de56
S3BinaryCacheStore:: Eliminate a string copy while uploading
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-16 16:21:30 +01:00
Eric Litak 0d38b4c792 armv5tel can be built by armv6l and armv7l (#1063) 2016-09-16 22:03:34 +02:00
Eelco Dolstra e07c0dcf5c Move some .drv parsing functions out of util 2016-09-14 10:54:57 +02:00
Eelco Dolstra 22d6e31fc6 Add a mechanism for derivation attributes to reference the derivation's outputs
For example, you can now say:

  configureFlags = "--prefix=${placeholder "out"} --includedir=${placeholder "dev"}";

The strings returned by the ‘placeholder’ builtin are replaced at
build time by the actual store paths corresponding to the specified
outputs.

Previously, you had to work around the inability to self-reference by doing stuff like:

  preConfigure = ''
    configureFlags+=" --prefix $out --includedir=$dev"
  '';

or rely on ad-hoc variable interpolation semantics in Autoconf or Make
(e.g. --prefix=\$(out)), which doesn't always work.
2016-08-17 17:19:32 +02:00
Eelco Dolstra ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra d64e0c1b64 Make derivationFromPath work on diverted stores 2016-06-02 18:43:36 +02:00
Eelco Dolstra 7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra 451ebf24ce Cache path info lookups in SQLite
This re-implements the binary cache database in C++, allowing it to be
used by other Store backends, in particular the S3 backend.
2016-04-20 14:12:38 +02:00
Eelco Dolstra fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Eelco Dolstra 6bd2c7bb38 OCD: foreach -> C++11 ranged for 2015-07-17 20:13:56 +02:00
Eelco Dolstra 1511aa9f48 Allow remote builds without sending the derivation closure
Previously, to build a derivation remotely, we had to copy the entire
closure of the .drv file to the remote machine, even though we only
need the top-level derivation. This is very wasteful: the closure can
contain thousands of store paths, and in some Hydra use cases, include
source paths that are very large (e.g. Git/Mercurial checkouts).

So now there is a new operation, StoreAPI::buildDerivation(), that
performs a build from an in-memory representation of a derivation
(BasicDerivation) rather than from a on-disk .drv file. The only files
that need to be in the Nix store are the sources of the derivation
(drv.inputSrcs), and the needed output paths of the dependencies (as
described by drv.inputDrvs). "nix-store --serve" exposes this
interface.

Note that this is a privileged operation, because you can construct a
derivation that builds any store path whatsoever. Fixing this will
require changing the hashing scheme (i.e., the output paths should be
computed from the other fields in BasicDerivation, allowing them to be
verified without access to other derivations). However, this would be
quite nice because it would allow .drv-free building (e.g. "nix-env
-i" wouldn't have to write any .drv files to disk).

Fixes #173.
2015-07-17 17:57:40 +02:00
Eelco Dolstra b65875f859 Export outputPaths function
This is useful for the new hydra-queue-runner.
2015-06-10 16:17:06 +02:00
Eelco Dolstra 11849a320e Use proper quotes everywhere 2014-08-20 18:03:48 +02:00
Eelco Dolstra dfa2f77d2e If a .drv cannot be parsed, show its path
Otherwise you just get ‘expected string `Derive(['’ which isn't very helpful.
2014-04-08 19:24:29 +02:00
Eelco Dolstra 8d8d47abd2 Only substitute wanted outputs of a derivation
If a derivation has multiple outputs, then we only want to download
those outputs that are actuallty needed.  So if we do "nix-build -A
openssl.man", then only the "man" output should be downloaded.
Likewise if another package depends on ${openssl.man}.

The tricky part is that different derivations can depend on different
outputs of a given derivation, so we may need to restart the
corresponding derivation goal if that happens.
2012-11-26 17:15:09 +01:00
Eelco Dolstra 46a369ad95 Make "nix-build -A <derivation>.<output>" do the right thing
For example, given a derivation with outputs "out", "man" and "bin":

  $ nix-build -A pkg

produces ./result pointing to the "out" output;

  $ nix-build -A pkg.man

produces ./result-man pointing to the "man" output;

  $ nix-build -A pkg.all

produces ./result, ./result-man and ./result-bin;

  $ nix-build -A pkg.all -A pkg2

produces ./result, ./result-man, ./result-bin and ./result-2.
2012-11-26 15:39:10 +01:00
Eelco Dolstra 0a7084567f Add a ‘--repair’ flag to nix-instantiate
This allows repairing corrupted derivations and other source files.
2012-10-03 15:09:18 -04:00
Eelco Dolstra 97421eb5ec Refactor settings processing
Put all Nix configuration flags in a Settings object.
2012-07-30 19:55:41 -04:00
Eelco Dolstra 13114daa3e * Ouch. A store upgrade could cause a substituter to be triggered,
causing a deadlock.
2011-09-12 09:07:43 +00:00
Eelco Dolstra 93227ff65c * Eliminate all uses of the global variable ‘store’ from libstore.
This should also fix:

    nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.

  which was caused by hashDerivationModulo() calling the ‘store’
  object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Eelco Dolstra b2027f70d9 * Fix a huuuuge security hole in the Nix daemon. It didn't check that
derivations added to the store by clients have "correct" output
  paths (meaning that the output paths are computed by hashing the
  derivation according to a certain algorithm).  This means that a
  malicious user could craft a special .drv file to build *any*
  desired path in the store with any desired contents (so long as the
  path doesn't already exist).  Then the attacker just needs to wait
  for a victim to come along and install the compromised path.

  For instance, if Alice (the attacker) knows that the latest Firefox
  derivation in Nixpkgs produces the path

    /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1

  then (provided this path doesn't already exist) she can craft a .drv
  file that creates that path (i.e., has it as one of its outputs),
  add it to the store using "nix-store --add", and build it with
  "nix-store -r".  So the fake .drv could write a Trojan to the
  Firefox path.  Then, if user Bob (the victim) comes along and does

    $ nix-env -i firefox
    $ firefox

  he executes the Trojan injected by Alice.

  The fix is to have the Nix daemon verify that derivation outputs are
  correct (in addValidPath()).  This required some refactoring to move
  the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
Eelco Dolstra 6bbfe95e30 * Don't use an ostringstream in unparseDerivation(), because it's
slow.  A `nix-env -qa --drv-path \*' on Nixpkgs was spending 40% of
  its time in unparseDerivation() because of this (now 11%).
2010-04-21 19:25:50 +00:00
Eelco Dolstra efc7a579e8 * Don't use the ATerm library for parsing/printing .drv files. 2010-04-19 13:46:58 +00:00
Eelco Dolstra d5eab2fc82 * Use foreach in a lot of places. 2009-04-21 11:52:16 +00:00
Eelco Dolstra b428adc267 * Strip off the `.nix' suffix from the attribute name for files in
~/.nix-defexpr,  otherwise the attribute cannot be selected with the
  `-A' option.  Useful if you want to stick a Nix expression directly
  in ~/.nix-defexpr.
2008-08-25 13:31:57 +00:00
Eelco Dolstra b618fa6eb6 * computeStorePathForText: take the references into account when
computing the store path (NIX-77).  This is an important security
  property in multi-user Nix stores.

  Note that this changes the store paths of derivations (since the
  derivation aterms are added using addTextToStore), but not most
  outputs (unless they use builtins.toFile).
2007-01-29 15:51:37 +00:00
Eelco Dolstra fcd9900d74 * Replace read-only calls to addTextToStore. 2006-12-01 21:00:39 +00:00
Eelco Dolstra e2ef5e07fd * Refactoring. There is now an abstract interface class StoreAPI
containing functions that operate on the Nix store.  One
  implementation is LocalStore, which operates on the Nix store
  directly.  The next step, to enable secure multi-user Nix, is to
  create a different implementation RemoteStore that talks to a
  privileged daemon process that uses LocalStore to perform the actual
  operations.
2006-11-30 17:43:04 +00:00
Eelco Dolstra d7efd76394 * Big cleanup of the semantics of paths, strings, contexts, string
concatenation and string coercion.  This was a big mess (see
  e.g. NIX-67).  Contexts are now folded into strings, so that they
  don't cause evaluation errors when they're not expected.  The
  semantics of paths has been clarified (see nixexpr-ast.def).
  toString() and coerceToString() have been merged.

  Semantic change: paths are now copied to the store when they're in a
  concatenation (and in most other situations - that's the
  formalisation of the meaning of a path).  So

    "foo " + ./bla

  evaluates to "foo /nix/store/hash...-bla", not "foo
  /path/to/current-dir/bla".  This prevents accidental impurities, and
  is more consistent with the treatment of derivation outputs, e.g.,
  `"foo " + bla' where `bla' is a derivation.  (Here `bla' would be
  replaced by the output path of `bla'.)
2006-10-16 15:55:34 +00:00
Eelco Dolstra 4be5443882 * Remove unnecessary inclusions of aterm2.h. 2006-09-04 22:08:40 +00:00
Eelco Dolstra 75068e7d75 * Use a proper namespace.
* Optimise header file usage a bit.
* Compile the parser as C++.
2006-09-04 21:06:23 +00:00
Eelco Dolstra c791e94aee * Removed a bunch of ATreverses. 2006-05-02 17:51:50 +00:00
Eelco Dolstra a24b78e9f1 * Maintain the references/referers relation also for derivations.
This simplifies garbage collection and `nix-store --query
  --requisites' since we no longer need to treat derivations
  specially.

* Better maintaining of the invariants, e.g., setReferences() can only
  be called on a valid/substitutable path.
2005-01-25 21:28:25 +00:00
Eelco Dolstra 6a0a2d5593 * Terminology fixes. 2005-01-20 16:01:07 +00:00
Eelco Dolstra 05f0430de1 * Another change to low-level derivations. The last one this year, I
promise :-) This allows derivations to specify on *what* output
  paths of input derivations they are dependent.  This helps to
  prevent unnecessary downloads.  For instance, a build might be
  dependent on the `devel' and `lib' outputs of some library
  component, but not the `docs' output.
2005-01-20 14:10:19 +00:00
Eelco Dolstra 96de272b48 * Renamed normalise.cc' -> build.cc', `storeexprs.cc' ->
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
  they have been built/added.  This is so that we can check whether
  the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
  holds.
2005-01-19 16:39:47 +00:00
Renamed from src/libstore/storeexpr.cc (Browse further)