This should also fix:
nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.
which was caused by hashDerivationModulo() calling the ‘store’
object (during store upgrades) before openStore() assigned it.
derivations added to the store by clients have "correct" output
paths (meaning that the output paths are computed by hashing the
derivation according to a certain algorithm). This means that a
malicious user could craft a special .drv file to build *any*
desired path in the store with any desired contents (so long as the
path doesn't already exist). Then the attacker just needs to wait
for a victim to come along and install the compromised path.
For instance, if Alice (the attacker) knows that the latest Firefox
derivation in Nixpkgs produces the path
/nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1
then (provided this path doesn't already exist) she can craft a .drv
file that creates that path (i.e., has it as one of its outputs),
add it to the store using "nix-store --add", and build it with
"nix-store -r". So the fake .drv could write a Trojan to the
Firefox path. Then, if user Bob (the victim) comes along and does
$ nix-env -i firefox
$ firefox
he executes the Trojan injected by Alice.
The fix is to have the Nix daemon verify that derivation outputs are
correct (in addValidPath()). This required some refactoring to move
the hash computation code to libstore.
by setting the ‘outputs’ attribute. For example:
stdenv.mkDerivation {
name = "aterm-2.5";
src = ...;
outputs = [ "out" "tools" "dev" ];
configureFlags = "--bindir=$(tools)/bin --includedir=$(dev)/include";
}
This derivation creates three outputs, named like this:
/nix/store/gcnqgllbh01p3d448q8q6pzn2nc2gpyl-aterm-2.5
/nix/store/gjf1sgirwfnrlr0bdxyrwzpw2r304j02-aterm-2.5-tools
/nix/store/hp6108bqfgxvza25nnxfs7kj88xi2vdx-aterm-2.5-dev
That is, the symbolic name of the output is suffixed to the store
path (except for the ‘out’ output). Each path is passed to the
builder through the corresponding environment variable, e.g.,
${tools}.
The main reason for multiple outputs is to allow parts of a package
to be distributed and garbage-collected separately. For instance,
most packages depend on Glibc for its libraries, but don't need its
header files. If these are separated into different store paths,
then a package that depends on the Glibc libraries only causes the
libraries and not the headers to be downloaded.
The main problem with multiple outputs is that if one output exists
while the others have been garbage-collected (or never downloaded in
the first place), and we want to rebuild the other outputs, then
this isn't possible because we can't clobber a valid output (it
might be in active use). This currently gives an error message
like:
error: derivation `/nix/store/1s9zw4c8qydpjyrayxamx2z7zzp5pcgh-aterm-2.5.drv' is blocked by its output paths
There are two solutions: 1) Do the build in a chroot. Then we don't
need to overwrite the existing path. 2) Use hash rewriting (see the
ASE-2005 paper). Scary but it should work.
This is not finished yet. There is not yet an easy way to refer to
non-default outputs in Nix expressions. Also, mutually recursive
outputs aren't detected yet and cause the garbage collector to
crash.
tree). This saves a lot of memory. The vector should be sorted so
that names can be looked up using binary search, but this is not the
case yet. (Surprisingly, looking up attributes using linear search
doesn't have a big impact on performance.)
Memory consumption for
$ nix-instantiate /etc/nixos/nixos/tests -A bittorrent.test --readonly-mode
on x86_64-linux with GC enabled is now 185 MiB (compared to 946
MiB on the trunk).
a pointer to a Value, rather than the Value directly. This improves
the effectiveness of garbage collection a lot: if the Value is
stored inside the set directly, then any live pointer to the Value
causes all other attributes in the set to be live as well.
because it defines _FILE_OFFSET_BITS. Without this, on
OpenSolaris the system headers define it to be 32, and then
the 32-bit stat() ends up being called with a 64-bit "struct
stat", or vice versa.
This also ensures that we get 64-bit file sizes everywhere.
* Remove the redundant call to stat() in parseExprFromFile().
The file cannot be a symlink because that's the exit condition
of the loop before.
values. This improves sharing and gives another speed up.
Evaluation of the NixOS system attribute is now almost 7 times
faster than the old evaluator.
use site, allowing environments to be stores as vectors of values
rather than maps. This should speed up evaluation and reduce the
number of allocations.
efficiently. The symbol table ensures that there is only one copy
of each symbol, thus allowing symbols to be compared efficiently
using a pointer equality test.
that there are some places in Nixpkgs (php_configurable /
composableDerivation, it seems) that call `derivation' with
incorrect arguments (namely, the `name' attribute missing) but get
away with it because of laziness.
* Removed exprToString and stringToExpr because there is no ATerm
representation to work on anymore (and exposing the internals of the
evaluator like this is not a good idea anyway).
allowed. So `name1@name2', `{attrs1}@{attrs2}' and so on are now no
longer legal. This is no big loss because they were not useful
anyway.
This also changes the output of builtins.toXML for @-patterns
slightly.
intersectAttrs returns the (right-biased) intersection between two
attribute sets, e.g. every attribute from the second set that also
exists in the first. functionArgs returns the set of attributes
expected by a function.
The main goal of these is to allow the elimination of most of
all-packages.nix. Most package instantiations in all-packages.nix
have this form:
foo = import ./foo.nix {
inherit a b c;
};
With intersectAttrs and functionArgs, this can be written as:
foo = callPackage (import ./foo.nix) { };
where
callPackage = f: args:
f ((builtins.intersectAttrs (builtins.functionArgs f) pkgs) // args);
I.e., foo.nix is called with all attributes from "pkgs" that it
actually needs (e.g., pkgs.a, pkgs.b and pkgs.c). (callPackage can
do any other generic package-level stuff we might want, such as
applying makeOverridable.) Of course, the automatically supplied
arguments can be overriden if needed, e.g.
foo = callPackage (import ./foo.nix) {
c = c_version_2;
};
but for the vast majority of packages, this won't be needed.
The advantages are to reduce the amount of typing needed to add a
dependency (from three sites to two), and to reduce the number of
trivial commits to all-packages.nix. For the former, there have
been two previous attempts:
- Use "args: with args;" in the package's function definition.
This however obscures the actual expected arguments of a
function, which is very bad.
- Use "{ arg1, arg2, ... }:" in the package's function definition
(i.e. use the ellipis "..." to allow arbitrary additional
arguments), and then call the function with all of "pkgs" as an
argument. But this inhibits error detection if you call it with
an misspelled (or obsolete) argument.
NixOS evaluation errors in particular look intimidating and
generally aren't very useful. Ideally the builtins.throw messages
should be self-contained.
sure that it works as expected when you pass it a derivation. That
is, we have to make sure that all build-time dependencies are built,
and that they are all in the input closure (otherwise remote builds
might fail, for example). This is ensured at instantiation time by
adding all derivations and their sources to inputDrvs and inputSrcs.
derivation should be a source rather than a derivation dependency of
the call to the NAR derivation. Otherwise the derivation (and all
its dependencies) will be built as a side-effect, which may not even
succeed.
SHA-256 outputs of fixed-output derivations. I.e. they now produce
the same store path:
$ nix-store --add x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
$ nix-store --add-fixed --recursive sha256 x
/nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x
the latter being the same as the path that a derivation
derivation {
name = "x";
outputHashAlgo = "sha256";
outputHashMode = "recursive";
outputHash = "...";
...
};
produces.
This does change the output path for such fixed-output derivations.
Fortunately they are quite rare. The most common use is fetchsvn
calls with SHA-256 hashes. (There are a handful of those is
Nixpkgs, mostly unstable development packages.)
* Documented the computation of store paths (in store-api.cc).
dependency. `storePath /nix/store/bla' gives exactly the same
result as `toPath /nix/store/bla', except that the former includes
/nix/store/bla in the dependency context of the string.
Useful in some generated Nix expressions like nix-push, which now
finally does the right thing wrt distributed builds. (Previously
the path to be packed wasn't an explicit dependency, so it wouldn't
be copied to the remote machine.)
functions that take a single argument (plain lambdas) into one AST
node (Function) that contains a Pattern node describing the
arguments. Current patterns are single lazy arguments (VarPat) and
matching against an attribute set (AttrsPat).
This refactoring allows other kinds of patterns to be added easily,
such as Haskell-style @-patterns, or list pattern matching.
logic through the `parseDrvName' and `compareVersions' primops.
This will allow expressions to easily check whether some dependency
is a specific needed version or falls in some version range. See
tests/lang/eval-okay-versions.nix for examples.
;-)
* Channels: fix channels that are plain lists of derivations (like
strategoxt-unstable) instead of functions (like nixpkgs-unstable).
This fixes the error message "error: the left-hand side of the
function call is neither a function nor a primop (built-in
operation) but a list".
computing the store path (NIX-77). This is an important security
property in multi-user Nix stores.
Note that this changes the store paths of derivations (since the
derivation aterms are added using addTextToStore), but not most
outputs (unless they use builtins.toFile).
* `sub' to subtract two numbers.
* `stringLength' to get the length of a string.
* `substring' to get a substring of a string. These should be enough
to allow most string operations to be expressed.
from a source directory. All files for which a predicate function
returns true are copied to the store. Typical example is to leave
out the .svn directory:
stdenv.mkDerivation {
...
src = builtins.filterSource
(path: baseNameOf (toString path) != ".svn")
./source-dir;
# as opposed to
# src = ./source-dir;
}
This is important because the .svn directory influences the hash in
a rather unpredictable and variable way.
containing functions that operate on the Nix store. One
implementation is LocalStore, which operates on the Nix store
directly. The next step, to enable secure multi-user Nix, is to
create a different implementation RemoteStore that talks to a
privileged daemon process that uses LocalStore to perform the actual
operations.
concatenation and string coercion. This was a big mess (see
e.g. NIX-67). Contexts are now folded into strings, so that they
don't cause evaluation errors when they're not expected. The
semantics of paths has been clarified (see nixexpr-ast.def).
toString() and coerceToString() have been merged.
Semantic change: paths are now copied to the store when they're in a
concatenation (and in most other situations - that's the
formalisation of the meaning of a path). So
"foo " + ./bla
evaluates to "foo /nix/store/hash...-bla", not "foo
/path/to/current-dir/bla". This prevents accidental impurities, and
is more consistent with the treatment of derivation outputs, e.g.,
`"foo " + bla' where `bla' is a derivation. (Here `bla' would be
replaced by the output path of `bla'.)
side should be a path, I guess.
* Handle paths that are in the store but not direct children of the
store directory.
* Ugh, hack to prevent double context wrapping.
attribute existence and to return an attribute from an attribute
set, respectively. Example: `hasAttr "foo" {foo = 1;}'. They
differ from the `?' and `.' operators in that the attribute name is
an arbitrary expression. (NIX-61)
and returns its path. This can be used to (for instance) write
builders inside a Nix expression, e.g.,
stdenv.mkDerivation {
builder = "
source $stdenv/setup
...
";
...
}
derivation attributes to flatten them into strings. This is
possible since string can nowadays be wrapped in contexts that
describe the derivations/sources referenced by the evaluation of the
string.
all the primops. This allows Nix expressions to test for new
primops and take appropriate action if they're not available. For
instance, rather than calling a primop `foo' directly, they could
say `if builtins ? foo then builtins.foo ... else ...'.
packages (provided that they have a `meta.description' attribute).
E.g.,
$ ./src/nix-env/nix-env -qa --description gcc
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for sparc-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for mips-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for arm-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x
instantiation, e.g. "nix-env -i" and "nix-env -qas" (but not
"nix-env -qa"). It turns out that many redundant calls to
addToStore(path) were made, which reads and hashes the entire path.
For instance, the bash bootstrap binary in Nixpkgs would be read and
hashed many times. As a result nix-env would spend around 92% of
its time in the function sha256_block (according to callgrind).
Some simple memoization fixes this.
dependencyClosure { ... searchPath = [ ../foo ../bar ]; ... }
* Primop `dirOf' to return the directory part of a path (e.g., dirOf
/a/b/c == /a/b).
* Primop `relativise' (according to Webster that's a real word!) that
given paths A and B returns a string representing path B relative
path to A; e.g., relativise /a/b/c a/b/x/y => "../x/y".
determination (e.g., finding the header files dependencies of a C
file) in Nix low-level builds automatically.
For instance, in the function `compileC' in make/lib/default.nix, we
find the header file dependencies of C file `main' as follows:
localIncludes =
dependencyClosure {
scanner = file:
import (findIncludes {
inherit file;
});
startSet = [main];
};
The function works by "growing" the set of dependencies, starting
with the set `startSet', and calling the function `scanner' for each
file to get its dependencies (which should yield a list of strings
representing relative paths). For instance, when `scanner' is
called on a file `foo.c' that includes the line
#include "../bar/fnord.h"
then `scanner' should yield ["../bar/fnord.h"]. This list of
dependencies is absolutised relative to the including file and added
to the set of dependencies. The process continues until no more
dependencies are found (hence its a closure).
`dependencyClosure' yields a list that contains in alternation a
dependency, and its relative path to the directory of the start
file, e.g.,
[ /bla/bla/foo.c
"foo.c"
/bla/bar/fnord.h
"../bar/fnord.h"
]
These relative paths are necessary for the builder that compiles
foo.c to reconstruct the relative directory structure expected by
foo.c.
The advantage of `dependencyClosure' over the old approach (using
the impure `__currentTime') is that it's completely pure, and more
efficient because it only rescans for dependencies (i.e., by
building the derivations yielded by `scanner') if sources have
actually changed. The old approach rescanned every time.
`removeAttrs attrs ["x", "y"]' returns the set `attrs' with the
attributes named `x' and `y' removed. It is not an error for the
named attributes to be missing from the input set.
* Make the `derivation' primitive much more lazy. The expression
`derivation attrs' now evaluates to (essentially)
attrs // {
type = "derivation";
outPath = derivation! attrs;
drvPath = derivation! attrs;
}
where `derivation!' is a primop that does the actual derivation
instantiation (i.e., it does what `derivation' used to do). The
advantage is that it allows commands such as `nix-env -qa' and
`nix-env -i' to be much faster since they no longer need to
instantiate all derivations, just the `name' attribute. (However,
`nix-env' doesn't yet take advantage of this since it still always
evaluates the `outPath' and `drvPath' attributes).
Also, this allows derivations to cyclically reference each other,
for example,
webServer = derivation {
...
hostName = "svn.cs.uu.nl";
services = [svnService];
};
svnService = derivation {
...
hostName = webServer.hostName;
};
Previously, this would yield a black hole (infinite recursion).
derivations. This is mostly to simplify the implementation of
nix-prefetch-{url, svn}, which now work properly in setuid
installations.
* Enforce valid store names in `nix-store --add / --add-fixed'.
* Set the references for the user environment manifest properly.
* Don't copy the manifest (this was accidental).
* Don't store derivation paths in the manifest (maybe this should be
made optional). This cleans up the semantics of nix-env, which were
weird.
* Hash on the output paths of activated components, not on derivation
paths. This is because we don't know the derivation path of already
installed components anymore, and it allows the installation of
components by store path (skipping Nix expressions entirely).
* Query options `--out-path' and `--drv-path' to show the output and
derivation paths of components, respectively (the latter replaces
the `--expr' query).
promise :-) This allows derivations to specify on *what* output
paths of input derivations they are dependent. This helps to
prevent unnecessary downloads. For instance, a build might be
dependent on the `devel' and `lib' outputs of some library
component, but not the `docs' output.
`derivations.cc', etc.
* Store the SHA-256 content hash of store paths in the database after
they have been built/added. This is so that we can check whether
the store has been messed with (a la `rpm --verify').
* When registering path validity, verify that the closure property
holds.
representation of closures as ATerms in the Nix store. Instead, the
file system pointer graph is now stored in the Nix database. This
has many advantages:
- It greatly simplifies the implementation (we can drop the notion
of `successors', and so on).
- It makes registering roots for the garbage collector much easier.
Instead of specifying the closure expression as a root, you can
simply specify the store path that must be retained as a root.
This could not be done previously, since there was no way to find
the closure store expression containing a given store path.
- Better traceability: it is now possible to query what paths are
referenced by a path, and what paths refer to a path.
* Formalise the notion of fixed-output derivations, i.e., derivations
for which a cryptographic hash of the output is known in advance.
Changes to such derivations should not propagate upwards through the
dependency graph. Previously this was done by specifying the hash
component of the output path through the `id' attribute, but this is
insecure since you can lie about it (i.e., you can specify any hash
and then produce a completely different output). Now the
responsibility for checking the output is moved from the builder to
Nix itself.
A fixed-output derivation can be created by specifying the
`outputHash' and `outputHashAlgo' attributes, the latter taking
values `md5', `sha1', and `sha256', and the former specifying the
actual hash in hexadecimal or in base-32 (auto-detected by looking
at the length of the attribute value). MD5 is included for
compatibility but should be considered deprecated.
* Removed the `drvPath' pseudo-attribute in derivation results. It's
no longer necessary.
* Cleaned up the support for multiple output paths in derivation store
expressions. Each output now has a unique identifier (e.g., `out',
`devel', `docs'). Previously there was no way to tell output paths
apart at the store expression level.
* `nix-hash' now has a flag `--base32' to specify that the hash should
be printed in base-32 notation.
* `fetchurl' accepts parameters `sha256' and `sha1' in addition to
`md5'.
* `nix-prefetch-url' now prints out a SHA-1 hash in base-32. (TODO: a
flag to specify the hash.)
bits, then encode them in a radix-32 representation (using digits
and letters except e, o, u, and t). This produces store paths like
/nix/store/4i0zb0z7f88mwghjirkz702a71dcfivn-aterm-2.3.1. The nice
thing about this is that the hash part of the file name is still 32
characters, as before with MD5.
(Of course, shortening SHA-256 to 160 bits makes it no better than
SHA-160 in theory, but hopefully it's a bit more resistant to
attacks; it's certainly a lot slower.)
* Start cleaning up unique store path generation (they weren't always
unique; in particular the suffix ("-aterm-2.2", "-builder.sh") was
not part of the hash, therefore changes to the suffix would cause
multiple store objects with the same hash).
Instead we generate data bindings (build and match functions) for
the constructors specified in `constructors.def'. In particular
this removes the conversions between AFuns and strings, and Nix
expression evaluation now seems 3 to 4 times faster.
The expression `with E1; E2' evaluates to E2 with all bindings in
the attribute set E1 substituted. E.g.,
with {x = 123;}; x
evaluates to 123. That is, the attribute set E1 is in scope in E2.
This is particularly useful when importing files containing lots
definitions. E.g., instead of
let {
inherit (import ./foo.nix) a b c d e f;
body = ... a ... f ...;
}
we can now say
with import ./foo.nix;
... a ... f ...
I.e., we don't have to say what variables should be brought into scope.
permission to the Nix store or database. E.g., `nix-env -qa' will
work, but `nix-env -qas' won't (the latter needs DB access). The
option `--readonly-mode' forces this mode; otherwise, it's only
activated when the database cannot be opened.
print a nice backtrace of the stack, rather than vomiting a gigantic
(and useless) aterm on the screen. Example:
error: while evaluating file `.../pkgs/system/test.nix':
while evaluating attribute `subversion' at `.../pkgs/system/all-packages-generic.nix', line 533:
while evaluating function at `.../pkgs/applications/version-management/subversion/default.nix', line 1:
assertion failed at `.../pkgs/applications/version-management/subversion/default.nix', line 13
Since the Nix expression language is lazy, the trace may be
misleading. The purpose is to provide a hint as to the location of
the problem.
`bla:' is now no longer parsed as a URL.
* Re-enabled support for the `args' attribute in derivations to
specify command line arguments to the builder, e.g.,
...
builder = /usr/bin/python;
args = ["-c" ./builder.py];
...
