Messages like
while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':
are redundant, because Nix already shows that it's evaluating a derivation:
while instantiating the derivation named `firefox-24.0' at `/home/eelco/Dev/nixpkgs/pkgs/applications/networking/browsers/firefox/default.nix:131:5':
while evaluating the derivation attribute `nativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:76:17':
Since we already cache files in normal form (fileEvalCache), caching
parse trees is redundant.
Note that getting rid of this cache doesn't actually save much memory
at the moment, because parse trees are currently not freed / GC'ed.
This reduces the difference between inherited and non-inherited
attribute handling to the choice of which env to use (in recs and lets)
by setting the AttrDef::e to a new ExprVar in the parser rather than
carrying a separate AttrDef::v VarRef member.
As an added bonus, this allows inherited attributes that inherit from a
with to delay forcing evaluation of the with's attributes.
Signed-off-by: Shea Levy <shea@shealevy.com>
Commit 20866a7031 added a ‘withAttrs’
field to Env, which is annoying because it makes every Env structure
bigger and we allocate millions of them. E.g. NixOS evaluation took
18 MiB more. So this commit squeezes ‘withAttrs’ into values[0].
Probably should use a union...
Evaluation of attribute sets is strict in the attribute names, which
means immediate evaluation of `with` attribute sets rules out some
potentially interesting use cases (e.g. where the attribute names of one
set depend in some way on another but we want to bring those names into
scope for some values in the second set).
The major example of this is overridable self-referential package sets
(e.g. all-packages.nix). With immediate `with` evaluation, the only
options for such sets are to either make them non-recursive and
explicitly use the name of the overridden set in non-overridden one
every time you want to reference another package, or make the set
recursive and use the `__overrides` hack. As shown in the test case that
comes with this commit, though, delayed `with` evaluation allows a nicer
third alternative.
Signed-off-by: Shea Levy <shea@shealevy.com>
We now print all output paths of a package, e.g.
openssl-1.0.0i bin=/nix/store/gq2mvh0wb9l90djvsagln3aqywqmr6vl-openssl-1.0.0i-bin;man=/nix/store/7zwf5r5hsdarl3n86dasvb4chm2xzw9n-openssl-1.0.0i-man;/nix/store/cj7xvk7fjp9q887359j75pw3pzjfmqf1-openssl-1.0.0i
or (in XML mode)
<item attrPath="openssl" name="openssl-1.0.0i" system="x86_64-linux">
<output name="bin" path="/nix/store/gq2mvh0wb9l90djvsagln3aqywqmr6vl-openssl-1.0.0i-bin" />
<output name="man" path="/nix/store/7zwf5r5hsdarl3n86dasvb4chm2xzw9n-openssl-1.0.0i-man" />
<output name="out" path="/nix/store/cj7xvk7fjp9q887359j75pw3pzjfmqf1-openssl-1.0.0i" />
</item>
This allows adding attributes like
attr = if stdenv.system == "bla" then something else null;
without changing the resulting derivation on non-<bla> platforms.
We once considered adding a special "ignore" value for this purpose,
but using null seems more elegant.
This can serve as a generic efficient list builder. For instance, the
function ‘catAttrs’ in Nixpkgs can be rewritten from
attr: l: fold (s: l: if hasAttr attr s then [(getAttr attr s)] ++ l else l) [] l
to
attr: l: builtins.concatLists (map (s: if hasAttr attr s then [(getAttr attr s)] else []) l)
Statistics before:
time elapsed: 1.08683
size of a value: 24
environments allocated: 1384376 (35809568 bytes)
list elements: 6946783 (55574264 bytes)
list concatenations: 37434
values allocated: 1760440 (42250560 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18273
number of thunks: 1297673
number of thunks avoided: 1380759
number of attr lookups: 430802
number of primop calls: 628912
number of function calls: 1333544
Statistics after (including new catAttrs):
time elapsed: 0.959854
size of a value: 24
environments allocated: 1010198 (26829296 bytes)
list elements: 1984878 (15879024 bytes)
list concatenations: 30488
values allocated: 1589760 (38154240 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18274
number of thunks: 1040925
number of thunks avoided: 1038428
number of attr lookups: 438419
number of primop calls: 474844
number of function calls: 959366
Setting the environment variable NIX_COUNT_CALLS to 1 enables some
basic profiling in the evaluator. It will count calls to functions
and primops as well as evaluations of attributes.
For example, to see where evaluation of a NixOS configuration spends
its time:
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 ./src/nix-instantiate/nix-instantiate '<nixos>' -A system --readonly-mode
...
calls to 39 primops:
239532 head
233962 tail
191252 hasAttr
...
calls to 1595 functions:
224157 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:19'
221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:14'
221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:10'
...
evaluations of 7088 attributes:
167377 undefined position
132459 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:119:41'
47322 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:13:21'
...
prevents files from being evaluated and stored as values multiple
times. For instance, evaluation of the ‘system’ attribute in NixOS
causes ‘nixpkgs/pkgs/lib/lists.nix’ to be evaluated 2019 times.
Caching gives a modest speedup and a decent memory footprint
reduction (e.g., from 1.44s to 1.28s, and from 81 MiB to 59 MiB with
GC_INITIAL_HEAP_SIZE=100000 on my system).
directory
/home/eelco/src/stdenv-updates
that you want to use as the directory for import such as
with (import <nixpkgs> { });
then you can say
$ nix-build -I nixpkgs=/home/eelco/src/stdenv-updates
brackets, e.g.
import <nixpkgs/pkgs/lib>
are resolved by looking them up relative to the elements listed in
the search path. This allows us to get rid of hacks like
import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"
The search path can be specified through the ‘-I’ command-line flag
and through the colon-separated ‘NIX_PATH’ environment variable,
e.g.,
$ nix-build -I /etc/nixos ...
If a file is not found in the search path, an error message is
lazily thrown.
derivations added to the store by clients have "correct" output
paths (meaning that the output paths are computed by hashing the
derivation according to a certain algorithm). This means that a
malicious user could craft a special .drv file to build *any*
desired path in the store with any desired contents (so long as the
path doesn't already exist). Then the attacker just needs to wait
for a victim to come along and install the compromised path.
For instance, if Alice (the attacker) knows that the latest Firefox
derivation in Nixpkgs produces the path
/nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1
then (provided this path doesn't already exist) she can craft a .drv
file that creates that path (i.e., has it as one of its outputs),
add it to the store using "nix-store --add", and build it with
"nix-store -r". So the fake .drv could write a Trojan to the
Firefox path. Then, if user Bob (the victim) comes along and does
$ nix-env -i firefox
$ firefox
he executes the Trojan injected by Alice.
The fix is to have the Nix daemon verify that derivation outputs are
correct (in addValidPath()). This required some refactoring to move
the hash computation code to libstore.
tree). This saves a lot of memory. The vector should be sorted so
that names can be looked up using binary search, but this is not the
case yet. (Surprisingly, looking up attributes using linear search
doesn't have a big impact on performance.)
Memory consumption for
$ nix-instantiate /etc/nixos/nixos/tests -A bittorrent.test --readonly-mode
on x86_64-linux with GC enabled is now 185 MiB (compared to 946
MiB on the trunk).
improves GC effectiveness a bit more (because a live value doesn't
keep other values in the environment plus the parent environments
alive), and removes the need for copy nodes.
a pointer to a Value, rather than the Value directly. This improves
the effectiveness of garbage collection a lot: if the Value is
stored inside the set directly, then any live pointer to the Value
causes all other attributes in the set to be live as well.
because it defines _FILE_OFFSET_BITS. Without this, on
OpenSolaris the system headers define it to be 32, and then
the 32-bit stat() ends up being called with a 64-bit "struct
stat", or vice versa.
This also ensures that we get 64-bit file sizes everywhere.
* Remove the redundant call to stat() in parseExprFromFile().
The file cannot be a symlink because that's the exit condition
of the loop before.
values. This improves sharing and gives another speed up.
Evaluation of the NixOS system attribute is now almost 7 times
faster than the old evaluator.
use site, allowing environments to be stores as vectors of values
rather than maps. This should speed up evaluation and reduce the
number of allocations.
precedence, i.e. `with {x=1;}; with {x=2;}; x' evaluates to 2'.
This has a simpler implementation and seems more natural. There
doesn't seem to be any code in Nixpkgs or NixOS that relies on the
old behaviour.
efficiently. The symbol table ensures that there is only one copy
of each symbol, thus allowing symbols to be compared efficiently
using a pointer equality test.
that there are some places in Nixpkgs (php_configurable /
composableDerivation, it seems) that call `derivation' with
incorrect arguments (namely, the `name' attribute missing) but get
away with it because of laziness.
concatenation and string coercion. This was a big mess (see
e.g. NIX-67). Contexts are now folded into strings, so that they
don't cause evaluation errors when they're not expected. The
semantics of paths has been clarified (see nixexpr-ast.def).
toString() and coerceToString() have been merged.
Semantic change: paths are now copied to the store when they're in a
concatenation (and in most other situations - that's the
formalisation of the meaning of a path). So
"foo " + ./bla
evaluates to "foo /nix/store/hash...-bla", not "foo
/path/to/current-dir/bla". This prevents accidental impurities, and
is more consistent with the treatment of derivation outputs, e.g.,
`"foo " + bla' where `bla' is a derivation. (Here `bla' would be
replaced by the output path of `bla'.)
packages (provided that they have a `meta.description' attribute).
E.g.,
$ ./src/nix-env/nix-env -qa --description gcc
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for sparc-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for mips-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x (cross-compiler for arm-linux)
gcc-4.0.2 GNU Compiler Collection, 4.0.x
instantiation, e.g. "nix-env -i" and "nix-env -qas" (but not
"nix-env -qa"). It turns out that many redundant calls to
addToStore(path) were made, which reads and hashes the entire path.
For instance, the bash bootstrap binary in Nixpkgs would be read and
hashed many times. As a result nix-env would spend around 92% of
its time in the function sha256_block (according to callgrind).
Some simple memoization fixes this.
