More precisely, in concatLists, if all lists except one are empty,
then just return the non-empty list. This reduces the number of list
element allocations by 32% when evaluating a NixOS system
configuration.
This can serve as a generic efficient list builder. For instance, the
function ‘catAttrs’ in Nixpkgs can be rewritten from
attr: l: fold (s: l: if hasAttr attr s then [(getAttr attr s)] ++ l else l) [] l
to
attr: l: builtins.concatLists (map (s: if hasAttr attr s then [(getAttr attr s)] else []) l)
Statistics before:
time elapsed: 1.08683
size of a value: 24
environments allocated: 1384376 (35809568 bytes)
list elements: 6946783 (55574264 bytes)
list concatenations: 37434
values allocated: 1760440 (42250560 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18273
number of thunks: 1297673
number of thunks avoided: 1380759
number of attr lookups: 430802
number of primop calls: 628912
number of function calls: 1333544
Statistics after (including new catAttrs):
time elapsed: 0.959854
size of a value: 24
environments allocated: 1010198 (26829296 bytes)
list elements: 1984878 (15879024 bytes)
list concatenations: 30488
values allocated: 1589760 (38154240 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18274
number of thunks: 1040925
number of thunks avoided: 1038428
number of attr lookups: 438419
number of primop calls: 474844
number of function calls: 959366
The one in Nixpkgs is O(n^2), this one is O(n). Big reduction in the
number of list allocations.
Statistics before (on a NixOS system config):
time elapsed: 1.17982
size of a value: 24
environments allocated: 1543334 (39624560 bytes)
list elements: 9612638 (76901104 bytes)
list concatenations: 37434
values allocated: 1854933 (44518392 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18272
number of thunks: 1392467
number of thunks avoided: 1507311
number of attr lookups: 430801
number of primop calls: 691600
number of function calls: 1492502
Statistics after:
time elapsed: 1.08683
size of a value: 24
environments allocated: 1384376 (35809568 bytes)
list elements: 6946783 (55574264 bytes)
list concatenations: 37434
values allocated: 1760440 (42250560 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18273
number of thunks: 1297673
number of thunks avoided: 1380759
number of attr lookups: 430802
number of primop calls: 628912
number of function calls: 1333544
Evaluation of a NixOS configuration spends quite a lot of time in the
"filter" function in Nixpkgs. As implemented in Nixpkgs, this is a
O(n^2) operation, so it's a good candidate for providing a more
efficient (i.e. primop) implementation. Using it gives a ~10% speed
increase and a significant reduction in the number of evaluations.
Statistics before (on a NixOS system config):
time elapsed: 1.3258
size of a value: 24
environments allocated: 1980939 (50127080 bytes)
list elements: 14679308 (117434464 bytes)
list concatenations: 50828
values allocated: 2098938 (50374512 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18271
number of thunks: 1645752
number of thunks avoided: 1921196
number of attr lookups: 430798
number of primop calls: 838807
number of function calls: 1930107
Statistics after:
time elapsed: 1.17982
size of a value: 24
environments allocated: 1543334 (39624560 bytes)
list elements: 9612638 (76901104 bytes)
list concatenations: 37434
values allocated: 1854933 (44518392 bytes)
attribute sets allocated: 392040
right-biased unions: 186334
values copied in right-biased unions: 591137
symbols in symbol table: 18272
number of thunks: 1392467
number of thunks avoided: 1507311
number of attr lookups: 430801
number of primop calls: 691600
number of function calls: 1492502
Setting the environment variable NIX_COUNT_CALLS to 1 enables some
basic profiling in the evaluator. It will count calls to functions
and primops as well as evaluations of attributes.
For example, to see where evaluation of a NixOS configuration spends
its time:
$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 ./src/nix-instantiate/nix-instantiate '<nixos>' -A system --readonly-mode
...
calls to 39 primops:
239532 head
233962 tail
191252 hasAttr
...
calls to 1595 functions:
224157 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:19'
221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:14'
221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:10'
...
evaluations of 7088 attributes:
167377 undefined position
132459 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:119:41'
47322 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:13:21'
...
The generated attrset has drvPath and outPath with the right string context, type 'derivation', outputName with
the right name, all with a list of outputs, and an attribute for each output.
I see three uses for this (though certainly there may be more):
* Using derivations generated by something besides nix-instantiate (e.g. guix)
* Allowing packages provided by channels to be used in nix expressions. If a channel installed a valid deriver
for each package it provides into the store, then those could be imported and used as dependencies or installed
in environment.systemPackages, for example.
* Enable hydra to be consistent in how it treats inputs that are outputs of another build. Right now, if an
input is passed as an argument to the job, it is passed as a derivation, but if it is accessed via NIX_PATH
(i.e. through the <> syntax), then it is a path that can be imported. This is problematic because the build
being depended upon may have been built with non-obvious arguments passed to its jobset file. With this
feature, hydra can just set the name of that input to the path to its drv file in NIX_PATH
For each output, this adds a corresponding attribute to the derivation that is
the same as the derivation except for outPath, which is set to the path specific
to that output. Additionally, an "all" attribute is added that is a list of all
of the output derivations. This has to be done outside of derivationStrict as
each output is itself a derivation that contains itself (and all other outputs)
as an attribute. The derivation itself is equivalent to the first output in the
outputs list (or "out" if that list isn't set).
This should also fix:
nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.
which was caused by hashDerivationModulo() calling the ‘store’
object (during store upgrades) before openStore() assigned it.
prevents files from being evaluated and stored as values multiple
times. For instance, evaluation of the ‘system’ attribute in NixOS
causes ‘nixpkgs/pkgs/lib/lists.nix’ to be evaluated 2019 times.
Caching gives a modest speedup and a decent memory footprint
reduction (e.g., from 1.44s to 1.28s, and from 81 MiB to 59 MiB with
GC_INITIAL_HEAP_SIZE=100000 on my system).
