Commit graph

265 commits

Author SHA1 Message Date
Eelco Dolstra 1672bcd230
Move evaluator-specific settings out of libstore 2018-05-30 13:29:50 +02:00
Eelco Dolstra 4bb8741b98
Make Env self-describing
If the Env denotes a 'with', then values[0] may be an Expr* cast to a
Value*. For code that generically traverses Values/Envs, it's useful
to know this.
2018-05-22 16:02:32 +02:00
Eelco Dolstra 9fd7cf98db
Memoise checkSourcePath()
This prevents hydra-eval-jobs from statting the same files over and
over again.
2018-05-22 13:02:14 +02:00
Eelco Dolstra 53ec5ac69f
Fix some random -Wconversion warnings 2018-05-02 13:56:34 +02:00
Tuomas Tynkkynen 7e0360504d libexpr: Optimize prim_derivationStrict by using more symbol comparisons 2018-02-17 16:54:21 +02:00
Shea Levy 081f14a169
Allow using RegisterPrimop to define constants.
This enables plugins to add new constants, as well as new primops.
2018-02-08 14:35:50 -05:00
Eelco Dolstra d4dcffd643
Add pure evaluation mode
In this mode, the following restrictions apply:

* The builtins currentTime, currentSystem and storePath throw an
  error.

* $NIX_PATH and -I are ignored.

* fetchGit and fetchMercurial require a revision hash.

* fetchurl and fetchTarball require a sha256 attribute.

* No file system access is allowed outside of the paths returned by
  fetch{Git,Mercurial,url,Tarball}. Thus 'nix build -f ./foo.nix' is
  not allowed.

Thus, the evaluation result is completely reproducible from the
command line arguments. E.g.

  nix build --pure-eval '(
    let
      nix = fetchGit { url = https://github.com/NixOS/nixpkgs.git; rev = "9c927de4b179a6dd210dd88d34bda8af4b575680"; };
      nixpkgs = fetchGit { url = https://github.com/NixOS/nixpkgs.git; ref = "release-17.09"; rev = "66b4de79e3841530e6d9c6baf98702aa1f7124e4"; };
    in (import (nix + "/release.nix") { inherit nix nixpkgs; }).build.x86_64-linux
  )'

The goal is to enable completely reproducible and traceable
evaluation. For example, a NixOS configuration could be fully
described by a single Git commit hash. 'nixos-rebuild' would do
something like

  nix build --pure-eval '(
    (import (fetchGit { url = file:///my-nixos-config; rev = "..."; })).system
  ')

where the Git repository /my-nixos-config would use further fetchGit
calls or Git externals to fetch Nixpkgs and whatever other
dependencies it has. Either way, the commit hash would uniquely
identify the NixOS configuration and allow it to reproduced.
2018-01-16 19:23:18 +01:00
Eelco Dolstra 74f75c8558
import, builtins.readFile: Handle diverted stores
Fixes #1791
2018-01-12 17:31:08 +01:00
Eelco Dolstra 812e027e1d
Add option allowed-uris
This allows network access in restricted eval mode.
2017-10-30 12:41:49 +01:00
Eelco Dolstra 4c9ff89c26
nix-build/nix-shell: Eliminate call to nix-instantiate / nix-store
Note that this removes the need for a derivation symlink, so the
--drv-path and --add-drv-link flags now do nothing.
2017-07-26 17:29:09 +02:00
Eelco Dolstra fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra 6de33a9c67
Add support for passing structured data to builders
Previously, all derivation attributes had to be coerced into strings
so that they could be passed via the environment. This is lossy
(e.g. lists get flattened, necessitating configureFlags
vs. configureFlagsArray, of which the latter cannot be specified as an
attribute), doesn't support attribute sets at all, and has size
limitations (necessitating hacks like passAsFile).

This patch adds a new mode for passing attributes to builders, namely
encoded as a JSON file ".attrs.json" in the current directory of the
builder. This mode is activated via the special attribute

  __structuredAttrs = true;

(The idea is that one day we can set this in stdenv.mkDerivation.)

For example,

  stdenv.mkDerivation {
    __structuredAttrs = true;
    name = "foo";
    buildInputs = [ pkgs.hello pkgs.cowsay ];
    doCheck = true;
    hardening.format = false;
  }

results in a ".attrs.json" file containing (sans the indentation):

  {
    "buildInputs": [],
    "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash",
    "configureFlags": [
      "--with-foo",
      "--with-bar=1 2"
    ],
    "doCheck": true,
    "hardening": {
      "format": false
    },
    "name": "foo",
    "nativeBuildInputs": [
      "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03"
    ],
    "propagatedBuildInputs": [],
    "propagatedNativeBuildInputs": [],
    "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv",
    "system": "x86_64-linux"
  }

"passAsFile" is ignored in this mode because it's not needed - large
strings are included directly in the JSON representation.

It is up to the builder to do something with the JSON
representation. For example, in bash-based builders, lists/attrsets of
string values could be mapped to bash (associative) arrays.
2017-01-26 20:40:33 +01:00
Eelco Dolstra 215b70f51e
Revert "Get rid of unicode quotes (#1140)"
This reverts commit f78126bfd6. There
really is no need for such a massive change...
2016-11-26 00:38:01 +01:00
Guillaume Maudoux f78126bfd6 Get rid of unicode quotes (#1140) 2016-11-25 15:48:27 +01:00
Eelco Dolstra 92d917b71a Fix GC build 2016-08-30 13:12:12 +02:00
Eelco Dolstra c42d1acfeb forceBool(): Show position info 2016-08-29 19:37:19 +02:00
Eelco Dolstra 26d92017d3 Add builtin function "partition"
The implementation of "partition" in Nixpkgs is O(n^2) (because of the
use of ++), and for some reason was causing stack overflows in
multi-threaded evaluation (not sure why).

This reduces "nix-env -qa --drv-path" runtime by 0.197s and memory
usage by 298 MiB (in non-Boehm mode).
2016-08-29 19:36:54 +02:00
Eelco Dolstra d74236d1f2 nix build: Use Nix search path
That is, unless --file is specified, the Nix search path is
synthesized into an attribute set. Thus you can say

  $ nix build nixpkgs.hello

assuming $NIX_PATH contains an entry of the form "nixpkgs=...". This
is more verbose than

  $ nix build hello

but is less ambiguous.
2016-08-23 17:11:19 +02:00
Eelco Dolstra 363f37d084 Make the search path lazier with non-fatal errors
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.
2016-04-14 15:32:24 +02:00
Eelco Dolstra 12b257f045 Make primop registration pluggable
This way we don't have to put all primops in one giant file.
2016-04-14 12:50:01 +02:00
Eelco Dolstra b3e8d72770 Merge pull request #762 from ctheune/ctheune-floats
Implement floats
2016-02-12 12:49:59 +01:00
Eelco Dolstra fa7cd5369b StoreAPI -> Store
Calling a class an API is a bit redundant...
2016-02-04 14:48:42 +01:00
Eelco Dolstra c10c61449f Eliminate the "store" global variable
Also, move a few free-standing functions into StoreAPI and Derivation.

Also, introduce a non-nullable smart pointer, ref<T>, which is just a
wrapper around std::shared_ptr ensuring that the pointer is never
null. (For reference-counted values, this is better than passing a
"T&", because the latter doesn't maintain the refcount. Usually, the
caller will have a shared_ptr keeping the value alive, but that's not
always the case, e.g., when passing a reference to a std::thread via
std::bind.)
2016-02-04 14:28:26 +01:00
Shea Levy 14080f3e4b Use __toString when coercing sets to strings.
For example, "${{ foo = "bar"; __toString = x: x.foo; }}" evaluates
to "bar".

With this, we can delay calling functions like mkDerivation,
buildPythonPackage, etc. until we actually need a derivation, enabling
overrides and other modifications to happen by simple attribute set
update.
2016-01-05 13:55:06 +01:00
Christian Theune 14ebde5289 First hit at providing support for floats in the language. 2016-01-05 00:40:40 +01:00
Mathnerd314 8a87521636 forceFunction: allow functors as well 2015-10-08 13:19:44 +02:00
Iwan Aucamp 75837651f1 Output line number on infinite recursion 2015-07-31 20:26:44 +02:00
Eelco Dolstra cb4320c1a0 Cleanup 2015-07-23 23:14:07 +02:00
Eelco Dolstra c8bb2371eb Optimize empty sets
This reduces the number of Bindings allocations by about 10%.
2015-07-23 23:11:08 +02:00
Nicolas B. Pierron db21cfa688 Move attribute set data structures into their own header file.
This modification moves Attr and Bindings structures into their own header
file which is dedicated to the attribute set representation. The goal of to
isolate pieces of code which are related to the attribute set
representation. Thus future modifications of the attribute set
representation will only have to modify these files, and not every other
file across the evaluator.
2015-07-14 19:23:17 +02:00
Eelco Dolstra 726f7f7fc9 Fix Boehm API violation
We were calling GC_INIT() after doing an allocation (in the baseEnv
construction), which is not allowed.
2015-03-19 20:02:37 +01:00
Eelco Dolstra 15d2d3c34e Add restricted evaluation mode
If ‘--option restrict-eval true’ is given, the evaluator will throw an
exception if an attempt is made to access any file outside of the Nix
search path. This is primarily intended for Hydra, where we don't want
people doing ‘builtins.readFile ~/.ssh/id_dsa’ or stuff like that.
2015-02-23 15:54:31 +01:00
Eelco Dolstra c2a8b5c42d Fix assertion failure in nix-env
$ nix-env -f ~/Dev/nixops/ -iA foo
  nix-env: src/libexpr/eval.hh:57: void nix::Bindings::push_back(const nix::Attr&): Assertion `size_ < capacity' failed.
  Aborted
2015-01-15 12:15:22 +01:00
Eelco Dolstra 153a943de7 Show position info for failing <...> lookups 2015-01-07 13:43:55 +01:00
Eelco Dolstra 8aedaf111e Remove canary stuff 2014-12-12 10:59:50 +01:00
Eelco Dolstra 4e340a983f forceString(): Accept pos argument 2014-11-25 10:23:36 +01:00
Shea Levy 997defa166 Add functors (callable attribute sets).
With this, attribute sets with a `__functor` attribute can be applied
just like normal functions. This can be used to attach arbitrary
metadata to a function without callers needing to treat it specially.
2014-11-15 16:12:05 -05:00
Shea Levy 6062b12160 Fix build on gcc < 4.7 2014-10-20 12:15:50 -04:00
Shea Levy d16e3c7f09 Export realiseContext in libnixexpr
Useful for importNative plugins
2014-10-17 22:15:09 -04:00
Eelco Dolstra 0ed1b924be Bindings: Remove copy constructor 2014-09-24 15:29:05 +02:00
Eelco Dolstra 1e0a799bef Rename strictForceValue -> forceValueDeep 2014-09-22 16:05:00 +02:00
Eelco Dolstra eff120d1b9 Add a function ‘valueSize’
It returns the size of value, including all other values and
environments reachable from it. It is intended for debugging memory
consumption issues.
2014-09-22 16:05:00 +02:00
Eelco Dolstra 93e4f01ee3 Inline Bindings::find() 2014-09-19 16:56:13 +02:00
Eelco Dolstra 5b58991a71 Store Attrs inside Bindings
This prevents a double allocation per attribute set.
2014-09-19 16:49:41 +02:00
Eelco Dolstra 6e5b02bee4 Add some instrumentation for debugging GC leaks 2014-09-17 15:19:07 +02:00
Eelco Dolstra fefd3650d4 Fix a segfault in ‘nix-env -qa’
This was triggered by 47e185847e, which
turned globals.state into a pointer.
2014-08-21 00:05:17 +02:00
Eelco Dolstra 62a6eeb1f3 Make the Nix search path declarative
Nix search path lookups like <nixpkgs> are now desugared to ‘findFile
nixPath <nixpkgs>’, where ‘findFile’ is a new primop. Thus you can
override the search path simply by saying

  let
    nixPath = [ { prefix = "nixpkgs"; path = "/my-nixpkgs"; } ];
  in ... <nixpkgs> ...

In conjunction with ‘scopedImport’ (commit
c273c15cb1), the Nix search path can be
propagated across imports, e.g.

  let

    overrides = {
      nixPath = [ ... ] ++ builtins.nixPath;
      import = fn: scopedImport overrides fn;
      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;
      builtins = builtins // overrides;
    };

  in scopedImport overrides ./nixos
2014-05-26 17:02:22 +02:00
Eelco Dolstra 39d72640c2 Ensure that -I flags get included in nixPath
Also fixes #261.
2014-05-26 16:52:31 +02:00
Eelco Dolstra c273c15cb1 Add primop ‘scopedImport’
‘scopedImport’ works like ‘import’, except that it takes a set of
attributes to be added to the lexical scope of the expression,
essentially extending or overriding the builtin variables.  For
instance, the expression

  scopedImport { x = 1; } ./foo.nix

where foo.nix contains ‘x’, will evaluate to 1.

This has a few applications:

* It allows getting rid of function argument specifications in package
  expressions. For instance, a package expression like:

    { stdenv, fetchurl, libfoo }:

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  can now we written as just

    stdenv.mkDerivation { ... buildInputs = [ libfoo ]; }

  and imported in all-packages.nix as:

    bar = scopedImport pkgs ./bar.nix;

  So whereas we once had dependencies listed in three places
  (buildInputs, the function, and the call site), they now only need
  to appear in one place.

* It allows overriding builtin functions. For instance, to trace all
  calls to ‘map’:

  let
    overrides = {
      map = f: xs: builtins.trace "map called!" (map f xs);

      # Ensure that our override gets propagated by calls to
      # import/scopedImport.
      import = fn: scopedImport overrides fn;

      scopedImport = attrs: fn: scopedImport (overrides // attrs) fn;

      # Also update ‘builtins’.
      builtins = builtins // overrides;
    };
  in scopedImport overrides ./bla.nix

* Similarly, it allows extending the set of builtin functions. For
  instance, during Nixpkgs/NixOS evaluation, the Nixpkgs library
  functions could be added to the default scope.

There is a downside: calls to scopedImport are not memoized, unlike
import. So importing a file multiple times leads to multiple parsings
/ evaluations. It would be possible to construct the AST only once,
but that would require careful handling of variables/environments.
2014-05-26 14:26:29 +02:00
Eelco Dolstra 4c5faad994 Show position info in Boolean operations 2014-04-04 22:43:52 +02:00
Eelco Dolstra bd9b1d97b4 Show position info in string concatenation / addition errors 2014-04-04 22:19:33 +02:00
Eelco Dolstra a5fe730940 forceString: Show position info 2014-04-04 21:14:11 +02:00
Eelco Dolstra 27b44b8cf7 forceAttrs: Show position info 2014-04-04 19:11:40 +02:00
Eelco Dolstra 96b695ccab forceList: Show position info 2014-04-04 19:05:36 +02:00
Eelco Dolstra b62d36963c forceInt: Show position info 2014-04-04 18:59:29 +02:00
Eelco Dolstra c28de6d96e Pass position information to primop calls
For example:

  error: `tail' called on an empty list, at
    /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/ex-2/default.nix:13:7
2014-04-04 18:59:29 +02:00
Eelco Dolstra b72c8d2e5b Include position info in function application
This allows error messages like:

  error: the anonymous function at `/etc/nixos/configuration.nix:1:1'
    called without required argument `foo', at
    `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:77:59'
2014-04-04 18:59:29 +02:00
Eelco Dolstra 2caab81660 Revert "Make ifs and asserts tail-recursive"
This reverts commit 273322c773.
2014-03-05 16:18:13 +01:00
Eelco Dolstra f14ef84a51 Warn about missing -I paths
Fixes #121.  Note that we don't warn about missing $NIX_PATH entries
because it's intended that some may be missing (cf. the default
$NIX_PATH on NixOS, which includes paths like /etc/nixos/nixpkgs for
backward compatibility).
2014-02-26 15:21:56 +01:00
Eelco Dolstra 94f9c14d52 Fix some clang warnings 2014-01-21 18:29:55 +01:00
Eelco Dolstra 77c13cdf56 Add a toJSON primop 2013-11-19 00:04:11 +01:00
Eelco Dolstra 285df765b9 Add a primop unsafeGetAttrPos to return the position of an attribute 2013-11-18 22:22:35 +01:00
Eelco Dolstra fc33fd86b7 Add a symbol __curPos that expands to the current source location
I.e. an attribute set { file = <string>; line = <int>; column = <int>; }.
2013-11-18 20:16:02 +01:00
Eelco Dolstra c897bac549 Make function calls tail-recursive 2013-11-12 11:32:23 +00:00
Eelco Dolstra 273322c773 Make ifs and asserts tail-recursive
The local Value object prevented g++ from making a tail call.  Not
clear why.  In any case, not using a temporary makes g++ do the tail
call.
2013-11-12 11:32:23 +00:00
Eelco Dolstra ea6bf0c21f Slightly optimize listToAttrs 2013-10-28 07:34:44 +01:00
Eelco Dolstra 5bc41d78ff Rename "attribute sets" to "sets"
We don't have any other kind of sets so calling them attribute sets is
unnecessarily verbose.
2013-10-24 16:41:04 +02:00
Eelco Dolstra f440558acc Don't show <nix/derivation.nix> in stack traces
Messages like

  while evaluating the attribute `outPath' at `/nix/store/212ngf4ph63mp6p1np2bapkfikpakfv7-nix-1.6/share/nix/corepkgs/derivation.nix:18:9':

are redundant, because Nix already shows that it's evaluating a derivation:

  while instantiating the derivation named `firefox-24.0' at `/home/eelco/Dev/nixpkgs/pkgs/applications/networking/browsers/firefox/default.nix:131:5':
  while evaluating the derivation attribute `nativeBuildInputs' at `/home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/default.nix:76:17':
2013-10-17 11:47:38 +02:00
Eelco Dolstra 221a2daf34 Merge VarRef into ExprVar 2013-10-08 14:24:53 +02:00
Eelco Dolstra 07a08bddf0 nix-env: Load files in ~/.nix-defexpr on demand
So if you do "nix-env -qa -A nixos", then other channels won't be
parsed/evaluated at all.
2013-09-03 15:45:32 +02:00
Eelco Dolstra 6f809194d7 Get rid of the parse tree cache
Since we already cache files in normal form (fileEvalCache), caching
parse trees is redundant.

Note that getting rid of this cache doesn't actually save much memory
at the moment, because parse trees are currently not freed / GC'ed.
2013-09-03 13:01:42 +02:00
Eelco Dolstra 57d18df7d0 Add some support code for nix-repl 2013-09-02 18:34:04 +02:00
Eelco Dolstra 33972629d7 Fix whitespace 2013-09-02 16:29:15 +02:00
Shea Levy afc6c1bad6 Simplify inherited attribute handling
This reduces the difference between inherited and non-inherited
attribute handling to the choice of which env to use (in recs and lets)
by setting the AttrDef::e to a new ExprVar in the parser rather than
carrying a separate AttrDef::v VarRef member.

As an added bonus, this allows inherited attributes that inherit from a
with to delay forcing evaluation of the with's attributes.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-08-26 11:31:56 +02:00
Eelco Dolstra d308aeaf53 Store Nix integers as longs
So on 64-bit systems, integers are now 64-bit.

Fixes #158.
2013-08-19 12:35:03 +02:00
Eelco Dolstra 46222bbc43 Typo 2013-08-14 22:32:49 +02:00
Eelco Dolstra 46ffcce0c3 In the profiler output, show function names (if available) 2013-08-02 18:29:23 +00:00
Eelco Dolstra 0a470fc345 Make Env smaller
Commit 20866a7031 added a ‘withAttrs’
field to Env, which is annoying because it makes every Env structure
bigger and we allocate millions of them.  E.g. NixOS evaluation took
18 MiB more.  So this commit squeezes ‘withAttrs’ into values[0].
Probably should use a union...
2013-07-31 12:44:21 +02:00
Shea Levy e068f49f7d Avoid thunks when a fromWith var can be looked up without evaluation
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-31 11:48:39 +02:00
Shea Levy 20866a7031 Delay evaulation of with attrs until a variable lookup needs them
Evaluation of attribute sets is strict in the attribute names, which
means immediate evaluation of `with` attribute sets rules out some
potentially interesting use cases (e.g. where the attribute names of one
set depend in some way on another but we want to bring those names into
scope for some values in the second set).

The major example of this is overridable self-referential package sets
(e.g. all-packages.nix). With immediate `with` evaluation, the only
options for such sets are to either make them non-recursive and
explicitly use the name of the overridden set in non-overridden one
every time you want to reference another package, or make the set
recursive and use the `__overrides` hack. As shown in the test case that
comes with this commit, though, delayed `with` evaluation allows a nicer
third alternative.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-31 11:48:39 +02:00
Eelco Dolstra 8eed07cda4 nix-env -q --out-path: Support multiple outputs
We now print all output paths of a package, e.g.

  openssl-1.0.0i  bin=/nix/store/gq2mvh0wb9l90djvsagln3aqywqmr6vl-openssl-1.0.0i-bin;man=/nix/store/7zwf5r5hsdarl3n86dasvb4chm2xzw9n-openssl-1.0.0i-man;/nix/store/cj7xvk7fjp9q887359j75pw3pzjfmqf1-openssl-1.0.0i

or (in XML mode)

  <item attrPath="openssl" name="openssl-1.0.0i" system="x86_64-linux">
    <output name="bin" path="/nix/store/gq2mvh0wb9l90djvsagln3aqywqmr6vl-openssl-1.0.0i-bin" />
    <output name="man" path="/nix/store/7zwf5r5hsdarl3n86dasvb4chm2xzw9n-openssl-1.0.0i-man" />
    <output name="out" path="/nix/store/cj7xvk7fjp9q887359j75pw3pzjfmqf1-openssl-1.0.0i" />
  </item>
2012-11-28 13:49:44 +01:00
Eelco Dolstra 6c98e6a5de Optionally ignore null-valued derivation attributes
This allows adding attributes like

  attr = if stdenv.system == "bla" then something else null;

without changing the resulting derivation on non-<bla> platforms.

We once considered adding a special "ignore" value for this purpose,
but using null seems more elegant.
2012-11-27 15:01:32 +01:00
Eelco Dolstra 408a7bfac1 nix-instantiate: Fix read-only evaluation 2012-11-26 17:41:10 +01:00
Eelco Dolstra 0a7084567f Add a ‘--repair’ flag to nix-instantiate
This allows repairing corrupted derivations and other source files.
2012-10-03 15:09:18 -04:00
Eelco Dolstra 198d0338be Add a primop ‘concatLists’
This can serve as a generic efficient list builder.  For instance, the
function ‘catAttrs’ in Nixpkgs can be rewritten from

  attr: l: fold (s: l: if hasAttr attr s then [(getAttr attr s)] ++ l else l) [] l

to

  attr: l: builtins.concatLists (map (s: if hasAttr attr s then [(getAttr attr s)] else []) l)

Statistics before:

  time elapsed: 1.08683
  size of a value: 24
  environments allocated: 1384376 (35809568 bytes)
  list elements: 6946783 (55574264 bytes)
  list concatenations: 37434
  values allocated: 1760440 (42250560 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18273
  number of thunks: 1297673
  number of thunks avoided: 1380759
  number of attr lookups: 430802
  number of primop calls: 628912
  number of function calls: 1333544

Statistics after (including new catAttrs):

  time elapsed: 0.959854
  size of a value: 24
  environments allocated: 1010198 (26829296 bytes)
  list elements: 1984878 (15879024 bytes)
  list concatenations: 30488
  values allocated: 1589760 (38154240 bytes)
  attribute sets allocated: 392040
  right-biased unions: 186334
  values copied in right-biased unions: 591137
  symbols in symbol table: 18274
  number of thunks: 1040925
  number of thunks avoided: 1038428
  number of attr lookups: 438419
  number of primop calls: 474844
  number of function calls: 959366
2012-08-13 01:53:10 -04:00
Eelco Dolstra 62f72eb9e1 Add some more evaluations stats 2012-08-12 23:41:48 -04:00
Eelco Dolstra e82767910c Add some basic profiling support to the evaluator
Setting the environment variable NIX_COUNT_CALLS to 1 enables some
basic profiling in the evaluator.  It will count calls to functions
and primops as well as evaluations of attributes.

For example, to see where evaluation of a NixOS configuration spends
its time:

$ NIX_SHOW_STATS=1 NIX_COUNT_CALLS=1 ./src/nix-instantiate/nix-instantiate '<nixos>' -A system --readonly-mode
...
calls to 39 primops:
    239532 head
    233962 tail
    191252 hasAttr
...
calls to 1595 functions:
    224157 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:19'
    221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:14'
    221767 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/lists.nix:17:10'
...
evaluations of 7088 attributes:
    167377 undefined position
    132459 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:119:41'
     47322 `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/lib/attrsets.nix:13:21'
...
2012-08-12 23:29:28 -04:00
Eelco Dolstra b7fd2c2822 Use "#pragma once" to prevent repeated header file inclusion 2012-07-18 14:59:03 -04:00
Eelco Dolstra 1217204c81 Remove dead code 2012-07-17 14:07:52 -04:00
Eelco Dolstra f491ae97d4 * Inline some functions and get rid of the indirection through
EvalState::eval().  This gives a 12% speedup on ‘nix-instantiate
  /etc/nixos/nixos/ -A system --readonly-mode’ (from 1.01s to 0.89s).
2012-02-04 13:50:25 +00:00
Eelco Dolstra 9fe24c5a0d * Don't create thunks for simple constants (integers, strings, paths)
and allocate them only once.
* Move Value and related functions into value.hh.
2012-01-07 17:26:33 +00:00
Eelco Dolstra b52966e821 * Remove dead code. 2012-01-04 21:47:48 +00:00
Eelco Dolstra 35f2a6ba82 * Don't use dynamic_cast, it's very slow. "nix-instantiate
/etc/nixos/nixos -A system" spent about 10% of its time in
  dynamic_cast.
2012-01-04 21:24:11 +00:00
Eelco Dolstra 2d663b502d * Cache the result of file evaluation (i.e, memoize evalFile()). This
prevents files from being evaluated and stored as values multiple
  times.  For instance, evaluation of the ‘system’ attribute in NixOS
  causes ‘nixpkgs/pkgs/lib/lists.nix’ to be evaluated 2019 times.

  Caching gives a modest speedup and a decent memory footprint
  reduction (e.g., from 1.44s to 1.28s, and from 81 MiB to 59 MiB with
  GC_INITIAL_HEAP_SIZE=100000 on my system).
2011-08-06 19:45:43 +00:00
Eelco Dolstra c7101dac0b * Allow redirections in search path entries. E.g. if you have a
directory

    /home/eelco/src/stdenv-updates

  that you want to use as the directory for import such as

    with (import <nixpkgs> { });

  then you can say

    $ nix-build -I nixpkgs=/home/eelco/src/stdenv-updates
2011-08-06 17:48:57 +00:00
Eelco Dolstra 1ecc97b6bd * Add a Nix expression search path feature. Paths between angle
brackets, e.g.

    import <nixpkgs/pkgs/lib>

  are resolved by looking them up relative to the elements listed in
  the search path.  This allows us to get rid of hacks like

    import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"

  The search path can be specified through the ‘-I’ command-line flag
  and through the colon-separated ‘NIX_PATH’ environment variable,
  e.g.,

    $ nix-build -I /etc/nixos ...

  If a file is not found in the search path, an error message is
  lazily thrown.
2011-08-06 16:05:24 +00:00
Eelco Dolstra 54945a2950 * Refactoring: move parseExprFromFile() and parseExprFromString() into
the EvalState class.
2011-08-06 13:02:55 +00:00
Eelco Dolstra b2027f70d9 * Fix a huuuuge security hole in the Nix daemon. It didn't check that
derivations added to the store by clients have "correct" output
  paths (meaning that the output paths are computed by hashing the
  derivation according to a certain algorithm).  This means that a
  malicious user could craft a special .drv file to build *any*
  desired path in the store with any desired contents (so long as the
  path doesn't already exist).  Then the attacker just needs to wait
  for a victim to come along and install the compromised path.

  For instance, if Alice (the attacker) knows that the latest Firefox
  derivation in Nixpkgs produces the path

    /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1

  then (provided this path doesn't already exist) she can craft a .drv
  file that creates that path (i.e., has it as one of its outputs),
  add it to the store using "nix-store --add", and build it with
  "nix-store -r".  So the fake .drv could write a Trojan to the
  Firefox path.  Then, if user Bob (the victim) comes along and does

    $ nix-env -i firefox
    $ firefox

  he executes the Trojan injected by Alice.

  The fix is to have the Nix daemon verify that derivation outputs are
  correct (in addValidPath()).  This required some refactoring to move
  the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
Eelco Dolstra 43535499f3 * When allocating an attribute set, reserve enough space for all
elements.  This prevents the vector from having to resize itself.
2010-10-24 20:09:37 +00:00
Eelco Dolstra e0b7fb8f27 * Keep attribute sets in sorted order to speed up attribute lookups.
* Simplify the representation of attributes in the AST.
* Change the behaviour of listToAttrs() in case of duplicate names.
2010-10-24 19:52:33 +00:00
Eelco Dolstra 2dc6d50941 * Don't create thunks for variable lookups (if possible). This
significantly reduces the number of values allocated (e.g. from 8.7m
  to 4.9m for the Bittorrent test).
2010-10-24 14:20:02 +00:00
Eelco Dolstra 0b305c534f * Store attribute sets as a vector instead of a map (i.e. a red-black
tree).  This saves a lot of memory.  The vector should be sorted so
  that names can be looked up using binary search, but this is not the
  case yet.  (Surprisingly, looking up attributes using linear search
  doesn't have a big impact on performance.)

  Memory consumption for

    $ nix-instantiate /etc/nixos/nixos/tests -A bittorrent.test --readonly-mode

  on x86_64-linux with GC enabled is now 185 MiB (compared to 946
  MiB on the trunk).
2010-10-24 00:41:29 +00:00
Eelco Dolstra 8ac06726b9 * Make Value smaller by not storing redundant PrimOp info.
* Clear pointers in Values after overwriting them to make sure that no
  objects are kept alive unnecessarily.
2010-10-23 20:07:47 +00:00
Eelco Dolstra 3f66cfb96b * Remove allocValues(). 2010-10-23 18:18:07 +00:00
Eelco Dolstra 4dee289550 * In environments, store pointers to values rather than values. This
improves GC effectiveness a bit more (because a live value doesn't
  keep other values in the environment plus the parent environments
  alive), and removes the need for copy nodes.
2010-10-22 15:51:52 +00:00
Eelco Dolstra 41c45a9b31 * Store Value nodes outside of attribute sets. I.e., Attr now stores
a pointer to a Value, rather than the Value directly.  This improves
  the effectiveness of garbage collection a lot: if the Value is
  stored inside the set directly, then any live pointer to the Value
  causes all other attributes in the set to be live as well.
2010-10-22 14:47:42 +00:00
Eelco Dolstra 64c3325b0b * Make building against the Boehm GC a configure option. 2010-10-22 13:39:15 +00:00
Eelco Dolstra 76feaf016a * Keep some more stats. 2010-10-20 15:48:00 +00:00
Eelco Dolstra e879a0371b * Use the Boehm garbage collector to reclaim unused memory in the Nix
expression evaluator.
2010-10-20 11:38:30 +00:00
Eelco Dolstra 705868a8a9 * Make sure that config.h is included before the system headers,
because it defines _FILE_OFFSET_BITS.  Without this, on 
  OpenSolaris the system headers define it to be 32, and then 
  the 32-bit stat() ends up being called with a 64-bit "struct 
  stat", or vice versa.

  This also ensures that we get 64-bit file sizes everywhere.

* Remove the redundant call to stat() in parseExprFromFile().
  The file cannot be a symlink because that's the exit condition
  of the loop before.
2010-10-04 17:55:38 +00:00
Eelco Dolstra 4aa9245083 * Hack needed for GCC 4.3.2 on OpenSolaris. 2010-10-04 10:51:16 +00:00
Eelco Dolstra f16fe2af8d * builtins.toXML: propagate the string context. This is a regression
from the old ATerm-based evaluator.
2010-06-10 10:29:50 +00:00
Eelco Dolstra 93cd5a4a13 * The << operator on values should be const. 2010-05-18 10:36:37 +00:00
Eelco Dolstra b2235d81d1 * Restore the __overrides feature that was lost somewhere in the
fast-eval branch.
2010-05-15 08:10:12 +00:00
Eelco Dolstra e2d5e40f4f * Keep track of the source positions of attributes. 2010-05-07 12:11:05 +00:00
Eelco Dolstra 0bc468f195 * Simplify the implementation of `with'. This gives a 7% speedup in
evaluating the NixOS system configuration.
2010-04-22 15:08:09 +00:00
Eelco Dolstra 2d7636529f * String equality tests should take the context into account. All the
evaluation test cases now succeed.
2010-04-22 09:54:11 +00:00
Eelco Dolstra 0777448ca6 * Fixed builtins.genericClosure. 2010-04-21 15:57:11 +00:00
Eelco Dolstra fe2d869e04 * Store user environment manifests as a Nix expression in
$out/manifest.nix rather than as an ATerm.

  (Hm, I thought I committed this two days ago...)
2010-04-21 15:08:58 +00:00
Eelco Dolstra 5c31995bb8 * Updated some more primops. 2010-04-16 15:13:47 +00:00
Eelco Dolstra 04c4bd3624 * Store lists as lists of pointers to values rather than as lists of
values.  This improves sharing and gives another speed up.
  Evaluation of the NixOS system attribute is now almost 7 times
  faster than the old evaluator.
2010-04-15 00:37:36 +00:00
Eelco Dolstra e41b5828db * Better stats. 2010-04-14 23:48:46 +00:00
Eelco Dolstra 267dc693d2 * Fix builtins. 2010-04-14 22:59:39 +00:00
Eelco Dolstra 81de12bc8f * Refactoring: move variable uses to a separate class. 2010-04-14 15:14:23 +00:00
Eelco Dolstra 9985230c00 * After parsing, compute level/displacement pairs for each variable
use site, allowing environments to be stores as vectors of values
  rather than maps.  This should speed up evaluation and reduce the
  number of allocations.
2010-04-14 14:42:32 +00:00
Eelco Dolstra 85d13c8f93 * Change the semantics of "with" so that inner "withs" take
precedence, i.e. `with {x=1;}; with {x=2;}; x' evaluates to 2'.
  This has a simpler implementation and seems more natural.  There
  doesn't seem to be any code in Nixpkgs or NixOS that relies on the
  old behaviour.
2010-04-14 08:37:08 +00:00
Eelco Dolstra 7d47498b5e * Evaluate lets directly (i.e. without desugaring to `rec { attrs...;
<let-body> = e; }.<let-body>).  This prevents the unnecessary
  allocation of an attribute set.
2010-04-13 13:42:25 +00:00
Eelco Dolstra ac1e8f40d4 * Use a symbol table to represent identifiers and attribute names
efficiently.  The symbol table ensures that there is only one copy
  of each symbol, thus allowing symbols to be compared efficiently
  using a pointer equality test.
2010-04-13 12:25:42 +00:00
Eelco Dolstra 4d6ad5be17 * Don't use ATerms for the abstract syntax trees anymore. Not
finished yet.
2010-04-12 18:30:11 +00:00
Eelco Dolstra f3dc7ab877 * Keep more statistics about stack space usage.
* Reduce stack space usage.
2010-04-09 12:00:49 +00:00
Eelco Dolstra af2a372bb0 * Update autoCallFunction() and findAlongAttrPath(). 2010-04-07 15:47:06 +00:00
Eelco Dolstra fc92244ba8 * Implemented the primops necessary for generating the NixOS manual. 2010-04-07 13:55:46 +00:00
Eelco Dolstra 7b851915bf * Improve sharing. 2010-04-01 12:04:57 +00:00
Eelco Dolstra 71f026292b * Make `derivation' lazy again for performance. It also turns out
that there are some places in Nixpkgs (php_configurable /
  composableDerivation, it seems) that call `derivation' with
  incorrect arguments (namely, the `name' attribute missing) but get
  away with it because of laziness.
2010-04-01 09:55:57 +00:00
Eelco Dolstra 979f163615 * Handle string contexts. `nix-instantiate' can now correctly compute
the `firefoxWrapper' attribute in Nixpkgs, and it's about 3 times
  faster than the trunk :-)
2010-03-31 19:52:29 +00:00
Eelco Dolstra 55e207b2dc * Cache parse trees to prevent repeated parsing of imported Nix
expressions.
2010-03-31 16:14:32 +00:00
Eelco Dolstra 3d94be61ea * Implemented derivations. 2010-03-31 15:38:03 +00:00
Eelco Dolstra 7f19e03c65 * More primops. 2010-03-30 22:39:48 +00:00
Eelco Dolstra 47df476daa * More operators / primops. 2010-03-30 18:05:54 +00:00
Eelco Dolstra c9170be2bd * More primops. 2010-03-30 15:18:20 +00:00
Eelco Dolstra c3aa615a5f * More primops. 2010-03-30 14:39:27 +00:00
Eelco Dolstra 5b72d8a749 * Implemented `map'. 2010-03-30 13:47:59 +00:00
Eelco Dolstra d78a05ab40 * Make `import' work. 2010-03-30 09:22:33 +00:00
Eelco Dolstra 31428c3a06 * Started integrating the new evaluator. 2010-03-29 14:37:56 +00:00
Eelco Dolstra 50d11b90ca * Allow unsafe (unspecified) comparisons between attrsets unless
NIX_NO_UNSAFE_EQ is set, for now.
2009-05-12 11:06:24 +00:00
Eelco Dolstra 05879db628 * Memoize strict evaluation. 2007-01-13 15:41:54 +00:00
Eelco Dolstra d7efd76394 * Big cleanup of the semantics of paths, strings, contexts, string
concatenation and string coercion.  This was a big mess (see
  e.g. NIX-67).  Contexts are now folded into strings, so that they
  don't cause evaluation errors when they're not expected.  The
  semantics of paths has been clarified (see nixexpr-ast.def).
  toString() and coerceToString() have been merged.

  Semantic change: paths are now copied to the store when they're in a
  concatenation (and in most other situations - that's the
  formalisation of the meaning of a path).  So

    "foo " + ./bla

  evaluates to "foo /nix/store/hash...-bla", not "foo
  /path/to/current-dir/bla".  This prevents accidental impurities, and
  is more consistent with the treatment of derivation outputs, e.g.,
  `"foo " + bla' where `bla' is a derivation.  (Here `bla' would be
  replaced by the output path of `bla'.)
2006-10-16 15:55:34 +00:00
Eelco Dolstra 2ab4bc44c7 * Builtin function `add' to add integers.
* Put common test functions in tests/lang/lib.nix.
2006-09-22 15:29:21 +00:00
Eelco Dolstra 75068e7d75 * Use a proper namespace.
* Optimise header file usage a bit.
* Compile the parser as C++.
2006-09-04 21:06:23 +00:00
Eelco Dolstra f93f7b75be * Okay, that's a bit harder than expected. 2006-08-30 13:10:04 +00:00