Commit graph

279 commits

Author SHA1 Message Date
pennae 34b72775cf make throw*Error member functions of EvalState
when we introduce position and symbol tables we'll need to do lookups to turn
indices into those tables into actual positions/symbols. having the error
functions as members of EvalState will avoid a lot of churn for adding lookups
into the tables for each caller.
2022-04-21 21:25:18 +02:00
pennae 90b5c0a1a6 turn primop names into strings
we don't *need* symbols here. the only advantage they have over strings is
making call-counting slightly faster, but that's a diagnostic feature and thus
needn't be optimized.

this also fixes a move bug that previously didn't show up: PrimOp structs were
accessed after being moved from, which technically invalidates them. previously
the names remained valid because Symbol copies on move, but strings are
invalidated. we now copy the entire primop struct instead of moving since primop
registration happen once and are not performance-sensitive.
2022-04-21 21:25:17 +02:00
Eelco Dolstra 5cd72598fe Add support for impure derivations
Impure derivations are derivations that can produce a different result
every time they're built. Example:

  stdenv.mkDerivation {
    name = "impure";
    __impure = true; # marks this derivation as impure
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    buildCommand = "date > $out";
  };

Some important characteristics:

* This requires the 'impure-derivations' experimental feature.

* Impure derivations are not "cached". Thus, running "nix-build" on
  the example above multiple times will cause a rebuild every time.

* They are implemented similar to CA derivations, i.e. the output is
  moved to a content-addressed path in the store. The difference is
  that we don't register a realisation in the Nix database.

* Pure derivations are not allowed to depend on impure derivations. In
  the future fixed-output derivations will be allowed to depend on
  impure derivations, thus forming an "impurity barrier" in the
  dependency graph.

* When sandboxing is enabled, impure derivations can access the
  network in the same way as fixed-output derivations. In relaxed
  sandboxing mode, they can access the local filesystem.
2022-03-31 13:43:20 +02:00
Eelco Dolstra fc35b11a7c Fix mismatched tag warning on clang 2022-03-25 15:22:22 +01:00
Eelco Dolstra 86b05ccd54 Only provide builtin.{getFlake,fetchClosure} is the corresponding experimental feature is enabled
This allows writing fallback code like

  if builtins ? fetchClosure then
    builtins.fetchClose { ... }
  else
    builtins.storePath ...
2022-03-25 14:04:18 +01:00
John Ericson 4d6a3806d2 Decode string context straight to using StorePaths
I gather decoding happens on demand, so I hope don't think this should
have any perf implications one way or the other.
2022-03-18 15:36:11 +00:00
John Ericson 91adfb8894 Create some type aliases for string Contexts 2022-03-11 22:30:10 +00:00
pennae 4d629c4f7a add HAVE_BOEHMGC guards to batched allocation functions 2022-03-09 00:18:50 +01:00
pennae c96460f352 force-inline a few much-used functions
these functions are called a whole lot, and they're all comparatively small.
always inlining them gives ~0.7% performance boost on eval.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.912 s ±  0.036 s    [User: 5.823 s, System: 0.856 s]
    Range (min … max):    6.849 s …  6.980 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     325.1 ms ±   2.5 ms    [User: 293.2 ms, System: 31.8 ms]
    Range (min … max):   322.2 ms … 332.8 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.636 s ±  0.024 s    [User: 2.352 s, System: 0.226 s]
    Range (min … max):    2.574 s …  2.681 s    20 runs
2022-03-08 23:30:18 +01:00
pennae 60ed4e908a cache singleton Envs just like Values
vast majority of envs is this size.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.946 s ±  0.041 s    [User: 5.875 s, System: 0.835 s]
    Range (min … max):    6.834 s …  7.005 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.3 ms ±   2.5 ms    [User: 299.2 ms, System: 30.9 ms]
    Range (min … max):   327.5 ms … 337.7 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.671 s ±  0.035 s    [User: 2.370 s, System: 0.232 s]
    Range (min … max):    2.597 s …  2.749 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs
2022-03-08 23:30:18 +01:00
Eelco Dolstra b55d79728c Add EvalState::coerceToStorePath() helper
This is useful whenever we want to evaluate something to a store path
(e.g. in get-drvs.cc).

Extracted from the lazy-trees branch (where we can require that a
store path must come from a store source tree accessor).
2022-03-02 23:58:58 +01:00
Eelco Dolstra d974d2ad59 fetch{url,Tarball}: Remove 'narHash' attribute
This was introduced in #6174. However fetch{url,Tarball} are legacy
and we shouldn't have an undocumented attribute that does the same
thing as one that already exists ('sha256').
2022-03-01 11:30:26 +01:00
Robert Hensing ee019d0afc Add EvalState::allowAndSetStorePathString helper
This switches addPath from `printStorePath` to `toRealPath`.
2022-02-28 21:37:49 +01:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra bd383d1b6f Make most calls to determinePos() lazy 2022-02-04 00:33:21 +01:00
Eelco Dolstra 4c755c3b3f Merge branch 'issue-3505' of https://github.com/kamadorueda/nix 2022-02-04 00:33:13 +01:00
pennae d439dceb3b optionally return string_view from coerceToString
we'll retain the old coerceToString interface that returns a string, but callers
that don't need the returned value to outlive the Value it came from can save
copies by using the new interface instead. for values that weren't stringy we'll
pass a new buffer argument that'll be used for storage and shouldn't be
inspected.
2022-01-27 22:15:30 +01:00
pennae 41d70a2fc8 return string_views from forceString*
once a string has been forced we already have dynamic storage allocated for it,
so we can easily reuse that storage instead of copying.
2022-01-27 17:15:43 +01:00
pennae 0d7fae6a57 convert a for more utilities to string_view 2022-01-27 17:15:43 +01:00
pennae fd5aa6ee3e allocate a GC root value for the Value cache pointer
keeping it as a simple data member means it won't be scanned by the GC, so
eventually the GC will collect a cache that is still referenced (resulting in
use-after-free of cache elements).

fixes #5962
2022-01-22 21:19:56 +01:00
Kevin Amado 1472e045a7
avoid unnecesary calls 2022-01-21 16:32:43 -05:00
Kevin Amado 49b0bb0206
forceValue: make pos mandatory
- Make passing the position to `forceValue` mandatory,
  this way we remember people that the position is
  important for better error messages
- Add pos to all `forceValue` calls
2022-01-21 16:32:43 -05:00
Eelco Dolstra 4af88a4c91
Merge pull request #5906 from pennae/primops-optimization
optimize primops and utils by caching more and copying less
2022-01-18 19:43:28 +01:00
Eelco Dolstra fc2443a67c
Merge pull request #5812 from pennae/small-perf-improvements
improve parser performance a bit
2022-01-17 19:49:52 +01:00
pennae ad60dfde2a also cache split regexes, not just match regexes
gives about 1% improvement on system eval, a bit less on nix search.

 # before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      7.419 s ±  0.045 s    [User: 6.362 s, System: 0.794 s]
    Range (min … max):    7.335 s …  7.517 s    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.921 s ±  0.023 s    [User: 2.626 s, System: 0.210 s]
    Range (min … max):    2.883 s …  2.957 s    20 runs

 # after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      7.370 s ±  0.059 s    [User: 6.333 s, System: 0.791 s]
    Range (min … max):    7.286 s …  7.541 s    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.891 s ±  0.033 s    [User: 2.606 s, System: 0.210 s]
    Range (min … max):    2.823 s …  2.958 s    20 runs
2022-01-14 14:04:17 +01:00
pennae 34e3bd10e3 avoid copies of parser input data
when given a string yacc will copy the entire input to a newly allocated
location so that it can add a second terminating NUL byte. since the
parser is a very internal thing to EvalState we can ensure that having
two terminating NUL bytes is always possible without copying, and have
the parser itself merely check that the expected NULs are present.

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     572.4 ms ±   2.3 ms    [User: 563.4 ms, System: 8.6 ms]
  Range (min … max):   566.9 ms … 579.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     381.7 ms ±   1.0 ms    [User: 348.3 ms, System: 33.1 ms]
  Range (min … max):   380.2 ms … 387.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.936 s ±  0.005 s    [User: 2.715 s, System: 0.221 s]
  Range (min … max):    2.923 s …  2.946 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     571.7 ms ±   2.4 ms    [User: 563.3 ms, System: 8.0 ms]
  Range (min … max):   566.7 ms … 579.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     376.6 ms ±   1.0 ms    [User: 345.8 ms, System: 30.5 ms]
  Range (min … max):   374.5 ms … 379.1 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.922 s ±  0.006 s    [User: 2.707 s, System: 0.215 s]
  Range (min … max):    2.906 s …  2.934 s    50 runs
2022-01-13 18:06:15 +01:00
pennae 1bebb1095a cache more often-used symbols for primops
there's a few symbols in primops we can create once and pick them out of
EvalState afterwards instead of creating them every time we need them. this
gives almost 1% speedup to an uncached nix search.
2022-01-13 13:58:33 +01:00
Eelco Dolstra 2b4c944823 Remove EvalState::mkAttrs() 2022-01-04 20:29:17 +01:00
Eelco Dolstra 17daec0b83 Move empty attrset optimisation 2022-01-04 19:23:11 +01:00
Eelco Dolstra cc08364315 Remove non-method mkString() 2022-01-04 18:24:42 +01:00
Eelco Dolstra 6d9a6d2cc3 Ensure that attrsets are sorted
Previously you had to remember to call value->attrs->sort() after
populating value->attrs. Now there is a BindingsBuilder helper that
wraps Bindings and ensures that sort() is called before you can use
it.
2022-01-04 18:00:33 +01:00
Eelco Dolstra 89faff93e2
Merge pull request #5807 from NixOS/5805-ca-ifd
Fix IFD with CA derivations
2021-12-21 18:47:34 +01:00
regnat d90f9d4b99 Fix IFD with CA derivations
Rewrite the string taken by the IFD-like primops to contain the actual
output paths of the derivations rather than the placeholders

Fix #5805
2021-12-21 09:36:50 +01:00
pennae 09b245690a bulk-allocate Value instances in the evaluator
calling GC_malloc for each value is significantly more expensive than
allocating a bunch of values at once with GC_malloc_many. "a bunch" here
is a GC block size, ie 16KiB or less.

this gives a 1.5% performance boost when evaluating our nixos system.

tested with

nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

 # on master

  Time (mean ± σ):      3.335 s ±  0.007 s    [User: 2.774 s, System: 0.293 s]
  Range (min … max):    3.315 s …  3.347 s    50 runs

 # with this change

  Time (mean ± σ):      3.288 s ±  0.006 s    [User: 2.728 s, System: 0.292 s]
  Range (min … max):    3.274 s …  3.307 s    50 runs
2021-12-20 23:01:28 +01:00
Eelco Dolstra 40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra 81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
regnat af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Eelco Dolstra d39692e6b3 Make builtins.{path,filterSource} work with chroot stores 2021-10-07 14:22:39 +02:00
Eelco Dolstra cfaad7168e Refactoring: Add allowPath() method 2021-10-07 12:11:00 +02:00
Maximilian Bosch 2b02ce0e48
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
I found it somewhat confusing to have an error like

    error: attribute 'getFlake' missing

if the required experimental-feature (`flakes`) is not enabled. Instead,
I'd expect Nix to throw an error just like it's the case when using e.g. `nix
flake` without `flakes` being enabled.

With this change, the error looks like this:

    $ nix-instantiate -E 'builtins.getFlake "nixpkgs"'
    error: Cannot call 'builtins.getFlake' because experimental Nix feature 'flakes' is disabled. You can enable it via '--extra-experimental-features flakes'.

           at «string»:1:1:

                1| builtins.getFlake "nixpkgs"
                 | ^

I didn't use `settings.requireExperimentalFeature` here on purpose
because this doesn't contain a position. Also, it doesn't seem as if we
need to catch the error and check for the missing feature here since
this already happens at evaluation time.
2021-09-29 11:57:15 +02:00
Eelco Dolstra 49a932fb18 nix --help: Display help using lowdown instead of man
Fixes #4476.
Fixes #5231.
2021-09-13 14:45:21 +02:00
Eelco Dolstra 7ee639f9db
Merge pull request #5066 from Radvendii/master
add antiquotations to paths
2021-09-01 12:55:04 +02:00
Taeer Bar-Yam 9da8f5e25d path antiquotations: canonizePath -> canonicalizePath 2021-08-31 08:02:04 -04:00
Eelco Dolstra 323cafcb4e
Merge pull request #5191 from hercules-ci/evalstate-lifetime-hygiene
EvalState lifetime hygiene
2021-08-30 12:23:09 +02:00
Robert Hensing 8656b130ea Fix use after free with vImportedDrvToDerivation 2021-08-29 20:42:57 +02:00
Robert Hensing 8bc76acc7c Move vCallFlake into EvalState
This fixes a use-after-free bug:

1. s = new EvalState();
2. callFlake()
3. static vCallFlake now references s
4. delete s;
5. s2 = new EvalState();
6. callFlake()
7. static vCallFlake still references s
8. crash

Nix 2.3 did not have a problem with recreating EvalState.
2021-08-29 20:42:49 +02:00
Robert Hensing f10465774f Force all Pos* to be non-null
This fixes a class of crashes and introduces ptr<T> to make the
code robust against this failure mode going forward.

Thanks regnat for the idea of a ref<T> without overhead!

Closes #4895
Closes #4893
Closes #5127
Closes #5113
2021-08-29 18:11:58 +02:00
Taeer Bar-Yam 8f9429dcab add antiquotations to paths 2021-08-06 06:46:05 -04:00
Eelco Dolstra c000cec27f
Merge pull request #5048 from tweag/flox-eval-store
--eval-store and faster closure copying
2021-07-27 12:20:32 +02:00