Commit graph

2622 commits

Author SHA1 Message Date
Eelco Dolstra 9acc770ce4
Remove pre-C++11 hackiness 2022-05-26 12:40:01 +02:00
Eelco Dolstra 762fa2b2ff
Merge pull request #6570 from edolstra/eof-in-string
Handle EOFs in string literals correctly
2022-05-26 08:13:39 +02:00
Ben Burdette 9a5ea6c359
Merge branch 'master' into debug-exploratory-PR 2022-05-25 10:41:10 -06:00
Ben Burdette b4c24a29c6 back to ref<EvalState> in NixRepl 2022-05-25 10:21:20 -06:00
Eelco Dolstra 27ebb97d0a
Handle EOFs in string literals correctly
We can't return a STR token without setting a valid StringToken,
otherwise the parser will crash.

Fixes #6562.
2022-05-25 17:58:13 +02:00
Eelco Dolstra 91b7d5373a
Style tweaks 2022-05-25 12:32:22 +02:00
Ben Burdette 13d02af079 remove redundant 'debugMode' flag 2022-05-22 21:45:24 -06:00
Ben Burdette 7ccb2700c0 comments 2022-05-22 19:15:58 -06:00
Ben Burdette 5bc97fdfa6 change state derefs 2022-05-22 19:12:03 -06:00
Ben Burdette 982c8a959b remove special tryEval behavior 2022-05-20 12:45:36 -06:00
Ben Burdette 884d591787 debugRepl ftn pointer 2022-05-20 10:33:50 -06:00
Ben Burdette 0600df86b8 'debugMode' 2022-05-19 17:01:23 -06:00
Ben Burdette 7ddef73d02 de-const evalState exceptions 2022-05-19 12:44:40 -06:00
Ben Burdette f9cdb6af8d Merge branch 'debug-exploratory-PR' into debuggerHook-eval-arg 2022-05-19 11:07:18 -06:00
Ben Burdette 357fb84dba use an expr->StaticEnv table in evalState 2022-05-19 10:48:10 -06:00
Théophane Hufschmitt 78dc64ec1e
Merge pull request #6544 from ncfavier/getFlake-no-write-lock-file
Do not attempt to write a lock file in builtins.getFlake
2022-05-18 16:41:13 +02:00
Naïm Favier 169384abb2
Do not attempt to write a lock file in builtins.getFlake
Fixes https://github.com/NixOS/nix/issues/6541
2022-05-18 15:45:06 +02:00
zhujun b8e44dc62b primop_match: fix example letter case in document 2022-05-18 14:05:26 +08:00
Ben Burdette 667074b586 first whack at passing evalState as an arg to debuggerHook. 2022-05-16 09:20:51 -06:00
Théophane Hufschmitt 43a2c13672 Make nix::eval_cache::int_t more idiomatic
Don’t explicitely give it a constructor, but use aggregate
initialization instead (also prevents having an implicit coertion, which
is probably good here)
2022-05-16 16:36:21 +02:00
Eli Kogan-Wang 27d0f6747d resolve redundant priority passing, wrap NixInt in eval-cache variant 2022-05-16 15:17:35 +02:00
Eli Kogan-Wang c81d24f1c7 Add int to eval-cache, bump eval cache schema version 2022-05-16 02:29:29 +02:00
elikoga 0cea59ab4b
Merge branch 'NixOS:master' into master 2022-05-16 01:01:37 +02:00
Ben Burdette 86ba0a702c fix thunk issue 2022-05-15 12:05:51 -06:00
Eli Kogan-Wang be2b19041e Integrate review changes 2022-05-13 22:02:28 +02:00
Eelco Dolstra bf89cd95a4
Merge pull request #6525 from J-Swift/feature/bash-prefix
Add `bash-prompt-prefix` option
2022-05-13 14:06:07 +02:00
Ben Burdette 2acdb90438 remove debug code 2022-05-12 14:20:45 -06:00
Ben Burdette 2d0d1ec99d remove debug code 2022-05-12 14:15:35 -06:00
Ben Burdette 4f48095c66 Merge branch 'debugThrow' into debug-exploratory-PR 2022-05-12 14:11:35 -06:00
Ben Burdette 1ea13084c9 template-ize debugThrow 2022-05-12 13:59:58 -06:00
Eelco Dolstra 54457382f9
Fix static build
https://hydra.nixos.org/build/176211267
2022-05-11 11:36:56 +02:00
Jimmy Reichley 2998527b18
Allow setting bash-prompt-prefix nix develop configuration 2022-05-10 16:53:22 -04:00
Ben Burdette 7cd7c7c91a
Merge branch 'master' into debug-exploratory-PR 2022-05-09 09:30:44 -06:00
Jan Tojnar 59d9551c25
libexpr: Fix manual link in error message
It was changed to the old manual in 8895fa70a4
2022-05-08 18:59:00 +02:00
Andreas Rammhold 059ae7f6c4
Add unit tests for libexpr (#5377)
* libexpr: fix builtins.split example

The example was previously indicating that multiple whitespaces would be
collapsed into a single captured whitespace. That isn't true and was
likely a mistake when being documented initially.

* Fix segfault on unitilized list when printing value

Since lists are just chunks of memory the individual elements in the
list might be unitilized when a programming error happens within Nix.

In this case the values are null-initialized (at least with Boehm GC)
and we can avoid a nullptr deref when printing them.

I ran into this issue while ensuring that new expression tests would
show the actual value on an assertion failure.

This is unlikely to cause any runtime performance regressions as
printing values is not really in the hot path (unless the repl is the
primary use case).

* Add operator<< for ValueTypes

* Add libexpr tests

This introduces tests for libexpr that evalulate various trivial Nix
language expressions and primop invocations that should be good smoke
tests wheter or not the implementation is behaving as expected.
2022-05-06 18:05:27 +02:00
Ben Burdette fc66f48812 debugError() 2022-05-06 09:09:49 -06:00
Ben Burdette 2c9fafdc9e trying debugThrow 2022-05-06 08:47:21 -06:00
Ben Burdette dea998b2f2 traceable_allocator 2022-05-05 20:26:10 -06:00
Ben Burdette f400c5466d rename valmap 2022-05-05 15:43:23 -06:00
Ben Burdette 09fcfee925 don't print the 'break' argument 2022-05-05 15:34:59 -06:00
Ben Burdette 0ac121a094
Merge branch 'NixOS:master' into debug-exploratory-PR 2022-05-05 13:54:20 -06:00
Eelco Dolstra 58645a78ab builtins.break: Return argument when debugging is not enabled 2022-05-05 17:17:03 +02:00
Eelco Dolstra dd8b91eebc Style fixes
In particular, use std::make_shared and enumerate(). Also renamed some
fields to fit naming conventions.
2022-05-05 17:17:03 +02:00
Alexander Shpilkin b3ed32d0fd Add forgotten null check 2022-05-04 22:13:49 +03:00
Eelco Dolstra c98648bef0 Merge remote-tracking branch 'origin/master' into debug-exploratory-PR 2022-05-04 14:10:21 +02:00
Alain Zscheile 1385b20078
Get rid of most .at calls (#6393)
Use one of `get` or `getOr` instead which will either return a null-pointer (with a nicer error message) or a default value when the key is missing.
2022-05-04 07:44:32 +02:00
Eelco Dolstra 4a79cba511 Allow selecting derivation outputs using 'installable!outputs'
E.g. 'nixpkgs#glibc^dev,static' or 'nixpkgs#glibc^*'.
2022-05-03 13:43:52 +02:00
Ben Burdette 172a83d22a line endings 2022-04-29 11:24:54 -06:00
Ben Burdette ca6cba8b81 fix 'suggestions' error 2022-04-29 10:51:10 -06:00
Ben Burdette 2a5632c70d incorporate PosIdx changes, symbol changes. 2022-04-29 10:02:17 -06:00
Guillaume Maudoux e93b59fbc5 Merge remote-tracking branch 'origin/master' into coerce-string 2022-04-29 00:12:25 +02:00
Ben Burdette 6e19947993 Merge branch 'master' into debug-merge-master 2022-04-28 12:32:57 -06:00
Guillaume Maudoux f6baa4d188 fixup! fix errors case and wording 2022-04-28 13:18:19 +02:00
Guillaume Maudoux 402ee8ab64 No point in passing string_views by reference 2022-04-28 13:02:39 +02:00
Guillaume Maudoux acf990c9ea fix errors case and wording 2022-04-28 12:54:14 +02:00
Eelco Dolstra 717298c749 Bump eval cache schema version 2022-04-26 17:17:51 +02:00
Eelco Dolstra 1ddabe1a01 nix: Respect meta.outputsToInstall, and use all outputs by default
'nix profile install' will now install all outputs listed in the
package's meta.outputsToInstall attribute, or all outputs if that
attribute doesn't exist. This makes it behave consistently with
nix-env. Fixes #6385.

Furthermore, for consistency, all other 'nix' commands do this as
well. E.g. 'nix build' will build and symlink the outputs in
meta.outputsToInstall, defaulting to all outputs. Previously, it only
built/symlinked the first output. Note that this means that selecting
a specific output using attrpath selection (e.g. 'nix build
nixpkgs#libxml2.dev') no longer works. A subsequent PR will add a way
to specify the desired outputs explicitly.
2022-04-26 17:17:27 +02:00
Eelco Dolstra b12c33510c EvalCache AttrKey: Use Symbol instead of std::string 2022-04-26 14:16:20 +02:00
Eelco Dolstra 474695975d EvalCache: Revert to using symbols in getAttr() 2022-04-26 14:01:42 +02:00
Eelco Dolstra fab731a9d4 Don't pass Symbol by reference
Since Symbol is just an integer, passing it by const reference is
never advantageous.
2022-04-26 13:25:17 +02:00
pennae a385e51a08 rename SymbolIdx -> Symbol, Symbol -> SymbolStr
after #6218 `Symbol` no longer confers a uniqueness invariant on the
string it wraps, it is now possible to create multiple symbols that
compare equal but whose string contents have different addresses. this
guarantee is now only provided by `SymbolIdx`, leaving `Symbol` only as
a string wrapper that knows about the intricacies of how symbols need to
be formatted for output.

this change renames `SymbolIdx` to `Symbol` to restore the previous
semantics of `Symbol` to that name. we also keep the wrapper type and
rename it to `SymbolStr` instead of returning plain strings from lookups
into the symbol table because symbols are formatted for output in many
places. theoretically we do not need `SymbolStr`, only a function that
formats a string for output as a symbol, but having to wrap every symbol
that appears in a message into eg `formatSymbol()` is error-prone and
inconvient.
2022-04-25 15:37:01 +02:00
Théophane Hufschmitt c4ffc8e2f8
Merge pull request #6218 from pennae/pos-symbol-tables
reduce the size of Attr from 3 pointers to 2 on 64 bit machines
2022-04-22 10:28:06 +02:00
Théophane Hufschmitt 7ca6fbc8ca Move ChunkedVector to its own header 2022-04-22 10:01:02 +02:00
Tom Bereknyei f25112d383 fix: builtins.toFile adds path to allowedPaths
The produced path is then allowed be imported or utilized elsewhere:
```
assert (43 == import (builtins.toFile "source" "43")); "good"
```

This will still fail on write-only stores.
2022-04-21 16:41:37 -04:00
pennae 8168a4cf4a shrink Attr by 8 bytes on 64bit machines
with position and symbol tables in place we can now shrink Attr by a full
pointer with some simple field reordering. since Attr is a very hot struct this
has substantial impact on memory use, decreasing GC allocations and heap size by
10-15% each. we also get a ~15% performance improvement due to reduced GC
loading.

pure parsing has taken a hit over the branch base because positions are now
slightly more expensive to create, but overall we get a noticeable improvement.

before (on memory-friendliness):

  Benchmark 1: nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.960 s ±  0.028 s    [User: 5.832 s, System: 0.897 s]
    Range (min … max):    6.886 s …  7.005 s    20 runs

  Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     328.1 ms ±   1.7 ms    [User: 295.8 ms, System: 32.2 ms]
    Range (min … max):   324.9 ms … 331.2 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.688 s ±  0.029 s    [User: 2.365 s, System: 0.238 s]
    Range (min … max):    2.642 s …  2.742 s    20 runs

after:

  Benchmark 1: nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.902 s ±  0.039 s    [User: 5.844 s, System: 0.783 s]
    Range (min … max):    6.820 s …  6.956 s    20 runs

  Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.7 ms ±   2.2 ms    [User: 300.6 ms, System: 30.0 ms]
    Range (min … max):   327.5 ms … 334.5 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.330 s ±  0.027 s    [User: 2.040 s, System: 0.234 s]
    Range (min … max):    2.272 s …  2.383 s    20 runs
2022-04-21 21:56:34 +02:00
pennae 8775be3393 store Symbols in a table as well, like positions
this slightly increases the amount of memory used for any given symbol, but this
increase is more than made up for if the symbol is referenced more than once in
the EvalState that holds it. on average every symbol should be referenced at
least twice (once to introduce a binding, once to use it), so we expect no
increase in memory on average.

symbol tables are limited to 2³² entries like position tables, and similar
arguments apply to why overflow is not likely: 2³² symbols would require as many
string instances (at 24 bytes each) and map entries (at 24 bytes or more each,
assuming that the map holds on average at most one item per bucket as the docs
say). a full symbol table would require at least 192GB of memory just for
symbols, which is well out of reach. (an ofborg eval of nixpks today creates
less than a million symbols!)
2022-04-21 21:56:31 +02:00
pennae 00a3280232 don't use Symbol in Pos to represent a path
PosTable deduplicates origin information, so using symbols for paths is no
longer necessary. moving away from path Symbols also reduces the usage of
symbols for things that are not keys in attribute sets, which will become
important in the future when we turn symbols into indices as well.
2022-04-21 21:46:10 +02:00
pennae 6526d1676b replace most Pos objects/ptrs with indexes into a position table
Pos objects are somewhat wasteful as they duplicate the origin file name and
input type for each object. on files that produce more than one Pos when parsed
this a sizeable waste of memory (one pointer per Pos). the same goes for
ptr<Pos> on 64 bit machines: parsing enough source to require 8 bytes to locate
a position would need at least 8GB of input and 64GB of expression memory. it's
not likely that we'll hit that any time soon, so we can use a uint32_t index to
locate positions instead.
2022-04-21 21:46:06 +02:00
pennae 34b72775cf make throw*Error member functions of EvalState
when we introduce position and symbol tables we'll need to do lookups to turn
indices into those tables into actual positions/symbols. having the error
functions as members of EvalState will avoid a lot of churn for adding lookups
into the tables for each caller.
2022-04-21 21:25:18 +02:00
pennae 39df15fb8e don't use full Pos for findPackageFilename/editorFor
only file and line of the returned position were ever used, it wasn't actually
used a position. as such we may as well use a path+int pair for only those two
values and remove a use of Pos that would not work well with a position table.
2022-04-21 21:25:18 +02:00
pennae 38de79fcf7 remove Bindings::need
a future commit will remove the ability to convert the symbol type used in
bindings to strings. since we only have two users we can inline the error check.
2022-04-21 21:25:18 +02:00
pennae ff0fd91ed2 remove Symbol::empty
the only use of this function is to determine whether a lambda has a non-set
formal, but this use is arguably better served by Symbol::set and using a
non-Symbol instead of an empty symbol in the parser when no such formal is present.
2022-04-21 21:25:18 +02:00
pennae 90b5c0a1a6 turn primop names into strings
we don't *need* symbols here. the only advantage they have over strings is
making call-counting slightly faster, but that's a diagnostic feature and thus
needn't be optimized.

this also fixes a move bug that previously didn't show up: PrimOp structs were
accessed after being moved from, which technically invalidates them. previously
the names remained valid because Symbol copies on move, but strings are
invalidated. we now copy the entire primop struct instead of moving since primop
registration happen once and are not performance-sensitive.
2022-04-21 21:25:17 +02:00
John Ericson 08b8657978 Merge branch 'path-info' into ca-drv-exotic 2022-04-19 22:39:57 +00:00
John Ericson 55caef36ed Merge remote-tracking branch 'upstream/master' into path-info 2022-04-19 22:27:21 +00:00
Eelco Dolstra 51712bf012
Merge pull request #6128 from ncfavier/fix-completion
Shell completion improvements
2022-04-19 13:45:33 +02:00
Eelco Dolstra d89840b103 Make InstallableFlake::toValue() and toDerivation() behave consistently
In particular, this means that 'nix eval` (which uses toValue()) no
longer auto-calls functions or functors (because
AttrCursor::findAlongAttrPath() doesn't).

Fixes #6152.

Also use ref<> in a few places, and don't return attrpaths from
getCursor() because cursors already have a getAttrPath() method.
2022-04-14 14:07:04 +02:00
Ben Burdette 8b197c492e remove comma 2022-04-09 21:54:41 -06:00
Ben Burdette a61841ac41 don't use std::map merge 2022-04-09 07:45:23 -06:00
Ben Burdette 27d45f9eb3 minor cleanup 2022-04-08 15:46:12 -06:00
Ben Burdette b8b8ec7101 move throw to preverve Error type; turn off debugger for tryEval 2022-04-08 12:34:27 -06:00
Ben Burdette 1a93ac8133 Merge remote-tracking branch 'upstream/master' into upstream-merge 2022-04-07 13:42:01 -06:00
Ben Burdette 50b52d5110 remove debug code 2022-04-07 12:03:18 -06:00
Eelco Dolstra 589f6f267b fetchClosure: Don't allow URL query parameters
Allowing this is a potential security hole, since it allows the user
to specify parameters like 'local-nar-cache'.
2022-04-06 11:52:51 +02:00
Eelco Dolstra fdfe737867 Fix handling of outputHash when outputHashAlgo is not specified
https://hydra.nixos.org/build/171351131
2022-04-01 12:40:49 +02:00
Eelco Dolstra c9a29d0d92
Merge pull request #6227 from NixOS/impure-derivations-ng
Impure derivations
2022-03-31 19:58:35 +02:00
Ben Burdette f41c18b221 comments 2022-03-31 09:39:18 -06:00
Ben Burdette 1096d17b65 show 'with' bindings as well as static 2022-03-31 09:37:36 -06:00
Eelco Dolstra 7537097284 Provide default values for outputHashAlgo and outputHashMode 2022-03-31 16:56:44 +02:00
Eelco Dolstra 5cd72598fe Add support for impure derivations
Impure derivations are derivations that can produce a different result
every time they're built. Example:

  stdenv.mkDerivation {
    name = "impure";
    __impure = true; # marks this derivation as impure
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
    buildCommand = "date > $out";
  };

Some important characteristics:

* This requires the 'impure-derivations' experimental feature.

* Impure derivations are not "cached". Thus, running "nix-build" on
  the example above multiple times will cause a rebuild every time.

* They are implemented similar to CA derivations, i.e. the output is
  moved to a content-addressed path in the store. The difference is
  that we don't register a realisation in the Nix database.

* Pure derivations are not allowed to depend on impure derivations. In
  the future fixed-output derivations will be allowed to depend on
  impure derivations, thus forming an "impurity barrier" in the
  dependency graph.

* When sandboxing is enabled, impure derivations can access the
  network in the same way as fixed-output derivations. In relaxed
  sandboxing mode, they can access the local filesystem.
2022-03-31 13:43:20 +02:00
Daniel Pauls fa83b865a2 libexpr: Throw the correct error in toJSON
BaseError::addTrace(...) returns a BaseError, but we want to
throw a TypeError instead.

Fixes #6336.
2022-03-30 15:50:13 +02:00
Ben Burdette c0a567e196 remove const_cast 2022-03-29 16:44:47 -06:00
Théophane Hufschmitt 390269ed87 Simplify the handling of the hash modulo
Rather than having four different but very similar types of hashes, make
only one, with a tag indicating whether it corresponds to a regular of
deferred derivation.

This implies a slight logical change: The original Nix+multiple-outputs
model assumed only one hash-modulo per derivation. Adding
multiple-outputs CA derivations changed this as these have one
hash-modulo per output. This change is now treating each derivation as
having one hash modulo per output.
This obviously means that we internally loose the guaranty that
all the outputs of input-addressed derivations have the same hash
modulo. But it turns out that it doesn’t matter because there’s nothing
in the code taking advantage of that fact (and it probably shouldn’t
anyways).

The upside is that it is now much easier to work with these hashes, and
we can get rid of a lot of useless `std::visit{ overloaded`.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2022-03-29 18:17:35 +02:00
Ben Burdette 1bec3fb337 add DebugTrace for error 2022-03-25 18:15:31 -06:00
John Ericson ff2a8ccfe1 Merge branch 'path-info' into ca-drv-exotic 2022-03-25 19:40:52 +00:00
John Ericson 0dc2974930 Merge remote-tracking branch 'upstream/master' into path-info 2022-03-25 19:25:08 +00:00
Eelco Dolstra fc35b11a7c Fix mismatched tag warning on clang 2022-03-25 15:22:22 +01:00
Eelco Dolstra 8c363eb3eb Document getFlake
Fixes #5523.
2022-03-25 14:19:55 +01:00
Eelco Dolstra 86b05ccd54 Only provide builtin.{getFlake,fetchClosure} is the corresponding experimental feature is enabled
This allows writing fallback code like

  if builtins ? fetchClosure then
    builtins.fetchClose { ... }
  else
    builtins.storePath ...
2022-03-25 14:04:18 +01:00
Eelco Dolstra f902f3c2cb Add experimental feature 'fetch-closure' 2022-03-24 21:33:33 +01:00
Eelco Dolstra e5f7029ba4 nix store make-content-addressed: Support --from / --to 2022-03-24 21:33:33 +01:00
Eelco Dolstra 98658ae9d2 Document fetchClosure 2022-03-24 21:33:33 +01:00
Eelco Dolstra 28186b7044 Add a test for fetchClosure and 'nix store make-content-addressed' 2022-03-24 21:33:33 +01:00
Eelco Dolstra 4120930ac1 fetchClosure: Only allow some "safe" store types 2022-03-24 21:33:33 +01:00
Eelco Dolstra 7ffda0af6e fetchClosure: Skip makeContentAddressed() if toPath is already valid 2022-03-24 21:33:33 +01:00
Eelco Dolstra 545c2d0d8c fetchClosure: Allow a path to be rewritten to CA on the fly
The advantage is that the resulting closure doesn't need to be signed,
so you don't need to configure any binary cache keys on the client.
2022-03-24 21:33:33 +01:00
Eelco Dolstra 7f6fe8ca1d Rename 2022-03-24 21:33:33 +01:00
Eelco Dolstra 41659418cf fetchClosure: Require a CA path in pure mode 2022-03-24 21:33:33 +01:00
Eelco Dolstra f4bafc412f Add builtins.fetchClosure
This allows closures to be imported at evaluation time, without
requiring the user to configure substituters. E.g.

  builtins.fetchClosure {
    storePath = /nix/store/f89g6yi63m1ywfxj96whv5sxsm74w5ka-python3.9-sqlparse-0.4.2;
    from = "https://cache.ngi0.nixos.org";
  }
2022-03-24 21:33:33 +01:00
Eelco Dolstra d67fe90375
Merge pull request #6305 from flox/genericClosure_doc
docs: genericClosure
2022-03-24 14:02:58 +01:00
Tom Bereknyei 0736f3651d docs: genericClosure 2022-03-24 08:03:59 -04:00
Sergei Trofimovich 9174d884d7 lexer: add error location to lexer errors
Before the change lexter errors did not report the location:

    $ nix build -f. mc
    error: path has a trailing slash
    (use '--show-trace' to show detailed location information)

Note that it's not clear what file generates the error.

After the change location is reported:

    $ src/nix/nix --extra-experimental-features nix-command build -f ~/nm mc
    error: path has a trailing slash

           at .../pkgs/development/libraries/glib/default.nix:54:18:

               53|   };
               54|   src = /tmp/foo/;
                 |                  ^
               55|
    (use '--show-trace' to show detailed location information)

Here we see both problematic file and the string itself.
2022-03-24 08:16:14 +00:00
Eelco Dolstra a0259a21a4 Don't hide repeated values while generating manifest.nix
Fixes #6243.
2022-03-22 13:18:56 +01:00
Eelco Dolstra 732296ddc0 printValue(): <REPEAT> -> «repeated»
This ensures that it doesn't get parsed as a valid Nix expression.
2022-03-22 13:00:27 +01:00
Eelco Dolstra e4ff430866
Merge pull request #6237 from obsidiansystems/store-path-string-context
Decode string context straight to using StorePaths
2022-03-22 10:29:46 +01:00
John Ericson 4d6a3806d2 Decode string context straight to using StorePaths
I gather decoding happens on demand, so I hope don't think this should
have any perf implications one way or the other.
2022-03-18 15:36:11 +00:00
John Ericson a544ed7684 Generalize DerivationType in preparation for impure derivations 2022-03-18 14:59:56 +00:00
John Ericson 049fae155a Avoid some pointless copying of drvs 2022-03-18 14:59:56 +00:00
John Ericson 8496be7def Use Deferred when building an input-addressed drv
Easier than using dummy path with input addressed.
2022-03-18 14:59:56 +00:00
Guillaume Maudoux c2b620f3ad Try to fix issues with macos clang, v3 2022-03-18 15:35:24 +01:00
Guillaume Maudoux 726f5836d8 Try to fix issues with macos clang, v2 2022-03-18 15:22:25 +01:00
Guillaume Maudoux 37e84316c2 Try to fix issues with macos clang 2022-03-18 14:48:49 +01:00
Guillaume Maudoux 9c42c00570 Fix some error kind mismatches 2022-03-18 10:22:47 +01:00
Guillaume Maudoux ad3fadb95a fixup! Merge remote-tracking branch 'origin/master' into coerce-string 2022-03-18 10:11:36 +01:00
Guillaume Maudoux ca5c3e86ab Merge remote-tracking branch 'origin/master' into coerce-string 2022-03-18 01:25:55 +01:00
Guillaume Maudoux 1942fed6d9 Revert extra colon at end os strings 2022-03-18 01:10:04 +01:00
Guillaume Maudoux e6d07e0d89 Refactor to use more traces and less string manipulations 2022-03-18 00:58:09 +01:00
John Ericson 197feed51d Clean up DerivationOutput, and headers
1. `DerivationOutput` now as the `std::variant` as a base class. And the
   variants are given hierarchical names under `DerivationOutput`.

   In 8e0d0689be @matthewbauer and I
   didn't know a better idiom, and so we made it a field. But this sort
   of "newtype" is anoying for literals downstream.

   Since then we leaned the base class, inherit the constructors trick,
   e.g. used in `DerivedPath`. Switching to use that makes this more
   ergonomic, and consistent.

2. `store-api.hh` and `derivations.hh` are now independent.

   In bcde5456cc I swapped the dependency,
   but I now know it is better to just keep on using incomplete types as
   much as possible for faster compilation and good separation of
   concerns.
2022-03-17 22:35:53 +00:00
Théophane Hufschmitt a0b517de57
Merge pull request #6242 from ncfavier/print-output-names
nix-env: always print output names in JSON and XML
2022-03-17 10:55:22 +01:00
Naïm Favier 5736661922
nix-env: always print output names in JSON and XML
The current `--out-path` flag has two disadvantages when one is only
concerned with querying the names of outputs:
- it requires evaluating every output's `outPath`, which takes
  significantly more resources and runs into more failures
- it destroys the information of the order of outputs so we can't tell
  which one is the main output

This patch makes the output names always present (replacing paths with
`null` in JSON if `--out-path` isn't given), and adds an `outputName`
field.
2022-03-16 21:26:19 +01:00
Ben Burdette 88a54108eb formatting 2022-03-16 12:09:47 -06:00
Ben Burdette 3dfab6e534 have only one debuggerHook declaration 2022-03-14 11:58:11 -06:00
Ben Burdette eaecaaa00b more debug_throw coverage of EvalErrors 2022-03-14 11:39:53 -06:00
John Ericson 91adfb8894 Create some type aliases for string Contexts 2022-03-11 22:30:10 +00:00
John Ericson 0948b8e94d Reduce variants for derivation hash modulo
This changes was taken from dynamic derivation (#4628). It` somewhat
undoes the refactors I first did for floating CA derivations, as the
benefit of hindsight + requirements of dynamic derivations made me
reconsider some things.

They aren't to consequential, but I figured they might be good to land
first, before the more profound changes @thufschmitt has in the works.
2022-03-11 21:20:37 +00:00
Eelco Dolstra aee56e0f89 Merge remote-tracking branch 'origin/eval-suggestions' 2022-03-11 12:02:26 +01:00
Eelco Dolstra 31a392dfe2 Merge pull request #5865 from pennae/memory-friendliness
be more memory friendly
2022-03-11 11:52:39 +01:00
John Ericson 938650700f Merge branch 'path-info' into ca-drv-exotic 2022-03-10 16:20:01 +00:00
John Ericson 8ba089597f Merge remote-tracking branch 'upstream/master' into path-info 2022-03-10 15:48:14 +00:00
pennae 4d629c4f7a add HAVE_BOEHMGC guards to batched allocation functions 2022-03-09 00:18:50 +01:00
pennae 47baa9d43c make Pos smaller
reduces peak hep memory use on eval of our test system from 264.4MB to 242.3MB,
possibly also a slight performance boost.

theoretically memory use could be cut down by another eight bytes per Pos on
average by turning it into a tuple containing an index into a global base
position table with row and column offsets, but that doesn't seem worth the
effort at this point.
2022-03-08 23:30:18 +01:00
pennae c96460f352 force-inline a few much-used functions
these functions are called a whole lot, and they're all comparatively small.
always inlining them gives ~0.7% performance boost on eval.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.912 s ±  0.036 s    [User: 5.823 s, System: 0.856 s]
    Range (min … max):    6.849 s …  6.980 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     325.1 ms ±   2.5 ms    [User: 293.2 ms, System: 31.8 ms]
    Range (min … max):   322.2 ms … 332.8 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.636 s ±  0.024 s    [User: 2.352 s, System: 0.226 s]
    Range (min … max):    2.574 s …  2.681 s    20 runs
2022-03-08 23:30:18 +01:00
pennae 60ed4e908a cache singleton Envs just like Values
vast majority of envs is this size.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.946 s ±  0.041 s    [User: 5.875 s, System: 0.835 s]
    Range (min … max):    6.834 s …  7.005 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.3 ms ±   2.5 ms    [User: 299.2 ms, System: 30.9 ms]
    Range (min … max):   327.5 ms … 337.7 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.671 s ±  0.035 s    [User: 2.370 s, System: 0.232 s]
    Range (min … max):    2.597 s …  2.749 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.935 s ±  0.052 s    [User: 5.852 s, System: 0.853 s]
    Range (min … max):    6.808 s …  7.026 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.8 ms ±   2.7 ms    [User: 299.0 ms, System: 30.8 ms]
    Range (min … max):   326.6 ms … 336.5 ms    20 runs

  Benchmark 3: nix flakes eval --raw --impure --file expr.nix
    Time (mean ± σ):      2.655 s ±  0.038 s    [User: 2.364 s, System: 0.220 s]
    Range (min … max):    2.574 s …  2.737 s    20 runs
2022-03-08 23:30:18 +01:00
pennae 4b2b0d3a55 remove GC_PTR_STORE_AND_DIRTY
turns out it's only necessary for MANUAL_VDB, which nix doesn't use. omitting
them gives a slight performance improvement on eval.

before:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.988 s ±  0.061 s    [User: 5.935 s, System: 0.845 s]
    Range (min … max):    6.865 s …  7.075 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     332.6 ms ±   3.9 ms    [User: 299.6 ms, System: 32.9 ms]
    Range (min … max):   328.1 ms … 339.1 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.681 s ±  0.049 s    [User: 2.382 s, System: 0.228 s]
    Range (min … max):    2.607 s …  2.776 s    20 runs

after:

  Benchmark 1: nix flakes search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      6.946 s ±  0.041 s    [User: 5.875 s, System: 0.835 s]
    Range (min … max):    6.834 s …  7.005 s    20 runs

  Benchmark 2: nix flakes eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     330.3 ms ±   2.5 ms    [User: 299.2 ms, System: 30.9 ms]
    Range (min … max):   327.5 ms … 337.7 ms    20 runs

  Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.671 s ±  0.035 s    [User: 2.370 s, System: 0.232 s]
    Range (min … max):    2.597 s …  2.749 s    20 runs
2022-03-08 23:30:18 +01:00
regnat f6078e474d Also display some suggestions for invalid formal arguments
```console
$ nix eval --expr '({ foo ? 1 }: foo) { fob = 2; }'
error: anonymous function at (string):1:2 called with unexpected argument 'fob'

       at «string»:1:1:

            1| ({ foo ? 1 }: foo) { fob = 2; }
             | ^
       Did you mean foo?
```

Not that because Nix will first check for _missing_ arguments before
checking for extra arguments, `({ foo }: foo) { fob = 1; }` will
complain about the missing `foo` argument (rather than extra `fob`) and
so won’t display a suggestion.
2022-03-08 16:40:22 +01:00
regnat 33b7514035 Try and make the darwin build happy 2022-03-08 16:07:17 +01:00
regnat 0c6e46e349 Add some suggestions to the evaluator
Make the evaluator show some suggestions when trying to access an
invalid field from an attrset.

```console
$ nix eval --expr '{ foo = 1; }.foa'
error: attribute 'foa' missing

       at «string»:1:1:

            1| { foo = 1; }.foa
             | ^
       Did you mean foo?
```
2022-03-08 06:21:45 +01:00
Eelco Dolstra 30ddd37873 Merge branch 'cli-suggestions' of https://github.com/thufschmitt/nix 2022-03-07 19:47:45 +01:00
regnat dcf4f77fac Merge or-suggestions.hh into suggestions.hh
No real need for keeping a separate header for such a simple class.

This requires changing a bit `OrSuggestions<T>::operator*` to not throw
an `Error` to prevent a cyclic dependency. But since this error is only
thrown on programmer error, we can replace the whole method by a direct
call to `std::get` which will raise its own assertion if needs be.
2022-03-07 17:49:02 +01:00
Robert Hensing da260f579d dupStringWithLen -> makeImmutableString
Refactor the `size == 0` logic into a new helper function that
replaces dupStringWithLen.
The name had to change, because unlike a `dup`-function, it does
not always allocate a new string.
2022-03-07 16:09:12 +01:00
Robert Hensing bbf55383e7 Value::mkPath: Avoid potential crash from null string_view 2022-03-07 16:09:12 +01:00
Robert Hensing 1b978596b5 Value::mkString: Avoid crash from null string_view 2022-03-07 16:09:12 +01:00
Naïm Favier da7d8daa77
Add shell completion for --override-flake
Requires moving the MixEvalArgs class from libexpr to libcmd because
that's where completeFlakeRef is.
2022-03-07 12:01:54 +01:00
Guillaume Maudoux 13c4dc6532 more fixes 2022-03-07 11:33:03 +01:00
regnat fd45d85b41 Move OrSuggestions to its own header
Prevents a recursive inclusion
2022-03-07 10:09:10 +01:00
regnat 98e361ad4c Also display suggestions for the commands using the eval cache
Make `nix build .#nix-armv8l-linux` work for example
2022-03-07 10:09:10 +01:00
regnat c0792b1546 Implement a suggestions mechanism
Each `Error` class now includes a set of suggestions, and these are printed by
the top-level handler.
2022-03-07 10:09:09 +01:00
Guillaume Maudoux 1b5a8db148 change error location for genericClosure operator errors 2022-03-05 21:19:04 +01:00
Guillaume Maudoux 4078015927 DRY addPrimOp 2022-03-05 21:18:30 +01:00
Guillaume Maudoux cbbbf36881 Use 'errorCtx' name everywhere 2022-03-04 22:55:14 +01:00
Guillaume Maudoux 57684d6247 fixup! s/forceValue/forceFunction/ where applicable 2022-03-04 22:51:56 +01:00
Guillaume Maudoux ed02fa3c40 s/forceValue/forceFunction/ where applicable 2022-03-04 22:15:30 +01:00
Guillaume Maudoux 3a5855353e Add detailed error mesage for coerceTo{String,Path} 2022-03-04 21:47:58 +01:00
Guillaume Maudoux be1f069746 Add error context for most basic coercions 2022-03-04 05:04:47 +01:00
Guillaume Maudoux 00e242feed Add some context to coercion error strings 2022-03-03 21:46:20 +01:00
Eelco Dolstra e9c04c3351 Be more aggressive in hiding repeated values
We now memoize on Bindings / list element vectors rather than Values,
so that e.g. two Values that point to the same Bindings will be
printed only once.
2022-03-03 13:33:34 +01:00
Eelco Dolstra ecff9d969a printValue(): Don't show repeated values
Fixes #6157.
2022-03-03 13:18:23 +01:00
Eelco Dolstra 6097790863 Fix segfault in headerCallback()
https://hydra.nixos.org/build/168594664
2022-03-03 11:11:16 +01:00
Eelco Dolstra b55d79728c Add EvalState::coerceToStorePath() helper
This is useful whenever we want to evaluate something to a store path
(e.g. in get-drvs.cc).

Extracted from the lazy-trees branch (where we can require that a
store path must come from a store source tree accessor).
2022-03-02 23:58:58 +01:00
Eelco Dolstra 0cb5af5000
Merge pull request #6185 from hercules-ci/fetchTree-reuse-local-paths
fetchTree: Use isValidPath, add comment
2022-03-01 13:15:42 +01:00
Robert Hensing b6deca7c0d fetchTree: Use isValidPath, add comment 2022-03-01 12:11:10 +01:00
Eelco Dolstra d974d2ad59 fetch{url,Tarball}: Remove 'narHash' attribute
This was introduced in #6174. However fetch{url,Tarball} are legacy
and we shouldn't have an undocumented attribute that does the same
thing as one that already exists ('sha256').
2022-03-01 11:30:26 +01:00
Eelco Dolstra 9ab81a9d38
Merge pull request #6183 from obsidiansystems/sort-config
Move some stuff from `Settings` to a new `FetchSettings`.
2022-03-01 10:52:55 +01:00
John Ericson ea71da395f Move some stuff from Settings to a new FetchSettings.
Starting work on #5638

The exact boundary between `FetchSettings` and `EvalSettings` is not
clear to me, but that's fine. First lets clean out `libstore`, and then
worry about what, if anything, should be the separation between those
two.
2022-03-01 01:39:25 +00:00
Robert Hensing ee019d0afc Add EvalState::allowAndSetStorePathString helper
This switches addPath from `printStorePath` to `toRealPath`.
2022-02-28 21:37:49 +01:00
Robert Hensing 158280d8e9 fetchTree: Do not re-fetch paths already present 2022-02-28 21:37:49 +01:00
Eelco Dolstra df552ff53e Remove std::string alias (for real this time)
Also use std::string_view in a few more places.
2022-02-25 16:13:02 +01:00
Eelco Dolstra 9d4697453f ExprAttrs::show(): Show attrs in sorted order
Issue #6160.
2022-02-25 09:21:04 +01:00
Eelco Dolstra 8518cebfce libfetchers: Rename immutable -> locked
This is more consistent with flake terminology.
2022-02-24 18:09:00 +01:00
Eelco Dolstra 36c7b12f33 Remove std::string alias 2022-02-21 16:37:25 +01:00
Eelco Dolstra 1ac2664472 Remove std::vector alias 2022-02-21 16:32:34 +01:00
Eelco Dolstra fe9afb65bb Remove std::set alias 2022-02-21 16:28:23 +01:00
Eelco Dolstra afcdc7606c Remove std::list alias 2022-02-21 16:25:12 +01:00
Ben Burdette 3d94d3ba91 Expr refs instead of pointers 2022-02-15 15:46:45 -07:00
Ben Burdette c9bc3735f6 quit repl from step mode 2022-02-15 09:49:25 -07:00
Ben Burdette e761bf0601 make an 'info' level error on break 2022-02-14 14:04:34 -07:00
Ben Burdette 4cffb130e3 for primops, enter the debugger at the last DebugTrace in the stack 2022-02-11 14:14:25 -07:00
Ben Burdette 3ff5ac3586 update the eval-inline throw fns 2022-02-10 16:01:49 -07:00
Ben Burdette bc67cb5ad1 remove fakeEnv stuff and instead use last context from the stack 2022-02-10 15:05:38 -07:00
Eelco Dolstra 93293fc66b
Merge pull request #6042 from pennae/fix-repl-a
fix nix repl not overriding existing bindings in :a
2022-02-06 13:17:58 +01:00
Ben Burdette 195db83148 a few merge fixes 2022-02-04 17:35:56 -07:00
Ben Burdette dbe3fd3735 Merge branch 'master' into debug-step 2022-02-04 15:09:40 -07:00
Ben Burdette 3ddf864e1b print value in break 2022-02-04 14:50:25 -07:00
pennae 1daf1babf9 fix nix repl not overriding existing bindings in :a
previously :a would override old bindings of a name with new values if the added
set contained names that were already bound. in nix 2.6 this doesn't happen any
more, which is potentially confusing.

fixes #6041
2022-02-04 15:27:59 +01:00
Eelco Dolstra bd383d1b6f Make most calls to determinePos() lazy 2022-02-04 00:33:21 +01:00
Eelco Dolstra 4c755c3b3f Merge branch 'issue-3505' of https://github.com/kamadorueda/nix 2022-02-04 00:33:13 +01:00
Ben Burdette 412d58f0bb break() primop; step and go debug commands 2022-02-03 13:15:21 -07:00
Eelco Dolstra fcb33440b6
Merge pull request #6036 from tweag/balsoft/and-yet-another-follows-bugfix
Flake follows: resolve all follows to absolute
2022-02-02 22:52:08 +01:00
Alexander Bantyev 169ea0b83f
Flake follows: resolve all follows to absolute
It's not possible in general to know in computeLocks, relative to
which path the follows was intended to be. So, we always resolve
follows to their absolute states when we encounter them (which can
either be in parseFlakeInput or computeLocks' fake input population).

Fixes https://github.com/NixOS/nix/issues/6013
Fixes https://github.com/NixOS/nix/issues/5609
Fixes https://github.com/NixOS/nix/issues/5697 (again)
2022-02-02 21:41:45 +03:00
Eelco Dolstra 17e3f353df Merge branch 'parser-improvements' of https://github.com/pennae/nix 2022-02-02 12:45:44 +01:00
Eelco Dolstra cd35bbbeef Merge branch 'more-stringviews' of https://github.com/pennae/nix 2022-02-02 12:38:37 +01:00
Thomas Koch 85b1427662 fix spelling mistakes reported by Debian's lintian tool 2022-01-30 10:51:39 +02:00
Eelco Dolstra 4bf6af7b55 Remove a repeated std::move in a for loop 2022-01-28 15:10:43 +01:00
pennae d439dceb3b optionally return string_view from coerceToString
we'll retain the old coerceToString interface that returns a string, but callers
that don't need the returned value to outlive the Value it came from can save
copies by using the new interface instead. for values that weren't stringy we'll
pass a new buffer argument that'll be used for storage and shouldn't be
inspected.
2022-01-27 22:15:30 +01:00
pennae 41d70a2fc8 return string_views from forceString*
once a string has been forced we already have dynamic storage allocated for it,
so we can easily reuse that storage instead of copying.
2022-01-27 17:15:43 +01:00
pennae 0d7fae6a57 convert a for more utilities to string_view 2022-01-27 17:15:43 +01:00
regnat fcdc60ed22 Don’t require NIX_PATH entries to be valid paths
It’s totally valid to have entries in `NIX_PATH` that aren’t valid paths
(they can even be arbitrary urls or `channel:<channel-name>`).

Fix #5998 and #5980
2022-01-27 16:26:39 +01:00
regnat f113ea6c73 Fix parsing of variable names that are a suffix of '__curPos'
Follow-up from #5969
Fix #5982
2022-01-25 10:49:27 +01:00
Eelco Dolstra 8cbbaf23e8 Allow builtins.{readFile,path} on invalid paths
Stop-gap measure to fix #5975.
2022-01-24 23:02:28 +01:00
Eelco Dolstra bed8270c0c Fix parsing of variable names that are a prefix of '__curPos'
Fixes

  $ nix-instantiate --parse -E 'x: with x; _'
  (x: (with x; __curPos))
2022-01-24 15:18:18 +01:00
Eelco Dolstra c4fc9b6a8d ExprConcatStrings::show(): Print values instead of pointers 2022-01-24 15:08:16 +01:00
pennae fd5aa6ee3e allocate a GC root value for the Value cache pointer
keeping it as a simple data member means it won't be scanned by the GC, so
eventually the GC will collect a cache that is still referenced (resulting in
use-after-free of cache elements).

fixes #5962
2022-01-22 21:19:56 +01:00
Kevin Amado 50efc5499a
determinePos: remove from critical path 2022-01-21 16:32:43 -05:00
Kevin Amado c3896e19d0
forceAttrs: make pos mandatory 2022-01-21 16:32:43 -05:00
Kevin Amado 1472e045a7
avoid unnecesary calls 2022-01-21 16:32:43 -05:00
Kevin Amado 49b0bb0206
forceValue: make pos mandatory
- Make passing the position to `forceValue` mandatory,
  this way we remember people that the position is
  important for better error messages
- Add pos to all `forceValue` calls
2022-01-21 16:32:43 -05:00
Eelco Dolstra 128098040b
Fix exception handling around realisePath()
This no longer worked correctly because 'path' is uninitialised when
an exception occurs, leading to errors like

       … while importing ''

       at /nix/store/rrzz5b1pshvzh1437ac9nkl06br81lkv-source/flake.nix:352:13:

So move the adding of the error context into realisePath().
2022-01-21 13:53:18 +01:00
pennae 7d4cc5515c defer formals duplicate check for incresed efficiency all round
if we defer the duplicate argument check for lambda formals we can use more
efficient data structures for the formals set, and we can get rid of the
duplication of formals names to boot. instead of a list of formals we've seen
and a set of names we'll keep a vector instead and run a sort+dupcheck step
before moving the parsed formals into a newly created lambda. this improves
performance on search and rebuild by ~1%, pure parsing gains more (about 4%).

this does reorder lambda arguments in the xml output, but the output is still
stable. this shouldn't be a problem since argument order is not semantically
important anyway.

 before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.550 s ±  0.060 s    [User: 6.470 s, System: 1.664 s]
    Range (min … max):    8.435 s …  8.666 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     346.7 ms ±   2.1 ms    [User: 312.4 ms, System: 34.2 ms]
    Range (min … max):   343.8 ms … 353.4 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.720 s ±  0.031 s    [User: 2.415 s, System: 0.231 s]
    Range (min … max):    2.662 s …  2.780 s    20 runs

 after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.462 s ±  0.063 s    [User: 6.398 s, System: 1.661 s]
    Range (min … max):    8.339 s …  8.542 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.1 ms ±   1.4 ms    [User: 296.8 ms, System: 32.3 ms]
    Range (min … max):   326.1 ms … 330.8 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.687 s ±  0.035 s    [User: 2.392 s, System: 0.228 s]
    Range (min … max):    2.626 s …  2.754 s    20 runs
2022-01-19 17:07:29 +01:00
pennae 9ac836d1d6 don't use Symbols for strings
string expressions by and large do not need the benefits a Symbol gives us,
instead they pollute the symbol table and cause unnecessary overhead for almost
all strings. the one place we can think of that benefits from them (attrpaths
with expressions) extracts the benefit in the parser, which we'll have to touch
anyway when changing ExprString to hold strings.

this gives a sizeable improvement on of 3-5% on all benchmarks we've run.

 before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.844 s ±  0.045 s    [User: 6.750 s, System: 1.663 s]
    Range (min … max):    8.758 s …  8.922 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     367.4 ms ±   3.3 ms    [User: 332.3 ms, System: 35.2 ms]
    Range (min … max):   364.0 ms … 375.2 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.810 s ±  0.030 s    [User: 2.517 s, System: 0.225 s]
    Range (min … max):    2.742 s …  2.854 s    20 runs

 after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.533 s ±  0.068 s    [User: 6.485 s, System: 1.642 s]
    Range (min … max):    8.404 s …  8.657 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     347.6 ms ±   3.1 ms    [User: 313.1 ms, System: 34.5 ms]
    Range (min … max):   343.3 ms … 354.6 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.709 s ±  0.032 s    [User: 2.414 s, System: 0.232 s]
    Range (min … max):    2.655 s …  2.788 s    20 runs
2022-01-19 14:48:00 +01:00
pennae 0a7746603e remove ExprIndStr
it can be replaced with StringToken if we add another bit if information to
StringToken, namely whether this string should take part in indentation scanning
or not. since all escaping terminates indentation scanning we need to set this
bit only for the non-escaped IND_STRING rule.

this improves performance by about 1%.

 before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.880 s ±  0.048 s    [User: 6.809 s, System: 1.643 s]
    Range (min … max):    8.781 s …  8.993 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     375.0 ms ±   2.2 ms    [User: 339.8 ms, System: 35.2 ms]
    Range (min … max):   371.5 ms … 379.3 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.831 s ±  0.040 s    [User: 2.536 s, System: 0.225 s]
    Range (min … max):    2.769 s …  2.912 s    20 runs

 after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.832 s ±  0.048 s    [User: 6.757 s, System: 1.657 s]
    Range (min … max):    8.743 s …  8.921 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     367.4 ms ±   3.2 ms    [User: 332.7 ms, System: 34.7 ms]
    Range (min … max):   364.6 ms … 374.6 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.810 s ±  0.030 s    [User: 2.517 s, System: 0.225 s]
    Range (min … max):    2.742 s …  2.854 s    20 runs
2022-01-19 13:39:42 +01:00
lincoln auster [they/them] 3e5a9ad7ff allow modifying lockfile commit msg with nix config option
This allows setting the commit-lockfile-summary option to a non-empty
string to override the commit summary while leaving the body unchanged.
2022-01-18 16:10:27 -07:00
Eelco Dolstra 4af88a4c91
Merge pull request #5906 from pennae/primops-optimization
optimize primops and utils by caching more and copying less
2022-01-18 19:43:28 +01:00
Théophane Hufschmitt dc4730ee94
Merge pull request #5720 from tomberek/flake_search
flakes: search up to git or filesystem boundary
2022-01-18 10:59:12 +01:00
Eelco Dolstra fc2443a67c
Merge pull request #5812 from pennae/small-perf-improvements
improve parser performance a bit
2022-01-17 19:49:52 +01:00
Alexander Bantyev e3690ab393
Add more tests for flake upward searching 2022-01-14 17:21:18 +03:00
tomberek 2dead20924
Apply suggestions from code review
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-01-14 09:16:34 -05:00
pennae ad60dfde2a also cache split regexes, not just match regexes
gives about 1% improvement on system eval, a bit less on nix search.

 # before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      7.419 s ±  0.045 s    [User: 6.362 s, System: 0.794 s]
    Range (min … max):    7.335 s …  7.517 s    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.921 s ±  0.023 s    [User: 2.626 s, System: 0.210 s]
    Range (min … max):    2.883 s …  2.957 s    20 runs

 # after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      7.370 s ±  0.059 s    [User: 6.333 s, System: 0.791 s]
    Range (min … max):    7.286 s …  7.541 s    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.891 s ±  0.033 s    [User: 2.606 s, System: 0.210 s]
    Range (min … max):    2.823 s …  2.958 s    20 runs
2022-01-14 14:04:17 +01:00
pennae c9fc975259 optimize removeAttrs builtin
use a sorted array of symbols to be removed instead of a set. this saves a lot
of memory allocations and slightly speeds up removal.
2022-01-14 14:01:52 +01:00
pennae 72f42093e7 optimize unescapeStr
mainly to avoid an allocation and a copy of a string that can be
modified in place (ever since EvalState holds on to the buffer, not the
generated parser itself).

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     571.7 ms ±   2.4 ms    [User: 563.3 ms, System: 8.0 ms]
  Range (min … max):   566.7 ms … 579.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     376.6 ms ±   1.0 ms    [User: 345.8 ms, System: 30.5 ms]
  Range (min … max):   374.5 ms … 379.1 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.922 s ±  0.006 s    [User: 2.707 s, System: 0.215 s]
  Range (min … max):    2.906 s …  2.934 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     570.4 ms ±   2.8 ms    [User: 561.3 ms, System: 8.6 ms]
  Range (min … max):   564.6 ms … 578.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     375.4 ms ±   1.3 ms    [User: 343.2 ms, System: 31.7 ms]
  Range (min … max):   373.4 ms … 378.2 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.925 s ±  0.006 s    [User: 2.704 s, System: 0.219 s]
  Range (min … max):    2.910 s …  2.942 s    50 runs
2022-01-13 18:06:15 +01:00
pennae 34e3bd10e3 avoid copies of parser input data
when given a string yacc will copy the entire input to a newly allocated
location so that it can add a second terminating NUL byte. since the
parser is a very internal thing to EvalState we can ensure that having
two terminating NUL bytes is always possible without copying, and have
the parser itself merely check that the expected NULs are present.

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     572.4 ms ±   2.3 ms    [User: 563.4 ms, System: 8.6 ms]
  Range (min … max):   566.9 ms … 579.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     381.7 ms ±   1.0 ms    [User: 348.3 ms, System: 33.1 ms]
  Range (min … max):   380.2 ms … 387.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.936 s ±  0.005 s    [User: 2.715 s, System: 0.221 s]
  Range (min … max):    2.923 s …  2.946 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     571.7 ms ±   2.4 ms    [User: 563.3 ms, System: 8.0 ms]
  Range (min … max):   566.7 ms … 579.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     376.6 ms ±   1.0 ms    [User: 345.8 ms, System: 30.5 ms]
  Range (min … max):   374.5 ms … 379.1 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.922 s ±  0.006 s    [User: 2.707 s, System: 0.215 s]
  Range (min … max):    2.906 s …  2.934 s    50 runs
2022-01-13 18:06:15 +01:00
pennae eee0bcee22 avoid allocations in SymbolTable::create
speeds up parsing by ~3%, system builds by a bit more than 1%

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     574.7 ms ±   2.8 ms    [User: 566.3 ms, System: 8.0 ms]
  Range (min … max):   569.2 ms … 580.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     394.4 ms ±   0.8 ms    [User: 361.8 ms, System: 32.3 ms]
  Range (min … max):   392.7 ms … 395.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.976 s ±  0.005 s    [User: 2.757 s, System: 0.218 s]
  Range (min … max):    2.966 s …  2.990 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     572.4 ms ±   2.3 ms    [User: 563.4 ms, System: 8.6 ms]
  Range (min … max):   566.9 ms … 579.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     381.7 ms ±   1.0 ms    [User: 348.3 ms, System: 33.1 ms]
  Range (min … max):   380.2 ms … 387.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.936 s ±  0.005 s    [User: 2.715 s, System: 0.221 s]
  Range (min … max):    2.923 s …  2.946 s    50 runs
2022-01-13 18:06:15 +01:00
pennae 61a9d16d5c don't strdup tokens in the lexer
every stringy token the lexer returns is turned into a Symbol and not
used further, so we don't have to strdup. using a string_view is
sufficient, but due to limitations of the current parser we have to use
a POD type that holds the same information.

gives ~2% on system build, 6% on search, 8% on parsing alone

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     610.6 ms ±   2.4 ms    [User: 602.5 ms, System: 7.8 ms]
  Range (min … max):   606.6 ms … 617.3 ms    50 runs

Benchmark 2: nix eval -f hackage-packages.nix
  Time (mean ± σ):     430.1 ms ±   1.4 ms    [User: 393.1 ms, System: 36.7 ms]
  Range (min … max):   428.2 ms … 434.2 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      3.032 s ±  0.005 s    [User: 2.808 s, System: 0.223 s]
  Range (min … max):    3.023 s …  3.041 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     574.7 ms ±   2.8 ms    [User: 566.3 ms, System: 8.0 ms]
  Range (min … max):   569.2 ms … 580.7 ms    50 runs

Benchmark 2: nix eval -f hackage-packages.nix
  Time (mean ± σ):     394.4 ms ±   0.8 ms    [User: 361.8 ms, System: 32.3 ms]
  Range (min … max):   392.7 ms … 395.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.976 s ±  0.005 s    [User: 2.757 s, System: 0.218 s]
  Range (min … max):    2.966 s …  2.990 s    50 runs
2022-01-13 18:06:14 +01:00
pennae 6401e443a4 move strings in derivationStrict
the temporary will be discarded anyway, so we can move out of it and save many
small allocations and copies.
2022-01-13 14:00:20 +01:00
pennae ef45787aae avoid string copies in attrNames sort comparison
symbols can also be cast to string_view, which compares the same but doesn't
require a copy of both symbol names on every comparison.
2022-01-13 14:00:19 +01:00
pennae 1bebb1095a cache more often-used symbols for primops
there's a few symbols in primops we can create once and pick them out of
EvalState afterwards instead of creating them every time we need them. this
gives almost 1% speedup to an uncached nix search.
2022-01-13 13:58:33 +01:00
Théophane Hufschmitt e61c4bc25a
Merge pull request #5887 from pennae/avoid-streams
avoid std::?stream overhead when it's not helpful
2022-01-12 10:52:40 +01:00
pennae 5838354d34 optimize ExprConcatStrings::eval
constructing an ostringstream for non-string concats (like integer addition) is
a small constant cost that we can avoid. for string concats we can keep all the
string temporaries we get from coerceToString and concatenate them in one go,
which saves a lot of intermediate temporaries and copies in ostringstream. we
can also avoid copying the concatenated string again by directly allocating it
in GC memory and moving ownership of the concatenated string into the target
value.

saves about 2% on system eval.

before:

  Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.837 s ±  0.031 s    [User: 2.562 s, System: 0.191 s]
    Range (min … max):    2.796 s …  2.892 s    20 runs

after:

  Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.790 s ±  0.035 s    [User: 2.532 s, System: 0.187 s]
    Range (min … max):    2.722 s …  2.836 s    20 runs
2022-01-12 10:07:21 +01:00
Eelco Dolstra f39d94a55b Merge branch 'readfile' of https://github.com/abbradar/nix 2022-01-10 16:55:28 +01:00
Nikolay Amiantov c66865dff1 builtins.readFile: Propagate path context
Co-authored-by: Shea Levy <shea@shealevy.com>
2022-01-09 13:07:00 +03:00
Ben Burdette a963674d88 optinoal error; compiles 2022-01-08 11:03:48 -07:00
Ben Burdette c51b527c28 add env to DebugTrace 2022-01-07 16:37:44 -07:00
Eelco Dolstra 3ec02c6743
Merge pull request #5546 from NixOS/allow-paths-in-flake-local-options
allow paths in flake local options
2022-01-06 21:55:20 +01:00
Ben Burdette bf8a065be0 add colors; remove headings 2022-01-05 12:28:31 -07:00
Ben Burdette a4d8a799b7 tidy up debugtraces 2022-01-05 12:21:18 -07:00
Eelco Dolstra 2b4c944823 Remove EvalState::mkAttrs() 2022-01-04 20:29:17 +01:00
Eelco Dolstra 17daec0b83 Move empty attrset optimisation 2022-01-04 19:23:11 +01:00
Eelco Dolstra ca5baf2392 Turn mkString(Symbol) into a method 2022-01-04 19:09:40 +01:00
Eelco Dolstra ed93aec3c3 Remove non-method mkPath() 2022-01-04 18:45:16 +01:00
Eelco Dolstra 263a8d293c Remove non-method mk<X> functions 2022-01-04 18:40:39 +01:00
Eelco Dolstra cc08364315 Remove non-method mkString() 2022-01-04 18:24:42 +01:00
Eelco Dolstra 6d9a6d2cc3 Ensure that attrsets are sorted
Previously you had to remember to call value->attrs->sort() after
populating value->attrs. Now there is a BindingsBuilder helper that
wraps Bindings and ensures that sort() is called before you can use
it.
2022-01-04 18:00:33 +01:00
Ben Burdette 1b6b33d43d filter out underscore names 2022-01-03 18:29:43 -07:00
Ben Burdette a47de1ac37 Merge branch 'master' into debug-exploratory-PR 2022-01-03 16:08:28 -07:00
pennae 00c993f48b add zipAttrsWith primop
nixpkgs can save a good bit of eval memory with this primop. zipAttrsWith is
used quite a bit around nixpkgs (eg in the form of recursiveUpdate), but the
most costly application for this primop is in the module system. it improves
the implementation of zipAttrsWith from nixpkgs by not checking an attribute
multiple times if it occurs more than once in the input list, allocates less
values and set elements, and just avoids many a temporary object in general.

nixpkgs has a more generic version of this operation, zipAttrsWithNames, but
this version is only used once so isn't suitable for being the base of a new
primop. if it were to be used more we should add a second primop instead.
2022-01-03 21:05:53 +01:00
Eelco Dolstra 96d08fcd66
Merge pull request #5839 from tweag/balsoft/yet-another-follows-bugfix
flake.cc: computeLocks: Only verify overrides when they could change
2022-01-03 20:51:58 +01:00
Sebastian Ullrich d0c8e9254e Fix IFD with chroot store 2021-12-29 19:00:02 +01:00
Lily Foster 92c7d33ee3 Sort attrs from tables in fromTOML
This was dropped in 10a8b5d for the migration from cpptoml to toml11 but
seems to be necessary for the attrsets to work correctly.

Fixes #5833
2021-12-28 10:11:00 -05:00
Alexander Bantyev 2664a216e5
flake.cc: computeLocks: Only verify overrides when they could change
When we check for disappeared overrides, we can get "false positives"
for follows and overrides which are defined in the dependencies of the
flake we are locking, since they are not parsed by
parseFlakeInputs. However, at that point we already know that the
overrides couldn't have possible been changed if the input itself
hasn't changed (since we check that oldLock->originalRef == *input.ref
for the input's parent). So, to prevent this, only perform this check
when it was possible that the flake changed (e.g. the flake we're
locking, or a new input, or the input has changed and mustRefetch ==
true).
2021-12-28 15:30:06 +03:00
Ben Burdette 5954cbf3e9 more cleanup 2021-12-27 18:29:55 -07:00
Ben Burdette 4610e02d04 remove debug code 2021-12-27 18:12:46 -07:00
Ben Burdette 9760fa8661 add DebugTrace for the current error 2021-12-27 17:35:27 -07:00
Ben Burdette 6801a423fc :d env 2021-12-27 16:28:45 -07:00
Ben Burdette ff82ba98b4 don't add builtins to extras, initEnv() in regular repl 2021-12-27 14:06:04 -07:00
Ben Burdette d0d5890445 don't add underscore names to extras 2021-12-27 13:47:35 -07:00
Yorick a4ab0a74d9
Fix accidental O(n^2 * log n) performance in NixRepl::addAttrsToScope
Only sort once, after adding all of the attrs first. This reduces my
`nix repl '<nixpkgs>'` loading time from 1.07s to 103ms.

Fixes #5823
2021-12-27 13:18:55 +01:00
Ben Burdette e5eebda194 DebugTrace 2021-12-23 13:36:39 -07:00
tomberek f404189368
Merge branch 'master' into flake_search 2021-12-23 15:22:52 -05:00
Ben Burdette deb1fd66e8 makeDebugTraceStacker 2021-12-23 09:08:41 -07:00
Ben Burdette 1bda6a01e1 indenting 2021-12-23 08:14:17 -07:00
Nicolas Mattia b16643b6fc Use int64_t and NixFloat in fromTOML types
This makes sure that values parsed from TOML have a proper size. Using
e.g. `double` caused issues on i686 where the size of `double` (32bit)
was too small to accommodate some values.
2021-12-23 14:12:49 +01:00
regnat dc89dfa7b3 Properly return false on builtins.pathExists /someNonAllowedPath
Follow-up from https://github.com/NixOS/nix/pull/5807 to fix https://github.com/NixOS/nix/pull/5807#issuecomment-1000135394
2021-12-23 10:49:33 +01:00
Ben Burdette bc20e54e00 stack traces basically working 2021-12-22 19:40:08 -07:00
Ben Burdette b4a59a5eec DebugStackTracker class in one place 2021-12-22 15:38:49 -07:00
Eelco Dolstra 89faff93e2
Merge pull request #5807 from NixOS/5805-ca-ifd
Fix IFD with CA derivations
2021-12-21 18:47:34 +01:00
Eelco Dolstra 25cea2b737 Merge branch 'nm-toml11' of https://github.com/nmattia/nix 2021-12-21 14:18:04 +01:00
regnat d90f9d4b99 Fix IFD with CA derivations
Rewrite the string taken by the IFD-like primops to contain the actual
output paths of the derivations rather than the placeholders

Fix #5805
2021-12-21 09:36:50 +01:00
regnat cbbd21ec07 Factor out the path realisation bit of IFD 2021-12-21 09:36:19 +01:00
pennae 09b245690a bulk-allocate Value instances in the evaluator
calling GC_malloc for each value is significantly more expensive than
allocating a bunch of values at once with GC_malloc_many. "a bunch" here
is a GC block size, ie 16KiB or less.

this gives a 1.5% performance boost when evaluating our nixos system.

tested with

nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

 # on master

  Time (mean ± σ):      3.335 s ±  0.007 s    [User: 2.774 s, System: 0.293 s]
  Range (min … max):    3.315 s …  3.347 s    50 runs

 # with this change

  Time (mean ± σ):      3.288 s ±  0.006 s    [User: 2.728 s, System: 0.292 s]
  Range (min … max):    3.274 s …  3.307 s    50 runs
2021-12-20 23:01:28 +01:00
Nicolas Mattia 9dea5768ef Clean up toml parsing code 2021-12-20 21:39:37 +01:00
Ben Burdette f317019edd :d error 2021-12-20 12:32:21 -07:00
Nicolas Mattia 5d06836b9f Use long in fromTOML 2021-12-17 23:15:26 +01:00
Nicolas Mattia 10a8b5d3ae Update primops/fromTOML.cc to use toml11 2021-12-17 22:13:04 +01:00
regnat ab902521b1 Allow paths in flake local settings
Fix #5505
2021-12-17 09:31:13 +01:00
Théophane Hufschmitt 8868da45a8
Merge pull request #5623 from yorickvP/fix-5621
flakes: fix boolean and int nixConfig values
2021-12-14 10:35:37 +01:00
Eelco Dolstra 32a62b0d25 Re-allow inputs.x.url = "/path/to/file" 2021-12-13 20:59:33 +01:00
Eelco Dolstra cc6406cc59 Merge branch 'better-interpolation-error-location' of https://github.com/greedy/nix 2021-12-13 19:20:48 +01:00
Tom Bereknyei b6cc0a704d flakes: search up to git or filesystem boundary
While parsing a flakeref, upon not finding a flake.nix, search upwards
until git or filesystem boundary.
2021-12-03 10:55:30 -05:00
Silvan Mosberger 90700736c7 Introduce builtins.groupBy primop
This function is very useful in nixpkgs, but its implementation in Nix
itself is rather slow due to it requiring a lot of attribute set and
list appends.
2021-12-02 21:54:51 +01:00
Alexander Bantyev 8cbf862e6f
Flakes: computeLocks: pass correct LockParent when reusing oldLock
Previously, when we were attempting to reuse the old lockfile
information in the computeLocks function, we have passed the parent of
the current input to the next computeLocks call. This was incorrect,
since the follows are resolved relative to the parent. This caused
issues when we tried to reuse oldLock but couldn't for some
reason (read: mustRefetch is true), in that case the follows were
resolved incorrectly.

Fix this by passing the correct parent, and adding some tests to
prevent this particular regression from happening again.

Closes https://github.com/NixOS/nix/issues/5697
2021-12-02 12:02:34 +03:00
Ben Burdette e82aec4efc fix merge issues 2021-11-30 14:15:02 -07:00
Yorick bd628cf3da
flakes: fix boolean and int nixConfig values
Some type confusion was causing ints to be pointers, and bools
to be ints. Fixes #5621
2021-11-29 15:53:22 +01:00
Andreas Rammhold 90d8178009
Don't move the arguments of the primOp
Moving arguments of the primOp into the registration structure makes it
impossible to initialize a second EvalState with the correct primOp
registration. It will end up registering all those "RegisterPrimOp"'s
with an arity of zero on all but the 2nd instance of the EvalState.

Not moving the memory will add a tiny bit of memory overhead during the
eval since we need a copy of all the argument lists of all the primOp's.
The overhead shouldn't be too bad as it is static (based on the amonut
of registered operations) and only occurs once during the interpreter
startup.
2021-11-28 02:06:47 +01:00
Eelco Dolstra 6f46434f32
Merge pull request #5648 from edolstra/list-iter
Support range-based for loop over list values
2021-11-25 17:13:25 +01:00
Ben Burdette 64c4ba8f66 Merge branch 'master' into debug-merge 2021-11-25 08:53:59 -07:00
Eelco Dolstra b6c8e57056 Support range-based for loop over list values 2021-11-25 16:31:39 +01:00
Ben Burdette 69e26c5c4b more cleanup 2021-11-25 08:23:07 -07:00
regnat c47027f3a1 Fix the error when accessing a forbidden path in pure eval
If we’re in pure eval mode, then tell that in the error message rather
than (wrongly) speaking about restricted mode.

Fix https://github.com/NixOS/nix/issues/5611
2021-11-25 14:48:01 +01:00
Eelco Dolstra d58f149140
Merge pull request #5631 from Infinisil/list-compare
Make lists be comparable
2021-11-24 15:48:05 +01:00
Silvan Mosberger 09471d2680 Make lists be comparable
Makes lists comparable using lexicographic comparison.

Increments builtins.langVersion in order for this change to be
detectable
2021-11-24 13:40:46 +01:00
Eelco Dolstra b367f1061c
Merge pull request #5624 from rofrol/typo-single-quote
Typo: change to normal single quote
2021-11-22 21:33:21 +01:00
Roman Frołow 0768c08d99 Typo: change to normal singlequote 2021-11-22 13:37:38 +01:00
Tom Bereknyei 4318ba2ec5 add real path to allowedPaths 2021-11-20 00:25:36 -05:00
Eelco Dolstra 0961c1068a Merge branch 'accept_conf' of https://github.com/tomberek/nix 2021-11-18 12:56:09 +01:00
Eelco Dolstra 0adced4b9e
Merge pull request #5580 from ksonj/fix/non-standard-ssh
Fix detection of scp-style URIs to support non-standard SSH ports
2021-11-18 12:30:29 +01:00
Kalle Jepsen 46d2a5a10b Simplify fix by disallowing / in front of @ to match scp style 2021-11-17 13:49:10 +01:00
Eelco Dolstra d03e89e5d1 Parse '(f x) y' the same as 'f x y'
(cherry picked from commit 5253cb4b68ad248f37b27849c0ebf3614e4f2777)
2021-11-17 09:53:57 +01:00
Kalle Jepsen 6d46b5b609 Fix detection of scp-style URIs to support non-standard SSH ports for git 2021-11-17 08:41:26 +01:00
Eelco Dolstra d7bae52b9d Call functors with both arguments at once
This is not really useful on its own, but it does recover the
'infinite recursion' error message for '{ __functor = x: x; } 1', and
is more efficient in conjunction with #3718.

Fixes #5515.
2021-11-16 22:34:17 +01:00
Eelco Dolstra e41cf8511f Don't hang when calling an attrset
Fixes #5565.
2021-11-16 17:44:19 +01:00
Eelco Dolstra 6463eaca14
Merge pull request #5472 from NixOS/async-realisation-substitution
async realisation substitution
2021-11-16 12:54:20 +01:00
Eelco Dolstra 9fe0343bfd Merge branch 'master' of github.com:NixOS/nix 2021-11-15 18:45:20 +01:00
Eelco Dolstra 671817a858 Simplify lockFlake() a bit 2021-11-15 18:44:27 +01:00
Eelco Dolstra d1d223838b Merge branch 'balsoft/fix-subinputs-not-getting-updated' of https://github.com/tweag/nix 2021-11-15 18:37:13 +01:00
Domen Kožar 164179983e
Merge pull request #5428 from kreisys/add-pos-to-json-type-error
toJSON: report error position for fancier output
2021-11-15 07:57:46 -06:00
Kevin Amado d0e9e18489
toXML: display errors position
- This change applies to builtins.toXML and inner workings
- Proof of concept:
  ```nix
  let e = builtins.toXML e; in e
  ```
- Before:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered
  ```
- After:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/poc.nix:1:9:

            1| let e = builtins.toXML e; in e
             |
  ```
2021-11-13 20:33:34 -05:00
Tom Bereknyei 30496af598 Adds an accept-flake-config flag 2021-11-12 09:50:07 -05:00
Alexander Bantyev 07bffe7998
Flakes: refetch the input when a follows disappears
When an input follows disappears, we can't just reuse the old lock
file entries since we may be missing some required ones. Refetch the
input when this happens.

Closes https://github.com/NixOS/nix/issues/5289
2021-11-11 12:05:18 +03:00
Ben Burdette 7e2a3db4eb cleanup 2021-11-09 13:14:49 -07:00
Ben Burdette 885f819922 remove dead code 2021-11-09 11:20:14 -07:00
Eelco Dolstra 67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev 0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra 736d6ab721
Merge pull request #5504 from NixOS/flake-options-and-daemon
Make the flake options work when using the daemon
2021-11-08 13:54:55 +01:00
Andreas Rammhold 8e7359db64
Remove unused "<let-body>" symbol
The requirement for the symbol has been removed since at least 7d47498.
2021-11-07 18:26:43 +01:00
Alexander Bantyev 9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
regnat 1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
Eelco Dolstra 40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra cbfbf71e08 Use callFunction() with an array for some calls with arity > 1 2021-11-04 15:03:57 +01:00
Eelco Dolstra bcf4780006 Add level / displacement types 2021-11-04 15:03:45 +01:00
Eelco Dolstra 81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
Eelco Dolstra ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
regnat 5b2aa61f1b Don’t require ca-derivations when __contentAddressed = false
If we explicitely opt-out of it, there’s no need to require the
experimental feature
2021-11-03 06:51:32 +01:00
Shay Bergmann a50c027ece
toJSON: improve pos accuracy, add trace 2021-10-27 19:48:48 +00:00
Eelco Dolstra 5a160171d0 Remove redundant 'warning:' 2021-10-27 18:14:12 +02:00
Shay Bergmann 769de259f0
toJSON: pass pos in case of a list as well 2021-10-26 14:43:15 +00:00
Eelco Dolstra 9ce84c64c5 Tweak fetchTree docs 2021-10-26 14:21:24 +02:00
regnat af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Shay Bergmann ba81e871b2
toJSON: report error position for fancier output
Given flake:

```nix
{ description = "nix json error provenance";
  inputs = {};
  outputs = { self }: {
    jsonFunction = _: "function";
    json = builtins.toJSON (_: "function");
  };
}

```
- Before:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON
```

- After:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON

       at /nix/store/b7imf1c2j4jnkg3ys7fsfbj02s5j0i4f-source/testflake/flake.nix:4:5:

            3|   outputs = { self }: {
            4|     jsonFunction = _: "function";
             |     ^
            5|     json = builtins.toJSON (_: "function");
```
2021-10-25 21:17:52 +00:00
Ben Burdette fb8377547b more code cleanup 2021-10-22 14:49:58 -06:00
Ben Burdette 71da988d47 more debug removal 2021-10-22 14:34:50 -06:00
Ben Burdette e54f17eb46 remove more debug code 2021-10-22 14:27:04 -06:00
Ben Burdette cbc2f0fe31 remove dead code 2021-10-22 14:02:47 -06:00
Kevin Amado 823dce945a
fetch: nicer infinite recursion errors
- This change applies to builtins.fetchurl and builtins.fetchTarball
- PoC: `let x = builtins.fetchurl x; in x`
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchurl x; in x
             |         ^
  ```

Mentions: #3505
2021-10-17 12:54:53 -05:00
Kevin Amado e5a27a3b4e
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree {
    type = "git";
    inherit x;
  };
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:3:10:

            2|   type = "git";
            3|   inherit x;
             |          ^
            4| };
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Kevin Amado 18e3d63341
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree x;
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchTree x;
             |         ^
            2| in x
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Kevin Amado 1bdeef8395
add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let
    x = builtins.fetchMercurial x;
  in
  x
  ```
- Before:
  ```bash
  $ nix-instantiate --show-trace --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  nix-instantiate --show-trace --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/test/default.nix:2:7:

            1| let
            2|   x = builtins.fetchMercurial x;
             |       ^
            3| in
  ```

Mentions: #3505
2021-10-14 23:23:05 -05:00
Ben Burdette 427fb8d158 comment out debugs 2021-10-11 16:48:10 -06:00
Ben Burdette 98eb13691a print staticenv bindings 2021-10-11 16:32:43 -06:00
Ben Burdette 2ee1fa4afd add nullable Expr argument 2021-10-11 14:42:29 -06:00
regnat 7466048d39 (partially) Revert "Don't copy in rethrow"
This reverts some parts of commit
8430a8f086 which was trying to rethrow
some exceptions while we weren’t in the context of a `catch` block,
causing some weird “terminate called without an active exception”
errors.

Fix #5368
2021-10-11 10:51:22 +02:00
John Ericson 195daa8299 Merge remote-tracking branch 'upstream/master' into ca-drv-exotic 2021-10-08 23:59:15 +00:00
Eelco Dolstra d39692e6b3 Make builtins.{path,filterSource} work with chroot stores 2021-10-07 14:22:39 +02:00
Eelco Dolstra c4dcf3cf25 Add a trace to all errors in addPath() 2021-10-07 13:47:15 +02:00
Eelco Dolstra 4806f2f6b0 Allow builtins.{path,filterSource} on paths with a context
We now build the context (so this has the side-effect of making
builtins.{path,filterSource} work on derivations outputs, if IFD is
enabled) and then check that the path has no references (which is what
we really care about).
2021-10-07 13:43:17 +02:00
Eelco Dolstra 7c50568788 Remove unnecessary call to queryMissing()
Worker::run() already does this.
2021-10-07 13:15:01 +02:00
Eelco Dolstra 972405edf5 Allow access to path copied to the store
Fixes https://github.com/NixOS/nix/pull/5163#issuecomment-931733912.
2021-10-07 12:15:22 +02:00
Eelco Dolstra cfaad7168e Refactoring: Add allowPath() method 2021-10-07 12:11:00 +02:00
Eelco Dolstra c9ee634f75
Merge pull request #5341 from andir/libexpr-formals
libexpr: remove matchAttrs boolean from ExprLambda
2021-10-07 11:58:56 +02:00
Eelco Dolstra 53e4794289
Merge pull request #5286 from ilkecan/add-a-warning-to-filterSource
Warn about the usage of filterSource with Nix store paths
2021-10-06 21:02:39 +02:00
ilkecan a4a6ef4fb2 Add a warning to filterSource
Warn about the usage of `filterSource` with Nix store paths
2021-10-06 19:25:33 +03:00
Eelco Dolstra 0dc8172458 Remove no-op call to realiseContext() 2021-10-06 18:08:37 +02:00
Eelco Dolstra c497fce011 Merge branch 'flakes_filterSource' of https://github.com/tomberek/nix 2021-10-06 18:08:18 +02:00
Eelco Dolstra 57a8eb4c01 fetchTree(): Parse type attribute first
The 'url' attribute depends on whether type == 'git', so this is needed for

  builtins.fetchTree {url = "git@github.com:NixOS/nix.git"; type = "git";}
2021-10-06 17:39:02 +02:00
Eelco Dolstra 83d86cc1b0 Cleanup 2021-10-06 17:30:10 +02:00
Andreas Rammhold cae41eebff libexpr: remove matchAttrs boolean from ExprLambda
The boolean is only used to determine if the formals are set to a
non-null pointer in all our cases. We can get rid of that allocation and
instead just compare the pointer value with NULL. Saving up to
sizeof(bool) + platform specific alignment per ExprLambda instace.
Probably not a lot of memory but perhaps a few kilobyte with nixpkgs?

This also gets rid of a potential issue with dereferencing formals based on
the value of the boolean that didn't have to be aligned with the formals
pointer but was in all our cases.
2021-10-06 17:24:06 +02:00
Eelco Dolstra 46753b5e9c Merge branch 'considerate/scp-like-urls' of https://github.com/considerate/nix 2021-10-06 17:19:27 +02:00
Ben Burdette aad27143c6 storing staticenv bindings 2021-10-02 13:47:36 -06:00
John Ericson edf67e1508 Merge branch 'path-info' into ca-drv-exotic 2021-10-01 17:25:22 +00:00
John Ericson 13b6b64589 Merge remote-tracking branch 'upstream/master' into path-info 2021-10-01 17:12:54 +00:00
John Ericson 9af9ab4212 Merge branch 'path-info' into ca-drv-exotic 2021-09-30 22:42:15 +00:00
John Ericson f4f3203aa7 Merge remote-tracking branch 'upstream/master' into path-info 2021-09-30 22:41:53 +00:00
John Ericson 242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Taeer Bar-Yam f14660d5e2 reset yylloc when yyless(0) is called 2021-09-29 19:47:01 -04:00
Eelco Dolstra fd01c48d34
Merge pull request #5301 from Ma27/builtins-missing-feature-error
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
2021-09-29 12:53:29 +02:00
Maximilian Bosch 2b02ce0e48
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
I found it somewhat confusing to have an error like

    error: attribute 'getFlake' missing

if the required experimental-feature (`flakes`) is not enabled. Instead,
I'd expect Nix to throw an error just like it's the case when using e.g. `nix
flake` without `flakes` being enabled.

With this change, the error looks like this:

    $ nix-instantiate -E 'builtins.getFlake "nixpkgs"'
    error: Cannot call 'builtins.getFlake' because experimental Nix feature 'flakes' is disabled. You can enable it via '--extra-experimental-features flakes'.

           at «string»:1:1:

                1| builtins.getFlake "nixpkgs"
                 | ^

I didn't use `settings.requireExperimentalFeature` here on purpose
because this doesn't contain a position. Also, it doesn't seem as if we
need to catch the error and check for the missing feature here since
this already happens at evaluation time.
2021-09-29 11:57:15 +02:00
Eelco Dolstra 8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Eelco Dolstra 362d8f925e
Merge pull request #5253 from edolstra/flake-ifd
Don't allow IFD in flakes by default
2021-09-24 10:48:23 +02:00
Ben Burdette b9d08b98da ok was unconditoinally throwing on any With var 2021-09-23 13:02:39 -06:00
Geoff Reedy 9d67332e4b Better eval error locations for interpolation and +
Previously, type or coercion errors for string interpolation, path
interpolation, and plus expressions were always reported at the
beginning of the outer expression. This leads to confusing evaluation
error messages making it hard to accurately diagnose and then fix the
error.

For example, errors were reported as follows.

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                 ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                     ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                  ^
```

This commit changes the ExprConcatStrings expression vector to store a
sequence of expressions *and* their expansion locations so that error
locations can be reported accurately. For interpolation, the error is
reported at the beginning of the entire `${foo}`, not at the beginning
of `foo` because I thought this was slightly clearer. The previous
errors are now reported as:

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                         ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                         ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                   ^
```

The error is reported at this kind of precise location even for
multi-line indented strings.

This probably helps with at least some of the cases mentioned in #561
2021-09-22 20:57:34 -06:00
Ben Burdette c07edb1932 staticenv should be With 2021-09-22 18:14:57 -06:00
Ben Burdette c7e3d830c1 more debug stuff 2021-09-22 16:22:53 -06:00
Eelco Dolstra d1bf7431bb Revert "Merge pull request #4922 from nrdxp/default-submodules"
This reverts commit 6678e98411, reversing
changes made to 90b2dd570c.
2021-09-22 17:25:25 +02:00
Eelco Dolstra ff28fffce2 Don't cache realiseContext() errors
Errors that depend on the configuration (such as whether
allow-import-from-derivation is set) should not be cached.
2021-09-22 14:00:56 +02:00
Eelco Dolstra 60cc975d22 Set input parent at construction time 2021-09-21 14:07:16 +02:00
Geoff Reedy cbe9ddfd1a Include subpath in flake fingerprint
Without this, flakes within the same tree and same lock data will have
the same fingerprint and the eval cache for one flake will be
incorrectly used for another.
2021-09-16 15:58:21 -06:00
Ben Burdette cd8c232b55 add cout debugging 2021-09-15 16:16:53 -06:00
Eelco Dolstra 1ec4efa6c8
Merge pull request #5257 from edolstra/dirty-lock-file
If we can't write a lock file, pretend the top-level flake is dirty
2021-09-15 20:18:23 +02:00
Eelco Dolstra c17f3c5e69 Merge branch 'mh/fix-chroot-eval' of https://github.com/obsidiansystems/nix 2021-09-15 18:37:58 +02:00
Eelco Dolstra 027344ce7e If we can't write a lock file, pretend the top-level flake is dirty
Alternative to #4639. You can still read flake.lock, but at least in
reproducible workflows like NixOS configurations where you require a
non-dirty tree, evaluation will fail because there is no rev.
2021-09-15 18:31:42 +02:00
Eelco Dolstra 991cc53386 Revert "Disallow reading flake.lock"
This reverts commit e5596113f7.
2021-09-15 18:30:37 +02:00
Eelco Dolstra e5596113f7 Disallow reading flake.lock
With --no-write-lock-file, it's possible that flake.lock is out of
sync with the actual inputs used by the evaluation. So doing fromJSON
(readFile ./flake.lock) will give wrong results.

Fixes #4639.
2021-09-14 21:09:11 +02:00
Ben Burdette 21071bfdeb shared_ptr for StaticEnv 2021-09-14 10:49:22 -06:00
Eelco Dolstra 16d4922dd2
Merge pull request #5240 from edolstra/builtin-help
nix --help: Display help using lowdown instead of man
2021-09-14 11:31:23 +02:00
kvtb c6fa7775de hashFile, hashString: realize context before calculation, and discard afterwards 2021-09-13 22:34:58 +02:00
Ben Burdette 176911102c printEnvPosChain 2021-09-13 11:57:25 -06:00
Eelco Dolstra 49a932fb18 nix --help: Display help using lowdown instead of man
Fixes #4476.
Fixes #5231.
2021-09-13 14:45:21 +02:00
Eelco Dolstra 9180239081
Merge pull request #5215 from ncfavier/patch-1
Fix use-registries logic in builtins.getFlake
2021-09-08 11:19:39 +02:00
Shea Levy 6678e98411
Merge pull request #4922 from nrdxp/default-submodules
libfetchers/git: fetch submodules by default
2021-09-07 05:48:23 -04:00
Naïm Favier b0d4190f19
Fix use-registries logic in builtins.getFlake 2021-09-05 15:28:44 +02:00
Eelco Dolstra 7ee639f9db
Merge pull request #5066 from Radvendii/master
add antiquotations to paths
2021-09-01 12:55:04 +02:00
Taeer Bar-Yam 1ffb9f1970 fix parse of /${foo}. was // + foo
I don't think this changes the way any program would behave, but it's a
cleaner internal representation.
2021-08-31 15:55:55 -04:00
Taeer Bar-Yam 9da8f5e25d path antiquotations: canonizePath -> canonicalizePath 2021-08-31 08:02:04 -04:00
Eelco Dolstra eda0fee160
Merge pull request #5175 from Pamplemousse/make
Don't overwrite user provided `lib*_LDFLAGS`
2021-08-30 12:44:29 +02:00
Eelco Dolstra 323cafcb4e
Merge pull request #5191 from hercules-ci/evalstate-lifetime-hygiene
EvalState lifetime hygiene
2021-08-30 12:23:09 +02:00
Robert Hensing 8656b130ea Fix use after free with vImportedDrvToDerivation 2021-08-29 20:42:57 +02:00
Robert Hensing 8bc76acc7c Move vCallFlake into EvalState
This fixes a use-after-free bug:

1. s = new EvalState();
2. callFlake()
3. static vCallFlake now references s
4. delete s;
5. s2 = new EvalState();
6. callFlake()
7. static vCallFlake still references s
8. crash

Nix 2.3 did not have a problem with recreating EvalState.
2021-08-29 20:42:49 +02:00
Robert Hensing f10465774f Force all Pos* to be non-null
This fixes a class of crashes and introduces ptr<T> to make the
code robust against this failure mode going forward.

Thanks regnat for the idea of a ref<T> without overhead!

Closes #4895
Closes #4893
Closes #5127
Closes #5113
2021-08-29 18:11:58 +02:00
Ben Burdette 310c689d31 remove more explicit valmap code 2021-08-25 13:18:27 -06:00
Ben Burdette d8a977a22e adding all the value names from env.values[0] 2021-08-25 11:19:09 -06:00
Pamplemousse a4c6d319a8 Don't overwrite user provided lib*_LDFLAGS
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-25 08:59:19 -07:00
Ben Burdette bd3b5329f9 print env bindings 2021-08-24 16:32:54 -06:00
Eelco Dolstra af94b54db3 Coding style 2021-08-23 13:05:42 +02:00
Tom Bereknyei d90582be33 Allow use of path and filterSource in flakes
As filterSource and path perform work, add paths to allowedPaths.
2021-08-22 18:45:42 -04:00
Ben Burdette 4b5f9b35f0 env to bindings 2021-08-18 21:25:26 -06:00
Ben Burdette 2272021536 more error fixes 2021-08-18 20:02:23 -06:00
Ben Burdette e82cf13b1e switch to fakeenvs 2021-08-18 17:53:10 -06:00
Ben Burdette b6eb38016b moving towards env in exceptions 2021-08-17 14:39:50 -06:00
Alex Zero b3c424f5a6
Fix follows paths in subordinate lockfiles 2021-08-17 15:25:42 +01:00
Piotr Szubiakowski 9f13cb31e8
Install pkg-config files in the correct location
Use `$(libdir)` while installing .pc files looks like a more generic
solution. For example, it will work for distributions like RHEL or
Fedora where .pc files are installed in `/usr/lib64/pkgconfig`.
2021-08-13 21:08:58 +00:00
Eelco Dolstra 6b8069b823 In flake lock file diffs, show the last-modified date of inputs if available
This is a bit more informative than just the hash.

Also, format the diffs a bit nicer.
2021-08-11 19:35:28 +02:00
Ben Burdette 030271184f trying env args; but unecessary? 2021-08-09 14:30:47 -06:00
Eelco Dolstra 8943e3176d
Merge pull request #5111 from Pamplemousse/clean
Minor maintenance cleaning
2021-08-09 20:05:03 +02:00
Pamplemousse 2de7a1fe67 libexpr: Squash similar conditions
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-09 10:10:11 -07:00
Alexander Bantyev 54c580b71f
Whitelist nixConfig.flake-registry
flake-registry should be safe to set to an aribtrary value, since it
is identical to just setting `inputs`.
2021-08-07 14:42:59 +03:00
Ben Burdette 89264d20e6 move valmap to hh; add to env 2021-08-06 11:09:27 -06:00
Taeer Bar-Yam 8f9429dcab add antiquotations to paths 2021-08-06 06:46:05 -04:00
Madeline Haraj cb1ffb7789 Use the store path as the context of the result of fetchTree, not the real path 2021-08-05 22:08:39 -04:00
Viktor Kronvall 7cdaae6b9c Support SCP-like URLs in builtins.fetchGit attrs
This extends https://github.com/NixOS/nix/pull/4978 with
supporting the SCP-like urls in expressions like

```nix
builtins.fetchGit {
  url = "git@github.com:NixOS/nix.git";
  ref = "master";
}
```
2021-07-30 08:17:46 +09:00
Eelco Dolstra 7816ef6c51
Merge pull request #5042 from alyssais/pthread
Enable pthreads for new libraries
2021-07-28 12:31:01 +02:00
Eelco Dolstra c000cec27f
Merge pull request #5048 from tweag/flox-eval-store
--eval-store and faster closure copying
2021-07-27 12:20:32 +02:00
Pamplemousse c2d7c0cdb9 libexpr: Remove unused code
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-26 09:14:34 -07:00
Alyssa Ross ae0c026fe9
Enable pthreads for new libraries
Otherwise the lack of pthread causes linking to fail for NetBSD.
2021-07-24 09:15:01 +00:00
Eelco Dolstra 2ff3035cf4 Support --eval-store in nix-instantiate and nix-build 2021-07-22 09:59:51 +02:00
Eelco Dolstra bef40c2949 Add --eval-store option 2021-07-22 09:59:51 +02:00
Pamplemousse c1c5dd7449 Avoid global counters
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-21 16:49:52 -07:00
Eelco Dolstra fc248cf59e Merge branch 'balsoft/no-registries' of https://github.com/serokell/nix 2021-07-21 10:11:40 +02:00
Robert Hensing ad24921de8 Rename findDerivationFilename -> findPackageFilename
It does not operate on a derivation and does not return a
derivation path. Instead it works at the language level,
where a distinct term "package" is more appropriate to
distinguish the parent object of `meta.position`; an
attribute which doesn't even make it into the derivation.
2021-07-19 18:10:10 +02:00
Alexander Bantyev 3e57e3480b
Add use-registries config option (and deprecate --no-registries flag)
Some people want to avoid using registries at all on their system; Instead
of having to add --no-registries to every command, this commit allows to
set use-registries = false in the config. --no-registries is still allowed
everywhere it was allowed previously, but is now deprecated.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-07-16 23:26:04 +03:00
Eelco Dolstra 77d5b37da3
Merge pull request #5011 from Pamplemousse/fix_adjustLoc
libexpr: Fix read out-of-bound on the heap
2021-07-15 09:23:59 +02:00
Pamplemousse 99f8fc995b libexpr: Fix read out-of-bound on the heap
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-14 09:09:42 -07:00
Niels Egberts ae0ed53b09 toString also coerces a set with an outPath attribute to a string
nix-repl> builtins.toString { outPath = "somestring"; }
"somestring"
2021-07-09 21:50:10 +01:00
Eelco Dolstra d2b8b23ae9
Merge pull request #4995 from NixOS/fetchgit-name-attribute
Fix the `name` attribute in builtins.fetchGit
2021-07-09 00:32:13 +02:00
regnat a654c1d81c Restore the possibility to use a name parameter in fetchGit
Accidentally broken by 7e5c79a2d2
2021-07-08 14:57:48 +02:00
Eelco Dolstra 8648143120
Merge pull request #4988 from NixOS/fetchgit-name-attribute
Add a name attribute to the fetchers
2021-07-08 14:33:49 +02:00
regnat 7e5c79a2d2 Forbid the name attribute for fetchTree
We need to support it for the “old” fetch* functions for backwards
compatibility, but we don’t need it for fetchTree (as it’s a new
function).
Given that changing the `name` messes-up the content hashing, we can
just forbid passing a custom `name` argument to it
2021-07-08 13:53:52 +02:00
Eelco Dolstra 2172e60f7a
Merge pull request #4935 from alyssais/host_os
Apply OS checks to host platform, not build
2021-07-08 12:40:54 +02:00
Eelco Dolstra 156666de3d
Merge pull request #4969 from serokell/balsoft/fix-nixConfig-flake-registry
flake.nixConfig: fix flake-registry config settings
2021-07-08 12:35:08 +02:00
regnat e4b082a52b allow fetchMercurial to take a name argument 2021-07-06 08:43:06 +02:00
Eelco Dolstra 1f93084149
Merge pull request #4978 from NixOS/scp-uris
Allow scp-style uris in `fetchgit`
2021-07-05 14:07:04 +02:00
Eelco Dolstra 24bc935462 Merge branch 'disable_gc' of https://github.com/Pamplemousse/nix 2021-07-05 13:53:52 +02:00
Domen Kožar f9d72855ae
Merge pull request #4967 from Pamplemousse/specific_errors
Prefer to throw specific errors
2021-07-03 10:34:56 +02:00
regnat ec2c6bd470 Allow scp-style uris in fetchgit
Fix #5303
2021-07-02 19:20:07 +02:00
Pamplemousse 4a7a8b87cd Prefer to throw specific errors
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-01 11:09:31 -07:00
Alexander Bantyev e756a59c72
fixup! flake.nixConfig: fix flake-registry config settings 2021-07-01 17:54:22 +03:00
Alexander Bantyev ef1e7ab840
flake.nixConfig: fix flake-registry config settings
Before this commit, nixConfig.flake-registry didn't have any real effect
on the evaluation, since config was applied after inputs were evaluated.
Change this behavior: apply the config in the beginning of flake::lockFile.
2021-07-01 00:23:47 +03:00
Pamplemousse 5be17a4b96 Allow to compile after ./configure --enable-gc=no
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-06-30 09:45:21 -07:00
Eelco Dolstra 5a18e2a533
Merge pull request #4959 from NixOS/override-non-flake-inputs
Keep the `isFlake` attribute for overriden inputs
2021-06-29 14:23:50 +02:00
regnat 4cff8188a5 Keep the isFlake attribute for overriden inputs
When overriding an input that isn’t a flake, mark the override as also not
being a flake.

Fix #3774
2021-06-29 13:56:48 +02:00
Robert Hensing bec83a6f95 BoehmGCStackAllocator: ignore stack protection page
This fixes a crash that looks like:

```
Thread 1 "nix-build" received signal SIGSEGV, Segmentation fault.
0x00007ffff7ad22a0 in GC_push_all_eager () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
(gdb) bt
0  0x00007ffff7ad22a0 in GC_push_all_eager () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
1  0x00007ffff7adeefb in GC_push_all_stacks () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
2  0x00007ffff7ad5ac7 in GC_mark_some () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
3  0x00007ffff7ad77bd in GC_stopped_mark () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
4  0x00007ffff7adbe3a in GC_try_to_collect_inner.part.0 () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
5  0x00007ffff7adc2a2 in GC_collect_or_expand () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
6  0x00007ffff7adc4f8 in GC_allocobj () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
7  0x00007ffff7adc88f in GC_generic_malloc_inner () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
8  0x00007ffff7ae1a04 in GC_generic_malloc_many () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
9  0x00007ffff7ae1c72 in GC_malloc_kind () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
10 0x00007ffff7e003d6 in nix::EvalState::allocValue() () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
11 0x00007ffff7e04b9c in nix::EvalState::callPrimOp(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
12 0x00007ffff7e0a773 in nix::EvalState::callFunction(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
13 0x00007ffff7e0a91d in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
14 0x00007ffff7e0a8f8 in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
15 0x00007ffff7e0e0e8 in nix::ExprOpNEq::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
16 0x00007ffff7e0d708 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
17 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
18 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
19 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
20 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
21 0x00007ffff7e09e19 in nix::ExprOpNot::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
22 0x00007ffff7e0a792 in nix::EvalState::callFunction(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
23 0x00007ffff7e8cba0 in nix::addPath(nix::EvalState&, nix::Pos const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Value*, nix::FileIngestionMethod, std::optional<nix::Hash>, nix::Value&)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#1}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
24 0x00007ffff752e6f9 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
25 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
26 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
27 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
28 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
29 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
30 0x00007ffff757f8c0 in void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, nix::VirtualStackAllocator, boost::coroutines2::detail::pull_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::control_block::control_block<nix::VirtualStackAllocator, nix::sinkToSource(std::function<void (nix::Sink&)>, std::function<void ()>)::SinkToSource::read(char*, unsigned long)::{lambda(boost::coroutines2::detail::push_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&)#1}>(boost::context::preallocated, nix::VirtualStackAllocator&&, nix::sinkToSource(std::function<void (nix::Sink&)>, std::function<void ()>)::SinkToSource::read(char*, unsigned long)::{lambda(boost::coroutines2::detail::push_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&)#1}&&)::{lambda(boost::context::fiber&&)#1}> >(boost::context::detail::transfer_t) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
31 0x00007ffff6f331ef in make_fcontext () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libboost_context.so.1.69.0
32 0x0000000000000000 in ?? ()
```
2021-06-24 12:20:39 +02:00
Alyssa Ross 4f80464645
Apply OS checks to host platform, not build
Previously, the build system used uname(1) output when it wanted to
check the operating system it was being built for, which meant that it
didn't take into-account cross-compilation when the build and host
operating systems were different.

To fix this, instead of consulting uname output, we consult the host
triple, specifically the third "kernel" part.

For "kernel"s with stable ABIs, like Linux or Cygwin, we can use a
simple ifeq to test whether we're compiling for that system, but for
other platforms, like Darwin, FreeBSD, or Solaris, we have to use a
more complicated check to take into account the version numbers at the
end of the "kernel"s.  I couldn't find a way to just strip these
version numbers in GNU Make without shelling out, which would be even
more ugly IMO.  Because these checks differ between kernels, and the
patsubst ones are quite fiddly, I've added variables for each host OS
we might want to check to make them easier to reuse.
2021-06-23 15:00:36 +00:00
Timothy DeHerrera 769ca4e26d
libfetchers/git: fetch submodules by default 2021-06-17 12:00:26 -06:00
Ben Burdette 129dd760e6 mapEnvBindings 2021-06-11 18:55:15 -06:00
Ben Burdette d22de1dd0c remove dead code 2021-06-09 15:38:08 -06:00
Ben Burdette 93ca9381da formatting; string arg for throwTypeError 2021-06-08 18:37:28 -06:00
Ben Burdette ebf530d31e line endings 2021-06-08 18:17:58 -06:00
Ben Burdette ff2e72054f another throwTypeError form 2021-06-08 14:44:41 -06:00
Alyssa Ross c57ab17687
Only link with libdl on Linux
Linux is (as far as I know) the only mainstream operating system that
requires linking with libdl for dlopen.  On BSD, libdl doesn't exist,
so on non-FreeBSD BSDs linking will currently fail.  On macOS, it's
apparently just a symlink to libSystem (macOS libc), presumably
present for compatibility with things that assume Linux.

So the right thing to do here is to only add -ldl on Linux, not to add
it for everything that isn't FreeBSD.
2021-06-01 08:05:21 +00:00
Ben Burdette 99304334ca showType(fun) 2021-05-14 18:09:30 -06:00
Ben Burdette 644567cf7e clean up w LocalNoInline macro 2021-05-14 13:40:00 -06:00
Ben Burdette 17af7dc326 throwAssertionError, throwUndefinedError -> valmap-ized 2021-05-14 11:29:26 -06:00
Ben Burdette d041dd874e throwEvalError form 4 2021-05-14 11:15:24 -06:00
Ben Burdette ed74eaa07f throwEvalError form 3 2021-05-14 11:09:18 -06:00
Ben Burdette 989a4181a8 throwEvalError form 2 2021-05-14 11:06:20 -06:00
Ben Burdette ab19d1685d throwEvalError; mapBindings 2021-05-13 16:00:48 -06:00
Ben Burdette 459bccc750 plain env pointer 2021-05-12 11:33:31 -06:00
Ben Burdette 0c2265da85 unique_ptr for valmap 2021-05-12 09:43:58 -06:00
Ben Burdette e7847ad7a1 map1/2 for stack usage 2021-05-11 15:38:49 -06:00
Ben Burdette a8fef9a6b1 throwTypeError with debugger/env 2021-05-10 18:36:57 -06:00
Eelco Dolstra 8768398d5f
Merge pull request #4787 from Ma27/builtins-ceil-floor
Implement `builtins.floor` and `builtins.ceil` using the C library functions internally
2021-05-10 20:37:27 +02:00
Maximilian Bosch bec33858bb
primops: math.h -> cmath
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-05-10 16:41:10 +02:00
Maximilian Bosch 7f7f99f350
Implement builtins.floor and builtins.ceil using the C library functions internally
Closes #4782

Note: even though the type is internally called `NixFloat`, it's
actually a `double`.
2021-05-10 12:19:32 +02:00
Matthew Bauer 2f63cc02de Create parent trusted list path before writing
This makes sure that $HOME/.local/share/nix exists before we try to
write to it.
2021-05-08 21:31:28 -05:00
Ben Burdette 2dd61411af debugger on autoCallFunction error 2021-05-03 14:37:33 -06:00
Ben Burdette 57c2dd5d85 fixes 2021-04-28 09:55:08 -06:00
Ben Burdette b466d51952 Merge branch 'master' into debugger-merge 2021-04-27 12:31:01 -06:00
Domen Kožar e3e78ee2a2
Merge pull request #4751 from Ma27/storepath-pos
primops/storePath: add trace to pure mode error
2021-04-27 19:05:23 +02:00
Maximilian Bosch e6a360a4d9
primops/storePath: add trace to pure mode error
As described in #4745 it's otherwise fairly hard to understand where
this is coming from. Say you have an expression which uses e.g.
`types.package`:

``` nix
{ outputs = { self, nixpkgs }: {
    packages.x86_64-linux.hello = let
      foo = nixpkgs.lib.evalModules {
        modules = [
          {
            options.foo.bar = with nixpkgs.lib; mkOption { type = types.package; };
          }
          {
            foo.bar = ./.;
          }
        ];
      };
    in builtins.trace foo.config.foo.bar.outPath nixpkgs.legacyPackages.x86_64-linux.hello;
    defaultPackage.x86_64-linux = self.packages.x86_64-linux.hello;
  };
}
```

Then you'll get an error trace like this:

```
error: 'builtins.storePath' is not allowed in pure evaluation mode

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:328:15:

          327|     let
          328|       path' = builtins.storePath path;
             |               ^
          329|       res =

       … while evaluating the attribute 'config.foo.bar.outPath'

       at /nix/store/p4h2x6r80njkb0j2rc1xjhhl99yri3zb-source/lib/attrsets.nix:332:11:

          331|           name = sanitizeDerivationName (builtins.substring 33 (-1) (baseNameOf path'));
          332|           outPath = path';
             |           ^
          333|           outputs = [ "out" ];

       … while evaluating the attribute 'packages.x86_64-linux.hello'

       at /nix/store/6c1rfsqzrhjw1235palzjmf5vihcpci7-source/flake.nix:3:5:

            2| { outputs = { self, nixpkgs }: {
            3|     packages.x86_64-linux.hello = let
             |     ^
            4|       foo = nixpkgs.lib.evalModules {
```

Fixes #4745
2021-04-27 17:24:38 +02:00
regnat 31313d1401 Replace the trailing markdown spaces by a backslash
They are equivalent according to
<https://spec.commonmark.org/0.29/#hard-line-breaks>,
and the trailing spaces tend to be a pain (because the make git
complain, editors tend to want to remove them − the `.editorconfig`
actually specifies that − etc..).
2021-04-23 14:37:21 +02:00
Eelco Dolstra 293220bed5
Merge pull request #4440 from Ma27/misc-pos-fixes
Miscellaneous improvements for positioning in eval-errors
2021-04-23 11:10:59 +02:00
Maximilian Bosch 6905291daf
libcmd/installables: force re-evaluation of cached failures
I think that it's not very helpful to get "cached failures" in a wrong
`flake.nix`. This can be very confusing when debugging a Nix expression.
See for instance NixOS/nixpkgs#118115.

In fact, the eval cache allows a forced reevaluation which is used for
e.g. `nix eval`.

This change makes sure that this is the case for `nix build` as well. So
rather than

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: cached failure of attribute 'defaultPackage.x86_64-linux'

the evaluation of already-evaluated (and failed) attributes looks like
this now:

    λ ma27 [~/Projects/exp] → ../nix/outputs/out/bin/nix build -L --rebuild --experimental-features 'nix-command flakes'
    error: attribute 'hell' missing

           at /nix/store/mrnvi9ss8zn5wj6gpn4bcd68vbh42mfh-source/flake.nix:6:35:

                5|
                6|     packages.x86_64-linux.hello = nixpkgs.legacyPackages.x86_64-linux.hell;
                 |                                   ^
                7|
    (use '--show-trace' to show detailed location information)
2021-04-19 19:51:04 +02:00
Maximilian Bosch 864ef0e93d
libexpr/primops: review 2021-04-13 23:12:39 +02:00
Maximilian Bosch ad7b47dc85
primops/libexpr: use new attr-call extractor everywhere; use function's pos if attr-set pos == noPos 2021-04-13 23:12:38 +02:00
Maximilian Bosch f60473a077
libexpr/primops: Move attr name extraction into its own function
This now takes care of providing positioning for both the faulting value
and the faulting function call in case of an error.
2021-04-13 23:12:38 +02:00
Maximilian Bosch 7c76964daa
libexpr: misc improvements for proper error position
When working on some more complex Nix code, there are sometimes rather
unhelpful or misleading error messages, especially if coerce-errors are
thrown.

This patch is a first steps towards improving that. I'm happy to file
more changes after that, but I'd like to gather some feedback first.

To summarize, this patch does the following things:

* Attrsets (a.k.a. `Bindings` in `libexpr`) now have a `Pos`. This is
  helpful e.g. to identify which attribute-set in `listToAttrs` is
  invalid.

* The `Value`-struct has a new method named `determinePos` which tries
  to guess the position of a value and falls back to a default if that's
  not possible.

  This can be used to provide better messages if a coercion fails.

* The new `determinePos`-API is used by `builtins.concatMap` now. With
  that change, Nix shows the exact position in the error where a wrong
  value was returned by the lambda.

  To make sure it's still obvious that `concatMap` is the problem,
  another stack-frame was added.

* The changes described above can be added to every other `primop`, but
  first I'd like to get some feedback about the overall approach.
2021-04-13 23:12:38 +02:00
Maximilian Bosch 3550a32b25
primops/derivation: use position of currently evaluated attribute
* The position of the `name`-attribute appears in the trace.
* If e.g. `meta` has no `outPath`-attribute, a `cannot coerce set to
  string` error will be thrown where `pos` points to `name =` which is
  highly misleading.
2021-04-13 23:12:38 +02:00
Graham Christensen d04969ffa5 EvalSettings::getDefaultNixPath: respect {restrict,pure}Eval
Otherwise Nix may look to invalid locations for channels.
2021-04-12 12:37:15 -04:00
John Ericson d0ed11ca72 Merge commit '1b6cf0d5f56e166a1cbbf38142375b7a92fc88f2' into ca-drv-exotic 2021-04-05 19:06:43 -04:00
John Ericson 386765e3ff Merge commit 'd5cef6c33a051dfc672cb1e5f4739948b167315b' into ca-drv-exotic 2021-04-05 19:06:37 -04:00
John Ericson cdc9f34a44 Merge commit 'e12308dd63f0ad27b22dcdb3da89c411eebcad2b' into ca-drv-exotic 2021-04-05 19:06:30 -04:00
John Ericson 1b6cf0d5f5 Merge remote-tracking branch 'upstream/master' into path-info 2021-04-05 18:47:33 -04:00