Commit graph

7965 commits

Author SHA1 Message Date
Tom Bereknyei 93299efc7c bundler: add tests and change defaults to use a derivation 2022-01-25 03:39:18 -05:00
Farid Zakaria 8ba7a2d3a8 Do not suppress errors in nix-env from feedback by Eelco 2022-01-24 19:12:13 -08:00
Tom Bereknyei c94db0535c Refactor bundler API
Bundlers now expect to be located at bundlers.<system>.<name> and are a
function from derivations to derivations.
2022-01-24 21:43:04 -05:00
Tom Bereknyei 3be810f5db bundler: pass drv attrset instead of path 2022-01-24 21:43:04 -05:00
Eelco Dolstra 8cbbaf23e8 Allow builtins.{readFile,path} on invalid paths
Stop-gap measure to fix #5975.
2022-01-24 23:02:28 +01:00
Eelco Dolstra bed8270c0c Fix parsing of variable names that are a prefix of '__curPos'
Fixes

  $ nix-instantiate --parse -E 'x: with x; _'
  (x: (with x; __curPos))
2022-01-24 15:18:18 +01:00
Eelco Dolstra c4fc9b6a8d ExprConcatStrings::show(): Print values instead of pointers 2022-01-24 15:08:16 +01:00
Eelco Dolstra 7afbdf2545 hiliteMatches(): Style fixes, pass more stuff by reference 2022-01-24 14:47:34 +01:00
Théophane Hufschmitt 4530574363
Merge pull request #5945 from afishhh/master
Make `nix search` highlight all regexes and matches
2022-01-24 13:02:51 +01:00
Eelco Dolstra 8c778e3d9d
Merge pull request #5964 from pennae/fix-bulk-alloc
allocate a GC root value for the Value cache pointer
2022-01-24 12:43:14 +01:00
Eelco Dolstra 1e2bfd3da5
Merge pull request #5965 from SuperSandro2000/nix-shell-rm
nix-shell: use clean rm, not any aliases/functions
2022-01-24 12:40:53 +01:00
Eelco Dolstra 076945c808
Merge pull request #5875 from hercules-ci/fix-large-drv-field-stack-overflow
Fix segfault or stack overflow caused by large derivation fields
2022-01-24 12:05:30 +01:00
regnat ffb28eaa1e Add a small documentation for hiliteMatches 2022-01-24 10:07:02 +01:00
Sandro Jäckel 040cb7304a
nix-shell: use clean rm, not any aliases/functions 2022-01-23 04:58:00 +01:00
pennae fd5aa6ee3e allocate a GC root value for the Value cache pointer
keeping it as a simple data member means it won't be scanned by the GC, so
eventually the GC will collect a cache that is still referenced (resulting in
use-after-free of cache elements).

fixes #5962
2022-01-22 21:19:56 +01:00
Kevin Amado 50efc5499a
determinePos: remove from critical path 2022-01-21 16:32:43 -05:00
Kevin Amado 3d2ad2b70b
forceList: make pos mandatory 2022-01-21 16:32:43 -05:00
Kevin Amado c3896e19d0
forceAttrs: make pos mandatory 2022-01-21 16:32:43 -05:00
Kevin Amado 1472e045a7
avoid unnecesary calls 2022-01-21 16:32:43 -05:00
Kevin Amado 49b0bb0206
forceValue: make pos mandatory
- Make passing the position to `forceValue` mandatory,
  this way we remember people that the position is
  important for better error messages
- Add pos to all `forceValue` calls
2022-01-21 16:32:43 -05:00
Fishhh f82a426502
Add some tests for hiliteMatches in libutil 2022-01-21 20:10:54 +01:00
Fishhh 1e0b7cdc3f
Move hilite_all into libutil and rename it to hiliteMatches
The signature was also changed so the function now accepts a vector
instead of an iterator
2022-01-21 20:10:46 +01:00
Robert Hensing 0407436b0f
derivations.cc: Use larger buffer in printString
If we want to be careful about hitting the stack protector page, we should use `-fstack-check` instead.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-01-21 17:25:37 +01:00
Eelco Dolstra 128098040b
Fix exception handling around realisePath()
This no longer worked correctly because 'path' is uninitialised when
an exception occurs, leading to errors like

       … while importing ''

       at /nix/store/rrzz5b1pshvzh1437ac9nkl06br81lkv-source/flake.nix:352:13:

So move the adding of the error context into realisePath().
2022-01-21 13:53:18 +01:00
Eelco Dolstra e85cf34ea3
Merge pull request #5949 from obsidiansystems/old-copyClosure
Add back `copyClosure` for plain `StorePath`s
2022-01-21 11:33:32 +01:00
Eelco Dolstra accfcfff6b
Merge pull request #5943 from NixOS/3204-fix-build-with-newer-nlohmann-json
Fix the build with nlohmann/json 3.10.4+
2022-01-21 11:31:39 +01:00
Eelco Dolstra c7223db871
Merge pull request #5942 from NixOS/5912-quieter-nix-why-depends
Make `nix why-depends` quieter by default
2022-01-21 10:18:28 +01:00
Théophane Hufschmitt fa53250c36
Improve the description of the --precise option
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2022-01-21 09:52:40 +01:00
John Ericson 5ee937523d Add back copyClosure for plain StorePaths
This was removed in 2e199673a5 when
`copyPath` transitioned to use `RealisedPath`. But then in
e9848beca7 we added it back just for
`realisedPath`.

I think it is a good utility function --- one can easily imagine it
becoming optimized in the future, and copying paths *violating* the
closure is a very niche feature.

So if we have `copyPaths` for both sorts of paths, I think we should
have `copyClosure` for both sorts too.
2022-01-20 20:57:44 +00:00
Hubert Głuchowski 9510ad10c5
Make hilite_all take an iterator of matches instead of a vector. 2022-01-20 17:12:15 +01:00
John Ericson 89f8917a32 Remove dead field in NixArgs
This has been unused since 170e86dff5

CC @thufschmitt
2022-01-19 20:26:14 +00:00
Hubert Głuchowski 87fdd23025
Make nix search highlight all matches of a regex 2022-01-19 20:45:31 +01:00
Hubert Głuchowski 92e8230215
Make nix search highlight all regexes 2022-01-19 19:08:02 +01:00
pennae 7d4cc5515c defer formals duplicate check for incresed efficiency all round
if we defer the duplicate argument check for lambda formals we can use more
efficient data structures for the formals set, and we can get rid of the
duplication of formals names to boot. instead of a list of formals we've seen
and a set of names we'll keep a vector instead and run a sort+dupcheck step
before moving the parsed formals into a newly created lambda. this improves
performance on search and rebuild by ~1%, pure parsing gains more (about 4%).

this does reorder lambda arguments in the xml output, but the output is still
stable. this shouldn't be a problem since argument order is not semantically
important anyway.

 before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.550 s ±  0.060 s    [User: 6.470 s, System: 1.664 s]
    Range (min … max):    8.435 s …  8.666 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     346.7 ms ±   2.1 ms    [User: 312.4 ms, System: 34.2 ms]
    Range (min … max):   343.8 ms … 353.4 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.720 s ±  0.031 s    [User: 2.415 s, System: 0.231 s]
    Range (min … max):    2.662 s …  2.780 s    20 runs

 after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.462 s ±  0.063 s    [User: 6.398 s, System: 1.661 s]
    Range (min … max):    8.339 s …  8.542 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     329.1 ms ±   1.4 ms    [User: 296.8 ms, System: 32.3 ms]
    Range (min … max):   326.1 ms … 330.8 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.687 s ±  0.035 s    [User: 2.392 s, System: 0.228 s]
    Range (min … max):    2.626 s …  2.754 s    20 runs
2022-01-19 17:07:29 +01:00
Robert Hensing dec7748119 Replace withBuffer by boost small_vector
Although this will leave gaps in the stack, the performance impact
of those should be insignificant and we get a simpler solution
this way.
2022-01-19 15:21:56 +01:00
Robert Hensing 624f18ad90 withBuffer: Make sure to hit the stack protector 2022-01-19 15:21:56 +01:00
Robert Hensing 6dd271b7b4 withBuffer: avoid allocating a std::function 2022-01-19 15:21:56 +01:00
Robert Hensing 55c58580be Add withBuffer
... to avoid non-standard, unidiomatic alloca.
2022-01-19 15:21:56 +01:00
Robert Hensing d038a67bd3 Fix segfault or stack overflow caused by large derivation fields
This removes a dynamic stack allocation, making the derivation
unparsing logic robust against overflows when large strings are
added to a derivation.
Overflow behavior depends on the platform and stack configuration.

For instance, x86_64-linux/glibc behaves as (somewhat) expected:

$ (ulimit -s 20000; nix-instantiate tests/lang/eval-okay-big-derivation-attr.nix)
error: stack overflow (possible infinite recursion)

$ (ulimit -s 40000; nix-instantiate tests/lang/eval-okay-big-derivation-attr.nix)
error: expression does not evaluate to a derivation (or a set or list of those)

However, on aarch64-darwin:

$ nix-instantiate big-attr.nix                                                                                                                                                                                                                                                       ~
zsh: segmentation fault  nix-instantiate big-attr.nix

This indicates a slight flaw in the single stack protection page
approach that is not encountered with normal stack frames.
2022-01-19 15:21:56 +01:00
pennae 9ac836d1d6 don't use Symbols for strings
string expressions by and large do not need the benefits a Symbol gives us,
instead they pollute the symbol table and cause unnecessary overhead for almost
all strings. the one place we can think of that benefits from them (attrpaths
with expressions) extracts the benefit in the parser, which we'll have to touch
anyway when changing ExprString to hold strings.

this gives a sizeable improvement on of 3-5% on all benchmarks we've run.

 before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.844 s ±  0.045 s    [User: 6.750 s, System: 1.663 s]
    Range (min … max):    8.758 s …  8.922 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     367.4 ms ±   3.3 ms    [User: 332.3 ms, System: 35.2 ms]
    Range (min … max):   364.0 ms … 375.2 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.810 s ±  0.030 s    [User: 2.517 s, System: 0.225 s]
    Range (min … max):    2.742 s …  2.854 s    20 runs

 after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.533 s ±  0.068 s    [User: 6.485 s, System: 1.642 s]
    Range (min … max):    8.404 s …  8.657 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     347.6 ms ±   3.1 ms    [User: 313.1 ms, System: 34.5 ms]
    Range (min … max):   343.3 ms … 354.6 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.709 s ±  0.032 s    [User: 2.414 s, System: 0.232 s]
    Range (min … max):    2.655 s …  2.788 s    20 runs
2022-01-19 14:48:00 +01:00
regnat e36add56cf Fix the build with nlohmann/json 3.10.4+ 2022-01-19 14:37:54 +01:00
regnat dd7c2e0695 Make nix why-depends quieter by default
Unless `--precise` is passed, make `nix why-depends` only show the
dependencies between the store paths, without introspecting them to
find the actual references.

This also makes it ~3x faster
2022-01-19 14:24:14 +01:00
pennae 0a7746603e remove ExprIndStr
it can be replaced with StringToken if we add another bit if information to
StringToken, namely whether this string should take part in indentation scanning
or not. since all escaping terminates indentation scanning we need to set this
bit only for the non-escaped IND_STRING rule.

this improves performance by about 1%.

 before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.880 s ±  0.048 s    [User: 6.809 s, System: 1.643 s]
    Range (min … max):    8.781 s …  8.993 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     375.0 ms ±   2.2 ms    [User: 339.8 ms, System: 35.2 ms]
    Range (min … max):   371.5 ms … 379.3 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.831 s ±  0.040 s    [User: 2.536 s, System: 0.225 s]
    Range (min … max):    2.769 s …  2.912 s    20 runs

 after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      8.832 s ±  0.048 s    [User: 6.757 s, System: 1.657 s]
    Range (min … max):    8.743 s …  8.921 s    20 runs

  nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
    Time (mean ± σ):     367.4 ms ±   3.2 ms    [User: 332.7 ms, System: 34.7 ms]
    Range (min … max):   364.6 ms … 374.6 ms    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.810 s ±  0.030 s    [User: 2.517 s, System: 0.225 s]
    Range (min … max):    2.742 s …  2.854 s    20 runs
2022-01-19 13:39:42 +01:00
lincoln auster [they/them] 3e5a9ad7ff allow modifying lockfile commit msg with nix config option
This allows setting the commit-lockfile-summary option to a non-empty
string to override the commit summary while leaving the body unchanged.
2022-01-18 16:10:27 -07:00
Eelco Dolstra 4af88a4c91
Merge pull request #5906 from pennae/primops-optimization
optimize primops and utils by caching more and copying less
2022-01-18 19:43:28 +01:00
Eelco Dolstra 9901cb96c7
Merge pull request #5918 from tweag/tty-logs
Print build logs when not connected to a TTY
2022-01-18 19:42:01 +01:00
Eelco Dolstra 04432f2510 Add examples 2022-01-18 17:37:04 +01:00
Eelco Dolstra 3876238546 Add Installable::toDrvPaths()
This is needed to get the path of a derivation that might not exist
(e.g. for 'nix store copy-log').

InstallableStorePath::toDerivedPaths() cannot be used for this because
it calls readDerivation(), so it fails if the store doesn't have the
derivation.
2022-01-18 17:28:18 +01:00
Eelco Dolstra 5b243a2b4b BinaryCacheStore: Implement addBuildLog() 2022-01-18 16:14:01 +01:00
Eelco Dolstra 4dda1f92aa Add command 'nix store copy-log'
Fixes #5222.
2022-01-18 14:08:49 +01:00
Eelco Dolstra 6448ea84ab Factor out --from / --to logic 2022-01-18 14:08:49 +01:00
Eelco Dolstra f6f0bcf11f
Doh 2022-01-18 14:06:51 +01:00
Eelco Dolstra 50be51d9a8 Doh 2022-01-18 13:50:25 +01:00
Eelco Dolstra 3157028fc1
Merge pull request #5932 from edolstra/remove-shared-strings
Remove shared strings
2022-01-18 11:14:18 +01:00
Eelco Dolstra d62a9390fc Get rid of std::shared_ptr<std::string> and ref<std::string>
These were needed back in the pre-C++11 era because we didn't have
move semantics. But now we do.
2022-01-18 11:12:30 +01:00
Théophane Hufschmitt dc4730ee94
Merge pull request #5720 from tomberek/flake_search
flakes: search up to git or filesystem boundary
2022-01-18 10:59:12 +01:00
regnat 5753f6efbb Fix the rendering of the example directory tree 2022-01-18 10:55:00 +01:00
Eelco Dolstra 52ee7ec002 StringSource: Use std::string_view 2022-01-17 22:20:23 +01:00
Eelco Dolstra 776eb97a43 serialise.hh: Use std::string_view 2022-01-17 22:20:23 +01:00
Eelco Dolstra fc2443a67c
Merge pull request #5812 from pennae/small-perf-improvements
improve parser performance a bit
2022-01-17 19:49:52 +01:00
Alexander Bantyev 34b66aab00
Update documentation for paths on command line
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-01-17 20:00:04 +03:00
Alexander Bantyev 8cf54f754d
Show build and substitution information when not connected to a TTY
When stderr is not connected to a tty, show "building" and
"substituting" messages, a-la nix-build et al.

Closes https://github.com/NixOS/nix/issues/4402

Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-01-17 19:57:29 +03:00
Eelco Dolstra bb24d1edd7
Merge pull request #5914 from NixOS/5911-fix-nix-why-depends-pager
Start the pager early-enough in `nix why-depends`
2022-01-17 10:10:31 +01:00
Farid Zakaria 61f02f7f20 Make queryJSON not bail immediately on an assertion or error 2022-01-15 19:36:07 -08:00
Alexander Bantyev f055cc5a0b
Document searching upwards and fix documentation for installables 2022-01-14 17:58:03 +03:00
Alexander Bantyev e3690ab393
Add more tests for flake upward searching 2022-01-14 17:21:18 +03:00
tomberek 2dead20924
Apply suggestions from code review
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
2022-01-14 09:16:34 -05:00
pennae ad60dfde2a also cache split regexes, not just match regexes
gives about 1% improvement on system eval, a bit less on nix search.

 # before

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      7.419 s ±  0.045 s    [User: 6.362 s, System: 0.794 s]
    Range (min … max):    7.335 s …  7.517 s    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.921 s ±  0.023 s    [User: 2.626 s, System: 0.210 s]
    Range (min … max):    2.883 s …  2.957 s    20 runs

 # after

  nix search --no-eval-cache --offline ../nixpkgs hello
    Time (mean ± σ):      7.370 s ±  0.059 s    [User: 6.333 s, System: 0.791 s]
    Range (min … max):    7.286 s …  7.541 s    20 runs

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.891 s ±  0.033 s    [User: 2.606 s, System: 0.210 s]
    Range (min … max):    2.823 s …  2.958 s    20 runs
2022-01-14 14:04:17 +01:00
pennae c9fc975259 optimize removeAttrs builtin
use a sorted array of symbols to be removed instead of a set. this saves a lot
of memory allocations and slightly speeds up removal.
2022-01-14 14:01:52 +01:00
regnat 81cd0a113b Start the pager early-enough in nix why-depends
`nix why-depends` is piping its output into a pager by default.
However the pager was only started after the first path is printed,
causing it to be excluded from the pager output.

(Actually the pager was started *inside* the recursive function that was
printing the dependency chain, so a new instance was started at each
level. It’s a little miracle that it worked at all).

Fix #5911
2022-01-14 13:41:45 +01:00
pennae 72f42093e7 optimize unescapeStr
mainly to avoid an allocation and a copy of a string that can be
modified in place (ever since EvalState holds on to the buffer, not the
generated parser itself).

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     571.7 ms ±   2.4 ms    [User: 563.3 ms, System: 8.0 ms]
  Range (min … max):   566.7 ms … 579.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     376.6 ms ±   1.0 ms    [User: 345.8 ms, System: 30.5 ms]
  Range (min … max):   374.5 ms … 379.1 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.922 s ±  0.006 s    [User: 2.707 s, System: 0.215 s]
  Range (min … max):    2.906 s …  2.934 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     570.4 ms ±   2.8 ms    [User: 561.3 ms, System: 8.6 ms]
  Range (min … max):   564.6 ms … 578.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     375.4 ms ±   1.3 ms    [User: 343.2 ms, System: 31.7 ms]
  Range (min … max):   373.4 ms … 378.2 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.925 s ±  0.006 s    [User: 2.704 s, System: 0.219 s]
  Range (min … max):    2.910 s …  2.942 s    50 runs
2022-01-13 18:06:15 +01:00
pennae 34e3bd10e3 avoid copies of parser input data
when given a string yacc will copy the entire input to a newly allocated
location so that it can add a second terminating NUL byte. since the
parser is a very internal thing to EvalState we can ensure that having
two terminating NUL bytes is always possible without copying, and have
the parser itself merely check that the expected NULs are present.

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     572.4 ms ±   2.3 ms    [User: 563.4 ms, System: 8.6 ms]
  Range (min … max):   566.9 ms … 579.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     381.7 ms ±   1.0 ms    [User: 348.3 ms, System: 33.1 ms]
  Range (min … max):   380.2 ms … 387.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.936 s ±  0.005 s    [User: 2.715 s, System: 0.221 s]
  Range (min … max):    2.923 s …  2.946 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     571.7 ms ±   2.4 ms    [User: 563.3 ms, System: 8.0 ms]
  Range (min … max):   566.7 ms … 579.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     376.6 ms ±   1.0 ms    [User: 345.8 ms, System: 30.5 ms]
  Range (min … max):   374.5 ms … 379.1 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.922 s ±  0.006 s    [User: 2.707 s, System: 0.215 s]
  Range (min … max):    2.906 s …  2.934 s    50 runs
2022-01-13 18:06:15 +01:00
pennae eee0bcee22 avoid allocations in SymbolTable::create
speeds up parsing by ~3%, system builds by a bit more than 1%

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     574.7 ms ±   2.8 ms    [User: 566.3 ms, System: 8.0 ms]
  Range (min … max):   569.2 ms … 580.7 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     394.4 ms ±   0.8 ms    [User: 361.8 ms, System: 32.3 ms]
  Range (min … max):   392.7 ms … 395.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.976 s ±  0.005 s    [User: 2.757 s, System: 0.218 s]
  Range (min … max):    2.966 s …  2.990 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     572.4 ms ±   2.3 ms    [User: 563.4 ms, System: 8.6 ms]
  Range (min … max):   566.9 ms … 579.1 ms    50 runs

Benchmark 2: nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  Time (mean ± σ):     381.7 ms ±   1.0 ms    [User: 348.3 ms, System: 33.1 ms]
  Range (min … max):   380.2 ms … 387.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.936 s ±  0.005 s    [User: 2.715 s, System: 0.221 s]
  Range (min … max):    2.923 s …  2.946 s    50 runs
2022-01-13 18:06:15 +01:00
pennae 61a9d16d5c don't strdup tokens in the lexer
every stringy token the lexer returns is turned into a Symbol and not
used further, so we don't have to strdup. using a string_view is
sufficient, but due to limitations of the current parser we have to use
a POD type that holds the same information.

gives ~2% on system build, 6% on search, 8% on parsing alone

 # before

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     610.6 ms ±   2.4 ms    [User: 602.5 ms, System: 7.8 ms]
  Range (min … max):   606.6 ms … 617.3 ms    50 runs

Benchmark 2: nix eval -f hackage-packages.nix
  Time (mean ± σ):     430.1 ms ±   1.4 ms    [User: 393.1 ms, System: 36.7 ms]
  Range (min … max):   428.2 ms … 434.2 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      3.032 s ±  0.005 s    [User: 2.808 s, System: 0.223 s]
  Range (min … max):    3.023 s …  3.041 s    50 runs

 # after

Benchmark 1: nix search --offline nixpkgs hello
  Time (mean ± σ):     574.7 ms ±   2.8 ms    [User: 566.3 ms, System: 8.0 ms]
  Range (min … max):   569.2 ms … 580.7 ms    50 runs

Benchmark 2: nix eval -f hackage-packages.nix
  Time (mean ± σ):     394.4 ms ±   0.8 ms    [User: 361.8 ms, System: 32.3 ms]
  Range (min … max):   392.7 ms … 395.7 ms    50 runs

Benchmark 3: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      2.976 s ±  0.005 s    [User: 2.757 s, System: 0.218 s]
  Range (min … max):    2.966 s …  2.990 s    50 runs
2022-01-13 18:06:14 +01:00
pennae 6401e443a4 move strings in derivationStrict
the temporary will be discarded anyway, so we can move out of it and save many
small allocations and copies.
2022-01-13 14:00:20 +01:00
pennae ef45787aae avoid string copies in attrNames sort comparison
symbols can also be cast to string_view, which compares the same but doesn't
require a copy of both symbol names on every comparison.
2022-01-13 14:00:19 +01:00
pennae 1bebb1095a cache more often-used symbols for primops
there's a few symbols in primops we can create once and pick them out of
EvalState afterwards instead of creating them every time we need them. this
gives almost 1% speedup to an uncached nix search.
2022-01-13 13:58:33 +01:00
pennae 44c92a1667 use more string_view in utils
there's a couple places that can be easily converted from using strings to using
string_views instead. gives a slight (~1%) boost to system eval.

 # before

  nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.946 s ±  0.026 s    [User: 2.655 s, System: 0.209 s]
    Range (min … max):    2.905 s …  2.995 s    20 runs

 # after

    Time (mean ± σ):      2.928 s ±  0.024 s    [User: 2.638 s, System: 0.211 s]
    Range (min … max):    2.893 s …  2.970 s    20 runs
2022-01-13 13:51:29 +01:00
Théophane Hufschmitt e61c4bc25a
Merge pull request #5887 from pennae/avoid-streams
avoid std::?stream overhead when it's not helpful
2022-01-12 10:52:40 +01:00
pennae 73fcc40fa4 use boost::lexical_cast for string2*
this avoids one copy from `s` into `str`, and possibly another copy needed to
construct `s` at the call site. lexical_cast is also more efficient in general.
2022-01-12 10:07:21 +01:00
pennae 5838354d34 optimize ExprConcatStrings::eval
constructing an ostringstream for non-string concats (like integer addition) is
a small constant cost that we can avoid. for string concats we can keep all the
string temporaries we get from coerceToString and concatenate them in one go,
which saves a lot of intermediate temporaries and copies in ostringstream. we
can also avoid copying the concatenated string again by directly allocating it
in GC memory and moving ownership of the concatenated string into the target
value.

saves about 2% on system eval.

before:

  Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.837 s ±  0.031 s    [User: 2.562 s, System: 0.191 s]
    Range (min … max):    2.796 s …  2.892 s    20 runs

after:

  Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
    Time (mean ± σ):      2.790 s ±  0.035 s    [User: 2.532 s, System: 0.187 s]
    Range (min … max):    2.722 s …  2.836 s    20 runs
2022-01-12 10:07:21 +01:00
Théophane Hufschmitt d023903b6f
Merge pull request #5883 from fzakaria/faridzakaria/check-meta
Check for `--meta` before emitting it for `--json`
2022-01-11 16:57:28 +01:00
Eelco Dolstra 8a446aff75
Merge pull request #5898 from layus/repair-path-links
Make --repair-path also repair corrupt optimised links
2022-01-11 14:14:44 +01:00
Guillaume Maudoux 9f9f39a24b Prefer RepairFlag over bool when applicable 2022-01-11 13:38:28 +01:00
Guillaume Maudoux e9a4abdb5d Make --repair-path also repair corrupt optimised links
There already existed a smoke test for the link content length,
but it appears that there exists some corruptions pernicious enough
to replace the file content with zeros, and keeping the same length.

--repair-path now goes as far as checking the content of the link,
making it true to its name and actually repairing the path for such
coruption cases.
2022-01-11 11:57:45 +01:00
Eelco Dolstra f39d94a55b Merge branch 'readfile' of https://github.com/abbradar/nix 2022-01-10 16:55:28 +01:00
Eelco Dolstra 19fd6e585d 'target' points to 'source' 2022-01-10 16:52:25 +01:00
Eelco Dolstra df11e75d0e Merge branch 'recursive-nix-fix' of https://github.com/L-as/nix 2022-01-10 16:51:59 +01:00
Nikolay Amiantov c66865dff1 builtins.readFile: Propagate path context
Co-authored-by: Shea Levy <shea@shealevy.com>
2022-01-09 13:07:00 +03:00
Ben Burdette 990bec78d3 clear screen and show top debug trace 2022-01-08 15:43:04 -07:00
Ben Burdette a963674d88 optinoal error; compiles 2022-01-08 11:03:48 -07:00
Ben Burdette c51b527c28 add env to DebugTrace 2022-01-07 16:37:44 -07:00
Eelco Dolstra fcddaa4b9b dump(): Use emplace() 2022-01-07 13:23:17 +01:00
Eelco Dolstra e1a0359b59 isInDir() / isDirOrInDir(): Use std::string_view 2022-01-07 13:23:00 +01:00
Eelco Dolstra be64fb9b51 DerivationGoal::loadDerivation(): Don't use derivationFromPath()
This causes a recursive call to ensurePath(), which is not a good
idea.
2022-01-07 13:22:34 +01:00
pennae 26a8b220eb avoid ostream sentries per json string character
we don't have to create an ostream sentry object for every character of a JSON
string we write. format a bunch of characters and flush them to the stream all
at once instead.

this doesn't affect small numbers of string characters, but larger numbers of
total JSON string characters written gain a lot. at 1MB of total string written
we gain almost 30%, at 16MB it's almost a factor of 3x. large numbers of JSON
string characters do occur naturally in a nixos system evaluation to generate
documentation (though this is now somewhat mitigated by caching the largest part
of nixos option docs).

benchmarked with

  hyperfine 'nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) {e})"' --warmup 1 -L e 1,4,256,4096,65536

before:

  Benchmark 1: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 1)"
    Time (mean ± σ):      12.5 ms ±   0.2 ms    [User: 9.2 ms, System: 4.0 ms]
    Range (min … max):    11.9 ms …  13.1 ms    223 runs

  Benchmark 2: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4)"
    Time (mean ± σ):      12.5 ms ±   0.2 ms    [User: 9.3 ms, System: 3.8 ms]
    Range (min … max):    11.9 ms …  13.2 ms    220 runs

  Benchmark 3: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 256)"
    Time (mean ± σ):      13.2 ms ±   0.3 ms    [User: 9.8 ms, System: 4.0 ms]
    Range (min … max):    12.6 ms …  14.3 ms    205 runs

  Benchmark 4: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4096)"
    Time (mean ± σ):      24.0 ms ±   0.4 ms    [User: 19.4 ms, System: 5.2 ms]
    Range (min … max):    22.7 ms …  25.8 ms    119 runs

  Benchmark 5: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 65536)"
    Time (mean ± σ):     196.0 ms ±   3.7 ms    [User: 171.2 ms, System: 25.8 ms]
    Range (min … max):   190.6 ms … 201.5 ms    14 runs

after:

  Benchmark 1: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 1)"
    Time (mean ± σ):      12.4 ms ±   0.3 ms    [User: 9.1 ms, System: 4.0 ms]
    Range (min … max):    11.7 ms …  13.3 ms    204 runs

  Benchmark 2: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4)"
    Time (mean ± σ):      12.4 ms ±   0.2 ms    [User: 9.2 ms, System: 3.9 ms]
    Range (min … max):    11.8 ms …  13.0 ms    214 runs

  Benchmark 3: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 256)"
    Time (mean ± σ):      12.6 ms ±   0.2 ms    [User: 9.5 ms, System: 3.8 ms]
    Range (min … max):    12.1 ms …  13.3 ms    209 runs

  Benchmark 4: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 4096)"
    Time (mean ± σ):      15.9 ms ±   0.2 ms    [User: 11.4 ms, System: 5.1 ms]
    Range (min … max):    15.2 ms …  16.4 ms    171 runs

  Benchmark 5: nix eval --raw --expr "let s = __concatStringsSep \"\" (__genList (_: \"c\") 256); in __toJSON (__genList (_: s) 65536)"
    Time (mean ± σ):      69.0 ms ±   0.9 ms    [User: 44.3 ms, System: 25.3 ms]
    Range (min … max):    67.2 ms …  70.9 ms    42 runs
2022-01-07 06:53:47 +01:00
Farid Zakaria e36fdbbfd9 Check for --meta before emitting it for --json
Check that the meta flag is present when emitting JSON query information
for `nix-env`.

fixes #5882
2022-01-06 14:40:02 -08:00
Eelco Dolstra f71d84672b Merge branch 'install-nlohmann-json-headers' of https://github.com/hercules-ci/nix 2022-01-06 22:34:03 +01:00
Eelco Dolstra 5c7d3b351f
Merge pull request #5878 from fzakaria/faridzakaria/json-outputs
Add outputs to JSON query
2022-01-06 22:33:11 +01:00
Eelco Dolstra 3ec02c6743
Merge pull request #5546 from NixOS/allow-paths-in-flake-local-options
allow paths in flake local options
2022-01-06 21:55:20 +01:00
Farid Zakaria 65257614ea Add outputs to JSON query
Emit output information when printing JSON information
and `--out-paths` is requested.

fixes #5877
2022-01-06 09:13:18 -08:00
Domen Kožar bdc577936f
Merge pull request #5749 from GavinRay97/patch-1
Better diagnostics if no valid signature found
2022-01-06 08:39:58 +01:00
Ben Burdette 84aeb74377 revert value-add 2022-01-05 14:25:45 -07:00
Ben Burdette bf8a065be0 add colors; remove headings 2022-01-05 12:28:31 -07:00
Ben Burdette a4d8a799b7 tidy up debugtraces 2022-01-05 12:21:18 -07:00
Eelco Dolstra 2b4c944823 Remove EvalState::mkAttrs() 2022-01-04 20:29:17 +01:00
Eelco Dolstra 17daec0b83 Move empty attrset optimisation 2022-01-04 19:23:11 +01:00
Eelco Dolstra ca5baf2392 Turn mkString(Symbol) into a method 2022-01-04 19:09:40 +01:00
Eelco Dolstra ed93aec3c3 Remove non-method mkPath() 2022-01-04 18:45:16 +01:00
Eelco Dolstra 263a8d293c Remove non-method mk<X> functions 2022-01-04 18:40:39 +01:00
Eelco Dolstra cc08364315 Remove non-method mkString() 2022-01-04 18:24:42 +01:00
Eelco Dolstra 6d9a6d2cc3 Ensure that attrsets are sorted
Previously you had to remember to call value->attrs->sort() after
populating value->attrs. Now there is a BindingsBuilder helper that
wraps Bindings and ensures that sort() is called before you can use
it.
2022-01-04 18:00:33 +01:00
Ben Burdette 1b6b33d43d filter out underscore names 2022-01-03 18:29:43 -07:00
Ben Burdette c669108981 merge cleanup 2022-01-03 18:13:16 -07:00
Ben Burdette a47de1ac37 Merge branch 'master' into debug-exploratory-PR 2022-01-03 16:08:28 -07:00
pennae 00c993f48b add zipAttrsWith primop
nixpkgs can save a good bit of eval memory with this primop. zipAttrsWith is
used quite a bit around nixpkgs (eg in the form of recursiveUpdate), but the
most costly application for this primop is in the module system. it improves
the implementation of zipAttrsWith from nixpkgs by not checking an attribute
multiple times if it occurs more than once in the input list, allocates less
values and set elements, and just avoids many a temporary object in general.

nixpkgs has a more generic version of this operation, zipAttrsWithNames, but
this version is only used once so isn't suitable for being the base of a new
primop. if it were to be used more we should add a second primop instead.
2022-01-03 21:05:53 +01:00
Eelco Dolstra 96d08fcd66
Merge pull request #5839 from tweag/balsoft/yet-another-follows-bugfix
flake.cc: computeLocks: Only verify overrides when they could change
2022-01-03 20:51:58 +01:00
Eelco Dolstra 70dfcbbb37
Merge pull request #5840 from tweag/balsoft/nix-repl-show-trace
nix repl: fix --show-trace and add the ability to set trace display
2022-01-03 20:41:59 +01:00
Sebastian Ullrich d0c8e9254e Fix IFD with chroot store 2021-12-29 19:00:02 +01:00
Alexander Bantyev a26351da02
Add ability to toggle show-trace from within the repl 2021-12-29 02:54:36 +03:00
Lily Foster 92c7d33ee3 Sort attrs from tables in fromTOML
This was dropped in 10a8b5d for the migration from cpptoml to toml11 but
seems to be necessary for the attrsets to work correctly.

Fixes #5833
2021-12-28 10:11:00 -05:00
Alexander Bantyev 581f774284
BaseError::calcWhat: take loggerSettings.showTrace into account
Text representation for errors should include the trace if
--show-trace is passed.
2021-12-28 15:53:21 +03:00
Alexander Bantyev 2664a216e5
flake.cc: computeLocks: Only verify overrides when they could change
When we check for disappeared overrides, we can get "false positives"
for follows and overrides which are defined in the dependencies of the
flake we are locking, since they are not parsed by
parseFlakeInputs. However, at that point we already know that the
overrides couldn't have possible been changed if the input itself
hasn't changed (since we check that oldLock->originalRef == *input.ref
for the input's parent). So, to prevent this, only perform this check
when it was possible that the flake changed (e.g. the flake we're
locking, or a new input, or the input has changed and mustRefetch ==
true).
2021-12-28 15:30:06 +03:00
Ben Burdette 5954cbf3e9 more cleanup 2021-12-27 18:29:55 -07:00
Ben Burdette 4610e02d04 remove debug code 2021-12-27 18:12:46 -07:00
Ben Burdette 9760fa8661 add DebugTrace for the current error 2021-12-27 17:35:27 -07:00
Ben Burdette 6801a423fc :d env 2021-12-27 16:28:45 -07:00
Ben Burdette 2a66c120e6 by refernce for addVarToScope 2021-12-27 14:48:34 -07:00
Ben Burdette ff82ba98b4 don't add builtins to extras, initEnv() in regular repl 2021-12-27 14:06:04 -07:00
Ben Burdette d0d5890445 don't add underscore names to extras 2021-12-27 13:47:35 -07:00
Yorick a4ab0a74d9
Fix accidental O(n^2 * log n) performance in NixRepl::addAttrsToScope
Only sort once, after adding all of the attrs first. This reduces my
`nix repl '<nixpkgs>'` loading time from 1.07s to 103ms.

Fixes #5823
2021-12-27 13:18:55 +01:00
Ben Burdette e5eebda194 DebugTrace 2021-12-23 13:36:39 -07:00
tomberek f404189368
Merge branch 'master' into flake_search 2021-12-23 15:22:52 -05:00
Ben Burdette deb1fd66e8 makeDebugTraceStacker 2021-12-23 09:08:41 -07:00
Ben Burdette 1bda6a01e1 indenting 2021-12-23 08:14:17 -07:00
Nicolas Mattia b16643b6fc Use int64_t and NixFloat in fromTOML types
This makes sure that values parsed from TOML have a proper size. Using
e.g. `double` caused issues on i686 where the size of `double` (32bit)
was too small to accommodate some values.
2021-12-23 14:12:49 +01:00
regnat dc89dfa7b3 Properly return false on builtins.pathExists /someNonAllowedPath
Follow-up from https://github.com/NixOS/nix/pull/5807 to fix https://github.com/NixOS/nix/pull/5807#issuecomment-1000135394
2021-12-23 10:49:33 +01:00
Ben Burdette bc20e54e00 stack traces basically working 2021-12-22 19:40:08 -07:00
Ben Burdette b4a59a5eec DebugStackTracker class in one place 2021-12-22 15:38:49 -07:00
Eelco Dolstra 7feb741e00
Merge pull request #5821 from edolstra/remove-affinity
Remove CPU locking
2021-12-22 20:31:19 +01:00
Eelco Dolstra b666a2ca8c
Merge pull request #5820 from ncfavier/completion-nospace
Don't insert spaces when completing attribute paths
2021-12-22 17:55:05 +01:00
Naïm Favier c4a03bc4ae
Fix attr path completion after a dot 2021-12-22 16:37:58 +01:00
Naïm Favier 1da1b2b345
Don't insert spaces when completing attribute paths 2021-12-22 16:17:01 +01:00
Eelco Dolstra 9747ea84b4 Remove CPU locking
This was already accidentally disabled in ba87b08. It also no longer
appears to be beneficial, and in fact slow things down, e.g. when
evaluating a NixOS system configuration:

  elapsed time:       median =      3.8170  mean =      3.8202  stddev =      0.0195  min =      3.7894  max =      3.8600  [rejected, p=0.00000, Δ=0.36929±0.02513]
2021-12-22 15:56:25 +01:00
Eelco Dolstra 89faff93e2
Merge pull request #5807 from NixOS/5805-ca-ifd
Fix IFD with CA derivations
2021-12-21 18:47:34 +01:00
Eelco Dolstra 25cea2b737 Merge branch 'nm-toml11' of https://github.com/nmattia/nix 2021-12-21 14:18:04 +01:00
Eelco Dolstra 3228a38e1c Merge branch 'why-depends-completion' of https://github.com/ncfavier/nix 2021-12-21 11:38:07 +01:00
regnat d90f9d4b99 Fix IFD with CA derivations
Rewrite the string taken by the IFD-like primops to contain the actual
output paths of the derivations rather than the placeholders

Fix #5805
2021-12-21 09:36:50 +01:00
regnat cbbd21ec07 Factor out the path realisation bit of IFD 2021-12-21 09:36:19 +01:00
pennae 09b245690a bulk-allocate Value instances in the evaluator
calling GC_malloc for each value is significantly more expensive than
allocating a bunch of values at once with GC_malloc_many. "a bunch" here
is a GC block size, ie 16KiB or less.

this gives a 1.5% performance boost when evaluating our nixos system.

tested with

nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'

 # on master

  Time (mean ± σ):      3.335 s ±  0.007 s    [User: 2.774 s, System: 0.293 s]
  Range (min … max):    3.315 s …  3.347 s    50 runs

 # with this change

  Time (mean ± σ):      3.288 s ±  0.006 s    [User: 2.728 s, System: 0.292 s]
  Range (min … max):    3.274 s …  3.307 s    50 runs
2021-12-20 23:01:28 +01:00
Nicolas Mattia 9dea5768ef Clean up toml parsing code 2021-12-20 21:39:37 +01:00
Ben Burdette f317019edd :d error 2021-12-20 12:32:21 -07:00
Naïm Favier 1827cf5a9b
Add shell completion for why-depends
Fixes #5806
2021-12-20 19:58:30 +01:00
Nicolas Mattia 5d06836b9f Use long in fromTOML 2021-12-17 23:15:26 +01:00
Nicolas Mattia 10a8b5d3ae Update primops/fromTOML.cc to use toml11 2021-12-17 22:13:04 +01:00
Nicolas Mattia c3f68b5db7 Replace cpptoml with toml11 2021-12-17 22:03:33 +01:00
Eelco Dolstra 6e6e998930
Merge pull request #5787 from edolstra/unshare-fs
Ignore EPERM when unsharing FS state
2021-12-17 12:00:09 +01:00
Théophane Hufschmitt 23ea1e46cc
Merge pull request #5047 from symphorien/fix-nix-channel
nix-channel: use nix-env -i --remove-all to upgrade
2021-12-17 10:05:48 +01:00
regnat ab902521b1 Allow paths in flake local settings
Fix #5505
2021-12-17 09:31:13 +01:00
Eelco Dolstra ec8f24ed3a Ignore EPERM when unsharing FS state
On Docker (but not podman), unshare(CLONE_FS) fails with EPERM. So
let's ignore it and hope nothing bad happens.

Attempted fix for #5777.
2021-12-16 22:02:50 +01:00
Noah Snelson c3f9344cae Fix typo in src/nix/shell.md
Changed `If not command` to `If no command`, also specified that the
default shell launched in `nix shell` can be specified with `$SHELL`.
2021-12-15 23:40:37 -08:00
Eelco Dolstra 18e4851752
Merge pull request #5769 from NixOS/ca/get-build-stats
Add a crude tracing mechansim for the build results
2021-12-14 10:53:51 +01:00
Théophane Hufschmitt 8868da45a8
Merge pull request #5623 from yorickvP/fix-5621
flakes: fix boolean and int nixConfig values
2021-12-14 10:35:37 +01:00
Théophane Hufschmitt 99d617bcde
Merge pull request #5771 from edolstra/single-file-flake-inputs
Re-allow inputs.x.url = "/path/to/file"
2021-12-14 09:17:24 +01:00
Théophane Hufschmitt 6234e1c811
Merge pull request #5702 from baloo/baloo/hide-non-reproducible-settings
reproducibility: hide non-reproducible settings from manual
2021-12-14 09:14:25 +01:00
Théophane Hufschmitt 3fc8042f77
Merge pull request #5366 from trofi/speedup-ca-query
ca-specific-schema.sql: add index on RealisationsRefs(referrer)
2021-12-14 09:08:17 +01:00
Eelco Dolstra 95bd5da341 queryMissing(): Use a bigger thread pool
This is a temporary hack until we make queryMissing() async.

Closes #5324.
2021-12-13 21:11:37 +01:00
Eelco Dolstra ade870764a
Merge pull request #5725 from ConnorBaker/master
Add support for .tar.gz shorthand (.tgz) to TarballInputScheme
2021-12-13 21:08:06 +01:00
Eelco Dolstra 32a62b0d25 Re-allow inputs.x.url = "/path/to/file" 2021-12-13 20:59:33 +01:00
Eelco Dolstra 46d86e06ba Simplify 2021-12-13 20:28:53 +01:00
Eelco Dolstra 9529de2eed Merge branch 'arm32-personality' of https://github.com/cleverca22/nix 2021-12-13 19:58:40 +01:00
Eelco Dolstra 6ae5f39ea0
Merge pull request #5770 from edolstra/gc-fixes
Fix macOS GC failures
2021-12-13 19:55:32 +01:00
Eelco Dolstra e2e5f3a78f Respect /etc/hosts
https://hydra.nixos.org/build/161439329
2021-12-13 19:37:30 +01:00
Eelco Dolstra cc6406cc59 Merge branch 'better-interpolation-error-location' of https://github.com/greedy/nix 2021-12-13 19:20:48 +01:00
Eelco Dolstra 2567b74c66 nix run: Tweak docs 2021-12-13 19:14:32 +01:00
Eelco Dolstra 775bdc0d9e Merge branch 'pname-main-program' of https://github.com/figsoda/nix 2021-12-13 19:09:36 +01:00
Eelco Dolstra d17d46cfc2 More debug info 2021-12-13 19:03:08 +01:00
Eelco Dolstra 05fcc681ac Retry on ECONNREFUSED
https://hydra.nixos.org/build/161439235
2021-12-13 19:00:19 +01:00
Eelco Dolstra 6d30f9e6fe Explicitly make GC roots client sockets blocking
On macOS / BSD, these sockets inherit the non-blocking flag of the
server soocket, which is not what we want.

https://github.com/dotnet/runtime/issues/25069
https://bugs.python.org/issue7995

https://hydra.nixos.org/build/161439304
2021-12-13 18:59:50 +01:00
regnat 2eec2f765a Add a crude tracing mechansim for the build results
Add a `_NIX_TRACE_BUILT_OUTPUTS` environment variable that can be set to
a filename in which the result of each build will be logged.

This is intentionally crude and undocumented as it’s only meant to be a
temporary thing to assess the usefulness of CA derivations.
Any other use would need a cleaner re-implementation first.
2021-12-13 17:02:14 +01:00
regnat 55dbb7f1cc More properly track the status of CA builds
Make the build of unresolved derivations return the same status as the
resolved one, except in the case of an `AlreadyValid` in which case it
will return `ResolvesToAlreadyValid` to mean that the outputs of the unresolved
derivation weren’t known, but the resolved one is.
2021-12-13 17:02:13 +01:00
Eelco Dolstra 84497119ad
Merge pull request #5761 from yorickvP/fix-5741
extract_archive: fix "Hard-link target '...'" error
2021-12-13 13:37:40 +01:00
Yorick 33e96820d5
EvalCommand::getEvalState: use gc tracable allocator for EvalState 2021-12-11 16:02:08 +01:00
Gavin Ray 3542d4fe16
Incorporate suggestions from @edolstra 2021-12-10 19:02:22 -05:00
Yorick 0e0de90b35
extract_archive: fix "Hard-link target '...'" error
Fixes #5741
2021-12-10 17:03:51 +01:00
Yorick be1055f2cc
extract_archive: use copy_pathname instead of set_pathname.
Libarchive documentation mentions that archive_entry_set_pathname
expects us to keep the passed string alive, which we don't.
2021-12-10 17:00:32 +01:00
Eelco Dolstra 20b1290103 Ignore system.nfs4_acl
Closes #1584.
2021-12-10 15:12:55 +01:00
Eelco Dolstra f9437b4f18 Merge branch 'master' of https://github.com/MagicRB/nix 2021-12-10 14:49:25 +01:00
regnat 55e3a1108c Properly take the last assignment in the REPL
When a variable is assigned in the REPL, make sure to remove any possible reference to the old one so that we correctly pick the new one afterwards

Fix #5706
2021-12-09 16:35:27 +01:00
Eelco Dolstra bc57b3854e
Merge pull request #5695 from obsidiansystems/tidy-logging
Tidy up the logging
2021-12-09 15:31:08 +01:00
Gavin Ray 971382cab0
Better diagnostics if no valid signature found
I downloaded Nix tonight, and immediately broke it by accidentally removing the default binary caching.
After figuring this out, I also failed to fix it properly, due to using the wrong key for Nix's default binary cache

If the diagnostic message would have been clearer about what/where a "signature" for a "substituter" is + comes from, it probably would have saved me a few hours.
Maybe we can save other noobs the same pain?
2021-12-08 19:55:34 -05:00
Eelco Dolstra 92ff5b4254 Tweak warning 2021-12-07 22:22:02 +01:00
Eelco Dolstra 21206caf13 Merge remote-tracking branch 'origin/5712-ca-drvs-with-old-daemon' 2021-12-07 22:06:22 +01:00
Eelco Dolstra 8b5088b62f Merge branch 'balsoft/nix-repl-log' of https://github.com/tweag/nix 2021-12-07 21:33:32 +01:00
Jan Tojnar ae21aab456 Update manual links
Fixes: https://github.com/NixOS/nixos-homepage/issues/762
2021-12-06 16:42:57 +01:00
Alexander Bantyev 33926ed1e7
nix repl: add :log
Add a :log command that shows logs for a derivation.

Closes https://github.com/NixOS/nix/issues/3504

Co-authored-by: Taeer Bar-Yam <taeer@bar-yam.me>
2021-12-06 16:52:50 +03:00
Alexander Bantyev 0a2fa2d684
RunPager: restore stdout upon pager exit
Before this change, stdout was closed after the pager exits. This is
fine for non-interactive commands where we want to exit right after
the pager exits anyways, but for interactive things (e.g. nix repl)
this breaks the output after we quit the pager.

Keep the initial stdout fd as part of RunPager, and restore it in
RunPager::~RunPager using dup2.
2021-12-06 16:51:55 +03:00
regnat fa58bff1ab Fallback when the daemon is too old for CA derivations
Fix #5712
2021-12-06 11:45:18 +01:00
Connor Baker 52f5fa948a Add support for .tar.gz shorthand (.tgz) to TarballInputScheme 2021-12-03 22:16:02 -05:00
Tom Bereknyei b6cc0a704d flakes: search up to git or filesystem boundary
While parsing a flakeref, upon not finding a flake.nix, search upwards
until git or filesystem boundary.
2021-12-03 10:55:30 -05:00
Silvan Mosberger 90700736c7 Introduce builtins.groupBy primop
This function is very useful in nixpkgs, but its implementation in Nix
itself is rather slow due to it requiring a lot of attribute set and
list appends.
2021-12-02 21:54:51 +01:00
Eelco Dolstra 2ff71b0213
Merge pull request #5713 from tweag/balsoft/another-follows-bugfix
Flakes: computeLocks: pass correct LockParent when reusing oldLock
2021-12-02 13:17:50 +01:00
Eelco Dolstra 77b3e4b4ee
Merge pull request #5708 from trofi/fix-build-on-gcc-11
src/libutil/util.hh: fix build on gcc-11
2021-12-02 13:16:15 +01:00
Alexander Bantyev 8cbf862e6f
Flakes: computeLocks: pass correct LockParent when reusing oldLock
Previously, when we were attempting to reuse the old lockfile
information in the computeLocks function, we have passed the parent of
the current input to the next computeLocks call. This was incorrect,
since the follows are resolved relative to the parent. This caused
issues when we tried to reuse oldLock but couldn't for some
reason (read: mustRefetch is true), in that case the follows were
resolved incorrectly.

Fix this by passing the correct parent, and adding some tests to
prevent this particular regression from happening again.

Closes https://github.com/NixOS/nix/issues/5697
2021-12-02 12:02:34 +03:00
David Arnold 7489811736
Fix parent path check boundary
- Previous to this commit the boundary was exclusive of the
  top level flake.
- This is wrong since the top level flake is still a valid
  relative reference.
- Now, the check boundary is inclusive of the top level flake.

Signed-off-by: Timothy DeHerrera <tim.deh@pm.me>
2021-12-01 17:01:48 -07:00
Sergei Trofimovich c32a5f4d38 src/libutil/util.hh: fix build on gcc-11
Due to missing <atomic> declaration the build fails as:

    src/libutil/util.hh:350:24: error: no match for 'operator||' (operand types are 'std::atomic<bool>' and 'bool')
      350 |     if (_isInterrupted || (interruptCheck && interruptCheck()))
          |         ~~~~~~~~~~~~~~ ^~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          |         |                                 |
          |         std::atomic<bool>                 bool
2021-12-01 22:08:05 +00:00
Eelco Dolstra 782837d934 Merge branch 'nix-env-install-suggestions' of https://github.com/tweag/nix 2021-12-01 21:47:28 +01:00
Eelco Dolstra fb662e0acf
Merge pull request #5696 from obsidiansystems/fix-5299
Fix #5299 and remove uncesssary unbounded buffer
2021-12-01 21:32:26 +01:00
Arthur Gautier 21520297da reproducibility: hide non-reproducible settings from manual
Because the manual is generated from default values which are themselves
generated from various sources (cpuid, bios settings (kvm), number of
cores). This commit hides non-reproducible settings from the manual
output.
2021-12-01 17:25:58 +01:00
Théophane Hufschmitt f3f32f0c30
Merge pull request #5599 from fzakaria/faridzakaria/fix-progress-bar-leak
Fix heap use after free in progress-bar.cc
2021-12-01 16:28:38 +01:00
Ben Burdette c151a9b426 fix linking 2021-11-30 15:14:23 -07:00
John Ericson 2f5c913d4a Fix #5299
No matter what, we need to resize the buffer to not have any scratch
space after we do the `read`. In the end of file case, `got` will be 0
from it's initial value.

Before, we forgot to resize in the EOF case with the break. Yes, we know
we didn't recieve any data in that case, but we still have the scatch
space to undo.

Co-Authored-By: Will Fancher <Will.Fancher@Obsidian.Systems>
2021-11-30 22:11:46 +00:00
John Ericson 95157b4e66 Push wopAddToStore old style stream adapters into smaller scopes
This doesn't fix the bug, but makes the code less difficult to read.
Also improve the comments, now that it is clear what part is needed in
each code path.
2021-11-30 22:10:58 +00:00
John Ericson 70a717f7a8 Nix daemon stream old wopAddToStore
No more buffering in string.
2021-11-30 21:55:53 +00:00
Ben Burdette e82aec4efc fix merge issues 2021-11-30 14:15:02 -07:00
John Ericson 1f15441103 Tidy up the logging
Use the macros more, so we properly skip work when the log level
excludes. Also log the daemon operation number on the daemon side.
2021-11-30 20:23:13 +00:00
Yorick bd628cf3da
flakes: fix boolean and int nixConfig values
Some type confusion was causing ints to be pointers, and bools
to be ints. Fixes #5621
2021-11-29 15:53:22 +01:00
Eelco Dolstra 6e0cbc666b createTempFile(): Mark file as CLOEEXEC
Fixes #5674.
2021-11-29 11:20:50 +01:00
Eelco Dolstra e1420c66a4
Merge pull request #5680 from andir/libexpr-dont-move-primop-args
Don't move the arguments of the primOp
2021-11-29 11:11:54 +01:00
Eelco Dolstra 3d36f48acb
Merge pull request #5667 from NixOS/5661-dont-complete-flakes-without-xp-feature
Don’t try to complete flakes is the feature isn’t enabled
2021-11-29 10:51:41 +01:00
Andreas Rammhold 90d8178009
Don't move the arguments of the primOp
Moving arguments of the primOp into the registration structure makes it
impossible to initialize a second EvalState with the correct primOp
registration. It will end up registering all those "RegisterPrimOp"'s
with an arity of zero on all but the 2nd instance of the EvalState.

Not moving the memory will add a tiny bit of memory overhead during the
eval since we need a copy of all the argument lists of all the primOp's.
The overhead shouldn't be too bad as it is static (based on the amonut
of registered operations) and only occurs once during the interpreter
startup.
2021-11-28 02:06:47 +01:00
Alex Kladov 1d5a881da5
Fix typo 2021-11-27 23:29:48 +03:00
regnat 3a0277305a Don’t try to complete flakes is the feature isn’t enabled
Fix #5661
2021-11-26 16:56:51 +01:00
Eelco Dolstra 3ef66cd23a nix repl: Reset the terminal on exceptional exits 2021-11-26 13:10:28 +01:00
Eelco Dolstra b653fb9ccf
Merge pull request #5662 from NixOS/5259-nix-shell-with-ca-bash
Make `nix-shell` work when nixpkgs is content-addressed
2021-11-26 12:47:30 +01:00
Alex Shabalin 6d166d19a6 Catch flake-related exception type in REPL
Closes https://github.com/NixOS/nix/issues/5656
2021-11-26 11:03:48 +01:00
regnat 05081bedc1 Make nix-shell work when nixpkgs is content-addressed
Fix #5259
2021-11-26 09:56:48 +01:00
Eelco Dolstra 5fcf7f04a9
Merge pull request #5384 from baloo/baloo/dns-timeout
preloadNSS / dns timeout
2021-11-25 17:37:53 +01:00
Eelco Dolstra d5d0621250
Merge pull request #5603 from obsidiansystems/recursive-nix-system-feature
"recursive-nix" system feature only with experimental feature
2021-11-25 17:20:58 +01:00
Eelco Dolstra 6f46434f32
Merge pull request #5648 from edolstra/list-iter
Support range-based for loop over list values
2021-11-25 17:13:25 +01:00
Ben Burdette 64c4ba8f66 Merge branch 'master' into debug-merge 2021-11-25 08:53:59 -07:00
Eelco Dolstra b6c8e57056 Support range-based for loop over list values 2021-11-25 16:31:39 +01:00
Ben Burdette 69e26c5c4b more cleanup 2021-11-25 08:23:07 -07:00
regnat c47027f3a1 Fix the error when accessing a forbidden path in pure eval
If we’re in pure eval mode, then tell that in the error message rather
than (wrongly) speaking about restricted mode.

Fix https://github.com/NixOS/nix/issues/5611
2021-11-25 14:48:01 +01:00
Eelco Dolstra 1f7584d24c
Merge pull request #5644 from tweag/fix-interrupted-data-race
Fix a minor data race with _isInterrupted
2021-11-25 13:55:03 +01:00
Las Safin 8388d2c7c6
Make recursive-nix work even when not privileged
Before this, `setns` would fail when switching to the mount namespace,
since we did not have the privileges to do so when not root.

Closes #5360
2021-11-25 11:15:11 +00:00
Eelco Dolstra d58f149140
Merge pull request #5631 from Infinisil/list-compare
Make lists be comparable
2021-11-24 15:48:05 +01:00
Alex Shabalin 2970ca18bf Fix a minor data race with _isInterrupted 2021-11-24 14:50:08 +01:00
Eelco Dolstra 884674a8e2 nix flake check: Fix markdown 2021-11-24 13:53:09 +01:00
Silvan Mosberger 09471d2680 Make lists be comparable
Makes lists comparable using lexicographic comparison.

Increments builtins.langVersion in order for this change to be
detectable
2021-11-24 13:40:46 +01:00
John Ericson 286eb81143 "recursive-nix" system feature only with experimental feature 2021-11-23 20:19:04 +00:00
Farid Zakaria d41af23a6c Fix heap use after free in progress-bar.cc
Fix some heap-use-after-free in progress-bar.cc

These are somewhat tricky failures here due to temporary variable
creation and string_view
2021-11-23 08:34:55 -08:00
Alex Shabalin b26cb0c9ac Fix use after free in content-address.cc
Inspired by https://github.com/NixOS/nix/pull/5599
2021-11-23 16:15:34 +01:00
Eelco Dolstra dbfcaa607a
Merge pull request #5628 from matthewbauer/rosetta2-new-path
Set new rosetta 2 path in sandbox
2021-11-23 10:46:50 +01:00
Matthew Bauer 08b1ac3e38 Set new rosetta 2 path in sandbox
see:

https://github.com/NixOS/nix/pull/5388

and

https://github.com/NixOS/nix/pull/5251
2021-11-22 14:42:31 -06:00
Eelco Dolstra b367f1061c
Merge pull request #5624 from rofrol/typo-single-quote
Typo: change to normal single quote
2021-11-22 21:33:21 +01:00
Eelco Dolstra 720ed47678
Merge pull request #5613 from tomberek/allow_realpath
add real path to allowedPaths
2021-11-22 15:10:47 +01:00
Eelco Dolstra dcaa8dfd10
Merge pull request #5620 from alyssais/wayland
nix-shell --pure: let variables for Wayland through
2021-11-22 15:09:52 +01:00
Alex Shabalin db2e4489a5 Unify #if linux 2021-11-22 13:57:56 +01:00
Roman Frołow 0768c08d99 Typo: change to normal singlequote 2021-11-22 13:37:38 +01:00
Alyssa Ross 664ee49e0d
nix-shell --pure: let variables for Wayland through
We let DISPLAY (X11) through, so we should let the Wayland equivalents
through as well.  Similarly, we let HOME through, so it should be okay
to allow XDG_RUNTIME_DIR (which is needed for connecting to Wayland
with WAYLAND_DISPLAY) through as well.  Otherwise graphical
applications will either fall back to X11 (if they support it), or
just not work (if they don't).
2021-11-21 16:14:19 +00:00
Tom Bereknyei 4318ba2ec5 add real path to allowedPaths 2021-11-20 00:25:36 -05:00
Alex Shabalin 86b7962807 Use warn to print a warning 2021-11-19 16:29:55 +01:00
Alex Shabalin a18d9269a5 Fix build warnings on MacOS 2021-11-19 15:22:31 +01:00
Alex Shabalin 4b28798bfc Offer suggestions for nix-env -i
Closes https://github.com/NixOS/nix/issues/972
2021-11-18 15:34:57 +01:00
Eelco Dolstra 0961c1068a Merge branch 'accept_conf' of https://github.com/tomberek/nix 2021-11-18 12:56:09 +01:00
Eelco Dolstra 0adced4b9e
Merge pull request #5580 from ksonj/fix/non-standard-ssh
Fix detection of scp-style URIs to support non-standard SSH ports
2021-11-18 12:30:29 +01:00
Eelco Dolstra 262a3c7ce3 Simplify 2021-11-18 12:12:31 +01:00
Eelco Dolstra acf381b061 Merge branch 'faridzakaria/fix-stack-buffer-overflow' of https://github.com/fzakaria/nix 2021-11-18 12:10:52 +01:00
Eelco Dolstra 205655e98b
Merge pull request #5586 from tweag/fix-stdout-for-editors-in-repl
Fix :e in repl
2021-11-18 11:46:44 +01:00
Eelco Dolstra 1d0c6a4b99
Merge pull request #5583 from jtojnar/patch-1
Fix XDG_CONFIG_DIRS fallback
2021-11-18 11:35:18 +01:00
Alex Shabalin 9653858ce6 Fix :e in repl
Closes https://github.com/NixOS/nix/issues/5487

Co-authored-by: Alexander Bantyev balsoft@balsoft.ru
2021-11-18 09:03:33 +01:00
Farid Zakaria bc14465e08 Fix stack buffer overflow
Fix a stack buffer overflow found by running MemorySanitizer.
2021-11-18 04:05:25 +00:00
Jan Tojnar e96faadcd6 Fix XDG_CONFIG_DIRS fallback
According to XDG Base Directory Specification,
it should fall back to /etc/xdg when the env var is not present.
2021-11-17 14:31:15 +01:00
Kalle Jepsen 46d2a5a10b Simplify fix by disallowing / in front of @ to match scp style 2021-11-17 13:49:10 +01:00
Eelco Dolstra d03e89e5d1 Parse '(f x) y' the same as 'f x y'
(cherry picked from commit 5253cb4b68ad248f37b27849c0ebf3614e4f2777)
2021-11-17 09:53:57 +01:00
Kalle Jepsen 6d46b5b609 Fix detection of scp-style URIs to support non-standard SSH ports for git 2021-11-17 08:41:26 +01:00
Eelco Dolstra d7bae52b9d Call functors with both arguments at once
This is not really useful on its own, but it does recover the
'infinite recursion' error message for '{ __functor = x: x; } 1', and
is more efficient in conjunction with #3718.

Fixes #5515.
2021-11-16 22:34:17 +01:00
Eelco Dolstra e41cf8511f Don't hang when calling an attrset
Fixes #5565.
2021-11-16 17:44:19 +01:00
Eelco Dolstra 8c93a481af Ignore errors unsharing/restoring the mount namespace
This prevents Nix from barfing when run in a container where it
doesn't have the appropriate privileges.
2021-11-16 14:48:44 +01:00
Eelco Dolstra ec608e3def
Merge pull request #5452 from Kha/git-extend-lock
git: extend cache dir lock over all mutating operations
2021-11-16 12:55:12 +01:00
Eelco Dolstra 6463eaca14
Merge pull request #5472 from NixOS/async-realisation-substitution
async realisation substitution
2021-11-16 12:54:20 +01:00
Eelco Dolstra 6d0aa8d175
Merge pull request #5559 from Kloenk/libfetchers-gitlab-headers
libfetchers: set free gitlab headers
2021-11-16 12:47:49 +01:00
Eelco Dolstra 9fe0343bfd Merge branch 'master' of github.com:NixOS/nix 2021-11-15 18:45:20 +01:00
Eelco Dolstra 671817a858 Simplify lockFlake() a bit 2021-11-15 18:44:27 +01:00
Eelco Dolstra d1d223838b Merge branch 'balsoft/fix-subinputs-not-getting-updated' of https://github.com/tweag/nix 2021-11-15 18:37:13 +01:00
Domen Kožar 164179983e
Merge pull request #5428 from kreisys/add-pos-to-json-type-error
toJSON: report error position for fancier output
2021-11-15 07:57:46 -06:00
kloenk 79d07d0980 libfetchers: set free gitlab headers 2021-11-14 12:23:46 +01:00
Kevin Amado d0e9e18489
toXML: display errors position
- This change applies to builtins.toXML and inner workings
- Proof of concept:
  ```nix
  let e = builtins.toXML e; in e
  ```
- Before:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered
  ```
- After:
  ```
  $ nix-instantiate --eval poc.nix
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/poc.nix:1:9:

            1| let e = builtins.toXML e; in e
             |
  ```
2021-11-13 20:33:34 -05:00
Tom Bereknyei 30496af598 Adds an accept-flake-config flag 2021-11-12 09:50:07 -05:00
Eelco Dolstra bceda30498 Typo 2021-11-12 13:41:15 +01:00
Eelco Dolstra 44aed58538
Merge pull request #5540 from samueldr/fix/key-subcommand
nix key: Fix error message and don't require flakes
2021-11-12 10:38:00 +01:00
Samuel Dionne-Riel 30e5c5c55f nix registry: Mark experimental
This is part of the flakes feature. Mark it as such.
2021-11-12 03:18:31 -05:00
Samuel Dionne-Riel c1dea92dd6 nix key: Fix error message and don't require flakes 2021-11-11 21:31:26 -05:00
Robert Hensing 3884f7a69a Install nlohmann_json headers
These headers are included by the libexpr, libfetchers, libstore
and libutil headers.
Considering that these are vendored sources, Nix should expose them,
as it is not a good idea for reverse dependencies to rely on a
potentially different source that can go out of sync.
2021-11-11 11:05:44 +01:00
Alexander Bantyev 07bffe7998
Flakes: refetch the input when a follows disappears
When an input follows disappears, we can't just reuse the old lock
file entries since we may be missing some required ones. Refetch the
input when this happens.

Closes https://github.com/NixOS/nix/issues/5289
2021-11-11 12:05:18 +03:00
Eelco Dolstra a218cfd6c7
Merge pull request #5528 from matthewbauer/recognize-nixosModule
Recognize singular "nixosModule" in nix flake show
2021-11-10 12:17:11 +01:00
Sergei Trofimovich edfc5b2f12 ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.

RealisationsRefs(referrer)
--------------------------

Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:

    $ nix edit -f . bash # add unused environment variable, like FOO="1"
    # populate RealisationsRefs, build fresh system
    $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
    $ nix edit -f . bash # add unused environment variable, like FOO="2"
    $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'

In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).

Before the change:

    $ time nix build -f nixos system ... # step 4 above
    real    34m3,613s
    user    0m5,232s
    sys     0m0,758s

Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.

After the change:

    $ time nix build -f nixos system ... # step 4 above
    real    4m50,061s
    user    0m5,038s
    sys     0m0,677s

Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.

Realisations(outputPath)
------------------------

After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.

The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.

Before the change:
    $ time nix-collect-garbage -d --max-freed 100G
    <interrupted before finish, took too long>
    real    23m32.411s
    user    17m49.679s
    sys     4m50.609s

Most of time was spent in re-scanning Realisations table on each path deletion.

After the change:
    $ time nix-collect-garbage -d --max-freed 100G

    real    8m43.226s
    user    6m16.317s
    sys     1m40.188s

Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-11-10 08:32:05 +00:00
Ben Burdette 7e2a3db4eb cleanup 2021-11-09 13:14:49 -07:00
Ben Burdette 885f819922 remove dead code 2021-11-09 11:20:14 -07:00
Matthew Bauer ceeecf2f9e Recognize singular "nixosModule" in nix flake show
This makes nixosModule appears as a "NixOS Module" like nixosModules
does.
2021-11-09 11:14:15 -06:00
Eelco Dolstra 67179472df
Merge pull request #5494 from tweag/balsoft/allow-references-in-addPath
Allow references in addPath
2021-11-09 15:57:39 +01:00
Alexander Bantyev 0b005bc9d6
addToStore, addToStoreFromDump: refactor: pass refs by const reference
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
2021-11-09 12:24:49 +03:00
Eelco Dolstra ff2af4d64e Unshare mount namespace in main()
Doing it as a side-effect of calling LocalStore::makeStoreWritable()
is very ugly.

Also, make sure that stopping the progress bar joins the update
thread, otherwise that thread should be unshared as well.
2021-11-08 22:00:45 +01:00
Eelco Dolstra 7a71621b7c Merge branch 'fix-writable-shell' of https://github.com/yorickvP/nix 2021-11-08 21:12:51 +01:00
Eelco Dolstra 6c2af1f201
Merge pull request #5434 from timothyklim/git-url-submodules
Support building flakes from a Git repo url with submodules
2021-11-08 14:13:42 +01:00
Eelco Dolstra 736d6ab721
Merge pull request #5504 from NixOS/flake-options-and-daemon
Make the flake options work when using the daemon
2021-11-08 13:54:55 +01:00
Andreas Rammhold 8e7359db64
Remove unused "<let-body>" symbol
The requirement for the symbol has been removed since at least 7d47498.
2021-11-07 18:26:43 +01:00
Alexander Bantyev 9d4dcff37a
addPath: allow paths with references
Since 4806f2f6b0, we can't have paths with
references passed to builtins.{path,filterSource}. This prevents many cases
of those functions called on IFD outputs from working. Resolve this by
passing the references found in the original path to the added path.
2021-11-05 22:41:30 +03:00
Théophane Hufschmitt 3d6ee223d6
Merge pull request #5476 from NixOS/ca/fix-build-in-nix-repl
Fix `nix repl`’s building of CA derivations
2021-11-05 17:35:18 +01:00
regnat 1f3c3a3785 Make the flake options work when using the daemon
When setting flake-local options (with the `nixConfig` field), forward
these options to the daemon in case we’re using one.

This is necessary in particular for options like `binary-caches` or
`post-build-hook` to make sense.

Fix <343239fc8a (r44356843)>
2021-11-05 16:19:16 +01:00
regnat 1a4c9ba50b Fix nix repl’s building of CA derivations
When running a `:b` command in the repl, after building the derivations
query the store for its outputs rather than just assuming that they are
known in the derivation itself (which isn’t true for CA derivations)

Fix #5328
2021-11-05 11:04:14 +01:00
Eelco Dolstra 40925337a9 Remove maxPrimOpArity 2021-11-04 15:04:07 +01:00
Eelco Dolstra acd6bddec7 Fix derivation primop 2021-11-04 15:04:00 +01:00
Eelco Dolstra cbfbf71e08 Use callFunction() with an array for some calls with arity > 1 2021-11-04 15:03:57 +01:00
Eelco Dolstra bcf4780006 Add level / displacement types 2021-11-04 15:03:45 +01:00
Eelco Dolstra 81e7c40264 Optimize primop calls
We now parse function applications as a vector of arguments rather
than as a chain of binary applications, e.g. 'substring 1 2 "foo"' is
parsed as

  ExprCall { .fun = <substring>, .args = [ <1>, <2>, <"foo"> ] }

rather than

  ExprApp (ExprApp (ExprApp <substring> <1>) <2>) <"foo">

This allows primops to be called immediately (if enough arguments are
supplied) without having to allocate intermediate tPrimOpApp values.

On

  $ nix-instantiate --dry-run '<nixpkgs/nixos/release-combined.nix>' -A nixos.tests.simple.x86_64-linux

this gives a substantial performance improvement:

  user CPU time:      median =      0.9209  mean =      0.9218  stddev =      0.0073  min =      0.9086  max =      0.9340  [rejected, p=0.00000, Δ=-0.21433±0.00677]
  elapsed time:       median =      1.0585  mean =      1.0584  stddev =      0.0024  min =      1.0523  max =      1.0623  [rejected, p=0.00000, Δ=-0.20594±0.00236]

because it reduces the number of tPrimOpApp allocations from 551990 to
42534 (i.e. only small minority of primop calls are partially
applied) which in turn reduces time spent in the garbage collector.
2021-11-04 15:03:40 +01:00
Eelco Dolstra ab35cbd675 StaticEnv: Use std::vector instead of std::map 2021-11-04 15:03:34 +01:00
Eelco Dolstra c4bd6a15c2 Add helper function to check whether a function arg is 'X' or '_X'
Also allow '_'.
2021-11-04 14:52:35 +01:00
Eelco Dolstra 1e7c796e66
Merge pull request #5475 from doronbehar/SQLiteWAL-vfs
libstore: Use unix-dotfile vfs if useSQLiteWAL is false
2021-11-03 21:20:27 +01:00
Christopher League 3f070cc417 In checkOverlay, accept underscored names for final/prev args.
Resolves #4416.
2021-11-03 09:25:27 -04:00
Doron Behar 14fcf17277 libstore: Use unix-dotfile vfs if useSQLiteWAL is false 2021-11-03 14:19:11 +02:00
regnat f4c869977c Make the DrvOutputSubstitutionGoal more async 2021-11-03 06:51:34 +01:00
regnat fbc70034b3 Make the realisation fetching from binary caches async
That way we can fetch several realisations from the same cache in
parallel
2021-11-03 06:51:34 +01:00
regnat 96670ed216 Expose an async interface for queryRealisation
Doesn’t change much so far because everything is still using it
synchronously, but should allow the binary cache to fetch stuff in
parallel
2021-11-03 06:51:34 +01:00
regnat 5b2aa61f1b Don’t require ca-derivations when __contentAddressed = false
If we explicitely opt-out of it, there’s no need to require the
experimental feature
2021-11-03 06:51:32 +01:00
Sebastian Ullrich b459a3e856 git: extend cache dir lock over all mutating operations 2021-11-01 09:14:56 +01:00
Eelco Dolstra e6795c4350 Style 2021-10-29 14:45:13 +02:00
Eelco Dolstra 6e30d9b69f Merge branch 'master' of https://github.com/alekswn/nix 2021-10-29 14:42:26 +02:00
Eelco Dolstra 33d04e8a8d Use nix::connect() to connect to the garbage collector 2021-10-28 22:51:11 +02:00
Eelco Dolstra 22c35ea5b8 Remove unused variable 2021-10-28 22:51:11 +02:00
Eelco Dolstra a7d4f3411e Merge remote-tracking branch 'origin/master' into non-blocking-gc 2021-10-28 14:56:55 +02:00
Shay Bergmann a50c027ece
toJSON: improve pos accuracy, add trace 2021-10-27 19:48:48 +00:00
Shay Bergmann 465a167c43
nix-instantiate: pass pos in the --eval --json code path 2021-10-27 19:01:32 +00:00
figsoda a2473823d7 run: use pname as a fallback for main program 2021-10-27 14:49:24 -04:00
Eelco Dolstra 5a160171d0 Remove redundant 'warning:' 2021-10-27 18:14:12 +02:00
Eelco Dolstra 1254e8753c build-remote: Implicitly add the 'builtin' system type to all machines
This makes 'nix-env -i --max-jobs 0' work with remote builders.
2021-10-27 14:25:13 +02:00
Eelco Dolstra f2280749b1 If max-jobs == 0, do preferLocalBuild on remote builders 2021-10-27 14:21:31 +02:00
Eelco Dolstra 6e684d1b87 daemon: Accept 'repeat' setting from untrusted users
Fixes #5352.
2021-10-27 13:09:13 +02:00
Eelco Dolstra 13a7a24ba5 Style 2021-10-27 13:02:37 +02:00
Shay Bergmann 769de259f0
toJSON: pass pos in case of a list as well 2021-10-26 14:43:15 +00:00
Timothy 8919b81dad Support building flakes from a Git repo url with submodules query parameter 2021-10-26 20:02:37 +07:00
Eelco Dolstra 0d9e050ba7 parseExperimentalFeature(): Initialize atomically 2021-10-26 14:29:48 +02:00
Eelco Dolstra 9ce84c64c5 Tweak fetchTree docs 2021-10-26 14:21:24 +02:00
regnat af99941279 Make experimental-features a proper type
Rather than having them plain strings scattered through the whole
codebase, create an enum containing all the known experimental features.

This means that
- Nix can now `warn` when an unkwown experimental feature is passed
  (making it much nicer to spot typos and spot deprecated features)
- It’s now easy to remove a feature altogether (once the feature isn’t
  experimental anymore or is dropped) by just removing the field for the
  enum and letting the compiler point us to all the now invalid usages
  of it.
2021-10-26 07:02:31 +02:00
Shay Bergmann ba81e871b2
toJSON: report error position for fancier output
Given flake:

```nix
{ description = "nix json error provenance";
  inputs = {};
  outputs = { self }: {
    jsonFunction = _: "function";
    json = builtins.toJSON (_: "function");
  };
}

```
- Before:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON
```

- After:

```console
❯ nix eval --json .#jsonFunction
error: cannot convert a function to JSON

       at /nix/store/b7imf1c2j4jnkg3ys7fsfbj02s5j0i4f-source/testflake/flake.nix:4:5:

            3|   outputs = { self }: {
            4|     jsonFunction = _: "function";
             |     ^
            5|     json = builtins.toJSON (_: "function");
```
2021-10-25 21:17:52 +00:00
Alexander Bantyev b9234142f5
addToStore, addToStoreFromDump: add references argument
Allow to pass a set of references to be added as info to the added paths.
2021-10-23 21:30:51 +03:00
Ben Burdette fb8377547b more code cleanup 2021-10-22 14:49:58 -06:00
Ben Burdette 71da988d47 more debug removal 2021-10-22 14:34:50 -06:00
Ben Burdette e54f17eb46 remove more debug code 2021-10-22 14:27:04 -06:00
Ben Burdette cbc2f0fe31 remove dead code 2021-10-22 14:02:47 -06:00
Ben Burdette 383ab600ee show expr on error 2021-10-22 13:41:04 -06:00
Kevin Amado 823dce945a
fetch: nicer infinite recursion errors
- This change applies to builtins.fetchurl and builtins.fetchTarball
- PoC: `let x = builtins.fetchurl x; in x`
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchurl x; in x
             |         ^
  ```

Mentions: #3505
2021-10-17 12:54:53 -05:00
Alexey Novikov e989c83b44 Add error reporting to machine spec paser
Currently machine specification (`/etc/nix/machine`) parser fails
with a vague exception if the file had incorrect format.
This commit adds verbose exceptions and unit-tests for the parser.
2021-10-17 12:45:56 +04:00
Alexey Novikov 64a3b045c1 Fix error detection in 'base64Decode()'
Fixed a bug in initialization of 'base64DecodeChars' variable.
Currently decoder do not fail on invalid Base64 strings.
Added test-case to verify the fix.

Also have made 'base64DecodeChars' to be computed at compile time.
And added a test case to encode/decode string with non-printable charactes.
2021-10-17 12:45:26 +04:00
Arthur Gautier fa4abe46e2 preloadNSS: document the preload mechanism
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-16 02:55:25 +00:00
Kevin Amado e5a27a3b4e
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree {
    type = "git";
    inherit x;
  };
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:3:10:

            2|   type = "git";
            3|   inherit x;
             |          ^
            4| };
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Kevin Amado 18e3d63341
fetchTree: add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let x = builtins.fetchTree x;
  in x
  ```
- Before:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  $ nix-instantiate --extra-experimental-features flakes --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/nix/test.nix:1:9:

            1| let x = builtins.fetchTree x;
             |         ^
            2| in x
  ```

Mentions: #3505
2021-10-15 19:25:19 -05:00
Eelco Dolstra 10f9a8e77d Add a test for the non-blocking GC 2021-10-15 16:52:37 +02:00
Eelco Dolstra ac54c6faa6 Fix main GC thread exiting 2021-10-15 16:36:48 +02:00
Yorick fcb8af550f
Restore parent mount namespace in restoreProcessContext
This ensures any started processes can't write to /nix/store (except
during builds). This partially reverts 01d07b1e, which happened because
of #2646.

The problem was only happening after nix downloads anything, causing
me to suspect the download thread. The problem turns out to be:
"A  process  can't  join a new mount namespace if it is sharing
filesystem-related attributes with another process", in this case this
process is the curl thread.

Ideally, we might kill it before spawning the shell process, but it's
inside a static variable in the getFileTransfer() function. So
instead, stop it from sharing FS state using unshare(). A strategy
such as the one from #5057 (single-threaded chroot helper binary) is
also very much on the table.

Fixes #4337.
2021-10-15 16:25:49 +02:00
Eelco Dolstra e0936ae38f Fix crash when a GC client disconnects
The client thread can't just delete its own thread object from
connections, it has to detach it.
2021-10-15 16:12:21 +02:00
Eelco Dolstra 304180d0de Memoize queryReferrers() 2021-10-15 12:20:23 +02:00
Arthur Gautier c345a4a1e8 fixup macos build
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 07:34:01 +00:00
Eelco Dolstra c0951299b3
Merge pull request #5383 from kvtb/patch-7
fix build with gcc11
2021-10-15 09:03:12 +02:00
Eelco Dolstra c574ab3907
Merge pull request #5388 from yvt/fix-oahd-path
Add another path where a Rosetta 2 daemon plist file is possibly located
2021-10-15 09:02:16 +02:00
Kevin Amado 1bdeef8395
add pos to EvalState::forceValue
- This way we improve error messages
  on infinite recursion
- Demo:
  ```nix
  let
    x = builtins.fetchMercurial x;
  in
  x
  ```
- Before:
  ```bash
  $ nix-instantiate --show-trace --strict
  error: infinite recursion encountered
  ```
- After:
  ```bash
  nix-instantiate --show-trace --strict
  error: infinite recursion encountered

       at /data/github/kamadorueda/test/default.nix:2:7:

            1| let
            2|   x = builtins.fetchMercurial x;
             |       ^
            3| in
  ```

Mentions: #3505
2021-10-14 23:23:05 -05:00
Arthur Gautier ca8989daf3 preloadNSS: warn if unable to open nss backend
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 02:59:00 +00:00
Arthur Gautier 85717eff15 preloadNSS: detect glibc
Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 02:59:00 +00:00
Arthur Gautier d1da45855c preloadNSS: Drop the dns query workaround
We can actually just load nss ourselves and call in nss to configure it
and we don't need to run a dummy query entirely to have nss load nss_dns
as a side-effect.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-10-15 02:58:18 +00:00
yvt a9d9e55551 Add another path where a Rosetta 2 configuration file is possibly located 2021-10-15 09:48:15 +09:00
Eelco Dolstra 17e6ebcc90 Speed up GC by marking entire closures as live 2021-10-14 14:13:57 +02:00
Eelco Dolstra 0154fa30cf Remove GCState 2021-10-14 13:52:49 +02:00
Eelco Dolstra 0317ffdad3 Move deleteFromStore() 2021-10-14 13:34:48 +02:00
Eelco Dolstra 0be8cc1466 pathInfoCache: Use the entire base name as the cache key
This fixes a bug in the garbage collector where if a path
/nix/store/abcd-foo is valid, but we do a
isValidPath("/nix/store/abcd-foo.lock") first, then a negative entry
for /nix/store/abcd is added to pathInfoCache, so /nix/store/abcd-foo
is subsequently considered invalid and deleted.
2021-10-14 13:28:22 +02:00
Eelco Dolstra eab934cb2a Make the canReachRoots() traversal non-recursive 2021-10-14 12:34:32 +02:00
Eelco Dolstra 09b14ea97a Cleanup 2021-10-14 10:04:13 +02:00
kvtb eae29b0385
fix build with gcc11 2021-10-13 18:03:33 +00:00
Eelco Dolstra 35c98a59c5 Fix GC when there are cycles in the referrers graph
(where "referrers" includes the reverse of derivation outputs and
derivers). Now we do a full traversal to look if we can reach any
root. If not, all paths reached can be deleted.
2021-10-13 12:12:44 +02:00
Eelco Dolstra 1785ba2980 Simplify 2021-10-13 12:12:44 +02:00
Eelco Dolstra dced45f146 strcpy -> memcpy
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2021-10-13 12:12:44 +02:00
Eelco Dolstra 262520fcfe Use a thread per connection 2021-10-13 12:12:44 +02:00
Eelco Dolstra ff453b06f9 Fix auto-gc 2021-10-13 12:12:44 +02:00
Eelco Dolstra 8614cf1334 Non-blocking garbage collector
The garbage collector no longer blocks other processes from
adding/building store paths or adding GC roots. To prevent the
collector from deleting store paths just added by another process,
processes need to connect to the garbage collector via a Unix domain
socket to register new temporary roots.
2021-10-13 12:12:44 +02:00
Eelco Dolstra 9947f1646a Remove syncWithGC() 2021-10-13 12:12:44 +02:00
Eelco Dolstra 8eac7dfad4 Remove trash directory 2021-10-13 12:12:44 +02:00
Eelco Dolstra 624dfde3df
Merge pull request #5362 from Artturin/nixunpack
nix develop: add --unpack
2021-10-13 11:39:08 +02:00
Eelco Dolstra 0fac86fd6f Style tweaks 2021-10-13 11:00:10 +02:00
Eelco Dolstra abd685d373 Merge branch 'feature/comp-level' of https://github.com/tomberek/nix 2021-10-13 10:45:44 +02:00
Eelco Dolstra 3e0c6aac9a
Merge pull request #5375 from edolstra/repl-ctrl-c
nix repl: Fix plugin-files warning
2021-10-12 17:11:39 +02:00
Eelco Dolstra 5176b072ed Cleanup 2021-10-12 16:43:00 +02:00
Eelco Dolstra 3a778ea8a0 Merge branch 'nix-repl-download-interruption' of https://github.com/Ma27/nix 2021-10-12 16:34:38 +02:00
Eelco Dolstra f6cdae5181 nix repl: Don't write to std::cout directly
Writing to std::cout doesn't play nice with ProgressBar.
2021-10-12 15:36:45 +02:00
Eelco Dolstra 9ebe02a81e nix repl: Don't build in a child process
Fixes #5356. This is a bit risky due to interrupts, but we have to
deal with those anyway (#5353).
2021-10-12 15:27:02 +02:00
Tom Bereknyei 03bb8f84e0 Add compression level for NARs
Based off on @dtzWill's #2276
2021-10-12 02:14:36 -04:00
Ben Burdette 427fb8d158 comment out debugs 2021-10-11 16:48:10 -06:00
Ben Burdette 98eb13691a print staticenv bindings 2021-10-11 16:32:43 -06:00
Ben Burdette 2ee1fa4afd add nullable Expr argument 2021-10-11 14:42:29 -06:00
regnat 7466048d39 (partially) Revert "Don't copy in rethrow"
This reverts some parts of commit
8430a8f086 which was trying to rethrow
some exceptions while we weren’t in the context of a `catch` block,
causing some weird “terminate called without an active exception”
errors.

Fix #5368
2021-10-11 10:51:22 +02:00
Artturin e399c6ab7f nix develop: add --unpack 2021-10-09 01:19:50 +03:00
Sergei Trofimovich d7d6fe44d6 libstore-tests: add libutil dependency (fix static link failure)
In https://github.com/NixOS/nix/pull/5350 we noticed link failures
pkgsStatic.nixUnstable. Adding explicit dependency on libutil fixes
libstore-tests linking.
2021-10-08 07:16:12 +01:00
Maximilian Bosch 0872659002
nix repl: properly deal with interruptions
When I stop a download with Ctrl-C in a `nix repl` of a flake, the REPL
refuses to do any other downloads:

    nix-repl> builtins.getFlake "nix-serve"
    [0.0 MiB DL] downloading 'https://api.github.com/repos/edolstra/nix-serve/tarball/e9828a9e01a14297d15ca41 error: download of 'e9828a9e01' was interrupted
    [0.0 MiB DL]
    nix-repl> builtins.getFlake "nix-serve"
    error: interrupted by the user
    [0.0 MiB DL]

To fix this issue, two changes were necessary:

* Reset the global `_isInterrupted` variable: only because a single
  operation was aborted, it should still be possible to continue the
  session.
* Recreate a `fileTransfer`-instance if the current one was shut down by
  an abort.
2021-10-07 23:58:02 +02:00
Eelco Dolstra d39692e6b3 Make builtins.{path,filterSource} work with chroot stores 2021-10-07 14:22:39 +02:00
Eelco Dolstra c4dcf3cf25 Add a trace to all errors in addPath() 2021-10-07 13:47:15 +02:00
Eelco Dolstra 4806f2f6b0 Allow builtins.{path,filterSource} on paths with a context
We now build the context (so this has the side-effect of making
builtins.{path,filterSource} work on derivations outputs, if IFD is
enabled) and then check that the path has no references (which is what
we really care about).
2021-10-07 13:43:17 +02:00
Eelco Dolstra 66c4b20d8b Typo 2021-10-07 13:34:04 +02:00
Eelco Dolstra 7c50568788 Remove unnecessary call to queryMissing()
Worker::run() already does this.
2021-10-07 13:15:01 +02:00
Eelco Dolstra 972405edf5 Allow access to path copied to the store
Fixes https://github.com/NixOS/nix/pull/5163#issuecomment-931733912.
2021-10-07 12:15:22 +02:00
Eelco Dolstra cfaad7168e Refactoring: Add allowPath() method 2021-10-07 12:11:00 +02:00
Eelco Dolstra c9ee634f75
Merge pull request #5341 from andir/libexpr-formals
libexpr: remove matchAttrs boolean from ExprLambda
2021-10-07 11:58:56 +02:00
Eelco Dolstra 53e4794289
Merge pull request #5286 from ilkecan/add-a-warning-to-filterSource
Warn about the usage of filterSource with Nix store paths
2021-10-06 21:02:39 +02:00
ilkecan a4a6ef4fb2 Add a warning to filterSource
Warn about the usage of `filterSource` with Nix store paths
2021-10-06 19:25:33 +03:00
Eelco Dolstra 0dc8172458 Remove no-op call to realiseContext() 2021-10-06 18:08:37 +02:00
Eelco Dolstra c497fce011 Merge branch 'flakes_filterSource' of https://github.com/tomberek/nix 2021-10-06 18:08:18 +02:00
Eelco Dolstra 57a8eb4c01 fetchTree(): Parse type attribute first
The 'url' attribute depends on whether type == 'git', so this is needed for

  builtins.fetchTree {url = "git@github.com:NixOS/nix.git"; type = "git";}
2021-10-06 17:39:02 +02:00
Eelco Dolstra 83d86cc1b0 Cleanup 2021-10-06 17:30:10 +02:00
Andreas Rammhold cae41eebff libexpr: remove matchAttrs boolean from ExprLambda
The boolean is only used to determine if the formals are set to a
non-null pointer in all our cases. We can get rid of that allocation and
instead just compare the pointer value with NULL. Saving up to
sizeof(bool) + platform specific alignment per ExprLambda instace.
Probably not a lot of memory but perhaps a few kilobyte with nixpkgs?

This also gets rid of a potential issue with dereferencing formals based on
the value of the boolean that didn't have to be aligned with the formals
pointer but was in all our cases.
2021-10-06 17:24:06 +02:00
Eelco Dolstra 46753b5e9c Merge branch 'considerate/scp-like-urls' of https://github.com/considerate/nix 2021-10-06 17:19:27 +02:00
Eelco Dolstra c6718a9d95 Don't reset the logger in a vfork
9c766a40cb broke logging from the
daemon, because commonChildInit is called when starting the build hook
in a vfork, so it ends up resetting the parent's logger. So don't
vfork.

It might be best to get rid of vfork altogether, but that may cause
problems, e.g. when we call an external program like git from the
evaluator.
2021-10-06 13:54:59 +02:00
Eelco Dolstra 08aa7daee0 Remove links to .md files in help output
Fixes #5337.
2021-10-06 13:01:18 +02:00
Eelco Dolstra fd57e7074f
Merge pull request #5335 from edolstra/socket-paths
Support arbitrary-length socket paths
2021-10-05 17:36:07 +02:00
Eelco Dolstra b14bc06955 Don't ignore SIGCHLD in createUnixDomainSocket() 2021-10-05 13:23:16 +02:00
Eelco Dolstra e4a5d64a81 Show failing PID 2021-10-05 13:19:55 +02:00
Eelco Dolstra b299560872 Typo 2021-10-05 11:24:09 +02:00
Eelco Dolstra 3b7f4c7d9d Add FIXME about ptsname 2021-10-05 11:04:46 +02:00
Eelco Dolstra 43d4d75e22 Connect/bind Unix domain sockets in a child process
In the child process, we can do a chdir() and avoid the problem of the
path not fitting into sockaddr_un.
2021-10-05 10:44:59 +02:00
Eelco Dolstra 77ebbc9f54 Add a test for RefScanSink and clean up the code
Issue #5322.
2021-10-04 14:29:42 +02:00
Eelco Dolstra ef34fd0656 scanForReferences(): Use a StorePathSet 2021-10-04 13:47:38 +02:00
Eelco Dolstra 6fadb5df05
Merge pull request #5310 from trofi/drop-spammy-rewrite
local-derivation-goal.cc: drop spammy "warning: rewriting hashes in..…
2021-10-03 19:53:06 +02:00
kvtb 638c73776a
fix creation of NAR files >4GB on 32-bit platforms
`size_t` is 32-bit on 32-bit platforms, while the file size can be bigger
2021-10-02 21:04:01 +00:00
Ben Burdette aad27143c6 storing staticenv bindings 2021-10-02 13:47:36 -06:00
Alex Kladov 3e884aa002
Fix typo 2021-10-02 10:17:45 +03:00
Sergei Trofimovich 621aa65325 local-derivation-goal.cc: downgrade "warning: rewriting hashes in..." down to debug
Before the changes when building the whole system with
`contentAddressedByDefault = true;` we get many noninformative messages:

    $ nix build -f nixos system --keep-going
    ...
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-clang-11.1.0.drv.chroot/nix/store/...-11.1.0-python'; cross fingers
    error: 2 dependencies of derivation '/nix/store/...-hub-2.14.2.drv' failed to build
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-dev'; cross fingers
    warning: rewriting hashes in '/nix/store/...-subversion-1.14.1.drv.chroot/nix/store/...-subversion-1.14.1-man'; cross fingers
    ...

Let's downgrade these messages down to debug().
2021-10-01 17:41:44 +01:00
Eelco Dolstra 7cc220825d
Merge pull request #5167 from Ma27/keep-failed-on-ssh-remote-build
nix-store --serve: pass on `settings.keepFailed` from SSH store
2021-10-01 16:35:02 +02:00
John Ericson 242f9bf3dc std::visit by reference
I had started the trend of doing `std::visit` by value (because a type
error once mislead me into thinking that was the only form that
existed). While the optomizer in principle should be able to deal with
extra coppying or extra indirection once the lambdas inlined, sticking
with by reference is the conventional default. I hope this might even
improve performance.
2021-09-30 21:35:09 +00:00
Taeer Bar-Yam f14660d5e2 reset yylloc when yyless(0) is called 2021-09-29 19:47:01 -04:00
Eelco Dolstra fd01c48d34
Merge pull request #5301 from Ma27/builtins-missing-feature-error
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
2021-09-29 12:53:29 +02:00
Maximilian Bosch 2b02ce0e48
libexpr: throw a more helpful eval-error if a builtin is not available due to a missing feature-flag
I found it somewhat confusing to have an error like

    error: attribute 'getFlake' missing

if the required experimental-feature (`flakes`) is not enabled. Instead,
I'd expect Nix to throw an error just like it's the case when using e.g. `nix
flake` without `flakes` being enabled.

With this change, the error looks like this:

    $ nix-instantiate -E 'builtins.getFlake "nixpkgs"'
    error: Cannot call 'builtins.getFlake' because experimental Nix feature 'flakes' is disabled. You can enable it via '--extra-experimental-features flakes'.

           at «string»:1:1:

                1| builtins.getFlake "nixpkgs"
                 | ^

I didn't use `settings.requireExperimentalFeature` here on purpose
because this doesn't contain a position. Also, it doesn't seem as if we
need to catch the error and check for the missing feature here since
this already happens at evaluation time.
2021-09-29 11:57:15 +02:00
Eelco Dolstra 6b5efeafd1
Merge pull request #5295 from Ma27/bmcheck-fod-mismatch
build: also throw hash-mismatch errors if `buildMode == bmCheck`
2021-09-29 10:13:45 +02:00
Maximilian Bosch bb1a851bcf
build: also throw hash-mismatch errors if buildMode == bmCheck
This actually bit me quite recently in `nixpkgs` because I assumed that
`nix-build --check` would also error out if hashes don't match anymore[1]
and so I wrongly assumed that I couldn't reproduce the mismatch error.

The fix is rather simple, during the output registration a so-called
`delayedException` is instantiated e.g. if a FOD hash-mismatch occurs.
However, in case of `nix-build --check` (or `--rebuild` in case of `nix
build`), the code-path where this exception is thrown will never be
reached.

By adding that check to the if-clause that causes an early exit in case
of `bmCheck`, the issue is gone. Also added a (previously failing)
test-case to demonstrate the problem.

[1] https://github.com/NixOS/nixpkgs/pull/139238, the underlying issue
    was that `nix-prefetch-git` returns different hashes than `fetchgit`
    because the latter one fetches submodules by default.
2021-09-27 15:44:39 +02:00
Eelco Dolstra 9c766a40cb Fix 'error: reading a line: Input/output error' in startBuilder()
With -vvvv, the ProgressBar was polluting the stderr of the child,
messing up its \2 message to the parent.
2021-09-27 14:44:21 +02:00
Eelco Dolstra 8430a8f086 Don't copy in rethrow 2021-09-27 14:38:10 +02:00
Eelco Dolstra 4b2b151131 nix path-info -r: Don't duplicate the root paths
This fixes

  $ nix path-info -r $(type -P ls)
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32
  /nix/store/5d821pjgzb90lw4zbg6xwxs7llm335wr-libunistring-0.9.10
  ...
  /nix/store/mrv4y369nw6hg4pw8d9p9bfdxj9pjw0x-acl-2.3.0
  /nix/store/vfilzcp8a467w3p0mp54ybq6bdzb8w49-coreutils-8.32

Also, output the paths in topologically sorted order like we used to.
2021-09-27 12:47:39 +02:00
Eelco Dolstra a15e65eef0 run(): Move 2021-09-27 11:12:06 +02:00
Eelco Dolstra 283e7da470
Merge pull request #5292 from edolstra/update-lowdown
Update lowdown
2021-09-27 11:10:05 +02:00
oxalica 81b8e910a0
Explicitly set initial branch name for git 2021-09-24 22:09:49 +08:00
Eelco Dolstra aedbc7b683 Use latest lowdown
This improves list rendering
(https://github.com/kristapsdz/lowdown/issues/73).
2021-09-24 15:22:51 +02:00
Eelco Dolstra 362d8f925e
Merge pull request #5253 from edolstra/flake-ifd
Don't allow IFD in flakes by default
2021-09-24 10:48:23 +02:00
Ben Burdette b9d08b98da ok was unconditoinally throwing on any With var 2021-09-23 13:02:39 -06:00
Eelco Dolstra ea9df6fe51 Shut down write side before draining the read side
This is important if the remote side *does* execute
nix-store/nix-daemon successfully, but stdout is polluted
(e.g. because the remote user's bashrc script prints something to
stdout). In that case we have to shutdown the write side to force the
remote nix process to exit.
2021-09-23 18:01:04 +02:00
Eelco Dolstra 994348e9e0 SSHStore / LegacySSHStore: Show a better error message if the remote is "nologin"
Instead of

  error: serialised integer 7161674624452356180 is too large for type 'j'

we now get

  error: 'nix-store --serve' protocol mismatch from 'sshtest@localhost', got 'This account is currently not available.'

Fixes https://github.com/NixOS/nixpkgs/issues/37287.
2021-09-23 17:50:29 +02:00
Eelco Dolstra 60642aa5e2 Remove risky char * 2021-09-23 12:07:50 +02:00
Eelco Dolstra 198fa786a1 Add some more instrumentation 2021-09-23 11:42:33 +02:00
Geoff Reedy 9d67332e4b Better eval error locations for interpolation and +
Previously, type or coercion errors for string interpolation, path
interpolation, and plus expressions were always reported at the
beginning of the outer expression. This leads to confusing evaluation
error messages making it hard to accurately diagnose and then fix the
error.

For example, errors were reported as follows.

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                 ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                     ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                  ^
```

This commit changes the ExprConcatStrings expression vector to store a
sequence of expressions *and* their expansion locations so that error
locations can be reported accurately. For interpolation, the error is
reported at the beginning of the entire `${foo}`, not at the beginning
of `foo` because I thought this was slightly clearer. The previous
errors are now reported as:

```
cannot coerce an integer to a string
1| let foo = 7; in "bar" + foo
 |                         ^

cannot add a string to an integer
1| let foo = "bar"; in 4 + foo
 |                         ^

cannot coerce an integer to a string
1| let foo = 7; in "x${foo}"
 |                   ^
```

The error is reported at this kind of precise location even for
multi-line indented strings.

This probably helps with at least some of the cases mentioned in #561
2021-09-22 20:57:34 -06:00
Ben Burdette c07edb1932 staticenv should be With 2021-09-22 18:14:57 -06:00
Ben Burdette c7e3d830c1 more debug stuff 2021-09-22 16:22:53 -06:00
Eelco Dolstra d1bf7431bb Revert "Merge pull request #4922 from nrdxp/default-submodules"
This reverts commit 6678e98411, reversing
changes made to 90b2dd570c.
2021-09-22 17:25:25 +02:00
Eelco Dolstra 8623a5b595 Disable IFD selectively
It's now disabled by default for the following:

* 'nix search' (this was already implied by read-only mode)
* 'nix flake show'
* 'nix flake check', but only on the hydraJobs output
2021-09-22 17:15:07 +02:00
Eelco Dolstra d8c10028d9 Make setDefault() typed 2021-09-22 14:15:35 +02:00
Eelco Dolstra ff28fffce2 Don't cache realiseContext() errors
Errors that depend on the configuration (such as whether
allow-import-from-derivation is set) should not be cached.
2021-09-22 14:00:56 +02:00
Eelco Dolstra bcd73ebf60 Add missing #include
Fixes #5282.
2021-09-22 12:14:50 +02:00
Eelco Dolstra 60cc975d22 Set input parent at construction time 2021-09-21 14:07:16 +02:00
Eelco Dolstra 06557299b3 Allow relative paths anywhere into the parent's store path 2021-09-21 13:45:11 +02:00
Eelco Dolstra 5cbb9c5406 path fetcher: Fix relative path check 2021-09-21 13:19:26 +02:00
Michael Bishop 374fe49ff7 set the PER_LINUX32 personality flag, when building for armv6l-linux or armv7l-linux
this prevents 32bit builds from detecting a 64bit kernel and picking the wrong target
2021-09-19 23:07:10 -03:00
Gregor Kleen fd67a0f927 Fix scheme argument to s3 cache 2021-09-17 10:45:19 +02:00
Geoff Reedy cbe9ddfd1a Include subpath in flake fingerprint
Without this, flakes within the same tree and same lock data will have
the same fingerprint and the eval cache for one flake will be
incorrectly used for another.
2021-09-16 15:58:21 -06:00
Ben Burdette cd8c232b55 add cout debugging 2021-09-15 16:16:53 -06:00
Eelco Dolstra 1ec4efa6c8
Merge pull request #5257 from edolstra/dirty-lock-file
If we can't write a lock file, pretend the top-level flake is dirty
2021-09-15 20:18:23 +02:00
Eelco Dolstra c17f3c5e69 Merge branch 'mh/fix-chroot-eval' of https://github.com/obsidiansystems/nix 2021-09-15 18:37:58 +02:00
Eelco Dolstra 027344ce7e If we can't write a lock file, pretend the top-level flake is dirty
Alternative to #4639. You can still read flake.lock, but at least in
reproducible workflows like NixOS configurations where you require a
non-dirty tree, evaluation will fail because there is no rev.
2021-09-15 18:31:42 +02:00
Eelco Dolstra 991cc53386 Revert "Disallow reading flake.lock"
This reverts commit e5596113f7.
2021-09-15 18:30:37 +02:00
Emily 56025ad3b1 sandbox: allow Rosetta 2 on Darwin
This allows sandboxed x86_64-darwin builds on aarch64-darwin.
2021-09-15 02:02:03 +01:00
Eelco Dolstra fda4efff87 Rename unnecessary git@ 2021-09-14 22:53:31 +02:00
Eelco Dolstra 5ee3ee1a6b
Merge pull request #5249 from edolstra/nix-profile
Add missing 'nix profile' subcommands
2021-09-14 22:27:19 +02:00
Eelco Dolstra e5596113f7 Disallow reading flake.lock
With --no-write-lock-file, it's possible that flake.lock is out of
sync with the actual inputs used by the evaluation. So doing fromJSON
(readFile ./flake.lock) will give wrong results.

Fixes #4639.
2021-09-14 21:09:11 +02:00
Eelco Dolstra b41968f15a nix profile history: Show profile date 2021-09-14 20:47:33 +02:00
Eelco Dolstra 4b738fc7a9 Add 'nix profile wipe-history' command 2021-09-14 20:35:41 +02:00
Eelco Dolstra f359b9981b Generations -> profile versions 2021-09-14 19:57:45 +02:00
Eelco Dolstra 229ad612b8 Fix quotes 2021-09-14 19:48:16 +02:00
Eelco Dolstra 817562e694 Add "nix profile rollback" command 2021-09-14 19:32:33 +02:00
Ben Burdette 21071bfdeb shared_ptr for StaticEnv 2021-09-14 10:49:22 -06:00
Eelco Dolstra 1fbaf36729 nix flake show --json: Add type info
For extensibility, every leaf node is now an object that contains at
least a type field (e.g. "type": "derivation").
2021-09-14 17:18:29 +02:00
Eelco Dolstra f3259af73e Merge branch 'tomberek/show_json' of https://github.com/tomberek/nix 2021-09-14 16:58:35 +02:00
Eelco Dolstra d589a6aa8a
Merge pull request #5247 from edolstra/stablize-ca-references
Remove the 'ca-references' feature check
2021-09-14 15:42:50 +02:00
Eelco Dolstra 75837bb595 Respect NO_COLOR in Markdown output 2021-09-14 14:57:58 +02:00
Eelco Dolstra 7e4dd0e279
Merge pull request #5248 from edolstra/flake-clone
Fix 'nix flake clone' on github repos
2021-09-14 14:14:59 +02:00
Eelco Dolstra 01a4fa3b6e Advise using --extra-experimental-features instead of --experimental-features
Fixes #3737.
2021-09-14 13:56:36 +02:00
Eelco Dolstra 761ac9d584 Remove the 'ca-references' feature check
Fixes #3422.
Fixes #4425.
2021-09-14 13:53:20 +02:00
Eelco Dolstra 07996c4810 github fetcher: Don't use HEAD branch when cloning
Fixes #4394.
2021-09-14 13:38:45 +02:00
Eelco Dolstra d72d31d529 github fetcher: Use git+https for cloning
git+ssh only works if you have SSH access.
2021-09-14 13:38:05 +02:00
Eelco Dolstra a4ae601b44
Merge pull request #5245 from edolstra/warnings
Change warning messages from yellow to magenta
2021-09-14 11:46:34 +02:00
Eelco Dolstra 16d4922dd2
Merge pull request #5240 from edolstra/builtin-help
nix --help: Display help using lowdown instead of man
2021-09-14 11:31:23 +02:00
Eelco Dolstra 58b5036c54 Change warnings from yellow to magenta
This matches gcc and clang.
2021-09-14 10:42:29 +02:00
Eelco Dolstra 4ffda0af7c ANSI_YELLOW -> ANSI_WARNING 2021-09-14 10:42:29 +02:00
Eelco Dolstra 76e368a3b4 Fix macOS build 2021-09-14 10:07:31 +02:00
Eelco Dolstra 1ba993d07c Fix clang warning 2021-09-14 08:15:33 +02:00
Eelco Dolstra 4ed66735b6 RunOptions: Use designated initializers
Also get rid of _killStderr because it wasn't actually checked
anywhere.
2021-09-13 23:31:04 +02:00
Eelco Dolstra c3e9acd1c0 Remove tabs 2021-09-13 23:06:33 +02:00
kvtb c6fa7775de hashFile, hashString: realize context before calculation, and discard afterwards 2021-09-13 22:34:58 +02:00
Ben Burdette 176911102c printEnvPosChain 2021-09-13 11:57:25 -06:00
Eelco Dolstra 49a932fb18 nix --help: Display help using lowdown instead of man
Fixes #4476.
Fixes #5231.
2021-09-13 14:45:21 +02:00
Eelco Dolstra 8796b1b5e3 Fix markdown error 2021-09-13 14:44:53 +02:00
Eelco Dolstra eadb45c4db Use Bindings::{get,need} instead of find 2021-09-13 13:53:24 +02:00
Eelco Dolstra a73be28717 renderMarkdownToTerminal(): Fix terminal width computation 2021-09-13 13:52:14 +02:00
Eelco Dolstra ff09acd1f9 Remove tabs 2021-09-10 11:00:50 +02:00
Eelco Dolstra 02ece164be Make installables const 2021-09-10 10:39:39 +02:00
Eelco Dolstra 1e43bc6dc0
Merge pull request #5224 from baloo/baloo/5089/force-nss_dns-load
preloadNSS: fixup nss_dns load
2021-09-09 10:37:41 +02:00
Tom Bereknyei dc25856d74 Ensure nix flake show produces valid json 2021-09-08 19:38:22 -04:00
Arthur Gautier 3b72741f23 preloadNSS: load NSS before threads are started
preloadNSS is not thread-safe, this commit moves it before we start the
first thread.

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-08 18:29:31 +00:00
Eelco Dolstra b71428c907 Merge branch 'fix-3976' of https://github.com/mkenigs/nix 2021-09-08 14:43:12 +02:00
Naïm Favier 7f0d177ce7
Add missing include in util.cc 2021-09-08 12:20:08 +02:00
Eelco Dolstra 9180239081
Merge pull request #5215 from ncfavier/patch-1
Fix use-registries logic in builtins.getFlake
2021-09-08 11:19:39 +02:00
Shea Levy 6678e98411
Merge pull request #4922 from nrdxp/default-submodules
libfetchers/git: fetch submodules by default
2021-09-07 05:48:23 -04:00
Arthur Gautier 0b42a0f781 preloadNSS: fixup nss_dns load
Before this commit, the dns lookup in preloadNSS would still go through
nscd. This did not have the effect of loading the nss_dns.so as expected
(nss_dns.so being out of reach from within the sandbox).

Should LOCALDOMAIN environment variable be defined, nss will completely
avoid nscd and will do its dns resolution on its own.

By temporarly setting LOCALDOMAIN variable before calling in NSS, we can
force NSS to load the shared libraries as expected.

Fixes #5089

Signed-off-by: Arthur Gautier <baloo@superbaloo.net>
2021-09-06 23:13:48 +00:00
Naïm Favier b0d4190f19
Fix use-registries logic in builtins.getFlake 2021-09-05 15:28:44 +02:00
Eelco Dolstra 90b2dd570c Add FIXME 2021-09-02 15:00:52 +02:00
Georges Dubus bc6e7ca046 Don't use read-only mode for nix build --dry-run
In dry run mode, new derivations can't be create, so running the command on anything that has not been evaluated before results in an error message of the form `don't know how to build these paths (may be caused by read-only store access)`.

For comparison, the classical `nix-build --dry-run` doesn't use read-only mode.

Closes #1795

(cherry picked from commit 54525682df707742e58311c32e9c9cb18de1e31f)
2021-09-02 14:50:19 +02:00
Eelco Dolstra b2e8120d25 parseInstallables(): Parse store paths as store paths
If the store path contains a flake, this means that a command like
"nix path-info /path" will show info about /path, not about the
default output of the flake in /path. If you want the latter, you can
explicitly ask for it by doing "nix path-info path:/path".

Fixes #4568.
2021-09-02 14:18:04 +02:00
Eelco Dolstra 77ca5e951c
Merge pull request #5203 from NixOS/fix-invalid-lockfile-names
Don’t create lockfiles with an invalid path name
2021-09-02 11:03:18 +02:00
Lucas Hoffmann d948415b71
Docs: Fix syntax error in json example in man page 2021-09-02 10:47:38 +02:00
regnat 497225b07d Don’t create lockfiles with an invalid path name
Store paths are only allowed to contain a limited subset of the
alphabet, which doesn’t include `!`. So don’t create lockfiles that
contain this `!` character as that would otherwise confuse (and break)
the gc.

Fix #5176
2021-09-02 09:57:41 +02:00
Eelco Dolstra f73d911628 Style 2021-09-01 21:46:25 +02:00
Eelco Dolstra 2cb7a502a4 Merge branch 'short-circuit-subst' of https://github.com/Kha/nix 2021-09-01 21:45:04 +02:00
Eelco Dolstra 7ee639f9db
Merge pull request #5066 from Radvendii/master
add antiquotations to paths
2021-09-01 12:55:04 +02:00
Taeer Bar-Yam 1ffb9f1970 fix parse of /${foo}. was // + foo
I don't think this changes the way any program would behave, but it's a
cleaner internal representation.
2021-08-31 15:55:55 -04:00
Taeer Bar-Yam 9da8f5e25d path antiquotations: canonizePath -> canonicalizePath 2021-08-31 08:02:04 -04:00
Maximilian Bosch 50edbc4ddf
nix-store --serve: pass on settings.keepFailed from SSH store
When doing e.g.

    nix-build -A package --keep-failed --option \
      builders \
      'ssh://mfhydra?remote-store=/home/bosch/store x86_64-linux - 10 4 big-parallel'

this doesn't work properly because this build-setting is ignored.

I changed this behavior by passing the `settings.keepFailed` through the
serve-protocol to remote machines to make sure that I can introspect the
build-directory (which is particularly helpful when I have to look at a
`config.log` from a failed build for instance).
2021-08-31 13:11:46 +02:00
Eelco Dolstra f3b8b4040e
Merge pull request #5071 from NixOS/dont-send-xp-features-to-the-daemon
Don’t send the experimental-features to the daemon
2021-08-31 11:55:39 +02:00
Eelco Dolstra 50a35860ee TarArchive: Small refactoring 2021-08-30 17:02:51 +02:00
Eelco Dolstra c7a7652725 Don't segfault if archive_entry_pathname() returns null
Issues #4499.
2021-08-30 17:02:39 +02:00
Eelco Dolstra eda0fee160
Merge pull request #5175 from Pamplemousse/make
Don't overwrite user provided `lib*_LDFLAGS`
2021-08-30 12:44:29 +02:00
Eelco Dolstra 323cafcb4e
Merge pull request #5191 from hercules-ci/evalstate-lifetime-hygiene
EvalState lifetime hygiene
2021-08-30 12:23:09 +02:00
regnat b2bce915ad Don’t accept experimental features from the client
If the client sends an “experimental features” setting, just ignore it
2021-08-30 10:48:05 +02:00
Robert Hensing 92778a5f80 Tidy 2021-08-30 09:52:43 +02:00
Robert Hensing 8656b130ea Fix use after free with vImportedDrvToDerivation 2021-08-29 20:42:57 +02:00
Robert Hensing 8bc76acc7c Move vCallFlake into EvalState
This fixes a use-after-free bug:

1. s = new EvalState();
2. callFlake()
3. static vCallFlake now references s
4. delete s;
5. s2 = new EvalState();
6. callFlake()
7. static vCallFlake still references s
8. crash

Nix 2.3 did not have a problem with recreating EvalState.
2021-08-29 20:42:49 +02:00
Robert Hensing f10465774f Force all Pos* to be non-null
This fixes a class of crashes and introduces ptr<T> to make the
code robust against this failure mode going forward.

Thanks regnat for the idea of a ref<T> without overhead!

Closes #4895
Closes #4893
Closes #5127
Closes #5113
2021-08-29 18:11:58 +02:00
Ben Burdette 310c689d31 remove more explicit valmap code 2021-08-25 13:18:27 -06:00
Ben Burdette d8a977a22e adding all the value names from env.values[0] 2021-08-25 11:19:09 -06:00
Pamplemousse a4c6d319a8 Don't overwrite user provided lib*_LDFLAGS
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-25 08:59:19 -07:00
Ben Burdette bd3b5329f9 print env bindings 2021-08-24 16:32:54 -06:00
Eelco Dolstra af94b54db3 Coding style 2021-08-23 13:05:42 +02:00
Eelco Dolstra c6b063c31a Merge branch 'fix/subflake-follows-fix' of https://github.com/ArctarusLimited/nix 2021-08-23 12:47:09 +02:00
Eelco Dolstra a93f72c084
Merge pull request #5160 from CertainLach/fix/invalid-characters-in-json-logger
Replace invalid characters in json logger
2021-08-23 11:10:55 +02:00
Tom Bereknyei d90582be33 Allow use of path and filterSource in flakes
As filterSource and path perform work, add paths to allowedPaths.
2021-08-22 18:45:42 -04:00
Yaroslav Bolyukin bbb3bcb165
Replace invalid characters in json logger
Fixes #5159

Signed-off-by: Yaroslav Bolyukin <iam@lach.pw>
2021-08-22 00:26:22 +03:00
Jan Tojnar 2d66a31f01 nix {bundle,run}: drop broken flake attr-path prefixes
“packages” was probably meant to be “packages.${system}.” but that
is already listed in `getDefaultFlakeAttrPathPrefixes` in `installables`,
which is probably why no one noticed it was broken.
2021-08-21 20:17:05 +02:00
Jan Tojnar d7b6c8f591 nix develop: Fix devShells lookup
It currently fails with the following error:

    error: flake 'git+file://…' does not provide attribute 'devShells.x86_64-linuxhaskell', 'packages.x86_64-linux.haskell', 'legacyPackages.x86_64-linux.haskell' or 'haskell'
2021-08-21 01:24:28 +02:00
Matthew Kenigsberg d7fe36116e
nix develop --phase: chdir to flake directory
For git+file and path flakes, chdir to flake directory so that phases
that expect to be in the flake directory can run

Fixes https://github.com/NixOS/nix/issues/3976
2021-08-19 15:42:13 -05:00
Ben Burdette 4b5f9b35f0 env to bindings 2021-08-18 21:25:26 -06:00
Ben Burdette 2272021536 more error fixes 2021-08-18 20:02:23 -06:00
Ben Burdette e82cf13b1e switch to fakeenvs 2021-08-18 17:53:10 -06:00
Tom Bereknyei a82de5b31b Enable JSON option to show flakes
`nix-flake show --json`
2021-08-17 23:18:19 -04:00
Ben Burdette b6eb38016b moving towards env in exceptions 2021-08-17 14:39:50 -06:00
Alex Zero b3c424f5a6
Fix follows paths in subordinate lockfiles 2021-08-17 15:25:42 +01:00
Eelco Dolstra 2cd1a5b8f3
Merge pull request #5134 from pszubiak/fix-pkgconfig-install-path
Install pkg-config files in the correct location
2021-08-16 11:56:49 +02:00
Magic_RB 8438114399
Add ignored_acls setting
Signed-off-by: Magic_RB <magic_rb@redalder.org>
2021-08-14 21:46:09 +02:00
Jörg Thalheim 21d0334e49 fix typo in ca-error message 2021-08-14 00:18:45 +02:00
Piotr Szubiakowski 9f13cb31e8
Install pkg-config files in the correct location
Use `$(libdir)` while installing .pc files looks like a more generic
solution. For example, it will work for distributions like RHEL or
Fedora where .pc files are installed in `/usr/lib64/pkgconfig`.
2021-08-13 21:08:58 +00:00
Eelco Dolstra 6b8069b823 In flake lock file diffs, show the last-modified date of inputs if available
This is a bit more informative than just the hash.

Also, format the diffs a bit nicer.
2021-08-11 19:35:28 +02:00
Pamplemousse 0da416f820 15f4d4f follow up
* libstore: `bz2` should not be linked
  * libutil: `zlib.h` should not be included

Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-10 16:04:53 -07:00
Ben Burdette 030271184f trying env args; but unecessary? 2021-08-09 14:30:47 -06:00
Eelco Dolstra 8943e3176d
Merge pull request #5111 from Pamplemousse/clean
Minor maintenance cleaning
2021-08-09 20:05:03 +02:00
Pamplemousse 2de7a1fe67 libexpr: Squash similar conditions
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-09 10:10:11 -07:00
Sebastian Ullrich a0f97b1f54 Expand docstring of Store::querySubstitutablePathInfos 2021-08-09 15:58:50 +02:00
Eelco Dolstra 27444d40cf
Merge pull request #5101 from fzakaria/faridzakaria/fix-nix-shell
nix-shell --pure: Let it work for any derivation
2021-08-09 14:52:11 +02:00
Eelco Dolstra 4c5ce2a345
Merge pull request #5104 from andir/refscan-race
Fix potential race-condition in reference scanning code
2021-08-09 14:49:24 +02:00
Eelco Dolstra 873df3ab03
Merge pull request #5106 from andir/libstore-waiter-set
libstore: use set instead of list for waiter list
2021-08-09 14:46:49 +02:00
Andreas Rammhold a9cb1ca32c libstore: use set instead of list for waiter list
This replaces the O(n) search complexity in our insert code with a
lookup of O(log n). It also makes removing waitees easier as we can use
the extract method provided by the set class.
2021-08-08 14:05:38 +02:00
Sebastian Ullrich 12f50b6510 Short-circuit querying substituters on success 2021-08-08 13:24:13 +02:00
Andreas Rammhold b2d3976163 Fix potential race-condition in reference scanning code
Previously the code ensures that the isBase32 array would only be
initialised once in a single-threaded context. If two threads happen to
call the function before the initialisation was completed both of them
would have completed the initialization step. This allowed for a
race-condition where one thread might be done with the initialization
but the other thread sets all the fields to false again. For a brief
moment the base32 detection would then produce false-negatives.
2021-08-07 19:10:25 +02:00
Alexander Bantyev 54c580b71f
Whitelist nixConfig.flake-registry
flake-registry should be safe to set to an aribtrary value, since it
is identical to just setting `inputs`.
2021-08-07 14:42:59 +03:00
Farid Zakaria fbf42c55ac nix-shell --pure: Let it work for any derivation
`nix-shell --pure` when applied to a non stdenv derivation doesn't seem
to clear the PATH. It expects the stdenv/setup file to do so.

This adds an explicit `unset PATH` by nix-build.cc (nix-shell) itself so
that it's not reliant on stdenv/setup anymore.

This does not break impure nix-shell since the PATH is persisted as the
variable `p` prior in the bash rcfile

fixes #5092
2021-08-06 15:30:49 -07:00
Ben Burdette 89264d20e6 move valmap to hh; add to env 2021-08-06 11:09:27 -06:00
Eelco Dolstra e277c0c479
Merge pull request #5075 from andir/libutil-base64-init
libutil: initialize the base64 decode array only once
2021-08-06 17:41:11 +02:00
Taeer Bar-Yam 8f9429dcab add antiquotations to paths 2021-08-06 06:46:05 -04:00
Madeline Haraj cb1ffb7789 Use the store path as the context of the result of fetchTree, not the real path 2021-08-05 22:08:39 -04:00
Michael Fellinger 5c99ec374c
fix git init race condition 2021-08-02 13:47:59 +02:00
ldesgoui 6eeb6f9c84 nix-shell -p: pass --arg values as nixpkgs params 2021-07-31 17:25:10 +02:00
Andreas Rammhold b9c9c25766
libutil: initialize the base64 decode array only once
Previously, despite having a boolean that tracked initialization, the
decode characters have been "calculated" every single time a base64
string was being decoded.

With this change we only initialize the decode array once in a
thread-safe manner.
2021-07-30 21:07:32 +02:00
Pamplemousse d64cb33e90 Remove unused struct in libstore
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-30 10:34:28 -07:00
regnat d9ba3385a9 Don’t send the experimental-features to the daemon
The experimental features are, well, experimental, and shouldn’t be
carelessly and transparently enabled.
Besides, some (`ca-derivations` at least) need to be enabled at startup
in order to work properly.
So it’s better to just require that daemon be started with the right
`experimental-features` option.

Fix #5017
2021-07-30 10:34:50 +02:00
Viktor Kronvall 7cdaae6b9c Support SCP-like URLs in builtins.fetchGit attrs
This extends https://github.com/NixOS/nix/pull/4978 with
supporting the SCP-like urls in expressions like

```nix
builtins.fetchGit {
  url = "git@github.com:NixOS/nix.git";
  ref = "master";
}
```
2021-07-30 08:17:46 +09:00
Eelco Dolstra 7816ef6c51
Merge pull request #5042 from alyssais/pthread
Enable pthreads for new libraries
2021-07-28 12:31:01 +02:00
Eelco Dolstra ed5ad59dc1 nix develop: Support chroot stores
Fixes #5024.
2021-07-27 14:24:03 +02:00
Eelco Dolstra c000cec27f
Merge pull request #5048 from tweag/flox-eval-store
--eval-store and faster closure copying
2021-07-27 12:20:32 +02:00
Eelco Dolstra 47002108d1 nix-instantiate: Fix --eval-store 2021-07-27 11:16:47 +02:00
Eelco Dolstra f52fa47e16
Merge pull request #5046 from ncfavier/patch-1
Fix --no-gc-warning
2021-07-27 09:55:25 +02:00
Eelco Dolstra 9deeab6337
Merge pull request #5043 from alyssais/vexing
libutil: use uniform initialization in _deletePath
2021-07-27 09:54:24 +02:00
Pamplemousse c2d7c0cdb9 libexpr: Remove unused code
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-26 09:14:34 -07:00
Eelco Dolstra da55210403
Merge pull request #5037 from Pamplemousse/less_globals
Avoid global counters
2021-07-26 14:29:03 +02:00
Eelco Dolstra 72c5bac39d Revert no longer necessary change 2021-07-26 13:50:18 +02:00
Eelco Dolstra fe1f34fa60 Low-latency closure copy
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.

For example, on a connection with an 50 ms delay:

Before:

  $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
    /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
    --derivation --no-check-sigs
  real    0m57.868s
  user    0m0.103s
  sys     0m0.056s

After:

  real    0m0.690s
  user    0m0.017s
  sys     0m0.011s
2021-07-26 13:31:09 +02:00
Naïm Favier cefa8b673f
Fix --no-gc-warning
Broken by 8e758d402b
2021-07-25 16:06:55 +02:00
Guillaume Girol 136a613cce nix-channel: use nix-env -i --remove-all to upgrade
Fixes #4110
2021-07-25 12:00:00 +00:00
Alyssa Ross 369ed71858
libutil: use uniform initialization in _deletePath
Otherwise I get a compiler error when building for NetBSD:

src/libutil/util.cc: In function 'void nix::_deletePath(const Path&, uint64_t&)':
src/libutil/util.cc:438:17: error: base operand of '->' is not a pointer
  438 |     AutoCloseFD dirfd(open(dir.c_str(), O_RDONLY));
      |                 ^~~~~
src/libutil/util.cc:439:10: error: 'dirfd' was not declared in this scope
  439 |     if (!dirfd) {
      |          ^~~~~
src/libutil/util.cc:444:17: error: 'dirfd' was not declared in this scope
  444 |     _deletePath(dirfd.get(), path, bytesFreed);
      |                 ^~~~~
2021-07-24 09:19:48 +00:00
Alyssa Ross ae0c026fe9
Enable pthreads for new libraries
Otherwise the lack of pthread causes linking to fail for NetBSD.
2021-07-24 09:15:01 +00:00
Eelco Dolstra 9957315ce0 RemoteStore::buildPaths(): Handle evalStore 2021-07-22 22:50:48 +02:00
Eelco Dolstra a7b7fcfb16 Remove redundant RealisedPath::closure() call 2021-07-22 22:43:18 +02:00
Eelco Dolstra 96c62fb66c Fix formatting error in 'nix store' manpage 2021-07-22 10:11:04 +02:00
Eelco Dolstra eb6db4fd38 buildPaths(): Add an evalStore argument
With this, we don't have to copy the entire .drv closure to the
destination store ahead of time (or at all). Instead, buildPaths()
reads .drv files from the eval store and copies inputSrcs to the
destination store if it needs to build a derivation.

Issue #5025.
2021-07-22 09:59:51 +02:00
Eelco Dolstra 668abd3e57 copyPaths: Pass store by reference 2021-07-22 09:59:51 +02:00
Eelco Dolstra 8d9f7048cd Use eval-store in more places
In particular, this now works:

  $ nix path-info --eval-store auto --store https://cache.nixos.org nixpkgs#hello

Previously this would fail as it would try to upload the hello .drv to
cache.nixos.org. Now the .drv is instantiated in the local store, and
then we check for the existence of the outputs in cache.nixos.org.
2021-07-22 09:59:51 +02:00
Eelco Dolstra 95e915a993 DummyStore: Remove redundant method 2021-07-22 09:59:51 +02:00
Eelco Dolstra 7321657746 nix-shell: Handle --eval-store correctly 2021-07-22 09:59:51 +02:00
Eelco Dolstra e9848beca7 nix-build: Copy drv closure between eval store and build store 2021-07-22 09:59:51 +02:00
Eelco Dolstra 2ff3035cf4 Support --eval-store in nix-instantiate and nix-build 2021-07-22 09:59:51 +02:00
Eelco Dolstra 3d9de41a5b Hacky fast closure copying mechanism 2021-07-22 09:59:51 +02:00
Eelco Dolstra bef40c2949 Add --eval-store option 2021-07-22 09:59:51 +02:00
Eelco Dolstra dfda499326 Downgrade warning message
If a store doesn't support GC, then we don't need to warn about the
inability to create roots.
2021-07-22 09:58:50 +02:00
Pamplemousse c1c5dd7449 Avoid global counters
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-21 16:49:52 -07:00
Eelco Dolstra 3bb8667a17 Tweak --no/use-registries doc strings 2021-07-21 14:27:37 +02:00
Eelco Dolstra fc248cf59e Merge branch 'balsoft/no-registries' of https://github.com/serokell/nix 2021-07-21 10:11:40 +02:00
Eelco Dolstra 140ccf1368 deletePath(): Return ENFILE instead of EBADF when out of file descriptors
Also remove an erroneous comment.
2021-07-20 20:59:45 +02:00
Eelco Dolstra c5fafca5a4 Merge remote-tracking branch 'origin/repl-flake-support' 2021-07-20 20:06:56 +02:00
regnat 8707773965 Properly lock the builds of CA derivations
Make sure that we can’t build the same derivation twice at the same
time.

Fix https://github.com/NixOS/nix/issues/5029
2021-07-20 06:57:56 +02:00
Robert Hensing 3bbf5558e0 nix repl: Update :edit help text
It supports functions as well. Also change `package` to
`derivation` because it operates at the language level and does
not open the derivation (which would be useful but not nearly
as much).
2021-07-19 18:13:26 +02:00
Robert Hensing ad24921de8 Rename findDerivationFilename -> findPackageFilename
It does not operate on a derivation and does not return a
derivation path. Instead it works at the language level,
where a distinct term "package" is more appropriate to
distinguish the parent object of `meta.position`; an
attribute which doesn't even make it into the derivation.
2021-07-19 18:10:10 +02:00
regnat 765a3a20cb Add a :load-flake command to the nix repl
`:lf ref` does essentially the same thing as
`:a (builtins.getFlake "ref")` (but cannonicalising `ref` if needs be,
so that `:lf .` works)

Fix #3803
2021-07-19 17:52:41 +02:00
Eelco Dolstra 58cb411db6
Merge pull request #5020 from NixOS/make-the-repl-reuse-the-whole-nix-config
Forward the whole Nix config to the repl subprocesses
2021-07-19 16:02:02 +02:00
Alexander Bantyev 3e57e3480b
Add use-registries config option (and deprecate --no-registries flag)
Some people want to avoid using registries at all on their system; Instead
of having to add --no-registries to every command, this commit allows to
set use-registries = false in the config. --no-registries is still allowed
everywhere it was allowed previously, but is now deprecated.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-07-16 23:26:04 +03:00
regnat a4ec6cb1da Be more lenient when realisations have a conflicting dependency set
- This can legitimately happen (for example because of a non-determinism
  causing a build-time dependency to be kept or not as a runtime
  reference)
- Because of older Nix versions, it can happen that we encounter a
  realisation with an (erroneously) empty set of dependencies, in which
  case we don’t want to fail, but just warn the user and try to fix it.
2021-07-16 11:48:41 +02:00
regnat c05bdef020 Forward the whole Nix config to the repl subprocesses
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the nix subprocesses in the repl
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available.

This is required for example to make `nix repl` work with a custom
`--store`
2021-07-16 09:48:45 +02:00
regnat 9b1f3cbc13 Forward the whole Nix config to the post-build-hook
Fill `NIX_CONFIG` with the value of the current Nix configuration before
calling the post-build-hook.
That way the whole configuration (including the possible
`experimental-features`, a possibly `--store` option or whatever) will
be made available to the hook
2021-07-15 18:41:56 +02:00
Eelco Dolstra 307977963c nlohmann_json: Update to 3.9.1, fix use of internal copy 2021-07-15 12:25:53 +02:00
Eelco Dolstra 77d5b37da3
Merge pull request #5011 from Pamplemousse/fix_adjustLoc
libexpr: Fix read out-of-bound on the heap
2021-07-15 09:23:59 +02:00
Pamplemousse 99f8fc995b libexpr: Fix read out-of-bound on the heap
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-14 09:09:42 -07:00
regnat 797e260e3a Make nix flake check aware of devShells 2021-07-13 17:25:27 +02:00
regnat 43d5c5f87b Make nix flake show display the devShells 2021-07-13 17:25:27 +02:00
regnat 037c86ee04 nix develop: Search in devShells.${system} by default
Make `nix develop .#foo` search `.#devShells.${system}.foo` first
2021-07-13 17:25:27 +02:00
Eelco Dolstra eb4788954d
Merge pull request #5006 from illustris/nscd
fixed-output derivations: fix incorrect responses for getpwuid
2021-07-13 17:20:37 +02:00
illustris 7bc17a903b fixed output derivations: fix incorrect responses for getpwuid
Passing nscd socket into the build environment causes unexpected behavior in programs that make getpwuid and other related calls.

relevant threads:
- https://github.com/NixOS/nix/issues/4991
- https://discourse.nixos.org/t/haunted-nix-build-breaks-isolation/13869
2021-07-13 15:34:14 +05:30
Eelco Dolstra 099df07e1e
Merge pull request #5000 from nielsegberts/master
toString also coerces a set with an outPath attribute to a string
2021-07-12 17:35:32 +02:00
Eelco Dolstra 02dff9e529 Style 2021-07-12 17:32:23 +02:00
Eelco Dolstra e06c272c12 Merge branch 'structured-attrs-shell' of https://github.com/Ma27/nix 2021-07-12 17:13:05 +02:00
Maximilian Bosch 04cd2da84c
Merge branch 'master' into structured-attrs-shell
Conflicts:
        src/nix/develop.cc
        src/nix/get-env.sh
        tests/shell.nix
2021-07-12 15:49:39 +02:00
Eelco Dolstra 91d2e8d5ad Error -> UsageError 2021-07-12 15:04:46 +02:00
Eelco Dolstra 8d97030bfd Merge branch 'days' of https://github.com/nielsegberts/nix 2021-07-12 15:02:24 +02:00
Niels Egberts 2cf14db857 Throw on unexpected input for --delete-older-than
'--delete-older-than 10' deletes the generations older than a single day, and '--delete-older-than 12m' deletes all generations older than 12 days.

This changes makes it throw on those invalid inputs, and gives an example of a valid input.
2021-07-11 11:24:43 +01:00
Niels Egberts ae0ed53b09 toString also coerces a set with an outPath attribute to a string
nix-repl> builtins.toString { outPath = "somestring"; }
"somestring"
2021-07-09 21:50:10 +01:00
Eelco Dolstra ceda58d112 Formatting 2021-07-09 14:03:51 +02:00
Eelco Dolstra 07790fdddf ref: Add equality operators 2021-07-09 14:03:48 +02:00
Eelco Dolstra 86fb01c4be nix print-dev-env: Add --json flag 2021-07-09 12:10:48 +02:00
Eelco Dolstra 5f6375a816 nix develop: Filter some bash magic variables 2021-07-09 01:18:44 +02:00
Eelco Dolstra b1f1347ade nix develop: Don't parse bash environment with regexes
Instead have get-env.sh dump the bash environment as JSON. This should
be a lot less error-prone.

Fixes #4992.
2021-07-09 01:18:37 +02:00
Eelco Dolstra d2b8b23ae9
Merge pull request #4995 from NixOS/fetchgit-name-attribute
Fix the `name` attribute in builtins.fetchGit
2021-07-09 00:32:13 +02:00
regnat a654c1d81c Restore the possibility to use a name parameter in fetchGit
Accidentally broken by 7e5c79a2d2
2021-07-08 14:57:48 +02:00
Eelco Dolstra 8648143120
Merge pull request #4988 from NixOS/fetchgit-name-attribute
Add a name attribute to the fetchers
2021-07-08 14:33:49 +02:00
regnat 7e5c79a2d2 Forbid the name attribute for fetchTree
We need to support it for the “old” fetch* functions for backwards
compatibility, but we don’t need it for fetchTree (as it’s a new
function).
Given that changing the `name` messes-up the content hashing, we can
just forbid passing a custom `name` argument to it
2021-07-08 13:53:52 +02:00
Eelco Dolstra 2172e60f7a
Merge pull request #4935 from alyssais/host_os
Apply OS checks to host platform, not build
2021-07-08 12:40:54 +02:00
Eelco Dolstra 156666de3d
Merge pull request #4969 from serokell/balsoft/fix-nixConfig-flake-registry
flake.nixConfig: fix flake-registry config settings
2021-07-08 12:35:08 +02:00
Eelco Dolstra 2c853e2a58 Merge branch 'master' of github.com:NixOS/nix 2021-07-07 10:03:04 +02:00
Eelco Dolstra 3b3e6bb1e5 Style tweaks 2021-07-07 10:02:55 +02:00
Eelco Dolstra e37ecd1282 Merge branch 'balsoft/registry' of https://github.com/serokell/nix 2021-07-07 09:57:15 +02:00
regnat 2c8240677e allow fetchFromGitHub to take a name argument 2021-07-06 08:43:19 +02:00
regnat e4b082a52b allow fetchMercurial to take a name argument 2021-07-06 08:43:06 +02:00
regnat a487a652ed allow fetchTarball to take a name argument 2021-07-06 08:42:47 +02:00
regnat 83615fcf8f Allow fetchGit to take a name argument
Fix #3388
2021-07-06 08:41:52 +02:00
Eelco Dolstra 0a8845720e
Merge pull request #4980 from Ma27/file-attr-completion
libcmd/installables: implement completion for Nix expressions passed via `-f`
2021-07-05 21:50:15 +02:00
Maximilian Bosch 3c5f69bb60
completeInstallable: also match for already typed prefixes 2021-07-05 21:37:33 +02:00
Eelco Dolstra 1f93084149
Merge pull request #4978 from NixOS/scp-uris
Allow scp-style uris in `fetchgit`
2021-07-05 14:07:04 +02:00
Eelco Dolstra cee426cc01
Merge pull request #4938 from tomcur/store-uri-parsing
Improve machine store URI parsing
2021-07-05 14:00:35 +02:00
Eelco Dolstra 24bc935462 Merge branch 'disable_gc' of https://github.com/Pamplemousse/nix 2021-07-05 13:53:52 +02:00
Michael Fellinger 087c5f5325
Fix devShell handling of env values including @ and % 2021-07-05 12:05:06 +02:00
Maximilian Bosch 70cb2ffacc
libcmd/installables: implement completion for Nix expressions passed via -f 2021-07-03 14:19:10 +02:00
Domen Kožar f9d72855ae
Merge pull request #4967 from Pamplemousse/specific_errors
Prefer to throw specific errors
2021-07-03 10:34:56 +02:00
regnat ec2c6bd470 Allow scp-style uris in fetchgit
Fix #5303
2021-07-02 19:20:07 +02:00
Yestin L. Harrison 0fe84bef72 Add $NO_COLOR check to ANSI escape conditions 2021-07-02 09:33:54 -06:00
Yestin L. Harrison 20cce079f2 Respect TERM=dumb more consistently 2021-07-01 18:19:01 -06:00
Pamplemousse 4a7a8b87cd Prefer to throw specific errors
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-01 11:09:31 -07:00
Alexander Bantyev e756a59c72
fixup! flake.nixConfig: fix flake-registry config settings 2021-07-01 17:54:22 +03:00
Alexander Bantyev ef1e7ab840
flake.nixConfig: fix flake-registry config settings
Before this commit, nixConfig.flake-registry didn't have any real effect
on the evaluation, since config was applied after inputs were evaluated.
Change this behavior: apply the config in the beginning of flake::lockFile.
2021-07-01 00:23:47 +03:00
Alexander Bantyev 811f3e8605
nix registry pin: add a way to pin to a custom locked 2021-06-30 22:16:40 +03:00
Alexander Bantyev 093ed47636
nix registry: add --registry flag 2021-06-30 22:13:32 +03:00
Pamplemousse 5be17a4b96 Allow to compile after ./configure --enable-gc=no
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-06-30 09:45:21 -07:00
Eelco Dolstra 139f7af5ec
Merge pull request #4958 from NixOS/nix-develop-remove-override-input-warning
develop: Discard the input{Overrides,Updates} when getting bash
2021-06-30 17:57:01 +02:00
Eelco Dolstra c92fbdb654
Merge pull request #4960 from NixOS/nix-repl-experimental-features
Forward the experimental features to the nix repl subprocesses
2021-06-29 15:10:02 +02:00
regnat 7daf0c6ef1 Forward the experimental features to the nix repl subprocesses
Pass the current experimental features using `NIX_CONFIG` to the various
Nix subprocesses that `nix repl` invokes.

This is quite a hack, but having `nix repl` call Nix with a subprocess
is a hack already, so I guess that’s fine.
2021-06-29 14:52:46 +02:00
Eelco Dolstra 5a18e2a533
Merge pull request #4959 from NixOS/override-non-flake-inputs
Keep the `isFlake` attribute for overriden inputs
2021-06-29 14:23:50 +02:00
regnat 4cff8188a5 Keep the isFlake attribute for overriden inputs
When overriding an input that isn’t a flake, mark the override as also not
being a flake.

Fix #3774
2021-06-29 13:56:48 +02:00
Eelco Dolstra f14c3b6f68
Merge pull request #4944 from hercules-ci/fix-gc-crash
Fix gc crash
2021-06-29 13:52:14 +02:00
regnat bf7960a4ed develop: Discard the input{Overrides,Updates} when getting bash
`nix develop` is getting bash from an (assumed existing) `nixpkgs`
flake. However, when doing so, it reuses the `lockFlags` passed to the
current flake, including the `--input-overrides` and `--input-update`
which generally don’t make sense anymore at that point (and trigger a
warning because of that)

Clear these overrides before getting the nixpkgs flake to get rid of the
warning.
2021-06-29 10:13:42 +02:00
Florian Franzen decc14d4b7
man: fix formatting of nix3-profile-remove 2021-06-28 16:27:03 +02:00
Eelco Dolstra 6182ae6898
Merge pull request #4942 from NixOS/ca/remove-lock-files
Eventually delete the CA paths lock files
2021-06-28 16:12:19 +02:00
Eelco Dolstra f5320299dd
Merge pull request #4937 from NixOS/ca/make-the-tests-useful
Make the CA tests actually test something
2021-06-28 16:06:49 +02:00
regnat 8b6fba2b63 Eventually delete the CA paths lock files
Mark the lockfiles as having to eventually be deleted so that they don’t
stay laying around in the store at the end of the build

Fix #4936
2021-06-24 15:45:05 +02:00
Maximilian Bosch 644415d391
Perform input rewrites only in LocalDerivationGoal 2021-06-24 15:25:21 +02:00
regnat 7746cb13dc Make CA derivations compatible with recursive Nix
Add an access-control list to the realisations in recursive-nix (similar
to the already existing one for store paths), so that we can build
content-addressed derivations in the restricted store.

Fix #4353
2021-06-24 14:53:10 +02:00
Robert Hensing bec83a6f95 BoehmGCStackAllocator: ignore stack protection page
This fixes a crash that looks like:

```
Thread 1 "nix-build" received signal SIGSEGV, Segmentation fault.
0x00007ffff7ad22a0 in GC_push_all_eager () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
(gdb) bt
0  0x00007ffff7ad22a0 in GC_push_all_eager () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
1  0x00007ffff7adeefb in GC_push_all_stacks () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
2  0x00007ffff7ad5ac7 in GC_mark_some () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
3  0x00007ffff7ad77bd in GC_stopped_mark () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
4  0x00007ffff7adbe3a in GC_try_to_collect_inner.part.0 () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
5  0x00007ffff7adc2a2 in GC_collect_or_expand () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
6  0x00007ffff7adc4f8 in GC_allocobj () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
7  0x00007ffff7adc88f in GC_generic_malloc_inner () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
8  0x00007ffff7ae1a04 in GC_generic_malloc_many () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
9  0x00007ffff7ae1c72 in GC_malloc_kind () from /nix/store/p1z58l18klf88iijpd0qi8yd2n9lhlk4-boehm-gc-8.0.4/lib/libgc.so.1
10 0x00007ffff7e003d6 in nix::EvalState::allocValue() () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
11 0x00007ffff7e04b9c in nix::EvalState::callPrimOp(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
12 0x00007ffff7e0a773 in nix::EvalState::callFunction(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
13 0x00007ffff7e0a91d in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
14 0x00007ffff7e0a8f8 in nix::ExprApp::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
15 0x00007ffff7e0e0e8 in nix::ExprOpNEq::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
16 0x00007ffff7e0d708 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
17 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
18 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
19 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
20 0x00007ffff7e0d695 in nix::ExprOpOr::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
21 0x00007ffff7e09e19 in nix::ExprOpNot::eval(nix::EvalState&, nix::Env&, nix::Value&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
22 0x00007ffff7e0a792 in nix::EvalState::callFunction(nix::Value&, nix::Value&, nix::Value&, nix::Pos const&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
23 0x00007ffff7e8cba0 in nix::addPath(nix::EvalState&, nix::Pos const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Value*, nix::FileIngestionMethod, std::optional<nix::Hash>, nix::Value&)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#1}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixexpr.so
24 0x00007ffff752e6f9 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
25 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
26 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
27 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
28 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
29 0x00007ffff752e8e2 in nix::dump(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nix::Sink&, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>&) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
30 0x00007ffff757f8c0 in void boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, nix::VirtualStackAllocator, boost::coroutines2::detail::pull_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::control_block::control_block<nix::VirtualStackAllocator, nix::sinkToSource(std::function<void (nix::Sink&)>, std::function<void ()>)::SinkToSource::read(char*, unsigned long)::{lambda(boost::coroutines2::detail::push_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&)#1}>(boost::context::preallocated, nix::VirtualStackAllocator&&, nix::sinkToSource(std::function<void (nix::Sink&)>, std::function<void ()>)::SinkToSource::read(char*, unsigned long)::{lambda(boost::coroutines2::detail::push_coroutine<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&)#1}&&)::{lambda(boost::context::fiber&&)#1}> >(boost::context::detail::transfer_t) () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libnixutil.so
31 0x00007ffff6f331ef in make_fcontext () from /nix/store/hzdzcv9d3bc8rlsaphh7x54zsf0x8nx6-nix-2.4pre20210601_5985b8b/lib/libboost_context.so.1.69.0
32 0x0000000000000000 in ?? ()
```
2021-06-24 12:20:39 +02:00
regnat be7a4a6a13 Make the post-build-hook also run for unresolved CA derivations
Fix #4837
2021-06-24 11:41:57 +02:00
Thomas Churchman e3d11f9a9c Improve machine store URI parsing 2021-06-23 22:09:22 +01:00
Alyssa Ross 4f80464645
Apply OS checks to host platform, not build
Previously, the build system used uname(1) output when it wanted to
check the operating system it was being built for, which meant that it
didn't take into-account cross-compilation when the build and host
operating systems were different.

To fix this, instead of consulting uname output, we consult the host
triple, specifically the third "kernel" part.

For "kernel"s with stable ABIs, like Linux or Cygwin, we can use a
simple ifeq to test whether we're compiling for that system, but for
other platforms, like Darwin, FreeBSD, or Solaris, we have to use a
more complicated check to take into account the version numbers at the
end of the "kernel"s.  I couldn't find a way to just strip these
version numbers in GNU Make without shelling out, which would be even
more ugly IMO.  Because these checks differ between kernels, and the
patsubst ones are quite fiddly, I've added variables for each host OS
we might want to check to make them easier to reuse.
2021-06-23 15:00:36 +00:00
Eelco Dolstra 0a535dd5ac
Merge pull request #4839 from NixOS/ca/gracefully-handle-duplicate-realisations
Gracefully handle duplicate realisations
2021-06-23 11:50:18 +02:00
Eelco Dolstra f9f773b332
Merge pull request #4908 from NixOS/ca/fix-nix-develop
Make `nix develop` work with CA derivations
2021-06-23 11:27:58 +02:00
regnat c878cee895 Assert that compatible realisations have the same dependencies
Should always hold, but that’s not necessarily obvious, so better
enforce it
2021-06-23 11:27:16 +02:00
regnat 16fb7d8d95 Display the diverging paths in case of a realisation mismatch 2021-06-23 11:27:16 +02:00
regnat 40f925b2da Fix indentation 2021-06-23 11:27:16 +02:00
regnat d32cf0c17a Gracefully ignore a substituter if it holds an incompatible realisation 2021-06-23 11:27:16 +02:00
regnat b8f7177a7b Properly fail when trying to register an incoherent realisation 2021-06-23 11:27:14 +02:00
Eelco Dolstra 7945055c63
Merge pull request #4842 from NixOS/ca/fix-nix-shell
Make `nix-shell` support content-addressed derivations
2021-06-23 11:26:22 +02:00
Eelco Dolstra 4a5aa1dbf6
Merge pull request #4838 from NixOS/ca/recursively-substitute-realisations
Recursively substitute the realisations
2021-06-23 10:33:25 +02:00
Eelco Dolstra 26d2c62225
Merge pull request #4906 from NixOS/collect-garbage-ca
Make `computeFSClosure` ca-aware
2021-06-23 10:07:08 +02:00
Eelco Dolstra d9a43d3137
Merge pull request #4905 from NixOS/ca-derivations-machine-feature
Add a ca-derivations required machine feature
2021-06-23 10:05:53 +02:00
Eelco Dolstra 3e4126b67c
Merge pull request #4926 from NixOS/ca/build-remote-signal-6
Fix the remote build of CA derivations
2021-06-23 10:04:35 +02:00
Eelco Dolstra 4f9508c3b5
Merge pull request #4836 from NixOS/ca/track-drvoutput-dependencies-2-le-retour
Track the dependencies of CA realisations
2021-06-23 10:03:31 +02:00
regnat ed0e21a88d Fix indentation 2021-06-23 08:16:34 +02:00
regnat 7c96a76dd7 Reformat the sql statements 2021-06-23 08:16:34 +02:00
Théophane Hufschmitt 8d09a4f9a0 Remove a useless string split
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
Théophane Hufschmitt c13d7d0b97 Pass more values by reference
Rather than copying them around everywhere

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-23 08:16:34 +02:00
Maximilian Bosch 6f206549ba
Move writeStructuredAttrsShell out of ParsedDerivation class 2021-06-22 21:14:20 +02:00
Maximilian Bosch 27ce722638
Prefix env vars for attrs.* files with NIX_ 2021-06-22 19:45:08 +02:00
Maximilian Bosch a92245b110
Remove now-obsolete typedef 2021-06-22 19:15:57 +02:00
Maximilian Bosch 3504c811a5
Add testcase for nix develop with __structuredAttrs 2021-06-22 19:15:57 +02:00
Maximilian Bosch f1e281c4fe
Split shell & json creation for build environments with structured attrs 2021-06-22 19:15:57 +02:00
Maximilian Bosch 447928bdb5
Fix usage of structured attrs for nix develop 2021-06-22 19:15:57 +02:00
Maximilian Bosch 3944a120ec
Set environment variables for .attrs.json & .attrs.sh
This way no derivation has to expect that these files are in the `cwd`
during the build. This is problematic for `nix-shell` where these files
would have to be inserted into the nix-shell's `cwd` which can become
problematic with e.g. recursive `nix-shell`.

To remain backwards-compatible, the location inside the build sandbox
will be kept, however using these files directly should be deprecated
from now on.
2021-06-22 19:15:57 +02:00
Maximilian Bosch 3b5429aec1
Source complete env in nix-shell with __structuredAttrs = true;
This is needed to push the adoption of structured attrs[1] forward. It's
now checked if a `__json` exists in the environment-map of the derivation
to be openend in a `nix-shell`.

Derivations with structured attributes enabled also make use of a file
named `.attrs.json` containing every environment variable represented as
JSON which is useful for e.g. `exportReferencesGraph`[2]. To
provide an environment similar to the build sandbox, `nix-shell` now
adds a `.attrs.json` to `cwd` (which is mostly equal to the one in the
build sandbox) and removes it using an exit hook when closing the shell.

To avoid leaking internals of the build-process to the `nix-shell`, the
entire logic to generate JSON and shell code for structured attrs was
moved into the `ParsedDerivation` class.

[1] https://nixos.mayflower.consulting/blog/2020/01/20/structured-attrs/
[2] https://nixos.org/manual/nix/unstable/expressions/advanced-attributes.html#advanced-attributes
2021-06-22 19:15:57 +02:00
regnat 3b58dbb356 nix-shell: Replace resolving failure error by an assertion
This shouldn’t happen in practice, so better make it explicit
2021-06-22 11:29:55 +02:00
regnat dcabb46124 Shorten a stupidly long sql query name 2021-06-21 16:28:06 +02:00
Eelco Dolstra 610baf359a
Merge pull request #4833 from NixOS/ca/json-realisations-in-worker-protocol
Always send the realisations as JSON
2021-06-21 16:12:17 +02:00
Eelco Dolstra 24e7353232
Merge pull request #4928 from NixOS/ca/remove-existing-invalid-store-path
Remove a possible existing store path when building CA derivations
2021-06-21 16:09:21 +02:00
Théophane Hufschmitt ce674cb2cf
Properly set the output env variables
Co-authored-by: John Ericson <git@JohnEricson.me>
2021-06-21 15:52:01 +02:00
regnat 608434722b Only symlink the requested outputs in nix build
Fix #4925
2021-06-21 15:47:47 +02:00
regnat 3784c66a46 Remove a possible existing store path when building CA derivations
In case a previous interrupted build left a garbage path laying around,
remove it before trying to move the path to its final location.

Fix #4858
2021-06-21 15:29:15 +02:00
regnat 498677cbed Fix the remote build of CA derivations
Make sure that the derivation we send to the remote builder is exactly
the one that we want to build locally so that the output ids are exactly
the same

Fix #4845
2021-06-21 14:18:33 +02:00
Eelco Dolstra 4202a3bc4e UDSRemoteStore: Support the 'root' store parameter
Useful when we're using a daemon with a chroot store, e.g.

  $ NIX_DAEMON_SOCKET_PATH=/tmp/chroot/nix/var/nix/daemon-socket/socket nix-daemon --store /tmp/chroot

Then the client can now connect with

  $ nix build --store unix:///tmp/chroot/nix/var/nix/daemon-socket/socket?root=/tmp/chroot nixpkgs#hello
2021-06-18 17:04:11 +02:00
Timothy DeHerrera 769ca4e26d
libfetchers/git: fetch submodules by default 2021-06-17 12:00:26 -06:00
Eelco Dolstra e6150de90d
nix develop: Filter out NIX_REMOTE
When recursive Nix is enabled, NIX_REMOTE is set to
unix:///build/.nix-socket, which doesn't work outside of the sandbox.
2021-06-15 12:06:01 +02:00
regnat 96d7170e12 Don’t check the deriver field on computeFSClosure
That doesn’t really make sense with CA derivations (and wasn’t even
really correct before because of FO derivations, though that probably
didn’t matter much in practice)
2021-06-12 12:24:53 +02:00
Ben Burdette edb5a28024 hintfmt for eye searing varnames 2021-06-11 18:55:40 -06:00
Ben Burdette 129dd760e6 mapEnvBindings 2021-06-11 18:55:15 -06:00
regnat 2cf591a134 Make nix develop work with CA derivations
Fix #4823
2021-06-11 13:35:13 +02:00
regnat 56605b4688 Make nix-shell support content-addressed derivations
Resolve the derivation before trying to load its environment −
essentially reproducing what the build loop does − so that we can
effectively access our dependencies (and not just their placeholders).

Fix #4821
2021-06-11 13:32:49 +02:00
regnat 7ac038fa4b Make computeFSClosure ca-aware
Fix #4820 by preventing nix-collect garbage from crashing if
`keep-outputs` or `keep-derivations` is true
2021-06-11 09:26:49 +02:00
regnat 7c077d2a0f Add a ca-derivations required machine feature
Make ca-derivations require a `ca-derivations` machine feature, and
ca-aware builders expose it.

That way, a network of builders can mix ca-aware and non-ca-aware
machines, and the scheduler will send them in the right place.
2021-06-11 09:12:53 +02:00
Ben Burdette d22de1dd0c remove dead code 2021-06-09 15:38:08 -06:00
Ben Burdette 93ca9381da formatting; string arg for throwTypeError 2021-06-08 18:37:28 -06:00
Ben Burdette ebf530d31e line endings 2021-06-08 18:17:58 -06:00
Ben Burdette a8df239752 highlight the extra vars 2021-06-08 14:44:53 -06:00
Ben Burdette ff2e72054f another throwTypeError form 2021-06-08 14:44:41 -06:00
keke 50dc88a56c
fix error: 'optional' in namespace 'std' does not name a template type 2021-06-02 18:09:03 +08:00
regnat 838f862f4f doc: Wrap at 80 characters 2021-06-02 11:26:04 +02:00
Théophane Hufschmitt 7565308d04
Fix a documentation typo
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-02 11:25:47 +02:00
regnat 7d651f5c3f throw_ -> reportError
This function might or might not throw depending on the value of
`keepGoing`, so naming it `throw_` was a bit confusing
2021-06-02 11:24:31 +02:00
regnat d12b12a15b Let nix flake check keep going when keep-going is set
When the `keep-going` option is set to `true`, make `nix flake check`
continue as much as it can before failing.

The UI isn’t perfect as-it-is as all the lines currently start with a
mostly useless `error (ignored): error:` prefix, but I’m not sure what
the best output would be, so I’ll leave it as-it-is for the time being

(This is a bit hijacking the `keep-going` flag as it’s supposed to be a
build-time only thing. But I think it’s faire to reuse it here).

Fix https://github.com/NixOS/nix/issues/4450
2021-06-02 11:13:12 +02:00
regnat 5985b8b527 Check the CA hash when importing stuff in the local store
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
2021-06-01 15:09:24 +02:00
Eelco Dolstra 48396d940e
Merge pull request #4866 from alyssais/libdl
Only link with libdl on Linux
2021-06-01 11:50:50 +02:00
Eelco Dolstra b8fbfc80fd
Merge pull request #4864 from jeremyschlatter/fix-typo
Fix typo in documentation
2021-06-01 11:49:46 +02:00
Eelco Dolstra caef6f4314
Merge pull request #4734 from p01arst0rm/fix-s3-ifdef
unified macro style for ENABLE_S3
2021-06-01 11:30:21 +02:00
Alyssa Ross c57ab17687
Only link with libdl on Linux
Linux is (as far as I know) the only mainstream operating system that
requires linking with libdl for dlopen.  On BSD, libdl doesn't exist,
so on non-FreeBSD BSDs linking will currently fail.  On macOS, it's
apparently just a symlink to libSystem (macOS libc), presumably
present for compatibility with things that assume Linux.

So the right thing to do here is to only add -ldl on Linux, not to add
it for everything that isn't FreeBSD.
2021-06-01 08:05:21 +00:00
Jeremy Schlatter f674f7f434
Fix typo in documentation 2021-06-01 00:05:34 -07:00
Timothy Klim 4da9ec772c Add .tar.zst support for TarballInputScheme 2021-05-29 16:03:26 +07:00
Eelco Dolstra b10256af51
Merge pull request #4849 from NixOS/ca/fix-nix-store--export
Make the Nar hash non modulo
2021-05-28 10:48:36 +02:00
regnat a22755721b Recursively substitute the realisations
Make sure that whenever we substitute a realisation, we also substitute
its entire closure
2021-05-26 18:44:17 +02:00
regnat 1f3ff0d193 Aso track the output path of the realisation dependencies 2021-05-26 17:09:21 +02:00
regnat cb46d70794 Add a db migration script 2021-05-26 16:59:09 +02:00
regnat 63ebfc73c5 Make copyPaths copy the whole realisations closure
Otherwise registering the realisations on the remote side might fail as
it now expects a complete closure
2021-05-26 16:59:09 +02:00
regnat 8c30acc3e8 Properly track the drvoutput references when building 2021-05-26 16:59:09 +02:00
regnat af3afd25ea Add a method to compute the closure of a realisation
Only considers the closure in term of `Realisation`, ignores all the
opaque inputs.

Dunno whether that’s the nicest solution, need to think it through a bit
2021-05-26 16:59:09 +02:00