Commit graph

15951 commits

Author SHA1 Message Date
John Ericson 15820a0507
Merge pull request #9720 from NixOS/fix-internal-api-docs
Fix Internal API docs
2024-01-08 16:33:38 -05:00
John Ericson fa2a27d720
Merge pull request #9710 from wegank/disable-lowdown
Make lowdown optional
2024-01-08 15:13:45 -05:00
John Ericson 29eb5ed1dc Fix Internal API docs
Because of source filtering, they were empty.

Fixes #9694
2024-01-08 14:47:42 -05:00
Weijia Wang bbd0a959e1 Make lowdown optional
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2024-01-08 20:37:42 +01:00
Rebecca Turner 4feb7d9f71
Combine AbstractPos, PosAdapter, and Pos
Also move `SourcePath` into `libutil`.

These changes allow `error.hh` and `error.cc` to access source path and
position information, which we can use to produce better error messages
(for example, we could consider omitting filenames when two or more
consecutive stack frames originate from the same file).
2024-01-08 10:59:41 -08:00
José Luis Lafuente 53fdcbca50
Add clang format configuration 2024-01-08 19:46:38 +01:00
Valentin Gagarin 6a243e5ed2
fix an old lost direct (#9458)
this part must have been moved quite a while ago, but apparently so far
no one noticed
2024-01-08 19:38:36 +01:00
John Ericson 3ecdda80be
Merge pull request #9709 from wegank/configure-ac-doc-gen
Fix typo in configure.ac
2024-01-08 12:19:36 -05:00
Weijia Wang 605eba3829 Fix typo in configure.ac 2024-01-08 17:31:27 +01:00
John Ericson ff6de4a9ee
Merge pull request #9662 from shlevy/flat-fixed-references-assert
Improve error message for fixed-outputs with references.
2024-01-08 10:46:37 -05:00
John Ericson c4c636284e
Only test bug fix with new enough deamon 2024-01-08 10:17:28 -05:00
Eelco Dolstra faf87b51f7
Show why GC socket connection was refused
Co-authored-by: John Ericson <git@JohnEricson.me>
2024-01-08 14:14:36 +01:00
Shea Levy eeb2f083c5
Improve error message for fixed-outputs with references.
This codepath is possible, e.g. with a dockerTools.pullImage of an image with a Nix store.
2024-01-07 07:32:31 -05:00
Valentin Gagarin 8e865f3aba
deduplicate installation instructions (#9507)
* deduplicate installation instructions

- reorder sections to present pinned installation more prominently
- remove outdated notes on the macOS installer rework
- update instructions to handle the installer tarball

Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
2024-01-06 22:45:25 +01:00
wiki-me a4d33e816e
Improve documentation around upgrading nix (#9679)
* Improve documentation around upgrading nix, add replacing nix channel with new one

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-01-06 18:01:10 +00:00
John Ericson 9651034dc2
Merge pull request #9698 from NixOS/no-boost-regex
Don't pull in libboost_regex
2024-01-05 09:49:13 -05:00
Eelco Dolstra 388c79d546 Don't pull in libboost_regex
We're not using and we don't want to pull in libicu (37 MiB).
2024-01-05 15:17:01 +01:00
Eelco Dolstra 359990dfdc
Merge pull request #9324 from 9999years/fix-8854-take-2
Don't attempt to `git add` ignored files
2024-01-05 14:57:32 +01:00
Eelco Dolstra dedbbbb451
Merge pull request #9670 from DavHau/log-lines
saner default for log-lines: change to 25
2024-01-05 14:39:50 +01:00
Eelco Dolstra cbf99c71c6
Merge pull request #9692 from 9999years/update-clang-stdenv
Remove `clang11Stdenv`
2024-01-05 10:11:44 +01:00
Rebecca Turner d8a2b06e20
Remove clang11Stdenv
Clang 11 doesn't have support for three-way-comparisons (<=>, "spaceship
operator", "consistent comparisons") and is older than `clangStdenv`.

`clangStdenv` is currently 12 on FreeBSD and Android and 16 on other
platforms:

32e718f00c/pkgs/top-level/all-packages.nix (L16629-L16644)

Let's start by removing Clang 11 from our distribution. Next we can
consider upgrading to Clang 17, which fully supports the spaceship
operator:

https://releases.llvm.org/17.0.1/tools/clang/docs/ReleaseNotes.html#what-s-new-in-clang-release
2024-01-04 11:43:56 -08:00
Robert Hensing 1ed245a606
Merge pull request #9690 from obsidiansystems/nixos-23.05-small-again
flake: Go back to regular `nixos-23.05-small`
2024-01-04 17:30:21 +01:00
Eelco Dolstra 965cfe9688
Merge pull request #9687 from edolstra/withFramedSink-ctrl-c-hang
withFramedSink(): Receive interrupts on the stderr thread
2024-01-04 17:05:18 +01:00
John Ericson 37ea1612c7 flake: Go back to regular nixos-23.05-small
Finally get off the ad-hoc staging commit!

Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/36c4ac09e9bebcec1fa7b7539cddb0c9e837409c' (2023-11-30)
  → 'github:NixOS/nixpkgs/2c9c58e98243930f8cb70387934daa4bc8b00373' (2023-12-31)
2024-01-04 10:45:00 -05:00
John Ericson 65fad9b84c
Merge pull request #9688 from obsidiansystems/signature-generalization-prep
Signer infrastructure: Prep for #9076
2024-01-03 17:25:37 -05:00
John Ericson 12bb8cdd38 Signer infrastructure: Prep for #9076
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.

(Split from that PR to allow reviewing in smaller chunks.)

Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
2024-01-03 16:13:55 -05:00
Eelco Dolstra 295a2ff8bd Make some more threads receive interrupts
Shouldn't hurt to do this. In particular, this should speed up
shutting down the PathSubstitutionGoal thread if it's copying from a
remote store.
2024-01-03 19:30:02 +01:00
Eelco Dolstra 24e70489e5 withFramedSink(): Receive interrupts on the stderr thread
Otherwise Nix deadlocks when Ctrl-C is received in withFramedSink():
the parent thread will wait forever for the stderr thread to shut
down.

Fixes the hang reported in https://github.com/NixOS/nix/issues/7245#issuecomment-1770560923.
2024-01-03 19:14:50 +01:00
Eelco Dolstra 315aade89d
Merge pull request #9681 from edolstra/eval-optimisations
Optimize empty list constants
2024-01-03 10:43:01 +01:00
Eelco Dolstra f5b97f1958
Merge pull request #9683 from obsidiansystems/fix-builds
Fix builds / CI jobs
2024-01-03 10:28:00 +01:00
Eelco Dolstra 484881f302 Move empty list constant 2024-01-03 10:23:27 +01:00
John Ericson 7b8af5f916 buildNoTests: Restore intent
The thing we wanted to test was that building Nix without building or
running tests, and without depending on libraries only needed by tests,
works.

But since 6c8f4ef350, we can also install
unit tests, and during the conversion to using `package.nix` this
started happening more often (they go to a separate output though, so
this should be fine).

This adds more `... = false` to restore the original intent: don't run
unit test or functional tests, and don't install unit tests.
2024-01-02 12:50:48 -05:00
John Ericson 86e9244437 Fix buildNoTest
`checkInputs` is not right for this because we don't just need these
deps when `doTest`, we also need them when `installUnitTests`.
2024-01-02 12:41:53 -05:00
John Ericson 2b20f36f95 Fix NetBSD build
There was still a mistake after my earlier
a7115a47ef and
e13fc0bbdb. This finally gets it right.
2024-01-02 12:33:51 -05:00
Eelco Dolstra 3f796514b3 Optimize empty list constants
This avoids a Value allocation for empty list constants. During a `nix
search nixpkgs`, about 82% of all thunked lists are empty, so this
removes about 3 million Value allocations.

Performance comparison on `nix search github:NixOS/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 --no-eval-cache`:

maximum RSS:        median = 3845432.0000  mean = 3845432.0000  stddev =      0.0000  min = 3845432.0000  max = 3845432.0000  [rejected?, p=0.00000, Δ=-70084.00000±0.00000]
soft page faults:   median = 965395.0000  mean = 965394.6667  stddev =      1.1181  min = 965392.0000  max = 965396.0000  [rejected?, p=0.00000, Δ=-17929.77778±38.59610]
system CPU time:    median =      1.8029  mean =      1.7702  stddev =      0.0621  min =      1.6749  max =      1.8417  [rejected, p=0.00064, Δ=-0.12873±0.09905]
user CPU time:      median =     14.1022  mean =     14.0633  stddev =      0.1869  min =     13.8118  max =     14.3190  [not rejected, p=0.03006, Δ=-0.18248±0.24928]
elapsed time:       median =     15.8205  mean =     15.8618  stddev =      0.2312  min =     15.5033  max =     16.1670  [not rejected, p=0.00558, Δ=-0.28963±0.29434]
2024-01-02 12:49:11 +01:00
Eelco Dolstra 3f834f5c64
Merge pull request #9657 from edolstra/fix-getNameFromURL
getNameFromURL(): Support uppercase characters in attribute names
2024-01-02 12:48:17 +01:00
Robert Hensing 83f5622545
Merge pull request #9658 from pennae/env-diet
reduce the size of Env by one pointer
2023-12-31 13:57:16 +01:00
Robert Hensing 3511430902
Merge pull request #9673 from pennae/drv-parse-opts
optimize derivation parsing
2023-12-31 13:49:03 +01:00
pennae 1fe66852ff reduce the size of Env by one pointer
since `up` and `values` are both pointer-aligned the type field will
also be pointer-aligned, wasting 48 bits of space on most machines. we
can get away with removing the type field altogether by encoding some
information into the `with` expr that created the env to begin with,
reducing the GC load for the absolutely massive amount of single-entry
envs we create for lambdas. this reduces memory usage of system eval by
quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB)
and gives similar savings in eval time.

running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'`

before:

  Time (mean ± σ):      5.576 s ±  0.003 s    [User: 5.197 s, System: 0.378 s]
  Range (min … max):    5.572 s …  5.581 s    10 runs

after:

  Time (mean ± σ):      5.408 s ±  0.002 s    [User: 5.019 s, System: 0.388 s]
  Range (min … max):    5.405 s …  5.411 s    10 runs
2023-12-30 18:55:13 +01:00
pennae c62686a95b reduce copies during drv parsing
many paths need not be heap-allocated, and derivation env name/valye
pairs can be moved into the map.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.883 s ±  0.016 s    [User: 5.250 s, System: 1.424 s]
  Range (min … max):    6.860 s …  6.905 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.868 s ±  0.027 s    [User: 5.194 s, System: 1.466 s]
  Range (min … max):    6.828 s …  6.913 s    10 runs
2023-12-30 18:44:15 +01:00
pennae 02c64abf1e use translation table for drv string parsing
the table is very small compared to cache sizes and a single indexed
load is faster than three comparisons.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.907 s ±  0.012 s    [User: 5.272 s, System: 1.429 s]
  Range (min … max):    6.893 s …  6.926 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.883 s ±  0.016 s    [User: 5.250 s, System: 1.424 s]
  Range (min … max):    6.860 s …  6.905 s    10 runs
2023-12-30 18:44:15 +01:00
pennae 79d3d412ca optimize derivation string parsing
a bunch of derivation strings contain no escape sequences. we can
optimize for this fact by first scanning for the end of a derivation
string and simply returning the contents unmodified if no escape
sequences were found. to make this even more efficient we can also use
BackedStringViews to avoid copies, avoiding heap allocations for
transient data.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
  Range (min … max):    6.926 s …  6.974 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.907 s ±  0.012 s    [User: 5.272 s, System: 1.429 s]
  Range (min … max):    6.893 s …  6.926 s    10 runs
2023-12-30 18:44:10 +01:00
Robert Hensing bf1b294bd8
Merge pull request #9617 from 9999years/stack-overflow-segfault
Fix segfault on infinite recursion in some cases
2023-12-30 10:40:25 +01:00
Rebecca Turner 7434caca05
Fix segfault on infinite recursion in some cases
This fixes a segfault on infinite function call recursion (rather than
infinite thunk recursion) by tracking the function call depth in
`EvalState`.

Additionally, to avoid printing extremely long stack traces, stack
frames are now deduplicated, with a `(19997 duplicate traces omitted)`
message. This should only really be triggered in infinite recursion
scenarios.

Before:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    Segmentation fault: 11

After:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    error: stack overflow

           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)' --show-trace
    error:
           … from call site
             at «string»:1:1:
                1| (x: x x) (x: x x)
                 | ^

           … while calling anonymous lambda
             at «string»:1:2:
                1| (x: x x) (x: x x)
                 |  ^

           … from call site
             at «string»:1:5:
                1| (x: x x) (x: x x)
                 |     ^

           … while calling anonymous lambda
             at «string»:1:11:
                1| (x: x x) (x: x x)
                 |           ^

           … from call site
             at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

           (19997 duplicate traces omitted)

           error: stack overflow
           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^
2023-12-29 22:16:44 -08:00
pennae 2cfc4ace35 malloc/memset even less
more buffers that can be uninitialized and on the stack. small
difference, but still worth doing.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
  Range (min … max):    6.943 s …  6.974 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
  Range (min … max):    6.926 s …  6.974 s    10 runs
2023-12-30 00:45:11 +01:00
pennae 99a691c8a1 don't use istreams in hot paths
istream sentry objects are very expensive for single-character
operations, and since we don't configure exception masks for the
istreams used here they don't even do anything. all we need is
end-of-string checks and an advancing position in an immutable memory
buffer, both of which can be had for much cheaper than istreams allow.

the effect of this change is most apparent on empty stores.

before:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      7.167 s ±  0.013 s    [User: 5.528 s, System: 1.431 s]
  Range (min … max):    7.147 s …  7.182 s    10 runs

after:

Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
  Range (min … max):    6.943 s …  6.974 s    10 runs
2023-12-30 00:45:10 +01:00
Eelco Dolstra a21c762dab
Merge pull request #9666 from unblevable/dervation-typo
Fix "dervation" typo
2023-12-27 21:52:37 +01:00
DavHau b6313f64f7 saner default for log-lines: change to 25
This seems to be a much saner default. 10 lines are just not enough in so many cases.
2023-12-27 19:57:27 +07:00
Brian Le e2399fc949 Change "dervation" typos to "derivation" 2023-12-26 17:12:28 -05:00
John Ericson e23983db82
Merge pull request #9661 from shlevy/ifd-buildStore
Build IFD in the build store when using eval-store.
2023-12-24 01:26:14 -05:00