Commit graph

9720 commits

Author SHA1 Message Date
Jacek Galowicz c924147c9d Drop parentheses from thunks 2024-01-16 17:00:46 +00:00
Jacek Galowicz 1885d579db Improve String Handling 2024-01-16 17:00:46 +00:00
Jacek Galowicz 0bc66e529f Use npos member variables instead of full type 2024-01-16 15:17:57 +00:00
Eelco Dolstra d005bade7f connect(): Propagate errno from the child process
This is necessary on macOS since addTempRoot() relies on errno.
2024-01-16 15:26:17 +01:00
Eelco Dolstra 0b1d93d2ba Sleep a bit between attempts to connect to the root server 2024-01-16 15:26:17 +01:00
Eelco Dolstra 302625e83b Merge remote-tracking branch 'origin/master' into handle-missing-gc-socket 2024-01-16 13:18:58 +01:00
Robert Hensing b2deff1947
Merge pull request #9747 from awakesecurity/mz/fix-quadratic-splitString
Fix performance of builtins.substring for empty substrings
2024-01-16 12:18:59 +01:00
John Ericson beed00c04e absPath: just take a std::string_view
1. Slightly more efficient

2. Easier to call

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2024-01-15 08:21:54 -05:00
Robert Hensing 268c49264a
Merge pull request #9763 from L-as/avoid-unnecessary-copy
Avoid unnecessary copy of goal log
2024-01-15 08:32:03 +01:00
Las Safin f61d951909
Avoid unnecessary copy of goal log
The data was (accidentally?) copied into a std::string,
even though the string is immediately converted into a std::string_view.
The code has been changed to construct a std::string_view directly,
such that one copy less happens.
2024-01-13 19:32:37 +00:00
John Ericson e739a5002d Avoid Windows macros in the parser and lexer
`FLOAT`, `INT`, and `IN` are identifers taken by macros.

The name `IN_KW` is chosen to match `OR_KW`, which is presumably named
that way for the same reason of dodging macros.
2024-01-12 19:51:36 -05:00
John Ericson c58da62a06
Merge pull request #9737 from obsidiansystems/sys-error-split
Separate `SystemError` from `SysError`
2024-01-12 12:41:36 -05:00
John Ericson 6208ca7209 Separate SystemError from SysError
Most of this is a `catch SysError` -> `catch SystemError` sed. This
is a rather pure-churn change I would like to get out of the way. **The
intersting part is `src/libutil/error.hh`.**

On Unix, we will only throw the `SysError` concrete class, which has
the same constructors that `SystemError` used to have.

On Windows, we will throw `WinError` *and* `SysError`. `WinError`
(which will be created in a later PR), will use a `DWORD` instead of
`int` error value, and `GetLastError()`, which is the Windows equivalent
of the `errno` machinery. Windows will *also* use `SysError` because
Window's "libc" (MSVCRT) implements the POSIX interface, and we use it
too.

As the docs describe, while we *throw* one of the 3 choices above (2
concrete classes or the alias), we should always *catch* `SystemError`.
This ensures no matter how the implementation changes for Windows (e.g.
between `SysError` and `WinError`) the catching logic stays the same
and stays correct.

Co-Authored-By volth <volth@volth.com>
Co-Authored-By Eugene Butler <eugene@eugene4.com>
2024-01-12 12:00:33 -05:00
Mel Zuser 25c889baac Fix performance of builtins.substring for empty substrings
When returning a 0-length substring, avoid calling coerceToString,
since it returns a string_view with the string's length, which is
expensive to compute for large strings.
2024-01-12 08:37:04 -08:00
Eelco Dolstra 52f949bbf5
Merge pull request #9656 from edolstra/nix-profile-stable-names
Make profile element names stable
2024-01-12 16:59:36 +01:00
Eelco Dolstra 0d55d660d5
Merge pull request #8043 from bobvanderlinden/pr-shell-env
nix: develop: use SHELL from rc script
2024-01-12 13:42:42 +01:00
Eelco Dolstra e21b3cf9db Merge remote-tracking branch 'origin/master' into nix-profile-stable-names 2024-01-12 13:36:27 +01:00
Eelco Dolstra 7c6f093abc .data() -> .c_str() to be on the safe side 2024-01-12 13:00:53 +01:00
Eelco Dolstra 66bd1b0298 Merge remote-tracking branch 'origin/master' into pr-shell-env 2024-01-12 12:56:26 +01:00
Eelco Dolstra 3e23759834 gc-non-blocking.sh: Add explanation
Also name the _NIX_TEST_GC_SYNC environment variables logically.
2024-01-12 12:38:55 +01:00
Eelco Dolstra 5703c31325 Merge remote-tracking branch 'origin/master' into handle-missing-gc-socket 2024-01-12 12:26:25 +01:00
Rebecca Turner df84dd4d8d
Restore ambiguous value printer for nix-instantiate
The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.
2024-01-11 16:34:36 -08:00
Rebecca Turner 0fa08b4516
Unify and refactor value printing
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).

This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.

Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)

Please read the tests for example output.

Future work:
- It would be nice to provide this function as a builtin, perhaps
  `builtins.toStringDebug` -- a printing function that never fails would
  be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
  command line, e.g. `--option to-string-max-attrs 1000`.
2024-01-11 16:34:36 -08:00
Rebecca Turner c9125603a5
Unindent print.hh declarations 2024-01-11 16:07:01 -08:00
John Ericson 84502674f8
Merge pull request #9736 from obsidiansystems/mingw-makefiles
Some small Makefile improvements to prepare for Windows support
2024-01-11 10:58:55 -05:00
John Ericson 113499d16f
Merge pull request #9634 from 9999years/combine-abstract-pos-and-pos
Combine `AbstractPos`, `PosAdapter`, and `Pos`
2024-01-11 10:47:27 -05:00
John Ericson 423484ad26 Only link with -pthread on Unix
We don't want this with MinGW.
2024-01-10 20:38:39 -05:00
John Ericson f9e5eb5f0a Make indentation in makesfiles consistent
Tab (as required) for rules, two spaces for `if`...`endif`.
2024-01-10 20:26:34 -05:00
John Ericson 2cea88dbc8 Improve build sytem support for readline instead of editline
Changes:

- CPP variable is now `USE_READLINE` not `READLINE`

- `configure.ac` supports with new CLI flag

- `package.nix` supports with new configuration option

- `flake.nix` CIs this (along with no markdown)

Remove old Ubuntu 16.04 stop-gap too, as that is now quite old.

Motivation:

- editline does not build for Windows, but readline *should*. (I am
  still working on this in Nixpkgs at this time, however. So there will
  be a follow-up Nix PR removing the windows-only skipping of the
  readline library once I am done.)

- Per
  https://salsa.debian.org/debian/nix/-/blob/master/debian/rules?ref_type=heads#L27
  and #2551, Debian builds Nix with readline. Now we better support and
  CI that build configuration.

This is picking up where #2551 left off, ensuring we test a few more
things not merely have CPP for them.

Co-authored-by: Weijia Wang <9713184+wegank@users.noreply.github.com>
2024-01-09 16:56:55 -05:00
Eelco Dolstra b91c935c2f
Merge pull request #9648 from cole-h/nix-shell-ordering
nix shell: reflect command line order in PATH order
2024-01-09 19:00:33 +01:00
John Ericson fa2a27d720
Merge pull request #9710 from wegank/disable-lowdown
Make lowdown optional
2024-01-08 15:13:45 -05:00
Weijia Wang bbd0a959e1 Make lowdown optional
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2024-01-08 20:37:42 +01:00
Rebecca Turner 4feb7d9f71
Combine AbstractPos, PosAdapter, and Pos
Also move `SourcePath` into `libutil`.

These changes allow `error.hh` and `error.cc` to access source path and
position information, which we can use to produce better error messages
(for example, we could consider omitting filenames when two or more
consecutive stack frames originate from the same file).
2024-01-08 10:59:41 -08:00
John Ericson ff6de4a9ee
Merge pull request #9662 from shlevy/flat-fixed-references-assert
Improve error message for fixed-outputs with references.
2024-01-08 10:46:37 -05:00
Eelco Dolstra faf87b51f7
Show why GC socket connection was refused
Co-authored-by: John Ericson <git@JohnEricson.me>
2024-01-08 14:14:36 +01:00
Shea Levy eeb2f083c5
Improve error message for fixed-outputs with references.
This codepath is possible, e.g. with a dockerTools.pullImage of an image with a Nix store.
2024-01-07 07:32:31 -05:00
Eelco Dolstra 359990dfdc
Merge pull request #9324 from 9999years/fix-8854-take-2
Don't attempt to `git add` ignored files
2024-01-05 14:57:32 +01:00
Eelco Dolstra dedbbbb451
Merge pull request #9670 from DavHau/log-lines
saner default for log-lines: change to 25
2024-01-05 14:39:50 +01:00
Eelco Dolstra 965cfe9688
Merge pull request #9687 from edolstra/withFramedSink-ctrl-c-hang
withFramedSink(): Receive interrupts on the stderr thread
2024-01-04 17:05:18 +01:00
John Ericson 12bb8cdd38 Signer infrastructure: Prep for #9076
This sets up infrastructure in libutil to allow for signing other than
by a secret key in memory. #9076 uses this to implement remote signing.

(Split from that PR to allow reviewing in smaller chunks.)

Co-Authored-By: Raito Bezarius <masterancpp@gmail.com>
2024-01-03 16:13:55 -05:00
Eelco Dolstra 295a2ff8bd Make some more threads receive interrupts
Shouldn't hurt to do this. In particular, this should speed up
shutting down the PathSubstitutionGoal thread if it's copying from a
remote store.
2024-01-03 19:30:02 +01:00
Eelco Dolstra 24e70489e5 withFramedSink(): Receive interrupts on the stderr thread
Otherwise Nix deadlocks when Ctrl-C is received in withFramedSink():
the parent thread will wait forever for the stderr thread to shut
down.

Fixes the hang reported in https://github.com/NixOS/nix/issues/7245#issuecomment-1770560923.
2024-01-03 19:14:50 +01:00
Eelco Dolstra 315aade89d
Merge pull request #9681 from edolstra/eval-optimisations
Optimize empty list constants
2024-01-03 10:43:01 +01:00
Eelco Dolstra 484881f302 Move empty list constant 2024-01-03 10:23:27 +01:00
John Ericson 2b20f36f95 Fix NetBSD build
There was still a mistake after my earlier
a7115a47ef and
e13fc0bbdb. This finally gets it right.
2024-01-02 12:33:51 -05:00
Eelco Dolstra 3f796514b3 Optimize empty list constants
This avoids a Value allocation for empty list constants. During a `nix
search nixpkgs`, about 82% of all thunked lists are empty, so this
removes about 3 million Value allocations.

Performance comparison on `nix search github:NixOS/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 --no-eval-cache`:

maximum RSS:        median = 3845432.0000  mean = 3845432.0000  stddev =      0.0000  min = 3845432.0000  max = 3845432.0000  [rejected?, p=0.00000, Δ=-70084.00000±0.00000]
soft page faults:   median = 965395.0000  mean = 965394.6667  stddev =      1.1181  min = 965392.0000  max = 965396.0000  [rejected?, p=0.00000, Δ=-17929.77778±38.59610]
system CPU time:    median =      1.8029  mean =      1.7702  stddev =      0.0621  min =      1.6749  max =      1.8417  [rejected, p=0.00064, Δ=-0.12873±0.09905]
user CPU time:      median =     14.1022  mean =     14.0633  stddev =      0.1869  min =     13.8118  max =     14.3190  [not rejected, p=0.03006, Δ=-0.18248±0.24928]
elapsed time:       median =     15.8205  mean =     15.8618  stddev =      0.2312  min =     15.5033  max =     16.1670  [not rejected, p=0.00558, Δ=-0.28963±0.29434]
2024-01-02 12:49:11 +01:00
Eelco Dolstra 3f834f5c64
Merge pull request #9657 from edolstra/fix-getNameFromURL
getNameFromURL(): Support uppercase characters in attribute names
2024-01-02 12:48:17 +01:00
Robert Hensing 83f5622545
Merge pull request #9658 from pennae/env-diet
reduce the size of Env by one pointer
2023-12-31 13:57:16 +01:00
Robert Hensing 3511430902
Merge pull request #9673 from pennae/drv-parse-opts
optimize derivation parsing
2023-12-31 13:49:03 +01:00
pennae 1fe66852ff reduce the size of Env by one pointer
since `up` and `values` are both pointer-aligned the type field will
also be pointer-aligned, wasting 48 bits of space on most machines. we
can get away with removing the type field altogether by encoding some
information into the `with` expr that created the env to begin with,
reducing the GC load for the absolutely massive amount of single-entry
envs we create for lambdas. this reduces memory usage of system eval by
quite a bit (reducing heap size of our system eval from 8.4GB to 8.23GB)
and gives similar savings in eval time.

running `nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'`

before:

  Time (mean ± σ):      5.576 s ±  0.003 s    [User: 5.197 s, System: 0.378 s]
  Range (min … max):    5.572 s …  5.581 s    10 runs

after:

  Time (mean ± σ):      5.408 s ±  0.002 s    [User: 5.019 s, System: 0.388 s]
  Range (min … max):    5.405 s …  5.411 s    10 runs
2023-12-30 18:55:13 +01:00