Commit graph

54 commits

Author SHA1 Message Date
Rebecca Turner 06e11778b5 Merge "Rewrite REPL test parser" into main 2024-04-06 21:36:26 +00:00
Rebecca Turner ee423f391d Rewrite REPL test parser
- Use a recursive descent parser so that it's easy to extend.
- Add `@args` to enable customizing command-line arguments
- Add `@should-start` to enable `nix repl` tests that error before
  entering the REPL
- Make sure to read all stdout output before comparing. This catches
  some extra output we were tossing out before!

Change-Id: I5522555df4c313024ab15cd10f9f04e7293bda3a
2024-04-05 13:14:21 -07:00
eldritch horrors 38dc6f5b69 Revert "libutil: drop Pool resources on exceptional free"
This reverts commit de2884b82b376d10de5c400d8e73bc7d98f195d2.

Change-Id: I1fa301149d7c2ed3d266a40c15b2d010e12e44e6
2024-04-05 20:13:02 +00:00
Rebecca Turner ea10fe7ab0 Add EscapeStringOptions and escapeString tests
Change-Id: I86ead2f969c9e03c9edfa51bbc92ee06393fd7d6
2024-04-03 16:25:54 +00:00
jade 73507a7167 Merge changes Ib62d3d68,Ic3e7affe into main
* changes:
  Make things that can throw not noexcept anymore
  Fix various clang-tidy lints
2024-03-31 15:38:48 +00:00
eldritch horrors 862f20a4ba libutil: remove Pool::flushBad
this was never actually used, and bad design in the first place—why
should a bad resource be put back into the idle pool? just drop it.

Change-Id: Idab8774bee19dadae0209d404c4fb86dd4aeba1e
2024-03-31 00:07:09 +00:00
eldritch horrors 620de98d0c libutil: drop Pool resources on exceptional free
if a scope owning a resource does not gracefully drop that resource
while handling exceptions from deeper down the call stack we should
assume the resource is invalid state and drop it. currently it *is*
true that such cases do not cause resources to be freed, but thanks
to validator misuses this has so far not caused any larger problem.

Change-Id: Ie4f91bcd60a64d05c5ff9d22cc97954816d13b97
2024-03-30 23:40:45 +00:00
Rebecca Turner 2a98ba8b97 Add pre-commit checks
The big ones here are `trim-trailing-whitespace` and `end-of-file-fixer`
(which makes sure that every file ends with exactly one newline
character).

Change-Id: Idca73b640883188f068f9903e013cf0d82aa1123
2024-03-29 22:57:40 -07:00
jade 1fa6a3e335 Fix various clang-tidy lints
* some things that can throw are marked noexcept
  yet the linter seems to think not. Maybe they can't throw in practice.
  I would rather not have the UB possibility in pretty obvious cold
  paths.
* various default-case-missing complaints
* a fair pile of casts from integer to character, which are in fact
  deliberate.
* an instance of <https://clang.llvm.org/extra/clang-tidy/checks/bugprone/move-forwarding-reference.html>
* bugprone-not-null-terminated-result on handing a string to curl in
  chunks of bytes. our usage is fine.
* reassigning a unique_ptr by CRIMES instead of using release(), then
  using release() and ignoring the result. wild. let's use release() for
  its intended purpose.

Change-Id: Ic3e7affef12383576213a8a7c8145c27e662513d
2024-03-29 20:26:38 -07:00
Rebecca Turner a5a25894c1 Move escapeString to its own file
Change-Id: Ie5c954ec73c46c9d3c679ef99a83a29cc7a08352
2024-03-29 16:26:29 -07:00
Winter Cute 6646b80396 meson: add missing explicit dependency on nlohmann_json
Without this, the Meson setup won't bail out if nlohmann_json is
missing, leading to subpar DX (and maybe worse, but I'm not entirely
sure).

Change-Id: I5913111060226b540dcf003257c99a08e84da0de
2024-03-29 14:16:58 -04:00
Rebecca Turner 877750b7c5 Merge "Move DebugChar into its own file" into main 2024-03-29 16:20:14 +00:00
Rebecca Turner 236bc046ba Merge "Remove HintFmt::operator%" into main 2024-03-29 01:13:45 +00:00
Rebecca Turner 5ec2efb686 Move DebugChar into its own file
Change-Id: Ia40549e5d0b78ece8dd0722c3a5a032b9915f24b
2024-03-28 15:54:12 -07:00
Rebecca Turner 62332c1250 Merge "Move shell_words into its own file" into main 2024-03-28 22:49:00 +00:00
jade edba570664 HOT SALE: 15% off your build times!
This was achieved by running maintainers/buildtime_report.sh on the
build directory of a meson build, then asking "why the heck is json
eating our build times", and strategically moving the json using bits
out of widely included headers.

It turns out that putting literally any metrics whatsoever into the
build had immediate and predictable results.

Results are 1382.5s frontend time -> 1175.4s frontend time, back end
time approximately invariant.

Related: #159

Change-Id: I7edea95c8536203325c8bb4dae5f32d727a21b2d
2024-03-27 03:52:57 +00:00
Rebecca Turner aee3d639b5 Move shell_words into its own file
Change-Id: I34c0ebfb6dcea49bf632d8880e04075335a132bf
2024-03-26 16:44:04 -07:00
Rebecca Turner 8e63eca912 Remove HintFmt::operator%
Change-Id: Ibcf1a7848b4b18ec9b0807628ff229079ae7a0fe
2024-03-26 15:40:05 -07:00
Qyriad e1ffe56793 meson: implement unit tests
Unit tests can be run with `meson test -C build --suite check`.
`--suite check` is optional, as right now that's the only test suite,
but when functional tests are added those will be in a separate suite.

Change-Id: I7f22f1cde4b489b3cdb5f9a36a544f0c409fcc1f
2024-03-26 00:43:33 +01:00
jade 47a237f7ec Merge "Delete hasPrefix and hasSuffix from the codebase" into main 2024-03-18 12:01:39 -06:00
eldritch horrors 86a1121d16 use byte indexed locations for PosIdx
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).

this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.

since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.

notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).

(cherry picked from commit 5d9fdab3de0ee17c71369ad05806b9ea06dfceda)
Change-Id: Ie0b2430cb120c09097afa8c0101884d94f4bbf34
2024-03-18 16:12:46 +01:00
jade 61e21b2557 Delete hasPrefix and hasSuffix from the codebase
These now have equivalents in the standard lib in C++20. This change was
performed with a custom clang-tidy check which I will submit later.
Executed like so:

ninja -C build && run-clang-tidy -checks='-*,nix-*' -load=build/libnix-clang-tidy.so -p .. -fix ../tests | tee -a clang-tidy-result

Change-Id: I62679e315ff9e7ce72a40b91b79c3e9fc01b27e9
2024-03-17 20:17:19 -07:00
jade 886a418d23 builtins.nixVersion: return fixed fake version
This builtin is only going to cause us problems because we are not Nix,
so let's just falsify being in the 2.18 series, since that is the
closest target that has any meaning.

In future we might want to have a better feature detection mechanism,
for when we actually add stuff to some builtin's attr set argument. But
builtins.nixVersion is just going to be hopelessly broken and it should
be stubbed out.

Fixes #144

Change-Id: Id7390b32a29c6147f2977737d81846320de5d67e
2024-03-17 00:32:19 -07:00
jade 18ed6c3bdf Implement a repl characterization test system
This allows for automating using the repl without needing a PTY, with
very easy to write test files.

Change-Id: Ia8d7854edd91f93477638942cb6fc261354e6035
2024-03-15 12:31:16 -07:00
jade 38571c50e6 Implement a parser for a literate testing system for the repl
This parser can be reused for other purposes. It's inspired by
https://bitheap.org/cram/

Although eelco's impostor exists https://github.com/mobusoperandi/eelco,
it is not very nice to depend on out of tree testing frameworks with no
way to customize them.

Change-Id: Ifca50177e09730182baf0ebf829c3505bbb0274a
2024-03-14 14:30:38 -07:00
eldritch horrors 06952cf7c4 support <program>_ENV variables
this lets us set per-test-program environment variables rather than only
a single, global default. this was supported in nix originally but
might've gone partially missing in the upstream backports process?

Change-Id: Iad0919841b1b6d11e0b7ebd3920449a62f544e77
2024-03-13 19:48:26 +01:00
jade 1b8662b85c import the revisions to the characterization test framework from cppnix
This has some Flaws for sure (like, it is going to be a bit stretched to
use for repl characterization), but it is a start.

Change-Id: I258c8beb3aee236f45818a03be83bcda858120c9
2024-03-11 14:14:43 -07:00
eldritch horrors 1958152d14 Pretty-print values in the REPL
Pretty-print values in the REPL by printing each item in a list or
attrset on a separate line. When possible, single-item lists and
attrsets are printed on one line, as long as they don't contain a nested
list, attrset, or thunk.

Before:
```
{ attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
```

After:
```
{
  attrs = {
    a = {
      b = {
        c = { };
      };
    };
  };
  list = [ 1 ];
  list' = [
    1
    2
    3
  ];
}
```

(cherry picked from commit c0a15fb7d03dfb8f53bc6726c414bc88aa362592)
Change-Id: Ia2b41849165a5ddb63f7a8c272a2476b3e4292df
2024-03-09 07:20:23 -07:00
eldritch horrors b221a14f0a Merge pull request #9925 from 9999years/fmt-cleanup
Cleanup `fmt.hh`

(cherry picked from commit 47a1dbb4b8e7913cbb9b4d604728b912e76e4ca0)
Change-Id: Id076a45cb39652f437fe3f8bda10c310a9894777
2024-03-09 07:00:13 -07:00
eldritch horrors 08252967a8 libexpr: Support structured error classes
While preparing PRs like #9753, I've had to change error messages in
dozens of code paths. It would be nice if instead of

    EvalError("expected 'boolean' but found '%1%'", showType(v))

we could write

    TypeError(v, "boolean")

or similar. Then, changing the error message could be a mechanical
refactor with the compiler pointing out places the constructor needs to
be changed, rather than the error-prone process of grepping through the
codebase. Structured errors would also help prevent the "same" error
from having multiple slightly different messages, and could be a first
step towards error codes / an error index.

This PR reworks the exception infrastructure in `libexpr` to
support exception types with different constructor signatures than
`BaseError`. Actually refactoring the exceptions to use structured data
will come in a future PR (this one is big enough already, as it has to
touch every exception in `libexpr`).

The core design is in `eval-error.hh`. Generally, errors like this:

    state.error("'%s' is not a string", getAttrPathStr())
      .debugThrow<TypeError>()

are transformed like this:

    state.error<TypeError>("'%s' is not a string", getAttrPathStr())
      .debugThrow()

The type annotation has moved from `ErrorBuilder::debugThrow` to
`EvalState::error`.

(cherry picked from commit c6a89c1a1659b31694c0fbcd21d78a6dd521c732)
Change-Id: Iced91ba4e00ca9e801518071fb43798936cbd05a
2024-03-09 04:47:05 -07:00
eldritch horrors 7673312ccc Merge pull request #9928 from 9999years/error-messages-in-nix-repl
Improve error printing in `nix repl`

(cherry picked from commit a8050d9b83052e4b5c52bf2d116381aedec3a93e)
Change-Id: I588f92d1dd4c546c98788b71403cc034f5e7129a
2024-03-09 03:37:35 -07:00
eldritch horrors c864923928 Merge pull request #9929 from 9999years/dont-print-values-in-magenta
Don't print the first bracket in values in magenta in error messages

(cherry picked from commit 46a0625a40aef6946a35f92fdacf0e6b4a14414f)
Change-Id: I8435565c87db182116140eaeea9df1243e67ea94
2024-03-09 03:37:35 -07:00
eldritch horrors 609a8e0d94 Merge pull request #9754 from 9999years/print-value-when-coercion-fails
Print the value in `error: cannot coerce` messages

(cherry picked from commit 5b7bfd2d6b89d7dd5f54c1ca6c8072358d31a84e)

===

test taken from 6e8d5983143ae576e3f4b1d2954a5267f2943a49; it was added
previously (and not backported because its pr was a mostly-revert), but
it's useful to have around.

Change-Id: Icbd14b55e3610ce7b774667bf14b82e6dc717982
2024-03-09 00:05:41 -07:00
eldritch horrors 2f7c3fa251 Merge pull request #9818 from Ma27/print-value-on-function-call-type-error
libexpr: print value of what is attempted to be called as function
(cherry picked from commit 50e5d7b883042852538371237e32a66bb22f0485)
Change-Id: I7cb6290bd8f244e83bfce3b2eed2a4c8b4f16a83
2024-03-09 00:05:41 -07:00
eldritch horrors 87e6ac5eb7 Merge pull request #9753 from 9999years/print-value-on-type-error
Print the value in `value is X while a Y is expected` error

(cherry picked from commit 5f72a97092da6af28a7d2b2a50d74e9d34fae7e1)
Change-Id: Idb4bc903ae59a0f5b6fb3b1da4d47970fe0a6efe
2024-03-09 00:05:41 -07:00
eldritch horrors 512c1f05c3 Unify and refactor value printing
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).

This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.

Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)

Please read the tests for example output.

Future work:
- It would be nice to provide this function as a builtin, perhaps
  `builtins.toStringDebug` -- a printing function that never fails would
  be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
  command line, e.g. `--option to-string-max-attrs 1000`.

(cherry picked from commit 0fa08b451682fb3311fe58112ff05c4fe5bee3a4, )

===

Restore ambiguous value printer for `nix-instantiate`

The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.

(cherry picked from commit df84dd4d8dd3fd6381ac2ca3064432ab31a16b79)

Change-Id: I7cca6b4b53cd0642f2d49af657d5676a8554c9f8
2024-03-09 03:50:06 +01:00
eldritch horrors 8a268359b0 Merge pull request #9560 from obsidiansystems/serve-proto-unkeyed-valid-path-info-serializer
Factor out `ServeProto::Serialiser<UnkeyedValidPathInfo>` and test

(cherry picked from commit 139982997eec493a0f74105c427953f6be77da6d)
Change-Id: I28e4ba5a681a90d81915a56e6dbaa5456d64f96d
2024-03-07 12:37:33 +01:00
eldritch horrors 1342c8f18e Merge pull request #10074 from lf-/jade/ban-implicit-fallthrough
Warn on implicit switch case fallthrough

(cherry picked from commit 21282c3c204597641402c6bcff8fc9ee7bc31fa1)
Change-Id: I5ebbdfb6c037d2c55254f37dd391c07c2ce7443e
2024-03-07 00:11:12 -07:00
eldritch horrors 96a3683308 Ban building Nix with NDEBUG
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.

For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.

There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.

I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.

Upstream-PR: https://github.com/NixOS/nix/pull/10126
Original-Change-Id: I513cceaac1371decb3d96231e6ef9181c910c218
Change-Id: I531a51f6348a746e8e41d88203b08f614898356c
2024-03-04 09:24:58 +01:00
eldritch horrors 2bb0a06346 Merge pull request #9841 from obsidiansystems/float-speed-factor
Convert `Machine::speedFactor` from a non-neg int to a non-neg float

(cherry picked from commit 69d0ae27e376e7c7c4f237716b0149223b8a805a)
Change-Id: I2afb5cf9e4fe1384985c58353946135c3d102b42
2024-03-04 08:40:38 +01:00
eldritch horrors 7614aa9797 Merge pull request #4093 from matthewbauer/eval-system
Add eval-system option

(cherry picked from commit 071dbbee33af9f27338c3e53e4ea067dbfa14010)
Change-Id: Ia81358c8cfb60241da07a4d0e84b9ee62a18a53f
2024-03-04 07:21:01 +01:00
eldritch horrors c208e918e5 fix: nlohmann::adl_serializer for std::optional (#9147)
This allows templates such as `NLOHMANN_DEFINE_TYPE_*` templates and other generators with things like `std::vector<std::optional<T>>`.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
(cherry picked from commit 02bd821f2e71372d31bbe6700bd68086cc2ee70a)
Change-Id: I8b0ebcf2af4226610dadd565962f2d2327415a03
2024-03-04 07:11:25 +01:00
eldritch horrors 3f3badffc9 Merge pull request #9395 from nbraud/buitlins
builtins.concatMap: Fix typo in error message

(cherry picked from commit 4292d997568eb30503e287f98e24821ff0bc2816)
Change-Id: Ia33d1b02e41f699ef0c8c2d6487c9f70b2cc8cf4
2024-03-04 05:56:04 +01:00
eldritch horrors 5e182235cb Merge pull request #7348 from thufschmitt/dont-use-vlas
Remove the usage of VLAs in the code

(cherry picked from commit ac4431e9d016e62fb5dc9ae36833bd0c6cdadeec)
Change-Id: Ifbf5fbfc2e27122362a2aaea4b62c7cf3ca46b1a
2024-03-04 05:51:23 +01:00
eldritch horrors 7ff1dca1fa Merge pull request #9247 from obsidiansystems/derivation-test-with-files
Turn derivation unit tests into unit characterization tests

(cherry picked from commit a6e587923c9d5d716fe0f0049bed96d1cc210bff)
Change-Id: Ia2a2e65aabfee8d5d52142b8fdaacbae4a27242c
2024-03-04 05:21:10 +01:00
eldritch horrors 7f590ea709 Merge pull request #6223 from obsidiansystems/worker-proto-with-version
Give `nix daemon` and `nix-store --serve` protocols separate serializers with version info

(cherry picked from commit 8b68bbb77745fda0d14939b6c23d31cc89da41ce)
Change-Id: Ia3d3b9fbaf9f0ae62ab225020b7d14790e793655
2024-03-04 04:59:31 +01:00
Robert Hensing ab40b2c5d0 Merge pull request #9157 from obsidiansystems/protocol-versions
Add protocol versions to `{Worker,Serve}Proto::*Conn`

(cherry picked from commit 4d17c59d8d059a5b39f1d1da2b58f2ec8da44861)
Change-Id: I497af39deb792e50c157a1305d8c9e722798740b
2024-03-04 04:43:35 +01:00
eldritch horrors e12e9f2452 Merge pull request #9137 from obsidiansystems/serve-protocol
Introduce separate Serve protocol serialisers

(cherry picked from commit d070d8b7460f412a657745698dba291c66792402)
Change-Id: Ibcc8639e8997bcd07f7a5318330a147bcadc411b
2024-03-04 04:37:05 +01:00
eldritch horrors 6897e238bd Merge pull request #9099 from obsidiansystems/common-proto
Factor out bits of the worker protocol to use elsewhere

(cherry picked from commit 4b1a97338f517f45e6169d3d8845c5caa5724e97)
Change-Id: If93afa0f8b1cf9b0e705b34fa71e6fd708752758
2024-03-04 04:36:58 +01:00
eldritch horrors 7e966819b5 Merge pull request #9098 from obsidiansystems/test-proto
Enable most of the third `BuildResult` worker protocol test

(cherry picked from commit d344c112f772282bacacd4c66a75df4022d16e12)
Change-Id: I7b2b72aa84c19a6069f9c12128d901262db6f91c
2024-03-04 04:36:28 +01:00