[Nix#9942] Better support for printing large values in `nix repl` and error messages #61

New issue

Open

opened 2024-03-16 06:44:42 +00:00 by lix-bot · 0 comments

lix-bot commented

2024-03-16 06:44:42 +00:00

Member

Upstream-Issue: NixOS/nix#9942

Currently, some error messages will print values in a "limited" fashion, after PRs like #9753. Using the value printer refactor from #9606, Nix will print at most 10 attributes, 10 list items, and 1024 bytes of each string in error messages before giving up and printing a message like «20373 attributes elided» to indicate the missing values:

$ nix repl nixpkgs
Loading installable 'flake:nixpkgs#'...
Added 5 variables.
nix-repl> builtins.map (lib.id) legacyPackages.aarch64-darwin
error:
       … while calling the 'map' builtin
         at «string»:1:1:
            1| builtins.map (lib.id) legacyPackages.aarch64-darwin
             | ^

       … while evaluating the second argument passed to builtins.map

       error: expected a list but found a set: { _type = "pkgs"; AAAAAASomeThingsFailToEvaluate = «thunk»; AMB-plugins = «thunk»; ArchiSteamFarm = «thunk»; AusweisApp2 = «thunk»; BeatSaberModManager = «thunk»; CHOWTapeModel = «thunk»; ChowCentaur = «thunk»; ChowKick = «thunk»; ChowPhaser = «thunk»; «20373 attributes elided» }

nix repl doesn't currently use this output-limiting functionality, but instead sets a maximum depth (of 1 level) to print to, with the :p command used to print the entire structure:

$ nix repl
nix-repl> { a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z = 1; }
{ a = { ... }; }

nix-repl> :p { a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z = 1; }
{ a = { b = { c = { d = { e = { f = { g = { h = { i = { j = { k = { l = { m = { n = { o = { p = { q = { r = { s = { t = { u = { v = { w = { x = { y = { z = 1; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }

But the maximum depth of 1 still produces virtually unbounded output for large flat attribute sets like nixpkgs. Evaluating legacyPackages.aarch64-darwin in nix repl nixpkgs will happily spend 4 minutes printing 400,000 lines of output, and from experience seems rather uninterested in stopping when Ctrl-C is pressed.

Describe the solution you'd like

I would like to set nix repl to pretty-print (as in #9931) a modest number of maximum attributes, list items, and strings. The printer will track if any values were omitted, and this information will be used to format a tip for the user. Roughly like this:

$ nix repl nixpkgs
nix-repl> legacyPackages.aarch64-darwin
{
  _type = "pkgs";
  AAAAAASomeThingsFailToEvaluate = «thunk»;
  AMB-plugins = «thunk»;
  ArchiSteamFarm = «thunk»;
  AusweisApp2 = «thunk»;
  BeatSaberModManager = «thunk»;
  CHOWTapeModel = «thunk»;
  ChowCentaur = «thunk»;
  ChowKick = «thunk»;
  ChowPhaser = «thunk»;
  «20373 attributes elided»
}
# Note: Some attributes were elided. Print the whole value with `:p` or
# increase `--option max-attrs-print ...` to see more of the value.

(NB: In practice, we'd want to print a decent amount of attributes, maybe 100, before eliding any.)

Combined with #9922 and #9944, we could also add a REPL command to change the maximum values to be printed at runtime:

nix-repl> :set max-attrs-print 100
nix-repl> legacyPackages.aarch64-darwin  # Print the first 100 attributes.

This could be used to improve error ergonomics as well, by providing an opt-in for printing more of failing values:

error:
       … while calling the 'map' builtin
         at «string»:1:1:
            1| builtins.map (lib.id) legacyPackages.aarch64-darwin
             | ^

       … while evaluating the second argument passed to builtins.map

       error: expected a list but found a set: { _type = "pkgs"; AAAAAASomeThingsFailToEvaluate = «thunk»; AMB-plugins = «thunk»; ArchiSteamFarm = «thunk»; AusweisApp2 = «thunk»; BeatSaberModManager = «thunk»; CHOWTapeModel = «thunk»; ChowCentaur = «thunk»; ChowKick = «thunk»; ChowPhaser = «thunk»; «20373 attributes elided» }

       note: Some attributes were elided. To print more attributes, use
       `--option max-attrs-print ...`.

Additional context

Why does this functionality require setting handlers (#9922)?

It may not seem obvious, but this functionality requires setting handlers (#9922). This is because C++ initializes static values in an arbitrary and undefined order before executing the main function. In print-options.hh, we have some code like this:

static PrintOptions errorPrintOptions = PrintOptions {
    .maxAttrs = 10,
    // ...
};

If we naïvely added a setting like this:

Setting<size_t> maxAttrsPrint{this, 10, "max-attrs-print",
    "The maximum number of attributes to print before eliding."};

And updated the errorPrintOptions definition to match:

static PrintOptions errorPrintOptions = PrintOptions {
    .maxAttrs = evalSettings.maxAttrsPrint,
    // ...
};

Then the compiler may very well decide to initialize errorPrintOptions before evalSettings, leading to zero attributes being printed in error messages. (Worse, the compiler could initialize the two statics in different threads, leading to a non-deterministic number of attributes being printed.) With a setting handler, we could ask the max-attrs-print setting to simply update errorPrintOptions.maxAttrs when it's set.

The alternative, which I think is probably a good idea regardless of whether settings handlers are added, is to make setting values (and many other statics in the Nix codebase) functions, so they run in a predictable, programmer-controlled order, from within the main function.

Priorities

Add 👍 to issues you find important.

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/9942 ### Is your feature request related to a problem? Please describe. Currently, some error messages will print values in a "limited" fashion, after PRs like #9753. Using the value printer refactor from #9606, Nix will print at most 10 attributes, 10 list items, and 1024 bytes of each string in error messages before giving up and printing a message like `«20373 attributes elided»` to indicate the missing values: ``` $ nix repl nixpkgs Loading installable 'flake:nixpkgs#'... Added 5 variables. nix-repl> builtins.map (lib.id) legacyPackages.aarch64-darwin error: … while calling the 'map' builtin at «string»:1:1: 1| builtins.map (lib.id) legacyPackages.aarch64-darwin | ^ … while evaluating the second argument passed to builtins.map error: expected a list but found a set: { _type = "pkgs"; AAAAAASomeThingsFailToEvaluate = «thunk»; AMB-plugins = «thunk»; ArchiSteamFarm = «thunk»; AusweisApp2 = «thunk»; BeatSaberModManager = «thunk»; CHOWTapeModel = «thunk»; ChowCentaur = «thunk»; ChowKick = «thunk»; ChowPhaser = «thunk»; «20373 attributes elided» } ``` `nix repl` doesn't currently use this output-limiting functionality, but instead sets a maximum depth (of 1 level) to print to, with the `:p` command used to print the entire structure: ``` $ nix repl nix-repl> { a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z = 1; } { a = { ... }; } nix-repl> :p { a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.u.v.w.x.y.z = 1; } { a = { b = { c = { d = { e = { f = { g = { h = { i = { j = { k = { l = { m = { n = { o = { p = { q = { r = { s = { t = { u = { v = { w = { x = { y = { z = 1; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; }; } ``` But the maximum depth of 1 still produces virtually unbounded output for large _flat_ attribute sets like `nixpkgs`. Evaluating `legacyPackages.aarch64-darwin` in `nix repl nixpkgs` will happily spend 4 minutes printing 400,000 lines of output, and from experience seems rather uninterested in stopping when Ctrl-C is pressed. ### Describe the solution you'd like I would like to set `nix repl` to pretty-print (as in #9931) a modest number of maximum attributes, list items, and strings. The printer will track if any values were omitted, and this information will be used to format a tip for the user. Roughly like this: ``` $ nix repl nixpkgs nix-repl> legacyPackages.aarch64-darwin { _type = "pkgs"; AAAAAASomeThingsFailToEvaluate = «thunk»; AMB-plugins = «thunk»; ArchiSteamFarm = «thunk»; AusweisApp2 = «thunk»; BeatSaberModManager = «thunk»; CHOWTapeModel = «thunk»; ChowCentaur = «thunk»; ChowKick = «thunk»; ChowPhaser = «thunk»; «20373 attributes elided» } # Note: Some attributes were elided. Print the whole value with `:p` or # increase `--option max-attrs-print ...` to see more of the value. ``` (NB: In practice, we'd want to print a decent amount of attributes, maybe 100, before eliding any.) Combined with #9922 and #9944, we could also add a REPL command to change the maximum values to be printed at runtime: ``` nix-repl> :set max-attrs-print 100 nix-repl> legacyPackages.aarch64-darwin # Print the first 100 attributes. ``` This could be used to improve error ergonomics as well, by providing an opt-in for printing more of failing values: ``` error: … while calling the 'map' builtin at «string»:1:1: 1| builtins.map (lib.id) legacyPackages.aarch64-darwin | ^ … while evaluating the second argument passed to builtins.map error: expected a list but found a set: { _type = "pkgs"; AAAAAASomeThingsFailToEvaluate = «thunk»; AMB-plugins = «thunk»; ArchiSteamFarm = «thunk»; AusweisApp2 = «thunk»; BeatSaberModManager = «thunk»; CHOWTapeModel = «thunk»; ChowCentaur = «thunk»; ChowKick = «thunk»; ChowPhaser = «thunk»; «20373 attributes elided» } note: Some attributes were elided. To print more attributes, use `--option max-attrs-print ...`. ``` ### Additional context <details> <summary>Why does this functionality require setting handlers (#9922)?</summary> It may not seem obvious, but this functionality requires setting handlers (#9922). This is because [C++ initializes `static` values in an arbitrary and undefined order](https://en.cppreference.com/w/cpp/language/siof) before executing the `main` function. In `print-options.hh`, we have some code like this: ```c++ static PrintOptions errorPrintOptions = PrintOptions { .maxAttrs = 10, // ... }; ``` If we naïvely added a setting like this: ```cpp Setting<size_t> maxAttrsPrint{this, 10, "max-attrs-print", "The maximum number of attributes to print before eliding."}; ``` And updated the `errorPrintOptions` definition to match: ```c++ static PrintOptions errorPrintOptions = PrintOptions { .maxAttrs = evalSettings.maxAttrsPrint, // ... }; ``` Then the compiler may very well decide to initialize `errorPrintOptions` before `evalSettings`, leading to _zero_ attributes being printed in error messages. (Worse, the compiler could initialize the two `static`s in different threads, leading to a non-deterministic number of attributes being printed.) With a setting handler, we could ask the `max-attrs-print` setting to simply update `errorPrintOptions.maxAttrs` when it's set. The alternative, which I think is probably a good idea regardless of whether settings handlers are added, is to make setting values (and many other `static`s in the Nix codebase) functions, so they run in a predictable, programmer-controlled order, from within the `main` function. </details> ### Priorities Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).