lix/doc/manual/rl-next/stack-overflow-segfaults.md
Rebecca Turner 7434caca05
Fix segfault on infinite recursion in some cases
This fixes a segfault on infinite function call recursion (rather than
infinite thunk recursion) by tracking the function call depth in
`EvalState`.

Additionally, to avoid printing extremely long stack traces, stack
frames are now deduplicated, with a `(19997 duplicate traces omitted)`
message. This should only really be triggered in infinite recursion
scenarios.

Before:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    Segmentation fault: 11

After:

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
    error: stack overflow

           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

    $ nix-instantiate --eval --expr '(x: x x) (x: x x)' --show-trace
    error:
           … from call site
             at «string»:1:1:
                1| (x: x x) (x: x x)
                 | ^

           … while calling anonymous lambda
             at «string»:1:2:
                1| (x: x x) (x: x x)
                 |  ^

           … from call site
             at «string»:1:5:
                1| (x: x x) (x: x x)
                 |     ^

           … while calling anonymous lambda
             at «string»:1:11:
                1| (x: x x) (x: x x)
                 |           ^

           … from call site
             at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^

           (19997 duplicate traces omitted)

           error: stack overflow
           at «string»:1:14:
                1| (x: x x) (x: x x)
                 |              ^
2023-12-29 22:16:44 -08:00

753 B

synopsis issues prs
Some stack overflow segfaults are fixed 9616 9617

The number of nested function calls has been restricted, to detect and report infinite function call recursions. The default maximum call depth is 10,000 and can be set with the max-call-depth option.

This fixes segfaults or the following unhelpful error message in many cases:

error: stack overflow (possible infinite recursion)

Before:

$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
Segmentation fault: 11

After:

$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
error: stack overflow

       at «string»:1:14:
            1| (x: x x) (x: x x)
             |              ^