Commit graph

2063 commits

Author SHA1 Message Date
Maximilian Bosch d3b41f3a7d tests: actually run mercurial tests
The binary to check for is called hg not hq.

Change-Id: I812a30f9347d5bf0573cdacc3fc887960887ee92
2024-05-04 16:48:07 +02:00
Maximilian Bosch 799d0132f0 Merge "Revert "Revert "Merge pull request #6621 from Kha/nested-follows""" into main 2024-05-04 08:52:29 +00:00
Maximilian Bosch 79d0ae6670 Merge "libstore/local-derivation-goal: prohibit creating setuid/setgid binaries" into main 2024-05-04 07:26:15 +00:00
Maximilian Bosch e91be79d8e tests/flakes/follow-paths: test that warning about non-existent input works recursively
When I added the warning that an input X has an override for a
non-existent input, the recursive flake input override fix wasn't
implemented yet[1].

This patch tests that both work together.

[1] https://github.com/NixOS/nix/pull/6663

Change-Id: I90dc032029b7160ab4a97d28c480c59d3a6f0150
2024-05-03 22:54:38 +00:00
Maximilian Bosch 0e38720502 Revert "Revert "Merge pull request #6621 from Kha/nested-follows""
This reverts commit a8b3d777fb.

This undoes the revert of PR#6621, which allows nested `follows`, i.e.

    {
      inputs = {
        foo.url = "github:bar/foo";
        foo.inputs.bar.inputs.nixpkgs = "nixpkgs";
      };
    }

does the expected thing now. This is useful to avoid the 1000 instances
of nixpkgs problem without having each flake in the dependency tree to
expose all of its transitive dependencies for modification.

This was in fact part of Nix before and the C++ changes applied w/o
conflicts. However, it got reverted then because people didn't want to
merge lazy-trees against it which was supposed to be merged soon back in
October 2022.

Fixes: lix-project/lix#201

Change-Id: I5ddef914135b695717b2ef88862d57ced5e7aa3c
2024-05-03 22:54:38 +00:00
Maximilian Bosch 045ee37438 libstore/local-derivation-goal: prohibit creating setuid/setgid binaries
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.

Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:

> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.

Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.

This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
is needed to have the correct syscall number available via
`__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:

    $ rg --pcre2 'define __NR_fchmodat2 (?!452)'
    sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
    58:#define __NR_fchmodat2 1073742276

    sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
    67:#define __NR_fchmodat2 6452

    sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
    62:#define __NR_fchmodat2 5452

    sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
    70:#define __NR_fchmodat2 4452

    sysdeps/unix/sysv/linux/alpha/arch-syscall.h
    59:#define __NR_fchmodat2 562

I added a small regression-test to the setuid integration-test that
attempts to set the suid bit on a file using the fchmodat2 syscall.
I confirmed that the test fails without the change in
local-derivation-goal.

Additionally, we require libseccomp 2.5.5 or greater now: as it turns
out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Lix against the project's flake.

To work around that

* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
  scheduled[3].

* the package now uses libseccomp 2.5.5 on its own already. This is to
  provide a quick fix since the correct fix for 23.11 is still a staging cycle
  away.

We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷

Relevant PRs in CppNix:

* https://github.com/NixOS/nix/pull/10591
* https://github.com/NixOS/nix/pull/10501

[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
[3] https://github.com/NixOS/nixpkgs/pull/306070

(cherry picked from commit ba6804518772e6afb403dd55478365d4b863c854)
Change-Id: I6921ab5a363188c6bff617750d00bb517276b7fe
2024-05-03 16:29:06 +02:00
Théophane Hufschmitt 8458d98b27 Rename nix show-config to nix config show
Part of #7672

My main motivation is to be able to use `nix.checkConfig`[1]. This
doesn't work with Lix currently since the module uses `nix show-config`
if the Nix version is <2.20pre and `nix config show` otherwise. I think
this is the only instance where nixpkgs checks for which Nix commands
exist that affects us now, so I figured we could just perform the rename
here as well[2] and still provide the current version number[3].

I don't have a strong opinion on whether to deprecate `nix show-config`,
the warning is added there automatically.

(cherry picked from commit f300e11b056dea414d7d77bbc6e5a7dc5d9ddd41)

[1] https://nixos.org/manual/nixos/stable/options.html#opt-nix.checkConfig
[2] I should add that I don't use the "official" ways of installing Lix
    because using the flake directly and callPackaging it seemed to fit
    better into my workflow: I already have a little mess to make
    sure Hydra from the flake uses the correct pkgs.nix and I didn't
    want to complicate it further while keeping a single package-set I
    can build in CI. Don't get me wrong, I think such a module for a
    quick-start is very important, just giving context on why I bother
    in the first place :)
[3] When we go public, I think it's worth considering to add support in
    nixpkgs itself for Lix.

Change-Id: I47b4239b05cbeda3c370d2fa56ea768b768768ac
2024-05-03 16:26:16 +02:00
eldritch horrors d55b158e24 libutil: make rewriteStrings sound
this is used in CA rewriting, replacement of placeholders in
derivations, generating scripts for devShells, and some more
places. in all of these transitive replacements are unsound,
and overlapping replacements would be as well. there even is
a test that transitive replacements do not happen (in the CA
RewriteSink suite), but none for overlapping replacements. a
minimally surprising binary rewriter surely would not do any
of these replacements, the only reason we have not seen this
break yet is probably that rewriteStrings is only called for
store paths and things that look like store paths (and those
should never overlap nor admit such transitive replacements)

Change-Id: I6fc29f939d5061d9f56c752624a823ece8437c07
2024-05-03 00:50:31 +00:00
Qyriad 076dfd30c6 Merge changes from topic "profile-v3" into main
* changes:
  nix3-profile: remove check "name" attr in manifests
  Add profile migration test
  nix3-profile: make element names stable
  getNameFromURL(): Support uppercase characters in attribute names
  nix3-profile: remove indices
  nix3-profile: allow using human-readable names to select packages
  implement parsing human-readable names from URLs
2024-05-02 20:15:48 +00:00
Eelco Dolstra 5d2031f92d Add profile migration test
(cherry picked from commit 72560f7bbef2ab3c02b8ca040fe084328bdd5fbe)

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Change-Id: I405e5848e2627a76940220fb6aebadfb8f094afb
2024-05-02 12:59:15 -06:00
Qyriad e0911eef73 nix3-profile: make element names stable
Based off of commit 6268a45b650f563bae2360e0540920a2959bdd40

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I0fcf069a8537c61ad6fc4eee1f3c193a708ea1c4
2024-05-02 12:59:15 -06:00
Eelco Dolstra ce70f02aff getNameFromURL(): Support uppercase characters in attribute names
In particular, this makes it handle 'legacyPackages' correctly.

(cherry picked from commit 936a3642264ac159f3f9093710be3465b70e0e89)

Upstream-PR: https://github.com/NixOS/nix/pull/9657
Change-Id: Icc4efe02f7f8e90a2970589f72fd3d3cd4418d95
2024-05-02 12:02:28 -06:00
Qyriad e98fc952a8 nix3-profile: remove indices
Based off of commit 3187bc9ac3dd193b9329ef68c73ac3cca794ed78

Upstream-PR: https://github.com/NixOS/nix/pull/9656
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Change-Id: I8ac4a33314cd1cf9de95404c20f58e883460acc7
2024-05-02 12:02:28 -06:00
Qyriad f88423813f nix3-profile: allow using human-readable names to select packages
These names are parsed from the URL provided for that package

Based off of commit 257b768436a0e8ab7887f9b790c5b92a7fe51ef5

Upstream-PR: https://github.com/NixOS/nix/pull/8678
Co-authored-by: Felix Uhl <felix.uhl@outlook.com>
Change-Id: I76d5f9cfb11d3d2915b3dd1db21d7bb49e91f4fb
2024-05-02 12:02:28 -06:00
Robert Hensing b7ce11c97d Disallow store path names that are . or .. (plus opt. -)
As discussed in the maintainer meeting on 2024-01-29.

Mainly this is to avoid a situation where the name is parsed and
treated as a file name, mostly to protect users.
.-* and ..-* are also considered invalid because they might strip
on that separator to remove versions. Doesn't really work, but that's
what we decided, and I won't argue with it, because .-* probably
doesn't seem to have a real world application anyway.
We do still permit a 1-character name that's just "-", which still
poses a similar risk in such a situation. We can't start disallowing
trailing -, because a non-zero number of users will need it and we've
seen how annoying and painful such a change is.

What matters most is preventing a situation where . or .. can be
injected, and to just get this done.

(cherry picked from commit f1b4663805a9dbcb1ace64ec110092d17c9155e0)
Change-Id: I900a8509933cee662f888c3c76fa8986b0058839
2024-05-02 19:34:38 +02:00
Robert Hensing 4b3dc66386 test: Generate distinct hashes
Gen::just is the constant generator. Don't just return that!

(cherry picked from commit 8406da28773f050e00a006e4812e3ecbf919a2a9)
Change-Id: Ibfd0bd40f90942077a4720086ce0cd3bfabef79d
2024-05-02 19:34:21 +02:00
Robert Hensing 2eec547d7d test: Generate distinct path names
Gen: :just is the constant generator. Don't just return that!

(cherry picked from commit 69bbd5852af9b2f0b794162bd1debcdf64fc6648)
Change-Id: Id6e58141f5a42a1f67bd11d48c87b32a3ebd0500
2024-05-02 19:34:04 +02:00
Robert Hensing fae6ae2122 parseStorePath: Support leading period
(cherry picked from commit b13e6a76b4f289c6db69ffaa7bd35b7e44f2a391)
Change-Id: Ie14be437d87d17248f80e1f009aa2a4311ddede6
2024-05-02 19:34:03 +02:00
Robert Hensing 3a058dc4b3 Revert "StorePath: reject names starting with '.'"
This reverts commit 24bda0c7b381e1a017023c6f7cb9661fae8560bd.

(cherry picked from commit 9ddd0f2af8fd95e1380027a70d0aa650ea2fd5e4)
Change-Id: Ideb547e2a8ac911cf39d58d3e0c1553867bdd776
2024-05-02 18:53:51 +02:00
Qyriad 1425aa0b7c implement parsing human-readable names from URLs
Based off of commit 257b768436a0e8ab7887f9b790c5b92a7fe51ef5

Upstream-PR: https://github.com/NixOS/nix/pull/8678
Co-authored-by: Felix Uhl <felix.uhl@outlook.com>
Change-Id: Idcb7f6191ca3310ef9dc854197f7798260c3f71d
2024-04-30 18:11:14 -06:00
Qyriad e2ab89a74b add VM test for nix upgrade-nix
This commit adds a new NixOS VM test, which tests that `nix upgrade-nix`
works on both kinds of profiles (manifest.nix and manifest.json).

Done as a separate commit from 831d18a13, since it relies on the
--store-path argument from 026c90e5f as well.

Change-Id: I5fc94b751d252862cb6cffb541a4c072faad9f3b
2024-04-29 01:19:21 +00:00
Maximilian Bosch 8773439a85 Merge "ssh-ng: Set log-fd for ssh to 4 by default" into main 2024-04-26 18:30:33 +00:00
Maximilian Bosch 104448e75d ssh-ng: Set log-fd for ssh to 4 by default
That's expected by `build-remote` and makes sure that errors are
correctly forwarded to the user. For instance, let's say that the
host-key of `example.org` is unknown and

    nix-build ../nixpkgs -A hello -j0 --builders 'ssh-ng://example.org'

is issued, then you get the following output:

    cannot build on 'ssh-ng://example.org?&': error: failed to start SSH connection to 'example.org'
    Failed to find a machine for remote build!
    derivation: yh46gakxq3kchrbihwxvpn5bmadcw90b-hello-2.12.1.drv
    required (system, features): (x86_64-linux, [])
    2 available machines:
    [...]

The relevant information (`Host key verification failed`) ends up in the
daemon's log, but that's not very obvious considering that the daemon
isn't very chatty normally.

This can be fixed - the same way as its done for legacy-ssh - by passing
fd 4 to the SSH wrapper. Now you'd get the following error:

    cannot build on 'ssh-ng://example.org': error: failed to start SSH connection to 'example.org': Host key verification failed.
    Failed to find a machine for remote build!
    [...]

...and now it's clear what's wrong.

Please note that this is won't end up in the derivation's log.

For previous discussion about this change see
https://github.com/NixOS/nix/pull/7659.

Change-Id: I5790856dbf58e53ea3e63238b015ea06c347cf92
2024-04-26 19:04:06 +02:00
Artemis Tosini c03de0df62 gc: Find roots using libproc on Darwin
Previously, the garbage collector found runtime roots on Darwin by
shelling out to `lsof -n -w -F n` then parsing the result.
However, this requires an lsof binary and can be extremely slow.

The official Apple lsof returns in a reasonable amount of time,
about 250ms in my tests, but the lsof packaged in nixpkgs is quite slow,
taking about 40 seconds to run the command.

Using libproc directly is about the same speed as Apple lsof,
and allows us to reënable several tests that were disabled on Darwin.

Change-Id: Ifa0adda7984e13c15535693baba835aae79a3577
2024-04-25 23:24:21 -04:00
Qyriad 7063170d5f tests: add error messages to the asserts in tarball flakes test
In hopes of avoiding opaque error messages like the one in
https://buildbot.lix.systems/#/builders/49/builds/1054/steps/1/logs/stdio

Traceback (most recent call last):
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/__init__.py", line 126, in main
    driver.run_tests()
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/driver.py", line 159, in run_tests
    self.test_script()
  File "/nix/store/wj6wh89jhd2492r781qsr09r9wydfs6m-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/driver.py", line 151, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 13, in <module>
AssertionError

Change-Id: Idd2212a1c3714ce58c7c3a9f34c2ca4313eb6d55
2024-04-22 16:13:36 -06:00
puck 272c2ff15f remove extraneous cache entry from github fetcher
This isn't necessary, as it's already covered by the tarball fetcher's
cache.

Change-Id: I85e35f5a61594f27b8f30d82145f92c5d6559e1f
2024-04-21 10:46:05 +00:00
eldritch horrors a326344253 tests: unhaunt the flakes nixos tests
these should really wait for networks to come up, otherwise they can fail.

fixes #235

Change-Id: I08989e8bdb0de280df74660ac43983de5c34fa9d
2024-04-18 20:09:19 +00:00
Ilya K 6d79aa3d70 Merge "libstore/build: set NO_NEW_PRIVS for the sandbox" into main 2024-04-16 05:33:41 +00:00
Qyriad 4e68deef80 abort with a descriptive message on bad HintFmt usage
Change-Id: Ic2f05572042343a8160fd971394372f5f2706fc4
2024-04-15 23:09:16 +00:00
Ilya K effc28f6f5 libstore/build: set NO_NEW_PRIVS for the sandbox
Change-Id: I711f64e2b68495ed9c85c1a4bd5025405805e43a
2024-04-15 10:25:29 +03:00
Ilya K b469c6509b libstore/build: just copy the magic /etc files into the sandbox
Saves us a bunch of thinking about how to handle symlinks, and prevents
the DNS config from changing on the fly under the build, which may or may
not be a good thing?

Change-Id: I071e6ae7e220884690b788d94f480866f428db71
2024-04-13 12:43:19 +03:00
Théophane Hufschmitt 07b627cc6d Merge pull request #10456 from NixOS/fixpermdeniedbind
Fix adding symlink to the sandbox paths

(cherry-picked from commit da1e977bf48cff2a635034c85e7c13878e38efc2)

Change-Id: I221c85a38180800ec6552d2e86a88df48398fad8
2024-04-11 15:43:58 +03:00
Rebecca Turner 9e7e927837 Print top-level errors normally in nix repl
Previously, errors while printing values in `nix repl` would be printed
in `«error: ...»` brackets rather than displayed normally:

```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
«error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.»
```

Now, errors will be displayed normally if they're emitted at the
top-level of an expression:

```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
error:
       … in the condition of the assert statement
         at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/lib/customisation.nix:268:17:
          267|     in commonAttrs // {
          268|       drvPath = assert condition; drv.drvPath;
             |                 ^
          269|       outPath = assert condition; drv.outPath;

       … in the left operand of the OR (||) operator
         at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/passthrufun.nix:28:45:
           27|         if lib.isDerivation value then
           28|           lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value
             |                                             ^
           29|         else

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.
```

Errors emitted in nested structures (like e.g. when printing `nixpkgs`)
will still be printed in brackets.

Change-Id: I25aeddf08c017582718cb9772a677bf51b9fc2ad
2024-04-09 08:34:40 -07:00
Rebecca Turner a95c62673b Merge "Add repl-overlays" into main 2024-04-09 05:29:48 +00:00
Rebecca Turner 727b43478c Add repl-overlays
Adds a `repl-overlays` option, which specifies files that can overlay
and modify the top-level bindings in `nix repl`. For example, with the
following contents in `~/.config/nix/repl.nix`:

    info: final: prev: let
      optionalAttrs = predicate: attrs:
        if predicate
        then attrs
        else {};
    in
      optionalAttrs (prev ? legacyPackages && prev.legacyPackages ? ${info.currentSystem})
      {
        pkgs = prev.legacyPackages.${info.currentSystem};
      }

We can run `nix repl` and use `pkgs` to refer to `legacyPackages.${currentSystem}`:

    $ nix repl --repl-overlays ~/.config/nix/repl.nix nixpkgs
    Lix 2.90.0
    Type :? for help.
    Loading installable 'flake:nixpkgs#'...
    Added 5 variables.
    Loading 'repl-overlays'...
    Added 6 variables.
    nix-repl> pkgs.bash
    «derivation /nix/store/g08b5vkwwh0j8ic9rkmd8mpj878rk62z-bash-5.2p26.drv»

Change-Id: Ic12e0f2f210b2f46e920c33088dfe1083f42391a
2024-04-08 17:11:47 -07:00
jade 54bc1f1b98 "but doctor, I AM the untrusted store": nix doctor had wrong trustedness
This probably snuck in in a refactor using truthiness or so. The
trustedness flag was having the optional fullness checked, rather than
the actual contained trust level.

Also adds some tests.

```
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: daemon
Version: 2.20.4
Trusted: 0
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix doctor
warning: 'doctor' is a deprecated alias for 'config check'
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
```

Fixes: lix-project/lix#232
Change-Id: I21576e2a0a755036edf8814133345987617ba3d0
2024-04-08 15:40:12 -07:00
Rebecca Turner 5ec256625e Merge "Don't run libstore unit tests in project root" into main 2024-04-08 22:08:34 +00:00
Rebecca Turner 12f2565f91 Don't run libstore unit tests in project root
This keeps the libstore unit tests from writing
`libstore-unit-tests.xml` to the project root.

Change-Id: I0d9909aabf9f3574cc1e72a5ae81daefba9a394b
2024-04-08 12:43:35 -07:00
Rebecca Turner 4a49a497eb Fix REPL test parser nits
Follow-up to https://gerrit.lix.systems/c/lix/+/546

Change-Id: Ie603f01e5520329bf879e061cea9e3fba45213fd
2024-04-08 11:14:34 -07:00
Rebecca Turner 4600eabdae Merge "Rename some parse failure tests to parse-fail-*" into main 2024-04-08 06:33:51 +00:00
Rebecca Turner 8ed8b9aef0 Merge "Add PathsSetting" into main 2024-04-08 06:31:11 +00:00
Robert Hensing b995c17f0e Merge pull request #10208 from 9999years/print-strings-directly
`:print` strings directly in `nix repl`

(cherry picked from commit 3539172fd2f7cee639ce46423c58beca4231f2db)
Change-Id: I1972f3bf3b56312851f38288509d371d37f21677
Upstream-PR: https://github.com/NixOS/nix/pull/10208
2024-04-07 19:10:43 -07:00
Rebecca Turner 6f863e8ccd Add PathsSetting
Change-Id: I1165f6ef033a5f757ca3716d3f8008ba36b01fd0
2024-04-07 18:22:15 -07:00
Rebecca Turner 06e11778b5 Merge "Rewrite REPL test parser" into main 2024-04-06 21:36:26 +00:00
Qyriad 4e11b0d04d Merge "meson: add missing tests: ca, dyn-drv, plugins, libstoreconsumer" into main 2024-04-06 01:05:09 +00:00
eldritch horrors e9e1b6963c Merge changes I1fa30114,I3ca208b6,Ide4c6e00,I74c46b9f,I05fa6a9d, ... into main
* changes:
  Revert "libutil: drop Pool resources on exceptional free"
  Revert "libutil: remove Pool::Handle::bad"
  Revert "libstore: remove one Resource::good flag"
  Revert "libstore: using throwing finally in withFramedSink"
  Revert "libutil: allow graceful dropping of Pool::Handle"
  Revert "libutil: drop Fs{Source,Sink}::good"
  libutil: guard Finally against invalid exception throws
2024-04-05 23:17:18 +00:00
Qyriad 405e41e288 Merge "meson: run repl characterization test" into main 2024-04-05 20:57:42 +00:00
Rebecca Turner ee423f391d Rewrite REPL test parser
- Use a recursive descent parser so that it's easy to extend.
- Add `@args` to enable customizing command-line arguments
- Add `@should-start` to enable `nix repl` tests that error before
  entering the REPL
- Make sure to read all stdout output before comparing. This catches
  some extra output we were tossing out before!

Change-Id: I5522555df4c313024ab15cd10f9f04e7293bda3a
2024-04-05 13:14:21 -07:00
eldritch horrors 38dc6f5b69 Revert "libutil: drop Pool resources on exceptional free"
This reverts commit de2884b82b376d10de5c400d8e73bc7d98f195d2.

Change-Id: I1fa301149d7c2ed3d266a40c15b2d010e12e44e6
2024-04-05 20:13:02 +00:00
Qyriad 07e1ddc23d meson: add missing tests: ca, dyn-drv, plugins, libstoreconsumer
Change-Id: I6a74ebaf93697cb99aadd6b51538c2766b0a808a
2024-04-04 20:43:38 -06:00
Rebecca Turner 0fec7d2db6 Rename some parse failure tests to parse-fail-*
These were mistakenly labeled `eval-fail-*`.

Note that the `lang.sh` runner passes `parse-fail-*` tests on stdin, so
filenames are removed from error messages.

Change-Id: I7f3a0d78b6cfa87af29aaa1b7af19d5a57fd4ade
2024-04-04 15:49:30 -07:00
Qyriad 3f4c7a68f4 meson: run repl characterization test
This was mistakenly left out of 69c3363f2.

Fixes #205.

Change-Id: Ib9a2088c9eee3a192623bee107ba32221d048640
2024-04-03 17:05:30 -06:00
Rebecca Turner 84018e1264 Make repl_characterization.cc constants string_views
Change-Id: I96455ee365799ae87d9ab433cf102c691616a45e
2024-04-03 16:25:54 +00:00
Rebecca Turner ea10fe7ab0 Add EscapeStringOptions and escapeString tests
Change-Id: I86ead2f969c9e03c9edfa51bbc92ee06393fd7d6
2024-04-03 16:25:54 +00:00
Théophane Hufschmitt 6c29016a09 Merge pull request #9920 from 9999years/forbid-nested-debuggers
Forbid nested debuggers

(cherry picked from commit e164b39ee90fd655dbb7f479fdd4fbe38cc883bd)
Change-Id: Iff62f40fd251116516a63e2d3f9fb5b21480b16d
2024-03-31 17:28:25 +00:00
jade 73507a7167 Merge changes Ib62d3d68,Ic3e7affe into main
* changes:
  Make things that can throw not noexcept anymore
  Fix various clang-tidy lints
2024-03-31 15:38:48 +00:00
eldritch horrors 862f20a4ba libutil: remove Pool::flushBad
this was never actually used, and bad design in the first place—why
should a bad resource be put back into the idle pool? just drop it.

Change-Id: Idab8774bee19dadae0209d404c4fb86dd4aeba1e
2024-03-31 00:07:09 +00:00
eldritch horrors 620de98d0c libutil: drop Pool resources on exceptional free
if a scope owning a resource does not gracefully drop that resource
while handling exceptions from deeper down the call stack we should
assume the resource is invalid state and drop it. currently it *is*
true that such cases do not cause resources to be freed, but thanks
to validator misuses this has so far not caused any larger problem.

Change-Id: Ie4f91bcd60a64d05c5ff9d22cc97954816d13b97
2024-03-30 23:40:45 +00:00
Rebecca Turner 2a98ba8b97 Add pre-commit checks
The big ones here are `trim-trailing-whitespace` and `end-of-file-fixer`
(which makes sure that every file ends with exactly one newline
character).

Change-Id: Idca73b640883188f068f9903e013cf0d82aa1123
2024-03-29 22:57:40 -07:00
jade 1fa6a3e335 Fix various clang-tidy lints
* some things that can throw are marked noexcept
  yet the linter seems to think not. Maybe they can't throw in practice.
  I would rather not have the UB possibility in pretty obvious cold
  paths.
* various default-case-missing complaints
* a fair pile of casts from integer to character, which are in fact
  deliberate.
* an instance of <https://clang.llvm.org/extra/clang-tidy/checks/bugprone/move-forwarding-reference.html>
* bugprone-not-null-terminated-result on handing a string to curl in
  chunks of bytes. our usage is fine.
* reassigning a unique_ptr by CRIMES instead of using release(), then
  using release() and ignoring the result. wild. let's use release() for
  its intended purpose.

Change-Id: Ic3e7affef12383576213a8a7c8145c27e662513d
2024-03-29 20:26:38 -07:00
Rebecca Turner a5a25894c1 Move escapeString to its own file
Change-Id: Ie5c954ec73c46c9d3c679ef99a83a29cc7a08352
2024-03-29 16:26:29 -07:00
Winter Cute 6646b80396 meson: add missing explicit dependency on nlohmann_json
Without this, the Meson setup won't bail out if nlohmann_json is
missing, leading to subpar DX (and maybe worse, but I'm not entirely
sure).

Change-Id: I5913111060226b540dcf003257c99a08e84da0de
2024-03-29 14:16:58 -04:00
Rebecca Turner 877750b7c5 Merge "Move DebugChar into its own file" into main 2024-03-29 16:20:14 +00:00
eldritch horrors 86b954a7af meson: increase functional test timeout
sometimes these fail with timeouts on loaded machines. let's up the
timeouts until we can pull the tests apart to more reasonable sizes

Change-Id: I2dfff2183cc1f3ff5e6107f43748ac046fe00d05
2024-03-29 02:19:36 +00:00
Rebecca Turner 236bc046ba Merge "Remove HintFmt::operator%" into main 2024-03-29 01:13:45 +00:00
Rebecca Turner 5ec2efb686 Move DebugChar into its own file
Change-Id: Ia40549e5d0b78ece8dd0722c3a5a032b9915f24b
2024-03-28 15:54:12 -07:00
Rebecca Turner 62332c1250 Merge "Move shell_words into its own file" into main 2024-03-28 22:49:00 +00:00
Qyriad 038daad218 meson: implement functional tests
Functional tests can be run with
`meson test -C build --suite installcheck`.

Notably, functional tests must be run *after* running `meson install`
(Lix's derivation runs the installcheck suite in installCheckPhase so it
does this correctly), due to some quirks between Meson and the testing
system.

As far as I can tell the functional tests are meant to be run after
installing anyway, but unfortunately I can't transparently make
`meson test --suite installcheck` depend on the install targets.

The script that runs the functional tests, meson/run-test.py, checks
that `meson install` has happened and fails fast with a (hopefully)
helpful error message if any of the functional tests are run before
installing.

TODO: this change needs reflection in developer documentation

Change-Id: I8dcb5fdfc0b6cb17580973d24ad930abd57018f6
2024-03-27 18:37:50 -06:00
jade edba570664 HOT SALE: 15% off your build times!
This was achieved by running maintainers/buildtime_report.sh on the
build directory of a meson build, then asking "why the heck is json
eating our build times", and strategically moving the json using bits
out of widely included headers.

It turns out that putting literally any metrics whatsoever into the
build had immediate and predictable results.

Results are 1382.5s frontend time -> 1175.4s frontend time, back end
time approximately invariant.

Related: lix-project/lix#159

Change-Id: I7edea95c8536203325c8bb4dae5f32d727a21b2d
2024-03-27 03:52:57 +00:00
Rebecca Turner aee3d639b5 Move shell_words into its own file
Change-Id: I34c0ebfb6dcea49bf632d8880e04075335a132bf
2024-03-26 16:44:04 -07:00
Rebecca Turner 8e63eca912 Remove HintFmt::operator%
Change-Id: Ibcf1a7848b4b18ec9b0807628ff229079ae7a0fe
2024-03-26 15:40:05 -07:00
Qyriad 207d24da4e Merge "meson: implement unit tests" into main 2024-03-26 20:06:09 +00:00
Qyriad e1ffe56793 meson: implement unit tests
Unit tests can be run with `meson test -C build --suite check`.
`--suite check` is optional, as right now that's the only test suite,
but when functional tests are added those will be in a separate suite.

Change-Id: I7f22f1cde4b489b3cdb5f9a36a544f0c409fcc1f
2024-03-26 00:43:33 +01:00
Théophane Hufschmitt 86881226b0 Merge pull request #8817 from iFreilicht/flake-update-lock-overhaul
Overhaul `nix flake update` and `nix flake lock` UX

(cherry picked from commit 12a0ae73dbb37becefa5a442eb4532ff0de9ce65)
Change-Id: Iff3b4f4235ebb1948ec612036b39ab29e4ca22b2
2024-03-25 17:36:24 -06:00
Tom Bereknyei 4494f9097f feat: notation to refer to no attribute search prefix
An attrPath prefix of "." indicates no need to try default attrPath prefixes. For example `nixpkgs#legacyPackages.x86_64-linux.ERROR` searches through

```
trying flake output attribute 'packages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute ''
trying flake output attribute 'legacyPackages.x86_64-linux.legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
trying flake output attribute 'legacyPackages.x86_64-linux.ERROR'
using cached attrset attribute 'legacyPackages.x86_64-linux'
```

And there is no way to specify that one does not want the automatic
search behavior. Now one can specify
`nixpkgs#.legacyPackages.x86_64-linux.ERROR` to only refer to the rooted
attribute path without any default injection of attribute search path or
system.

Change-Id: Iac1334e1470137b7ce11dcf845513810230638ec
(cherry picked from commit d4aed18883b361133607296fb6cd789c47427a38)
2024-03-25 15:30:36 +00:00
Lunaphied 185ecf1f45 Improve new CLI UX by supporting short -E flag for --expr
Change-Id: I55881c846da8416a92a14deedfa5bbbf09a122fb
2024-03-24 21:17:51 -06:00
eldritch horrors afb839a0c9 libexpr: associate let exprs with the correct StaticEnv
static env association is from expr to its enclosing scope, but let
exprs set their association to their *inner* scope. this skips one level
of envs and will cause segfaults if the parent is a with expr.

fixes #145

Change-Id: I1d22146110f071ede21b4eed7ed34b5850ef2ef3
2024-03-18 14:15:22 -07:00
eldritch horrors b3599166ad libexpr: sort binding name in debugger
not doing this exposes the binding name order to the annoying
interference of parse order on symbol order, which wouldn't be so bad if
it didn't make the tests less reliable and, importantly, dependent on
linker behavior (due to primop initialization being done in static
initializer, and the order of static initializers being defined only
within a single translation unit).

fixes #143

Change-Id: I3cf417893fbcf19e9ad3ff8986deb7cbcf3ca511
2024-03-18 20:03:31 +01:00
jade 47a237f7ec Merge "Delete hasPrefix and hasSuffix from the codebase" into main 2024-03-18 12:01:39 -06:00
eldritch horrors 86a1121d16 use byte indexed locations for PosIdx
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).

this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.

since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.

notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).

(cherry picked from commit 5d9fdab3de0ee17c71369ad05806b9ea06dfceda)
Change-Id: Ie0b2430cb120c09097afa8c0101884d94f4bbf34
2024-03-18 16:12:46 +01:00
eldritch horrors c39150e6bb diagnose "unexpected EOF" at EOF
this needs a string comparison because there seems to be no other way to
get that information out of bison. usually the location info is going to
be correct (pointing at a bad token), but since EOF isn't a token as
such it'll be wrong in that this case.

this hasn't shown up much so far because a single line ending *is* a
token, so any file formatted in the usual manner (ie, ending in a line
ending) would have its EOF position reported correctly.

(cherry picked from commit 855fd5a1bb781e4f722c1d757ba43e866d370132)
Change-Id: I120c56a962f4286b1ae3b71da7b71ce8ec3e0535
2024-03-18 16:12:46 +01:00
eldritch horrors 4c072c7c5f match line endings used by parser and error reports
the parser treats a plain \r as a newline, error reports do not. this
can lead to interesting divergences if anything makes use of this
feature, with error reports pointing to wrong locations in the input (or
even outside the input altogether).

(cherry picked from commit 2be6b143289e5479cc4a2667bb84e879116c2447)
Change-Id: Ieb7f7655bac8cb0cf5734c60bd41723388f2973c
2024-03-18 16:12:46 +01:00
eldritch horrors 9cf92c012d report inherit attr errors at the duplicate name
previously we reported the error at the beginning of the binding
block (for plain inherits) or the beginning of the attr list (for
inherit-from), effectively hiding where exactly the error happened.

this also carries over to runtime positions of attributes in sets as
reported by unsafeGetAttrPos. we're not worried about this changing
observable eval behavior because it *is* marked unsafe, and the new
behavior is much more useful.

(cherry picked from commit 1edd6fada53553b89847ac3981ac28025857ca02)
Change-Id: I2f50eb9f3dc3977db4eb3e3da96f1cb37ccd5174
2024-03-18 16:12:45 +01:00
eldritch horrors d826427f02 normalize formal order on ExprLambda::show
we already normalize attr order to lexicographic, doing the same for
formals makes sense. doubly so because the order of formals would
otherwise depend on the context of the expression, which is not quite as
useful as one might expect.

(cherry picked from commit 4147ecfb1c51f3fe3b4adcbd4e753fd487dab645)
Change-Id: I3fd0dbdef3ac7447a3a03ff20bb514a0d0f23fb1
2024-03-18 07:56:34 -06:00
eldritch horrors 314f044c2b keep copies of parser inputs that are in-memory only
the parser modifies its inputs, which means that sharing them between
the error context reporting system and the parser itself can confuse the
reporting system. usually this led to early truncation of error context
reports which, while not dangerous, can be quite confusing.

(cherry picked from commit d384ecd553aa997270b79ee98d02f7cf7e1849e6)
Change-Id: I677646b5675b12b2faa787943646aa36dc6e6ee3
2024-03-18 07:56:23 -06:00
jade 61e21b2557 Delete hasPrefix and hasSuffix from the codebase
These now have equivalents in the standard lib in C++20. This change was
performed with a custom clang-tidy check which I will submit later.
Executed like so:

ninja -C build && run-clang-tidy -checks='-*,nix-*' -load=build/libnix-clang-tidy.so -p .. -fix ../tests | tee -a clang-tidy-result

Change-Id: I62679e315ff9e7ce72a40b91b79c3e9fc01b27e9
2024-03-17 20:17:19 -07:00
jade 886a418d23 builtins.nixVersion: return fixed fake version
This builtin is only going to cause us problems because we are not Nix,
so let's just falsify being in the 2.18 series, since that is the
closest target that has any meaning.

In future we might want to have a better feature detection mechanism,
for when we actually add stuff to some builtin's attr set argument. But
builtins.nixVersion is just going to be hopelessly broken and it should
be stubbed out.

Fixes lix-project/lix#144

Change-Id: Id7390b32a29c6147f2977737d81846320de5d67e
2024-03-17 00:32:19 -07:00
eldritch horrors 11f35afa6f diagnose duplicated attrs at correct path
diagnose attr duplication at the path the duplication was detected, not
at the path the current attribute wanted to place. doing the latter is
only correct if a leaf attribute was duplicated, not if an attrpath was
set to a non-attrset in one binding and a (potentially implied) attrset
in another binding.

fixes #124

Change-Id: Ic4aa9cc12a9874d4e7897c6f64408f10aa36fc82
2024-03-16 22:12:49 +01:00
jade 0d85875c3a Allow dlopen of plugins to fail
It happens with some frequency that plugins that might be unimportant to
the evaluation at hand mismatch with the nix version, leading to
spurious load failures. Let's make these non fatal.

Change-Id: Iba10e951d171725ccf1a121bcd9be1e1d6ad69eb
2024-03-15 12:31:16 -07:00
jade 7d361f1a82 Test that :st does ... something
Change-Id: I97c00b5eb1288f68d8c2b484436cc185d040b8b2
2024-03-15 12:31:16 -07:00
jade af066af7f3 repl_characterization: Also verify the stack trace exists
Change-Id: I8b2d8211a24011fae1586a1182d7d0772a039cd7
2024-03-15 12:31:16 -07:00
jade 78513b1fc8 repl_characterization: eat newlines after commands and source-dir paths
This is because they are unrepresentable in the source files with
commentary but not in the output, so we should just eat them in
normalization. It's ok.

Change-Id: I2cb7e8b3fc7b00874885bb287cbaa200b41cb16b
2024-03-15 12:31:16 -07:00
jade 8a8715af89 Add regression tests for #9917, #9918
Change-Id: Ib0591e1499c5dba5e5a83ee75a899c9d16986827
2024-03-15 12:31:16 -07:00
jade 18ed6c3bdf Implement a repl characterization test system
This allows for automating using the repl without needing a PTY, with
very easy to write test files.

Change-Id: Ia8d7854edd91f93477638942cb6fc261354e6035
2024-03-15 12:31:16 -07:00
jade 38571c50e6 Implement a parser for a literate testing system for the repl
This parser can be reused for other purposes. It's inspired by
https://bitheap.org/cram/

Although eelco's impostor exists https://github.com/mobusoperandi/eelco,
it is not very nice to depend on out of tree testing frameworks with no
way to customize them.

Change-Id: Ifca50177e09730182baf0ebf829c3505bbb0274a
2024-03-14 14:30:38 -07:00
eldritch horrors c26599b143 libexpr: fix elided value counting in printer
using the total-attrs-printed and total-list-items-printed counters to
calculate how many attrs were elided only works properly if no nesting
is involved. once things do nest the global counter can exceed the size
of the currently printed object, leading to unsigned wrapping and great
overestimation of elided counts. counting locally in addition to global
counts fixes this.

these are functional tests because creating these objects requires the
evaluator to not be a huge amount of code, and we also want defaults to
be tested for cli usage.

fixes #14

Change-Id: Icb9a0cb21b2f4bacbc5e9dcdd8c0b9055b4088a7
2024-03-14 01:52:19 -06:00
eldritch horrors 06952cf7c4 support <program>_ENV variables
this lets us set per-test-program environment variables rather than only
a single, global default. this was supported in nix originally but
might've gone partially missing in the upstream backports process?

Change-Id: Iad0919841b1b6d11e0b7ebd3920449a62f544e77
2024-03-13 19:48:26 +01:00
jade 1b8662b85c import the revisions to the characterization test framework from cppnix
This has some Flaws for sure (like, it is going to be a bit stretched to
use for repl characterization), but it is a start.

Change-Id: I258c8beb3aee236f45818a03be83bcda858120c9
2024-03-11 14:14:43 -07:00
eldritch horrors b667b4cded evaluate inherit (from) exprs only once per directive
desugaring inherit-from to syntactic duplication of the source expr also
duplicates side effects of the source expr (such as trace calls) and
expensive computations (such as derivationStrict).

(cherry picked from commit cefd0302b55b3360dbca59cfcb4bf6a750d6cdcf)
Change-Id: Iff519f991adef2e51683ba2c552d37a3df7a179e
2024-03-10 03:18:32 -06:00
eldritch horrors 2a84123631 group inherit by source during Expr::show
for plain inherits this is really just a stylistic choice, but for
inherit-from it actually fixes an exponential size increase problem
during expr printing (as may happen during assertion failure reporting,
on during duplicate attr detection in the parser)

(cherry picked from commit ecf8b12d60ad2929f9998666cf0966475b91e291)
Change-Id: Ie55f0cb01a37e766414c31f8d40f51c2c7d106b0
2024-03-10 03:18:32 -06:00
eldritch horrors bf19eebb9b use the same bindings print for ExprAttrs and ExprLet
this also has the effect of sorting let bindings lexicographically
rather than by symbol creation order as was previously done, giving a
better canonicalization in the process.

(cherry picked from commit 6c08fba533ef31cad2bdc03ba72ecf58dc8ee5a0)
Change-Id: Ia887f629305645bb8a165fbbc0d32e620912595a
2024-03-10 03:18:32 -06:00
eldritch horrors 3e43f4aeff add test for inherit expr printing
(cherry picked from commit 73065a400d176b21f518c1f4ece90c31318b218d)
Change-Id: I9356d8084d241a7904b66554d7c4194f8433edf7
2024-03-10 03:18:32 -06:00
eldritch horrors 0c19e6b56c add test for inherit-from semantics
(cherry picked from commit 8669c02468994887be91072ac58b1ee43380d354)
Change-Id: If8513316bf4b4b559c5bb63842c856f016816802
2024-03-10 03:18:32 -06:00
eldritch horrors 7c882a5075 Merge "make the multi-node vm tests a bit more reliable" into main 2024-03-10 03:18:08 -06:00
eldritch horrors 34fb7a7e9d make the multi-node vm tests a bit more reliable
without these changes the tests will very repeatably (although not very
reliably) wedge in our runs. the ssh command starts, opens a sessions,
does something, the session closes again, but the test does not move on.
adding *just* the redirect and not the unit waits is not sufficient
either, it needs both. this feels like a bug in the nixos testing
framework somewhere, but digging that far is not in the cards right now.

Change-Id: Idab577b83a36cc4899bb5ffbb3d9adc04e83e51c
2024-03-10 10:10:52 +01:00
eldritch horrors a9b813cc3b Merge pull request #10066 from 9999years/print-all-frames
Do not skip any stack frames when `--show-trace` is given

(cherry picked from commit 0b47783d0a879875d558f0b56e49584f25ceb2d0)
Change-Id: Ia0f18266dbcf97543110110c655c219c7a3e3270
2024-03-09 10:17:26 -07:00
eldritch horrors 1958152d14 Pretty-print values in the REPL
Pretty-print values in the REPL by printing each item in a list or
attrset on a separate line. When possible, single-item lists and
attrsets are printed on one line, as long as they don't contain a nested
list, attrset, or thunk.

Before:
```
{ attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
```

After:
```
{
  attrs = {
    a = {
      b = {
        c = { };
      };
    };
  };
  list = [ 1 ];
  list' = [
    1
    2
    3
  ];
}
```

(cherry picked from commit c0a15fb7d03dfb8f53bc6726c414bc88aa362592)
Change-Id: Ia2b41849165a5ddb63f7a8c272a2476b3e4292df
2024-03-09 07:20:23 -07:00
eldritch horrors b221a14f0a Merge pull request #9925 from 9999years/fmt-cleanup
Cleanup `fmt.hh`

(cherry picked from commit 47a1dbb4b8e7913cbb9b4d604728b912e76e4ca0)
Change-Id: Id076a45cb39652f437fe3f8bda10c310a9894777
2024-03-09 07:00:13 -07:00
eldritch horrors 08252967a8 libexpr: Support structured error classes
While preparing PRs like #9753, I've had to change error messages in
dozens of code paths. It would be nice if instead of

    EvalError("expected 'boolean' but found '%1%'", showType(v))

we could write

    TypeError(v, "boolean")

or similar. Then, changing the error message could be a mechanical
refactor with the compiler pointing out places the constructor needs to
be changed, rather than the error-prone process of grepping through the
codebase. Structured errors would also help prevent the "same" error
from having multiple slightly different messages, and could be a first
step towards error codes / an error index.

This PR reworks the exception infrastructure in `libexpr` to
support exception types with different constructor signatures than
`BaseError`. Actually refactoring the exceptions to use structured data
will come in a future PR (this one is big enough already, as it has to
touch every exception in `libexpr`).

The core design is in `eval-error.hh`. Generally, errors like this:

    state.error("'%s' is not a string", getAttrPathStr())
      .debugThrow<TypeError>()

are transformed like this:

    state.error<TypeError>("'%s' is not a string", getAttrPathStr())
      .debugThrow()

The type annotation has moved from `ErrorBuilder::debugThrow` to
`EvalState::error`.

(cherry picked from commit c6a89c1a1659b31694c0fbcd21d78a6dd521c732)
Change-Id: Iced91ba4e00ca9e801518071fb43798936cbd05a
2024-03-09 04:47:05 -07:00
eldritch horrors 7673312ccc Merge pull request #9928 from 9999years/error-messages-in-nix-repl
Improve error printing in `nix repl`

(cherry picked from commit a8050d9b83052e4b5c52bf2d116381aedec3a93e)
Change-Id: I588f92d1dd4c546c98788b71403cc034f5e7129a
2024-03-09 03:37:35 -07:00
eldritch horrors c864923928 Merge pull request #9929 from 9999years/dont-print-values-in-magenta
Don't print the first bracket in values in magenta in error messages

(cherry picked from commit 46a0625a40aef6946a35f92fdacf0e6b4a14414f)
Change-Id: I8435565c87db182116140eaeea9df1243e67ea94
2024-03-09 03:37:35 -07:00
eldritch horrors b6b31d255a Merge pull request #9926 from 9999years/fix-cycle-detection-in-nix-repl
Fix cycle detection in `nix repl`

(cherry picked from commit e190c20c3394fd1a5cd9be1afc3f30ab32dcd36b)
Change-Id: Ie385e781b9f0b7171ca653bcd53a990bb41f9e4b
2024-03-09 03:37:35 -07:00
eldritch horrors 609a8e0d94 Merge pull request #9754 from 9999years/print-value-when-coercion-fails
Print the value in `error: cannot coerce` messages

(cherry picked from commit 5b7bfd2d6b89d7dd5f54c1ca6c8072358d31a84e)

===

test taken from 6e8d5983143ae576e3f4b1d2954a5267f2943a49; it was added
previously (and not backported because its pr was a mostly-revert), but
it's useful to have around.

Change-Id: Icbd14b55e3610ce7b774667bf14b82e6dc717982
2024-03-09 00:05:41 -07:00
eldritch horrors 2f7c3fa251 Merge pull request #9818 from Ma27/print-value-on-function-call-type-error
libexpr: print value of what is attempted to be called as function
(cherry picked from commit 50e5d7b883042852538371237e32a66bb22f0485)
Change-Id: I7cb6290bd8f244e83bfce3b2eed2a4c8b4f16a83
2024-03-09 00:05:41 -07:00
eldritch horrors 87e6ac5eb7 Merge pull request #9753 from 9999years/print-value-on-type-error
Print the value in `value is X while a Y is expected` error

(cherry picked from commit 5f72a97092da6af28a7d2b2a50d74e9d34fae7e1)
Change-Id: Idb4bc903ae59a0f5b6fb3b1da4d47970fe0a6efe
2024-03-09 00:05:41 -07:00
eldritch horrors 512c1f05c3 Unify and refactor value printing
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).

This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.

Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)

Please read the tests for example output.

Future work:
- It would be nice to provide this function as a builtin, perhaps
  `builtins.toStringDebug` -- a printing function that never fails would
  be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
  command line, e.g. `--option to-string-max-attrs 1000`.

(cherry picked from commit 0fa08b451682fb3311fe58112ff05c4fe5bee3a4, )

===

Restore ambiguous value printer for `nix-instantiate`

The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.

(cherry picked from commit df84dd4d8dd3fd6381ac2ca3064432ab31a16b79)

Change-Id: I7cca6b4b53cd0642f2d49af657d5676a8554c9f8
2024-03-09 03:50:06 +01:00
puck 80b79d0137 flake.nix: upgrade to nixos-23.11
This also bypasses the Objective-C fork safety during tests.

Change-Id: I92bf9f911e8a1fbd32eae13255f9a9dabde40b21
2024-03-08 23:59:01 +00:00
eldritch horrors 232f4feace Merge pull request #9324 from 9999years/fix-8854-take-2
Don't attempt to `git add` ignored files

(cherry picked from commit 359990dfdc713c80aabd7ea6f7e4528628fbe108)

===

also added a regression test that isn't upstream to be sure we're
actually fixing the bug.

Change-Id: I8267a3d0ece9909d8008b7435b90e7b3eee366f6
2024-03-07 14:50:43 +01:00
eldritch horrors 8a268359b0 Merge pull request #9560 from obsidiansystems/serve-proto-unkeyed-valid-path-info-serializer
Factor out `ServeProto::Serialiser<UnkeyedValidPathInfo>` and test

(cherry picked from commit 139982997eec493a0f74105c427953f6be77da6d)
Change-Id: I28e4ba5a681a90d81915a56e6dbaa5456d64f96d
2024-03-07 12:37:33 +01:00
eldritch horrors 9eb58f5209 Merge pull request #9032 from Ma27/structured-attrs-env-vars
structured attrs: improve support / usage of NIX_ATTRS_{SH,JSON}_FILE

(cherry picked from commit 3c042f3b0b0a7ef9c47bf049f5410dbd4aac9e90)
Change-Id: I7e41838338ee1edf31fff6f9e354c3db2bba6c0e
2024-03-07 10:46:47 +01:00
eldritch horrors ca03f7cc28 Merge pull request #9676 from DavHau/git-testsuite
initialize test suite for git fetchers

(cherry picked from commit 0bd9e10aea747df51c8a5af124864c722cbeafde)
Change-Id: Idf94a47794190c3e1de07fc4e7848741c4e9ffed
2024-03-07 10:25:03 +01:00
eldritch horrors b87f059ed4 Merge pull request #9631 from cole-h/fixup-check-warnings
Fix warnings when running checks

(cherry picked from commit 75e10e42f3c63fd9b9c8cf222b992ab77e497854)
Change-Id: Id955008fe045f23f72fae2a2cdf8f7ccddd1e6b9
2024-03-07 09:58:15 +01:00
eldritch horrors 06e92450bd Merge pull request #8544 from edolstra/handle-missing-gc-socket
LocalStore: :addTempRoot(): Handle ENOENT
(cherry picked from commit 7115edc85af060ef235ac0270245ab46cc828f7c)
Change-Id: Ie6b1596049c3fde09b98f2f0727899f98e48e6b1
2024-03-07 00:43:51 -07:00
eldritch horrors b14f88e0d4 Merge pull request #9985 from alois31/symlink-resolution
Restore `builtins.pathExists` behavior on broken symlinks

(cherry picked from commit d53c8901ef7f2033855dd99063522e3d56a19dab)

===

note that this variant differs markedly from the source commit because
we haven't endured quite as much lazy trees.

Change-Id: I0facf282f21fe0db4134be5c65a8368c1b3a06fc
2024-03-07 00:43:51 -07:00
eldritch horrors 1342c8f18e Merge pull request #10074 from lf-/jade/ban-implicit-fallthrough
Warn on implicit switch case fallthrough

(cherry picked from commit 21282c3c204597641402c6bcff8fc9ee7bc31fa1)
Change-Id: I5ebbdfb6c037d2c55254f37dd391c07c2ce7443e
2024-03-07 00:11:12 -07:00
eldritch horrors f10e673272 tests/nixos/remote-builds*: Inline module + format
(cherry picked from commit 5167351efbee5c5a7390510eb720c31c6976f4d9)
Change-Id: I0caba23b589ed428d08895d7b8f0c22532bd259e
2024-03-06 21:32:47 -07:00
eldritch horrors 097acae35b tests/nixos: Test remote build against older versions
(cherry picked from commit e502d1cf945fb3cdd0ca1e1c16ec330ccab51c7b)
Change-Id: If6a1758b6457c5dae9305829c4d71d1905cfca22
2024-03-06 21:32:47 -07:00
eldritch horrors 25cae6d377 Merge pull request #9604 from obsidiansystems/test-substitute-on-destination
Test `nix copy --substitute-on-destination`

(cherry picked from commit ed93e51f4b0a8149521e3cf5b4f3ff4eabf5b060)
Change-Id: I33f8b591f42acf96fd18f27758dd1141b4bbbb5f
2024-03-07 03:34:00 +01:00
eldritch horrors 706f0df55b Merge pull request #9280 from R-VdP/rvdp/fix_remote_logging_phase_reporting
Include phase reporting in log file for ssh-ng builds

(cherry picked from commit b1e7d7cad625095656fff05ac4aedeb12135110a)
Change-Id: I4076669b0ba160412f7c628ca9113f9abbc8c303
2024-03-06 19:11:12 -07:00
puck 6f36a8834c Copy the output of fixed-output derivations before registering them
It is possible to exfiltrate a file descriptor out of the build sandbox
of FODs, and use it to modify the store path after it has been
registered. To avoid that issue, don't register the output of the build,
but a copy of it (that will be free of any leaked file descriptor).

Test that we can't leverage abstract unix domain sockets to leak file
descriptors out of the sandbox and modify the path after it has been
registered.

(cherry picked from commit 2dadfeb690e7f4b8f97298e29791d202fdba5ca6)
(tests cherry picked from commit c854ae5b3078ac5d99fa75fe148005044809e18c)

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Theophane Hufschmitt <theophane.hufschmitt@tweag.io>
Co-authored-by: Tom Bereknyei <tomberek@gmail.com>

Change-Id: I87cd58f1c0a4f7b7a610d354206b33301e47b1a4
2024-03-07 01:44:58 +00:00
eldritch horrors 2e1f5e2666 Merge pull request #9105 from Ericson2314/split-out-nixos-tests
Define NixOS tests in `tests/nixos/default.nix` rather than `flake.nix`

(cherry picked from commit c29b8ba142a0650d1182ca838ddc1b2d273dcd2a)
Change-Id: Ieae1b6476d95024485df7067e008013bc5542039
2024-03-05 21:11:59 +01:00
eldritch horrors 96a3683308 Ban building Nix with NDEBUG
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.

For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.

There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.

I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.

Upstream-PR: https://github.com/NixOS/nix/pull/10126
Original-Change-Id: I513cceaac1371decb3d96231e6ef9181c910c218
Change-Id: I531a51f6348a746e8e41d88203b08f614898356c
2024-03-04 09:24:58 +01:00
eldritch horrors 647d762ab5 Merge pull request #9933 from pennae/debugger-fix
fix debugger crashing while printing envs

(cherry picked from commit 9b8b4860913afdb7f9cb1e1e81f9a9dd192cd1c1)
Change-Id: Ica6a17e4b3e3b8b8093719f442c28be7f13be09d
2024-03-04 08:57:44 +01:00
eldritch horrors 9ef32cf3b8 Merge pull request #9930 from rvl/print-dev-env-unbound-variables
print-dev-env: Avoid using unbound shellHook variable
(cherry picked from commit 25385a408ef7281d966c8732608833e224b32586)
Change-Id: Id1aca13942adfa628f7bdcfc0b6fc03bb3bab9ea
2024-03-04 08:51:49 +01:00
eldritch horrors 74272a9bc4 Merge pull request #9861 from 9999years/colored-diff-in-lang-tests
Color `diff` output in `tests/functional/lang` tests

(cherry picked from commit 1dc55c0f2f034bce6e3de4a5cda96d686b10a7f8)
Change-Id: Ie9b3fc3446bd3caa0fd8885de88639516a2ff862
2024-03-04 08:50:00 +01:00
eldritch horrors 2bb0a06346 Merge pull request #9841 from obsidiansystems/float-speed-factor
Convert `Machine::speedFactor` from a non-neg int to a non-neg float

(cherry picked from commit 69d0ae27e376e7c7c4f237716b0149223b8a805a)
Change-Id: I2afb5cf9e4fe1384985c58353946135c3d102b42
2024-03-04 08:40:38 +01:00
eldritch horrors 044c117a9f Merge pull request #9481 from iFreilicht/disallow-nix-search-without-search-terms
nix search: Disallow empty regex

(cherry picked from commit 1c260fa6d1f47d83954792771d0614db163cc3bc)
Change-Id: Iaaf3605c24a342fcb05d0b534a9f305533d3b5fa
2024-03-04 08:40:12 +01:00
eldritch horrors b6bb869e52 Merge pull request #9632 from cole-h/nix-daemon-testing
Use `nix daemon` in the test suite

(cherry picked from commit e6e160a0758c0354ed810b0ec9846ed885adcc11)
Change-Id: I537a25d3d48f609cd77b2c3a8ad68e87aebabfe8
2024-03-04 08:17:32 +01:00
eldritch horrors e81ee8b337 Merge pull request #9785 from hercules-ci/test-substring-negative
tests/functional/lang: Test substring with negative length

(cherry picked from commit 86156d05dd33f856d8804f89669a7fe9b81f1a0d)
Change-Id: I2e2086027a43f8111ba5068ac16590eaa0b798d4
2024-03-04 08:07:16 +01:00
eldritch horrors 2c85fcce87 Merge pull request #9747 from awakesecurity/mz/fix-quadratic-splitString
Fix performance of builtins.substring for empty substrings

(cherry picked from commit b2deff1947c2fe57fdbf1a472eb9003eb407f8d3)
Change-Id: I4ddfc8d26a4781c9520fff9807849a073ee7bed8
2024-03-04 08:06:36 +01:00
eldritch horrors e310bbf913 Merge pull request #8043 from bobvanderlinden/pr-shell-env
nix: develop: use SHELL from rc script
(cherry picked from commit 0d55d660d5ea081630aa9606b047eb755cff1528)
Change-Id: I83be6c63b282d7f01a0defa78d9e787c77f1f02d
2024-03-04 07:59:32 +01:00
eldritch horrors 180984178d Merge pull request #9648 from cole-h/nix-shell-ordering
nix shell: reflect command line order in PATH order

(cherry picked from commit b91c935c2faf08ced2c763dcd2a831f26d84fa86)
Change-Id: If16c120bb74857c2817366e74e5b0877eb997260
2024-03-04 07:50:55 +01:00
eldritch horrors 5f2eaa1b35 Merge pull request #9662 from shlevy/flat-fixed-references-assert
Improve error message for fixed-outputs with references.

(cherry picked from commit ff6de4a9ee6c3862db9ee5f09ff9c3f43ae7a088)
Change-Id: I733c49760b9a3f1b76a6bece3b250b8579cd6cac
2024-03-04 07:50:02 +01:00
eldritch horrors 96f1a404d0 Merge pull request #9617 from 9999years/stack-overflow-segfault
Fix segfault on infinite recursion in some cases

(cherry picked from commit bf1b294bd81ca76c5ec9fe3ecd52196bf52a8300)
Change-Id: Id137541426ec8536567835953fccf986a3aebf16
2024-03-04 07:35:20 +01:00
eldritch horrors 859da103ae Merge pull request #9637 from 9999years/fix-logging-test-on-macos
Fix `logging.sh` test on macOS

(cherry picked from commit 94a03a810e948f65776c3386d9a5234db551dc9e)
Change-Id: I65dde18ee53ccded31790e0cf1f836c3ad6dc492
2024-03-04 07:26:34 +01:00
eldritch horrors 2ddf453e0d Merge pull request #9619 from 9999years/remove-blank-lines-in-errors
Remove some blank lines from stack traces

(cherry picked from commit cea83544362bd6426e9bc0b7dd0d611c19b0e3fd)
Change-Id: Ic1af8b09e9994d3c69fd3b37ae47a4bb786a15a1
2024-03-04 07:23:44 +01:00
eldritch horrors 7614aa9797 Merge pull request #4093 from matthewbauer/eval-system
Add eval-system option

(cherry picked from commit 071dbbee33af9f27338c3e53e4ea067dbfa14010)
Change-Id: Ia81358c8cfb60241da07a4d0e84b9ee62a18a53f
2024-03-04 07:21:01 +01:00
eldritch horrors 13f78c557b Merge pull request #9589 from obsidiansystems/floating-content-addressing-derivations-eval-store
Fix building CA derivations with and eval store

(cherry picked from commit dfc0cee7024a082d90a4f68296f55a82dfd52126)
Change-Id: I28feb5a36d4fe75f0ed3e3e2db6eb56b67d0f371
2024-03-04 07:15:08 +01:00
eldritch horrors 2ba9788003 Merge pull request #9555 from 9999years/positions-in-errors
Pass positions when evaluating

(cherry picked from commit c8458bd731eb1c74159bebe459ea00165e056b65)
Change-Id: I1b4a5d58973be6264ffdb23b4492da200fdb71be
2024-03-04 07:11:25 +01:00
eldritch horrors e9243c849a Merge pull request #9529 from wh0/patch-1
tests: avoid a chroot store without sandbox support
(cherry picked from commit 5910140f252280f6be429d1cb5e91e69999f5b43)
Change-Id: Ic2255261334012f36a465a96073f50669952dc26
2024-03-04 07:11:25 +01:00