Commit graph

259 commits

Author SHA1 Message Date
eldritch horrors 112fd6c971 rewrite the parser with pegtl instead of flex/bison
this gives about 20% performance improvements on pure parsing. obviously
it'll be less on full eval, but depending on how much parsing has to be
done (eg whether nixpkgs haskell modules are included or not) it ranges
anywhere from 4% to 10% in our tests.

this has been tested with thousands of core hours of fuzz testing to
ensure that the ASTs produced by the new parser are exactly the same as
the ones produced by the old parser. error messages will
change (sometimes a lot) and are currently not perfect, but we'd rather
leave that open for improvement than having this work rot forever.

Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
2024-03-16 18:07:01 +01:00
Cole Helbling 84727bebb4
Backport PR#9633 by cole-h: package: don't set sysconfdir in devShells
(cherry-picked from commit ba0087316acc2aba999cabe5e1a159da636b2569)

Change-Id: I5a50afb3b7b65516df798ee51b74f06727a91928
2024-03-14 14:06:31 -07:00
Rebecca Turner 9f242fae76
Only set LOCALE_ARCHIVE on Linux
`macOS` does not have `glibcLocales`:

    error:
       … while calling the 'derivationStrict' builtin

         at /derivation-internal.nix:9:12:

            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'nix-2.90.0'
         whose name attribute is located at /nix/store/y0c95bwyvs80pm69hdd4b11pyq2ghiwh-source
/pkgs/stdenv/generic/make-derivation.nix:348:7

       … while evaluating attribute 'LOCALE_ARCHIVE' of derivation 'nix-2.90.0'

         at /nix/store/ng5qzbyv4902b4pw7g35caqw5cnmryf9-source/flake.nix:331:15:

          330|               # Required to make non-NixOS Linux not complain about missing loc

Change-Id: I4464484a0eca12b5e073d49d900b6f25886245c1
2024-03-14 14:06:23 -07:00
puck caded0d55e Merge "Delete the existing installer" into main 2024-03-14 13:14:21 -06:00
puck 93cc063344 Delete the existing installer
We're not going to use it.

Fixes: #31
Change-Id: Ib17a2eb6cae1ecbbf9ad1062e576ba6107a3c13b
2024-03-14 18:15:46 +00:00
jade ab5ff86917 flake: dev shells should include the LOCALE_ARCHIVE so configure does not complain
Change-Id: Id661bd8c90696cb663a25503878a697f596d1e77
2024-03-13 12:25:23 -07:00
Qyriad 36a8d151e1 package: cleanup of all intermediaries
Change-Id: I0da5182de6b01c192cfcba407959d659d70c6dc9
2024-03-11 04:26:35 -06:00
Qyriad 529a01ade2 package: migrate devShells
Change-Id: Ic63721667edd4bef79aa699a0de8411639e5159b
2024-03-11 04:26:35 -06:00
Qyriad b072c069b7 package: migrate internal-api-docs
Change-Id: I344d73a412c2c6e4bb2eb14bd4859056324f1ba7
2024-03-11 04:26:35 -06:00
Qyriad 4ad3446311 package: migrate testNixVersions
Change-Id: I71845f8a6d7b77c3617d055e726ed4a28cd05fa3
2024-03-11 04:26:35 -06:00
Qyriad 15380b4c6e package: refactor Nix out of flake.nix and into package.nix
This series takes a somewhat different approach from the flake rework
done in NixOS/nix. The package.nix here does not provide callPackage
options for all the various settings in the build, and instead the other
places Nix derivations are used (like internal-api-docs) will .overrideAttrs
the normal Nix package derivation. This more closely matches how these
things were structured originally, and results in less churn and more
atomicity in these changes.

In the future, package.nix likely will migrate to have more build
options in the callPackage arguments, but we are also planning to
rewrite the build system anyway.

Change-Id: I170c4e5a4184bab62e1fd75e56db876d4ff116cf
2024-03-11 04:26:35 -06:00
eldritch horrors 71e0114708 remove getDerivations deduplication
deduplication does not currently work fully, showing derivations
multiple times if they have different underlying values. this can happen
by selecting the same derivation twice for two different attributes of a
set, using inherit-from (which reduces to the previous), importing
nixpkgs twice, or any other number of things.

since users already have to deal with duplicates for this reason it
won't hurt to add *more* duplicates. the alternative would be to
deduplicate fully, which would drop derivations that are currently
returned and those pose a regression risk.

Change-Id: I64b397351237e10375d270f1bddecb71f62aa131
2024-03-10 03:18:32 -06:00
eldritch horrors 896e525681 Disable bear on all the things with darwin as hostPlatform
Just `stdenv.isDarwin` isn't enough because it doesn't apply to the
build platform, which mean that cross packages building from darwin to
another platform will have `isDarwin` set to false.
Replace it by `stdenv.buildPlatform.isDarwin`.

(cherry picked from commit a0cb75d96f76a3be48b9319e26d8ad78ef4e4525)

(h/t jade for finding this one)
Change-Id: If3cb74e6feaa5d51de550d9a140c71683c2214cd
2024-03-09 00:03:50 -07:00
puck 80b79d0137 flake.nix: upgrade to nixos-23.11
This also bypasses the Objective-C fork safety during tests.

Change-Id: I92bf9f911e8a1fbd32eae13255f9a9dabde40b21
2024-03-08 23:59:01 +00:00
puck a3eba9d717 flake.nix: remove unused hydraJobs
This is in preparation for setting buildbot up to use hydraJobs instead
of checks.

Change-Id: Ie971e7db3940d9fbb7ed2f53d6429f607e00051f
2024-03-08 17:48:35 +00:00
eldritch horrors c67b392385 Merge pull request #9608 from NixOS/default-lowdown
Remove custom lowdown

(cherry picked from commit 790cf13c268c7197c276cc02efda4cfe64a3a688)
Change-Id: Ie01f9a69e81e793fc7ac869de943da370dc75e36
2024-03-05 23:36:11 -07:00
eldritch horrors 2e1f5e2666 Merge pull request #9105 from Ericson2314/split-out-nixos-tests
Define NixOS tests in `tests/nixos/default.nix` rather than `flake.nix`

(cherry picked from commit c29b8ba142a0650d1182ca838ddc1b2d273dcd2a)
Change-Id: Ieae1b6476d95024485df7067e008013bc5542039
2024-03-05 21:11:59 +01:00
eldritch horrors 4c56629a52 Merge pull request #9692 from 9999years/update-clang-stdenv
Remove `clang11Stdenv`

(cherry picked from commit cbf99c71c6f58579174c1c7649a5421c1d2ba9b0)
Change-Id: I716ce1a54f2006c4a9dc9716e8529fe7858ecebb
2024-03-04 07:48:42 +01:00
eldritch horrors 298be50d7d Merge pull request #9690 from obsidiansystems/nixos-23.05-small-again
flake: Go back to regular `nixos-23.05-small`
(cherry picked from commit 1ed245a60672c123c1348a63061fb4d64fb95212)
Change-Id: I33f5fe20cd6bc658a461f560e458b440b3b7e18e
2024-03-04 07:47:58 +01:00
eldritch horrors 2e2f2579fa Merge pull request #9575 from hercules-ci/shell-on-hydra
flake.nix: Cache shell inputs through hydra

(cherry picked from commit b1842a4f05113967e8c20c9ebfd79edc39da259b)
Change-Id: I9e59b3c7dbf441e74cff3e4f24bbf65170ac505b
2024-03-04 07:12:28 +01:00
eldritch horrors 116e48fad3 Merge pull request #9573 from hercules-ci/rl-next-md-frontmatter
rl-next: Fix and support markdown frontmatter syntax
(cherry picked from commit 69b7876a0810269ad71807594cfd99b26cd8a5ff)
Change-Id: I8bfb8967af0943080fdd70d257c34abaf0a9fedf
2024-03-04 07:12:09 +01:00
eldritch horrors bc32950ac5 Merge pull request #9518 from NixOS/fix-static-build
Update Nixpkgs to fix static build

(cherry picked from commit 9817788f6b25fefa731b838391f859e886ae0d24)
Change-Id: Ibe07a263f4d8f3b230775fb993881e5151a53784
2024-03-04 07:11:25 +01:00
eldritch horrors 001be52794 Merge pull request #9430 from hercules-ci/remove-vlas
Fix stack overflow in `filter`

(cherry picked from commit cb7f25869daa2491eeac5fee49ad8f31b2218c15)
Change-Id: Ib90f97a9805bbb4d0e2741551d490f054fc0a675
2024-03-04 07:11:25 +01:00
eldritch horrors e8a488236c Merge pull request #9393 from hercules-ci/changelog-d
Automatically compile hand-written release notes with `changelog-d`

(cherry picked from commit 928f0c13414d20c1af88b30bd6700fd730ee0bab)
Change-Id: Ia0685835c52edf185b64dd696b19305746c077e5
2024-03-04 07:11:19 +01:00
eldritch horrors 7d8b34475a Merge pull request #9152 from obsidiansystems/split-out-perl-nix
Factor out Perl bindings Nix package

(cherry picked from commit d12c614ac75171421844f3706d89913c3d841460)
Change-Id: Ie75ccfec8cc815ea95ae9848cb03e33aa8cee5ae
2024-03-04 04:37:44 +01:00
eldritch horrors da0aa66d98 Merge pull request #9131 from obsidiansystems/delete-bootstrap-script
Get rid of `bootstrap.sh`

(cherry picked from commit aaef47a08eaf54a8856dd25c784fd85d8d7b0e22)
Change-Id: I1a74bed0c23d6fda06d5dfd8ecad443b9122da12
2024-03-04 04:36:52 +01:00
eldritch horrors fd1299cef3 Merge pull request #9106 from Ericson2314/positive-source-filtering
Use positive source filtering for the standalone functional tests job and Perl bindings

(cherry picked from commit 6b6bd9003062c86a49d4384381941cf57f269c45)
Change-Id: I896be67654f893d543ed6beb5d0d0d6c6d36e027
2024-03-04 04:36:42 +01:00
eldritch horrors c36ba796dd Merge pull request #9096 from NixOS/dont-rebuild-twice-on-ci
Don't run the tests that require building if we're not building

(cherry picked from commit 8cfa582f436db8066eff74cb084990367e014ce6)
Change-Id: I6085ca6107349669407340d7a5e52639a2febc90
2024-03-04 04:36:33 +01:00
eldritch horrors f17e7b1855 Merge pull request #8923 from obsidiansystems/test-proto
Unit test some worker protocol serializers

(cherry picked from commit c6faef61a6f31c71146aee5d88168e861df9a22a)
Change-Id: I99e36f5f17eb7642211a4e42a16b143424f164b4
2024-03-04 04:36:14 +01:00
John Ericson bc4aa3d5db Merge pull request #8887 from obsidiansystems/bsd-cross-ci
Support cross compiling to BSD and CI it

(cherry picked from commit 1f3fc08c5994ca69c84c9e745d59ec2bb2fd820a)
Change-Id: I415e92952afc661cfb5ef91a76c0637678a04a19
2024-03-04 04:36:04 +01:00
John Ericson f7f37035c8 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

(cherry picked from commit 91b6833686a6a6d9eac7f3f66393ec89ef1d3b57)
(cherry picked from commit a61e42adb528b3d40ce43e07c79368d779a8b624)
2023-12-01 13:05:03 -05:00
John Ericson 30dcc19d1f Put functional tests in tests/functional
I think it is bad for these reasons when `tests/` contains a mix of
functional and integration tests

 - Concepts is harder to understand, the documentation makes a good
   unit vs functional vs integration distinction, but when the
   integration tests are just two subdirs within `tests/` this is not
   clear.

 - Source filtering in the `flake.nix` is more complex. We need to
   filter out some of the dirs from `tests/`, rather than simply pick
   the dirs we want and take all of them. This is a good sign the
   structure of what we are trying to do is not matching the structure
   of the files.

With this change we have a clean:
```shell-session
$ git show 'HEAD:tests'
tree HEAD:tests

functional/
installer/
nixos/
```

(cherry picked from commit 68c81c737571794f7246db53fb4774e94fcf4b7e)
2023-12-01 12:06:43 -05:00
Gerg-L 277ba90779 flake: complete update to 23.05
(cherry picked from commit f264d9ff0862277523a207c7afaed4894a40dc11)
2023-09-25 08:43:54 +00:00
Eelco Dolstra 44fb119218 Mark official release 2023-09-20 12:49:01 +02:00
Eelco Dolstra 126e2645f2 Disable rapidcheck tests in the coverage run
https://hydra.nixos.org/build/233688539
2023-09-19 16:04:00 +02:00
Eelco Dolstra c8afa01bc2 Try aws-sdk-cpp fix 2023-09-19 14:51:50 +02:00
Vladimír Čunát 539cc5e5f0 flake: update nixpkgs: 22.11 -> 23.05
The lowdown input can't be updated; `nix build` would fail to find it.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-09-01 14:07:25 +02:00
Robert Hensing 3384f70a3d nixpkgsLibTests: Only test our Nix
Interface has changed upstream.
It *should* be fine to test 23.05's other Nix versions as those
*should* succeed, but that's not the case and it's obfuscating
our terrible CI setup's log.
2023-08-30 19:34:35 -04:00
p01arst0rm 7d82341633 update system definitions 2023-08-23 19:28:24 +01:00
Robert Hensing 21a188a2b4 Add gc root for nixpkgs/lib content 2023-08-16 16:01:46 +02:00
Robert Hensing 63e0b5d081 GC root for fetched nixpkgs/lib content 2023-08-16 15:46:37 +02:00
Robert Hensing b13fc7101f Add positive source filter
Source filtering is a really cool Nix feature that lets us avoid a
lot of rebuilds, which speeds up the iteration cycle a lot in cases
where the relevant source files aren't actually modified.

We used to have a source filter that marked a few files as irrelevant,
but this is the wrong approach, as we have many more files that are
irrelevant. We may call this negative filtering.

This commit switches the source filtering to positive filtering, which
is a lot more robust. Instead of marking which files we don't need
we marked the files that we do need.

It's a superior approach because it is fail safe. Instead of allowing
build performance problems to creep in over time, we require that all
source inputs are declared.

I shouldn't have to explain that declaring inputs is a good practice,
so I'll stop over-explaining here.

I do have to acknowledge that this will cause a build failure when the
filter is incomplete. This is *good*, because it's the only realistic
way we could be reminded of these problems. These events will be
infrequent, so the small cost of extending the filter is worth it,
compared to the hidden cost of longer dev cycles for things like tests,
docker image, etc, etc.

(Also rebuilding Nix for stupid unnecessary reasons makes my blood boil)
2023-08-16 14:21:59 +02:00
Eelco Dolstra 1ad3328c5e Allow tarball URLs to redirect to a lockable immutable URL
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.

So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
2023-06-13 14:17:45 +02:00
Philipp Otterbein ca6b759f4e fix failing configure in nix-tests 2023-04-09 02:33:53 +02:00
Eelco Dolstra 2425121a59 Remove nixpkgsFor flake output
Fixes "warning: unknown flake output 'nixpkgsFor'".
2023-03-31 16:08:16 +02:00
Théophane Hufschmitt e32ca3cf16
Merge pull request #8018 from tweag/ssh-password-prompt
SSH: don't erase password prompt if it is displayed
2023-03-31 12:06:10 +02:00
Alexander Bantyev 80f9231b69
Filter tests/nixos from source 2023-03-24 14:29:28 +04:00
Alexander Bantyev 85a2d1d94f
Add a test for nix copy over ssh
Check that nix copy can copy stuff, refuses to copy unsigned paths by
default, and doesn't hide the ssh password prompt.
2023-03-22 09:45:08 +04:00
Eelco Dolstra 658847179a Fix internal-api rendering in Hydra
Currently it gives a 500 error with "Do not know how to serve path
'/nix/store/bym5sm8z2wpavnvzancb9gjdlgyzs1l8-nix-internal-api-docs-2.15.0pre20230320_e37f436/share/doc/nix/internal-api'."
2023-03-20 13:13:57 +01:00
Théophane Hufschmitt 9ec1a3ae60
Merge pull request #7989 from sysedwinistrator/flake-compat-sha256-mr
add flake-compat to flake.nix and use sha256 in default.nix
2023-03-14 17:12:50 +01:00