Commit graph

33 commits

Author SHA1 Message Date
Nikodem Rabuliński e8a603fb2f Merge changes Icf26010a,Ib6161567 into main
* changes:
  Always initialize curl in parent process on darwin
  Fix failing darwin tests
2024-05-07 21:26:24 +00:00
Nikodem Rabuliński f894cce79b
Always initialize curl in parent process on darwin
Because of an objc quirk[1], calling curl_global_init for the first time
after fork() will always result in a crash.
Up until now the solution has been to set
OBJC_DISABLE_INITIALIZE_FORK_SAFETY for every nix process to ignore
that error.
This is less than ideal because we were setting it in package.nix,
which meant that running nix tests locally would fail because
that variable was not set.
Instead of working around that error we address it at the core -
by calling curl_global_init inside initLibStore, which should mean
curl will already have been initialized by the time we try to do so in
a forked process.

[1] 01edf1705f/runtime/objc-initialize.mm (L614-L636)

Change-Id: Icf26010a8be655127cc130efb9c77b603a6660d0
2024-05-07 20:43:17 +02:00
jade bf1a22aa8e Remove README.md from build dependencies
Now we will simply hit cache every time anyone changes readme, yay!

Change-Id: I5906f589d319ff6d43cbd2b467887e08f7474283
2024-05-06 18:23:57 -07:00
Maximilian Bosch 3580a4b7bf package: remove assert for libseccomp version
This has the following downsides:

* you cannot build Lix against nixos-unstable.
* this will immediately break as soon as libseccomp will hit
  nixos-23.11 (given that people will probably use the package.nix via
  our overlay or override nixpkgs via `follows`).

Hence, removing the assert again and add a better FIXME comment.

Change-Id: I284e10cf08e1873fef70ed869a1638aa89792422
2024-05-04 11:25:29 +02:00
Maximilian Bosch 045ee37438 libstore/local-derivation-goal: prohibit creating setuid/setgid binaries
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.

Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:

> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.

Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.

This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
is needed to have the correct syscall number available via
`__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:

    $ rg --pcre2 'define __NR_fchmodat2 (?!452)'
    sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
    58:#define __NR_fchmodat2 1073742276

    sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
    67:#define __NR_fchmodat2 6452

    sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
    62:#define __NR_fchmodat2 5452

    sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
    70:#define __NR_fchmodat2 4452

    sysdeps/unix/sysv/linux/alpha/arch-syscall.h
    59:#define __NR_fchmodat2 562

I added a small regression-test to the setuid integration-test that
attempts to set the suid bit on a file using the fchmodat2 syscall.
I confirmed that the test fails without the change in
local-derivation-goal.

Additionally, we require libseccomp 2.5.5 or greater now: as it turns
out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Lix against the project's flake.

To work around that

* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
  scheduled[3].

* the package now uses libseccomp 2.5.5 on its own already. This is to
  provide a quick fix since the correct fix for 23.11 is still a staging cycle
  away.

We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷

Relevant PRs in CppNix:

* https://github.com/NixOS/nix/pull/10591
* https://github.com/NixOS/nix/pull/10501

[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
[3] https://github.com/NixOS/nixpkgs/pull/306070

(cherry picked from commit ba6804518772e6afb403dd55478365d4b863c854)
Change-Id: I6921ab5a363188c6bff617750d00bb517276b7fe
2024-05-03 16:29:06 +02:00
Qyriad f24223931d meson: remove unnecessary parts of cross file
Meson cross files layer, the last value of each key takes effect.

https: //mesonbuild.com/Machine-files.html#loading-multiple-machine-files
Change-Id: I22d886f71cd51f0ce520d3fc22aed4bcf074bb91
2024-04-23 10:20:20 -06:00
Qyriad b913a939b0 meson: flip the switch!!
This commit makes Meson the default buildsystem for Lix.
The Make buildsystem is now deprecated and will be removed soon, but has
not yet, which will be done in a later commit when all seems good. The
mesonBuild jobs have been removed, and have not been replaced with
equivalent jobs to ensure the Make buildsystem still works.

The full, new commands in a development shell are:

$ meson setup ./build "--prefix=$out" $mesonFlags

(A simple `meson setup ./build` will also build, but will do a different
thing, not having the settings from package.nix applied.)

$ meson compile -C build
$ meson test -C build --suite=check
$ meson install -C build
$ meson test -C build --suite=installcheck

(Check and installcheck may both be done after install, allowing you to
omit the --suite argument entirely, but this is the order package.nix
runs them in.)

If tests fail and Meson helpfully has no output for why, use the
`--print-error-logs` option to `meson test`. Why this is not the default
I cannot explain.

If you change a setting in the buildsystem, most cases will
automatically regenerate the Meson configuration, but some cases, like
trying to build a specific target whose name is new to the buildsystem
(e.g. `meson compile -C build src/libmelt/libmelt.dylib`, when
`libmelt.dylib` did not exist as a target the last time the buildsystem
was generated), then you can reconfigure using new settings but
existing options, and only recompiling stuff affected by the changes:

$ meson setup --reconfigure build

Note that changes to the default values in `meson.options` or in the
`default_options :` argument to project() are NOT propagated with
`--reconfigure`.

If you want a totally clean build, you can use:

$ meson setup --wipe build

That will work regardless of if `./build` exists or not.

Specific, named targets may be addressed in
`meson build -C build <target>` with the "target ID" if there is one,
which is the first string argument passed to target functions that
have one, and unrelated to the variable name, e.g.:

libexpr_dylib = library('nixexpr', …)

can be addressed with:

$ meson compile -C build nixexpr

All targets may be addressed as their output, relative to the build
directory, e.g.:

$ meson compile -C build src/libexpr/libnixexpr.so

But Meson does not consider intermediate files like object files
targets. To build a specific object file, use Ninja directly and
specify the output file relative to the build directory:

$ ninja -C build src/libexpr/libnixexpr.so.p/nixexpr.cc.o

To inspect the canonical source of truth on what the state of the
buildsystem configuration is, use:

$ meson introspect

Have fun!

Change-Id: Ia3e7b1e6fae26daf3162e655b4ded611a5cd57ad
2024-04-22 21:41:58 -06:00
Qyriad 05e3b1d39e meson: fix cross compilation
This should fix cross compilation in the base case, but this is
difficult to test as cross compilation is broken in many different
places right now. This should bring Meson back up to cross parity with
the Make buildsystem though.

Change-Id: If09be8142d1fc975a82b994143ff35be1297dad8
2024-04-22 21:41:58 -06:00
Qyriad 111db8b38f meson: correctly embed sandbox shell when asked
Change-Id: I2f6c0d42245204a516d2e424eea26a6391e975ad
2024-04-18 16:15:58 -06:00
Qyriad b81eec6ed5 build internal API docs with Meson
This commit adds the capability for building the Doxygen internal API
docs in the Meson buildsystem, and also makes doing so the default for
the internal-api-docs hydra job. Aside from the /nix-support directory,
which differed only by the hash part of a store path, the outputs of
hydraJobs.internal-api-docs before and after this commit were
bit-for-bit identical on my machine.

Change-Id: I98f0017891c25b06866c15f7652fe74f706ec8e1
2024-04-15 19:05:07 -06:00
Rebecca Turner cfbcf12276 Format Nix code with nixfmt
Change-Id: I61efeb666ff7481c05fcb247168290e86a250151
2024-04-08 13:00:00 -07:00
jade 0145d45806 nix-doc -> lix-doc, make self-contained in package.nix
package.nix previously needed this callPackage'd externally, which
didn't make a lot of sense to us since this is an internal dependency.
Thus we changed it to make it more self contained.

Change-Id: I4935bc0bc80e1a132bc9b1519e917791da95037c
2024-04-08 04:05:13 +00:00
Qyriad 0c5e2cfb33 package: just use fileset from lib
The following command is now sufficient to build Lix from outside of the
flake:

nix-build -E 'let pkgs = import <nixpkgs> { }; in pkgs.callPackage
./package.nix { build-release-notes = false; nix-doc = pkgs.callPackage
./nix-doc/package.nix { }; }'

Change-Id: Ie6b14b446480ac07c7266d4fba20042b04cc35b9
2024-04-07 23:10:05 +00:00
Qyriad 3ac2dd0613 package: put boehmgc patch logic in package.nix
In our view it really doesn't make sense to not have this in in
package.nix in some way. These patches aren't just for performance or
something -- Lix flat out doesn't build without these patches.

(Arguably that makes them a buildsystem responsibility as well, but that
can wait for when we're ready to start adding subproject fallback
dependency resolution to Meson.)

This is a step towards making `package.nix` more self-sufficient and
`callPackage`able without excessive external logic.

With this change the following command is enough to build Lix from out
of the flake:

nix-build -E 'let pkgs = import <nixpkgs> { }; in pkgs.callPackage
./package.nix { build-release-notes = false; inherit (pkgs.lib) fileset;
nix-doc = pkgs.callPackage ./nix-doc/package.nix { }; }'

Change-Id: Ia37fe8171f87d3293033de8be07d9bab12716f1d
2024-04-07 22:44:04 +00:00
Qyriad 11ad383327 package: add python to nativeBuildInputs
Commit c21d11ac0 "docs: replace sed invocation with an mdbook
preprocessor for @docroot@" added a direct build dependency on Python.
This has been accidentally working so far because Python is already a
*transitive* dependency of Lix's derivation.

Change-Id: I32d6b4f2665dbbfad7014613457dd58aa4ec73da
2024-04-05 22:28:27 -06:00
Rebecca Turner 5081109592 Fix Boost with make build
This prevents the autotools build from discovering Boost in Homebrew
installation directories on macOS.

Change-Id: I624309165c9371c391fd657424ba4c4f3182b385
2024-04-05 11:09:59 -07:00
Lunaphied 7a1054fa5f repl: improve :doc builtin repl command to support lambdas.
For a long time `nix repl` has supported displaying documentation set on
builtins, however, it has long been convention to use Markdown comments
on Nix functions themselves for documentation. This exposes that
information to `nix repl` users in a nice and formatted way.

NixOS/rfcs#145 doc-comments are primarily what this feature is intended
to consume, however, support for lambda documentation in the repl is
experimental. We do our best effort to support the RFC here.

These changes are based on [the nix-doc library](https://github.com/lf-/nix-doc) and
are licensed under the terms described in the relevant source files.

Change-Id: Ic6fe947d39a22540705d890737e336c4720b0a22
2024-04-03 13:47:22 -06:00
Qyriad 4f5e532b8b meson: implement perl bindings
Change-Id: Ie1bfb0aa784e6136a82d518a652d0ae60c4b047a
2024-04-02 11:22:42 -06:00
Qyriad 81e50fef70 Merge "meson: implement functional tests" into main 2024-03-28 20:38:05 +00:00
jade 61d394e344 Merge "Build with traps on signed overflow" into main 2024-03-28 20:27:32 +00:00
jade 14207e1cf8 Build with traps on signed overflow
This is UB, we should not be doing it, and we can cheaply turn it into
crashes reliably. We would much rather have crashes than the program
doing something silly.

Benchmarks, but i wonder if they are nonsense because they get identical
times across compilers?!

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `result-clang/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 375.5 ± 24.0 | 353.8 | 408.8 | 1.00 |
| `result-gcc/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 407.9 ± 26.0 | 385.1 | 449.5 | 1.09 ± 0.10 |
| `result-clangsan/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 382.2 ± 26.6 | 354.9 | 419.0 | 1.02 ± 0.10 |
| `result-gccsan/bin/nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix` | 408.6 ± 24.6 | 384.5 | 441.9 | 1.09 ± 0.10 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `result-clang/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.199 ± 0.167 | 16.930 | 17.499 | 1.01 ± 0.01 |
| `result-gcc/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.409 ± 0.126 | 17.242 | 17.633 | 1.02 ± 0.01 |
| `result-clangsan/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.080 ± 0.137 | 16.879 | 17.350 | 1.00 |
| `result-gccsan/bin/nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello` | 17.396 ± 0.160 | 17.131 | 17.660 | 1.02 ± 0.01 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `result-clang/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.267 ± 0.069 | 6.197 | 6.415 | 1.02 ± 0.01 |
| `result-gcc/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.232 ± 0.045 | 6.180 | 6.311 | 1.01 ± 0.01 |
| `result-clangsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.162 ± 0.020 | 6.133 | 6.196 | 1.00 |
| `result-gccsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 6.229 ± 0.031 | 6.199 | 6.289 | 1.01 ± 0.01 |

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `GC_INITIAL_HEAP_SIZE=10g result-clang/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.683 ± 0.044 | 4.630 | 4.761 | 1.00 |
| `GC_INITIAL_HEAP_SIZE=10g result-gcc/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.750 ± 0.041 | 4.680 | 4.812 | 1.01 ± 0.01 |
| `GC_INITIAL_HEAP_SIZE=10g result-clangsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.703 ± 0.040 | 4.640 | 4.760 | 1.00 ± 0.01 |
| `GC_INITIAL_HEAP_SIZE=10g result-gccsan/bin/nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'` | 4.766 ± 0.037 | 4.727 | 4.844 | 1.02 ± 0.01 |

Change-Id: I616ca3eab670317587d47b41870d8ac963c019ae
2024-03-27 23:54:04 -07:00
Winter Cute 80405d0626 Stop vendoring toml11
We don't apply any patches to it, and vendoring it locks users into
bugs (it hasn't been updated since its introduction in late 2021).

Closes lix-project/lix#164

Change-Id: Ied071c841fc30b0dfb575151afd1e7f66970fdb9
2024-03-27 21:04:00 -04:00
Qyriad 038daad218 meson: implement functional tests
Functional tests can be run with
`meson test -C build --suite installcheck`.

Notably, functional tests must be run *after* running `meson install`
(Lix's derivation runs the installcheck suite in installCheckPhase so it
does this correctly), due to some quirks between Meson and the testing
system.

As far as I can tell the functional tests are meant to be run after
installing anyway, but unfortunately I can't transparently make
`meson test --suite installcheck` depend on the install targets.

The script that runs the functional tests, meson/run-test.py, checks
that `meson install` has happened and fails fast with a (hopefully)
helpful error message if any of the functional tests are run before
installing.

TODO: this change needs reflection in developer documentation

Change-Id: I8dcb5fdfc0b6cb17580973d24ad930abd57018f6
2024-03-27 18:37:50 -06:00
eldritch horrors 279e30e7ef build: replace changelog-d with local script
hacking changelog-d to support not just github but also forgejo and
gerrit is a lot more complicated than it's worth, even moreso since
the entire thing can just as well be done with ~60 lines of python.
this new script is also much cheaper to instantiate (being python),
so having it enabled in all shells is far less of a hassle.

we've also adjusted existing release notes that referenced a gerrit
cl to auto-link to the cl in question, making the diff a bit bigger

closes lix-project/lix#176

Change-Id: I8ba7dd0070aad9ba4474401731215fcf5d9d2130
2024-03-27 03:09:14 +00:00
Qyriad e1ffe56793 meson: implement unit tests
Unit tests can be run with `meson test -C build --suite check`.
`--suite check` is optional, as right now that's the only test suite,
but when functional tests are added those will be in a separate suite.

Change-Id: I7f22f1cde4b489b3cdb5f9a36a544f0c409fcc1f
2024-03-26 00:43:33 +01:00
Qyriad b4d07656ff build: optionally build and install with meson
This commit adds several meson.build, which successfully build and
install Lix executables, libraries, and headers. Meson does not yet
build docs, Perl bindings, or run tests, which will be added in
following commits. As such, this commit does not remove the existing
build system, or make it the default, and also as such, this commit has
several FIXMEs and TODOs as notes for what should be done before the
existing autoconf + make buildsystem can be removed and Meson made the
default. This commit does not modify any source files.

A Meson-enabled build is also added as a Hydra job, and to
`nix flake check`.

Change-Id: I667c8685b13b7bab91e281053f807a11616ae3d4
2024-03-22 08:36:50 -06:00
Cole Helbling 84727bebb4
Backport PR#9633 by cole-h: package: don't set sysconfdir in devShells
(cherry-picked from commit ba0087316acc2aba999cabe5e1a159da636b2569)

Change-Id: I5a50afb3b7b65516df798ee51b74f06727a91928
2024-03-14 14:06:31 -07:00
eldritch horrors 06952cf7c4 support <program>_ENV variables
this lets us set per-test-program environment variables rather than only
a single, global default. this was supported in nix originally but
might've gone partially missing in the upstream backports process?

Change-Id: Iad0919841b1b6d11e0b7ebd3920449a62f544e77
2024-03-13 19:48:26 +01:00
Qyriad 36a8d151e1 package: cleanup of all intermediaries
Change-Id: I0da5182de6b01c192cfcba407959d659d70c6dc9
2024-03-11 04:26:35 -06:00
Qyriad b072c069b7 package: migrate internal-api-docs
Change-Id: I344d73a412c2c6e4bb2eb14bd4859056324f1ba7
2024-03-11 04:26:35 -06:00
Qyriad 4ad3446311 package: migrate testNixVersions
Change-Id: I71845f8a6d7b77c3617d055e726ed4a28cd05fa3
2024-03-11 04:26:35 -06:00
Qyriad 875b76d0c7 package: use pname, version, and dontBuild (first change with diff hash)
The src fileset, preConfigure, and separateDebugInfo also respond to doBuild if its overridden

This commit is logically just a continuation of the previous commit's
refactor, but exists separately to delineate when the core Nix
derivation hash changed (this commit).

Change-Id: I67a61bc9608d91b6a833ebc5c3894b2d2e694050
2024-03-11 04:26:35 -06:00
Qyriad 15380b4c6e package: refactor Nix out of flake.nix and into package.nix
This series takes a somewhat different approach from the flake rework
done in NixOS/nix. The package.nix here does not provide callPackage
options for all the various settings in the build, and instead the other
places Nix derivations are used (like internal-api-docs) will .overrideAttrs
the normal Nix package derivation. This more closely matches how these
things were structured originally, and results in less churn and more
atomicity in these changes.

In the future, package.nix likely will migrate to have more build
options in the callPackage arguments, but we are also planning to
rewrite the build system anyway.

Change-Id: I170c4e5a4184bab62e1fd75e56db876d4ff116cf
2024-03-11 04:26:35 -06:00