this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.
this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.
test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:
- 40% faster on lix's package.nix at 10000 iterations
- 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
- equivalent on all of nixpkgs concatenated at 100 iterations
(excluding invalid files, each file surrounded with parens)
more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.
Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.408s ± 0.025s
user: 0.355s | system: 0.033s
median: 0.389s
range: 0.388s ... 0.442s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.332s ± 0.024s
user: 0.279s | system: 0.033s
median: 0.314s
range: 0.313s ... 0.361s
relative: 0.814
---
old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 6.133s ± 0.022s
user: 5.395s | system: 0.437s
median: 6.128s
range: 6.099s ... 6.183s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 5.925s ± 0.025s
user: 5.176s | system: 0.456s
median: 5.934s
range: 5.861s ... 5.943s
relative: 0.966
---
GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.503s ± 0.027s
user: 3.731s | system: 0.547s
median: 4.499s
range: 4.478s ... 4.541s
relative: 1
GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.285s ± 0.031s
user: 3.504s | system: 0.571s
median: 4.281s
range: 4.221s ... 4.328s
relative: 0.951
---
old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
mean: 16.475s ± 0.07s
user: 14.088s | system: 1.572s
median: 16.495s
range: 16.351s ... 16.536s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
mean: 15.973s ± 0.013s
user: 13.558s | system: 1.615s
median: 15.973s
range: 15.946s ... 15.99s
relative: 0.97
---
Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
This is motivated by flakes being bad and all the stuff that calls
things by "system" being utterly unable to cope with cross compilation.
So if we go shove it in package.nix it is suddenly usable from cross
contexts.
Usage:
```
nix build -L .#nix-riscv64-linux.binaryTarball
```
Change-Id: I702ebf2ac5bd9d1c57662f968b000073134df336
It builds. I have not tested the binaries since I don't have hardware,
but I would be rather surprised if it were broken, given that nix *runs*
on this platform.
Change-Id: I0b474ffcd4a431bf117a303d0b65fa6532113f48
We should not let these regress in CI by having broken dependencies or
similar. Still need to fix the evaluation error checking in
buildbot-nix, but this is a useful step regardless.
Fixes: #383
Change-Id: I3883184165440e66256c989117f2ab2e54c3aafd
-- message from cl/1418 --
The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.
Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227
-- jade resubmit changes --
This is a resubmission of https://gerrit.lix.systems/c/lix/+/1418, which
was reverted in https://gerrit.lix.systems/c/lix/+/1432 for breaking CI
evaluation without being detected.
I have run `nix flake check -Lv` on this one before submission and it
passes on my machine and crucially without eval errors, so the CI result
should be accurate.
It seems like someone renamed forbiddenDependenciesRegex to
forbiddenDependenciesRegexes in nixpkgs and also changed the type
incompatibly. That's pretty silly, but at least it's just an eval error.
Also, `xonsh` regressed the availability of `xonsh-unwrapped`, but it
was fixed by us in https://github.com/NixOS/nixpkgs/pull/317636, which
is now in our channel, so we update nixpkgs compared to the original
iteration of this to simply get that.
We originally had a regression related to some reorganization of the
nixpkgs lib test suite in which there was broken parameter passing.
This, too, we got quickfixed in nixpkgs, so we don't need any changes
for it: https://github.com/NixOS/nixpkgs/pull/317772
Related: https://gerrit.lix.systems/c/lix/+/1428
Fixes: #385
Change-Id: I26d41ea826fec900ebcad0f82a727feb6bcd28f3
I have checked the image can build things and inspected `diff -ru`
compared to the old image. As far as I can tell it is more or less
the same besides the later git change.
Layers are now 65MB or less, and we aren't against the maxLayers limit
for the broken automatic layering to do anything but shove one store
path in a layer (which is good behaviour, actually).
This uses nix2container which streams images, so the build time is much
shorter.
I have also taken the opportunity to, in addition to fixing the 400MB
single layer (terrible, and what motivated this in the first place),
delete about 200MB of closure size inflicted by git vs gitMinimal
causing both perl and python to get into closure.
People mostly use this thing for CI, so I don't really think you need
advanced git operations, and large git can be added at the user side if
really motivated.
With love for whichever container developer somewhat ironically assumed
that one would not run skopeo in a minimal container that doesn't have a
/var/tmp.
Fixes: #378
Change-Id: Icc3aa20e64446276716fbbb87535fd5b50628010
This can release x86_64-linux binaries to staging, with ephemeral keys.
I think it's good enough to review at least at this point, so we don't
keep adding more stuff to it to make it harder to review.
Change-Id: Ie95e8f35d1252f5d014e819566f170b30eda152e
We realized that there's really no good place to put these dev facing
bulletins, and the user-facing release notes aren't really the worst
place to put them, I guess, and we do kind of hope that it converts
users to devs.
Change-Id: Id9387b2964fe291cb5a3f74ad6344157f19b540c
Here's my guide so far:
$ rg '((?!(recursive).*) Nix
(?!(daemon|store|expression|Rocks!|Packages|language|derivation|archive|account|user|sandbox|flake).*))'
-g '!doc/' --pcre2
All items from this query have been tackled. For the documentation side:
that's for #162.
Additionally, all remaining references to github.com/NixOS/nix which
were not relevant were also replaced.
Fixes: #148.
Fixes: #162.
Change-Id: Ib3451fae5cb8ab8cd9ac9e4e4551284ee6794545
Signed-off-by: Raito Bezarius <raito@lix.systems>
Surely if you have unreleased changes you want them on a page right?
`officialRelease` means "this is a *release version*", which is a
reasonable case to not want it, but we are not that here.
I understand wanting to be able to turn it off for deps reasons or
something, but other than that, uhh, seems better to just turn it on
always; it is basically free compute-wise to the point we run it on
pre-commit.
Part two of fixing lix#297.
Fixes: #297
Change-Id: I0f8dd1ae42458df371aef529c456e47a7ac04ae0
Now instead of a derivation overridden from Lix, we use a mkShell
derivation parameterized on an already called package.nix. This also
lets callPackage take care of the buildPackages distinction for the
devShell.
Change-Id: I5ddfec40d83fa6136032da7606fe6d3d5014ef42
ClangBuildAnalyzer doesn't build on i686-linux due to
`long long int`/`size_t` conversion errors, so let's just exclude it
from the devshell on that platform
Change-Id: If1077a7b3860db4381999c8e304f6d4b2bc96a05
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.
Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:
> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.
Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.
This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
is needed to have the correct syscall number available via
`__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:
$ rg --pcre2 'define __NR_fchmodat2 (?!452)'
sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
58:#define __NR_fchmodat2 1073742276
sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
67:#define __NR_fchmodat2 6452
sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
62:#define __NR_fchmodat2 5452
sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
70:#define __NR_fchmodat2 4452
sysdeps/unix/sysv/linux/alpha/arch-syscall.h
59:#define __NR_fchmodat2 562
I added a small regression-test to the setuid integration-test that
attempts to set the suid bit on a file using the fchmodat2 syscall.
I confirmed that the test fails without the change in
local-derivation-goal.
Additionally, we require libseccomp 2.5.5 or greater now: as it turns
out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Lix against the project's flake.
To work around that
* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
scheduled[3].
* the package now uses libseccomp 2.5.5 on its own already. This is to
provide a quick fix since the correct fix for 23.11 is still a staging cycle
away.
We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷
Relevant PRs in CppNix:
* https://github.com/NixOS/nix/pull/10591
* https://github.com/NixOS/nix/pull/10501
[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
[3] https://github.com/NixOS/nixpkgs/pull/306070
(cherry picked from commit ba6804518772e6afb403dd55478365d4b863c854)
Change-Id: I6921ab5a363188c6bff617750d00bb517276b7fe
This commit makes Meson the default buildsystem for Lix.
The Make buildsystem is now deprecated and will be removed soon, but has
not yet, which will be done in a later commit when all seems good. The
mesonBuild jobs have been removed, and have not been replaced with
equivalent jobs to ensure the Make buildsystem still works.
The full, new commands in a development shell are:
$ meson setup ./build "--prefix=$out" $mesonFlags
(A simple `meson setup ./build` will also build, but will do a different
thing, not having the settings from package.nix applied.)
$ meson compile -C build
$ meson test -C build --suite=check
$ meson install -C build
$ meson test -C build --suite=installcheck
(Check and installcheck may both be done after install, allowing you to
omit the --suite argument entirely, but this is the order package.nix
runs them in.)
If tests fail and Meson helpfully has no output for why, use the
`--print-error-logs` option to `meson test`. Why this is not the default
I cannot explain.
If you change a setting in the buildsystem, most cases will
automatically regenerate the Meson configuration, but some cases, like
trying to build a specific target whose name is new to the buildsystem
(e.g. `meson compile -C build src/libmelt/libmelt.dylib`, when
`libmelt.dylib` did not exist as a target the last time the buildsystem
was generated), then you can reconfigure using new settings but
existing options, and only recompiling stuff affected by the changes:
$ meson setup --reconfigure build
Note that changes to the default values in `meson.options` or in the
`default_options :` argument to project() are NOT propagated with
`--reconfigure`.
If you want a totally clean build, you can use:
$ meson setup --wipe build
That will work regardless of if `./build` exists or not.
Specific, named targets may be addressed in
`meson build -C build <target>` with the "target ID" if there is one,
which is the first string argument passed to target functions that
have one, and unrelated to the variable name, e.g.:
libexpr_dylib = library('nixexpr', …)
can be addressed with:
$ meson compile -C build nixexpr
All targets may be addressed as their output, relative to the build
directory, e.g.:
$ meson compile -C build src/libexpr/libnixexpr.so
But Meson does not consider intermediate files like object files
targets. To build a specific object file, use Ninja directly and
specify the output file relative to the build directory:
$ ninja -C build src/libexpr/libnixexpr.so.p/nixexpr.cc.o
To inspect the canonical source of truth on what the state of the
buildsystem configuration is, use:
$ meson introspect
Have fun!
Change-Id: Ia3e7b1e6fae26daf3162e655b4ded611a5cd57ad
This commit adds the capability for building the Doxygen internal API
docs in the Meson buildsystem, and also makes doing so the default for
the internal-api-docs hydra job. Aside from the /nix-support directory,
which differed only by the hash part of a store path, the outputs of
hydraJobs.internal-api-docs before and after this commit were
bit-for-bit identical on my machine.
Change-Id: I98f0017891c25b06866c15f7652fe74f706ec8e1
This is in our style guide, we can cheaply enforce it, let's do it.
```
$ pre-commit
check-case-conflicts.....................................................Passed
check-executables-have-shebangs..........................................Passed
check-headers............................................................Failed
- hook id: check-headers
- exit code: 1
Missing pattern @file in file src/libexpr/value.hh
We found some header files that don't conform to the style guide.
The Lix style guide requests that header files:
- Begin with `#pragma once` so they only get parsed once
- Contain a doxygen comment (`/**` or `///`) containing `@file`, for
example, `///@file`, which will make doxygen generate docs for them.
When adding that, consider also adding a `@brief` with a sentence
explaining what the header is for.
For more details: https://wiki.lix.systems/link/3#bkmrk-header-files
check-merge-conflicts....................................................Passed
check-shebang-scripts-are-executable.....................................Passed
check-symlinks.......................................(no files to check)Skipped
end-of-file-fixer........................................................Passed
mixed-line-endings.......................................................Passed
no-commit-to-branch......................................................Passed
release-notes........................................(no files to check)Skipped
treefmt..................................................................Passed
trim-trailing-whitespace.................................................Passed
```
Fixes: #233
Change-Id: I77150b9298c844ffedd0f85cc5250ae9208502e3
The flake for pre-commit-checks is rather questionable. We ignored
it so it uses our own nixpkgs and doesn't reimport nixpkgs. This should
save a couple of seconds of eval time!
Change-Id: I4584982beb32e0122f791fa29f6a544bdbb9e201
package.nix previously needed this callPackage'd externally, which
didn't make a lot of sense to us since this is an internal dependency.
Thus we changed it to make it more self contained.
Change-Id: I4935bc0bc80e1a132bc9b1519e917791da95037c
Some of this code existed for installer tests, and indeed its removal is
an indication that our daemon cross-compatibility tests were removed.
Although these are not like, super critical tests, we would like to
restore them.
See: #33
Change-Id: I75c733b25c00eca3a9676d498703bbfc1d6ec21b
The following command is now sufficient to build Lix from outside of the
flake:
nix-build -E 'let pkgs = import <nixpkgs> { }; in pkgs.callPackage
./package.nix { build-release-notes = false; nix-doc = pkgs.callPackage
./nix-doc/package.nix { }; }'
Change-Id: Ie6b14b446480ac07c7266d4fba20042b04cc35b9
follow-up to 32eaa8a29[1] "flake: move release note checks to hydraJobs",
this commit fixes a load-bearing typo for`checks.rl-next` and
`checks.rl-next-dev`.
[1]: 32eaa8a2910793538deab31f85534faf7e722ef7
Change-Id: I9383ed21f7eccc337c0c2f65525418b735a94a1d
In our view it really doesn't make sense to not have this in in
package.nix in some way. These patches aren't just for performance or
something -- Lix flat out doesn't build without these patches.
(Arguably that makes them a buildsystem responsibility as well, but that
can wait for when we're ready to start adding subproject fallback
dependency resolution to Meson.)
This is a step towards making `package.nix` more self-sufficient and
`callPackage`able without excessive external logic.
With this change the following command is enough to build Lix from out
of the flake:
nix-build -E 'let pkgs = import <nixpkgs> { }; in pkgs.callPackage
./package.nix { build-release-notes = false; inherit (pkgs.lib) fileset;
nix-doc = pkgs.callPackage ./nix-doc/package.nix { }; }'
Change-Id: Ia37fe8171f87d3293033de8be07d9bab12716f1d
For a long time `nix repl` has supported displaying documentation set on
builtins, however, it has long been convention to use Markdown comments
on Nix functions themselves for documentation. This exposes that
information to `nix repl` users in a nice and formatted way.
NixOS/rfcs#145 doc-comments are primarily what this feature is intended
to consume, however, support for lambda documentation in the repl is
experimental. We do our best effort to support the RFC here.
These changes are based on [the nix-doc library](https://github.com/lf-/nix-doc) and
are licensed under the terms described in the relevant source files.
Change-Id: Ic6fe947d39a22540705d890737e336c4720b0a22
eldritch horrors
Robert Hensing
jade
Pierre Bourdon
raito
Maximilian Bosch
Qyriad
Patrick Jackson
Rebecca Turner
Lunaphied
Artemis Tosini