Commit graph

101 commits

Author SHA1 Message Date
jade 0c76195351 build: remove expect as a dependency
I was packaging Lix 2.91 for nixpkgs and was annoyed at the expect
dependency. Turns out that you can replace unbuffer with a pretty-short
Python script.

It became less short after I found out that Linux was converting \n to
\r\n in the terminal subsystem, which was not very funny, but is at
least solved by twiddling termios bits.

Change-Id: I8a2700abcbbf6a9902e01b05b40fa9340c0ab90c
2024-08-10 16:10:16 -07:00
jade 3775b6ac88 package: remove unused autotools code, empty file
I noticed there was some stuff setting configureFlags that definitely do
not do anything with meson, so let's rip them out.

As for the empty file, it was added when I was thinking I needed a fake
C++ target to convince meson to create the necessary dependencies. That
was not in fact possible so it should have never been committed.

Change-Id: Ied4723d8a5d21aed85f352c48b080ab2c977a496
2024-08-09 23:22:11 -07:00
jade 3b902683e9 Merge changes I0373ac01,I7b543967,I537103eb into main
* changes:
  releng: fix the git push
  releng: clarify/update docs, add instructions after tag
  Fix is_maintenance_branch heuristic
2024-08-08 23:12:11 +00:00
jade f3ef0899c7 build: integrate clang-tidy into CI
This still has utterly unacceptably bad output format design that I
would not inflict on anyone I like, but it *does* now exist, and you
*can* find the errors in the log.

Future work would obviously be to fix that and integrate the actual
errors into Gerrit using codechecker or so.

Followup issue: lix-project/lix#457

Fixes: lix-project/lix#147
Change-Id: Ifca22e443d357762125f4ad6bc4f568af3a26c62
2024-08-08 14:53:17 -07:00
jade 8a86f38bca Fix is_maintenance_branch heuristic
This was broken because Nix language's version comparison does not know
how to deal with versions like -rc1 and considers them newer, which is
in this case not desirable.

That in turn led to not tagging 2.90.0 docker images as "latest" since
the heuristic was wrong.

This commit also adds some more cross-checking and failsafes in case the
person running releng does not have a local main branch that is up to
date.

Fixes: lix-project/lix#443
Change-Id: I537103ebab58ae978c00e06972abe14432dd9c80
2024-08-07 20:14:45 -07:00
alois31 780998f4ea Merge "package: improve support for building without BDW-GC" into main 2024-08-07 07:07:28 +00:00
alois31 f84997cbef
package: don't hide system-wide manual pages
When MANPATH is unset or contains an empty component, a reasonable default is
used. Previously (after 3dced96741), when MANPATH
was unset, the shell hook would only place a location containing the Lix manual
pages there, and system-wide manual pages would become unavailable in the
development shell, which is undesired. Fix the issue by including an empty
component in this case.

Change-Id: Ib3c67a831d709fe2a87520e15917eebb59397bd1
2024-08-06 17:18:05 +02:00
alois31 58758c0f87
package: improve support for building without BDW-GC
Expose an option for disabling the BDW-GC build dependency entirely. Fix the
place where one of its headers was included (unnecessarily) without proper
guarding. Finally, use this machinery to exclude BDW-GC from the ASAN builds
entirely (its usage has already been disabled due to compatibility issues
anyway), to ensure this configuration is not regressed again.

Change-Id: I2ebe8094abf67e7d1e99eed971de3e99d071c10b
2024-08-03 06:14:41 +02:00
Jeremy List c907d805bf Merge "package: make aws-sdk-cpp build input optional" into main 2024-08-02 11:42:13 +00:00
Isabel 9eb374dc6d Merge "nix flake show: add the description if it exists" into main 2024-08-02 07:56:06 +00:00
Jeremy List f41190552f
package: make aws-sdk-cpp build input optional
I have added an option to turn off this build input because I'm much
more comfortable when I don't have that type of thing on my computer.
Its default value is true in order to avoid impacting anyone who depends
on AWS features.

Change-Id: Ic57f3c9b9468f422e9fbdcf3ba0fe96177631067
2024-08-02 09:14:48 +12:00
jade 5eecdd3ae9 releng: move officialRelease to version.json
This was causing a few bits of suffering downstream, in particular, in
the NixOS module, which, after this change, can have the
`officialRelease` stuff in *it* completely deleted since we now have
correct defaulting in package.nix for it.

It also eliminates some automated editing of Nix files, which is
certainly always welcome to eliminate.

Fixes: lix-project/lix#406
Change-Id: Id12f3018cff4633e379dbfcbe26b7bc84922bdaf
2024-07-31 14:13:39 -07:00
jade b5c6ce7a53 Add -Werror CI job
We should cause CLs that introduce compiler warnings to fail CI. Sadly
this will only cover Clang, but it will cover Clang for free, so it's
truly impossible to say if it's bad or not.

Change-Id: I45ca20d77251af9671d5cbe0d29cb08c5f1d03c2
2024-07-31 14:13:39 -07:00
jade e51263057f ci: add a asan+ubsan test run on x86_64-linux
This should at least catch out blatantly bad patches that don't pass the
test suite with ASan. We don't do this to the integration tests since
they run on relatively limited-memory VMs and so it may not be super
safe to run an evaluator with leak driven garbage collection for them.

Fixes: lix-project/lix#403
Fixes: lix-project/lix#319
Change-Id: I5267b02626866fd33e8b4d8794344531af679f78
2024-07-31 14:13:39 -07:00
V. a98dce2a1f devendor pegtl
Change-Id: I609a58985fc5210806d0959049a48976ae079c30
2024-07-26 11:22:34 +04:00
alois31 d945e89e19 Merge changes I45d3895f,I541be3ea,Ibe51416d into main
* changes:
  libstore/build: block io_uring
  libstore/build: use an allowlist approach to syscall filtering
  libstore/build: always treat seccomp setup failures as fatal
2024-07-26 07:08:35 +00:00
alois31 127ee1a101
libstore/build: use an allowlist approach to syscall filtering
Previously, system call filtering (to prevent builders from storing files with
setuid/setgid permission bits or extended attributes) was performed using a
blocklist. While this looks simple at first, it actually carries significant
security and maintainability risks: after all, the kernel may add new syscalls
to achieve the same functionality one is trying to block, and it can even be
hard to actually add the syscall to the blocklist when building against a C
library that doesn't know about it yet. For a recent demonstration of this
happening in practice to Nix, see the introduction of fchmodat2 [0] [1].

The allowlist approach does not share the same drawback. While it does require
a rather large list of harmless syscalls to be maintained in the codebase,
failing to update this list (and roll out the update to all users) in time has
rather benign effects; at worst, very recent programs that already rely on new
syscalls will fail with an error the same way they would on a slightly older
kernel that doesn't support them yet. Most importantly, no unintended new ways
of performing dangerous operations will be silently allowed.

Another possible drawback is reduced system call performance due to the larger
filter created by the allowlist requiring more computation [2]. However, this
issue has not convincingly been demonstrated yet in practice, for example in
systemd or various browsers. To the contrary, it has been measured that the the
actual filter constructed here has approximately the same overhead as a very
simple filter blocking only one system call.

This commit tries to keep the behavior as close to unchanged as possible. The
system call list is in line with libseccomp 2.5.5 and glibc 2.39, which are the
latest versions at the point of writing. Since libseccomp 2.5.5 is already a
requirement and the distributions shipping this together with older versions of
glibc are mostly not a thing any more, this should not lead to more build
failures any more.

[0] https://github.com/NixOS/nixpkgs/issues/300635
[1] https://github.com/NixOS/nix/issues/10424
[2] https://github.com/flatpak/flatpak/pull/4462#issuecomment-1061690607

Change-Id: I541be3ea9b249bcceddfed6a5a13ac10b11e16ad
2024-07-25 18:24:40 +02:00
jade 98e8cf9c63 package.nix: remove dead code
Change-Id: Ic0dfcfe27dbf13da4f7f74f5fab8ce6fa718d28f
2024-07-23 21:53:43 +02:00
Isabel d2422771eb
nix flake show: add the description if it exists
(cherry picked from commit 8cd1d02f90eb9915e640c5d370d919fad9833c65)

nix flake show: Only print up to the first new line if it exists.

(cherry picked from commit 5281a44927bdb51bfe6e5de12262d815c98f6fe7)

add tests

(cherry picked from commit 74ae0fbdc70a5079a527fe143c4832d1357011f7)

Handle long strings, embedded new lines and empty descriptions

(cherry picked from commit 2ca7b3afdbbd983173a17fa0a822cf7623601367)

Account for total length of 80

(cherry picked from commit 1cc808c18cbaaf26aaae42bb1d7f7223f25dd364)

docs: add nix flake show description release note

fix: remove white space

nix flake show: trim length based on terminal size

test: account for terminal size

docs(flake-description): before and after commands; add myself to credits

Upstream-PR: https://github.com/NixOS/nix/pull/10980
Change-Id: Ie1c667dc816b3dd81e65a1f5395e57ea48ee0362
2024-07-23 13:21:15 +01:00
V. 85e3b9b871 De-vendor nixfmt
Change-Id: I1a051be495318a507d07f6d0a6b157616e26774c
2024-07-22 21:09:58 +04:00
eldritch horrors dfedbc154f remove sourceToSink, sinkToSource, and boehm patch
Change-Id: I1379841299713175d0225b82a67f50660f9eb5e2
2024-07-16 01:50:16 +00:00
Lunaphied 014410cbf0 Merge "lix-doc: update dependencies and refactor" into main 2024-07-10 17:47:27 +00:00
piegames 3dced96741 devShell: Fix errors with environment variables
Previous code assumed that $MANPATH always exists

Change-Id: I5a4d012045ba6ff9086373b3f46a75d82285d393
2024-07-10 14:22:47 +02:00
Lunaphied 41963df4a5 lix-doc: update dependencies and refactor
This updates the version of rnix used and refactors the code generally
to be more precise and capable in it's identification of both lambdas
and determining which documentation comments are attached.

Change-Id: Ib0dddabd71f772c95077f9d7654023b37a7a1fd2
2024-07-10 02:25:48 +00:00
jade f7d54cb6b1 packaging: make pegtl use the __forDefaults mechanism
This avoids needing to pass it in when callPackage'ing Lix from external
code.

Change-Id: Ie07e84a151e38614064609a2f6dbff165e193be7
2024-06-26 00:44:46 -07:00
jade 4ac2c496d4 Merge "change shebangs of all .sh scripts to bash" into main 2024-06-25 22:18:26 +00:00
jade aceef13682 Merge changes If0ddec6b,Iaa63ed18 into main
* changes:
  Add some release notes for things we did
  packaging: don't build internal api docs by default in dev shells
2024-06-25 22:16:04 +00:00
eldritch horrors e6cd67591b libexpr: rewrite the parser with pegtl instead of flex/bison
this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.

this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.

test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:

  - 40% faster on lix's package.nix at 10000 iterations
  - 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
  - equivalent on all of nixpkgs concatenated at 100 iterations
    (excluding invalid files, each file surrounded with parens)

more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.

Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.408s ± 0.025s
            user: 0.355s | system: 0.033s
  median:   0.389s
  range:    0.388s ... 0.442s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.332s ± 0.024s
            user: 0.279s | system: 0.033s
  median:   0.314s
  range:    0.313s ... 0.361s
  relative: 0.814

---

old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     6.133s ± 0.022s
            user: 5.395s | system: 0.437s
  median:   6.128s
  range:    6.099s ... 6.183s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     5.925s ± 0.025s
            user: 5.176s | system: 0.456s
  median:   5.934s
  range:    5.861s ... 5.943s
  relative: 0.966

---

GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.503s ± 0.027s
            user: 3.731s | system: 0.547s
  median:   4.499s
  range:    4.478s ... 4.541s
  relative: 1

GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.285s ± 0.031s
            user: 3.504s | system: 0.571s
  median:   4.281s
  range:    4.221s ... 4.328s
  relative: 0.951

---

old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     16.475s ± 0.07s
            user: 14.088s | system: 1.572s
  median:   16.495s
  range:    16.351s ... 16.536s
  relative: 1

new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     15.973s ± 0.013s
            user: 13.558s | system: 1.615s
  median:   15.973s
  range:    15.946s ... 15.99s
  relative: 0.97

---

Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
2024-06-25 12:24:58 +00:00
jade 1245340e44 packaging: don't build internal api docs by default in dev shells
These are totally available and you can just turn them on, but they have
very bad dependency tracking and thus bloat incremental change times,
which is not really ok.

Change-Id: Iaa63ed18a789e74fcb757248cd24c3b194afcc80
2024-06-24 15:57:38 -07:00
vigress8 c7af89c797 change shebangs of all .sh scripts to bash
On operating systems where /bin/sh is not Bash, some scripts are invalid
because of bashisms, and building Lix fails with errors like this:
`render-manpage.sh: 3: set: Illegal option -o pipefail`
This modifies all scripts that use a `/bin/sh` shebang to `/usr/bin/env
bash`, including currently POSIX-compliant ones, to prevent any future
confusion.

Change-Id: Ia074cc6db42d40fc59a63726f6194ea0149ea5e0
2024-06-24 14:00:43 -07:00
jade d5637ee790 devShell: guard against running from another directory
I was working on nix-eval-jobs with a dev shell with some shenanigans to
run against a locally built Lix and it was getting really annoying when
`nix develop ../lix#` was messing up my other git repo's hooks.

This is a fix via blunt force, but it is at least obvious how it works.

Change-Id: Ia29eeb5be57ab6a2c88451c00ea18a51e4dfe65e
2024-06-24 13:41:38 -07:00
jade 068576042b packaging: Move binaryTarball to a passthru attr in package.nix
This is motivated by flakes being bad and all the stuff that calls
things by "system" being utterly unable to cope with cross compilation.
So if we go shove it in package.nix it is suddenly usable from cross
contexts.

Usage:

```
nix build -L .#nix-riscv64-linux.binaryTarball
```

Change-Id: I702ebf2ac5bd9d1c57662f968b000073134df336
2024-06-13 15:14:22 -07:00
Pierre Bourdon 248ecb11af dev shell: Add bashInteractive
This was bothersome to me as a zsh+direnv user.

Change-Id: Ia5b54cc63647a5c6ced2b5412e972dac1abf8184
2024-06-12 15:34:23 -07:00
jade 479055aee8 Misc workaround removals since 24.05 upgrade
Change-Id: I9491b103333cb0e25c245199e88365ded7800d2e
2024-06-12 15:34:23 -07:00
Pierre Bourdon f7b6552699 [resubmit] flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
-- message from cl/1418 --

The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

-- jade resubmit changes --

This is a resubmission of https://gerrit.lix.systems/c/lix/+/1418, which
was reverted in https://gerrit.lix.systems/c/lix/+/1432 for breaking CI
evaluation without being detected.

I have run `nix flake check -Lv` on this one before submission and it
passes on my machine and crucially without eval errors, so the CI result
should be accurate.

It seems like someone renamed forbiddenDependenciesRegex to
forbiddenDependenciesRegexes in nixpkgs and also changed the type
incompatibly. That's pretty silly, but at least it's just an eval error.

Also, `xonsh` regressed the availability of `xonsh-unwrapped`, but it
was fixed by us in https://github.com/NixOS/nixpkgs/pull/317636, which
is now in our channel, so we update nixpkgs compared to the original
iteration of this to simply get that.

We originally had a regression related to some reorganization of the
nixpkgs lib test suite in which there was broken parameter passing.
This, too, we got quickfixed in nixpkgs, so we don't need any changes
for it: https://github.com/NixOS/nixpkgs/pull/317772

Related: https://gerrit.lix.systems/c/lix/+/1428
Fixes: lix-project/lix#385

Change-Id: I26d41ea826fec900ebcad0f82a727feb6bcd28f3
2024-06-12 15:34:22 -07:00
jade 8a3d063a49 Merge changes from topic "releng" into main
* changes:
  releng: add prod environment, ready for release
  releng: automatically figure out if we should tag latest for docker
  releng: support multiarch docker images
  manual: rewrite the docker guide now that we have images
  Rewrite docker to be sensible and smaller
  Implement docker upload in the releng tools
2024-06-11 04:45:12 +00:00
jade a986a8dfa1 Merge "Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"" into main 2024-06-10 04:48:12 +00:00
jade 8a09465c3a Revert "flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)"
This reverts commit 28a079f841.

Reason for revert: This caused a pile of regressions in CI, and does not pass nix flake check. Some number of them are fixed in CL: https://gerrit.lix.systems/c/lix/+/1429 but there's more to be fixed.

We should defer this after 2.90.

Change-Id: Ib839d0fcb08eb52094af2b521145e3c1b4e0556f
2024-06-10 04:29:13 +00:00
jade 9aeb314e6a releng: support multiarch docker images
If we don't want to have separate registry tags by architecture (EWWWW),
we need to be able to build multiarch docker images. This is pretty
simple, and just requires making a manifest pointing to each of the
component images.

I was *going* to just do this API prodding with manifest-tool, but it
doesn't support putting metadata on the outer manifest, which is
actually kind of a problem because it then doesn't render the metadata
on github. So I guess we get a simple little containers API
implementation that is 90% auth code.

Change-Id: I8bdd118d4cbc13b23224f2fb174b232432686bea
2024-06-09 20:33:24 -07:00
jade 7dfa2a761e Merge changes from topic "releng" into main
* changes:
  releng: support pushing the manual to docs also
  Expose officialRelease from the flake
  Put into place initial release engineering
2024-06-09 08:28:52 +00:00
jade ff95b980d4 Implement docker upload in the releng tools
This uses skopeo to not think about docker daemons. I, however, noticed
that the docker image we had would have totally terrible cache hits, so
I rewrote it.

Fixes: lix-project/lix#252

Change-Id: I3c5b6c1f3ba0b9dfcac212b2148f390e0cd542b7
2024-06-09 00:30:12 -07:00
Pierre Bourdon 28a079f841
flake: update nixpkgs pin 23.11->24.05 (+ boehmgc compat changes)
The boehmgc changes are bundled into this commit because doing otherwise
would require an annoying dance of "adding compatibility for < 8.2.6 and
>= 8.2.6" then updating the pin then removing the (now unneeded)
compatibility. It doesn't seem worth the trouble to me given the low
complexity of said changes.

Rebased coroutine-sp-fallback.diff patch taken from https://github.com/NixOS/nixpkgs/pull/317227

Change-Id: I8c590e9fe25c0f566d0cfeacb96d8cf50abf12e8
2024-06-09 01:25:53 +02:00
jade 4f94531209 Merge changes from topic "releng" into main
* changes:
  Add meson release note
  Move version to a JSON file so we can have release names
  Remove rl-next-dev
2024-06-07 03:53:31 +00:00
jade bdf1b264ad Expose officialRelease from the flake
Change-Id: If87beb3f31dfb5d59862294ac2e1c821ea864277
2024-06-06 20:53:08 -07:00
Qyriad ec768df004 Merge changes Ic4be41eb,I48db2385 into main
* changes:
  devshells: only enable pch for clang
  build: expose option to enable or disable precompiled std headers
2024-06-06 22:21:52 +00:00
jade 9c77c62e73 Move version to a JSON file so we can have release names
Change-Id: I5ff3396a302565ee5ee6c2db97e048e403779076
2024-06-06 15:08:12 -07:00
jade 1659404626 Add xonsh to the shell
Change-Id: If8f3825d2bdcc3f1d00583a11d890c1c8ab37b9f
2024-06-06 14:50:27 -07:00
jade e0748377dc pname: nix -> lix
This had a regression last time: https://gerrit.lix.systems/c/lix/+/1196

But f3f68fcfa fixed upgrade-nix to not be broken, so this should be ok tbh.

Change-Id: I48ea1359790878bb8ead5d8a4b3f61caa4aabfb5
2024-06-06 20:42:29 +00:00
Qyriad 766e718f67 devshells: only enable pch for clang
clangd seems to break if GCC is using precompiled headers for C++'s
standard library, so this sets -Denable-pch-std=${stdenv.cc.isClang}

Fixes #374.

Change-Id: Ic4be41ebe7576ebcb9c208275596f953c2003109
2024-06-06 12:48:13 -06:00
jade 0f99ed43f1 build-time: remove 20% more by PCH'ing C++ stdlib
It seems like someone implemented precompiled headers a long time ago
and then it never got ported to meson or maybe didn't work at all.

This is, however, blessedly easy to simply implement. I went looking for
`#define` that could affect the result of precompiling the headers, and
as far as I can tell we aren't doing any of that, so this should truly
just be free build time savings.

Previous state:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

New state:
**** Time summary:
Compilation (567 times):
  Parsing (frontend):         1123.0 s
  Codegen & opts (backend):   1078.1 s

I wonder if the "regression" in codegen time is just doing the PCH
operation a few times, because meson does it per-target.

Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30 21:54:21 +00:00