Commit graph

15744 commits

Author SHA1 Message Date
jade 82dc712d93 releng: add prod environment, ready for release
I am *reasonably* confident that this releng infrastructure can actually
build a Lix 2.90 and release it successfully. Let's make it possible to
do, and add some cute colours to the confirmation message.

Change-Id: I85e498b6fb49ffc5e75c0a72c5e45fb1f69030d3
2024-06-09 20:33:24 -07:00
jade ce71d0e9ab releng: automatically figure out if we should tag latest for docker
For example, when releasing from release-2.90, if `main` has a 2.91 tag
ancestor, we know that 2.91 was released, so we should *not* tag latest.

Change-Id: Ia56b17a2ee03bbec74b7c271c742858c690d450d
2024-06-09 20:33:24 -07:00
jade 9aeb314e6a releng: support multiarch docker images
If we don't want to have separate registry tags by architecture (EWWWW),
we need to be able to build multiarch docker images. This is pretty
simple, and just requires making a manifest pointing to each of the
component images.

I was *going* to just do this API prodding with manifest-tool, but it
doesn't support putting metadata on the outer manifest, which is
actually kind of a problem because it then doesn't render the metadata
on github. So I guess we get a simple little containers API
implementation that is 90% auth code.

Change-Id: I8bdd118d4cbc13b23224f2fb174b232432686bea
2024-06-09 20:33:24 -07:00
jade 4392d89eea manual: rewrite the docker guide now that we have images
Change-Id: I5bdf47e67059ae4099552750a47ae070dbe094df
2024-06-09 20:33:24 -07:00
jade 9bb7fb8f69 Rewrite docker to be sensible and smaller
I have checked the image can build things and inspected `diff -ru`
compared to the old image. As far as I can tell it is more or less
the same besides the later git change.

Layers are now 65MB or less, and we aren't against the maxLayers limit
for the broken automatic layering to do anything but shove one store
path in a layer (which is good behaviour, actually).

This uses nix2container which streams images, so the build time is much
shorter.

I have also taken the opportunity to, in addition to fixing the 400MB
single layer (terrible, and what motivated this in the first place),
delete about 200MB of closure size inflicted by git vs gitMinimal
causing both perl and python to get into closure.

People mostly use this thing for CI, so I don't really think you need
advanced git operations, and large git can be added at the user side if
really motivated.

With love for whichever container developer somewhat ironically assumed
that one would not run skopeo in a minimal container that doesn't have a
/var/tmp.

Fixes: #378

Change-Id: Icc3aa20e64446276716fbbb87535fd5b50628010
2024-06-09 20:33:24 -07:00
jade ff95b980d4 Implement docker upload in the releng tools
This uses skopeo to not think about docker daemons. I, however, noticed
that the docker image we had would have totally terrible cache hits, so
I rewrote it.

Fixes: #252

Change-Id: I3c5b6c1f3ba0b9dfcac212b2148f390e0cd542b7
2024-06-09 00:30:12 -07:00
jade 98e8475147 releng: support pushing the manual to docs also
Change-Id: Ifd0b51425ee4955e0230fb2804a6f54ef0fe16e9
2024-06-06 20:53:08 -07:00
jade bdf1b264ad Expose officialRelease from the flake
Change-Id: If87beb3f31dfb5d59862294ac2e1c821ea864277
2024-06-06 20:53:08 -07:00
jade c32a01f9eb Put into place initial release engineering
This can release x86_64-linux binaries to staging, with ephemeral keys.
I think it's good enough to review at least at this point, so we don't
keep adding more stuff to it to make it harder to review.

Change-Id: Ie95e8f35d1252f5d014e819566f170b30eda152e
2024-06-06 20:53:08 -07:00
jade 611b1de441 Add meson release note
Change-Id: I4d2d08dc77a3ab4dce9fbb129c1487aa8c9f1722
2024-06-06 15:08:12 -07:00
jade 9c77c62e73 Move version to a JSON file so we can have release names
Change-Id: I5ff3396a302565ee5ee6c2db97e048e403779076
2024-06-06 15:08:12 -07:00
jade 24057dcb6a Remove rl-next-dev
We realized that there's really no good place to put these dev facing
bulletins, and the user-facing release notes aren't really the worst
place to put them, I guess, and we do kind of hope that it converts
users to devs.

Change-Id: Id9387b2964fe291cb5a3f74ad6344157f19b540c
2024-06-06 15:08:12 -07:00
jade 1659404626 Add xonsh to the shell
Change-Id: If8f3825d2bdcc3f1d00583a11d890c1c8ab37b9f
2024-06-06 14:50:27 -07:00
jade e0748377dc pname: nix -> lix
This had a regression last time: https://gerrit.lix.systems/c/lix/+/1196

But f3f68fcfa fixed upgrade-nix to not be broken, so this should be ok tbh.

Change-Id: I48ea1359790878bb8ead5d8a4b3f61caa4aabfb5
2024-06-06 20:42:29 +00:00
jade 8f9bcd20eb Merge "libstore/filetransfer: fix no-s3 build" into main 2024-06-06 03:08:14 +00:00
Linus Heckemann 609b721425 libstore/filetransfer: fix no-s3 build
Fixes a compiler error that looks like:

error: could not convert '[...]' from 'future<void>' to 'future<nix::FileTransferResult>'
Change-Id: I4aeadfeba0dadfdf133f25e6abce90ede7a86ca6
2024-06-05 15:50:57 -07:00
Pierre Bourdon 6e59b4b407 Merge "tests/nixos: make the tarball-flakes test better reflect real use cases" into main 2024-06-04 15:14:02 +00:00
Pierre Bourdon a3256a9375
tests/nixos: make the tarball-flakes test better reflect real use cases
In most real world cases, the Link header is set on the redirect, not on
the final file. This regressed in Lix earlier and while new unit tests
were added to cover it, this integration test should probably have also
caught it.

Change-Id: I2a9d8d952fff36f2c22cfd751451c2b523f7045c
2024-06-04 08:12:59 +02:00
Olmo Kramer 72d85acba4 nix flake update: add test for multiple inputs from nix#10073
Upstream-PR: https://github.com/NixOS/nix/pull/10073
Change-Id: I53fcb43b387e55439e062e208877afeb88493bb4
2024-06-03 21:50:33 +00:00
Nikodem Rabuliński 5d3910330d Show message about --update-input being replaced by nix flake update
Fixes: #283

Change-Id: I6ee23874cb09f51d788521273076a25ba8764859
2024-06-03 21:50:33 +00:00
Nikodem Rabuliński cc3674ea93 Accept multiple arguments to nix flake update
Fixes: #194

Change-Id: Ia7bd4f7640384be9827dbb7e2c594f0aa5f1aff8
2024-06-03 21:50:33 +00:00
Qyriad c55e93ca23 Revert "nix3: always use the same verbosity default (info)"
This reverts commit d0390b5cf2.

Other parts of the codebase will need to be adjusted in response to a
default verbosity change. Let's just push this to after 2.90.

Fixes #362.
Fixes #367.

Change-Id: I04648473579146851bda41d764adc1ef954c355d
2024-06-01 18:29:19 -06:00
Qyriad d374a9908f Merge "build: fix static linking with a hack" into main 2024-06-01 19:17:13 +00:00
raito b8cb7abcf0 chore: rebrand Nix to Lix when it makes sense
Here's my guide so far:

$ rg '((?!(recursive).*) Nix
(?!(daemon|store|expression|Rocks!|Packages|language|derivation|archive|account|user|sandbox|flake).*))'
-g '!doc/' --pcre2

All items from this query have been tackled. For the documentation side:
that's for #162.

Additionally, all remaining references to github.com/NixOS/nix which
were not relevant were also replaced.

Fixes: #148.
Fixes: #162.
Change-Id: Ib3451fae5cb8ab8cd9ac9e4e4551284ee6794545
Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-06-01 20:31:24 +02:00
Linus Heckemann 5312e60be6 Merge "libfetchers: allow fetching gitlab refs with >1 commit" into main 2024-06-01 09:54:11 +00:00
Qyriad e54d4c9381 build: fix static linking with a hack
This causes libstore, libexpr, libfetchers, and libutil to be linked
with -Wl,--whole-archive to executables, when building statically.

libstore for the store backends, libexpr for the primops, libfetchers
for the fetcher backends I assume(?), and libutil for the nix::logger
initializer (which notably shows in pre-main constructors when HOME is
not owned by the user. cursed.).

This workaround should be removed when #359 is fixed.

Fixes #306.

Change-Id: Ie9ef0154e09a6ed97920ee8ab23810ca5e2de84c
2024-05-31 21:47:16 -06:00
jade c7ca87461d Merge "build-remote: truncate+hash store URI used in lockfile paths" into main 2024-05-31 19:22:32 +00:00
jade 7081889faa Merge "truncate WAL files on exit" into main 2024-05-31 19:21:30 +00:00
jade adedac70fa Merge changes Ifcb0d310,I664366b8,Ibe7cf546 into main
* changes:
  gitignore: delete 90% of it
  build-time: remove 20% more by PCH'ing C++ stdlib
  shellHook: make it actually run
2024-05-31 19:19:29 +00:00
Linus Heckemann 82de36f77a libfetchers: allow fetching gitlab refs with >1 commit
Change-Id: I945c4c5512def9eff728bb67fe3c03ae17f99d6d
2024-05-31 21:12:04 +02:00
jade a75d7a5777 Merge "libutil: fix args assert being thrown on Darwin in nix-eval-jobs" into main 2024-05-31 18:57:13 +00:00
alois31 ff08d95420 Merge "libstore/build: copy ca-certificates too" into main 2024-05-31 16:44:18 +00:00
annalee 713cd7e9e7 truncate WAL files on exit
Fix for https://github.com/NixOS/nix/issues/10300

18a2620273  enabled persistent WAL files that will never get truncated. to fix this, journal_size_limit is set to 2^40, which results in the WAL files being truncated to 0 on exit, as well as limiting the WAL files to 2^40 bytes following a checkpoint.

this aligns lix with the nix change: https://github.com/NixOS/nix/pull/10301

https://www.sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal
https://www.sqlite.org/pragma.html#pragma_journal_size_limit
ed517a7082/src/wal.c (L2518)

PR-Link: https://github.com/lix-project/lix/pull/9

Co-Authored-By: paparodeo <170618376+paparodeo@users.noreply.github.com>
Change-Id: I90ec1a467c92c582ff8c07dd363a4cf789782214
2024-05-31 12:22:15 +00:00
Lunaphied d4b7e6baca build-remote: truncate+hash store URI used in lockfile paths
Fixes: #157
Fixes: #221

Previously the entire escaped store URI was included. This would cause
build failures if a very long or deeply nested path was being used in
the store.

Now, we use the first 48 characters of the URL (escaped), then 16 bytes
of hash of the entire URL. This should never collide and limits the
length of the file name to a bit over 64, which is fine.

Change-Id: Ic1ba690a94e83749567c2c29460b8d1bcf2ac413
2024-05-31 12:18:24 +00:00
jade ac78c1dcd5 libutil: fix args assert being thrown on Darwin in nix-eval-jobs
This is because a dynamic_cast<nix::RootArgs *> of a (n-e-j) MyArgs
returns nullptr even though MyArgs has virtual nix::RootArgs as a
parent.

class MyArgs : virtual public nix::MixEvalArgs,
               virtual public nix::MixCommonArgs,
               virtual nix::RootArgs { ... };

So this should work right?? But it does not. We found out that it's
caused by -fvisibility=hidden in n-e-j, but honestly this code was bad
anyway.

The trivial solution is to simply stop relying on RTTI working properly
here, which is probably better OO architecture anyway. However, I am not
100% confident *this* is sound, since we have this horrible hierarchy:

      Args (defines getRoot)
     /        |           \
RootArgs  MixCommonArgs  MixEvalArgs
(overrides)

I am not confident that this is guaranteed to resolve from Args always
in the case of this override.

Assertion failed: (res), function getRoot, file src/libutil/args.cc, line 67.
6MyArgsProcess 60503 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #4: 0x0000000100b1a41c liblixutil.dylib`nix::Args::processArgs(std::__1::list<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, bool) [inlined] nix::Args::getRoot(this=0x00000001000d0688) at args.cc:67:5 [opt]
   64       std::cout << typeid(*p).name();
   65
   66       auto * res = dynamic_cast<RootArgs *>(p);
-> 67       assert(res);
   68       return *res;
   69   }
   70
Target 0: (nix-eval-jobs) stopped.

(lldb) p this
(MyArgs *) 0x00000001000d0688
(lldb) p *this
(nix::Args) {
  longFlags = size=180  { ... }
  shortFlags = size=4  { ... }
  expectedArgs = size=1  { ... }
  processedArgs = size=0 {}
  hiddenCategories = size=1 {
    [0] = "Options to override configuration settings"
  }
  parent = nullptr
}

We also found that if we did this:
class [[gnu::visibility("default")]] RootArgs : virtual public Args

it would work properly (???!). This is of course, very strange, because
objdump -Ct output on liblixexpr.dylib is identical both with and
without it.

Possibly related: https://www.qt.io/blog/quality-assurance/one-way-dynamic_cast-across-library-boundaries-can-fail-and-how-to-fix-it

Fixes: lix-project/nix-eval-jobs#2
Change-Id: I6b9ed968ed56420a9c4d2dffd18999d78c2761bd
2024-05-31 12:17:06 +00:00
terru - 0c6cb34de6 Merge "document context-dependent keywords" into main 2024-05-31 10:07:50 +00:00
alois31 cf756fdf3c libstore/build: copy ca-certificates too
In b469c6509b, the ca-certificates file was
missed. It should be copied too so that we don't end up bind-mounting a broken
symlink.

Change-Id: Ic9b292d602eb94b0e78f77f2a27a19d24665783c
2024-05-31 07:54:18 +00:00
jade a6b33cb3b2 gitignore: delete 90% of it
*laughs in meson putting it all in build/*

Change-Id: Ifcb0d3104cf9e64c4de91c3a92828899a209d00d
2024-05-30 22:24:55 +00:00
jade 0f99ed43f1 build-time: remove 20% more by PCH'ing C++ stdlib
It seems like someone implemented precompiled headers a long time ago
and then it never got ported to meson or maybe didn't work at all.

This is, however, blessedly easy to simply implement. I went looking for
`#define` that could affect the result of precompiling the headers, and
as far as I can tell we aren't doing any of that, so this should truly
just be free build time savings.

Previous state:
Compilation (551 times):
  Parsing (frontend):         1302.1 s
  Codegen & opts (backend):    956.3 s

New state:
**** Time summary:
Compilation (567 times):
  Parsing (frontend):         1123.0 s
  Codegen & opts (backend):   1078.1 s

I wonder if the "regression" in codegen time is just doing the PCH
operation a few times, because meson does it per-target.

Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30 21:54:21 +00:00
jade e6e5cacabe shellHook: make it actually run
When we changed this in I91cb6eb6668f3a8eace36ecbdb01eb367861d77b to
not run in nested shells, we didn't predict that `nix develop` would do
something ridiculous and append -env to things silently. `nix-shell` of
course does not do this, so we need to tolerate both.

Change-Id: Ibe7cf546823d7358ebc0414ecbe154e3e3194f45
2024-05-30 21:54:21 +00:00
Linus Heckemann 3df013597d libfetchers: handle nonexistent refs in GitLab repos more gracefully
Before:

$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null

After:

$ outputs/out/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error:
       … while updating the lock file of flake 'git+file:///home/linus/projects/lix?ref=refs/heads/fix-gitlab-nonexistent&rev=915f16a619a36237a099b9aa9afed6d14ff613b4'

       … while updating the flake input 'nixpkgs'

       … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'

       error: No commits returned by GitLab API -- does the ref really exist?

Change-Id: Id9bc79d98348500e152ed519bb3ac79a3d15c38d
2024-05-30 21:53:51 +00:00
jade 2f104bbe3b Merge "Revert "tests/filetransfer: reënable on Darwin"" into main 2024-05-30 21:53:38 +00:00
jade 533d469875 Revert "tests/filetransfer: reënable on Darwin"
This reverts commit 285bc67318.

Reason for revert: #364

For some reason this broke `main` even though the change we are reverting passed CI! Mysterious, haunted, etc. Needs more debugging, let's turn it off for now.

Change-Id: Ica4819d61cd35b83eb52985bfcb657e858f025a9
2024-05-30 21:38:32 +00:00
Qyriad 260db1ea64 Merge "build: fix static aws-cpp-sdk" into main 2024-05-30 20:53:49 +00:00
jade c161687b5f Merge "tests/filetransfer: reënable on Darwin" into main 2024-05-30 14:58:23 +00:00
jade 18aa3e1d57 Merge "Remove 100s of CPU time (10%) from build times (1465s -> 1302s)" into main 2024-05-30 14:57:37 +00:00
jade 53d40888ff Merge "unix-domain-socket.cc: add comment explaining why bindConnectProcHelper" into main 2024-05-30 14:56:57 +00:00
Maximilian Bosch 3d78b4847e Merge "release-notes: add missing credits/category to consistent-nix-build entry" into main 2024-05-30 14:38:47 +00:00
Maximilian Bosch 6abac7aacc release-notes: add missing credits/category to consistent-nix-build entry
Change-Id: I737422a2ff9d66be30cc432f8c1ddba9b1e71f4f
2024-05-30 14:05:12 +00:00
Ilya K da95bf8c82 libstore/filetransfer: remove debug print
foo.

Change-Id: I7d7db22f68046d2ecf3b594b4ee6fd9c9dac4be1
2024-05-30 16:42:45 +03:00