libutil: BrotliDecompression{Sink -> Source}

Change-Id: I9579dd08f7bd0f927bde9d3128515b0cee15f320

.github/CODEOWNERS

@@ -1,18 +0,0 @@
-# Pull requests concerning the listed files will automatically invite the respective maintainers as reviewers.
-# This file is not used for denoting any kind of ownership, but is merely a tool for handling notifications.
-#
-# Merge permissions are required for maintaining an entry in this file.
-# For documentation on this mechanism, see https://help.github.com/articles/about-codeowners/
-
-# Default reviewers if nothing else matches
-* @edolstra
-
-# This file
-.github/CODEOWNERS @edolstra
-
-# Public documentation
-/doc @fricklerhandwerk
-*.md @fricklerhandwerk
-
-# Libstore layer
-/src/libstore @thufschmitt

.github/ISSUE_TEMPLATE/installer.md

@@ -9,7 +9,7 @@ assignees: ''
 
 ## Platform
 
-<!-- select the platform on which you tried to install Nix -->
+<!-- select the platform on which you tried to install Lix -->
 
 - [ ] Linux: <!-- state your distribution, e.g. Arch Linux, Ubuntu, ... -->
 - [ ] macOS

.github/ISSUE_TEMPLATE/missing_documentation.md

@@ -19,9 +19,10 @@ assignees: ''
 
 <!-- make sure this issue is not redundant or obsolete -->
 
-- [ ] checked [latest Nix manual] \([source])
+- [ ] checked [latest Lix manual] \([source]\)
-- [ ] checked [open documentation issues and pull requests] for possible duplicates
+- [ ] checked [documentation issues] and [recent documentation changes] for possible duplicates
 
-[latest Nix manual]: https://nixos.org/manual/nix/unstable/
+[latest Nix manual]: https://docs.lix.systems/manual/lix/nightly
-[source]: https://github.com/NixOS/nix/tree/master/doc/manual/src
+[source]: https://git.lix.systems/lix-project/lix/src/main/doc/manual/src
-[open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation
+[documentation issues]: https://git.lix.systems/lix-project/lix/issues?labels=151&state=all
+[recent documentation changes]: https://gerrit.lix.systems/q/p:lix+path:%22%5Edoc/manual/.*%22

.github/STALE-BOT.md

@@ -1,35 +0,0 @@
-# Stale bot information
-
-- Thanks for your contribution!
-- To remove the stale label, just leave a new comment.
-- _How to find the right people to ping?_ → [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
-- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
-
-## Suggestions for PRs
-
-1. GitHub sometimes doesn't notify people who commented / reviewed a PR previously, when you (force) push commits. If you have addressed the reviews you can [officially ask for a review](https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from those who commented to you or anyone else.
-2. If it is unfinished but you plan to finish it, please mark it as a draft.
-3. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work.
-4. To get things rolling again, rebase the PR against the target branch and address valid comments.
-5. If you need a review to move forward, ask in [the Discourse thread for PRs that need help](https://discourse.nixos.org/t/prs-in-distress/3604).
-6. If all you need is a merge, check the git history to find and [request reviews](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from people who usually merge related contributions.
-
-## Suggestions for issues
-
-1. If it is resolved (either for you personally, or in general), please consider closing it.
-2. If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough.
-3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [our Discourse Forum](https://discourse.nixos.org/).
-4. As with all open source projects, your best option is to submit a Pull Request that addresses this issue. We :heart: this attitude!
-
-**Memorandum on closing issues**
-
-Don't be afraid to close an issue that holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.
-
-## Useful GitHub search queries
-
-- [Open PRs with any stale-bot interaction](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+)
-- [Open PRs with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22)
-- [Open PRs with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)
-- [Open Issues with any stale-bot interaction](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+)
-- [Open Issues with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22+)
-- [Open Issues with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)

.github/dependabot.yml

@@ -1,6 +0,0 @@
-version: 2
-updates:
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "weekly"

.github/labeler.yml

@@ -1,23 +0,0 @@
-"documentation":
-  - doc/manual/*
-  - src/nix/**/*.md
-
-"store":
-  - src/libstore/store-api.*
-  - src/libstore/*-store.*
-
-"fetching":
-  - src/libfetchers/**/*
-
-"repl":
-  - src/libcmd/repl.*
-  - src/nix/repl.*
-
-"new-cli":
-  - src/nix/**/*
-
-"with-tests":
-  # Unit tests
-  - src/*/tests/**/*
-  # Functional and integration tests
-  - tests/functional/**/*

.github/stale.yml

@@ -1,9 +0,0 @@
-# Configuration for probot-stale - https://github.com/probot/stale
-daysUntilStale: 180
-daysUntilClose: false
-exemptLabels:
-  - "critical"
-  - "never-stale"
-staleLabel: "stale"
-markComment: false
-closeComment: false

boehmgc-coroutine-sp-fallback.diff

@@ -1,56 +1,8 @@
-diff --git a/darwin_stop_world.c b/darwin_stop_world.c
-index 0468aaec..b348d869 100644
---- a/darwin_stop_world.c
-+++ b/darwin_stop_world.c
-@@ -356,6 +356,7 @@ GC_INNER void GC_push_all_stacks(void)
-   int nthreads = 0;
-   word total_size = 0;
-   mach_msg_type_number_t listcount = (mach_msg_type_number_t)THREAD_TABLE_SZ;
-+  size_t stack_limit;
-   if (!EXPECT(GC_thr_initialized, TRUE))
-     GC_thr_init();
-
-@@ -411,6 +412,19 @@ GC_INNER void GC_push_all_stacks(void)
-             GC_push_all_stack_sections(lo, hi, p->traced_stack_sect);
-           }
-           if (altstack_lo) {
-+            // When a thread goes into a coroutine, we lose its original sp until
-+            // control flow returns to the thread.
-+            // While in the coroutine, the sp points outside the thread stack,
-+            // so we can detect this and push the entire thread stack instead,
-+            // as an approximation.
-+            // We assume that the coroutine has similarly added its entire stack.
-+            // This could be made accurate by cooperating with the application
-+            // via new functions and/or callbacks.
-+            stack_limit = pthread_get_stacksize_np(p->id);
-+            if (altstack_lo >= altstack_hi || altstack_lo < altstack_hi - stack_limit) { // sp outside stack
-+              altstack_lo = altstack_hi - stack_limit;
-+            }
-+
-             total_size += altstack_hi - altstack_lo;
-             GC_push_all_stack(altstack_lo, altstack_hi);
-           }
-diff --git a/include/gc.h b/include/gc.h
-index edab6c22..f2c61282 100644
---- a/include/gc.h
-+++ b/include/gc.h
-@@ -2172,6 +2172,11 @@ GC_API void GC_CALL GC_win32_free_heap(void);
-         (*GC_amiga_allocwrapper_do)(a,GC_malloc_atomic_ignore_off_page)
- #endif /* _AMIGA && !GC_AMIGA_MAKINGLIB */
-
-+#if !__APPLE__
-+/* Patch doesn't work on apple */
-+#define NIX_BOEHM_PATCH_VERSION 1
-+#endif
-+
- #ifdef __cplusplus
-   } /* extern "C" */
- #endif
 diff --git a/pthread_stop_world.c b/pthread_stop_world.c
-index b5d71e62..aed7b0bf 100644
+index 2b45489..0e6d8ef 100644
 --- a/pthread_stop_world.c
 +++ b/pthread_stop_world.c
-@@ -768,6 +768,8 @@ STATIC void GC_restart_handler(int sig)
+@@ -776,6 +776,8 @@ STATIC void GC_restart_handler(int sig)
  /* world is stopped.  Should not fail if it isn't.                      */
  GC_INNER void GC_push_all_stacks(void)
  {
@@ -59,20 +11,23 @@ index b5d71e62..aed7b0bf 100644
      GC_bool found_me = FALSE;
      size_t nthreads = 0;
      int i;
-@@ -851,6 +853,37 @@ GC_INNER void GC_push_all_stacks(void)
+@@ -868,6 +870,40 @@ GC_INNER void GC_push_all_stacks(void)
-           hi = p->altstack + p->altstack_size;
+             hi = p->altstack + p->altstack_size;
+ #         endif
            /* FIXME: Need to scan the normal stack too, but how ? */
-           /* FIXME: Assume stack grows down */
 +        } else {
-+#ifdef HAVE_PTHREAD_ATTR_GET_NP
++          #ifdef HAVE_PTHREAD_ATTR_GET_NP
-+          if (!pthread_attr_init(&pattr)
++          if (pthread_attr_init(&pattr) != 0) {
-+              || !pthread_attr_get_np(p->id, &pattr))
++            ABORT("GC_push_all_stacks: pthread_attr_init failed!");
-+#else /* HAVE_PTHREAD_GETATTR_NP */
++          }
-+          if (pthread_getattr_np(p->id, &pattr))
++          if (pthread_attr_get_np(p->id, &pattr) != 0) {
-+#endif
++            ABORT("GC_push_all_stacks: pthread_attr_get_np failed!");
-+          {
++          }
++          #else
++          if (pthread_getattr_np(p->id, &pattr)) {
 +            ABORT("GC_push_all_stacks: pthread_getattr_np failed!");
 +          }
++          #endif
 +          if (pthread_attr_getstacksize(&pattr, &stack_limit)) {
 +            ABORT("GC_push_all_stacks: pthread_attr_getstacksize failed!");
 +          }
@@ -95,5 +50,5 @@ index b5d71e62..aed7b0bf 100644
 +          #error "STACK_GROWS_UP not supported in boost_coroutine2 (as of june 2021), so we don't support it in Nix."
 +          #endif
          }
-         GC_push_all_stack_sections(lo, hi, traced_stack_sect);
+ #       ifdef STACKPTR_CORRECTOR_AVAILABLE
- #       ifdef STACK_GROWS_UP
+           if (GC_sp_corrector != 0)

boehmgc-traceable_allocator-public.diff

@@ -1,12 +0,0 @@
-diff --git a/include/gc_allocator.h b/include/gc_allocator.h
-index 597c7f13..587286be 100644
---- a/include/gc_allocator.h
-+++ b/include/gc_allocator.h
-@@ -312,6 +312,7 @@ public:
-
- template<>
- class traceable_allocator<void> {
-+public:
-   typedef size_t      size_type;
-   typedef ptrdiff_t   difference_type;
-   typedef void*       pointer;

doc/manual/meson.build

@@ -110,6 +110,7 @@ manual = custom_target(
     builtins_md,
     builtin_constants_md,
     rl_next_generated,
+    nix,
   ],
   output : [
     'manual',
@@ -186,6 +187,7 @@ foreach command : nix_nested_manpages
       ],
       input : [
         manual_md,
+        nix,
       ],
       output : command[0] + '-' + page + '.1',
       install : true,
@@ -298,6 +300,7 @@ foreach page : nix3_manpages
     input : [
       'render-manpage.sh',
       manual_md,
+      nix,
     ],
     output : page + '.1',
     install : true,
@@ -341,6 +344,7 @@ foreach entry : nix_manpages
       'render-manpage.sh',
       manual_md,
       entry.get(3, []),
+      nix,
     ],
     output : '@0@.@1@'.format(entry[0], entry[1]),
     install : true,

doc/manual/rl-next/addDrvOutputDependencies.md

@@ -1,14 +0,0 @@
----
-synopsis: "Add a builtin `addDrvOutputDependencies`"
-prs: 9216
-issues: 7910
-credits: [ericson, horrors]
-category: Features
----
-
-This builtin allows taking a `drvPath`-like string and turning it into a string
-with context such that, when it lands in a derivation, it will create
-dependencies on *all the outputs* in its closure (!). Although `drvPath` does this
-today, this builtin starts forming a path to migrate to making `drvPath` have a
-more normal and less surprising string context behaviour (see linked issue and
-PR for more details).

doc/manual/rl-next/always-allow-substitutes.md

@@ -1,13 +0,0 @@
----
-synopsis: "Add an option `always-allow-substitutes` to ignore `allowSubstitutes` in derivations"
-prs: 8047
-credits: [lovesegfault, horrors]
-category: Improvements
----
-
-You can set this setting to force a system to always allow substituting even
-trivial derivations like `pkgs.writeText`. This is useful for
-[`nix-fast-build --skip-cached`][skip-cached] and similar to be able to also
-ignore trivial derivations.
-
-[skip-cached]: https://github.com/Mic92/nix-fast-build?tab=readme-ov-file#avoiding-redundant-package-downloads

doc/manual/rl-next/better-errors-in-nix-repl.md

@@ -1,42 +0,0 @@
----
-synopsis: Concise error printing in `nix repl`
-prs: 9928
-cls: 811
-category: Improvements
-credits: 9999years
----
-
-Previously, if an element of a list or attribute set threw an error while
-evaluating, `nix repl` would print the entire error (including source location
-information) inline. This output was clumsy and difficult to parse:
-
-```
-nix-repl> { err = builtins.throw "uh oh!"; }
-{ err = «error:
-       … while calling the 'throw' builtin
-         at «string»:1:9:
-            1| { err = builtins.throw "uh oh!"; }
-             |         ^
-
-       error: uh oh!»; }
-```
-
-Now, only the error message is displayed, making the output much more readable.
-```
-nix-repl> { err = builtins.throw "uh oh!"; }
-{ err = «error: uh oh!»; }
-```
-
-However, if the whole expression being evaluated throws an error, source
-locations and (if applicable) a stack trace are printed, just like you'd expect:
-
-```
-nix-repl> builtins.throw "uh oh!"
-error:
-       … while calling the 'throw' builtin
-         at «string»:1:1:
-            1| builtins.throw "uh oh!"
-             | ^
-
-       error: uh oh!
-```

doc/manual/rl-next/build-timing-analysis.md

@@ -1,22 +0,0 @@
----
-synopsis: Clang build timing analysis
-cls: 587
-category: Development
----
-
-We now have Clang build profiling available, which generates Chrome
-tracing files for each compilation unit. To enable it, run `meson configure
-build -Dprofile-build=enabled` in a Clang stdenv (`nix develop
-.#native-clangStdenvPackages`) then rerun the compilation.
-
-If you want to make the build go faster, do a clang build with meson, then run
-`maintainers/buildtime_report.sh build`, then contemplate how to improve the
-build time.
-
-You can also look at individual object files' traces in
-<https://ui.perfetto.dev>.
-
-See [the wiki page][improving-build-times-wiki] for more details on how to do
-this.
-
-[improving-build-times-wiki]: https://wiki.lix.systems/link/8#bkmrk-page-title

doc/manual/rl-next/consistent-nix-build.md

@@ -1,9 +0,0 @@
----
-synopsis: Show all FOD errors with `nix build --keep-going`
-credits: [ma27]
-category: Improvements
-cls: [1108]
----
-
-`nix build --keep-going` now behaves consistently with `nix-build --keep-going`. This means
-that if e.g. multiple FODs fail to build, all hash mismatches are displayed.

doc/manual/rl-next/cve-fod-fix.md

@@ -1,21 +0,0 @@
----
-synopsis: "Fix CVE-2024-27297 (GHSA-2ffj-w4mj-pg37)"
-cls: 266
-credits: [puck, jade, thufschmitt, tomberek, valentin]
-category: Fixes
----
-
-Since Lix fixed-output derivations run in the host network namespace (which we
-wish to change in the future, see
-[lix#285](https://git.lix.systems/lix-project/lix/issues/285)), they may open
-abstract-namespace Unix sockets to each other and to programs on the host. Lix
-contained a now-fixed time-of-check/time-of-use vulnerability where one
-derivation could send writable handles to files in their final location in the
-store to another over an abstract-namespace Unix socket, exit, then the other
-derivation could wait for Lix to hash the paths and overwrite them.
-
-The impact of this vulnerability is that two malicious fixed-output derivations
-could create a poisoned path for the sources to Bash or similarly important
-software containing a backdoor, leading to local privilege execution.
-
-CppNix advisory: https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37

doc/manual/rl-next/debugger-locals-for-let-expressions.md

@@ -1,11 +0,0 @@
----
-synopsis: "`--debugger` can now access bindings from `let` expressions"
-prs: 9918
-issues: 8827
-category: Fixes
-credits: 9999years
----
-
-Breakpoints and errors in the bindings of a `let` expression can now access
-those bindings in the debugger. Previously, only the body of `let` expressions
-could access those bindings.

doc/manual/rl-next/debugger-on-trace.md

@@ -1,11 +0,0 @@
----
-synopsis: Enter the `--debugger` when `builtins.trace` is called if `debugger-on-trace` is set
-prs: 9914
-category: Features
-credits: 9999years
----
-
-If the `debugger-on-trace` option is set and `--debugger` is given,
-`builtins.trace` calls will behave similarly to `builtins.break` and will enter
-the debug REPL. This is useful for determining where warnings are being emitted
-from.

doc/manual/rl-next/deprecate-online-flake-registry.md

@@ -1,16 +0,0 @@
----
-synopsis: "Deprecate the online flake registries and vendor the default registry"
-cls: 1127
-credits: midnightveil
-issues: [fj#183, fj#110, fj#116, 8953, 9087]
-category: Breaking Changes
----
-
-The online flake registry [https://channels.nixos.org/flake-registry.json](https://channels.nixos.org/flake-registry.json) is not pinned in any way,
-and the targets of the indirections can both update or change entirely at any
-point. Furthermore, it is refetched on every use of a flake reference, even if
-there is a local flake reference, and even if you are offline (which breaks).
-
-For now, we deprecate the (any) online flake registry, and vendor a copy of the
-current online flake registry. This makes it work offline, and ensures that
-it won't change in the future.

doc/manual/rl-next/drop-vendored-toml11.md

@@ -1,9 +0,0 @@
----
-synopsis: Stop vendoring toml11
-cls: 675
-category: Packaging
-credits: winter
----
-
-We don't apply any patches to it, and vendoring it locks users into
-bugs (it hasn't been updated since its introduction in late 2021).

doc/manual/rl-next/drv-string-parse-hang.md

@@ -1,8 +0,0 @@
----
-synopsis: Fix handling of truncated `.drv` files.
-prs: 9673
-category: Fixes
-credits: horrors
----
-
-Previously a `.drv` that was truncated in the middle of a string would case nix to enter an infinite loop, eventually exhausting all memory and crashing.

doc/manual/rl-next/dup-attr-errors.md

@@ -1,24 +0,0 @@
----
-synopsis: Duplicate attribute reports are more accurate
-cls: 557
-credits: horrors
-category: Improvements
----
-
-Duplicate attribute errors are now more accurate, showing the path at which an error was detected rather than the full, possibly longer, path that caused the error.
-Error reports are now
-```ShellSession
-$ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
-error: attribute 'a.b' already defined at «string»:1:3
-       at «string»:1:12:
-            1| { a.b = 1; a.b.c.d = 1;
-             |            ^
-```
-instead of
-```ShellSession
-$ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
-error: attribute 'a.b.c.d' already defined at «string»:1:3
-       at «string»:1:12:
-            1| { a.b = 1; a.b.c.d = 1;
-             |            ^
-```

doc/manual/rl-next/empty-search-regex.md

@@ -1,8 +0,0 @@
----
-synopsis: Disallow empty search regex in `nix search`
-prs: 9481
-credits: [iFreilicht, horrors]
-category: Miscellany
----
-
-[`nix search`](@docroot@/command-ref/new-cli/nix3-search.md) now requires a search regex to be passed. To show all packages, use `^`.

doc/manual/rl-next/enable-coredumps.md

@@ -1,13 +0,0 @@
----
-synopsis: "Add an option `enable-core-dumps` that enables core dumps from builds"
-cls: 1088
-credits: midnightveil
-category: Features
----
-
-In the past, Lix disabled core dumps by setting the soft `RLIMIT_CORE` to 0
-unconditionally. Although this rlimit could be altered from the builder since
-it is just the soft limit, this was kind of annoying to do. By passing
-`--option enable-core-dumps true` to an offending build, one can now cause the
-core dumps to be handled by the system in the normal way (winding up in
-`coredumpctl`, say, on Linux).

doc/manual/rl-next/enter-debugger-more-reliably-in-let-and-calls.md

@@ -1,27 +0,0 @@
----
-synopsis: The `--debugger` will start more reliably in `let` expressions and function calls
-prs: 9917
-issues: 6649
-credits: [9999years, horrors]
-category: Fixes
----
-
-Previously, if you attempted to evaluate this file with the debugger:
-
-```nix
-let
-  a = builtins.trace "before inner break" (
-    builtins.break "hello"
-  );
-  b = builtins.trace "before outer break" (
-    builtins.break a
-  );
-in
-  b
-```
-
-Lix would correctly enter the debugger at `builtins.break a`, but if you asked
-it to `:continue`, it would skip over the `builtins.break "hello"` expression
-entirely.
-
-Now, Lix will correctly enter the debugger at both breakpoints.

doc/manual/rl-next/env-size-reduction.md

@@ -1,10 +0,0 @@
----
-synopsis: Reduce eval memory usage and wall time
-prs: 9658
-cls: 207
-credits: horrors
-category: Improvements
----
-
-Reduce the size of the `Env` struct used in the evaluator by a pointer, or 8 bytes on most modern machines.
-This reduces memory usage during eval by around 2% and wall time by around 3%.

doc/manual/rl-next/eval-system.md

@@ -1,14 +0,0 @@
----
-synopsis: Add new `eval-system` setting
-prs: 4093
-credits: [matthewbauer, horrors]
-category: Features
----
-
-Add a new `eval-system` option.
-Unlike `system`, it just overrides the value of `builtins.currentSystem`.
-This is more useful than overriding `system`, because you can build these derivations on remote builders which can work on the given system.
-In contrast, `system` also effects scheduling which will cause Lix to build those derivations locally even if that doesn't make sense.
-
-`eval-system` only takes effect if it is non-empty.
-If empty (the default) `system` is used as before, so there is no breakage.

doc/manual/rl-next/fchmodat2-sandbox.md

@@ -1,10 +0,0 @@
----
-synopsis: Creating setuid/setgid binaries with fchmodat2 is now prohibited by the build sandbox
-prs: 10501
-credits: ma27
-category: Fixes
----
-
-The build sandbox blocks any attempt to create setuid/setgid binaries, but didn't check
-for the use of the `fchmodat2` syscall which was introduced in Linux 6.6 and is used by
-glibc >=2.39. This is fixed now.

doc/manual/rl-next/fix-nested-follows.md

@@ -1,24 +0,0 @@
----
-synopsis: Fix nested flake input `follows`
-prs: 6621
-cls: 994
-credits: [Kha, ma27]
-category: Fixes
-significance: significant
----
-
-Previously nested-input overrides were ignored; that is, the following did not
-override anything, in spite of the `nix3-flake` manual documenting it working:
-
-```
-{
-  inputs = {
-    foo.url = "github:bar/foo";
-    foo.inputs.bar.inputs.nixpkgs = "nixpkgs";
-  };
-}
-```
-
-This is useful to avoid the 1000 instances of nixpkgs problem without having
-each flake in the dependency tree to expose all of its transitive dependencies
-for modification.

doc/manual/rl-next/fix-silent-unknown-options.md

@@ -1,30 +0,0 @@
----
-synopsis: Warn on unknown settings anywhere in the command line
-prs: 10701
-credits: [cole-h]
-category: Improvements
----
-
-All `nix` commands will now properly warn when an unknown option is specified anywhere in the command line.
-
-Before:
-
-```console
-$ nix-instantiate --option foobar baz --expr '{}'
-warning: unknown setting 'foobar'
-$ nix-instantiate '{}' --option foobar baz --expr
-$ nix eval --expr '{}' --option foobar baz
-{ }
-```
-
-After:
-
-```console
-$ nix-instantiate --option foobar baz --expr '{}'
-warning: unknown setting 'foobar'
-$ nix-instantiate '{}' --option foobar baz --expr
-warning: unknown setting 'foobar'
-$ nix eval --expr '{}' --option foobar baz
-warning: unknown setting 'foobar'
-{ }
-```

doc/manual/rl-next/forbid-nested-debuggers.md

@@ -1,34 +0,0 @@
----
-synopsis: Nested debuggers are no longer supported
-prs: 9920
-credits: 9999years
-category: Improvements
----
-
-Previously, evaluating an expression that throws an error in the debugger would
-enter a second, nested debugger:
-
-```
-nix-repl> builtins.throw "what"
-error: what
-
-
-Starting REPL to allow you to inspect the current state of the evaluator.
-
-Welcome to Nix 2.18.1. Type :? for help.
-
-nix-repl>
-```
-
-Now, it just prints the error message like `nix repl`:
-
-```
-nix-repl> builtins.throw "what"
-error:
-       … while calling the 'throw' builtin
-         at «string»:1:1:
-            1| builtins.throw "what"
-             | ^
-
-       error: what
-```

doc/manual/rl-next/formal-order.md

@@ -1,9 +0,0 @@
----
-synopsis: consistent order of lambda formals in printed expressions
-prs: 9874
-credits: horrors
-category: Fixes
----
-
-Always print lambda formals in lexicographic order rather than the internal, creation-time based symbol order.
-This makes printed formals independent of the context they appear in.

doc/manual/rl-next/gc-roots-darwin.md

@@ -1,8 +0,0 @@
----
-synopsis: Find GC roots using libproc on Darwin
-cls: 723
-credits: artemist
-category: Improvements
----
-
-Previously, the garbage collector found runtime roots on Darwin by shelling out to `lsof -n -w -F n` then parsing the result. The version of `lsof` packaged in Nixpkgs is very slow on Darwin, so Lix now uses `libproc` directly to speed up GC root discovery, in some tests taking 250ms now instead of 40s.

doc/manual/rl-next/inherit-error-positions.md

@@ -1,8 +0,0 @@
----
-synopsis: fix duplicate attribute error positions for `inherit`
-prs: 9874
-credits: horrors
-category: Fixes
----
-
-When an inherit caused a duplicate attribute error, the position of the error was not reported correctly, placing the error with the inherit itself or at the start of the bindings block instead of the offending attribute name.

doc/manual/rl-next/inherit-from-by-need.md

@@ -1,9 +0,0 @@
----
-synopsis: "`inherit (x) ...` evaluates `x` only once"
-prs: 9847
-category: Fixes
-credits: horrors
----
-
-`inherit (x) a b ...` now evaluates the expression `x` only once for all inherited attributes rather than once for each inherited attribute.
-This does not usually have a measurable impact, but side-effects (such as `builtins.trace`) would be duplicated and expensive expressions (such as derivations) could cause a measurable slowdown.

doc/manual/rl-next/leading-period.md

@@ -1,12 +0,0 @@
----
-synopsis: Store paths are allowed to start with `.`
-issues: 912
-prs: [9867, 9091, 9095, 9120, 9121, 9122, 9130, 9219, 9224]
-credits: [roberth, horrors]
-category: Fixes
----
-
-Leading periods were allowed by accident in Nix 2.4. The Nix team has considered this to be a bug, but this behavior has since been relied on by users, leading to unnecessary difficulties.
-From now on, leading periods are officially, definitively supported. The names `.` and `..` are disallowed, as well as those starting with `.-` or `..-`.
-
-Nix versions that denied leading periods are documented [in the issue](https://github.com/NixOS/nix/issues/912#issuecomment-1919583286).

doc/manual/rl-next/linux-sandbox-consistency.md

@@ -1,12 +0,0 @@
----
-synopsis: Enforce syscall filtering and no-new-privileges on Linux
-cls: 1063
-category: Breaking Changes
-credits: alois31
----
-
-In order to improve consistency of the build environment, system call filtering and no-new-privileges are now unconditionally enabled on Linux.
-The `filter-syscalls` and `allow-new-privileges` options which could be used to disable these features under some circumstances have been removed.
-
-In order to support building on architectures without libseccomp support, the option to disable syscall filtering at build time remains.
-However, other uses of this option are heavily discouraged, since it would reduce the security of the sandbox substantially.

doc/manual/rl-next/macos-stack-size.md

@@ -1,9 +0,0 @@
----
-synopsis: Increase default stack size on macOS
-prs: 9860
-credits: 9999years
-category: Improvements
----
-
-Increase the default stack size on macOS to the same value as on Linux, subject to system restrictions to maximum stack size.
-This should reduce the number of stack overflow crashes on macOS when evaluating Nix code with deep call stacks.

doc/manual/rl-next/meson.md

@@ -1,15 +0,0 @@
----
-synopsis: Lix is built with meson
-# and many more
-cls: [580, 627, 628, 707, 711, 712, 719]
-credits: [Qyriad, horrors, jade, 9999years, winter]
-category: Packaging
----
-
-Lix is built exclusively with the meson build system thanks to a huge team-wide
-effort, and the legacy `make`/`autoconf` based build system has been removed
-altogether. This improves maintainability of Lix, enables things like saving
-20% of compile times with precompiled headers, and generally makes the build
-less able to produce obscure incremental compilation bugs.
-
-Non-Nix-based downstream packaging needs rewriting accordingly.

doc/manual/rl-next/more-logs.md

@@ -1,9 +0,0 @@
----
-synopsis: Show more log context for failed builds
-prs: 9670
-credits: DavHau
-category: Improvements
----
-
-Show 25 lines of log tail instead of 10 for failed builds.
-This increases the chances of having useful information in the shown logs.

doc/manual/rl-next/new-assertions.md

@@ -1,26 +0,0 @@
----
-synopsis: Lix turns more internal bugs into crashes
-cls: [797, 626]
-credits: jade
-category: Packaging
-significance: significant
----
-
-Lix now enables build options such as trapping on signed overflow and enabling
-libstdc++ assertions by default. These may find new bugs in Lix, which will
-present themselves as Lix processes aborting, potentially without an error
-message.
-
-If Lix processes abort on your machine, this is a bug. Please file a bug,
-ideally with the core dump (or information from it).
-
-On Linux, run `coredumpctl list`, find the crashed process's PID at
-the bottom of the list, then run `coredumpctl info THE-PID`. You can then paste
-the output into a bug report.
-
-On macOS, open the Console app from Applications/Utilities, select Crash
-Reports, select the crash report in question. Right click on it, select Open In
-Finder, then include that file in your bug report. [See the Apple
-documentation][apple-crashreport] for more details.
-
-[apple-crashreport]: https://developer.apple.com/documentation/xcode/acquiring-crash-reports-and-diagnostic-logs#Locate-crash-reports-and-memory-logs-on-the-device

doc/manual/rl-next/nix-config-show.md

@@ -1,12 +0,0 @@
----
-synopsis: rename 'nix show-config' to 'nix config show'
-issues: 7672
-prs: 9477
-cls: 993
-credits: [thufschmitt, ma27]
-category: Improvements
----
-
-`nix show-config` was renamed to `nix config show` to be more consistent with the rest of the command-line interface.
-
-Running `nix show-config` will now print a deprecation warning saying to use `nix config show` instead.

doc/manual/rl-next/nix-env-json-drv-path.md

@@ -1,8 +0,0 @@
----
-synopsis: Fix `nix-env --query --drv-path --json`
-prs: 9257
-credits: [Artturin, horrors]
-category: Fixes
----
-
-Fixed a bug where `nix-env --query` ignored `--drv-path` when `--json` was set.

doc/manual/rl-next/nix-eval-derivations.md

@@ -1,9 +0,0 @@
----
-synopsis: Print derivation paths in `nix eval`
-cls: 446
-credits: 9999years
-category: Improvements
----
-
-`nix eval` previously printed derivations as attribute sets, so commands that print derivations (e.g. `nix eval nixpkgs#bash`) would infinitely loop and segfault.
-It now prints the `.drv` path the derivation generates instead.

doc/manual/rl-next/nix-flake-check-logs-actions.md

@@ -1,37 +0,0 @@
----
-synopsis: "`nix flake check` logs the checks"
-issues: 8882
-prs: 8893
-cls: [259, 260, 261, 262]
-credits: [9999years, raito, horrors]
-category: Improvements
-significance: significant
----
-
-`nix flake check` now logs the checks it runs and the derivations it evaluates:
-
-```
-$ nix flake check -v
-evaluating flake...
-checking flake output 'checks'...
-checking derivation 'checks.aarch64-darwin.ghciwatch-tests'...
-derivation evaluated to /nix/store/nh7dlvsrhds4cxl91mvgj4h5cbq6skmq-ghciwatch-test-0.3.0.drv
-checking derivation 'checks.aarch64-darwin.ghciwatch-clippy'...
-derivation evaluated to /nix/store/9cb5a6wmp6kf6hidqw9wphidvb8bshym-ghciwatch-clippy-0.3.0.drv
-checking derivation 'checks.aarch64-darwin.ghciwatch-doc'...
-derivation evaluated to /nix/store/8brdd3jbawfszpbs7vdpsrhy80as1il8-ghciwatch-doc-0.3.0.drv
-checking derivation 'checks.aarch64-darwin.ghciwatch-fmt'...
-derivation evaluated to /nix/store/wjhs0l1njl5pyji53xlmfjrlya0wmz8p-ghciwatch-fmt-0.3.0.drv
-checking derivation 'checks.aarch64-darwin.ghciwatch-audit'...
-derivation evaluated to /nix/store/z0mps8dyj2ds7c0fn0819y5h5611033z-ghciwatch-audit-0.3.0.drv
-checking flake output 'packages'...
-checking derivation 'packages.aarch64-darwin.default'...
-derivation evaluated to /nix/store/41abbdyglw5x9vcsvd89xan3ydjf8d7r-ghciwatch-0.3.0.drv
-checking flake output 'apps'...
-checking flake output 'devShells'...
-checking derivation 'devShells.aarch64-darwin.default'...
-derivation evaluated to /nix/store/bc935gz7dylzmcpdb5cczr8gngv8pmdb-nix-shell.drv
-running 5 flake checks...
-warning: The check omitted these incompatible systems: aarch64-linux, x86_64-darwin, x86_64-linux
-Use '--all-systems' to check all.
-```

doc/manual/rl-next/nix-flake-update-ux.md

@@ -1,19 +0,0 @@
----
-synopsis: "Overhaul `nix flake update` and `nix flake lock` UX"
-prs: 8817
-credits: [iFreilicht, Lunaphied, thufschmitt]
-category: Breaking Changes
----
-
-The interface for creating and updating lock files has been overhauled:
-
-- [`nix flake lock`](@docroot@/command-ref/new-cli/nix3-flake-lock.md) only creates lock files and adds missing inputs now.
-It will *never* update existing inputs.
-
-- [`nix flake update`](@docroot@/command-ref/new-cli/nix3-flake-update.md) does the same, but *will* update inputs.
-- Passing no arguments will update all inputs of the current flake, just like it already did.
-- Passing input names as arguments will ensure only those are updated. This replaces the functionality of `nix flake lock --update-input`
-- To operate on a flake outside the current directory, you must now pass `--flake path/to/flake`.
-
-- The flake-specific flags `--recreate-lock-file` and `--update-input` have been removed from all commands operating on installables.
-They are superceded by `nix flake update`.

doc/manual/rl-next/nix-profile-names.md

@@ -1,11 +0,0 @@
----
-synopsis: "`nix profile` now allows referring to elements by human-readable name, and no longer accepts indices"
-prs: 8678
-cls: [978, 980]
-category: Breaking Changes
-credits: [iFreilicht, Qyriad, edolstra]
----
-
-[`nix profile`](@docroot@/command-ref/new-cli/nix3-profile.md) now uses names to refer to installed packages when running [`list`](@docroot@/command-ref/new-cli/nix3-profile-list.md), [`remove`](@docroot@/command-ref/new-cli/nix3-profile-remove.md) or [`upgrade`](@docroot@/command-ref/new-cli/nix3-profile-upgrade.md) as opposed to indices. Indices have been removed. Profile element names are generated when a package is installed and remain the same until the package is removed.
-
-**Warning**: The `manifest.nix` file used to record the contents of profiles has changed. Lix will automatically upgrade profiles to the new version when you modify the profile. After that, the profile can no longer be used by older versions of Lix.

doc/manual/rl-next/nix-store-prefetch-unpack.md

@@ -1,18 +0,0 @@
----
-synopsis: "Add an option `--unpack` to unpack archives in `nix store prefetch-file`"
-prs: 9805
-cls: 224
-credits: [yshui, horrors]
-category: Improvements
----
-
-It is now possible to fetch an archive then NAR-hash it (as in, hash it in the
-same manner as `builtins.fetchTarball` or fixed-output derivations with
-recursive hash type) in one command.
-
-Example:
-
-```
-~ » nix store prefetch-file --name source --unpack https://git.lix.systems/lix-project/lix/archive/2.90-beta.1.tar.gz
-Downloaded 'https://git.lix.systems/lix-project/lix/archive/2.90-beta.1.tar.gz' to '/nix/store/yvfqnq52ryjc3janw02ziv7kr6gd0cs1-source' (hash 'sha256-REWlo2RYHfJkxnmZTEJu3Cd/2VM+wjjpPy7Xi4BdDTQ=').
-```

doc/manual/rl-next/nixversion-fake.md

@@ -1,17 +0,0 @@
----
-synopsis: "`builtins.nixVersion` and `builtins.langVersion` return fixed values"
-cls: [558, 1144]
-credits: jade
-category: Breaking Changes
----
-
-`builtins.nixVersion` now returns a fixed value `"2.18.3-lix"`.
-
-`builtins.langVersion` returns a fixed value `6`, matching CppNix 2.18.
-
-This prevents feature detection assuming that features that exist in Nix
-post-Lix-branch-off might exist, even though the Lix version is greater than
-the Nix version.
-
-In the future, check for builtins for feature detection. If a feature cannot be
-detected by *those* means, please file a Lix bug.

doc/manual/rl-next/no-cache-eval-errors.md

@@ -1,10 +0,0 @@
----
-synopsis: re-evaluate cached evaluation errors
-cls: 771
-credits: Qyriad
-category: Fixes
----
-
-"cached failure of [expr]" errors have been removed: expressions already in the
-eval cache as a failure will now simply be re-evaluated, removing the need to
-set `--no-eval-cache` or similar to see the error.

doc/manual/rl-next/print-in-repl.md

@@ -1,55 +0,0 @@
----
-synopsis: "REPL printing improvements"
-prs: [9931, 10208]
-cls: [375, 492]
-credits: [9999years, horrors]
-category: Improvements
----
-
-The REPL printer has been improved to do the following:
-- If a string is passed to `:print`, it is printed literally to the screen
-- Structures will be printed as multiple lines when necessary
-
-Before:
-
-```
-nix-repl> { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
-{ attrs = { ... }; list = [ ... ]; list' = [ ... ]; }
-
-nix-repl> :p { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
-{ attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
-
-nix-repl> :p "meow"
-"meow"
-```
-
-After:
-
-```
-nix-repl> { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
-{
-  attrs = { ... };
-  list = [ ... ];
-  list' = [ ... ];
-}
-
-nix-repl> :p { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
-{
-  attrs = {
-    a = {
-      b = {
-        c = { };
-      };
-    };
-  };
-  list = [ 1 ];
-  list' = [
-    1
-    2
-    3
-  ];
-}
-
-nix-repl> :p "meow"
-meow
-```

doc/manual/rl-next/print-value-in-coercion-error.md

@@ -1,26 +0,0 @@
----
-synopsis: Coercion errors include the failing value
-issues: 561
-prs: 9754
-credits: [9999years, horrors]
-category: Improvements
----
-
-The `error: cannot coerce a <TYPE> to a string` message now includes the value
-which caused the error.
-
-Before:
-
-```
-       error: cannot coerce a set to a string
-```
-
-After:
-
-```
-       error: cannot coerce a set to a string: { aesSupport = «thunk»;
-       avx2Support = «thunk»; avx512Support = «thunk»; avxSupport = «thunk»;
-       canExecute = «thunk»; config = «thunk»; darwinArch = «thunk»; darwinMinVersion
-       = «thunk»; darwinMinVersionVariable = «thunk»; darwinPlatform = «thunk»; «84
-       attributes elided»}
-```

doc/manual/rl-next/print-value-in-installable-flake-error.md

@@ -1,20 +0,0 @@
----
-synopsis: New-cli flake commands that expect derivations now print the failing value and its type
-credits: Qyriad
-category: Improvements
-cls: 1177
----
-
-In errors like `flake output attribute 'legacyPackages.x86_64-linux.lib' is not a derivation or path`, the message now includes the failing value and type.
-
-Before:
-
-```
-    error: flake output attribute 'nixosConfigurations.yuki.config' is not a derivation or path
-````
-
-After:
-
-```
-    error: expected flake output attribute 'nixosConfigurations.yuki.config' to be a derivation or path but found a set: { appstream = «thunk»; assertions = «thunk»; boot = { bcache = «thunk»; binfmt = «thunk»; binfmtMiscRegistrations = «thunk»; blacklistedKernelModules = «thunk»; bootMount = «thunk»; bootspec = «thunk»; cleanTmpDir = «thunk»; consoleLogLevel = «thunk»; «43 attributes elided» }; «48 attributes elided» }
-```

doc/manual/rl-next/print-value-in-type-error.md

@@ -1,25 +0,0 @@
----
-synopsis: Type errors include the failing value
-issues: 561
-prs: 9753
-credits: [9999years, horrors]
-category: Improvements
----
-
-In errors like `value is an integer while a list was expected`, the message now
-includes the failing value.
-
-Before:
-
-```
-       error: value is a set while a string was expected
-```
-
-After:
-
-```
-       error: expected a string but found a set: { ghc810 = «thunk»;
-       ghc8102Binary = «thunk»; ghc8107 = «thunk»; ghc8107Binary = «thunk»;
-       ghc865Binary = «thunk»; ghc90 = «thunk»; ghc902 = «thunk»; ghc92 = «thunk»;
-       ghc924Binary = «thunk»; ghc925 = «thunk»;  «17 attributes elided»}
-```

doc/manual/rl-next/reduce-debugger-clutter.md

@@ -1,39 +0,0 @@
----
-synopsis: "Visual clutter in `--debugger` is reduced"
-prs: 9919
-category: Improvements
-credits: [9999years, horrors]
----
-
-Before:
-```
-info: breakpoint reached
-
-
-Starting REPL to allow you to inspect the current state of the evaluator.
-
-Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
-
-nix-repl> :continue
-error: uh oh
-
-
-Starting REPL to allow you to inspect the current state of the evaluator.
-
-Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
-
-nix-repl>
-```
-
-After:
-
-```
-info: breakpoint reached
-
-Nix 2.20.0pre20231222_dirty debugger
-Type :? for help.
-nix-repl> :continue
-error: uh oh
-
-nix-repl>
-```

doc/manual/rl-next/rename-lixexpr.md

@@ -1,31 +0,0 @@
----
-synopsis: Rename all the libraries nixexpr, nixstore, etc to lixexpr, lixstore, etc
-credits: jade
-category: Breaking Changes
----
-
-The Lix C++ API libraries have had the following changes:
-- Includes moved from `include/nix/` to `include/lix/`
-- `pkg-config` files renamed from `nix-expr` to `lix-expr` and so on.
-- Libraries renamed from `libnixexpr.so` to `liblixexpr.so` and so on.
-
-There are other changes between Nix 2.18 and Lix, since these APIs are not
-stable. However, this change in particular is a deliberate compatibility break
-to force downstreams linking to Lix to specifically handle Lix and avoid Lix
-accidentally getting ensnared in compatibility code for newer CppNix.
-
-Migration path:
-
-- expr.hh      -> lix/libexpr/expr.hh
-- nix/config.h -> lix/config.h
-
-To apply this migration automatically, remove all `<nix/>` from includes, so `#include <nix/expr.hh>` -> `#include <expr.hh>`.
-Then, the correct paths will be resolved from the tangled mess, and the clang-tidy automated fix will work.
-
-Then run the following for out of tree projects (header filter is set to only fix instances in headers in `../src` relative to the compiler's working directory, as would be the case in nix-eval-jobs or other things built with meson, e.g.):
-
-```console
-lix_root=$HOME/lix
-(cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build')
-run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -header-filter '\.\./src/.*\.h' -fix src
-```

doc/manual/rl-next/repl-complete-colon.md

@@ -0,0 +1,8 @@
+---
+synopsis: "`nix repl` now allows tab-completing the special repl :colon commands"
+cls: 1367
+credits: Qyriad
+category: Improvements
+---
+
+The REPL (`nix repl`) supports pressing `<TAB>` to complete a partial expression, but now also supports completing the special :colon commands as well (`:b`, `:edit`, `:doc`, etc), if the line starts with a colon.

doc/manual/rl-next/repl-doc-command.md

@@ -1,16 +0,0 @@
----
-synopsis: Experimental REPL support for documentation comments using `:doc`
-cls: 564
-category: Features
-credits: [Lunaphied, jade]
-significance: significant
----
-
-Using `:doc` in the REPL now supports showing documentation comments when defined on a function.
-
-Previously this was only able to document builtins, however it now will show comments defined on a lambda as well.
-
-This support is experimental and relies on an embedded version of [nix-doc](https://github.com/lf-/nix-doc).
-
-The logic also supports limited Markdown formatting of doccomments and should easily support any [RFC 145](https://github.com/NixOS/rfcs/blob/master/rfcs/0145-doc-strings.md)
-compatible documentation comments in addition to simple commented documentation.

doc/manual/rl-next/repl-fix-history.md

@@ -1,9 +0,0 @@
----
-synopsis: "`nix repl` history is saved more reliably"
-cls: 1164
-credits: puck
----
-
-`nix repl` now saves its history file after each line, rather than at the end
-of the session; ensuring that it will remember what you typed even after it
-crashes.

doc/manual/rl-next/repl-interrupt.md

@@ -1,10 +0,0 @@
----
-synopsis: Interrupting builds in the REPL works more than once
-cls: 1097
-category: Fixes
-credits: alois31
----
-
-Builds in the REPL can be interrupted by pressing Ctrl+C.
-Previously, this only worked once per REPL session; further attempts would be ignored.
-This issue is now fixed, so that builds can be canceled consistently.

doc/manual/rl-next/repl-overlays.md

@@ -1,39 +0,0 @@
----
-synopsis: Add `repl-overlays` option
-prs: 10203
-cls: 504
-credits: 9999years
-significance: significant
-category: Features
----
-
-A `repl-overlays` option has been added, which specifies files that can overlay
-and modify the top-level bindings in `nix repl`. For example, with the
-following contents in `~/.config/nix/repl.nix`:
-
-```nix
-info: final: prev: let
-  optionalAttrs = predicate: attrs:
-    if predicate
-    then attrs
-    else {};
-in
-  optionalAttrs (prev ? legacyPackages && prev.legacyPackages ? ${info.currentSystem})
-  {
-    pkgs = prev.legacyPackages.${info.currentSystem};
-  }
-```
-
-We can run `nix repl` and use `pkgs` to refer to `legacyPackages.${currentSystem}`:
-
-```ShellSession
-$ nix repl --repl-overlays ~/.config/nix/repl.nix nixpkgs
-Lix 2.90.0
-Type :? for help.
-Loading installable 'flake:nixpkgs#'...
-Added 5 variables.
-Loading 'repl-overlays'...
-Added 6 variables.
-nix-repl> pkgs.bash
-«derivation /nix/store/g08b5vkwwh0j8ic9rkmd8mpj878rk62z-bash-5.2p26.drv»
-```

doc/manual/rl-next/repl-tstp.md

@@ -1,7 +0,0 @@
----
-synopsis: "REPL now supports CTRL+Z to suspend"
-credits: [Qyriad]
-category: Improvements
----
-
-Editline is now built with SIGTSTP support, so now typing CTRL+Z in the REPL will suspend the REPL and allow it to be resumed later or backgrounded.

doc/manual/rl-next/shebang-single-quotes.md

@@ -1,13 +0,0 @@
----
-synopsis: Allow single quotes in nix-shell shebangs
-prs: 8470
-credits: [ncfavier, horrors]
-category: Improvements
----
-
-Example:
-
-```bash
-#! /usr/bin/env nix-shell
-#! nix-shell -i bash --packages 'terraform.withPlugins (plugins: [ plugins.openstack ])'
-```

doc/manual/rl-next/short-expr-flag.md

@@ -1,16 +0,0 @@
----
-synopsis: reintroduce shortened `-E` form for `--expr` to new CLI
-cls: 605
-credits: Lunaphied
-category: Improvements
----
-
-In the old CLI, it was possible to supply a shorter `-E` flag instead of fully
-specifying `--expr` every time you wished to provide an expression that would
-be evaluated to produce the given command's input. This was retained for the
-`--file` flag when the new CLI utilities were written with `-f`, but `-E` was
-dropped.
-
-We now restore the `-E` short form for better UX. This is most useful for
-`nix eval` but most any command that takes an Installable argument should benefit
-from it as well.

doc/manual/rl-next/source-location-in-while-evaluating-attribute.md

@@ -1,25 +0,0 @@
----
-synopsis: "In the debugger, `while evaluating the attribute` errors now include position information"
-prs: 9915
-credits: 9999years
-category: Fixes
----
-
-Before:
-
-```
-0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
-0x600001522598
-```
-
-After:
-
-```
-0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
-/nix/store/hg65h51xnp74ikahns9hyf3py5mlbbqq-source/overrides/default.nix:132:27
-
-   131|
-   132|       bootstrappingBase = pkgs.${self.python.pythonAttr}.pythonForBuild.pkgs;
-      |                           ^
-   133|     in
-```

doc/manual/rl-next/source-positions-in-errors.md

@@ -1,44 +0,0 @@
----
-synopsis: Source locations are printed more consistently in errors
-issues: 561
-prs: 9555
-credits: [9999years, horrors]
-category: Improvements
----
-
-Source location information is now included in error messages more
-consistently. Given this code:
-
-```nix
-let
-  attr = {foo = "bar";};
-  key = {};
-in
-  attr.${key}
-```
-
-Previously, Nix would show this unhelpful message when attempting to evaluate
-it:
-
-```
-error:
-       … while evaluating an attribute name
-
-       error: value is a set while a string was expected
-```
-
-Now, the error message displays where the problematic value was found:
-
-```
-error:
-       … while evaluating an attribute name
-
-         at bad.nix:4:11:
-
-            3|   key = {};
-            4| in attr.${key}
-             |           ^
-            5|
-
-       error: expected a string but found a set: { }
-```

doc/manual/rl-next/ssh-ng-phase-reporting.md

@@ -1,8 +0,0 @@
----
-synopsis: Include phase reporting in log file for ssh-ng builds
-prs: 9280
-credits: r-vdp
-category: Fixes
----
-
-Store phase information of remote builds run via `ssh-ng` remotes in the local log file, matching logging behavior of local builds.

doc/manual/rl-next/ssh-ng-substitute.md

@@ -1,9 +0,0 @@
----
-synopsis: Fix `ssh-ng://` remotes not respecting `--substitute-on-destination`
-prs: 9600
-credits: SharzyL
-category: Fixes
----
-
-`nix copy ssh-ng://` now respects `--substitute-on-destination`, as does `nix-copy-closure` and other commands that operate on remote `ssh-ng` stores.
-Previously this was always set by `builders-use-substitutes` setting.

doc/manual/rl-next/stack-overflow-segfaults.md

@@ -1,35 +0,0 @@
----
-synopsis: Some stack overflow segfaults are fixed
-issues: 9616
-prs: 9617
-cls: 205
-category: Improvements
-credits: [9999years, horrors]
----
-
-The number of nested function calls has been restricted, to detect and report
-infinite function call recursions. The default maximum call depth is 10,000 and
-can be set with [the `max-call-depth`
-option](@docroot@/command-ref/conf-file.md#conf-max-call-depth).
-
-This fixes segfaults or the following unhelpful error message in many cases:
-
-    error: stack overflow (possible infinite recursion)
-
-Before:
-
-```
-$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
-Segmentation fault: 11
-```
-
-After:
-
-```
-$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
-error: stack overflow
-
-       at «string»:1:14:
-            1| (x: x x) (x: x x)
-             |              ^
-```

doc/manual/rl-next/upgrade-nix-override.md

@@ -1,8 +0,0 @@
----
-synopsis: add `--store-path` argument to `nix upgrade-nix`, to manually specify the Nix to upgrade to
-cls: 953
-credits: Qyriad
-category: Features
----
-
-`nix upgrade-nix` by default downloads a manifest to find the new Nix version to upgrade to, but now you can specify `--store-path` to upgrade Nix to an arbitrary version from the Nix store.

doc/manual/rl-next/upgrade-nix-profile-compat.md

@@ -1,10 +0,0 @@
----
-synopsis: using `nix profile` on `/nix/var/nix/profiles/default` no longer breaks `nix upgrade-nix`
-cls: 952
-credits: Qyriad
-category: Fixes
----
-
-On non-NixOS, Nix is conventionally installed into a `nix-env` style profile at /nix/var/nix/profiles/default.
-Like any `nix-env` profile, using `nix profile` on it automatically migrates it to a `nix profile` style profile, which is incompatible with `nix-env`.
-`nix upgrade-nix` previously relied solely on `nix-env` to do the upgrade, but now will work fine with either kind of profile.

doc/manual/rl-next/upstart-removal.md

@@ -1,10 +0,0 @@
----
-synopsis: Upstart scripts removed
-cls: 574
-category: Packaging
-credits: jade
----
-
-Upstart scripts have been removed from Lix, since Upstart is obsolete and has
-not been shipped by any major distributions for many years. If these are
-necessary to your use case, please back port them to your packaging.

doc/manual/rl-next/warn-ignored-client-settings.md

@@ -1,9 +0,0 @@
----
-synopsis: Warn about ignored client settings
-cls: 1026
-credits: jade
-category: Improvements
----
-
-Emit a warning for every client-provided setting the daemon ignores because the requesting client is not run by a trusted user.
-Previously this was only a debug message.

doc/manual/rl-next/with-error-reporting.md

@@ -1,34 +0,0 @@
----
-synopsis: Better error reporting for `with` expressions
-prs: 9658
-cls: 207
-credits: horrors
-category: Improvements
----
-
-`with` expressions using non-attrset values to resolve variables are now reported with proper positions.
-
-Previously an incorrect `with` expression would report no position at all, making it hard to determine where the error originated:
-
-```
-nix-repl> with 1; a
-error:
-       … <borked>
-
-         at «none»:0: (source not available)
-
-       error: value is an integer while a set was expected
-```
-
-Now position information is preserved and reported as with most other errors:
-
-```
-nix-repl> with 1; a
-error:
-       … while evaluating the first subexpression of a with expression
-         at «string»:1:1:
-            1| with 1; a
-             | ^
-
-       error: expected a set but found an integer: 1
-```

doc/manual/src/SUMMARY.md

@@ -196,53 +196,55 @@
   - [C++ style guide](contributing/cxx.md)
 - [Release Notes](release-notes/release-notes.md)
   - [Upcoming release](release-notes/rl-next.md)
-  - [Release 2.18 (2023-09-20)](release-notes/rl-2.18.md)
+  <!-- RELENG-AUTO-INSERTION-MARKER (see releng/release_notes.py) -->
-  - [Release 2.17 (2023-07-24)](release-notes/rl-2.17.md)
+  - [Lix 2.90 (FIXME date)](release-notes/rl-2.90.md)
-  - [Release 2.16 (2023-05-31)](release-notes/rl-2.16.md)
+  - [Nix 2.18 (2023-09-20)](release-notes/rl-2.18.md)
-  - [Release 2.15 (2023-04-11)](release-notes/rl-2.15.md)
+  - [Nix 2.17 (2023-07-24)](release-notes/rl-2.17.md)
-  - [Release 2.14 (2023-02-28)](release-notes/rl-2.14.md)
+  - [Nix 2.16 (2023-05-31)](release-notes/rl-2.16.md)
-  - [Release 2.13 (2023-01-17)](release-notes/rl-2.13.md)
+  - [Nix 2.15 (2023-04-11)](release-notes/rl-2.15.md)
-  - [Release 2.12 (2022-12-06)](release-notes/rl-2.12.md)
+  - [Nix 2.14 (2023-02-28)](release-notes/rl-2.14.md)
-  - [Release 2.11 (2022-08-25)](release-notes/rl-2.11.md)
+  - [Nix 2.13 (2023-01-17)](release-notes/rl-2.13.md)
-  - [Release 2.10 (2022-07-11)](release-notes/rl-2.10.md)
+  - [Nix 2.12 (2022-12-06)](release-notes/rl-2.12.md)
-  - [Release 2.9 (2022-05-30)](release-notes/rl-2.9.md)
+  - [Nix 2.11 (2022-08-25)](release-notes/rl-2.11.md)
-  - [Release 2.8 (2022-04-19)](release-notes/rl-2.8.md)
+  - [Nix 2.10 (2022-07-11)](release-notes/rl-2.10.md)
-  - [Release 2.7 (2022-03-07)](release-notes/rl-2.7.md)
+  - [Nix 2.9 (2022-05-30)](release-notes/rl-2.9.md)
-  - [Release 2.6 (2022-01-24)](release-notes/rl-2.6.md)
+  - [Nix 2.8 (2022-04-19)](release-notes/rl-2.8.md)
-  - [Release 2.5 (2021-12-13)](release-notes/rl-2.5.md)
+  - [Nix 2.7 (2022-03-07)](release-notes/rl-2.7.md)
-  - [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md)
+  - [Nix 2.6 (2022-01-24)](release-notes/rl-2.6.md)
-  - [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md)
+  - [Nix 2.5 (2021-12-13)](release-notes/rl-2.5.md)
-  - [Release 2.2 (2019-01-11)](release-notes/rl-2.2.md)
+  - [Nix 2.4 (2021-11-01)](release-notes/rl-2.4.md)
-  - [Release 2.1 (2018-09-02)](release-notes/rl-2.1.md)
+  - [Nix 2.3 (2019-09-04)](release-notes/rl-2.3.md)
-  - [Release 2.0 (2018-02-22)](release-notes/rl-2.0.md)
+  - [Nix 2.2 (2019-01-11)](release-notes/rl-2.2.md)
-  - [Release 1.11.10 (2017-06-12)](release-notes/rl-1.11.10.md)
+  - [Nix 2.1 (2018-09-02)](release-notes/rl-2.1.md)
-  - [Release 1.11 (2016-01-19)](release-notes/rl-1.11.md)
+  - [Nix 2.0 (2018-02-22)](release-notes/rl-2.0.md)
-  - [Release 1.10 (2015-09-03)](release-notes/rl-1.10.md)
+  - [Nix 1.11.10 (2017-06-12)](release-notes/rl-1.11.10.md)
-  - [Release 1.9 (2015-06-12)](release-notes/rl-1.9.md)
+  - [Nix 1.11 (2016-01-19)](release-notes/rl-1.11.md)
-  - [Release 1.8 (2014-12-14)](release-notes/rl-1.8.md)
+  - [Nix 1.10 (2015-09-03)](release-notes/rl-1.10.md)
-  - [Release 1.7 (2014-04-11)](release-notes/rl-1.7.md)
+  - [Nix 1.9 (2015-06-12)](release-notes/rl-1.9.md)
-  - [Release 1.6.1 (2013-10-28)](release-notes/rl-1.6.1.md)
+  - [Nix 1.8 (2014-12-14)](release-notes/rl-1.8.md)
-  - [Release 1.6 (2013-09-10)](release-notes/rl-1.6.md)
+  - [Nix 1.7 (2014-04-11)](release-notes/rl-1.7.md)
-  - [Release 1.5.2 (2013-05-13)](release-notes/rl-1.5.2.md)
+  - [Nix 1.6.1 (2013-10-28)](release-notes/rl-1.6.1.md)
-  - [Release 1.5 (2013-02-27)](release-notes/rl-1.5.md)
+  - [Nix 1.6 (2013-09-10)](release-notes/rl-1.6.md)
-  - [Release 1.4 (2013-02-26)](release-notes/rl-1.4.md)
+  - [Nix 1.5.2 (2013-05-13)](release-notes/rl-1.5.2.md)
-  - [Release 1.3 (2013-01-04)](release-notes/rl-1.3.md)
+  - [Nix 1.5 (2013-02-27)](release-notes/rl-1.5.md)
-  - [Release 1.2 (2012-12-06)](release-notes/rl-1.2.md)
+  - [Nix 1.4 (2013-02-26)](release-notes/rl-1.4.md)
-  - [Release 1.1 (2012-07-18)](release-notes/rl-1.1.md)
+  - [Nix 1.3 (2013-01-04)](release-notes/rl-1.3.md)
-  - [Release 1.0 (2012-05-11)](release-notes/rl-1.0.md)
+  - [Nix 1.2 (2012-12-06)](release-notes/rl-1.2.md)
-  - [Release 0.16 (2010-08-17)](release-notes/rl-0.16.md)
+  - [Nix 1.1 (2012-07-18)](release-notes/rl-1.1.md)
-  - [Release 0.15 (2010-03-17)](release-notes/rl-0.15.md)
+  - [Nix 1.0 (2012-05-11)](release-notes/rl-1.0.md)
-  - [Release 0.14 (2010-02-04)](release-notes/rl-0.14.md)
+  - [Nix 0.16 (2010-08-17)](release-notes/rl-0.16.md)
-  - [Release 0.13 (2009-11-05)](release-notes/rl-0.13.md)
+  - [Nix 0.15 (2010-03-17)](release-notes/rl-0.15.md)
-  - [Release 0.12 (2008-11-20)](release-notes/rl-0.12.md)
+  - [Nix 0.14 (2010-02-04)](release-notes/rl-0.14.md)
-  - [Release 0.11 (2007-12-31)](release-notes/rl-0.11.md)
+  - [Nix 0.13 (2009-11-05)](release-notes/rl-0.13.md)
-  - [Release 0.10.1 (2006-10-11)](release-notes/rl-0.10.1.md)
+  - [Nix 0.12 (2008-11-20)](release-notes/rl-0.12.md)
-  - [Release 0.10 (2006-10-06)](release-notes/rl-0.10.md)
+  - [Nix 0.11 (2007-12-31)](release-notes/rl-0.11.md)
-  - [Release 0.9.2 (2005-09-21)](release-notes/rl-0.9.2.md)
+  - [Nix 0.10.1 (2006-10-11)](release-notes/rl-0.10.1.md)
-  - [Release 0.9.1 (2005-09-20)](release-notes/rl-0.9.1.md)
+  - [Nix 0.10 (2006-10-06)](release-notes/rl-0.10.md)
-  - [Release 0.9 (2005-09-16)](release-notes/rl-0.9.md)
+  - [Nix 0.9.2 (2005-09-21)](release-notes/rl-0.9.2.md)
-  - [Release 0.8.1 (2005-04-13)](release-notes/rl-0.8.1.md)
+  - [Nix 0.9.1 (2005-09-20)](release-notes/rl-0.9.1.md)
-  - [Release 0.8 (2005-04-11)](release-notes/rl-0.8.md)
+  - [Nix 0.9 (2005-09-16)](release-notes/rl-0.9.md)
-  - [Release 0.7 (2005-01-12)](release-notes/rl-0.7.md)
+  - [Nix 0.8.1 (2005-04-13)](release-notes/rl-0.8.1.md)
-  - [Release 0.6 (2004-11-14)](release-notes/rl-0.6.md)
+  - [Nix 0.8 (2005-04-11)](release-notes/rl-0.8.md)
-  - [Release 0.5 and earlier](release-notes/rl-0.5.md)
+  - [Nix 0.7 (2005-01-12)](release-notes/rl-0.7.md)
+  - [Nix 0.6 (2004-11-14)](release-notes/rl-0.6.md)
+  - [Nix 0.5 and earlier](release-notes/rl-0.5.md)

doc/manual/src/contributing/testing.md

@@ -12,44 +12,51 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
 > An example of some files, demonstrating much of what is described below
 >
 > ```
-> src
-> ├── libexpr
-> │   ├── value/context.hh
-> │   ├── value/context.cc
-> │   │
-> │   …
->     └── tests
-> │       ├── value/context.hh
-> │       ├── value/context.cc
-> │       │
-> │       …
-> │
-> ├── unit-test-data
-> │   ├── libstore
-> │   │   ├── worker-protocol/content-address.bin
-> │   │   …
-> │   …
 > …
+> ├── src
+> │   ├── libexpr
+> │   │   ├── …
+> │   │   ├── value
+> │   │   │   ├── context.cc
+> │   │   │   └── context.hh
+> │   …   …
+> ├── tests
+> │   …
+> │   └── unit
+> │       ├── libcmd
+> │       │   └── args.cc
+> │       ├── libexpr
+> │       │   ├── …
+> │       │   └── value
+> │       │       ├── context.cc
+> │       │       └── print.cc
+> │       ├── libexpr-support
+> │       │   └── tests
+> │       │       ├── libexpr.hh
+> │       │       └── value
+> │       │           ├── context.cc
+> │       │           └── context.hh
+> │       ├── libstore
+> │       │   ├── common-protocol.cc
+> │       │   ├── data
+> │       │   │   ├── libstore
+> │       │   │   │   ├── common-protocol
+> │       │   │   │   │   ├── content-address.bin
+> │       │   │   │   │   ├── drv-output.bin
+> …       …   …   …   …   …
 > ```
 
-<!-- FIXME(Lix): this might get renamed to liblixexpr, etc? -->
+The unit tests for each Lix library (`liblixexpr`, `liblixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
 
-The unit tests for each Lix library (`libnixexpr`, `libnixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
+The data is in `tests/unit/LIBNAME/data/LIBNAME`, with one subdir per library, with the same name as where the code goes.
-
+For example, `liblixstore` code is in `src/libstore`, and its test data is in `tests/unit/libstore/data/libstore`.
-The data is in `unit-test-data`, with one subdir per library, with the same name as where the code goes.
+The path to the unit test data directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
-For example, `libnixstore` code is in `src/libstore`, and its test data is in `unit-test-data/libstore`.
-The path to the `unit-test-data` directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
-
-> **Note**
-> Due to the way googletest works, downstream unit test executables will actually include and re-run upstream library tests.
-> Therefore it is important that the same value for `_NIX_TEST_UNIT_DATA` be used with the tests for each library.
-> That is why we have the test data nested within a single `unit-test-data` directory.
 
 ### Running tests
 
 You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test.
 
-To get a list of tests, use `meson test -C build --list`.
+To get a list of tests, use `meson test -C build --list` (or `just test --list` for short).
 
 For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test).
 
@@ -74,20 +81,27 @@ See [below](#characterization-testing-1) for a broader discussion of characteriz
 Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used.
 For example:
 ```shell-session
-$ _NIX_TEST_ACCEPT=1 make libstore-tests-exe_RUN
+$ _NIX_TEST_ACCEPT=1 just test --suite check libstore-unit-tests
 ...
-[  SKIPPED ] WorkerProtoTest.string_read
+../tests/unit/libstore/common-protocol.cc:27: Skipped
-[  SKIPPED ] WorkerProtoTest.string_write
+Cannot read golden master because another test is also updating it
-[  SKIPPED ] WorkerProtoTest.storePath_read
+
-[  SKIPPED ] WorkerProtoTest.storePath_write
+../tests/unit/libstore/common-protocol.cc:62: Skipped
+Updating golden master
+
+../tests/unit/libstore/common-protocol.cc:27: Skipped
+Cannot read golden master because another test is also updating it
+
+../tests/unit/libstore/common-protocol.cc:62: Skipped
+Updating golden master
 ...
 ```
-will regenerate the "golden master" expected result for the `libnixstore` characterization tests.
+will regenerate the "golden master" expected result for the `liblixstore` characterization tests.
 The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
 
 ## Functional tests
 
-The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/local.mk`.
+The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/meson.build`.
 Each test is a bash script.
 
 ### Running the whole test suite
@@ -185,7 +199,7 @@ edit it like so:
 ```
 
 <div class="warning">
-FIXME(meson): the command here may be incorrect for meson.
+FIXME(meson): the command here is incorrect for meson and this whole functionality may need rebuilding.
 </div>
 
 Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
@@ -209,15 +223,11 @@ This technique is to include the exact output/behavior of a former version of Ni
 
 For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
 
-<div class="warning">
-FIXME(meson): this is incorrect for meson. `_NIX_TEST_ACCEPT=1` is still valid but the test invocation needs to change.
-</div>
-
 It is frequently useful to regenerate the expected output.
 To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
 For example:
 ```bash
-_NIX_TEST_ACCEPT=1 make tests/functional/lang.sh.test
+_NIX_TEST_ACCEPT=1 just test --suite installcheck -v functional-lang
 ```
 
 An interesting situation to document is the case when these tests are "overfitted".
@@ -320,3 +330,135 @@ solved this need?
 ~~>
 
 -->
+
+## Magic environment variables
+
+FIXME: maybe this section should be moved elsewhere or turned partially into user docs, but I just need a complete index for now.
+I actually want to ban people calling getenv without writing documentation, and produce a comprehensive list of env-vars used by Lix and enforce it.
+
+This is a non-exhaustive list of almost all environment variables, magic or not, accepted or used by various parts of the test suite as well as Lix itself.
+Please add more if you find them.
+
+I looked for these in the testsuite with the following bad regexes:
+
+```
+rg '(?:[^A-Za-z]|^)(_[A-Z][^-\[ }/:");$(]+)' -r '$1' --no-filename --only-matching tests | sort -u > vars.txt
+rg '\$\{?([A-Z][^-\[ }/:");]+)' -r '$1' --no-filename --only-matching tests | sort -u > vars.txt
+```
+
+I grepped `src/` for `get[eE]nv\("` to find the mentions in Lix code.
+
+### Used by Lix testing support code
+
+- `_NIX_TEST_ACCEPT` (optional) - Writes out the result of a characterization test as the new expected value.
+  **Expected value**: 1
+
+- `_NIX_TEST_UNIT_DATA` - The path to the directory for the data for a given unit test suite.
+
+  **Expected value**: `tests/unit/libstore/data/libstore` or similar
+
+
+### Used by Lix
+
+- `_NIX_FORCE_HTTP` - Forces file URIs to be treated as remote ones.
+
+  Used by `src/libfetchers/git.cc`, `src/libstore/http-binary-cache-store.cc`,
+  `src/libstore/local-binary-cache-store.cc`. Seems to be for forcing Git
+  clones of `git+file://` URLs, making the HTTP binary
+  cache store accept `file://` URLs (presumably passing them to curl?), and
+  unknown reasons for the local binary cache.
+
+  FIXME(jade): is this obscuring a bug in https://git.lix.systems/lix-project/lix/issues/200?
+
+  **Expected value**: 1
+- `NIX_ATTRS_SH_FILE`, `NIX_ATTRS_JSON_FILE` (output) - Set by Lix builders; see
+  `structuredAttrs` documentation.
+- `NIX_BIN_DIR`, `NIX_STORE_DIR` (or its inconsistently-used old alias `NIX_STORE`), `NIX_DATA_DIR`,
+  `NIX_LOG_DIR`, `NIX_LOG_DIR`, `NIX_STATE_DIR`, `NIX_CONF_DIR` -
+  Overrides compile-time configuration of various locations used by Lix. See `src/libstore/globals.cc`.
+
+  **Expected value**: a directory
+- `NIX_DAEMON_SOCKET_PATH` (optional) - Overrides the daemon socket path from `$NIX_STATE_DIR/daemon-socket/socket`.
+
+  **Expected value**: path to a socket
+- `NIX_LOG_FD` (output) - An FD number for logs in `internal-json` format to be sent to.
+  Used for, mostly, "setPhase" in nixpkgs setup.sh, but can also be creatively used to print verbose log messages from derivations.
+
+  **Provided value**: number corresponding to an FD in the builder
+- `NIX_PATH` - Search path for `<whatever>`. Documented elsewhere in the manual.
+
+  **Expected value**: `:` separated list of things that are not necessarily pointing to filesystem paths
+- `NIX_REMOTE` - The default value of the Lix setting `store`.
+
+  **Expected value**: "daemon", usually. Could be "auto" or any other value acceptable in `store`.
+- `NIX_BUILD_SHELL` - Documented elsewhere; the shell to invoke with `nix-shell` but not `nix develop`/`nix shell`.
+  The latter ignoring it altogether seems like a bug.
+
+  **Expected value**: the path to an executable shell
+- `PRINT_PATH` - Undocumented. Used by `nix-prefetch-url` as an alternative form of `--print-path`. Why???
+- `_NIX_IN_TEST` - If present with any value, makes `fetchClosure` accept file URLs in addition to HTTP ones. Why is this not `_NIX_FORCE_HTTP`??
+
+  Not used anywhere else.
+- `NIX_ALLOW_EVAL` - Used by eval-cache tests to block evaluation if set to `0`.
+
+  **Expected value**: 1 or 0
+- `EDITOR` - Used by `editorFor()`, which has some extremely sketchy editor-detection code for jumping to line numbers.
+- `LISTEN_FDS` and `LISTEN_PID` - Used for systemd socket activation using the systemd socket activation protocol.
+- `NIX_PAGER` (alternatively, `PAGER`) - Used to select a pager for Lix output. Why does this not use libutil `getEnv()`?
+- `LESS` (output) - Sets the pager settings for `less` when invoked by Lix.
+- `NIX_IGNORE_SYMLINK_STORE` - When set, Lix allows the store to be a symlink. Why do we support this?
+
+  Apparently [someone was using it enough to fix it](https://github.com/NixOS/nix/pull/4038).
+- `NIX_SSL_CERT_FILE` (alternatively, `SSL_CERT_FILE`) - Used to set CA certificates for libcurl.
+
+  **Expected value**: "/etc/ssl/certs/ca-certificates.crt" or similar
+- `NIX_REMOTE_SYSTEMS` - Used to set `builders`. Can we please deprecate this?
+- `NIX_USER_CONF_FILES` - `:` separated list of config files to load before
+  `/nix/nix.conf` under each of `XDG_CONFIG_DIRS`.
+- `NIX_CONFIG` - Newline separated configuration to load into Lix.
+- `NIX_GET_COMPLETIONS` - Returns completions.
+  Unsure of the exact format, someone should document it; either way my shell never had any completions.
+
+  **Expected value**: number of completions to return.
+- `IN_SYSTEMD` - Used to switch the logging format so that systemd gets the correct log levels. I think.
+- `NIX_HELD_LOCKS` - Not used, what is this for?? We should surely remove it right after searching github?
+- `GC_INITIAL_HEAP_SIZE` - Used to set the initial heap size, processed by boehmgc.
+- `NIX_COUNT_CALLS` - Documented elsewhere; prints call counts for profiling purposes.
+- `NIX_SHOW_STATS` - Documented elsewhere; prints various evaluation statistics like function calls, gc info, and similar.
+- `NIX_SHOW_STATS_PATH` - Writes those statistics into a file at the given path instead of stdout. Undocumented.
+- `NIX_SHOW_SYMBOLS` - Dumps the symbol table into the show-stats json output.
+- `TERM` - If `dumb` or unset, disables ANSI colour output.
+- `NO_COLOR`, `NOCOLOR` - Disables ANSI colour output.
+- `_NIX_DEVELOPER_SHOW_UNKNOWN_LOCATIONS` - Highlights unknown locations in errors.
+- `NIX_PROFILE` - Selects which profile `nix-env` will operate on. Documented elsewhere.
+- `NIX_SSHOPTS` - Options passed to `ssh(1)` when using a ssh remote store.
+  Incorrectly documented on `nix-copy-closure` which is *surely* not the only place they are used??
+- `_NIX_TEST_NO_LSOF` - Used on non-Linux, non-macOS platforms to disable using `lsof` when finding gc roots.
+
+  Since https://git.lix.systems/lix-project/lix/issues/156 was fixed, this should probably just be removed as it was a bad workaround for a macOS issue.
+- `_NIX_TEST_GC_SYNC_1` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
+- `_NIX_TEST_GC_SYNC_2` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
+- `_NIX_TEST_FREE_SPACE_FILE` - Path to a file containing a decimal number with the free space that the GC is to believe it has.
+- Various XDG vars
+- `NIX_DEBUG_SQLITE_TRACES` - Dump all sqlite queries to the log at `notice` level.
+- `_NIX_TEST_NO_SANDBOX` - Disables actually setting up the sandbox on macOS while leaving other logic the same. Unused on other platforms.
+- `_NIX_TRACE_BUILT_OUTPUTS` - Dumps all the derivation paths alongside their outputs as lines into a file of the given name.
+
+### Used by the functional test framework
+
+- `NIX_DAEMON_PACKAGE` - Runs the test suite against an alternate Nix daemon with the current client.
+
+  **Expected value**: something like `/nix/store/...-nix-2.18.2`
+- `NIX_CLIENT_PACKAGE` - Runs the test suite against an alternate Nix client with the current daemon.
+
+  **Expected value**: something like `/nix/store/...-nix-2.18.2`
+- `NIX_TESTS_CA_BY_DEFAULT` - Pass `__contentAddressed`, `outputHashMode` and `outputHashAlgo` to builds of some input-addressed derivations in the test suite.
+
+  **Expected value**: 1
+- `TEST_DATA` - Not an environment variable! This is used in repl characterization tests to refer to `tests/functional/repl_characterization/data`.
+  More specifically, that path is replaced with the string `$TEST_DATA` in output for reproducibility.
+- `TEST_HOME` (output) - Set to the temporary directory that is set as `$HOME` inside the tests, underneath `$TEST_ROOT`.
+- `TEST_ROOT` (output) - Set to the temporary directory that is created for each test to mess with.
+- `_NIX_TEST_DAEMON_PID` (output) - Used to track the daemon pid to be able to kill it.
+
+  **Provided value**: Daemon pid as a base-10 integer, e.g. 2345

doc/manual/src/installation/installing-docker.md

@@ -1,64 +1,62 @@
 # Using Lix within Docker
 
-Currently the Lix project doesn't ship docker images. However, we have the infrastructure to do it, it's just not yet been done. See https://git.lix.systems/lix-project/lix/issues/252
+Lix is available on the following two container registries:
-
+- [ghcr.io/lix-project/lix](https://ghcr.io/lix-project/lix)
-<!--
+- [git.lix.systems/lix-project/lix](https://git.lix.systems/lix-project/-/packages/container/lix)
 
 To run the latest stable release of Lix with Docker run the following command:
 
 ```console
-$ docker run -ti nixos/nix
+~ » sudo podman run -it ghcr.io/lix-project/lix:latest
-Unable to find image 'nixos/nix:latest' locally
+Trying to pull ghcr.io/lix-project/lix:latest...
-latest: Pulling from nixos/nix
+
-5843afab3874: Pull complete
+bash-5.2# nix --version
-b52bf13f109c: Pull complete
+nix (Lix, like Nix) 2.90.0
-1e2415612aa3: Pull complete
-Digest: sha256:27f6e7f60227e959ee7ece361f75d4844a40e1cc6878b6868fe30140420031ff
-Status: Downloaded newer image for nixos/nix:latest
-35ca4ada6e96:/# nix --version
-nix (Nix) 2.3.12
-35ca4ada6e96:/# exit
 ```
 
 # What is included in Lix's Docker image?
 
-The official Docker image is created using `pkgs.dockerTools.buildLayeredImage`
+The official Docker image is created using [nix2container]
 (and not with `Dockerfile` as it is usual with Docker images). You can still
 base your custom Docker image on it as you would do with any other Docker
 image.
 
-The Docker image is also not based on any other image and includes minimal set
+[nix2container]: https://github.com/nlewo/nix2container
-of runtime dependencies that are required to use Lix:
 
- - pkgs.nix
+The Docker image is also not based on any other image and includes the nixpkgs
- - pkgs.bashInteractive
+that Lix was built with along with a minimal set of tools in the system profile:
- - pkgs.coreutils-full
+
- - pkgs.gnutar
+- bashInteractive
- - pkgs.gzip
+- cacert.out
- - pkgs.gnugrep
+- coreutils-full
- - pkgs.which
+- curl
- - pkgs.curl
+- findutils
- - pkgs.less
+- gitMinimal
- - pkgs.wget
+- gnugrep
- - pkgs.man
+- gnutar
- - pkgs.cacert.out
+- gzip
- - pkgs.findutils
+- iana-etc
+- less
+- libxml2
+- lix
+- man
+- openssh
+- sqlite
+- wget
+- which
 
 # Docker image with the latest development version of Lix
 
-To get the latest image that was built by [Hydra](https://hydra.nixos.org) run
+FIXME: There are not currently images of development versions of Lix. Tracking issue: https://git.lix.systems/lix-project/lix/issues/381
-the following command:
+
+You can build a Docker image from source yourself and copy it to either:
+
+Podman: `nix run '.#dockerImage.copyTo' containers-storage:lix`
+
+Docker: `nix run '.#dockerImage.copyToDockerDaemon'`
+
+Then:
 
 ```console
-$ curl -L https://hydra.nixos.org/job/nix/master/dockerImage.x86_64-linux/latest/download/1 | docker load
+$ docker run -ti lix
-$ docker run -ti nix:2.5pre20211105
 ```
-
-You can also build a Docker image from source yourself:
-
-```console
-$ nix build ./\#hydraJobs.dockerImage.x86_64-linux
-$ docker load -i ./result/image.tar.gz
-$ docker run -ti nix:2.5pre20211105
-```
--->

doc/manual/src/installation/multi-user.md

@@ -1,32 +1,121 @@
 # Multi-User Mode
 
-To allow a Nix store to be shared safely among multiple users, it is
+To allow a Nix store to be shared safely among multiple users, it is important that users cannot meaningfully influence the execution of derivation builds such that they could inject malicious code into them without changing their (either input- or output- addressed) hash.
-important that users are not able to run builders that modify the Nix
+If they could do so, they could install a Trojan horse in some package and compromise the accounts of other users.
-store or database in arbitrary ways, or that interfere with builds
-started by other users. If they could do so, they could install a Trojan
-horse in some package and compromise the accounts of other users.
 
-To prevent this, the Nix store and database are owned by some privileged
+To prevent this, the Nix store and database are owned by some privileged user (usually `root`) and builders are executed under unprivileged system user accounts (usually named `nixbld1`, `nixbld2`, etc.).
-user (usually `root`) and builders are executed under special user
+When an unprivileged user runs a Nix command, actions that operate on the Nix store (such as builds) are forwarded to a *Nix daemon* running under the owner of the Nix store/database that performs the operation.
-accounts (usually named `nixbld1`, `nixbld2`, etc.). When a unprivileged
-user runs a Nix command, actions that operate on the Nix store (such as
-builds) are forwarded to a *Nix daemon* running under the owner of the
-Nix store/database that performs the operation.
 
-> **Note**
+The buried lede in the above sentence is that *currently*, even in multi-user mode using a daemon, if executing as the user that owns the store, Lix directly manipulates the store unless `--store daemon` is specified.
->
+[We intend to change this in the future][multi-user-should-not-be-root].
-> Multi-user mode has one important limitation: only root and a set of
-> trusted users specified in `nix.conf` can specify arbitrary binary
-> caches. So while unprivileged users may install packages from
-> arbitrary Nix expressions, they may not get pre-built binaries.
 
-## Setting up the build users
+<div class="warning">
+The Lix team considers the goal of the sandbox to be primarily for preventing reproducibility mistakes, and does not consider multi-user mode to be a strong security boundary between users.
+
+Do not evaluate or build untrusted, potentially-malicious, Nix language code on machines that you care deeply about maintaining user isolation on.
+
+Although we would consider any sandbox escapes to be serious security bugs and we intend to fix them, we are not confident enough in the daemon's security to call the daemon a security boundary.
+</div>
+
+[multi-user-should-not-be-root]: https://git.lix.systems/lix-project/lix/issues/18
+
+## Trust model
+
+There are two categories of users of the Lix daemon: trusted users and untrusted users.
+The Lix daemon only allows connections from users that are either trusted users, or are specified in, or are members of groups specified in, [`allowed-users`](../command-ref/conf-file.md#conf-allowed-users) in `nix.conf`.
+Trusted users are users and users of groups specified in [`trusted-users`](../command-ref/conf-file.md#conf-trusted-users) in `nix.conf`.
+
+All users of the Lix daemon may do the following to bring things into the Nix store:
+
+- Users may load derivations and output-addressed files into the store with `nix-store --add` or through Nix language code.
+- Users may locally build derivations, either of the output-addressed or input-addressed variety, creating output paths.
+
+  Note that [fixed-output derivations only consider name and hash](https://github.com/NixOS/nix/issues/969), so it is possible to write a fixed-output derivation for something important with a bogus hash and have it resolve to something else already built in the store.
+
+  On systems with `sandbox` enabled (default on Linux; [not *yet* on macOS][sandbox-enable-macos]), derivations are either:
+    - Input-addressed, so they are run in the sandbox with no network access, with the following exceptions:
+
+      - The (poorly named, since it is not *just* about chroot) property `__noChroot` is set on the derivation and `sandbox` is set to `relaxed`.
+      - On macOS, the derivation property `__darwinAllowLocalNetworking` allows network access to localhost from input-addressed derivations regardless of the `sandbox` setting value. This property exists with such semantics because macOS has no network namespace equivalent to isolate individual processes' localhost networking.
+    - Output-addressed, so they are run with network access but their result must match an expected hash.
+
+  Trusted users may set any setting, including `sandbox = false`, so the sandbox state can be different at runtime from what is described in `nix.conf` for builds invoked with such settings.
+- Users may copy appropriately-signed derivation outputs into the store.
+
+  By default, any paths *copied into a store* (such as by substitution) must have signatures from [`trusted-public-keys`](../command-ref/conf-file.md#conf-trusted-public-keys) unless they are [output-addressed](../glossary.md#gloss-output-addressed-store-object).
+
+  Unsigned paths may be copied into a store if [`require-sigs`](../command-ref/conf-file.md#conf-require-sigs) is disabled in the daemon's configuration (not default), or if the client is a trusted user and passed `--no-check-sigs` to `nix copy`.
+- Users may request that the daemon substitutes appropriately-signed derivation outputs from a binary cache in the daemon's [`substituters`](../command-ref/conf-file.md#conf-substituters) list.
+
+  Untrusted clients may also specify additional values for `substituters` (via e.g. `--extra-substituters` on a Nix command) that are listed in [`trusted-substituters`](../command-ref/conf-file.md#conf-trusted-substituters).
+
+  A client could in principle substitute such paths itself then copy them to the daemon (see clause above) if they are appropriately signed but are *not* from a trusted substituter, however this is not implemented in the current Lix client to our knowledge, at the time of writing.
+  This probably means that `trusted-substituters` is a redundant setting except insofar as such substitution would have to be done on the client rather than as root on the daemon; and it is highly defensible to not allow random usage of our HTTP client running as root.
+
+[sandbox-enable-macos]: https://git.lix.systems/lix-project/lix/issues/386
+
+### The Lix daemon as a security non-boundary
+
+The Lix team and wider community does not consider the Lix daemon to be a *security boundary* against malicious Nix language code.
+
+Although we do our best to make it secure, we do not recommend sharing a Lix daemon with potentially malicious users.
+That means that public continuous integration (CI) builds of untrusted Nix code should not share builders with CI that writes into a cache used by trusted infrastructure.
+
+For example, [hydra.nixos.org], which is the builder for [cache.nixos.org], does not execute untrusted Nix language code; a separate system, [ofborg] is used for CI of nixpkgs pull requests.
+The build output of pull request CI is never pushed to [cache.nixos.org], and those systems are considered entirely untrusted.
+
+This is because, among other things, the Lix sandbox is *more* susceptible to kernel exploits than Docker, which, unlike Lix, blocks nested user namespaces via `seccomp` in its default policy, and there have been many kernel bugs only exposed to unprivileged users via user namespaces allowing otherwise-root-only system calls.
+In general, the Lix sandbox is set up to be relatively unrestricted while maintaining its goals of building useful, reproducible software; security is not its primary goal.
+
+The Lix sandbox is a custom *non-rootless* Linux container implementation that has not been audited to nearly the same degree as Docker and similar systems.
+Also, the Lix daemon is a complex and historied C++ executable running as root with very little privilege separation.
+All of this means that a security hole in the Lix daemon gives immediate root access.
+Systems like Docker (especially non-rootless Docker) should *themselves* probably not be used in a multi-tenant manner with mutually distrusting tenants, but the Lix daemon *especially* should not be used as such as of this writing.
+
+The primary purpose of the sandbox is to strongly encourage packages to be reproducible, a goal which it is generally quite successful at.
+
+[hydra.nixos.org]: https://hydra.nixos.org
+[ofborg]: https://github.com/NixOS/ofborg
+[cache.nixos.org]: https://cache.nixos.org
+
+### Trusted users
+
+Trusted users are permitted to set any setting and bypass security restrictions on the daemon.
+They are currently in widespread use for a couple of reasons such as remote builds (which we [intend to fix](https://git.lix.systems/lix-project/lix/issues/171)).
+
+Trusted users are effectively root on Nix daemons running as root (the default configuration) for *at least* the following reasons, and should be thus thought of as equivalent to passwordless sudo.
+This is not a comprehensive list.
+
+- They may copy an unsigned malicious built output into the store for `systemd` or anything else that will run as root, then when the system is upgraded, that path will be used from the local store rather than substituted.
+- They may set the following settings that are commands the daemon will run as root:
+  - `build-hook`
+  - `diff-hook`
+  - `pre-build-hook`
+  - `post-build-hook`
+- They may set `build-users-group`.
+
+  In particular, they may set it to empty string, which runs builds as root with respect to the rest of the system (!!).
+  We, too, [think that is absurd and intend to not accept such a configuration](https://git.lix.systems/lix-project/lix/issues/242).
+  It is then simply an exercise to the reader to find a daemon that does `SCM_CREDENTIALS` over a `unix(7)` socket and lets you run commands as root, and mount it into the sandbox with `extra-sandbox-paths`.
+
+  At the very least, the Lix daemon itself (since `root` is a trusted user by default) and probably `systemd` qualify for this.
+- They may set the `builders` list, which will have ssh run as root.
+  We aren't sure if there is a way to abuse this for command execution but it's plausible.
+
+Note that setting `accept-flake-config` allows arbitrary Nix flakes to set Nix settings in the `nixConfig` stanza.
+Do not set this setting or pass `--accept-flake-config` while executing untrusted Nix language code as a trusted user for the reasons above!
+
+## Build users
 
 The *build users* are the special UIDs under which builds are performed.
-They should all be members of the *build users group* `nixbld`. This
+A build user is selected for a build by looking in the group specified by [`build-users-group`](../command-ref/conf-file.md#conf-build-users-group), by default, `nixbld`, then a member of that group not currently executing a build is selected for the build.
-group should have no other members. The build users should not be
+The build users should not be members of any other group.
-members of any other group. On Linux, you can create the group and users
+
-as follows:
+There can never be more concurrent builds than the number of build users, unless using [`auto-allocate-uids`](../command-ref/conf-file.md#conf-auto-allocate-uids) ([tracking issue][auto-allocate-uids-issue]).
+
+[auto-allocate-uids-issue]: https://git.lix.systems/lix-project/lix/issues/387
+
+If, for some reason, you need to create such users manually, the following command will create 10 build users on Linux:
 
 ```console
 $ groupadd -r nixbld
@@ -35,43 +124,12 @@ $ for n in $(seq 1 10); do useradd -c "Nix build user $n" \
     nixbld$n; done
 ```
 
-This creates 10 build users. There can never be more concurrent builds
-than the number of build users, so you may want to increase this if you
-expect to do many builds at the same time.
-
 ## Running the daemon
 
-The [Nix daemon](../command-ref/nix-daemon.md) should be started as
+The [Nix daemon](../command-ref/nix-daemon.md) can be started manually as follows (as `root`):
-follows (as `root`):
 
 ```console
-$ nix-daemon
+# nix-daemon
 ```
 
-You’ll want to put that line somewhere in your system’s boot scripts.
+In standard installations of Lix, the daemon is started by a `systemd` unit (Linux) or `launchd` service (macOS).
-
-To let unprivileged users use the daemon, they should set the
-[`NIX_REMOTE` environment variable](../command-ref/env-common.md) to
-`daemon`. So you should put a line like
-
-```console
-export NIX_REMOTE=daemon
-```
-
-into the users’ login scripts.
-
-## Restricting access
-
-To limit which users can perform Nix operations, you can use the
-permissions on the directory `/nix/var/nix/daemon-socket`. For instance,
-if you want to restrict the use of Nix to the members of a group called
-`nix-users`, do
-
-```console
-$ chgrp nix-users /nix/var/nix/daemon-socket
-$ chmod ug=rwx,o= /nix/var/nix/daemon-socket
-```
-
-This way, users who are not in the `nix-users` group cannot connect to
-the Unix domain socket `/nix/var/nix/daemon-socket/socket`, so they
-cannot perform Nix operations.

doc/manual/src/installation/nix-security.md

@@ -7,9 +7,8 @@ management operations. All other users can then use the installed
 packages, but they cannot perform package management operations
 themselves.
 
-Alternatively, you can configure Lix in “multi-user mode”. In this
+Alternatively, you can configure Lix in “multi-user mode”. In this model, all users can perform package management operations — for instance, every user can install software for themselves without requiring root privileges.
-model, all users can perform package management operations — for
+Lix does its best to ensure that this is secure.
-instance, every user can install software without requiring root
+For instance, it would be considered a serious security bug for one untrusted user to be able to overwrite a package used by another user with a Trojan horse.
-privileges. Lix ensures that this is secure. For instance, it’s not
+
-possible for one user to overwrite a package used by another user with a
+Nevertheless, the Lix team does not consider multi-user mode a strong security boundary, and does not recommend running untrusted user-supplied Nix language code on privileged machines, even if it is secure to the best of our knowledge at any moment in time.
-Trojan horse.

doc/manual/src/release-notes/rl-2.0.md

@@ -229,7 +229,7 @@ This release has the following new features:
     <https://cache.nixos.org/> or <ssh://machine>. The following store
     types are supported:
 
-      - `LocalStore` (stori URI `local` or an absolute path) and the
+      - `LocalStore` (store URI `local` or an absolute path) and the
         misnamed `RemoteStore` (`daemon`) provide access to a local Nix
         store, the latter via the Nix daemon. You can use `auto` or the
         empty string to auto-select a local or daemon store depending on

doc/manual/src/release-notes/rl-2.90.md

@@ -0,0 +1,929 @@
+# Lix 2.90 "Vanilla Ice Cream" (FIXME date)
+
+
+# Lix 2.90.0 (FIXME date)
+
+## Breaking Changes
+- Deprecate the online flake registries and vendor the default registry [fj#183](https://git.lix.systems/lix-project/lix/issues/183) [fj#110](https://git.lix.systems/lix-project/lix/issues/110) [fj#116](https://git.lix.systems/lix-project/lix/issues/116) [#8953](https://github.com/NixOS/nix/issues/8953) [#9087](https://github.com/NixOS/nix/issues/9087) [cl/1127](https://gerrit.lix.systems/c/lix/+/1127)
+
+  The online flake registry [https://channels.nixos.org/flake-registry.json](https://channels.nixos.org/flake-registry.json) is not pinned in any way,
+  and the targets of the indirections can both update or change entirely at any
+  point. Furthermore, it is refetched on every use of a flake reference, even if
+  there is a local flake reference, and even if you are offline (which breaks).
+
+  For now, we deprecate the (any) online flake registry, and vendor a copy of the
+  current online flake registry. This makes it work offline, and ensures that
+  it won't change in the future.
+
+  Many thanks to [julia](https://git.lix.systems/midnightveil) for this.
+- Enforce syscall filtering and no-new-privileges on Linux [cl/1063](https://gerrit.lix.systems/c/lix/+/1063)
+
+  In order to improve consistency of the build environment, system call filtering and no-new-privileges are now unconditionally enabled on Linux.
+  The `filter-syscalls` and `allow-new-privileges` options which could be used to disable these features under some circumstances have been removed.
+
+  In order to support building on architectures without libseccomp support, the option to disable syscall filtering at build time remains.
+  However, other uses of this option are heavily discouraged, since it would reduce the security of the sandbox substantially.
+
+  Many thanks to [alois31](https://git.lix.systems/alois31) for this.
+- Overhaul `nix flake update` and `nix flake lock` UX [#8817](https://github.com/NixOS/nix/pull/8817)
+
+  The interface for creating and updating lock files has been overhauled:
+
+  - [`nix flake lock`](@docroot@/command-ref/new-cli/nix3-flake-lock.md) only creates lock files and adds missing inputs now.
+  It will *never* update existing inputs.
+
+  - [`nix flake update`](@docroot@/command-ref/new-cli/nix3-flake-update.md) does the same, but *will* update inputs.
+  - Passing no arguments will update all inputs of the current flake, just like it already did.
+  - Passing input names as arguments will ensure only those are updated. This replaces the functionality of `nix flake lock --update-input`
+  - To operate on a flake outside the current directory, you must now pass `--flake path/to/flake`.
+
+  - The flake-specific flags `--recreate-lock-file` and `--update-input` have been removed from all commands operating on installables.
+  They are superceded by `nix flake update`.
+
+  Many thanks to [iFreilicht](https://github.com/iFreilicht), [Lunaphied](https://git.lix.systems/Lunaphied), and [Théophane Hufschmitt](https://github.com/thufschmitt) for this.
+- `nix profile` now allows referring to elements by human-readable name, and no longer accepts indices [#8678](https://github.com/NixOS/nix/pull/8678) [cl/978](https://gerrit.lix.systems/c/lix/+/978) [cl/980](https://gerrit.lix.systems/c/lix/+/980)
+
+  [`nix profile`](@docroot@/command-ref/new-cli/nix3-profile.md) now uses names to refer to installed packages when running [`list`](@docroot@/command-ref/new-cli/nix3-profile-list.md), [`remove`](@docroot@/command-ref/new-cli/nix3-profile-remove.md) or [`upgrade`](@docroot@/command-ref/new-cli/nix3-profile-upgrade.md) as opposed to indices. Indices have been removed. Profile element names are generated when a package is installed and remain the same until the package is removed.
+
+  **Warning**: The `manifest.nix` file used to record the contents of profiles has changed. Lix will automatically upgrade profiles to the new version when you modify the profile. After that, the profile can no longer be used by older versions of Lix.
+
+  Many thanks to [iFreilicht](https://github.com/iFreilicht), [Qyriad](https://git.lix.systems/Qyriad), and [Eelco Dolstra](https://github.com/edolstra) for this.
+- `builtins.nixVersion` and `builtins.langVersion` return fixed values [cl/558](https://gerrit.lix.systems/c/lix/+/558) [cl/1144](https://gerrit.lix.systems/c/lix/+/1144)
+
+  `builtins.nixVersion` now returns a fixed value `"2.18.3-lix"`.
+
+  `builtins.langVersion` returns a fixed value `6`, matching CppNix 2.18.
+
+  This prevents feature detection assuming that features that exist in Nix
+  post-Lix-branch-off might exist, even though the Lix version is greater than
+  the Nix version.
+
+  In the future, check for builtins for feature detection. If a feature cannot be
+  detected by *those* means, please file a Lix bug.
+
+  Many thanks to [jade](https://git.lix.systems/jade) for this.
+- Rename all the libraries nixexpr, nixstore, etc to lixexpr, lixstore, etc
+
+  The Lix C++ API libraries have had the following changes:
+  - Includes moved from `include/nix/` to `include/lix/`
+  - `pkg-config` files renamed from `nix-expr` to `lix-expr` and so on.
+  - Libraries renamed from `libnixexpr.so` to `liblixexpr.so` and so on.
+
+  There are other changes between Nix 2.18 and Lix, since these APIs are not
+  stable. However, this change in particular is a deliberate compatibility break
+  to force downstreams linking to Lix to specifically handle Lix and avoid Lix
+  accidentally getting ensnared in compatibility code for newer CppNix.
+
+  Migration path:
+
+  - expr.hh      -> lix/libexpr/expr.hh
+  - nix/config.h -> lix/config.h
+
+  To apply this migration automatically, remove all `<nix/>` from includes, so `#include <nix/expr.hh>` -> `#include <expr.hh>`.
+  Then, the correct paths will be resolved from the tangled mess, and the clang-tidy automated fix will work.
+
+  Then run the following for out of tree projects (header filter is set to only fix instances in headers in `../src` relative to the compiler's working directory, as would be the case in nix-eval-jobs or other things built with meson, e.g.):
+
+  ```console
+  lix_root=$HOME/lix
+  (cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build')
+  run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -header-filter '\.\./src/.*\.h' -fix src
+  ```
+
+  Many thanks to [jade](https://git.lix.systems/jade) for this.
+
+## Features
+- Experimental REPL support for documentation comments using `:doc` [cl/564](https://gerrit.lix.systems/c/lix/+/564)
+
+  Using `:doc` in the REPL now supports showing documentation comments when defined on a function.
+
+  Previously this was only able to document builtins, however it now will show comments defined on a lambda as well.
+
+  This support is experimental and relies on an embedded version of [nix-doc](https://github.com/lf-/nix-doc).
+
+  The logic also supports limited Markdown formatting of doccomments and should easily support any [RFC 145](https://github.com/NixOS/rfcs/blob/master/rfcs/0145-doc-strings.md)
+  compatible documentation comments in addition to simple commented documentation.
+
+  Many thanks to [Lunaphied](https://git.lix.systems/Lunaphied) and [jade](https://git.lix.systems/jade) for this.
+- Add `repl-overlays` option [#10203](https://github.com/NixOS/nix/pull/10203) [cl/504](https://gerrit.lix.systems/c/lix/+/504)
+
+  A `repl-overlays` option has been added, which specifies files that can overlay
+  and modify the top-level bindings in `nix repl`. For example, with the
+  following contents in `~/.config/nix/repl.nix`:
+
+  ```nix
+  info: final: prev: let
+    optionalAttrs = predicate: attrs:
+      if predicate
+      then attrs
+      else {};
+  in
+    optionalAttrs (prev ? legacyPackages && prev.legacyPackages ? ${info.currentSystem})
+    {
+      pkgs = prev.legacyPackages.${info.currentSystem};
+    }
+  ```
+
+  We can run `nix repl` and use `pkgs` to refer to `legacyPackages.${currentSystem}`:
+
+  ```ShellSession
+  $ nix repl --repl-overlays ~/.config/nix/repl.nix nixpkgs
+  Lix 2.90.0
+  Type :? for help.
+  Loading installable 'flake:nixpkgs#'...
+  Added 5 variables.
+  Loading 'repl-overlays'...
+  Added 6 variables.
+  nix-repl> pkgs.bash
+  «derivation /nix/store/g08b5vkwwh0j8ic9rkmd8mpj878rk62z-bash-5.2p26.drv»
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Add a builtin `addDrvOutputDependencies` [#7910](https://github.com/NixOS/nix/issues/7910) [#9216](https://github.com/NixOS/nix/pull/9216)
+
+  This builtin allows taking a `drvPath`-like string and turning it into a string
+  with context such that, when it lands in a derivation, it will create
+  dependencies on *all the outputs* in its closure (!). Although `drvPath` does this
+  today, this builtin starts forming a path to migrate to making `drvPath` have a
+  more normal and less surprising string context behaviour (see linked issue and
+  PR for more details).
+
+  Many thanks to [John Ericson](https://github.com/ericson2314) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Enter the `--debugger` when `builtins.trace` is called if `debugger-on-trace` is set [#9914](https://github.com/NixOS/nix/pull/9914)
+
+  If the `debugger-on-trace` option is set and `--debugger` is given,
+  `builtins.trace` calls will behave similarly to `builtins.break` and will enter
+  the debug REPL. This is useful for determining where warnings are being emitted
+  from.
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Add an option `enable-core-dumps` that enables core dumps from builds [cl/1088](https://gerrit.lix.systems/c/lix/+/1088)
+
+  In the past, Lix disabled core dumps by setting the soft `RLIMIT_CORE` to 0
+  unconditionally. Although this rlimit could be altered from the builder since
+  it is just the soft limit, this was kind of annoying to do. By passing
+  `--option enable-core-dumps true` to an offending build, one can now cause the
+  core dumps to be handled by the system in the normal way (winding up in
+  `coredumpctl`, say, on Linux).
+
+  Many thanks to [julia](https://git.lix.systems/midnightveil) for this.
+- Add new `eval-system` setting [#4093](https://github.com/NixOS/nix/pull/4093)
+
+  Add a new `eval-system` option.
+  Unlike `system`, it just overrides the value of `builtins.currentSystem`.
+  This is more useful than overriding `system`, because you can build these derivations on remote builders which can work on the given system.
+  In contrast, `system` also effects scheduling which will cause Lix to build those derivations locally even if that doesn't make sense.
+
+  `eval-system` only takes effect if it is non-empty.
+  If empty (the default) `system` is used as before, so there is no breakage.
+
+  Many thanks to [matthewbauer](https://github.com/matthewbauer) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- add `--store-path` argument to `nix upgrade-nix`, to manually specify the Nix to upgrade to [cl/953](https://gerrit.lix.systems/c/lix/+/953)
+
+  `nix upgrade-nix` by default downloads a manifest to find the new Nix version to upgrade to, but now you can specify `--store-path` to upgrade Nix to an arbitrary version from the Nix store.
+
+  Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
+
+## Improvements
+- `nix flake check` logs the checks [#8882](https://github.com/NixOS/nix/issues/8882) [#8893](https://github.com/NixOS/nix/pull/8893) [cl/259](https://gerrit.lix.systems/c/lix/+/259) [cl/260](https://gerrit.lix.systems/c/lix/+/260) [cl/261](https://gerrit.lix.systems/c/lix/+/261) [cl/262](https://gerrit.lix.systems/c/lix/+/262)
+
+  `nix flake check` now logs the checks it runs and the derivations it evaluates:
+
+  ```
+  $ nix flake check -v
+  evaluating flake...
+  checking flake output 'checks'...
+  checking derivation 'checks.aarch64-darwin.ghciwatch-tests'...
+  derivation evaluated to /nix/store/nh7dlvsrhds4cxl91mvgj4h5cbq6skmq-ghciwatch-test-0.3.0.drv
+  checking derivation 'checks.aarch64-darwin.ghciwatch-clippy'...
+  derivation evaluated to /nix/store/9cb5a6wmp6kf6hidqw9wphidvb8bshym-ghciwatch-clippy-0.3.0.drv
+  checking derivation 'checks.aarch64-darwin.ghciwatch-doc'...
+  derivation evaluated to /nix/store/8brdd3jbawfszpbs7vdpsrhy80as1il8-ghciwatch-doc-0.3.0.drv
+  checking derivation 'checks.aarch64-darwin.ghciwatch-fmt'...
+  derivation evaluated to /nix/store/wjhs0l1njl5pyji53xlmfjrlya0wmz8p-ghciwatch-fmt-0.3.0.drv
+  checking derivation 'checks.aarch64-darwin.ghciwatch-audit'...
+  derivation evaluated to /nix/store/z0mps8dyj2ds7c0fn0819y5h5611033z-ghciwatch-audit-0.3.0.drv
+  checking flake output 'packages'...
+  checking derivation 'packages.aarch64-darwin.default'...
+  derivation evaluated to /nix/store/41abbdyglw5x9vcsvd89xan3ydjf8d7r-ghciwatch-0.3.0.drv
+  checking flake output 'apps'...
+  checking flake output 'devShells'...
+  checking derivation 'devShells.aarch64-darwin.default'...
+  derivation evaluated to /nix/store/bc935gz7dylzmcpdb5cczr8gngv8pmdb-nix-shell.drv
+  running 5 flake checks...
+  warning: The check omitted these incompatible systems: aarch64-linux, x86_64-darwin, x86_64-linux
+  Use '--all-systems' to check all.
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt), [Raito Bezarius](https://git.lix.systems/raito), and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Add an option `always-allow-substitutes` to ignore `allowSubstitutes` in derivations [#8047](https://github.com/NixOS/nix/pull/8047)
+
+  You can set this setting to force a system to always allow substituting even
+  trivial derivations like `pkgs.writeText`. This is useful for
+  [`nix-fast-build --skip-cached`][skip-cached] and similar to be able to also
+  ignore trivial derivations.
+
+  [skip-cached]: https://github.com/Mic92/nix-fast-build?tab=readme-ov-file#avoiding-redundant-package-downloads
+
+  Many thanks to [lovesegfault](https://github.com/lovesegfault) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Concise error printing in `nix repl` [#9928](https://github.com/NixOS/nix/pull/9928) [cl/811](https://gerrit.lix.systems/c/lix/+/811)
+
+  Previously, if an element of a list or attribute set threw an error while
+  evaluating, `nix repl` would print the entire error (including source location
+  information) inline. This output was clumsy and difficult to parse:
+
+  ```
+  nix-repl> { err = builtins.throw "uh oh!"; }
+  { err = «error:
+         … while calling the 'throw' builtin
+           at «string»:1:9:
+              1| { err = builtins.throw "uh oh!"; }
+               |         ^
+
+         error: uh oh!»; }
+  ```
+
+  Now, only the error message is displayed, making the output much more readable.
+  ```
+  nix-repl> { err = builtins.throw "uh oh!"; }
+  { err = «error: uh oh!»; }
+  ```
+
+  However, if the whole expression being evaluated throws an error, source
+  locations and (if applicable) a stack trace are printed, just like you'd expect:
+
+  ```
+  nix-repl> builtins.throw "uh oh!"
+  error:
+         … while calling the 'throw' builtin
+           at «string»:1:1:
+              1| builtins.throw "uh oh!"
+               | ^
+
+         error: uh oh!
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Show all FOD errors with `nix build --keep-going` [cl/1108](https://gerrit.lix.systems/c/lix/+/1108)
+
+  `nix build --keep-going` now behaves consistently with `nix-build --keep-going`. This means
+  that if e.g. multiple FODs fail to build, all hash mismatches are displayed.
+
+  Many thanks to [ma27](https://git.lix.systems/ma27) for this.
+- Duplicate attribute reports are more accurate [cl/557](https://gerrit.lix.systems/c/lix/+/557)
+
+  Duplicate attribute errors are now more accurate, showing the path at which an error was detected rather than the full, possibly longer, path that caused the error.
+  Error reports are now
+  ```ShellSession
+  $ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
+  error: attribute 'a.b' already defined at «string»:1:3
+         at «string»:1:12:
+              1| { a.b = 1; a.b.c.d = 1;
+               |            ^
+  ```
+  instead of
+  ```ShellSession
+  $ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
+  error: attribute 'a.b.c.d' already defined at «string»:1:3
+         at «string»:1:12:
+              1| { a.b = 1; a.b.c.d = 1;
+               |            ^
+  ```
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Reduce eval memory usage and wall time [#9658](https://github.com/NixOS/nix/pull/9658) [cl/207](https://gerrit.lix.systems/c/lix/+/207)
+
+  Reduce the size of the `Env` struct used in the evaluator by a pointer, or 8 bytes on most modern machines.
+  This reduces memory usage during eval by around 2% and wall time by around 3%.
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Warn on unknown settings anywhere in the command line [#10701](https://github.com/NixOS/nix/pull/10701)
+
+  All `nix` commands will now properly warn when an unknown option is specified anywhere in the command line.
+
+  Before:
+
+  ```console
+  $ nix-instantiate --option foobar baz --expr '{}'
+  warning: unknown setting 'foobar'
+  $ nix-instantiate '{}' --option foobar baz --expr
+  $ nix eval --expr '{}' --option foobar baz
+  { }
+  ```
+
+  After:
+
+  ```console
+  $ nix-instantiate --option foobar baz --expr '{}'
+  warning: unknown setting 'foobar'
+  $ nix-instantiate '{}' --option foobar baz --expr
+  warning: unknown setting 'foobar'
+  $ nix eval --expr '{}' --option foobar baz
+  warning: unknown setting 'foobar'
+  { }
+  ```
+
+  Many thanks to [Cole Helbling](https://github.com/cole-h) for this.
+- Nested debuggers are no longer supported [#9920](https://github.com/NixOS/nix/pull/9920)
+
+  Previously, evaluating an expression that throws an error in the debugger would
+  enter a second, nested debugger:
+
+  ```
+  nix-repl> builtins.throw "what"
+  error: what
+
+
+  Starting REPL to allow you to inspect the current state of the evaluator.
+
+  Welcome to Nix 2.18.1. Type :? for help.
+
+  nix-repl>
+  ```
+
+  Now, it just prints the error message like `nix repl`:
+
+  ```
+  nix-repl> builtins.throw "what"
+  error:
+         … while calling the 'throw' builtin
+           at «string»:1:1:
+              1| builtins.throw "what"
+               | ^
+
+         error: what
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Find GC roots using libproc on Darwin [cl/723](https://gerrit.lix.systems/c/lix/+/723)
+
+  Previously, the garbage collector found runtime roots on Darwin by shelling out to `lsof -n -w -F n` then parsing the result. The version of `lsof` packaged in Nixpkgs is very slow on Darwin, so Lix now uses `libproc` directly to speed up GC root discovery, in some tests taking 250ms now instead of 40s.
+
+  Many thanks to [Artemis Tosini](https://git.lix.systems/artemist) for this.
+- Increase default stack size on macOS [#9860](https://github.com/NixOS/nix/pull/9860)
+
+  Increase the default stack size on macOS to the same value as on Linux, subject to system restrictions to maximum stack size.
+  This should reduce the number of stack overflow crashes on macOS when evaluating Nix code with deep call stacks.
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Show more log context for failed builds [#9670](https://github.com/NixOS/nix/pull/9670)
+
+  Show 25 lines of log tail instead of 10 for failed builds.
+  This increases the chances of having useful information in the shown logs.
+
+  Many thanks to [DavHau](https://github.com/DavHau) for this.
+- rename 'nix show-config' to 'nix config show' [#7672](https://github.com/NixOS/nix/issues/7672) [#9477](https://github.com/NixOS/nix/pull/9477) [cl/993](https://gerrit.lix.systems/c/lix/+/993)
+
+  `nix show-config` was renamed to `nix config show` to be more consistent with the rest of the command-line interface.
+
+  Running `nix show-config` will now print a deprecation warning saying to use `nix config show` instead.
+
+  Many thanks to [Théophane Hufschmitt](https://github.com/thufschmitt) and [ma27](https://git.lix.systems/ma27) for this.
+- Print derivation paths in `nix eval` [cl/446](https://gerrit.lix.systems/c/lix/+/446)
+
+  `nix eval` previously printed derivations as attribute sets, so commands that print derivations (e.g. `nix eval nixpkgs#bash`) would infinitely loop and segfault.
+  It now prints the `.drv` path the derivation generates instead.
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Add an option `--unpack` to unpack archives in `nix store prefetch-file` [#9805](https://github.com/NixOS/nix/pull/9805) [cl/224](https://gerrit.lix.systems/c/lix/+/224)
+
+  It is now possible to fetch an archive then NAR-hash it (as in, hash it in the
+  same manner as `builtins.fetchTarball` or fixed-output derivations with
+  recursive hash type) in one command.
+
+  Example:
+
+  ```
+  ~ » nix store prefetch-file --name source --unpack https://git.lix.systems/lix-project/lix/archive/2.90-beta.1.tar.gz
+  Downloaded 'https://git.lix.systems/lix-project/lix/archive/2.90-beta.1.tar.gz' to '/nix/store/yvfqnq52ryjc3janw02ziv7kr6gd0cs1-source' (hash 'sha256-REWlo2RYHfJkxnmZTEJu3Cd/2VM+wjjpPy7Xi4BdDTQ=').
+  ```
+
+  Many thanks to [yshui](https://github.com/yshui) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- REPL printing improvements [#9931](https://github.com/NixOS/nix/pull/9931) [#10208](https://github.com/NixOS/nix/pull/10208) [cl/375](https://gerrit.lix.systems/c/lix/+/375) [cl/492](https://gerrit.lix.systems/c/lix/+/492)
+
+  The REPL printer has been improved to do the following:
+  - If a string is passed to `:print`, it is printed literally to the screen
+  - Structures will be printed as multiple lines when necessary
+
+  Before:
+
+  ```
+  nix-repl> { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
+  { attrs = { ... }; list = [ ... ]; list' = [ ... ]; }
+
+  nix-repl> :p { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
+  { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
+
+  nix-repl> :p "meow"
+  "meow"
+  ```
+
+  After:
+
+  ```
+  nix-repl> { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
+  {
+    attrs = { ... };
+    list = [ ... ];
+    list' = [ ... ];
+  }
+
+  nix-repl> :p { attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
+  {
+    attrs = {
+      a = {
+        b = {
+          c = { };
+        };
+      };
+    };
+    list = [ 1 ];
+    list' = [
+      1
+      2
+      3
+    ];
+  }
+
+  nix-repl> :p "meow"
+  meow
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Coercion errors include the failing value [#561](https://github.com/NixOS/nix/issues/561) [#9754](https://github.com/NixOS/nix/pull/9754)
+
+  The `error: cannot coerce a <TYPE> to a string` message now includes the value
+  which caused the error.
+
+  Before:
+
+  ```
+         error: cannot coerce a set to a string
+  ```
+
+  After:
+
+  ```
+         error: cannot coerce a set to a string: { aesSupport = «thunk»;
+         avx2Support = «thunk»; avx512Support = «thunk»; avxSupport = «thunk»;
+         canExecute = «thunk»; config = «thunk»; darwinArch = «thunk»; darwinMinVersion
+         = «thunk»; darwinMinVersionVariable = «thunk»; darwinPlatform = «thunk»; «84
+         attributes elided»}
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- New-cli flake commands that expect derivations now print the failing value and its type [cl/1177](https://gerrit.lix.systems/c/lix/+/1177)
+
+  In errors like `flake output attribute 'legacyPackages.x86_64-linux.lib' is not a derivation or path`, the message now includes the failing value and type.
+
+  Before:
+
+  ```
+      error: flake output attribute 'nixosConfigurations.yuki.config' is not a derivation or path
+  ````
+
+  After:
+
+  ```
+      error: expected flake output attribute 'nixosConfigurations.yuki.config' to be a derivation or path but found a set: { appstream = «thunk»; assertions = «thunk»; boot = { bcache = «thunk»; binfmt = «thunk»; binfmtMiscRegistrations = «thunk»; blacklistedKernelModules = «thunk»; bootMount = «thunk»; bootspec = «thunk»; cleanTmpDir = «thunk»; consoleLogLevel = «thunk»; «43 attributes elided» }; «48 attributes elided» }
+  ```
+
+  Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
+- Type errors include the failing value [#561](https://github.com/NixOS/nix/issues/561) [#9753](https://github.com/NixOS/nix/pull/9753)
+
+  In errors like `value is an integer while a list was expected`, the message now
+  includes the failing value.
+
+  Before:
+
+  ```
+         error: value is a set while a string was expected
+  ```
+
+  After:
+
+  ```
+         error: expected a string but found a set: { ghc810 = «thunk»;
+         ghc8102Binary = «thunk»; ghc8107 = «thunk»; ghc8107Binary = «thunk»;
+         ghc865Binary = «thunk»; ghc90 = «thunk»; ghc902 = «thunk»; ghc92 = «thunk»;
+         ghc924Binary = «thunk»; ghc925 = «thunk»;  «17 attributes elided»}
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Visual clutter in `--debugger` is reduced [#9919](https://github.com/NixOS/nix/pull/9919)
+
+  Before:
+  ```
+  info: breakpoint reached
+
+
+  Starting REPL to allow you to inspect the current state of the evaluator.
+
+  Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
+
+  nix-repl> :continue
+  error: uh oh
+
+
+  Starting REPL to allow you to inspect the current state of the evaluator.
+
+  Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
+
+  nix-repl>
+  ```
+
+  After:
+
+  ```
+  info: breakpoint reached
+
+  Nix 2.20.0pre20231222_dirty debugger
+  Type :? for help.
+  nix-repl> :continue
+  error: uh oh
+
+  nix-repl>
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- REPL now supports CTRL+Z to suspend
+
+  Editline is now built with SIGTSTP support, so now typing CTRL+Z in the REPL will suspend the REPL and allow it to be resumed later or backgrounded.
+
+  Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
+- Allow single quotes in nix-shell shebangs [#8470](https://github.com/NixOS/nix/pull/8470)
+
+  Example:
+
+  ```bash
+  #! /usr/bin/env nix-shell
+  #! nix-shell -i bash --packages 'terraform.withPlugins (plugins: [ plugins.openstack ])'
+  ```
+
+  Many thanks to [ncfavier](https://github.com/ncfavier) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- reintroduce shortened `-E` form for `--expr` to new CLI [cl/605](https://gerrit.lix.systems/c/lix/+/605)
+
+  In the old CLI, it was possible to supply a shorter `-E` flag instead of fully
+  specifying `--expr` every time you wished to provide an expression that would
+  be evaluated to produce the given command's input. This was retained for the
+  `--file` flag when the new CLI utilities were written with `-f`, but `-E` was
+  dropped.
+
+  We now restore the `-E` short form for better UX. This is most useful for
+  `nix eval` but most any command that takes an Installable argument should benefit
+  from it as well.
+
+  Many thanks to [Lunaphied](https://git.lix.systems/Lunaphied) for this.
+- Source locations are printed more consistently in errors [#561](https://github.com/NixOS/nix/issues/561) [#9555](https://github.com/NixOS/nix/pull/9555)
+
+  Source location information is now included in error messages more
+  consistently. Given this code:
+
+  ```nix
+  let
+    attr = {foo = "bar";};
+    key = {};
+  in
+    attr.${key}
+  ```
+
+  Previously, Nix would show this unhelpful message when attempting to evaluate
+  it:
+
+  ```
+  error:
+         … while evaluating an attribute name
+
+         error: value is a set while a string was expected
+  ```
+
+  Now, the error message displays where the problematic value was found:
+
+  ```
+  error:
+         … while evaluating an attribute name
+
+           at bad.nix:4:11:
+
+              3|   key = {};
+              4| in attr.${key}
+               |           ^
+              5|
+
+         error: expected a string but found a set: { }
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Some stack overflow segfaults are fixed [#9616](https://github.com/NixOS/nix/issues/9616) [#9617](https://github.com/NixOS/nix/pull/9617) [cl/205](https://gerrit.lix.systems/c/lix/+/205)
+
+  The number of nested function calls has been restricted, to detect and report
+  infinite function call recursions. The default maximum call depth is 10,000 and
+  can be set with [the `max-call-depth`
+  option](@docroot@/command-ref/conf-file.md#conf-max-call-depth).
+
+  This fixes segfaults or the following unhelpful error message in many cases:
+
+      error: stack overflow (possible infinite recursion)
+
+  Before:
+
+  ```
+  $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
+  Segmentation fault: 11
+  ```
+
+  After:
+
+  ```
+  $ nix-instantiate --eval --expr '(x: x x) (x: x x)'
+  error: stack overflow
+
+         at «string»:1:14:
+              1| (x: x x) (x: x x)
+               |              ^
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Warn about ignored client settings [cl/1026](https://gerrit.lix.systems/c/lix/+/1026)
+
+  Emit a warning for every client-provided setting the daemon ignores because the requesting client is not run by a trusted user.
+  Previously this was only a debug message.
+
+  Many thanks to [jade](https://git.lix.systems/jade) for this.
+- Better error reporting for `with` expressions [#9658](https://github.com/NixOS/nix/pull/9658) [cl/207](https://gerrit.lix.systems/c/lix/+/207)
+
+  `with` expressions using non-attrset values to resolve variables are now reported with proper positions.
+
+  Previously an incorrect `with` expression would report no position at all, making it hard to determine where the error originated:
+
+  ```
+  nix-repl> with 1; a
+  error:
+         … <borked>
+
+           at «none»:0: (source not available)
+
+         error: value is an integer while a set was expected
+  ```
+
+  Now position information is preserved and reported as with most other errors:
+
+  ```
+  nix-repl> with 1; a
+  error:
+         … while evaluating the first subexpression of a with expression
+           at «string»:1:1:
+              1| with 1; a
+               | ^
+
+         error: expected a set but found an integer: 1
+  ```
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+
+## Fixes
+- Fix nested flake input `follows` [#6621](https://github.com/NixOS/nix/pull/6621) [cl/994](https://gerrit.lix.systems/c/lix/+/994)
+
+  Previously nested-input overrides were ignored; that is, the following did not
+  override anything, in spite of the `nix3-flake` manual documenting it working:
+
+  ```
+  {
+    inputs = {
+      foo.url = "github:bar/foo";
+      foo.inputs.bar.inputs.nixpkgs = "nixpkgs";
+    };
+  }
+  ```
+
+  This is useful to avoid the 1000 instances of nixpkgs problem without having
+  each flake in the dependency tree to expose all of its transitive dependencies
+  for modification.
+
+  Many thanks to [Kha](https://github.com/Kha) and [ma27](https://git.lix.systems/ma27) for this.
+- Fix CVE-2024-27297 (GHSA-2ffj-w4mj-pg37) [cl/266](https://gerrit.lix.systems/c/lix/+/266)
+
+  Since Lix fixed-output derivations run in the host network namespace (which we
+  wish to change in the future, see
+  [lix#285](https://git.lix.systems/lix-project/lix/issues/285)), they may open
+  abstract-namespace Unix sockets to each other and to programs on the host. Lix
+  contained a now-fixed time-of-check/time-of-use vulnerability where one
+  derivation could send writable handles to files in their final location in the
+  store to another over an abstract-namespace Unix socket, exit, then the other
+  derivation could wait for Lix to hash the paths and overwrite them.
+
+  The impact of this vulnerability is that two malicious fixed-output derivations
+  could create a poisoned path for the sources to Bash or similarly important
+  software containing a backdoor, leading to local privilege execution.
+
+  CppNix advisory: https://github.com/NixOS/nix/security/advisories/GHSA-2ffj-w4mj-pg37
+
+  Many thanks to [puck](https://git.lix.systems/puck), [jade](https://git.lix.systems/jade), [Théophane Hufschmitt](https://github.com/thufschmitt), [Tom Bereknyei](https://github.com/tomberek), and [Valentin Gagarin](https://github.com/fricklerhandwerk) for this.
+- `--debugger` can now access bindings from `let` expressions [#8827](https://github.com/NixOS/nix/issues/8827) [#9918](https://github.com/NixOS/nix/pull/9918)
+
+  Breakpoints and errors in the bindings of a `let` expression can now access
+  those bindings in the debugger. Previously, only the body of `let` expressions
+  could access those bindings.
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Fix handling of truncated `.drv` files. [#9673](https://github.com/NixOS/nix/pull/9673)
+
+  Previously a `.drv` that was truncated in the middle of a string would case nix to enter an infinite loop, eventually exhausting all memory and crashing.
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+- The `--debugger` will start more reliably in `let` expressions and function calls [#6649](https://github.com/NixOS/nix/issues/6649) [#9917](https://github.com/NixOS/nix/pull/9917)
+
+  Previously, if you attempted to evaluate this file with the debugger:
+
+  ```nix
+  let
+    a = builtins.trace "before inner break" (
+      builtins.break "hello"
+    );
+    b = builtins.trace "before outer break" (
+      builtins.break a
+    );
+  in
+    b
+  ```
+
+  Lix would correctly enter the debugger at `builtins.break a`, but if you asked
+  it to `:continue`, it would skip over the `builtins.break "hello"` expression
+  entirely.
+
+  Now, Lix will correctly enter the debugger at both breakpoints.
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Creating setuid/setgid binaries with fchmodat2 is now prohibited by the build sandbox [#10501](https://github.com/NixOS/nix/pull/10501)
+
+  The build sandbox blocks any attempt to create setuid/setgid binaries, but didn't check
+  for the use of the `fchmodat2` syscall which was introduced in Linux 6.6 and is used by
+  glibc >=2.39. This is fixed now.
+
+  Many thanks to [ma27](https://git.lix.systems/ma27) for this.
+- consistent order of lambda formals in printed expressions [#9874](https://github.com/NixOS/nix/pull/9874)
+
+  Always print lambda formals in lexicographic order rather than the internal, creation-time based symbol order.
+  This makes printed formals independent of the context they appear in.
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+- fix duplicate attribute error positions for `inherit` [#9874](https://github.com/NixOS/nix/pull/9874)
+
+  When an inherit caused a duplicate attribute error, the position of the error was not reported correctly, placing the error with the inherit itself or at the start of the bindings block instead of the offending attribute name.
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+- `inherit (x) ...` evaluates `x` only once [#9847](https://github.com/NixOS/nix/pull/9847)
+
+  `inherit (x) a b ...` now evaluates the expression `x` only once for all inherited attributes rather than once for each inherited attribute.
+  This does not usually have a measurable impact, but side-effects (such as `builtins.trace`) would be duplicated and expensive expressions (such as derivations) could cause a measurable slowdown.
+
+  Many thanks to [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Store paths are allowed to start with `.` [#912](https://github.com/NixOS/nix/issues/912) [#9867](https://github.com/NixOS/nix/pull/9867) [#9091](https://github.com/NixOS/nix/pull/9091) [#9095](https://github.com/NixOS/nix/pull/9095) [#9120](https://github.com/NixOS/nix/pull/9120) [#9121](https://github.com/NixOS/nix/pull/9121) [#9122](https://github.com/NixOS/nix/pull/9122) [#9130](https://github.com/NixOS/nix/pull/9130) [#9219](https://github.com/NixOS/nix/pull/9219) [#9224](https://github.com/NixOS/nix/pull/9224)
+
+  Leading periods were allowed by accident in Nix 2.4. The Nix team has considered this to be a bug, but this behavior has since been relied on by users, leading to unnecessary difficulties.
+  From now on, leading periods are officially, definitively supported. The names `.` and `..` are disallowed, as well as those starting with `.-` or `..-`.
+
+  Nix versions that denied leading periods are documented [in the issue](https://github.com/NixOS/nix/issues/912#issuecomment-1919583286).
+
+  Many thanks to [Robert Hensing](https://github.com/roberth) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- Fix `nix-env --query --drv-path --json` [#9257](https://github.com/NixOS/nix/pull/9257)
+
+  Fixed a bug where `nix-env --query` ignored `--drv-path` when `--json` was set.
+
+  Many thanks to [Artturin](https://github.com/Artturin) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- re-evaluate cached evaluation errors [cl/771](https://gerrit.lix.systems/c/lix/+/771)
+
+  "cached failure of [expr]" errors have been removed: expressions already in the
+  eval cache as a failure will now simply be re-evaluated, removing the need to
+  set `--no-eval-cache` or similar to see the error.
+
+  Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
+- Interrupting builds in the REPL works more than once [cl/1097](https://gerrit.lix.systems/c/lix/+/1097)
+
+  Builds in the REPL can be interrupted by pressing Ctrl+C.
+  Previously, this only worked once per REPL session; further attempts would be ignored.
+  This issue is now fixed, so that builds can be canceled consistently.
+
+  Many thanks to [alois31](https://git.lix.systems/alois31) for this.
+- In the debugger, `while evaluating the attribute` errors now include position information [#9915](https://github.com/NixOS/nix/pull/9915)
+
+  Before:
+
+  ```
+  0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
+  0x600001522598
+  ```
+
+  After:
+
+  ```
+  0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
+  /nix/store/hg65h51xnp74ikahns9hyf3py5mlbbqq-source/overrides/default.nix:132:27
+
+     131|
+     132|       bootstrappingBase = pkgs.${self.python.pythonAttr}.pythonForBuild.pkgs;
+        |                           ^
+     133|     in
+  ```
+
+  Many thanks to [wiggles](https://git.lix.systems/rbt) for this.
+- Include phase reporting in log file for ssh-ng builds [#9280](https://github.com/NixOS/nix/pull/9280)
+
+  Store phase information of remote builds run via `ssh-ng` remotes in the local log file, matching logging behavior of local builds.
+
+  Many thanks to [r-vdp](https://github.com/r-vdp) for this.
+- Fix `ssh-ng://` remotes not respecting `--substitute-on-destination` [#9600](https://github.com/NixOS/nix/pull/9600)
+
+  `nix copy ssh-ng://` now respects `--substitute-on-destination`, as does `nix-copy-closure` and other commands that operate on remote `ssh-ng` stores.
+  Previously this was always set by `builders-use-substitutes` setting.
+
+  Many thanks to [SharzyL](https://github.com/SharzyL) for this.
+- using `nix profile` on `/nix/var/nix/profiles/default` no longer breaks `nix upgrade-nix` [cl/952](https://gerrit.lix.systems/c/lix/+/952)
+
+  On non-NixOS, Nix is conventionally installed into a `nix-env` style profile at /nix/var/nix/profiles/default.
+  Like any `nix-env` profile, using `nix profile` on it automatically migrates it to a `nix profile` style profile, which is incompatible with `nix-env`.
+  `nix upgrade-nix` previously relied solely on `nix-env` to do the upgrade, but now will work fine with either kind of profile.
+
+  Many thanks to [Qyriad](https://git.lix.systems/Qyriad) for this.
+
+## Packaging
+- Lix turns more internal bugs into crashes [cl/797](https://gerrit.lix.systems/c/lix/+/797) [cl/626](https://gerrit.lix.systems/c/lix/+/626)
+
+  Lix now enables build options such as trapping on signed overflow and enabling
+  libstdc++ assertions by default. These may find new bugs in Lix, which will
+  present themselves as Lix processes aborting, potentially without an error
+  message.
+
+  If Lix processes abort on your machine, this is a bug. Please file a bug,
+  ideally with the core dump (or information from it).
+
+  On Linux, run `coredumpctl list`, find the crashed process's PID at
+  the bottom of the list, then run `coredumpctl info THE-PID`. You can then paste
+  the output into a bug report.
+
+  On macOS, open the Console app from Applications/Utilities, select Crash
+  Reports, select the crash report in question. Right click on it, select Open In
+  Finder, then include that file in your bug report. [See the Apple
+  documentation][apple-crashreport] for more details.
+
+  [apple-crashreport]: https://developer.apple.com/documentation/xcode/acquiring-crash-reports-and-diagnostic-logs#Locate-crash-reports-and-memory-logs-on-the-device
+
+  Many thanks to [jade](https://git.lix.systems/jade) for this.
+- Stop vendoring toml11 [cl/675](https://gerrit.lix.systems/c/lix/+/675)
+
+  We don't apply any patches to it, and vendoring it locks users into
+  bugs (it hasn't been updated since its introduction in late 2021).
+
+  Many thanks to [winter](https://git.lix.systems/winter) for this.
+- Lix is built with meson [cl/580](https://gerrit.lix.systems/c/lix/+/580) [cl/627](https://gerrit.lix.systems/c/lix/+/627) [cl/628](https://gerrit.lix.systems/c/lix/+/628) [cl/707](https://gerrit.lix.systems/c/lix/+/707) [cl/711](https://gerrit.lix.systems/c/lix/+/711) [cl/712](https://gerrit.lix.systems/c/lix/+/712) [cl/719](https://gerrit.lix.systems/c/lix/+/719)
+
+  Lix is built exclusively with the meson build system thanks to a huge team-wide
+  effort, and the legacy `make`/`autoconf` based build system has been removed
+  altogether. This improves maintainability of Lix, enables things like saving
+  20% of compile times with precompiled headers, and generally makes the build
+  less able to produce obscure incremental compilation bugs.
+
+  Non-Nix-based downstream packaging needs rewriting accordingly.
+
+  Many thanks to [Qyriad](https://git.lix.systems/Qyriad), [eldritch horrors](https://git.lix.systems/pennae), [jade](https://git.lix.systems/jade), [wiggles](https://git.lix.systems/rbt), and [winter](https://git.lix.systems/winter) for this.
+- Upstart scripts removed [cl/574](https://gerrit.lix.systems/c/lix/+/574)
+
+  Upstart scripts have been removed from Lix, since Upstart is obsolete and has
+  not been shipped by any major distributions for many years. If these are
+  necessary to your use case, please back port them to your packaging.
+
+  Many thanks to [jade](https://git.lix.systems/jade) for this.
+
+## Development
+- Clang build timing analysis [cl/587](https://gerrit.lix.systems/c/lix/+/587)
+
+  We now have Clang build profiling available, which generates Chrome
+  tracing files for each compilation unit. To enable it, run `meson configure
+  build -Dprofile-build=enabled` in a Clang stdenv (`nix develop
+  .#native-clangStdenvPackages`) then rerun the compilation.
+
+  If you want to make the build go faster, do a clang build with meson, then run
+  `maintainers/buildtime_report.sh build`, then contemplate how to improve the
+  build time.
+
+  You can also look at individual object files' traces in
+  <https://ui.perfetto.dev>.
+
+  See [the wiki page][improving-build-times-wiki] for more details on how to do
+  this.
+
+  [improving-build-times-wiki]: https://wiki.lix.systems/link/8#bkmrk-page-title
+
+## Miscellany
+- Disallow empty search regex in `nix search` [#9481](https://github.com/NixOS/nix/pull/9481)
+
+  [`nix search`](@docroot@/command-ref/new-cli/nix3-search.md) now requires a search regex to be passed. To show all packages, use `^`.
+
+  Many thanks to [iFreilicht](https://github.com/iFreilicht) and [eldritch horrors](https://git.lix.systems/pennae) for this.
+- `nix repl` history is saved more reliably [cl/1164](https://gerrit.lix.systems/c/lix/+/1164)
+
+  `nix repl` now saves its history file after each line, rather than at the end
+  of the session; ensuring that it will remember what you typed even after it
+  crashes.
+
+  Many thanks to [puck](https://git.lix.systems/puck) for this.

docker.nix

@@ -1,5 +1,8 @@
 {
   pkgs ? import <nixpkgs> { },
+  # Git commit ID, if available
+  lixRevision ? null,
+  nix2container,
   lib ? pkgs.lib,
   name ? "lix",
   tag ? "latest",
@@ -12,26 +15,51 @@
   flake-registry ? null,
 }:
 let
+  layerContents = with pkgs; [
+    # pulls in glibc and openssl, about 60MB
+    { contents = [ coreutils-full ]; }
+    # some stuff that is low in the closure graph and small ish, mostly to make
+    # incremental lix updates cheaper
+    {
+      contents = [
+        curl
+        libxml2
+        sqlite
+      ];
+    }
+    # 50MB of git
+    { contents = [ gitMinimal ]; }
+    # 144MB of nixpkgs
+    {
+      contents = [ channel ];
+      inProfile = false;
+    }
+  ];
+
+  # These packages are left to be auto layered by nix2container, since it is
+  # less critical that they get layered sensibly and they tend to not be deps
+  # of anything in particular
+  autoLayered = with pkgs; [
+    bashInteractive
+    gnutar
+    gzip
+    gnugrep
+    which
+    less
+    wget
+    man
+    cacert.out
+    findutils
+    iana-etc
+    openssh
+    nix
+  ];
+
   defaultPkgs =
-    with pkgs;
+    lib.lists.flatten (
-    [
+      map (x: if !(x ? inProfile) || x.inProfile then x.contents else [ ]) layerContents
-      nix
+    )
-      bashInteractive
+    ++ autoLayered
-      coreutils-full
-      gnutar
-      gzip
-      gnugrep
-      which
-      curl
-      less
-      wget
-      man
-      cacert.out
-      findutils
-      iana-etc
-      git
-      openssh
-    ]
     ++ extraPkgs;
 
   users =
@@ -139,16 +167,17 @@ let
     ))
     + "\n";
 
+  nixpkgs = pkgs.path;
+  channel = pkgs.runCommand "channel-nixpkgs" { } ''
+    mkdir $out
+    ${lib.optionalString bundleNixpkgs ''
+      ln -s ${nixpkgs} $out/nixpkgs
+      echo "[]" > $out/manifest.nix
+    ''}
+  '';
+
   baseSystem =
     let
-      nixpkgs = pkgs.path;
-      channel = pkgs.runCommand "channel-nixos" { inherit bundleNixpkgs; } ''
-        mkdir $out
-        if [ "$bundleNixpkgs" ]; then
-          ln -s ${nixpkgs} $out/nixpkgs
-          echo "[]" > $out/manifest.nix
-        fi
-      '';
       rootEnv = pkgs.buildPackages.buildEnv {
         name = "root-profile-env";
         paths = defaultPkgs;
@@ -187,7 +216,7 @@ let
       profile = pkgs.buildPackages.runCommand "user-environment" { } ''
         mkdir $out
         cp -a ${rootEnv}/* $out/
-        ln -s ${manifest} $out/manifest.nix
+        ln -sf ${manifest} $out/manifest.nix
       '';
       flake-registry-path =
         if (flake-registry == null) then
@@ -236,6 +265,7 @@ let
           ln -s /nix/var/nix/profiles/share $out/usr/
 
           mkdir -p $out/nix/var/nix/gcroots
+          ln -s /nix/var/nix/profiles $out/nix/var/nix/gcroots/profiles
 
           mkdir $out/tmp
 
@@ -248,14 +278,14 @@ let
           mkdir -p $out/nix/var/nix/profiles/per-user/root
 
           ln -s ${profile} $out/nix/var/nix/profiles/default-1-link
-          ln -s $out/nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default
+          ln -s /nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default
           ln -s /nix/var/nix/profiles/default $out/root/.nix-profile
 
           ln -s ${channel} $out/nix/var/nix/profiles/per-user/root/channels-1-link
-          ln -s $out/nix/var/nix/profiles/per-user/root/channels-1-link $out/nix/var/nix/profiles/per-user/root/channels
+          ln -s /nix/var/nix/profiles/per-user/root/channels-1-link $out/nix/var/nix/profiles/per-user/root/channels
 
           mkdir -p $out/root/.nix-defexpr
-          ln -s $out/nix/var/nix/profiles/per-user/root/channels $out/root/.nix-defexpr/channels
+          ln -s /nix/var/nix/profiles/per-user/root/channels $out/root/.nix-defexpr/channels
           echo "${channelURL} ${channelName}" > $out/root/.nix-channels
 
           mkdir -p $out/bin $out/usr/bin
@@ -273,43 +303,99 @@ let
           ln -s $globalFlakeRegistryPath $out/nix/var/nix/gcroots/auto/$rootName
         '')
       );
-in
-pkgs.dockerTools.buildLayeredImageWithNixDb {
 
-  inherit name tag maxLayers;
+  layers = builtins.foldl' (
+    layersList: el:
+    let
+      layer = nix2container.buildLayer {
+        deps = el.contents;
+        layers = layersList;
+      };
+    in
+    layersList ++ [ layer ]
+  ) [ ] layerContents;
 
-  contents = [ baseSystem ];
+  image = nix2container.buildImage {
 
-  extraCommands = ''
+    inherit name tag maxLayers;
-    rm -rf nix-support
-    ln -s /nix/var/nix/profiles nix/var/nix/gcroots/profiles
-  '';
-  fakeRootCommands = ''
-    chmod 1777 tmp
-    chmod 1777 var/tmp
-  '';
 
-  config = {
+    inherit layers;
-    Cmd = [ "/root/.nix-profile/bin/bash" ];
+
-    Env = [
+    copyToRoot = [ baseSystem ];
-      "USER=root"
+
-      "PATH=${
+    initializeNixDatabase = true;
-        lib.concatStringsSep ":" [
+
-          "/root/.nix-profile/bin"
+    perms = [
-          "/nix/var/nix/profiles/default/bin"
+      {
-          "/nix/var/nix/profiles/default/sbin"
+        path = baseSystem;
-        ]
+        regex = "(/var)?/tmp";
-      }"
+        mode = "1777";
-      "MANPATH=${
+      }
-        lib.concatStringsSep ":" [
-          "/root/.nix-profile/share/man"
-          "/nix/var/nix/profiles/default/share/man"
-        ]
-      }"
-      "SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
-      "GIT_SSL_CAINFO=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
-      "NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
-      "NIX_PATH=/nix/var/nix/profiles/per-user/root/channels:/root/.nix-defexpr/channels"
     ];
+
+    config = {
+      Cmd = [ "/root/.nix-profile/bin/bash" ];
+      Env = [
+        "USER=root"
+        "PATH=${
+          lib.concatStringsSep ":" [
+            "/root/.nix-profile/bin"
+            "/nix/var/nix/profiles/default/bin"
+            "/nix/var/nix/profiles/default/sbin"
+          ]
+        }"
+        "MANPATH=${
+          lib.concatStringsSep ":" [
+            "/root/.nix-profile/share/man"
+            "/nix/var/nix/profiles/default/share/man"
+          ]
+        }"
+        "SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
+        "GIT_SSL_CAINFO=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
+        "NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
+        "NIX_PATH=/nix/var/nix/profiles/per-user/root/channels:/root/.nix-defexpr/channels"
+      ];
+
+      Labels = {
+        "org.opencontainers.image.title" = "Lix";
+        "org.opencontainers.image.source" = "https://git.lix.systems/lix-project/lix";
+        "org.opencontainers.image.vendor" = "Lix project";
+        "org.opencontainers.image.version" = pkgs.nix.version;
+        "org.opencontainers.image.description" = "Minimal Lix container image, with some batteries included.";
+      } // lib.optionalAttrs (lixRevision != null) { "org.opencontainers.image.revision" = lixRevision; };
+    };
+
+    meta = {
+      description = "Docker image for Lix. This is built with nix2container; see that project's README for details";
+      longDescription = ''
+        Docker image for Lix, built with nix2container.
+        To copy it to your docker daemon, nix run .#dockerImage.copyToDockerDaemon
+        To copy it to podman, nix run .#dockerImage.copyTo containers-storage:lix
+      '';
+    };
   };
+in
+image
+// {
+  # We don't ship the tarball as the default output because it is a strange thing to want imo
+  tarball =
+    pkgs.buildPackages.runCommand "docker-image-tarball-${pkgs.nix.version}"
+      {
+        nativeBuildInputs = [ pkgs.buildPackages.bubblewrap ];
+        meta.description = "Docker image tarball with Lix for ${pkgs.system}";
+      }
+      ''
+        mkdir -p $out/nix-support
+        image=$out/image.tar
+        # bwrap for foolish temp dir selection code that forces /var/tmp:
+        # https://github.com/containers/skopeo.git/blob/60ee543f7f7c242f46cc3a7541d9ac8ab1c89168/vendor/github.com/containers/image/v5/internal/tmpdir/tmpdir.go#L15-L18
+        mkdir -p $TMPDIR/fake-var/tmp
+        args=(--unshare-user --bind "$TMPDIR/fake-var" /var)
+        for dir in /*; do
+          args+=(--dev-bind "/$dir" "/$dir")
+        done
+        bwrap ''${args[@]} -- ${lib.getExe image.copyTo} docker-archive:$image
+        gzip $image
+        echo "file binary-dist $image" >> $out/nix-support/hydra-build-products
+      '';
 }

flake.lock

@@ -16,18 +16,34 @@
         "type": "github"
       }
     },
+    "nix2container": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1712990762,
+        "narHash": "sha256-hO9W3w7NcnYeX8u8cleHiSpK2YJo7ecarFTUlbybl7k=",
+        "owner": "nlewo",
+        "repo": "nix2container",
+        "rev": "20aad300c925639d5d6cbe30013c8357ce9f2a2e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nlewo",
+        "repo": "nix2container",
+        "type": "github"
+      }
+    },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1715123187,
+        "lastModified": 1718111384,
-        "narHash": "sha256-0czuu757t53lK6uWeo1a5/jJbCd9t4sOtLDFpts60DM=",
+        "narHash": "sha256-7tSst0S5FOmcgvNtfy6cjZX5w8CabCVAfAeCkhY4OVg=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "0c592f9a288bdf764b6f24c757277c0e49757a46",
+        "rev": "a508a44af0c1b1b57785c34d8b54783536273eeb",
         "type": "github"
       },
       "original": {
         "owner": "NixOS",
-        "ref": "nixos-23.11-small",
+        "ref": "nixos-24.05-small",
         "repo": "nixpkgs",
         "type": "github"
       }
@@ -67,6 +83,7 @@
     "root": {
       "inputs": {
         "flake-compat": "flake-compat",
+        "nix2container": "nix2container",
         "nixpkgs": "nixpkgs",
         "nixpkgs-regression": "nixpkgs-regression",
         "pre-commit-hooks": "pre-commit-hooks"

flake.nix

@@ -2,12 +2,16 @@
   description = "The purely functional package manager";
 
   inputs = {
-    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11-small";
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05-small";
     nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
     pre-commit-hooks = {
       url = "github:cachix/git-hooks.nix";
       flake = false;
     };
+    nix2container = {
+      url = "github:nlewo/nix2container";
+      flake = false;
+    };
     flake-compat = {
       url = "github:edolstra/flake-compat";
       flake = false;
@@ -20,6 +24,7 @@
       nixpkgs,
       nixpkgs-regression,
       pre-commit-hooks,
+      nix2container,
       flake-compat,
     }:
 
@@ -82,10 +87,11 @@
       crossSystems = [
         "armv6l-linux"
         "armv7l-linux"
-        # FIXME: doesn't evaluate, plausibly fixed in >=24.05, so recheck when
+        "riscv64-linux"
-        # we update to 24.05
+        # FIXME: still broken in 24.05: fails to build rustc(??) due to missing -lstdc++ dep
-        # "x86_64-freebsd13"
+        # "x86_64-freebsd"
-        "x86_64-netbsd"
+        # FIXME: broken dev shell due to python
+        # "x86_64-netbsd"
       ];
 
       stdenvs = [
@@ -130,13 +136,11 @@
                   {
                     system = crossSystem;
                   }
-                  // lib.optionalAttrs (crossSystem == "x86_64-freebsd13") { useLLVM = true; };
+                  // lib.optionalAttrs (crossSystem == "x86_64-freebsd") { useLLVM = true; };
               overlays = [
                 (overlayFor (p: p.${stdenv}))
                 (final: prev: { nixfmt = final.callPackage ./nix-support/nixfmt.nix { }; })
               ];
-
-              config.permittedInsecurePackages = [ "nix-2.13.6" ];
             };
           stdenvs = forAllStdenvs (make-pkgs null);
           native = stdenvs.stdenvPackages;
@@ -148,8 +152,6 @@
         }
       );
 
-      binaryTarball = nix: pkgs: pkgs.callPackage ./nix-support/binary-tarball.nix { inherit nix; };
-
       overlayFor =
         getStdenv: final: prev:
         let
@@ -162,7 +164,6 @@
           nixUnstable = prev.nixUnstable;
 
           check-headers = final.buildPackages.callPackage ./maintainers/check-headers.nix { };
-          clangbuildanalyzer = final.buildPackages.callPackage ./misc/clangbuildanalyzer.nix { };
 
           default-busybox-sandbox-shell = final.busybox.override {
             useMusl = true;
@@ -210,6 +211,11 @@
         # Binary package for various platforms.
         build = forAllSystems (system: self.packages.${system}.nix);
 
+        devShell = forAllSystems (system: {
+          default = self.devShells.${system}.default;
+          clang = self.devShells.${system}.native-clangStdenvPackages;
+        });
+
         rl-next = forAllSystems (
           system:
           let
@@ -227,14 +233,17 @@
           }
         );
 
+        # Completion tests for the Nix REPL.
+        repl-completion = forAllSystems (
+          system: nixpkgsFor.${system}.native.callPackage ./tests/repl-completion.nix { }
+        );
+
         # Perl bindings for various platforms.
-        perlBindings = forAllSystems (system: nixpkgsFor.${system}.native.nix.perl-bindings);
+        perlBindings = forAllSystems (system: nixpkgsFor.${system}.native.nix.passthru.perl-bindings);
 
         # Binary tarball for various platforms, containing a Nix store
         # with the closure of 'nix' package.
-        binaryTarball = forAllSystems (
+        binaryTarball = forAllSystems (system: nixpkgsFor.${system}.native.nix.passthru.binaryTarball);
-          system: binaryTarball nixpkgsFor.${system}.native.nix nixpkgsFor.${system}.native
-        );
 
         # docker image with Lix inside
         dockerImage = lib.genAttrs linux64BitSystems (system: self.packages.${system}.dockerImage);
@@ -277,9 +286,20 @@
 
           nixpkgsLibTests = forAllSystems (
             system:
-            import (nixpkgs + "/lib/tests/release.nix") {
+            let
+              inherit (self.packages.${system}) nix;
               pkgs = nixpkgsFor.${system}.native;
-              nixVersions = [ self.packages.${system}.nix ];
+              testWithNix = import (nixpkgs + "/lib/tests/test-with-nix.nix") { inherit pkgs lib nix; };
+            in
+            pkgs.symlinkJoin {
+              name = "nixpkgs-lib-tests";
+              paths =
+                [ testWithNix ]
+                # FIXME: This is disabled on darwin due to a nixpkgs bug https://github.com/NixOS/nixpkgs/issues/319147
+                # After that is fixed, it should be restored to use lib/tests/release.nix as before, rather than this reimplementation.
+                ++ lib.optionals pkgs.stdenv.isLinux [
+                  (import (nixpkgs + "/pkgs/test/release") { inherit pkgs lib nix; })
+                ];
             }
           );
         };
@@ -306,12 +326,16 @@
       checks = forAvailableSystems (
         system:
         {
+          # devShells and packages already get checked by nix flake check, so
+          # this is just jobs that are special
+
           binaryTarball = self.hydraJobs.binaryTarball.${system};
           perlBindings = self.hydraJobs.perlBindings.${system};
           nixpkgsLibTests = self.hydraJobs.tests.nixpkgsLibTests.${system};
           rl-next = self.hydraJobs.rl-next.${system}.user;
           # Will be empty attr set on i686-linux, and filtered out by forAvailableSystems.
           pre-commit = self.hydraJobs.pre-commit.${system};
+          repl-completion = self.hydraJobs.repl-completion.${system};
         }
         // (lib.optionalAttrs (builtins.elem system linux64BitSystems)) {
           dockerImage = self.hydraJobs.dockerImage.${system};
@@ -330,19 +354,13 @@
             dockerImage =
               let
                 pkgs = nixpkgsFor.${system}.native;
-                image = import ./docker.nix {
+                nix2container' = import nix2container { inherit pkgs system; };
-                  inherit pkgs;
-                  tag = pkgs.nix.version;
-                };
               in
-              pkgs.runCommand "docker-image-tarball-${pkgs.nix.version}"
+              import ./docker.nix {
-                { meta.description = "Docker image with Lix for ${system}"; }
+                inherit pkgs;
-                ''
+                nix2container = nix2container'.nix2container;
-                  mkdir -p $out/nix-support
+                tag = pkgs.nix.version;
-                  image=$out/image.tar.gz
+              };
-                  ln -s ${image} $image
-                  echo "file binary-dist $image" >> $out/nix-support/hydra-build-products
-                '';
           }
           // builtins.listToAttrs (
             map (crossSystem: {

maintainers/release-notes

@@ -1,179 +0,0 @@
-#!/usr/bin/env nix-shell
-#!nix-shell -i bash ../shell.nix -I nixpkgs=channel:nixos-unstable-small
-#                                   ^^^^^^^
-#                Only used for bash. shell.nix goes to the flake.
-
-# --- CONFIGURATION ---
-
-# This does double duty for
-# - including rl-next
-# - marking where to insert new links (right after)
-SUMMARY_MARKER_LINE='release-notes/rl-next.md'
-
-# --- LIB ---
-
-log() {
-  echo 1>&2 "release-notes:" "$@"
-}
-logcmd() {
-  local cmd="$1"
-  shift
-  logcmd2 "$cmd" "${*@Q}" "$cmd" "$@"
-}
-logcmd2() {
-  local fakecmd="$1"
-  local fakeargs="$2"
-  shift
-  shift
-  printf 1>&2 "release-notes: \033[34;1m$fakecmd\033[0m "
-  echo "$fakeargs" 1>&2
-  "$@"
-}
-die() {
-  # ANSI red
-  printf 1>&2 "release-notes: \033[31;1merror:\033[0m"
-  echo 1>&2 "" "$@"
-  exit 1
-}
-confirm() {
-  local answer
-  echo 1>&2 "$@" "[y/n]"
-  read -r answer
-  case "$answer" in
-    y|Y|yes|Yes|YES)
-      return 0
-      ;;
-    n|N|no|No|NO)
-      return 1
-      ;;
-    *)
-      echo 1>&2 "please answer y or n"
-      confirm "$@"
-      ;;
-  esac
-}
-report_done() {
-  logcmd2 "git" "show" git -c pager.show=false show
-  printf 1>&2 "release-notes: \033[32;1mdone\033[0m\n"
-}
-
-# --- PARSE ARGS ---
-
-if [[ $# -gt 0 ]]; then
-  die "Release notes takes no arguments, but make sure to set VERSION."
-fi
-
-# --- CHECKS ---
-
-if [[ ! -e flake.nix ]] || [[ ! -e .git ]]; then
-  die "must run in repo root"
-  exit 1
-fi
-
-# repo must be clean
-if ! git diff --quiet; then
-  die "repo is dirty, please commit or stash changes"
-fi
-
-if ! git diff --quiet --cached; then
-  die "repo has staged changes, please commit or stash them"
-fi
-
-if ! grep -F "$SUMMARY_MARKER_LINE" doc/manual/src/SUMMARY.md >/dev/null; then
-  # would have been nice to catch this early, but won't be worth the extra infra
-  die "SUMMARY.md is missing the marker line '$SUMMARY_MARKER_LINE', which would be used for inserting a new release notes page. Please fix the script."
-fi
-
-if [[ ! -n "${VERSION:-}" ]]; then
-  die "please set the VERSION environment variable before invoking this script"
-  exit 1
-fi
-
-# version_major_minor:      MAJOR.MINOR
-# version_full:             MAJOR.MINOR.PATCH
-# IS_PATCH:                 true if this is a patch release; append instead of create
-if grep -E '^[0-9]+\.[0-9]+$' <<< "$VERSION" >/dev/null; then
-  log 'is minor'
-  IS_PATCH=false
-  version_full="$VERSION.0"
-  version_major_minor="$VERSION"
-elif grep -E '^[0-9]+\.[0-9]+\.0$' <<< "$VERSION" >/dev/null; then
-  log 'is minor (.0)'
-  IS_PATCH=false
-  version_full="$VERSION"
-  version_major_minor="$(echo "$VERSION" | sed -e 's/\.0$//')"
-elif grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' <<< "$VERSION" >/dev/null; then
-  log 'is patch'
-  IS_PATCH=true
-  version_full="$VERSION"
-  version_major_minor="$(echo "$VERSION" | sed -e 's/\.[0-9]*$//')"
-else
-  die "VERSION must be MAJOR.MINOR[.PATCH], where each is a number, e.g. 2.20 or 2.20.1 (VERSION was set to $VERSION)"
-fi
-
-unset VERSION
-
-log "version_major_minor=$version_major_minor"
-log "version_full=$version_full"
-log "IS_PATCH=$IS_PATCH"
-
-basename=rl-${version_major_minor}.md
-file=doc/manual/src/release-notes/$basename
-
-if ! $IS_PATCH; then
-  if [[ -e $file ]]; then
-    die "release notes file $file already exists. If you'd like to make a minor release, pass a patch version, e.g. 2.20.1"
-  fi
-fi
-
-# --- DEFAULTS ---
-
-if [[ ! -n "${DATE:-}" ]]; then
-  DATE="$(date +%Y-%m-%d)"
-  log "DATE not set, using $DATE"
-fi
-
-case "$DATE" in
-  [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9])
-    ;;
-  *)
-    die "DATE must be YYYY-MM-DD, e.g. 2021-12-31 (DATE was set to $DATE)"
-    ;;
-esac
-
-# --- DO THE WORK ---
-
-# menu
-title="Release $version_major_minor ($DATE)"
-# section on page
-section_title="Release $version_full ($DATE)"
-
-(
-  # TODO add minor number, and append?
-  echo "# $section_title"
-  echo
-  build-release-notes --change-authors doc/manual/change-authors.yml doc/manual/rl-next
-) | tee -a $file
-
-log "Wrote $file"
-
-if ! $IS_PATCH; then
-  NEW_SUMMARY_LINE="  - [$title](release-notes/$basename)"
-
-  # find the marker line, insert new link after it
-  escaped_marker="$(echo "$SUMMARY_MARKER_LINE" | sed -e 's/\//\\\//g' -e 's/ /\\ /g')"
-  escaped_line="$(echo "$NEW_SUMMARY_LINE" | sed -e 's/\//\\\//g' -e 's/ /\\ /g')"
-  logcmd sed -i -e "/$escaped_marker/a $escaped_line" doc/manual/src/SUMMARY.md
-fi
-
-for f in doc/manual/rl-next/*.md; do
-  if [[ config != "$(basename $f)" ]]; then
-    logcmd git rm $f
-  fi
-done
-
-logcmd git add $file doc/manual/src/SUMMARY.md
-logcmd git status
-logcmd git commit -m "release notes: $version_full"
-
-report_done

meson.build

@@ -129,6 +129,20 @@ endif
 
 cxx = meson.get_compiler('cpp')
 
+
+# clangd breaks when GCC is using precompiled headers lmao
+# https://git.lix.systems/lix-project/lix/issues/374
+should_pch = get_option('enable-pch-std')
+summary('PCH C++ stdlib', should_pch, bool_yn : true)
+if should_pch
+  # Unlike basically everything else that takes a file, Meson requires the arguments to
+  # cpp_pch : to be strings and doesn't accept files(). So absolute path it is.
+  cpp_pch = [meson.project_source_root() / 'src/pch/precompiled-headers.hh']
+else
+  cpp_pch = []
+endif
+
+
 # Translate some historical and Mesony CPU names to Lixy CPU names.
 # FIXME(Qyriad): the 32-bit x86 code is not tested right now, because cross compilation for Lix
 # to those architectures is currently broken for other reasons, namely:
@@ -153,6 +167,7 @@ message('canonical Nix system name:', host_system)
 
 is_linux = host_machine.system() == 'linux'
 is_darwin = host_machine.system() == 'darwin'
+is_freebsd = host_machine.system() == 'freebsd'
 is_x64 = host_machine.cpu_family() == 'x86_64'
 
 # Per-platform arguments that you should probably pass to shared_module() invocations.
@@ -168,23 +183,18 @@ elif is_linux
   # Clang sanitizers on Linux.
   # FIXME(Qyriad): is that true?
 endif
-deps = [ ]
 configdata = { }
 
 #
 # Dependencies
 #
 
-boehm = dependency('bdw-gc', required : get_option('gc'))
+boehm = dependency('bdw-gc', required : get_option('gc'), version : '>=8.2.6')
-if boehm.found()
-  deps += boehm
-endif
 configdata += {
   'HAVE_BOEHMGC': boehm.found().to_int(),
 }
 
 boost = dependency('boost', required : true, modules : ['context', 'coroutine', 'container'])
-deps += boost
 
 # cpuid only makes sense on x86_64
 cpuid_required = is_x64 ? get_option('cpuid') : false
@@ -192,7 +202,6 @@ cpuid = dependency('libcpuid', 'cpuid', required : cpuid_required)
 configdata += {
   'HAVE_LIBCPUID': cpuid.found().to_int(),
 }
-deps += cpuid
 
 # seccomp only makes sense on Linux
 seccomp_required = is_linux ? get_option('seccomp-sandboxing') : false
@@ -205,19 +214,17 @@ configdata += {
 }
 
 libarchive = dependency('libarchive', required : true)
-deps += libarchive
 
 brotli = [
   dependency('libbrotlicommon', required : true),
   dependency('libbrotlidec', required : true),
   dependency('libbrotlienc', required : true),
 ]
-deps += brotli
 
 openssl = dependency('libcrypto', 'openssl', required : true)
-deps += openssl
 
-aws_sdk = dependency('aws-cpp-sdk-core', required : false)
+# FIXME: confirm we actually support such old versions of aws-sdk-cpp
+aws_sdk = dependency('aws-cpp-sdk-core', required : false, version : '>=1.8')
 aws_sdk_transfer = dependency('aws-cpp-sdk-transfer', required : aws_sdk.found(), fallback : ['aws_sdk', 'aws_cpp_sdk_transfer_dep'])
 if aws_sdk.found()
   # The AWS pkg-config adds -std=c++11.
@@ -229,13 +236,6 @@ if aws_sdk.found()
     links : true,
     sources : true,
   )
-  deps += aws_sdk
-  s = aws_sdk.version().split('.')
-  configdata += {
-    'AWS_VERSION_MAJOR': s[0].to_int(),
-    'AWS_VERSION_MINOR': s[1].to_int(),
-    'AWS_VERSION_PATCH': s[2].to_int(),
-  }
   aws_sdk_transfer = aws_sdk_transfer.partial_dependency(
     compile_args : false,
     includes : true,
@@ -256,7 +256,6 @@ if aws_s3.found()
     links : true,
     sources : true,
   )
-  deps += aws_s3
 endif
 
 configdata += {
@@ -264,26 +263,20 @@ configdata += {
 }
 
 sqlite = dependency('sqlite3', 'sqlite', version : '>=3.6.19', required : true)
-deps += sqlite
 
 sodium = dependency('libsodium', 'sodium', required : true)
-deps += sodium
 
 curl = dependency('libcurl', 'curl', required : true)
-deps += curl
 
 editline = dependency('libeditline', 'editline', version : '>=1.14', required : true)
-deps += editline
 
 lowdown = dependency('lowdown', version : '>=0.9.0', required : true)
-deps += lowdown
 
 # HACK(Qyriad): rapidcheck's pkg-config doesn't include the libs lol
 # Note: technically we 'check' for rapidcheck twice, for the internal-api-docs handling above,
 # but Meson will cache the result of the first one, and the required : arguments are different.
 rapidcheck_meson = dependency('rapidcheck', required : enable_tests)
 rapidcheck = declare_dependency(dependencies : rapidcheck_meson, link_args : ['-lrapidcheck'])
-deps += rapidcheck
 
 gtest = [
   dependency('gtest', required : enable_tests),
@@ -291,13 +284,10 @@ gtest = [
   dependency('gmock', required : enable_tests),
   dependency('gmock_main', required : enable_tests),
 ]
-deps += gtest
 
 toml11 = dependency('toml11', version : '>=3.7.0', required : true, method : 'cmake')
-deps += toml11
 
 nlohmann_json = dependency('nlohmann_json', required : true)
-deps += nlohmann_json
 
 # lix-doc is a Rust project provided via buildInputs and unfortunately doesn't have any way to be detected.
 # Just declare it manually to resolve this.
@@ -305,7 +295,6 @@ deps += nlohmann_json
 # FIXME: build this with meson in the future after we drop Make (with which we
 # *absolutely* are not going to make it work)
 lix_doc = declare_dependency(link_args : [ '-llix_doc' ])
-deps += lix_doc
 
 #
 # Build-time tools
@@ -469,6 +458,12 @@ if cxx.get_linker_id() in ['ld.bfd', 'ld.gold']
   add_project_link_arguments('-Wl,--no-copy-dt-needed-entries', language : 'cpp')
 endif
 
+if is_freebsd
+  # FreeBSD's `environ` is defined in `crt1.o`, not `libc.so`,
+  # so the linker thinks it's undefined
+  add_project_link_arguments('-Wl,-z,undefs', language: 'cpp')
+endif
+
 # Generate Chromium tracing files for each compiled file, which enables
 # maintainers/buildtime_report.sh BUILD-DIR to simply work in clang builds.
 #

meson.options

@@ -64,3 +64,7 @@ option('internal-api-docs', type : 'feature', value : 'auto',
 option('profile-dir', type : 'string', value : 'etc/profile.d',
   description : 'the path to install shell profile files',
 )
+
+option('enable-pch-std', type : 'boolean', value : true,
+  description : 'whether to use precompiled headers for C++\'s standard library (breaks clangd if you\'re using GCC)',
+)

misc/clangbuildanalyzer.nix

@@ -1,33 +0,0 @@
-# Upstreaming here, can be deleted once it's upstreamed:
-# https://github.com/NixOS/nixpkgs/pull/297102
-{
-  stdenv,
-  lib,
-  cmake,
-  fetchFromGitHub,
-}:
-stdenv.mkDerivation (finalAttrs: {
-  pname = "clangbuildanalyzer";
-  version = "1.5.0";
-
-  src = fetchFromGitHub {
-    owner = "aras-p";
-    repo = "ClangBuildAnalyzer";
-    rev = "v${finalAttrs.version}";
-    sha256 = "sha256-kmgdk634zM0W0OoRoP/RzepArSipa5bNqdVgdZO9gxo=";
-  };
-
-  nativeBuildInputs = [ cmake ];
-
-  meta = {
-    description = "Tool for analyzing Clang's -ftrace-time files";
-    homepage = "https://github.com/aras-p/ClangBuildAnalyzer";
-    maintainers = with lib.maintainers; [ lf- ];
-    license = lib.licenses.unlicense;
-    platforms = lib.platforms.unix;
-    # `long long int` != `size_t`
-    # There's no convenient lib.platforms.32bit or anything, but it's easy enough to do ourselves.
-    badPlatforms = lib.filter (plat: (lib.systems.elaborate plat).is32bit) lib.platforms.all;
-    mainProgram = "ClangBuildAnalyzer";
-  };
-})

package.nix

@@ -39,12 +39,10 @@
   pkg-config,
   python3,
   rapidcheck,
-  skopeo,
   sqlite,
   toml11,
   util-linuxMinimal ? utillinuxMinimal,
   utillinuxMinimal ? null,
-  xonsh-unwrapped,
   xz,
 
   busybox-sandbox-shell,
@@ -71,8 +69,6 @@
         # `boehmgc-nix` then this will almost certainly have duplicate patches, which means
         # the patches won't apply and we'll get a build failure.
         ./boehmgc-coroutine-sp-fallback.diff
-        # https://github.com/ivmai/bdwgc/pull/586
-        ./boehmgc-traceable_allocator-public.diff
       ];
     };
 
@@ -380,8 +376,6 @@ stdenv.mkDerivation (finalAttrs: {
     platforms = lib.platforms.unix;
   };
 
-  passthru.perl-bindings = pkgs.callPackage ./perl { inherit fileset stdenv; };
-
   # Export the patched version of boehmgc.
   # flake.nix exports that into its overlay.
   passthru = {
@@ -394,13 +388,19 @@ stdenv.mkDerivation (finalAttrs: {
     mkDevShell =
       {
         mkShell,
-        just,
+
-        nixfmt,
+        bashInteractive,
-        glibcLocales,
-        pre-commit-checks,
         clang-tools,
-        llvmPackages,
         clangbuildanalyzer,
+        glibcLocales,
+        just,
+        llvmPackages,
+        nixfmt,
+        skopeo,
+        xonsh,
+
+        # Lix specific packages
+        pre-commit-checks,
         contribNotice,
       }:
       let
@@ -408,20 +408,16 @@ stdenv.mkDerivation (finalAttrs: {
           # Required to make non-NixOS Linux not complain about missing locale files during configure in a dev shell
           LOCALE_ARCHIVE = "${lib.getLib pkgs.glibcLocales}/lib/locale/locale-archive";
         };
-        # for some reason that seems accidental and was changed in
-        # NixOS 24.05-pre, clang-tools is pinned to LLVM 14 when
-        # default LLVM is newer.
-        clang-tools_llvm = clang-tools.override { inherit llvmPackages; };
 
         pythonPackages = (
           p: [
             p.yapf
             p.python-frontmatter
-            (p.toPythonModule xonsh-unwrapped)
+            p.requests
+            p.xdg-base-dirs
+            (p.toPythonModule xonsh.passthru.unwrapped)
           ]
         );
-        # FIXME: This will explode when we switch to 24.05 if we don't backport
-        # https://github.com/NixOS/nixpkgs/pull/317636 first
         pythonEnv = python3.withPackages pythonPackages;
 
         # pkgs.mkShell uses pkgs.stdenv by default, regardless of inputsFrom.
@@ -440,13 +436,22 @@ stdenv.mkDerivation (finalAttrs: {
           # For Meson to find Boost.
           env = finalAttrs.env;
 
-          # I guess this is necessary because mesonFlags to mkDerivation doesn't propagate in inputsFrom,
+          mesonFlags =
-          # which only propagates stuff set in hooks? idk.
+            # I guess this is necessary because mesonFlags to mkDerivation doesn't propagate in inputsFrom,
-          inherit (finalAttrs) mesonFlags;
+            # which only propagates stuff set in hooks? idk.
+            finalAttrs.mesonFlags
+            # Clangd breaks when GCC is using precompiled headers, so for the devshell specifically
+            # we make precompiled C++ stdlib conditional on using Clang.
+            # https://git.lix.systems/lix-project/lix/issues/374
+            ++ [ (lib.mesonBool "enable-pch-std" stdenv.cc.isClang) ];
 
           packages =
-            lib.optional (stdenv.cc.isClang && hostPlatform == buildPlatform) clang-tools_llvm
+            lib.optional (stdenv.cc.isClang && hostPlatform == buildPlatform) clang-tools
             ++ [
+              # Why are we providing a bashInteractive? Well, when you run
+              # `bash` from inside `nix develop`, say, because you are using it
+              # via direnv, you will by default get bash (unusable edition).
+              bashInteractive
               pythonEnv
               # docker image tool
               skopeo
@@ -502,5 +507,11 @@ stdenv.mkDerivation (finalAttrs: {
           '';
         }
       );
+
+    perl-bindings = pkgs.callPackage ./perl { inherit fileset stdenv; };
+
+    binaryTarball = pkgs.callPackage ./nix-support/binary-tarball.nix {
+      nix = finalAttrs.finalPackage;
+    };
   };
 })

releng/__init__.py

@@ -1,13 +1,72 @@
 from xonsh.main import setup
+
 setup()
 del setup
 
-from releng import environment
+import logging
-from releng import create_release
+import sys
-from releng import keys
+
-from releng import version
+import xonsh.base_shell
-from releng import cli
+
-from releng import docker
+from . import environment
+from . import create_release
+from . import keys
+from . import version
+from . import cli
+from . import docker
+from . import docker_assemble
+from . import release_notes
+from . import gitutils
+
+
+def setup_logging():
+    """
+    Sets up logging to work properly. The following are intended to work:
+    - ipython/xonsh configuration files adding log handlers out of band
+    - Reloading the module in xonsh/ipython not causing Bonus Loggers (which is
+      why we check if there is already a handler. This also helps the previous
+      case)
+    - Importing the releng module from xonsh and poking at it interactively
+    """
+
+    LEVELS = {
+        # Root logger must be DEBUG so that anything else can be DEBUG
+        None: logging.DEBUG,
+        # Everything in releng
+        __name__: logging.DEBUG,
+        # Log spam caused by prompt_toolkit
+        'asyncio': logging.INFO,
+    }
+
+    for name, level in LEVELS.items():
+        logger = logging.getLogger(name)
+        logger.setLevel(level)
+
+    root_logger = logging.getLogger()
+
+    fmt = logging.Formatter('{asctime} {levelname} {name}: {message}',
+                            datefmt='%b %d %H:%M:%S',
+                            style='{')
+
+    if not any(
+            isinstance(h, logging.StreamHandler) for h in root_logger.handlers):
+        stderr = sys.stderr
+        # XXX: Horrible hack required by the virtual stderr xonsh uses for each entered
+        # command getting closed after the command is run: we need to pull out
+        # the real stderr because this survives across multiple command runs.
+        #
+        # This only applies when running xonsh in interactive mode and importing releng.
+        if isinstance(sys.stderr, xonsh.base_shell._TeeStd):
+            stderr = stderr.std  # type: ignore
+
+        hand = logging.StreamHandler(stream=stderr)
+        hand.set_name('releng root handler')
+        hand.setFormatter(fmt)
+        root_logger.addHandler(hand)
+
+
+setup_logging()
+
 
 def reload():
     import importlib
@@ -17,3 +76,6 @@ def reload():
     importlib.reload(version)
     importlib.reload(cli)
     importlib.reload(docker)
+    importlib.reload(docker_assemble)
+    importlib.reload(gitutils)
+    importlib.reload(release_notes)

releng/cli.py

@@ -2,16 +2,16 @@ from . import create_release
 from . import docker
 from .environment import RelengEnvironment
 from . import environment
-import functools
 import argparse
 import sys
 
 
 def do_build(args):
     if args.target == 'all':
-        create_release.build_artifacts(no_check_git=args.no_check_git)
+        create_release.build_artifacts(args.profile, no_check_git=args.no_check_git)
     elif args.target == 'manual':
-        eval_result = create_release.eval_jobs()
+        # n.b. args.profile does nothing here, you will just get the x86_64-linux manual no matter what.
+        eval_result = create_release.eval_jobs(args.profile)
         create_release.build_manual(eval_result)
     else:
         raise ValueError('invalid target, unreachable')
@@ -28,7 +28,8 @@ def do_upload(env: RelengEnvironment, args):
         docker.check_all_logins(env)
         create_release.upload_artifacts(env,
                                         force_push_tag=args.force_push_tag,
-                                        noconfirm=args.noconfirm)
+                                        noconfirm=args.noconfirm,
+                                        no_check_git=args.no_check_git)
     elif args.target == 'manual':
         create_release.upload_manual(env)
     else:
@@ -79,10 +80,18 @@ def main():
     build.add_argument('--target',
                        choices=['manual', 'all'],
                        help='Whether to build everything or just the manual')
+    build.add_argument('--profile',
+                       default='all',
+                       choices=('all', 'x86_64-linux-only'),
+                       help='Which systems to build targets for.')
     build.set_defaults(cmd=do_build)
 
     upload = sps.add_parser(
         'upload', help='Upload artifacts to cache and releases bucket')
+    upload.add_argument(
+        '--no-check-git',
+        action='store_true',
+        help="Don't check git state before uploading. For testing.")
     upload.add_argument('--force-push-tag',
                         action='store_true',
                         help='Force push the tag. For testing.')

releng/create_release.xsh

@@ -6,10 +6,14 @@ from pathlib import Path
 import tempfile
 import hashlib
 import datetime
+
 from . import environment
 from .environment import RelengEnvironment
 from . import keys
+from . import docker
 from .version import VERSION, RELEASE_NAME, MAJOR
+from .gitutils import verify_are_on_tag, git_preconditions
+from . import release_notes
 
 $RAISE_SUBPROC_ERROR = True
 $XONSH_SHOW_TRACEBACK = True
@@ -25,9 +29,6 @@ RELENG_MSG = "Release created with releng/create_release.xsh"
 BUILD_CORES = 16
 MAX_JOBS = 2
 
-# TODO
-RELEASE_SYSTEMS = ["x86_64-linux"]
-
 
 def setup_creds(env: RelengEnvironment):
     key = keys.get_ephemeral_key(env)
@@ -37,15 +38,6 @@ def setup_creds(env: RelengEnvironment):
     $AWS_ENDPOINT_URL = environment.S3_ENDPOINT
 
 
-def git_preconditions():
-    # verify there is nothing in index ready to stage
-    proc = !(git diff-index --quiet --cached HEAD --)
-    assert proc.rtn == 0
-    # verify there is nothing *stageable* and tracked
-    proc = !(git diff-files --quiet)
-    assert proc.rtn == 0
-
-
 def official_release_commit_tag(force_tag=False):
     print('[+] Setting officialRelease in flake.nix and tagging')
     prev_branch = $(git symbolic-ref --short HEAD).strip()
@@ -89,11 +81,9 @@ def realise(paths: list[str]):
     nix-store @(args) @(paths)
 
 
-def eval_jobs():
+def eval_jobs(build_profile):
-    nej_output = $(nix-eval-jobs --workers 4 --gc-roots-dir @(GCROOTS_DIR) --force-recurse --flake '.#release-jobs')
+    nej_output = $(nix-eval-jobs --workers 4 --gc-roots-dir @(GCROOTS_DIR) --force-recurse --flake f'.#release-jobs.{build_profile}')
-    return [x for x in (json.loads(s) for s in nej_output.strip().split('\n'))
+    return [json.loads(s) for s in nej_output.strip().split('\n')]
-        if x['system'] in RELEASE_SYSTEMS
-    ]
 
 
 def upload_drv_paths_and_outputs(env: RelengEnvironment, paths: list[str]):
@@ -116,17 +106,17 @@ def upload_drv_paths_and_outputs(env: RelengEnvironment, paths: list[str]):
         raise subprocess.CalledProcessError(rv, proc.args)
 
 
-def make_manifest(eval_result):
+def make_manifest(builds_by_system):
-    manifest = {vs['system']: vs['outputs']['out'] for vs in eval_result}
+
     def manifest_line(system, out):
         return f'  {system} = "{out}";'
 
     manifest_text = textwrap.dedent("""\
         # This file was generated by releng/create_release.xsh in Lix
         {{
-          {lines}
+        {lines}
         }}
-    """).format(lines='\n'.join(manifest_line(s, p) for (s, p) in manifest.items()))
+    """).format(lines='\n'.join(manifest_line(s, p) for (s, p) in builds_by_system.items()))
 
     return manifest_text
 
@@ -152,6 +142,18 @@ def sha256_file(f: Path):
     return hasher.hexdigest()
 
 
+def extract_builds_by_system(eval_result):
+    # This could be a dictionary comprehension, but we want to be absolutely
+    # sure we don't have duplicates.
+    ret = {}
+    for attr in eval_result:
+        if attr['attrPath'][0] != 'build':
+            continue
+        assert attr['system'] not in ret
+        ret[attr['system']] = attr['outputs']['out']
+    return ret
+
+
 def make_artifacts_dir(eval_result, d: Path):
     d.mkdir(exist_ok=True, parents=True)
     version_dir = d / 'lix' / f'lix-{VERSION}'
@@ -160,12 +162,14 @@ def make_artifacts_dir(eval_result, d: Path):
     tarballs_drv = next(p for p in eval_result if p['attr'] == 'tarballs')
     cp --no-preserve=mode -r @(tarballs_drv['outputs']['out'])/* @(version_dir)
 
+    builds_by_system = extract_builds_by_system(eval_result)
+
     # FIXME: upgrade-nix searches for manifest.nix at root, which is rather annoying
     with open(d / 'manifest.nix', 'w') as h:
-        h.write(make_manifest(eval_result))
+        h.write(make_manifest(builds_by_system))
 
     with open(version_dir / 'manifest.nix', 'w') as h:
-        h.write(make_manifest(eval_result))
+        h.write(make_manifest(builds_by_system))
 
     print('[+] Make sources tarball')
 
@@ -181,51 +185,7 @@ def make_artifacts_dir(eval_result, d: Path):
 
 
 def prepare_release_notes():
-    print('[+] Preparing release notes')
+    rl_path = release_notes.build_release_notes_to_file()
-    RELEASE_NOTES_PATH = Path('doc/manual/rl-next')
-
-    if RELEASE_NOTES_PATH.isdir():
-        notes_body = subprocess.check_output(['build-release-notes', '--change-authors', 'doc/manual/change-authors.yml', 'doc/manual/rl-next']).decode()
-    else:
-        # I guess nobody put release notes on their changes?
-        print('[-] Warning: seemingly missing any release notes, not worrying about it')
-        notes_body = ''
-
-    rl_path = Path(f'doc/manual/src/release-notes/rl-{MAJOR}.md')
-
-    existing_rl = ''
-    try:
-        with open(rl_path, 'r') as fh:
-            existing_rl = fh.read()
-    except FileNotFoundError:
-        pass
-
-    date = datetime.datetime.now().strftime('%Y-%m-%d')
-
-    minor_header = f'# Lix {VERSION} ({date})'
-
-    header = f'# Lix {MAJOR} "{RELEASE_NAME}"'
-    if existing_rl.startswith(header):
-        # strip the header off for minor releases
-        lines = existing_rl.splitlines()
-        header = lines[0]
-        existing_rl = '\n'.join(lines[1:])
-    else:
-        header += f' ({date})\n\n'
-
-    header += '\n' + minor_header + '\n'
-
-    notes = header
-    notes += notes_body
-    notes += "\n\n"
-    notes += existing_rl
-
-    # make pre-commit happy about one newline
-    notes = notes.rstrip()
-    notes += "\n"
-
-    with open(rl_path, 'w') as fh:
-        fh.write(notes)
 
     commit_msg = textwrap.dedent("""\
         release: release notes for {VERSION}
@@ -233,36 +193,41 @@ def prepare_release_notes():
         {RELENG_MSG}
     """).format(VERSION=VERSION, RELENG_MSG=RELENG_MSG)
 
-    git add @(rl_path)
+    git add @(rl_path) @(release_notes.SUMMARY)
-    git rm doc/manual/rl-next/*.md
+    git rm --ignore-unmatch 'doc/manual/rl-next/*.md'
 
     git commit -m @(commit_msg)
 
 
-def verify_are_on_tag():
+def upload_artifacts(env: RelengEnvironment, noconfirm=False, no_check_git=False, force_push_tag=False):
-    current_tag = $(git describe --tag).strip()
+    if not no_check_git:
-    assert current_tag == VERSION
+        verify_are_on_tag()
-
+        git_preconditions()
-
-def upload_artifacts(env: RelengEnvironment, noconfirm=False, force_push_tag=False):
     assert 'AWS_SECRET_ACCESS_KEY' in __xonsh__.env
 
     tree @(ARTIFACTS)
 
+    env_part = f'environment {env.name}'
     not noconfirm and confirm(
-        f'Would you like to release {ARTIFACTS} as {VERSION}? Type "I want to release this" to confirm\n',
+        f'Would you like to release {ARTIFACTS} as {VERSION} in {env.colour(env_part)}? Type "I want to release this to {env.name}" to confirm\n',
-        'I want to release this'
+        f'I want to release this to {env.name}'
     )
 
+    docker_images = list((ARTIFACTS / f'lix/lix-{VERSION}').glob(f'lix-{VERSION}-docker-image-*.tar.gz'))
+    assert docker_images
+
     print('[+] Upload to cache')
     with open(DRVS_TXT) as fh:
         upload_drv_paths_and_outputs(env, [x.strip() for x in fh.readlines() if x])
 
+    print('[+] Upload docker images')
+    for target in env.docker_targets:
+        docker.upload_docker_images(target, docker_images)
 
     print('[+] Upload to release bucket')
     aws s3 cp --recursive @(ARTIFACTS)/ @(env.releases_bucket)/
     print('[+] Upload manual')
-    upload_manual()
+    upload_manual(env)
 
     print('[+] git push tag')
     git push @(['-f'] if force_push_tag else []) @(env.git_repo) f'{VERSION}:refs/tags/{VERSION}'
@@ -277,11 +242,11 @@ def do_tag_merge(force_tag=False, no_check_git=False):
 
 
 def build_manual(eval_result):
-    manual = next(x['outputs']['doc'] for x in eval_result if x['attr'] == 'build.x86_64-linux')
+    (drv, manual) = next((x['drvPath'], x['outputs']['doc']) for x in eval_result if x['attr'] == 'build.x86_64-linux')
     print('[+] Building manual')
-    realise([manual])
+    realise([drv])
 
-    cp --no-preserve=mode -vr @(manual)/share/doc/nix @(MANUAL)
+    cp --no-preserve=mode -T -vr @(manual)/share/doc/nix/manual @(MANUAL)
 
 
 def upload_manual(env: RelengEnvironment):
@@ -297,14 +262,14 @@ def upload_manual(env: RelengEnvironment):
         aws s3 sync @(MANUAL)/ @(env.docs_bucket)/manual/lix/stable/
 
 
-def build_artifacts(no_check_git=False):
+def build_artifacts(build_profile, no_check_git=False):
     rm -rf release/
     if not no_check_git:
         verify_are_on_tag()
         git_preconditions()
 
     print('[+] Evaluating')
-    eval_result = eval_jobs()
+    eval_result = eval_jobs(build_profile)
     drv_paths = [x['drvPath'] for x in eval_result]
 
     print('[+] Building')
@@ -312,7 +277,8 @@ def build_artifacts(no_check_git=False):
     build_manual(eval_result)
 
     with open(DRVS_TXT, 'w') as fh:
-        fh.write('\n'.join(drv_paths))
+        # don't bother putting the release tarballs themselves because they are duplicate and huge
+        fh.write('\n'.join(x['drvPath'] for x in eval_result if x['attr'] != 'lix-release-tarballs'))
 
     make_artifacts_dir(eval_result, ARTIFACTS)
     print(f'[+] Done! See {ARTIFACTS}')

releng/docker.xsh

@@ -0,0 +1,83 @@
+import json
+import logging
+from pathlib import Path
+import tempfile
+
+import requests
+
+from .environment import DockerTarget, RelengEnvironment
+from .version import VERSION, MAJOR
+from . import gitutils
+from .docker_assemble import Registry, OCIIndex, OCIIndexItem
+from . import docker_assemble
+
+log = logging.getLogger(__name__)
+log.setLevel(logging.INFO)
+
+def check_all_logins(env: RelengEnvironment):
+    for target in env.docker_targets:
+        check_login(target)
+
+def check_login(target: DockerTarget):
+    log.info('Checking login for %s', target.registry_name)
+    skopeo login @(target.registry_name())
+
+def upload_docker_images(target: DockerTarget, paths: list[Path]):
+    if not paths: return
+
+    sess = requests.Session()
+    sess.headers['User-Agent'] = 'lix-releng'
+
+    tag_names = [DockerTarget.resolve(tag, version=VERSION, major=MAJOR) for tag in target.tags]
+
+    # latest only gets tagged for the current release branch of Lix
+    if not gitutils.is_maintenance_branch('HEAD'):
+        tag_names.append('latest')
+
+    meta = {}
+
+    reg = docker_assemble.Registry(sess)
+    manifests = []
+
+    with tempfile.TemporaryDirectory() as tmp:
+        tmp = Path(tmp)
+
+        for path in paths:
+            digest_file = tmp / (path.name + '.digest')
+            tmp_image = tmp / 'tmp-image.tar.gz'
+
+            # insecure-policy: we don't have any signature policy, we are just uploading an image
+            #
+            # Absurd: we copy it into an OCI image first so we can get the hash
+            # we need to upload it untagged, because skopeo has no "don't tag
+            # this" option.
+            # The reason for this is that forgejo's container registry throws
+            # away old versions of tags immediately, so we cannot use a temp
+            # tag, and it *does* reduce confusion to not upload tags that
+            # should not be used.
+            #
+            # Workaround for: https://github.com/containers/skopeo/issues/2354
+            log.info('skopeo copy to temp oci-archive %s', tmp_image)
+            skopeo --insecure-policy copy --format oci --all --digestfile @(digest_file) docker-archive:@(path) oci-archive:@(tmp_image)
+
+            inspection = json.loads($(skopeo inspect oci-archive:@(tmp_image)))
+
+            docker_arch = inspection['Architecture']
+            docker_os = inspection['Os']
+            meta = inspection['Labels']
+
+            log.info('Pushing image %s for %s to %s', path, docker_arch, target.registry_path)
+
+            digest = digest_file.read_text().strip()
+            skopeo --insecure-policy copy --preserve-digests --all oci-archive:@(tmp_image) f'docker://{target.registry_path}@{digest}'
+
+            # skopeo doesn't give us the manifest size directly, so we just ask the registry
+            metadata = reg.image_info(target.registry_path, digest)
+
+            manifests.append(OCIIndexItem(metadata=metadata, architecture=docker_arch, os=docker_os))
+
+    log.info('Pushed images to %r, building a bigger and more menacing manifest from %r with metadata %r', target, manifests, meta)
+    # send the multiarch manifest to each tag
+    index = OCIIndex(manifests=manifests, annotations=meta)
+    for tag in tag_names:
+        reg.upload_index(target.registry_path, tag, index)

releng/docker_assemble.py

@@ -0,0 +1,391 @@
+from typing import Any, Literal, Optional
+import re
+from pathlib import Path
+import json
+import dataclasses
+import time
+from urllib.parse import unquote
+import urllib.request
+import logging
+
+import requests.auth
+import requests
+import xdg_base_dirs
+
+log = logging.getLogger(__name__)
+log.setLevel(logging.INFO)
+
+DEBUG_REQUESTS = False
+if DEBUG_REQUESTS:
+    urllib3_logger = logging.getLogger('requests.packages.urllib3')
+    urllib3_logger.setLevel(logging.DEBUG)
+    urllib3_logger.propagate = True
+
+# So, there is a bunch of confusing stuff happening in this file. The gist of why it's Like This is:
+#
+# nix2container does not concern itself with tags (reasonably enough):
+# https://github.com/nlewo/nix2container/issues/59
+#
+# This is fine. But then we noticed: docker images don't play nice if you have
+# multiple architectures you want to abstract over if you don't do special
+# things. Those special things are images with manifests containing multiple
+# images.
+#
+# Docker has a data model vaguely analogous to git: you have higher level
+# objects referring to a bunch of content-addressed blobs.
+#
+# A multiarch image is more or less just a manifest that refers to more
+# manifests; in OCI it is an Index.
+#
+# See the API spec here: https://github.com/opencontainers/distribution-spec/blob/v1.0.1/spec.md#definitions
+# And the Index spec here: https://github.com/opencontainers/image-spec/blob/v1.0.1/image-index.md
+#
+# skopeo doesn't *know* how to make multiarch *manifests*:
+# https://github.com/containers/skopeo/issues/1136
+#
+# There is a tool called manifest-tool that is supposed to do this
+# (https://github.com/estesp/manifest-tool) but it doesn't support putting in
+# annotations on the outer image, and I *really* didn't want to write golang to
+# fix that. Thus, a little bit of homebrew containers code.
+#
+# Essentially what we are doing in here is splatting a bunch of images into the
+# registry without tagging them (with a silly workaround to skopeo issues),
+# then simply sending a new composite manifest ourselves.
+
+DockerArchitecture = Literal['amd64'] | Literal['arm64']
+MANIFEST_MIME = 'application/vnd.oci.image.manifest.v1+json'
+INDEX_MIME = 'application/vnd.oci.image.index.v1+json'
+
+
+@dataclasses.dataclass(frozen=True, order=True)
+class ImageMetadata:
+    size: int
+    digest: str
+    """sha256:SOMEHEX"""
+
+
+@dataclasses.dataclass(frozen=True, order=True)
+class OCIIndexItem:
+    """Information about an untagged uploaded image."""
+
+    metadata: ImageMetadata
+
+    architecture: DockerArchitecture
+
+    os: str = 'linux'
+
+    def serialize(self):
+        return {
+            'mediaType': MANIFEST_MIME,
+            'size': self.metadata.size,
+            'digest': self.metadata.digest,
+            'platform': {
+                'architecture': self.architecture,
+                'os': self.os,
+            }
+        }
+
+
+@dataclasses.dataclass(frozen=True)
+class OCIIndex:
+    manifests: list[OCIIndexItem]
+
+    annotations: dict[str, str]
+
+    def serialize(self):
+        return {
+            'schemaVersion': 2,
+            'manifests': [item.serialize() for item in sorted(self.manifests)],
+            'annotations': self.annotations
+        }
+
+
+@dataclasses.dataclass
+class TaggingOperation:
+    manifest: OCIIndex
+    tags: list[str]
+    """Tags this image is uploaded under"""
+
+
+runtime_dir = xdg_base_dirs.xdg_runtime_dir()
+config_dir = xdg_base_dirs.xdg_config_home()
+
+AUTH_FILES = ([runtime_dir / 'containers/auth.json'] if runtime_dir else []) + \
+    [config_dir / 'containers/auth.json', Path.home() / '.docker/config.json']
+
+
+# Copied from Werkzeug https://github.com/pallets/werkzeug/blob/62e3ea45846d06576199a2f8470be7fe44c867c1/src/werkzeug/http.py#L300-L325
+def parse_list_header(value: str) -> list[str]:
+    """Parse a header value that consists of a list of comma separated items according
+    to `RFC 9110 <https://httpwg.org/specs/rfc9110.html#abnf.extension>`__.
+
+    This extends :func:`urllib.request.parse_http_list` to remove surrounding quotes
+    from values.
+
+    .. code-block:: python
+
+        parse_list_header('token, "quoted value"')
+        ['token', 'quoted value']
+
+    This is the reverse of :func:`dump_header`.
+
+    :param value: The header value to parse.
+    """
+    result = []
+
+    for item in urllib.request.parse_http_list(value):
+        if len(item) >= 2 and item[0] == item[-1] == '"':
+            item = item[1:-1]
+
+        result.append(item)
+
+    return result
+
+
+# https://www.rfc-editor.org/rfc/rfc2231#section-4
+_charset_value_re = re.compile(
+    r"""
+    ([\w!#$%&*+\-.^`|~]*)'  # charset part, could be empty
+    [\w!#$%&*+\-.^`|~]*'  # don't care about language part, usually empty
+    ([\w!#$%&'*+\-.^`|~]+)  # one or more token chars with percent encoding
+    """,
+    re.ASCII | re.VERBOSE,
+)
+
+
+# Copied from: https://github.com/pallets/werkzeug/blob/62e3ea45846d06576199a2f8470be7fe44c867c1/src/werkzeug/http.py#L327-L394
+def parse_dict_header(value: str) -> dict[str, str | None]:
+    """Parse a list header using :func:`parse_list_header`, then parse each item as a
+    ``key=value`` pair.
+
+    .. code-block:: python
+
+        parse_dict_header('a=b, c="d, e", f')
+        {"a": "b", "c": "d, e", "f": None}
+
+    This is the reverse of :func:`dump_header`.
+
+    If a key does not have a value, it is ``None``.
+
+    This handles charsets for values as described in
+    `RFC 2231 <https://www.rfc-editor.org/rfc/rfc2231#section-3>`__. Only ASCII, UTF-8,
+    and ISO-8859-1 charsets are accepted, otherwise the value remains quoted.
+
+    :param value: The header value to parse.
+
+    .. versionchanged:: 3.0
+        Passing bytes is not supported.
+
+    .. versionchanged:: 3.0
+        The ``cls`` argument is removed.
+
+    .. versionchanged:: 2.3
+        Added support for ``key*=charset''value`` encoded items.
+
+    .. versionchanged:: 0.9
+       The ``cls`` argument was added.
+    """
+    result: dict[str, str | None] = {}
+
+    for item in parse_list_header(value):
+        key, has_value, value = item.partition("=")
+        key = key.strip()
+
+        if not has_value:
+            result[key] = None
+            continue
+
+        value = value.strip()
+        encoding: str | None = None
+
+        if key[-1] == "*":
+            # key*=charset''value becomes key=value, where value is percent encoded
+            # adapted from parse_options_header, without the continuation handling
+            key = key[:-1]
+            match = _charset_value_re.match(value)
+
+            if match:
+                # If there is a charset marker in the value, split it off.
+                encoding, value = match.groups()
+                assert encoding
+                encoding = encoding.lower()
+
+            # A safe list of encodings. Modern clients should only send ASCII or UTF-8.
+            # This list will not be extended further. An invalid encoding will leave the
+            # value quoted.
+            if encoding in {"ascii", "us-ascii", "utf-8", "iso-8859-1"}:
+                # invalid bytes are replaced during unquoting
+                value = unquote(value, encoding=encoding)
+
+        if len(value) >= 2 and value[0] == value[-1] == '"':
+            value = value[1:-1]
+
+        result[key] = value
+
+    return result
+
+
+def parse_www_authenticate(www_authenticate):
+    scheme, _, rest = www_authenticate.partition(' ')
+    scheme = scheme.lower()
+    rest = rest.strip()
+
+    parsed = parse_dict_header(rest.rstrip('='))
+    return parsed
+
+
+class AuthState:
+
+    def __init__(self, auth_files: list[Path] = AUTH_FILES):
+        self.auth_map: dict[str, str] = {}
+        for f in auth_files:
+            self.auth_map.update(AuthState.load_auth_file(f))
+        self.token_cache: dict[str, str] = {}
+
+    @staticmethod
+    def load_auth_file(path: Path) -> dict[str, str]:
+        if path.exists():
+            with path.open() as fh:
+                try:
+                    json_obj = json.load(fh)
+                    return {k: v['auth'] for k, v in json_obj['auths'].items()}
+                except (json.JSONDecodeError, KeyError) as e:
+                    log.exception('JSON decode error in %s', path, exc_info=e)
+        return {}
+
+    def get_token(self, hostname: str) -> Optional[str]:
+        return self.token_cache.get(hostname)
+
+    def obtain_token(self, session: requests.Session, token_endpoint: str,
+                     scope: str, service: str, image_path: str) -> str:
+        authority, _, _ = image_path.partition('/')
+        if tok := self.get_token(authority):
+            return tok
+
+        creds = self.find_credential_for(image_path)
+        if not creds:
+            raise ValueError('No credentials available for ' + image_path)
+
+        resp = session.get(token_endpoint,
+                           params={
+                               'client_id': 'lix-releng',
+                               'scope': scope,
+                               'service': service,
+                           },
+                           headers={
+                               'Authorization': 'Basic ' + creds
+                           }).json()
+        token = resp['token']
+        self.token_cache[authority] = token
+        return token
+
+    def find_credential_for(self, image_path: str):
+        trails = image_path.split('/')
+        for i in range(len(trails)):
+            prefix = '/'.join(trails[:len(trails) - i])
+            if prefix in self.auth_map:
+                return self.auth_map[prefix]
+
+        return None
+
+
+class RegistryAuthenticator(requests.auth.AuthBase):
+    """Authenticates to an OCI compliant registry"""
+
+    def __init__(self, auth_state: AuthState, session: requests.Session,
+                 image: str):
+        self.auth_map: dict[str, str] = {}
+        self.image = image
+        self.session = session
+        self.auth_state = auth_state
+
+    def response_hook(self, r: requests.Response,
+                      **kwargs: Any) -> requests.Response:
+        if r.status_code == 401:
+            www_authenticate = r.headers.get('www-authenticate', '').lower()
+            parsed = parse_www_authenticate(www_authenticate)
+            assert parsed
+
+            tok = self.auth_state.obtain_token(
+                self.session,
+                parsed['realm'],  # type: ignore
+                parsed['scope'],  # type: ignore
+                parsed['service'],  # type: ignore
+                self.image)
+
+            new_req = r.request.copy()
+            new_req.headers['Authorization'] = 'Bearer ' + tok
+
+            return self.session.send(new_req)
+        else:
+            return r
+
+    def __call__(self,
+                 r: requests.PreparedRequest) -> requests.PreparedRequest:
+        authority, _, _ = self.image.partition('/')
+        auth_may = self.auth_state.get_token(authority)
+
+        if auth_may:
+            r.headers['Authorization'] = 'Bearer ' + auth_may
+
+        r.register_hook('response', self.response_hook)
+        return r
+
+
+class Registry:
+
+    def __init__(self, session: requests.Session):
+        self.auth_state = AuthState()
+        self.session = session
+
+    def image_info(self, image_path: str, manifest_id: str) -> ImageMetadata:
+        authority, _, path = image_path.partition('/')
+        resp = self.session.head(
+            f'https://{authority}/v2/{path}/manifests/{manifest_id}',
+            headers={'Accept': MANIFEST_MIME},
+            auth=RegistryAuthenticator(self.auth_state, self.session,
+                                       image_path))
+        resp.raise_for_status()
+        return ImageMetadata(int(resp.headers['content-length']),
+                             resp.headers['docker-content-digest'])
+
+    def delete_tag(self, image_path: str, tag: str):
+        authority, _, path = image_path.partition('/')
+        resp = self.session.delete(
+            f'https://{authority}/v2/{path}/manifests/{tag}',
+            headers={'Content-Type': INDEX_MIME},
+            auth=RegistryAuthenticator(self.auth_state, self.session,
+                                       image_path))
+        resp.raise_for_status()
+
+    def _upload_index(self, image_path: str, tag: str, index: OCIIndex):
+        authority, _, path = image_path.partition('/')
+        body = json.dumps(index.serialize(),
+                          separators=(',', ':'),
+                          sort_keys=True)
+
+        resp = self.session.put(
+            f'https://{authority}/v2/{path}/manifests/{tag}',
+            data=body,
+            headers={'Content-Type': INDEX_MIME},
+            auth=RegistryAuthenticator(self.auth_state, self.session,
+                                       image_path))
+        resp.raise_for_status()
+
+        return resp.headers['Location']
+
+    def upload_index(self,
+                     image_path: str,
+                     tag: str,
+                     index: OCIIndex,
+                     retries=20,
+                     retry_delay=1):
+        # eventual consistency lmao
+        for _ in range(retries):
+            try:
+                return self._upload_index(image_path, tag, index)
+            except requests.HTTPError as e:
+                if e.response.status_code != 404:
+                    raise
+
+            time.sleep(retry_delay)

releng/environment.py

@@ -1,5 +1,9 @@
-import dataclasses
+from typing import Callable
 import urllib.parse
+import re
+import functools
+import subprocess
+import dataclasses
 
 S3_HOST = 's3.lix.systems'
 S3_ENDPOINT = 'https://s3.lix.systems'
@@ -19,12 +23,19 @@ DEFAULT_STORE_URI_BITS = {
 @dataclasses.dataclass
 class DockerTarget:
     registry_path: str
+    """Registry path without the tag, e.g. ghcr.io/lix-project/lix"""
 
-    def resolve(self, version: str) -> str:
+    tags: list[str]
-        """Applies templates:
+    """List of tags this image should take. There must be at least one."""
-        - version: the Lix version
+
+    @staticmethod
+    def resolve(item: str, version: str, major: str) -> str:
         """
-        return self.registry_path.format(version=version)
+        Applies templates:
+        - version: the Lix version e.g. 2.90.0
+        - major: the major Lix version e.g. 2.90
+        """
+        return item.format(version=version, major=major)
 
     def registry_name(self) -> str:
         [a, _, _] = self.registry_path.partition('/')
@@ -34,6 +45,7 @@ class DockerTarget:
 @dataclasses.dataclass
 class RelengEnvironment:
     name: str
+    colour: Callable[[str], str]
 
     cache_store_overlay: dict[str, str]
     cache_bucket: str
@@ -49,21 +61,69 @@ class RelengEnvironment:
         return self.cache_bucket + "?" + urllib.parse.urlencode(qs)
 
 
+SGR = '\x1b['
+RED = '31;1m'
+GREEN = '32;1m'
+RESET = '0m'
+
+
+def sgr(colour: str, text: str) -> str:
+    return f'{SGR}{colour}{text}{SGR}{RESET}'
+
+
 STAGING = RelengEnvironment(
     name='staging',
+    colour=functools.partial(sgr, GREEN),
     docs_bucket='s3://staging-docs',
     cache_bucket='s3://staging-cache',
     cache_store_overlay={'secret-key': 'staging.key'},
     releases_bucket='s3://staging-releases',
     git_repo='ssh://git@git.lix.systems/lix-project/lix-releng-staging',
     docker_targets=[
-        DockerTarget(
+        # latest will be auto tagged if appropriate
-            'git.lix.systems/lix-project/lix-releng-staging/lix:{version}'),
+        DockerTarget('git.lix.systems/lix-project/lix-releng-staging',
+                     tags=['{version}', '{major}']),
+        DockerTarget('ghcr.io/lix-project/lix-releng-staging',
+                     tags=['{version}', '{major}']),
+    ],
+)
+
+GERRIT_REMOTE_RE = re.compile(r'^ssh://(\w+@)?gerrit.lix.systems:2022/lix$')
+
+
+def guess_gerrit_remote():
+    """
+    Deals with people having unknown gerrit username.
+    """
+    out = [
+        x.split()[1] for x in subprocess.check_output(
+            ['git', 'remote', '-v']).decode().splitlines()
+    ]
+    return next(x for x in out if GERRIT_REMOTE_RE.match(x))
+
+
+PROD = RelengEnvironment(
+    name='production',
+    colour=functools.partial(sgr, RED),
+    docs_bucket='s3://docs',
+    cache_bucket='s3://cache',
+    # FIXME: we should decrypt this with age into a tempdir in the future, but
+    # the issue is how to deal with the recipients file. For now, we should
+    # just delete it after doing a release.
+    cache_store_overlay={'secret-key': 'prod.key'},
+    releases_bucket='s3://releases',
+    git_repo=guess_gerrit_remote(),
+    docker_targets=[
+        # latest will be auto tagged if appropriate
+        DockerTarget('git.lix.systems/lix-project/lix',
+                     tags=['{version}', '{major}']),
+        DockerTarget('ghcr.io/lix-project/lix', tags=['{version}', '{major}']),
     ],
 )
 
 ENVIRONMENTS = {
     'staging': STAGING,
+    'production': PROD,
 }
 
 

releng/gitutils.xsh

@@ -0,0 +1,39 @@
+import subprocess
+import json
+
+from .version import VERSION
+
+
+def version_compare(v1: str, v2: str):
+    return json.loads($(nix-instantiate --eval --json --argstr v1 @(v1) --argstr v2 @(v2) --expr '{v1, v2}: builtins.compareVersions v1 v2'))
+
+
+def latest_tag_on_branch(branch: str) -> str:
+    return $(git describe --abbrev=0 @(branch) e>/dev/null).strip()
+
+
+def is_maintenance_branch(branch: str) -> bool:
+    try:
+        main_tag = latest_tag_on_branch('main')
+        current_tag = latest_tag_on_branch(branch)
+
+        return version_compare(current_tag, main_tag) < 0
+    except subprocess.CalledProcessError:
+        # This is the case before Lix releases 2.90, since main *has* no
+        # release tag on it.
+        # FIXME: delete this case after 2.91
+        return False
+
+
+def verify_are_on_tag():
+    current_tag = $(git describe --tag).strip()
+    assert current_tag == VERSION
+
+
+def git_preconditions():
+    # verify there is nothing in index ready to stage
+    proc = !(git diff-index --quiet --cached HEAD --)
+    assert proc.rtn == 0
+    # verify there is nothing *stageable* and tracked
+    proc = !(git diff-files --quiet)
+    assert proc.rtn == 0

releng/release-jobs.nix

@@ -3,8 +3,27 @@ let
   inherit (pkgs) lib;
   lix = hydraJobs.build.x86_64-linux;
 
-  systems = [ "x86_64-linux" ];
+  # This is all so clumsy because we can't use arguments to functions in
-  dockerSystems = [ "x86_64-linux" ];
+  # flakes, and certainly not with n-e-j.
+  profiles = {
+    # Used for testing
+    x86_64-linux-only = {
+      systems = [ "x86_64-linux" ];
+      dockerSystems = [ "x86_64-linux" ];
+    };
+    all = {
+      systems = [
+        "x86_64-linux"
+        "aarch64-linux"
+        "aarch64-darwin"
+        "x86_64-darwin"
+      ];
+      dockerSystems = [
+        "x86_64-linux"
+        "aarch64-linux"
+      ];
+    };
+  };
 
   doTarball =
     {
@@ -27,13 +46,14 @@ let
       sha256sum --binary $filename | cut -f1 -d' ' > $out/$basename.sha256
     '';
 
-  targets =
+  targetsFor =
+    { systems, dockerSystems }:
     builtins.map (system: {
       target = hydraJobs.binaryTarball.${system};
       targetName = "*.tar.xz";
     }) systems
     ++ builtins.map (system: {
-      target = hydraJobs.dockerImage.${system};
+      target = hydraJobs.dockerImage.${system}.tarball;
       targetName = "image.tar.gz";
       rename = "lix-${lix.version}-docker-image-${system}.tar.gz";
     }) dockerSystems;
@@ -44,14 +64,29 @@ let
     tar -cvzf "$out/lix-${lix.version}-manual.tar.gz" lix-${lix.version}-manual
   '';
 
-  tarballs = pkgs.runCommand "lix-release-tarballs" { } ''
+  tarballsFor =
-    mkdir -p $out
+    { systems, dockerSystems }:
-    ${lib.concatMapStringsSep "\n" doTarball targets}
+    pkgs.runCommand "lix-release-tarballs" { } ''
-    cp ${manualTar}/*.tar.gz $out
+      mkdir -p $out
-    cp -r ${lix.doc}/share/doc/nix/manual $out
+      ${lib.concatMapStringsSep "\n" doTarball (targetsFor {
-  '';
+        inherit systems dockerSystems;
+      })}
+      ${doTarball {
+        target = manualTar;
+        targetName = "lix-*.tar.gz";
+      }}
+      cp -r ${lix.doc}/share/doc/nix/manual $out
+    '';
 in
-{
+(builtins.mapAttrs (
+  _:
+  { systems, dockerSystems }:
+  {
+    build = lib.filterAttrs (x: _: builtins.elem x systems) hydraJobs.build;
+    tarballs = tarballsFor { inherit systems dockerSystems; };
+  }
+) profiles)
+// {
   inherit (hydraJobs) build;
-  inherit tarballs;
+  inherit tarballsFor;
 }

releng/release_notes.py

@@ -0,0 +1,81 @@
+from pathlib import Path
+import subprocess
+import datetime
+
+from .version import MAJOR, VERSION, RELEASE_NAME
+
+MANUAL = Path('doc/manual')
+RELEASE_NOTES_BASE = MANUAL / 'src/release-notes'
+VERSION_RL = RELEASE_NOTES_BASE / f'rl-{MAJOR}.md'
+SUMMARY = MANUAL / 'src/SUMMARY.md'
+
+def add_to_summary(date: str):
+    # N.B: This kind of duplicates gitutils.is_maintenance_branch, but it's a more clear
+    # check that allows potentially releasing a -rc without release notes being
+    # moved, then in .0 actually move the release notes in place.
+    if VERSION_RL.exists():
+        return
+
+    MARKER = '<!-- RELENG-AUTO-INSERTION-MARKER'
+
+    new_lines = []
+    for line in SUMMARY.read_text().splitlines():
+        new_lines.append(line)
+        if MARKER in line:
+            indent, _, _ = line.partition(MARKER)
+            new_lines.append(f'{indent}- [Lix {MAJOR} ({date})](release-notes/rl-{MAJOR}.md)')
+
+    # make pre-commit happy about one newline
+    text = '\n'.join(new_lines).rstrip()
+    text += '\n'
+    SUMMARY.write_text(text)
+
+def build_release_notes_to_file():
+    date = datetime.datetime.now().strftime('%Y-%m-%d')
+    add_to_summary(date)
+
+    print('[+] Preparing release notes')
+    RELEASE_NOTES_PATH = Path('doc/manual/rl-next')
+
+    if RELEASE_NOTES_PATH.is_dir():
+        notes_body = subprocess.check_output(['build-release-notes', '--change-authors', 'doc/manual/change-authors.yml', RELEASE_NOTES_PATH]).decode()
+    else:
+        # I guess nobody put release notes on their changes?
+        print('[-] Warning: seemingly missing any release notes, not worrying about it')
+        notes_body = ''
+
+    rl_path = Path(RELEASE_NOTES_BASE / f'rl-{MAJOR}.md')
+
+    existing_rl = ''
+    try:
+        with open(rl_path, 'r') as fh:
+            existing_rl = fh.read()
+    except FileNotFoundError:
+        pass
+
+    minor_header = f'# Lix {VERSION} ({date})'
+
+    header = f'# Lix {MAJOR} "{RELEASE_NAME}"'
+    if existing_rl.startswith(header):
+        # strip the header off for minor releases
+        lines = existing_rl.splitlines()
+        header = lines[0]
+        existing_rl = '\n'.join(lines[1:])
+    else:
+        header += f' ({date})\n\n'
+
+    header += '\n' + minor_header + '\n'
+
+    notes = header
+    notes += notes_body
+    notes += "\n\n"
+    notes += existing_rl
+
+    # make pre-commit happy about one newline
+    notes = notes.rstrip()
+    notes += "\n"
+
+    with open(rl_path, 'w') as fh:
+        fh.write(notes)
+
+    return rl_path

src/libcmd/installables.cc

@@ -214,7 +214,7 @@ void SourceExprCommand::completeInstallable(AddCompletions & completions, std::s
 
             evalSettings.pureEval = false;
             auto state = getEvalState();
-            Expr *e = state->parseExprFromFile(
+            Expr & e = state->parseExprFromFile(
                 resolveExprPath(state->checkSourcePath(lookupFileArg(*state, *file)))
                 );
 
@@ -434,13 +434,13 @@ Installables SourceExprCommand::parseInstallables(
         auto vFile = state->allocValue();
 
         if (file == "-") {
-            auto e = state->parseStdin();
+            auto & e = state->parseStdin();
             state->eval(e, *vFile);
         }
         else if (file)
             state->evalFile(lookupFileArg(*state, *file), *vFile);
         else {
-            auto e = state->parseExprFromString(*expr, state->rootPath(CanonPath::fromCwd()));
+            auto & e = state->parseExprFromString(*expr, state->rootPath(CanonPath::fromCwd()));
             state->eval(e, *vFile);
         }