Merge "build internal API docs with Meson" into main

This commit adds the capability for building the Doxygen internal API
docs in the Meson buildsystem, and also makes doing so the default for
the internal-api-docs hydra job. Aside from the /nix-support directory,
which differed only by the hash part of a store path, the outputs of
hydraJobs.internal-api-docs before and after this commit were
bit-for-bit identical on my machine.

Change-Id: I98f0017891c25b06866c15f7652fe74f706ec8e1

.clang-format

@@ -48,4 +48,3 @@ PointerAlignment: Middle
 SortIncludes: Never
 SpaceAfterCStyleCast: true
 SpaceAfterTemplateKeyword: false
-

.clang-tidy

@@ -0,0 +1,18 @@
+UseColor: true
+Checks:
+  - -*
+  - bugprone-*
+    # too many warnings
+  - -bugprone-assignment-in-if-condition
+    # too many warnings
+  - -bugprone-narrowing-conversions
+    # kind of nonsense
+  - -bugprone-easily-swappable-parameters
+    # too many warnings for now
+  - -bugprone-implicit-widening-of-multiplication-result
+    # Lix's exception handling is Questionable
+  - -bugprone-empty-catch
+    # many warnings
+  - -bugprone-unchecked-optional-access
+    # many warnings, seems like a questionable lint
+  - -bugprone-branch-clone

.github/workflows/hydra_status.yml

@@ -17,4 +17,3 @@ jobs:
       with:
         fetch-depth: 0
     - run: bash scripts/check-hydra-status.sh
-

.gitignore

@@ -10,6 +10,7 @@ perl/Makefile.config
 /stamp-h1
 /svn-revision
 /libtool
+/config
 
 # /doc/manual/
 /doc/manual/*.1
@@ -20,15 +21,9 @@ perl/Makefile.config
 /doc/manual/conf-file.json
 /doc/manual/language.json
 /doc/manual/xp-features.json
-/doc/manual/src/SUMMARY.md
-/doc/manual/src/SUMMARY-rl-next.md
-/doc/manual/src/command-ref/new-cli
-/doc/manual/src/command-ref/conf-file.md
 /doc/manual/src/command-ref/experimental-features-shortlist.md
 /doc/manual/src/contributing/experimental-feature-descriptions.md
-/doc/manual/src/language/builtins.md
+/doc/manual/src/release-notes/rl-next-generated.md
-/doc/manual/src/language/builtin-constants.md
-/doc/manual/src/release-notes/rl-next.md
 
 # /scripts/
 /scripts/nix-profile.sh
@@ -159,3 +154,5 @@ result-*
 buildtime.bin
 
 .envrc.local
+# We generate this with a Nix shell hook
+/.pre-commit-config.yaml

COPYING

@@ -55,7 +55,7 @@ modified by someone else and passed on, the recipients should know
 that what they have is not the original version, so that the original
 author's reputation will not be affected by problems that might be
 introduced by others.
-
+
   Finally, software patents pose a constant threat to the existence of
 any free program.  We wish to make sure that a company cannot
 effectively restrict the users of a free program by obtaining a
@@ -111,7 +111,7 @@ modification follow.  Pay close attention to the difference between a
 "work based on the library" and a "work that uses the library".  The
 former contains code derived from the library, whereas the latter must
 be combined with the library in order to run.
-
+
 		  GNU LESSER GENERAL PUBLIC LICENSE
    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
 
@@ -158,7 +158,7 @@ Library.
   You may charge a fee for the physical act of transferring a copy,
 and you may at your option offer warranty protection in exchange for a
 fee.
-
+
   2. You may modify your copy or copies of the Library or any portion
 of it, thus forming a work based on the Library, and copy and
 distribute such modifications or work under the terms of Section 1
@@ -216,7 +216,7 @@ instead of to this License.  (If a newer version than version 2 of the
 ordinary GNU General Public License has appeared, then you can specify
 that version instead if you wish.)  Do not make any other change in
 these notices.
-
+
   Once this change is made in a given copy, it is irreversible for
 that copy, so the ordinary GNU General Public License applies to all
 subsequent copies and derivative works made from that copy.
@@ -267,7 +267,7 @@ Library will still fall under Section 6.)
 distribute the object code for the work under the terms of Section 6.
 Any executables containing that work also fall under Section 6,
 whether or not they are linked directly with the Library itself.
-
+
   6. As an exception to the Sections above, you may also combine or
 link a "work that uses the Library" with the Library to produce a
 work containing portions of the Library, and distribute that work
@@ -329,7 +329,7 @@ restrictions of other proprietary libraries that do not normally
 accompany the operating system.  Such a contradiction means you cannot
 use both them and the Library together in an executable that you
 distribute.
-
+
   7. You may place library facilities that are a work based on the
 Library side-by-side in a single library together with other library
 facilities not covered by this License, and distribute such a combined
@@ -370,7 +370,7 @@ subject to these terms and conditions.  You may not impose any further
 restrictions on the recipients' exercise of the rights granted herein.
 You are not responsible for enforcing compliance by third parties with
 this License.
-
+
   11. If, as a consequence of a court judgment or allegation of patent
 infringement or for any other reason (not limited to patent issues),
 conditions are imposed on you (whether by court order, agreement or
@@ -422,7 +422,7 @@ conditions either of that version or of any later version published by
 the Free Software Foundation.  If the Library does not specify a
 license version number, you may choose any version ever published by
 the Free Software Foundation.
-
+
   14. If you wish to incorporate parts of the Library into other free
 programs whose distribution conditions are incompatible with these,
 write to the author to ask for permission.  For software which is
@@ -456,7 +456,7 @@ SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
 DAMAGES.
 
 		     END OF TERMS AND CONDITIONS
-
+
            How to Apply These Terms to Your New Libraries
 
   If you develop a new library, and you want it to be of the greatest
@@ -500,5 +500,3 @@ necessary.  Here is a sample; alter the names:
   Ty Coon, President of Vice
 
 That's all there is to it!
-
-

Makefile.config.in

@@ -19,6 +19,7 @@ LIBBROTLI_LIBS = @LIBBROTLI_LIBS@
 LIBCURL_LIBS = @LIBCURL_LIBS@
 LIBSECCOMP_LIBS = @LIBSECCOMP_LIBS@
 LOWDOWN_LIBS = @LOWDOWN_LIBS@
+NIXDOC_LIBS = -llix_doc
 OPENSSL_LIBS = @OPENSSL_LIBS@
 PACKAGE_NAME = @PACKAGE_NAME@
 PACKAGE_VERSION = @PACKAGE_VERSION@

bench/.gitignore

@@ -0,0 +1,3 @@
+bench-*.json
+bench-*.md
+nixpkgs

bench/README.md

@@ -0,0 +1,91 @@
+# Benchmarking scripts for Lix
+
+These are very much WIP, and have a few clumsy assumptions that we would
+somewhat rather be fixed, but we have committed them to let others be able to
+do benchmarking in the mean time.
+
+## Benchmarking procedure
+
+Build some Lixes you want to compare, by whichever means you wish.
+
+Get a computer that is not busy and *strongly preferably* is bare-metal or at
+least not a cloud VM (e.g. go make coffee when running benchmarks).
+
+From the root of a Lix checkout, run `./bench/bench.sh resultlink-one
+resultlink-two`, where `resultlink-one` and `resultlink-two` are the result
+links from the builds you want to test (they can be any directory with bin/nix
+in it, however).
+
+To get the summary again, run `./bench/summarize.jq bench/bench-*.json`.
+
+## Example results
+
+(vim tip: `:r !bench/summarize.jq bench/bench-*.json` to dump it directly into
+your editor)
+
+```
+result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6
+c6fe19ccb59cac54b5b3f25e160870 hello
+  mean:     15.993s ± 0.081s
+            user: 13.321s | system: 1.865s
+  median:   15.994s
+  range:    15.829s ... 16.096s
+  relative: 1
+result/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19cc
+b59cac54b5b3f25e160870 hello
+  mean:     15.897s ± 0.075s
+            user: 13.248s | system: 1.843s
+  median:   15.88s
+  range:    15.807s ... 16.047s
+  relative: 0.994
+
+---
+
+result/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
+  mean:     0.4s ± 0.024s
+            user: 0.335s | system: 0.046s
+  median:   0.386s
+  range:    0.379s ... 0.43s
+  relative: 1
+
+result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
+  mean:     0.404s ± 0.024s
+            user: 0.338s | system: 0.046s
+  median:   0.386s
+  range:    0.384s ... 0.436s
+  relative: 1.008
+
+---
+
+result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
+  mean:     5.838s ± 0.023s
+            user: 5.083s | system: 0.464s
+  median:   5.845s
+  range:    5.799s ... 5.867s
+  relative: 1
+
+result/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
+  mean:     5.788s ± 0.044s
+            user: 5.056s | system: 0.439s
+  median:   5.79s
+  range:    5.715s ... 5.876s
+  relative: 0.991
+
+---
+
+GC_INITIAL_HEAP_SIZE=10g result-asserts/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
+  mean:     4.147s ± 0.021s
+            user: 3.457s | system: 0.487s
+  median:   4.147s
+  range:    4.123s ... 4.195s
+  relative: 1
+
+GC_INITIAL_HEAP_SIZE=10g result/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
+  mean:     4.149s ± 0.027s
+            user: 3.483s | system: 0.456s
+  median:   4.142s
+  range:    4.126s ... 4.215s
+  relative: 1
+
+---
+```

bench/bench.sh

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+shopt -s inherit_errexit
+
+scriptdir=$(cd "$(dirname -- "$0")" ; pwd -P)
+cd "$scriptdir/.."
+
+if [[ $# -lt 2 ]]; then
+    # FIXME(jade): it is a reasonable use case to want to run a benchmark run
+    # on just one build. However, since we are using hyperfine in comparison
+    # mode, we would have to combine the JSON ourselves to support that, which
+    # would probably be better done by writing a benchmarking script in
+    # not-bash.
+    echo "Fewer than two result dirs given, nothing to compare!" >&2
+    echo "Pass some directories (with names indicating which alternative they are) with bin/nix in them" >&2
+    echo "Usage: ./bench/bench.sh result-1 result-2 [result-3...]" >&2
+    exit 1
+fi
+
+_exit=""
+trap "$_exit" EXIT
+
+# XXX: yes this is very silly. flakes~!!
+nix build --impure --expr '(builtins.getFlake "git+file:.").inputs.nixpkgs.outPath' -o bench/nixpkgs
+
+export NIX_REMOTE="$(mktemp -d)"
+_exit='rm -rfv "$NIX_REMOTE"; $_exit'
+export NIX_PATH="nixpkgs=bench/nixpkgs:nixos-config=bench/configuration.nix"
+
+builds=("$@")
+
+flake_args="--extra-experimental-features 'nix-command flakes'"
+
+hyperfineArgs=(
+    --parameter-list BUILD "$(IFS=,; echo "${builds[*]}")"
+    --warmup 2 --runs 10
+)
+
+declare -A cases
+cases=(
+    [search]="{BUILD}/bin/nix $flake_args search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello"
+    [rebuild]="{BUILD}/bin/nix $flake_args eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
+    [rebuild-lh]="GC_INITIAL_HEAP_SIZE=10g {BUILD}/bin/nix eval $flake_args --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
+    [parse]="{BUILD}/bin/nix $flake_args eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix"
+)
+
+benches=(
+    rebuild
+    rebuild-lh
+    search
+    parse
+)
+
+for k in "${benches[@]}"; do
+    taskset -c 2,3 \
+        chrt -f 50 \
+        hyperfine "${hyperfineArgs[@]}"  --export-json="bench/bench-${k}.json" --export-markdown="bench/bench-${k}.md" "${cases[$k]}"
+done
+
+echo "Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)"
+bench/summarize.jq bench/*.json

bench/configuration.nix

@@ -0,0 +1,325 @@
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
+
+{
+  boot = {
+    initrd = {
+      availableKernelModules = [
+        "xhci_pci"
+        "ahci"
+      ];
+      kernelModules = [ "dm-snapshot" ];
+      luks.devices = {
+        croot = {
+          device = "/dev/sdb";
+          allowDiscards = true;
+        };
+      };
+    };
+    kernelModules = [ "kvm-intel" ];
+    kernelPackages = pkgs.linuxPackages_latest;
+
+    loader = {
+      systemd-boot.enable = true;
+      efi.canTouchEfiVariables = true;
+    };
+  };
+
+  hardware = {
+    enableRedistributableFirmware = true;
+    cpu.intel.updateMicrocode = true;
+    opengl.driSupport32Bit = true;
+    opengl.extraPackages = with pkgs; [
+      vaapiIntel
+      intel-media-driver
+      intel-compute-runtime
+    ];
+  };
+
+  fileSystems = {
+    "/" = {
+      device = "/dev/sda2";
+      fsType = "xfs";
+      options = [ "noatime" ];
+    };
+
+    "/boot" = {
+      device = "/dev/sda1";
+      fsType = "vfat";
+    };
+
+    "/nas" = {
+      device = "nas:/";
+      fsType = "nfs4";
+      options = [
+        "ro"
+        "x-systemd.automount"
+      ];
+    };
+  };
+  swapDevices = [ { device = "/dev/swap"; } ];
+
+  networking = {
+    useDHCP = false;
+    hostName = "host";
+    wireless = {
+      enable = true;
+      interfaces = [ "eth1" ];
+    };
+    interfaces = {
+      eth0.useDHCP = true;
+      eth1.useDHCP = true;
+    };
+    wg-quick.interfaces = {
+      wg0 = {
+        address = [ "2001:db8::1" ];
+        privateKeyFile = "/etc/secrets/wg0.key";
+        peers = [
+          {
+            publicKey = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
+            endpoint = "[2001:db8::2]:61021";
+            allowedIPs = [ "2001::db8:1::/64" ];
+          }
+        ];
+      };
+    };
+
+    firewall.allowedUDPPorts = [ 4567 ];
+  };
+
+  i18n = {
+    defaultLocale = "en_US.UTF-8";
+    inputMethod.enabled = "ibus";
+  };
+
+  services = {
+    xserver = {
+      enable = true;
+      layout = "us";
+      xkbVariant = "altgr-intl";
+      xkbOptions = "ctrl:nocaps";
+      libinput.enable = true;
+      wacom.enable = true;
+      videoDrivers = [ "modesetting" ];
+      modules = [ pkgs.xf86_input_wacom ];
+
+      displayManager.sx.enable = true;
+      windowManager.i3.enable = true;
+    };
+
+    udev.extraHwdb = ''
+      # not like this mattered at all
+      # we're not running udev from here
+    '';
+
+    udev.extraRules = ''
+      # ACTION=="add", SUBSYSTEM=="input", ...
+    '';
+  };
+
+  sound.enable = true;
+  hardware.pulseaudio = {
+    enable = true;
+    package = pkgs.pulseaudioFull;
+    daemon.config = {
+      lock-memory = "yes";
+      realtime-scheduling = "yes";
+      rlimit-rtprio = "-1";
+    };
+  };
+
+  programs = {
+    light.enable = true;
+    wireshark = {
+      enable = true;
+      package = pkgs.wireshark-qt;
+    };
+    gnupg.agent = {
+      enable = true;
+    };
+  };
+
+  fonts.packages = with pkgs; [
+    font-awesome
+    noto-fonts
+    noto-fonts-cjk
+    noto-fonts-emoji
+    noto-fonts-extra
+    dejavu_fonts
+    powerline-fonts
+    source-code-pro
+    cantarell-fonts
+  ];
+
+  users = {
+    mutableUsers = false;
+
+    users = {
+      user = {
+        isNormalUser = true;
+        group = "user";
+        extraGroups = [
+          "wheel"
+          "video"
+          "audio"
+          "dialout"
+          "users"
+          "kvm"
+          "wireshark"
+        ];
+        password = "unimportant";
+      };
+    };
+
+    groups = {
+      user = { };
+    };
+  };
+
+  security = {
+    pam.loginLimits = [
+      {
+        domain = "@audio";
+        item = "memlock";
+        type = "-";
+        value = "unlimited";
+      }
+      {
+        domain = "@audio";
+        item = "rtprio";
+        type = "-";
+        value = "99";
+      }
+      {
+        domain = "@audio";
+        item = "nofile";
+        type = "soft";
+        value = "99999";
+      }
+      {
+        domain = "@audio";
+        item = "nofile";
+        type = "hard";
+        value = "99999";
+      }
+    ];
+
+    sudo.extraRules = [
+      {
+        users = [ "user" ];
+        commands = [
+          {
+            command = "${pkgs.linuxPackages.cpupower}/bin/cpupower";
+            options = [ "NOPASSWD" ];
+          }
+        ];
+      }
+    ];
+  };
+
+  environment.systemPackages = with pkgs; [
+    a2jmidid
+    age
+    ardour
+    bemenu
+    blender
+    breeze-icons
+    breeze-qt5
+    bubblewrap
+    calf
+    claws-mail
+    darktable
+    duperemove
+    emacs
+    feh
+    file
+    firefox
+    fluidsynth
+    gnome3.adwaita-icon-theme
+    gnuplot
+    graphviz
+    helm
+    i3status-rust
+    inkscape
+    jack2
+    jq
+    krita
+    ldns
+    libqalculate
+    libreoffice
+    man-pages
+    nheko
+    nix-diff
+    nix-index
+    nix-output-monitor
+    open-music-kontrollers.patchmatrix
+    pamixer
+    pavucontrol
+    pciutils
+    picom
+    pwgen
+    redshift
+    ripgrep
+    rlwrap
+    silver-searcher
+    soundfont-fluid
+    whois
+    wol
+    xclip
+    xdot
+    xdotool
+    xorg.xkbcomp
+    yt-dlp
+    zathura
+    borgbackup
+    linuxPackages.cpupower
+    mtr
+    kitty
+    xf86_input_wacom
+  ];
+
+  environment.pathsToLink = [ "/share/soundfonts" ];
+
+  systemd.user.services.run-python = {
+    after = [ "network-online.target" ];
+    script = ''
+      exec ${pkgs.python3}/bin/python
+    '';
+    serviceConfig = {
+      CapabilityBoundingSet = [ "" ];
+      KeyringMode = "private";
+      LockPersonality = true;
+      MemoryDenyWriteExecute = true;
+      NoNewPrivileges = true;
+      PrivateDevices = true;
+      PrivateTmp = true;
+      PrivateUsers = true;
+      ProcSubset = "pid";
+      ProtectClock = true;
+      ProtectControlGroups = true;
+      ProtectHome = true;
+      ProtectHostname = true;
+      ProtectKernelLogs = true;
+      ProtectKernelModules = true;
+      ProtectKernelTunables = true;
+      ProtectProc = "invisible";
+      ProtectSystem = "strict";
+      RestrictAddressFamilies = "AF_INET AF_INET6";
+      RestrictNamespaces = true;
+      RestrictRealtime = true;
+      RestrictSUIDSGID = true;
+      SystemCallArchitectures = "native";
+      SystemCallFilter = [
+        "@system-service"
+        "~ @resources @privileged"
+      ];
+      UMask = "077";
+    };
+  };
+
+  system.stateVersion = "23.11";
+}

bench/summarize.jq

@@ -0,0 +1,22 @@
+#!/usr/bin/env -S jq -Mrf
+
+def round3:
+    . * 1000 | round | . / 1000
+    ;
+
+def stats($first):
+    [
+        "  mean:     \(.mean | round3)s ± \(.stddev | round3)s",
+        "            user: \(.user | round3)s | system: \(.system | round3)s",
+        "  median:   \(.median | round3)s",
+        "  range:    \(.min | round3)s ... \(.max | round3)s",
+        "  relative: \(.mean / $first.mean | round3)"
+    ]
+    | join("\n")
+    ;
+
+def fmt($first):
+    "\(.command)\n" + (. | stats($first))
+    ;
+
+[.results | .[0] as $first | .[] | fmt($first)] | join("\n\n") | (. + "\n\n---\n")

clang-tidy/.editorconfig

@@ -0,0 +1,4 @@
+# LLVM style code is 2-space indented
+[*.{cc,hh}]
+indent_style = space
+indent_size = 2

clang-tidy/FixIncludes.cc

@@ -0,0 +1,81 @@
+#include "FixIncludes.hh"
+#include <clang-tidy/ClangTidyCheck.h>
+#include <clang/Basic/Diagnostic.h>
+#include <clang/Basic/SourceManager.h>
+#include <clang/Lex/PPCallbacks.h>
+#include <clang/Lex/Preprocessor.h>
+#include <llvm/ADT/StringRef.h>
+#include <llvm/Support/Debug.h>
+#include <memory>
+#include <set>
+#include <string>
+
+namespace nix::clang_tidy {
+
+using namespace clang;
+using namespace clang::tidy;
+
+class FixIncludesCallbacks : public PPCallbacks {
+public:
+  ClangTidyCheck &Check;
+  Preprocessor &PP;
+  FixIncludesCallbacks(ClangTidyCheck &Check, Preprocessor &PP)
+      : Check(Check), PP(PP) {}
+
+private:
+  bool Ignore = false;
+  virtual void LexedFileChanged(FileID FID, LexedFileChangeReason Reason,
+                                SrcMgr::CharacteristicKind FileType,
+                                FileID PrevFID, SourceLocation Loc) override;
+
+  virtual void InclusionDirective(SourceLocation HashLoc,
+                                  const Token &IncludeTok, StringRef FileName,
+                                  bool IsAngled, CharSourceRange FilenameRange,
+                                  OptionalFileEntryRef File,
+                                  StringRef SearchPath, StringRef RelativePath,
+                                  const Module *Imported,
+                                  SrcMgr::CharacteristicKind FileType) override;
+};
+
+void FixIncludesCallbacks::LexedFileChanged(FileID, LexedFileChangeReason,
+                                            SrcMgr::CharacteristicKind FileType,
+                                            FileID, SourceLocation) {
+  Ignore = FileType != SrcMgr::C_User;
+}
+
+void FixIncludesCallbacks::InclusionDirective(
+    SourceLocation, const Token &, StringRef, bool,
+    CharSourceRange FilenameRange, OptionalFileEntryRef File, StringRef,
+    StringRef, const Module *, SrcMgr::CharacteristicKind) {
+  if (Ignore)
+    return;
+
+  // FIXME: this is kinda evil, but this is a one-time fixup
+  const std::string SourceDir = "src/";
+
+  if (File && File->getNameAsRequested().contains(SourceDir)) {
+    StringRef Name = File->getNameAsRequested();
+    auto Idx = Name.find(SourceDir);
+    assert(Idx != std::string::npos);
+    StringRef Suffix = Name.drop_front(Idx + SourceDir.length());
+
+    if (!Suffix.starts_with("lib")) {
+      llvm::dbgs() << "ignored: " << Suffix << "\n";
+      return;
+    }
+
+    auto Diag = Check.diag(FilenameRange.getBegin(),
+                           "include needs to specify the source subdir");
+
+    Diag << FilenameRange
+         << FixItHint::CreateReplacement(FilenameRange,
+                                         ("\"" + Suffix + "\"").str());
+  }
+}
+
+void FixIncludesCheck::registerPPCallbacks(const SourceManager &,
+                                           Preprocessor *PP, Preprocessor *) {
+  PP->addPPCallbacks(std::make_unique<FixIncludesCallbacks>(*this, *PP));
+}
+
+}; // namespace nix::clang_tidy

clang-tidy/FixIncludes.hh

@@ -0,0 +1,21 @@
+#pragma once
+///@file
+
+#include <clang-tidy/ClangTidyCheck.h>
+#include <clang/ASTMatchers/ASTMatchFinder.h>
+#include <llvm/ADT/StringRef.h>
+
+namespace nix::clang_tidy {
+
+using namespace clang;
+using namespace clang::tidy;
+
+class FixIncludesCheck : public ClangTidyCheck {
+    public:
+    FixIncludesCheck(StringRef Name, ClangTidyContext *Context)
+            : ClangTidyCheck(Name, Context) {}
+
+    void registerPPCallbacks(const SourceManager &SM, Preprocessor *PP, Preprocessor *ModuleExpanderPP) override;
+};
+
+};

clang-tidy/LixClangTidyChecks.cc

@@ -1,5 +1,6 @@
 #include <clang-tidy/ClangTidyModule.h>
 #include <clang-tidy/ClangTidyModuleRegistry.h>
+#include "FixIncludes.hh"
 #include "HasPrefixSuffix.hh"
 
 namespace nix::clang_tidy {
@@ -9,9 +10,10 @@ using namespace clang::tidy;
 class NixClangTidyChecks : public ClangTidyModule {
     public:
         void addCheckFactories(ClangTidyCheckFactories &CheckFactories) override {
-            CheckFactories.registerCheck<HasPrefixSuffixCheck>("nix-hasprefixsuffix");
+            CheckFactories.registerCheck<HasPrefixSuffixCheck>("lix-hasprefixsuffix");
+            CheckFactories.registerCheck<FixIncludesCheck>("lix-fixincludes");
         }
 };
 
-static ClangTidyModuleRegistry::Add<NixClangTidyChecks> X("nix-module", "Adds nix specific checks");
+static ClangTidyModuleRegistry::Add<NixClangTidyChecks> X("lix-module", "Adds lix specific checks");
 };

clang-tidy/meson.build

@@ -1,8 +1,13 @@
-project('nix-clang-tidy', ['cpp', 'c'],
+project('lix-clang-tidy', ['cpp', 'c'],
   version : '0.1',
   default_options : ['warning_level=3', 'cpp_std=c++20'])
 
 llvm = dependency('Clang', version: '>= 14', modules: ['libclang'])
-sources = ['HasPrefixSuffix.cc', 'NixClangTidyChecks.cc']
+sources = files(
-shared_module('nix-clang-tidy', sources,
+  'HasPrefixSuffix.cc',
+  'LixClangTidyChecks.cc',
+  'FixIncludes.cc',
+)
+
+shared_module('lix-clang-tidy', sources,
   dependencies: llvm)

config/install-sh

@@ -1,527 +0,0 @@
-#!/bin/sh
-# install - install a program, script, or datafile
-
-scriptversion=2011-11-20.07; # UTC
-
-# This originates from X11R5 (mit/util/scripts/install.sh), which was
-# later released in X11R6 (xc/config/util/install.sh) with the
-# following copyright and license.
-#
-# Copyright (C) 1994 X Consortium
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to
-# deal in the Software without restriction, including without limitation the
-# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-# sell copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
-# X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
-# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNEC-
-# TION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-#
-# Except as contained in this notice, the name of the X Consortium shall not
-# be used in advertising or otherwise to promote the sale, use or other deal-
-# ings in this Software without prior written authorization from the X Consor-
-# tium.
-#
-#
-# FSF changes to this file are in the public domain.
-#
-# Calling this script install-sh is preferred over install.sh, to prevent
-# 'make' implicit rules from creating a file called install from it
-# when there is no Makefile.
-#
-# This script is compatible with the BSD install script, but was written
-# from scratch.
-
-nl='
-'
-IFS=" ""	$nl"
-
-# set DOITPROG to echo to test this script
-
-# Don't use :- since 4.3BSD and earlier shells don't like it.
-doit=${DOITPROG-}
-if test -z "$doit"; then
-  doit_exec=exec
-else
-  doit_exec=$doit
-fi
-
-# Put in absolute file names if you don't have them in your path;
-# or use environment vars.
-
-chgrpprog=${CHGRPPROG-chgrp}
-chmodprog=${CHMODPROG-chmod}
-chownprog=${CHOWNPROG-chown}
-cmpprog=${CMPPROG-cmp}
-cpprog=${CPPROG-cp}
-mkdirprog=${MKDIRPROG-mkdir}
-mvprog=${MVPROG-mv}
-rmprog=${RMPROG-rm}
-stripprog=${STRIPPROG-strip}
-
-posix_glob='?'
-initialize_posix_glob='
-  test "$posix_glob" != "?" || {
-    if (set -f) 2>/dev/null; then
-      posix_glob=
-    else
-      posix_glob=:
-    fi
-  }
-'
-
-posix_mkdir=
-
-# Desired mode of installed file.
-mode=0755
-
-chgrpcmd=
-chmodcmd=$chmodprog
-chowncmd=
-mvcmd=$mvprog
-rmcmd="$rmprog -f"
-stripcmd=
-
-src=
-dst=
-dir_arg=
-dst_arg=
-
-copy_on_change=false
-no_target_directory=
-
-usage="\
-Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE
-   or: $0 [OPTION]... SRCFILES... DIRECTORY
-   or: $0 [OPTION]... -t DIRECTORY SRCFILES...
-   or: $0 [OPTION]... -d DIRECTORIES...
-
-In the 1st form, copy SRCFILE to DSTFILE.
-In the 2nd and 3rd, copy all SRCFILES to DIRECTORY.
-In the 4th, create DIRECTORIES.
-
-Options:
-     --help     display this help and exit.
-     --version  display version info and exit.
-
-  -c            (ignored)
-  -C            install only if different (preserve the last data modification time)
-  -d            create directories instead of installing files.
-  -g GROUP      $chgrpprog installed files to GROUP.
-  -m MODE       $chmodprog installed files to MODE.
-  -o USER       $chownprog installed files to USER.
-  -s            $stripprog installed files.
-  -t DIRECTORY  install into DIRECTORY.
-  -T            report an error if DSTFILE is a directory.
-
-Environment variables override the default commands:
-  CHGRPPROG CHMODPROG CHOWNPROG CMPPROG CPPROG MKDIRPROG MVPROG
-  RMPROG STRIPPROG
-"
-
-while test $# -ne 0; do
-  case $1 in
-    -c) ;;
-
-    -C) copy_on_change=true;;
-
-    -d) dir_arg=true;;
-
-    -g) chgrpcmd="$chgrpprog $2"
-	shift;;
-
-    --help) echo "$usage"; exit $?;;
-
-    -m) mode=$2
-	case $mode in
-	  *' '* | *'	'* | *'
-'*	  | *'*'* | *'?'* | *'['*)
-	    echo "$0: invalid mode: $mode" >&2
-	    exit 1;;
-	esac
-	shift;;
-
-    -o) chowncmd="$chownprog $2"
-	shift;;
-
-    -s) stripcmd=$stripprog;;
-
-    -t) dst_arg=$2
-	# Protect names problematic for 'test' and other utilities.
-	case $dst_arg in
-	  -* | [=\(\)!]) dst_arg=./$dst_arg;;
-	esac
-	shift;;
-
-    -T) no_target_directory=true;;
-
-    --version) echo "$0 $scriptversion"; exit $?;;
-
-    --)	shift
-	break;;
-
-    -*)	echo "$0: invalid option: $1" >&2
-	exit 1;;
-
-    *)  break;;
-  esac
-  shift
-done
-
-if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then
-  # When -d is used, all remaining arguments are directories to create.
-  # When -t is used, the destination is already specified.
-  # Otherwise, the last argument is the destination.  Remove it from $@.
-  for arg
-  do
-    if test -n "$dst_arg"; then
-      # $@ is not empty: it contains at least $arg.
-      set fnord "$@" "$dst_arg"
-      shift # fnord
-    fi
-    shift # arg
-    dst_arg=$arg
-    # Protect names problematic for 'test' and other utilities.
-    case $dst_arg in
-      -* | [=\(\)!]) dst_arg=./$dst_arg;;
-    esac
-  done
-fi
-
-if test $# -eq 0; then
-  if test -z "$dir_arg"; then
-    echo "$0: no input file specified." >&2
-    exit 1
-  fi
-  # It's OK to call 'install-sh -d' without argument.
-  # This can happen when creating conditional directories.
-  exit 0
-fi
-
-if test -z "$dir_arg"; then
-  do_exit='(exit $ret); exit $ret'
-  trap "ret=129; $do_exit" 1
-  trap "ret=130; $do_exit" 2
-  trap "ret=141; $do_exit" 13
-  trap "ret=143; $do_exit" 15
-
-  # Set umask so as not to create temps with too-generous modes.
-  # However, 'strip' requires both read and write access to temps.
-  case $mode in
-    # Optimize common cases.
-    *644) cp_umask=133;;
-    *755) cp_umask=22;;
-
-    *[0-7])
-      if test -z "$stripcmd"; then
-	u_plus_rw=
-      else
-	u_plus_rw='% 200'
-      fi
-      cp_umask=`expr '(' 777 - $mode % 1000 ')' $u_plus_rw`;;
-    *)
-      if test -z "$stripcmd"; then
-	u_plus_rw=
-      else
-	u_plus_rw=,u+rw
-      fi
-      cp_umask=$mode$u_plus_rw;;
-  esac
-fi
-
-for src
-do
-  # Protect names problematic for 'test' and other utilities.
-  case $src in
-    -* | [=\(\)!]) src=./$src;;
-  esac
-
-  if test -n "$dir_arg"; then
-    dst=$src
-    dstdir=$dst
-    test -d "$dstdir"
-    dstdir_status=$?
-  else
-
-    # Waiting for this to be detected by the "$cpprog $src $dsttmp" command
-    # might cause directories to be created, which would be especially bad
-    # if $src (and thus $dsttmp) contains '*'.
-    if test ! -f "$src" && test ! -d "$src"; then
-      echo "$0: $src does not exist." >&2
-      exit 1
-    fi
-
-    if test -z "$dst_arg"; then
-      echo "$0: no destination specified." >&2
-      exit 1
-    fi
-    dst=$dst_arg
-
-    # If destination is a directory, append the input filename; won't work
-    # if double slashes aren't ignored.
-    if test -d "$dst"; then
-      if test -n "$no_target_directory"; then
-	echo "$0: $dst_arg: Is a directory" >&2
-	exit 1
-      fi
-      dstdir=$dst
-      dst=$dstdir/`basename "$src"`
-      dstdir_status=0
-    else
-      # Prefer dirname, but fall back on a substitute if dirname fails.
-      dstdir=`
-	(dirname "$dst") 2>/dev/null ||
-	expr X"$dst" : 'X\(.*[^/]\)//*[^/][^/]*/*$' \| \
-	     X"$dst" : 'X\(//\)[^/]' \| \
-	     X"$dst" : 'X\(//\)$' \| \
-	     X"$dst" : 'X\(/\)' \| . 2>/dev/null ||
-	echo X"$dst" |
-	    sed '/^X\(.*[^/]\)\/\/*[^/][^/]*\/*$/{
-		   s//\1/
-		   q
-		 }
-		 /^X\(\/\/\)[^/].*/{
-		   s//\1/
-		   q
-		 }
-		 /^X\(\/\/\)$/{
-		   s//\1/
-		   q
-		 }
-		 /^X\(\/\).*/{
-		   s//\1/
-		   q
-		 }
-		 s/.*/./; q'
-      `
-
-      test -d "$dstdir"
-      dstdir_status=$?
-    fi
-  fi
-
-  obsolete_mkdir_used=false
-
-  if test $dstdir_status != 0; then
-    case $posix_mkdir in
-      '')
-	# Create intermediate dirs using mode 755 as modified by the umask.
-	# This is like FreeBSD 'install' as of 1997-10-28.
-	umask=`umask`
-	case $stripcmd.$umask in
-	  # Optimize common cases.
-	  *[2367][2367]) mkdir_umask=$umask;;
-	  .*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;;
-
-	  *[0-7])
-	    mkdir_umask=`expr $umask + 22 \
-	      - $umask % 100 % 40 + $umask % 20 \
-	      - $umask % 10 % 4 + $umask % 2
-	    `;;
-	  *) mkdir_umask=$umask,go-w;;
-	esac
-
-	# With -d, create the new directory with the user-specified mode.
-	# Otherwise, rely on $mkdir_umask.
-	if test -n "$dir_arg"; then
-	  mkdir_mode=-m$mode
-	else
-	  mkdir_mode=
-	fi
-
-	posix_mkdir=false
-	case $umask in
-	  *[123567][0-7][0-7])
-	    # POSIX mkdir -p sets u+wx bits regardless of umask, which
-	    # is incompatible with FreeBSD 'install' when (umask & 300) != 0.
-	    ;;
-	  *)
-	    tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$
-	    trap 'ret=$?; rmdir "$tmpdir/d" "$tmpdir" 2>/dev/null; exit $ret' 0
-
-	    if (umask $mkdir_umask &&
-		exec $mkdirprog $mkdir_mode -p -- "$tmpdir/d") >/dev/null 2>&1
-	    then
-	      if test -z "$dir_arg" || {
-		   # Check for POSIX incompatibilities with -m.
-		   # HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or
-		   # other-writable bit of parent directory when it shouldn't.
-		   # FreeBSD 6.1 mkdir -m -p sets mode of existing directory.
-		   ls_ld_tmpdir=`ls -ld "$tmpdir"`
-		   case $ls_ld_tmpdir in
-		     d????-?r-*) different_mode=700;;
-		     d????-?--*) different_mode=755;;
-		     *) false;;
-		   esac &&
-		   $mkdirprog -m$different_mode -p -- "$tmpdir" && {
-		     ls_ld_tmpdir_1=`ls -ld "$tmpdir"`
-		     test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1"
-		   }
-		 }
-	      then posix_mkdir=:
-	      fi
-	      rmdir "$tmpdir/d" "$tmpdir"
-	    else
-	      # Remove any dirs left behind by ancient mkdir implementations.
-	      rmdir ./$mkdir_mode ./-p ./-- 2>/dev/null
-	    fi
-	    trap '' 0;;
-	esac;;
-    esac
-
-    if
-      $posix_mkdir && (
-	umask $mkdir_umask &&
-	$doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir"
-      )
-    then :
-    else
-
-      # The umask is ridiculous, or mkdir does not conform to POSIX,
-      # or it failed possibly due to a race condition.  Create the
-      # directory the slow way, step by step, checking for races as we go.
-
-      case $dstdir in
-	/*) prefix='/';;
-	[-=\(\)!]*) prefix='./';;
-	*)  prefix='';;
-      esac
-
-      eval "$initialize_posix_glob"
-
-      oIFS=$IFS
-      IFS=/
-      $posix_glob set -f
-      set fnord $dstdir
-      shift
-      $posix_glob set +f
-      IFS=$oIFS
-
-      prefixes=
-
-      for d
-      do
-	test X"$d" = X && continue
-
-	prefix=$prefix$d
-	if test -d "$prefix"; then
-	  prefixes=
-	else
-	  if $posix_mkdir; then
-	    (umask=$mkdir_umask &&
-	     $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break
-	    # Don't fail if two instances are running concurrently.
-	    test -d "$prefix" || exit 1
-	  else
-	    case $prefix in
-	      *\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;;
-	      *) qprefix=$prefix;;
-	    esac
-	    prefixes="$prefixes '$qprefix'"
-	  fi
-	fi
-	prefix=$prefix/
-      done
-
-      if test -n "$prefixes"; then
-	# Don't fail if two instances are running concurrently.
-	(umask $mkdir_umask &&
-	 eval "\$doit_exec \$mkdirprog $prefixes") ||
-	  test -d "$dstdir" || exit 1
-	obsolete_mkdir_used=true
-      fi
-    fi
-  fi
-
-  if test -n "$dir_arg"; then
-    { test -z "$chowncmd" || $doit $chowncmd "$dst"; } &&
-    { test -z "$chgrpcmd" || $doit $chgrpcmd "$dst"; } &&
-    { test "$obsolete_mkdir_used$chowncmd$chgrpcmd" = false ||
-      test -z "$chmodcmd" || $doit $chmodcmd $mode "$dst"; } || exit 1
-  else
-
-    # Make a couple of temp file names in the proper directory.
-    dsttmp=$dstdir/_inst.$$_
-    rmtmp=$dstdir/_rm.$$_
-
-    # Trap to clean up those temp files at exit.
-    trap 'ret=$?; rm -f "$dsttmp" "$rmtmp" && exit $ret' 0
-
-    # Copy the file name to the temp name.
-    (umask $cp_umask && $doit_exec $cpprog "$src" "$dsttmp") &&
-
-    # and set any options; do chmod last to preserve setuid bits.
-    #
-    # If any of these fail, we abort the whole thing.  If we want to
-    # ignore errors from any of these, just make sure not to ignore
-    # errors from the above "$doit $cpprog $src $dsttmp" command.
-    #
-    { test -z "$chowncmd" || $doit $chowncmd "$dsttmp"; } &&
-    { test -z "$chgrpcmd" || $doit $chgrpcmd "$dsttmp"; } &&
-    { test -z "$stripcmd" || $doit $stripcmd "$dsttmp"; } &&
-    { test -z "$chmodcmd" || $doit $chmodcmd $mode "$dsttmp"; } &&
-
-    # If -C, don't bother to copy if it wouldn't change the file.
-    if $copy_on_change &&
-       old=`LC_ALL=C ls -dlL "$dst"	2>/dev/null` &&
-       new=`LC_ALL=C ls -dlL "$dsttmp"	2>/dev/null` &&
-
-       eval "$initialize_posix_glob" &&
-       $posix_glob set -f &&
-       set X $old && old=:$2:$4:$5:$6 &&
-       set X $new && new=:$2:$4:$5:$6 &&
-       $posix_glob set +f &&
-
-       test "$old" = "$new" &&
-       $cmpprog "$dst" "$dsttmp" >/dev/null 2>&1
-    then
-      rm -f "$dsttmp"
-    else
-      # Rename the file to the real destination.
-      $doit $mvcmd -f "$dsttmp" "$dst" 2>/dev/null ||
-
-      # The rename failed, perhaps because mv can't rename something else
-      # to itself, or perhaps because mv is so ancient that it does not
-      # support -f.
-      {
-	# Now remove or move aside any old file at destination location.
-	# We try this two ways since rm can't unlink itself on some
-	# systems and the destination file might be busy for other
-	# reasons.  In this case, the final cleanup might fail but the new
-	# file should still install successfully.
-	{
-	  test ! -f "$dst" ||
-	  $doit $rmcmd -f "$dst" 2>/dev/null ||
-	  { $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null &&
-	    { $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; }
-	  } ||
-	  { echo "$0: cannot unlink or rename $dst" >&2
-	    (exit 1); exit 1
-	  }
-	} &&
-
-	# Now rename the file to the real destination.
-	$doit $mvcmd "$dsttmp" "$dst"
-      }
-    fi || exit 1
-
-    trap '' 0
-  fi
-done
-
-# Local variables:
-# eval: (add-hook 'write-file-hooks 'time-stamp)
-# time-stamp-start: "scriptversion="
-# time-stamp-format: "%:y-%02m-%02d.%02H"
-# time-stamp-time-zone: "UTC"
-# time-stamp-end: "; # UTC"
-# End:

default.nix

@@ -1,10 +1,9 @@
-(import
+(import (
-  (
+  let
-    let lock = builtins.fromJSON (builtins.readFile ./flake.lock); in
+    lock = builtins.fromJSON (builtins.readFile ./flake.lock);
+  in
   fetchTarball {
     url = "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz";
     sha256 = lock.nodes.flake-compat.locked.narHash;
   }
-  )
+) { src = ./.; }).defaultNix
-  { src = ./.; }
-).defaultNix

doc/internal-api/doxygen.cfg.in

@@ -14,6 +14,8 @@ PROJECT_NAME           = "Nix"
 
 PROJECT_NUMBER         = @PACKAGE_VERSION@
 
+OUTPUT_DIRECTORY       = @docdir@
+
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a
 # quick idea about the purpose of the project. Keep the description short.

doc/internal-api/meson.build

@@ -0,0 +1,33 @@
+doxygen_cfg = configure_file(
+  input : 'doxygen.cfg.in',
+  output : 'doxygen.cfg',
+  configuration : {
+    'PACKAGE_VERSION': meson.project_version(),
+    'RAPIDCHECK_HEADERS': rapidcheck_meson.get_variable('includedir'),
+    'docdir' : meson.current_build_dir(),
+  },
+)
+
+internal_api_docs = custom_target(
+  'internal-api-docs',
+  command : [
+    bash,
+    # Meson can you please just give us a `workdir` argument to custom targets...
+    '-c',
+    # We have to prefix the doxygen_cfg path with the project build root
+    # because of the cd in front.
+    'cd @0@ && @1@ @2@/@INPUT0@'.format(
+      meson.project_source_root(),
+      doxygen.full_path(),
+      meson.project_build_root(),
+    ),
+  ],
+  input : [
+    doxygen_cfg,
+  ],
+  output : 'html',
+  install : true,
+  install_dir : datadir / 'doc/nix/internal-api',
+)
+
+alias_target('internal-api-html', internal_api_docs)

doc/manual/book.toml

@@ -7,10 +7,22 @@ additional-js = ["redirects.js"]
 edit-url-template = "https://github.com/NixOS/nix/tree/master/doc/manual/{path}"
 git-repository-url = "https://github.com/NixOS/nix"
 
+# Handles replacing @docroot@ with a path to ./src relative to that markdown file,
+# {{#include handlebars}}, and the @generated@ syntax used within these. it mostly
+# but not entirely replaces the links preprocessor (which we cannot simply use due
+# to @generated@ files living in a different directory to make meson happy). we do
+# not want to disable the links preprocessor entirely though because that requires
+# disabling *all* built-in preprocessors and selectively reenabling those we want.
+[preprocessor.substitute]
+command = "python3 doc/manual/substitute.py"
+before = ["anchors", "links"]
+
 [preprocessor.anchors]
 renderers = ["html"]
 command = "jq --from-file doc/manual/anchors.jq"
 
+[output.markdown]
+
 [output.linkcheck]
 # no Internet during the build (in the sandbox)
 follow-web-links = false

doc/manual/generate-builtin-constants.nix

@@ -5,7 +5,13 @@ in
 
 builtinsInfo:
 let
-  showBuiltin = name: { doc, type, impure-only }:
+  showBuiltin =
+    name:
+    {
+      doc,
+      type,
+      impure-only,
+    }:
     let
       type' = optionalString (type != null) " (${type})";
 

doc/manual/generate-builtins.nix

@@ -5,7 +5,14 @@ in
 
 builtinsInfo:
 let
-  showBuiltin = name: { doc, args, arity, experimental-feature }:
+  showBuiltin =
+    name:
+    {
+      doc,
+      args,
+      arity,
+      experimental-feature,
+    }:
     let
       experimentalNotice = optionalString (experimental-feature != null) ''
         This function is only available if the [${experimental-feature}](@docroot@/contributing/experimental-features.md#xp-feature-${experimental-feature}) experimental feature is enabled.

doc/manual/generate-deps.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+
+import glob
+import sys
+
+# meson expects makefile-style dependency declarations, i.e.
+#
+#   target: dependency...
+#
+# meson seems to pass depfiles straight on to ninja even though
+# it also parses the file itself (or at least has code to do so
+# in its tree), so we must live by ninja's rules: only slashes,
+# spaces and octothorpes can be escaped, anything else is taken
+# literally. since the rules for these aren't even the same for
+# all three we will just fail when we encounter any of them (if
+# asserts are off for some reason the depfile will likely point
+# to nonexistant paths, making everything phony and thus fine.)
+for path in glob.glob(sys.argv[1] + '/**', recursive=True):
+    assert '\\' not in path
+    assert ' ' not in path
+    assert '#' not in path
+    print("ignored:", path)

doc/manual/generate-manpage.nix

@@ -1,8 +1,26 @@
 let
   inherit (builtins)
-    attrNames attrValues fromJSON listToAttrs mapAttrs
+    attrNames
-    concatStringsSep concatMap length lessThan replaceStrings sort;
+    attrValues
-  inherit (import ./utils.nix) concatStrings optionalString filterAttrs trim squash unique showSettings;
+    fromJSON
+    listToAttrs
+    mapAttrs
+    concatStringsSep
+    concatMap
+    length
+    lessThan
+    replaceStrings
+    sort
+    ;
+  inherit (import ./utils.nix)
+    concatStrings
+    optionalString
+    filterAttrs
+    trim
+    squash
+    unique
+    showSettings
+    ;
 in
 
 inlineHTML: commandDump:
@@ -11,7 +29,13 @@ let
 
   commandInfo = fromJSON commandDump;
 
-  showCommand = { command, details, filename, toplevel }:
+  showCommand =
+    {
+      command,
+      details,
+      filename,
+      toplevel,
+    }:
     let
 
       result = ''
@@ -35,26 +59,27 @@ let
         ${maybeOptions}
       '';
 
-      showSynopsis = command: args:
+      showSynopsis =
+        command: args:
         let
-          showArgument = arg: "*${arg.label}*" + optionalString (! arg ? arity) "...";
+          showArgument = arg: "*${arg.label}*" + optionalString (!arg ? arity) "...";
           arguments = concatStringsSep " " (map showArgument args);
-        in ''
+        in
+        ''
           `${command}` [*option*...] ${arguments}
         '';
 
-      maybeSubcommands = optionalString (details ? commands && details.commands != {})
+      maybeSubcommands = optionalString (details ? commands && details.commands != { }) ''
-         ''
         where *subcommand* is one of the following:
 
         ${subcommands}
       '';
 
-      subcommands = if length categories > 1
+      subcommands = if length categories > 1 then listCategories else listSubcommands details.commands;
-        then listCategories
-        else listSubcommands details.commands;
 
-      categories = sort (x: y: x.id < y.id) (unique (map (cmd: cmd.category) (attrValues details.commands)));
+      categories = sort (x: y: x.id < y.id) (
+        unique (map (cmd: cmd.category) (attrValues details.commands))
+      );
 
       listCategories = concatStrings (map showCategory categories);
 
@@ -71,11 +96,11 @@ let
       '';
 
       # TODO: move this confusing special case out of here when implementing #8496
-      maybeStoreDocs = optionalString
+      maybeStoreDocs = optionalString (details ? doc) (
-        (details ? doc)
+        replaceStrings [ "@stores@" ] [ storeDocs ] details.doc
-        (replaceStrings ["@stores@"] [storeDocs] details.doc);
+      );
 
-      maybeOptions = optionalString (details.flags != {}) ''
+      maybeOptions = optionalString (details.flags != { }) ''
         # Options
 
         ${showOptions details.flags toplevel.flags}
@@ -85,7 +110,8 @@ let
         > See [`man nix.conf`](@docroot@/command-ref/conf-file.md#command-line-flags) for overriding configuration settings with command line flags.
       '';
 
-      showOptions = options: commonOptions:
+      showOptions =
+        options: commonOptions:
         let
           allOptions = options // commonOptions;
           showCategory = cat: ''
@@ -94,42 +120,60 @@ let
             ${listOptions (filterAttrs (n: v: v.category == cat) allOptions)}
           '';
           listOptions = opts: concatStringsSep "\n" (attrValues (mapAttrs showOption opts));
-          showOption = name: option:
+          showOption =
+            name: option:
             let
               result = trim ''
                 - ${item}
                   ${option.description}
               '';
-              item = if inlineHTML
+              item =
-                then ''<span id="opt-${name}">[`--${name}`](#opt-${name})</span> ${shortName} ${labels}''
+                if inlineHTML then
-                else "`--${name}` ${shortName} ${labels}";
+                  ''<span id="opt-${name}">[`--${name}`](#opt-${name})</span> ${shortName} ${labels}''
-              shortName = optionalString
+                else
-                (option ? shortName)
+                  "`--${name}` ${shortName} ${labels}";
-                ("/ `-${option.shortName}`");
+              shortName = optionalString (option ? shortName) ("/ `-${option.shortName}`");
-              labels = optionalString
+              labels = optionalString (option ? labels) (concatStringsSep " " (map (s: "*${s}*") option.labels));
-                (option ? labels)
+            in
-                (concatStringsSep " " (map (s: "*${s}*") option.labels));
+            result;
-            in result;
           categories = sort lessThan (unique (map (cmd: cmd.category) (attrValues allOptions)));
-        in concatStrings (map showCategory categories);
+        in
-    in squash result;
+        concatStrings (map showCategory categories);
+    in
+    squash result;
 
   appendName = filename: name: (if filename == "nix" then "nix3" else filename) + "-" + name;
 
-  processCommand = { command, details, filename, toplevel }:
+  processCommand =
+    {
+      command,
+      details,
+      filename,
+      toplevel,
+    }:
     let
       cmd = {
         inherit command;
         name = filename + ".md";
-        value = showCommand { inherit command details filename toplevel; };
+        value = showCommand {
+          inherit
+            command
+            details
+            filename
+            toplevel
+            ;
         };
-      subcommand = subCmd: processCommand {
+      };
+      subcommand =
+        subCmd:
+        processCommand {
           command = command + " " + subCmd;
           details = details.commands.${subCmd};
           filename = appendName filename subCmd;
           inherit toplevel;
         };
-    in [ cmd ] ++ concatMap subcommand (attrNames details.commands or {});
+    in
+    [ cmd ] ++ concatMap subcommand (attrNames details.commands or { });
 
   manpages = processCommand {
     command = "nix";
@@ -138,14 +182,15 @@ let
     toplevel = commandInfo.args;
   };
 
-  tableOfContents = let
-    showEntry = page:
-      "    - [${page.command}](command-ref/new-cli/${page.name})";
-    in concatStringsSep "\n" (map showEntry manpages) + "\n";
-
   storeDocs =
     let
-      showStore = name: { settings, doc, experimentalFeature }:
+      showStore =
+        name:
+        {
+          settings,
+          doc,
+          experimentalFeature,
+        }:
         let
           experimentalFeatureNote = optionalString (experimentalFeature != null) ''
             > **Warning**
@@ -161,7 +206,8 @@ let
             extra-experimental-features = ${experimentalFeature}
             ```
           '';
-        in ''
+        in
+        ''
           ## ${name}
 
           ${doc}
@@ -172,6 +218,7 @@ let
 
           ${showSettings { inherit inlineHTML; } settings}
         '';
-    in concatStrings (attrValues (mapAttrs showStore commandInfo.stores));
+    in
-
+    concatStrings (attrValues (mapAttrs showStore commandInfo.stores));
-in (listToAttrs manpages) // { "SUMMARY.md" = tableOfContents; }
+in
+listToAttrs manpages

doc/manual/generate-xp-features-shortlist.nix

@@ -2,8 +2,8 @@ with builtins;
 with import ./utils.nix;
 
 let
-  showExperimentalFeature = name: doc:
+  showExperimentalFeature = name: doc: ''
-    ''
     - [`${name}`](@docroot@/contributing/experimental-features.md#xp-feature-${name})
   '';
-in xps: indent "  " (concatStrings (attrValues (mapAttrs showExperimentalFeature xps)))
+in
+xps: indent "  " (concatStrings (attrValues (mapAttrs showExperimentalFeature xps)))

doc/manual/generate-xp-features.nix

@@ -2,10 +2,12 @@ with builtins;
 with import ./utils.nix;
 
 let
-  showExperimentalFeature = name: doc:
+  showExperimentalFeature =
+    name: doc:
     squash ''
       ## [`${name}`]{#xp-feature-${name}}
 
       ${doc}
     '';
-in xps: (concatStringsSep "\n" (attrValues (mapAttrs showExperimentalFeature xps)))
+in
+xps: (concatStringsSep "\n" (attrValues (mapAttrs showExperimentalFeature xps)))

doc/manual/json-to-tree.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+
+"""
+This script is a helper for this project's Meson buildsystem, to replace its
+usage of `nix eval --write-to`. Writing a JSON object as a nested directory
+tree is more generic, easier to maintain, and far, far less cursed. Nix
+has 'good' support for JSON output. Let's just use it.
+"""
+
+import argparse
+from pathlib import Path
+import json
+import sys
+
+name = 'json-to-tree.py'
+
+def log(*args, **kwargs):
+    kwargs['file'] = sys.stderr
+    return print(f'{name}:', *args, **kwargs)
+
+def write_dict_to_directory(current_directory: Path, data: dict, files_written=0):
+    current_directory.mkdir(parents=True, exist_ok=True)
+    for key, value in data.items():
+        nested_path = current_directory / key
+        match value:
+            case dict(nested_data):
+                files_written += write_dict_to_directory(nested_path, nested_data)
+
+            case str(content):
+                nested_path.write_text(content)
+                files_written += 1
+
+            case rest:
+                assert False, \
+                    f'should have been called on a dict or string, not {type(rest)=}\n\t{rest=}'
+
+    return files_written
+
+def main():
+    parser = argparse.ArgumentParser(name)
+    parser.add_argument('-i', '--input', type=argparse.FileType('r'), default='-',
+        help='The JSON input to operate on and output as a directory tree',
+    )
+    parser.add_argument('-o', '--output', type=Path, required=True,
+        help='The place to put the directory tree',
+    )
+    args = parser.parse_args()
+
+    json_string = args.input.read()
+
+    try:
+        data = json.loads(json_string)
+    except json.JSONDecodeError:
+        log(f'could not decode JSON from input: {json_string}')
+        raise
+
+
+    files_written = write_dict_to_directory(args.output, data)
+    log(f'wrote {files_written} files')
+
+sys.exit(main())

doc/manual/local.mk

@@ -17,14 +17,14 @@ man-pages := $(foreach n, \
 	nix-hash.1 nix-copy-closure.1 \
 	nix.conf.5 nix-daemon.8 \
 	nix-profiles.5 \
-, $(d)/$(n))
+, doc/manual/generated/in/$(n))
 
 # man pages for subcommands
 # convert from `$(d)/src/command-ref/nix-{1}/{2}.md` to `$(d)/nix-{1}-{2}.1`
 # FIXME: unify with how nix3-cli man pages are generated
 man-pages += $(foreach subcommand, \
 	$(filter-out %opt-common.md %env-common.md, $(wildcard $(d)/src/command-ref/nix-*/*.md)), \
-	$(d)/$(subst /,-,$(subst $(d)/src/command-ref/,,$(subst .md,.1,$(subcommand)))))
+	doc/manual/generated/in/$(subst /,-,$(subst $(d)/src/command-ref/,,$(subst .md,.1,$(subcommand)))))
 
 clean-files += $(d)/*.1 $(d)/*.5 $(d)/*.8
 
@@ -39,118 +39,91 @@ dummy-env = env -i \
 
 nix-eval = $(dummy-env) $(doc_nix) eval --experimental-features nix-command -I nix/corepkgs=corepkgs --store dummy:// --impure --raw
 
-# re-implement mdBook's include directive to make it usable for terminal output and for proper @docroot@ substitution
+doc/manual/generated/in/nix-env-%.1: doc/manual/generated/out
-define process-includes
+	$(trace-gen) doc/manual/render-manpage.sh \
-	while read -r line; do \
+		--out-no-smarty "$(subst nix-env-,nix-env --,$$(basename "$@" .1))" 1 \
-		set -euo pipefail; \
+		doc/manual/generated/out/markdown/command-ref/nix-env/$*.md \
-		filename="$$(dirname $(1))/$$(sed 's/{{#include \(.*\)}}/\1/'<<< $$line)"; \
+		$@
-		test -f "$$filename" || ( echo "#include-d file '$$filename' does not exist." >&2; exit 1; ); \
-		matchline="$$(sed 's|/|\\/|g' <<< $$line)"; \
-		sed -i "/$$matchline/r $$filename" $(2); \
-		sed -i "s/$$matchline//" $(2); \
-	done < <(grep '{{#include' $(1))
-endef
 
-$(d)/nix-env-%.1: $(d)/src/command-ref/nix-env/%.md
+doc/manual/generated/in/nix-store-%.1: doc/manual/generated/out
-	@printf "Title: %s\n\n" "$(subst nix-env-,nix-env --,$$(basename "$@" .1))" > $^.tmp
+	$(trace-gen) doc/manual/render-manpage.sh \
-	$(render-subcommand)
+		--out-no-smarty "$(subst nix-store-,nix-store --,$$(basename "$@" .1))" 1 \
-
+		doc/manual/generated/out/markdown/command-ref/nix-store/$*.md \
-$(d)/nix-store-%.1: $(d)/src/command-ref/nix-store/%.md
+		$@
-	@printf -- 'Title: %s\n\n' "$(subst nix-store-,nix-store --,$$(basename "$@" .1))" > $^.tmp
-	$(render-subcommand)
-
-# FIXME: there surely is some more deduplication to be achieved here with even darker Make magic
-define render-subcommand
-  @cat $^ >> $^.tmp
-	@$(call process-includes,$^,$^.tmp)
-	$(trace-gen) lowdown -sT man --nroff-nolinks -M section=1 $^.tmp -o $@
-	@# fix up `lowdown`'s automatic escaping of `--`
-	@# https://github.com/kristapsdz/lowdown/blob/edca6ce6d5336efb147321a43c47a698de41bb7c/entity.c#L202
-	@sed -i 's/\e\[u2013\]/--/' $@
-	@rm $^.tmp
-endef
 
 
-$(d)/%.1: $(d)/src/command-ref/%.md
+doc/manual/generated/in/%.1: doc/manual/generated/out
-	@printf "Title: %s\n\n" "$$(basename $@ .1)" > $^.tmp
+	$(trace-gen) doc/manual/render-manpage.sh "$$(basename $@ .1)" 1 \
-	@cat $^ >> $^.tmp
+		doc/manual/generated/out/markdown/command-ref/$*.md \
-	@$(call process-includes,$^,$^.tmp)
+		$@
-	$(trace-gen) lowdown -sT man --nroff-nolinks -M section=1 $^.tmp -o $@
-	@rm $^.tmp
 
-$(d)/%.8: $(d)/src/command-ref/%.md
+doc/manual/generated/in/%.8: doc/manual/generated/out
-	@printf "Title: %s\n\n" "$$(basename $@ .8)" > $^.tmp
+	$(trace-gen) doc/manual/render-manpage.sh "$$(basename $@ .8)" 8 \
-	@cat $^ >> $^.tmp
+		doc/manual/generated/out/markdown/command-ref/$*.md \
-	$(trace-gen) lowdown -sT man --nroff-nolinks -M section=8 $^.tmp -o $@
+		$@
-	@rm $^.tmp
 
-$(d)/nix.conf.5: $(d)/src/command-ref/conf-file.md
+doc/manual/generated/in/nix.conf.5: doc/manual/generated/out
-	@printf "Title: %s\n\n" "$$(basename $@ .5)" > $^.tmp
+	$(trace-gen) doc/manual/render-manpage.sh "$$(basename $@ .5)" 5 \
-	@cat $^ >> $^.tmp
+		doc/manual/generated/out/markdown/command-ref/conf-file.md \
-	@$(call process-includes,$^,$^.tmp)
+		$@
-	$(trace-gen) lowdown -sT man --nroff-nolinks -M section=5 $^.tmp -o $@
-	@rm $^.tmp
 
-$(d)/nix-profiles.5: $(d)/src/command-ref/files/profiles.md
+doc/manual/generated/in/nix-profiles.5: doc/manual/generated/out
-	@printf "Title: %s\n\n" "$$(basename $@ .5)" > $^.tmp
+	$(trace-gen) doc/manual/render-manpage.sh "$$(basename $@ .5)" 5 \
-	@cat $^ >> $^.tmp
+		doc/manual/generated/out/markdown/command-ref/files/profiles.md \
-	$(trace-gen) lowdown -sT man --nroff-nolinks -M section=5 $^.tmp -o $@
+		$@
-	@rm $^.tmp
 
-$(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/SUMMARY-rl-next.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md
+doc/manual/generated/in/command-ref/new-cli: doc/manual/generated/in/nix.json $(d)/utils.nix $(d)/generate-manpage.nix $(doc_nix)
-	@cp $< $@
+	@mkdir -p doc/manual/generated/in/command-ref
-	@$(call process-includes,$@,$@)
-
-$(d)/src/command-ref/new-cli: $(d)/nix.json $(d)/utils.nix $(d)/generate-manpage.nix $(doc_nix)
 	@rm -rf $@ $@.tmp
 	$(trace-gen) $(nix-eval) --write-to $@.tmp --expr 'import doc/manual/generate-manpage.nix true (builtins.readFile $<)'
 	@mv $@.tmp $@
 
-$(d)/src/command-ref/conf-file.md: $(d)/conf-file.json $(d)/utils.nix $(d)/src/command-ref/conf-file-prefix.md $(d)/src/command-ref/experimental-features-shortlist.md $(doc_nix)
+doc/manual/generated/in/command-ref/conf-file.md: doc/manual/generated/in/conf-file.json $(d)/utils.nix doc/manual/generated/in/command-ref/experimental-features-shortlist.md $(doc_nix)
-	@cat doc/manual/src/command-ref/conf-file-prefix.md > $@.tmp
+	@mkdir -p doc/manual/generated/in/command-ref
-	$(trace-gen) $(nix-eval) --expr '(import doc/manual/utils.nix).showSettings { inlineHTML = true; } (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp;
+	$(trace-gen) $(nix-eval) --expr '(import doc/manual/utils.nix).showSettings { inlineHTML = true; } (builtins.fromJSON (builtins.readFile $<))' >> $@
-	@mv $@.tmp $@
 
-$(d)/nix.json: $(doc_nix)
+doc/manual/generated/in/nix.json: $(doc_nix)
+	@mkdir -p doc/manual/generated/in
 	$(trace-gen) $(dummy-env) $(doc_nix) __dump-cli > $@.tmp
 	@mv $@.tmp $@
 
-$(d)/conf-file.json: $(doc_nix)
+doc/manual/generated/in/conf-file.json: $(doc_nix)
+	@mkdir -p doc/manual/generated/in
 	$(trace-gen) $(dummy-env) $(doc_nix) show-config --json --experimental-features nix-command > $@.tmp
 	@mv $@.tmp $@
 
-$(d)/src/contributing/experimental-feature-descriptions.md: $(d)/xp-features.json $(d)/utils.nix $(d)/generate-xp-features.nix $(doc_nix)
+doc/manual/generated/in/contributing/experimental-feature-descriptions.md: doc/manual/generated/in/xp-features.json $(d)/utils.nix $(d)/generate-xp-features.nix $(doc_nix)
+	@mkdir -p doc/manual/generated/in/contributing
 	@rm -rf $@ $@.tmp
 	$(trace-gen) $(nix-eval) --write-to $@.tmp --expr 'import doc/manual/generate-xp-features.nix (builtins.fromJSON (builtins.readFile $<))'
 	@mv $@.tmp $@
 
-$(d)/src/command-ref/experimental-features-shortlist.md: $(d)/xp-features.json $(d)/utils.nix $(d)/generate-xp-features-shortlist.nix $(doc_nix)
+doc/manual/generated/in/command-ref/experimental-features-shortlist.md: doc/manual/generated/in/xp-features.json $(d)/utils.nix $(d)/generate-xp-features-shortlist.nix $(doc_nix)
+	@mkdir -p doc/manual/generated/in/command-ref
 	@rm -rf $@ $@.tmp
 	$(trace-gen) $(nix-eval) --write-to $@.tmp --expr 'import doc/manual/generate-xp-features-shortlist.nix (builtins.fromJSON (builtins.readFile $<))'
 	@mv $@.tmp $@
 
-$(d)/xp-features.json: $(doc_nix)
+doc/manual/generated/in/xp-features.json: $(doc_nix)
 	$(trace-gen) $(dummy-env) NIX_PATH=nix/corepkgs=corepkgs $(doc_nix) __dump-xp-features > $@.tmp
 	@mv $@.tmp $@
 
-$(d)/src/language/builtins.md: $(d)/language.json $(d)/generate-builtins.nix $(d)/src/language/builtins-prefix.md $(doc_nix)
+doc/manual/generated/in/language/builtins.md: doc/manual/generated/in/language.json $(d)/generate-builtins.nix $(doc_nix)
-	@cat doc/manual/src/language/builtins-prefix.md > $@.tmp
+	@mkdir -p doc/manual/generated/in/language
-	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<)).builtins' >> $@.tmp;
+	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<)).builtins' >> $@
-	@cat doc/manual/src/language/builtins-suffix.md >> $@.tmp
-	@mv $@.tmp $@
 
-$(d)/src/language/builtin-constants.md: $(d)/language.json $(d)/generate-builtin-constants.nix $(d)/src/language/builtin-constants-prefix.md $(doc_nix)
+doc/manual/generated/in/language/builtin-constants.md: doc/manual/generated/in/language.json $(d)/generate-builtin-constants.nix $(doc_nix)
-	@cat doc/manual/src/language/builtin-constants-prefix.md > $@.tmp
+	@mkdir -p doc/manual/generated/in/language
-	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtin-constants.nix (builtins.fromJSON (builtins.readFile $<)).constants' >> $@.tmp;
+	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtin-constants.nix (builtins.fromJSON (builtins.readFile $<)).constants' >> $@
-	@cat doc/manual/src/language/builtin-constants-suffix.md >> $@.tmp
-	@mv $@.tmp $@
 
-$(d)/language.json: $(doc_nix)
+doc/manual/generated/in/language.json: $(doc_nix)
+	@mkdir -p doc/manual/generated/in
 	$(trace-gen) $(dummy-env) NIX_PATH=nix/corepkgs=corepkgs $(doc_nix) __dump-language > $@.tmp
 	@mv $@.tmp $@
 
 # Generate "Upcoming release" notes (or clear it and remove from menu)
-$(d)/src/release-notes/rl-next.md: $(d)/rl-next $(d)/rl-next/*
+doc/manual/generated/in/release-notes/rl-next-generated.md: $(d)/rl-next $(d)/rl-next/*
+	@mkdir -p doc/manual/generated/in/release-notes
 	@if type -p build-release-notes > /dev/null; then \
 		echo "  GEN   " $@; \
 		build-release-notes doc/manual/rl-next > $@; \
@@ -159,14 +132,6 @@ $(d)/src/release-notes/rl-next.md: $(d)/rl-next $(d)/rl-next/*
 		true > $@; \
 	fi
 
-$(d)/src/SUMMARY-rl-next.md: $(d)/src/release-notes/rl-next.md
-	$(trace-gen) true
-	@if [ -s $< ]; then \
-		echo '  - [Upcoming release](release-notes/rl-next.md)' > $@; \
-	else \
-	  true > $@; \
-	fi
-
 # Generate the HTML manual.
 .PHONY: manual-html
 manual-html: $(docdir)/manual/index.html
@@ -183,9 +148,9 @@ $(mandir)/man1/nix3-manpages: doc/manual/generated/man1/nix3-manpages
 	@mkdir -p $(DESTDIR)$$(dirname $@)
 	$(trace-install) install -m 0644 $$(dirname $<)/* $(DESTDIR)$$(dirname $@)
 
-doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
+doc/manual/generated/man1/nix3-manpages: doc/manual/generated/out
 	@mkdir -p $(DESTDIR)$$(dirname $@)
-	$(trace-gen) for i in doc/manual/src/command-ref/new-cli/*.md; do \
+	$(trace-gen) for i in doc/manual/generated/out/markdown/command-ref/new-cli/*.md; do \
 		name=$$(basename $$i .md); \
 		tmpFile=$$(mktemp); \
 		if [[ $$name = SUMMARY ]]; then continue; fi; \
@@ -196,23 +161,18 @@ doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
 	done
 	@touch $@
 
-$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/contributing/experimental-feature-descriptions.md $(d)/src/command-ref/conf-file.md $(d)/src/language/builtins.md $(d)/src/language/builtin-constants.md $(d)/src/release-notes/rl-next.md
+doc/manual/generated/out: $(MANUAL_SRCS) $(d)/book.toml $(d)/anchors.jq $(d)/custom.css $(d)/src/SUMMARY.md doc/manual/generated/in/command-ref/new-cli doc/manual/generated/in/command-ref/experimental-features-shortlist.md doc/manual/generated/in/contributing/experimental-feature-descriptions.md doc/manual/generated/in/command-ref/conf-file.md doc/manual/generated/in/language/builtins.md doc/manual/generated/in/language/builtin-constants.md doc/manual/generated/in/release-notes/rl-next-generated.md $(d)/substitute.py
+	@rm -rf $@
 	$(trace-gen) \
-		tmp="$$(mktemp -d)"; \
+		MDBOOK_SUBSTITUTE_SEARCH=doc/manual/generated/in \
-		cp -r doc/manual "$$tmp"; \
+		RUST_LOG=warn \
-		find "$$tmp" -name '*.md' | while read -r file; do \
+		mdbook build doc/manual -d generated/out 2>&1 \
-			$(call process-includes,$$file,$$file); \
+			| { grep -Fv "because fragment resolution isn't implemented" || :; }
-		done; \
+	@find $@ -iname meson.build -delete
-		find "$$tmp" -name '*.md' | while read -r file; do \
+
-			docroot="$$(realpath --relative-to="$$(dirname "$$file")" $$tmp/manual/src)"; \
+$(docdir)/manual/index.html: doc/manual/generated/out
-			sed -i "s,@docroot@,$$docroot,g" "$$file"; \
+	@mkdir -p $(DESTDIR)$(docdir)
-		done; \
-		set -euo pipefail; \
-		RUST_LOG=warn mdbook build "$$tmp/manual" -d $(DESTDIR)$(docdir)/manual.tmp 2>&1 \
-			| { grep -Fv "because fragment resolution isn't implemented" || :; }; \
-		rm -rf "$$tmp/manual"
 	@rm -rf $(DESTDIR)$(docdir)/manual
-	@mv $(DESTDIR)$(docdir)/manual.tmp/html $(DESTDIR)$(docdir)/manual
+	@cp -r $</html $(DESTDIR)$(docdir)/manual
-	@rm -rf $(DESTDIR)$(docdir)/manual.tmp
 
 endif

doc/manual/meson.build

@@ -0,0 +1,348 @@
+nix_env_for_docs = {
+  'HOME': '/dummy',
+  'NIX_CONF_DIR': '/dummy',
+  'NIX_SSL_CERT_FILE': '/dummy/no-ca-bundle.crt',
+  'NIX_STATE_DIR': '/dummy',
+  'NIX_CONFIG': 'cores = 0',
+}
+
+nix_for_docs = [ nix, '--experimental-features', 'nix-command' ]
+nix_eval_for_docs_common = nix_for_docs + [
+  'eval',
+  '-I', 'nix/corepkgs=corepkgs',
+  '--store', 'dummy://',
+  '--impure',
+]
+nix_eval_for_docs = nix_eval_for_docs_common + '--raw'
+
+conf_file_json = custom_target(
+  command : nix_for_docs + [ 'show-config', '--json' ],
+  capture : true,
+  output : 'conf-file.json',
+  env : nix_env_for_docs,
+)
+
+nix_conf_file_md_body = custom_target(
+  command : nix_eval_for_docs + [
+    '--expr',
+    '(import @INPUT0@).showSettings { inlineHTML = true; } (builtins.fromJSON (builtins.readFile @INPUT1@))',
+  ],
+  capture : true,
+  input : [
+    'utils.nix',
+    conf_file_json,
+  ],
+  output : 'conf-file.md.body',
+  env : nix_env_for_docs,
+)
+
+nix_conf_file_md = custom_target(
+  command : [ 'cat', '@INPUT@' ],
+  capture : true,
+  input : [
+    'src/command-ref/conf-file.md',
+    nix_conf_file_md_body,
+  ],
+  output : 'conf-file.md',
+)
+
+nix_exp_features_json = custom_target(
+  command : [ nix, '__dump-xp-features' ],
+  capture : true,
+  output : 'xp-features.json',
+)
+
+language_json = custom_target(
+  command: [nix, '__dump-language'],
+  output : 'language.json',
+  capture : true,
+  env : nix_env_for_docs,
+)
+
+nix3_cli_json = custom_target(
+  command : [ nix, '__dump-cli' ],
+  capture : true,
+  output : 'nix.json',
+  env : nix_env_for_docs,
+)
+
+generate_manual_deps = files(
+  'generate-deps.py',
+)
+
+# Generates builtins.md and builtin-constants.md.
+subdir('src/language')
+# Generates new-cli pages, experimental-features-shortlist.md, and conf-file.md.
+subdir('src/command-ref')
+# Generates experimental-feature-descriptions.md.
+subdir('src/contributing')
+# Generates rl-next-generated.md.
+subdir('src/release-notes')
+
+manual = custom_target(
+  'manual',
+  command : [
+    bash,
+    '-euo', 'pipefail',
+    '-c',
+    '''
+        @0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
+        cd @SOURCE_ROOT@
+        @1@ build doc/manual -d @2@ | { grep -Fv "because fragment resolution isn't implemented" || :; }
+        rm -rf @2@/manual
+        mv @2@/html @2@/manual
+        find @2@/manual -iname meson.build -delete
+    '''.format(
+      python.full_path(),
+      mdbook.full_path(),
+      meson.current_build_dir(),
+    ),
+  ],
+  input : [
+    generate_manual_deps,
+    'book.toml',
+    'anchors.jq',
+    'custom.css',
+    nix3_cli_files,
+    experimental_features_shortlist_md,
+    experimental_feature_descriptions_md,
+    conf_file_md,
+    builtins_md,
+    builtin_constants_md,
+    rl_next_generated,
+  ],
+  output : [
+    'manual',
+    'markdown',
+  ],
+  depfile : 'manual.d',
+  env : {
+    'RUST_LOG': 'info',
+    'MDBOOK_SUBSTITUTE_SEARCH': meson.current_build_dir() / 'src',
+  },
+)
+manual_html = manual[0]
+manual_md = manual[1]
+
+install_subdir(
+  manual_html.full_path(),
+  install_dir : datadir / 'doc/nix',
+)
+
+nix_nested_manpages = [
+  [ 'nix-env',
+    [
+      'delete-generations',
+      'install',
+      'list-generations',
+      'query',
+      'rollback',
+      'set-flag',
+      'set',
+      'switch-generation',
+      'switch-profile',
+      'uninstall',
+      'upgrade',
+    ],
+  ],
+  [ 'nix-store',
+    [
+      'add-fixed',
+      'add',
+      'delete',
+      'dump-db',
+      'dump',
+      'export',
+      'gc',
+      'generate-binary-cache-key',
+      'import',
+      'load-db',
+      'optimise',
+      'print-env',
+      'query',
+      'read-log',
+      'realise',
+      'repair-path',
+      'restore',
+      'serve',
+      'verify',
+      'verify-path',
+    ],
+  ],
+]
+
+foreach command : nix_nested_manpages
+  foreach page : command[1]
+    title = command[0] + ' --' + page
+    section = '1'
+    custom_target(
+      command : [
+        './render-manpage.sh',
+        '--out-no-smarty',
+        title,
+        section,
+        '@INPUT0@/command-ref' / command[0] / (page + '.md'),
+        '@OUTPUT0@',
+      ],
+      input : [
+        manual_md,
+      ],
+      output : command[0] + '-' + page + '.1',
+      install : true,
+      install_dir : mandir / 'man1',
+    )
+  endforeach
+endforeach
+
+nix3_manpages = [
+  'nix3-build',
+  'nix3-bundle',
+  'nix3-copy',
+  'nix3-daemon',
+  'nix3-derivation-add',
+  'nix3-derivation',
+  'nix3-derivation-show',
+  'nix3-develop',
+  'nix3-doctor',
+  'nix3-edit',
+  'nix3-eval',
+  'nix3-flake-archive',
+  'nix3-flake-check',
+  'nix3-flake-clone',
+  'nix3-flake-info',
+  'nix3-flake-init',
+  'nix3-flake-lock',
+  'nix3-flake',
+  'nix3-flake-metadata',
+  'nix3-flake-new',
+  'nix3-flake-prefetch',
+  'nix3-flake-show',
+  'nix3-flake-update',
+  'nix3-fmt',
+  'nix3-hash-file',
+  'nix3-hash',
+  'nix3-hash-path',
+  'nix3-hash-to-base16',
+  'nix3-hash-to-base32',
+  'nix3-hash-to-base64',
+  'nix3-hash-to-sri',
+  'nix3-help',
+  'nix3-help-stores',
+  'nix3-key-convert-secret-to-public',
+  'nix3-key-generate-secret',
+  'nix3-key',
+  'nix3-log',
+  'nix3-nar-cat',
+  'nix3-nar-dump-path',
+  'nix3-nar-ls',
+  'nix3-nar',
+  'nix3-path-info',
+  'nix3-print-dev-env',
+  'nix3-profile-diff-closures',
+  'nix3-profile-history',
+  'nix3-profile-install',
+  'nix3-profile-list',
+  'nix3-profile',
+  'nix3-profile-remove',
+  'nix3-profile-rollback',
+  'nix3-profile-upgrade',
+  'nix3-profile-wipe-history',
+  'nix3-realisation-info',
+  'nix3-realisation',
+  'nix3-registry-add',
+  'nix3-registry-list',
+  'nix3-registry',
+  'nix3-registry-pin',
+  'nix3-registry-remove',
+  'nix3-repl',
+  'nix3-run',
+  'nix3-search',
+  'nix3-shell',
+  'nix3-show-config',
+  'nix3-store-add-file',
+  'nix3-store-add-path',
+  'nix3-store-cat',
+  'nix3-store-copy-log',
+  'nix3-store-copy-sigs',
+  'nix3-store-delete',
+  'nix3-store-diff-closures',
+  'nix3-store-dump-path',
+  'nix3-store-gc',
+  'nix3-store-ls',
+  'nix3-store-make-content-addressed',
+  'nix3-store',
+  'nix3-store-optimise',
+  'nix3-store-path-from-hash-part',
+  'nix3-store-ping',
+  'nix3-store-prefetch-file',
+  'nix3-store-repair',
+  'nix3-store-sign',
+  'nix3-store-verify',
+  'nix3-upgrade-nix',
+  'nix3-why-depends',
+  'nix',
+]
+
+foreach page : nix3_manpages
+  section = '1'
+  custom_target(
+    command : [
+      bash,
+      '@INPUT0@',
+      page,
+      section,
+      '@INPUT1@/command-ref/new-cli/@0@.md'.format(page),
+      '@OUTPUT@',
+    ],
+    input : [
+      'render-manpage.sh',
+      manual_md,
+    ],
+    output : page + '.1',
+    install : true,
+    install_dir : mandir / 'man1',
+  )
+endforeach
+
+nix_manpages = [
+  [ 'nix-env', 1 ],
+  [ 'nix-store', 1 ],
+  [ 'nix-build', 1 ],
+  [ 'nix-shell', 1 ],
+  [ 'nix-instantiate', 1 ],
+  [ 'nix-collect-garbage', 1 ],
+  [ 'nix-prefetch-url', 1 ],
+  [ 'nix-channel', 1 ],
+  [ 'nix-hash', 1 ],
+  [ 'nix-copy-closure', 1 ],
+  [ 'nix.conf', 5, 'conf-file.md' ],
+  [ 'nix-daemon', 8 ],
+  [ 'nix-profiles', 5, 'files/profiles.md' ],
+]
+
+foreach entry : nix_manpages
+  title = entry[0]
+  # nix.conf.5 and nix-profiles.5 are based off of conf-file.md and files/profiles.md,
+  # rather than a stem identical to its mdbook source.
+  # Therefore we use an optional third element of this array to override the name pattern
+  md_file = entry.get(2, title + '.md')
+  section = entry[1].to_string()
+  custom_target(
+    command : [
+      bash,
+      '@INPUT0@',
+      title,
+      section,
+      '@INPUT1@/command-ref/@0@'.format(md_file),
+      '@OUTPUT@',
+    ],
+    input : [
+      'render-manpage.sh',
+      manual_md,
+      entry.get(3, []),
+    ],
+    output : '@0@.@1@'.format(entry[0], entry[1]),
+    install : true,
+    install_dir : mandir / 'man@0@'.format(entry[1]),
+  )
+endforeach

doc/manual/process-includes.sh

@@ -0,0 +1,32 @@
+#!/bin/sh
+
+set -euo pipefail
+
+infile="$1"
+outfile="$2"
+shift 2
+
+# set a search path for includes. the old makefile-based system splorked
+# everything into the source tree and was thus able to not have a search
+# path, but the meson system generates intermediate files into dedicated
+# directories separate from the source. we still retain the implicit old
+# behavior for now as the base search path, once meson is the default we
+# can revisit this and remove the implicit search path entry. it's fine.
+set -- "$(dirname "$infile")" "$@"
+
+# re-implement mdBook's include directive to make it usable for terminal output and for proper @docroot@ substitution
+(grep '{{#include' "$infile" || true) | while read -r line; do
+    found=false
+    include="$(printf "$line" | sed 's/{{#include \(.*\)}}/\1/')"
+    for path in "$@"; do
+        filename="$path/$include"
+        if [ -e "$filename" ]; then
+            found=true
+            matchline="$(printf "$line" | sed 's|/|\\/|g')"
+            sed -i "/$matchline/r $filename" "$outfile"
+            sed -i "s/$matchline//" "$outfile"
+            break
+        fi
+    done
+    $found || ( echo "#include-d file '$filename' does not exist." >&2; exit 1; )
+done

doc/manual/render-manpage.sh

@@ -0,0 +1,25 @@
+#!/bin/sh
+
+set -euo pipefail
+
+lowdown_args=
+
+if [ "$1" = --out-no-smarty ]; then
+    lowdown_args=--out-no-smarty
+    shift
+fi
+
+[ "$#" = 4 ] || {
+    echo "wrong number of args passed" >&2
+    exit 1
+}
+
+title="$1"
+section="$2"
+infile="$3"
+outfile="$4"
+
+(
+    printf "Title: %s\n\n" "$title"
+    cat "$infile"
+) | lowdown -sT man --nroff-nolinks $lowdown_args -M section="$section" -o "$outfile"

doc/manual/rl-next/better-errors-in-nix-repl.md

@@ -1,6 +1,7 @@
 ---
 synopsis: Concise error printing in `nix repl`
 prs: 9928
+cls: 811
 ---
 
 Previously, if an element of a list or attribute set threw an error while
@@ -37,4 +38,3 @@ error:
 
        error: uh oh!
 ```
-

doc/manual/rl-next/forbid-nested-debuggers.md

@@ -0,0 +1,32 @@
+---
+synopsis: Nested debuggers are no longer supported
+prs: 9920
+---
+
+Previously, evaluating an expression that throws an error in the debugger would
+enter a second, nested debugger:
+
+```
+nix-repl> builtins.throw "what"
+error: what
+
+
+Starting REPL to allow you to inspect the current state of the evaluator.
+
+Welcome to Nix 2.18.1. Type :? for help.
+
+nix-repl>
+```
+
+Now, it just prints the error message like `nix repl`:
+
+```
+nix-repl> builtins.throw "what"
+error:
+       … while calling the 'throw' builtin
+         at «string»:1:1:
+            1| builtins.throw "what"
+             | ^
+
+       error: what
+```

doc/manual/rl-next/new-assertions.md

@@ -0,0 +1,23 @@
+---
+synopsis: Lix turns more internal bugs into crashes
+cls: [797, 626]
+---
+
+Lix now enables build options such as trapping on signed overflow and enabling
+libstdc++ assertions by default. These may find new bugs in Lix, which will
+present themselves as Lix processes aborting, potentially without an error
+message.
+
+If Lix processes abort on your machine, this is a bug. Please file a bug,
+ideally with the core dump (or information from it).
+
+On Linux, run `coredumpctl list`, find the crashed process's PID at
+the bottom of the list, then run `coredumpctl info THE-PID`. You can then paste
+the output into a bug report.
+
+On macOS, open the Console app from Applications/Utilities, select Crash
+Reports, select the crash report in question. Right click on it, select Open In
+Finder, then include that file in your bug report. [See the Apple
+documentation][apple-crashreport] for more details.
+
+[apple-crashreport]: https://developer.apple.com/documentation/xcode/acquiring-crash-reports-and-diagnostic-logs#Locate-crash-reports-and-memory-logs-on-the-device

doc/manual/rl-next/no-cache-eval-errors.md

@@ -0,0 +1,8 @@
+---
+synopsis: re-evaluate cached evaluation errors
+cls: 771
+---
+
+"cached failure of [expr]" errors have been removed: expressions already in the
+eval cache as a failure will now simply be re-evaluated, removing the need to
+set `--no-eval-cache` or similar to see the error.

doc/manual/rl-next/repl-doc-command.md

@@ -0,0 +1,13 @@
+---
+synopsis: Experimental REPL support for documentation comments using `:doc`
+cls: 564
+---
+
+Using `:doc` in the REPL now supports showing documentation comments when defined on a function.
+
+Previously this was only able to document builtins, however it now will show comments defined on a lambda as well.
+
+This support is experimental and relies on an embedded version of [nix-doc](https://github.com/lf-/nix-doc).
+
+The logic also supports limited Markdown formatting of doccomments and should easily support any [RFC 145](https://github.com/NixOS/rfcs/blob/master/rfcs/0145-doc-strings.md)
+compatible documentation comments in addition to simple commented documentation.

doc/manual/rl-next/repl-overlays.md

@@ -0,0 +1,36 @@
+---
+synopsis: Add `repl-overlays` option
+prs: 10203
+cls: 504
+---
+
+A `repl-overlays` option has been added, which specifies files that can overlay
+and modify the top-level bindings in `nix repl`. For example, with the
+following contents in `~/.config/nix/repl.nix`:
+
+```nix
+info: final: prev: let
+  optionalAttrs = predicate: attrs:
+    if predicate
+    then attrs
+    else {};
+in
+  optionalAttrs (prev ? legacyPackages && prev.legacyPackages ? ${info.currentSystem})
+  {
+    pkgs = prev.legacyPackages.${info.currentSystem};
+  }
+```
+
+We can run `nix repl` and use `pkgs` to refer to `legacyPackages.${currentSystem}`:
+
+```ShellSession
+$ nix repl --repl-overlays ~/.config/nix/repl.nix nixpkgs
+Lix 2.90.0
+Type :? for help.
+Loading installable 'flake:nixpkgs#'...
+Added 5 variables.
+Loading 'repl-overlays'...
+Added 6 variables.
+nix-repl> pkgs.bash
+«derivation /nix/store/g08b5vkwwh0j8ic9rkmd8mpj878rk62z-bash-5.2p26.drv»
+```

doc/manual/src/SUMMARY.md

@@ -88,7 +88,92 @@
     - [nix-instantiate](command-ref/nix-instantiate.md)
     - [nix-prefetch-url](command-ref/nix-prefetch-url.md)
   - [Experimental Commands](command-ref/experimental-commands.md)
-{{#include ./command-ref/new-cli/SUMMARY.md}}
+    - [nix](command-ref/new-cli/nix.md)
+    - [nix build](command-ref/new-cli/nix3-build.md)
+    - [nix bundle](command-ref/new-cli/nix3-bundle.md)
+    - [nix copy](command-ref/new-cli/nix3-copy.md)
+    - [nix daemon](command-ref/new-cli/nix3-daemon.md)
+    - [nix derivation](command-ref/new-cli/nix3-derivation.md)
+    - [nix derivation add](command-ref/new-cli/nix3-derivation-add.md)
+    - [nix derivation show](command-ref/new-cli/nix3-derivation-show.md)
+    - [nix develop](command-ref/new-cli/nix3-develop.md)
+    - [nix doctor](command-ref/new-cli/nix3-doctor.md)
+    - [nix edit](command-ref/new-cli/nix3-edit.md)
+    - [nix eval](command-ref/new-cli/nix3-eval.md)
+    - [nix flake](command-ref/new-cli/nix3-flake.md)
+    - [nix flake archive](command-ref/new-cli/nix3-flake-archive.md)
+    - [nix flake check](command-ref/new-cli/nix3-flake-check.md)
+    - [nix flake clone](command-ref/new-cli/nix3-flake-clone.md)
+    - [nix flake info](command-ref/new-cli/nix3-flake-info.md)
+    - [nix flake init](command-ref/new-cli/nix3-flake-init.md)
+    - [nix flake lock](command-ref/new-cli/nix3-flake-lock.md)
+    - [nix flake metadata](command-ref/new-cli/nix3-flake-metadata.md)
+    - [nix flake new](command-ref/new-cli/nix3-flake-new.md)
+    - [nix flake prefetch](command-ref/new-cli/nix3-flake-prefetch.md)
+    - [nix flake show](command-ref/new-cli/nix3-flake-show.md)
+    - [nix flake update](command-ref/new-cli/nix3-flake-update.md)
+    - [nix fmt](command-ref/new-cli/nix3-fmt.md)
+    - [nix hash](command-ref/new-cli/nix3-hash.md)
+    - [nix hash file](command-ref/new-cli/nix3-hash-file.md)
+    - [nix hash path](command-ref/new-cli/nix3-hash-path.md)
+    - [nix hash to-base16](command-ref/new-cli/nix3-hash-to-base16.md)
+    - [nix hash to-base32](command-ref/new-cli/nix3-hash-to-base32.md)
+    - [nix hash to-base64](command-ref/new-cli/nix3-hash-to-base64.md)
+    - [nix hash to-sri](command-ref/new-cli/nix3-hash-to-sri.md)
+    - [nix help](command-ref/new-cli/nix3-help.md)
+    - [nix help-stores](command-ref/new-cli/nix3-help-stores.md)
+    - [nix key](command-ref/new-cli/nix3-key.md)
+    - [nix key convert-secret-to-public](command-ref/new-cli/nix3-key-convert-secret-to-public.md)
+    - [nix key generate-secret](command-ref/new-cli/nix3-key-generate-secret.md)
+    - [nix log](command-ref/new-cli/nix3-log.md)
+    - [nix nar](command-ref/new-cli/nix3-nar.md)
+    - [nix nar cat](command-ref/new-cli/nix3-nar-cat.md)
+    - [nix nar dump-path](command-ref/new-cli/nix3-nar-dump-path.md)
+    - [nix nar ls](command-ref/new-cli/nix3-nar-ls.md)
+    - [nix path-info](command-ref/new-cli/nix3-path-info.md)
+    - [nix print-dev-env](command-ref/new-cli/nix3-print-dev-env.md)
+    - [nix profile](command-ref/new-cli/nix3-profile.md)
+    - [nix profile diff-closures](command-ref/new-cli/nix3-profile-diff-closures.md)
+    - [nix profile history](command-ref/new-cli/nix3-profile-history.md)
+    - [nix profile install](command-ref/new-cli/nix3-profile-install.md)
+    - [nix profile list](command-ref/new-cli/nix3-profile-list.md)
+    - [nix profile remove](command-ref/new-cli/nix3-profile-remove.md)
+    - [nix profile rollback](command-ref/new-cli/nix3-profile-rollback.md)
+    - [nix profile upgrade](command-ref/new-cli/nix3-profile-upgrade.md)
+    - [nix profile wipe-history](command-ref/new-cli/nix3-profile-wipe-history.md)
+    - [nix realisation](command-ref/new-cli/nix3-realisation.md)
+    - [nix realisation info](command-ref/new-cli/nix3-realisation-info.md)
+    - [nix registry](command-ref/new-cli/nix3-registry.md)
+    - [nix registry add](command-ref/new-cli/nix3-registry-add.md)
+    - [nix registry list](command-ref/new-cli/nix3-registry-list.md)
+    - [nix registry pin](command-ref/new-cli/nix3-registry-pin.md)
+    - [nix registry remove](command-ref/new-cli/nix3-registry-remove.md)
+    - [nix repl](command-ref/new-cli/nix3-repl.md)
+    - [nix run](command-ref/new-cli/nix3-run.md)
+    - [nix search](command-ref/new-cli/nix3-search.md)
+    - [nix shell](command-ref/new-cli/nix3-shell.md)
+    - [nix show-config](command-ref/new-cli/nix3-show-config.md)
+    - [nix store](command-ref/new-cli/nix3-store.md)
+    - [nix store add-file](command-ref/new-cli/nix3-store-add-file.md)
+    - [nix store add-path](command-ref/new-cli/nix3-store-add-path.md)
+    - [nix store cat](command-ref/new-cli/nix3-store-cat.md)
+    - [nix store copy-log](command-ref/new-cli/nix3-store-copy-log.md)
+    - [nix store copy-sigs](command-ref/new-cli/nix3-store-copy-sigs.md)
+    - [nix store delete](command-ref/new-cli/nix3-store-delete.md)
+    - [nix store diff-closures](command-ref/new-cli/nix3-store-diff-closures.md)
+    - [nix store dump-path](command-ref/new-cli/nix3-store-dump-path.md)
+    - [nix store gc](command-ref/new-cli/nix3-store-gc.md)
+    - [nix store ls](command-ref/new-cli/nix3-store-ls.md)
+    - [nix store make-content-addressed](command-ref/new-cli/nix3-store-make-content-addressed.md)
+    - [nix store optimise](command-ref/new-cli/nix3-store-optimise.md)
+    - [nix store path-from-hash-part](command-ref/new-cli/nix3-store-path-from-hash-part.md)
+    - [nix store ping](command-ref/new-cli/nix3-store-ping.md)
+    - [nix store prefetch-file](command-ref/new-cli/nix3-store-prefetch-file.md)
+    - [nix store repair](command-ref/new-cli/nix3-store-repair.md)
+    - [nix store sign](command-ref/new-cli/nix3-store-sign.md)
+    - [nix store verify](command-ref/new-cli/nix3-store-verify.md)
+    - [nix upgrade-nix](command-ref/new-cli/nix3-upgrade-nix.md)
+    - [nix why-depends](command-ref/new-cli/nix3-why-depends.md)
   - [Files](command-ref/files.md)
     - [nix.conf](command-ref/conf-file.md)
     - [Profiles](command-ref/files/profiles.md)
@@ -109,7 +194,7 @@
   - [CLI guideline](contributing/cli-guideline.md)
   - [C++ style guide](contributing/cxx.md)
 - [Release Notes](release-notes/release-notes.md)
-{{#include ./SUMMARY-rl-next.md}}
+  - [Upcoming release](release-notes/rl-next.md)
   - [Release 2.18 (2023-09-20)](release-notes/rl-2.18.md)
   - [Release 2.17 (2023-07-24)](release-notes/rl-2.17.md)
   - [Release 2.16 (2023-05-31)](release-notes/rl-2.16.md)

doc/manual/src/_redirects

@@ -27,4 +27,3 @@
 /package-management/channels* /command-ref/nix-channel 301!
 
 /package-management/s3-substituter* /command-ref/new-cli/nix3-help-stores#s3-binary-cache-store 301!
-

doc/manual/src/architecture/architecture.md

@@ -112,4 +112,3 @@ Build inputs used as instructions to a build task are marked accordingly:
 |                                                                    |
 +--------------------------------------------------------------------+
 ```
-

doc/manual/src/command-ref/conf-file.md

@@ -17,7 +17,7 @@ By default Nix reads settings from the following places, in that order:
 
 1. If [`NIX_USER_CONF_FILES`](./env-common.md#env-NIX_USER_CONF_FILES) is set, then each path separated by `:` will be loaded in reverse order.
 
-   Otherwise it will look for `nix/nix.conf` files in `XDG_CONFIG_DIRS` and [`XDG_CONFIG_HOME`](./env-common.md#env-XDG_CONFIG_HOME).
+   Otherwise it will look for `nix/nix.conf` files in `XDG_CONFIG_DIRS` and `XDG_CONFIG_HOME`.
    If unset, `XDG_CONFIG_DIRS` defaults to `/etc/xdg`, and `XDG_CONFIG_HOME` defaults to `$HOME/.config` as per [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html).
 
 1. If [`NIX_CONFIG`](./env-common.md#env-NIX_CONFIG) is set, its contents are treated as the contents of a configuration file.
@@ -68,3 +68,4 @@ The `extra-` prefix is supported for settings that take a list of items (e.g. `-
 
 # Available settings
 
+{{#include @generated@/command-ref/conf-file.md}}

doc/manual/src/command-ref/env-common.md

@@ -115,9 +115,9 @@ For backwards compatibility, Nix commands will follow the standard only when [`u
 
 The following environment variables are used to determine locations of various state and configuration files:
 
-- [`XDG_CONFIG_HOME`]{#env-XDG_CONFIG_HOME} (default `~/.config`)
+- `XDG_CONFIG_HOME` (default `~/.config`)
-- [`XDG_STATE_HOME`]{#env-XDG_STATE_HOME} (default `~/.local/state`)
+- `XDG_STATE_HOME` (default `~/.local/state`)
-- [`XDG_CACHE_HOME`]{#env-XDG_CACHE_HOME} (default `~/.cache`)
+- `XDG_CACHE_HOME` (default `~/.cache`)
 
 
 [XDG Base Directory Specification]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

doc/manual/src/command-ref/meson.build

@@ -0,0 +1,66 @@
+xp_features_json = custom_target(
+  command : [nix, '__dump-xp-features'],
+  capture : true,
+  output : 'xp-features.json',
+)
+
+experimental_features_shortlist_md = custom_target(
+  command : nix_eval_for_docs + [
+    '--expr',
+    'import @INPUT0@ (builtins.fromJSON (builtins.readFile @INPUT1@))',
+  ],
+  input : [
+    '../../generate-xp-features-shortlist.nix',
+    xp_features_json,
+  ],
+  capture : true,
+  output : 'experimental-features-shortlist.md',
+  env : nix_env_for_docs,
+)
+
+# Intermediate step for manpage generation.
+# This splorks the output of generate-manpage.nix as JSON,
+# which gets written as a directory tree below.
+nix3_cli_files_json = custom_target(
+  command : nix_eval_for_docs_common + [
+    '--json',
+    '--expr',
+    'import @INPUT0@ true (builtins.readFile @INPUT1@)',
+  ],
+  input : [
+    '../../generate-manpage.nix',
+    nix3_cli_json,
+  ],
+  capture : true,
+  output : 'new-cli.json',
+  env : nix_env_for_docs,
+)
+nix3_cli_files = custom_target(
+  command : [
+    python,
+    '@INPUT0@',
+    '-i', '@INPUT1@',
+    '-o', '@OUTPUT@',
+  ],
+  input : [
+    '../../json-to-tree.py',
+    nix3_cli_files_json,
+  ],
+  output : 'new-cli',
+)
+
+conf_file_md = custom_target(
+  command : [
+    nix_eval_for_docs,
+    '--expr',
+    '(import @INPUT0@).showSettings { inlineHTML = true; } (builtins.fromJSON (builtins.readFile @INPUT1@))',
+  ],
+  capture : true,
+  input : [
+    '../../utils.nix',
+    conf_file_json,
+    experimental_features_shortlist_md,
+  ],
+  output : 'conf-file.md',
+  env : nix_env_for_docs,
+)

doc/manual/src/command-ref/new-cli/nix.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix.md}}

doc/manual/src/command-ref/new-cli/nix3-build.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-build.md}}

doc/manual/src/command-ref/new-cli/nix3-bundle.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-bundle.md}}

doc/manual/src/command-ref/new-cli/nix3-copy.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-copy.md}}

doc/manual/src/command-ref/new-cli/nix3-daemon.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-daemon.md}}

doc/manual/src/command-ref/new-cli/nix3-derivation-add.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-derivation-add.md}}

doc/manual/src/command-ref/new-cli/nix3-derivation-show.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-derivation-show.md}}

doc/manual/src/command-ref/new-cli/nix3-derivation.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-derivation.md}}

doc/manual/src/command-ref/new-cli/nix3-develop.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-develop.md}}

doc/manual/src/command-ref/new-cli/nix3-doctor.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-doctor.md}}

doc/manual/src/command-ref/new-cli/nix3-edit.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-edit.md}}

doc/manual/src/command-ref/new-cli/nix3-eval.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-eval.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-archive.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-archive.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-check.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-check.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-clone.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-clone.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-info.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-info.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-init.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-init.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-lock.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-lock.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-metadata.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-metadata.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-new.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-new.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-prefetch.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-prefetch.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-show.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-show.md}}

doc/manual/src/command-ref/new-cli/nix3-flake-update.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake-update.md}}

doc/manual/src/command-ref/new-cli/nix3-flake.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-flake.md}}

doc/manual/src/command-ref/new-cli/nix3-fmt.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-fmt.md}}

doc/manual/src/command-ref/new-cli/nix3-hash-file.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-hash-file.md}}

doc/manual/src/command-ref/new-cli/nix3-hash-path.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-hash-path.md}}

doc/manual/src/command-ref/new-cli/nix3-hash-to-base16.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-hash-to-base16.md}}

doc/manual/src/command-ref/new-cli/nix3-hash-to-base32.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-hash-to-base32.md}}

doc/manual/src/command-ref/new-cli/nix3-hash-to-base64.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-hash-to-base64.md}}

doc/manual/src/command-ref/new-cli/nix3-hash-to-sri.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-hash-to-sri.md}}

doc/manual/src/command-ref/new-cli/nix3-hash.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-hash.md}}

doc/manual/src/command-ref/new-cli/nix3-help-stores.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-help-stores.md}}

doc/manual/src/command-ref/new-cli/nix3-help.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-help.md}}

doc/manual/src/command-ref/new-cli/nix3-key-convert-secret-to-public.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-key-convert-secret-to-public.md}}

doc/manual/src/command-ref/new-cli/nix3-key-generate-secret.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-key-generate-secret.md}}

doc/manual/src/command-ref/new-cli/nix3-key.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-key.md}}

doc/manual/src/command-ref/new-cli/nix3-log.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-log.md}}

doc/manual/src/command-ref/new-cli/nix3-nar-cat.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-nar-cat.md}}

doc/manual/src/command-ref/new-cli/nix3-nar-dump-path.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-nar-dump-path.md}}

doc/manual/src/command-ref/new-cli/nix3-nar-ls.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-nar-ls.md}}

doc/manual/src/command-ref/new-cli/nix3-nar.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-nar.md}}

doc/manual/src/command-ref/new-cli/nix3-path-info.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-path-info.md}}

doc/manual/src/command-ref/new-cli/nix3-print-dev-env.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-print-dev-env.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-diff-closures.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-diff-closures.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-history.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-history.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-install.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-install.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-list.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-list.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-remove.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-remove.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-rollback.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-rollback.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-upgrade.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-upgrade.md}}

doc/manual/src/command-ref/new-cli/nix3-profile-wipe-history.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile-wipe-history.md}}

doc/manual/src/command-ref/new-cli/nix3-profile.md

@@ -0,0 +1 @@
+{{#include @generated@/command-ref/new-cli/nix3-profile.md}}