focus on golden-path covering most scenarios

This should handle installation scenarios we can handle with
anything resembling confidence. Goal is approximating the existing
setup--not enforcing a best-practice...

Approaches (+ installer-handled, - manual) and configs each covers:

+ no change needed; /nix OK on boot volume:
  All pre-Catalina (regardless of T2 or FileVault use)

+ create new unencrypted volume:
  Catalina, pre-T2, no FileVault

+ create new encrypted-at-rest volume:
  Catalina, pre-T2, FileVault
  Catalina, T2, no FileVault

- require user to pre-create encrypted volume
  Catalina, T2, FileVault

doc/manual/installation/installing-binary.xml

@@ -6,16 +6,30 @@
 
 <title>Installing a Binary Distribution</title>
 
-<para>If you are using Linux or macOS, the easiest way to install Nix
+<para>
-is to run the following command:
+  If you are using Linux or macOS versions up to 10.14 (Mojave), the
+  easiest way to install Nix is to run the following command:
+</para>
 
 <screen>
   $ sh &lt;(curl https://nixos.org/nix/install)
 </screen>
 
-As of Nix 2.1.0, the Nix installer will always default to creating a
+<para>
-single-user installation, however opting in to the multi-user
+  If you're using macOS 10.15 (Catalina) or newer, consult
-installation is highly recommended.
+  <link linkend="sect-macos-installation">the macOS installation instructions</link>
+  before installing.
+</para>
+
+<para>
+  As of Nix 2.1.0, the Nix installer will always default to creating a
+  single-user installation, however opting in to the multi-user
+  installation is highly recommended.
+  <!-- TODO: this explains *neither* why the default version is
+  single-user, nor why we'd recommend multi-user over the default.
+  True prospective users don't have much basis for evaluating this.
+  What's it to me? Who should pick which? Why? What if I pick wrong?
+  -->
 </para>
 
 <section xml:id="sect-single-user-installation">
@@ -36,7 +50,7 @@ run this under your usual user account, <emphasis>not</emphasis> as
 root.  The script will invoke <command>sudo</command> to create
 <filename>/nix</filename> if it doesn’t already exist.  If you don’t
 have <command>sudo</command>, you should manually create
-<command>/nix</command> first as root, e.g.:
+<filename>/nix</filename> first as root, e.g.:
 
 <screen>
 $ mkdir /nix
@@ -47,7 +61,7 @@ The install script will modify the first writable file from amongst
 <filename>.bash_profile</filename>, <filename>.bash_login</filename>
 and <filename>.profile</filename> to source
 <filename>~/.nix-profile/etc/profile.d/nix.sh</filename>. You can set
-the <command>NIX_INSTALLER_NO_MODIFY_PROFILE</command> environment
+the <envar>NIX_INSTALLER_NO_MODIFY_PROFILE</envar> environment
 variable before executing the install script to disable this
 behaviour.
 </para>
@@ -81,12 +95,10 @@ $ rm -rf /nix
   <para>
     You can instruct the installer to perform a multi-user
     installation on your system:
-
-    <screen>
-  sh &lt;(curl https://nixos.org/nix/install) --daemon
-</screen>
   </para>
 
+  <screen>sh &lt;(curl https://nixos.org/nix/install) --daemon</screen>
+
   <para>
     The multi-user installation of Nix will create build users between
     the user IDs 30001 and 30032, and a group with the group ID 30000.
@@ -136,82 +148,253 @@ sudo rm /Library/LaunchDaemons/org.nixos.nix-daemon.plist
 
 </section>
 
-<section xml:id="sect-apfs-volume-installation">
+<section xml:id="sect-macos-installation">
-  <title>APFS Volume Installation</title>
+  <title>macOS Installation</title>
 
   <para>
-    The root filesystem is read-only as of macOS 10.15 Catalina, all writable
+    Starting with macOS 10.15 (Catalina), the root filesystem is read-only.
-    paths were moved to a separate data volume.  This means creating or writing
+    This means <filename>/nix</filename> can no longer live on your system
-    to <filename>/nix</filename> is not allowed.  While changing the default prefix
+    volume, and that you'll need a workaround to install Nix.
-    would be possible, it's a very intrusive change that has side effects we want to
-    avoid for now.
   </para>
 
   <para>
-    For common writable locations <literal>firmlinks</literal> were introduced,
+    The recommended approach, which creates an unencrypted APFS volume
-    described by Apple as a "bi-directional wormhole" between two filesystems.
+    for your Nix store and a "synthetic" empty directory to mount it
-    Essentially a bind mount for APFS volumes.  However this is (currently) not
+    over at <filename>/nix</filename>, is least likely to impair Nix
-    user configurable and only available for paths like <filename>/Users</filename>.
+    or your system.
   </para>
 
+  <note><para>
+    With all separate-volume approaches, it's possible something on
+    your system (particularly daemons/services and restored apps) may
+    need access to your Nix store before the volume is mounted. Adding
+    additional encryption makes this more likely.
+  </para></note>
+
   <para>
-    For special cases like NFS mount points or package manager roots <link xlink:href="https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man5/synthetic.conf.5.html">synthetic.conf(5)</link>
+    If you're using a recent Mac with a
-    provides a mechanism for some limited, user-controlled file-creation at <filename>/</filename>.
+    <link xlink:href="https://www.apple.com/euro/mac/shared/docs/Apple_T2_Security_Chip_Overview.pdf">T2 chip</link>,
-    This only applies at boot time, however <command>apfs.util</command> can be used
+    your drive will still be encrypted at rest (in which case "unencrypted"
-    to trigger the creation (not deletion) of new entries without a reboot.
+    is a bit of a misnomer). To use this approach, just install Nix with:
-    It would be ideal if this could create firmlinks, however a symlink or mountpoint
-    are the only options.
   </para>
 
-<screen>
+  <screen>$ sh &lt;(curl https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume</screen>
-alice$ /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B
-</screen>
 
-  <itemizedlist>
+  <para>
-    <listitem>
+    If you don't like the sound of this, you'll want to weigh the
-      <para>
+    other approaches and tradeoffs detailed in this section.
-        The simplest solution is creating a symlink with <filename>/etc/synthetic.conf</filename>
+  </para>
-        to the data volume. (not recommended)
-      </para>
 
-<screen>
+  <note>
-nix    /System/Volumes/Data/nix
+    <title>Eventual solutions?</title>
-</screen>
+    <para>
+      All of the known workarounds have drawbacks, but we hope
+      better solutions will be available in the future. Some that
+      we have our eye on are:
+    </para>
+    <orderedlist>
+      <listitem>
+        <para>
+          A true firmlink would enable the Nix store to live on the
+          primary data volume without the build problems caused by
+          the symlink approach. End users cannot currently
+          create true firmlinks.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          If the Nix store volume shared FileVault encryption
+          with the primary data volume (probably by using the same
+          volume group and role), FileVault encryption could be
+          easily supported by the installer without requiring
+          manual setup by each user.
+        </para>
+      </listitem>
+    </orderedlist>
+  </note>
 
-<screen>
+  <section xml:id="sect-macos-installation-change-store-prefix">
-alice$ ls -l /
+    <title>Change the Nix store path prefix</title>
-lrwxr-xr-x   1 root  wheel    25 Jan  1  2019 nix -> /System/Volumes/Data/nix
+    <para>
-</screen>
+      Changing the default prefix for the Nix store is a simple
+      approach which enables you to leave it on your root volume,
+      where it can take full advantage of FileVault encryption if
+      enabled. Unfortunately, this approach also opts your device out
+      of some benefits that are enabled by using the same prefix
+      across systems:
 
-      <para>
+      <itemizedlist>
-        However builds that detect or resolve this symlink will leak the canonical
+        <listitem>
-        location or even fail in certain cases, making this approach undesirable.
+          <para>
-      </para>
+            Your system won't be able to take advantage of the binary
-    </listitem>
+            cache (unless someone is able to stand up and support
+            duplicate caching infrastructure), which means you'll
+            spend more time waiting for builds.
+          </para>
+        </listitem>
+        <listitem>
+          <para>
+            It's harder to build and deploy packages to Linux systems.
+          </para>
+        </listitem>
+        <!-- TODO: may be more here -->
+      </itemizedlist>
 
-    <listitem>
+      <!-- TODO: Yes, but how?! -->
-      <para>
-        An empty directory can also be created using <filename>/etc/synthetic.conf</filename>,
-        this won't be writable but can be used as a mount point.  And with
-        <literal>APFS</literal> it's relatively easy to create an separate
-        volume for nix instead.
-      </para>
 
-<screen>
+      It would also possible (and often requested) to just apply this
-nix
+      change ecosystem-wide, but it's an intrusive process that has
-</screen>
+      side effects we want to avoid for now.
+      <!-- magnificent hand-wavy gesture -->
+    </para>
+    <para>
+    </para>
+  </section>
 
-<screen>
+  <section xml:id="sect-macos-installation-encrypted-volume">
-alice$ /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B
+    <title>Use a separate encrypted volume</title>
-alice$ sudo diskutil apfs addVolume diskX APFS 'Nix Store' -mountpoint /nix
+    <para>
-alice$ mount
+      If you like, you can also add encryption to the recommended
-/dev/disk1s6 on /nix (apfs, local, journaled)
+      approach taken by the installer. You can do this by pre-creating
-</screen>
+      an encrypted volume before you run the installer--or you can
+      run the installer and encrypt the volume it creates later.
+      <!-- TODO: see later note about whether this needs both add-encryption and from-scratch directions -->
+    </para>
+    <para>
+      In either case, adding encryption to a second volume isn't quite
+      as simple as enabling FileVault for your boot volume. Before you
+      dive in, there are a few things to weigh:
+    </para>
+    <orderedlist>
+      <listitem>
+        <para>
+          The additional volume won't be encrypted with your existing
+          FileVault key, so you'll need another mechanism to decrypt
+          the volume.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          You can store the password in Keychain to automatically
+          decrypt the volume on boot--but it'll have to wait on Keychain
+          and may not mount before your GUI apps restore. If any of
+          your launchd agents or apps depend on Nix-installed software
+          (for example, if you use a Nix-installed login shell), the
+          restore may fail or break.
+        </para>
+        <para>
+          On a case-by-case basis, you may be able to work around this
+          problem by using <command>wait4path</command> to block
+          execution until your executable is available.
+        </para>
+        <para>
+          It's also possible to decrypt and mount the volume earlier
+          with a login hook--but this mechanism appears to be
+          deprecated and its future is unclear.
+        </para>
+      </listitem>
+      <listitem>
+        <para>
+          You can hard-code the password in the clear, so that your
+          store volume can be decrypted before Keychain is available.
+        </para>
+      </listitem>
+    </orderedlist>
+    <para>
+      If you are comfortable navigating these tradeoffs, you can encrypt the volume with
+      something along the lines of:
+      <!-- TODO:
+      I don't know if this also needs from-scratch instructions?
+      can we just recommend use-the-installer-and-then-encrypt?
+      -->
+    </para>
+    <!--
+    TODO: it looks like this option can be encryptVolume|encrypt|enableFileVault
 
-      <para>
+    It may be more clear to use encryptVolume, here? FileVault seems
-        This does make the installation more complicated, requiring both
+    heavily associated with the boot-volume behavior; I worry
-        <filename>/etc/synthetic.conf</filename> as well as <filename>/etc/fstab</filename>
+    a little that it can mislead here, especially as it gets
-      </para>
+    copied around minus doc context...?
+    -->
+    <screen>alice$ diskutil apfs enableFileVault /nix -user disk</screen>
+
+    <!-- TODO: and then go into detail on the mount/decrypt approaches? -->
+  </section>
+
+  <section xml:id="sect-macos-installation-symlink">
+    <!--
+    Maybe a good razor is: if we'd hate having to support someone who
+    installed Nix this way, it shouldn't even be detailed?
+    -->
+    <title>Symlink the Nix store to a custom location</title>
+    <para>
+      Another simple approach is using <filename>/etc/synthetic.conf</filename>
+      to symlink the Nix store to the data volume. This option also
+      enables your store to share any configured FileVault encryption.
+      Unfortunately, builds that resolve the symlink may leak the
+      canonical path or even fail.
+    </para>
+    <para>
+      Because of these downsides, we can't recommend this approach.
+    </para>
+    <!-- Leaving out instructions for this one. -->
+  </section>
+
+  <section xml:id="sect-macos-installation-recommended-notes">
+    <title>Notes on the recommended approach</title>
+    <para>
+      This section goes into a little more detail on the recommended
+      approach. You don't need to understand it to run the installer,
+      but it can serve as a helpful reference if you run into trouble.
+    </para>
+    <orderedlist>
+      <listitem>
+        <para>
+          In order to compose user-writable locations into the new
+          read-only system root, Apple introduced a new concept called
+          <literal>firmlinks</literal>, which it describes as a
+          "bi-directional wormhole" between two filesystems. You can
+          see the current firmlinks in <filename>/usr/share/firmlinks</filename>.
+          Unfortunately, firmlinks aren't (currently?) user-configurable.
+        </para>
+
+        <para>
+          For special cases like NFS mount points or package manager roots,
+          <link xlink:href="https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man5/synthetic.conf.5.html">synthetic.conf(5)</link>
+          supports limited user-controlled file-creation (of symlinks,
+          and synthetic empty directories) at <filename>/</filename>.
+          To create a synthetic empty directory for mounting at <filename>/nix</filename>,
+          add the following line to <filename>/etc/synthetic.conf</filename>
+          (create it if necessary):
+        </para>
+
+        <screen>nix</screen>
+      </listitem>
+
+      <listitem>
+        <para>
+          This configuration is applied at boot time, but you can use
+          <command>apfs.util</command> to trigger creation (not deletion)
+          of new entries without a reboot:
+        </para>
+
+        <screen>alice$ /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B</screen>
+      </listitem>
+
+      <listitem>
+        <para>
+          Create the new APFS volume with diskutil:
+        </para>
+
+        <screen>alice$ sudo diskutil apfs addVolume diskX APFS 'Nix Store' -mountpoint /nix</screen>
+      </listitem>
+
+      <listitem>
+        <para>
+          Using <command>vifs</command>, add the new mount to
+          <filename>/etc/fstab</filename>. If it doesn't already have
+          other entries, it should look something like:
+        </para>
 
 <screen>
 #
@@ -219,29 +402,16 @@ alice$ mount
 #
 # Failure to do so is unsupported and may be destructive.
 #
-LABEL=Nix\040Store /nix apfs rw
+LABEL=Nix\040Store /nix apfs rw,nobrowse
 </screen>
 
-      <para>
+        <para>
-        On macOS volumes are also mounted quite late, launchd services or other
+          The nobrowse setting will keep Spotlight from indexing this
-        things that start during login will start before our volume is mounted.
+          volume, and keep it from showing up on your desktop.
-        For these cases eg. <command>wait4path</command> must be used for
+        </para>
-        things that depend on <filename>/nix</filename>.
+      </listitem>
-      </para>
+    </orderedlist>
-
+  </section>
-      <para>
-        This new volume also won't be encrypted by default, and enabling it
-        requires extra setup.  For machines with a <link xlink:href="https://www.apple.com/euro/mac/shared/docs/Apple_T2_Security_Chip_Overview.pdf">T2 chip</link>
-        all data is already entrypted at rest, older hardware won't even when
-        FileVault is enabled for the rest of the system.
-      </para>
-
-<screen>
-alice$ diskutil apfs enableFileVault /nix -user disk
-</screen>
-
-    </listitem>
-  </itemizedlist>
 
 </section>
 

scripts/create-darwin-volume.sh

@@ -59,10 +59,45 @@ test_nix() {
     test -d "/nix"
 }
 
-test_filevault() {
+test_t2_chip_present(){
+    # Use xartutil to see if system has a t2 chip.
+    #
+    # This isn't well-documented on its own; until it is,
+    # let's keep track of knowledge/assumptions.
+    #
+    # Warnings:
+    # - Don't search "xart" if porn will cause you trouble :)
+    # - Other xartutil flags do dangerous things. Don't run them
+    #   naively. If you must, search "xartutil" first.
+    #
+    # Assumptions:
+    # - the "xART session seeds recovery utility"
+    #   appears to interact with xartstorageremoted
+    # - `sudo xartutil --list` lists xART sessions
+    #   and their seeds and exits 0 if successful. If
+    #   not, it exits 1 and prints an error such as:
+    #   xartutil: ERROR: No supported link to the SEP present
+    # - xART sessions/seeds are present when a T2 chip is
+    #   (and not, otherwise)
+    # - the presence of a T2 chip means a newly-created
+    #   volume on the primary drive will be
+    #   encrypted at rest
+    # - all together: `sudo xartutil --list`
+    #   should exit 0 if a new Nix Store volume will
+    #   be encrypted at rest, and exit 1 if not.
+    sudo xartutil --list >/dev/null 2>/dev/null
+}
+
+test_filevault_in_use() {
     disk=$1
-    apfs_volumes_for "$disk" | volume_list_true FileVault | grep -q true || return
+    #      list vols on disk | get value of Filevault key | value is true
-    ! sudo xartutil --list >/dev/null 2>/dev/null
+    apfs_volumes_for "$disk" | volume_list_true FileVault | grep -q true
+}
+
+# use after error msg for conditions we don't understand
+suggest_report_error(){
+    # ex "error: something sad happened :(" >&2
+    echo "       please report this @ https://github.com/nixos/nix/issues" >&2
 }
 
 main() {
@@ -89,7 +124,8 @@ main() {
         echo "Configuring /etc/synthetic.conf..." >&2
         echo nix | sudo tee /etc/synthetic.conf
         if ! test_synthetic_conf; then
-            echo "error: failed to configure synthetic.conf" >&2
+            echo "error: failed to configure synthetic.conf;" >&2
+            suggest_report_error
             exit 1
         fi
     fi
@@ -101,7 +137,8 @@ main() {
             sudo mkdir -p /nix 2>/dev/null || true
         fi
         if ! test_nix; then
-            echo "error: failed to bootstrap /nix, a reboot might be required" >&2
+            echo "error: failed to bootstrap /nix; if a reboot doesn't help," >&2
+            suggest_report_error
             exit 1
         fi
     fi
@@ -111,10 +148,25 @@ main() {
     if [ -z "$volume" ]; then
         echo "Creating a Nix Store volume..." >&2
 
-        if test_filevault "$disk"; then
+        if test_filevault_in_use "$disk"; then
-            echo "error: FileVault detected, refusing to create unencrypted volume" >&2
+            # TODO: Not sure if it's in-scope now, but `diskutil apfs list`
-            echo "See https://nixos.org/nix/manual/#sect-apfs-volume-installation" >&2
+            # shows both filevault and encrypted at rest status, and it
-            exit 1
+            # may be the more semantic way to test for this? It'll show
+            # `FileVault:                 No (Encrypted at rest)`
+            # `FileVault:                 No`
+            # `FileVault:                 Yes (Unlocked)`
+            # and so on.
+            if test_t2_chip_present; then
+                echo "warning: boot volume is FileVault-encrypted, but the Nix store volume" >&2
+                echo "         is only encrypted at rest." >&2
+                echo "         See https://nixos.org/nix/manual/#sect-macos-installation" >&2
+            else
+                echo "error: refusing to create Nix store volume because the boot volume is" >&2
+                echo "       FileVault encrypted, but encryption-at-rest is not available." >&2
+                echo "       Manually create a volume for the store and re-run this script." >&2
+                echo "       See https://nixos.org/nix/manual/#sect-macos-installation" >&2
+                exit 1
+            fi
         fi
 
         sudo diskutil apfs addVolume "$disk" APFS 'Nix Store' -mountpoint /nix
@@ -128,13 +180,6 @@ main() {
         label=$(echo "$volume" | sed 's/ /\\040/g')
         printf "\$a\nLABEL=%s /nix apfs rw,nobrowse\n.\nwq\n" "$label" | EDITOR=ed sudo vifs
     fi
-
-    echo "" >&2
-    echo "The following options can be enabled to disable spotlight indexing" >&2
-    echo "of the volume, which might be desirable." >&2
-    echo "" >&2
-    echo "   $ sudo mdutil -i off /nix" >&2
-    echo "" >&2
 }
 
 main "$@"

scripts/install-nix-from-closure.sh

@@ -52,7 +52,7 @@ while [ $# -gt 0 ]; do
             NIX_INSTALLER_NO_CHANNEL_ADD=1;;
         --no-modify-profile)
             NIX_INSTALLER_NO_MODIFY_PROFILE=1;;
-        --create-volume)
+        --darwin-use-unencrypted-nix-store-volume)
             CREATE_DARWIN_VOLUME=1;;
         *)
             (
@@ -77,12 +77,13 @@ while [ $# -gt 0 ]; do
                 echo ""
             ) >&2
 
-            if [ "$(uname -s)" = "Darwin" ]; then
+            # darwin and Catalina+
+            if [ "$(uname -s)" = "Darwin" ] && [ "$macos_major" -gt 14 ]; then
                 (
-                    echo " --create-volume: Create an APFS volume for the store and create the /nix"
+                    echo " --darwin-use-unencrypted-nix-store-volume: Create an APFS volume for the Nix"
-                    echo "                  mountpoint for it using synthetic.conf."
+                    echo "              store and mount it at /nix. This is the recommended way to create"
-                    echo "                  (required on macOS >=10.15)"
+                    echo "              /nix with a read-only / on macOS >=10.15."
-                    echo "                  See https://nixos.org/nix/manual/#sect-apfs-volume-installation"
+                    echo "              See: https://nixos.org/nix/manual/#sect-macos-installation"
                     echo ""
                 ) >&2
             fi
@@ -98,12 +99,12 @@ if [ "$(uname -s)" = "Darwin" ]; then
     fi
 
     info=$(diskutil info -plist / | xpath "/plist/dict/key[text()='Writable']/following-sibling::true[1]" 2> /dev/null)
-    if ! [ -e $dest ] && [ -n "$info" ]; then
+    if ! [ -e $dest ] && [ -n "$info" ] && [ "$macos_major" -gt 14 ]; then
         (
             echo ""
             echo "Installing on macOS >=10.15 requires relocating the store to an apfs volume."
-            echo "Use sh <(curl https://nixos.org/nix/install) --create-volume or run the preparation steps manually."
+            echo "Use sh <(curl https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume or run the preparation steps manually."
-            echo "See https://nixos.org/nix/manual/#sect-apfs-volume-installation"
+            echo "See https://nixos.org/nix/manual/#sect-macos-installation"
             echo ""
         ) >&2
         exit 1