lix-project/lix

Fork 59

bind-mount nix build sandbox before building derivations to allow `noexec` on `/tmp` #610

New issue

Closed

opened 2024-12-28 23:21:07 +00:00 by lordgrimmauld · 3 comments

lordgrimmauld commented

2024-12-28 23:21:07 +00:00

Member

/tmpcan be mounted with mount option noexec. This is paranoid, but quite a common thing while hardening linux systems. Currently, the nix-daemon expects the TMPDIR to be mounted executable. Specifically, some nix build tasks fail if this assumption about TMPDIR is unmet.

This is an issue i came across during hardening experiments on NixOS, but can very well break on non-NixOS systems as well. To reproduce, setting up a VM and mounting /tmp with noexec is not quite enough. Not all nix builds require the mount to be executable. One failure mode i found is any rust package depending on serde, but there are most certainly more failing packages.

I believe this is an issue, because this makes packages sometimes fail in a way that is entirely dependent on the host setup. This means this directly impacts reproducibility.

Describe the solution you'd like

The nix daemon (here lix) should explicitly list assumptions and make sure they are met. The ideal way is to use a bind mount to create an executable mount for the build sandbox. mount bind /tmp/noexec-dir /tmp/exec-dir -o exec will make the kernel drop the noexec option otherwise inherited from the parent file system. This bind mount can even optionally be in-place.

Adding this bind mount would mean TMPDIR would still be respected, but with explicit assumptions about the underlying file system. Bind mounts basically cost nothing to create or to destroy.

Adding these changes would mean lix could build derivations in a more reproducible way while keeping the security benefits users might chose with a noexec /tmp directory. These changes should be compatible with any system running a kernel that supports bind mounts. This might not be all systems that Lix supports! Linux would work, but some of the more exotic platforms might not support this. In which case, there might need to be special casing. Not sure how best to handle this.

Describe alternatives you've considered

One obvious option is to just not support noexec on the TMPDIR. This would mean any user choosing noexec for /tmp would have to provide an explicit TMPDIR mounted as executable. This is my current workaround [EXTERNAL LINK]. However, this is very inconvenient on non-NixOS systems where this immediately means messing with the fstab and manually changing systemd rules. And double irritating on specifically hardened impermanent systems like TailsOS.

An alternative to using a bind mount can be using a tmpfs for the build sandbox. @atemu had voiced some concerns about this approach, as RAM limitations as well as other weirdness specific to tmpfs might cause unexpected side effects. I believe it is worth considering, but not likely to be a viable alternative to a simple bind mount.

Additional context

bind mounts can give exec

Yes, bind mounts do allow exec if mounted with -o exec even if their mount parent is noexec. I did receive some confusion about this, see screenshot for confirmation.

tmpfs on nixos

For those not trusting external links, the current workaround is this:

let
 	  nix_build = "/nix/build-sandbox";
in {
 	  fileSystems."${nix_build}" = {
	    # can execute
	    device = "none";
	    fsType = "tmpfs";
	    options = [
	      "defaults"
	      "size=30%"
	      "mode=755"
	      "exec"
	    ];
	  };
	
	  systemd.services.nix-daemon.environment.TMPDIR = nix_build;
}

bind mount on nixos

Similarly, bind would work as a workaround too:

let
 	  nix_build = "/nix/build-sandbox";
in {
 	  fileSystems."${nix_build}" = {
	    # can execute
 	    device = "/tmp";
	    options = [
	      "bind"
	      "exec"
	    ];
	  };
	
	  systemd.services.nix-daemon.environment.TMPDIR = nix_build;
}

Test VM with `/tmp` as noexec

This VM can be built with nix-build '<nixpkgs/nixos>' -A vm -I nixpkgs=channel:nixos-24.11 -I nixos-config=./configuration.nix.
CAREFUL: requires a lot of RAM so you can use this VM to try and build nix packages.

{ pkgs, ... }:
{
  imports = [
    <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix>
  ];

  users.users.test = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    initialPassword = "test";
  };

  nix.package = pkgs.lix;
  nix.settings.experimental-features = [
    "nix-command"
    "flakes"
    "pipe-operator"
  ];
  
  system.stateVersion = "24.11";

  virtualisation.graphics = false;
  virtualisation.memorySize = 8192;

  virtualisation.fileSystems."/tmp" = {
    fsType = "tmpfs";
    device = "none";
    options = [
      "mode=777"
      "noexec" # the interesting bit
    ];
  };
}

example for a failing build

nix build github:lordgrimmauld/aa-alias-manager for example, there is more packages, this is where i first found it.
Log (notice the permission denied/never executed cased by noexec):

[test@nixos:~]$ nix log /nix/store/ijfw64iagbc44ys0l9bw811arkwg9xrk-aa-alias-manager-unstable-2024-11-01.drv | cat
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/ijfw64iagbc44ys0l9bw811arkwg9xrk-aa-alias-manager-unstable-2024-11-01.drv^*'
Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/qk2nw1m4qjfj7g0cj39mm4pphpjiffpc-source
source root is source
Executing cargoSetupPostUnpackHook
Finished cargoSetupPostUnpackHook
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Executing cargoSetupPostPatchHook
Validating consistency between /build/source/Cargo.lock and /build/cargo-vendor-dir/Cargo.lock
Finished cargoSetupPostPatchHook
Running phase: updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
Running phase: buildPhase
@nix { "action": "setPhase", "phase": "buildPhase" }
Executing cargoBuildHook
cargoBuildHook flags: -j 1 --target x86_64-unknown-linux-gnu --offline --profile release
   Compiling proc-macro2 v1.0.89
error: failed to run custom build command for `proc-macro2 v1.0.89`

Caused by:
  could not execute process `/build/source/target/release/build/proc-macro2-35dfd481bb3f98b2/build-script-build` (never executed)

Caused by:
  Permission denied (os error 13)

[test@nixos:~]$

## Is your feature request related to a problem? Please describe. `/tmp`can be mounted with mount option `noexec`. This is paranoid, but quite a common thing while hardening linux systems. Currently, the nix-daemon expects the `TMPDIR` to be mounted executable. Specifically, some `nix build` tasks fail if this assumption about `TMPDIR` is unmet. This is an issue i came across during hardening experiments on NixOS, but can very well break on non-NixOS systems as well. To reproduce, setting up a VM and mounting `/tmp` with `noexec` is not quite enough. Not all nix builds require the mount to be executable. One failure mode i found is any rust package depending on serde, but there are most certainly more failing packages. I believe this is an issue, because this makes packages sometimes fail in a way that is entirely dependent on the host setup. This means this directly impacts reproducibility. ## Describe the solution you'd like The nix daemon (here lix) should explicitly list assumptions and make sure they are met. The ideal way is to use a bind mount to create an executable mount for the build sandbox. `mount bind /tmp/noexec-dir /tmp/exec-dir -o exec` will make the kernel drop the `noexec` option otherwise inherited from the parent file system. This bind mount can even optionally be in-place. Adding this bind mount would mean `TMPDIR` would still be respected, but with *explicit assumptions* about the underlying file system. Bind mounts basically cost nothing to create or to destroy. Adding these changes would mean lix could build derivations in a more reproducible way while keeping the security benefits users might chose with a `noexec` `/tmp` directory. These changes should be compatible with any system running a kernel that supports bind mounts. *This might not be all systems that Lix supports!* Linux would work, but some of the more exotic platforms might not support this. In which case, there might need to be special casing. Not sure how best to handle this. ## Describe alternatives you've considered One obvious option is to just not support `noexec` on the `TMPDIR`. This would mean any user choosing `noexec` for `/tmp` would have to provide an explicit `TMPDIR` mounted as executable. [This is my current workaround [EXTERNAL LINK]](https://git.grimmauld.de/Grimmauld/grimm-nixos-laptop/src/commit/883d5edcd99af1b6d70742bacdd23f5b0eeb4bea/specific/grimm-nixos-ssd/hardware-configuration.nix#L115-L127.). However, this is very inconvenient on non-NixOS systems where this immediately means messing with the fstab and manually changing systemd rules. And double irritating on specifically hardened impermanent systems like TailsOS. An alternative to using a bind mount can be using a `tmpfs` for the build sandbox. @atemu had voiced some concerns about this approach, as RAM limitations as well as other weirdness specific to tmpfs might cause unexpected side effects. I believe it is worth considering, but not likely to be a viable alternative to a simple bind mount. ## Additional context ### bind mounts can give exec Yes, bind mounts do allow `exec` if mounted with `-o exec` *even if their mount parent is `noexec`*. I did receive some confusion about this, see screenshot for confirmation. ![image](/attachments/7c8ffe32-e9db-4972-900a-35a8ad5b1c8e) ### tmpfs on nixos For those not trusting external links, the current workaround is this: ```nix let nix_build = "/nix/build-sandbox"; in { fileSystems."${nix_build}" = { # can execute device = "none"; fsType = "tmpfs"; options = [ "defaults" "size=30%" "mode=755" "exec" ]; }; systemd.services.nix-daemon.environment.TMPDIR = nix_build; } ``` ### bind mount on nixos Similarly, bind would work as a workaround too: ```nix let nix_build = "/nix/build-sandbox"; in { fileSystems."${nix_build}" = { # can execute device = "/tmp"; options = [ "bind" "exec" ]; }; systemd.services.nix-daemon.environment.TMPDIR = nix_build; } ``` ### Test VM with `/tmp` as noexec This VM can be built with `nix-build '<nixpkgs/nixos>' -A vm -I nixpkgs=channel:nixos-24.11 -I nixos-config=./configuration.nix`. CAREFUL: requires a lot of RAM so you can use this VM to try and build nix packages. ```nix { pkgs, ... }: { imports = [ <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix> ]; users.users.test = { isNormalUser = true; extraGroups = [ "wheel" ]; initialPassword = "test"; }; nix.package = pkgs.lix; nix.settings.experimental-features = [ "nix-command" "flakes" "pipe-operator" ]; system.stateVersion = "24.11"; virtualisation.graphics = false; virtualisation.memorySize = 8192; virtualisation.fileSystems."/tmp" = { fsType = "tmpfs"; device = "none"; options = [ "mode=777" "noexec" # the interesting bit ]; }; } ``` ### example for a failing build `nix build github:lordgrimmauld/aa-alias-manager` for example, there is more packages, this is where i first found it. Log (notice the permission denied/never executed cased by `noexec`): ``` [test@nixos:~]$ nix log /nix/store/ijfw64iagbc44ys0l9bw811arkwg9xrk-aa-alias-manager-unstable-2024-11-01.drv | cat warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/ijfw64iagbc44ys0l9bw811arkwg9xrk-aa-alias-manager-unstable-2024-11-01.drv^*' Running phase: unpackPhase @nix { "action": "setPhase", "phase": "unpackPhase" } unpacking source archive /nix/store/qk2nw1m4qjfj7g0cj39mm4pphpjiffpc-source source root is source Executing cargoSetupPostUnpackHook Finished cargoSetupPostUnpackHook Running phase: patchPhase @nix { "action": "setPhase", "phase": "patchPhase" } Executing cargoSetupPostPatchHook Validating consistency between /build/source/Cargo.lock and /build/cargo-vendor-dir/Cargo.lock Finished cargoSetupPostPatchHook Running phase: updateAutotoolsGnuConfigScriptsPhase @nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" } Running phase: configurePhase @nix { "action": "setPhase", "phase": "configurePhase" } Running phase: buildPhase @nix { "action": "setPhase", "phase": "buildPhase" } Executing cargoBuildHook cargoBuildHook flags: -j 1 --target x86_64-unknown-linux-gnu --offline --profile release Compiling proc-macro2 v1.0.89 error: failed to run custom build command for `proc-macro2 v1.0.89` Caused by: could not execute process `/build/source/target/release/build/proc-macro2-35dfd481bb3f98b2/build-script-build` (never executed) Caused by: Permission denied (os error 13) [test@nixos:~]$ ```

image.png

2.1 MiB

raito added the

Area/store

label

2024-12-28 23:26:44 +00:00

lordgrimmauld commented

2025-01-18 10:38:19 +00:00

Author

Member

This seems somewhat related to lix-project/lix@3c8096e5cb in which @lilyball allowed setting a specific temp directory for lix. That change makes this workaround significantly less cursed, setting the temp directory for the daemon in an actually intended place! Nice to see that.

Assuring that temp directory is mounted with exec permissions might still be a good idea, but at this point a simple warning/error about the temp directory not being executable could be a valid option too. That would mean the bind mount would still have to happen manually, but at least the failure mode could be loud.

This seems somewhat related to https://git.lix.systems/lix-project/lix/commit/3c8096e5cb94fcb0a007d3f5af94addb6ffc3e5d in which @lilyball allowed setting a specific temp directory for lix. That change makes this workaround significantly less cursed, setting the temp directory for the daemon in an actually intended place! Nice to see that. Assuring that temp directory is mounted with `exec` permissions might still be a good idea, but at this point a simple warning/error about the temp directory not being executable could be a valid option too. That would mean the bind mount would still have to happen manually, but at least the failure mode could be loud.

👍 1

raito commented

2025-01-18 12:44:51 +00:00

Owner

This seems somewhat related to lix-project/lix@3c8096e5cb in which @lilyball allowed setting a specific temp directory for lix. That change makes this workaround significantly less cursed, setting the temp directory for the daemon in an actually intended place! Nice to see that.

Assuring that temp directory is mounted with exec permissions might still be a good idea, but at this point a simple warning/error about the temp directory not being executable could be a valid option too. That would mean the bind mount would still have to happen manually, but at least the failure mode could be loud.

Do you feel like you could send us such a change?

> This seems somewhat related to https://git.lix.systems/lix-project/lix/commit/3c8096e5cb94fcb0a007d3f5af94addb6ffc3e5d in which @lilyball allowed setting a specific temp directory for lix. That change makes this workaround significantly less cursed, setting the temp directory for the daemon in an actually intended place! Nice to see that. > > Assuring that temp directory is mounted with `exec` permissions might still be a good idea, but at this point a simple warning/error about the temp directory not being executable could be a valid option too. That would mean the bind mount would still have to happen manually, but at least the failure mode could be loud. Do you feel like you could send us such a change?