(cautiously) enable the sandbox by default on macOS #386

New issue

Open

opened 2024-06-11 01:35:18 +00:00 by jade · 3 comments

jade commented

2024-06-11 01:35:18 +00:00

Owner

Currently the sandbox is disabled by default on macOS. However, to the best of our knowledge, it is not disabled for good reasons. That is to say, it is enabled on Hydra, and it is generally (with almost no exceptions) not tolerated to disable it in nixpkgs.

One exception, e.g.: 8fd4097529/pkgs/development/mobile/xcodeenv/compose-xcodewrapper.nix (L1-L44)

I think that we could set it to relaxed for a release or two before defaulting to on.

cc @puck

Currently the sandbox is disabled by default on macOS. However, to the best of our knowledge, it is not disabled for *good reasons*. That is to say, it is enabled on Hydra, and it is generally (with almost no exceptions) not tolerated to disable it in nixpkgs. One exception, e.g.: https://github.com/nixos/nixpkgs/blob/8fd4097529b4a71e0de000c90c8c4279f534eada/pkgs/development/mobile/xcodeenv/compose-xcodewrapper.nix#L1-L44 I think that we *could* set it to `relaxed` for a release or two before defaulting to on. cc @puck

jade added this to the post-release milestone

2024-06-11 01:35:18 +00:00

jade added the

stability

OS/macOS

Area/store

labels

2024-06-11 01:35:18 +00:00

szlend commented

2024-06-12 21:01:46 +00:00

Hydra doesn’t have the sandbox enabled. I’ve had to fix things that worked because of sandboxing but failed because Hydra picked up the system Python due to not having sandboxing enabled.

It also has some severe limitations. It can’t handle derivations with a lot of paths, and networking cannot be namespaced, so running multiple builds at the same time can interfere with each other. That’s why there is a special __darwinAllowLocalNetworking variable to enable it on a case by case basis.

But if you do something like nix build nixpkgs#legacyPackages.{aarch64,x86_64}-darwin#python3Packages.request, they will probably interefere with each other and cause build failures.

To put it another way: I used to run with relaxed sandbox. When I started working on the stdenv last year, I eventually stopped because it was such an enormous time sink fixing everything that is broken when sandboxing is enabled (even relaxed sandboxing). Users can get by with sandboxing enabled because Hydra’s lack of sandboxing hides the problems.

I think Nix should have some sandbox profile enabled to block system paths but not really try to implement one beyond that. Darwin just does not have the tools to do it well, but stopping derivations from picking up Homebrew stuff is better than doing nothing with the sandbox completely disabled.

Enabling it by default will require extensive work on nixpkgs or find a better compromise.

I would love to have a functioning sandbox on Darwin, but if we're talking about Hydra for nixpkgs, then I believe the sandbox is still flat out disabled there. At least according to the main Darwin maintainers: > Hydra doesn’t have the sandbox enabled. I’ve had to fix things that worked because of sandboxing but failed because Hydra picked up the system Python due to not having sandboxing enabled. > It also has some severe limitations. It can’t handle derivations with a lot of paths, and networking cannot be namespaced, so running multiple builds at the same time can interfere with each other. That’s why there is a special __darwinAllowLocalNetworking variable to enable it on a case by case basis. > But if you do something like nix build nixpkgs#legacyPackages.{aarch64,x86_64}-darwin#python3Packages.request, they will probably interefere with each other and cause build failures. > To put it another way: I used to run with relaxed sandbox. When I started working on the stdenv last year, I eventually stopped because it was such an enormous time sink fixing everything that is broken when sandboxing is enabled (even relaxed sandboxing). Users can get by with sandboxing enabled because Hydra’s lack of sandboxing hides the problems. > I think Nix should have some sandbox profile enabled to block system paths but not really try to implement one beyond that. Darwin just does not have the tools to do it well, but stopping derivations from picking up Homebrew stuff is better than doing nothing with the sandbox completely disabled. Enabling it by default will require extensive work on nixpkgs or find a better compromise.

szlend commented

2024-06-23 11:44:28 +00:00

Pulling this from a conversation on Matrix:

I think the issue is not how the sandbox is configured, but more how it works. It doesn't really provide any isolation like Linux namespaces or chroot. It's more akin to SElinux or AppArmor where you just define access control to resources and there's no one profile you can just use all across the board. So you have to pass this complexity down to package maintainers. Most of them being Linux/NixOS users and (understandably) not wanting to deal with this

I think one interesting approach (as suggested by nixpkgs darwin maintainers) could be making the default sandbox profile much more permissive. Basically just prevent picking up homebrew/system dependencies, prevent network access, etc. We could then introduce progressively stricter sandbox profiles that package maintainers could opt-in into.

This way we don't block Linux package maintainers that don't want to deal with darwin sandbox issues while still allowing darwin maintainers to improve reproducibility if they want to. Also this way we could have some sort of sandbox enabled on Hydra without having to fix thousands of packages.

Having a permissive sandbox is still preferable to no sandbox at all. We shouldn't let perfection get in the way of progress.

Pulling this from a conversation on Matrix: > I think the issue is not how the sandbox is configured, but more how it works. It doesn't really provide any isolation like Linux namespaces or chroot. It's more akin to SElinux or AppArmor where you just define access control to resources and there's no one profile you can just use all across the board. So you have to pass this complexity down to package maintainers. Most of them being Linux/NixOS users and (understandably) not wanting to deal with this I think one interesting approach (as suggested by nixpkgs darwin maintainers) could be making the *default* sandbox profile much more permissive. Basically just prevent picking up homebrew/system dependencies, prevent network access, etc. We could then introduce progressively stricter sandbox profiles that package maintainers could opt-in into. This way we don't block Linux package maintainers that don't want to deal with darwin sandbox issues while still allowing darwin maintainers to improve reproducibility if they want to. Also this way we could have *some* sort of sandbox enabled on Hydra without having to fix thousands of packages. Having a permissive sandbox is still preferable to no sandbox at all. We shouldn't let perfection get in the way of progress.

szlend commented

2024-06-23 11:56:22 +00:00

It also has some severe limitations. It can’t handle derivations with a lot of paths, and networking cannot be namespaced, so running multiple builds at the same time can interfere with each other. That’s why there is a special __darwinAllowLocalNetworking variable to enable it on a case by case basis.

But if you do something like nix build nixpkgs#legacyPackages.{aarch64,x86_64}-darwin#python3Packages.request, they will probably interefere with each other and cause build failures.

I wonder if __darwinAllowLocalNetworking could be extended to explicitly declare which ports the derivation needs. The Lix build scheduler could then reorder builds so no two derivations with the same resource requirements (ports) run at the same time.

Edit: Tbh we could do this already without extending __darwinAllowLocalNetworking. It would just mean that any derivation tagged with __darwinAllowLocalNetworking would be built sequentially. Which is reasonable as long as it excludes FOD.

> It also has some severe limitations. It can’t handle derivations with a lot of paths, and networking cannot be namespaced, so running multiple builds at the same time can interfere with each other. That’s why there is a special __darwinAllowLocalNetworking variable to enable it on a case by case basis. > But if you do something like nix build nixpkgs#legacyPackages.{aarch64,x86_64}-darwin#python3Packages.request, they will probably interefere with each other and cause build failures. I wonder if `__darwinAllowLocalNetworking` could be extended to explicitly declare which ports the derivation needs. The Lix build scheduler could then reorder builds so no two derivations with the same resource requirements (ports) run at the same time. Edit: Tbh we could do this already without extending `__darwinAllowLocalNetworking`. It would just mean that any derivation tagged with `__darwinAllowLocalNetworking` would be built sequentially. Which is reasonable as long as it excludes FOD.

jade referenced this issue

2025-03-06 04:41:30 +00:00

Port macOS sandbox profile size fix #718

jade referenced this issue

2025-07-28 02:20:10 +00:00

sandbox setting is a horrifying field of landmines of implementation complexity #936