lix-project/lix

Fork 56

custom CA cert needs to be explicitly added to extra-sandbox-paths #885

New issue

Closed

opened 2025-06-27 19:12:57 +00:00 by flokli · 9 comments

flokli commented

2025-06-27 19:12:57 +00:00

Describe the bug

Opened after a discussion with @Raito.

While teaching an FOD fetcher to accept lib.fetchers.proxyImpureEnvVars, I noticed that for some reason setting HTTP_PROXY, HTTPS_PROXY and NIX_SSL_CERT_FILE in nix-daemon.service is not sufficient.

I still need to pass `--option extra-sandbox-paths /nix/store/…-mitmproxy-ca-cert.pem to get the build to be able to access the file, otherwise I get a "file not found error" from the fetcher when trying to read the file.

I assumed NIX_SSL_CERT_FILE would act as an override to the ssl-cert-file option, and there already is logic in place to make things passed that way available in the sandbox, but that code doesn't seem to trigger.

Steps To Reproduce

Checkout https://github.com/NixOS/nixpkgs/pull/420608 and follow the testing instructions, but don't set extra-sandbox-paths

Expected behavior

I'd expect NIX_SSL_CERT_FILE as an override to the Nix' ssl-cert-file option to also allow access to this path from inside the build sandbox.

`nix --version` output

nix (Lix, like Nix) 2.91.2

## Describe the bug Opened after a discussion with @Raito. While teaching an FOD fetcher to accept `lib.fetchers.proxyImpureEnvVars`, I noticed that for some reason setting `HTTP_PROXY`, `HTTPS_PROXY` and `NIX_SSL_CERT_FILE` in `nix-daemon.service` is not sufficient. I still need to pass `--option extra-sandbox-paths /nix/store/…-mitmproxy-ca-cert.pem to get the build to be able to access the file, otherwise I get a "file not found error" from the fetcher when trying to read the file. I assumed `NIX_SSL_CERT_FILE` would act as an override to the `ssl-cert-file` option, and there already is logic in place to make things passed that way available in the sandbox, but that code doesn't seem to trigger. ## Steps To Reproduce Checkout https://github.com/NixOS/nixpkgs/pull/420608 and follow the testing instructions, but don't set `extra-sandbox-paths` ## Expected behavior I'd expect `NIX_SSL_CERT_FILE` as an override to the Nix' `ssl-cert-file` option to also allow access to this path from inside the build sandbox. ## `nix --version` output nix (Lix, like Nix) 2.91.2

flokli added the

bug

label

2025-06-27 19:12:57 +00:00

emilazy commented

2025-06-28 18:02:55 +00:00

Member

Related: https://gerrit.lix.systems/c/lix/+/2906? (I thought it worked on Linux but not Darwin but I assume you’re seeing it not‐work on Linux too?)

Related: <https://gerrit.lix.systems/c/lix/+/2906>? (I thought it worked on Linux but not Darwin but I assume you’re seeing it not‐work on Linux too?)

flokli commented

2025-06-28 18:58:32 +00:00

Author

I'm seeing it non-working on Linux, yes.

tomasajt commented

2025-06-29 15:22:57 +00:00

I have not been using nix for that long so I do not know if it has been any different previously, but I don't think it ever worked on Linux before.

There is this logic, which seems to have been made to support using the configured caFile:

                 if (settings.caFile != "" && pathExists(settings.caFile)) {
                     // For the same reasons as above, copy the CA certificates file too.
                     // It should be even less likely to change during the build than resolv.conf.
                     createDirs(chrootRootDir + "/etc/ssl/certs");
                     copyFile(settings.caFile, chrootRootDir + "/etc/ssl/certs/ca-certificates.crt", { .followSymlinks = true });
                 }

But it only puts the caFile under a hardcoded location, not to the same path as configured by NIX_SSL_CERT_FILE.
Using extra-sandbox-paths does actually use the same path, this is why it works fine.

As I said in https://github.com/NixOS/nix/issues/12698#issuecomment-3014208342 , I don't think having NIX_SSL_CERT_FILE as an impure env var is a good solution since we can't guarantee the same layout in the build sandbox vs in the normal files system.
Having an env var (possible called NIX_SSL_CERT_FILE, or anything else) automagically point to a location where to caFile was copied to (possible a hardcoded location, like it is currently done with linux) is a better solution.

This would get rid of our dependency on the cacert from the nixpkgs of the derivation and instead depend on just the cacert referenced in the nix-conf file.

(I have not looked into the internals of the cacert package itself, so maybe there are some other things to take into consideration before stripping it out of nixpkgs)
Of course, another thing to consider is the fact that this wouldn't really be backwards compatible if we completely removed the usage of the cacert hook (since old nix versions wouldn't have the magic env var pointing to a correct caFile)

I have not been using nix for that long so I do not know if it has been any different previously, but I don't think it ever worked on Linux before. There is this logic, which seems to have been made to support using the configured caFile: https://git.lix.systems/lix-project/lix/src/commit/33122e79dfe51f143158a25a087f7aa0a2bf19d0/lix/libstore/build/local-derivation-goal.cc#L1146-L1151 But it only puts the caFile under a hardcoded location, not to the same path as configured by NIX_SSL_CERT_FILE. Using `extra-sandbox-paths` does actually use the same path, this is why it works fine. As I said in https://github.com/NixOS/nix/issues/12698#issuecomment-3014208342 , I don't think having NIX_SSL_CERT_FILE as an impure env var is a good solution since we can't guarantee the same layout in the build sandbox vs in the normal files system. Having an env var (possible called NIX_SSL_CERT_FILE, or anything else) automagically point to a location where to caFile was copied to (possible a hardcoded location, like it is currently done with linux) is a better solution. This would get rid of our dependency on the cacert from the nixpkgs of the derivation and instead depend on just the cacert referenced in the nix-conf file. (I have not looked into the internals of the cacert package itself, so maybe there are some other things to take into consideration before stripping it out of nixpkgs) Of course, another thing to consider is the fact that this wouldn't really be backwards compatible if we completely removed the usage of the cacert hook (since old nix versions wouldn't have the magic env var pointing to a correct caFile)

emilazy commented

2025-07-02 14:48:13 +00:00

Member

Here is my proposal that should work uniformly on both Linux and Darwin and avoid the complex canonicalization logic in cl/2906. Would appreciate feedback from @tomasajt and @lilyball.

We ensure that from a derivation build’s point of view, NIX_SSL_CERT_FILE is always set, and set to a path underneath the parent of the logical store directory as seen by that build (e.g., /nix/var/nix/ca-certificates/…).
On Linux, we ensure that this path is accessible from inside the build’s filesystem namespace one way or another.
On Darwin, we physically copy the daemon’s NIX_SSL_CERT_FILE/SSL_CERT_FILE/ssl-cert-file to the location, and expose it in the sandbox. Since we control the path, we do not need the complex logic in cl/2906. We may have to do this on Linux for unsandboxed builds too? I do not know if Linux builds ever skip the filesystem namespace.

Exactly what the path should be, I am not sure. It must be under /nix on Darwin, as we cannot map other paths and do not own any other, and on Darwin the physical and logical store directories are always equal for builds being run on the local machine. Since exposing the physical store directory on Linux would be bad for chroot stores and so on, the path as seen by the builder must therefore be under the parent of the logical store directory.

The trivial option is to make it /nix/var/nix/ca-certificates.crt, copy it once at daemon startup, and then never update it. I am not sure if that works well with --store local, since it could be overwritten in the middle of another build. (On the other hand, that’s probably not a big problem as long as it’s atomic.) It also is a somewhat suboptimal UX when the certificates get updated, since you have to restart the daemon.

A fancier option would be to atomically copy it periodically or at the start of a build, and allow it to be overwritten in the middle of another FOD build. On Linux you could just do a bind mount and skip the actual copying logic, I guess.

An even fancier option would be to copy at the start of every build to /nix/var/nix/ca-certificates/‹hash of file›.crt, and collect unused ones when no build no longer references them.

And at this point I’m realizing that… actually, can’t we just treat it as a nix-store --add, copy it into /nix/store, pass it into the sandbox as if it was a real dependency, and root it for the duration of a build? That seems like simple logic that gives us all the properties we’d want. Cost: hashing ~500 KiB per FOD to determine whether we need to copy it to the store, I suppose? But we could do this even with the “never update it except when the daemon starts” scheme. So that’s probably better.

Here is my proposal that should work uniformly on both Linux and Darwin and avoid the complex canonicalization logic in cl/2906. Would appreciate feedback from @tomasajt and @lilyball. 1. We ensure that from a derivation build’s point of view, `NIX_SSL_CERT_FILE` is always set, and set to a path underneath the parent of the logical store directory as seen by that build (e.g., `/nix/var/nix/ca-certificates/…`). 2. On Linux, we ensure that this path is accessible from inside the build’s filesystem namespace one way or another. 3. On Darwin, we physically copy the daemon’s `NIX_SSL_CERT_FILE`/`SSL_CERT_FILE`/`ssl-cert-file` to the location, and expose it in the sandbox. Since we control the path, we do not need the complex logic in cl/2906. We may have to do this on Linux for unsandboxed builds too? I do not know if Linux builds ever skip the filesystem namespace. Exactly what the path should be, I am not sure. It must be under `/nix` on Darwin, as we cannot map other paths and do not own any other, and on Darwin the physical and logical store directories are always equal for builds being run on the local machine. Since exposing the physical store directory on Linux would be bad for `chroot` stores and so on, the path as seen by the builder must therefore be under the parent of the logical store directory. The trivial option is to make it `/nix/var/nix/ca-certificates.crt`, copy it once at daemon startup, and then never update it. I am not sure if that works well with `--store local`, since it could be overwritten in the middle of another build. (On the other hand, that’s probably not a big problem as long as it’s atomic.) It also is a somewhat suboptimal UX when the certificates get updated, since you have to restart the daemon. A fancier option would be to atomically copy it periodically or at the start of a build, and allow it to be overwritten in the middle of another FOD build. On Linux you could just do a bind mount and skip the actual copying logic, I guess. An even fancier option would be to copy at the start of every build to `/nix/var/nix/ca-certificates/‹hash of file›.crt`, and collect unused ones when no build no longer references them. And at this point I’m realizing that… actually, can’t we just treat it as a `nix-store --add`, copy it into `/nix/store`, pass it into the sandbox as if it was a real dependency, and root it for the duration of a build? That seems like simple logic that gives us all the properties we’d want. Cost: hashing ~500 KiB per FOD to determine whether we need to copy it to the store, I suppose? But we could do this even with the “never update it except when the daemon starts” scheme. So that’s probably better.

flokli commented

2025-07-04 09:44:14 +00:00

Author

And at this point I’m realizing that… actually, can’t we just treat it as a nix-store --add, copy it into /nix/store, pass it into the sandbox as if it was a real dependency, and root it for the duration of a build? That seems like simple logic that gives us all the properties we’d want. Cost: hashing ~500 KiB per FOD to determine whether we need to copy it to the store, I suppose? But we could do this even with the “never update it except when the daemon starts” scheme. So that’s probably better.

Importing whatever NIX_SSL_CERT_FILE / settings.caFile into the store, making that visible in the build (but not part of the references, only if also explicitly in the build inputs), and setting NIX_SSL_CERT_FILE in the build environment to this sounds like a very nice approach, that should work both in Linux and MacOS.

I'd much prefer that over different code paths, and trying to deal with mutations in /nix/var/nix/ca-certificates.crt and all the things they entail :-)

> And at this point I’m realizing that… actually, can’t we just treat it as a nix-store --add, copy it into /nix/store, pass it into the sandbox as if it was a real dependency, and root it for the duration of a build? That seems like simple logic that gives us all the properties we’d want. Cost: hashing ~500 KiB per FOD to determine whether we need to copy it to the store, I suppose? But we could do this even with the “never update it except when the daemon starts” scheme. So that’s probably better. Importing whatever NIX_SSL_CERT_FILE / settings.caFile into the store, making that visible in the build (but not part of the references, only if also explicitly in the build inputs), and setting `NIX_SSL_CERT_FILE` in the build environment to this sounds like a very nice approach, that should work both in Linux and MacOS. I'd much prefer that over different code paths, and trying to deal with mutations in /nix/var/nix/ca-certificates.crt and all the things they entail :-)

👍 3

emilazy commented

2025-07-04 10:12:49 +00:00

Member

Yeah, I had to write the rest of that comment to come up with the obviously correct solution 😅

There’s still design points about whether to do the copy once at daemon / local build startup or to keep it updated for changes (so that e.g. adding enterprise CAs through nix-darwin works seamlessly), and for the latter I’m not sure how much overhead an additional store addition per FOD build would have. But store paths seem like the way forward regardless.

Edit: Oh, and we will want to ensure the path is a disallowedReferences, I guess. Even though the FOD hash should prevent doing anything too weird with that.

Yeah, I had to write the rest of that comment to come up with the obviously correct solution 😅 There’s still design points about whether to do the copy once at daemon / `local` build startup or to keep it updated for changes (so that e.g. adding enterprise CAs through nix-darwin works seamlessly), and for the latter I’m not sure how much overhead an additional store addition per FOD build would have. But store paths seem like the way forward regardless. Edit: Oh, and we will want to ensure the path is a `disallowedReferences`, I guess. Even though the FOD hash should prevent doing anything too weird with that.

tomasajt commented

2025-07-04 10:31:08 +00:00

Edit: Oh, and we will want to ensure the path is a disallowedReferences, I guess. Even though the FOD hash should prevent doing anything too weird with that.

No store paths are allowed inside the output of a FOD, so it should be fine, yeah

Also, just regarding my earlier comment about the cacert hook: I guess we can't really remove it, because 1: it sets up other env vars like SSL_CERT_FILE, 2: it would act as a compatibility layer for older nixes that might not get this change.

> Edit: Oh, and we will want to ensure the path is a `disallowedReferences`, I guess. Even though the FOD hash should prevent doing anything too weird with that. No store paths are allowed inside the output of a FOD, so it should be fine, yeah --- Also, just regarding my earlier comment about the cacert hook: I guess we can't really remove it, because 1: it sets up other env vars like SSL_CERT_FILE, 2: it would act as a compatibility layer for older nixes that might not get this change.

lix-bot commented

2025-07-08 20:10:51 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

comment in cl/2906 ("fix: pass caFile and all symlinks to darwin sandbox")
commit message in cl/3765 ("libstore/build: rewire builder's environment in presence of a global CA")

This issue was mentioned on Gerrit on the following CLs: * comment in [cl/2906](https://gerrit.lix.systems/c/lix/+/2906) ("fix: pass caFile and all symlinks to darwin sandbox") * commit message in [cl/3765](https://gerrit.lix.systems/c/lix/+/3765) ("libstore/build: rewire builder's environment in presence of a global CA")

raito added the

release-blocker

label

2025-07-15 12:56:08 +00:00

raito commented

2025-07-26 00:33:47 +00:00

Owner

I decided not to go for a solution that inserts the CA file in the store because this is not needed in the end.

My view is this is more the interaction of two features that looks like they compose but they are not, reinforced by the fact that nixpkgs set a very weird default as @tomasajt pointed it out.

I propose a solution where we push a warning, fix nixpkgs as we go and make the interaction of the two features a hard error later on.

In the meantime, the CL should restore the functionality as expected in most of the happy cases and leave the possibility for advanced users to still reach out to extra-sandbox-paths if needed.

I decided not to go for a solution that inserts the CA file in the store because this is not needed in the end. My view is this is more the interaction of two features that looks like they compose but *they are not*, reinforced by the fact that nixpkgs set a very weird default as @tomasajt pointed it out. I propose a solution where we push a warning, fix nixpkgs as we go and make the interaction of the two features a hard error later on. In the meantime, the CL should restore the functionality as expected in most of the happy cases and leave the possibility for advanced users to still reach out to `extra-sandbox-paths` if needed.

raito added the

labels

2025-07-26 00:35:07 +00:00