lix-project/lix

Fork 54

Allow client to pass environment variables to FOD builder #590

New issue

Open

opened 2024-12-03 08:33:23 +00:00 by szlend · 9 comments

szlend commented

2024-12-03 08:33:23 +00:00

Currently there's no decent way of passing secrets to builders that need to be able to download files from sources that require authentication.

Current methods:

builtins.fetchTree/builtins.fetchGit - client-side fetching, blocks evaluation, restricted to git/github/gitlab etc..
pkgs.fetch* - FOD builder fetching, requires credentials on the filesystem (netrc)

The ideal solution would be #558, however that's still pretty far away.

Describe the solution you'd like

Nix 2.19 introduced a new experimental feature that I actually quite like, called configurable-impure-env (PR). This allows you to pass impure environmental variables to the daemon. I think it's a pretty decent simple solution until we can do #558.

Example usage:

NIX_CONIFG="impure-env = TOKEN=$(fetch-token-from-key-store)" nix build .#foo

Open questions:

Should this be an experimental feature?
Should impure-env require a user to be trusted?
- If the user does not need to be trusted, we probably want some sort of env var filtering so users can't just override stuff like HTTP_PROXY.
Better CLI UX? Something like --impure-env FOO=bar (set explicitly) / --impure-env FOO (inherit from env)
- Could be a future improvement

Describe alternatives you've considered

We could just not do anything until #558 and keep passing credentials through netrc.

Additional context

Happy to port this from Nix 2.19 myself if maintainers agree and the way we wanna implement this is clear.

## Is your feature request related to a problem? Please describe. Currently there's no decent way of passing secrets to builders that need to be able to download files from sources that require authentication. Current methods: - `builtins.fetchTree`/`builtins.fetchGit` - client-side fetching, blocks evaluation, restricted to git/github/gitlab etc.. - `pkgs.fetch*` - FOD builder fetching, requires credentials on the filesystem (`netrc`) The ideal solution would be #558, however that's still pretty far away. ## Describe the solution you'd like Nix 2.19 introduced a new experimental feature that I actually quite like, called `configurable-impure-env` ([PR](https://github.com/NixOS/nix/pull/8830)). This allows you to pass impure environmental variables to the daemon. I think it's a pretty decent simple solution until we can do #558. Example usage: ``` NIX_CONIFG="impure-env = TOKEN=$(fetch-token-from-key-store)" nix build .#foo ``` Open questions: - Should this be an experimental feature? - Should `impure-env` require a user to be trusted? - If the user does not need to be trusted, we probably want some sort of env var filtering so users can't just override stuff like `HTTP_PROXY`. - Better CLI UX? Something like `--impure-env FOO=bar` (set explicitly) / `--impure-env FOO` (inherit from env) - Could be a future improvement ## Describe alternatives you've considered We could just not do anything until #558 and keep passing credentials through `netrc`. ## Additional context Happy to port this from Nix 2.19 myself if maintainers agree and the way we wanna implement this is clear.

raito commented

2024-12-03 15:16:37 +00:00

Owner

Hello there,

I discussed this with few team members in private who are working in adjacent areas. While I agree / recognize that netrc is insufficient, we need more documentation / examples of where netrc fell short (scenarios, usecases, etc.) so that we can keep our priority on this topic updated and clear. Most of the workloads I work with private environments are, today, already possible with the current netrc capabilities, albeit with Nix code and sandbox impure paths.

Now, onto the feature, one problem with this sort of thing is what are the various policies we should apply, explain, advise to users with multi-tenant contexts (single tenant context is a special case of multi-tenancy here.)

For example, consider that you have a community builder as a remote builder, and you pass a GitHub authentication token as an impure environment variable, any operators of that community builder who are logging or watching for environments over there will be able to capture it, constituting a dangerous scenario.

More than an experimental feature, I think this is a dangerous-feature, because it is a super well-designed assault rifle pointed to the feet of our users. Now, with netrc, the potential for leakage also exist but is better understood with the experience we have with netrc, arguably also netrc forces people to design custom solutions based on sandbox impure paths which needs to be provisioned uniformly on any remote builder or environments you care about.

So we are inclined towards implementing:

client-side building first and foremost
providing policy hooks to decide whether to carry pluggable authentication data on remote builders in non-client contexts (if the scheduling decides so)
designing a pluggable authentication protocol

Some of the architectural work that is being conducted right now should provide the building blocks to achieve this, without paying the cost that CppNix is self-inflicting themselves with, all that technical debt.

Nonetheless, as you remarked, this will take a bit of time.

The position I would aim for is that I would prefer to see more energy spent on implementing the correct path rather than deal with impure-env that might be a distraction which solves a tiny surface of the problem statement.

That being said, I do believe that helping the scenario where you use impure sandbox paths which are shaped as a socket that you can ask for secrets (whether you put a Vault proxy behind or just a simple socat that sends back your secret) can be made more secure by helping the sandbox prove to the socket who they are. I'd like to see more towards identity-aware build contexts which can help for more than just obtaining temporary secrets to do things, but also signing, etc.

Note that this is my opinion as a core team member, @jade who partaked in the thread may have a different one.

Hello there, I discussed this with few team members in private who are working in adjacent areas. While I agree / recognize that netrc is insufficient, we need more documentation / examples of where netrc fell short (scenarios, usecases, etc.) so that we can keep our priority on this topic updated and clear. Most of the workloads I work with private environments are, today, already possible with the current netrc capabilities, albeit with Nix code and sandbox impure paths. Now, onto the feature, one problem with this sort of thing is what are the various policies we should apply, explain, advise to users with multi-tenant contexts (single tenant context is a special case of multi-tenancy here.) For example, consider that you have a community builder as a remote builder, and you pass a GitHub authentication token as an impure environment variable, any operators of that community builder who are logging or watching for environments over there will be able to capture it, constituting a dangerous scenario. More than an experimental feature, I think this is a `dangerous-feature`, because it is a super well-designed assault rifle pointed to the feet of our users. Now, with netrc, the potential for leakage also exist but is better understood with the experience we have with netrc, arguably also netrc forces people to design custom solutions based on sandbox impure paths which needs to be provisioned uniformly on any remote builder or environments you care about. So we are inclined towards implementing: - client-side building first and foremost - providing policy hooks to decide whether to carry pluggable authentication data on remote builders in non-client contexts (if the scheduling decides so) - designing a pluggable authentication protocol Some of the architectural work that is being conducted right now should provide the building blocks to achieve this, without paying the cost that CppNix is self-inflicting themselves with, all that technical debt. Nonetheless, as you remarked, this will take a bit of time. The position I would aim for is that I would prefer to see more energy spent on implementing the correct path rather than deal with impure-env that might be a distraction which solves a tiny surface of the problem statement. That being said, I do believe that helping the scenario where you use impure sandbox paths which are shaped as a socket that you can ask for secrets (whether you put a Vault proxy behind or just a simple `socat` that sends back your secret) can be made more secure by helping the sandbox prove to the socket who they are. I'd like to see more towards identity-aware build contexts which can help for more than just obtaining temporary secrets to do things, but also signing, etc. Note that this is my opinion as a core team member, @jade who partaked in the thread may have a different one.

szlend commented

2024-12-03 16:12:53 +00:00

Author

That being said, I do believe that helping the scenario where you use impure sandbox paths which are shaped as a socket that you can ask for secrets (whether you put a Vault proxy behind or just a simple socat that sends back your secret) can be made more secure by helping the sandbox prove to the socket who they are. I'd like to see more towards identity-aware build contexts which can help for more than just obtaining temporary secrets to do things, but also signing, etc.

Exactly what I was looking into before discovering configurable-impure-env.

However I couldn't figure out how to prove the build process that's requesting credentials is who it says it is. Without that I felt like it was just smoke and mirrors really, so I just went with the simpler solution.

> That being said, I do believe that helping the scenario where you use impure sandbox paths which are shaped as a socket that you can ask for secrets (whether you put a Vault proxy behind or just a simple socat that sends back your secret) can be made more secure by helping the sandbox prove to the socket who they are. I'd like to see more towards identity-aware build contexts which can help for more than just obtaining temporary secrets to do things, but also signing, etc. Exactly what I was looking into before discovering `configurable-impure-env`. However I couldn't figure out how to prove the build process that's requesting credentials is who it says it is. Without that I felt like it was just smoke and mirrors really, so I just went with the simpler solution.

raito commented

2024-12-03 22:50:30 +00:00

Owner

However I couldn't figure out how to prove the build process that's requesting credentials is who it says it is. Without that I felt like it was just smoke and mirrors really, so I just went with the simpler solution.

This is something we could provide in our sandboxes, but we need probably a bit of help on how to design this "right" to ensure that we don't forget anyone for their workloads.

> However I couldn't figure out how to prove the build process that's requesting credentials is who it says it is. Without that I felt like it was just smoke and mirrors really, so I just went with the simpler solution. This is something we could provide in our sandboxes, but we need probably a bit of help on how to design this "right" to ensure that we don't forget anyone for their workloads.

jade added the

Area/store

label

2024-12-06 22:07:43 +00:00

lix-bot commented

2024-12-18 12:33:43 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

comment in cl/2318 ("Allow CLI to pass environment variables to FOD builder")

This issue was mentioned on Gerrit on the following CLs: * comment in [cl/2318](https://gerrit.lix.systems/c/lix/+/2318) ("Allow CLI to pass environment variables to FOD builder")

stephank commented

2024-12-19 08:04:56 +00:00

Ah, sorry. I pushed the port to Gerrit after having skimmed this too quickly, and drew wrong conclusions.

(It wasn't wasted effort, fwiw. It was useful as a quick workaround for a work project. Perhaps the patch floating around is useful in the same way to others.)

While I agree / recognize that netrc is insufficient, we need more documentation / examples of where netrc fell short (scenarios, usecases, etc.) so that we can keep our priority on this topic updated and clear. Most of the workloads I work with private environments are, today, already possible with the current netrc capabilities, albeit with Nix code and sandbox impure paths.

I was dealing with PHP specifically. Composer uses ~/.config/composer/auth.json, or the same format JSON in a COMPOSER_AUTH variable. I was filling COMPOSER_AUTH in a php.mkComposerVendor based derivation.

But I think most language package managers lean towards a configuration file? The COMPOSER_AUTH env var feels like an outlier.

Either way, I'm not sure if you're solving packager manager issues with netrc, but if you are, I'm curious what that looks like.

That being said, I do believe that helping the scenario where you use impure sandbox paths which are shaped as a socket that you can ask for secrets (whether you put a Vault proxy behind or just a simple socat that sends back your secret) can be made more secure by helping the sandbox prove to the socket who they are.

Docker, Kubernetes, systemd currently all lean towards offering secrets as plain files. If access is limited to the sandbox, does a socket gain anything? Is the idea to offer even finer grained access than per sandbox / per build?

(But also, precedent for files doesn't have much weight if there's no specific integration in the tools we intend to run. As long as the result is easy to work with, I guess.)

Now, onto the feature, one problem with this sort of thing is what are the various policies we should apply, explain, advise to users with multi-tenant contexts (single tenant context is a special case of multi-tenancy here.)

User/group policies make sense, but this reads like there are more ideas?

Wild idea might be to lock specific credentials down to specific FOD hashes, which might work with sufficiently powerful prefetch and foo2nix tooling, but it's probably too fine grained. Every update would require registering a new hash in configuration. Plus it'd probably make onboarding steep for new tools. I have a feeling a lot of people use the 'copy hash from nix error message' flow, which may limit what we can usefully match in policies. 🙃

Ah, sorry. I pushed the port to Gerrit after having skimmed this too quickly, and drew wrong conclusions. (It wasn't wasted effort, fwiw. It was useful as a quick workaround for a work project. Perhaps the patch floating around is useful in the same way to others.) > While I agree / recognize that netrc is insufficient, we need more documentation / examples of where netrc fell short (scenarios, usecases, etc.) so that we can keep our priority on this topic updated and clear. Most of the workloads I work with private environments are, today, already possible with the current netrc capabilities, albeit with Nix code and sandbox impure paths. I was dealing with PHP specifically. Composer uses `~/.config/composer/auth.json`, or the same format JSON in a `COMPOSER_AUTH` variable. I was filling `COMPOSER_AUTH` in a `php.mkComposerVendor` based derivation. But I think most language package managers lean towards a configuration file? The `COMPOSER_AUTH` env var feels like an outlier. Either way, I'm not sure if you're solving packager manager issues with netrc, but if you are, I'm curious what that looks like. > That being said, I do believe that helping the scenario where you use impure sandbox paths which are shaped as a socket that you can ask for secrets (whether you put a Vault proxy behind or just a simple socat that sends back your secret) can be made more secure by helping the sandbox prove to the socket who they are. Docker, Kubernetes, systemd currently all lean towards offering secrets as plain files. If access is limited to the sandbox, does a socket gain anything? Is the idea to offer even finer grained access than per sandbox / per build? (But also, precedent for files doesn't have much weight if there's no specific integration in the tools we intend to run. As long as the result is easy to work with, I guess.) > Now, onto the feature, one problem with this sort of thing is what are the various policies we should apply, explain, advise to users with multi-tenant contexts (single tenant context is a special case of multi-tenancy here.) User/group policies make sense, but this reads like there are more ideas? Wild idea might be to lock specific credentials down to specific FOD hashes, which might work with sufficiently powerful prefetch and foo2nix tooling, but it's probably too fine grained. Every update would require registering a new hash in configuration. Plus it'd probably make onboarding steep for new tools. I have a feeling a lot of people use the 'copy hash from nix error message' flow, which may limit what we can usefully match in policies. 🙃

raito commented

2024-12-19 11:10:33 +00:00

Owner

Docker, Kubernetes, systemd currently all lean towards offering secrets as plain files. If access is limited to the sandbox, does a socket gain anything? Is the idea to offer even finer grained access than per sandbox / per build?

https://systemd.io/CREDENTIALS/ talks a lot about AF_UNIX socket :).

Files are fine as well. Sockets are the general case.

User/group policies make sense, but this reads like there are more ideas?

There are more ideas :), but I have no time to expand for now: I hope I can show various proof of concepts.

> Docker, Kubernetes, systemd currently all lean towards offering secrets as plain files. If access is limited to the sandbox, does a socket gain anything? Is the idea to offer even finer grained access than per sandbox / per build? https://systemd.io/CREDENTIALS/ talks a lot about `AF_UNIX` socket :). Files are fine as well. Sockets are the general case. > User/group policies make sense, but this reads like there are more ideas? There are more ideas :), but I have no time to expand for now: I hope I can show various proof of concepts.

truh commented

2025-08-20 11:12:38 +00:00

I'd like impure-env for setting NIXPKGS_ALLOW_UNFREE=1 and there is seemingly no other way to run build unfree applications without passing --impure every time. --impure also seems overly broad just to set a single nixpkgs flag.

Just commenting with since the issue so far only considers the auth use case.

I'd like impure-env for setting `NIXPKGS_ALLOW_UNFREE=1` and there is seemingly no other way to run build unfree applications without passing `--impure` every time. `--impure` also seems overly broad just to set a single nixpkgs flag. Just commenting with since the issue so far only considers the auth use case.

pennae commented

2025-08-20 11:20:54 +00:00

Owner

Just commenting with since the issue so far only considers the auth use case.

this issue is about env vars making their way into builders, but NIXPKGS_ALLOW_UNFREE only affects evaluation. what you're asking is only superficially related.

> Just commenting with since the issue so far only considers the auth use case. this issue is about env vars making their way into *builders*, but `NIXPKGS_ALLOW_UNFREE` only affects evaluation. what you're asking is only superficially related.

truh commented

2025-08-20 12:02:58 +00:00

Right, this doesn't actually work with NixOS/nix either.