lix-project/lix

Fork 54

`remote-store=` store URI option does not act on `build-dir` #876

New issue

Closed

opened 2025-06-24 21:42:13 +00:00 by raito · 8 comments

raito commented

2025-06-24 21:42:13 +00:00

Owner

Describe the bug

This is more a surprise than an actual bug report. The Lix core team who mitigated the issue intended to make this a system setting and not a store setting.

In situations where you are using remote-store= to separate stores when remote building, you may find yourself writing to /nix/var/nix/builds and you may get new EPERM errors because you do not have such permissions to create directories and builds there.

Indeed, remote-store is not build-dir and there's no remote-build-dir option. This could be seen as an oversight or intended. This issue serves as discussion for this topic.

Steps To Workaround

Set NIX_STATE_DIR= in the environment of nix-daemon --stdio.

Multiple ways exist to perform this: PAM session variables, SSH ForceCommand, etc.

Expected behavior

A clear answer to "should build-dir be a store option?"

`nix --version` output

Affecting Lix 2.91.2, 2.92.2, Lix 2.93.1 and HEAD.

Additional context

Came up in a Hydra deployment.

## Describe the bug This is more a surprise than an actual bug report. The Lix core team who mitigated the issue intended to make this a system setting and not a store setting. In situations where you are using `remote-store=` to separate stores when remote building, you may find yourself writing to `/nix/var/nix/builds` and you may get new EPERM errors because you do not have such permissions to create directories and builds there. Indeed, `remote-store` is not `build-dir` and there's no `remote-build-dir` option. This could be seen as an oversight or intended. This issue serves as discussion for this topic. ## Steps To Workaround Set `NIX_STATE_DIR=` in the environment of `nix-daemon --stdio`. Multiple ways exist to perform this: PAM session variables, SSH `ForceCommand`, etc. ## Expected behavior A clear answer to "should `build-dir` be a store option?" ## `nix --version` output Affecting Lix 2.91.2, 2.92.2, Lix 2.93.1 and HEAD. ## Additional context Came up in a Hydra deployment.

raito added the

bug

label

2025-06-24 21:42:13 +00:00

lheckemann commented

2025-06-24 21:45:14 +00:00

Member

I'd argue that the responsibility here lies with the chroot store implementation, and remote-store itself is mostly irrelevant. Won't you have the same problem if you try to nix-build '<nixpkgs>' -A hello --store /mnt without the SSH stuff being involved? I think that should work as before, but it sounds like it won't currently.

I'd argue that the responsibility here lies with the chroot store implementation, and `remote-store` itself is mostly irrelevant. Won't you have the same problem if you try to `nix-build '<nixpkgs>' -A hello --store /mnt` without the SSH stuff being involved? I think that should work as before, but it sounds like it won't currently.

raito added

and removed

labels

2025-06-24 21:48:14 +00:00

raito commented

2025-06-24 21:59:27 +00:00

Author

Owner

@lheckemann wrote in #876 (comment):

I'd argue that the responsibility here lies with the chroot store implementation, and remote-store itself is mostly irrelevant. Won't you have the same problem if you try to nix-build '<nixpkgs>' -A hello --store /mnt without the SSH stuff being involved? I think that should work as before, but it sounds like it won't currently.

It should, no? Your command goes via the daemon which has permissions to write in /nix/var/nix/builds.

The problem arise when you do NIX_REMOTE=local nix-build '<nixpkgs>' -A hello --store /mnt.

@lheckemann wrote in https://git.lix.systems/lix-project/lix/issues/876#issuecomment-12682: > I'd argue that the responsibility here lies with the chroot store implementation, and `remote-store` itself is mostly irrelevant. Won't you have the same problem if you try to `nix-build '<nixpkgs>' -A hello --store /mnt` without the SSH stuff being involved? I think that should work as before, but it sounds like it won't currently. It should, no? Your command goes via the daemon which has permissions to write in `/nix/var/nix/builds`. The problem arise when you do `NIX_REMOTE=local nix-build '<nixpkgs>' -A hello --store /mnt`.

lheckemann commented

2025-06-24 22:03:20 +00:00

Member

No, --store /mnt is equivalent to --store local?root=/mnt (not 100% sure on the name of the root= parameter) -- it has nothing to do with the daemon anymore.

No, `--store /mnt` is equivalent to `--store local?root=/mnt` (not 100% sure on the name of the `root=` parameter) -- it has nothing to do with the daemon anymore.

👍 1

lheckemann commented

2025-06-24 22:03:55 +00:00

Member

(sorry for double-posting) --store is the same setting as NIX_REMOTE

(sorry for double-posting) `--store` is the same setting as `NIX_REMOTE`

👍 1

lheckemann commented

2025-06-25 10:48:57 +00:00

Member

Can confirm that the security fix breaks building in chroot stores with no SSH involvement:

$ nix build --store ~/scratch nixpkgs#hello --rebuild
error: creating directory '/nix/var/nix/builds/nix-build-hello-2.12.2.drv-0': Permission denied

chroot stores should presumably use their own prefix, or (if possible, not sure about this) perform the chroot early enough that /nix is already the store that's being built in when setting up the build dir.

In any case, I'm strongly of the opinion that this shouldn't require any extra steps when invoking the build.

Can confirm that the security fix breaks building in chroot stores with no SSH involvement: ``` $ nix build --store ~/scratch nixpkgs#hello --rebuild error: creating directory '/nix/var/nix/builds/nix-build-hello-2.12.2.drv-0': Permission denied ``` chroot stores should presumably use their own prefix, or (if possible, not sure about this) perform the chroot early enough that `/nix` is already the store that's being built in when setting up the build dir. In any case, I'm strongly of the opinion that this shouldn't require any extra steps when invoking the build.

raito commented

2025-06-25 10:59:29 +00:00

Author

Owner

OK, this is serious and we will probably need to release new minors.

In the meantime, we probably need to be very forthcoming about workarounds to survive.

OK, this is serious and we will probably need to release new minors. In the meantime, we probably need to be very forthcoming about workarounds to survive.

pennae commented

2025-06-25 14:03:23 +00:00

Owner

if we're going to have a built-in workaround that moves build dirs for chroot store it should probably happen somewhere around here

if we're going to have a built-in workaround that moves build dirs for chroot store it should probably happen [somewhere around here](https://git.lix.systems/lix-project/lix/src/commit/42e2bd045c9e51a59fdab038dc4e6f9e86c4922c/lix/libstore/store-api.cc#L1401)

lix-bot commented

2025-06-27 14:08:03 +00:00

Member

This issue was mentioned on Gerrit on the following CLs:

commit message in cl/3492 ("libstore: make buildDir a LocalStore config")
commit message in cl/3501 ("libstore: fallback on creating a safe space in the default tempdir")
commit message in cl/3502 ("libstore: fallback on creating a safe space in the default tempdir")
commit message in cl/3503 ("libstore: fallback on creating a safe space in the default tempdir")
commit message in cl/3504 ("libstore: fallback on creating a safe space in the default tempdir")

This issue was mentioned on Gerrit on the following CLs: * commit message in [cl/3492](https://gerrit.lix.systems/c/lix/+/3492) ("libstore: make `buildDir` a `LocalStore` config") * commit message in [cl/3501](https://gerrit.lix.systems/c/lix/+/3501) ("libstore: fallback on creating a safe space in the default tempdir") * commit message in [cl/3502](https://gerrit.lix.systems/c/lix/+/3502) ("libstore: fallback on creating a safe space in the default tempdir") * commit message in [cl/3503](https://gerrit.lix.systems/c/lix/+/3503) ("libstore: fallback on creating a safe space in the default tempdir") * commit message in [cl/3504](https://gerrit.lix.systems/c/lix/+/3504) ("libstore: fallback on creating a safe space in the default tempdir")