lix-project/lix

Fork 56

Setting tempdir in nix3-develop, nix3-print-dev-env can lead to issues #979

New issue

Closed

opened 2025-08-25 13:09:24 +00:00 by teofilc · 5 comments

teofilc commented

2025-08-25 13:09:24 +00:00

Summary

In the nix3-develop codepath, we set NIX_BUILD_TOP, TMP, and a bunch of synonyms thereof to a new mktemp -d created directory.

Diferences from `/tmp`

This can lead to subtle bugs because now our TMP directory differs from the normal one in a few ways.

The file permissions are different. /tmp has 1777 on my system. But folders created by mktemp -d have different permissions, eg, 0700
The folder is owned by a different user. I don't think this is necessarily a problem, since it's normally a less privileged user.
The folder may be cleaned up by systemd-tmpfiles. There is no limit to how long a shell might be used for, so after a 3 week holiday, systemd-tmpfiles might clean up your TMP directory and now your shell is suddenly broken.

Bugs

For instance https://github.com/NixOS/nix/issues/10753 was triggered by the following scenario.

We create a tmpdir as root and set TMP to that. This doesn't have permissions for arbitrary users to create folders in it
We run a build as the nixbld user with the sandbox off (since we are using docker)
We try to create a tmpdir but fail, since we don't have permission to create folders under /tmp

If systemd-tmpfiles deletes your TMP folder, then your shell is broken and you need to re source it. This can be painful for users who use tools like haskell.nix which have extremely bad eval times (so users sometimes keep a shell open for months (sic!)). I think there is an issue for this somewhere but I can't find it.

## Summary In the nix3-develop codepath, we set `NIX_BUILD_TOP`, `TMP`, and a bunch of synonyms thereof to a new `mktemp -d` created directory. ## Diferences from `/tmp` This can lead to subtle bugs because now our `TMP` directory differs from the normal one in a few ways. 1. The file permissions are different. `/tmp` has 1777 on my system. But folders created by `mktemp -d` have different permissions, eg, 0700 2. The folder is owned by a different user. I don't think this is necessarily a problem, since it's normally a less privileged user. 3. The folder may be cleaned up by systemd-tmpfiles. There is no limit to how long a shell might be used for, so after a 3 week holiday, systemd-tmpfiles might clean up your `TMP` directory and now your shell is suddenly broken. ## Bugs For instance https://github.com/NixOS/nix/issues/10753 was triggered by the following scenario. 1. We create a tmpdir as `root` and set `TMP` to that. This doesn't have permissions for arbitrary users to create folders in it 2. We run a build as the `nixbld` user with the sandbox off (since we are using docker) 3. We try to create a tmpdir but fail, since we don't have permission to create folders under /tmp If systemd-tmpfiles deletes your `TMP` folder, then your shell is broken and you need to re source it. This can be painful for users who use tools like `haskell.nix` which have extremely bad eval times (so users sometimes keep a shell open for months (sic!)). I think there is an issue for this somewhere but I can't find it.

teofilc added the

bug

label

2025-08-25 13:09:24 +00:00

teofilc referenced this issue

2025-08-25 13:10:18 +00:00

Inconsistent behavior in tempdir manipulations between nix-shell and nix3-develop #978

raito added the

release-blocker

label

2025-08-25 13:22:38 +00:00

raito commented

2025-10-13 10:42:46 +00:00

Owner

OK, I have more bandwidth to think about this.

I'm not so sure yet what is the right way to go about this, basically, we have no guarantees whatsoever about what happens in /tmp neither.

I hear that some users may want to have persistent shells, in that case, those should never be staged in any tmp-ish directory, more like, a conscious choice to choose a persistent shell or not in /nix/var/nix/shells or something, and you can have manual GC or automatic GC (not a fan of adding a new GC system for shells).

What do you see as solution for your needs?

OK, I have more bandwidth to think about this. I'm not so sure yet what is the right way to go about this, basically, we have no guarantees whatsoever about what happens in `/tmp` neither. I hear that some users may want to have persistent shells, in that case, those should never be staged in any tmp-ish directory, more like, a conscious choice to choose a persistent shell or not in `/nix/var/nix/shells` or something, and you can have manual GC or automatic GC (not a fan of adding a new GC system for shells). What do you see as solution for your needs?

teofilc commented

2025-10-13 13:18:42 +00:00

Author

I think having the option to use manually GC'd temp dir makes sense, and is probably the best thing we can do.

I think this sort of thing is a result of shells expressing tow fundamentally different things: a way to debug a derivation build; and a way to set up development environments. The former needs to recreate an accurate temp dir, while the later doesn't.

I think having the option to use manually GC'd temp dir makes sense, and is probably the best thing we can do. I think this sort of thing is a result of shells expressing tow fundamentally different things: a way to debug a derivation build; and a way to set up development environments. The former needs to recreate an accurate temp dir, while the later doesn't.

👍 1

raito commented

2025-11-07 01:45:44 +00:00

Owner

@teofilc We looked again at it.

I think the best we can offer is documentation in the future on this because AFAIK, this already works:

$ TMPDIR=~/.cache/shells nix develop nixpkgs#systemd
[raito@Thors ~]$ echo "$NIX_BUILD_TOP"
/home/raito/.cache/shells/nix-shell.EOQBtw

already works on HEAD.

I will let you confirm me if there's anything wrong with that, but from my PoV, nix3-develop and nix-shell are consistent on this behavior, solving the inconsistencies.

@teofilc We looked again at it. I think the best we can offer is documentation in the future on this because AFAIK, this already works: ``` $ TMPDIR=~/.cache/shells nix develop nixpkgs#systemd [raito@Thors ~]$ echo "$NIX_BUILD_TOP" /home/raito/.cache/shells/nix-shell.EOQBtw ``` already works on HEAD. I will let you confirm me if there's anything wrong with that, but from my PoV, nix3-develop and nix-shell are consistent on this behavior, solving the inconsistencies.

teofilc commented

2025-11-12 23:55:50 +00:00

Author

@raito that sounds like a good workaround to me!