[Nix#9675] nix creates processes in cgroup inner nodes #77

New issue

Open

opened 2024-03-16 06:44:47 +00:00 by lix-bot · 0 comments

lix-bot commented

2024-03-16 06:44:47 +00:00

Member

Upstream-Issue: NixOS/nix#9675

Describe the bug

I run a shared build cluster at my company. I'm trying to limit my system's resource utilisation such that one rogue derivation cannot take out a build node. I'm trying to use the Nix cgroup support to help achieve this.

What I had envisaged was watching for new Nix build cgroups being created, and then setting memory limits on them such that memory leaks lead to killed processes.

Unfortunately, this is hard to implement because Nix creates the cgroups as children of the cgroup of the nix-daemon's cgroup.
This is not a recommended configuration with cgroupsv2 as here, inner nodes are not allowed to contain processes.

While the nix-daemon process lives in the parent cgroup, Linux does not allow any controls to be set on any of the child cgroups (limiting their utility considerably!).

The systemd docs say the below:

https://systemd.io/CGROUP_DELEGATION/

🙁 The i-like-islands option. If all you care about is your own cgroup tree, and you want to have to do as little as possible with systemd and no interest in integration with the rest of the system, then this is a valid option. For this all you have to do is turn on Delegate= for your main manager daemon. Then figure out the cgroup systemd placed your daemon in: you can now freely create sub-cgroups beneath it. Don’t forget the no-processes-in-inner-nodes rule however: you have to move your main daemon process out of that cgroup (and into a sub-cgroup) before you can start further processes in any of your sub-cgroups.

Nix is basically doing this, but forgot 'you have to move your main daemon process out of that cgroup' bit.

Expected behavior

Nix creates its build cgroups parented by a cgroup which does not contain processes.

nix-env --version output

All Nix releases that support cgroups at time of writing.

Additional context

My workaround for now was to change the nix-daemon's unit to launch in the root cgroup, as this has no such limitation (execstart cgexec -g ":/" nix-daemon). Then, a combination of inotifywait (on /nix/var/nix/cgroups) to figure out what cgroups matter, xargs, and a bash function which resolves that link to find the cgroup dir and sets the properties is all we need :).

Priorities

Add 👍 to issues you find important.

Upstream-Issue: https://git.lix.systems/NixOS/nix/issues/9675 **Describe the bug** I run a shared build cluster at my company. I'm trying to limit my system's resource utilisation such that one rogue derivation cannot take out a build node. I'm trying to use the Nix cgroup support to help achieve this. What I had envisaged was watching for new Nix build cgroups being created, and then setting memory limits on them such that memory leaks lead to killed processes. Unfortunately, this is hard to implement because Nix creates the cgroups as children of the cgroup of the nix-daemon's cgroup. This is not a recommended configuration with cgroupsv2 as here, inner nodes are not allowed to contain processes. While the nix-daemon process lives in the parent cgroup, Linux does not allow any controls to be set on any of the child cgroups (limiting their utility considerably!). The systemd docs say the below: https://systemd.io/CGROUP_DELEGATION/ > 🙁 The i-like-islands option. If all you care about is your own cgroup tree, and you want to have to do as little as possible with systemd and no interest in integration with the rest of the system, then this is a valid option. For this all you have to do is turn on Delegate= for your main manager daemon. Then figure out the cgroup systemd placed your daemon in: you can now freely create sub-cgroups beneath it. Don’t forget the no-processes-in-inner-nodes rule however: you have to move your main daemon process out of that cgroup (and into a sub-cgroup) before you can start further processes in any of your sub-cgroups. Nix is basically doing this, but forgot 'you have to move your main daemon process out of that cgroup' bit. **Expected behavior** Nix creates its build cgroups parented by a cgroup which does not contain processes. **`nix-env --version` output** All Nix releases that support cgroups at time of writing. **Additional context** My workaround for now was to change the nix-daemon's unit to launch in the root cgroup, as this has no such limitation (execstart cgexec -g ":/" nix-daemon). Then, a combination of inotifywait (on /nix/var/nix/cgroups) to figure out what cgroups matter, xargs, and a bash function which resolves that link to find the cgroup dir and sets the properties is all we need :). **Priorities** Add :+1: to [issues you find important](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc).