[Tracking issue] Cgroups in Lix #537

New issue

Open

opened 2024-09-29 16:00:35 +00:00 by raito · 0 comments

raito commented

2024-09-29 16:00:35 +00:00

Owner

With https://gerrit.lix.systems/c/lix/+/1936/8, we get a pretty OK story for cgroups on Linux.

I would like to discuss stabilization of cgroups (to enable by default, yes, yes) and what are the missing pieces.

2026 situation

Store-bypassing operations are a mess

We can post error messages but users are not always equipped to remediate. What do we do?

This can happen even in breaking situations where a deployment tool does not allow to pass --no-use-cgroups to get functionality back.

2024 situation

Daemon and single user mode

The current situation is that you can run Lix bypassing the daemon, this means that it makes sense to continue using cgroups for the local user doing that.

In the future, we may just have a daemon colocated with the single user and this situation will disappear and reduce a bunch of non-DRY code.

Cgroups for non-builds

Not everything is a build, some things are substitutions that may lead to a build, for example.

We may still want to have nix-daemon/nix-substitution/nix-build-uid-$uid as a path here.

Here's a list of things that can happen:

substitutions
path repairs
ensure that a path exist: is it a build? is it a substitution? is it just a check? propose cgroup hierarchies!
garbage collection (+ repairs)
recursive nix ops (surprise, this is not well supported by CppNix code!)

Delegation

We should probably use nsdelegate mount option and remount the cgroupv2 hierarchy in the sandbox setup, this avoids having to keep track of which files we need to chown to the builder (a TODO is in the code mentioned above).

systemd remote control

We can let systemd do most of our job using StartTransientUnit, via a D-Bus dependency.

More tests

We should probably test:

Interaction with recursive Nix
Apply more controls and verify they work out of the box

We want to ensure we can let operators do equal partition of build resources, etc.

cc @pennae @jade

With https://gerrit.lix.systems/c/lix/+/1936/8, we get a pretty OK story for cgroups on Linux. I would like to discuss stabilization of cgroups (to enable by default, yes, yes) and what are the missing pieces. # 2026 situation ## Store-bypassing operations are a mess We can post error messages but users are not always equipped to remediate. What do we do? This can happen even in breaking situations where a deployment tool does not allow to pass `--no-use-cgroups` to get functionality back. # 2024 situation ### Daemon and single user mode _The current situation_ is that you can run Lix bypassing the daemon, this means that it makes sense to continue using cgroups for the local user doing that. In the future, we may just have a daemon colocated with the single user and this situation will disappear and reduce a bunch of non-DRY code. ### Cgroups for non-builds Not everything is a build, some things are substitutions that may lead to a build, for example. We may still want to have `nix-daemon/nix-substitution/nix-build-uid-$uid` as a path here. Here's a list of things that can happen: - substitutions - path repairs - ensure that a path exist: is it a build? is it a substitution? is it just a check? propose cgroup hierarchies! - garbage collection (+ repairs) - recursive nix ops (surprise, this is not well supported by CppNix code!) ### Delegation We should probably use `nsdelegate` mount option and remount the cgroupv2 hierarchy in the sandbox setup, this avoids having to keep track of which files we need to chown to the builder (a TODO is in the code mentioned above). ### systemd remote control We can let systemd do most of our job using `StartTransientUnit`, via a D-Bus dependency. ### More tests We should probably test: - Interaction with recursive Nix - Apply more controls and verify they work out of the box We want to ensure we can let operators do equal partition of build resources, etc. cc @pennae @jade