Make build-related global settings into store settings #886

New issue

Open

opened 2025-06-27 19:43:24 +00:00 by lheckemann · 1 comment

lheckemann commented

2025-06-27 19:43:24 +00:00

Member

The initial trigger for this was configuring the forkos hydra: we want to use the same build machines in different configurations for big-parallel and other jobs. For example, we have the following lines in the hydra machines file:

ssh://hydra-aarch64-big-parallel-bm-12?remote-store=/mnt&cores=60 aarch64-linux /run/agenix/hydra-ssh-key-priv 2 1 kvm,nixos-test,big-parallel big-parallel
ssh://hydra-aarch64-bm-12?remote-store=/mnt&cores=20 aarch64-linux /run/agenix/hydra-ssh-key-priv 8 1 kvm,nixos-test -

With a dumb hack for hydra, this allows building big-parallel jobs with cores set to 60 while limiting other jobs to 20.

Describe the solution you'd like

The dumb implementation in this special case aside, I think cores should in fact be a store setting to allow doing this in a more principled fashion -- both for hydra (once we finally make it use the store abstraction rather than its own special sauce) and for standalone Lix, via settings that allow specifying stores (--eval-store, --store and --builders are the ones that come to mind for me).

A number of other settings that control builds should, IMHO, also be moved from the global scope to become store-specific. Some examples (not exhaustive):

build-dir -- could be used to ensure builds run on a specific filesystem if required
sandbox (and all settings beginning with sandbox-)
extra-platforms
pasta-path
pre-build-hook
post-build-hook
secret-key-files

Describe alternatives you've considered

Keep the status quo; the global settings affect all local stores used, and remote stores cannot be configured.

Anything that needs to do this could also split builds across multiple processes to avoid sharing the global settings. This would mean much more work on the part of consumers, especially ones that don't already take over scheduling (like hydra).

## Is your feature request related to a problem? Please describe. The initial trigger for this was configuring the forkos hydra: we want to use the same build machines in different configurations for big-parallel and other jobs. For example, we have the following lines in the hydra machines file: ``` ssh://hydra-aarch64-big-parallel-bm-12?remote-store=/mnt&cores=60 aarch64-linux /run/agenix/hydra-ssh-key-priv 2 1 kvm,nixos-test,big-parallel big-parallel ssh://hydra-aarch64-bm-12?remote-store=/mnt&cores=20 aarch64-linux /run/agenix/hydra-ssh-key-priv 8 1 kvm,nixos-test - ``` With a [dumb hack](https://git.lix.systems/lix-project/hydra/pulls/27) for hydra, this allows building big-parallel jobs with `cores` set to 60 while limiting other jobs to 20. ## Describe the solution you'd like The dumb implementation in this special case aside, I think `cores` should in fact be a store setting to allow doing this in a more principled fashion -- both for hydra (once we finally make it use the store abstraction rather than its own special sauce) and for standalone Lix, via settings that allow specifying stores (`--eval-store`, `--store` and `--builders` are the ones that come to mind for me). A number of other settings that control builds should, IMHO, also be moved from the global scope to become store-specific. Some examples (not exhaustive): - `build-dir` -- could be used to ensure builds run on a specific filesystem if required - `sandbox` (and all settings beginning with `sandbox-`) - `extra-platforms` - `pasta-path` - `pre-build-hook` - `post-build-hook` - `secret-key-files` ## Describe alternatives you've considered Keep the status quo; the global settings affect all local stores used, and remote stores cannot be configured. Anything that needs to do this could also split builds across multiple processes to avoid sharing the global settings. This would mean much more work on the part of consumers, especially ones that don't already take over scheduling (like hydra).

lheckemann commented

2025-07-01 10:30:44 +00:00

Author

Member

See also https://gerrit.lix.systems/c/lix/+/3492 which did this for build-dir but was abandoned in favour of a hotfix with less far-reaching implications