Parallel evaluation of flake outputs #382

New issue

Open

opened 2024-06-09 12:22:10 +00:00 by jalil-salame · 2 comments

jalil-salame commented

2024-06-09 12:22:10 +00:00

I was looking at nix-fast-build since I felt that nix flake check was taking too long (~1min in my flake-based nixos configuration) and got stuck not being able to run the checks that nix flake check does by default (e.g. checking overlays, nixosConfigurations, etc.) as these are not available to nix-eval-jobs, the underlying tool nix-fast-build uses to parallelize nix builds.

Describe the solution you'd like

I would like parallel execution of the check in nix flake check to be done inside nix (lix) itself; that way the default checks could also be done in parallel. This could also be used to build a CLI that achieves the same purpose as nix-eval-jobs, but is able to share more internal state (and thus be even faster).

Describe alternatives you've considered

Expose the default flake checks such that they can be called through nix build (and thus be available for nix-eval-jobs).
Do nothing (the current speed is still less than a Rust clean build XD)

Additional context

When I say the default checks are not available I mean that if you define a outputs = {...}: {checks.x86_64-linux.lint = ...;} you can then call nix build .#checks.x86_64-linux.lint and evaluate the check. You cannot do this for the checks that work by default on packages, devShells, overlays, etc.

## Is your feature request related to a problem? Please describe. I was looking at [`nix-fast-build`][1] since I felt that `nix flake check` was taking too long (~1min in my flake-based nixos configuration) and got stuck not being able to run the checks that `nix flake check` does by default (e.g. checking overlays, nixosConfigurations, etc.) as these are not available to [`nix-eval-jobs`][2], the underlying tool [`nix-fast-build`][1] uses to parallelize nix builds. [1]: <https://github.com/Mic92/nix-fast-build> "nix-fast-build GitHub repo" [2]: <https://github.com/nix-community/nix-eval-jobs> "nix-eval-jobs GitHub repo" ## Describe the solution you'd like I would like parallel execution of the check in `nix flake check` to be done inside `nix` (`lix`) itself; that way the default checks could also be done in parallel. This could also be used to build a CLI that achieves the same purpose as [`nix-eval-jobs`][2], but is able to share more internal state (and thus be even faster). ## Describe alternatives you've considered - Expose the default flake checks such that they can be called through `nix build` (and thus be available for [`nix-eval-jobs`][2]). - Do nothing (the current speed is still less than a Rust clean build XD) ## Additional context When I say the default checks are not available I mean that if you define a `outputs = {...}: {checks.x86_64-linux.lint = ...;}` you can then call `nix build .#checks.x86_64-linux.lint` and evaluate the check. You cannot do this for the checks that work by default on `packages`, `devShells`, `overlays`, etc.

jade commented

2024-06-09 23:54:39 +00:00

Owner

we would really like to make nix-eval-jobs kind of thing part of lix. see also: #25

we would really like to make nix-eval-jobs kind of thing part of lix. see also: https://git.lix.systems/lix-project/lix/issues/25

jade added the

Area/evaluator

Area/flakes

labels 2024-06-09 23:55:09 +00:00

jalil-salame commented

2024-06-10 07:27:56 +00:00

Author

This is slightly different from nix-eval-jobs:

$ nix-eval-jobs --extra-experimental-features flakes --force-recurse --flake .#checks | jq -r 'select(.system == "x86_64-linux") | .drvPath'
...
/nix/store/s411j1c214b30hsc3dgmfv310r6p4qnq-fmt-src.drv
/nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv
/nix/store/yziw1bipzvvqz9cs8bmbn9dp0jq8rnj8-nixvim-check.drv
/nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv

Using the example command from #25 (comment) on the public part of my NixOS configuration you only get four derivations (the ones explicitly defined in the .#checks), not the actual ~12 checks that run with nix flake check.

actual run of nix flake check in the repo (~20 checks)

$ nix flake check
fetching git input 'git+file:///home/jalil/Dev/configuration.nix'
evaluating flake...
checking flake output 'overlays'...
checking overlay 'overlays.unstable'...
checking overlay 'overlays.jpassmenu'...
checking overlay 'overlays.audiomenu'...
checking overlay 'overlays.nixvim'...
checking flake output 'checks'...
checking derivation checks.x86_64-linux.nvim...
derivation evaluated to /nix/store/yziw1bipzvvqz9cs8bmbn9dp0jq8rnj8-nixvim-check.drv
checking derivation checks.x86_64-linux.fmt...
derivation evaluated to /nix/store/s411j1c214b30hsc3dgmfv310r6p4qnq-fmt-src.drv
checking derivation checks.x86_64-linux.lint...
derivation evaluated to /nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv
checking derivation checks.x86_64-linux.typos...
derivation evaluated to /nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv
checking flake output 'packages'...
checking derivation packages.x86_64-linux.nvim...
derivation evaluated to /nix/store/7cw874cxrk008237aswv9qgwj31h591m-nixvim.drv
checking derivation packages.x86_64-linux.docs...
derivation evaluated to /nix/store/4l8gi62lsxq8l4ig427xlcczsm9sq0h4-nixos-configuration-book.drv
checking derivation packages.x86_64-linux.nixos-markdown...
derivation evaluated to /nix/store/fm863rs80b9jzfirmcbhgi2gq434bpgn-options.md.drv
checking derivation packages.x86_64-linux.nvim-markdown...
derivation evaluated to /nix/store/khafbjfbp0ywagcc7qw4zhvfv8fdf3d2-options.md.drv
checking derivation packages.x86_64-linux.home-markdown...
derivation evaluated to /nix/store/jal464n78gy2gbnihahq7m50issf56hr-options.md.drv
checking flake output 'formatter'...
checking flake output 'nixosConfigurations'...
checking NixOS configuration 'nixosConfigurations.vm'...
copying path '/nix/store/s13rwnsg2qrwcxk6fadza2ys1irsrg8f-base16-schemes-unstable-2024-01-14' from 'https://cache.nixos.org'...
checking flake output 'nixosModules'...
checking NixOS module 'nixosModules.default'...
checking NixOS module 'nixosModules.vm'...
checking NixOS module 'nixosModules.homeManagerModuleSandalone'...
checking NixOS module 'nixosModules.homeManagerModuleNixOS'...
checking NixOS module 'nixosModules.nixosModule'...
checking NixOS module 'nixosModules.gemini'...
checking NixOS module 'nixosModules.capricorn'...
checking NixOS module 'nixosModules.libra'...
checking flake output 'devShells'...
checking derivation devShells.x86_64-linux.default...
derivation evaluated to /nix/store/j61zsfiqngpsiphj8jvd2xmass0s8p6i-nix-shell.drv
running 4 flake checks...
building '/nix/store/s411j1c214b30hsc3dgmfv310r6p4qnq-fmt-src.drv'...
building '/nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv'...
copying path '/nix/store/ag2axck3lz6r5gs61a2bc678r5ky1icj-docker-buildx-0.14.1' from 'https://cache.nixos.org'...
copying path '/nix/store/jw7hxwpr8szr4d74l0x502xpsjs35vyc-docker-compose-2.27.1' from 'https://cache.nixos.org'...
copying path '/nix/store/494gsm86cdaiyan1sa26qdapzwg1pw2j-docker-plugins' from 'https://cache.nixos.org'...
copying path '/nix/store/wgk94qcrj11dr35m6pw0pgwiqr3dpzc8-docker-24.0.9' from 'https://cache.nixos.org'...
building '/nix/store/yziw1bipzvvqz9cs8bmbn9dp0jq8rnj8-nixvim-check.drv'...
warning: The check omitted these incompatible systems: aarch64-linux
Use '--all-systems' to check all.

Another problem with nix-eval-jobs, is that you need to run four instances of nix build, which all re-evaluate the inputs. I don't think this is a big issue, but it certainly could be made more efficient by first evaluating the inputs and then splitting off the computation.

PS: I assume you all know this better than I do, but I prefer having it written down so we can all be on the same page c:

This is slightly different from `nix-eval-jobs`: ```console $ nix-eval-jobs --extra-experimental-features flakes --force-recurse --flake .#checks | jq -r 'select(.system == "x86_64-linux") | .drvPath' ... /nix/store/s411j1c214b30hsc3dgmfv310r6p4qnq-fmt-src.drv /nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv /nix/store/yziw1bipzvvqz9cs8bmbn9dp0jq8rnj8-nixvim-check.drv /nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv ``` Using the example command from https://git.lix.systems/lix-project/lix/issues/25#issuecomment-783 on the public part of [my NixOS configuration][1] you only get four derivations (the ones explicitly defined in the `.#checks`), not the actual ~12 checks that run with `nix flake check`. <details><summary>actual run of <code>nix flake check</code> in the repo (~20 checks)</summary> ```console $ nix flake check fetching git input 'git+file:///home/jalil/Dev/configuration.nix' evaluating flake... checking flake output 'overlays'... checking overlay 'overlays.unstable'... checking overlay 'overlays.jpassmenu'... checking overlay 'overlays.audiomenu'... checking overlay 'overlays.nixvim'... checking flake output 'checks'... checking derivation checks.x86_64-linux.nvim... derivation evaluated to /nix/store/yziw1bipzvvqz9cs8bmbn9dp0jq8rnj8-nixvim-check.drv checking derivation checks.x86_64-linux.fmt... derivation evaluated to /nix/store/s411j1c214b30hsc3dgmfv310r6p4qnq-fmt-src.drv checking derivation checks.x86_64-linux.lint... derivation evaluated to /nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv checking derivation checks.x86_64-linux.typos... derivation evaluated to /nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv checking flake output 'packages'... checking derivation packages.x86_64-linux.nvim... derivation evaluated to /nix/store/7cw874cxrk008237aswv9qgwj31h591m-nixvim.drv checking derivation packages.x86_64-linux.docs... derivation evaluated to /nix/store/4l8gi62lsxq8l4ig427xlcczsm9sq0h4-nixos-configuration-book.drv checking derivation packages.x86_64-linux.nixos-markdown... derivation evaluated to /nix/store/fm863rs80b9jzfirmcbhgi2gq434bpgn-options.md.drv checking derivation packages.x86_64-linux.nvim-markdown... derivation evaluated to /nix/store/khafbjfbp0ywagcc7qw4zhvfv8fdf3d2-options.md.drv checking derivation packages.x86_64-linux.home-markdown... derivation evaluated to /nix/store/jal464n78gy2gbnihahq7m50issf56hr-options.md.drv checking flake output 'formatter'... checking flake output 'nixosConfigurations'... checking NixOS configuration 'nixosConfigurations.vm'... copying path '/nix/store/s13rwnsg2qrwcxk6fadza2ys1irsrg8f-base16-schemes-unstable-2024-01-14' from 'https://cache.nixos.org'... checking flake output 'nixosModules'... checking NixOS module 'nixosModules.default'... checking NixOS module 'nixosModules.vm'... checking NixOS module 'nixosModules.homeManagerModuleSandalone'... checking NixOS module 'nixosModules.homeManagerModuleNixOS'... checking NixOS module 'nixosModules.nixosModule'... checking NixOS module 'nixosModules.gemini'... checking NixOS module 'nixosModules.capricorn'... checking NixOS module 'nixosModules.libra'... checking flake output 'devShells'... checking derivation devShells.x86_64-linux.default... derivation evaluated to /nix/store/j61zsfiqngpsiphj8jvd2xmass0s8p6i-nix-shell.drv running 4 flake checks... building '/nix/store/s411j1c214b30hsc3dgmfv310r6p4qnq-fmt-src.drv'... building '/nix/store/f8v8rshsszvmvm26z04glscl117p8pbl-lint-src.drv'... copying path '/nix/store/ag2axck3lz6r5gs61a2bc678r5ky1icj-docker-buildx-0.14.1' from 'https://cache.nixos.org'... copying path '/nix/store/jw7hxwpr8szr4d74l0x502xpsjs35vyc-docker-compose-2.27.1' from 'https://cache.nixos.org'... copying path '/nix/store/494gsm86cdaiyan1sa26qdapzwg1pw2j-docker-plugins' from 'https://cache.nixos.org'... copying path '/nix/store/wgk94qcrj11dr35m6pw0pgwiqr3dpzc8-docker-24.0.9' from 'https://cache.nixos.org'... building '/nix/store/yziw1bipzvvqz9cs8bmbn9dp0jq8rnj8-nixvim-check.drv'... warning: The check omitted these incompatible systems: aarch64-linux Use '--all-systems' to check all. ``` </details> Another problem with `nix-eval-jobs`, is that you need to run four instances of `nix build`, which all re-evaluate the `inputs`. I don't think this is a big issue, but it certainly could be made more efficient by first evaluating the inputs and then splitting off the computation. PS: I assume you all know this better than I do, but I prefer having it written down so we can all be on the same page c: [1]: <https://github.com/jalil-salame/configuration.nix> "Jalil's public NixOS configuration"

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#382

No description provided.