92 lines
3.2 KiB
Markdown
92 lines
3.2 KiB
Markdown
|
# Benchmarking scripts for Lix
|
||
|
|
|
||
|
|
These are very much WIP, and have a few clumsy assumptions that we would
|
||
|
|
somewhat rather be fixed, but we have committed them to let others be able to
|
||
|
|
do benchmarking in the mean time.
|
||
|
|
|
||
|
|
## Benchmarking procedure
|
||
|
|
|
||
|
|
Build some Lixes you want to compare, by whichever means you wish.
|
||
|
|
|
||
|
|
Get a computer that is not busy and *strongly preferably* is bare-metal or at
|
||
|
|
least not a cloud VM (e.g. go make coffee when running benchmarks).
|
||
|
|
|
||
|
|
From the root of a Lix checkout, run `./bench/bench.sh resultlink-one
|
||
|
|
resultlink-two`, where `resultlink-one` and `resultlink-two` are the result
|
||
|
|
links from the builds you want to test (they can be any directory with bin/nix
|
||
|
|
in it, however).
|
||
|
|
|
||
|
|
To get the summary again, run `./bench/summarize.jq bench/bench-*.json`.
|
||
|
|
|
||
|
|
## Example results
|
||
|
|
|
||
|
|
(vim tip: `:r !bench/summarize.jq bench/bench-*.json` to dump it directly into
|
||
|
|
your editor)
|
||
|
|
|
||
|
|
```
|
||
|
|
result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6
|
||
|
|
c6fe19ccb59cac54b5b3f25e160870 hello
|
||
|
|
mean: 15.993s ± 0.081s
|
||
|
|
user: 13.321s | system: 1.865s
|
||
|
|
median: 15.994s
|
||
|
|
range: 15.829s ... 16.096s
|
||
|
|
relative: 1
|
||
|
|
result/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19cc
|
||
|
|
b59cac54b5b3f25e160870 hello
|
||
|
|
mean: 15.897s ± 0.075s
|
||
|
|
user: 13.248s | system: 1.843s
|
||
|
|
median: 15.88s
|
||
|
|
range: 15.807s ... 16.047s
|
||
|
|
relative: 0.994
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
result/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
|
||
|
|
mean: 0.4s ± 0.024s
|
||
|
|
user: 0.335s | system: 0.046s
|
||
|
|
median: 0.386s
|
||
|
|
range: 0.379s ... 0.43s
|
||
|
|
relative: 1
|
||
|
|
|
||
|
|
result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
|
||
|
|
mean: 0.404s ± 0.024s
|
||
|
|
user: 0.338s | system: 0.046s
|
||
|
|
median: 0.386s
|
||
|
|
range: 0.384s ... 0.436s
|
||
|
|
relative: 1.008
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
|
||
|
|
mean: 5.838s ± 0.023s
|
||
|
|
user: 5.083s | system: 0.464s
|
||
|
|
median: 5.845s
|
||
|
|
range: 5.799s ... 5.867s
|
||
|
|
relative: 1
|
||
|
|
|
||
|
|
result/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
|
||
|
|
mean: 5.788s ± 0.044s
|
||
|
|
user: 5.056s | system: 0.439s
|
||
|
|
median: 5.79s
|
||
|
|
range: 5.715s ... 5.876s
|
||
|
|
relative: 0.991
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
GC_INITIAL_HEAP_SIZE=10g result-asserts/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
|
||
|
|
mean: 4.147s ± 0.021s
|
||
|
|
user: 3.457s | system: 0.487s
|
||
|
|
median: 4.147s
|
||
|
|
range: 4.123s ... 4.195s
|
||
|
|
relative: 1
|
||
|
|
|
||
|
|
GC_INITIAL_HEAP_SIZE=10g result/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
|
||
|
|
mean: 4.149s ± 0.027s
|
||
|
|
user: 3.483s | system: 0.456s
|
||
|
|
median: 4.142s
|
||
|
|
range: 4.126s ... 4.215s
|
||
|
|
relative: 1
|
||
|
|
|
||
|
|
---
|
||
|
|
```
|