lix/bench

Benchmarking scripts for Lix

These are very much WIP, and have a few clumsy assumptions that we would somewhat rather be fixed, but we have committed them to let others be able to do benchmarking in the mean time.

Benchmarking procedure

Build some Lixes you want to compare, by whichever means you wish.

Get a computer that is not busy and strongly preferably is bare-metal or at least not a cloud VM (e.g. go make coffee when running benchmarks).

From the root of a Lix checkout, run ./bench/bench.sh resultlink-one resultlink-two, where resultlink-one and resultlink-two are the result links from the builds you want to test (they can be any directory with bin/nix in it, however).

To get the summary again, run ./bench/summarize.jq bench/bench-*.json.

Example results

(vim tip: :r !bench/summarize.jq bench/bench-*.json to dump it directly into your editor)

result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6
c6fe19ccb59cac54b5b3f25e160870 hello
  mean:     15.993s ± 0.081s
            user: 13.321s | system: 1.865s
  median:   15.994s
  range:    15.829s ... 16.096s
  relative: 1
result/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19cc
b59cac54b5b3f25e160870 hello
  mean:     15.897s ± 0.075s
            user: 13.248s | system: 1.843s
  median:   15.88s
  range:    15.807s ... 16.047s
  relative: 0.994

---

result/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.4s ± 0.024s
            user: 0.335s | system: 0.046s
  median:   0.386s
  range:    0.379s ... 0.43s
  relative: 1

result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  mean:     0.404s ± 0.024s
            user: 0.338s | system: 0.046s
  median:   0.386s
  range:    0.384s ... 0.436s
  relative: 1.008

---

result-asserts/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     5.838s ± 0.023s
            user: 5.083s | system: 0.464s
  median:   5.845s
  range:    5.799s ... 5.867s
  relative: 1

result/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     5.788s ± 0.044s
            user: 5.056s | system: 0.439s
  median:   5.79s
  range:    5.715s ... 5.876s
  relative: 0.991

---

GC_INITIAL_HEAP_SIZE=10g result-asserts/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.147s ± 0.021s
            user: 3.457s | system: 0.487s
  median:   4.147s
  range:    4.123s ... 4.195s
  relative: 1

GC_INITIAL_HEAP_SIZE=10g result/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
  mean:     4.149s ± 0.027s
            user: 3.483s | system: 0.456s
  median:   4.142s
  range:    4.126s ... 4.215s
  relative: 1

---