Run benchmarks in CI #23

New issue

Open

opened 2024-03-12 17:40:38 +00:00 by jade · 4 comments

jade commented

2024-03-12 17:40:38 +00:00

Owner

Nix does not run benchmarks in CI. We can run benchmarks in CI.

jade added the

stability

performance

labels 2024-03-12 17:40:38 +00:00

pennae commented

2024-03-12 17:51:14 +00:00

Owner

dumping a nix benchmark script here that we used for a number of tests and also gave to cppnix. the real horrorific script also has runs in callgrind and cachegrind aside from just hyperfine, but considering that cppnix doesn't even run something that produces simple numbers we did not burden them with it then

#!/usr/bin/env bash

set -euo pipefail
shopt -s inherit_errexit

_exit=""
trap "$_exit" EXIT

export NIX_REMOTE="$(mktemp -d)"
_exit='rm -rfv "$NIX_REMOTE"; $_exit'
export NIX_PATH="nixpkgs=../nixpkgs:nixos-config=bench/configuration.nix"

_nix="src/nix/nix"
# _nix="result/bin/nix"; echo -e "\e[1m\e[91mUSING FLAKE-BUILT NIX\e[0m" >&2
nix="$_nix --extra-experimental-features 'nix-command flakes'"

hyperfineArgs=(
    --warmup 2 --runs 10
)

declare -A cases
cases=(
    [search]="$nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello"
    [rebuild]="$nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
    [rebuild-lh]="GC_INITIAL_HEAP_SIZE=10g $nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
    [parse]="$nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix"
)

benches=(
    search
    rebuild
    rebuild-lh
    parse
)

run=()
for k in "${benches[@]}"; do
    run+=("${cases[$k]}")
done

taskset -c 2,3 \
    chrt -f 50 \
    hyperfine "${hyperfineArgs[@]}" "${run[@]}"

dumping a nix benchmark script here that we used for a number of tests and also gave to cppnix. the *real* horrorific script also has runs in callgrind and cachegrind aside from just hyperfine, but considering that cppnix doesn't even run something that produces simple numbers we did not burden them with it then ``` #!/usr/bin/env bash set -euo pipefail shopt -s inherit_errexit _exit="" trap "$_exit" EXIT export NIX_REMOTE="$(mktemp -d)" _exit='rm -rfv "$NIX_REMOTE"; $_exit' export NIX_PATH="nixpkgs=../nixpkgs:nixos-config=bench/configuration.nix" _nix="src/nix/nix" # _nix="result/bin/nix"; echo -e "\e[1m\e[91mUSING FLAKE-BUILT NIX\e[0m" >&2 nix="$_nix --extra-experimental-features 'nix-command flakes'" hyperfineArgs=( --warmup 2 --runs 10 ) declare -A cases cases=( [search]="$nix search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello" [rebuild]="$nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'" [rebuild-lh]="GC_INITIAL_HEAP_SIZE=10g $nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'" [parse]="$nix eval -f ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix" ) benches=( search rebuild rebuild-lh parse ) run=() for k in "${benches[@]}"; do run+=("${cases[$k]}") done taskset -c 2,3 \ chrt -f 50 \ hyperfine "${hyperfineArgs[@]}" "${run[@]}" ```

jade referenced this issue

2024-03-16 22:22:13 +00:00

heavier nightly/weekly ci jobs #142