forked from lix-project/lix
Sergei Trofimovich
edfc5b2f12
For a typical desktop system (~2K packages) we can easily get 100K entries in RealisationsRefs. Without indices query for RealisationsRefs requires linear scan. RealisationsRefs(referrer) -------------------------- Inefficiency is seen as a 100% CPU load of nix-daemon for the following scenario: $ nix edit -f . bash # add unused environment variable, like FOO="1" # populate RealisationsRefs, build fresh system $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }' $ nix edit -f . bash # add unused environment variable, like FOO="2" $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }' In this case `bash `will be rebuilt a few times and then rest of CPU time is spent on scanning RealisationsRefs table (about 5 CPU-minutes on my machine). Before the change: $ time nix build -f nixos system ... # step 4 above real 34m3,613s user 0m5,232s sys 0m0,758s Of all this time about 29.5 minutes are taken by nix-daemon's CPU time. After the change: $ time nix build -f nixos system ... # step 4 above real 4m50,061s user 0m5,038s sys 0m0,677s Of all this time about 1 minute is taken by nix-daemon's CPU time. Most of the time is spent polling for non-existent realisations on cache-nixos.org. Realisations(outputPath) ------------------------ After running CA system for two weeks I got ~1M entries in Realisations table. `nix-collect-garbage` became very slow (seemingly 100 path deletions per second). It happens due to a slow cascading delete from Realisations triggered by deletion from ValidPaths. The fix is to add an index on primary key from ValidPaths(id) that triggers cascading deletions. Before the change: $ time nix-collect-garbage -d --max-freed 100G <interrupted before finish, took too long> real 23m32.411s user 17m49.679s sys 4m50.609s Most of time was spent in re-scanning Realisations table on each path deletion. After the change: $ time nix-collect-garbage -d --max-freed 100G real 8m43.226s user 6m16.317s sys 1m40.188s Time is spent scanning sqlite indices and in kernel when unlinking directories. |
||
---|---|---|
.. | ||
build | ||
builtins | ||
tests | ||
binary-cache-store.cc | ||
binary-cache-store.hh | ||
builtins.hh | ||
ca-specific-schema.sql | ||
content-address.cc | ||
content-address.hh | ||
crypto.cc | ||
crypto.hh | ||
daemon.cc | ||
daemon.hh | ||
derivations.cc | ||
derivations.hh | ||
derived-path.cc | ||
derived-path.hh | ||
dummy-store.cc | ||
export-import.cc | ||
filetransfer.cc | ||
filetransfer.hh | ||
fs-accessor.hh | ||
gc.cc | ||
globals.cc | ||
globals.hh | ||
http-binary-cache-store.cc | ||
legacy-ssh-store.cc | ||
local-binary-cache-store.cc | ||
local-fs-store.cc | ||
local-fs-store.hh | ||
local-store.cc | ||
local-store.hh | ||
local.mk | ||
lock.cc | ||
lock.hh | ||
machines.cc | ||
machines.hh | ||
misc.cc | ||
names.cc | ||
names.hh | ||
nar-accessor.cc | ||
nar-accessor.hh | ||
nar-info-disk-cache.cc | ||
nar-info-disk-cache.hh | ||
nar-info.cc | ||
nar-info.hh | ||
nix-store.pc.in | ||
optimise-store.cc | ||
parsed-derivations.cc | ||
parsed-derivations.hh | ||
path-info.cc | ||
path-info.hh | ||
path-with-outputs.cc | ||
path-with-outputs.hh | ||
path.cc | ||
path.hh | ||
pathlocks.cc | ||
pathlocks.hh | ||
profiles.cc | ||
profiles.hh | ||
realisation.cc | ||
realisation.hh | ||
references.cc | ||
references.hh | ||
remote-fs-accessor.cc | ||
remote-fs-accessor.hh | ||
remote-store.cc | ||
remote-store.hh | ||
s3-binary-cache-store.cc | ||
s3-binary-cache-store.hh | ||
s3.hh | ||
sandbox-defaults.sb | ||
sandbox-minimal.sb | ||
sandbox-network.sb | ||
schema.sql | ||
serve-protocol.hh | ||
sqlite.cc | ||
sqlite.hh | ||
ssh-store.cc | ||
ssh.cc | ||
ssh.hh | ||
store-api.cc | ||
store-api.hh | ||
uds-remote-store.cc | ||
uds-remote-store.hh | ||
worker-protocol.hh |