A modern, delicious implementation of the Nix package manager, focused on correctness, usability, and growth — and committed to doing right by its community
Find a file
Sergei Trofimovich edfc5b2f12 ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.

RealisationsRefs(referrer)
--------------------------

Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:

    $ nix edit -f . bash # add unused environment variable, like FOO="1"
    # populate RealisationsRefs, build fresh system
    $ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
    $ nix edit -f . bash # add unused environment variable, like FOO="2"
    $ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'

In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).

Before the change:

    $ time nix build -f nixos system ... # step 4 above
    real    34m3,613s
    user    0m5,232s
    sys     0m0,758s

Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.

After the change:

    $ time nix build -f nixos system ... # step 4 above
    real    4m50,061s
    user    0m5,038s
    sys     0m0,677s

Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.

Realisations(outputPath)
------------------------

After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.

The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.

Before the change:
    $ time nix-collect-garbage -d --max-freed 100G
    <interrupted before finish, took too long>
    real    23m32.411s
    user    17m49.679s
    sys     4m50.609s

Most of time was spent in re-scanning Realisations table on each path deletion.

After the change:
    $ time nix-collect-garbage -d --max-freed 100G

    real    8m43.226s
    user    6m16.317s
    sys     1m40.188s

Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-11-10 08:32:05 +00:00
.github Bump actions/checkout from 2.3.5 to 2.4.0 2021-11-08 22:00:58 +00:00
config Run autoupdate 2021-06-01 11:42:38 +02:00
contrib function-trace: always show the trace 2019-09-18 23:23:21 +02:00
doc/manual Fix some typos in CLI guideline. 2021-11-08 08:24:15 +01:00
m4 autoconf: Fix C++17 detection not working on Ubuntu 16.04. 2019-07-03 04:32:25 +02:00
maintainers Update release script 2021-11-02 11:53:25 +01:00
misc Merge pull request #4935 from alyssais/host_os 2021-07-08 12:40:54 +02:00
mk mk/libraries.mk: fix trace-ld and trace-ar expansions 2021-10-08 22:59:42 +01:00
nix-rust Apply OS checks to host platform, not build 2021-06-23 15:00:36 +00:00
perl Expose a perl method to query a derivation 2021-07-30 11:55:14 +02:00
scripts darwin-install: fix already-mounted store volumes 2021-11-04 14:09:40 -05:00
src ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath) 2021-11-10 08:32:05 +00:00
tests Merge pull request #5504 from NixOS/flake-options-and-daemon 2021-11-08 13:54:55 +01:00
.dir-locals.el .dir-locals.el: Set c-block-comment-prefix 2020-07-10 11:21:06 +02:00
.editorconfig Add .editorconfig 2017-06-05 22:57:28 +01:00
.gitignore gitignore the libstore-tests executable 2021-10-08 10:12:27 +02:00
.version Bump version to 2.5 2021-10-07 17:39:30 +02:00
boehmgc-coroutine-sp-fallback.diff Fix leaking pthread_attr_t 2021-11-03 22:54:16 +01:00
bootstrap.sh bootstrap: Simplify & make more robust. 2011-09-06 12:11:05 +00:00
configure.ac configure: explicit dependency on lowdown library 2021-09-14 07:54:37 +02:00
COPYING * Change this to LGPL to keep the government happy. 2006-04-25 16:41:06 +00:00
default.nix Simplify shell.nix and default.nix 2020-06-17 19:21:46 +02:00
flake.lock flake.lock: Update 2021-10-06 12:41:23 +02:00
flake.nix Merge remote-tracking branch 'origin/master' into non-blocking-gc 2021-10-28 14:56:55 +02:00
local.mk Remove 'dist' target 2020-12-03 16:17:58 +01:00
Makefile Add a test for RefScanSink and clean up the code 2021-10-04 14:29:42 +02:00
Makefile.config.in Apply OS checks to host platform, not build 2021-06-23 15:00:36 +00:00
precompiled-headers.h Config: Use nlohmann/json 2020-08-20 11:02:16 +02:00
README.md throw freenode down the memory hole 2021-05-27 21:48:39 +02:00
shell.nix Add back flake-compat shell.nix 2020-07-17 14:58:59 +00:00

Nix

Open Collective supporters Test

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. Please refer to the Nix manual for more details.

Installation

On Linux and macOS the easiest way to install Nix is to run the following shell command (as a user other than root):

$ curl -L https://nixos.org/nix/install | sh

Information on additional installation methods is available on the Nix download page.

Building And Developing

See our Hacking guide in our manual for instruction on how to build nix from source with nix-build or how to get a development environment.

Additional Resources

License

Nix is released under the LGPL v2.1.