2020-10-20 13:14:02 +00:00
|
|
|
|
-- Extension of the sql schema for content-addressed derivations.
|
|
|
|
|
-- Won't be loaded unless the experimental feature `ca-derivations`
|
|
|
|
|
-- is enabled
|
|
|
|
|
|
|
|
|
|
create table if not exists Realisations (
|
2021-05-07 11:57:01 +00:00
|
|
|
|
id integer primary key autoincrement not null,
|
2020-10-20 13:14:02 +00:00
|
|
|
|
drvPath text not null,
|
|
|
|
|
outputName text not null, -- symbolic output id, usually "out"
|
|
|
|
|
outputPath integer not null,
|
2021-03-08 10:56:33 +00:00
|
|
|
|
signatures text, -- space-separated list
|
2020-10-20 13:14:02 +00:00
|
|
|
|
foreign key (outputPath) references ValidPaths(id) on delete cascade
|
|
|
|
|
);
|
2021-05-07 11:57:01 +00:00
|
|
|
|
|
|
|
|
|
create index if not exists IndexRealisations on Realisations(drvPath, outputName);
|
|
|
|
|
|
2021-11-03 15:30:22 +00:00
|
|
|
|
-- We can end-up in a weird edge-case where a path depends on itself because
|
|
|
|
|
-- it’s an output of a CA derivation, that happens to be the same as one of its
|
|
|
|
|
-- dependencies.
|
|
|
|
|
-- In that case we have a dependency loop (path -> realisation1 -> realisation2
|
|
|
|
|
-- -> path) that we need to break by removing the dependencies between the
|
|
|
|
|
-- realisations
|
|
|
|
|
create trigger if not exists DeleteSelfRefsViaRealisations before delete on ValidPaths
|
|
|
|
|
begin
|
2021-11-05 10:17:22 +00:00
|
|
|
|
delete from RealisationsRefs where realisationReference in (
|
2021-11-03 15:30:22 +00:00
|
|
|
|
select id from Realisations where outputPath = old.id
|
|
|
|
|
);
|
|
|
|
|
end;
|
|
|
|
|
|
2021-05-07 11:57:01 +00:00
|
|
|
|
create table if not exists RealisationsRefs (
|
|
|
|
|
referrer integer not null,
|
|
|
|
|
realisationReference integer,
|
|
|
|
|
foreign key (referrer) references Realisations(id) on delete cascade,
|
|
|
|
|
foreign key (realisationReference) references Realisations(id) on delete restrict
|
|
|
|
|
);
|
2022-03-26 16:18:51 +00:00
|
|
|
|
-- used by deletion trigger
|
|
|
|
|
create index if not exists IndexRealisationsRefsRealisationReference on RealisationsRefs(realisationReference);
|
ca-specific-schema.sql: add index on RealisationsRefs(referrer) and (outputPath)
For a typical desktop system (~2K packages) we can easily get 100K
entries in RealisationsRefs. Without indices query for RealisationsRefs
requires linear scan.
RealisationsRefs(referrer)
--------------------------
Inefficiency is seen as a 100% CPU load of nix-daemon for the following
scenario:
$ nix edit -f . bash # add unused environment variable, like FOO="1"
# populate RealisationsRefs, build fresh system
$ nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
$ nix edit -f . bash # add unused environment variable, like FOO="2"
$ time nix build -f nixos system --arg config '{ contentAddressedByDefault = true; }'
In this case `bash `will be rebuilt a few times and then rest of CPU
time is spent on scanning RealisationsRefs table (about 5 CPU-minutes
on my machine).
Before the change:
$ time nix build -f nixos system ... # step 4 above
real 34m3,613s
user 0m5,232s
sys 0m0,758s
Of all this time about 29.5 minutes are taken by nix-daemon's CPU time.
After the change:
$ time nix build -f nixos system ... # step 4 above
real 4m50,061s
user 0m5,038s
sys 0m0,677s
Of all this time about 1 minute is taken by nix-daemon's CPU time.
Most of the time is spent polling for non-existent realisations on
cache-nixos.org.
Realisations(outputPath)
------------------------
After running CA system for two weeks I got ~1M entries in Realisations
table. `nix-collect-garbage` became very slow (seemingly 100 path deletions
per second). It happens due to a slow cascading delete from Realisations
triggered by deletion from ValidPaths.
The fix is to add an index on primary key from ValidPaths(id) that
triggers cascading deletions.
Before the change:
$ time nix-collect-garbage -d --max-freed 100G
<interrupted before finish, took too long>
real 23m32.411s
user 17m49.679s
sys 4m50.609s
Most of time was spent in re-scanning Realisations table on each path deletion.
After the change:
$ time nix-collect-garbage -d --max-freed 100G
real 8m43.226s
user 6m16.317s
sys 1m40.188s
Time is spent scanning sqlite indices and in kernel when unlinking directories.
2021-10-09 17:47:05 +00:00
|
|
|
|
|
|
|
|
|
-- used by QueryRealisationReferences
|
|
|
|
|
create index if not exists IndexRealisationsRefs on RealisationsRefs(referrer);
|
|
|
|
|
-- used by cascade deletion when ValidPaths is deleted
|
|
|
|
|
create index if not exists IndexRealisationsRefsOnOutputPath on Realisations(outputPath);
|