`nix collect-garbage` is kind of slow at deleting paths #418

New issue

Open

opened 2024-06-24 23:50:57 +00:00 by jade · 1 comment

jade commented

2024-06-24 23:50:57 +00:00

Owner

I bet it's entirely single threaded. We should look into that.

jade added the

performance

label 2024-06-24 23:50:57 +00:00

jade commented

2024-06-28 05:31:41 +00:00

Author

Owner

The deleter queue is also initially populated with the current contents of the .garbage directory in case it is interrupted.

By doing it this way, we get:

atomicity of deleting paths (i.e. i suspect that currently you can corrupt a store path by half-deleting it and then interrupting that)
wayyyy faster GC since we can dispatch deletion to a thread pool

Redesign idea that would avoid a lot of rearchitecture (since the current gc preserves invariants like that paths always have their dependencies): run the GC in two parts: one thread, which is exactly identical to the current GC except it replaces the recursive-delete operation with another which moves paths into a garbage bin, say, `/nix/store/.garbage`, and adds it to a deleter queue. Then the deleter queue is consumed by a thread pool that does not care about ordering. The deleter queue is also initially populated with the current contents of the `.garbage` directory in case it is interrupted. By doing it this way, we get: - atomicity of deleting paths (i.e. i suspect that currently you can corrupt a store path by half-deleting it and then interrupting that) - wayyyy faster GC since we can dispatch deletion to a thread pool

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#418

No description provided.

nix collect-garbage is kind of slow at deleting paths #418

`nix collect-garbage` is kind of slow at deleting paths #418