WIP: feat: make eval store configurable #451

Draft

fricklerhandwerk wants to merge 1 commit from fricklerhandwerk/nix-eval-store into main

fricklerhandwerk commented

2024-12-10 13:08:53 +00:00

(Migrated from github.com)

evaluations write garbage, which we don't use, to disk if we hit the physical store, so this should speed things up a bit

RaitoBezarius (Migrated from github.com) reviewed

2024-12-10 13:08:53 +00:00

RaitoBezarius commented

2024-12-10 19:03:58 +00:00

(Migrated from github.com)

TODO to unblock:

add tmpfs creation in the nixos modules
configure that evaluation store dir
configure it for the staging deployment

TODO to unblock: - [ ] add tmpfs creation in the nixos modules - [ ] configure that evaluation store dir - [ ] configure it for the staging deployment

Erethon commented

2024-12-12 00:12:35 +00:00

(Migrated from github.com)

Something to keep in mind if we're to implement this (and we might need to document it even if we keep using /nix/store): The tracker produces a lot of derivations, currently in the production instance we have ~1.3 million files in /nix/store and that's only with 24.11 and part of 24.05 ingested. This makes it easy for a filesystem to run out of inodes (it happened earlier in the production instance and I had to intervene).

A tmpfs with default settings has number_of_pages / 2 inodes. This means that on a typical system with 16GB of memory and a pagesize of 4096, we get ~4 million pages, which means ~2 million inodes. Two million inodes might not be enough for our usecase, so we'll have to mount the tmpfs with more inodes instead of the default.

Then again assuming a derivation is ~3kB in size (which I think is fairly typical) and we have ~1.5m of those, that's already almost 5GB of data, so we'll face memory issues as well as inode issues. If we really don't need those derivations, we should have a way to clean them as soon as we're done with them. And ideally not have them touch the disk but use a tmpfs, as per the original idea.

Something to keep in mind if we're to implement this (and we might need to document it even if we keep using `/nix/store`): The tracker produces a lot of derivations, currently in the production instance we have ~1.3 million files in `/nix/store` and that's only with 24.11 and part of 24.05 ingested. This makes it easy for a filesystem to run out of inodes (it happened earlier in the production instance and I had to intervene). A tmpfs with default settings has `number_of_pages / 2` inodes. This means that on a typical system with 16GB of memory and a pagesize of 4096, we get ~4 million pages, which means ~2 million inodes. Two million inodes might not be enough for our usecase, so we'll have to mount the tmpfs with more inodes instead of the default. Then again assuming a derivation is ~3kB in size (which I think is fairly typical) and we have ~1.5m of those, that's already almost 5GB of data, so we'll face memory issues as well as inode issues. If we really don't need those derivations, we should have a way to clean them as soon as we're done with them. And ideally not have them touch the disk but use a tmpfs, as per the original idea.

👍 1

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fricklerhandwerk/nix-eval-store:fricklerhandwerk/nix-eval-store

git switch fricklerhandwerk/nix-eval-store

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff fricklerhandwerk/nix-eval-store

git switch fricklerhandwerk/nix-eval-store

git rebase main

git switch main

git merge --ff-only fricklerhandwerk/nix-eval-store

git switch fricklerhandwerk/nix-eval-store

git rebase main

git switch main

git merge --no-ff fricklerhandwerk/nix-eval-store

git switch main

git merge --squash fricklerhandwerk/nix-eval-store