lix-project/lix

Fork 31

duplicate info from the sqlite database as xattrs #298

New issue

Open

opened 2024-05-11 02:41:33 +00:00 by qyriad · 4 comments

qyriad commented

2024-05-11 02:41:33 +00:00

Owner

store database corruption is suffering. the store should be the ultimate source of truth and anything outside of it should be restorable from it.

we've agreed on this before but not written it down

store database corruption is suffering. the store should be the ultimate source of truth and anything outside of it should be restorable from it. we've agreed on this before but not written it down

qyriad added the

E/hard

Area/store

labels 2024-05-11 02:41:33 +00:00

qyriad added this to the post-release milestone 2024-05-11 02:41:39 +00:00

raito commented

2024-05-11 12:04:24 +00:00

Owner

One of the pre-requisite:

/nix/store should contain top-level directories or .drv.

cc @pennae

One of the pre-requisite: - /nix/store should contain top-level directories or `.drv`. cc @pennae

pennae commented

2024-05-11 14:08:59 +00:00

Owner

realistically that means we can either do store metadata xattrs or full user xattrs, but never both. adding an additional layer to store paths (giving eg out=/nix/store/gibberish-hello-8.4/content) would likely be a breaking change and make the store measurable less efficient. we could disallow user xattrs on $out exactly but not restrict them for subpaths, which should be very reasonable for almost any use.

once we have that we can link each store path to its deriver via xattr, and likewise each CA realisation to its source. signature information and other metadata can go into that too, reference information may need a fallback to be dumped into a file instead though (if we otherwise exceed xattr size limits). at that point we're very close to maintaining a shadow hierarchy of metadata files, but xattrs do offer tangible performance benefits (and, importantly for ext4, do not use an inode). having the fallback will allow us to support filesystems that don't have xattrs though, which may or not actually be necessary.

this has some unfortunate interactions with user-specified xattrs, which we probably should support and definitely keep in mind while designing this. xattrs are very nice but the interface is truly horrid. for example, did you know that xfs will you set multiple gigabytes worth of xattrs on an inode but cannot ever return more than about 10000 xattr names, which in total cannot cover more than 650MB? did you know that ext4 will impose mkfs-option-dependent limits on xattr sizes? if we want to use xattrs we will have to be sure that we *alone* set xattrs on that inode, or things get very messy very quickly. realistically that means we can either do store metadata xattrs or full user xattrs, but never both. adding an additional layer to store paths (giving eg `out=/nix/store/gibberish-hello-8.4/content`) would likely be a breaking change *and* make the store measurable less efficient. we *could* disallow user xattrs on `$out` exactly but not restrict them for subpaths, which should be very reasonable for almost any use. once we have that we can link each store path to its deriver via xattr, and likewise each CA realisation to its source. signature information and other metadata can go into that too, reference information may need a fallback to be dumped into a file instead though (if we otherwise exceed xattr size limits). at that point we're very close to maintaining a shadow hierarchy of metadata files, but xattrs do offer tangible performance benefits (and, importantly for ext4, do not use an inode). having the fallback will allow us to support filesystems that don't have xattrs though, which may or not actually be necessary.

jade commented

2024-05-11 21:31:28 +00:00

Owner

hmmm. so i think it might be reasonable to say the top level item of a nar may not have xattrs, and this probably wouldn't make anyone cry too much, right?

pennae commented

2024-05-11 22:04:26 +00:00

Owner

hmmm. so i think it might be reasonable to say the top level item of a nar may not have xattrs, and this probably wouldn't make anyone cry too much, right?

yes, that would be a corollary. especially since the store path root is usually to be considered equivalent to /, /usr, or /opt on an FHS distro, which no single thing can reasonably expect to be able to set xattrs on.

> hmmm. so i think it might be reasonable to say the top level item of a nar may not have xattrs, and this probably wouldn't make anyone cry too much, right? yes, that would be a corollary. especially since the store path root is usually to be considered equivalent to `/`, `/usr`, or `/opt` on an FHS distro, which no single thing can reasonably expect to be able to set xattrs on.

No milestone

No project

No assignees

4 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-project/lix#298

No description provided.