lix-community/nix-security-tracker
0
0
Fork 0
Code Issues 94 Pull requests 9 Projects Releases Packages Wiki Activity

WIP: Add a model for package #537

Draft
florentc wants to merge 1 commit from tweag/florentc/package-schema into main
pull from: tweag/florentc/package-schema
merge into: lix-community:main
Conversation 5 Commits 1 Files changed 2 +52
florentc commented 2025-05-07 14:37:43 +00:00 (Migrated from github.com)
Copy link

As a step to implement #532, this WIP PR introduces a db model for packages. For now, a package only has a name. The nix derivation table gains a column, package, for the many-to-one relation from derivation to package. The provided migration ensures the creation of packages associated to all derivations and links each derivation to the right package.

The idea is that user profiles will be in relation with packages to model subscriptions. This will make it possible to subscribe to a package name as we discussed with @fricklerhandwerk and make queries easier to link a "package" and the many derivations. Eventually, we could envision to migrate some metadata from derivations, which are in fact shared and package-related, to this package model.

As a step to implement #532, this WIP PR introduces a db model for packages. For now, a package only has a name. The nix derivation table gains a column, package, for the many-to-one relation from derivation to package. The provided migration ensures the creation of packages associated to all derivations and links each derivation to the right package. The idea is that user profiles will be in relation with packages to model subscriptions. This will make it possible to subscribe to a package name as we discussed with @fricklerhandwerk and make queries easier to link a "package" and the many derivations. Eventually, we could envision to migrate some metadata from derivations, which are in fact shared and package-related, to this package model.
fricklerhandwerk (Migrated from github.com) reviewed 2025-05-07 14:50:42 +00:00
src/website/shared/migrations/0049_nixpackage_nixderivation_package.py
@ -0,0 +5,4 @@
NixPackage = apps.get_model('shared', 'NixPackage')
NixDerivation = apps.get_model('shared', 'NixDerivation')
with transaction.atomic():
fricklerhandwerk (Migrated from github.com) commented 2025-05-07 14:50:42 +00:00
Copy link

This will probably stop the world for a long time

This will probably stop the world for a long time
florentc commented 2025-05-07 14:53:36 +00:00 (Migrated from github.com)
Copy link

@fricklerhandwerk I wanted your opinion/advice about implementing setting up the package of a derivation automatically and eventually making the package field non nullable. What would be the go-to way to automatically get_or_create a package and assign it to the package field?

Should I override the save method of the model or use some signal (post_save or other?) or something else

This would look something like this in both cases:

package, _ = NixPackage.objects.get_or_create(name=self.attribute)
self.package = package
@fricklerhandwerk I wanted your opinion/advice about implementing setting up the package of a derivation automatically and eventually making the package field non nullable. What would be the go-to way to automatically get_or_create a package and assign it to the package field? Should I override the `save` method of the model or use some signal (post_save or other?) or something else This would look something like this in both cases: ```python package, _ = NixPackage.objects.get_or_create(name=self.attribute) self.package = package ```
florentc (Migrated from github.com) reviewed 2025-05-07 14:56:00 +00:00
src/website/shared/migrations/0049_nixpackage_nixderivation_package.py
@ -0,0 +5,4 @@
NixPackage = apps.get_model('shared', 'NixPackage')
NixDerivation = apps.get_model('shared', 'NixDerivation')
with transaction.atomic():
florentc (Migrated from github.com) commented 2025-05-07 14:56:00 +00:00
Copy link

It did take something like 5 min. Do you know how it could be optimized?

It did take something like 5 min. Do you know how it could be optimized?
fricklerhandwerk commented 2025-05-08 07:42:49 +00:00 (Migrated from github.com)
Copy link

To be honest, I don't know, but I agree we'd need something like that.

Although I wonder whether it's a good idea to store the "package" aggregation at the relational layer. We need that information very quickly for (anything like) search (#215), so it needs to be cached anyway. So why not instead let the NixPackage model have a nullable JSONField that would store the derivations already together with whatever data we need, channel and version in particular? Then we could populate it on first access (e.g. when it's None and then update whenever a new NixDerivation is ingested, which would need a post_save signal on NixDerivation). At least for now, or maybe we'd do a management command that would populate the cache, just like with suggestions.

To be honest, I don't know, but I agree we'd need something like that. Although I wonder whether it's a good idea to store the "package" aggregation at the relational layer. We need that information very quickly for (anything like) search (#215), so it needs to be cached anyway. So why not instead let the `NixPackage` model have a nullable `JSONField` that would store the derivations already together with whatever data we need, channel and version in particular? Then we could populate it on first access (e.g. when it's `None` and then update whenever a new `NixDerivation` is ingested, which would need a `post_save` signal on `NixDerivation`). At least for now, or maybe we'd do a management command that would populate the cache, just like with suggestions.
fricklerhandwerk (Migrated from github.com) reviewed 2025-05-08 07:59:01 +00:00
src/website/shared/migrations/0049_nixpackage_nixderivation_package.py
@ -0,0 +5,4 @@
NixPackage = apps.get_model('shared', 'NixPackage')
NixDerivation = apps.get_model('shared', 'NixDerivation')
with transaction.atomic():
fricklerhandwerk (Migrated from github.com) commented 2025-05-08 07:59:01 +00:00
Copy link

Generally something like here: https://github.com/Nix-Security-WG/nix-security-tracker/pull/538/files

But see my top level comment for why we may not need that anyway

Generally something like here: https://github.com/Nix-Security-WG/nix-security-tracker/pull/538/files But see my top level comment for why we may not need that anyway
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin tweag/florentc/package-schema:tweag/florentc/package-schema
git switch tweag/florentc/package-schema

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff tweag/florentc/package-schema
git switch tweag/florentc/package-schema
git rebase main
git switch main
git merge --ff-only tweag/florentc/package-schema
git switch tweag/florentc/package-schema
git rebase main
git switch main
git merge --no-ff tweag/florentc/package-schema
git switch main
git merge --squash tweag/florentc/package-schema
git switch main
git merge --ff-only tweag/florentc/package-schema
git switch main
git merge tweag/florentc/package-schema
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
automation

   
backend

something the web server needs to provide the frontend with

  
bug

Something isn't working

  
contributor experience

   
data

something about quality or quantity of ingested data

  
deployment

   
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
nice to have

Nice to have, but not necessary for the time being

  
notifications

   
package maintainer

user story for package maintainers

  
performance

   
skin

Anything related to the visual presentation

  
tech debt

Activities that reduce tech debt

  
user story

description or implementation of a workflow

No labels
automation
backend
bug
contributor experience
data
deployment
documentation
duplicate
good first issue
help wanted
nice to have
notifications
package maintainer
performance
skin
tech debt
user story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-community/nix-security-tracker#537
No description provided.
Delete branch "tweag/florentc/package-schema"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?