lix-community/nix-security-tracker
0
0
Fork 0
Code Issues 94 Pull requests 9 Projects Releases Packages Wiki Activity

feat: history log setup #101

Closed
alejandrosame wants to merge 1 commit from feat/history-log into main
pull from: feat/history-log
merge into: lix-community:main
Conversation 11 Commits 1 Files changed 7 +306 -1
alejandrosame commented 2023-12-11 04:20:53 +00:00 (Migrated from github.com)
Copy link

The history log is implemented with the package 'django-pghistory', which uses pg triggers.

The summary of changes is as follows:

  • Changed DATABASE_URL to signal the dependency on postgresql. Migrations won't work anymore with sqlite.
  • Added python packages 'django-pghistory' and 'django-pgtrigger' (a dependency of the former inside django app).
  • A basic NixpkgsIssue snapshot model, tracking cve, derivations and statusfields.

This constitutes the building blocks for all activity log we want to register.

The history log is implemented with the package 'django-pghistory', which uses pg triggers. The summary of changes is as follows: - Changed DATABASE_URL to signal the dependency on postgresql. Migrations won't work anymore with sqlite. - Added python packages 'django-pghistory' and 'django-pgtrigger' (a dependency of the former inside django app). - A basic NixpkgsIssue snapshot model, tracking `cve`, `derivations` and `status`fields. This constitutes the building blocks for all activity log we want to register.
fricklerhandwerk (Migrated from github.com) reviewed 2023-12-11 04:20:53 +00:00
fricklerhandwerk (Migrated from github.com) requested changes 2023-12-11 13:41:58 +00:00
fricklerhandwerk (Migrated from github.com) left a comment
Copy link

This now requires a substantial rebase

This now requires a substantial rebase
nix/overlay.nix
fricklerhandwerk (Migrated from github.com) commented 2023-12-11 09:57:08 +00:00
Copy link

The prefix should't be needed with #86

The prefix should't be needed with #86
pkgs/python/django-pghistory/default.nix
fricklerhandwerk (Migrated from github.com) commented 2023-12-11 09:58:13 +00:00
Copy link
{ lib, fetchFromGitHub, buildPythonPackage, django_4, poetry-core }:

buildPythonPackage rec {
  pname = "django-pghistory";
  version = "3.1.0";
  pyproject = true;

  src = fetchFromGitHub {
    owner = "Opus10";
    repo = "django-pghistory";
    rev = "${version}";
    hash = "sha256-+/JmbKd4Qe+hR0UtBg4wm682pss9WCQhe4xE5UZDIik=";
  };

  propagatedBuildInputs = [ django_4 poetry-core ];

Also with #86, and similar for the other package

```suggestion { lib, fetchFromGitHub, buildPythonPackage, django_4, poetry-core }: buildPythonPackage rec { pname = "django-pghistory"; version = "3.1.0"; pyproject = true; src = fetchFromGitHub { owner = "Opus10"; repo = "django-pghistory"; rev = "${version}"; hash = "sha256-+/JmbKd4Qe+hR0UtBg4wm682pss9WCQhe4xE5UZDIik="; }; propagatedBuildInputs = [ django_4 poetry-core ]; ``` Also with #86, and similar for the other package
src/website/tracker/settings.py
fricklerhandwerk (Migrated from github.com) commented 2023-12-11 10:06:26 +00:00
Copy link

https://github.com/Nix-Security-WG/nix-security-tracker/pull/66/files#diff-42e2af48200a6b6083704ad7f3ceadc6138e7cfe19200e80f0229a035e18edc6R182

@RaitoBezarius Is the linked comment correct?

https://github.com/Nix-Security-WG/nix-security-tracker/pull/66/files#diff-42e2af48200a6b6083704ad7f3ceadc6138e7cfe19200e80f0229a035e18edc6R182 @RaitoBezarius Is the linked comment correct?
fricklerhandwerk (Migrated from github.com) reviewed 2023-12-11 13:45:42 +00:00
src/website/shared/models/cve.py
fricklerhandwerk (Migrated from github.com) commented 2023-12-11 13:45:09 +00:00
Copy link

We don't want to snapshot everything, as that will require a lot of churn and diffing the snapshots and such. Also the DB can be expected to be large. What we need to track already:

  • status changed
  • description snapshots
  • cve added/removed
  • derivation added/removed

All this indexed by user.

We don't want to snapshot everything, as that will require a lot of churn and diffing the snapshots and such. Also the DB can be expected to be large. What we need to track already: - status changed - description snapshots - cve added/removed - derivation added/removed All this indexed by user.
alejandrosame (Migrated from github.com) reviewed 2023-12-11 15:54:32 +00:00
nix/overlay.nix
alejandrosame (Migrated from github.com) commented 2023-12-11 15:54:32 +00:00
Copy link

applied

applied
alejandrosame (Migrated from github.com) reviewed 2023-12-11 15:54:44 +00:00
pkgs/python/django-pghistory/default.nix
alejandrosame (Migrated from github.com) commented 2023-12-11 15:54:44 +00:00
Copy link

applied

applied
alejandrosame (Migrated from github.com) reviewed 2023-12-11 15:56:46 +00:00
src/website/tracker/settings.py
alejandrosame (Migrated from github.com) commented 2023-12-11 15:56:46 +00:00
Copy link

Assuming you are referring to the placement of this settings before the user settings logic, applied already.

Assuming you are referring to the placement of this settings before the user settings logic, applied already.
alejandrosame (Migrated from github.com) reviewed 2023-12-11 16:22:33 +00:00
src/website/shared/models/cve.py
alejandrosame (Migrated from github.com) commented 2023-12-11 16:22:33 +00:00
Copy link

I added tracking per field. pghistory already diffs changes, instead of snapshotting the whole row.

For the CVE and derivation addition/removal I'll need to have clear the representation to contruct the proper query, for now it tracks the field directly in the NixpkgsIssue model.

I added tracking per field. pghistory already diffs changes, instead of snapshotting the whole row. For the CVE and derivation addition/removal I'll need to have clear the representation to contruct the proper query, for now it tracks the field directly in the NixpkgsIssue model.
alejandrosame commented 2023-12-11 16:27:30 +00:00 (Migrated from github.com)
Copy link

There are migration errors in main that prevent me from finishing a proper rebase. They should be fixed there first so I can rebase again.

There are migration errors in main that prevent me from finishing a proper rebase. They should be fixed there first so I can rebase again.
fricklerhandwerk (Migrated from github.com) reviewed 2023-12-11 17:34:06 +00:00
src/website/shared/models/cve.py
@ -479,6 +302,11 @@ class IssueStatus(models.TextChoices):
WONTFIX = "W", _("wontfix")
fricklerhandwerk (Migrated from github.com) commented 2023-12-11 17:34:06 +00:00
Copy link
    fields=["cve", "derivations", "status"],

We should defer the decision on how to deal with description changes to later.

```suggestion fields=["cve", "derivations", "status"], ``` We should defer the decision on how to deal with description changes to later.
alejandrosame commented 2023-12-12 16:33:42 +00:00 (Migrated from github.com)
Copy link

This PR is in a working state again.

After much fiddling with the default admin views of pghistory, I don't find them very usable. Taking this as starting point, I'll now create an Activity Log app to put there usable custom views of the different tables tracking the changes we want to summarise.

This PR is in a working state again. After much fiddling with the default admin views of `pghistory`, I don't find them very usable. Taking this as starting point, I'll now create an `Activity Log` app to put there usable custom views of the different tables tracking the changes we want to summarise.
alejandrosame commented 2024-11-28 03:04:31 +00:00 (Migrated from github.com)
Copy link

Implemented in #366

Implemented in #366

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
fricklerhandwerk
Labels
Clear labels   
automation

   
backend

something the web server needs to provide the frontend with

  
bug

Something isn't working

  
contributor experience

   
data

something about quality or quantity of ingested data

  
deployment

   
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
nice to have

Nice to have, but not necessary for the time being

  
notifications

   
package maintainer

user story for package maintainers

  
performance

   
skin

Anything related to the visual presentation

  
tech debt

Activities that reduce tech debt

  
user story

description or implementation of a workflow

No labels
automation
backend
bug
contributor experience
data
deployment
documentation
duplicate
good first issue
help wanted
nice to have
notifications
package maintainer
performance
skin
tech debt
user story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-community/nix-security-tracker#101
No description provided.
Delete branch "feat/history-log"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?