lix-community/nix-security-tracker
0
0
Fork 0
Code Issues 94 Pull requests 9 Projects Releases Packages Wiki Activity

Periodically generate a full database dump #198

New issue
Closed
opened 2024-09-26 12:54:01 +00:00 by proofconstruction · 10 comments
proofconstruction commented 2024-09-26 12:54:01 +00:00 (Migrated from github.com)
Copy link

Currently, ingesting the full CVE list and a copy of nixpkgs can take several hours, dramatically reducing iteration speed.

To facilitate faster development, it would be useful to asynchronously generate DB dumps and cache them for easy downloading by developers.

Currently, ingesting the full CVE list and a copy of nixpkgs can take several hours, dramatically reducing iteration speed. To facilitate faster development, it would be useful to asynchronously generate DB dumps and cache them for easy downloading by developers.
RaitoBezarius commented 2024-10-08 14:51:04 +00:00 (Migrated from github.com)
Copy link

Fixed by 5422c157fb.

Fixed by https://github.com/Nix-Security-WG/nix-security-tracker/commit/5422c157fb20c71cd6c0a7dd6825d5ce81527f99.
erictapen commented 2024-10-08 15:04:09 +00:00 (Migrated from github.com)
Copy link

What's the URL endpoint for the downloadable dumps now? Didn't see that at first glance.

What's the URL endpoint for the downloadable dumps now? Didn't see that at first glance.
RaitoBezarius commented 2024-10-08 15:06:19 +00:00 (Migrated from github.com)
Copy link

https://dumps.sectracker.nixpkgs.lahfa.xyz/

https://dumps.sectracker.nixpkgs.lahfa.xyz/
👍 1
erictapen commented 2024-10-21 13:11:01 +00:00 (Migrated from github.com)
Copy link

@RaitoBezarius the action is failing currently, so only one dump from 12 days is available on the bucket. Could you have a look? I'm still dumping directly from the staging host to get a working db, which is time consuming.

https://github.com/Nix-Security-WG/nix-security-tracker/actions/runs/11433700475/job/31806093828#step:9:18

@RaitoBezarius the action is failing currently, so only one dump from 12 days is available on the bucket. Could you have a look? I'm still dumping directly from the staging host to get a working db, which is time consuming. https://github.com/Nix-Security-WG/nix-security-tracker/actions/runs/11433700475/job/31806093828#step:9:18
fricklerhandwerk commented 2024-10-28 13:53:48 +00:00 (Migrated from github.com)
Copy link

Restoring the dump fails for me with:

pg_restore: processing data for table "public.shared_nixderivationmeta_source_provenances"
pg_restore: processing data for table "public.shared_nixderivationoutput"
pg_restore: error: could not uncompress data: incorrect data check
Restoring the dump fails for me with: ``` pg_restore: processing data for table "public.shared_nixderivationmeta_source_provenances" pg_restore: processing data for table "public.shared_nixderivationoutput" pg_restore: error: could not uncompress data: incorrect data check ```
RaitoBezarius commented 2024-10-28 13:54:29 +00:00 (Migrated from github.com)
Copy link

Can you provide a reproducer?

Can you provide a reproducer?
fricklerhandwerk commented 2024-10-28 13:55:59 +00:00 (Migrated from github.com)
Copy link

I'm following the steps in CONTRIBUTING.md. It seems the download repeatedly doesn't finish around the 1GiB mark:

$ curl https://dumps.sectracker.nixpkgs.lahfa.xyz/web-security-tracker --output dump
...
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)
I'm following [the steps in CONTRIBUTING.md](https://github.com/Nix-Security-WG/nix-security-tracker/blob/main/CONTRIBUTING.md#set-up-a-local-container). It seems the download repeatedly doesn't finish around the 1GiB mark: ```console $ curl https://dumps.sectracker.nixpkgs.lahfa.xyz/web-security-tracker --output dump ... curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) ```
fricklerhandwerk commented 2024-10-30 10:53:28 +00:00 (Migrated from github.com)
Copy link

Okay it seems being in a certain region of Germany messes up routing. It works from my VPS in Frankfurt...

Okay it seems being in a certain region of Germany messes up routing. It works from my VPS in Frankfurt...
RaitoBezarius commented 2024-10-30 11:08:07 +00:00 (Migrated from github.com)
Copy link

@fricklerhandwerk Can you run from your weird German region:

$ tracepath dumps.sectracker.nixpkgs.lahfa.xyz

and show me the result?

@fricklerhandwerk Can you run from your weird German region: ``` $ tracepath dumps.sectracker.nixpkgs.lahfa.xyz ``` and show me the result?
fricklerhandwerk commented 2024-10-31 10:20:27 +00:00 (Migrated from github.com)
Copy link

Another problem is that the dump is not being made periodically. There seems to be a problem with the GitHub Action.

Another problem is that the dump is not being made periodically. There seems to be a problem with the GitHub Action.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
npins-auto-update
No results found.
Labels
Clear labels   
automation

   
backend

something the web server needs to provide the frontend with

  
bug

Something isn't working

  
contributor experience

   
data

something about quality or quantity of ingested data

  
deployment

   
documentation

Improvements or additions to documentation

  
duplicate

This issue or pull request already exists

  
good first issue

Good for newcomers

  
help wanted

Extra attention is needed

  
nice to have

Nice to have, but not necessary for the time being

  
notifications

   
package maintainer

user story for package maintainers

  
performance

   
skin

Anything related to the visual presentation

  
tech debt

Activities that reduce tech debt

  
user story

description or implementation of a workflow

No labels
automation
backend
bug
contributor experience
data
deployment
documentation
duplicate
good first issue
help wanted
nice to have
notifications
package maintainer
performance
skin
tech debt
user story
Milestone
Clear milestone
No items
No milestone
1 Quick triaging
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: lix-community/nix-security-tracker#198
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?