the-distro/infra
8
0
Fork 3
Code Issues 35 Pull requests 4 Packages Projects 1 Releases Wiki Activity

Deploy channel scripts #8

New issue
Open
opened 2024-07-01 17:21:28 +00:00 by delroth · 5 comments
delroth commented 2024-07-01 17:21:28 +00:00
Owner
Copy link
No description provided.
delroth added this to the Bootstrap infra project 2024-07-01 17:21:28 +00:00
raito commented 2024-07-04 18:07:37 +00:00
Owner
Copy link
- https://github.com/NixOS/nixos-channel-scripts/blob/master/mirror-nixos-branch.pl needs to be removed of any github assumption - https://github.com/NixOS/infra/blob/master/modules/hydra-mirror.nix needs to be isolated and brought in this repo with its dependencies, cleaned up of any github assumption as well
raito commented 2024-08-01 22:34:42 +00:00
Owner
Copy link

#91 is a first step, it takes care of the 2nd item of the previous message more or less.

https://git.lix.systems/the-distro/infra/pulls/91 is a first step, it takes care of the 2nd item of the previous message more or less.
raito commented 2024-08-01 22:45:46 +00:00
Owner
Copy link

Plan to rewrite the Perl script, here's what it's doing:

(1) Dry run feature
(2) Configuration file (state directory for nixpkgs, S3 targets, Hydra URL) -- currently hardcoded
(3) Protection against spurious channel rollback
(4) Idempotency (pushing twice the same release has no effect)

S3 operations

It writes the following in the root of the release directory:

  • src-url : evaluation URL
  • git-revision: git revision
  • binary-cache-url: the binary cache URL associated (hardcoded to cno)
  • store-paths.xz if available, otherwise store-paths by manual fetching (?)

If it's a NixOS channel:

  • nixexprs.tar.xz by downloading the nixos.channel product
  • packages.json.br by downloading the nixpkgs.tarball product of type json-br
  • options.json.br by downloading the nixos.options product of type json-br
  • nixos.iso_minimal.aarch64-linux
  • nixos.iso_minimal.x86_64-linux

If it's not small: add nixos.iso_plasma(5|6).(aarch64|x86_64)-linux.

Add the OVA appliance as well (nixos.ova.x86_64-linux).

Otherwise (nixpkgs, etc.): add nixexprs.tar.xz from the tarball product, of type source-dist and packages.json.br from the tarball product of type json-br.

If it's a NixOS channel, generate the SQLite database and add it to nixexprs.tar.xz in a mutable way (https://github.com/NixOS/nixos-channel-scripts/blob/master/mirror-nixos-branch.pl#L247-L256).

Write an HTML page for that particular release.

Finally, upload all the contents with some Content-Encoding fiddling and checksums computation which are added to the HTML page at the end to the S3 release bucket.

Git operations

git remote update origin <&2 && git push origin $rev:refs/heads/$channelName &>2 is performed.

S3 redirection operations

Redirections are added from the channel name to the release prefix name for:

  • /
  • /nixexprs.tar.xz
  • /git-revision
  • /packages.json.br
  • /store-paths.xz

If the channel is NixOS-type:

  • /options.json.br is added
  • all images are aliased as well to latest-$type-$arch.iso
  • OVA appliance is aliased as well
Plan to rewrite the Perl script, here's what it's doing: (1) Dry run feature (2) Configuration file (state directory for nixpkgs, S3 targets, Hydra URL) -- currently hardcoded (3) Protection against spurious channel rollback (4) Idempotency (pushing twice the same release has no effect) ### S3 operations It writes the following in the root of the release directory: - `src-url` : evaluation URL - `git-revision`: git revision - `binary-cache-url`: the binary cache URL associated (hardcoded to cno) - `store-paths.xz` if available, otherwise `store-paths` by manual fetching (?) If it's a NixOS channel: - `nixexprs.tar.xz` by downloading the `nixos.channel` product - `packages.json.br` by downloading the `nixpkgs.tarball` product of type `json-br` - `options.json.br` by downloading the `nixos.options` product of type `json-br` - `nixos.iso_minimal.aarch64-linux` - `nixos.iso_minimal.x86_64-linux` If it's not small: add `nixos.iso_plasma(5|6).(aarch64|x86_64)-linux`. Add the OVA appliance as well (`nixos.ova.x86_64-linux`). Otherwise (nixpkgs, etc.): add `nixexprs.tar.xz` from the `tarball` product, of type `source-dist` and `packages.json.br` from the `tarball` product of type `json-br`. If it's a NixOS channel, generate the SQLite database and add it to `nixexprs.tar.xz` in a mutable way (https://github.com/NixOS/nixos-channel-scripts/blob/master/mirror-nixos-branch.pl#L247-L256). Write an HTML page for that particular release. Finally, upload all the contents with some Content-Encoding fiddling and checksums computation which are added to the HTML page at the end to the S3 release bucket. ### Git operations `git remote update origin <&2 && git push origin $rev:refs/heads/$channelName &>2` is performed. ### S3 redirection operations Redirections are added from the channel name to the release prefix name for: - `/` - `/nixexprs.tar.xz` - `/git-revision` - `/packages.json.br` - `/store-paths.xz` If the channel is NixOS-type: - `/options.json.br` is added - all images are aliased as well to `latest-$type-$arch.iso` - OVA appliance is aliased as well
raito commented 2024-08-01 22:57:26 +00:00
Owner
Copy link

Proposed rewrite in Rust is to separate the concerns and get incremental feature parity:

(1) A configuration file to centralize all interesting parameters
(2) Implement each download and store operations with a work queue design
(3) Drop all the legacy cruft (compression, etc, etc.)
(4) Drop the SQLite database mutable generation stuff for now
(5) Implement the full S3 uploading process
(6) Git operations

For now, I suggest to put aside the S3 redirections.

Proposed rewrite in Rust is to separate the concerns and get incremental feature parity: (1) A configuration file to centralize all interesting parameters (2) Implement each download and store operations with a work queue design (3) Drop all the legacy cruft (compression, etc, etc.) (4) Drop the SQLite database mutable generation stuff for now (5) Implement the full S3 uploading process (6) Git operations For now, I suggest to put aside the S3 redirections.
raito self-assigned this 2024-08-10 18:59:21 +00:00
raito commented 2024-08-10 18:59:46 +00:00
Owner
Copy link

The rewrite in Rust is moving at a steady pace here: https://git.lix.systems/the-distro/channel-scripts/src/branch/rust.
I'm taking the lock because I have almost all the context now.

The rewrite in Rust is moving at a steady pace here: https://git.lix.systems/the-distro/channel-scripts/src/branch/rust. I'm taking the lock because I have almost all the context now.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
cache-ipv4
format
channel-scripts
acls
hexchen/colmena-checker
hexy/credentials
buildbot
ckie/moarr-v4
gerrit-metrics
moar-builders
unplug-epyc
hydra-wire-up
alertmanager
fodwatch
cadvisor-containers
gerrit-http-clones
meta01
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Bootstrap infra
Assignees
Clear assignees
No assignees
raito
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: the-distro/infra#8
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?