lix-project/lix
18
37
Fork 14
Code Issues 286 Code Review (Gerrit) Projects 1 Releases Wiki Activity
lix/releng
History
jade 9bb7fb8f69 Rewrite docker to be sensible and smaller 
I have checked the image can build things and inspected `diff -ru`
compared to the old image. As far as I can tell it is more or less
the same besides the later git change.

Layers are now 65MB or less, and we aren't against the maxLayers limit
for the broken automatic layering to do anything but shove one store
path in a layer (which is good behaviour, actually).

This uses nix2container which streams images, so the build time is much
shorter.

I have also taken the opportunity to, in addition to fixing the 400MB
single layer (terrible, and what motivated this in the first place),
delete about 200MB of closure size inflicted by git vs gitMinimal
causing both perl and python to get into closure.

People mostly use this thing for CI, so I don't really think you need
advanced git operations, and large git can be added at the user side if
really motivated.

With love for whichever container developer somewhat ironically assumed
that one would not run skopeo in a minimal container that doesn't have a
/var/tmp.

Fixes: #378

Change-Id: Icc3aa20e64446276716fbbb87535fd5b50628010
2024-06-09 20:33:24 -07:00
..
__init__.py Implement docker upload in the releng tools 2024-06-09 00:30:12 -07:00
__main__.py Put into place initial release engineering 2024-06-06 20:53:08 -07:00
cli.py Implement docker upload in the releng tools 2024-06-09 00:30:12 -07:00
create_release.xsh Implement docker upload in the releng tools 2024-06-09 00:30:12 -07:00
docker.xsh Implement docker upload in the releng tools 2024-06-09 00:30:12 -07:00
environment.py Implement docker upload in the releng tools 2024-06-09 00:30:12 -07:00
keys.py releng: support pushing the manual to docs also 2024-06-06 20:53:08 -07:00
README.md Put into place initial release engineering 2024-06-06 20:53:08 -07:00
release-jobs.nix Rewrite docker to be sensible and smaller 2024-06-09 20:33:24 -07:00
version.py Put into place initial release engineering 2024-06-06 20:53:08 -07:00

README.md

Release engineering

This directory contains the release engineering scripts for Lix.

Release process

Prerequisites

  • FIXME: validation via misc tests in nixpkgs, among other things? What other validation do we need before we can actually release?
  • Have a release post ready to go as a PR on the website repo.
  • No release-blocker bugs.

Process

The following process can be done either against the staging environment or the live environment.

For staging, the buckets are staging-releases, staging-cache, etc.

FIXME: obtainment of signing key for signing cache paths?

First, we prepare the release. python -m releng prepare is used for this.

  • Gather everything in doc/manual/rl-next and put it in doc/manual/src/release-notes/rl-MAJOR.md.

Then we tag the release with python -m releng tag:

  • Git HEAD is detached.
  • officialRelease = true is set in flake.nix, this is committed, and a release is tagged.
  • The tag is merged back into the last branch (either main for new releases or release-MAJOR for maintenance releases) with git merge -s ours VERSION creating a history link but ignoring the tree of the release tag.
  • Git HEAD is once again detached onto the release tag.

Then, we build the release artifacts with python -m releng build:

  • Source tarball is generated with git archive, then checksummed.
  • Manifest for nix upgrade-nix is produced and put in s3://releases at /manifest.nix and /lix/lix-VERSION.
  • Release is built: hydraJobs.binaryTarball jobs are built, and joined into a derivation that depends on all of them and adds checksum files. This and the sources go into s3://releases/lix/lix-VERSION.

At this point we have a release/artifacts and release/manual directory which are ready to publish, and tags ready for publication. No keys are required to do this part.

Next, we do the publication with python -m releng upload:

  • Artifacts for this release are uploaded:

    • s3://releases/manifest.nix, changing the default version of Lix for nix upgrade-nix.
    • s3://releases/lix/lix-VERSION/ gets the following contents
      • Binary tarballs
      • Docs: manual/ (FIXME: should we actually do this? what about putting it on docs.lix.systems? I think doing both is correct, since the Web site should not be an archive of random old manuals)
      • Docs as tarball in addition to web.
      • Source tarball
      • Docker image (FIXME: upload to forgejo registry and github registry in the future)
    • s3://docs/manual/lix/MAJOR
    • s3://docs/manual/lix/stable

  • The tag is uploaded to the remote repo.

  • Manually build the installer using the scripts in the installer repo and upload.

    FIXME: This currently requires a local Apple Macintosh® aarch64 computer, but we could possibly automate doing it from the aarch64-darwin builder.

  • Manually Push the main/release branch directly to gerrit.

  • If this is a new major release, branch-off to release-MAJOR and push that branch directly to gerrit as well (FIXME: special creds for doing this as a service account so we don't need to have the gerrit perms to shoot ourselves in the foot by default because pushing to main is bad?).

    FIXME: automate branch-off to release-* branch.

  • Manually (FIXME?) switch back to the release branch, which now has the correct revision.

  • Post!!

Installer

The installer is cross-built to several systems from a Mac using build-all.xsh and upload-to-lix.xsh in the installer repo (FIXME: currently at least; maybe this should be moved here?) .

It installs a binary tarball (FIXME: it should be taught to substitute from cache instead) from some URL; this is the hydraJobs.binaryTarball. The default URLs differ by architecture and are configured here.

Infrastructure summary

  • releases.lix.systems (s3://releases):
    • Each release gets a directory: https://releases.lix.systems/lix/lix-2.90-beta.1
      • Binary tarballs: nix-2.90.0-beta.1-x86_64-linux.tar.xz, from hydraJobs.binaryTarball
      • Manifest: manifest.nix, an attrset of the store paths by architecture.
    • Manifest for nix upgrade-nix to the latest release at /manifest.nix.
  • cache.lix.systems (s3://cache):
    • Receives all artifacts for released versions of Lix; is a plain HTTP binary cache.
  • install.lix.systems (s3://install): 
    ~ » aws s3 ls s3://install/lix/
                           PRE lix-2.90-beta.0/
                           PRE lix-2.90-beta.1/
                           PRE lix-2.90.0pre20240411/
                           PRE lix-2.90.0pre20240412/
2024-05-05 18:59:11    6707344 lix-installer-aarch64-darwin
2024-05-05 18:59:16    7479768 lix-installer-aarch64-linux
2024-05-05 18:59:14    7982208 lix-installer-x86_64-darwin
2024-05-05 18:59:17    8978352 lix-installer-x86_64-linux

~ » aws s3 ls s3://install/lix/lix-2.90-beta.1/
2024-05-05 18:59:01    6707344 lix-installer-aarch64-darwin
2024-05-05 18:59:06    7479768 lix-installer-aarch64-linux
2024-05-05 18:59:03    7982208 lix-installer-x86_64-darwin
2024-05-05 18:59:07    8978352 lix-installer-x86_64-linux