lix/releng
jade ce71d0e9ab releng: automatically figure out if we should tag latest for docker
For example, when releasing from release-2.90, if `main` has a 2.91 tag
ancestor, we know that 2.91 was released, so we should *not* tag latest.

Change-Id: Ia56b17a2ee03bbec74b7c271c742858c690d450d
2024-06-09 20:33:24 -07:00
..
__init__.py releng: automatically figure out if we should tag latest for docker 2024-06-09 20:33:24 -07:00
__main__.py Put into place initial release engineering 2024-06-06 20:53:08 -07:00
cli.py Implement docker upload in the releng tools 2024-06-09 00:30:12 -07:00
create_release.xsh releng: automatically figure out if we should tag latest for docker 2024-06-09 20:33:24 -07:00
docker.xsh releng: support multiarch docker images 2024-06-09 20:33:24 -07:00
docker_assemble.py releng: support multiarch docker images 2024-06-09 20:33:24 -07:00
environment.py releng: automatically figure out if we should tag latest for docker 2024-06-09 20:33:24 -07:00
gitutils.xsh releng: automatically figure out if we should tag latest for docker 2024-06-09 20:33:24 -07:00
keys.py releng: support pushing the manual to docs also 2024-06-06 20:53:08 -07:00
README.md Put into place initial release engineering 2024-06-06 20:53:08 -07:00
release-jobs.nix Rewrite docker to be sensible and smaller 2024-06-09 20:33:24 -07:00
version.py Put into place initial release engineering 2024-06-06 20:53:08 -07:00

Release engineering

This directory contains the release engineering scripts for Lix.

Release process

Prerequisites

  • FIXME: validation via misc tests in nixpkgs, among other things? What other validation do we need before we can actually release?
  • Have a release post ready to go as a PR on the website repo.
  • No release-blocker bugs.

Process

The following process can be done either against the staging environment or the live environment.

For staging, the buckets are staging-releases, staging-cache, etc.

FIXME: obtainment of signing key for signing cache paths?

First, we prepare the release. python -m releng prepare is used for this.

  • Gather everything in doc/manual/rl-next and put it in doc/manual/src/release-notes/rl-MAJOR.md.

Then we tag the release with python -m releng tag:

  • Git HEAD is detached.
  • officialRelease = true is set in flake.nix, this is committed, and a release is tagged.
  • The tag is merged back into the last branch (either main for new releases or release-MAJOR for maintenance releases) with git merge -s ours VERSION creating a history link but ignoring the tree of the release tag.
  • Git HEAD is once again detached onto the release tag.

Then, we build the release artifacts with python -m releng build:

  • Source tarball is generated with git archive, then checksummed.
  • Manifest for nix upgrade-nix is produced and put in s3://releases at /manifest.nix and /lix/lix-VERSION.
  • Release is built: hydraJobs.binaryTarball jobs are built, and joined into a derivation that depends on all of them and adds checksum files. This and the sources go into s3://releases/lix/lix-VERSION.

At this point we have a release/artifacts and release/manual directory which are ready to publish, and tags ready for publication. No keys are required to do this part.

Next, we do the publication with python -m releng upload:

  • Artifacts for this release are uploaded:

    • s3://releases/manifest.nix, changing the default version of Lix for nix upgrade-nix.
    • s3://releases/lix/lix-VERSION/ gets the following contents
      • Binary tarballs
      • Docs: manual/ (FIXME: should we actually do this? what about putting it on docs.lix.systems? I think doing both is correct, since the Web site should not be an archive of random old manuals)
      • Docs as tarball in addition to web.
      • Source tarball
      • Docker image (FIXME: upload to forgejo registry and github registry in the future)
    • s3://docs/manual/lix/MAJOR
    • s3://docs/manual/lix/stable
  • The tag is uploaded to the remote repo.

  • Manually build the installer using the scripts in the installer repo and upload.

    FIXME: This currently requires a local Apple Macintosh® aarch64 computer, but we could possibly automate doing it from the aarch64-darwin builder.

  • Manually Push the main/release branch directly to gerrit.

  • If this is a new major release, branch-off to release-MAJOR and push that branch directly to gerrit as well (FIXME: special creds for doing this as a service account so we don't need to have the gerrit perms to shoot ourselves in the foot by default because pushing to main is bad?).

    FIXME: automate branch-off to release-* branch.

  • Manually (FIXME?) switch back to the release branch, which now has the correct revision.

  • Post!!

Installer

The installer is cross-built to several systems from a Mac using build-all.xsh and upload-to-lix.xsh in the installer repo (FIXME: currently at least; maybe this should be moved here?) .

It installs a binary tarball (FIXME: it should be taught to substitute from cache instead) from some URL; this is the hydraJobs.binaryTarball. The default URLs differ by architecture and are configured here.

Infrastructure summary

  • releases.lix.systems (s3://releases):
    • Each release gets a directory: https://releases.lix.systems/lix/lix-2.90-beta.1
      • Binary tarballs: nix-2.90.0-beta.1-x86_64-linux.tar.xz, from hydraJobs.binaryTarball
      • Manifest: manifest.nix, an attrset of the store paths by architecture.
    • Manifest for nix upgrade-nix to the latest release at /manifest.nix.
  • cache.lix.systems (s3://cache):
    • Receives all artifacts for released versions of Lix; is a plain HTTP binary cache.
  • install.lix.systems (s3://install):
    ~ » aws s3 ls s3://install/lix/
                               PRE lix-2.90-beta.0/
                               PRE lix-2.90-beta.1/
                               PRE lix-2.90.0pre20240411/
                               PRE lix-2.90.0pre20240412/
    2024-05-05 18:59:11    6707344 lix-installer-aarch64-darwin
    2024-05-05 18:59:16    7479768 lix-installer-aarch64-linux
    2024-05-05 18:59:14    7982208 lix-installer-x86_64-darwin
    2024-05-05 18:59:17    8978352 lix-installer-x86_64-linux
    
    ~ » aws s3 ls s3://install/lix/lix-2.90-beta.1/
    2024-05-05 18:59:01    6707344 lix-installer-aarch64-darwin
    2024-05-05 18:59:06    7479768 lix-installer-aarch64-linux
    2024-05-05 18:59:03    7982208 lix-installer-x86_64-darwin
    2024-05-05 18:59:07    8978352 lix-installer-x86_64-linux