lix/releng
jade 98d0249d5c releng: fix broken manifest from 2.90-rc1
Well that is embarrassing. I think the proper thing is to just quickly
ship a -rc2, so I will open a backport of this.

    $ curl https://releases.lix.systems/manifest.nix
    # This file was generated by releng/create_release.xsh in Lix
    {
        aarch64-linux = "/nix/store/mrbknq000af7iaqhk53bnpk1fvfrc1xp-lix-2.90.0-rc1";
      aarch64-darwin = "/nix/store/z1bdccwsk34iv491aygh0mm1lgpf7yy1-lix-2.90.0-rc1";
      x86_64-darwin = "/nix/store/xqvfpdhzck44v6kyhgi9f8v0xybksb6a-lix-2.90.0-rc1";
      x86_64-linux = "/nix/store/h2ml0nx4477r84y82jgm8y80jpr72gqw-lix-release-tarballs";
    }

Change-Id: I9cf007c850c2faf995a3a9d92457517b8501d1a1
2024-06-16 19:06:45 -07:00
..
__init__.py releng: automatically add to the summary page 2024-06-15 18:46:18 -07:00
__main__.py Put into place initial release engineering 2024-06-06 20:53:08 -07:00
cli.py releng: support multiple systems 2024-06-13 14:36:03 -07:00
create_release.xsh releng: fix broken manifest from 2.90-rc1 2024-06-16 19:06:45 -07:00
docker.xsh releng: fix upload of multiarch images to forgejo 2024-06-13 17:12:45 -07:00
docker_assemble.py releng: fix upload of multiarch images to forgejo 2024-06-13 17:12:45 -07:00
environment.py releng: add prod environment, ready for release 2024-06-09 20:33:24 -07:00
gitutils.xsh releng: fix git checking 2024-06-13 15:24:21 -07:00
keys.py releng: support pushing the manual to docs also 2024-06-06 20:53:08 -07:00
README.md Put into place initial release engineering 2024-06-06 20:53:08 -07:00
release-jobs.nix releng: add sha256 for the manual tarball 2024-06-13 17:14:06 -07:00
release_notes.py releng: automatically add to the summary page 2024-06-15 18:46:18 -07:00
version.py Put into place initial release engineering 2024-06-06 20:53:08 -07:00

Release engineering

This directory contains the release engineering scripts for Lix.

Release process

Prerequisites

  • FIXME: validation via misc tests in nixpkgs, among other things? What other validation do we need before we can actually release?
  • Have a release post ready to go as a PR on the website repo.
  • No release-blocker bugs.

Process

The following process can be done either against the staging environment or the live environment.

For staging, the buckets are staging-releases, staging-cache, etc.

FIXME: obtainment of signing key for signing cache paths?

First, we prepare the release. python -m releng prepare is used for this.

  • Gather everything in doc/manual/rl-next and put it in doc/manual/src/release-notes/rl-MAJOR.md.

Then we tag the release with python -m releng tag:

  • Git HEAD is detached.
  • officialRelease = true is set in flake.nix, this is committed, and a release is tagged.
  • The tag is merged back into the last branch (either main for new releases or release-MAJOR for maintenance releases) with git merge -s ours VERSION creating a history link but ignoring the tree of the release tag.
  • Git HEAD is once again detached onto the release tag.

Then, we build the release artifacts with python -m releng build:

  • Source tarball is generated with git archive, then checksummed.
  • Manifest for nix upgrade-nix is produced and put in s3://releases at /manifest.nix and /lix/lix-VERSION.
  • Release is built: hydraJobs.binaryTarball jobs are built, and joined into a derivation that depends on all of them and adds checksum files. This and the sources go into s3://releases/lix/lix-VERSION.

At this point we have a release/artifacts and release/manual directory which are ready to publish, and tags ready for publication. No keys are required to do this part.

Next, we do the publication with python -m releng upload:

  • Artifacts for this release are uploaded:

    • s3://releases/manifest.nix, changing the default version of Lix for nix upgrade-nix.
    • s3://releases/lix/lix-VERSION/ gets the following contents
      • Binary tarballs
      • Docs: manual/ (FIXME: should we actually do this? what about putting it on docs.lix.systems? I think doing both is correct, since the Web site should not be an archive of random old manuals)
      • Docs as tarball in addition to web.
      • Source tarball
      • Docker image (FIXME: upload to forgejo registry and github registry in the future)
    • s3://docs/manual/lix/MAJOR
    • s3://docs/manual/lix/stable
  • The tag is uploaded to the remote repo.

  • Manually build the installer using the scripts in the installer repo and upload.

    FIXME: This currently requires a local Apple Macintosh® aarch64 computer, but we could possibly automate doing it from the aarch64-darwin builder.

  • Manually Push the main/release branch directly to gerrit.

  • If this is a new major release, branch-off to release-MAJOR and push that branch directly to gerrit as well (FIXME: special creds for doing this as a service account so we don't need to have the gerrit perms to shoot ourselves in the foot by default because pushing to main is bad?).

    FIXME: automate branch-off to release-* branch.

  • Manually (FIXME?) switch back to the release branch, which now has the correct revision.

  • Post!!

Installer

The installer is cross-built to several systems from a Mac using build-all.xsh and upload-to-lix.xsh in the installer repo (FIXME: currently at least; maybe this should be moved here?) .

It installs a binary tarball (FIXME: it should be taught to substitute from cache instead) from some URL; this is the hydraJobs.binaryTarball. The default URLs differ by architecture and are configured here.

Infrastructure summary

  • releases.lix.systems (s3://releases):
    • Each release gets a directory: https://releases.lix.systems/lix/lix-2.90-beta.1
      • Binary tarballs: nix-2.90.0-beta.1-x86_64-linux.tar.xz, from hydraJobs.binaryTarball
      • Manifest: manifest.nix, an attrset of the store paths by architecture.
    • Manifest for nix upgrade-nix to the latest release at /manifest.nix.
  • cache.lix.systems (s3://cache):
    • Receives all artifacts for released versions of Lix; is a plain HTTP binary cache.
  • install.lix.systems (s3://install):
    ~ » aws s3 ls s3://install/lix/
                               PRE lix-2.90-beta.0/
                               PRE lix-2.90-beta.1/
                               PRE lix-2.90.0pre20240411/
                               PRE lix-2.90.0pre20240412/
    2024-05-05 18:59:11    6707344 lix-installer-aarch64-darwin
    2024-05-05 18:59:16    7479768 lix-installer-aarch64-linux
    2024-05-05 18:59:14    7982208 lix-installer-x86_64-darwin
    2024-05-05 18:59:17    8978352 lix-installer-x86_64-linux
    
    ~ » aws s3 ls s3://install/lix/lix-2.90-beta.1/
    2024-05-05 18:59:01    6707344 lix-installer-aarch64-darwin
    2024-05-05 18:59:06    7479768 lix-installer-aarch64-linux
    2024-05-05 18:59:03    7982208 lix-installer-x86_64-darwin
    2024-05-05 18:59:07    8978352 lix-installer-x86_64-linux