Hydra, for Lix
Find a file
Maximilian Bosch 9b62c52e5c hydra-queue-runner: drop broken connections from pool
Closes #1336

When restarting postgresql, the connections are still reused in
`hydra-queue-runner` causing errors like this

    main thread: Lost connection to the database server.
    queue monitor: Lost connection to the database server.

and no more builds being processed.

`hydra-evaluator` doesn't have that issue since it crashes right away.
We could let it retry indefinitely as well (see below), but I don't
want to change too much.

If the DB is still unreachable 10s later, the process will stop with a
non-zero exit code because of a missing DB connection. This however
isn't such a big deal because it will be immediately restarted
afterwards. With the current configuration, Hydra will never give up,
but restart (and retry) infinitely. To me that seems reasonable, i.e. to
retry DB connections on a long-running process. If this doesn't work
out, the monitoring should fire anyways because the queue fills up, but
I'm open to discuss that.

Please note that this isn't reproducible with the DB and the queue
runner on the same machine when using `services.hydra-dev`, because of
the `Requires=` dependency `hydra-queue-runner.service` ->
`hydra-init.service` -> `postgresql.service` that causes the queue
runner to be restarted on `systemctl restart postgresql`.

Internally, Hydra uses Nix's pool data structure: it basically has N
slots (here DB connections) and whenever a new one is requested, an idle
slot is provided or a new one is created (when N slots are active, it'll
be waited until one slot is free). The issue in the code here is however
that whenever an error is encountered, the slot is released, however the
same broken connection will be reused the next time. By using
`Pool::Handle::markBad`, Nix will drop a broken slot. This is now being
done when `pqxx::broken_connection` was caught.
2024-03-16 22:10:40 +01:00
.github test: use ubuntu-latest 2023-03-06 07:56:05 -08:00
datadog
doc Merge pull request #1349 from NixOS/ca-no-new-col 2024-01-26 17:54:02 -05:00
examples Extend Setup Information 2020-05-02 16:04:20 +02:00
foreman foreman: run the dev server with --restart and --debug 2021-12-16 10:20:25 -05:00
nixos-modules Reorganize hydra modules 2024-01-25 14:55:07 -05:00
src hydra-queue-runner: drop broken connections from pool 2024-03-16 22:10:40 +01:00
t lazy-load evaluation errors 2024-03-16 22:10:40 +01:00
.editorconfig Initialize a basic editorconfig 2021-08-06 14:59:40 -04:00
.gitignore Reorganize hydra modules 2024-01-25 14:55:07 -05:00
.perlcriticrc perlcritic: level 1 2021-12-14 10:24:34 -05:00
.yath.rc tests: move to t, allow yath test from root 2021-03-05 09:49:06 -08:00
configure.ac Filter out (mosts test) when !doCheck 2024-01-25 14:55:07 -05:00
COPYING
default.nix Simplify default.nix and shell.nix 2020-06-17 19:19:55 +02:00
flake.lock flake.lock: Update 2024-01-25 15:57:39 -05:00
flake.nix Reorganize hydra modules 2024-01-25 14:55:07 -05:00
hydra-api.yaml hydra-api.yaml: document all_builds (/eval/{eval-id}/builds) 2023-08-30 15:08:11 +02:00
INSTALL
Makefile.am Reorganize hydra modules 2024-01-25 14:55:07 -05:00
package.nix package: move foreman to nativeCheckInputs 2024-01-26 17:30:07 +01:00
Procfile Procfile: sort alphabetically 2021-04-05 16:10:09 +00:00
README.md Cleanup deps 2023-11-30 10:48:17 -05:00
shell.nix Remove yet another URL literal 2022-07-10 13:31:21 +02:00
version.txt Rename version to version.txt 2021-07-05 19:47:58 +01:00

Hydra

CI

Hydra is a Continuous Integration service for Nix based projects.

Installation And Setup

Note: The instructions provided below are intended to enable new users to get a simple, local installation up and running. They are by no means sufficient for running a production server, let alone a public instance.

Enabling The Service

Running Hydra is currently only supported on NixOS. The hydra module allows for an easy setup. The following configuration can be used for a simple setup that performs all builds on localhost (Please refer to the Options page for all available options):

{
  services.hydra = {
    enable = true;
    hydraURL = "http://localhost:3000";
    notificationSender = "hydra@localhost";
    buildMachinesFiles = [];
    useSubstitutes = true;
  };
}

Creating An Admin User

Once the Hydra service has been configured as above and activate you should already be able to access the UI interface at the specified URL. However some actions require an admin user which has to be created first:

$ su - hydra
$ hydra-create-user <USER> --full-name '<NAME>' \
    --email-address '<EMAIL>' --password-prompt --role admin

Afterwards you should be able to log by clicking on "Sign In" on the top right of the web interface using the credentials specified by hydra-create-user. Once you are logged in you can click "Admin -> Create Project" to configure your first project.

Creating A Simple Project And Jobset

In order to evaluate and build anything you need to create projects that contain jobsets. Hydra supports imperative and declarative projects and many different configurations. The steps below will guide you through the required steps to creating a minimal imperative project configuration.

Creating A Project

Log in as administrator, click "Admin" and select "Create project". Fill the form as follows:

  • Identifier: hello
  • Display name: hello
  • Description: hello project

Click "Create project".

Creating A Jobset

After creating a project you are forwarded to the project page. Click "Actions" and choose "Create jobset". Fill the form with the following values:

  • Identifier: hello
  • Nix expression: examples/hello.nix in hydra
  • Check interval: 60
  • Scheduling shares: 1

We have to add two inputs for this jobset. One for nixpkgs and one for hydra (which we are referencing in the Nix expression above):

  • Input name: nixpkgs

  • Type: Git checkout

  • Value: https://github.com/nixos/nixpkgs-channels nixos-20.03

  • Input name: hydra

  • Type: Git checkout

  • Value: https://github.com/nixos/hydra

Make sure State at the top of the page is set to "Enabled" and click on "Create jobset". This concludes the creation of a jobset that evaluates ./examples/hello.nix once a minute. Clicking "Evaluations" should list the first evaluation of the newly created jobset after a brief delay.

Building And Developing

Building Hydra

You can build Hydra via nix-build using the provided default.nix:

$ nix-build

Development Environment

You can use the provided shell.nix to get a working development environment:

$ nix-shell
$ autoreconfPhase
$ configurePhase # NOTE: not ./configure
$ make

Executing Hydra During Development

When working on new features or bug fixes you need to be able to run Hydra from your working copy. This can be done using foreman:

$ nix-shell
$ # hack hack
$ make
$ foreman start

Have a look at the Procfile if you want to see how the processes are being started. In order to avoid conflicts with services that might be running on your host, hydra and postgress are started on custom ports:

  • hydra-server: 63333 with the username "alice" and the password "foobar"
  • postgresql: 64444

Note that this is only ever meant as an ad-hoc way of executing Hydra during development. Please make use of the NixOS module for actually running Hydra in production.

Checking your patches

After making your changes, verify the test suite passes and perlcritic is still happy.

Start by following the steps in Development Environment.

Then, you can run the tests and the perlcritic linter together with:

$ nix-shell
$ make check

You can run a single test with:

$ nix-shell
$ yath test ./t/foo/bar.t

And you can run just perlcritic with:

$ nix-shell
$ make perlcritic

JSON API

You can also interface with Hydra through a JSON API. The API is defined in hydra-api.yaml and you can test and explore via the swagger editor

Additional Resources

License

Hydra is licensed under GPL-3.0

Icons provided free by EmojiOne.