Commit graph

4146 commits

Author SHA1 Message Date
Maximilian Bosch 9b62c52e5c hydra-queue-runner: drop broken connections from pool
Closes #1336

When restarting postgresql, the connections are still reused in
`hydra-queue-runner` causing errors like this

    main thread: Lost connection to the database server.
    queue monitor: Lost connection to the database server.

and no more builds being processed.

`hydra-evaluator` doesn't have that issue since it crashes right away.
We could let it retry indefinitely as well (see below), but I don't
want to change too much.

If the DB is still unreachable 10s later, the process will stop with a
non-zero exit code because of a missing DB connection. This however
isn't such a big deal because it will be immediately restarted
afterwards. With the current configuration, Hydra will never give up,
but restart (and retry) infinitely. To me that seems reasonable, i.e. to
retry DB connections on a long-running process. If this doesn't work
out, the monitoring should fire anyways because the queue fills up, but
I'm open to discuss that.

Please note that this isn't reproducible with the DB and the queue
runner on the same machine when using `services.hydra-dev`, because of
the `Requires=` dependency `hydra-queue-runner.service` ->
`hydra-init.service` -> `postgresql.service` that causes the queue
runner to be restarted on `systemctl restart postgresql`.

Internally, Hydra uses Nix's pool data structure: it basically has N
slots (here DB connections) and whenever a new one is requested, an idle
slot is provided or a new one is created (when N slots are active, it'll
be waited until one slot is free). The issue in the code here is however
that whenever an error is encountered, the slot is released, however the
same broken connection will be reused the next time. By using
`Pool::Handle::markBad`, Nix will drop a broken slot. This is now being
done when `pqxx::broken_connection` was caught.
2024-03-16 22:10:40 +01:00
Maximilian Bosch ef6be80f54 Use submit event in login form
It's a pet peeve from me when logging into my personal Hydra that I
always have to press the button rather than hitting Return after entering
my password.

Reason for that is that the form doesn't have a "submit" button, so far
it was always listened to the "click" event. Submit does that and you
can hit Return alternatively.
2024-03-16 22:10:40 +01:00
Ilya K 969eb3eeac urlencode drv names when fetching logs
Otherwise names with special characters like + break things.
2024-03-16 22:10:40 +01:00
Pierre Bourdon 18466e8326 queue-runner: try larger pipe buffer sizes 2024-03-16 22:10:40 +01:00
ajs124 6ed21490ee lazy-load evaluation errors
Closes #1362
2024-03-16 22:10:40 +01:00
John Ericson 838648c0ce
Merge pull request #1349 from NixOS/ca-no-new-col
Allow building content-addressed derivations with hydra, minimally
2024-01-26 17:54:02 -05:00
John Ericson 6ac4292912
Merge pull request #1351 from Ma27/hacking-fixes
Small fixes for the development environment
2024-01-26 17:22:42 -05:00
John Ericson b503280256 Add migration to drop non-null constraints 2024-01-26 11:53:58 -05:00
Maximilian Bosch b4c91b5a6a
package: move foreman to nativeCheckInputs
In 1bd195a513 strictDeps was set for the
Hydra package. As a result, `checkInputs` aren't available anymore in
the local dev-shell which is the sole purpose of foreman, to start
services and a database for development.
2024-01-26 17:30:07 +01:00
Maximilian Bosch 8477009310
doc/manual: fix instructions in contribution guidelines
In 5db374cb50 the `bootstrap` script was
removed, however it's still referenced in the contribution guidelines.
Change that to `autoreconfPhase` as intended by the commit.
2024-01-26 17:28:07 +01:00
John Ericson c62eaf248f Remove now-unneeded workaround 2024-01-26 01:20:07 -05:00
John Ericson 13b5f007ef Merge branch 'master' into ca-no-new-col 2024-01-26 01:19:45 -05:00
John Ericson 7f5889559e
Merge pull request #1350 from NixOS/remove-old-workaround
Remove now-unneeded workaround
2024-01-26 01:13:37 -05:00
John Ericson 5ee0e443e4 Remove now-unneeded workaround 2024-01-26 01:08:11 -05:00
John Ericson 323b556dc8 Minimal CA support
This verison has a worse UI, but also chnages the schema less: One
non-null constraint is removed, but no new columns are added.

Co-Authored-By: Andrea Ciceri <andrea.ciceri@autistici.org>
Co-Authored-By: regnat <rg@regnat.ovh>
2024-01-26 00:34:58 -05:00
John Ericson 458b9e4242
Merge pull request #1348 from NixOS/ca-prep
More CA derivations prep
2024-01-25 21:53:40 -05:00
John Ericson fcde5908d8 More CA derivations prep
Again, with care not to change the schema in any way.
2024-01-25 21:32:22 -05:00
John Ericson 083ef46c12
Merge pull request #1344 from delroth/google-popup
web: disable Sign in with Google popup
2024-01-25 16:36:16 -05:00
John Ericson 8a02bb7c36
Merge pull request #1347 from NixOS/simplify-req-features
Simplify `StoreConfig::getDefaultSystemFeatures` call
2024-01-25 16:17:25 -05:00
John Ericson c64eed7d07 Simplify StoreConfig::getDefaultSystemFeatures call
That method is now static.
2024-01-25 15:58:07 -05:00
John Ericson aed130cd17 flake.lock: Update
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/03e96b9dc011a16a0f6db9c7cb021ff93f8dcf88' (2024-01-19)
  → 'github:NixOS/nix/2c4bb93ba5a97e7078896ebc36385ce172960e4e' (2024-01-25)
2024-01-25 15:57:39 -05:00
John Ericson 7a6c401d42
Merge pull request #1346 from obsidiansystems/flake-reorg
Clean up the flake/build in a a few ways
2024-01-25 15:55:47 -05:00
John Ericson b5ed0787f7 Replace "not Perl" and "Perl again" with something more self-explanatory 2024-01-25 14:55:10 -05:00
John Ericson c5f37eca91 Reorganize hydra modules 2024-01-25 14:55:07 -05:00
John Ericson 73b6c1fb11 Filter out (mosts test) when !doCheck 2024-01-25 14:55:07 -05:00
John Ericson 4bbc7b8f75 Use the Nixpkgs fileset library to filter source
Now I can change Nix files without causing rebuilds.
2024-01-25 14:55:07 -05:00
John Ericson d6d6d1b649 flake.nix: Temporarily add a second Nixpkgs for lib.fileset
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/b38e5a665e9d0aa7975beb0ed12e42d13a392e74' (2023-12-13)
  → 'github:NixOS/nix/03e96b9dc011a16a0f6db9c7cb021ff93f8dcf88' (2024-01-19)
• Added input 'nixpkgs-for-fileset':
    'github:NixOS/nixpkgs/a77ab169a83a4175169d78684ddd2e54486ac651' (2024-01-24)
2024-01-25 14:55:07 -05:00
John Ericson 1bd195a513 Clean up deps
- `strictDeps`

- Ensure it builds with and without `doCheck`
2024-01-25 14:55:07 -05:00
John Ericson 1471aacadc Split out a package.nix
Just like we did with Nix.
2024-01-25 14:55:06 -05:00
John Ericson 62ddeb0ff0
Merge pull request #1345 from SuperSandro2000/patch-2
Remove automake, libtool
2024-01-25 14:47:07 -05:00
Sandro a876e46894
Remove automake, libtool
Those are already part of autoreconfHook
2024-01-25 17:12:40 +01:00
Pierre Bourdon 6df06b089e
web: disable Sign in with Google popup 2024-01-25 09:27:46 +01:00
John Ericson cc50fdff6f
Merge pull request #1343 from obsidiansystems/default-machine-file-features
Use `StoreConfig::getDefaultSystemFeatures` for default machine config
2024-01-24 21:44:32 -05:00
John Ericson b1fa6b3aac Use StoreConfig::getDefaultSystemFeatures for default machine config
We have to oddly make a `StoreConfig` subclass to get it, but
https://github.com/NixOS/nix/pull/9848 will fix that.

The purpose of this is to ensure that, absent an explicit config,
`localhost` includes `ca-derivations` and `recursive-nix` if those
experimental features are enabled.

Very much the complement of #1342, the previous PR.
2024-01-24 21:37:13 -05:00
John Ericson f6a2b7562a
Merge pull request #1342 from obsidiansystems/dedup-required-system-features
Use `nix::ParsedDerivation::getRequiredSystemFeatures()`
2024-01-24 21:13:49 -05:00
John Ericson 07cb5d1b7c Use nix::ParsedDerivation::getRequiredSystemFeatures()
A slight dedup, and also ensures that floating CA derivations require a
`ca-derivations` experimental feature. This fixes the scheduling issue
that @SuperSandro2000 found.
2024-01-24 21:04:14 -05:00
John Ericson d45e14fd43
Merge pull request #1316 from NixOS/ca-derivations-prep
Prepare for CA derivation support with lower impact changes
2024-01-24 18:12:42 -05:00
John Ericson d02e20a4c1
Merge pull request #1341 from NixOS/machine-dedup
Use Nix's `Machine` type in a minimal way
2024-01-23 15:38:19 -05:00
John Ericson 70e5469303 Use Nix's Machine type in a mimimal way
This is *just* using the fields from that type, and only where the types
coincide. (There are two fields with different types, `speedFactor` most
interestingly.) No code is reused, so we can be sure that no behavior is
changed.

Once the types are reconciled on the Nix side, then we can start
carefully actually reusing code.

Progress on #1164
2024-01-23 12:18:57 -05:00
John Ericson 2e6ee28f9b Machine -> ::Machine so we don't conflict with Nix's 2024-01-23 11:03:19 -05:00
John Ericson f5c0efb11e
Merge pull request #1340 from NixOS/start-using-nix-ssh
Replace `Child` with `SSHMaster::Connection`
2024-01-23 01:17:26 -05:00
John Ericson 4e8fbaa3d6 Replace Child with SSHMaster::Connection
Nix defines basically an identical struct for the same purpose, so let's
just use that.
2024-01-23 01:11:46 -05:00
John Ericson 588a0c5269 Merge remote-tracking branch 'upstream/master' into ca-derivations-prep 2023-12-23 19:19:54 -05:00
John Ericson 02e453fc8c
Merge pull request #1329 from NixOS/small-std-optional-cleanup
Clean up `std::optional` dereferencing in the queue runner
2023-12-23 19:18:41 -05:00
John Ericson 75f26f1fc4 Clean up std::optional dereferencing in the queue runner
Instead of doing this partial operation a number of times, assert (with
a comment, get a reference to the thing inside, and use that just once.
(This refactor was done twice, "just once" for each time.)
2023-12-23 19:10:58 -05:00
John Ericson 3c89067f52
Merge pull request #1328 from JackKelly-Bellroy/doc-store-uri
Document the `store_uri` parameter by way of example
2023-12-23 03:57:10 -05:00
Jack Kelly abd858d3dc Document the store_uri parameter by way of example 2023-12-19 07:54:40 +10:00
John Ericson 163dbf7f54
Merge pull request #1327 from NixOS/latest-2.19
flake.lock: Update Nix
2023-12-14 00:57:19 -05:00
John Ericson 642156372f Merge branch 'latest-2.19' into ca-derivations-prep 2023-12-14 00:33:22 -05:00
John Ericson 7517c134c5 flake.lock: Update Nix
Newer 2.19-maintenance has some `restricted-eval` fixes that benefit
Hydra.

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/50f8f1c8bc019a4c0fd098b9ac674b94cfc6af0d' (2023-12-11)
  → 'github:NixOS/nix/b38e5a665e9d0aa7975beb0ed12e42d13a392e74' (2023-12-13)
2023-12-14 00:32:15 -05:00