Commit graph

2362 commits

Author SHA1 Message Date
Bas van Dijk
38122544ed GitInput: only convert integer option values to int
The previous code converted option values to ints when the value
contained a digit somewhere. This is too eager since it also converts
strings like `release-0.2` to an int which should not happen.

We now only convert to int when the value is an integer.
2020-05-13 11:41:52 +02:00
Eelco Dolstra
15a45f1a8a
Fix build 2020-05-12 16:14:20 +02:00
Eelco Dolstra
e379628db0
hydra-eval-jobset: Pass --no-allow-import-from-derivation
https://github.com/NixOS/nixpkgs/issues/87592
2020-05-12 15:17:50 +02:00
Bas van Dijk
f32a2a48d7
Merge pull request #740 from knl/add-githubrefs-plugin
Add GithubRefs plugin
2020-05-08 13:21:45 +02:00
Eelco Dolstra
96a514c169
Remove the "releases" feature
We haven't used this in many years (it was really only used for nix
and patchelf releases).
2020-05-06 12:39:21 +02:00
Ben Wolsieffer
f020f7efef
hydra-queue-runner: don't try to distribute builds on localhost 2020-05-03 00:05:52 -04:00
Emery Hemingway
e93c36aab1 SoTest: read credentials from file 2020-04-26 12:12:04 +05:30
Nikola Knezevic
f03e7ef800 Add GithubRefs plugin
This plugin is a counterpart to GithubPulls plugin. Instead of fetching pull
requests, it will fetch all references (branches and tags) that start with a
particular prefix.

The plugin is a copy of GithubPulls plugin with appropriate changes to call the
right API and parse the config matching the need.
2020-04-23 10:45:37 +02:00
Emery Hemingway
a63e349476 Add SoTest plugin
https://opensource.sotest.io/
https://docs.sotest.io/
2020-04-21 15:25:44 +05:30
721c764951
Remove Hydra::Helper::nix::txn_do from the Perl code
To quote the function's comment:

  Awful hack to handle timeouts in SQLite: just retry the transaction.
  DBD::SQLite *has* a 30 second retry window, but apparently it
  doesn't work.

Since SQLite is now dropped entirely, this wrapper can be removed
completely.
2020-04-16 00:42:40 +02:00
efcbc08686
Get rid of dependency to SQLite
SQLite isn't properly supported by Hydra for a few years now[1], but
Hydra still depends on it. Apart from a slightly bigger closure this can
cause confusion by users since Hydra picks up SQLite rather than
PostgreSQL by default if HYDRA_DBI isn't configured properly[2]

[1] 78974abb69
[2] https://logs.nix.samueldr.com/nixos-dev/2020-04-10#3297342;
2020-04-16 00:42:40 +02:00
Graham Christensen
7b705758ec
Merge pull request #732 from Ma27/fix-build
Fix build against latest Nix
2020-04-09 09:02:45 -04:00
Bas van Dijk
6e358189ad Separate the build IDs in the build_finished payload with tabs
hydra-notify splits the payload on tabs so we shouldn't separate the
IDs with spaces.
2020-04-08 12:05:25 +02:00
2f9d422172
Fix build against latest Nix 2020-04-07 13:55:38 +02:00
Eelco Dolstra
4cabb37ebd
Merge pull request #730 from NixOS/flake
Flake support
2020-04-07 11:18:38 +02:00
Kevin Quick
a055796ef5
Merge branch 'master' into libpqxx_undeprecate 2020-04-01 11:54:41 -07:00
Eelco Dolstra
2d092a6fbc
Merge pull request #702 from kquick/fix_api_push
Handle case where jobset has no defined errormsg for api/jobsets
2020-04-01 13:09:05 +02:00
Eelco Dolstra
8564089186
Merge pull request #707 from basvandijk/fix-aggregate-status
Fix printing aggregate status
2020-04-01 13:04:53 +02:00
Eelco Dolstra
6c0bb909df
Merge pull request #704 from basvandijk/sleep-after-catch-exception
hydra-queue-runner: sleep 5s after handling an exception
2020-04-01 13:04:31 +02:00
Eelco Dolstra
9668bc01da
Merge pull request #726 from knl/document-slack-notifications-plugin
Document SlackNotifications plugin
2020-04-01 13:04:00 +02:00
Eelco Dolstra
9727892b61 Don't spam the journal with hydra-queue-runner status dumps
(cherry picked from commit 15ae932488)
2020-03-31 22:19:07 +02:00
Eelco Dolstra
ccd046ca3d Keep track of the number of unsupported steps
(cherry picked from commit 45ffe578b6)
2020-03-31 22:19:03 +02:00
Eelco Dolstra
4417f9f260 Abort unsupported build steps
If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.

(cherry picked from commit f5cdbfe21d)
2020-03-31 22:19:01 +02:00
Eelco Dolstra
68a59f34a0 Remove tabs
(cherry picked from commit 5b731004da)
2020-03-31 22:18:58 +02:00
7889aef792
hydra-eval-jobs: Mirror eval errors in STDERR
Otherwise, errors will not be shown to end-users, which makes debugging
long evals pretty much impossible.

(cherry picked from commit 76299b9174)
2020-03-31 12:48:33 +02:00
76299b9174 hydra-eval-jobs: Mirror eval errors in STDERR
Otherwise, errors will not be shown to end-users, which makes debugging
long evals pretty much impossible.
2020-03-30 16:11:44 -04:00
Eelco Dolstra
15ae932488
Don't spam the journal with hydra-queue-runner status dumps 2020-03-26 15:30:37 +01:00
Eelco Dolstra
45ffe578b6
Keep track of the number of unsupported steps 2020-03-26 15:27:13 +01:00
Eelco Dolstra
f5cdbfe21d
Abort unsupported build steps
If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.
2020-03-26 15:00:04 +01:00
Eelco Dolstra
5b731004da
Remove tabs 2020-03-26 14:59:52 +01:00
Nikola Knezevic
1bee6e3d8a Add debug logging
This will help us track potential problems with the plugin.
2020-03-26 13:27:44 +01:00
Nikola Knezevic
986fde8888 Refactor code
Extract the conditions before the loop, as they do not change due to channel
definition.
2020-03-26 13:27:44 +01:00
Nikola Knezevic
956f009672 Add documentation for SlackNotification plugin 2020-03-26 13:27:44 +01:00
Eelco Dolstra
d0688b93e1
Merge remote-tracking branch 'origin/master' into flake 2020-03-26 11:44:11 +01:00
Graham Christensen
12cc46cdb3
fixup: hydra-init: correct reference to hydra-backill-ids 2020-03-24 11:22:14 -04:00
73694087a0
Use the stopsign as icon for timeouts and exceeded log-limits
When I browse failed builds in a jobset-eval on Hydra, I regularly
mistake actual build-failures with temporary issues like timeouts (that
probably disappear at the next eval).

To prevent this kind of issue, I figured that using the stopsign-svg for
builds with timeouts or exceeded log-limits is a reasonable choice for
the following reasons:

* A user can now distinguish between actual build-errors (like
  compilation-failures or oversized outputs) and (usually) temporary issues
  (like a bloated log or a timeout).

* The stopsign is also used for aborted jobs that are shown in a
  different tab and can't be confused with timeouts for that reason.
2020-03-21 21:08:33 +01:00
Nikola Knezevic
01ae944c80 Add host tag to InfluxDB metrics
This should help us discern machines in environments with multiple Hydra deployments.
2020-03-12 14:23:07 +01:00
Eelco Dolstra
4b5bb4e760
Merge remote-tracking branch 'origin/master' into flake 2020-03-04 15:28:23 +01:00
Eelco Dolstra
123bee1db5
Restore job type checking 2020-03-04 15:16:26 +01:00
Eelco Dolstra
69a6f3448a
Fix calling job functions
Fixes #718.
2020-03-04 15:16:04 +01:00
Eelco Dolstra
3cc1deb125
Merge pull request #721 from grahamc/one-by-one
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
2020-03-04 08:44:43 +01:00
Graham Christensen
994430b94b
treewide: allow nix command 2020-03-03 22:52:20 -05:00
Graham Christensen
117b9ecef1
Nix.pm: readNixFile: pass «--experimental-features nix-command»
Declarative jobsets were broken by the Nix update, causing
nix cat-file to break silently.

This commit restores declarative jobsets, based on top of a commit
making it easier to see what broke.
2020-03-03 22:36:21 -05:00
Graham Christensen
113a312f67
handleDeclarativeJobsetBuild: handle errors from readNixFile 2020-03-03 22:32:13 -05:00
Graham Christensen
5fae9d96a2
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
In the past, jobsets which are automatically evaluated are evaluated
regularly, on a schedule. This schedule means a new evaluation is
created every checkInterval seconds (assuming something changed.)

This model works well for architectures where our build farm can
easily keep up with demand.

This commit adds a new type of evaluation, called ONE_AT_A_TIME, which
only schedules a new evaluation if the previous evaluation of the
jobset has no unfinished builds.

This model of evaluation lets us have 'low-tier' architectures.

For example, we could now have a jobset for ARMv7l builds, where
the buildfarm only has a single, underpowered ARMv7l builder.
Configuring that jobset as ONE_AT_A_TIME will create an evaluation
and then won't schedule another evaluation until every job of
the existing evaluation is complete.

This way, the cache will have a complete collection of pre-built
software for some commits, but the underpowered architecture will
never become backlogged in ancient revisions.
2020-03-03 19:28:44 -05:00
Graham Christensen
eaa65f51f4
hydra-evaluator: make the logic of the scheduler easier to read 2020-03-03 18:17:21 -05:00
Eelco Dolstra
eb5873ae53
Fix build 2020-02-20 11:19:45 +01:00
Eelco Dolstra
c642f787ee
hydra-eval-jobs: Parallelize
(cherry picked from commit be8eb9d00d)
2020-02-20 10:28:58 +01:00
Eelco Dolstra
adf61e5cf8
Fix build
(cherry picked from commit 639c660abf)
2020-02-20 10:26:45 +01:00
Eelco Dolstra
e4f5156c41
Build against nix-master
(cherry picked from commit e7f2139e25)
2020-02-20 10:24:04 +01:00
Eelco Dolstra
b790a00729
Disable deprecation warnings
(cherry picked from commit 950e8bef6c)
2020-02-20 10:22:21 +01:00
Eelco Dolstra
2a50daa377
Update aggregate handling
(cherry picked from commit cf961ac893)
2020-02-20 10:13:39 +01:00
Eelco Dolstra
15187b059b
Remove hydra-eval-guile-jobs
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.

(cherry picked from commit 23c9ca3e94)
2020-02-20 09:58:12 +01:00
Eelco Dolstra
be8eb9d00d
hydra-eval-jobs: Parallelize 2020-02-19 21:10:22 +01:00
Eelco Dolstra
6f1d68bda4
Revert "hydra-eval-jobs -> nix eval-hydra-jobs"
This reverts commit 345512a6d0.
2020-02-19 20:36:52 +01:00
Eelco Dolstra
cf961ac893
Update aggregate handling 2020-02-17 16:33:25 +01:00
Eelco Dolstra
68bdf834ee
Fix hydra-evaluator --unlock 2020-02-15 16:40:16 +01:00
Eelco Dolstra
345512a6d0
hydra-eval-jobs -> nix eval-hydra-jobs 2020-02-15 15:59:34 +01:00
Eelco Dolstra
23c9ca3e94
Remove hydra-eval-guile-jobs
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.
2020-02-15 15:59:34 +01:00
Eelco Dolstra
639c660abf
Fix build 2020-02-15 14:54:35 +01:00
Graham Christensen
ae1b9face3
Merge remote-tracking branch 'origin/master' into flake 2020-02-11 12:52:46 -05:00
Graham Christensen
027668f0db
hydra.sql: add an index for slow queries in production
These queries used to use (jobset, project) based indexes,
and the addition of jobset_id makes most of those indexes
unusable now.
2020-02-11 12:52:28 -05:00
Graham Christensen
53896ff19b
Merge remote-tracking branch 'origin/master' into flake 2020-02-11 08:36:40 -05:00
Graham Christensen
834793468f
fixup: d'oh, make the migrations from #710 part-2 sequential 2020-02-11 08:36:14 -05:00
Eelco Dolstra
881b7449fd Merge remote-tracking branch 'origin/master' into flake 2020-02-11 14:23:16 +01:00
Graham Christensen
2637a7ad76
Builds: index literally what latest-finished queries 2020-02-11 07:06:21 -05:00
Graham Christensen
f0f41eaaff
LatestSucceededForJob{,set}: Filter with jobset_id 2020-02-11 07:06:20 -05:00
Graham Christensen
66fbbd9692
Jobs.builds: Fetch via Jobs.jobset_id 2020-02-11 07:06:20 -05:00
Graham Christensen
7c71f9df28
Jobsets.builds: Fetch via Jobsets.id 2020-02-11 07:06:20 -05:00
Graham Christensen
3c392b8cd8
Jobsets.jobs: Fetch via Jobsets.id 2020-02-11 07:06:20 -05:00
Graham Christensen
8ef08f1385
Builds.jobset_id: make not-null 2020-02-11 07:06:20 -05:00
Graham Christensen
2cdcc7f188
Jobs.jobset_id: make not-null 2020-02-11 07:06:17 -05:00
Eelco Dolstra
100e09a5b3
Merge remote-tracking branch 'origin/master' into flake
Also update flake.lock
2020-02-10 17:58:10 +01:00
Graham Christensen
c4cc72f944
hydra-init: Warn about the schema version migration 2020-02-10 11:43:03 -05:00
Graham Christensen
f69260118b
hydra-backfill-ids: create to add jobset_id values to Builds and Jobs
Vacuum every 10 iterations, update 10k at a time.
2020-02-10 11:43:03 -05:00
Graham Christensen
f3a561aecd
Builds: populate Builds.jobset_id in hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Graham Christensen
ddf00fa627
Builds: add a nullable jobset_id foreign key to Jobsets.
Also, adds an explicitly named "builds" accessor to the Jobsets
Schema object, which uses the project/jobset name.
2020-02-10 11:43:02 -05:00
Graham Christensen
624f1d8d2d
Jobs: populate Jobs.jobset_id field when writing from hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Graham Christensen
efa1f1d4fb
Jobs: add a nullable jobset_id foreign key to Jobsets.
Also, adds an explicitly named "jobs" accessor to the Jobsets
Schema object, which uses the project/jobset name.
2020-02-10 11:43:02 -05:00
Graham Christensen
e00030563b
Jobsets: add a SERIAL, unique, non-null id column
A postgresql column which is non-null and unique is treated with
the same optimisations as a primary key, so we have no need to
try and recreate the `id` as the primary key.

No read paths are impacted by this change, and the database will
automatically create an ID for each insert. Thus, no code needs to
change.
2020-02-10 11:42:59 -05:00
Graham Christensen
6fe57ab5fa
Copy the flake migration from the flake branch
hydra.nixos.org is already running this rev, and it should be safe to
apply to everyone else. If we make changes to this migration, we'll
need to write another migration anyway.
2020-02-09 15:21:28 -05:00
Graham Christensen
c2f932a7e3
sql: Generate models from postgresql
Lowercasing is due to postgresql not having case-sensitive table names.
It always technically workde before, but those table names never
existed literally.

The switch to generating from postgresql is to handle an upcoming
addition of an auto-incrementign ID to the Jobset table. Sqlite doesn't
seem to be able to handle the table having an auto incrementing ID
field which isn't the primary key, but we can't change the primary
key trivially.

Since hydra doesn't support sqlite and hasn't for many year anyway,
it is easier to just generate from pgsql directly.
2020-02-06 12:23:47 -05:00
Eelco Dolstra
ba8814a245 Fix build 2020-02-03 18:43:45 +01:00
Eelco Dolstra
bb94677526 Merge remote-tracking branch 'origin/master' into flake 2020-02-03 17:49:01 +01:00
Bas van Dijk
5fab5e935b Remove the "log diff" buttons
because they're referencing the removed `logdiff` API.

This API was removed in 4d1816b152.

Fixes #409
2020-01-26 21:45:38 +01:00
Bas van Dijk
bd9b656c54 Fix printing aggregate status
`nrMembers` is undefined and it should have clearly be `nrConstituents`
which is calculated just before.

Fixes #693.
2020-01-26 20:15:18 +01:00
Kevin Quick
cdd9d6e071
Update haserrormsg logic implementation. 2020-01-20 10:40:33 -08:00
Bas van Dijk
08bfff9d13 hydra-queue-runner: sleep 5s after handling an exception
instead of immediately calling `readMachinesFiles` again which could
immediately throw another exception again.
2020-01-14 13:34:35 +01:00
Kevin Quick
ec8b2970e0
Remove added strictness to allow multiple query returns. 2020-01-12 10:18:52 -08:00
Kevin Quick
7bb6b5e206
Update libpqxx usage to move away from deprecated API interactions. 2020-01-11 22:38:40 -08:00
Kevin Quick
de24771a8e
Handle case where jobset has no defined errormsg for api/jobsets 2020-01-11 17:01:44 -08:00
Hamish Mackenzie
c40c887e50
Fixes for macOS
Building on macOS with the latest nixpkgs master and NixOS/nixpkgs#77147
fails.  It seems some `std::experimental` (optional) for instance are
not available as `experimental`, but are in `std`.  Also `toJSON` is
missing for `atomic< unsigned long long >`.
2020-01-07 12:38:06 +13:00
Eelco Dolstra
e7f2139e25 Build against nix-master 2019-12-30 22:49:26 +01:00
Graham Christensen
5cac40a438
Merge remote-tracking branch 'origin/master' into flake 2019-12-29 16:37:25 -05:00
Graham Christensen
d0f1bda0b8
job prometheus endpoint: drop nixname, too variable 2019-12-29 16:37:13 -05:00
Graham Christensen
d24de1b5de
Merge remote-tracking branch 'origin/master' into flake 2019-12-28 20:58:03 -05:00
Graham Christensen
64cdc3413c
job prometheus endpoint: d'oh 2019-12-28 20:57:27 -05:00
Graham Christensen
511c2db8aa
Merge remote-tracking branch 'origin/master' into flake 2019-12-28 20:22:54 -05:00
Graham Christensen
d5445bfc1d
job: create a prometheus endpoint
Export the most recent stop time and exit status in
a prometheus-friendly format.
2019-12-28 16:41:49 -05:00
Nikola Knezevic
06abfd6b2f hydra-send-stats: Cleanup removed metrics
In 2946899504 these metrics got removed
due to refactoring of how notifications work.
2019-11-13 11:42:58 +01:00