Commit graph

2314 commits

Author SHA1 Message Date
Nikola Knezevic e9922c460e Add missing SQL upgrade script for NOT NULL on type
`type` column in `Jobsets` is defined as NOT NULL. However, the original upgrade
script adding this column ommited the constraint.
2020-05-18 10:59:55 +02:00
Nikola Knezevic 575113396d Handle missing values in declarative jobsets
The current implementation will pass all values to `create_or_update` method. The
missing values will end up as `undef` (or `NULL`) when assigned to `%update`.
Thus, for columns that are NOT NULL, when, for example, flakes are not used,
will result in a horrible:

    DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed:
    ERROR:  null value in column "type" violates not-null constraint

    DETAIL:  Failing row contains (.jobsets, 118, hydra, hydra jobsets, src, hydra/jobsets.nix, null,
    null, null, 1589536378, 1, 0, 0, , 3, 30, 100, null, null, 1589536379, null, null). [for Statement
    "UPDATE jobsets SET checkinterval = ?, description = ?, enableemail = ?, nixexprinput = ?,
    nixexprpath = ?, type = ? WHERE ( ( name = ? AND project = ? ) )" with ParamValues: 1='30',
    2='hydra jobsets', 3='0', 4='src', 5='hydra/jobsets.nix', 6=undef, 7='.jobsets', 8='hydra'] at
    /nix/store/lsf81ip9ybxihk5praf2n0nh14a6i9j0-hydra-0.1.19700101.DIRTY/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 50

This change just omits adding such values to `%update`, which results in
PostgreSQL assigning the default values.
2020-05-15 20:33:54 +02:00
Graham Christensen 548fd8eadd
schema/Builds: use jobset_id instead of jobset name matches
This was clearly an error in the original part-2 of the diff, and
specifically breaks when two projects have a jobset of the same name.
2020-05-13 10:12:56 -04:00
Bas van Dijk 38122544ed GitInput: only convert integer option values to int
The previous code converted option values to ints when the value
contained a digit somewhere. This is too eager since it also converts
strings like `release-0.2` to an int which should not happen.

We now only convert to int when the value is an integer.
2020-05-13 11:41:52 +02:00
Eelco Dolstra 15a45f1a8a
Fix build 2020-05-12 16:14:20 +02:00
Eelco Dolstra e379628db0
hydra-eval-jobset: Pass --no-allow-import-from-derivation
https://github.com/NixOS/nixpkgs/issues/87592
2020-05-12 15:17:50 +02:00
Bas van Dijk f32a2a48d7
Merge pull request #740 from knl/add-githubrefs-plugin
Add GithubRefs plugin
2020-05-08 13:21:45 +02:00
Eelco Dolstra 96a514c169
Remove the "releases" feature
We haven't used this in many years (it was really only used for nix
and patchelf releases).
2020-05-06 12:39:21 +02:00
Ben Wolsieffer f020f7efef
hydra-queue-runner: don't try to distribute builds on localhost 2020-05-03 00:05:52 -04:00
Emery Hemingway e93c36aab1 SoTest: read credentials from file 2020-04-26 12:12:04 +05:30
Nikola Knezevic f03e7ef800 Add GithubRefs plugin
This plugin is a counterpart to GithubPulls plugin. Instead of fetching pull
requests, it will fetch all references (branches and tags) that start with a
particular prefix.

The plugin is a copy of GithubPulls plugin with appropriate changes to call the
right API and parse the config matching the need.
2020-04-23 10:45:37 +02:00
Emery Hemingway a63e349476 Add SoTest plugin
https://opensource.sotest.io/
https://docs.sotest.io/
2020-04-21 15:25:44 +05:30
Maximilian Bosch 721c764951
Remove Hydra::Helper::nix::txn_do from the Perl code
To quote the function's comment:

  Awful hack to handle timeouts in SQLite: just retry the transaction.
  DBD::SQLite *has* a 30 second retry window, but apparently it
  doesn't work.

Since SQLite is now dropped entirely, this wrapper can be removed
completely.
2020-04-16 00:42:40 +02:00
Maximilian Bosch efcbc08686
Get rid of dependency to SQLite
SQLite isn't properly supported by Hydra for a few years now[1], but
Hydra still depends on it. Apart from a slightly bigger closure this can
cause confusion by users since Hydra picks up SQLite rather than
PostgreSQL by default if HYDRA_DBI isn't configured properly[2]

[1] 78974abb69
[2] https://logs.nix.samueldr.com/nixos-dev/2020-04-10#3297342;
2020-04-16 00:42:40 +02:00
Graham Christensen 7b705758ec
Merge pull request #732 from Ma27/fix-build
Fix build against latest Nix
2020-04-09 09:02:45 -04:00
Bas van Dijk 6e358189ad Separate the build IDs in the build_finished payload with tabs
hydra-notify splits the payload on tabs so we shouldn't separate the
IDs with spaces.
2020-04-08 12:05:25 +02:00
Maximilian Bosch 2f9d422172
Fix build against latest Nix 2020-04-07 13:55:38 +02:00
Eelco Dolstra 4cabb37ebd
Merge pull request #730 from NixOS/flake
Flake support
2020-04-07 11:18:38 +02:00
Kevin Quick a055796ef5
Merge branch 'master' into libpqxx_undeprecate 2020-04-01 11:54:41 -07:00
Eelco Dolstra 2d092a6fbc
Merge pull request #702 from kquick/fix_api_push
Handle case where jobset has no defined errormsg for api/jobsets
2020-04-01 13:09:05 +02:00
Eelco Dolstra 8564089186
Merge pull request #707 from basvandijk/fix-aggregate-status
Fix printing aggregate status
2020-04-01 13:04:53 +02:00
Eelco Dolstra 6c0bb909df
Merge pull request #704 from basvandijk/sleep-after-catch-exception
hydra-queue-runner: sleep 5s after handling an exception
2020-04-01 13:04:31 +02:00
Eelco Dolstra 9668bc01da
Merge pull request #726 from knl/document-slack-notifications-plugin
Document SlackNotifications plugin
2020-04-01 13:04:00 +02:00
Eelco Dolstra 9727892b61 Don't spam the journal with hydra-queue-runner status dumps
(cherry picked from commit 15ae932488)
2020-03-31 22:19:07 +02:00
Eelco Dolstra ccd046ca3d Keep track of the number of unsupported steps
(cherry picked from commit 45ffe578b6)
2020-03-31 22:19:03 +02:00
Eelco Dolstra 4417f9f260 Abort unsupported build steps
If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.

(cherry picked from commit f5cdbfe21d)
2020-03-31 22:19:01 +02:00
Eelco Dolstra 68a59f34a0 Remove tabs
(cherry picked from commit 5b731004da)
2020-03-31 22:18:58 +02:00
Samuel Dionne-Riel 7889aef792
hydra-eval-jobs: Mirror eval errors in STDERR
Otherwise, errors will not be shown to end-users, which makes debugging
long evals pretty much impossible.

(cherry picked from commit 76299b9174)
2020-03-31 12:48:33 +02:00
Samuel Dionne-Riel 76299b9174 hydra-eval-jobs: Mirror eval errors in STDERR
Otherwise, errors will not be shown to end-users, which makes debugging
long evals pretty much impossible.
2020-03-30 16:11:44 -04:00
Eelco Dolstra 15ae932488
Don't spam the journal with hydra-queue-runner status dumps 2020-03-26 15:30:37 +01:00
Eelco Dolstra 45ffe578b6
Keep track of the number of unsupported steps 2020-03-26 15:27:13 +01:00
Eelco Dolstra f5cdbfe21d
Abort unsupported build steps
If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.
2020-03-26 15:00:04 +01:00
Eelco Dolstra 5b731004da
Remove tabs 2020-03-26 14:59:52 +01:00
Nikola Knezevic 1bee6e3d8a Add debug logging
This will help us track potential problems with the plugin.
2020-03-26 13:27:44 +01:00
Nikola Knezevic 986fde8888 Refactor code
Extract the conditions before the loop, as they do not change due to channel
definition.
2020-03-26 13:27:44 +01:00
Nikola Knezevic 956f009672 Add documentation for SlackNotification plugin 2020-03-26 13:27:44 +01:00
Eelco Dolstra d0688b93e1
Merge remote-tracking branch 'origin/master' into flake 2020-03-26 11:44:11 +01:00
Graham Christensen 12cc46cdb3
fixup: hydra-init: correct reference to hydra-backill-ids 2020-03-24 11:22:14 -04:00
Maximilian Bosch 73694087a0
Use the stopsign as icon for timeouts and exceeded log-limits
When I browse failed builds in a jobset-eval on Hydra, I regularly
mistake actual build-failures with temporary issues like timeouts (that
probably disappear at the next eval).

To prevent this kind of issue, I figured that using the stopsign-svg for
builds with timeouts or exceeded log-limits is a reasonable choice for
the following reasons:

* A user can now distinguish between actual build-errors (like
  compilation-failures or oversized outputs) and (usually) temporary issues
  (like a bloated log or a timeout).

* The stopsign is also used for aborted jobs that are shown in a
  different tab and can't be confused with timeouts for that reason.
2020-03-21 21:08:33 +01:00
Eelco Dolstra 4b5bb4e760
Merge remote-tracking branch 'origin/master' into flake 2020-03-04 15:28:23 +01:00
Eelco Dolstra 123bee1db5
Restore job type checking 2020-03-04 15:16:26 +01:00
Eelco Dolstra 69a6f3448a
Fix calling job functions
Fixes #718.
2020-03-04 15:16:04 +01:00
Eelco Dolstra 3cc1deb125
Merge pull request #721 from grahamc/one-by-one
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
2020-03-04 08:44:43 +01:00
Graham Christensen 994430b94b
treewide: allow nix command 2020-03-03 22:52:20 -05:00
Graham Christensen 117b9ecef1
Nix.pm: readNixFile: pass «--experimental-features nix-command»
Declarative jobsets were broken by the Nix update, causing
nix cat-file to break silently.

This commit restores declarative jobsets, based on top of a commit
making it easier to see what broke.
2020-03-03 22:36:21 -05:00
Graham Christensen 113a312f67
handleDeclarativeJobsetBuild: handle errors from readNixFile 2020-03-03 22:32:13 -05:00
Graham Christensen 5fae9d96a2
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
In the past, jobsets which are automatically evaluated are evaluated
regularly, on a schedule. This schedule means a new evaluation is
created every checkInterval seconds (assuming something changed.)

This model works well for architectures where our build farm can
easily keep up with demand.

This commit adds a new type of evaluation, called ONE_AT_A_TIME, which
only schedules a new evaluation if the previous evaluation of the
jobset has no unfinished builds.

This model of evaluation lets us have 'low-tier' architectures.

For example, we could now have a jobset for ARMv7l builds, where
the buildfarm only has a single, underpowered ARMv7l builder.
Configuring that jobset as ONE_AT_A_TIME will create an evaluation
and then won't schedule another evaluation until every job of
the existing evaluation is complete.

This way, the cache will have a complete collection of pre-built
software for some commits, but the underpowered architecture will
never become backlogged in ancient revisions.
2020-03-03 19:28:44 -05:00
Graham Christensen eaa65f51f4
hydra-evaluator: make the logic of the scheduler easier to read 2020-03-03 18:17:21 -05:00
Eelco Dolstra eb5873ae53
Fix build 2020-02-20 11:19:45 +01:00
Eelco Dolstra c642f787ee
hydra-eval-jobs: Parallelize
(cherry picked from commit be8eb9d00d)
2020-02-20 10:28:58 +01:00