Commit graph

2965 commits

Author SHA1 Message Date
Eelco Dolstra 1831866a52
Merge pull request #692 from knl/emit-hostname-with-influxdb-metrics
Add host tag to InfluxDB metrics
2020-06-10 10:57:17 +02:00
Eelco Dolstra 6168e7fa44
Merge pull request #781 from knl/remove-SSL_CERT_FILE
Remove SSL_CERT_FILE
2020-06-06 12:32:26 +02:00
Nikola Knezevic e5c757e219 Remove SSL_CERT_FILE
This was slated to be removed with Nix 16.03. We're now in 20.03, so
it's good to clean that up a bit.
2020-06-05 18:53:16 +02:00
Eelco Dolstra 56b1660c4d
Merge pull request #775 from knl/path-input-cache-validity-fix
Make PathInput plugin cache validity configurable
2020-06-05 17:22:07 +02:00
Eelco Dolstra 5f1faf4cb4
Merge pull request #779 from knl/there-is-no-hydra-postgresql.sql-anymore
Remove references to hydra-postgresql.sql
2020-06-05 17:21:40 +02:00
Eelco Dolstra c0b4686b32
Merge pull request #780 from knl/remove-dead-method
Remove dead method from Nix.pm
2020-06-05 17:21:15 +02:00
Nikola Knezevic e34d40d4f8 Remove dead method from Nix.pm
This method has been moved to hydra-eval-jobset a long time ago.
2020-06-05 15:01:36 +02:00
Nikola Knezevic 3acdd21569 Remove references to hydra-postgresql.sql
As of https://github.com/NixOS/hydra/pull/737 (removal of sqlite
dependency), the only supported database is Postgresql.

This change removes all references to hydra-postgresql.sql file. This
file is generated using a cpp on hydra.sql, but doesn't differ from
hydra.sql at all.
2020-06-05 13:42:56 +02:00
Nikola Knezevic fceaed2b24 Make PathInput plugin cache validity configurable
PathInput plugin keeps a cache of path evaluations. This cache is simple, and
path is not checked more than once every N seconds, where N=30. The caching is
there to avoid expensive calls to `nix-store --add`.

This change makes the validity period configurable. The main use case is
`api-test.pl` which was implemented wrong for a while, as the invocation of
`hydra-eval-jobset` would return the previous evaluation, claiming there are no
changes. The test has been fixed to check better for a new evaluation.
2020-06-04 12:26:47 +02:00
Eelco Dolstra 0b300e80ad
Merge pull request #773 from Ma27/fix-hydra-send-stats
Fix syntax error in `hydra-send-stats`
2020-06-02 16:25:25 +02:00
Maximilian Bosch d4822a5f4b
Fix syntax error in hydra-send-stats 2020-06-02 15:30:42 +02:00
Eelco Dolstra 750e2e618a
Merge pull request #770 from NixOS/remove-jobs
Remove the Jobs table
2020-06-01 10:25:41 +02:00
Eelco Dolstra d5844897da
Merge pull request #771 from knl/fix-unprocessed-notificationpendingsince
Fix: Set notificationpendingsince for dependent builds
2020-05-28 11:15:24 +02:00
Nikola Knezevic 7d52946982 Set notificationpendingsince for dependent builds
`build_finished` Postgres event will never be fired for the dependent builds.

For example, on our Hydra, the following query always returns increasing
numbers, even though all notifications have been delivered:

```
hydra=> select count(1) from builds where notificationpendingsince is not null;
 count
-------
  4583
(1 row)
```

Thus, we have to iterate over all dependent builds and mark their
`notificationpendingsince` as `null`, otherwise they will pile up until
the next restart of hydra-notify, when they will get delivered.
2020-05-28 10:45:41 +02:00
Nikola Knezevic 7148923d30 Make --no-allow-import-from-derivation configurable in hydra-eval-jobset
When deploying Hydra different than hydra.nixos.org one may encounter a problem
as building any job that uses IFD fails with:

May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/1jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/av3jr8ix4qcadq2wm3y3hplvxwzlhl4y-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize
'/nix/store/2jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but
'allow-import-from-derivation' is false"

The recent change enforced passing `--no-allow-import-from-derivation`
to `hydra-eval-job` unconditionally. This change makes it configurable and
defaults to **NOT PASSING IT** -- most of the deployments allow IFDs.

The configuration option is called `allow_import_from_derivation` and
defaults to `true`. It is interpreted as a boolean, with only true option being
`true`.
2020-05-28 10:15:46 +02:00
Eelco Dolstra 8adb433e3b
Remove the Jobs table
This table has been superfluous for a long time.
2020-05-27 20:09:36 +02:00
Eelco Dolstra af9f635a5a
Use PostgreSQL 11
This is what we use for hydra.nixos.org.
2020-05-27 17:34:14 +02:00
Eelco Dolstra a1e08d8819
Merge pull request #768 from knl/improve-handling-of-perl-block-eval-errors
Improve handling of Perl's block eval errors
2020-05-27 12:20:25 +02:00
Nikola Knezevic f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with "1":
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
Eelco Dolstra 471b036eef
Merge pull request #765 from cransom/foreman-hydra-notify
Add hydra-notify to devshell
2020-05-25 16:36:43 +02:00
Casey Ransom 8a169ffa56 Add hydra-notify to devshell
I came across https://github.com/NixOS/hydra/issues/751 and realized
that hydra-notify is responsible for creating the additional jobsets in
a declarative file. My declarative testing works in dev now.
2020-05-20 15:38:31 -04:00
Bas van Dijk 3c6b724f92
Merge pull request #764 from knl/render-jobset.tt-correctly
Render the jobset page correctly when there are fetch errors
2020-05-20 17:58:07 +02:00
Nikola Knezevic 0f21e4067b Render the jobset page correctly when there are fetch errors
The original code would return standard "Please come back later" page when there
are only fetch errors on a newly setup declarative project. The problem is that
there are two types of errors: standard errors and fetch errors. Each is
acompanied by a corresponding field for time of occurence. Standard errors use
'errortime', while fetch errors have 'lastchecktime' set to the time of the
error. Unfortunately, jobset.tt file was only using 'errortime' for displaying
the time. This would result in the following errors in logs:

    Couldn't render template "date error - bad time/date string:  expects 'hⓂ️s dⓂ️y'  got: ''

This change includes using 'lastchecktime' when rendering the error times.
2020-05-20 17:47:18 +02:00
Bas van Dijk 03f14f46d7
Merge pull request #762 from knl/fix-declarative-jobset-missing-type-value
Fix declarative jobset missing type value
2020-05-18 11:04:00 +02:00
Nikola Knezevic e9922c460e Add missing SQL upgrade script for NOT NULL on type
`type` column in `Jobsets` is defined as NOT NULL. However, the original upgrade
script adding this column ommited the constraint.
2020-05-18 10:59:55 +02:00
Nikola Knezevic 575113396d Handle missing values in declarative jobsets
The current implementation will pass all values to `create_or_update` method. The
missing values will end up as `undef` (or `NULL`) when assigned to `%update`.
Thus, for columns that are NOT NULL, when, for example, flakes are not used,
will result in a horrible:

    DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed:
    ERROR:  null value in column "type" violates not-null constraint

    DETAIL:  Failing row contains (.jobsets, 118, hydra, hydra jobsets, src, hydra/jobsets.nix, null,
    null, null, 1589536378, 1, 0, 0, , 3, 30, 100, null, null, 1589536379, null, null). [for Statement
    "UPDATE jobsets SET checkinterval = ?, description = ?, enableemail = ?, nixexprinput = ?,
    nixexprpath = ?, type = ? WHERE ( ( name = ? AND project = ? ) )" with ParamValues: 1='30',
    2='hydra jobsets', 3='0', 4='src', 5='hydra/jobsets.nix', 6=undef, 7='.jobsets', 8='hydra'] at
    /nix/store/lsf81ip9ybxihk5praf2n0nh14a6i9j0-hydra-0.1.19700101.DIRTY/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 50

This change just omits adding such values to `%update`, which results in
PostgreSQL assigning the default values.
2020-05-15 20:33:54 +02:00
Eelco Dolstra c4104fe1fa
Merge pull request #759 from gilligan/devshell
Devshell
2020-05-15 14:21:40 +02:00
Tobias Pflug 006233d7f3 README.md update
- drop any mention of runHydra
- link foreman and mention Procfile
2020-05-15 13:59:36 +02:00
Tobias Pflug 31262f14fb Address PR comments:
- scripts -> foreman
- drop runHydra
- drop devShell
- move postgresql to buildInputs
2020-05-15 13:48:21 +02:00
Tobias Pflug fc0eb02ffe Run hydra-dev-server
Execute hydra-dev-server instead of hydra-server

Co-authored-by: Graham Christensen <graham@grahamc.com>
2020-05-15 12:37:16 +02:00
Tobias Pflug 1bcb8d0be9 Use pg_isready for readiness check
Co-authored-by: Graham Christensen <graham@grahamc.com>
2020-05-15 12:37:16 +02:00
Tobias Pflug d9d58b7055 Use pg_ctl for readiness check
Using `pg_ctl status` is more reliable than relying checking an
open port via netcat.
2020-05-15 12:37:16 +02:00
Tobias Pflug 45ac8e0cbc README.md: info about runHydra/foreman
Add sections about using `runHydra` and `foreman`
2020-05-15 12:37:16 +02:00
Tobias Pflug 0add1cc6d8 Default to devShell in shell.nix 2020-05-15 12:37:16 +02:00
Tobias Pflug 07a4465260 Use custom ports for postgres/hydra
Use custom ports so hydra and postgres can run in environments where
the default ports are in use already.
2020-05-15 12:37:16 +02:00
Andreas Rammhold eb06a435ab Add devShell for faster feedback
This adds a `devShell` which unlike `runHydra` doesn't start hydra
automatically and doesn't receive hydra as build input. It is better
suited for interactive development cycles:

```
$ nix-shell -A devShell
$ ./bootstrap
$ configurePhase
$ make
$ # hack hack hack
$ foreman start
  # test test test
  <C-c>
$ # hack hack hack
```
2020-05-15 12:37:16 +02:00
Andreas Rammhold d1237c315d Add runHydra shell
runHyda automatically starts hydra and postgres:

```
$ nix-shell -A runHydra
```

The shell receives hydra from the working copy as buildInput.
Running hydra, queue-runner, evaluator and postgres is managed
by foreman (https://github.com/ddollar/foreman) and configured
in `Procfile`.
2020-05-15 12:37:16 +02:00
Eelco Dolstra a614199449
Merge pull request #760 from knl/patch-1
Correct the link to hydra-api.yml file
2020-05-14 16:38:13 +02:00
Nikola Knežević 090c05be5d
Correct the link to hydra-api.yml file
The correct filename is `hydra-api.yml` not `hydra.yml`.
2020-05-14 14:15:30 +02:00
Eelco Dolstra 16ecd4f59d
Merge pull request #757 from grahamc/fixup-migration-p2
schema/Builds: use jobset_id instead of jobset name matches
2020-05-13 16:43:02 +02:00
Graham Christensen 548fd8eadd
schema/Builds: use jobset_id instead of jobset name matches
This was clearly an error in the original part-2 of the diff, and
specifically breaks when two projects have a jobset of the same name.
2020-05-13 10:12:56 -04:00
Bas van Dijk 301e4e088e
Merge pull request #756 from basvandijk/only-convert-integer-option-values-to-ints
GitInput: only convert integer option values to int
2020-05-13 12:09:20 +02:00
Bas van Dijk 38122544ed GitInput: only convert integer option values to int
The previous code converted option values to ints when the value
contained a digit somewhere. This is too eager since it also converts
strings like `release-0.2` to an int which should not happen.

We now only convert to int when the value is an integer.
2020-05-13 11:41:52 +02:00
Eelco Dolstra 15a45f1a8a
Fix build 2020-05-12 16:14:20 +02:00
Eelco Dolstra ace051e8d9
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/3aaceeb7e2d3fb8a07a1aa5a21df1dca6bbaa0ef' -> 'github:NixOS/nix/14a3a62bfca6c572b9a415cfa80cdbd7ad4326b3'
2020-05-12 16:03:29 +02:00
Eelco Dolstra e379628db0
hydra-eval-jobset: Pass --no-allow-import-from-derivation
https://github.com/NixOS/nixpkgs/issues/87592
2020-05-12 15:17:50 +02:00
Eelco Dolstra cb06f0af47
flake.nix: Remove edition field 2020-05-12 14:03:06 +02:00
Bas van Dijk c2bc84ddb7
Merge pull request #755 from knl/migrate-tests-to-the-python-DSL
Convert all tests to the Python DSL
2020-05-11 22:11:58 +02:00
Nikola Knezevic 3efa8223c9 Convert all tests to the Python DSL
Since Perl-based NixOS tests will be deprecated in nixpkgs 20.09, this
change migrates all tests to the new Python-based NixOS tests.
2020-05-11 22:05:30 +02:00
Tobias Pflug 41d70395e4
Add hydra OpenAPI description (#750)
* Add swagger config

* Add hydra api info to the README

* hydra.yaml: added some more descriptions

* Add /login

* Add 404/403 responses

* hydra.yaml -> hydra-api.yaml

* Address PR comments

- drop releases/releasename
- document dependency
- document defaultpath

* Fix syntax

* Add project creation

Add `PUT /project/{id}`

* Add /search

* Add "/api/jobsets" endpoint

* Add /api/push endpoint

* Add jobset PUT definition

* Add eval endpoint

* Remove duplicated key

* Fix typo

* Fix structural errors

* fix another error

* Link to hydra-api.yaml from master

* Add openapi yaml validation step to CI
2020-05-11 10:04:46 -04:00