Commit graph

2900 commits

Author SHA1 Message Date
Eelco Dolstra d5844897da
Merge pull request #771 from knl/fix-unprocessed-notificationpendingsince
Fix: Set notificationpendingsince for dependent builds
2020-05-28 11:15:24 +02:00
Nikola Knezevic 7d52946982 Set notificationpendingsince for dependent builds
`build_finished` Postgres event will never be fired for the dependent builds.

For example, on our Hydra, the following query always returns increasing
numbers, even though all notifications have been delivered:

```
hydra=> select count(1) from builds where notificationpendingsince is not null;
 count
-------
  4583
(1 row)
```

Thus, we have to iterate over all dependent builds and mark their
`notificationpendingsince` as `null`, otherwise they will pile up until
the next restart of hydra-notify, when they will get delivered.
2020-05-28 10:45:41 +02:00
Nikola Knezevic 7148923d30 Make --no-allow-import-from-derivation configurable in hydra-eval-jobset
When deploying Hydra different than hydra.nixos.org one may encounter a problem
as building any job that uses IFD fails with:

May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/1jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/av3jr8ix4qcadq2wm3y3hplvxwzlhl4y-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize
'/nix/store/2jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but
'allow-import-from-derivation' is false"

The recent change enforced passing `--no-allow-import-from-derivation`
to `hydra-eval-job` unconditionally. This change makes it configurable and
defaults to **NOT PASSING IT** -- most of the deployments allow IFDs.

The configuration option is called `allow_import_from_derivation` and
defaults to `true`. It is interpreted as a boolean, with only true option being
`true`.
2020-05-28 10:15:46 +02:00
Eelco Dolstra a1e08d8819
Merge pull request #768 from knl/improve-handling-of-perl-block-eval-errors
Improve handling of Perl's block eval errors
2020-05-27 12:20:25 +02:00
Nikola Knezevic f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with "1":
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
Eelco Dolstra 471b036eef
Merge pull request #765 from cransom/foreman-hydra-notify
Add hydra-notify to devshell
2020-05-25 16:36:43 +02:00
Casey Ransom 8a169ffa56 Add hydra-notify to devshell
I came across https://github.com/NixOS/hydra/issues/751 and realized
that hydra-notify is responsible for creating the additional jobsets in
a declarative file. My declarative testing works in dev now.
2020-05-20 15:38:31 -04:00
Bas van Dijk 3c6b724f92
Merge pull request #764 from knl/render-jobset.tt-correctly
Render the jobset page correctly when there are fetch errors
2020-05-20 17:58:07 +02:00
Nikola Knezevic 0f21e4067b Render the jobset page correctly when there are fetch errors
The original code would return standard "Please come back later" page when there
are only fetch errors on a newly setup declarative project. The problem is that
there are two types of errors: standard errors and fetch errors. Each is
acompanied by a corresponding field for time of occurence. Standard errors use
'errortime', while fetch errors have 'lastchecktime' set to the time of the
error. Unfortunately, jobset.tt file was only using 'errortime' for displaying
the time. This would result in the following errors in logs:

    Couldn't render template "date error - bad time/date string:  expects 'hⓂ️s dⓂ️y'  got: ''

This change includes using 'lastchecktime' when rendering the error times.
2020-05-20 17:47:18 +02:00
Bas van Dijk 03f14f46d7
Merge pull request #762 from knl/fix-declarative-jobset-missing-type-value
Fix declarative jobset missing type value
2020-05-18 11:04:00 +02:00
Nikola Knezevic e9922c460e Add missing SQL upgrade script for NOT NULL on type
`type` column in `Jobsets` is defined as NOT NULL. However, the original upgrade
script adding this column ommited the constraint.
2020-05-18 10:59:55 +02:00
Nikola Knezevic 575113396d Handle missing values in declarative jobsets
The current implementation will pass all values to `create_or_update` method. The
missing values will end up as `undef` (or `NULL`) when assigned to `%update`.
Thus, for columns that are NOT NULL, when, for example, flakes are not used,
will result in a horrible:

    DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed:
    ERROR:  null value in column "type" violates not-null constraint

    DETAIL:  Failing row contains (.jobsets, 118, hydra, hydra jobsets, src, hydra/jobsets.nix, null,
    null, null, 1589536378, 1, 0, 0, , 3, 30, 100, null, null, 1589536379, null, null). [for Statement
    "UPDATE jobsets SET checkinterval = ?, description = ?, enableemail = ?, nixexprinput = ?,
    nixexprpath = ?, type = ? WHERE ( ( name = ? AND project = ? ) )" with ParamValues: 1='30',
    2='hydra jobsets', 3='0', 4='src', 5='hydra/jobsets.nix', 6=undef, 7='.jobsets', 8='hydra'] at
    /nix/store/lsf81ip9ybxihk5praf2n0nh14a6i9j0-hydra-0.1.19700101.DIRTY/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 50

This change just omits adding such values to `%update`, which results in
PostgreSQL assigning the default values.
2020-05-15 20:33:54 +02:00
Eelco Dolstra c4104fe1fa
Merge pull request #759 from gilligan/devshell
Devshell
2020-05-15 14:21:40 +02:00
Tobias Pflug 006233d7f3 README.md update
- drop any mention of runHydra
- link foreman and mention Procfile
2020-05-15 13:59:36 +02:00
Tobias Pflug 31262f14fb Address PR comments:
- scripts -> foreman
- drop runHydra
- drop devShell
- move postgresql to buildInputs
2020-05-15 13:48:21 +02:00
Tobias Pflug fc0eb02ffe Run hydra-dev-server
Execute hydra-dev-server instead of hydra-server

Co-authored-by: Graham Christensen <graham@grahamc.com>
2020-05-15 12:37:16 +02:00
Tobias Pflug 1bcb8d0be9 Use pg_isready for readiness check
Co-authored-by: Graham Christensen <graham@grahamc.com>
2020-05-15 12:37:16 +02:00
Tobias Pflug d9d58b7055 Use pg_ctl for readiness check
Using `pg_ctl status` is more reliable than relying checking an
open port via netcat.
2020-05-15 12:37:16 +02:00
Tobias Pflug 45ac8e0cbc README.md: info about runHydra/foreman
Add sections about using `runHydra` and `foreman`
2020-05-15 12:37:16 +02:00
Tobias Pflug 0add1cc6d8 Default to devShell in shell.nix 2020-05-15 12:37:16 +02:00
Tobias Pflug 07a4465260 Use custom ports for postgres/hydra
Use custom ports so hydra and postgres can run in environments where
the default ports are in use already.
2020-05-15 12:37:16 +02:00
Andreas Rammhold eb06a435ab Add devShell for faster feedback
This adds a `devShell` which unlike `runHydra` doesn't start hydra
automatically and doesn't receive hydra as build input. It is better
suited for interactive development cycles:

```
$ nix-shell -A devShell
$ ./bootstrap
$ configurePhase
$ make
$ # hack hack hack
$ foreman start
  # test test test
  <C-c>
$ # hack hack hack
```
2020-05-15 12:37:16 +02:00
Andreas Rammhold d1237c315d Add runHydra shell
runHyda automatically starts hydra and postgres:

```
$ nix-shell -A runHydra
```

The shell receives hydra from the working copy as buildInput.
Running hydra, queue-runner, evaluator and postgres is managed
by foreman (https://github.com/ddollar/foreman) and configured
in `Procfile`.
2020-05-15 12:37:16 +02:00
Eelco Dolstra a614199449
Merge pull request #760 from knl/patch-1
Correct the link to hydra-api.yml file
2020-05-14 16:38:13 +02:00
Nikola Knežević 090c05be5d
Correct the link to hydra-api.yml file
The correct filename is `hydra-api.yml` not `hydra.yml`.
2020-05-14 14:15:30 +02:00
Eelco Dolstra 16ecd4f59d
Merge pull request #757 from grahamc/fixup-migration-p2
schema/Builds: use jobset_id instead of jobset name matches
2020-05-13 16:43:02 +02:00
Graham Christensen 548fd8eadd
schema/Builds: use jobset_id instead of jobset name matches
This was clearly an error in the original part-2 of the diff, and
specifically breaks when two projects have a jobset of the same name.
2020-05-13 10:12:56 -04:00
Bas van Dijk 301e4e088e
Merge pull request #756 from basvandijk/only-convert-integer-option-values-to-ints
GitInput: only convert integer option values to int
2020-05-13 12:09:20 +02:00
Bas van Dijk 38122544ed GitInput: only convert integer option values to int
The previous code converted option values to ints when the value
contained a digit somewhere. This is too eager since it also converts
strings like `release-0.2` to an int which should not happen.

We now only convert to int when the value is an integer.
2020-05-13 11:41:52 +02:00
Eelco Dolstra 15a45f1a8a
Fix build 2020-05-12 16:14:20 +02:00
Eelco Dolstra ace051e8d9
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/3aaceeb7e2d3fb8a07a1aa5a21df1dca6bbaa0ef' -> 'github:NixOS/nix/14a3a62bfca6c572b9a415cfa80cdbd7ad4326b3'
2020-05-12 16:03:29 +02:00
Eelco Dolstra e379628db0
hydra-eval-jobset: Pass --no-allow-import-from-derivation
https://github.com/NixOS/nixpkgs/issues/87592
2020-05-12 15:17:50 +02:00
Eelco Dolstra cb06f0af47
flake.nix: Remove edition field 2020-05-12 14:03:06 +02:00
Bas van Dijk c2bc84ddb7
Merge pull request #755 from knl/migrate-tests-to-the-python-DSL
Convert all tests to the Python DSL
2020-05-11 22:11:58 +02:00
Nikola Knezevic 3efa8223c9 Convert all tests to the Python DSL
Since Perl-based NixOS tests will be deprecated in nixpkgs 20.09, this
change migrates all tests to the new Python-based NixOS tests.
2020-05-11 22:05:30 +02:00
Tobias Pflug 41d70395e4
Add hydra OpenAPI description (#750)
* Add swagger config

* Add hydra api info to the README

* hydra.yaml: added some more descriptions

* Add /login

* Add 404/403 responses

* hydra.yaml -> hydra-api.yaml

* Address PR comments

- drop releases/releasename
- document dependency
- document defaultpath

* Fix syntax

* Add project creation

Add `PUT /project/{id}`

* Add /search

* Add "/api/jobsets" endpoint

* Add /api/push endpoint

* Add jobset PUT definition

* Add eval endpoint

* Remove duplicated key

* Fix typo

* Fix structural errors

* fix another error

* Link to hydra-api.yaml from master

* Add openapi yaml validation step to CI
2020-05-11 10:04:46 -04:00
Bas van Dijk f32a2a48d7
Merge pull request #740 from knl/add-githubrefs-plugin
Add GithubRefs plugin
2020-05-08 13:21:45 +02:00
Eelco Dolstra 88ef3e68a2
Merge pull request #752 from NixOS/remove-releases
Remove the "releases" feature
2020-05-06 18:20:45 +02:00
Eelco Dolstra 96a514c169
Remove the "releases" feature
We haven't used this in many years (it was really only used for nix
and patchelf releases).
2020-05-06 12:39:21 +02:00
Eelco Dolstra ace30b4184
Merge pull request #749 from lopsided98/localhost-no-remote
Don't distribute localhost builds to other builders
2020-05-04 16:36:53 +02:00
Eelco Dolstra 65819a225d
Merge pull request #748 from gilligan/flake-compat-comment
Add comment about edolstra/flake-compat
2020-05-04 16:36:00 +02:00
Ben Wolsieffer f020f7efef
hydra-queue-runner: don't try to distribute builds on localhost 2020-05-03 00:05:52 -04:00
Tobias Pflug 9d80a6de66 Add comment about edolstra/flake-compat
Add a comment to default.nix and shell.nix briefly outlining what
flake-compat/default.nix does.
2020-05-02 18:56:30 +02:00
Graham Christensen c82f51751d
Merge pull request #747 from gilligan/extend-readme
Extend Setup Information
2020-05-02 10:21:51 -04:00
Tobias Pflug c8c308c0e2 Extend Setup Information
Add information on how to create a minimal setup: project/jobset.
2020-05-02 16:04:20 +02:00
Graham Christensen fd38524843
Merge pull request #746 from gilligan/add-ci-badge
Add CI badge to README.md
2020-05-02 08:36:38 -04:00
Tobias Pflug 32c35ed6e6 Add CI badge to README.md 2020-05-02 14:29:19 +02:00
Graham Christensen 9f2a7f86b3
Merge pull request #745 from gilligan/remove-png
Remove .png
2020-05-02 08:24:35 -04:00
Tobias Pflug 8400b005f3 Remove .png
File was added to the repo by accident
2020-05-02 14:19:14 +02:00
Graham Christensen 4dcae3100e
Merge pull request #744 from gilligan/revamp-readme
Update README
2020-05-01 18:02:25 -04:00