0
0
Fork 0
forked from lix-project/hydra
Commit graph

2348 commits

Author SHA1 Message Date
Casey Ransom 03be8ae7a1 Remove image dependency on hydra.nixos.org
Also standardize the slack image size on 256 pixels.
2020-09-01 15:13:12 -04:00
Eelco Dolstra 405c52b589
Fix build 2020-08-27 17:46:36 +02:00
Eelco Dolstra 1113c2895a Fix build 2020-08-07 21:42:09 +02:00
Eelco Dolstra f8e15bc311 Revive putBytes 2020-08-04 18:25:21 +02:00
Eelco Dolstra 4b5813051b
unsigned long long -> uint64_t 2020-08-04 11:38:22 +02:00
Eelco Dolstra 7d3ba616a9
Fix build 2020-08-04 11:33:29 +02:00
Eelco Dolstra 77c33c1d71
Restore NoCheckSigs
https://github.com/NixOS/nixpkgs/pull/93945#issuecomment-668244478
2020-08-04 10:53:06 +02:00
Eelco Dolstra 8722927c08
Copy paths in the right order 2020-07-28 13:46:57 +02:00
Eelco Dolstra 58a8b1c91c
Keep the SHA-1 column in existing installations 2020-07-28 11:47:44 +02:00
Eelco Dolstra a0e24f446b
Remove unused getMemSize() function 2020-07-27 20:40:57 +02:00
Eelco Dolstra 5b4df3ad5a
Get data needed by getBuildOutput() from the incoming NAR in a streaming fashion 2020-07-27 20:38:59 +02:00
Eelco Dolstra d4e4be4fd1
Remove SHA-1 hash from BuildProducts
SHA-1 is deprecated and it will be expensive to compute with the
streaming NAR handler.
2020-07-27 18:24:10 +02:00
Eelco Dolstra 7622cbfe37
buildRemote(): Copy paths to the destination store in O(1) memory 2020-07-27 18:11:04 +02:00
Eelco Dolstra cbcf6359b4
Remove TokenServer in preparation of making NAR copying O(1) memory 2020-07-27 14:57:22 +02:00
Eelco Dolstra e5f6fc2e4e
Quick hack to fix compilation 2020-07-27 14:53:43 +02:00
Bas van Dijk 48678df8b6
updateDeclarativeJobset: only set the emailresponsible column when defined () 2020-07-08 19:08:11 -04:00
Eelco Dolstra 7985757a1d
Fix build 2020-07-08 12:50:02 +02:00
Eelco Dolstra b0163e9eae
Fix project creation by non-admin users 2020-07-08 12:26:46 +02:00
Eelco Dolstra bb32aafa4a
Fix build 2020-06-23 13:56:44 +02:00
Eelco Dolstra 1831866a52
Merge pull request from knl/emit-hostname-with-influxdb-metrics
Add host tag to InfluxDB metrics
2020-06-10 10:57:17 +02:00
Eelco Dolstra 56b1660c4d
Merge pull request from knl/path-input-cache-validity-fix
Make PathInput plugin cache validity configurable
2020-06-05 17:22:07 +02:00
Eelco Dolstra 5f1faf4cb4
Merge pull request from knl/there-is-no-hydra-postgresql.sql-anymore
Remove references to hydra-postgresql.sql
2020-06-05 17:21:40 +02:00
Nikola Knezevic e34d40d4f8 Remove dead method from Nix.pm
This method has been moved to hydra-eval-jobset a long time ago.
2020-06-05 15:01:36 +02:00
Nikola Knezevic 3acdd21569 Remove references to hydra-postgresql.sql
As of https://github.com/NixOS/hydra/pull/737 (removal of sqlite
dependency), the only supported database is Postgresql.

This change removes all references to hydra-postgresql.sql file. This
file is generated using a cpp on hydra.sql, but doesn't differ from
hydra.sql at all.
2020-06-05 13:42:56 +02:00
Nikola Knezevic fceaed2b24 Make PathInput plugin cache validity configurable
PathInput plugin keeps a cache of path evaluations. This cache is simple, and
path is not checked more than once every N seconds, where N=30. The caching is
there to avoid expensive calls to `nix-store --add`.

This change makes the validity period configurable. The main use case is
`api-test.pl` which was implemented wrong for a while, as the invocation of
`hydra-eval-jobset` would return the previous evaluation, claiming there are no
changes. The test has been fixed to check better for a new evaluation.
2020-06-04 12:26:47 +02:00
Maximilian Bosch d4822a5f4b
Fix syntax error in hydra-send-stats 2020-06-02 15:30:42 +02:00
Eelco Dolstra 750e2e618a
Merge pull request from NixOS/remove-jobs
Remove the Jobs table
2020-06-01 10:25:41 +02:00
Eelco Dolstra d5844897da
Merge pull request from knl/fix-unprocessed-notificationpendingsince
Fix: Set notificationpendingsince for dependent builds
2020-05-28 11:15:24 +02:00
Nikola Knezevic 7d52946982 Set notificationpendingsince for dependent builds
`build_finished` Postgres event will never be fired for the dependent builds.

For example, on our Hydra, the following query always returns increasing
numbers, even though all notifications have been delivered:

```
hydra=> select count(1) from builds where notificationpendingsince is not null;
 count
-------
  4583
(1 row)
```

Thus, we have to iterate over all dependent builds and mark their
`notificationpendingsince` as `null`, otherwise they will pile up until
the next restart of hydra-notify, when they will get delivered.
2020-05-28 10:45:41 +02:00
Nikola Knezevic 7148923d30 Make --no-allow-import-from-derivation configurable in hydra-eval-jobset
When deploying Hydra different than hydra.nixos.org one may encounter a problem
as building any job that uses IFD fails with:

May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/1jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/av3jr8ix4qcadq2wm3y3hplvxwzlhl4y-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize
'/nix/store/2jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but
'allow-import-from-derivation' is false"

The recent change enforced passing `--no-allow-import-from-derivation`
to `hydra-eval-job` unconditionally. This change makes it configurable and
defaults to **NOT PASSING IT** -- most of the deployments allow IFDs.

The configuration option is called `allow_import_from_derivation` and
defaults to `true`. It is interpreted as a boolean, with only true option being
`true`.
2020-05-28 10:15:46 +02:00
Eelco Dolstra 8adb433e3b
Remove the Jobs table
This table has been superfluous for a long time.
2020-05-27 20:09:36 +02:00
Nikola Knezevic f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with "1":
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
Nikola Knezevic 0f21e4067b Render the jobset page correctly when there are fetch errors
The original code would return standard "Please come back later" page when there
are only fetch errors on a newly setup declarative project. The problem is that
there are two types of errors: standard errors and fetch errors. Each is
acompanied by a corresponding field for time of occurence. Standard errors use
'errortime', while fetch errors have 'lastchecktime' set to the time of the
error. Unfortunately, jobset.tt file was only using 'errortime' for displaying
the time. This would result in the following errors in logs:

    Couldn't render template "date error - bad time/date string:  expects 'hⓂ️s dⓂ️y'  got: ''

This change includes using 'lastchecktime' when rendering the error times.
2020-05-20 17:47:18 +02:00
Nikola Knezevic e9922c460e Add missing SQL upgrade script for NOT NULL on type
`type` column in `Jobsets` is defined as NOT NULL. However, the original upgrade
script adding this column ommited the constraint.
2020-05-18 10:59:55 +02:00
Nikola Knezevic 575113396d Handle missing values in declarative jobsets
The current implementation will pass all values to `create_or_update` method. The
missing values will end up as `undef` (or `NULL`) when assigned to `%update`.
Thus, for columns that are NOT NULL, when, for example, flakes are not used,
will result in a horrible:

    DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed:
    ERROR:  null value in column "type" violates not-null constraint

    DETAIL:  Failing row contains (.jobsets, 118, hydra, hydra jobsets, src, hydra/jobsets.nix, null,
    null, null, 1589536378, 1, 0, 0, , 3, 30, 100, null, null, 1589536379, null, null). [for Statement
    "UPDATE jobsets SET checkinterval = ?, description = ?, enableemail = ?, nixexprinput = ?,
    nixexprpath = ?, type = ? WHERE ( ( name = ? AND project = ? ) )" with ParamValues: 1='30',
    2='hydra jobsets', 3='0', 4='src', 5='hydra/jobsets.nix', 6=undef, 7='.jobsets', 8='hydra'] at
    /nix/store/lsf81ip9ybxihk5praf2n0nh14a6i9j0-hydra-0.1.19700101.DIRTY/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 50

This change just omits adding such values to `%update`, which results in
PostgreSQL assigning the default values.
2020-05-15 20:33:54 +02:00
Graham Christensen 548fd8eadd
schema/Builds: use jobset_id instead of jobset name matches
This was clearly an error in the original part-2 of the diff, and
specifically breaks when two projects have a jobset of the same name.
2020-05-13 10:12:56 -04:00
Bas van Dijk 38122544ed GitInput: only convert integer option values to int
The previous code converted option values to ints when the value
contained a digit somewhere. This is too eager since it also converts
strings like `release-0.2` to an int which should not happen.

We now only convert to int when the value is an integer.
2020-05-13 11:41:52 +02:00
Eelco Dolstra 15a45f1a8a
Fix build 2020-05-12 16:14:20 +02:00
Eelco Dolstra e379628db0
hydra-eval-jobset: Pass --no-allow-import-from-derivation
https://github.com/NixOS/nixpkgs/issues/87592
2020-05-12 15:17:50 +02:00
Bas van Dijk f32a2a48d7
Merge pull request from knl/add-githubrefs-plugin
Add GithubRefs plugin
2020-05-08 13:21:45 +02:00
Eelco Dolstra 96a514c169
Remove the "releases" feature
We haven't used this in many years (it was really only used for nix
and patchelf releases).
2020-05-06 12:39:21 +02:00
Ben Wolsieffer f020f7efef
hydra-queue-runner: don't try to distribute builds on localhost 2020-05-03 00:05:52 -04:00
Emery Hemingway e93c36aab1 SoTest: read credentials from file 2020-04-26 12:12:04 +05:30
Nikola Knezevic f03e7ef800 Add GithubRefs plugin
This plugin is a counterpart to GithubPulls plugin. Instead of fetching pull
requests, it will fetch all references (branches and tags) that start with a
particular prefix.

The plugin is a copy of GithubPulls plugin with appropriate changes to call the
right API and parse the config matching the need.
2020-04-23 10:45:37 +02:00
Emery Hemingway a63e349476 Add SoTest plugin
https://opensource.sotest.io/
https://docs.sotest.io/
2020-04-21 15:25:44 +05:30
Maximilian Bosch 721c764951
Remove Hydra::Helper::nix::txn_do from the Perl code
To quote the function's comment:

  Awful hack to handle timeouts in SQLite: just retry the transaction.
  DBD::SQLite *has* a 30 second retry window, but apparently it
  doesn't work.

Since SQLite is now dropped entirely, this wrapper can be removed
completely.
2020-04-16 00:42:40 +02:00
Maximilian Bosch efcbc08686
Get rid of dependency to SQLite
SQLite isn't properly supported by Hydra for a few years now[1], but
Hydra still depends on it. Apart from a slightly bigger closure this can
cause confusion by users since Hydra picks up SQLite rather than
PostgreSQL by default if HYDRA_DBI isn't configured properly[2]

[1] 78974abb69
[2] https://logs.nix.samueldr.com/nixos-dev/2020-04-10#3297342;
2020-04-16 00:42:40 +02:00
Graham Christensen 7b705758ec
Merge pull request from Ma27/fix-build
Fix build against latest Nix
2020-04-09 09:02:45 -04:00
Bas van Dijk 6e358189ad Separate the build IDs in the build_finished payload with tabs
hydra-notify splits the payload on tabs so we shouldn't separate the
IDs with spaces.
2020-04-08 12:05:25 +02:00
Maximilian Bosch 2f9d422172
Fix build against latest Nix 2020-04-07 13:55:38 +02:00