Commit graph

3137 commits

Author SHA1 Message Date
Eelco Dolstra 858eb41fab
Merge pull request #795 from NixOS/no-nar-accessors
Stream NARs from the remote to the destination store
2020-07-28 15:03:57 +02:00
Eelco Dolstra 8722927c08
Copy paths in the right order 2020-07-28 13:46:57 +02:00
Eelco Dolstra 58a8b1c91c
Keep the SHA-1 column in existing installations 2020-07-28 11:47:44 +02:00
Kevin Quick 3e73a2f709
Fix issue #614: restart queue/evaluator on sufficient disk space avai… (#777)
* Fix issue #614: restart queue/evaluator on sufficient disk space available.

* Only try to stop the service if it is currently running.

* Use named variables and added restarting message.
2020-07-27 15:46:57 -04:00
Eelco Dolstra a0e24f446b
Remove unused getMemSize() function 2020-07-27 20:40:57 +02:00
Eelco Dolstra 5b4df3ad5a
Get data needed by getBuildOutput() from the incoming NAR in a streaming fashion 2020-07-27 20:38:59 +02:00
Eelco Dolstra d4e4be4fd1
Remove SHA-1 hash from BuildProducts
SHA-1 is deprecated and it will be expensive to compute with the
streaming NAR handler.
2020-07-27 18:24:10 +02:00
Eelco Dolstra 7622cbfe37
buildRemote(): Copy paths to the destination store in O(1) memory 2020-07-27 18:11:04 +02:00
Eelco Dolstra cbcf6359b4
Remove TokenServer in preparation of making NAR copying O(1) memory 2020-07-27 14:57:22 +02:00
Eelco Dolstra e5f6fc2e4e
Quick hack to fix compilation 2020-07-27 14:53:43 +02:00
Eelco Dolstra 90f03158f3
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/1ab9da915422405452118ebb17b88cdfc90b1e10' -> 'github:NixOS/nix/d7c0f094cbcfe1ae4ccc3d54baec00b66ccb1ed0'
2020-07-27 14:02:36 +02:00
dependabot[bot] 63188667b8
Bump cachix/install-nix-action from v8 to v10 (#791)
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from v8 to v10.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Commits](https://github.com/cachix/install-nix-action/compare/v8...63cf434de4e4292c6960639d56c5dd550e789d77)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-07-14 20:50:41 +02:00
Bas van Dijk 48678df8b6
updateDeclarativeJobset: only set the emailresponsible column when defined (#788) 2020-07-08 19:08:11 -04:00
Eelco Dolstra 7985757a1d
Fix build 2020-07-08 12:50:02 +02:00
Eelco Dolstra 6c9e407bee
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/334e26bfc2ce82912602e8a0f9f9c7e0fb5c3221' -> 'github:NixOS/nix/1ab9da915422405452118ebb17b88cdfc90b1e10'
* Updated 'nixpkgs': 'github:NixOS/nixpkgs/a84b797b28eb104db758b5cb2b61ba8face6744b' -> 'github:NixOS/nixpkgs/c9d124e39dbeefc53c7b3e09fbfc2c26bcbd4845'
2020-07-08 12:27:26 +02:00
Eelco Dolstra b0163e9eae
Fix project creation by non-admin users 2020-07-08 12:26:46 +02:00
Eelco Dolstra d0deebc4fc
Merge pull request #790 from Mic92/dependabot
automatically update github actions
2020-06-30 11:35:23 +02:00
Jörg Thalheim 2d2b0d11dc
automatically update github actions
Same as https://github.com/NixOS/nix/pull/3757
2020-06-30 08:34:01 +01:00
Eelco Dolstra bb32aafa4a
Fix build 2020-06-23 13:56:44 +02:00
Eelco Dolstra 1726adbad7
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/2e4bd782115c5842781db31f6e06c162f8afe780' -> 'github:NixOS/nix/334e26bfc2ce82912602e8a0f9f9c7e0fb5c3221'
2020-06-23 11:47:15 +02:00
Eelco Dolstra 2dd06329eb Simplify default.nix and shell.nix 2020-06-17 19:19:55 +02:00
Eelco Dolstra 03e8a7fef4 flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/14a3a62bfca6c572b9a415cfa80cdbd7ad4326b3' -> 'github:NixOS/nix/2e4bd782115c5842781db31f6e06c162f8afe780'
* Updated 'nix/nixpkgs': 'github:NixOS/nixpkgs/b88ff468e9850410070d4e0ccd68c7011f15b2be' -> 'github:NixOS/nixpkgs/70717a337f7ae4e486ba71a500367cad697e5f09'
* Updated 'nixpkgs': 'github:NixOS/nixpkgs/29eddfc36d720dcc4822581175217543b387b1e8' -> 'github:NixOS/nixpkgs/a84b797b28eb104db758b5cb2b61ba8face6744b'
2020-06-17 19:19:41 +02:00
Eelco Dolstra 1831866a52
Merge pull request #692 from knl/emit-hostname-with-influxdb-metrics
Add host tag to InfluxDB metrics
2020-06-10 10:57:17 +02:00
Eelco Dolstra 6168e7fa44
Merge pull request #781 from knl/remove-SSL_CERT_FILE
Remove SSL_CERT_FILE
2020-06-06 12:32:26 +02:00
Nikola Knezevic e5c757e219 Remove SSL_CERT_FILE
This was slated to be removed with Nix 16.03. We're now in 20.03, so
it's good to clean that up a bit.
2020-06-05 18:53:16 +02:00
Eelco Dolstra 56b1660c4d
Merge pull request #775 from knl/path-input-cache-validity-fix
Make PathInput plugin cache validity configurable
2020-06-05 17:22:07 +02:00
Eelco Dolstra 5f1faf4cb4
Merge pull request #779 from knl/there-is-no-hydra-postgresql.sql-anymore
Remove references to hydra-postgresql.sql
2020-06-05 17:21:40 +02:00
Eelco Dolstra c0b4686b32
Merge pull request #780 from knl/remove-dead-method
Remove dead method from Nix.pm
2020-06-05 17:21:15 +02:00
Nikola Knezevic e34d40d4f8 Remove dead method from Nix.pm
This method has been moved to hydra-eval-jobset a long time ago.
2020-06-05 15:01:36 +02:00
Nikola Knezevic 3acdd21569 Remove references to hydra-postgresql.sql
As of https://github.com/NixOS/hydra/pull/737 (removal of sqlite
dependency), the only supported database is Postgresql.

This change removes all references to hydra-postgresql.sql file. This
file is generated using a cpp on hydra.sql, but doesn't differ from
hydra.sql at all.
2020-06-05 13:42:56 +02:00
Nikola Knezevic fceaed2b24 Make PathInput plugin cache validity configurable
PathInput plugin keeps a cache of path evaluations. This cache is simple, and
path is not checked more than once every N seconds, where N=30. The caching is
there to avoid expensive calls to `nix-store --add`.

This change makes the validity period configurable. The main use case is
`api-test.pl` which was implemented wrong for a while, as the invocation of
`hydra-eval-jobset` would return the previous evaluation, claiming there are no
changes. The test has been fixed to check better for a new evaluation.
2020-06-04 12:26:47 +02:00
Eelco Dolstra 0b300e80ad
Merge pull request #773 from Ma27/fix-hydra-send-stats
Fix syntax error in `hydra-send-stats`
2020-06-02 16:25:25 +02:00
Maximilian Bosch d4822a5f4b
Fix syntax error in hydra-send-stats 2020-06-02 15:30:42 +02:00
Eelco Dolstra 750e2e618a
Merge pull request #770 from NixOS/remove-jobs
Remove the Jobs table
2020-06-01 10:25:41 +02:00
Eelco Dolstra d5844897da
Merge pull request #771 from knl/fix-unprocessed-notificationpendingsince
Fix: Set notificationpendingsince for dependent builds
2020-05-28 11:15:24 +02:00
Nikola Knezevic 7d52946982 Set notificationpendingsince for dependent builds
`build_finished` Postgres event will never be fired for the dependent builds.

For example, on our Hydra, the following query always returns increasing
numbers, even though all notifications have been delivered:

```
hydra=> select count(1) from builds where notificationpendingsince is not null;
 count
-------
  4583
(1 row)
```

Thus, we have to iterate over all dependent builds and mark their
`notificationpendingsince` as `null`, otherwise they will pile up until
the next restart of hydra-notify, when they will get delivered.
2020-05-28 10:45:41 +02:00
Nikola Knezevic 7148923d30 Make --no-allow-import-from-derivation configurable in hydra-eval-jobset
When deploying Hydra different than hydra.nixos.org one may encounter a problem
as building any job that uses IFD fails with:

May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/1jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/av3jr8ix4qcadq2wm3y3hplvxwzlhl4y-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize
'/nix/store/2jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but
'allow-import-from-derivation' is false"

The recent change enforced passing `--no-allow-import-from-derivation`
to `hydra-eval-job` unconditionally. This change makes it configurable and
defaults to **NOT PASSING IT** -- most of the deployments allow IFDs.

The configuration option is called `allow_import_from_derivation` and
defaults to `true`. It is interpreted as a boolean, with only true option being
`true`.
2020-05-28 10:15:46 +02:00
Eelco Dolstra 8adb433e3b
Remove the Jobs table
This table has been superfluous for a long time.
2020-05-27 20:09:36 +02:00
Eelco Dolstra af9f635a5a
Use PostgreSQL 11
This is what we use for hydra.nixos.org.
2020-05-27 17:34:14 +02:00
Eelco Dolstra a1e08d8819
Merge pull request #768 from knl/improve-handling-of-perl-block-eval-errors
Improve handling of Perl's block eval errors
2020-05-27 12:20:25 +02:00
Nikola Knezevic f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with &quot;1&quot;:
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
Eelco Dolstra 471b036eef
Merge pull request #765 from cransom/foreman-hydra-notify
Add hydra-notify to devshell
2020-05-25 16:36:43 +02:00
Casey Ransom 8a169ffa56 Add hydra-notify to devshell
I came across https://github.com/NixOS/hydra/issues/751 and realized
that hydra-notify is responsible for creating the additional jobsets in
a declarative file. My declarative testing works in dev now.
2020-05-20 15:38:31 -04:00
Bas van Dijk 3c6b724f92
Merge pull request #764 from knl/render-jobset.tt-correctly
Render the jobset page correctly when there are fetch errors
2020-05-20 17:58:07 +02:00
Nikola Knezevic 0f21e4067b Render the jobset page correctly when there are fetch errors
The original code would return standard "Please come back later" page when there
are only fetch errors on a newly setup declarative project. The problem is that
there are two types of errors: standard errors and fetch errors. Each is
acompanied by a corresponding field for time of occurence. Standard errors use
'errortime', while fetch errors have 'lastchecktime' set to the time of the
error. Unfortunately, jobset.tt file was only using 'errortime' for displaying
the time. This would result in the following errors in logs:

    Couldn't render template "date error - bad time/date string:  expects 'hⓂ️s dⓂ️y'  got: ''

This change includes using 'lastchecktime' when rendering the error times.
2020-05-20 17:47:18 +02:00
Bas van Dijk 03f14f46d7
Merge pull request #762 from knl/fix-declarative-jobset-missing-type-value
Fix declarative jobset missing type value
2020-05-18 11:04:00 +02:00
Nikola Knezevic e9922c460e Add missing SQL upgrade script for NOT NULL on type
`type` column in `Jobsets` is defined as NOT NULL. However, the original upgrade
script adding this column ommited the constraint.
2020-05-18 10:59:55 +02:00
Nikola Knezevic 575113396d Handle missing values in declarative jobsets
The current implementation will pass all values to `create_or_update` method. The
missing values will end up as `undef` (or `NULL`) when assigned to `%update`.
Thus, for columns that are NOT NULL, when, for example, flakes are not used,
will result in a horrible:

    DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed:
    ERROR:  null value in column "type" violates not-null constraint

    DETAIL:  Failing row contains (.jobsets, 118, hydra, hydra jobsets, src, hydra/jobsets.nix, null,
    null, null, 1589536378, 1, 0, 0, , 3, 30, 100, null, null, 1589536379, null, null). [for Statement
    "UPDATE jobsets SET checkinterval = ?, description = ?, enableemail = ?, nixexprinput = ?,
    nixexprpath = ?, type = ? WHERE ( ( name = ? AND project = ? ) )" with ParamValues: 1='30',
    2='hydra jobsets', 3='0', 4='src', 5='hydra/jobsets.nix', 6=undef, 7='.jobsets', 8='hydra'] at
    /nix/store/lsf81ip9ybxihk5praf2n0nh14a6i9j0-hydra-0.1.19700101.DIRTY/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 50

This change just omits adding such values to `%update`, which results in
PostgreSQL assigning the default values.
2020-05-15 20:33:54 +02:00
Eelco Dolstra c4104fe1fa
Merge pull request #759 from gilligan/devshell
Devshell
2020-05-15 14:21:40 +02:00
Tobias Pflug 006233d7f3 README.md update
- drop any mention of runHydra
- link foreman and mention Procfile
2020-05-15 13:59:36 +02:00