Commit graph

2944 commits

Author SHA1 Message Date
Eelco Dolstra
529a003de5 flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/4e7f1c7f11fb22d7954b2dacbe87c61548fab82d' -> 'github:NixOS/nix/edfd676e059578fb574ce78d1a2cc66d018d3b16'
2020-08-07 21:20:39 +02:00
Eelco Dolstra
f8e15bc311 Revive putBytes 2020-08-04 18:25:21 +02:00
Eelco Dolstra
56495289cd flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/ed52cf632bf9db90311014600ca0f042ef57ca20' -> 'github:NixOS/nix/4e7f1c7f11fb22d7954b2dacbe87c61548fab82d'
2020-08-04 16:02:38 +02:00
Eelco Dolstra
4b5813051b
unsigned long long -> uint64_t 2020-08-04 11:38:22 +02:00
Eelco Dolstra
7d3ba616a9
Fix build 2020-08-04 11:33:29 +02:00
Eelco Dolstra
7b8a3acbf0
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/d7c0f094cbcfe1ae4ccc3d54baec00b66ccb1ed0' -> 'github:NixOS/nix/ed52cf632bf9db90311014600ca0f042ef57ca20'
2020-08-04 10:54:40 +02:00
Eelco Dolstra
77c33c1d71
Restore NoCheckSigs
https://github.com/NixOS/nixpkgs/pull/93945#issuecomment-668244478
2020-08-04 10:53:06 +02:00
Eelco Dolstra
858eb41fab
Merge pull request #795 from NixOS/no-nar-accessors
Stream NARs from the remote to the destination store
2020-07-28 15:03:57 +02:00
Eelco Dolstra
8722927c08
Copy paths in the right order 2020-07-28 13:46:57 +02:00
Eelco Dolstra
58a8b1c91c
Keep the SHA-1 column in existing installations 2020-07-28 11:47:44 +02:00
Kevin Quick
3e73a2f709
Fix issue #614: restart queue/evaluator on sufficient disk space avai… (#777)
* Fix issue #614: restart queue/evaluator on sufficient disk space available.

* Only try to stop the service if it is currently running.

* Use named variables and added restarting message.
2020-07-27 15:46:57 -04:00
Eelco Dolstra
a0e24f446b
Remove unused getMemSize() function 2020-07-27 20:40:57 +02:00
Eelco Dolstra
5b4df3ad5a
Get data needed by getBuildOutput() from the incoming NAR in a streaming fashion 2020-07-27 20:38:59 +02:00
Eelco Dolstra
d4e4be4fd1
Remove SHA-1 hash from BuildProducts
SHA-1 is deprecated and it will be expensive to compute with the
streaming NAR handler.
2020-07-27 18:24:10 +02:00
Eelco Dolstra
7622cbfe37
buildRemote(): Copy paths to the destination store in O(1) memory 2020-07-27 18:11:04 +02:00
Eelco Dolstra
cbcf6359b4
Remove TokenServer in preparation of making NAR copying O(1) memory 2020-07-27 14:57:22 +02:00
Eelco Dolstra
e5f6fc2e4e
Quick hack to fix compilation 2020-07-27 14:53:43 +02:00
Eelco Dolstra
90f03158f3
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/1ab9da915422405452118ebb17b88cdfc90b1e10' -> 'github:NixOS/nix/d7c0f094cbcfe1ae4ccc3d54baec00b66ccb1ed0'
2020-07-27 14:02:36 +02:00
dependabot[bot]
63188667b8
Bump cachix/install-nix-action from v8 to v10 (#791)
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from v8 to v10.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Commits](https://github.com/cachix/install-nix-action/compare/v8...63cf434de4e4292c6960639d56c5dd550e789d77)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-07-14 20:50:41 +02:00
Bas van Dijk
48678df8b6
updateDeclarativeJobset: only set the emailresponsible column when defined (#788) 2020-07-08 19:08:11 -04:00
Eelco Dolstra
7985757a1d
Fix build 2020-07-08 12:50:02 +02:00
Eelco Dolstra
6c9e407bee
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/334e26bfc2ce82912602e8a0f9f9c7e0fb5c3221' -> 'github:NixOS/nix/1ab9da915422405452118ebb17b88cdfc90b1e10'
* Updated 'nixpkgs': 'github:NixOS/nixpkgs/a84b797b28eb104db758b5cb2b61ba8face6744b' -> 'github:NixOS/nixpkgs/c9d124e39dbeefc53c7b3e09fbfc2c26bcbd4845'
2020-07-08 12:27:26 +02:00
Eelco Dolstra
b0163e9eae
Fix project creation by non-admin users 2020-07-08 12:26:46 +02:00
Eelco Dolstra
d0deebc4fc
Merge pull request #790 from Mic92/dependabot
automatically update github actions
2020-06-30 11:35:23 +02:00
2d2b0d11dc
automatically update github actions
Same as https://github.com/NixOS/nix/pull/3757
2020-06-30 08:34:01 +01:00
Eelco Dolstra
bb32aafa4a
Fix build 2020-06-23 13:56:44 +02:00
Eelco Dolstra
1726adbad7
flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/2e4bd782115c5842781db31f6e06c162f8afe780' -> 'github:NixOS/nix/334e26bfc2ce82912602e8a0f9f9c7e0fb5c3221'
2020-06-23 11:47:15 +02:00
Eelco Dolstra
2dd06329eb Simplify default.nix and shell.nix 2020-06-17 19:19:55 +02:00
Eelco Dolstra
03e8a7fef4 flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/14a3a62bfca6c572b9a415cfa80cdbd7ad4326b3' -> 'github:NixOS/nix/2e4bd782115c5842781db31f6e06c162f8afe780'
* Updated 'nix/nixpkgs': 'github:NixOS/nixpkgs/b88ff468e9850410070d4e0ccd68c7011f15b2be' -> 'github:NixOS/nixpkgs/70717a337f7ae4e486ba71a500367cad697e5f09'
* Updated 'nixpkgs': 'github:NixOS/nixpkgs/29eddfc36d720dcc4822581175217543b387b1e8' -> 'github:NixOS/nixpkgs/a84b797b28eb104db758b5cb2b61ba8face6744b'
2020-06-17 19:19:41 +02:00
Eelco Dolstra
1831866a52
Merge pull request #692 from knl/emit-hostname-with-influxdb-metrics
Add host tag to InfluxDB metrics
2020-06-10 10:57:17 +02:00
Eelco Dolstra
6168e7fa44
Merge pull request #781 from knl/remove-SSL_CERT_FILE
Remove SSL_CERT_FILE
2020-06-06 12:32:26 +02:00
Nikola Knezevic
e5c757e219 Remove SSL_CERT_FILE
This was slated to be removed with Nix 16.03. We're now in 20.03, so
it's good to clean that up a bit.
2020-06-05 18:53:16 +02:00
Eelco Dolstra
56b1660c4d
Merge pull request #775 from knl/path-input-cache-validity-fix
Make PathInput plugin cache validity configurable
2020-06-05 17:22:07 +02:00
Eelco Dolstra
5f1faf4cb4
Merge pull request #779 from knl/there-is-no-hydra-postgresql.sql-anymore
Remove references to hydra-postgresql.sql
2020-06-05 17:21:40 +02:00
Eelco Dolstra
c0b4686b32
Merge pull request #780 from knl/remove-dead-method
Remove dead method from Nix.pm
2020-06-05 17:21:15 +02:00
Nikola Knezevic
e34d40d4f8 Remove dead method from Nix.pm
This method has been moved to hydra-eval-jobset a long time ago.
2020-06-05 15:01:36 +02:00
Nikola Knezevic
3acdd21569 Remove references to hydra-postgresql.sql
As of https://github.com/NixOS/hydra/pull/737 (removal of sqlite
dependency), the only supported database is Postgresql.

This change removes all references to hydra-postgresql.sql file. This
file is generated using a cpp on hydra.sql, but doesn't differ from
hydra.sql at all.
2020-06-05 13:42:56 +02:00
Nikola Knezevic
fceaed2b24 Make PathInput plugin cache validity configurable
PathInput plugin keeps a cache of path evaluations. This cache is simple, and
path is not checked more than once every N seconds, where N=30. The caching is
there to avoid expensive calls to `nix-store --add`.

This change makes the validity period configurable. The main use case is
`api-test.pl` which was implemented wrong for a while, as the invocation of
`hydra-eval-jobset` would return the previous evaluation, claiming there are no
changes. The test has been fixed to check better for a new evaluation.
2020-06-04 12:26:47 +02:00
Eelco Dolstra
0b300e80ad
Merge pull request #773 from Ma27/fix-hydra-send-stats
Fix syntax error in `hydra-send-stats`
2020-06-02 16:25:25 +02:00
d4822a5f4b
Fix syntax error in hydra-send-stats 2020-06-02 15:30:42 +02:00
Eelco Dolstra
750e2e618a
Merge pull request #770 from NixOS/remove-jobs
Remove the Jobs table
2020-06-01 10:25:41 +02:00
Eelco Dolstra
d5844897da
Merge pull request #771 from knl/fix-unprocessed-notificationpendingsince
Fix: Set notificationpendingsince for dependent builds
2020-05-28 11:15:24 +02:00
Nikola Knezevic
7d52946982 Set notificationpendingsince for dependent builds
`build_finished` Postgres event will never be fired for the dependent builds.

For example, on our Hydra, the following query always returns increasing
numbers, even though all notifications have been delivered:

```
hydra=> select count(1) from builds where notificationpendingsince is not null;
 count
-------
  4583
(1 row)
```

Thus, we have to iterate over all dependent builds and mark their
`notificationpendingsince` as `null`, otherwise they will pile up until
the next restart of hydra-notify, when they will get delivered.
2020-05-28 10:45:41 +02:00
Nikola Knezevic
7148923d30 Make --no-allow-import-from-derivation configurable in hydra-eval-jobset
When deploying Hydra different than hydra.nixos.org one may encounter a problem
as building any job that uses IFD fails with:

May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/1jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/av3jr8ix4qcadq2wm3y3hplvxwzlhl4y-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize
'/nix/store/2jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but
'allow-import-from-derivation' is false"

The recent change enforced passing `--no-allow-import-from-derivation`
to `hydra-eval-job` unconditionally. This change makes it configurable and
defaults to **NOT PASSING IT** -- most of the deployments allow IFDs.

The configuration option is called `allow_import_from_derivation` and
defaults to `true`. It is interpreted as a boolean, with only true option being
`true`.
2020-05-28 10:15:46 +02:00
Eelco Dolstra
8adb433e3b
Remove the Jobs table
This table has been superfluous for a long time.
2020-05-27 20:09:36 +02:00
Eelco Dolstra
af9f635a5a
Use PostgreSQL 11
This is what we use for hydra.nixos.org.
2020-05-27 17:34:14 +02:00
Eelco Dolstra
a1e08d8819
Merge pull request #768 from knl/improve-handling-of-perl-block-eval-errors
Improve handling of Perl's block eval errors
2020-05-27 12:20:25 +02:00
Nikola Knezevic
f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with &quot;1&quot;:
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
Eelco Dolstra
471b036eef
Merge pull request #765 from cransom/foreman-hydra-notify
Add hydra-notify to devshell
2020-05-25 16:36:43 +02:00
Casey Ransom
8a169ffa56 Add hydra-notify to devshell
I came across https://github.com/NixOS/hydra/issues/751 and realized
that hydra-notify is responsible for creating the additional jobsets in
a declarative file. My declarative testing works in dev now.
2020-05-20 15:38:31 -04:00