This makes the test faster (by removing it and replacing it with a
`TestScmInput` module that exports the `testScmInput` subroutine). Now,
all the input tests can be run in parallel.
Some of the `tests/jobs/*-update.sh` scripts were "broken" (e.g. tests
failed for various reasons on my machine), so I fixed those up as well.
Co-authored-by: gustavderdrache <gustavderdrache@gmail.com>
PathInput plugin keeps a cache of path evaluations. This cache is simple, and
path is not checked more than once every N seconds, where N=30. The caching is
there to avoid expensive calls to `nix-store --add`.
This change makes the validity period configurable. The main use case is
`api-test.pl` which was implemented wrong for a while, as the invocation of
`hydra-eval-jobset` would return the previous evaluation, claiming there are no
changes. The test has been fixed to check better for a new evaluation.
SQLite isn't properly supported by Hydra for a few years now[1], but
Hydra still depends on it. Apart from a slightly bigger closure this can
cause confusion by users since Hydra picks up SQLite rather than
PostgreSQL by default if HYDRA_DBI isn't configured properly[2]
[1] 78974abb69
[2] https://logs.nix.samueldr.com/nixos-dev/2020-04-10#3297342;
In Nixpkgs 20.03, Mercurial fails if PYTHONPATH is set:
$ hg
Traceback (most recent call last):
File "/nix/store/q7s856v6nw4dffdrm9k3w38qs35i8kr3-mercurial-5.2.2/bin/..hg-wrapped-wrapped", line 37, in <module>
dispatch.run()
File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/importlib/util.py", line 245, in __getattribute__
self.__spec__.loader.exec_module(self)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/nix/store/q7s856v6nw4dffdrm9k3w38qs35i8kr3-mercurial-5.2.2/lib/python3.7/site-packages/mercurial/dispatch.py", line 10, in <module>
import difflib
File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/difflib.py", line 1084, in <module>
import re
File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/re.py", line 143, in <module>
class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
(cherry picked from commit 4009d4295e)
This adds a `InfluxDBNotification` plugin which is configured as:
```
<influxdb>
url = http://127.0.0.1:8086
db = hydra
</influxdb>
```
which will write a notification for every finished job to the
configured database in InfluxDB looking like:
```
hydra_build_status,cached=false,job=job,jobset=default,project=sample,repo=default,result=success,status=success,system=x86_64-linux build_id="1",build_status=0i,closure_size=584i,duration=0i,main_build_id="1",queued=0i,size=168i 1564156212
```
Prior, tests would all fail to build, causing, roughly, the following
error (roughly, because I added some debug log messages :)):
ok 68 - Evaluating jobs/build-products.nix should result in 2 builds
Queue runner stderr: using 4185024512 bytes for the NAR buffer
locking path '/build/source/tests/data/queue-runner/lock'
lock acquired on '/build/source/tests/data/queue-runner/lock.lock'
warning: unknown setting 'max-connection-age'
warning: unknown setting 'max-connections'
dispatcher woken up
dispatcher woken up
dispatcher sleeping for 7674380800s
adding new machine ‘localhost’
dispatcher woken up
checking the queue for builds > 0...
dispatcher sleeping for 7674380800s
sending notification about build 1
loading build 18 (tests:build-products:simple)
considering derivation ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
sending notification about build 2
creating build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
added build 18 (top-level step /build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv, 1 new steps)
got 1 new runnable steps from 1 new builds
step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ is now runnable
dispatcher woken up
dispatcher sleeping for 7674380800s
performing step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ 1 times on ‘localhost’ (needed by build 18
and 0 others)
sending closure of ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ to ‘localhost’
building ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ on ‘localhost’
killing process 10462
marking build 18 as failed
finishing build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
ok 69 - Build 'simple' from jobs/build-products.nix should exit with code 0
ok 70 - newbuild->finished was '1' instead of 1
not ok 71 - newbuild->buildstatus was '1' instead of 0
not ok 72 - Build 'simple' from jobs/build-products.nix should have buildstatus 0
Can't call method "name" on an undefined value at ./evaluation-tests.pl line 173.
FAIL: evaluation-tests.pl
This rewrites the top-level loop of hydra-evaluator in C++. The Perl
stuff is moved into hydra-eval-jobset. (Rewriting the entire evaluator
would be nice but is a bit too much work.) The new version has some
advantages:
* It can run multiple jobset evaluations in parallel.
* It uses PostgreSQL notifications so it doesn't have to poll the
database. So if a jobset is triggered via the web interface or from
a GitHub / Bitbucket webhook, evaluation of the jobset will start
almost instantaneously (assuming the evaluator is not at its
concurrency limit).
* It imposes a timeout on evaluations. So if e.g. hydra-eval-jobset
hangs connecting to a Mercurial server, it will eventually be
killed.
Maximilian Bosch