It now receives notifications about started/finished builds/steps via
PostgreSQL. This gets rid of the (substantial) overhead of starting
hydra-notify for every event. It also allows other programs (even on
other machines) to listen to Hydra notifications.
This adds a `InfluxDBNotification` plugin which is configured as:
```
<influxdb>
url = http://127.0.0.1:8086
db = hydra
</influxdb>
```
which will write a notification for every finished job to the
configured database in InfluxDB looking like:
```
hydra_build_status,cached=false,job=job,jobset=default,project=sample,repo=default,result=success,status=success,system=x86_64-linux build_id="1",build_status=0i,closure_size=584i,duration=0i,main_build_id="1",queued=0i,size=168i 1564156212
```
The creation of the `pg_trgm` extension needs superuser power. So,
this patch makes the extension creation in the Hydra NixOS module when
a local database is used.
If it is not possible to create this extension (remote database for
instance with nosuperuser), the creation of the `pg_trgm` index is
skipped (this index speedup queries on builds.drvpath) and warnings
are emitted:
initialising the Hydra database schema...
WARNING: Can not create extension pg_trgm: permission denied to create extension "pg_trgm"
WARNING: HINT: Temporary provide superuser role to your Hydra Postgresql user and run the script src/sql/upgrade-57.sql
WARNING: The pg_trgm index on builds.drvpath has been skipped (slower complex queries on builds.drvpath)
This allows to keep smooth migrations: the migration process doesn't
require a manual step (but this manual step is recommended on big
remote databases).
The search query uses the LIKE operator which requires a sequential
scan (it can't use the already existing B-tree index). This new
index (trigram) avoids a sequential scan of the builds table when the
LIKE operator is used.
Here is the analyze of a request on the builds table with this index:
explain analyze select * from builds where drvpath like '%k3r71gz0gv16ld8rhcp2bb8gb5w1xc4b%';
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on builds (cost=128.00..132.01 rows=1 width=492) (actual time=0.070..0.077 rows=1 loops=1)
Recheck Cond: (drvpath ~~ '%k3r71gz0gv16ld8rhcp2bb8gb5w1xc4b%'::text)
-> Bitmap Index Scan on indextrgmbuildsondrvpath (cost=0.00..128.00 rows=1 width=0) (actual time=0.047..0.047 rows=3 loops=1)
Index Cond: (drvpath ~~ '%k3r71gz0gv16ld8rhcp2bb8gb5w1xc4b%'::text)
Total runtime: 0.206 ms
(5 rows)
Currently, a full store path has to be provided to search in
builds. This patch permits to search jobs with a output path or
derivation hash.
Usecase: we are building Docker images with Hydra. The tag of the
Docker image is the hash of the image output path. This patch would
allow us to find back the build job from the tag of a running
container image.
This plugin expects as inputs to a jobset the following:
- gitlab_status_repo => Name of the repository input pointing to that
status updates should be POST'ed, i.e. the jobset has a git input
"nixexprs": "https://gitlab.example.com/project/nixexprs", in which
case "gitlab_status_repo" would be "nixexprs".
- gitlab_project_id => ID of the project in Gitlab, i.e. in the above
case the ID in gitlab of "nixexprs"
Without this patch running the following on MacOS:
nix-build release.nix -A build.x86_64-linux
results in the following error during the configuration phase (note that Nix
should be configured with a x86_64-linux build machine):
building '/nix/store/jb6ca1gmplyb69ayd43z7fb0y9npxd53-hydra-0.1.2581.8b5948f4cf12424c04df67a6eb136c9846fb2cfd.drv' on 'ssh://my-linux-build-machine'...
...
checking whether /nix/store/s6bhdppx66bkgf741vk4d29hgsj1h1zp-hydra-perl-deps/bin/nix-store is recent enough... ./configure: line 16254: /nix/store/s6bhdppx66bkgf741vk4d29hgsj1h1zp-hydra-perl-deps/bin/nix-store: cannot execute binary file: Exec format error
no
configure: error: `/nix/store/s6bhdppx66bkgf741vk4d29hgsj1h1zp-hydra-perl-deps/bin/nix-store' doesn't support `--timeout'; please use a newer version.
build time elapsed: 0m1.624s 0m1.774s 0m9.366s 0m6.110s
builder for '/nix/store/jb6ca1gmplyb69ayd43z7fb0y9npxd53-hydra-0.1.2581.8b5948f4cf12424c04df67a6eb136c9846fb2cfd.drv' failed with exit code 1
This problem is that the `nix` dependency of hydra is selected from a nixpkgs
set configured with a default `system` parameter,
i.e. `builtin.currentSystem`. This means that the hydra derivation which is
build for and on Linux depends on the nix derivation build for Darwin.
The fix is to select nix from the nixpkgs set configured with a system specified
by the user.
Prior, tests would all fail to build, causing, roughly, the following
error (roughly, because I added some debug log messages :)):
ok 68 - Evaluating jobs/build-products.nix should result in 2 builds
Queue runner stderr: using 4185024512 bytes for the NAR buffer
locking path '/build/source/tests/data/queue-runner/lock'
lock acquired on '/build/source/tests/data/queue-runner/lock.lock'
warning: unknown setting 'max-connection-age'
warning: unknown setting 'max-connections'
dispatcher woken up
dispatcher woken up
dispatcher sleeping for 7674380800s
adding new machine ‘localhost’
dispatcher woken up
checking the queue for builds > 0...
dispatcher sleeping for 7674380800s
sending notification about build 1
loading build 18 (tests:build-products:simple)
considering derivation ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
sending notification about build 2
creating build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
added build 18 (top-level step /build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv, 1 new steps)
got 1 new runnable steps from 1 new builds
step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ is now runnable
dispatcher woken up
dispatcher sleeping for 7674380800s
performing step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ 1 times on ‘localhost’ (needed by build 18
and 0 others)
sending closure of ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ to ‘localhost’
building ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ on ‘localhost’
killing process 10462
marking build 18 as failed
finishing build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
ok 69 - Build 'simple' from jobs/build-products.nix should exit with code 0
ok 70 - newbuild->finished was '1' instead of 1
not ok 71 - newbuild->buildstatus was '1' instead of 0
not ok 72 - Build 'simple' from jobs/build-products.nix should have buildstatus 0
Can't call method "name" on an undefined value at ./evaluation-tests.pl line 173.
FAIL: evaluation-tests.pl
The hydra-queue-runner opens a connection to the builder. If the
builder is 'localhost' it starts `nix-store`, otherwise it starts
'ssh'.
Currently, if the hydra-queue-runner can not start `nix-store` (not in
the PATH for instance), the error message is:
cannot connect to ‘localhost’: error: cannot start ssh: No such file
or directory
This is not useful since ssh is actually not started:/
With this patch the error message is now:
cannot connect to ‘localhost’: error: cannot start nix-store: No such file
or directory
Maximilian Bosch
Samuel Dionne-Riel