The search query uses the LIKE operator which requires a sequential
scan (it can't use the already existing B-tree index). This new
index (trigram) avoids a sequential scan of the builds table when the
LIKE operator is used.
Here is the analyze of a request on the builds table with this index:
explain analyze select * from builds where drvpath like '%k3r71gz0gv16ld8rhcp2bb8gb5w1xc4b%';
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on builds (cost=128.00..132.01 rows=1 width=492) (actual time=0.070..0.077 rows=1 loops=1)
Recheck Cond: (drvpath ~~ '%k3r71gz0gv16ld8rhcp2bb8gb5w1xc4b%'::text)
-> Bitmap Index Scan on indextrgmbuildsondrvpath (cost=0.00..128.00 rows=1 width=0) (actual time=0.047..0.047 rows=3 loops=1)
Index Cond: (drvpath ~~ '%k3r71gz0gv16ld8rhcp2bb8gb5w1xc4b%'::text)
Total runtime: 0.206 ms
(5 rows)
Currently, a full store path has to be provided to search in
builds. This patch permits to search jobs with a output path or
derivation hash.
Usecase: we are building Docker images with Hydra. The tag of the
Docker image is the hash of the image output path. This patch would
allow us to find back the build job from the tag of a running
container image.
This plugin expects as inputs to a jobset the following:
- gitlab_status_repo => Name of the repository input pointing to that
status updates should be POST'ed, i.e. the jobset has a git input
"nixexprs": "https://gitlab.example.com/project/nixexprs", in which
case "gitlab_status_repo" would be "nixexprs".
- gitlab_project_id => ID of the project in Gitlab, i.e. in the above
case the ID in gitlab of "nixexprs"
Without this patch running the following on MacOS:
nix-build release.nix -A build.x86_64-linux
results in the following error during the configuration phase (note that Nix
should be configured with a x86_64-linux build machine):
building '/nix/store/jb6ca1gmplyb69ayd43z7fb0y9npxd53-hydra-0.1.2581.8b5948f4cf12424c04df67a6eb136c9846fb2cfd.drv' on 'ssh://my-linux-build-machine'...
...
checking whether /nix/store/s6bhdppx66bkgf741vk4d29hgsj1h1zp-hydra-perl-deps/bin/nix-store is recent enough... ./configure: line 16254: /nix/store/s6bhdppx66bkgf741vk4d29hgsj1h1zp-hydra-perl-deps/bin/nix-store: cannot execute binary file: Exec format error
no
configure: error: `/nix/store/s6bhdppx66bkgf741vk4d29hgsj1h1zp-hydra-perl-deps/bin/nix-store' doesn't support `--timeout'; please use a newer version.
build time elapsed: 0m1.624s 0m1.774s 0m9.366s 0m6.110s
builder for '/nix/store/jb6ca1gmplyb69ayd43z7fb0y9npxd53-hydra-0.1.2581.8b5948f4cf12424c04df67a6eb136c9846fb2cfd.drv' failed with exit code 1
This problem is that the `nix` dependency of hydra is selected from a nixpkgs
set configured with a default `system` parameter,
i.e. `builtin.currentSystem`. This means that the hydra derivation which is
build for and on Linux depends on the nix derivation build for Darwin.
The fix is to select nix from the nixpkgs set configured with a system specified
by the user.
Prior, tests would all fail to build, causing, roughly, the following
error (roughly, because I added some debug log messages :)):
ok 68 - Evaluating jobs/build-products.nix should result in 2 builds
Queue runner stderr: using 4185024512 bytes for the NAR buffer
locking path '/build/source/tests/data/queue-runner/lock'
lock acquired on '/build/source/tests/data/queue-runner/lock.lock'
warning: unknown setting 'max-connection-age'
warning: unknown setting 'max-connections'
dispatcher woken up
dispatcher woken up
dispatcher sleeping for 7674380800s
adding new machine ‘localhost’
dispatcher woken up
checking the queue for builds > 0...
dispatcher sleeping for 7674380800s
sending notification about build 1
loading build 18 (tests:build-products:simple)
considering derivation ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
sending notification about build 2
creating build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
added build 18 (top-level step /build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv, 1 new steps)
got 1 new runnable steps from 1 new builds
step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ is now runnable
dispatcher woken up
dispatcher sleeping for 7674380800s
performing step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ 1 times on ‘localhost’ (needed by build 18
and 0 others)
sending closure of ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ to ‘localhost’
building ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ on ‘localhost’
killing process 10462
marking build 18 as failed
finishing build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
ok 69 - Build 'simple' from jobs/build-products.nix should exit with code 0
ok 70 - newbuild->finished was '1' instead of 1
not ok 71 - newbuild->buildstatus was '1' instead of 0
not ok 72 - Build 'simple' from jobs/build-products.nix should have buildstatus 0
Can't call method "name" on an undefined value at ./evaluation-tests.pl line 173.
FAIL: evaluation-tests.pl
The hydra-queue-runner opens a connection to the builder. If the
builder is 'localhost' it starts `nix-store`, otherwise it starts
'ssh'.
Currently, if the hydra-queue-runner can not start `nix-store` (not in
the PATH for instance), the error message is:
cannot connect to ‘localhost’: error: cannot start ssh: No such file
or directory
This is not useful since ssh is actually not started:/
With this patch the error message is now:
cannot connect to ‘localhost’: error: cannot start nix-store: No such file
or directory
Some time ago the data structure for maintainer descriptions in
`nixpkgs` changed from a simple attr set with maintainer emails as
values to an attribute set where the maintainer' nick is associated to
an attribute set with email, GitHub handle and full name.
Hydra can either parse a Nix list or fetches `shortName` from the
associated attribute set (which is used for `meta.licenses` as each
value in it contains a `shortName`). This behavior needs to be
replicated for maintainers to retrieve the emails for `hydra-notify`.
This change is backwards-compatible since `queryMetaStrings` is still
able to understand lists, so old versions of `nixpkgs` or packages using
the old maintainer data structure remain usable.
This is because setting only the initial heap size to more than
the default value (or the configured value) will cause all initial evals
until maxHeapSize expands to the given value to abort.
The 1.1 multiplier comes from the the configured defaults on NixOS' hydra,
and from the previous multiplier used before
7876cf677c.
This is caused by the autoconf check for `nix-store` which is equivalent
to running this:
```
$ nix-store --timeout 123 -q
```
This would open the pager on a 2.1.1 version of nix.
```
$ nix-store --version
nix-store (Nix) 2.1.1
```
Setting `PAGER` to `cat` ensures the pager doesn't block the
configurePhase.
```
$ PAGER=cat nix-store --timeout 123 -q
```
In order to access protected or private repositories. Using the target
repository URL along with the merge-request ref instead of the source
repository url and branch is necessary to avoid running into issues if
the source repository is not actually accessible to the user Hydra is
authenticating as.
Thanks Alexei Robyn for this patch.
Maximilian Bosch
Samuel Dionne-Riel