Commit graph

114 commits

Author SHA1 Message Date
Eelco Dolstra 8adb433e3b
Remove the Jobs table
This table has been superfluous for a long time.
2020-05-27 20:09:36 +02:00
Maximilian Bosch efcbc08686
Get rid of dependency to SQLite
SQLite isn't properly supported by Hydra for a few years now[1], but
Hydra still depends on it. Apart from a slightly bigger closure this can
cause confusion by users since Hydra picks up SQLite rather than
PostgreSQL by default if HYDRA_DBI isn't configured properly[2]

[1] 78974abb69
[2] https://logs.nix.samueldr.com/nixos-dev/2020-04-10#3297342;
2020-04-16 00:42:40 +02:00
Eelco Dolstra 53e1294825 Fix Mercurial tests
In Nixpkgs 20.03, Mercurial fails if PYTHONPATH is set:

  $ hg
  Traceback (most recent call last):
    File "/nix/store/q7s856v6nw4dffdrm9k3w38qs35i8kr3-mercurial-5.2.2/bin/..hg-wrapped-wrapped", line 37, in <module>
      dispatch.run()
    File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/importlib/util.py", line 245, in __getattribute__
      self.__spec__.loader.exec_module(self)
    File "<frozen importlib._bootstrap_external>", line 728, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/nix/store/q7s856v6nw4dffdrm9k3w38qs35i8kr3-mercurial-5.2.2/lib/python3.7/site-packages/mercurial/dispatch.py", line 10, in <module>
      import difflib
    File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/difflib.py", line 1084, in <module>
      import re
    File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/re.py", line 143, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'

(cherry picked from commit 4009d4295e)
2020-03-31 22:19:19 +02:00
Nikola Knezevic 06bdc8f85c Added the InfluxDBNotification plugin including a NixOS test
This adds a `InfluxDBNotification` plugin which is configured as:

```
<influxdb>
  url = http://127.0.0.1:8086
  db = hydra
</influxdb>
```

which will write a notification for every finished job to the
configured database in InfluxDB looking like:

```
hydra_build_status,cached=false,job=job,jobset=default,project=sample,repo=default,result=success,status=success,system=x86_64-linux build_id="1",build_status=0i,closure_size=584i,duration=0i,main_build_id="1",queued=0i,size=168i 1564156212
```
2019-07-26 17:47:03 +02:00
danbst 4171ab4c4f tests: change postgresql socket dir to /tmp
In https://github.com/NixOS/nixpkgs/pull/57677 default postgresql socket directory was changed
to `/run/postgresql`, which doesn't exist (and can't be created) in
Nix build environment. We'll use /tmp as socket dir explicitly then.

Fixes build failure https://hydra.nixos.org/build/91221682

Cc @aszlig @edolstra
2019-03-29 23:46:38 +02:00
Graham Christensen 215ca5da9c
Merge pull request #607 from nlewo/json-search
Add JSON search API endpoint
2019-03-18 15:08:32 -05:00
Graham Christensen 4160132278 tests: disable sandbox
Prior, tests would all fail to build, causing, roughly, the following
error (roughly, because I added some debug log messages :)):

    ok 68 - Evaluating jobs/build-products.nix should result in 2 builds
    Queue runner stderr: using 4185024512 bytes for the NAR buffer
    locking path '/build/source/tests/data/queue-runner/lock'
    lock acquired on '/build/source/tests/data/queue-runner/lock.lock'
    warning: unknown setting 'max-connection-age'
    warning: unknown setting 'max-connections'
    dispatcher woken up
    dispatcher woken up
    dispatcher sleeping for 7674380800s
    adding new machine ‘localhost’
    dispatcher woken up
    checking the queue for builds > 0...
    dispatcher sleeping for 7674380800s
    sending notification about build 1
    loading build 18 (tests:build-products:simple)
    considering derivation ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
    sending notification about build 2
    creating build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’
    added build 18 (top-level step /build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv, 1 new steps)
    got 1 new runnable steps from 1 new builds
    step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ is now runnable
    dispatcher woken up
    dispatcher sleeping for 7674380800s
    performing step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ 1 times on ‘localhost’ (needed by build 18
    and 0 others)
    sending closure of ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ to ‘localhost’
    building ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’ on ‘localhost’
    killing process 10462
    marking build 18 as failed
    finishing build step ‘/build/source/tests/nix/store/24h0i450d4k00a4jhhk6r7qpqdvzskw6-build-product-simple.drv’

    ok 69 - Build 'simple' from jobs/build-products.nix should exit with code 0
    ok 70 - newbuild->finished was '1' instead of 1
    not ok 71 - newbuild->buildstatus was '1' instead of 0
    not ok 72 - Build 'simple' from jobs/build-products.nix should have buildstatus 0
    Can't call method "name" on an undefined value at ./evaluation-tests.pl line 173.
    FAIL: evaluation-tests.pl
2019-02-01 04:31:11 +01:00
Antoine Eiche 0d2a2d8923 Add json output for the search API endpoint
This commit also add a test of this feature.

Note the search JSON output doesn't contain any jobs because they can
not be exported to JSON yet.

The JSON output on a search query matching a build looks like:

```
{
  "builds": [
    {
      "buildoutputs": {
        "out": {
          "path": "/nix/store/wdag3pznrvqk01byk989irg7rq3q2a2c-job"
        }
      },
      "finished": 0,
      "releasename": null,
      "starttime": null,
      "project": "sample",
      "buildproducts": {},
      "timestamp": 1541007629,
      "buildstatus": null,
      "nixname": "job",
      "drvpath": "/nix/store/n9zqndn7j7nyr6gg3bmxvw26cfmdwv2n-job.drv",
      "job": "job",
      "id": 1,
      "stoptime": null,
      "priority": 100,
      "system": "x86_64-linux",
      "jobsetevals": [
        1
      ],
      "jobset": "default",
      "buildmetrics": {}
    }
  ],
  "projects": [],
  "jobsets": [],
  "buildsdrv": []
}
```
2018-11-01 09:31:15 +01:00
Antoine Eiche dd1e4a92c0 api-test: make created project and jobset visible 2018-10-31 18:51:02 +01:00
Shea Levy c40077ece1 Fix tests 2017-05-24 10:00:38 -04:00
Eelco Dolstra 81ac547d2b
Move most of AddBuilds to hydra-eval-jobset
Having this stuff in a separate module is a remnant of the time when
hydra-server could add builds to the queue directly. This is no longer
the case.
2017-02-21 17:20:48 +01:00
Eelco Dolstra cc866bff1e Fix API tests
POST requests require a referrer now.

http://hydra.nixos.org/build/42775011
2016-10-27 13:15:09 +02:00
Eelco Dolstra e0b2921ff2 Concurrent hydra-evaluator
This rewrites the top-level loop of hydra-evaluator in C++. The Perl
stuff is moved into hydra-eval-jobset. (Rewriting the entire evaluator
would be nice but is a bit too much work.) The new version has some
advantages:

* It can run multiple jobset evaluations in parallel.

* It uses PostgreSQL notifications so it doesn't have to poll the
  database. So if a jobset is triggered via the web interface or from
  a GitHub / Bitbucket webhook, evaluation of the jobset will start
  almost instantaneously (assuming the evaluator is not at its
  concurrency limit).

* It imposes a timeout on evaluations. So if e.g. hydra-eval-jobset
  hangs connecting to a Mercurial server, it will eventually be
  killed.
2016-10-14 14:22:12 +02:00
Eelco Dolstra e94ef4627c Tests: Clear $NIX_REMOTE_SYSTEMS 2016-10-06 17:19:16 +02:00
Eelco Dolstra f2724d088d Test: Show error output from hydra-queue-runner on timeout 2016-10-06 17:18:59 +02:00
Shea Levy 93f9846425 git-rev: Remove superfluous git tag 2016-04-12 16:30:42 -04:00
Shea Levy 10e036238c Add eval test for specific git revision 2016-04-12 16:27:11 -04:00
Eelco Dolstra b1ce76c2b4 Fix test
nix-support/failed is supposed to be a file, not a directory.
2016-02-26 19:54:32 +01:00
Eelco Dolstra 48d6f0de2a Fix the API test 2015-08-12 16:07:19 +02:00
Eelco Dolstra b5815e2aa6 Run PostgreSQL during "make check" 2015-06-25 16:47:39 +02:00
Eelco Dolstra 18a3c3ff1c Update "make check" for the new queue runner
Also, if the machines file contains an entry for localhost, then run
"nix-store --serve" directly, without going through SSH.
2015-06-25 16:47:39 +02:00
Eelco Dolstra d6354cbe1f Fix Perl error in tests.api
Doing "su hydra" causes Perl to be invoked with a wrong PERL5LIB
(pointing to root's profile), leading to "Can't locate strict.pm".
2015-06-15 11:56:51 +02:00
Eelco Dolstra c93ff1e817 Doh 2015-05-26 15:25:21 +02:00
Eelco Dolstra f53e3c738d Fix test
http://hydra.nixos.org/build/14852570
2014-09-26 13:47:42 +02:00
Eelco Dolstra a052bba1bb Fix test 2014-04-24 16:08:04 +02:00
Eelco Dolstra c92410c147 Use hydra-module.nix in the tests 2013-11-06 18:10:52 +01:00
Eelco Dolstra 64cbdddea8 Test whether incorrect passwords are rejected 2013-11-06 16:12:08 +01:00
Eelco Dolstra 8c2acdd294 make clean: Delete darcs-repo 2013-11-06 14:29:55 +00:00
Shea Levy 5da3dcfd57 Make the outputs' outpaths available via the Build JSON API
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-29 14:41:48 -04:00
Shea Levy 4eef090a45 Flesh out the API a bit
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-29 13:54:11 -04:00
Shea Levy 5d3df69a0b Cleanup build serialization
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-25 11:09:11 -04:00
Shea Levy e04477003e Cleanup eval serialization
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-25 11:09:11 -04:00
Shea Levy 917660442e Add eager fetching of relations and enable that for jobset->jobsetinputs
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-25 11:09:11 -04:00
Shea Levy 04d8adaad3 Cleanup Jobset serialization
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-24 11:39:40 -04:00
Shea Levy 78dcd3e619 Cleanup Project model
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-24 11:39:40 -04:00
Shea Levy ee8275cfdb Start api cleanup with the User model
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-24 11:39:40 -04:00
Shea Levy 88217f5ad5 Stupid brittle tests
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-14 15:18:04 -04:00
Eelco Dolstra ed132cf1fc Remove obsolete JobStatus source 2013-10-03 15:05:53 +02:00
Shea Levy 1a313ad566 Allow dashes in jobset input names
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-23 17:53:35 -04:00
Shea Levy 74388353b5 Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:

* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
  format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
  meant to represent a directory, you should include the trailing slash,
  e.g. "cache/"). Default "".

After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.

This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.

This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix

Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-18 18:32:58 +02:00
Petr Rockai 66f3e60e2a Add a test for darcs inputs. 2013-09-07 13:28:53 +02:00
Petr Rockai 5a35912956 Add support for darcs repositories. 2013-09-07 13:28:51 +02:00
Eelco Dolstra fda9b66dc7 Doh 2013-08-20 17:37:15 +02:00
Eelco Dolstra edb88ef452 Remove unused ActiveJobs source 2013-08-20 15:22:46 +02:00
Eelco Dolstra d30b49ebac Forgot to commit 2013-08-14 03:50:07 +02:00
Eelco Dolstra 452c8e36d1 Materialize the number of finished builds
The NrBuilds table tracks the value of ‘select count(*) from Builds
where finished = 0’, keeping it up to date via a trigger.  This is
necessary to make the /all page fast, since otherwise it needs to do a
sequential scan on the Builds table.
2013-08-12 20:19:10 +02:00
Eelco Dolstra b47d9814e3 Clear $HYDRA_CONFIG in the tests
Otherwise one might accidentally send out HipChat notifications when
running the tests...
2013-07-19 14:42:34 +02:00
Shea Levy 002ac9ef63 Merge in the first bits of the API work
The catalyst-action-rest branch from shlevy/hydra was an exploration of
using Catalyst::Action::REST to create a JSON API for hydra. This commit
merges in the best bits from that experiment, with the goal that further
API endpoints can be added incrementally.

In addition to migrating more endpoints, there is potential for
improvement in what's already been done:
* The web interface can be updated to use the same non-GET endpoints as
  the JSON interface (using x-tunneled-method) instead of having a
  separate endpoint
* The web rendering should use the $c->stash->{resource} data structure
  where applicable rather than putting the same data in two places in
  the stash
* Which columns to render for each endpoint is a completely debatable
  question
* Hydra::Component::ToJSON should turn has_many relations that have
  strings as their primary keys into objects instead of arrays

Fixes NixOS/hydra#98

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-02 14:00:46 -04:00
Eelco Dolstra d11b08aa51 Fix the tests 2013-03-29 01:34:59 +01:00
Sander van der Burg 3884725bfd Added build product test files 2013-03-20 18:08:25 +01:00