Commit graph

104 commits

Author SHA1 Message Date
Eelco Dolstra 81ac547d2b
Move most of AddBuilds to hydra-eval-jobset
Having this stuff in a separate module is a remnant of the time when
hydra-server could add builds to the queue directly. This is no longer
the case.
2017-02-21 17:20:48 +01:00
Eelco Dolstra cc866bff1e Fix API tests
POST requests require a referrer now.

http://hydra.nixos.org/build/42775011
2016-10-27 13:15:09 +02:00
Eelco Dolstra e0b2921ff2 Concurrent hydra-evaluator
This rewrites the top-level loop of hydra-evaluator in C++. The Perl
stuff is moved into hydra-eval-jobset. (Rewriting the entire evaluator
would be nice but is a bit too much work.) The new version has some
advantages:

* It can run multiple jobset evaluations in parallel.

* It uses PostgreSQL notifications so it doesn't have to poll the
  database. So if a jobset is triggered via the web interface or from
  a GitHub / Bitbucket webhook, evaluation of the jobset will start
  almost instantaneously (assuming the evaluator is not at its
  concurrency limit).

* It imposes a timeout on evaluations. So if e.g. hydra-eval-jobset
  hangs connecting to a Mercurial server, it will eventually be
  killed.
2016-10-14 14:22:12 +02:00
Eelco Dolstra e94ef4627c Tests: Clear $NIX_REMOTE_SYSTEMS 2016-10-06 17:19:16 +02:00
Eelco Dolstra f2724d088d Test: Show error output from hydra-queue-runner on timeout 2016-10-06 17:18:59 +02:00
Shea Levy 93f9846425 git-rev: Remove superfluous git tag 2016-04-12 16:30:42 -04:00
Shea Levy 10e036238c Add eval test for specific git revision 2016-04-12 16:27:11 -04:00
Eelco Dolstra b1ce76c2b4 Fix test
nix-support/failed is supposed to be a file, not a directory.
2016-02-26 19:54:32 +01:00
Eelco Dolstra 48d6f0de2a Fix the API test 2015-08-12 16:07:19 +02:00
Eelco Dolstra b5815e2aa6 Run PostgreSQL during "make check" 2015-06-25 16:47:39 +02:00
Eelco Dolstra 18a3c3ff1c Update "make check" for the new queue runner
Also, if the machines file contains an entry for localhost, then run
"nix-store --serve" directly, without going through SSH.
2015-06-25 16:47:39 +02:00
Eelco Dolstra d6354cbe1f Fix Perl error in tests.api
Doing "su hydra" causes Perl to be invoked with a wrong PERL5LIB
(pointing to root's profile), leading to "Can't locate strict.pm".
2015-06-15 11:56:51 +02:00
Eelco Dolstra c93ff1e817 Doh 2015-05-26 15:25:21 +02:00
Eelco Dolstra f53e3c738d Fix test
http://hydra.nixos.org/build/14852570
2014-09-26 13:47:42 +02:00
Eelco Dolstra a052bba1bb Fix test 2014-04-24 16:08:04 +02:00
Eelco Dolstra c92410c147 Use hydra-module.nix in the tests 2013-11-06 18:10:52 +01:00
Eelco Dolstra 64cbdddea8 Test whether incorrect passwords are rejected 2013-11-06 16:12:08 +01:00
Eelco Dolstra 8c2acdd294 make clean: Delete darcs-repo 2013-11-06 14:29:55 +00:00
Shea Levy 5da3dcfd57 Make the outputs' outpaths available via the Build JSON API
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-29 14:41:48 -04:00
Shea Levy 4eef090a45 Flesh out the API a bit
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-29 13:54:11 -04:00
Shea Levy 5d3df69a0b Cleanup build serialization
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-25 11:09:11 -04:00
Shea Levy e04477003e Cleanup eval serialization
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-25 11:09:11 -04:00
Shea Levy 917660442e Add eager fetching of relations and enable that for jobset->jobsetinputs
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-25 11:09:11 -04:00
Shea Levy 04d8adaad3 Cleanup Jobset serialization
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-24 11:39:40 -04:00
Shea Levy 78dcd3e619 Cleanup Project model
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-24 11:39:40 -04:00
Shea Levy ee8275cfdb Start api cleanup with the User model
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-24 11:39:40 -04:00
Shea Levy 88217f5ad5 Stupid brittle tests
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-10-14 15:18:04 -04:00
Eelco Dolstra ed132cf1fc Remove obsolete JobStatus source 2013-10-03 15:05:53 +02:00
Shea Levy 1a313ad566 Allow dashes in jobset input names
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-23 17:53:35 -04:00
Shea Levy 74388353b5 Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:

* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
  format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
  meant to represent a directory, you should include the trailing slash,
  e.g. "cache/"). Default "".

After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.

This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.

This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix

Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-18 18:32:58 +02:00
Petr Rockai 66f3e60e2a Add a test for darcs inputs. 2013-09-07 13:28:53 +02:00
Petr Rockai 5a35912956 Add support for darcs repositories. 2013-09-07 13:28:51 +02:00
Eelco Dolstra fda9b66dc7 Doh 2013-08-20 17:37:15 +02:00
Eelco Dolstra edb88ef452 Remove unused ActiveJobs source 2013-08-20 15:22:46 +02:00
Eelco Dolstra d30b49ebac Forgot to commit 2013-08-14 03:50:07 +02:00
Eelco Dolstra 452c8e36d1 Materialize the number of finished builds
The NrBuilds table tracks the value of ‘select count(*) from Builds
where finished = 0’, keeping it up to date via a trigger.  This is
necessary to make the /all page fast, since otherwise it needs to do a
sequential scan on the Builds table.
2013-08-12 20:19:10 +02:00
Eelco Dolstra b47d9814e3 Clear $HYDRA_CONFIG in the tests
Otherwise one might accidentally send out HipChat notifications when
running the tests...
2013-07-19 14:42:34 +02:00
Shea Levy 002ac9ef63 Merge in the first bits of the API work
The catalyst-action-rest branch from shlevy/hydra was an exploration of
using Catalyst::Action::REST to create a JSON API for hydra. This commit
merges in the best bits from that experiment, with the goal that further
API endpoints can be added incrementally.

In addition to migrating more endpoints, there is potential for
improvement in what's already been done:
* The web interface can be updated to use the same non-GET endpoints as
  the JSON interface (using x-tunneled-method) instead of having a
  separate endpoint
* The web rendering should use the $c->stash->{resource} data structure
  where applicable rather than putting the same data in two places in
  the stash
* Which columns to render for each endpoint is a completely debatable
  question
* Hydra::Component::ToJSON should turn has_many relations that have
  strings as their primary keys into objects instead of arrays

Fixes NixOS/hydra#98

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-02 14:00:46 -04:00
Eelco Dolstra d11b08aa51 Fix the tests 2013-03-29 01:34:59 +01:00
Sander van der Burg 3884725bfd Added build product test files 2013-03-20 18:08:25 +01:00
Sander van der Burg 03189bf62b Implemented quoted strings support in hydra-build-products to allow file names with spaces + testcase 2013-03-20 18:05:21 +01:00
Eelco Dolstra fcd511c4de Revert "Remove now-unused SystemTypes table"
This reverts commit 71d020735b.
Unfortunately there are still some cases where we need to set Hydra's
concurrency separately.  (Ideally, Hydra would start *all* queued
builds in parallel and let Nix take care of everything...)
2013-03-05 17:42:16 +01:00
Shea Levy 71d020735b Remove now-unused SystemTypes table 2013-03-04 17:46:37 -05:00
Shea Levy 233e485a55 Remove the BuildMachines and BuildMachinesSystemTypes tables 2013-03-04 16:32:16 -05:00
Eelco Dolstra a77161e40a Allow users to edit their own settings
Also, don't use the flash anymore for going back to the referer.
2013-03-04 15:25:23 +01:00
Shea Levy dba8dc9668 Fix query-all-tables test count, there are more tables now 2013-02-19 14:21:57 +01:00
Eelco Dolstra 58dd49e645 Fix handling of IPC::Run::run exit status
Turns out that the exit status is returned in $?, not as the return
value of run().  So our error checking was completely bogus.
2013-01-23 15:56:28 +01:00
Eelco Dolstra 2d36b75d3e Fix indentation 2013-01-23 12:49:26 +00:00
Eelco Dolstra e4c123ef7c Fix indentation 2013-01-23 12:48:02 +00:00
Eelco Dolstra 67aefde62c Remove trailing whitespace 2013-01-22 14:41:02 +01:00