Commit graph

76 commits

Author SHA1 Message Date
Shea Levy 1a313ad566 Allow dashes in jobset input names
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-23 17:53:35 -04:00
Shea Levy 74388353b5 Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:

* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
  format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
  meant to represent a directory, you should include the trailing slash,
  e.g. "cache/"). Default "".

After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.

This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.

This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix

Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-18 18:32:58 +02:00
Petr Rockai 66f3e60e2a Add a test for darcs inputs. 2013-09-07 13:28:53 +02:00
Petr Rockai 5a35912956 Add support for darcs repositories. 2013-09-07 13:28:51 +02:00
Eelco Dolstra fda9b66dc7 Doh 2013-08-20 17:37:15 +02:00
Eelco Dolstra edb88ef452 Remove unused ActiveJobs source 2013-08-20 15:22:46 +02:00
Eelco Dolstra d30b49ebac Forgot to commit 2013-08-14 03:50:07 +02:00
Eelco Dolstra 452c8e36d1 Materialize the number of finished builds
The NrBuilds table tracks the value of ‘select count(*) from Builds
where finished = 0’, keeping it up to date via a trigger.  This is
necessary to make the /all page fast, since otherwise it needs to do a
sequential scan on the Builds table.
2013-08-12 20:19:10 +02:00
Eelco Dolstra b47d9814e3 Clear $HYDRA_CONFIG in the tests
Otherwise one might accidentally send out HipChat notifications when
running the tests...
2013-07-19 14:42:34 +02:00
Shea Levy 002ac9ef63 Merge in the first bits of the API work
The catalyst-action-rest branch from shlevy/hydra was an exploration of
using Catalyst::Action::REST to create a JSON API for hydra. This commit
merges in the best bits from that experiment, with the goal that further
API endpoints can be added incrementally.

In addition to migrating more endpoints, there is potential for
improvement in what's already been done:
* The web interface can be updated to use the same non-GET endpoints as
  the JSON interface (using x-tunneled-method) instead of having a
  separate endpoint
* The web rendering should use the $c->stash->{resource} data structure
  where applicable rather than putting the same data in two places in
  the stash
* Which columns to render for each endpoint is a completely debatable
  question
* Hydra::Component::ToJSON should turn has_many relations that have
  strings as their primary keys into objects instead of arrays

Fixes NixOS/hydra#98

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-07-02 14:00:46 -04:00
Eelco Dolstra d11b08aa51 Fix the tests 2013-03-29 01:34:59 +01:00
Sander van der Burg 3884725bfd Added build product test files 2013-03-20 18:08:25 +01:00
Sander van der Burg 03189bf62b Implemented quoted strings support in hydra-build-products to allow file names with spaces + testcase 2013-03-20 18:05:21 +01:00
Eelco Dolstra fcd511c4de Revert "Remove now-unused SystemTypes table"
This reverts commit 71d020735b.
Unfortunately there are still some cases where we need to set Hydra's
concurrency separately.  (Ideally, Hydra would start *all* queued
builds in parallel and let Nix take care of everything...)
2013-03-05 17:42:16 +01:00
Shea Levy 71d020735b Remove now-unused SystemTypes table 2013-03-04 17:46:37 -05:00
Shea Levy 233e485a55 Remove the BuildMachines and BuildMachinesSystemTypes tables 2013-03-04 16:32:16 -05:00
Eelco Dolstra a77161e40a Allow users to edit their own settings
Also, don't use the flash anymore for going back to the referer.
2013-03-04 15:25:23 +01:00
Shea Levy dba8dc9668 Fix query-all-tables test count, there are more tables now 2013-02-19 14:21:57 +01:00
Eelco Dolstra 58dd49e645 Fix handling of IPC::Run::run exit status
Turns out that the exit status is returned in $?, not as the return
value of run().  So our error checking was completely bogus.
2013-01-23 15:56:28 +01:00
Eelco Dolstra 2d36b75d3e Fix indentation 2013-01-23 12:49:26 +00:00
Eelco Dolstra e4c123ef7c Fix indentation 2013-01-23 12:48:02 +00:00
Eelco Dolstra 67aefde62c Remove trailing whitespace 2013-01-22 14:41:02 +01:00
Rob Vermaas 209c51ca6c * Remove accidentally commited foo/bar. 2012-06-25 13:37:06 +02:00
Rob Vermaas 9a7811aa92 * Add 'set -e' to all scm test setup scripts, to catch errors as early as possible
* Fix git setup
2012-06-25 13:33:36 +02:00
Rob Vermaas 478d3863bb Add bar with 1 2012-06-25 13:00:55 +02:00
Rob Vermaas ebececb0ee Increase depth 2 2012-06-25 12:24:03 +02:00
Rob Vermaas 9e8d3cdcc5 Increase depth 1 2012-06-25 12:24:03 +02:00
Rob Vermaas 2cdcbd587a Increase depth 0 2012-06-25 12:24:03 +02:00
Eelco Dolstra 854513be8e I should test first 2012-04-30 17:57:13 +02:00
Nicolas Pierron 69f9175986 Add tests for "git describe --long".
* Move test logic to scm-update.sh scripts.
    * Check for "git describe --long".
2012-04-29 20:59:43 -07:00
Rob Vermaas b49e138e9c Merge pull request #3 from nbp/candidates/fetch-git
Optimize fetch-git.
2012-04-24 10:18:28 -07:00
Eelco Dolstra 0446282a63 Fix the test 2012-04-15 22:56:15 +00:00
Nicolas Pierron 19d9955e89 Optimize fetch-git. 2012-04-14 18:17:35 -07:00
Eelco Dolstra fa62c8b7f2 Fix tests 2012-03-13 13:18:40 +01:00
Eelco Dolstra 93efae0c23 Fix "make check" 2012-03-12 21:13:50 +01:00
Eelco Dolstra 3f1bbf7ef4 Better fix 2012-03-12 21:06:42 +01:00
Eelco Dolstra 259afadbab Set $PERL5LIB properly in hydra-init in "make check" 2012-03-12 20:56:41 +01:00
Eelco Dolstra 541238030d Provide a command ‘hydra-init’ to initialise/upgrade the database
For schema upgrades, hydra-init executes the files
src/sql/upgrade-<N>.sql, each of which upgrades the schema from
version N-1 to N.  The upgrades are wrapped in a transaction.
2012-02-28 20:16:16 +01:00
Eelco Dolstra 09e26a3011 Fix broken test 2011-12-05 15:21:43 +01:00
Eelco Dolstra 12a7b5737d Test environment cleanups
In particular, clear $NIX_BUILD_HOOK to prevent tests
from failing mysteriously.  Also print stdout/stderr
output of hydra-evaluator.
2011-12-04 22:50:03 +01:00
Rob Vermaas ae26e6db2a fix assignment in evalSucceeds (always returned 3, therefore evalSucceeds always succeeded 2011-12-01 08:31:07 -05:00
Eelco Dolstra dd6e5d49ba Fix the test 2011-11-30 18:14:48 +01:00
Eelco Dolstra 82d17a2d0b Rename hydra_*.pl to hydra-*
The underscores are ugly and the .pl extension is an implementation
detail that shouldn't be visible to the outside.

Also, get rid of the *.in files.  It's not really necessary to
generate them.  And I was always modifying the wrong file.
2011-11-30 18:14:48 +01:00
Rob Vermaas a9f422dcff add NIX_MANIFESTS_DIR to tests 2011-04-19 11:55:49 +00:00
Rob Vermaas 385b70c898 hydra: fixed and re-enabled evaluation tests 2011-03-24 13:45:01 +00:00
Rob Vermaas 2c9be852cb disabled evaluation tests until i figure out why it fails in buildfarm and succeeds locally 2011-03-21 12:36:01 +00:00
Rob Vermaas 55b392a20d wrong altnr for jobset inputs in tests 2011-03-21 12:30:47 +00:00
Rob Vermaas 9f33d4c98d 2011-03-17 13:25:27 +00:00
Rob Vermaas 10e10e8a8c set executable 2011-03-17 12:49:39 +00:00
Rob Vermaas 597e7c04e1 cleanup 2011-03-17 11:55:39 +00:00