Commit graph

17 commits

Author SHA1 Message Date
Shea Levy 74388353b5 Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:

* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
  format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
  meant to represent a directory, you should include the trailing slash,
  e.g. "cache/"). Default "".

After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.

This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.

This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix

Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-18 18:32:58 +02:00
Eelco Dolstra a77161e40a Allow users to edit their own settings
Also, don't use the flash anymore for going back to the referer.
2013-03-04 15:25:23 +01:00
Eelco Dolstra 58dd49e645 Fix handling of IPC::Run::run exit status
Turns out that the exit status is returned in $?, not as the return
value of run().  So our error checking was completely bogus.
2013-01-23 15:56:28 +01:00
Eelco Dolstra 2d36b75d3e Fix indentation 2013-01-23 12:49:26 +00:00
Eelco Dolstra 67aefde62c Remove trailing whitespace 2013-01-22 14:41:02 +01:00
Eelco Dolstra 854513be8e I should test first 2012-04-30 17:57:13 +02:00
Nicolas Pierron 69f9175986 Add tests for "git describe --long".
* Move test logic to scm-update.sh scripts.
    * Check for "git describe --long".
2012-04-29 20:59:43 -07:00
Nicolas Pierron 19d9955e89 Optimize fetch-git. 2012-04-14 18:17:35 -07:00
Eelco Dolstra fa62c8b7f2 Fix tests 2012-03-13 13:18:40 +01:00
Eelco Dolstra 93efae0c23 Fix "make check" 2012-03-12 21:13:50 +01:00
Eelco Dolstra 12a7b5737d Test environment cleanups
In particular, clear $NIX_BUILD_HOOK to prevent tests
from failing mysteriously.  Also print stdout/stderr
output of hydra-evaluator.
2011-12-04 22:50:03 +01:00
Rob Vermaas ae26e6db2a fix assignment in evalSucceeds (always returned 3, therefore evalSucceeds always succeeded 2011-12-01 08:31:07 -05:00
Eelco Dolstra dd6e5d49ba Fix the test 2011-11-30 18:14:48 +01:00
Rob Vermaas 55b392a20d wrong altnr for jobset inputs in tests 2011-03-21 12:30:47 +00:00
Rob Vermaas 9f33d4c98d 2011-03-17 13:25:27 +00:00
Rob Vermaas 597e7c04e1 cleanup 2011-03-17 11:55:39 +00:00
Rob Vermaas 890a7867b4 add first evaluations tests 2011-03-16 13:18:12 +00:00