Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:
* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
meant to represent a directory, you should include the trailing slash,
e.g. "cache/"). Default "".
After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.
This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.
This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix
Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-03 14:53:56 +00:00
|
|
|
use strict;
|
2021-08-19 20:36:43 +00:00
|
|
|
use warnings;
|
Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:
* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
meant to represent a directory, you should include the trailing slash,
e.g. "cache/"). Default "".
After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.
This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.
This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix
Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-03 14:53:56 +00:00
|
|
|
use File::Basename;
|
|
|
|
use Hydra::Model::DB;
|
|
|
|
use Hydra::Helper::Nix;
|
|
|
|
use Nix::Store;
|
|
|
|
use Cwd;
|
|
|
|
|
|
|
|
my $db = Hydra::Model::DB->new;
|
|
|
|
|
|
|
|
use Test::Simple tests => 6;
|
|
|
|
|
|
|
|
$db->resultset('Users')->create({ username => "root", emailaddress => 'root@invalid.org', password => '' });
|
|
|
|
|
|
|
|
$db->resultset('Projects')->create({name => "tests", displayname => "", owner => "root"});
|
|
|
|
my $project = $db->resultset('Projects')->update_or_create({name => "tests", displayname => "", owner => "root"});
|
|
|
|
my $jobset = $project->jobsets->create({name => "basic", nixexprinput => "jobs", nixexprpath => "default.nix", emailoverride => ""});
|
|
|
|
|
|
|
|
my $jobsetinput;
|
|
|
|
|
|
|
|
$jobsetinput = $jobset->jobsetinputs->create({name => "jobs", type => "path"});
|
|
|
|
$jobsetinput->jobsetinputalts->create({altnr => 0, value => getcwd . "/jobs"});
|
2016-10-13 13:53:05 +00:00
|
|
|
system("hydra-eval-jobset " . $jobset->project->name . " " . $jobset->name);
|
Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:
* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
meant to represent a directory, you should include the trailing slash,
e.g. "cache/"). Default "".
After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.
This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.
This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix
Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-03 14:53:56 +00:00
|
|
|
|
|
|
|
my $successful_hash;
|
|
|
|
foreach my $build ($jobset->builds->search({finished => 0})) {
|
|
|
|
system("hydra-build " . $build->id);
|
|
|
|
my @outputs = $build->buildoutputs->all;
|
|
|
|
my $hash = substr basename($outputs[0]->path), 0, 32;
|
|
|
|
if ($build->job->name eq "job") {
|
|
|
|
ok(-e "/tmp/s3/hydra/$hash.nar", "The nar of a successful matched build is uploaded");
|
|
|
|
ok(-e "/tmp/s3/hydra/$hash.narinfo", "The narinfo of a successful matched build is uploaded");
|
|
|
|
$successful_hash = $hash;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
system("hydra-s3-backup-collect-garbage");
|
|
|
|
ok(-e "/tmp/s3/hydra/$successful_hash.nar", "The nar of a build that's a root is not removed by gc");
|
|
|
|
ok(-e "/tmp/s3/hydra/$successful_hash.narinfo", "The narinfo of a build that's a root is not removed by gc");
|
|
|
|
|
|
|
|
my $gcRootsDir = getGCRootsDir;
|
|
|
|
opendir DIR, $gcRootsDir or die;
|
|
|
|
while(readdir DIR) {
|
|
|
|
next if $_ eq "." or $_ eq "..";
|
|
|
|
unlink "$gcRootsDir/$_";
|
|
|
|
}
|
|
|
|
closedir DIR;
|
|
|
|
system("hydra-s3-backup-collect-garbage");
|
|
|
|
ok(not -e "/tmp/s3/hydra/$successful_hash.nar", "The nar of a build that's not a root is removed by gc");
|
|
|
|
ok(not -e "/tmp/s3/hydra/$successful_hash.narinfo", "The narinfo of a build that's not a root is removed by gc");
|