Commit graph

161 commits

Author SHA1 Message Date
Eelco Dolstra 0b5a7472be Add a job for the manual 2016-10-10 15:26:18 +02:00
Eelco Dolstra 633724cc8a services.hydra -> services.hydra-dev
Since Nixpkgs 16.09 contains a Hydra module, we were getting a
conflict. I guess we should have a way to override modules in
Nixpkgs...
2016-10-10 11:29:15 +02:00
Eelco Dolstra 2f0d2f8d91 Bump Nix 2016-10-10 11:18:17 +02:00
Eelco Dolstra fb0d2d2dda Build against the bleeding edge of Nix 2016-10-06 17:26:12 +02:00
Eelco Dolstra 934829a1a9 Remove NetStatsd since it's also in Nixpkgs 2016-10-06 14:50:59 +02:00
Domen Kožar 23bf99b8e2 Use pixz instead of bzip2 for compressing nars
According to following two benchmarks:

- https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/
- http://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

xz has better compression ratio than bzip2 at lowest compression rate.

https://github.com/vasi/pixz has been chosen as it can scale compressing
over multiple cores linearly.

We're using this in snabblab for a month now and it has improved CPU
wise the main Hydra server.
2016-08-12 11:39:07 +02:00
Alexander Ried 2c78a92ac5 release.nix: add missing dependency on nix 2016-05-09 18:35:24 +02:00
Alexander Ried 3f5ccfc7f3 release.nix use makeBinPath to construct PATH
makeBinPath takes care to use the correct output.

nix-repl> lib.makeSearchPath "bin" [pkgs.nix]
"/nix/store/zpp83pr21ihxwsr15l6mkzwkr49zj71d-nix-1.11.2-dev/bin"

nix-repl> lib.makeBinPath [pkgs.nix]
"/nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/bin"
2016-04-29 04:23:54 +02:00
Eelco Dolstra aa4c1fb1ab Fix version 2016-03-22 17:26:50 +01:00
Eelco Dolstra aba2356932 Restore path in nix-shell 2016-03-22 16:59:05 +01:00
Eelco Dolstra 1332463b02 Don't wrap C++ programs 2016-03-22 13:35:09 +01:00
Eelco Dolstra e624652dd8 Use patched aws-sdk-cpp 2016-03-22 13:11:30 +01:00
Eelco Dolstra a727643286 inNixShell considered harmful 2016-03-22 13:10:37 +01:00
Eelco Dolstra 74426e6820 Simplify running nix-shell
This also removes building a separate source tarball or building a PDF
manual since it's unlikely anybody cares.
2016-03-22 12:53:28 +01:00
Eelco Dolstra 2d40888e2e Add an S3-backed binary cache store 2016-02-18 16:18:50 +01:00
Eelco Dolstra 993647d1e3 Use Google's verifier 2016-01-14 12:54:47 +01:00
Eelco Dolstra 5a580b1bb2 Add support for logging in via a Google account
The required configuration in hydra.conf:

  enable_google_login = 1
  google_client_id = 238429sdjkds....apps.googleusercontent.com

and optionally persona_allowed_domains to restrict to one or more
domains.
2016-01-13 17:32:52 +01:00
Eelco Dolstra 1b1fdb3f8d Revert "Don't wrap hydra-queue-runner / hydra-eval-jobs"
This reverts commit f8ee98f455.
2015-10-31 15:32:19 +01:00
Eelco Dolstra f8ee98f455 Don't wrap hydra-queue-runner / hydra-eval-jobs 2015-10-30 18:13:23 +01:00
Eelco Dolstra 53d270c007 Enable debug info 2015-10-30 18:12:30 +01:00
Eelco Dolstra 7b9f488949 Check Nix version 2015-10-14 13:17:24 +02:00
Eelco Dolstra 69e9f73cf6 Use nixUnstable for the tarball
Not needed for building, only for dev-shell.
2015-08-17 11:17:54 +02:00
Eelco Dolstra 48d6f0de2a Fix the API test 2015-08-12 16:07:19 +02:00
Eelco Dolstra 8092149a9f Use nixUnstable 2015-07-30 16:55:11 +02:00
Eelco Dolstra f933762358 Disable building the PDF manual 2015-07-08 16:51:05 +02:00
Eelco Dolstra 296213e90a Drop obsolete graphviz dependency 2015-07-08 16:30:58 +02:00
Eelco Dolstra b5815e2aa6 Run PostgreSQL during "make check" 2015-06-25 16:47:39 +02:00
Eelco Dolstra 62219adaf3 Send queue runner stats to statsd
This is currently done by a separate program that periodically
calls "hydra-queue-runner --status". Eventually, I'll do this
in the queue runner directly.

Fixes #220.
2015-06-23 14:56:43 +02:00
Eelco Dolstra a757b783f4 Update $PATH 2015-06-22 16:15:16 +02:00
Eelco Dolstra a0c4120232 Don't copy src for nix-shell 2015-06-19 17:45:11 +02:00
Eelco Dolstra a984c0badc Merge branch 'master' into build-ng 2015-06-15 18:21:07 +02:00
Eelco Dolstra bf87d3a6ed Use stable Nix 2015-06-15 14:51:38 +02:00
Eelco Dolstra d6354cbe1f Fix Perl error in tests.api
Doing "su hydra" causes Perl to be invoked with a wrong PERL5LIB
(pointing to root's profile), leading to "Can't locate strict.pm".
2015-06-15 11:56:51 +02:00
Eelco Dolstra abca7a87da Cleanup 2015-06-12 18:15:58 +02:00
Eelco Dolstra f06ec78859 Handle building from a dirty Git tree 2015-06-12 18:15:58 +02:00
Eelco Dolstra 18e0a62e24 Disable 32-bit builds again
They're failing consistently:

http://hydra.nixos.org/job/hydra/master/build.i686-linux/all
2015-06-12 15:35:14 +02:00
Eelco Dolstra dc446c3980 Start of single-process hydra-queue-runner 2015-05-28 17:39:29 +02:00
Eelco Dolstra c93ff1e817 Doh 2015-05-26 15:25:21 +02:00
Chris Warburton d64ba15357 Allow running on i686-linux 2015-05-17 18:41:27 +01:00
Eelco Dolstra 8523130ebb Use Email::MIME instead of Email::Simple
Email::Simple cannot handle non-ASCII characters.

Fixes #191.
2014-11-19 14:45:46 +01:00
Eelco Dolstra d28ee3fd3f Remove Term::Size::Any dependency
This is now propagated by Catalyst::Runtime.
2014-11-12 13:49:04 +01:00
Eelco Dolstra eb2965071e Depend on Term::Size::Any
This shuts up a warning from Catalyst.
2014-11-06 15:36:46 +01:00
Eelco Dolstra 594fb7a009 Disable the S3 backup test
This test has never succeeded
(http://hydra.nixos.org/job/hydra/master/tests.s3backup.x86_64-linux)
so until somebody fixes it, there is not much point in building it.
2014-09-24 15:21:01 +02:00
Eelco Dolstra 4727165832 Use pkgconfig to find Nix 2014-09-18 12:24:05 +02:00
Eelco Dolstra 1306291c6c Handle utf-8 properly
Fixes errors like:

  Caught exception in engine "Wide character in syswrite at /nix/store/498lwsrn5kkdh1q8kn3vcpd3457w6m7a-hydra-perl-deps/lib/perl5/site_perl/5.16.3/Starman/Server.pm line 547."

Note that these errors didn't happen if the database encoding was set
to SQL_ASCII (which was the case for hydra.nixos.org, explaining why
it didn't get these errors). However, now the encoding must be
UTF8. To change it, do:

  update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'hydra';
2014-05-01 16:33:25 +02:00
Eelco Dolstra c92410c147 Use hydra-module.nix in the tests 2013-11-06 18:10:52 +01:00
Eelco Dolstra bb2976693c Use hydra-init / hydra-create-user in the tests 2013-11-06 16:12:07 +01:00
Eelco Dolstra 0fd6bb4e3f Drop building on i686-linux
Nobody cares.
2013-11-05 00:09:33 +01:00
Eelco Dolstra 0babdf3532 Adjust to the NixOS/Nixpkgs merge 2013-10-11 10:58:34 +02:00
Shea Levy 74388353b5 Add a plugin for backing up builds in s3
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:

* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
  format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
  meant to represent a directory, you should include the trailing slash,
  e.g. "cache/"). Default "".

After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.

This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.

This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix

Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-18 18:32:58 +02:00