snaakey/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
2297 commits 1 branch 0 tags 20 MiB
Commit graph

1920 commits

Author SHA1 Message Date
Eelco Dolstra ddc9f3cc6a Temporarily disable machines on any exception, not just connection failures 2016-03-22 16:54:40 +01:00
Eelco Dolstra 0aecd65e59 /queue-runner-status: Include info about temporarily disabled machines 2016-03-22 16:54:06 +01:00
Eelco Dolstra ac23bd1539 Revert "Apply IndexBuildsOnJobFinishedId to unfinished builds only" 
This reverts commit 1de5ce7a0e.
 2016-03-16 17:04:20 +01:00
Eelco Dolstra 7d8bf1b0f2 Shorten host names 2016-03-16 15:23:56 +01:00
Eelco Dolstra d5cffd4bc7 Make "Running builds" and "Machine status" pages faster 2016-03-16 15:19:18 +01:00
Eelco Dolstra 1de5ce7a0e Apply IndexBuildsOnJobFinishedId to unfinished builds only 2016-03-16 15:17:10 +01:00
Eelco Dolstra 520c8a5826 Use faster query to determine number of running builds 
The previous query

  select count(*) from builds b left join buildsteps s on s.build = b.id where busy = 1 and finished = 0

is suddenly taking several minutes. Probably PostgreSQL decided to use
a suboptimal query plan.
 2016-03-16 13:41:43 +01:00
Eelco Dolstra 405a43c171 Queue summary: Make rows clickable 2016-03-10 16:48:06 +01:00
Eelco Dolstra 5535bc28ca Tweak 2016-03-10 16:46:15 +01:00
Eelco Dolstra 60e7930d2b Bump memory limit a bit 2016-03-10 16:46:01 +01:00
Eelco Dolstra 75e7b35477 Fix retry of transient failures 2016-03-10 16:44:26 +01:00
Eelco Dolstra de71d5b622 Fix showing machine name for aborted build steps 2016-03-10 16:42:36 +01:00
Eelco Dolstra 33da40f272 Doh 2016-03-09 17:31:57 +01:00
Eelco Dolstra 4b9c76e502 hydra-queue-runner: Ensure regular status dumps 2016-03-09 17:11:34 +01:00
Eelco Dolstra 4151be7e69 Make the output size limit configurable 
The maximum output size per build step (as the sum of the NARs of each
output) can be set via hydra.conf, e.g.

  max-output-size = 1000000000

The default is 2 GiB.

Also refactored the build error / status handling a bit.
 2016-03-09 17:00:09 +01:00
Eelco Dolstra dc790c5f7e Fix bad format string 2016-03-09 16:59:35 +01:00
Eelco Dolstra 80ff78b1b6 Unify build and step status codes 
Also remove the obsolete status code 5 from the database.
 2016-03-09 15:30:43 +01:00
Eelco Dolstra 9127f5bbc3 hydra-queue-runner: Limit memory usage 
When using a binary cache store, the queue runner receives NARs from
the build machines, compresses them, and uploads them to the
cache. However, keeping multiple large NARs in memory can cause the
queue runner to run out of memory. This can happen for instance when
it's processing multiple ISO images concurrently.

The fix is to use a TokenServer to prevent the builder threads to
store more than a certain total size of NARs concurrently (at the
moment, this is hard-coded at 4 GiB). Builder threads that cause the
limit to be exceeded will block until other threads have finished.

The 4 GiB limit does not include certain other allocations, such as
for xz compression or for FSAccessor::readFile(). But since these are
unlikely to be more than the size of the NARs and hydra.nixos.org has
32 GiB RAM, it should be fine.
 2016-03-09 14:30:13 +01:00
Eelco Dolstra 49a4639377 Add a more concise queue page 
The old page didn't scale very well if you have 150K builds in the
queue, in fact it tended to make browsers hang. The new one just
shows, for each jobset, the number of queued builds. The actual builds
can be seen by going to the corresponding jobset page and looking at
the evals.
 2016-03-08 19:44:51 +01:00
Eelco Dolstra b77a43b83d Get rid of "will retry" messages after "maybe cancelling..." 2016-03-08 13:09:39 +01:00
Eelco Dolstra 718fef29ef Keep track of time required to load builds 2016-03-08 13:09:29 +01:00
Eelco Dolstra 2feb17c681 Some more logging 2016-03-08 13:08:07 +01:00
Eelco Dolstra 45b237453a hydra-queue-runner: Recycle finishedDrvs 
This should prevent the queue monitor thread from looking up the same
derivations over and over again.
 2016-03-08 11:52:13 +01:00
Eelco Dolstra 2ab8e9a1e0 hydra-queue-runner: Fix handling of missing derivations 
This barfed with 'queue monitor: ERROR: column "errormsg" of relation
"builds" does not exist' due to the removal of the errorMsg column.
 2016-03-07 19:05:24 +01:00
Eelco Dolstra e7ce225558 Fix build 2016-03-04 17:51:32 +01:00
Eelco Dolstra 76104accda Return unique store paths 2016-03-03 11:32:30 +01:00
Eelco Dolstra 86a2d6471c Fix a boost format string abort 2016-03-02 20:06:48 +01:00
Eelco Dolstra e7655fdcbc Fix latest-finished 2016-03-02 18:06:20 +01:00
Eelco Dolstra 232ca8fea2 Fix build 2016-03-02 17:05:07 +01:00
Eelco Dolstra e45bbfbef0 Fix .nixpkg channel uri 
Fixes #274.
 2016-03-02 15:38:40 +01:00
Eelco Dolstra 8b4f90b0d4 .nixpkgs: Drop obsolete manifest URI 2016-03-02 15:24:23 +01:00
Eelco Dolstra ec82bc2517 Add /eval/NNN/store-paths action to return store paths in an eval 
Needed by the NixOS channel scripts since we can no longer get a
MANIFEST from Hydra.
 2016-03-02 15:17:22 +01:00
Eelco Dolstra a74251af2b Disable channels on binary cached based Hydra instances 2016-03-02 15:08:53 +01:00
Eelco Dolstra 2d6b585cb3 Merge branch 'slack-plugin' of https://github.com/shlevy/hydra 2016-03-02 15:03:03 +01:00
Eelco Dolstra b98a061c24 Add some instrumentation to keep track of dispatcher cost 2016-03-02 14:18:39 +01:00
Eelco Dolstra 6beee0ab49 Fix segfault sorting runnable steps 
Same problem as d744362e4a.

    at /nix/store/ksvsbr7pg4z69bv6fbbc8h7x7rm2104m-gcc-4.9.3/include/c++/4.9.3/bits/predefined_ops.h:166
    __last@entry=..., __comp=...) at /nix/store/ksvsbr7pg4z69bv6fbbc8h7x7rm2104m-gcc-4.9.3/include/c++/4.9.3/bits/stl_algo.h:1827
    __comp=...) at /nix/store/ksvsbr7pg4z69bv6fbbc8h7x7rm2104m-gcc-4.9.3/include/c++/4.9.3/bits/stl_algo.h:4717
 2016-03-02 13:59:24 +01:00
Shea Levy 0f5937503e SlackNotification: Use bigger images 2016-03-01 11:25:18 -05:00
Shea Levy 006ac1fc03 Add slack plugin. 
Respects <slack> blocks in the hydra config, with attributes:

* jobs: a regexp matching the job name (in the format project:jobset:job)
* url: The URL to a slack incoming webhook
* force: If true, always send messages. Otherwise, only when the build status changes

Multiple <slack> blocks are allowed
 2016-02-29 14:48:36 -05:00
Eelco Dolstra bc958c508b Merge branch 'binary-cache' 2016-02-29 18:29:07 +01:00
Eelco Dolstra 7cd08c7c46 Warn if PostgreSQL appears stalled 2016-02-29 15:10:30 +01:00
Eelco Dolstra 922dc541c2 Add log message 2016-02-29 11:58:06 +01:00
Eelco Dolstra 610a8d67ae Better AWS error messages 2016-02-26 22:40:27 +01:00
Eelco Dolstra 1a055e7e9e Reduce severity level of some message 2016-02-26 21:31:08 +01:00
Eelco Dolstra 6bb860fd6e Add FIXME 2016-02-26 21:15:05 +01:00
Eelco Dolstra e8cdfe5171 hydra-server: Don't barf if the binary cache public key can't be read 2016-02-26 21:14:40 +01:00
Eelco Dolstra 53ca41ef9f Use US standard S3 region 2016-02-26 20:57:47 +01:00
Eelco Dolstra c635f5d0ea Fix Makefile.am 2016-02-26 19:54:55 +01:00
Eelco Dolstra 07e5fc5618 Hackery to make downloads work when using a binary cache 2016-02-26 17:28:26 +01:00
Eelco Dolstra b00bdefa98 Fix hydra-server signing 2016-02-26 17:28:16 +01:00
Eelco Dolstra 9de336de7c Proxy local binary caches via hydra-server 2016-02-26 17:27:30 +01:00
Eelco Dolstra b9afaadfb3 Keep better bytesReceived/bytesSent stats 2016-02-26 16:17:05 +01:00
Eelco Dolstra 6d741d2ffa Prevent download of NARs we just uploaded 2016-02-26 15:21:44 +01:00
Eelco Dolstra 02190b0fef Support hydra-build-products on binary cache stores 2016-02-26 14:45:03 +01:00
Eelco Dolstra 8e24ad6f0d Sync with Nix 2016-02-25 10:58:31 +01:00
Eelco Dolstra 8321a3eb27 Sync with Nix 2016-02-24 14:04:31 +01:00
Eelco Dolstra 7b509237cd Bleh Automake 2016-02-22 18:05:15 +01:00
Eelco Dolstra 6c3ae36648 hydra-queue-runner: Get store mode configuration from hydra.conf 
To use the local Nix store (default):

  store_mode = direct

To use a local binary cache:

  store_mode = local-binary-cache
  binary_cache_dir = /var/lib/hydra/binary-cache

To use an S3 bucket:

  store_mode = s3-binary-cache
  binary_cache_s3_bucket = my-nix-bucket

Also, respect binary_cache_{secret,public}_key_file for signing the
binary cache.
 2016-02-22 17:23:06 +01:00
Eelco Dolstra 94817d77d9 BinaryCacheStore: Respect build-use-substitutes 2016-02-22 17:21:39 +01:00
Eelco Dolstra 5668aa5f71 After uploading a .narinfo, add it to the LRU cache 2016-02-20 10:35:16 +01:00
Eelco Dolstra 88a05763cc Pool local store connections 2016-02-20 00:04:08 +01:00
Eelco Dolstra 1cefd6cac8 Fix log message 2016-02-20 00:02:37 +01:00
Eelco Dolstra a593ebc58e Add missing file 2016-02-19 20:58:40 +01:00
Eelco Dolstra 2b76094a23 S3BinaryCacheStore::isValidPath(): Do a GET instead of HEAD 2016-02-19 17:41:11 +01:00
Eelco Dolstra bd76f9120a Cache .narinfo lookups 2016-02-19 16:19:40 +01:00
Eelco Dolstra a0f74047da Keep some statistics for the binary cache stores 2016-02-19 14:24:23 +01:00
Eelco Dolstra dc4a00347d Use a single BinaryCacheStore for all threads 
This will make it easier to do caching / keep stats. Also, we won't
have S3Client's connection pooling if we create multiple S3Client
instances.
 2016-02-18 17:31:19 +01:00
Eelco Dolstra 00a7be13a2 Make queue runner internal status available under /queue-runner-status 2016-02-18 17:11:46 +01:00
Eelco Dolstra 8c9fc677c1 Typo 2016-02-18 16:43:24 +01:00
Eelco Dolstra db3fcc0f5e Enable substitution on the build machines 
If properly configured, this allows them to get store paths directly
from S3, rather than having to receive them from the queue runner.
 2016-02-18 16:42:05 +01:00
Eelco Dolstra 2d40888e2e Add an S3-backed binary cache store 2016-02-18 16:18:50 +01:00
Eelco Dolstra 0e254ca66d Refactor local binary cache code into a subclass 2016-02-18 14:06:17 +01:00
Eelco Dolstra a992f688d1 Rename class 2016-02-18 13:02:20 +01:00
Eelco Dolstra de77cc2910 Rename file 2016-02-18 13:02:20 +01:00
Michael Mortensen d9a4f7a211 Construct uri using uri_for, which respects X-Request-Base. 2016-02-17 21:27:44 +01:00
Eelco Dolstra ce5790285a Merge remote-tracking branch 'origin/master' into binary-cache 2016-02-17 11:54:59 +01:00
Eelco Dolstra d7a123fcd4 Keep track of the time we spend copying to/from build machines 2016-02-17 10:30:23 +01:00
Eelco Dolstra 25022bf5fd hydra-queue-runner: Support generating a signed binary cache 2016-02-16 16:41:42 +01:00
Eelco Dolstra 744cee134e hydra-queue-runner: Compress binary cache NARs using xz 2016-02-15 21:56:53 +01:00
Eelco Dolstra 2d0dd7fb49 hydra-queue-runner: Write directly to a binary cache 2016-02-15 21:10:29 +01:00
Eelco Dolstra e46acbf05b Remove the errorMsg column from the Builds table 
The queue runner no longer uses this field, and it doesn't provide
very interesting historical data (mostly SSH failures), but it takes
up a lot of space. Also, it contained some bad UTF-8 which was
preventing an upgrade to Postgres 9.5, so a good occasion to get rid
of it.
 2016-02-12 17:21:55 +01:00
Eelco Dolstra 5c5bb3762c Fix display of builds that don't have an eval 
Like http://hydra.nixos.org/build/22872.
 2016-02-12 16:27:25 +01:00
Eelco Dolstra 92d8b59361 Process Nix API changes 2016-02-11 15:59:47 +01:00
Eelco Dolstra 993647d1e3 Use Google's verifier 2016-01-14 12:54:47 +01:00
Eelco Dolstra 5a580b1bb2 Add support for logging in via a Google account 
The required configuration in hydra.conf:

  enable_google_login = 1
  google_client_id = 238429sdjkds....apps.googleusercontent.com

and optionally persona_allowed_domains to restrict to one or more
domains.
 2016-01-13 17:32:52 +01:00
Eelco Dolstra f11ce7e219 Bump evaluation timeout to 6 hours 
This is necessary given the current size of the Nixpkgs/NixOS
jobsets. Once we have a Nix store + Postgres on SSD, we can reduce
this again.

Should really make this configurable...
 2016-01-07 16:19:54 +01:00
Eelco Dolstra 0ca6bb79af Shut up warning 2016-01-07 16:19:54 +01:00
Eelco Dolstra 97f8c61928 Fix hydra-queue-runner --build-one 2015-12-29 17:53:33 +01:00
Eelco Dolstra 29db16bc69 Fix 'Can't locate object method "buildstepoutputs"' 
Template::Toolkit braindamage strikes again. See
8f85fe67d4.
 2015-12-15 11:55:57 +01:00
Eelco Dolstra b081133dcc Move GC roots to /nix/var/nix/gcroots/hydra 
The uid split a while back caused the web interface to create GC roots
in /nix/var/nix/gcroots/per-user/hydra-www, where they wouldn't be
purged by hydra-update-gc-roots. Thus restarted builds would
accumulate forever. The fix is to keep the roots in a shared directory
with gid=hydra.
 2015-12-14 13:31:24 +01:00
Eelco Dolstra e04fb5c8e7 Sort build steps 2015-12-14 13:01:35 +01:00
Eelco Dolstra 8f7614030e Better fix for dots in jobset names 2015-11-17 11:31:11 +01:00
Eelco Dolstra ea80a75204 Move getBaseUrl 2015-11-17 11:21:20 +01:00
Rob Vermaas dddb9a281d Allow dots in job specifier of input type 'Previous build' 2015-11-17 08:36:46 +00:00
Eelco Dolstra b6d2c6f03a Further fix for the channel regression 2015-11-02 14:58:23 +01:00
aszlig 20b412224e ListBuilds: Fix legacy channel URLs. 
Regression introduced by 1fdc258de0.

The commit introduced a channel/custom PathPart which uses the new
custom channel expressions, but I forgot to remove CaptureArgs, so the
URL really is channel/latest/ignored-value.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Reported-by: Peter Simons <simons@cryp.to>
 2015-11-02 14:51:42 +01:00
Eelco Dolstra c087472c71 Remove superfluous "has" function 2015-11-02 14:29:12 +01:00
Eelco Dolstra a5879cbeb8 Consistent terminology: "Clear" -> "Cancel" 2015-11-02 14:22:25 +01:00
Eelco Dolstra e1f9a209c5 Fix cancelling non-current builds 2015-11-02 14:20:27 +01:00
Eelco Dolstra 2d128d2a6f Don't show redundant "removing machine..." messages 2015-10-30 18:22:43 +01:00
Eelco Dolstra d8d188301d Fix division-by-zero crash 
Not clear why step_->jobsets was empty...
 2015-10-30 18:01:48 +01:00