raito/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
2404 commits 2 branches 0 tags 17 MiB
Commit graph

2404 commits

This branch
This branch
All branches
Author SHA1 Message Date
Eelco Dolstra
008d610467 getQueuedBuilds(): Don't catch errors while loading a build from the queue 
Otherwise we never recover from reset daemon connections, e.g.

  hydra-queue-runner[16106]: while loading build 599369: cannot start daemon worker: reading from file: Connection reset by peer
  hydra-queue-runner[16106]: while loading build 599236: writing to file: Broken pipe
  ...

The error is now handled queueMonitor(), causing the next call to
queueMonitorLoop() to create a new connection.
 2015-06-26 21:06:35 +02:00
Eelco Dolstra
f5e5a1b96e Don't wake up the queue runner for cached evals 2015-06-26 20:59:14 +02:00
Eelco Dolstra
401f5bdce2 Add a unit for hydra-send-stats 2015-06-26 15:24:12 +02:00
Eelco Dolstra
17924ce012 Note in the log when the queue runner / evaluator gets killed 2015-06-26 11:56:11 +02:00
Eelco Dolstra
aabe514d07 Update the hydra-queue-runner unit 2015-06-26 11:29:30 +02:00
Eelco Dolstra
73a78c2073 Give services.hydra.package a reasonable default 2015-06-26 11:29:19 +02:00
Eelco Dolstra
9a041f9a36 Restart builds failed due to unsupported system type 2015-06-26 11:28:38 +02:00
Eelco Dolstra
2f4676bd97 JSONObject doesn't handle 64-bit integers 2015-06-25 16:59:48 +02:00
Eelco Dolstra
c54a04688e Fix email sender address when notification_sender is not set 2015-06-25 16:49:01 +02:00
Eelco Dolstra
c6fcce3b3b Moar stats 2015-06-25 16:47:39 +02:00
Eelco Dolstra
b5815e2aa6 Run PostgreSQL during "make check" 2015-06-25 16:47:39 +02:00
Eelco Dolstra
18a3c3ff1c Update "make check" for the new queue runner 
Also, if the machines file contains an entry for localhost, then run
"nix-store --serve" directly, without going through SSH.
 2015-06-25 16:47:39 +02:00
Eelco Dolstra
32210905d8 Automatically reload $NIX_REMOTE_SYSTEMS when it changes 
Otherwise, you'd have to restart the queue runner to add or remove
machines.
 2015-06-25 16:47:25 +02:00
Eelco Dolstra
1a0e1eb5a0 More stats 2015-06-24 13:19:27 +02:00
Eelco Dolstra
3f8891b6ff Fix incorrect debug message 2015-06-23 17:53:15 +02:00
Eelco Dolstra
62219adaf3 Send queue runner stats to statsd 
This is currently done by a separate program that periodically
calls "hydra-queue-runner --status". Eventually, I'll do this
in the queue runner directly.

Fixes #220.
 2015-06-23 14:56:43 +02:00
Eelco Dolstra
af5cbe97aa createStep(): Cache finished derivations 
This gets rid of a lot of redundant calls to readDerivation().
 2015-06-23 03:25:31 +02:00
Eelco Dolstra
681f63a382 Typo 2015-06-23 02:15:11 +02:00
Eelco Dolstra
524ee295e0 Fix sending notifications in the successful case 2015-06-23 02:13:06 +02:00
Eelco Dolstra
4db7c51b5c Rate-limit the number of threads copying closures at the same time 
Having a hundred threads doing I/O at the same time is bad on magnetic
disks because of the excessive disk seeks. So allow only 4 threads to
copy closures in parallel.
 2015-06-23 01:49:14 +02:00
Eelco Dolstra
a317d24b29 hydra-queue-runner: Send build notifications 
Since our notification plugins are written in Perl, sending
notification from C++ requires a small Perl helper named
‘hydra-notify’.
 2015-06-23 00:14:49 +02:00
Eelco Dolstra
5312e1209b Keep per-machine stats 2015-06-22 17:11:17 +02:00
Eelco Dolstra
d06366e7cf Remove obsolete comment 2015-06-22 16:59:50 +02:00
Eelco Dolstra
e069ee960e Doh 2015-06-22 16:58:40 +02:00
Eelco Dolstra
a757b783f4 Update $PATH 2015-06-22 16:15:16 +02:00
Eelco Dolstra
e32ee3d5b9 Remove hydra-build and the old hydra-queue-runner 2015-06-22 15:43:15 +02:00
Eelco Dolstra
41ba7418e2 hydra-queue-runner: More stats 2015-06-22 15:34:33 +02:00
Eelco Dolstra
62b53a0a47 Guard against concurrent invocations of hydra-queue-runner 2015-06-22 14:24:03 +02:00
Eelco Dolstra
fbd7c02217 Periodically dump/log status 2015-06-22 14:15:43 +02:00
Eelco Dolstra
4f4141e1db Add command ‘hydra-queue-runner --status’ to show current status 2015-06-22 14:06:44 +02:00
Eelco Dolstra
44a2b74f5a Keep track of the number of build steps that are being built 
(As opposed to being in the closure copying stage.)
 2015-06-22 11:23:00 +02:00
Eelco Dolstra
fed71d3fe9 Move "created" field into Step::State 2015-06-22 11:07:52 +02:00
Eelco Dolstra
90a08db241 hydra-queue-runner: Fix assertion failure 2015-06-22 10:59:07 +02:00
Eelco Dolstra
d744362e4a hydra-queue-runner: Fix segfault sorting machines by load 
While sorting machines by load, the load of a machine
(machine->currentJobs) can be changed by other threads. If that
happens, the comparator is no longer a proper ordering, in which case
std::sort() can segfault. So we now make a copy of currentJobs before
sorting.
 2015-06-21 16:21:42 +02:00
Eelco Dolstra
a0eff6fc15 Fix machine selection 2015-06-19 17:45:26 +02:00
Eelco Dolstra
a0c4120232 Don't copy src for nix-shell 2015-06-19 17:45:11 +02:00
Eelco Dolstra
81abb6e166 Improve parsing of hydra-build-products 2015-06-19 17:20:20 +02:00
Eelco Dolstra
e13477bdf2 Robustness 2015-06-19 16:35:49 +02:00
Eelco Dolstra
f196967c43 Don't create a propagated build step to the same build 2015-06-19 15:33:37 +02:00
Eelco Dolstra
7afc61691b Doh 2015-06-19 15:27:49 +02:00
Eelco Dolstra
133d298e26 Asynchronously compress build logs 2015-06-19 15:06:12 +02:00
Eelco Dolstra
8e408048e2 Create build step for non-top-level cached failures 
This fixes the missing build step on failures like

  http://hydra.nixos.org/build/23222231
 2015-06-19 11:33:15 +02:00
Eelco Dolstra
77c8bfd392 Improve logging for aborts 2015-06-19 10:37:22 +02:00
Eelco Dolstra
8db1ae2855 Less verbosity 2015-06-18 17:43:13 +02:00
Eelco Dolstra
89b629eeb1 Fix finishing steps that are not top-level of any build 2015-06-18 17:37:35 +02:00
Eelco Dolstra
9cdbff2fdf Handle concurrent finishing of the same build 
There is a slight possibility that the queue monitor and a builder
thread simultaneously decide to mark a build as finished. That's fine,
as long as we ensure the DB update is idempotent (as ensured by doing
"update Builds set finished = 1 ... where finished = 0").
 2015-06-18 17:12:51 +02:00
Eelco Dolstra
948473c909 Fix race between the queue monitor and the builder threads 2015-06-18 16:30:28 +02:00
Eelco Dolstra
9c03b11ca8 Simplify retry handling 2015-06-18 14:51:50 +02:00
Eelco Dolstra
e039f5f840 Create failed build steps for cached failures 2015-06-18 04:35:37 +02:00
Eelco Dolstra
92ea800cfb Set finishedInDB in a few more places 2015-06-18 04:19:21 +02:00