snaakey/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
2063 commits 1 branch 0 tags 20 MiB
Commit graph

1737 commits

Author SHA1 Message Date
Eelco Dolstra 62219adaf3 Send queue runner stats to statsd 
This is currently done by a separate program that periodically
calls "hydra-queue-runner --status". Eventually, I'll do this
in the queue runner directly.

Fixes #220.
 2015-06-23 14:56:43 +02:00
Eelco Dolstra af5cbe97aa createStep(): Cache finished derivations 
This gets rid of a lot of redundant calls to readDerivation().
 2015-06-23 03:25:31 +02:00
Eelco Dolstra 681f63a382 Typo 2015-06-23 02:15:11 +02:00
Eelco Dolstra 524ee295e0 Fix sending notifications in the successful case 2015-06-23 02:13:06 +02:00
Eelco Dolstra 4db7c51b5c Rate-limit the number of threads copying closures at the same time 
Having a hundred threads doing I/O at the same time is bad on magnetic
disks because of the excessive disk seeks. So allow only 4 threads to
copy closures in parallel.
 2015-06-23 01:49:14 +02:00
Eelco Dolstra a317d24b29 hydra-queue-runner: Send build notifications 
Since our notification plugins are written in Perl, sending
notification from C++ requires a small Perl helper named
‘hydra-notify’.
 2015-06-23 00:14:49 +02:00
Eelco Dolstra 5312e1209b Keep per-machine stats 2015-06-22 17:11:17 +02:00
Eelco Dolstra d06366e7cf Remove obsolete comment 2015-06-22 16:59:50 +02:00
Eelco Dolstra e069ee960e Doh 2015-06-22 16:58:40 +02:00
Eelco Dolstra e32ee3d5b9 Remove hydra-build and the old hydra-queue-runner 2015-06-22 15:43:15 +02:00
Eelco Dolstra 41ba7418e2 hydra-queue-runner: More stats 2015-06-22 15:34:33 +02:00
Eelco Dolstra 62b53a0a47 Guard against concurrent invocations of hydra-queue-runner 2015-06-22 14:24:03 +02:00
Eelco Dolstra fbd7c02217 Periodically dump/log status 2015-06-22 14:15:43 +02:00
Eelco Dolstra 4f4141e1db Add command ‘hydra-queue-runner --status’ to show current status 2015-06-22 14:06:44 +02:00
Eelco Dolstra 44a2b74f5a Keep track of the number of build steps that are being built 
(As opposed to being in the closure copying stage.)
 2015-06-22 11:23:00 +02:00
Eelco Dolstra fed71d3fe9 Move "created" field into Step::State 2015-06-22 11:07:52 +02:00
Eelco Dolstra 90a08db241 hydra-queue-runner: Fix assertion failure 2015-06-22 10:59:07 +02:00
Eelco Dolstra d744362e4a hydra-queue-runner: Fix segfault sorting machines by load 
While sorting machines by load, the load of a machine
(machine->currentJobs) can be changed by other threads. If that
happens, the comparator is no longer a proper ordering, in which case
std::sort() can segfault. So we now make a copy of currentJobs before
sorting.
 2015-06-21 16:21:42 +02:00
Eelco Dolstra a0eff6fc15 Fix machine selection 2015-06-19 17:45:26 +02:00
Eelco Dolstra 81abb6e166 Improve parsing of hydra-build-products 2015-06-19 17:20:20 +02:00
Eelco Dolstra e13477bdf2 Robustness 2015-06-19 16:35:49 +02:00
Eelco Dolstra f196967c43 Don't create a propagated build step to the same build 2015-06-19 15:33:37 +02:00
Eelco Dolstra 7afc61691b Doh 2015-06-19 15:27:49 +02:00
Eelco Dolstra 133d298e26 Asynchronously compress build logs 2015-06-19 15:06:12 +02:00
Eelco Dolstra 8e408048e2 Create build step for non-top-level cached failures 
This fixes the missing build step on failures like

  http://hydra.nixos.org/build/23222231
 2015-06-19 11:33:15 +02:00
Eelco Dolstra 77c8bfd392 Improve logging for aborts 2015-06-19 10:37:22 +02:00
Eelco Dolstra 8db1ae2855 Less verbosity 2015-06-18 17:43:13 +02:00
Eelco Dolstra 89b629eeb1 Fix finishing steps that are not top-level of any build 2015-06-18 17:37:35 +02:00
Eelco Dolstra 9cdbff2fdf Handle concurrent finishing of the same build 
There is a slight possibility that the queue monitor and a builder
thread simultaneously decide to mark a build as finished. That's fine,
as long as we ensure the DB update is idempotent (as ensured by doing
"update Builds set finished = 1 ... where finished = 0").
 2015-06-18 17:12:51 +02:00
Eelco Dolstra 948473c909 Fix race between the queue monitor and the builder threads 2015-06-18 16:30:28 +02:00
Eelco Dolstra 9c03b11ca8 Simplify retry handling 2015-06-18 14:51:50 +02:00
Eelco Dolstra e039f5f840 Create failed build steps for cached failures 2015-06-18 04:35:37 +02:00
Eelco Dolstra 92ea800cfb Set finishedInDB in a few more places 2015-06-18 04:19:21 +02:00
Eelco Dolstra 47367451c7 hydra-queue-runner: Set isCachedBuild 2015-06-18 03:28:58 +02:00
Eelco Dolstra 8257812d0a Acquire exclusive table lock earlier 2015-06-18 02:44:29 +02:00
Eelco Dolstra 69be3cfe93 hydra-queue-runner: Handle status queries on the main thread 
Doing it on the queue monitor thread was problematic because
processing the queue can take a while.
 2015-06-18 01:57:01 +02:00
Eelco Dolstra a40ca6b76e hydra-queue-runner: Improve dispatcher 
We now take the machine speed factor into account, just like
build-remote.pl.
 2015-06-18 01:52:20 +02:00
Eelco Dolstra 3855131185 hydra-queue-runner: Improve SSH flags 2015-06-18 00:50:48 +02:00
Eelco Dolstra f57d0b0c54 hydra-queue-runner: Maintain count of active build steps 2015-06-18 00:24:56 +02:00
Eelco Dolstra 59dae60558 hydra-queue-runner: More stats 2015-06-17 22:38:12 +02:00
Eelco Dolstra ec8e8edc86 hydra-queue-runner: Handle $HYDRA_DBI 2015-06-17 22:11:01 +02:00
Eelco Dolstra 4d9c74335d Add forgotten file 2015-06-17 21:39:28 +02:00
Eelco Dolstra ce9e859a9c hydra-queue-runner: Implement --unlock 2015-06-17 21:35:20 +02:00
Eelco Dolstra ca48818b30 Fix remote building 2015-06-17 17:28:59 +02:00
Eelco Dolstra 11be780948 Handle failure with output 2015-06-17 17:11:42 +02:00
Eelco Dolstra b1a75c7f63 getQueuedBuilds(): Handle dependent builds first 
If a build A depends on a derivation that is the top-level derivation
of some build B, then we should process B before A (meaning we
shouldn't make the derivation runnable before B has been
added). Otherwise, the derivation will be "accounted" to A rather than
B (so the build step will show up in the wrong build).
 2015-06-17 14:46:02 +02:00
Eelco Dolstra c6d504edbb Handle SSH hosts without a @ 2015-06-17 13:49:18 +02:00
Eelco Dolstra 745efce828 hydra-queue-runner: Implement timeouts 
Also, keep track of timeouts in the database as a distinct build
status.
 2015-06-17 13:32:33 +02:00
Eelco Dolstra 2da4987bc2 Don't lock the CPU 2015-06-17 11:48:38 +02:00
Eelco Dolstra b91a616520 Automatically retry aborted builds 
Aborted builds are now put back on the runnable queue and retried
after a certain time interval (currently 60 seconds for the first
retry, then tripled on each subsequent retry).
 2015-06-17 11:45:20 +02:00
Eelco Dolstra e02654b3a0 Prefer cached failure over unsupported system type 2015-06-16 18:00:39 +02:00
Eelco Dolstra a984c0badc Merge branch 'master' into build-ng 2015-06-15 18:21:07 +02:00
Eelco Dolstra 42e7301c08 Add status dump facility 
Doing

  $ psql hydra -c 'notify dump_status'

will cause hydra-queue-runner to dump some internal status info on
stderr.
 2015-06-15 18:20:14 +02:00
Eelco Dolstra dd104f14ea Make the queue monitor more robust, and better debug output 2015-06-15 16:54:52 +02:00
Eelco Dolstra 147eb4fd15 Support requiredSystemFeatures 2015-06-15 16:33:50 +02:00
Eelco Dolstra 508ab7f8a2 Tweak build steps 2015-06-15 15:48:05 +02:00
Eelco Dolstra 21aaa0596b Fail builds with previously failed steps early 2015-06-15 15:31:42 +02:00
Eelco Dolstra c00bf7cd1a Check non-runnable steps for unsupported system type 2015-06-15 15:13:03 +02:00
Eelco Dolstra 5019fceb20 Add a error type for "unsupported system type" 2015-06-15 15:07:04 +02:00
Eelco Dolstra 541fbd62cc Immediately abort builds that require an unsupported system type 2015-06-15 14:51:49 +02:00
Eelco Dolstra f9cd5adae8 Queue monitor: Get only the fields we need 2015-06-11 18:09:50 +02:00
Eelco Dolstra c974fb893b Support cancelling builds 2015-06-11 18:07:45 +02:00
Eelco Dolstra c08883966c Use PostgreSQL notifications for queue events 
Hydra-queue-runner now no longer polls the queue periodically, but
instead sleeps until it receives a notification from PostgreSQL about
a change to the queue (build added, build cancelled or build
restarted).

Also, for the "build added" case, we now only check for builds with an
ID greater than the previous greatest ID. This is much more efficient
if the queue is large.
 2015-06-11 17:41:59 +02:00
Eelco Dolstra 672bbb1c67 hydra-update-gc-roots: Get Builds and BuildOutputs in the same query 
This greatly reduces the number of roundtrips to the database.
 2015-06-11 14:10:21 +02:00
Eelco Dolstra 9989a90e99 Keep the most recent successful build of current jobs 
Fixes #140.
 2015-06-11 14:10:21 +02:00
Eelco Dolstra d72a88b562 Don't try to handle SIGINT 
It just makes things unnecessarily complicated. We can just exit
without cleaning anything up, since the only thing to do is unmark
builds and build steps as busy. But we can do that by having systemd
call "hydra-queue-runner --unlock" from ExecStopPost.
 2015-06-10 15:55:46 +02:00
Eelco Dolstra a4fb93c119 Lock builds for a shorter amount of time 2015-06-10 15:36:21 +02:00
Eelco Dolstra 6d738a31bf Keep track of failed paths in the Hydra database 
I.e. don't use Nix's failed paths feature anymore. Easier to keep
everything in one place.
 2015-06-10 14:57:16 +02:00
Eelco Dolstra c68036f8b0 Pass ssh key 2015-06-10 14:57:07 +02:00
Eelco Dolstra 7dd1f0097e Finish copyClosure 2015-06-09 16:03:41 +02:00
Eelco Dolstra c93aa92563 Create BuildSteps race-free 
If multiple threads create a step for the same build, they could get
the same "max(stepnr)" and allocate conflicting new step numbers. So
lock the BuildSteps table while doing this. We could use a different
isolation level, but this is easier.
 2015-06-09 15:03:20 +02:00
Eelco Dolstra 61d4060522 Record the machine used for a build step 2015-06-09 14:57:49 +02:00
Eelco Dolstra 08633508da Fix colspan 2015-06-09 14:42:02 +02:00
Eelco Dolstra ca1fbdd058 Mark builds as busy 2015-06-09 14:31:43 +02:00
Eelco Dolstra 8b12ac1f6d Basic remote building 
This removes the need for Nix's build-remote.pl.

Build logs are now written to $HYDRA_DATA/build-logs because
hydra-queue-runner doesn't have write permission to /nix/var/log.
 2015-06-09 14:21:21 +02:00
Eelco Dolstra 3a6cb2f270 Implement a database connection pool 2015-05-29 20:55:13 +02:00
Eelco Dolstra 214b95706c On SIGINT, shut down the builder threads 
Note that they don't get interrupted at the moment (so on SIGINT, any
running builds will need to finish first).
 2015-05-29 20:02:15 +02:00
Eelco Dolstra e778821940 Make concurrency more robust 2015-05-29 17:14:20 +02:00
Eelco Dolstra 8640e30787 Very basic multi-threaded queue runner 2015-05-29 01:31:12 +02:00
Eelco Dolstra 604fdb908f Pass null values to libpqxx properly 2015-05-28 19:06:17 +02:00
Eelco Dolstra dc446c3980 Start of single-process hydra-queue-runner 2015-05-28 17:39:29 +02:00
Eelco Dolstra ff8cb93fe7 reproduce.tt: Wrap script to prevent problems piping into bash 2015-05-26 17:11:29 +02:00
Eelco Dolstra c32a244bd7 reproduce.tt: Add --print-flags option 
Useful when scripting stuff (e.g. if you want to do a nix-instantiate
instead of a nix-build).
 2015-05-26 16:36:08 +02:00
Eelco Dolstra f5ecb7a81a reproduce.tt: Use -I instead of $NIX_PATH 2015-05-26 16:07:00 +02:00
Eelco Dolstra b8a0956443 reproduce.tt: Add option to fetch but not build 2015-05-26 16:01:37 +02:00
Eelco Dolstra 91ecee0e5d Fix reproduce script 2015-05-26 15:54:38 +02:00
Eelco Dolstra 1ab0f96079 Rename c -> hydra-eval-jobs 2015-05-26 13:58:01 +02:00
Eelco Dolstra e003665146 Split timeSpent query into 2 separate queries, as postgresql isn't able to figure out a decent query plan. With 120k jobs in queue, this makes some queries go from 100s to 1-2s. 2015-05-01 12:28:29 +02:00
Eelco Dolstra d9ab964203 UTF-8 fix 2015-04-14 15:20:56 +02:00
Eelco Dolstra 01cd6397cb Better error message 2015-04-14 15:16:24 +02:00
Eelco Dolstra 5a61aefe4f Use Nix's restricted evaluation mode 2015-04-14 15:16:00 +02:00
Eelco Dolstra 5d8a563e90 Merge pull request #204 from aszlig/tail-improvements 
Improvements of tail-reload output
 2015-04-14 12:44:36 +02:00
Eelco Dolstra 63306aaf5a hydra-evaluator: Add some debug code 2015-04-09 17:35:04 +02:00
Eelco Dolstra a2dc92d871 Die tabs die 2015-04-09 17:22:10 +02:00
Eelco Dolstra 9e664cf8b0 Fix not-null constraint violation inserting build step 
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  null value in column "machine" violates not-null constraint
 2015-04-07 13:34:00 +02:00
aszlig 7a48ad375a
plain-reload: Scroll down at document load. 
When visiting the tail-reload page, for a short amount of time the
"unscrolled" version is shown. To circumvent that, let's scroll down
immediately at the first possibility to fill the gap between the loading
of the document and the first AJAX request coming in.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2015-04-03 00:00:27 +02:00
aszlig c251598241
plain-reload: Constrain tail output to 50 lines. 
There are quite a lot of build outputs which have lines with a length
exceeding the width of the taillog <pre/> and thus visually produce more
lines than 50. This causes the tail "box" to change height frequently
and to get to the bottom you need to scroll down.

We now set a fixed line-height to 120% of the font size and cap the
maximum height based on that value (50 * 1.2 = 60). It's probably not
nice to override the line-height, but max-lines is currently only
available using browser-specific property names. But after all it's just
for the tail output, if people complain about the line-height, we can
still change it :-)

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2015-04-02 17:29:48 +02:00
aszlig be051bcadc
plain-reload.tt: Properly escape tail content. 
We're just implicitly escaping the tail content by not using .load() but
explicitly setting the text content using .text(), so that escaping
isn't needed on our side.

This should get rid of a few formatting errors and possibly XSS if
someone manages to place JS code in the tail of a build and manages to
lurk a user to that tail output.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2015-04-02 16:55:59 +02:00
Eelco Dolstra aec0a35114 Fix duplicate row-link 2015-03-24 14:31:27 +01:00
Rob Vermaas 125dd80dd8 Respect hideJobset in renderBuildListBody. 2015-03-24 08:38:40 +00:00
Rob Vermaas cad82493b0 Restore build and eval IDs. 2015-03-23 15:11:38 +00:00
Rob Vermaas 91e7e6f6ab Use DAG range to determine commits between mercurial revisions. Previously it would show also commits in other branches. 2015-03-23 13:52:06 +00:00
Eelco Dolstra 3b5e84cc73 Tweak debug output 2015-03-23 12:59:41 +01:00
Eelco Dolstra 479632efa3 Call initGC() 2015-03-19 20:16:38 +01:00
Eelco Dolstra 4c0e3e4703 Debug tweak 2015-03-18 22:03:55 +01:00
Eelco Dolstra ad2b7646ac Don't show missing paths in logs 2015-03-04 15:44:04 +01:00
Eelco Dolstra 522bcea8fc Hide build IDs from build lists 
Like eval IDs, build IDs don't convey useful information.

Also, make the job name link to the build rather than the job. When
people click on a build, they expect to go to the build page, not the
job page.
 2015-02-26 13:28:14 +01:00
Eelco Dolstra 62805dd73c Queue: Remove the scheduling priority 
Scheduling is mostly based on jobset shares these days. So showing and
sorting by priority just wastes space and gives the incorrect
impression that Hydra executes builds in the order shown on the queue
page.
 2015-02-26 13:16:25 +01:00
Eelco Dolstra 7f6faee08a Show some eval stats 2015-02-26 13:11:33 +01:00
Eelco Dolstra f835ae492f Likewise remove eval IDs from eval lists 2015-02-26 13:06:53 +01:00
Eelco Dolstra c04c8c2bf1 Jobset job tab: Show eval date rather than eval ID 
Eval IDs carry no useful information, so it's better to show the date
of the eval. If the date is recent, a relative date is given (e.g. "3h
ago").
 2015-02-26 13:01:01 +01:00
Eelco Dolstra fa1e043526 Make input value wider 
Fixes #187.
 2015-02-25 17:42:47 +01:00
Eelco Dolstra adc72d2409 Record which build a failed build step was propagated from 2015-02-25 16:42:32 +01:00
Eelco Dolstra 48af914e28 When propagating failure, propagate the duration and machine 
Previously the duration would just show as "0" and you would have to
search for the original build to get the duration.
 2015-02-25 16:21:54 +01:00
Eelco Dolstra 2f946bdfd7 Render machine correctly if it doesn't contain @ 2015-02-25 16:21:29 +01:00
Eelco Dolstra 9282bcb95f Show eval input changes in deterministic order 2015-02-25 13:13:12 +01:00
Eelco Dolstra 7c048cbbcf Allow a different state directory for hydra-server 
This is necessary if hydra-server runs under a different uid than the
other Hydra processes.
 2015-02-24 11:40:24 +01:00
Eelco Dolstra 00e03b9db5 For consistency, Use base32 hashes in manifests 2015-02-19 12:44:52 +01:00
Eelco Dolstra 69adcd4be1 Update binary cache signing to Nix 1.9 2015-02-17 20:15:37 +01:00
Eelco Dolstra ea815d55b8 More robot exclusions 2015-01-13 13:50:13 +01:00
Eelco Dolstra 80116b648e Exclude robots from all of /build 2015-01-13 13:45:39 +01:00
Eelco Dolstra a12135fc51 Don't use Perl's -w flag 2014-12-12 17:39:52 +01:00
Eelco Dolstra 6a0b9a3476 hydra-build: Handle new trace messages 
Now build step duration no longer includes network overhead (i.e.
time to copy closures to the build machine).
 2014-12-12 17:20:50 +01:00
Eelco Dolstra c0ca5489e1 Don't use given/when 
These give warnings in Perl >= 5.18:

  given is experimental at /home/hydra/src/hydra/src/lib/Hydra/Helper/CatalystUtils.pm line 241.
  when is experimental at /home/hydra/src/hydra/src/lib/Hydra/Helper/CatalystUtils.pm line 242.
  ...
 2014-12-12 11:27:17 +01:00
Shea Levy e56e743f97 S3Backup: Create a proper nar 2014-12-10 23:06:52 -05:00
Shea Levy 208bbbb4b0 s3backup: Create temporary file in temp dir 2014-12-10 22:06:32 -05:00
Domen Kožar 8de15ce38d make sure status images always display for a list of builds 2014-12-03 13:54:22 +01:00
Eelco Dolstra 9e00d98d34 Shut up some Perl 5.20 warnings 2014-11-25 00:27:52 +01:00
Eelco Dolstra 5e265e6739 Tweak blame message 2014-11-19 15:24:31 +01:00
Eelco Dolstra 7b5b434014 Fix getResponsibleAuthors 2014-11-19 15:22:30 +01:00
Eelco Dolstra 0d28e9372b Remove dead code 2014-11-19 14:59:36 +01:00
Eelco Dolstra 2d46741840 Fix message 2014-11-19 14:52:42 +01:00
Eelco Dolstra 8523130ebb Use Email::MIME instead of Email::Simple 
Email::Simple cannot handle non-ASCII characters.

Fixes #191.
 2014-11-19 14:45:46 +01:00
Eelco Dolstra 41bc918382 Fix robots.txt 2014-11-18 14:55:52 +01:00
Rob Vermaas d5db1d3bc1 Revert "Make evaluation fail with proper error when a input of type build is not available." 
This reverts commit a1b5020562.
 2014-11-18 11:13:34 +01:00
Rob Vermaas fd36d79581 Fix Changes section on build page. Now shows the changes of the first evaluation a build was part of compared to the previous one. Also separated previous builds from the changes section, as that was confusing sometimes. 2014-11-18 11:00:28 +01:00
Rob Vermaas a1b5020562 Make evaluation fail with proper error when a input of type build is not available. 2014-11-18 11:00:28 +01:00
Eelco Dolstra c503876a7e Don't cache nix-cache-info very long 2014-11-06 15:30:24 +01:00
Eelco Dolstra c250407f3c Set Expires headers for Hydra's binary cache 
This allows caching by reverse proxies.
 2014-11-06 14:31:48 +01:00
Rob Vermaas 55eb55478d Allow HipChat notification on all builds, in stead of status changes only. 2014-10-21 07:50:12 +00:00
Eelco Dolstra aad3a11eca Build page: Show output / closure sizes 2014-10-07 18:12:00 +02:00
Eelco Dolstra 83d4930101 Reorder 2014-10-07 17:16:41 +02:00
Eelco Dolstra 52ec8abf97 Update flot to 0.8.3 2014-10-07 13:35:31 +02:00
Eelco Dolstra dfe5325490 Scale the Y axis to the visible points 2014-10-07 13:10:10 +02:00
Eelco Dolstra 1b12daa032 Move the build time chart to the job page 2014-10-07 12:59:09 +02:00
Eelco Dolstra 0bb027e633 Move the store path size chart to the job page 
The history is a property of the job, not individual builds.
 2014-10-07 11:53:17 +02:00
Eelco Dolstra 5f474b252c Add a chart to the job pages showing the closure size over time 2014-10-07 11:23:15 +02:00
Eelco Dolstra 3687c9c30b Validate build IDs 2014-10-01 17:05:39 +02:00
Eelco Dolstra 5a7efc0469 Fix aggregate handling 2014-10-01 15:34:05 +02:00
Eelco Dolstra 3f51ac6603 Partially revert 1c20cfdf24 
Not clear why update-dbix messed up the Jobsets and Projects
classes. This will probably happen again in the future...
 2014-10-01 15:28:18 +02:00