raito/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
2569 commits 2 branches 0 tags 17 MiB
Commit graph

2127 commits

Author SHA1 Message Date
Eelco Dolstra 3ded87329d Keep track of how many threads are waiting 2015-07-10 19:10:14 +02:00
Eelco Dolstra 06d75699a7 Fix restarting a build 2015-07-10 16:56:53 +02:00
Eelco Dolstra 7f865a30d5 hydra-evaluator: Fix input change check 
Because inputs were processed in random order by inputsToArgs, the
inputs hash could be different every time, leading to unnecessary
re-evaluations.
 2015-07-10 16:44:06 +02:00
Eelco Dolstra 3e7bbec40b hydra-evaluator: Send statistics to statsd 2015-07-10 16:40:50 +02:00
Eelco Dolstra 5919e911db Don't show how long a machine has been idle 
Without an index on (machine, stoptime desc), this requires a
sequential scan. And adding a whole index for this seems
overkill. (Possibly the queue runner could maintain this info more
efficiently.)
 2015-07-10 15:41:57 +02:00
Eelco Dolstra 3bb9e17e5c IndexJobsetEvalsOnJobsetId: Only index jobsets with new builds 2015-07-10 15:41:57 +02:00
Eelco Dolstra b09f7e0989 Add page showing latest build steps 2015-07-10 15:41:57 +02:00
Eelco Dolstra 0da08df4eb Stream logs if possible and remove size limit 2015-07-08 19:05:17 +02:00
Eelco Dolstra d8acaf2181 Index BuildSteps on propagatedFrom 
This significantly speeds up deleting Builds, since it removes the
need for a sequential scan on BuildSteps.
 2015-07-08 12:12:44 +02:00
Eelco Dolstra bbee81efae Use triggers for all notifications on Builds table changes 2015-07-08 12:05:32 +02:00
Eelco Dolstra 89fb723ace Notify the queue runner when a build is deleted 2015-07-08 11:43:35 +02:00
Eelco Dolstra 95c4294560 Allow cancelling builds marked as busy 
Note that if there are active build *steps*, this won't cancel them.
 2015-07-07 14:08:46 +02:00
Eelco Dolstra 35b7c4f82b Allow only 1 thread to send a closure to a given machine at the same time 
This prevents a race where multiple threads see that machine X is
missing path P, and start sending it concurrently. Nix handles this
correctly, but it's still wasteful (especially for the case where P ==
GHC).

A more refined scheme would be to have per machine, per path locks.
 2015-07-07 14:06:48 +02:00
Eelco Dolstra 16696a4aee Namespace cleanup 2015-07-07 10:29:43 +02:00
Eelco Dolstra 63745b8e25 Move buildRemote() into State 2015-07-07 10:25:33 +02:00
Eelco Dolstra df29527531 Refactor 2015-07-07 10:17:21 +02:00
Eelco Dolstra dd4f6e695e Merge branch 'master' into build-ng 2015-07-06 17:17:51 +02:00
Eelco Dolstra ccf6e6062c Store full Mercurial revision hashes 2015-07-06 17:17:17 +02:00
Eelco Dolstra 309ef5baa9 Merge branch 'master' into build-ng 2015-07-06 15:57:09 +02:00
Eelco Dolstra b85e9ef1cd Support using Git revisions as branch names 2015-07-06 15:56:24 +02:00
Eelco Dolstra b03de925cb Allow a jobset to be created from an evaluation 
Fixes #150.
 2015-07-06 15:56:20 +02:00
Eelco Dolstra dffb629b8a Unify Hydra's NixOS module with the one used for hydra.nixos.org 
In particular, the queue runner and web server now run under different
UIDs.
 2015-07-02 01:01:44 +02:00
Eelco Dolstra 3e0f5f664a GitInput plugin: Don't clone during getCommits 
This doesn't work if hydra-queue-runner has no write access to the scm
directory, and in any case races with the evaluator.
 2015-07-02 00:44:40 +02:00
Eelco Dolstra ae52fc7f61 Remove display of queue runner log file (it no longer exists) 2015-07-02 00:18:33 +02:00
Eelco Dolstra e35b704d80 Drop the 5 minute minimum interval between triggered evals 2015-07-01 14:45:39 +02:00
Eelco Dolstra 85a1ce99c9 Only include Persona JS when Persona is enabled 2015-07-01 14:24:18 +02:00
Eelco Dolstra 3c665dac82 Remove superfluous HYDRA_LOGO environment variable 2015-07-01 11:34:19 +02:00
Eelco Dolstra 7e6135a8c6 Don't repeat links to build step logs 
Hydra only stores the last log for a particular derivation, so only
show log links for the last one.
 2015-06-30 00:27:31 +02:00
Eelco Dolstra 2ece42b2b9 Support preferLocalBuild 
Derivations with "preferLocalBuild = true" can now be executed on
specific machines (typically localhost) by setting the mandary system
features field to include "local". For example:

  localhost x86_64-linux,i686-linux - 10 100 - local

says that "localhost" can *only* do builds with "preferLocalBuild =
true". The speed factor of 100 will make the machine almost always win
over other machines.
 2015-06-30 00:20:19 +02:00
Eelco Dolstra 008d610467 getQueuedBuilds(): Don't catch errors while loading a build from the queue 
Otherwise we never recover from reset daemon connections, e.g.

  hydra-queue-runner[16106]: while loading build 599369: cannot start daemon worker: reading from file: Connection reset by peer
  hydra-queue-runner[16106]: while loading build 599236: writing to file: Broken pipe
  ...

The error is now handled queueMonitor(), causing the next call to
queueMonitorLoop() to create a new connection.
 2015-06-26 21:06:35 +02:00
Eelco Dolstra f5e5a1b96e Don't wake up the queue runner for cached evals 2015-06-26 20:59:14 +02:00
Eelco Dolstra 401f5bdce2 Add a unit for hydra-send-stats 2015-06-26 15:24:12 +02:00
Eelco Dolstra 9a041f9a36 Restart builds failed due to unsupported system type 2015-06-26 11:28:38 +02:00
Eelco Dolstra 2f4676bd97 JSONObject doesn't handle 64-bit integers 2015-06-25 16:59:48 +02:00
Eelco Dolstra c54a04688e Fix email sender address when notification_sender is not set 2015-06-25 16:49:01 +02:00
Eelco Dolstra c6fcce3b3b Moar stats 2015-06-25 16:47:39 +02:00
Eelco Dolstra 18a3c3ff1c Update "make check" for the new queue runner 
Also, if the machines file contains an entry for localhost, then run
"nix-store --serve" directly, without going through SSH.
 2015-06-25 16:47:39 +02:00
Eelco Dolstra 32210905d8 Automatically reload $NIX_REMOTE_SYSTEMS when it changes 
Otherwise, you'd have to restart the queue runner to add or remove
machines.
 2015-06-25 16:47:25 +02:00
Eelco Dolstra 1a0e1eb5a0 More stats 2015-06-24 13:19:27 +02:00
Eelco Dolstra 3f8891b6ff Fix incorrect debug message 2015-06-23 17:53:15 +02:00
Eelco Dolstra 62219adaf3 Send queue runner stats to statsd 
This is currently done by a separate program that periodically
calls "hydra-queue-runner --status". Eventually, I'll do this
in the queue runner directly.

Fixes #220.
 2015-06-23 14:56:43 +02:00
Eelco Dolstra af5cbe97aa createStep(): Cache finished derivations 
This gets rid of a lot of redundant calls to readDerivation().
 2015-06-23 03:25:31 +02:00
Eelco Dolstra 681f63a382 Typo 2015-06-23 02:15:11 +02:00
Eelco Dolstra 524ee295e0 Fix sending notifications in the successful case 2015-06-23 02:13:06 +02:00
Eelco Dolstra 4db7c51b5c Rate-limit the number of threads copying closures at the same time 
Having a hundred threads doing I/O at the same time is bad on magnetic
disks because of the excessive disk seeks. So allow only 4 threads to
copy closures in parallel.
 2015-06-23 01:49:14 +02:00
Eelco Dolstra a317d24b29 hydra-queue-runner: Send build notifications 
Since our notification plugins are written in Perl, sending
notification from C++ requires a small Perl helper named
‘hydra-notify’.
 2015-06-23 00:14:49 +02:00
Eelco Dolstra 5312e1209b Keep per-machine stats 2015-06-22 17:11:17 +02:00
Eelco Dolstra d06366e7cf Remove obsolete comment 2015-06-22 16:59:50 +02:00
Eelco Dolstra e069ee960e Doh 2015-06-22 16:58:40 +02:00
Eelco Dolstra e32ee3d5b9 Remove hydra-build and the old hydra-queue-runner 2015-06-22 15:43:15 +02:00
Eelco Dolstra 41ba7418e2 hydra-queue-runner: More stats 2015-06-22 15:34:33 +02:00
Eelco Dolstra 62b53a0a47 Guard against concurrent invocations of hydra-queue-runner 2015-06-22 14:24:03 +02:00
Eelco Dolstra fbd7c02217 Periodically dump/log status 2015-06-22 14:15:43 +02:00
Eelco Dolstra 4f4141e1db Add command ‘hydra-queue-runner --status’ to show current status 2015-06-22 14:06:44 +02:00
Eelco Dolstra 44a2b74f5a Keep track of the number of build steps that are being built 
(As opposed to being in the closure copying stage.)
 2015-06-22 11:23:00 +02:00
Eelco Dolstra fed71d3fe9 Move "created" field into Step::State 2015-06-22 11:07:52 +02:00
Eelco Dolstra 90a08db241 hydra-queue-runner: Fix assertion failure 2015-06-22 10:59:07 +02:00
Eelco Dolstra d744362e4a hydra-queue-runner: Fix segfault sorting machines by load 
While sorting machines by load, the load of a machine
(machine->currentJobs) can be changed by other threads. If that
happens, the comparator is no longer a proper ordering, in which case
std::sort() can segfault. So we now make a copy of currentJobs before
sorting.
 2015-06-21 16:21:42 +02:00
Eelco Dolstra a0eff6fc15 Fix machine selection 2015-06-19 17:45:26 +02:00
Eelco Dolstra 81abb6e166 Improve parsing of hydra-build-products 2015-06-19 17:20:20 +02:00
Eelco Dolstra e13477bdf2 Robustness 2015-06-19 16:35:49 +02:00
Eelco Dolstra f196967c43 Don't create a propagated build step to the same build 2015-06-19 15:33:37 +02:00
Eelco Dolstra 7afc61691b Doh 2015-06-19 15:27:49 +02:00
Eelco Dolstra 133d298e26 Asynchronously compress build logs 2015-06-19 15:06:12 +02:00
Eelco Dolstra 8e408048e2 Create build step for non-top-level cached failures 
This fixes the missing build step on failures like

  http://hydra.nixos.org/build/23222231
 2015-06-19 11:33:15 +02:00
Eelco Dolstra 77c8bfd392 Improve logging for aborts 2015-06-19 10:37:22 +02:00
Eelco Dolstra 8db1ae2855 Less verbosity 2015-06-18 17:43:13 +02:00
Eelco Dolstra 89b629eeb1 Fix finishing steps that are not top-level of any build 2015-06-18 17:37:35 +02:00
Eelco Dolstra 9cdbff2fdf Handle concurrent finishing of the same build 
There is a slight possibility that the queue monitor and a builder
thread simultaneously decide to mark a build as finished. That's fine,
as long as we ensure the DB update is idempotent (as ensured by doing
"update Builds set finished = 1 ... where finished = 0").
 2015-06-18 17:12:51 +02:00
Eelco Dolstra 948473c909 Fix race between the queue monitor and the builder threads 2015-06-18 16:30:28 +02:00
Eelco Dolstra 9c03b11ca8 Simplify retry handling 2015-06-18 14:51:50 +02:00
Eelco Dolstra e039f5f840 Create failed build steps for cached failures 2015-06-18 04:35:37 +02:00
Eelco Dolstra 92ea800cfb Set finishedInDB in a few more places 2015-06-18 04:19:21 +02:00
Eelco Dolstra 47367451c7 hydra-queue-runner: Set isCachedBuild 2015-06-18 03:28:58 +02:00
Eelco Dolstra 8257812d0a Acquire exclusive table lock earlier 2015-06-18 02:44:29 +02:00
Eelco Dolstra 69be3cfe93 hydra-queue-runner: Handle status queries on the main thread 
Doing it on the queue monitor thread was problematic because
processing the queue can take a while.
 2015-06-18 01:57:01 +02:00
Eelco Dolstra a40ca6b76e hydra-queue-runner: Improve dispatcher 
We now take the machine speed factor into account, just like
build-remote.pl.
 2015-06-18 01:52:20 +02:00
Eelco Dolstra 3855131185 hydra-queue-runner: Improve SSH flags 2015-06-18 00:50:48 +02:00
Eelco Dolstra f57d0b0c54 hydra-queue-runner: Maintain count of active build steps 2015-06-18 00:24:56 +02:00
Eelco Dolstra 59dae60558 hydra-queue-runner: More stats 2015-06-17 22:38:12 +02:00
Eelco Dolstra ec8e8edc86 hydra-queue-runner: Handle $HYDRA_DBI 2015-06-17 22:11:01 +02:00
Eelco Dolstra 4d9c74335d Add forgotten file 2015-06-17 21:39:28 +02:00
Eelco Dolstra ce9e859a9c hydra-queue-runner: Implement --unlock 2015-06-17 21:35:20 +02:00
Eelco Dolstra ca48818b30 Fix remote building 2015-06-17 17:28:59 +02:00
Eelco Dolstra 11be780948 Handle failure with output 2015-06-17 17:11:42 +02:00
Eelco Dolstra b1a75c7f63 getQueuedBuilds(): Handle dependent builds first 
If a build A depends on a derivation that is the top-level derivation
of some build B, then we should process B before A (meaning we
shouldn't make the derivation runnable before B has been
added). Otherwise, the derivation will be "accounted" to A rather than
B (so the build step will show up in the wrong build).
 2015-06-17 14:46:02 +02:00
Eelco Dolstra c6d504edbb Handle SSH hosts without a @ 2015-06-17 13:49:18 +02:00
Eelco Dolstra 745efce828 hydra-queue-runner: Implement timeouts 
Also, keep track of timeouts in the database as a distinct build
status.
 2015-06-17 13:32:33 +02:00
Eelco Dolstra 2da4987bc2 Don't lock the CPU 2015-06-17 11:48:38 +02:00
Eelco Dolstra b91a616520 Automatically retry aborted builds 
Aborted builds are now put back on the runnable queue and retried
after a certain time interval (currently 60 seconds for the first
retry, then tripled on each subsequent retry).
 2015-06-17 11:45:20 +02:00
Eelco Dolstra e02654b3a0 Prefer cached failure over unsupported system type 2015-06-16 18:00:39 +02:00
Eelco Dolstra a984c0badc Merge branch 'master' into build-ng 2015-06-15 18:21:07 +02:00
Eelco Dolstra 42e7301c08 Add status dump facility 
Doing

  $ psql hydra -c 'notify dump_status'

will cause hydra-queue-runner to dump some internal status info on
stderr.
 2015-06-15 18:20:14 +02:00
Eelco Dolstra dd104f14ea Make the queue monitor more robust, and better debug output 2015-06-15 16:54:52 +02:00
Eelco Dolstra 147eb4fd15 Support requiredSystemFeatures 2015-06-15 16:33:50 +02:00
Eelco Dolstra 508ab7f8a2 Tweak build steps 2015-06-15 15:48:05 +02:00
Eelco Dolstra 21aaa0596b Fail builds with previously failed steps early 2015-06-15 15:31:42 +02:00
Eelco Dolstra c00bf7cd1a Check non-runnable steps for unsupported system type 2015-06-15 15:13:03 +02:00
Eelco Dolstra 5019fceb20 Add a error type for "unsupported system type" 2015-06-15 15:07:04 +02:00
Eelco Dolstra 541fbd62cc Immediately abort builds that require an unsupported system type 2015-06-15 14:51:49 +02:00
Eelco Dolstra f9cd5adae8 Queue monitor: Get only the fields we need 2015-06-11 18:09:50 +02:00
Eelco Dolstra c974fb893b Support cancelling builds 2015-06-11 18:07:45 +02:00
Eelco Dolstra c08883966c Use PostgreSQL notifications for queue events 
Hydra-queue-runner now no longer polls the queue periodically, but
instead sleeps until it receives a notification from PostgreSQL about
a change to the queue (build added, build cancelled or build
restarted).

Also, for the "build added" case, we now only check for builds with an
ID greater than the previous greatest ID. This is much more efficient
if the queue is large.
 2015-06-11 17:41:59 +02:00
Eelco Dolstra 672bbb1c67 hydra-update-gc-roots: Get Builds and BuildOutputs in the same query 
This greatly reduces the number of roundtrips to the database.
 2015-06-11 14:10:21 +02:00
Eelco Dolstra 9989a90e99 Keep the most recent successful build of current jobs 
Fixes #140.
 2015-06-11 14:10:21 +02:00
Eelco Dolstra d72a88b562 Don't try to handle SIGINT 
It just makes things unnecessarily complicated. We can just exit
without cleaning anything up, since the only thing to do is unmark
builds and build steps as busy. But we can do that by having systemd
call "hydra-queue-runner --unlock" from ExecStopPost.
 2015-06-10 15:55:46 +02:00
Eelco Dolstra a4fb93c119 Lock builds for a shorter amount of time 2015-06-10 15:36:21 +02:00
Eelco Dolstra 6d738a31bf Keep track of failed paths in the Hydra database 
I.e. don't use Nix's failed paths feature anymore. Easier to keep
everything in one place.
 2015-06-10 14:57:16 +02:00
Eelco Dolstra c68036f8b0 Pass ssh key 2015-06-10 14:57:07 +02:00
Eelco Dolstra 7dd1f0097e Finish copyClosure 2015-06-09 16:03:41 +02:00
Eelco Dolstra c93aa92563 Create BuildSteps race-free 
If multiple threads create a step for the same build, they could get
the same "max(stepnr)" and allocate conflicting new step numbers. So
lock the BuildSteps table while doing this. We could use a different
isolation level, but this is easier.
 2015-06-09 15:03:20 +02:00
Eelco Dolstra 61d4060522 Record the machine used for a build step 2015-06-09 14:57:49 +02:00
Eelco Dolstra 08633508da Fix colspan 2015-06-09 14:42:02 +02:00
Eelco Dolstra ca1fbdd058 Mark builds as busy 2015-06-09 14:31:43 +02:00
Eelco Dolstra 8b12ac1f6d Basic remote building 
This removes the need for Nix's build-remote.pl.

Build logs are now written to $HYDRA_DATA/build-logs because
hydra-queue-runner doesn't have write permission to /nix/var/log.
 2015-06-09 14:21:21 +02:00
Eelco Dolstra 3a6cb2f270 Implement a database connection pool 2015-05-29 20:55:13 +02:00
Eelco Dolstra 214b95706c On SIGINT, shut down the builder threads 
Note that they don't get interrupted at the moment (so on SIGINT, any
running builds will need to finish first).
 2015-05-29 20:02:15 +02:00
Eelco Dolstra e778821940 Make concurrency more robust 2015-05-29 17:14:20 +02:00
Eelco Dolstra 8640e30787 Very basic multi-threaded queue runner 2015-05-29 01:31:12 +02:00
Eelco Dolstra 604fdb908f Pass null values to libpqxx properly 2015-05-28 19:06:17 +02:00
Eelco Dolstra dc446c3980 Start of single-process hydra-queue-runner 2015-05-28 17:39:29 +02:00
Eelco Dolstra ff8cb93fe7 reproduce.tt: Wrap script to prevent problems piping into bash 2015-05-26 17:11:29 +02:00
Eelco Dolstra c32a244bd7 reproduce.tt: Add --print-flags option 
Useful when scripting stuff (e.g. if you want to do a nix-instantiate
instead of a nix-build).
 2015-05-26 16:36:08 +02:00
Eelco Dolstra f5ecb7a81a reproduce.tt: Use -I instead of $NIX_PATH 2015-05-26 16:07:00 +02:00
Eelco Dolstra b8a0956443 reproduce.tt: Add option to fetch but not build 2015-05-26 16:01:37 +02:00
Eelco Dolstra 91ecee0e5d Fix reproduce script 2015-05-26 15:54:38 +02:00
Eelco Dolstra 1ab0f96079 Rename c -> hydra-eval-jobs 2015-05-26 13:58:01 +02:00
Eelco Dolstra e003665146 Split timeSpent query into 2 separate queries, as postgresql isn't able to figure out a decent query plan. With 120k jobs in queue, this makes some queries go from 100s to 1-2s. 2015-05-01 12:28:29 +02:00
Eelco Dolstra d9ab964203 UTF-8 fix 2015-04-14 15:20:56 +02:00
Eelco Dolstra 01cd6397cb Better error message 2015-04-14 15:16:24 +02:00
Eelco Dolstra 5a61aefe4f Use Nix's restricted evaluation mode 2015-04-14 15:16:00 +02:00
Eelco Dolstra 5d8a563e90 Merge pull request #204 from aszlig/tail-improvements 
Improvements of tail-reload output
 2015-04-14 12:44:36 +02:00
Eelco Dolstra 63306aaf5a hydra-evaluator: Add some debug code 2015-04-09 17:35:04 +02:00
Eelco Dolstra a2dc92d871 Die tabs die 2015-04-09 17:22:10 +02:00
Eelco Dolstra 9e664cf8b0 Fix not-null constraint violation inserting build step 
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  null value in column "machine" violates not-null constraint
 2015-04-07 13:34:00 +02:00
aszlig 7a48ad375a
plain-reload: Scroll down at document load. 
When visiting the tail-reload page, for a short amount of time the
"unscrolled" version is shown. To circumvent that, let's scroll down
immediately at the first possibility to fill the gap between the loading
of the document and the first AJAX request coming in.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2015-04-03 00:00:27 +02:00
aszlig c251598241
plain-reload: Constrain tail output to 50 lines. 
There are quite a lot of build outputs which have lines with a length
exceeding the width of the taillog <pre/> and thus visually produce more
lines than 50. This causes the tail "box" to change height frequently
and to get to the bottom you need to scroll down.

We now set a fixed line-height to 120% of the font size and cap the
maximum height based on that value (50 * 1.2 = 60). It's probably not
nice to override the line-height, but max-lines is currently only
available using browser-specific property names. But after all it's just
for the tail output, if people complain about the line-height, we can
still change it :-)

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2015-04-02 17:29:48 +02:00
aszlig be051bcadc
plain-reload.tt: Properly escape tail content. 
We're just implicitly escaping the tail content by not using .load() but
explicitly setting the text content using .text(), so that escaping
isn't needed on our side.

This should get rid of a few formatting errors and possibly XSS if
someone manages to place JS code in the tail of a build and manages to
lurk a user to that tail output.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2015-04-02 16:55:59 +02:00
Eelco Dolstra aec0a35114 Fix duplicate row-link 2015-03-24 14:31:27 +01:00
Rob Vermaas 125dd80dd8 Respect hideJobset in renderBuildListBody. 2015-03-24 08:38:40 +00:00
Rob Vermaas cad82493b0 Restore build and eval IDs. 2015-03-23 15:11:38 +00:00
Rob Vermaas 91e7e6f6ab Use DAG range to determine commits between mercurial revisions. Previously it would show also commits in other branches. 2015-03-23 13:52:06 +00:00
Eelco Dolstra 3b5e84cc73 Tweak debug output 2015-03-23 12:59:41 +01:00
Eelco Dolstra 479632efa3 Call initGC() 2015-03-19 20:16:38 +01:00
Eelco Dolstra 4c0e3e4703 Debug tweak 2015-03-18 22:03:55 +01:00
Eelco Dolstra ad2b7646ac Don't show missing paths in logs 2015-03-04 15:44:04 +01:00
Eelco Dolstra 522bcea8fc Hide build IDs from build lists 
Like eval IDs, build IDs don't convey useful information.

Also, make the job name link to the build rather than the job. When
people click on a build, they expect to go to the build page, not the
job page.
 2015-02-26 13:28:14 +01:00
Eelco Dolstra 62805dd73c Queue: Remove the scheduling priority 
Scheduling is mostly based on jobset shares these days. So showing and
sorting by priority just wastes space and gives the incorrect
impression that Hydra executes builds in the order shown on the queue
page.
 2015-02-26 13:16:25 +01:00
Eelco Dolstra 7f6faee08a Show some eval stats 2015-02-26 13:11:33 +01:00
Eelco Dolstra f835ae492f Likewise remove eval IDs from eval lists 2015-02-26 13:06:53 +01:00
Eelco Dolstra c04c8c2bf1 Jobset job tab: Show eval date rather than eval ID 
Eval IDs carry no useful information, so it's better to show the date
of the eval. If the date is recent, a relative date is given (e.g. "3h
ago").
 2015-02-26 13:01:01 +01:00
Eelco Dolstra fa1e043526 Make input value wider 
Fixes #187.
 2015-02-25 17:42:47 +01:00
Eelco Dolstra adc72d2409 Record which build a failed build step was propagated from 2015-02-25 16:42:32 +01:00
Eelco Dolstra 48af914e28 When propagating failure, propagate the duration and machine 
Previously the duration would just show as "0" and you would have to
search for the original build to get the duration.
 2015-02-25 16:21:54 +01:00
Eelco Dolstra 2f946bdfd7 Render machine correctly if it doesn't contain @ 2015-02-25 16:21:29 +01:00
Eelco Dolstra 9282bcb95f Show eval input changes in deterministic order 2015-02-25 13:13:12 +01:00
Eelco Dolstra 7c048cbbcf Allow a different state directory for hydra-server 
This is necessary if hydra-server runs under a different uid than the
other Hydra processes.
 2015-02-24 11:40:24 +01:00
Eelco Dolstra 00e03b9db5 For consistency, Use base32 hashes in manifests 2015-02-19 12:44:52 +01:00
Eelco Dolstra 69adcd4be1 Update binary cache signing to Nix 1.9 2015-02-17 20:15:37 +01:00
Eelco Dolstra ea815d55b8 More robot exclusions 2015-01-13 13:50:13 +01:00
Eelco Dolstra 80116b648e Exclude robots from all of /build 2015-01-13 13:45:39 +01:00
Eelco Dolstra a12135fc51 Don't use Perl's -w flag 2014-12-12 17:39:52 +01:00
Eelco Dolstra 6a0b9a3476 hydra-build: Handle new trace messages 
Now build step duration no longer includes network overhead (i.e.
time to copy closures to the build machine).
 2014-12-12 17:20:50 +01:00
Eelco Dolstra c0ca5489e1 Don't use given/when 
These give warnings in Perl >= 5.18:

  given is experimental at /home/hydra/src/hydra/src/lib/Hydra/Helper/CatalystUtils.pm line 241.
  when is experimental at /home/hydra/src/hydra/src/lib/Hydra/Helper/CatalystUtils.pm line 242.
  ...
 2014-12-12 11:27:17 +01:00
Shea Levy e56e743f97 S3Backup: Create a proper nar 2014-12-10 23:06:52 -05:00
Shea Levy 208bbbb4b0 s3backup: Create temporary file in temp dir 2014-12-10 22:06:32 -05:00
Domen Kožar 8de15ce38d make sure status images always display for a list of builds 2014-12-03 13:54:22 +01:00
Eelco Dolstra 9e00d98d34 Shut up some Perl 5.20 warnings 2014-11-25 00:27:52 +01:00
Eelco Dolstra 5e265e6739 Tweak blame message 2014-11-19 15:24:31 +01:00
Eelco Dolstra 7b5b434014 Fix getResponsibleAuthors 2014-11-19 15:22:30 +01:00
Eelco Dolstra 0d28e9372b Remove dead code 2014-11-19 14:59:36 +01:00
Eelco Dolstra 2d46741840 Fix message 2014-11-19 14:52:42 +01:00
Eelco Dolstra 8523130ebb Use Email::MIME instead of Email::Simple 
Email::Simple cannot handle non-ASCII characters.

Fixes #191.
 2014-11-19 14:45:46 +01:00
Eelco Dolstra 41bc918382 Fix robots.txt 2014-11-18 14:55:52 +01:00
Rob Vermaas d5db1d3bc1 Revert "Make evaluation fail with proper error when a input of type build is not available." 
This reverts commit a1b5020562.
 2014-11-18 11:13:34 +01:00
Rob Vermaas fd36d79581 Fix Changes section on build page. Now shows the changes of the first evaluation a build was part of compared to the previous one. Also separated previous builds from the changes section, as that was confusing sometimes. 2014-11-18 11:00:28 +01:00
Rob Vermaas a1b5020562 Make evaluation fail with proper error when a input of type build is not available. 2014-11-18 11:00:28 +01:00
Eelco Dolstra c503876a7e Don't cache nix-cache-info very long 2014-11-06 15:30:24 +01:00
Eelco Dolstra c250407f3c Set Expires headers for Hydra's binary cache 
This allows caching by reverse proxies.
 2014-11-06 14:31:48 +01:00
Rob Vermaas 55eb55478d Allow HipChat notification on all builds, in stead of status changes only. 2014-10-21 07:50:12 +00:00
Eelco Dolstra aad3a11eca Build page: Show output / closure sizes 2014-10-07 18:12:00 +02:00
Eelco Dolstra 83d4930101 Reorder 2014-10-07 17:16:41 +02:00
Eelco Dolstra 52ec8abf97 Update flot to 0.8.3 2014-10-07 13:35:31 +02:00
Eelco Dolstra dfe5325490 Scale the Y axis to the visible points 2014-10-07 13:10:10 +02:00
Eelco Dolstra 1b12daa032 Move the build time chart to the job page 2014-10-07 12:59:09 +02:00
Eelco Dolstra 0bb027e633 Move the store path size chart to the job page 
The history is a property of the job, not individual builds.
 2014-10-07 11:53:17 +02:00
Eelco Dolstra 5f474b252c Add a chart to the job pages showing the closure size over time 2014-10-07 11:23:15 +02:00
Eelco Dolstra 3687c9c30b Validate build IDs 2014-10-01 17:05:39 +02:00
Eelco Dolstra 5a7efc0469 Fix aggregate handling 2014-10-01 15:34:05 +02:00
Eelco Dolstra 3f51ac6603 Partially revert 1c20cfdf24 
Not clear why update-dbix messed up the Jobsets and Projects
classes. This will probably happen again in the future...
 2014-10-01 15:28:18 +02:00
Eelco Dolstra 9b38b5f134 Remove the longDescription field 
It's not useful and takes up a lot of space.
 2014-09-30 15:44:09 +02:00
Eelco Dolstra f4acc9a522 Create Builds with iscurrent set 
This should eliminate a subsequent update.
 2014-09-30 15:44:09 +02:00
Eelco Dolstra 5b4de2dee6 hydra-evaluator: Reduce verbosity 2014-09-30 15:44:08 +02:00
Eelco Dolstra 09a96c642a hydra-eval-jobs: Use JSON instead of XML 
XML::Simple is pretty slow - reading the output for the Nixpkgs jobset
takes half a minute or so. JSON is pretty much instantaneous.
 2014-09-30 15:44:08 +02:00
Eelco Dolstra d9a5143fcb Use partial indexes 
There is no point in indexing rows with common column values like
"finished = 1", since those are the majority of the table. Only the
exceptions ("finished = 0") are interesting. Having smaller tables
should make updates/insertions faster.
 2014-09-30 15:44:08 +02:00
Eelco Dolstra 773c7e89cf Get rid of some obsolete indexes 2014-09-30 15:44:08 +02:00
Eelco Dolstra 1c20cfdf24 Drop the errorMsg column in the Jobs table 
We're not using it anywhere.
 2014-09-29 19:46:11 +02:00
Eelco Dolstra beb94d5fb3 Machine status page: Add link to build step 2014-09-25 20:24:55 +02:00
Eelco Dolstra 2054759fdf Status page: Show running builds rather than build steps 
The active build steps list was redundant, because the "Machine
status" page also shows them. So instead show active builds.
 2014-09-25 20:17:33 +02:00
Eelco Dolstra 23a9fea26c Remove the ability to add multiple jobset input alternatives 
Now each jobset input has exactly one value. (Actually, adding
multiple values was already broken, so all this does is clean up the
UI...)
 2014-09-25 16:43:17 +02:00
Eelco Dolstra c28e7ac692 Fix XML wellformedness 2014-09-25 16:06:07 +02:00
Eelco Dolstra b04c8adbd5 build.tt: Show jobset eval inputs instead of build inputs 
We're not recording build inputs anymore, so we now just show all
inputs of the build's first evaluation.
 2014-09-25 14:49:09 +02:00
Eelco Dolstra 01f4037d6f hydra-eval-jobs: Don't keep track of used inputs 
We no longer store this in the database, so it's not necessary for
hydra-eval-jobs to do this.
 2014-09-25 13:38:43 +02:00
Eelco Dolstra 6284fd540d Disallow multiple jobs with the same name 
This has been deprecated since a8db329839.

Issue #60.
 2014-09-24 18:12:59 +02:00
Eelco Dolstra 0d5a38a40b Really fix hydra-eval-jobs 
Fixes #183.
 2014-09-24 15:55:19 +02:00
Eelco Dolstra b11cb37044 Handle derivations without a system attribute 2014-09-22 16:53:40 +02:00
Eelco Dolstra eb2f3eb8d4 Fix build 2014-09-22 16:47:02 +02:00
Eelco Dolstra 4727165832 Use pkgconfig to find Nix 2014-09-18 12:24:05 +02:00
Eelco Dolstra c1bf3bb0f2 hydra-eval-jobs: Add --dry-run option 2014-09-12 14:30:01 +02:00
Eelco Dolstra 748c3409b4 Don't maintain BuildInputs anymore 
We don't need to record inputs per build anymore because we have
JobsetEvalInputs now.
 2014-09-06 19:06:07 +02:00
aszlig 9c7f303255 Use mktemp for tempdir creation in prefetchers. 
This incorporates the following two commits from <nixpkgs>:

NixOS/nixpkgs@f83af95f8a
NixOS/nixpkgs@5e7a1cf955

Hydra was the original reason why I was fixing tempdir creation in the
first place. Seeing that Hydra ships its own versions of these scripts,
we need to patch them here as well.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2014-08-29 12:40:11 +02:00
Eelco Dolstra 2f7e928f4e Bump default silent timeout to 2 hours 2014-08-25 11:35:56 +02:00
Eelco Dolstra d6b61f4925 More charset hackery 2014-08-23 16:39:20 +02:00
Sander van der Burg ede4a10353 Compose release name for an evaluation if none has been defined 2014-08-23 16:20:58 +02:00
Eelco Dolstra 319e053368 Hopefully fix UTF-8 chars in Hipchat notification messages 2014-08-18 14:01:39 +02:00
Eelco Dolstra 2ffceef875 Tweak 2014-08-17 23:54:12 +02:00
Eelco Dolstra 28b528c1ab Ensure proper escaping of /build/.../contents pages 2014-08-17 23:43:22 +02:00
Eelco Dolstra f50066ab6f Fix encoding of Hydra logs 2014-08-17 23:43:01 +02:00
Eelco Dolstra a80bfceaca Remove timeout detection hack 2014-08-17 19:26:03 +02:00
Eelco Dolstra 42b23133e8 Fix UTF-8 handling of log files 2014-08-13 18:53:29 +02:00
Eelco Dolstra a2b27c7cf2 Preserve whitespace in string inputs 2014-08-13 17:25:08 +02:00
Eelco Dolstra 9334b84d6f Fix registerRoot 
Fixes #175.
 2014-08-13 16:29:00 +02:00
Eelco Dolstra 3520315fea Fix build and handling of string inputs starting with a dash 
Fixes #176.
 2014-08-13 16:24:26 +02:00
Eelco Dolstra 69e3aa0438 Write Hydra roots as regular files instead of symlinks 
Note that this requires at least NixOS/Nix@1c208f2b7e.
 2014-08-01 17:24:55 +02:00
Ronny Esterluss 7351752066 fixed string comparison 2014-07-31 22:18:28 +02:00
Tino Breddin 5ebc2e5419 only checkout the target branch when topgit is used 2014-07-31 22:18:28 +02:00
Ronny Esterluss 55aaf807eb added clean command 2014-07-31 21:53:25 +02:00
Ronny Esterluss ee840be313 forcing branch switch 2014-07-31 21:53:25 +02:00
Eelco Dolstra 6b88be040e Make restartBuilds faster 2014-07-18 00:03:26 +02:00
Eelco Dolstra 365de86ead Fix hydra-update-gc-roots 2014-07-16 23:20:58 +02:00
Eelco Dolstra fb5f01097b Fix race between hydra-eval-jobs and hydra-update-gc-roots 
If hydra-eval-jobs creates a new root, and hydra-update-gc-roots runs
before hydra-evaluator has had a chance to add the corresponding build
to the database, then hydra-update-gc-roots will remove the root. If
subsequently the Nix garbage collector kicks in, it may remove the
build's .drv file before the build is performed. Since evaluation of
the Nixpkgs and NixOS jobsets nowadays takes a lot of time (e.g. an
hour), the probability of this happening is fairly high.

The quick fix is not to delete roots that are less than a day old. So
long as evaluation doesn't take longer than a day, this should be fine
;-)

Fixes #166.
 2014-07-14 13:18:07 +02:00
Shea Levy 114f8a26ee Fix build for new nixUnstable 2014-06-16 12:24:17 -04:00
Rob Vermaas 268f273a3e Add value and dependency fields to eval json output. 2014-05-08 17:08:31 +02:00
Eelco Dolstra da73eb5f8a Include dependency column in JSON output 2014-05-08 16:36:15 +02:00
Rob Vermaas 1c92d760fa Merge pull request #161 from thoughtpolice/coverity 
hydra: add Coverity Scan plugin
 2014-05-03 17:46:35 +02:00
Austin Seipp 4166974657 hydra: add Coverity Scan plugin 
This adds a Hydra plugin for users to submit their open source projects
to the Coverity Scan system for analysis.

First, add a <coverityscan> section to your Hydra config, including the
access token, project name, and email, and a regex specifying jobs to
upload:

    <coverityscan>
      project = testrix
      jobs    = foobar:.*:coverity.*
      email   = aseipp@pobox.com
      token   = ${builtins.readFile ./coverity-token}
    </coverityscan>

This will upload the scan results for any job whose name matches
'coverity.*' in any jobset in the Hydra 'foobar' project, for the
Coverity Scan project named 'testrix'.

Note that one upload will occur per job matched by the regular
expression - so be careful with how many builds you upload.

The jobs which are matched by the jobs specification must have a file in
their output path of the form:

  $out/tarballs/...-cov-int.(xz|lzma|zip|bz2|tgz)

The file must have the 'cov-int' directory produced by `cov-build` in
the root.

(You can also output something into
$out/nix-support/hydra-build-products for the Hydra UI.)

This file will be found in the store, and uploaded to the service
directly using your access credentials. Note the exact extension: don't
use .tar.xz, only use .xz specifically.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
 2014-05-02 14:15:25 -05:00
Eelco Dolstra 1306291c6c Handle utf-8 properly 
Fixes errors like:

  Caught exception in engine "Wide character in syswrite at /nix/store/498lwsrn5kkdh1q8kn3vcpd3457w6m7a-hydra-perl-deps/lib/perl5/site_perl/5.16.3/Starman/Server.pm line 547."

Note that these errors didn't happen if the database encoding was set
to SQL_ASCII (which was the case for hydra.nixos.org, explaining why
it didn't get these errors). However, now the encoding must be
UTF8. To change it, do:

  update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'hydra';
 2014-05-01 16:33:25 +02:00
Eelco Dolstra 67c79a16a8 Remove unnecessary join against releases table 
This gets rid of the warning:

  DBIx::Class::Storage::DBI::select_single(): Query returned more than one row.  SQL that returns multiple rows is DEPRECATED for ->find and ->single at /home/eelco/Dev/hydra/src/script/../lib/Hydra/Controller/Project.pm line 15
 2014-05-01 13:03:34 +02:00
Eelco Dolstra 6c6ce7a781 Fix cloning jobsets 2014-04-30 09:26:53 +02:00
Eelco Dolstra 4d9edd0ca8 Also redirect /job requests that refer to a renamed jobset 2014-04-24 11:05:16 +02:00
Eelco Dolstra 23416f9960 Do a permanent redirect 2014-04-24 10:52:46 +02:00
Eelco Dolstra affec8881d Remove support for views 
They're replaced by aggregates, which are declarative, faster, and
have a better interface.
 2014-04-23 23:22:44 +02:00
Eelco Dolstra 5e0d10a392 When renaming a jobset, add a redirect from the old name 2014-04-23 23:15:22 +02:00
Eelco Dolstra be63c50560 Show whether a job still exists 
In the dashboard and on the job page, indicate whether the job appears
in the latest jobset eval. That way, the user gets some indication if
a job has accidentally disappeared (e.g. due to an evaluation error).
 2014-04-08 18:50:37 +02:00
Ludovic Courtès 4471cae07e hydra-eval-guile-jobs: Add "This file is part of Hydra". 2014-04-08 18:22:21 +02:00
Ludovic Courtès cc46456598 hydra-eval-guile-jobs: Register derivations as GC roots. 
* src/script/hydra-eval-guile-jobs.in (register-gc-root): New
  procedure.
  (job-evaluations->sxml): Add #:gc-roots-dir parameter.  Call
  'register-gc-root'.
  (job-evaluations->xml): Add #:gc-roots-dir parameter; pass it to
  'job-evaluations->sxml'.
  (eval-guile-jobs): Warn when --gc-roots-dir isn't passed.  Pass
  GC-ROOTS-DIR to 'job-evaluations->xml'.
 2014-04-08 18:22:21 +02:00
Ludovic Courtès e3e8c1bc74 hydra-eval-guile-jobs: Honor $NIX_STORE_DIR. 
* src/script/hydra-eval-guile-jobs.in (strip-store-path): Honor
  $NIX_STORE_DIR.
 2014-04-08 18:22:21 +02:00
Eelco Dolstra 4cf0d7c4b3 hydra-update-gc-roots: Keep derivations of failed builds 
By keeping the derivations of failed builds in the most recent
evaluations, we ensure that failed builds can be restarted.
 2014-04-08 17:54:11 +02:00
Eelco Dolstra 4aa01e021e Fix building against the latest Nix 2014-04-08 17:08:09 +02:00
Shea Levy d087aeb0d9 Fix adding jobset inputs 2014-03-06 08:22:59 -05:00
Shea Levy 024a5e7d75 edit-jobset.tt: Retain old inputs with dashes in their name 
Fixes #149
 2014-03-06 07:50:49 -05:00
Eelco Dolstra 2d8f2503c5 Build against latest Nix 2014-03-05 11:11:59 +01:00
Ludovic Courtès ccdb6f2b23 Honor $NIX_STORE_DIR. 2014-02-28 14:09:12 +01:00
Your Name f409d2b4f6 Allow notify option to be set for hipchat notification. (https://www.hipchat.com/docs/api/method/rooms/message) 2014-02-28 09:27:33 +00:00
Your Name 421950aede Use minutes in build history charts. 2014-02-28 09:16:45 +00:00
Ludovic Courtès 61448ca2bd guile: Raise default absolute timeout to from 2h to 20h. 2014-02-27 22:19:52 +01:00
Shea Levy b29d7b1091 Only email responsible authors when the build has changed 2014-02-26 11:05:46 -05:00
Eelco Dolstra 8dcef7702f Add jobset/job menus on buid log pages 2014-02-26 11:49:28 +01:00
Eelco Dolstra 66b8c1a9e0 Return 410 Gone (rather than 500) if an output is no longer available 2014-02-26 11:38:02 +01:00
Rob Vermaas 6327edd63f Add a convenient way to get logs of a path/drv. Requested by phreedom. 2014-02-19 10:21:59 +00:00
Eelco Dolstra 73a0cb57c5 Jobset evals page: Don't show nrFailed if it's zero 2014-02-18 11:08:12 +01:00
Hydra d4d9896f9f Re-add the history tab (store size and build time), now as lazy-loaded tab on build page. 2014-01-28 11:55:19 +01:00
Rob Vermaas ac177d0984 Add nixexprpath/nixexprinput to GET of jobset (API). 2014-01-23 12:44:02 +01:00
Eelco Dolstra 2398f2ce49 Spelling fix 2014-01-15 16:09:02 +01:00
Eelco Dolstra 492fe121ea Jobset eval page: Fix handling of aborted/cancelled builds 
The fix in 85d51074 was not quite right.
 2014-01-15 16:06:47 +01:00
Eelco Dolstra cad2c07a35 Fix broken redirects 2014-01-14 18:17:33 +01:00
Eelco Dolstra 6572869596 clear-queue-non-current: Work around PostgreSQL query optimizer stupidity 2014-01-14 18:14:10 +01:00
Eelco Dolstra 62809f484f hydra-eval-jobs: Fix building against the latest unstable Nix 2014-01-12 17:37:56 +01:00
Rob Vermaas 1d392f64c6 Fix typo. 2014-01-10 15:20:19 +01:00
Rob Vermaas bd7b6fc401 Allow Hydra to run as a private instance by requiring a login. 
Use the following in your hydra.conf to make your instance a
private Hydra instance (public is the default):

  private 1

Currently, this will not allow you to use the API, channels
and the binary cache when running in private mode. We will add
solutions for these functionalities later.
 2014-01-10 11:04:28 +01:00
Rob Vermaas 3e1a98668f Allow configuring a set of domains to allow logins from Persona. 
E.g. add the following to only allow Persona logins from email.com and gmail.com email addresses.

  persona_allowed_domains email.com,gmail.com
 2014-01-09 13:31:02 +01:00
Eelco Dolstra a598fe7e81 Allow Hydra's binary cache to be signed 
This requires adding the following to hydra.conf:

  binary_cache_key_name = <key-name>
  binary_cache_private_key_file = <path-to-private-key>

e.g.

  binary_cache_key_name = hydra.nixos.org-1
  binary_cache_private_key_file = /home/hydra/cache-key.sec
 2014-01-08 15:19:17 +01:00
Shea Levy de26b55afe User.pm: Handle params from JSON properly 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-12-12 09:38:57 -05:00
Rob Vermaas f25f8dbfa3 Higher timeout for prefetching bazaar inputs. 2013-12-11 17:34:45 +01:00
Shea Levy a92a57f3b0 hydra-evaluator: When evaluating a single jobset, exit with a failure code if evaluation fails 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-11-27 14:29:05 -05:00
Eelco Dolstra dd4e57fb0c Allow passing a specific build as an input 
Fixes #62.
 2013-11-11 21:36:26 +00:00
Eelco Dolstra 8f104396ec Support passing a jobset evaluation as an input 
All successful, non-garbage-collected builds in the evaluation are
passed in a attribute set.  So if you declare a Hydra input named
‘foo’ of type ‘eval’, you get a set with members ‘foo.<jobname>’.  For
instance, if you passed a Nixpkgs eval as an input named ‘nixpkgs’,
then you could get the Firefox build for x86_64-linux as
‘nixpkgs.firefox.x86_64-linux’.

Inputs of type ‘eval’ can be specified in three ways:

* As the number of the evaluation.

* As a jobset identifier (‘<project>:<jobset>’), which will yield the
  latest finished evaluation of that jobset.  Note that there is no
  guarantee that any job in that evaluation has succeeded, so it might
  not be very useful.

* As a job identifier (‘<project>:<jobset>:<job>’), which will yield
  the latest finished evaluation of that jobset in which <job>
  succeeded.  In conjunction with aggregate jobs, this allows you to
  make sure that the evaluation contains the desired builds.
 2013-11-11 21:17:22 +00:00
Eelco Dolstra 7b35e4d0de Add a link to the latest finished eval of a jobset 2013-11-11 14:49:39 +01:00
Eelco Dolstra a0fdf8d2ba Use local copy of the Persona sign in button 2013-11-06 22:20:57 +01:00
Eelco Dolstra 4fb3d67f49 Make /login give a proper JSON response again 2013-11-06 16:12:07 +01:00
Eelco Dolstra 60e7e4fbe9 Remove obsolete/broken timeline feature 2013-11-06 15:20:30 +01:00
Eelco Dolstra e8d92445d7 Keep sessions for a week 2013-11-06 15:15:35 +01:00
Eelco Dolstra 5723fda7fc Keep session state across hydra-server restarts 2013-11-06 15:14:31 +01:00
Eelco Dolstra 33f01da507 Fix hydra_logo setting 2013-11-06 15:10:26 +01:00
Eelco Dolstra df1a33c63f Fix layout at all page sizes 2013-11-06 14:50:21 +01:00
Eelco Dolstra 3c45992182 Make the logo configurable via hydra.conf 2013-11-06 14:40:30 +01:00
Eelco Dolstra b44baf3746 Add a flag to enable Persona support 
It's disabled by default.  To enable, add the following to hydra.conf:

  enable_persona = 1
 2013-11-06 14:38:36 +01:00
Eelco Dolstra d118a575e1 Use page-header on the front page 2013-11-06 14:11:56 +01:00
Eelco Dolstra 8928c35ecd Slight layout tweak 2013-11-06 14:07:43 +01:00
Eelco Dolstra 55f9d23933 Add a command `hydra-create-user' for managing user accounts 2013-11-06 13:36:29 +01:00
Eelco Dolstra 3315d1ea51 Remove obsolete hydra-control script 2013-11-06 11:42:49 +00:00
Eelco Dolstra 7a18e5f0c1 Remove unused file 2013-11-06 11:42:04 +00:00
Eelco Dolstra 63100b0ee8 Merge branch 'master' into persona 2013-11-06 12:35:36 +01:00
Eelco Dolstra 27a740d4b1 Don't show an empty projects list 2013-11-06 12:15:11 +01:00
Eelco Dolstra a0b58b6b62 Fix typo that caused empty news item to show up 2013-11-06 12:15:11 +01:00
Eelco Dolstra ecadcef642 Prevent a division by zero in hydra-queue-runner 
Fixes #131.
 2013-11-06 12:15:11 +01:00
Shea Levy f1c187198c Stupid capitalization scheme 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-11-05 12:39:55 -05:00
Shea Levy 2e1c07c5f7 Argh 
ALWAYS RUN THE TESTS

Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-11-05 12:31:36 -05:00
aszlig cfd4843290 topbar: Include "Create project" in menu. 
For users who only have the "create-projects" role, actually display the
item in the menu as the only option.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2013-11-05 17:03:31 +01:00
aszlig f6a83bcc5b user: Fix spelling of create-projects role. 
This actually is right in the manual but displayed incorrectly on the
web interface.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2013-11-05 17:03:31 +01:00
Eelco Dolstra 8a0200edd2 Doh 2013-11-05 16:13:09 +01:00
Eelco Dolstra ae85d2df23 Add a "My jobsets" tab to the dashboard 2013-11-05 16:05:29 +01:00
Shea Levy d5574230e4 typo 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-11-05 09:57:15 -05:00
Shea Levy 217ba725a3 Restore accidentally-removed fields to the Builds API 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-11-05 09:43:32 -05:00
Shea Levy 385ceaff4d Show buildinput and buildproduct information in the Builds API 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-11-05 09:42:05 -05:00
Eelco Dolstra 589271bd72 In the "My jobs" tab, ignore disabled projects/jobsets 2013-11-05 14:53:52 +01:00
Eelco Dolstra 27b795f46e Doh 2013-11-05 14:46:20 +01:00
Eelco Dolstra fc827a003a Be paranoid about the Persona email address 2013-11-05 14:40:40 +01:00
Eelco Dolstra b53bd868fb Add a "My jobs" tab to the dashboard 
It shows all current jobs of which the user is a maintainer.
 2013-11-05 14:35:49 +01:00
Eelco Dolstra 3c981bed02 Slight cleanup in the Persona sign in code 2013-11-05 14:10:20 +01:00
Eelco Dolstra bdd6e99fbd Clean up user editing 2013-11-05 14:02:04 +01:00
Eelco Dolstra bd927eba66 Make sign in a modal dialog box rather than a separate page 2013-11-05 13:13:02 +01:00
Eelco Dolstra 299ac07cbc Re-enable adding new users via the web interface 
Also unify user creation and user editing.
 2013-11-05 12:41:10 +01:00
Eelco Dolstra 36c961a376 Store the account type ("hydra" or "persona") explicitly in the database 2013-11-05 11:46:05 +01:00
Eelco Dolstra 14b9b201e1 Don't show a password reset button for Persona accounts 2013-11-05 11:18:44 +01:00
Eelco Dolstra b54cfbf032 Merge branch 'master' into persona 
Conflicts:
	src/lib/Hydra/Helper/CatalystUtils.pm
	src/root/layout.tt
	src/root/topbar.tt
	src/root/user.tt
 2013-11-05 11:11:48 +01:00
Eelco Dolstra 32b6ddf772 hydra-eval-jobs: Support meta.license being a list 
TODO: PostgreSQL has an array column type, we should use it for the
license and maintainers columns.
 2013-11-04 22:50:32 +01:00
Eelco Dolstra 647d6bc3f1 Force creation of a new uncached eval if jobs have been removed 
Previously we only checked if jobs had been added.  We should probably
rename the "hasNewBuilds" field in the database.
 2013-11-01 19:24:52 +01:00
Eelco Dolstra a04c117eb6 Revert "Remove wacky "sysbuild" filtering" 
This reverts commit 2d7e106d29.

Unfortunately some jobsets still depend on this behaviour.  They could
probably do something like "assert system == input.system; ..." but
changing them all is undesirable.
 2013-11-01 18:30:36 +01:00
Eelco Dolstra cbd56824a2 Use /etc/nix/machines 2013-10-30 11:21:23 +01:00
Eelco Dolstra a919b503fb Restore api/get-info 2013-10-29 19:53:23 +01:00
Shea Levy 5da3dcfd57 Make the outputs' outpaths available via the Build JSON API 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-29 14:41:48 -04:00
Shea Levy 4eef090a45 Flesh out the API a bit 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-29 13:54:11 -04:00
Shea Levy b51b8cddf4 Add in missing needed join 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-25 14:06:52 -04:00
Shea Levy 8e82ba0093 Get rid of old ToJSON code path 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-25 11:09:11 -04:00
Shea Levy a89096ec5e Clean up root controller 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-25 11:09:11 -04:00
Shea Levy 5d3df69a0b Cleanup build serialization 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-25 11:09:11 -04:00
Shea Levy e04477003e Cleanup eval serialization 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-25 11:09:11 -04:00
Shea Levy 917660442e Add eager fetching of relations and enable that for jobset->jobsetinputs 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-25 11:09:11 -04:00
Shea Levy ec6568f9b7 Fix jobset input handling in the API 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-25 11:09:06 -04:00
Shea Levy 04d8adaad3 Cleanup Jobset serialization 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-24 11:39:40 -04:00
Shea Levy 78dcd3e619 Cleanup Project model 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-24 11:39:40 -04:00
Shea Levy ee8275cfdb Start api cleanup with the User model 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-24 11:39:40 -04:00
Eelco Dolstra f551de21f7 Fix creating jobsets in a RESTful way 2013-10-24 17:35:58 +02:00
Eelco Dolstra c719bebcb4 Remove redundant newlines 2013-10-17 11:20:27 +02:00
Shea Levy 806af3fe68 Fix typos 
I could've sworn I tested this...

Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-16 08:57:48 -04:00
Shea Levy 0db950931a Merge remote-tracking branch 'upstream/who-broke-builds' into upstream-master 
Include information about who changed the build status in notification
emails, and enable optional per-input notification of said committers.

Conflicts due to two branches modifying the database schema.

Signed-off-by: Shea Levy <shea@shealevy.com>

Conflicts:
	src/lib/Hydra/Schema/Jobsets.pm
	src/sql/upgrade-23.sql
 2013-10-15 09:49:20 -04:00
Eelco Dolstra 2127d133cd Add a dashboard 
Currently the dashboard allows users to get a quick overview of the
status of jobs they're interested in, but more will be added,
e.g. viewing all your jobsets or all jobs of which you're a
maintainer.
 2013-10-14 20:07:26 +02:00
Eelco Dolstra 09b5679ee7 Clean up the authorization code a bit 2013-10-14 18:01:04 +02:00
Eelco Dolstra 86e9abeb15 Use c.req.captures 2013-10-14 17:43:31 +02:00
Eelco Dolstra 854d419b2c Use redirectJSON 2013-10-14 17:35:14 +02:00
Eelco Dolstra c4e39d4769 Add one-shot jobsets 
There are jobsets that are evaluated only once, that is, after they've
been evaluated, they're disabled automatically.  This is primarily
useful for doing releases: for instance, doing an evaluation with
"officialRelease" set to "true" should be done only once.
 2013-10-11 12:01:52 +02:00
Eelco Dolstra f592ce0026 Fix extreme slowness in hydra-queue-runner 
If there are builds in the queue that depend on another scheduled
build, then hydra-queue-runner will start the dependency first and
block the dependent builds.  This is implemented in
findBuildDependencyInQueue.  However, if there are tens of thousands
of such dependent builds, since each call to
findBuildDependencyInQueue may take a second or so, hydra-queue-runner
will spend hours just deciding which builds *not* to do.  Thus very
little progress is made.

So now, when a build is started, we immediately check which builds are
"blocked" by it (i.e. depend on it), and remove such builds from
consideration.
 2013-10-11 10:54:02 +02:00
Eelco Dolstra a49457b2fd Don't break inside durations 2013-10-10 12:43:27 +02:00
Shea Levy 26470f1656 Check all inputs for blame but only email selected inputs 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 14:47:24 -04:00
Shea Levy 804617f075 Email responsible authors if requested 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy f8b80c99c2 Include who-broke-the-build information in notification emails 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 272d9e235d Remove unused assignment 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 2c90857689 getResponsibleAuthors: Respect checkResponsible 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 58ad3b4b6c Enable setting checkresponsible in the edit jobset form 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 3e1f930928 Enable setting emailresponsible in the edit jobset form 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 3e4a4e3761 Propagate checkresponsible from JobsetInput to BuildInput 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 07157f8125 Update Schema classes 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 6342464110 Add DB columns for when to notify responsible committers and which inputs should be checked 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Shea Levy 7913701d54 Factor a getResponsibleAuthors helper out of the HipChatNotification 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-10-08 13:24:49 -04:00
Rob Vermaas 20f1bf215a Make actions dropdown easier to find by making the dropdown title bold 2013-10-08 13:32:46 +02:00
Eelco Dolstra 0ec03aa0f4 Handle builds with multiple outputs correctly in Hydra channels 2013-10-07 17:06:17 +02:00
Eelco Dolstra 5ccff14f6b In Hydra channels, show only packages matching the user's system type 
Fixes NixOS/nix#169.
 2013-10-07 14:53:27 +02:00
Eelco Dolstra 5294a0a8a0 Register restarted derivations as GC roots 2013-10-04 17:11:42 +02:00
Eelco Dolstra 052bab169d Add a jobset eval action to restart all aborted/cancelled builds 2013-10-04 17:01:47 +02:00
Eelco Dolstra 85d51074b9 Jobset eval page: Show all aborted builds under the "aborted jobs" tab 
Previously some might be included under the "new jobs" tab
 2013-10-04 16:36:22 +02:00
Eelco Dolstra e334ff541d Jobset eval page: Show the number of builds in each tab 2013-10-04 16:35:56 +02:00
Eelco Dolstra aa49b128a8 Remove the "clone build" feature 2013-10-04 15:43:51 +02:00
Eelco Dolstra 7818bb75ed Add an action to cancel all builds in a jobset eval 2013-10-04 15:40:43 +02:00
Eelco Dolstra 3e54f0a6ab Fix an uninitialized value warning 2013-10-04 14:47:30 +02:00
Eelco Dolstra ee5b655535 Maintain the order of the input alternatives 2013-10-03 20:03:57 +02:00
Eelco Dolstra 550bf210fe Use more flash messages 2013-10-03 19:54:22 +02:00
Eelco Dolstra 383bc62803 Restore link to clearvcscache 2013-10-03 19:43:21 +02:00
Eelco Dolstra 720c3892a3 Use delete instead of delete_all 
DBIC's delete_all method fetches all rows separately, which is slow.
 2013-10-03 19:42:44 +02:00
Eelco Dolstra f32077b5e8 Simplify jobset cloning 
We can just show the normal "edit jobset" page for the original jobset
and then do a PUT request to create a new jobset.

Also simplified updating the jobset inputs.  We can just delete all of
them and recreate them from the user parameters.  That's safe because
it's done in a transaction.
 2013-10-03 19:28:25 +02:00
Eelco Dolstra 232f46c750 Use the REST API in the web interface for editing jobsets 2013-10-03 18:49:37 +02:00
Eelco Dolstra 63062f7bba Instead hard breaks in multi-line error messages 2013-10-03 18:05:37 +02:00
Eelco Dolstra 851c3329d0 Implement DELETE for jobsets and use it in the web interface 2013-10-03 17:54:40 +02:00
Eelco Dolstra ae13dd3e67 Use the REST API in the web interface for editing projects 2013-10-03 17:23:41 +02:00
Eelco Dolstra 8355dc57ee Ensure proper encoding of error messages 2013-10-03 17:20:00 +02:00
Eelco Dolstra acdeaa1d80 Fix tabs 2013-10-03 15:14:55 +02:00
Eelco Dolstra 97c76f34df Remove more dead code 2013-10-03 15:14:40 +02:00
Eelco Dolstra ed132cf1fc Remove obsolete JobStatus source 2013-10-03 15:05:53 +02:00
Eelco Dolstra c7a039bd05 Remove the Errors page 
The same info is readily available under the most recent jobset eval
page.
 2013-10-03 15:00:28 +02:00
Eelco Dolstra 64d617a6b2 Remove already disabled jobstatus code 2013-10-03 14:56:07 +02:00
Eelco Dolstra adcdfcde6b Remove some obsolete JSON APIs 2013-10-03 14:50:56 +02:00
Eelco Dolstra c8e5faf81e Clean up error handling 2013-10-03 14:45:23 +02:00
Eelco Dolstra 4ddb173ca1 Use notFound instead of status_not_found 2013-10-03 14:16:21 +02:00
Eelco Dolstra 2166c478ef Fix and clean up editing project settings 2013-10-03 14:05:10 +02:00
Eelco Dolstra b1f7096935 Restore old findBuildDependencyInQueue behaviour 2013-10-03 13:08:32 +02:00
Eelco Dolstra b1a26e6caa Revert "Add a dependency_lookup configuration option to enable (slow) dependency lookup in queue. This behaviour was disabled temporarily in accefbb79 due to slowness in very large queues, but some people might be dependent on it, so it is configurable until the previous behaviour is implemented more efficiently." 
This reverts commit 24f5a6b15f.
 2013-10-03 13:07:32 +02:00
Eelco Dolstra 9f3b47c963 Allow only project owners to delete projects / jobsets 
Giant gaping security hole.
 2013-10-03 13:06:16 +02:00
Eelco Dolstra 7a5ec00ea1 Fix a "Argument isn't numeric" warning if there are not jobsets 2013-10-03 13:04:20 +02:00
Eelco Dolstra 77fd72070f Fix showing the project's display name 2013-10-03 10:17:53 +00:00
Rob Vermaas 24f5a6b15f Add a dependency_lookup configuration option to enable (slow) dependency lookup in queue. This behaviour was disabled temporarily in accefbb79 due to slowness in very large queues, but some people might be dependent on it, so it is configurable until the previous behaviour is implemented more efficiently. 2013-10-03 09:09:18 +00:00
Eelco Dolstra cc1fcf657c Show aborted/cancelled builds separately in jobset eval pages 2013-10-03 01:54:42 +02:00
Eelco Dolstra 9801cb76db Add an action menu to evaluations 2013-10-03 01:34:17 +02:00
Eelco Dolstra 4fa2821eea Move more actions from the top bar 2013-10-03 01:17:52 +02:00
Eelco Dolstra a5cfae078e Remove the Build menu from the top bar 
It's now a dropdown menu in the tabs thingy, which subsumes the
"Reproduce locally" button.  This makes the actions in the menu a bit
more visible, IMHO.
 2013-10-02 19:10:00 +02:00
Eelco Dolstra f50477141d Add an input type "nix" for passing arbitrary Nix expressions 2013-09-30 12:03:25 +02:00
Eelco Dolstra 4dd1197d89 Fix uninitialized value warning 2013-09-30 10:01:09 +00:00
Eelco Dolstra af2b0c8bad Remove dead code 2013-09-30 11:57:38 +02:00
Eelco Dolstra baafe77489 Fix HTML error 
From Mats Erik Andersson.
 2013-09-30 11:18:48 +02:00
Eelco Dolstra d46ebeea99 Distinguish between permanent evaluation errors and transient input errors 
Fixes #112.
 2013-09-25 16:21:16 +02:00
Eelco Dolstra e1c9e28589 Handle UTF-8 characters in eval error messages 2013-09-25 15:51:03 +02:00
Eelco Dolstra a8db329839 Warn against multiple jobs with the same name 2013-09-25 15:30:59 +02:00
aszlig 0b202580ce templates: Use uri_for to reference static paths. 
This commit is provided by (zsh syntax):

sed -i 's|/static[^"]*|[% c.uri_for("&") %]|;s/\[% size %\]/${size}/' **/*.tt

And the reason for this change is to make it easier to change the base
path with headers like X-Request-Base to be served within a URI prefix,
especially when behind a reverse proxy.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
 2013-09-25 14:37:18 +02:00
Eelco Dolstra a2491f76a4 Use the same start/stop time for the build steps as for the build 2013-09-25 01:00:20 +02:00
Eelco Dolstra f037a318e3 *headdesk* 
DBIC::Class helpfully doesn't warn you when you're matching against
unselected columns.  So this query actually returned all builds...
 2013-09-25 01:00:20 +02:00
Eelco Dolstra 2d7e106d29 Remove wacky "sysbuild" filtering 
It's none of our business if a jobset wants to return a build that
uses a build for another system as an input...
 2013-09-25 01:00:20 +02:00
Rob Vermaas b1e29e50a7 Only send email notification of evaluation error when the evaluation error has changed. Fixes #121. 2013-09-24 12:01:57 -04:00
Eelco Dolstra 550d6b79b1 Allow dots in jobset names (like "release-13.09") 2013-09-24 15:15:44 +02:00
Shea Levy 1a313ad566 Allow dashes in jobset input names 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-09-23 17:53:35 -04:00
Shea Levy 6d5a3d0580 Derivations with multiple outputs break the 'link name is store path' assumption 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-09-22 21:26:59 -04:00
Shea Levy 98c9559bf2 Delay initializing the s3 client until it's needed to silence warnings 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-09-22 18:48:37 -04:00
Eelco Dolstra 77dbf55abb hydra-queue-runner: Tweaked the selection method 
Pick the jobset that has used the smallest fraction of its share,
rather than the jobset furthest below its share in absolute terms.
This gives jobsets with a small share a quicker start (but they
will also run out of their share quicker).
 2013-09-21 19:54:58 +00:00
Eelco Dolstra cf43c605cd hydra-queue-runner: Cache the lookup of time spent per jobset 2013-09-21 19:54:46 +00:00
Eelco Dolstra 4cdf1a270d hydra-queue-runner: Set the start time properly 2013-09-21 19:38:02 +00:00
Eelco Dolstra 52ce662710 hydra-queue-runner: Don't kill builds we just started 2013-09-21 20:51:43 +02:00
Eelco Dolstra accefbb798 hydra-queue-runner: Disable findBuildDependencyInQueue for now 
It's way too slow.
 2013-09-21 20:35:02 +02:00
Eelco Dolstra 24c757845d Fix lazy tab loading 2013-09-21 19:25:01 +02:00
Eelco Dolstra 9602499c1c hydra-evaluator: Do the actual work in a subprocess 
This should get rid of the slow memory leaks exhibited by
hydra-evaluator.
 2013-09-21 15:49:27 +00:00
Eelco Dolstra 4ed877360b hydra-queue-runner: Improved scheduling 
Each jobset now has a "scheduling share" that determines how much of
the build farm's time it is entitled to.  For instance, if a jobset
has 100 shares and the total number of shares of all jobsets is 1000,
it's entitled to 10% of the build farm's time.  When there is a free
build slot for a given system type, the queue runner will select the
jobset that is furthest below its scheduling share over a certain time
window (currently, the last day).  Withing that jobset, it will pick
the build with the highest priority.

So meta.schedulingPriority now only determines the order of builds
within a jobset, not between jobsets.  This makes it much easier to
prioritise one jobset over another (e.g. nixpkgs:trunk over
nixpkgs:stdenv).
 2013-09-21 14:57:01 +00:00
Eelco Dolstra 7efe793ee6 Fix the Darcs schema upgrade 2013-09-19 12:00:29 +00:00
Shea Levy 74388353b5 Add a plugin for backing up builds in s3 
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:

* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
  format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
  meant to represent a directory, you should include the trailing slash,
  e.g. "cache/"). Default "".

After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.

This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.

This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix

Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.

Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-09-18 18:32:58 +02:00
Eelco Dolstra 4705af48b8 hydra-build: Hack to handle timeouts 2013-09-18 13:06:35 +00:00
Eelco Dolstra e4cbd057e2 Fix an uninitialized value warning 2013-09-18 12:51:51 +00:00
Eelco Dolstra e54b536bb7 hydra-update-gc-roots: Don't keep the most recent successful view result 
Views are deprecated.
 2013-09-18 11:12:33 +00:00
Eelco Dolstra 2845d46d21 hydra-update-gc-roots: Keep more evals 
We now keep *all* unfinished evaluations of a jobset, in addition to
the <keepnr> most recent finished evaluations.

The main motivation is to ensure that mirror-{nixos,nixpkgs} work
properly: if building an evaluation takes too long, some of its builds
may already have been garbage-collected by the time the others finish.
 2013-09-18 11:10:10 +00:00
Eelco Dolstra 3b70a807b5 Allow setting keepnr to 0 2013-09-18 11:27:37 +02:00
Eelco Dolstra c667b9112d Don't trigger evaluation of disabled jobsets 2013-09-18 11:25:52 +02:00
Eelco Dolstra c14a333062 Revive release button on eval page 2013-09-17 16:39:50 +02:00
Rob Vermaas 597fd827b1 Merge pull request #81 from mornfall/master 
DARCS support
 2013-09-14 06:18:48 -07:00
Eelco Dolstra 3f68076577 hydra-build: Don't send a giant query to the database 
We had Postgres barfing with this error:

  DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: stack depth limit exceeded

because the ‘drvpath => [ @dependentDrvs ]’ in failDependents can
cause a query of unbounded size.  (In this specific case there was a
failure of Bison, which has > 10000 dependent derivations.)  So now we
just get all scheduled builds from the DB.
 2013-09-10 11:01:29 +00:00
Eelco Dolstra 35aad40692 Kill builds that produce more than 64 MiB of log output 2013-09-10 10:33:55 +00:00
Petr Rockai 0041c336d1 DarcsInput: Avoid a chdir. 2013-09-07 14:47:49 +02:00
Petr Rockai d31e4469bb Put a 5-second CPU time limit on the log processing pipeline. 2013-09-07 13:28:53 +02:00
Petr Rockai 1e1a1f0838 Also allow : (colon) in pathCompRE. 2013-09-07 13:28:53 +02:00
Petr Rockai bb38bdfe35 Allow : and + in build product filenames. 2013-09-07 13:28:53 +02:00
Petr Rockai dcf386cfed Make getDrvLogPath work with both bucketed and non-bucketed nix logs. 2013-09-07 13:28:53 +02:00
Petr Rockai bb8059e50c Add support for fetching URLs as inputs. 2013-09-07 13:28:53 +02:00
Petr Rockai 5a35912956 Add support for darcs repositories. 2013-09-07 13:28:51 +02:00
Eelco Dolstra 5078730cb5 Use evalFile() instead of parseExprFromFile() 2013-09-06 15:17:27 +00:00
Shea Levy 788c2bebaa Not every buildstep has a defined drvpath 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-09-03 14:43:08 -04:00
Eelco Dolstra 04e6eda4c7 Remove unused function 2013-09-03 20:29:52 +02:00
Eelco Dolstra d914845ad7 Use a distinct icon for cancelled builds 2013-09-03 17:37:47 +02:00
Eelco Dolstra e8fb659d63 Fix some wellformedness issues 
Also fixed some duplicate IDs reported by Mats Erik Andersson.
 2013-09-03 17:35:21 +02:00
Eelco Dolstra 614f07755a Don't show logs bigger than 64 MB 2013-08-30 20:44:04 +00:00
Eelco Dolstra 6655ada236 Serve raw uncompressed logs directly 2013-08-30 18:11:03 +00:00
Eelco Dolstra fd7e37ef89 Try harder to find build logs 
Due to the fixed-output derivation hashing scheme, there can be
multiple derivations of the same output path.  But build logs are
indexed by derivation path.  Thus, we may not be able to find the
log of a build or build step using its derivation.  So as a fallback,
Hydra now looks for other derivations with the same output paths.
 2013-08-30 13:53:25 +00:00
Eelco Dolstra 559a98cfee Use a different icon for aborted builds 2013-08-28 17:40:51 +02:00
Eelco Dolstra 2f7071ee9d Tabify the eval page 
The different types of types (new, removed, newly failing, and so
on) now each have their own tab.
 2013-08-28 17:40:51 +02:00
Eelco Dolstra 07747b7e88 Allow filtering jobs in eval pages 2013-08-28 17:40:51 +02:00
Eelco Dolstra d0bcaa6284 Allow comparing an eval against the jobset one day/week/month before 2013-08-28 17:40:51 +02:00
Eelco Dolstra ccc09c565e Sort jobsets by name 2013-08-28 17:40:51 +02:00
Eelco Dolstra 58a6fdc5ed Remove the job status pages 
They're mostly redundant since there is a faster "jobs" tab on
the jobset pages now.  The only thing the latter lacks is the
ability to see status change times, but those are quite expensive
to compute, and are visible on build pages if you really need them.
 2013-08-28 17:40:51 +02:00
Eelco Dolstra 42c4ef856f Fix incorrect removed jobs in eval pages 
PostgreSQL and Perl have different sort orders, in particular when
comparing job names such as "aspell.x86_64-linux" and
"aspellDicts.cs.i686-freebsd".  This confused the evaluation
comparison code, causing some jobs to appear as "removed".
So now we do all the sorting in Perl.

Fixes #105.
 2013-08-28 17:40:50 +02:00
Eelco Dolstra d886ff9973 Integrate the "Job status" and "All jobs" tabs 
The job status tabs now has a toggle to show inactive jobs, rendering
the "All jobs" tab redundant.
 2013-08-28 17:40:50 +02:00
Eelco Dolstra 9002b69c2d By default, show the first 250 jobs only 
This is particularly useful for the Nixpkgs jobsets, which now have
~24K jobs.
 2013-08-28 17:40:50 +02:00
Eelco Dolstra 10d959a9ff Show visual feedback that the filter is being applied 2013-08-28 17:40:50 +02:00
Eelco Dolstra 11acf0be28 Load a tab only once 2013-08-28 17:40:50 +02:00
Eelco Dolstra 410060ec8a Jobset page: Add a new tab to show job status in a matrix 2013-08-28 17:40:50 +02:00
Eelco Dolstra 82e073d043 Show build status 2 and 5 in the same way 
(Status 5 is obsolete.)
 2013-08-28 17:40:50 +02:00
Eelco Dolstra e69c7ac360 "limit" -> "rows" 2013-08-28 17:40:50 +02:00
Eelco Dolstra 7685596aa8 Transpose the aggregate constituents table 
This way, it grows vertically rather than horizontally.  Horizontal
may be more "logical", but this is more practical.
 2013-08-28 17:40:50 +02:00
Eelco Dolstra 7725038821 On aggregate job pages, show a matrix showing all the constituent builds 2013-08-28 17:40:50 +02:00
Rob Vermaas bf42392fe4 Fix typo. 2013-08-27 15:12:41 +02:00
Eelco Dolstra a57957df84 Handle job aliases in AggregateConstituents 
Aggregate constituents are derivations.  However there can be multiple
builds in an evaluation that have the same derivation, i.e. they can
alias each other (e.g. "emacs", "emacs24" and "emacs24Packages.emacs"
in Nixpkgs).  Previously we picked a build arbitrarily for the
AggregateConstituents table.  Now we pick the one with the shortest
name (e.g. "emacs").
 2013-08-27 11:48:02 +02:00
Rob Vermaas a98075f386 HipChat notification: do not include latest commits of all inputs in 'who-broke-the-build' list. Use only committers from inputs that have actually changed since previous build. 2013-08-26 11:06:10 +00:00
Eelco Dolstra 02cba75610 Add an action to download a specific output of a build as a .nar.bz2 
E.g. http://hydra/build/3515983/output/out downloads the output named
"out" as a bzip2-compressed NAR.
 2013-08-21 14:30:38 +02:00
Eelco Dolstra edb88ef452 Remove unused ActiveJobs source 2013-08-20 15:22:46 +02:00
Eelco Dolstra 14e418cafa Don't show bogus last-checked times 2013-08-16 18:26:01 +02:00
Eelco Dolstra 6264995198 Remove the jobs status page 
The per-system presentation doesn't make much sense any more given
issue #60.  It should be replaced by (say) a grid showing each job per
evaluation.
 2013-08-16 17:16:15 +02:00
Eelco Dolstra e54c361a95 Remove per-platform links from the job page 
Having different builds within a job is obsolete (issue #60), one
should have different job per platform (e.g. build.x86_64-linux).
 2013-08-16 16:39:42 +02:00
Eelco Dolstra a9c6f522e6 clear_queue_non_current: Don't use isCurrent 2013-08-16 16:38:09 +02:00
Eelco Dolstra 46f8b25c1f Keep builds that failed with output 
The user may want to look at the output, so they shouldn't be
GC'ed right away.
 2013-08-16 16:36:06 +02:00
Eelco Dolstra d16738e130 hydra-update-gc-roots: Keep the most recent evaluations 
We now keep all builds in the N most recent evaluations of a jobset,
rather than the N most recent builds of every job.  Note that this
means that typically fewer builds will be kept (since jobs may be
unchanged across evaluations).
 2013-08-16 16:21:30 +02:00
Eelco Dolstra 8e1ade4422 Fix display of non-aggregate builds 2013-08-15 13:57:47 +02:00
Eelco Dolstra 242072bbd6 Hide the views tab for project that don't have them 
Views are obsolete (replaced by the declarative "aggregate" build
mechanism) so we don't want people creating new ones.
 2013-08-15 13:54:23 +02:00
Eelco Dolstra c9a0e12804 Hide project/jobset in constituent list 2013-08-15 03:35:18 +02:00
Eelco Dolstra 72a0fa6ec5 Sort constituents by job name 2013-08-15 03:28:21 +02:00
Eelco Dolstra 06c74085b5 Make "Add to release" a modal dialog 2013-08-15 03:07:20 +02:00
Eelco Dolstra d92d83a82a Fix broken redirect when editing a release 2013-08-15 02:57:36 +02:00
Eelco Dolstra 1776d9118f Rename aggregate members to constituents 2013-08-15 02:33:10 +02:00
Eelco Dolstra e4141afcc9 On the build page, show how many aggregate constituents failed 
(Also, renamed aggregate "member" to "constituent", since "member" is
rather vague.)
 2013-08-15 02:17:06 +02:00
Eelco Dolstra 81322de94e Show aggregate members 2013-08-15 00:30:19 +02:00
Eelco Dolstra d58142b3f0 Store aggregate members in the database 
For presentation purposes, we need to know what builds are part of an
aggregate build.  So at evaluation time, look at the "members"
attribute, find the corresponding builds in the eval, and create a
mapping in the AggregateMembers table.
 2013-08-14 01:59:29 +02:00
Eelco Dolstra c27f4bbaf5 Add a redirect from an eval to a named member 
E.g. /eval/798867/job/tarball redirects to the build with job name
"tarball".
 2013-08-13 01:56:45 +02:00
Eelco Dolstra a25ea193f7 Add a /eval action to /build to go from a build to the latest finished eval 2013-08-13 01:43:50 +02:00
Eelco Dolstra fcdca0d4de Fix some XML wellformedness issues 2013-08-13 00:41:37 +02:00
Eelco Dolstra 4792256e43 Add some more fields to the JSON output of /build 2013-08-12 22:57:53 +02:00
Eelco Dolstra bef263c930 Add a ‘latest-finished’ action 
It redirects to the latest successful build from a finished
evaluation.  This is mostly useful for the Nixpkgs/NixOS mirroring
script, which need the latest finished evaluation in which some
aggregate job (such as ‘tested’ in NixOS) succeeded.
 2013-08-12 22:17:04 +02:00
Eelco Dolstra 84acccb3ea Index builds on stop time 
This is necessary to make the /all page fast, since it sorts builds on
descending stop time.
 2013-08-12 20:19:10 +02:00
Eelco Dolstra 452c8e36d1 Materialize the number of finished builds 
The NrBuilds table tracks the value of ‘select count(*) from Builds
where finished = 0’, keeping it up to date via a trigger.  This is
necessary to make the /all page fast, since otherwise it needs to do a
sequential scan on the Builds table.
 2013-08-12 20:19:10 +02:00
Eelco Dolstra 182f725612 Don't pass an undefined input 2013-08-12 18:15:11 +02:00
Eelco Dolstra d96df42c03 GitInput.pm: Don't do a chdir to the Git clone 
Doing a chdir in the parent is evil.  For instance, we had Hydra core
dumps ending up in the cloned directory.  Therefore, the function
‘run’ allows doing a chdir in the child.  The function ‘grab’ returns
the child's stdout and throws an exception if the child fails.
 2013-08-12 17:46:26 +02:00
Rob Vermaas 90eedcf256 HipChat notification: add support for Mercurial inputs for determining who might have broken the build. 2013-08-07 08:53:32 +00:00
Shea Levy 96e987bbfa Use inputTypes from plugins to determine valid input types 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-07-29 17:42:49 -04:00
Shea Levy 687ca429c3 Pass project and jobset to fetchInput 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-07-29 15:33:22 -04:00
Shea Levy f231c23b75 Only serialize JSON and HTML, not the C::C::REST defaults 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-07-26 14:25:25 -04:00
Shea Levy eab13d8736 ToJSON is automatically called recursively 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-07-26 13:54:07 -04:00
Shea Levy d6b23272e3 Don't try to serialize if there's nothing to serialize 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-07-26 12:04:27 -04:00
Shea Levy f7bcf9fc19 The field is actually enabled, not disabled 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-07-25 17:59:13 -04:00
Shea Levy 1404d33005 Show when a Project's jobsets are disabled 
Signed-off-by: Shea Levy <shea@shealevy.com>
 2013-07-25 17:48:28 -04:00
Rob Vermaas 513c030268 Do not use local clone for Bazaar inputs. 2013-07-22 20:42:17 +02:00