Commit graph

1353 commits

Author SHA1 Message Date
Eelco Dolstra 40debf1515 hydra-queue-runner: Don't unlock builds we just started
This happens if the previous iteration took more than 60 seconds.
Then the queue runner may think that builds failed to start properly
and unlock them, e.g.

build 5264936 pid 19248 died, unlocking
build 5264951 pid 19248 died, unlocking
build 5257073 pid 19248 died, unlocking
...
2013-06-07 20:15:37 +00:00
Eelco Dolstra 5d9b7c6ab2 Speed up findBuildDependencyInQueue
This was taking a long time due to the giant SQL query.

Issue #99.
2013-06-07 20:15:32 +00:00
Eelco Dolstra 8e36343b62 hydra-queue-runner: Start as many builds as possible on each iteration
Because we don't start a build if a dependency is already building,
it's possible that some or all of the $extraAllowed highest-priority
builds in the queue are not eligible.  E.g. with $extraAllowed = 32,
we might start only 3 builds even though there are thousands in the
queue.  The fix is to try all queued builds until $extraAllowed have
been started.

Issue #99.
2013-06-07 20:15:20 +00:00
Eelco Dolstra 2974fea1a7 Use https://ajax.googleapis.com if we're on https 2013-06-07 15:45:37 +02:00
Eelco Dolstra f4ecf27def Fix Perl warning about $path masking earlier definition 2013-06-02 23:32:09 +02:00
Eelco Dolstra 8e246a835a Don't do camel case in filenames 2013-06-02 23:30:46 +02:00
Eelco Dolstra 66c695e1c3 Fix ‘Unable to handle files with the extension 'conf'’
Config::Any uses Module::Pluggable to seach for plugins, so it needs
the patched Module::Pluggable in Nixpkgs (rather than the one in Perl
itself) to properly find plugins in symlink trees created by buildEnv.
2013-06-02 23:29:24 +02:00
Shea Levy 591acda6f9 reproduce.tt: Save the revCount when reproducing more than once
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-05-30 14:06:32 -04:00
Shea Levy 8e716ff062 reproduce.tt: Create the checkout dir before an hg clone
For some reason, hg clone from a local (path-based) repo will fail if
the parent directory of the destination directory doesn't exist (though
it succeeds when cloning from an http repo).

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-05-30 13:31:53 -04:00
Eelco Dolstra 1277a8fc1c Don't break certain table columns
For instance, if the "timestamp" column becomes too small, then every
timestamp is spread out across two lines, doubling the size of the
table.
2013-05-29 14:58:44 +02:00
Eelco Dolstra 47c49cf3d4 Show links on products of type "doc" 2013-05-29 11:39:23 +02:00
Eelco Dolstra 77e37c1ea3 Don't add a nix-build build product unless $out is a directory 2013-05-28 18:43:05 +02:00
Eelco Dolstra 1f1615e80b Support revision control systems via plugins 2013-05-25 15:36:58 -04:00
Eelco Dolstra 5e0542d3af Use buildEnv to combine Hydra's Perl dependencies
This makes the Perl search path ($PERL5LIB) much shorter, cutting down
the number of stat() calls when starting hydra-server from ~432000 to
~9000.
2013-05-24 15:26:47 -04:00
Eelco Dolstra e0c411220e built.tt: Handle the case where the original build is not known
And remove some parentheses.
2013-05-24 12:43:31 -04:00
Eelco Dolstra 9ac363d32a Fill in starttime/stoptime for cached builds 2013-05-24 12:43:02 -04:00
Petr Rockai 6c0577b226 Respect X-Request-Base header coming from a frontend proxy. 2013-05-23 13:07:40 -04:00
Eelco Dolstra 5e5b344527 Make latest/latest-for more efficient
Getting all matching rows from the database is expensive.
2013-05-23 12:18:38 -04:00
Eelco Dolstra 7b5e92868b Remove unused Jobs columns
Actually, we can *almost* get rid of the Jobs table entirely...
2013-05-23 12:12:19 -04:00
Eelco Dolstra b1f89bf92d Ensure that stoptime and starttime are set for finished builds 2013-05-23 12:05:12 -04:00
Eelco Dolstra 57b2bb0674 Let Builds.timestamp refer to the time the build was added
Previously, for scheduled builds, "timestamp" contained the time the
build was added to the queue, while for finished builds, it was the
time the build finished.  Now it's always the former.
2013-05-23 10:45:49 -04:00
Eelco Dolstra e09622eb29 Include # of failed builds in Subject 2013-05-21 18:30:05 -04:00
Shea Levy 4b1a8384db reproduce.tt: Set revCount for hg repos
The revision counting changes depending on which revision is cloned
initially, so clone the default branch first and then checkout the
required revision to match hydra's revCount.

Signed-off-by: Shea Levy <shea@shealevy.com>
2013-05-16 11:24:17 -04:00
Rob Vermaas 666f398e98 Always show the links related to the build product. 2013-05-16 11:29:54 +00:00
Eelco Dolstra 444b03bad2 Omit the system type if it's part of the job name 2013-05-15 11:43:06 +02:00
Eelco Dolstra ac6a152ec5 Improve email notifications status messages
Add "Failed with output", and shorten "Failed with non-zero exit code"
to "Failed".
2013-05-15 11:31:37 +02:00
Rob Vermaas 43785dfca9 Merge pull request #85 from peti/dont-clutter-system-log-with-debug-messages
hydra-queue-runner: don't clutter the system log with debug message
2013-05-10 14:52:13 -07:00
Eelco Dolstra 3939974df8 Set build status to 1 if the primary build failed 2013-05-10 00:51:45 +02:00
Eelco Dolstra 102359bf44 Add separate build step status codes for cached failures and timeouts 2013-05-09 22:13:01 +00:00
Eelco Dolstra a6d8566faf If a build aborts, mark any remaining active build steps as aborted
See e.g. http://hydra.nixos.org/build/4915744.

P.S. existing active build steps of finished builds can be marked as
aborted by running:

update buildsteps set busy = 0, status = 4
  where (build, stepnr) in
    (select s.build, s.stepnr from buildsteps s join builds b on s.build = b.id where b.finished = 1 and s.busy = 1);
2013-05-09 18:03:34 +02:00
Eelco Dolstra 7b78596412 Install plugins 2013-05-09 17:02:16 +02:00
Eelco Dolstra 184d23dc41 Remove debug line 2013-05-09 15:10:30 +02:00
Eelco Dolstra 3ca7b84297 Don't float the search bar to the right in collapsed mode 2013-05-09 14:36:51 +02:00
Eelco Dolstra 26b3da5d13 Fix tests
http://hydra.nixos.org/build/4902041
2013-05-09 14:29:04 +02:00
Eelco Dolstra de89c566f8 Use Template::Toolkit to generate email messages
Also, make the messages much shorter by not including build details.
2013-05-09 14:23:15 +02:00
Eelco Dolstra b378d94087 If multiple dependent builds fail, send a single email 2013-05-09 11:39:58 +02:00
Eelco Dolstra 038db3abeb Pass failing dependent builds to buildFinished 2013-05-08 18:54:01 +02:00
Eelco Dolstra 44b8d6f449 Use OO-style plugins
This is mostly so we don't have to pass around common parameters like
"db" and "config", and we don't have to check for the existence of
methods.

A plugin now looks like this:

  package Hydra::Plugin::TwitterNotification;

  use parent 'Hydra::Plugin';

  sub buildFinished {
      my ($self, $build, $dependents) = @_;
      print STDERR "tweeting about build ", $build->id, "\n";
      # Send tweet...
      # Hydra database is $self->{db}.
  }
2013-05-08 18:35:24 +02:00
Eelco Dolstra a420a33f66 Use "can" to check whether a plugin supports buildFinished 2013-05-08 17:57:48 +02:00
Eelco Dolstra 1d8bb0764b Add a plugin mechanism
You can now add plugins to Hydra by writing a module called
Hydra::Plugin::<whatever> and putting it in Perl's search path.  The
only plugin operation currently supported in buildFinished, called
when hydra-build has finished doing a build.

For instance, a Twitter notification plugin would look like this:

  package Hydra::Plugin::TwitterNotification;

  sub buildFinished {
      my ($self, $db, $config, $build, $dependents) = @_;
      print STDERR "tweeting about build ", $build->id, "\n";
      # send tweet...
  }

  1;
2013-05-08 17:30:30 +02:00
Eelco Dolstra f447c7d9db Add separation between news items 2013-05-07 17:04:42 +02:00
Eelco Dolstra a84db1c7cc Use extra-binary-caches 2013-05-07 15:57:20 +02:00
Eelco Dolstra 9947415ef0 Remove Twitter notification support
Turns out Twitter is not an ideal medium for sending build
notifications :-)
2013-05-03 18:33:11 +02:00
Eelco Dolstra f762d111f1 If a build step fail, immediately fail all queued builds that depend on it
This prevents unnecessary work, but it's mostly a refactoring to
support combining notification emails.
2013-05-03 18:30:13 +02:00
Eelco Dolstra 507e5bb190 Drop unused "disabled" columns 2013-05-03 16:39:17 +02:00
Eelco Dolstra 906b129f6a Fix findBuildDependencyInQueue
Previously this function didn't actually have a lot of effect.  If a
build A had a dependency B, Hydra would start B first.  But on the
next scan through the queue, it would start A anyway, because of the
"busy => 0" restriction.

Now the queue runner won't start a build if a dependency is already
running.  (This is not necessarily optimal, since the build may have
other dependencies that don't correspond to a build in the queue but
could run.  One day we'll start all Hydra builds in parallel...)

Also, for performance, use computeFSClosure instead of "nix-store
-qR".  And don't bother with topological sorting because it didn't
have an effect anyway since the database returns dependencies in
arbitrary order.
2013-05-03 16:27:26 +02:00
Eelco Dolstra 9e72c64eff Restore the links to the contents pages
These were accidentally lost in
80e0e7c84a.
2013-05-03 14:13:14 +02:00
Eelco Dolstra e96916533f Allow a per-jobset check interval
This allows checking a jobset (say) at most once a day.  It's also
possible to disable polling by setting the interval to 0.  This is
useful for jobsets that use push notification or are manually
evaluated.
2013-05-02 17:55:47 +02:00
Eelco Dolstra edaeb892e7 reproduce.tt: Use color 2013-05-02 17:55:47 +02:00
Eelco Dolstra a468a28840 reproduce.tt: Check for the presence of commands like nix-build 2013-05-02 17:55:47 +02:00