Restarted builds whose derivation has been garbage-collected in the
meantime caused hydra-queue-runner to get stuck in a loop saying:
Jun 14 11:54:25 lucifer hydra-queue-runner[31844]: system type `x86_64-darwin': 0 active, 2 allowed, started 2 builds
Jun 14 11:54:25 lucifer hydra-queue-runner[31844]: {UNKNOWN}: path `/nix/store/wcizsch2garjlvs4pswrar47i1hwjaia-inconsolata.drv' is not valid at
/nix/store/ypkdm4v13yrk941rvp8h0y425a5ww6nm-hydra-0.1pre1353-40debf1/bin/.hydra-queue-runner-wrapped line 51. at
/nix/store/kjpsc2zdaxnd44azxyw60f2px839m1cd-hydra-perl-deps/lib/perl5/site_perl/5.16.2/Catalyst/Model/DBIC/Schema.pm line 501
This happens if the previous iteration took more than 60 seconds.
Then the queue runner may think that builds failed to start properly
and unlock them, e.g.
build 5264936 pid 19248 died, unlocking
build 5264951 pid 19248 died, unlocking
build 5257073 pid 19248 died, unlocking
...
Because we don't start a build if a dependency is already building,
it's possible that some or all of the $extraAllowed highest-priority
builds in the queue are not eligible. E.g. with $extraAllowed = 32,
we might start only 3 builds even though there are thousands in the
queue. The fix is to try all queued builds until $extraAllowed have
been started.
Issue #99.
For some reason, hg clone from a local (path-based) repo will fail if
the parent directory of the destination directory doesn't exist (though
it succeeds when cloning from an http repo).
Signed-off-by: Shea Levy <shea@shealevy.com>
Previously, for scheduled builds, "timestamp" contained the time the
build was added to the queue, while for finished builds, it was the
time the build finished. Now it's always the former.
The revision counting changes depending on which revision is cloned
initially, so clone the default branch first and then checkout the
required revision to match hydra's revCount.
Signed-off-by: Shea Levy <shea@shealevy.com>
See e.g. http://hydra.nixos.org/build/4915744.
P.S. existing active build steps of finished builds can be marked as
aborted by running:
update buildsteps set busy = 0, status = 4
where (build, stepnr) in
(select s.build, s.stepnr from buildsteps s join builds b on s.build = b.id where b.finished = 1 and s.busy = 1);
This is mostly so we don't have to pass around common parameters like
"db" and "config", and we don't have to check for the existence of
methods.
A plugin now looks like this:
package Hydra::Plugin::TwitterNotification;
use parent 'Hydra::Plugin';
sub buildFinished {
my ($self, $build, $dependents) = @_;
print STDERR "tweeting about build ", $build->id, "\n";
# Send tweet...
# Hydra database is $self->{db}.
}
You can now add plugins to Hydra by writing a module called
Hydra::Plugin::<whatever> and putting it in Perl's search path. The
only plugin operation currently supported in buildFinished, called
when hydra-build has finished doing a build.
For instance, a Twitter notification plugin would look like this:
package Hydra::Plugin::TwitterNotification;
sub buildFinished {
my ($self, $db, $config, $build, $dependents) = @_;
print STDERR "tweeting about build ", $build->id, "\n";
# send tweet...
}
1;
Previously this function didn't actually have a lot of effect. If a
build A had a dependency B, Hydra would start B first. But on the
next scan through the queue, it would start A anyway, because of the
"busy => 0" restriction.
Now the queue runner won't start a build if a dependency is already
running. (This is not necessarily optimal, since the build may have
other dependencies that don't correspond to a build in the queue but
could run. One day we'll start all Hydra builds in parallel...)
Also, for performance, use computeFSClosure instead of "nix-store
-qR". And don't bother with topological sorting because it didn't
have an effect anyway since the database returns dependencies in
arbitrary order.
This allows checking a jobset (say) at most once a day. It's also
possible to disable polling by setting the interval to 0. This is
useful for jobsets that use push notification or are manually
evaluated.
This caused exceptions like:
Caught exception in Hydra::Controller::Build->view_build "writing to file: Broken pipe at /nix/store/ihdb3widsq1dk7sbl5vqjxfcxb5ypad4-hydra-0.1pre1297-8158093/libexec/hydra/lib/Hydra/Controller/Build.pm line 59."
because the connection to the Nix daemon would be terminated due to a
protocol violation (calling queryPathInfo with an empty string).
Returning only the first 20 results can cause NixOS/Nixpkgs channel
generation to fail, if the first 20 view results correspond to
evaluations that haven't finished yet. Then URLs like
/view/nixos/tested/latest-finished will return 500 rather than the
latest finished view.
Avoid the frequently printed
hydra-queue-runner[10293]: system type `x86_64-linux': 2 active, 2 allowed, starting 0 builds
message. That information is only interesting when some build are
actually started.
You can now do:
bash <(curl http://hydra-server/build/1238757/reproduce)
to download and execute a script that reproduces a Hydra build
locally. This script fetches all inputs (e.g. Git repositories) and
then invokes nix-build.
The downloaded sources are stored in /tmp/build-<buildid> and reused
between invocations of the script.
Any additional command line options are passed to nix-build. So
bash <(curl http://hydra-server/build/1238757/reproduce) --run-env
will drop you in a shell where you can interactively hack on the
build, e.g.
$ source $stdenv/setup
$ set +e
$ unpackPhase
$ cd $sourceRoot
$ configurePhase
$ emacs foo.c &
$ make
and so on.
Build product paths cannot reference locations outside of the Nix
store. We previously disallowed paths from being symlinks, but this
didn't take into account that parent path elements can be symlinks as
well. So a build product /nix/store/bla.../foo/passwd, with
/nix/store/bla.../foo being a symlink to /etc, would still work.
So now we check all paths encountered during path resolution.
Symlinks are allowed again so long as they point to the Nix store.
This is a followup to commit
10882a1ffd ("Add multiple output
support").
* src/script/hydra-eval-guile-jobs.in (job-evaluations->sxml): Return
several `output' tags in the body, and remove the `outPath' attribute
of `job'.
Chaining paths only works properly when PathPart is used. Before this
fix, the affected URIs bypassed the top-level 'admin' sub.
Signed-off-by: Shea Levy <shea@shealevy.com>
