Include information about who changed the build status in notification
emails, and enable optional per-input notification of said committers.
Conflicts due to two branches modifying the database schema.
Signed-off-by: Shea Levy <shea@shealevy.com>
Conflicts:
src/lib/Hydra/Schema/Jobsets.pm
src/sql/upgrade-23.sql
Currently the dashboard allows users to get a quick overview of the
status of jobs they're interested in, but more will be added,
e.g. viewing all your jobsets or all jobs of which you're a
maintainer.
Each jobset now has a "scheduling share" that determines how much of
the build farm's time it is entitled to. For instance, if a jobset
has 100 shares and the total number of shares of all jobsets is 1000,
it's entitled to 10% of the build farm's time. When there is a free
build slot for a given system type, the queue runner will select the
jobset that is furthest below its scheduling share over a certain time
window (currently, the last day). Withing that jobset, it will pick
the build with the highest priority.
So meta.schedulingPriority now only determines the order of builds
within a jobset, not between jobsets. This makes it much easier to
prioritise one jobset over another (e.g. nixpkgs:trunk over
nixpkgs:stdenv).
For presentation purposes, we need to know what builds are part of an
aggregate build. So at evaluation time, look at the "members"
attribute, find the corresponding builds in the eval, and create a
mapping in the AggregateMembers table.
The NrBuilds table tracks the value of ‘select count(*) from Builds
where finished = 0’, keeping it up to date via a trigger. This is
necessary to make the /all page fast, since otherwise it needs to do a
sequential scan on the Builds table.
The catalyst-action-rest branch from shlevy/hydra was an exploration of
using Catalyst::Action::REST to create a JSON API for hydra. This commit
merges in the best bits from that experiment, with the goal that further
API endpoints can be added incrementally.
In addition to migrating more endpoints, there is potential for
improvement in what's already been done:
* The web interface can be updated to use the same non-GET endpoints as
the JSON interface (using x-tunneled-method) instead of having a
separate endpoint
* The web rendering should use the $c->stash->{resource} data structure
where applicable rather than putting the same data in two places in
the stash
* Which columns to render for each endpoint is a completely debatable
question
* Hydra::Component::ToJSON should turn has_many relations that have
strings as their primary keys into objects instead of arrays
FixesNixOS/hydra#98
Signed-off-by: Shea Levy <shea@shealevy.com>
This allows checking a jobset (say) at most once a day. It's also
possible to disable polling by setting the interval to 0. This is
useful for jobsets that use push notification or are manually
evaluated.
This reverts commit 71d020735b.
Unfortunately there are still some cases where we need to set Hydra's
concurrency separately. (Ideally, Hydra would start *all* queued
builds in parallel and let Nix take care of everything...)
External machines can now notify Hydra that it should check a
repository by sending a GET or PUSH request to /api/push, providing a
list of jobsets to be checked and/or a list of repository URLs. In
the latter case, all jobsets that have any of the specified
repositories as an input will be checked.
For instance, you can configure GitHub or BitBucket to send a request
to the URL
http://hydra.example.org/api/push?repos=git://github.com/NixOS/nixpkgs.git
to trigger evaluation of all jobsets that have
git://github.com/NixOS/nixpkgs.git as an input, or to the URL
http://hydra.example.org/api/push?jobsets=patchelf:trunk,nixpkgs:trunk
to trigger evaluation of just the specified jobsets.
It's pointless to store these, since Nix knows where the logs are.
Also handle (in fact require) Nix's new log storage scheme. Also some
cleanups in the build page.
This gets rid of the openHydraDB function and ensures that we
open the database in a consistent way.
Also drop the PostgreSQL sequence hacks. They don't seem to be
necessary anymore.
* Don't use isCurrent anymore; instead look up builds in the previous
jobset evaluation. (The isCurrent field is still maintained because
it's still used in some other places.)
* To determine whether to perform an evaluation, compare the hash of
the current inputs with the inputs of the previous jobset
evaluation, rather than checking if there was ever an evaluation
with those inputs. This way, if the inputs of an evaluation change
back to a previous state, we get a new jobset evaluation in the
database (and thus the latest jobset evaluation correctly represents
the latest state of the jobset).
* Improve performance by removing some unnecessary operations and
adding an index.
The singleton table SchemaVersion contains the current version
of the Hydra database schema. This can be used to upgrade the
schema on the fly.
Also reran the DBIx::Class schema loader.
* Store the system type in the BuildSteps table.
* Don't query the queue size when serving static pages. This prevents
two unnecessary database queries per request.
recording the builds that are part of a jobset evaluation. We need
this to be able to answer queries such as "return the latest NixOS
ISO for which the installation test succeeded". This wasn't previously
possible because the database didn't record which builds of (say)
the `isoMinimal' job and the `tests.installer.simple' job came from
the same evaluation of the nixos:trunk jobset.
Keeping a record of evaluations is also useful for logging purposes.
faster, from about 4.5s to 1.0s for the global "latest" channel.
Note that the query is only fast if the "IndexBuildsOnJob" and
"IndexBuildsOnJobAndIsCurrent" indices are dropped - if they exist,
PostgreSQL will use those instead of the more efficient
"IndexBuildsOnJobFinishedId" index. Looks like a bug in the planner
to me...
releases as a dynamic view on the database was misguided, since
doing thing like adding a new job to a release set will invalidate
all old releases. So we rename release sets to views, and we'll
reintroduce releases as separate, static entities in the database.
the derivations that the jobset currently contains. This is
necessary to allow the "latest" channel to contain the correct
builds when the sources of a jobset are reverted.
highest ID rather than the highest timestamp. Otherwise, if a build
from revision N finishes after a build from revision N + 1, then
the build from revision N will end up in the channel. Thus, the
channel contents will be out of sync.
This is still not quite correct: if a revision *reverts* to an older
build, the channel will still end up out of sync, because Hydra
won't schedule the build again (after all, it has already done it).
A better fix would be to add a separate timestamp denoting when the
build was last "current" (i.e. corresponding to the "head revision"
of its job).
distinguish between jobs with the same name in different jobsets
(e.g. "trunk" vs "stdenv-branch" for Nixpkgs).
* Renamed the "attrName" field of Builds to "job".
* Renamed the "id" field of BuildSteps to "build".
Note: to upgrade old databases, do a dump with an old Sqlite first;
dumping with a new Sqlite will silently discard (!) the contents of
the ReleaseSetJobs table.
failed in a previous build. This is essential for Nixpkgs: we don't
want to keep doing the same failed dependency (say, Glibc) over and
over again for a few hundred jobs.
