The check to see whether a build had been scheduled in a previous
evaluation took about 200 ms for the nixpkgs:trunk jobset. Given
that it has more than 15000 builds, this added up to a lot. Now
it takes 0.2 ms per build.
The action .../jobset/<project>/<jobset>/latest-eval redirects to the
latest evaluation of the jobset that has no unfinished builds. Thus,
for instance,
http://hydra.nixos.org/jobset/nixpkgs/trunk/latest-eval/channel
is the channel containing the latest consistent set of Nixpkgs builds.
The URI parameter "compare=..." can denote either an arbitrary
evaluation ID, or the name of a jobset in the same project. In the
latter case, the comparison is made against the latest completed
evaluation of the specified jobset.
This happened in a pathological case in Nixpkgs: the "grub" job is
evaluated for i686-linux and x86_64-linux, but in the latter case it
returns the same derivation as in the former case. So only one build
should be added.
This gets rid of the openHydraDB function and ensures that we
open the database in a consistent way.
Also drop the PostgreSQL sequence hacks. They don't seem to be
necessary anymore.
When checking whether the jobset is unchanged, we need to compare with
the previous JobsetEval regardless of whether it had new builds.
Otherwise we'll keep adding new JobsetEval rows.
Because of the way DBIx::Class does prepared statements, even
innocuous queries such
$c->model('DB::Builds)->search({finished => 0})
can be extremely slow. This is because DBIx::Class prepares a
PostgreSQL statement
select ... from Builds where finished = ?
and since Builds is very large and there is a large fraction of rows
with "finished = 1", the PostgreSQL query planner decides to implement
this query with a sequential scan of the Builds table (despite the
existence of an index on "finished"), which is extremely slow. It
would be nice if we could tell DBIx::Class that constants should be
part of the prepared statement, i.e.
select ... from Builds where finished = 0
but AFAIK we can't.
In particular the /pkg action is now O(lg n) instead of O(n) in the
number of packages in the channel, and listing the channel contents
no longer requires calling isValidPath() on all packages.
Derivations (and thus build time dependencies) are no longer included
in the channel, because they're not GC roots. Thus they could
disappear unexpectedly.
This isn't perfect because it doesn't handle the case where a
previous build hasn't finished yet. But at least it won't send mail
for old builds that fail while a newer build has already succeeded.
* Don't use isCurrent anymore; instead look up builds in the previous
jobset evaluation. (The isCurrent field is still maintained because
it's still used in some other places.)
* To determine whether to perform an evaluation, compare the hash of
the current inputs with the inputs of the previous jobset
evaluation, rather than checking if there was ever an evaluation
with those inputs. This way, if the inputs of an evaluation change
back to a previous state, we get a new jobset evaluation in the
database (and thus the latest jobset evaluation correctly represents
the latest state of the jobset).
* Improve performance by removing some unnecessary operations and
adding an index.
Since it read the actual roots after determining the set of desired
roots, there was a possibility that it would delete roots added by
hydra-evaluator or hydra-build while hydra-update-gc-roots was
running. This could cause a derivation to be garbage-collected before
the build was performed, for instance. Now the actual roots are read
first, so any root added after that time is not deleted.
The hydra-update-gc-roots script is taking around 95 minutes on our
Hydra instance (though a lot of that is I/O wait). This patch
significantly reduces the number of database queries. In particular,
the N most recent successful builds for each job in a jobset are now
determined in a single query. Also, it removes the calls to
readlink().
Prepared statements are sometimes much slower than unprepared
statements, because the planner doesn't have access to the query
parameters. This is the case for the active build steps query (in
/status), where a prepared statement is three orders of magnitude
slower. So disable the use of prepared statements in this case.
(Since the query parameters are constant here, it would be nicer if we
could tell DBIx::Class to prepare a statement with those parameters
fixed. But I don't know an easy way to do so.)
For schema upgrades, hydra-init executes the files
src/sql/upgrade-<N>.sql, each of which upgrades the schema from
version N-1 to N. The upgrades are wrapped in a transaction.
Since a build may be a member of multiple jobset evaluations, we need
to do a "select distinct" here. But maybe we should only show builds
from a single evaluation (e.g. the most recent), since showing builds
from several may be confusing.
The abbreviated Git revision hash (e.g. "267480b") is typically
contained in ‘gitTag’ as well, but the latter can contain other
elements as well, e.g., the delta to the closest tag. That may
be undesirable in version strings, so this is an alternative.
The ‘revCount’ attribute is the number of commits in the history
of the revision. This is useful if you need a monotonically
increasing version number.
The ‘gitTag’ attribute is the output of ‘git describe’, e.g.
‘v1.0.4-14-g2414721’ to indicate that the current revision is 14
commits after the tag ‘v1.0.4’.
The singleton table SchemaVersion contains the current version
of the Hydra database schema. This can be used to upgrade the
schema on the fly.
Also reran the DBIx::Class schema loader.
In hydra-evaluator, reuse an SVN working copy between runs (similar to
what we do with Git and other input types). This reduces network
traffic in the common case.
Also, don't use nix-prefetch-svn. It doesn't do anything useful.
The underscores are ugly and the .pl extension is an implementation
detail that shouldn't be visible to the outside.
Also, get rid of the *.in files. It's not really necessary to
generate them. And I was always modifying the wrong file.
The "all" channel fundamentally doesn't scale, because it needs
to fetch N builds from the database (where N is potentially a very
large number), then check whether they are still valid. And it's
not very useful anyway.
