Commit graph

77 commits

Author SHA1 Message Date
Eelco Dolstra 47f877c5bb Evaluator cleanups
* Don't use isCurrent anymore; instead look up builds in the previous
  jobset evaluation.  (The isCurrent field is still maintained because
  it's still used in some other places.)

* To determine whether to perform an evaluation, compare the hash of
  the current inputs with the inputs of the previous jobset
  evaluation, rather than checking if there was ever an evaluation
  with those inputs.  This way, if the inputs of an evaluation change
  back to a previous state, we get a new jobset evaluation in the
  database (and thus the latest jobset evaluation correctly represents
  the latest state of the jobset).

* Improve performance by removing some unnecessary operations and
  adding an index.
2012-03-12 20:47:29 +01:00
Eelco Dolstra 29d5a02b94 Speed up hydra-update-gc-roots
The hydra-update-gc-roots script is taking around 95 minutes on our
Hydra instance (though a lot of that is I/O wait).  This patch
significantly reduces the number of database queries.  In particular,
the N most recent successful builds for each job in a jobset are now
determined in a single query.  Also, it removes the calls to
readlink().
2012-03-12 20:47:29 +01:00
Eelco Dolstra 5be004c999 Drop the merged tables after conversion 2012-03-12 20:47:29 +01:00
Eelco Dolstra 8d65ab6158 Don't use a prepared statement for the active build steps query
Prepared statements are sometimes much slower than unprepared
statements, because the planner doesn't have access to the query
parameters.  This is the case for the active build steps query (in
/status), where a prepared statement is three orders of magnitude
slower.  So disable the use of prepared statements in this case.

(Since the query parameters are constant here, it would be nicer if we
could tell DBIx::Class to prepare a statement with those parameters
fixed.  But I don't know an easy way to do so.)
2012-03-12 20:47:29 +01:00
Eelco Dolstra 68a867da67 Merge the BuildResultInfo table into the Builds table 2012-03-12 20:47:29 +01:00
Eelco Dolstra 25334715f8 Merge the BuildSchedulingInfo table into the Builds table
This simplifies the code and improves performance since it reduces
the number of joins.
2012-03-12 20:47:29 +01:00
Eelco Dolstra 541238030d Provide a command ‘hydra-init’ to initialise/upgrade the database
For schema upgrades, hydra-init executes the files
src/sql/upgrade-<N>.sql, each of which upgrades the schema from
version N-1 to N.  The upgrades are wrapped in a transaction.
2012-02-28 20:16:16 +01:00
Rob Vermaas 203a9b9ad7 make filesize field for buildproducts a bigint to support big files 2011-12-10 14:48:33 -05:00
Eelco Dolstra 9032c55aa6 Keep track of the database schema version
The singleton table SchemaVersion contains the current version
of the Hydra database schema.  This can be used to upgrade the
schema on the fly.

Also reran the DBIx::Class schema loader.
2011-12-05 14:29:29 +01:00
Eelco Dolstra be1935f66a Rename hydra_eval_jobs to hydra-eval-jobs 2011-11-30 18:14:48 +01:00
Eelco Dolstra 88b4c4f55d Add another index to speed up the jobstatus page 2011-11-29 18:48:11 +01:00
Eelco Dolstra 8d0ea54fa9 Speed up the /jobset overview pages a lot 2011-11-29 18:09:04 +01:00
Eelco Dolstra 0f639021d0 Add an index to speed up the /project/<name> pages 2011-11-29 17:07:37 +01:00
Eelco Dolstra 6c4ad48be7 Gratuitous whitespace. 2011-11-18 20:35:27 +01:00
Rob Vermaas 2bbbfd7a61 remove fk constraint jobsets.nixExprInput -> JobsetInputs.name, would only work on postgresql in postpone-fk-checking mode (at end of transaction in stead of through the transaction) which is not the default 2011-03-07 16:07:16 +00:00
Rob Vermaas c645fa55ff initial bzr support 2011-02-08 13:11:08 +00:00
Rob Vermaas 11ad35443b hydra: make size columns bigger (integer -> bigint) 2010-12-08 10:14:57 +00:00
Eelco Dolstra b4ce69ed8d * Index the Builds table on drvPath to prevent a full table scan
in findBuildDependencyInQueue in hydra_queue_runner.
2010-11-19 15:49:55 +00:00
Rob Vermaas dcdbb1d814 hydra: store logfile/output path/closure size 2010-11-11 11:03:50 +00:00
Rob Vermaas a391c78aaf hydra: buildmachine disabled by default, add some default constraints 2010-10-13 12:51:29 +00:00
Rob Vermaas 398993f688 hydra: add some admin for adding/enabling/etc build machines 2010-10-13 12:32:57 +00:00
Rob Vermaas c37b90ee3e hydra: use autoconf/-make 2010-09-30 14:29:15 +00:00
Eelco Dolstra f6715fa0ef * Added a status page that shows all the currently executing build steps.
* Store the system type in the BuildSteps table.
* Don't query the queue size when serving static pages.  This prevents
  two unnecessary database queries per request.
2010-08-31 15:27:46 +00:00
Eelco Dolstra 2a69745a88 * Store the name of the machine that performed a build step in the
BuildSteps table.
2010-08-31 14:08:59 +00:00
Rob Vermaas 9d9bf8b264 hydra: make nr of build to keep configurable per jobset 2010-08-10 06:48:45 +00:00
Eelco Dolstra 489e4f7697 * Create some more indices on tables that have a foreign key reference
to the Builds table.  Otherwise deleting builds (e.g. when flushing
  the queue) is very slow.
2010-08-05 14:06:02 +00:00
Rob Vermaas 0802559b03 hydra: do not perform git clone every time. in stead work on local clone and pull 2010-07-28 12:48:29 +00:00
Rob Vermaas d6131fba50 initial support for mercurial 2010-07-27 11:14:24 +00:00
Rob Vermaas 5f7ebaad37 added hide feature for project/jobset 2010-06-04 14:43:28 +00:00
Rob Vermaas bb7f82840b Hydra: Add support for maxSilent meta attribute (also already added timeout, but not implemented the actual timeout for the build yet) 2010-05-26 08:03:59 +00:00
Rob Vermaas 001113f7f9 Hydra/28: Rename "scheduler" to "evaluator" 2010-05-11 11:10:03 +00:00
Rob Vermaas 7a79d17a36 added newsitems, added some admin options to clear various caches. 2010-04-27 13:29:08 +00:00
Eelco Dolstra ba0c531f63 * Remove an empty file. 2010-03-10 15:20:06 +00:00
Rob Vermaas 2c132a2748 * hydra: project members can do stuff that the owner can do 2010-03-10 10:02:25 +00:00
Eelco Dolstra 7daca03e78 * Store jobset evaluations in the database explicitly. This includes
recording the builds that are part of a jobset evaluation.  We need
  this to be able to answer queries such as "return the latest NixOS
  ISO for which the installation test succeeded".  This wasn't previously
  possible because the database didn't record which builds of (say)
  the `isoMinimal' job and the `tests.installer.simple' job came from
  the same evaluation of the nixos:trunk jobset.

  Keeping a record of evaluations is also useful for logging purposes.
2010-03-05 15:41:10 +00:00
Eelco Dolstra 79a1489c95 * Remove the triggers to simulate foreign key constraints on SQLite,
as SQLite now supports them natively (though we still need to turn
  them on in Hydra).
2010-03-05 10:33:36 +00:00
Rob Vermaas 7208f79828 inital version of links to diff in scm 2010-02-25 15:32:56 +00:00
Rob Vermaas 87ea0cb2b9 fix wrong comment 2010-02-25 09:22:00 +00:00
Eelco Dolstra d8cc0bbb5d * Make the "latest succeeded" query (used by the "latest" channel)
faster, from about 4.5s to 1.0s for the global "latest" channel.
  Note that the query is only fast if the "IndexBuildsOnJob" and
  "IndexBuildsOnJobAndIsCurrent" indices are dropped - if they exist,
  PostgreSQL will use those instead of the more efficient
  "IndexBuildsOnJobFinishedId" index.  Looks like a bug in the planner
  to me...
2010-02-12 14:49:32 +00:00
Rob Vermaas 68c60b4c66 * hydra: added index, actual build time (buildstep with same outpath, so without deps) of the build 2010-02-11 12:23:46 +00:00
Eelco Dolstra 14a1e6db61 * More indices. 2010-02-11 10:42:37 +00:00
Eelco Dolstra 59e4f65298 * Speed up the jobset index page. Especially the query to get the
inactive jobs was quite slow.
* "IndexBy" -> "IndexOn".
2010-02-09 13:47:20 +00:00
Eelco Dolstra 9f9e18e3a6 2010-01-26 20:27:33 +00:00
Eelco Dolstra 268364bfd8 2010-01-26 20:19:25 +00:00
Eelco Dolstra 0c5f8dfb47 * Another essential index. 2010-01-26 20:03:20 +00:00
Eelco Dolstra f468caf192 2010-01-19 17:59:13 +00:00
Rob Vermaas 8a01999220 hydra
* remove trailing spaces from email notification
 * option to disable email notification for jobset
2010-01-06 13:07:59 +00:00
Rob Vermaas 044edfb764 * email notification of evaluation errors to project owner (if desired) 2009-12-18 12:07:45 +00:00
Rob Vermaas 06dc6d8f86 * for git inputs, check latest revision of branch (defaults to master for now), if there is change, only use input if last checkout was > hour ago. 2009-11-19 08:15:49 +00:00
Rob Vermaas 40920935fc remove revision from cached cvs inputs, and added missing files 2009-11-17 15:24:51 +00:00