ma27/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
1515 commits 4 branches 0 tags 26 MiB
Commit graph

241 commits

Author SHA1 Message Date
Eelco Dolstra 47f877c5bb Evaluator cleanups 
* Don't use isCurrent anymore; instead look up builds in the previous
  jobset evaluation.  (The isCurrent field is still maintained because
  it's still used in some other places.)

* To determine whether to perform an evaluation, compare the hash of
  the current inputs with the inputs of the previous jobset
  evaluation, rather than checking if there was ever an evaluation
  with those inputs.  This way, if the inputs of an evaluation change
  back to a previous state, we get a new jobset evaluation in the
  database (and thus the latest jobset evaluation correctly represents
  the latest state of the jobset).

* Improve performance by removing some unnecessary operations and
  adding an index.
 2012-03-12 20:47:29 +01:00
Eelco Dolstra bc82a82593 Fix a race condition in hydra-update-gc-roots 
Since it read the actual roots after determining the set of desired
roots, there was a possibility that it would delete roots added by
hydra-evaluator or hydra-build while hydra-update-gc-roots was
running.  This could cause a derivation to be garbage-collected before
the build was performed, for instance.  Now the actual roots are read
first, so any root added after that time is not deleted.
 2012-03-12 20:47:29 +01:00
Eelco Dolstra 29d5a02b94 Speed up hydra-update-gc-roots 
The hydra-update-gc-roots script is taking around 95 minutes on our
Hydra instance (though a lot of that is I/O wait).  This patch
significantly reduces the number of database queries.  In particular,
the N most recent successful builds for each job in a jobset are now
determined in a single query.  Also, it removes the calls to
readlink().
 2012-03-12 20:47:29 +01:00
Eelco Dolstra 8d65ab6158 Don't use a prepared statement for the active build steps query 
Prepared statements are sometimes much slower than unprepared
statements, because the planner doesn't have access to the query
parameters.  This is the case for the active build steps query (in
/status), where a prepared statement is three orders of magnitude
slower.  So disable the use of prepared statements in this case.

(Since the query parameters are constant here, it would be nicer if we
could tell DBIx::Class to prepare a statement with those parameters
fixed.  But I don't know an easy way to do so.)
 2012-03-12 20:47:29 +01:00
Eelco Dolstra 68a867da67 Merge the BuildResultInfo table into the Builds table 2012-03-12 20:47:29 +01:00
Eelco Dolstra 25334715f8 Merge the BuildSchedulingInfo table into the Builds table 
This simplifies the code and improves performance since it reduces
the number of joins.
 2012-03-12 20:47:29 +01:00
Eelco Dolstra 19fe4b9b4a Add hydra-init to the tarball 2012-03-05 15:35:13 +01:00
Eelco Dolstra 541238030d Provide a command ‘hydra-init’ to initialise/upgrade the database 
For schema upgrades, hydra-init executes the files
src/sql/upgrade-<N>.sql, each of which upgrades the schema from
version N-1 to N.  The upgrades are wrapped in a transaction.
 2012-02-28 20:16:16 +01:00
Eelco Dolstra da26294fdb Don't install hydra-create 
It doesn't do anything useful for users.
 2012-02-28 15:33:28 +01:00
Ludovic Courtès 1f448673f0 hydra-build: Do not send email following an abortion. 2012-02-14 11:48:44 +01:00
Eelco Dolstra a3cc2ce796 Remove redundant dot in status emails 
E.g.:

  This is to let you know that Hydra build 1998534 of job patchelf:trunk:deb_ubuntu1110x86_64 is 'Success'..
 2012-02-07 14:51:47 +01:00
Ludovic Courtès 5644c1c7d9 hydra-build: Add system info to the subject and extra headers. 2012-01-31 18:27:33 +01:00
Eelco Dolstra b1da85140d Fix sysbuild input type handling 2011-12-05 17:13:20 +01:00
Eelco Dolstra 3b9a62c964 Pass inputs to release expressions using -I 
This means that you can now write (for instance)

  with <nixpkgs> { ... };

in release.nix, rather than

  { nixpkgs } :

  with nixpkgs { ... };
 2011-12-05 13:08:43 +01:00
Eelco Dolstra 29846d7f2f Do incremental SVN checkouts 
In hydra-evaluator, reuse an SVN working copy between runs (similar to
what we do with Git and other input types).  This reduces network
traffic in the common case.

Also, don't use nix-prefetch-svn.  It doesn't do anything useful.
 2011-12-04 22:05:43 +01:00
Ludovic Courtès 6c4e83d454 More command renaming. 2011-12-02 15:58:26 +01:00
Eelco Dolstra be1935f66a Rename hydra_eval_jobs to hydra-eval-jobs 2011-11-30 18:14:48 +01:00
Eelco Dolstra 82d17a2d0b Rename hydra_*.pl to hydra-* 
The underscores are ugly and the .pl extension is an implementation
detail that shouldn't be visible to the outside.

Also, get rid of the *.in files.  It's not really necessary to
generate them.  And I was always modifying the wrong file.
 2011-11-30 18:14:48 +01:00
Eelco Dolstra 2c328e4b20 Forgot to change Nix to Nix::Store in one place 2011-11-30 17:22:31 +01:00
Eelco Dolstra c613b885f2 Use the new Nix Perl bindings 2011-11-30 15:25:28 +01:00
Rob Vermaas 41c649a2fd remove .hg-archival.txt in nix-prefetch-hg 2011-11-03 19:44:05 -04:00
Rob Vermaas ab81ce689d add nix-prefetch- scripts for now, were externals in svn 2011-10-28 22:12:51 -04:00
Eelco Dolstra 5f93e6437f * Update to Catalyst 5.9. 2011-10-07 13:45:23 +00:00
Rob Vermaas f34ec401ba fix for buildsteps starting at 0, probably something changed in catalyst 2011-09-08 22:48:38 +00:00
Eelco Dolstra 9f78a942bc 2011-08-19 16:09:14 +00:00
Ludovic Courtès dcbdd0ad26 hydra_build.pl: Honor `$build->timeout'. 2011-07-04 13:55:02 +00:00
Rob Vermaas 5d166150f9 using backquote as argument resulted in only first line as first argument to removeAsciiEscapes 2011-06-10 10:48:51 +00:00
Rob Vermaas 48b435c2de remove ascii escapes from log in tail page and emails 2011-06-10 09:53:15 +00:00
Rob Vermaas fca3019c7b qualify ordery_by clauses when necessary, remove unnecessary order_by's, reported by Ludo, resulted in errors in sqlite 2011-04-01 07:40:06 +00:00
Rob Vermaas 2849279f80 do not send email for builds with status 'aborted' 2011-03-15 12:12:56 +00:00
Rob Vermaas 03d12ca68e Hydra/57: Unknown failure -> Failed 2011-03-15 11:51:00 +00:00
Rob Vermaas 560f41a7e5 if HYDRA_CONFIG is not set, take default 2011-03-07 15:49:48 +00:00
Rob Vermaas 586f131ebd use HYDRA_DATA/hydra.conf as default location for hydra.conf 2011-03-07 15:06:32 +00:00
Rob Vermaas d4e9f5383b Hydra/56: handle failed builds with result only at build level, not buildsteps 2011-03-07 13:55:43 +00:00
Rob Vermaas 98824f2aaf hydra: nix-prefetch-bzr missing in tarball 2011-02-09 12:44:46 +00:00
Rob Vermaas 6e213fab97 hydra: add nix-prefetch-* to tarball 2011-01-14 14:11:08 +00:00
Rob Vermaas 2892c92569 move nix-prefetch-* into tarball 2011-01-14 14:04:46 +00:00
Rob Vermaas 039d1bed21 hydra: add support for succeedOnFailure feature in stdenv 2010-12-22 14:55:05 +00:00
Rob Vermaas 0cee7c0909 hydra: handle case where build to keep set to 0 2010-11-23 09:05:09 +00:00
Eelco Dolstra 91fe70522f * Doh. Remove debug statement. 2010-11-22 12:20:04 +00:00
Eelco Dolstra a93e272364 * Add a NarSize field to Hydra manifests. This allows nix-env 
to predict how much disk space a package will require.
* Compute the output / closure size using the info stored in the
  Nix database (rather than doing a slow "du").
 2010-11-19 15:44:20 +00:00
Eelco Dolstra 738712fca0 * Merge the GC branch. 2010-11-19 11:01:31 +00:00
Rob Vermaas dcdbb1d814 hydra: store logfile/output path/closure size 2010-11-11 11:03:50 +00:00
Rob Vermaas 753f5679cb hydra: remove HYDRA_HOME env var, not used 2010-10-08 10:45:44 +00:00
Rob Vermaas 5e0e9e058d hydra: removed need for HYDRA_BUILD_BASEURL env variable 2010-10-08 10:45:42 +00:00
Rob Vermaas c37b90ee3e hydra: use autoconf/-make 2010-09-30 14:29:15 +00:00
Rob Vermaas 38d50806b9 hydra: if evaluator sees cached build, also add the buildproducts 2010-09-07 11:29:52 +00:00
Rob Vermaas c5ceece093 hydra: moved getbuildlog 2010-09-02 08:56:29 +00:00
Rob Vermaas d7487ab0d9 hydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue. 2010-09-01 10:50:57 +00:00
Rob Vermaas 2e02291092 fixed email bug 2010-09-01 08:52:54 +00:00
Eelco Dolstra 1495e04d9d * Speed up findBuildDependencyInQueue by doing only one SQL query for 
all the dependencies (`drvpath => [ @drvs ]' is an OR).
 2010-08-31 16:19:33 +00:00
Eelco Dolstra f6715fa0ef * Added a status page that shows all the currently executing build steps. 
* Store the system type in the BuildSteps table.
* Don't query the queue size when serving static pages.  This prevents
  two unnecessary database queries per request.
 2010-08-31 15:27:46 +00:00
Eelco Dolstra 2a69745a88 * Store the name of the machine that performed a build step in the 
BuildSteps table.
 2010-08-31 14:08:59 +00:00
Rob Vermaas 7bd11b87f0 hydra: only use keepnr to determine if builds should be kept 2010-08-10 08:00:28 +00:00
Rob Vermaas 9d9bf8b264 hydra: make nr of build to keep configurable per jobset 2010-08-10 06:48:45 +00:00
Rob Vermaas 34e309f0f7 hydra: if a project or jobset is hidden (and thus disabled) do not keep builds 2010-08-09 18:36:16 +00:00
Rob Vermaas 066fd3e41f hydra: fixed email notification bug, when build is performed for the first time (it always said succeeded in the body of the mail 2010-08-09 13:08:27 +00:00
Rob Vermaas 87007fb97c hydra: fix enable email notification bug 2010-08-06 08:35:06 +00:00
Rob Vermaas bb7f82840b Hydra: Add support for maxSilent meta attribute (also already added timeout, but not implemented the actual timeout for the build yet) 2010-05-26 08:03:59 +00:00
Rob Vermaas 001113f7f9 Hydra/28: Rename "scheduler" to "evaluator" 2010-05-11 11:10:03 +00:00
Rob Vermaas 4550ced942 Hydra/28: Rename "scheduler" to "evaluator" 2010-05-11 11:09:58 +00:00
Rob Vermaas e18fe1078a rename var 2010-04-27 08:01:38 +00:00
Rob Vermaas 2bfddaf4ae Hydra/23: added some X-headers with meta info in email notifications, added more descriptive status 2010-04-26 14:36:56 +00:00
Rob Vermaas 1d0598272e Hydra/17: in queue runner, prefer builds in the queue that are a dependency of another build (with higher priority) 2010-04-23 11:33:06 +00:00
Rob Vermaas 742846e476 Hydra/18: fixed uninitialized value error when logfile is null 2010-04-19 10:25:17 +00:00
Rob Vermaas 8fb3d901cb Merge branch 'queue-17' 2010-04-13 08:42:44 +00:00
Rob Vermaas c38eca41a8 deleted some old scripts 2010-03-23 20:04:01 +00:00
Eelco Dolstra 82a325a8cd * $currentBuilds{id} can be 0. 2010-03-05 17:48:00 +00:00
Eelco Dolstra 7daca03e78 * Store jobset evaluations in the database explicitly. This includes 
recording the builds that are part of a jobset evaluation.  We need
  this to be able to answer queries such as "return the latest NixOS
  ISO for which the installation test succeeded".  This wasn't previously
  possible because the database didn't record which builds of (say)
  the `isoMinimal' job and the `tests.installer.simple' job came from
  the same evaluation of the nixos:trunk jobset.

  Keeping a record of evaluations is also useful for logging purposes.
 2010-03-05 15:41:10 +00:00
Eelco Dolstra 77d4a8c027 * Keep the 3 most recent builds for every platform for every job. 
Previously it just kept the 3 most recent builds for every job
  (contrary to what the comment said).
 2010-02-15 10:21:11 +00:00
Eelco Dolstra 6dd87f5851 * Typo. 2010-02-10 12:27:16 +00:00
Rob Vermaas 3b504b2370 * hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job. 2010-01-19 14:15:31 +00:00
Rob Vermaas 4a9234bc1d probably shouldn't leave in the debug statements 2010-01-12 08:40:41 +00:00
Rob Vermaas 3763748dbb revert unmeant previous change which caused errors to stay, even after successful evaluation of jobs 2010-01-12 08:39:30 +00:00
Rob Vermaas 791a6eddd6 hack to try and prevent too many newlines 2010-01-07 13:53:05 +00:00
Rob Vermaas 1c0d34d395 removed debug print, added last 50 lines in failure emails 2010-01-06 13:57:41 +00:00
Rob Vermaas 8a01999220 hydra 
* remove trailing spaces from email notification
 * option to disable email notification for jobset
 2010-01-06 13:07:59 +00:00
Rob Vermaas 044edfb764 * email notification of evaluation errors to project owner (if desired) 2009-12-18 12:07:45 +00:00
Eelco Dolstra 7eda090e74 * Prevent repeated evaluation of a jobset with the same inputs. This 
should make the Hydra scheduler a lot less CPU-intensive, since it
  won't run hydra_eval_jobs all the time.
 2009-11-17 13:55:22 +00:00
Eelco Dolstra e25f112b82 * Implemented the clone feature. 2009-10-26 17:01:23 +00:00
Eelco Dolstra f72367407a * Some renaming. 2009-10-26 15:55:19 +00:00
Eelco Dolstra e1373fa1c3 * Refactoring: move fetchInput out of hydra_scheduler into a separate 
module, since Controller/Build.pm needs it to create a new build.
 2009-10-26 15:39:14 +00:00
Eelco Dolstra cd27c046b0 * Fix an apparent incompatibility with recent DBIx::Class. 2009-10-26 13:55:54 +00:00
Eelco Dolstra cb2493eca9 * Store the jobset's nixExprPath and nixExprInput fields in a build to 
allow it to be cloned (re-executed with modified inputs) later and
  to provide some traceability.
 2009-10-26 13:33:48 +00:00
Eelco Dolstra 69f290413c * Do not garbage collect releases. 2009-10-26 12:47:29 +00:00
Eelco Dolstra 2cf42489e2 * More renaming. 2009-10-20 12:35:01 +00:00
Eelco Dolstra 8f9417f822 * Be a bit less aggressive in rescheduling builds that have already 
been done.
 2009-10-19 12:36:15 +00:00
Rob Vermaas 3ebe5e1069 2009-10-15 13:35:52 +00:00
Rob Vermaas 1c5ab05521 * added support for twitter notification 
3 environment variables are important: 
      TWITTER_USER
      TWITTER_PASS
      HYDRA_BUILD_BASEURL
   - twitter notification is off when TWITTER_USER and TWITTER_PASS are not defined
   - if HYDRA_BUILD_BASEURL is not defined, no URL is put in the twitter messages
 2009-10-15 13:23:15 +00:00
Eelco Dolstra 71bc5b14a0 * Add --fallback to prevent problems with obsolete substitutes. 2009-10-08 11:51:49 +00:00
Eelco Dolstra 6cedee5476 * Allow jobsets to be disabled. 2009-10-08 11:39:16 +00:00
Eelco Dolstra 158a746e94 * Don't bother with the Jobs.active column anymore. 2009-10-08 11:19:39 +00:00
Eelco Dolstra f158697796 * Shut up DBIx::Class. 2009-10-08 11:19:17 +00:00
Eelco Dolstra 076ef05578 * In the scheduler, don't check if we've already done a build (except 
against the set of current builds for the job).  This ensures that
  the builds with the highest ID are what we want in the channel, even
  in case of reverts.
 2009-10-07 13:18:12 +00:00
Eelco Dolstra e9cf409d80 * Mark the "current" builds in a jobset, i.e. those corresponding to 
the derivations that the jobset currently contains.  This is
  necessary to allow the "latest" channel to contain the correct
  builds when the sources of a jobset are reverted.
 2009-10-02 16:06:28 +00:00
Eelco Dolstra cbeccda5eb * Typo. 2009-08-12 13:50:59 +00:00
Eelco Dolstra 5e72c17540 * Include more info in notification emails. 2009-08-12 13:31:27 +00:00
Eelco Dolstra c1e6797d39 * Allow overriding the sender email address. 2009-07-24 18:06:34 +00:00
Rob Vermaas f89644ddb7 time out to 3600 for now 2009-07-13 11:42:56 +00:00
Eelco Dolstra 1cc99505b7 2009-07-10 14:43:21 +00:00
Eelco Dolstra abe0a8eb9d * Get the URI for use in notification mails from the Hydra config 
file.
 2009-07-10 09:47:42 +00:00
Eelco Dolstra bbf718e64d * Revert for now due to Postgres breakage. 2009-07-09 10:53:31 +00:00
Eelco Dolstra 44454ebeac * Send email if a build fails. 2009-07-08 15:52:55 +00:00
Eelco Dolstra 3119cc7edc 2009-07-07 16:15:38 +00:00
Eelco Dolstra 24a3b39463 * Store meta.maintainers. 2009-07-07 14:33:51 +00:00
Eelco Dolstra 5bdd5e7152 * Added a maintainers field to the Builds table. 
* Regenerated the schema bindings with the latest DBIx::Class.
 2009-07-07 13:59:59 +00:00
Eelco Dolstra 000fffeb11 2009-04-25 11:27:46 +00:00
Eelco Dolstra 109cc35edf * Randomly permute the order in which builds are added. This is 
mainly to prevent all those Nixpkgs builds named "kde*" from
  building at the same time.  Since they all have the same slow
  dependencies (qt, kdelibs) this tends to block the buildfarm.
 2009-04-23 15:40:36 +00:00
Eelco Dolstra c48ec3d340 * Autoflush stdout. 2009-04-22 22:59:54 +00:00
Eelco Dolstra 97a6011628 * Hack around those SQLite timeouts: just retry the transaction. 2009-04-22 22:43:04 +00:00
Eelco Dolstra 80691a39f5 * Don't log redundant build steps in case of cached failures. 2009-04-22 14:41:12 +00:00
Eelco Dolstra db4ce0df06 2009-03-31 16:09:04 +00:00
Eelco Dolstra 7b7c3f2265 * Perl sucks. "getBuildLog $drvPath" doesn't mean the same as 
"getBuildLog($drvPath)" if you call it in a hash, and quietly screws
  up the rest of the hash.
 2009-03-26 15:32:19 +00:00
Eelco Dolstra f0f9f12e42 * Use Nix's negative caching. 2009-03-26 12:53:39 +00:00
Eelco Dolstra c75977ce5a * Record the input containing the Nix expression (release.nix) in the 
build inputs.  Otherwise we can't (for example) reproduce a build
  later.
 2009-03-23 21:42:59 +00:00
Eelco Dolstra 0da0384753 * Top-level: don't exit on errors. 2009-03-23 01:13:37 +00:00
Eelco Dolstra d9424b6364 * Show the input bindings in evaluation error messages. 2009-03-20 17:06:50 +00:00
Eelco Dolstra 2755c895ff * In job inputs of type "build", allow the project and jobset names of 
the input build to be specified, as well as constraints on the
  inputs of the inputs build.  For instance, you can require that a
  build has input `system = "i686-linux"'.

  This is important when one binary build serves as an input to
  another binary build.  Obviously, we shouldn't pass a build on
  i686-linux as an input to another on i686-darwin.  Hence the
  necessity for constraint.

  The constraint are currently quite limited.  What you really want to
  say is that the "system" input of the other build has to match the
  "system" input of this build.  But those require a bit more work
  since they introduce dependencies between inputs.
 2009-03-20 14:50:09 +00:00
Eelco Dolstra b39e2c5e32 * Doh. 2009-03-18 17:13:13 +00:00
Eelco Dolstra 702da969a6 * Ordering by timestamp isn't a good idea here since a newer revision 
might finish building before an older revision, and therefore have a
  lower timestamp.
 2009-03-18 16:46:01 +00:00
Eelco Dolstra a1848b08f2 2009-03-16 17:51:42 +00:00
Eelco Dolstra bf1480cc25 * Doh. 2009-03-16 17:46:46 +00:00
Eelco Dolstra 5853a26b13 * Don't discard old build steps when restarting a build. 2009-03-16 16:56:47 +00:00
Eelco Dolstra a6e84d8431 * Option to show the tail of a log. 2009-03-16 12:16:33 +00:00
Eelco Dolstra d2fc382498 * Register GC roots properly. 2009-03-15 11:56:11 +00:00
Eelco Dolstra 32f0665d2c * Allow users to change the value of a build's "keep" flag, which 
prevents the build output from being garbage collected.
 2009-03-14 23:56:57 +00:00
Eelco Dolstra 7c7c43335d * Job status: show the active jobs. 2009-03-13 17:32:08 +00:00
Eelco Dolstra ae364b9e5f * Represent jobs explicitly in the DB. 2009-03-13 14:49:25 +00:00
Eelco Dolstra f2f586d842 * Disambiguate jobs by jobset name. I.e. jobs with the same name in 
different jobsets are not considered the same job.
 2009-03-12 23:46:17 +00:00
Eelco Dolstra a623ad157e * Add the name of the jobset to ReleaseSetJobs, otherwise we can't 
distinguish between jobs with the same name in different jobsets
  (e.g. "trunk" vs "stdenv-branch" for Nixpkgs).
* Renamed the "attrName" field of Builds to "job".
* Renamed the "id" field of BuildSteps to "build".
 2009-03-12 14:18:30 +00:00
Eelco Dolstra 70276f59ea * Remove redundant -fork. 2009-03-11 15:52:39 +00:00
Eelco Dolstra f945fbb5bd * Use Catalyst's HTTP::Prefork engine by default, with some limits to 
prevent the server from being overloaded.
 2009-03-11 15:48:06 +00:00
Eelco Dolstra 7ea69e7a9d * Disregard failing substituters. 2009-03-10 11:49:18 +00:00
Eelco Dolstra 18f25a1226 * Doh. 2009-03-10 08:52:42 +00:00
Eelco Dolstra fca7fb20c4 * Negative caching: don't perform a build if a dependency already 
failed in a previous build.  This is essential for Nixpkgs: we don't
  want to keep doing the same failed dependency (say, Glibc) over and
  over again for a few hundred jobs.
 2009-03-09 17:21:10 +00:00
Eelco Dolstra 8725dc03ec * Use ->update({...}) properly. 2009-03-09 16:22:41 +00:00
Eelco Dolstra e87896e4b4 * eval-jobs -> hydra_eval_jobs. 2009-03-09 15:16:11 +00:00
Eelco Dolstra a74bfdf533 * Inputs of type "build" must now be declared explicitly. 2009-03-09 13:58:43 +00:00
Eelco Dolstra 8c58448afc * hydra_scheduler: use eval-jobs. 2009-03-09 13:04:46 +00:00
Eelco Dolstra 36fdd7f37f * For products that are directories (like manuals), allow a default 
suffix other than index.html to be declared.  E.g. if a build does

    echo "doc manual $out manual.html" >> $out/nix-support/hydra-build-products

  the default link for the product is

    http://localhost:3000/build/417/download/1/manual.html

  but other files are also accessible, e.g.
    
    http://localhost:3000/build/417/download/1/style.css
 2009-03-06 13:34:53 +00:00
Eelco Dolstra 97ed2052ba * Move everything up one directory. 2009-03-05 13:41:57 +00:00