Commit graph

2212 commits

Author SHA1 Message Date
Shea Levy 86aa851d8d Merge branch 'circleci-plugin' 2017-04-24 11:28:09 -04:00
zraexy b3c4c4ef0d Make all content of rotated table header clickable 2017-04-20 15:58:17 -08:00
Shea Levy 4c26cf0c81 getHydraConfig: Allow multiple includes 2017-04-20 08:00:09 -04:00
Shea Levy f4b4ee5751 Add CircleCI plugin. 2017-04-19 15:20:09 -04:00
Eelco Dolstra cc85208fe4
Fix build 2017-04-18 20:50:18 +02:00
Joel Rivera af2f8122e3 Allow to configure the timeout value for the GitInput plugin in different places.
1. From the hydra configuration file.

    The configuration is loaded from the "git-input" block.
    Currently only the "timeout" variable is been looked up in the file.

    <git-input>
      # general timeout
      timeout = 400
      <input-name>
        # specific timeout for a particular input name
        timeout = 400
      </input-name>

      # use quotes when the input name has spaces
      <"foot with spaces">
        # specific timeout for a particular input name
        timeout = 400
      </"foo with spaces">
    </git-input>

2. As an argument in the input value after the repo url and branch (and after the deepClone if is defined)
   "timeout=<value>"

The preference on which value is used:

 1. input value
 2. Block with the name of the input in the <git-input> block
 3. "timeout" inside the <git-input> block
 4. Default value of 600 seconds. (original hard-coded value)

The code is generalized for more values to be configured, it might be too much
for a single value on a single plugin.
2017-04-13 19:10:53 -05:00
Shea Levy fed37ea640 nix-prefetch-git: Only remove .git, not e.g. .gitkeep (#454) 2017-04-12 21:08:04 +02:00
Eelco Dolstra c2c2633e50
Fix link to cached build log
Fixes #462.
2017-04-11 14:25:48 +02:00
Eelco Dolstra 433b94ee68
Fix Mercurial diff
It only showed the log messages of $rev1 and $rev2. Now it shows the
range $rev1::$rev2, excluding $rev1.
2017-04-11 13:39:54 +02:00
Eelco Dolstra 585d3274e3
Better error message if jobset doesn't evaluate to an attrset 2017-04-11 13:01:15 +02:00
Shea Levy 7e3f12fe10 Merge branch 'decl-project-remote-store' 2017-04-10 11:36:41 -04:00
Eelco Dolstra 426aea1236
hydra-queue-runner: Allow multiple concurrent daemon connections 2017-04-06 18:50:53 +02:00
Eelco Dolstra 5810042a3b
Periodically clear Store's path info cache
Otherwise the queue runner can consider paths as valid that have been
garbage-collected since the first time it queried them.
2017-04-06 17:20:23 +02:00
Eelco Dolstra 8364f4ec70
Upload log files to the right location
We were mixing up builds and steps. So for example

  https://cache.nixos.org/log/2w66a98iqbjdppc5s2b8qvhi3gprvy45-freecell-solver-4.8.0.drv

at the moment contains the log for
/nix/store/442r9d5ihbcpgq8q9dhijhvhlmplzp96-perl-namespace-autoclean-0.28.drv
because the latter is a step in http://hydra.nixos.org/build/51300420.
Oops.
2017-04-06 13:05:30 +02:00
Eelco Dolstra fec895a642
hydra-server: Support logs in S3 2017-04-05 17:55:56 +02:00
Eelco Dolstra 4f11cf45dc
Fix build cancellation
We nowadays ignore SIGINT, so the sshd child process inherited this
and ignored SIGINT as well.
2017-04-05 11:01:57 +02:00
Shea Levy a738f826e8 declarative projects: Pull jobset spec build from the remote store
Fixes #443
2017-04-01 10:38:49 -04:00
Will Dietz 90870f1416 NixExprs.pm: encode default.nix as utf-8 to fix missing chars/data. (#455)
cc #254
2017-04-01 11:18:51 +02:00
Eelco Dolstra 147ba3ca31
Set proper charset on log files 2017-03-31 18:00:08 +02:00
Shea Levy ec1c8efd43 Merge branch 'eval-no-build-hook' 2017-03-30 13:19:37 -04:00
Eelco Dolstra 8771f7f913 Merge pull request #382 from shlevy/cached-build-notifications
Send BuildFinished notifications on cached build results.
2017-03-29 18:52:20 +02:00
Eelco Dolstra 57bc0eaead
hydra-queue-runner: Limit concurrent database connections
Adding a 96-core aarch64 build machine to the build farm caused the
potential number of database connections to increase a lot, so we
started hitting the Postgres connection limit.
2017-03-21 11:53:46 +01:00
Shea Levy 3b5fd2d857 hydra-eval-jobs: Disable the build hook.
For some reason this causes an unexpected EOF when doing import-from-derivation.
2017-03-20 12:57:05 -04:00
Eelco Dolstra 150228d7de
Upload build logs to the binary cache 2017-03-15 16:59:57 +01:00
Eelco Dolstra 7e6486e694
Move log compression to a plugin 2017-03-15 16:59:57 +01:00
Eelco Dolstra 285754aff6
hydra-evaluator improvements
* The "Jobset" page now shows when evaluations are in progress (rather
  than just pending).

* Restored the ability to do a single evaluation from the command line
  by doing "hydra-evaluator <project> <jobset>".

* Fix some consistency issues between jobset status in PostgreSQL and
  in hydra-evaluator. In particular, "lastCheckedTime" was never
  updated internally.
2017-03-15 16:59:57 +01:00
Eelco Dolstra 242ea8009f
hydra-evaluator: Don't use the "timeout" command
"timeout" does not propagate PR_SET_PDEATHSIG, so it prevents
hydra-eval-jobset from being killed when hydra-evaluator quits.
2017-03-15 16:59:56 +01:00
Eelco Dolstra d1afb42f12
Supress debug message 2017-03-15 16:59:56 +01:00
Eelco Dolstra 73900e9f5f Fix std::stoi exception 2017-03-08 15:07:52 +01:00
Eelco Dolstra edebdf33f0
hydra-queue-runner: Handle SIGINT 2017-03-03 12:41:00 +01:00
Eelco Dolstra 500c27e4d5
Add hydra.conf option "nar_buffer_size" to configure memoryTokens limit
It defaults to half the physical RAM.
2017-03-03 12:37:27 +01:00
Rob Vermaas fd754d678e
Do not trigger eval on jobset change when check interval is 0 (disabled). 2017-02-23 09:15:26 +00:00
Eelco Dolstra 9f6afb3375
Lock paths in the scm cache
This is necessary now that hydra-evaluator runs multiple evaluations
in parallel, to prevent corruption of Git/Mercurial clones.
2017-02-21 18:12:52 +01:00
Eelco Dolstra 52753e49c2
hydra-evaluator: Handle SIGINT
This makes "systemctl stop hydra-evaluator" work promptly again.
2017-02-21 17:54:31 +01:00
Eelco Dolstra 81ac547d2b
Move most of AddBuilds to hydra-eval-jobset
Having this stuff in a separate module is a remnant of the time when
hydra-server could add builds to the queue directly. This is no longer
the case.
2017-02-21 17:20:48 +01:00
Eelco Dolstra 7081f97f3a
Pedantry: CLOCK_REALTIME -> CLOCK_MONOTONIC 2017-02-21 16:17:17 +01:00
Eelco Dolstra 53b1f7da64 Decrease memoryTokens 2017-02-03 14:44:52 +01:00
Eelco Dolstra a366f362e1 Use latest nixUnstable 2017-02-03 14:39:18 +01:00
Renzo Carbonara 63d8c8d631 Revert "Sort inputs table" (#448)
This reverts commit c103163825.

Fixes #412 #445
2017-01-30 18:20:08 +01:00
Renzo Carbonara de55303197 Fix SQL syntax for migration 50. It failed with PostgreSQL 9.4. (#416) 2016-12-09 18:18:17 +01:00
Eelco Dolstra 8a120006f0
Fix version test 2016-12-08 16:03:50 +01:00
Eelco Dolstra 9989e6c0f4
Get exact build start/stop times from the remote 2016-12-07 16:10:21 +01:00
Eelco Dolstra f6081668dc
Allow determinism checking for entire jobsets
Setting

  xxx-jobset-repeats = patchelf:master:2

will cause Hydra to perform every build step in the specified jobset 2
additional times (i.e. 3 times in total). Non-determinism is not fatal
unless the derivation has the attribute "isDeterministic = true"; we
just note the lack of determinism in the Hydra database. This will
allow us to get stats about the (lack of) reproducibility of all of
Nixpkgs.
2016-12-07 15:57:13 +01:00
Eelco Dolstra 8bb36e79bd
Support testing build determinism
Builds can now specify the attribute "isDeterministic = true" to tell
Hydra to build with build-repeat > 0. If there is a mismatch between
rounds, the step / build fails with a suitable status.

Maybe this should be a meta attribute, but that makes it invisible to
hydra-queue-runner, and it seems reasonable to make a claim of
mandatory determinism part of the derivation (since e.g. enabling this
flag should trigger a rebuild).
2016-12-06 17:46:06 +01:00
Eelco Dolstra d0ad3fd806
Revert "Remove the global channel"
This reverts commit 2f6c2f5622.

This unintentionally also removed /all, which is actually somewhat
useful.
2016-12-06 14:25:09 +01:00
Leonid Onokhov 8758d719c6 Remove logfile.css from templates (#428)
It is no longer used, and causes unnecessary 404s
2016-11-29 15:26:04 +01:00
Eelco Dolstra 2f6c2f5622
Remove the global channel
It's useless and it makes it very easy to kill the server by fetching
/channel/latest/closure.
2016-11-17 18:17:02 +01:00
Eelco Dolstra 98e7e37832
hydra-server: Fix and simplify robots.txt 2016-11-17 18:13:57 +01:00
Eelco Dolstra afb8765ae4
hydra-queue-runner: Bump memory limit to reflect more accurate accounting 2016-11-16 17:51:18 +01:00
Eelco Dolstra b4d32a3085
hydra-queue-runner: More accurate memory accounting
We now take into account the memory necessary for compressing the NAR
being exported to the binary cache, plus xz compression overhead.

Also, we now release the memory tokens for the NAR accessor *after*
releasing the NAR accessor. Previously the memory for the NAR accessor
might still be in use while another thread does an allocation, causing
the maximum to be exceeded temporarily.

Also, use notify_all instead of notify_one to wake up memory token
waiters. This is not very nice, but not every waiter is requesting the
same number of tokens, so some might be able to proceed.
2016-11-16 17:48:50 +01:00
Eelco Dolstra cb5e438a08 Bump Nix
Fixes #398.
2016-11-09 19:15:13 +01:00
Eelco Dolstra e2a012de8c hydra-evaluator: Set jobset error if hydra-eval-jobset dies 2016-11-08 17:08:54 +01:00
Eelco Dolstra 4576b3d35b hydra-evaluator: Show friendlier exit status 2016-11-08 17:00:17 +01:00
Eelco Dolstra 1ecc8a4f40 hydra-queue-runner: Fix a race keeping cancelled steps alive
If a step is cancelled just as its builder step is starting,
doBuildStep() will return sRetry. This causes builder() to make the
step runnable again, since the queue monitor may have added new builds
referencing it. The idea is that if the latter condition is not true,
the step's reference count will drop to zero and it will be
deleted. However, if the dispatcher thread sees and locks the step
before the reference count can drop to zero in the builder thread, the
dispatcher thread will start a new builder thread for the step. Thus
the step can be kept alive for an indefinite amount of time.

The fix is for State::builder() to use a weak pointer to the step, to
ensure that the step's reference count can drop to zero *before* it's
added to the runnable queue.
2016-11-08 11:47:49 +01:00
Eelco Dolstra de9d7bcf25 hydra-queue-runner: Handle exceptions in the dispatcher thread
E.g. "resource unavailable" when creating new threads.
2016-11-08 11:25:43 +01:00
Eelco Dolstra 7863d2e1da Step cancellation: Don't use pthread_cancel()
This was a bad idea because pthread_cancel() is unsalvageable broken
in C++. Destructors are not allowed to throw exceptions (especially in
C++11), but pthread_cancel() can cause a __cxxabiv1::__forced_unwind
exception inside any destructor that invokes a cancellation
point. (This exception can be caught but *must* be rethrown.) So let's
just kill the builder process instead.
2016-11-07 19:38:24 +01:00
Eelco Dolstra 95aa1f0590 Merge pull request #407 from grahamc/images
Update build images to be more friendly to users with color blindness.
2016-11-07 14:20:17 +01:00
Graham Christensen 6f6562bc5f
Update build images to be more friendly to users with color blindness.
Closes #406
2016-11-07 06:53:27 -05:00
Eelco Dolstra d7453bd8be hydra-queue-runner: Fix message 2016-11-02 12:44:18 +01:00
Eelco Dolstra 4f08c85c69 hydra-queue-runner: Fix assertion failure
It was hitting

    assert(reservation.unique());

Since we do want the machine reservation to be released before calling
wakeDispatcher(), let's use a different object for keeping track of
active steps.
2016-11-02 12:41:00 +01:00
Eelco Dolstra 07decd6915 Also hide disabled projects on the overview page by default 2016-11-01 13:15:55 +01:00
Eelco Dolstra 9072adece8 Fix broken logic for showing/hiding jobsets 2016-11-01 13:03:14 +01:00
Eelco Dolstra ef711ce845 Handle Referer not having a trailing slash
Fixes #415.
2016-11-01 11:00:59 +01:00
Eelco Dolstra b3169ce438 Kill active build steps when builds are cancelled
We now kill active build steps when there are no more referring
builds. This is useful e.g. for preventing cancelled multi-hour TPC-H
benchmark runs from hogging build machines.
2016-10-31 14:58:29 +01:00
Eelco Dolstra a816ef873d Warn against empty machines file 2016-10-31 11:40:36 +01:00
Eelco Dolstra a2be29377e Hide hidden jobsets for project admins by default
This unclutters project pages.

Fixes #390.
2016-10-31 11:40:36 +01:00
Graham Christensen 3c8f00c76f Clean up unused images (#404)
* Drop information_{32,64,128,256}.png because it appears to be unused.

* Delete unused images
2016-10-30 17:06:44 +01:00
Eelco Dolstra 2b790ceefa Remove the pretty-printed log
It didn't work anymore due to the removal of nix-log2xml. And in any
case XSL is the work of the devil.
2016-10-27 14:43:17 +02:00
Eelco Dolstra 41a020e323 Fix typo 2016-10-27 13:43:05 +02:00
Eelco Dolstra 3b84d4711b Bump Nix 2016-10-26 15:10:56 +02:00
Eelco Dolstra 0b00d51baf Prevent orphaned build steps
If two active steps of the same build failed, then the first would be
marked as "failed", but the second would end up as "orphaned", causing
it to be marked as "aborted" later on. Now it's correctly marked as
"failed".
2016-10-26 14:42:28 +02:00
Eelco Dolstra 8e1d791d0c Truncate the log just before starting the remote build
This gets rid of all those remote substitution messages that were
polluting the build logs.
2016-10-26 13:41:51 +02:00
Eelco Dolstra a12e9478e7 When manually scheduling an eval, force re-instantiation of store derivations
Without this, if (failed or aborted) derivations have been
garbage-collected, there is no way to restart them, which is very
annoying. Now we set a forceEval flag in the jobset to cause it to be
re-evaluated even if none of the inputs have changed.
2016-10-24 20:20:20 +02:00
Eelco Dolstra 140cbe9302 Set utf-8 charset for raw logs
Also fix incorrect use of Content-Encoding header in View::Plain.
2016-10-24 17:14:33 +02:00
Eelco Dolstra 3fcfa20d1a Fix regression caused by ee2e9f53
‘basicDrv.inputSrcs’ also contains the outputs of inputDrvs. These
don't necessarily exist in the local store, so copying them may cause
an exception. We should only copy the real inputSrcs.
2016-10-24 16:49:11 +02:00
Eelco Dolstra f7ff7f741b Fix a couple of encoding issues 2016-10-24 16:49:11 +02:00
Graham Christensen 3586bf28e4 Build status: copy the alt attribute to the title attribute for on-hover tool tips. (#405) 2016-10-23 13:21:21 +02:00
Eelco Dolstra a3efdcdfd9 Use std::regex 2016-10-21 18:06:26 +02:00
Eelco Dolstra 5d8fbc6ca1 Fix referer check 2016-10-21 17:56:34 +02:00
Eelco Dolstra c928c41ee1 Add XSRF protection for POST requests
Some Hydra API requests were vulnerable to XSRF attacks, e.g. you
could have a form on another website using http://hydra/logout as the
form action. So we now require POST requests to come from the same
origin.

Reported by Hans-Christian Esperer.
2016-10-20 16:11:33 +02:00
Eelco Dolstra bbe45ed844 Remove Persona support
Persona is no longer supported by Mozilla, so let's remove it.
2016-10-20 14:14:04 +02:00
Eelco Dolstra e0b2921ff2 Concurrent hydra-evaluator
This rewrites the top-level loop of hydra-evaluator in C++. The Perl
stuff is moved into hydra-eval-jobset. (Rewriting the entire evaluator
would be nice but is a bit too much work.) The new version has some
advantages:

* It can run multiple jobset evaluations in parallel.

* It uses PostgreSQL notifications so it doesn't have to poll the
  database. So if a jobset is triggered via the web interface or from
  a GitHub / Bitbucket webhook, evaluation of the jobset will start
  almost instantaneously (assuming the evaluator is not at its
  concurrency limit).

* It imposes a timeout on evaluations. So if e.g. hydra-eval-jobset
  hangs connecting to a Mercurial server, it will eventually be
  killed.
2016-10-14 14:22:12 +02:00
Eelco Dolstra 16feddd5d4 Drop obsolete -laws-cpp-sdk-s3 2016-10-14 14:22:12 +02:00
Eelco Dolstra dd5af7637d Remove finally.hh 2016-10-14 14:22:12 +02:00
Eelco Dolstra ee2e9f5335 Update to reflect BinaryCacheStore changes
BinaryCacheStore no longer implements buildPaths() and ensurePath(),
so we need to use copyPath() / copyClosure().
2016-10-07 20:23:05 +02:00
Eelco Dolstra f2724d088d Test: Show error output from hydra-queue-runner on timeout 2016-10-06 17:18:59 +02:00
Eelco Dolstra 6a313c691b hydra-queue-runner: Fix build 2016-10-06 16:58:54 +02:00
Alexander Ried 7089142fdc Add error/warnings for deprecated store specification 2016-10-06 15:10:14 +02:00
Alexander Ried a73f211bf2 Use store-api for binary cache instantiation 2016-10-06 15:09:44 +02:00
Alexander Ried 1c2f6281b9 Remove signing parameter (nix#f435f82) 2016-10-06 15:09:12 +02:00
Alexander Ried 232e6e8556 Replace buildVerbosity with verboseBuild (nix#5761827) 2016-10-06 15:08:02 +02:00
Alexander Ried 492d16074c Remove s3binarystore (moved to nix in d155d80) 2016-10-06 15:07:21 +02:00
Eelco Dolstra 706e06e8d7 hydra-eval-jobs: Fix build 2016-10-06 15:05:05 +02:00
Eelco Dolstra b1512a152a Fix build failure on GCC 5.4 2016-09-30 17:05:07 +02:00
Michele Guerini Rocco fb5e7f9570 Use the new logo (#396) 2016-09-29 13:08:15 +02:00
Eelco Dolstra 682b4bcc73 Merge pull request #311 from domenkozar/product-list-table
product list: convert to a proper table
2016-08-19 13:40:42 +02:00
Eelco Dolstra b03e3c593c Merge pull request #338 from expipiplus1/apache-include
Enable apache style includes in the config file
2016-08-19 13:32:44 +02:00
Shea Levy 5962367ffc Send BuildFinished notifications on cached build results.
Fixes #342
2016-08-17 06:40:12 -04:00
Domen Kožar 2840a5e089 push_github: return empty response instead of a template, fixes #309 2016-08-17 00:21:18 +02:00
Joe Hermaszewski 12194447ee GitHub push suffix (#332)
* Fix api/push-github

Fixes #309

* Don't require .git suffix on jobset inputs
2016-08-17 00:00:09 +02:00
Domen Kožar 03e8538746 Merge pull request #331 from expipiplus1/github-push
Fix api/push-github
2016-08-16 23:59:12 +02:00
Eelco Dolstra 71440e7bd7 Merge pull request #379 from domenkozar/sort-inputs-tables
Sort inputs table
2016-08-12 14:10:14 +02:00
Domen Kožar c103163825 Sort inputs table 2016-08-12 13:05:28 +02:00
Domen Kožar 23bf99b8e2 Use pixz instead of bzip2 for compressing nars
According to following two benchmarks:

- https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/
- http://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

xz has better compression ratio than bzip2 at lowest compression rate.

https://github.com/vasi/pixz has been chosen as it can scale compressing
over multiple cores linearly.

We're using this in snabblab for a month now and it has improved CPU
wise the main Hydra server.
2016-08-12 11:39:07 +02:00
Eelco Dolstra dd32033657 hydra-update-gc-roots: Only keep outputs when in direct mode
This prevents the server from gradually filling up due to store paths
fetched by hydra-server that then get turned into a GC root by
hydra-update-gc-roots.
2016-07-12 19:23:10 +02:00
Joe Hermaszewski 35a67d19ba Print jobset name when evaluator fails 2016-07-11 17:37:09 +01:00
Joe Hermaszewski 02d56dda58 Trigger ssh like urls from github pushes
URLs like "git@github.com:owner/repo.git

This commit removed the need for a forward slash between 'github.com' and
'owner'.
2016-06-21 16:34:54 +01:00
Joe Hermaszewski 02baff987a Enable apache style includes in the config file 2016-06-19 16:59:24 +01:00
Joe Hermaszewski eb9aba0ff4 Fix api/push-github
Fixes #309
2016-06-14 10:36:43 +01:00
Eelco Dolstra a55942603a Provide a plugin hook for when build steps finish
Fixes #318.
2016-05-27 14:35:32 +02:00
Eelco Dolstra f70946efca Allow public dashboards
Dashboards can now be marked as publically visible in the user
preferences. The dashboard URL has changed from /user/<name>/dashboard
to /dashboard/<name> because /user/<name> requires being logged in as
<name> or as an admin.
2016-05-27 14:35:32 +02:00
Domen Kožar 1a99761a8e product-list: remove redundant Icon table header 2016-05-19 13:32:53 +01:00
Domen Kožar 026f891ed6 product list: convert to a proper table 2016-05-12 20:06:22 +01:00
Shea Levy aa7cc6d599 Declarative projects: Include the fetch info for the declarative input in the project eval 2016-05-09 08:54:27 -04:00
Shea Levy 4392d3e21d Enable declarative projects.
This allows fully declarative project specifications. This is best
illustrated by example:

* I create a new project, setting the declarative spec file to
  "spec.json" and the declarative input to a git repo pointing
  at git://github.com/shlevy/declarative-hydra-example.git
* hydra creates a special ".jobsets" jobset alongside the project
* Just before evaluating the ".jobsets" jobset, hydra fetches
  declarative-hydra-example.git, reads spec.json as a jobset spec,
  and updates the jobset's configuration accordingly:
{
    "enabled": 1,
    "hidden": false,
    "description": "Jobsets",
    "nixexprinput": "src",
    "nixexprpath": "default.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}
* When the "jobsets" job of the ".jobsets" jobset completes, hydra
  reads its output as a JSON representation of a dictionary of
  jobset specs and creates a jobset named "master" configured
  accordingly (In this example, this is the same configuration as
  .jobsets itself, except using release.nix instead of default.nix):
{
    "enabled": 1,
    "hidden": false,
    "description": "js",
    "nixexprinput": "src",
    "nixexprpath": "release.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}
2016-05-09 08:54:27 -04:00
Eelco Dolstra b829be29b6 Lazy load Google and Persona login stuff 2016-04-28 17:13:02 +02:00
Eelco Dolstra 0c4e8b13a1 Update jquery and jquery-ui and use SRI checking 2016-04-28 15:13:54 +02:00
Eelco Dolstra b50a105ca7 S3BinaryCacheStore: Use disk cache 2016-04-20 15:29:40 +02:00
Eelco Dolstra b2b978eda0 Fix JSON output for builds 2016-04-18 11:44:59 +02:00
Eelco Dolstra afb86638cd Updates for negative .narinfo caching 2016-04-15 15:39:20 +02:00
Eelco Dolstra 177bf25d64 Queue monitor: Bail out earlier if a step has failed previously
Currently, the hydra.nixos.org queue contains 1000s of Darwin builds
that all depend on a stdenv-darwin that previously failed. However,
before, first createStep() would construct a dependency graph for each
build, then getQueuedBuilds() would discover that one of the steps had
failed previously and discard all those steps. Since the graph
construction involves a lot of uncached calls to isValidPath(), this
took several seconds per build.

Now createStep() detects the previous failure right away and bails
out.
2016-04-15 14:32:16 +02:00
Eelco Dolstra ef72569cc3 Merge pull request #280 from shlevy/github-status-api
Add a plugin to interact with the github status API.
2016-04-14 20:03:45 +02:00
Eelco Dolstra d6f188a01a Typo 2016-04-13 16:45:40 +02:00
Eelco Dolstra b1e36b550c max-output-size -> max_output_size
To be consistent with other Catalyst/Hydra config option names.
2016-04-13 16:30:52 +02:00
Eelco Dolstra 077ed3f571 Periodically clear orphaned build steps
These are build steps that remain "busy" in the database even though
they have finished, because they couldn't be updated (e.g. due to a
PostgreSQL connection problem). To prevent them from showing up as
busy in the "Machine status" page, we now periodically purge them.
2016-04-13 16:30:52 +02:00
Eelco Dolstra f3f661bac1 Reuse build products / metrics stored in the database
Previously, if the queue monitor thread encounters a build that Hydra
has previously built, it downloaded the output paths from the binary
cache, just to determine the build products and metrics. This is very
inefficient. In particular, when doing something like merging
nixpkgs:staging into nixpkgs:master, the queue monitor thread will be
locked up for a long time fetching files from S3, causing the build
farm to be mostly idle.

Of course this is entirely unnecessary, since the build
products/metrics are already in the Hydra database. So now we just
look up a previous build with the same output path, and copy the
products/metrics.
2016-04-13 16:30:52 +02:00
Eelco Dolstra a7755678fe Drop unused BuildProducts.description column 2016-04-13 16:30:52 +02:00
Eelco Dolstra 8c7edb1005 Fix narrowing conversion 2016-04-13 16:30:52 +02:00
Eelco Dolstra 00c78440b1 Disambiguate "marking build as succeeded" message 2016-04-13 16:30:52 +02:00
Eelco Dolstra ad834343b5 Fix build against current Nix master 2016-04-13 16:30:52 +02:00
Shea Levy ae24e7fb31 GithubStatus: Include the full job name and build ID in the context.
Build ID can be omitted by setting excludeBuildFromContext in config
2016-04-12 15:19:05 -04:00
Shea Levy 0d0b925af1 Add a plugin to interact with the github status API.
Mutliple <githubstatus> sections are possible:
* jobs: regexp for jobs to match
* inputs: the input which corresponds to the github repo/rev whose
  status we want to report. Can be repeated
* authorization: Verbatim contents of the Authorization header. See
  https://developer.github.com/v3/#authentication.
2016-04-12 14:42:01 -04:00
Shea Levy 9b37cb89ae Add buildStarted plugin hook 2016-04-12 14:42:01 -04:00
Eelco Dolstra ed88bbaac0 Set Vary to Accept
Otherwise, the browser may mix up HTML and JSON responses if it has
requested both. For example, hitting the back button to return to a
job metric page will show a JSON response, because that was the last
thing the browser fetched for that URL.

This requires Catalyst::Action::Rest >= 1.20.
2016-03-25 14:48:12 +01:00
Eelco Dolstra 32adc53070 Add tooltips to metrics showing the exact value of the data point 2016-03-25 14:32:36 +01:00
Eelco Dolstra 3e2911803d Add link to metrics 2016-03-25 13:57:17 +01:00
Eelco Dolstra dab16fb26b Lazy load the metrics tab 2016-03-25 13:49:06 +01:00
Eelco Dolstra 7a72f64e5e Move chart code to common.js 2016-03-25 13:33:10 +01:00
Eelco Dolstra dc2010eafc Fix rendering of metrics with dots in their name 2016-03-25 13:24:43 +01:00
Eelco Dolstra ef63dd77e3 Fix metric alignment 2016-03-25 12:08:18 +01:00
Eelco Dolstra 759bd38ef2 Sort metrics by name 2016-03-25 11:56:25 +01:00
Eelco Dolstra 6fc4dc4e27 /queue-summary: Show number of queued builds by system type 2016-03-22 17:03:26 +01:00
Eelco Dolstra ddc9f3cc6a Temporarily disable machines on any exception, not just connection failures 2016-03-22 16:54:40 +01:00
Eelco Dolstra 0aecd65e59 /queue-runner-status: Include info about temporarily disabled machines 2016-03-22 16:54:06 +01:00
Eelco Dolstra ac23bd1539 Revert "Apply IndexBuildsOnJobFinishedId to unfinished builds only"
This reverts commit 1de5ce7a0e.
2016-03-16 17:04:20 +01:00
Eelco Dolstra 7d8bf1b0f2 Shorten host names 2016-03-16 15:23:56 +01:00
Eelco Dolstra d5cffd4bc7 Make "Running builds" and "Machine status" pages faster 2016-03-16 15:19:18 +01:00
Eelco Dolstra 1de5ce7a0e Apply IndexBuildsOnJobFinishedId to unfinished builds only 2016-03-16 15:17:10 +01:00
Eelco Dolstra 520c8a5826 Use faster query to determine number of running builds
The previous query

  select count(*) from builds b left join buildsteps s on s.build = b.id where busy = 1 and finished = 0

is suddenly taking several minutes. Probably PostgreSQL decided to use
a suboptimal query plan.
2016-03-16 13:41:43 +01:00
Eelco Dolstra 405a43c171 Queue summary: Make rows clickable 2016-03-10 16:48:06 +01:00
Eelco Dolstra 5535bc28ca Tweak 2016-03-10 16:46:15 +01:00
Eelco Dolstra 60e7930d2b Bump memory limit a bit 2016-03-10 16:46:01 +01:00
Eelco Dolstra 75e7b35477 Fix retry of transient failures 2016-03-10 16:44:26 +01:00
Eelco Dolstra de71d5b622 Fix showing machine name for aborted build steps 2016-03-10 16:42:36 +01:00
Eelco Dolstra 33da40f272 Doh 2016-03-09 17:31:57 +01:00
Eelco Dolstra 4b9c76e502 hydra-queue-runner: Ensure regular status dumps 2016-03-09 17:11:34 +01:00
Eelco Dolstra 4151be7e69 Make the output size limit configurable
The maximum output size per build step (as the sum of the NARs of each
output) can be set via hydra.conf, e.g.

  max-output-size = 1000000000

The default is 2 GiB.

Also refactored the build error / status handling a bit.
2016-03-09 17:00:09 +01:00
Eelco Dolstra dc790c5f7e Fix bad format string 2016-03-09 16:59:35 +01:00
Eelco Dolstra 80ff78b1b6 Unify build and step status codes
Also remove the obsolete status code 5 from the database.
2016-03-09 15:30:43 +01:00
Eelco Dolstra 9127f5bbc3 hydra-queue-runner: Limit memory usage
When using a binary cache store, the queue runner receives NARs from
the build machines, compresses them, and uploads them to the
cache. However, keeping multiple large NARs in memory can cause the
queue runner to run out of memory. This can happen for instance when
it's processing multiple ISO images concurrently.

The fix is to use a TokenServer to prevent the builder threads to
store more than a certain total size of NARs concurrently (at the
moment, this is hard-coded at 4 GiB). Builder threads that cause the
limit to be exceeded will block until other threads have finished.

The 4 GiB limit does not include certain other allocations, such as
for xz compression or for FSAccessor::readFile(). But since these are
unlikely to be more than the size of the NARs and hydra.nixos.org has
32 GiB RAM, it should be fine.
2016-03-09 14:30:13 +01:00
Eelco Dolstra 49a4639377 Add a more concise queue page
The old page didn't scale very well if you have 150K builds in the
queue, in fact it tended to make browsers hang. The new one just
shows, for each jobset, the number of queued builds. The actual builds
can be seen by going to the corresponding jobset page and looking at
the evals.
2016-03-08 19:44:51 +01:00
Eelco Dolstra b77a43b83d Get rid of "will retry" messages after "maybe cancelling..." 2016-03-08 13:09:39 +01:00
Eelco Dolstra 718fef29ef Keep track of time required to load builds 2016-03-08 13:09:29 +01:00
Eelco Dolstra 2feb17c681 Some more logging 2016-03-08 13:08:07 +01:00
Eelco Dolstra 45b237453a hydra-queue-runner: Recycle finishedDrvs
This should prevent the queue monitor thread from looking up the same
derivations over and over again.
2016-03-08 11:52:13 +01:00
Eelco Dolstra 2ab8e9a1e0 hydra-queue-runner: Fix handling of missing derivations
This barfed with 'queue monitor: ERROR: column "errormsg" of relation
"builds" does not exist' due to the removal of the errorMsg column.
2016-03-07 19:05:24 +01:00
Eelco Dolstra e7ce225558 Fix build 2016-03-04 17:51:32 +01:00
Eelco Dolstra 76104accda Return unique store paths 2016-03-03 11:32:30 +01:00
Eelco Dolstra 86a2d6471c Fix a boost format string abort 2016-03-02 20:06:48 +01:00
Eelco Dolstra e7655fdcbc Fix latest-finished 2016-03-02 18:06:20 +01:00
Eelco Dolstra 232ca8fea2 Fix build 2016-03-02 17:05:07 +01:00
Eelco Dolstra e45bbfbef0 Fix .nixpkg channel uri
Fixes #274.
2016-03-02 15:38:40 +01:00
Eelco Dolstra 8b4f90b0d4 .nixpkgs: Drop obsolete manifest URI 2016-03-02 15:24:23 +01:00
Eelco Dolstra ec82bc2517 Add /eval/NNN/store-paths action to return store paths in an eval
Needed by the NixOS channel scripts since we can no longer get a
MANIFEST from Hydra.
2016-03-02 15:17:22 +01:00
Eelco Dolstra a74251af2b Disable channels on binary cached based Hydra instances 2016-03-02 15:08:53 +01:00
Eelco Dolstra 2d6b585cb3 Merge branch 'slack-plugin' of https://github.com/shlevy/hydra 2016-03-02 15:03:03 +01:00
Eelco Dolstra b98a061c24 Add some instrumentation to keep track of dispatcher cost 2016-03-02 14:18:39 +01:00
Eelco Dolstra 6beee0ab49 Fix segfault sorting runnable steps
Same problem as d744362e4a.

    at /nix/store/ksvsbr7pg4z69bv6fbbc8h7x7rm2104m-gcc-4.9.3/include/c++/4.9.3/bits/predefined_ops.h:166
    __last@entry=..., __comp=...) at /nix/store/ksvsbr7pg4z69bv6fbbc8h7x7rm2104m-gcc-4.9.3/include/c++/4.9.3/bits/stl_algo.h:1827
    __comp=...) at /nix/store/ksvsbr7pg4z69bv6fbbc8h7x7rm2104m-gcc-4.9.3/include/c++/4.9.3/bits/stl_algo.h:4717
2016-03-02 13:59:24 +01:00
Shea Levy 0f5937503e SlackNotification: Use bigger images 2016-03-01 11:25:18 -05:00
Shea Levy 006ac1fc03 Add slack plugin.
Respects <slack> blocks in the hydra config, with attributes:

* jobs: a regexp matching the job name (in the format project:jobset:job)
* url: The URL to a slack incoming webhook
* force: If true, always send messages. Otherwise, only when the build status changes

Multiple <slack> blocks are allowed
2016-02-29 14:48:36 -05:00
Eelco Dolstra bc958c508b Merge branch 'binary-cache' 2016-02-29 18:29:07 +01:00
Eelco Dolstra 7cd08c7c46 Warn if PostgreSQL appears stalled 2016-02-29 15:10:30 +01:00
Eelco Dolstra 922dc541c2 Add log message 2016-02-29 11:58:06 +01:00
Eelco Dolstra 610a8d67ae Better AWS error messages 2016-02-26 22:40:27 +01:00
Eelco Dolstra 1a055e7e9e Reduce severity level of some message 2016-02-26 21:31:08 +01:00
Eelco Dolstra 6bb860fd6e Add FIXME 2016-02-26 21:15:05 +01:00
Eelco Dolstra e8cdfe5171 hydra-server: Don't barf if the binary cache public key can't be read 2016-02-26 21:14:40 +01:00
Eelco Dolstra 53ca41ef9f Use US standard S3 region 2016-02-26 20:57:47 +01:00
Eelco Dolstra c635f5d0ea Fix Makefile.am 2016-02-26 19:54:55 +01:00
Eelco Dolstra 07e5fc5618 Hackery to make downloads work when using a binary cache 2016-02-26 17:28:26 +01:00
Eelco Dolstra b00bdefa98 Fix hydra-server signing 2016-02-26 17:28:16 +01:00
Eelco Dolstra 9de336de7c Proxy local binary caches via hydra-server 2016-02-26 17:27:30 +01:00
Eelco Dolstra b9afaadfb3 Keep better bytesReceived/bytesSent stats 2016-02-26 16:17:05 +01:00
Eelco Dolstra 6d741d2ffa Prevent download of NARs we just uploaded 2016-02-26 15:21:44 +01:00
Eelco Dolstra 02190b0fef Support hydra-build-products on binary cache stores 2016-02-26 14:45:03 +01:00
Eelco Dolstra 8e24ad6f0d Sync with Nix 2016-02-25 10:58:31 +01:00
Eelco Dolstra 8321a3eb27 Sync with Nix 2016-02-24 14:04:31 +01:00
Eelco Dolstra 7b509237cd Bleh Automake 2016-02-22 18:05:15 +01:00
Eelco Dolstra 6c3ae36648 hydra-queue-runner: Get store mode configuration from hydra.conf
To use the local Nix store (default):

  store_mode = direct

To use a local binary cache:

  store_mode = local-binary-cache
  binary_cache_dir = /var/lib/hydra/binary-cache

To use an S3 bucket:

  store_mode = s3-binary-cache
  binary_cache_s3_bucket = my-nix-bucket

Also, respect binary_cache_{secret,public}_key_file for signing the
binary cache.
2016-02-22 17:23:06 +01:00
Eelco Dolstra 94817d77d9 BinaryCacheStore: Respect build-use-substitutes 2016-02-22 17:21:39 +01:00
Eelco Dolstra 5668aa5f71 After uploading a .narinfo, add it to the LRU cache 2016-02-20 10:35:16 +01:00
Eelco Dolstra 88a05763cc Pool local store connections 2016-02-20 00:04:08 +01:00
Eelco Dolstra 1cefd6cac8 Fix log message 2016-02-20 00:02:37 +01:00
Eelco Dolstra a593ebc58e Add missing file 2016-02-19 20:58:40 +01:00
Eelco Dolstra 2b76094a23 S3BinaryCacheStore::isValidPath(): Do a GET instead of HEAD 2016-02-19 17:41:11 +01:00
Eelco Dolstra bd76f9120a Cache .narinfo lookups 2016-02-19 16:19:40 +01:00
Eelco Dolstra a0f74047da Keep some statistics for the binary cache stores 2016-02-19 14:24:23 +01:00
Eelco Dolstra dc4a00347d Use a single BinaryCacheStore for all threads
This will make it easier to do caching / keep stats. Also, we won't
have S3Client's connection pooling if we create multiple S3Client
instances.
2016-02-18 17:31:19 +01:00
Eelco Dolstra 00a7be13a2 Make queue runner internal status available under /queue-runner-status 2016-02-18 17:11:46 +01:00
Eelco Dolstra 8c9fc677c1 Typo 2016-02-18 16:43:24 +01:00
Eelco Dolstra db3fcc0f5e Enable substitution on the build machines
If properly configured, this allows them to get store paths directly
from S3, rather than having to receive them from the queue runner.
2016-02-18 16:42:05 +01:00
Eelco Dolstra 2d40888e2e Add an S3-backed binary cache store 2016-02-18 16:18:50 +01:00
Eelco Dolstra 0e254ca66d Refactor local binary cache code into a subclass 2016-02-18 14:06:17 +01:00
Eelco Dolstra a992f688d1 Rename class 2016-02-18 13:02:20 +01:00
Eelco Dolstra de77cc2910 Rename file 2016-02-18 13:02:20 +01:00
Michael Mortensen d9a4f7a211 Construct uri using uri_for, which respects X-Request-Base. 2016-02-17 21:27:44 +01:00
Eelco Dolstra ce5790285a Merge remote-tracking branch 'origin/master' into binary-cache 2016-02-17 11:54:59 +01:00
Eelco Dolstra d7a123fcd4 Keep track of the time we spend copying to/from build machines 2016-02-17 10:30:23 +01:00
Eelco Dolstra 25022bf5fd hydra-queue-runner: Support generating a signed binary cache 2016-02-16 16:41:42 +01:00
Eelco Dolstra 744cee134e hydra-queue-runner: Compress binary cache NARs using xz 2016-02-15 21:56:53 +01:00
Eelco Dolstra 2d0dd7fb49 hydra-queue-runner: Write directly to a binary cache 2016-02-15 21:10:29 +01:00
Eelco Dolstra e46acbf05b Remove the errorMsg column from the Builds table
The queue runner no longer uses this field, and it doesn't provide
very interesting historical data (mostly SSH failures), but it takes
up a lot of space. Also, it contained some bad UTF-8 which was
preventing an upgrade to Postgres 9.5, so a good occasion to get rid
of it.
2016-02-12 17:21:55 +01:00
Eelco Dolstra 5c5bb3762c Fix display of builds that don't have an eval
Like http://hydra.nixos.org/build/22872.
2016-02-12 16:27:25 +01:00
Eelco Dolstra 92d8b59361 Process Nix API changes 2016-02-11 15:59:47 +01:00
Eelco Dolstra 993647d1e3 Use Google's verifier 2016-01-14 12:54:47 +01:00
Eelco Dolstra 5a580b1bb2 Add support for logging in via a Google account
The required configuration in hydra.conf:

  enable_google_login = 1
  google_client_id = 238429sdjkds....apps.googleusercontent.com

and optionally persona_allowed_domains to restrict to one or more
domains.
2016-01-13 17:32:52 +01:00
Eelco Dolstra f11ce7e219 Bump evaluation timeout to 6 hours
This is necessary given the current size of the Nixpkgs/NixOS
jobsets. Once we have a Nix store + Postgres on SSD, we can reduce
this again.

Should really make this configurable...
2016-01-07 16:19:54 +01:00
Eelco Dolstra 0ca6bb79af Shut up warning 2016-01-07 16:19:54 +01:00
Eelco Dolstra 97f8c61928 Fix hydra-queue-runner --build-one 2015-12-29 17:53:33 +01:00
Eelco Dolstra 29db16bc69 Fix 'Can't locate object method "buildstepoutputs"'
Template::Toolkit braindamage strikes again. See
8f85fe67d4.
2015-12-15 11:55:57 +01:00
Eelco Dolstra b081133dcc Move GC roots to /nix/var/nix/gcroots/hydra
The uid split a while back caused the web interface to create GC roots
in /nix/var/nix/gcroots/per-user/hydra-www, where they wouldn't be
purged by hydra-update-gc-roots. Thus restarted builds would
accumulate forever. The fix is to keep the roots in a shared directory
with gid=hydra.
2015-12-14 13:31:24 +01:00
Eelco Dolstra e04fb5c8e7 Sort build steps 2015-12-14 13:01:35 +01:00
Eelco Dolstra 8f7614030e Better fix for dots in jobset names 2015-11-17 11:31:11 +01:00
Eelco Dolstra ea80a75204 Move getBaseUrl 2015-11-17 11:21:20 +01:00
Rob Vermaas dddb9a281d Allow dots in job specifier of input type 'Previous build' 2015-11-17 08:36:46 +00:00
Eelco Dolstra b6d2c6f03a Further fix for the channel regression 2015-11-02 14:58:23 +01:00
aszlig 20b412224e ListBuilds: Fix legacy channel URLs.
Regression introduced by 1fdc258de0.

The commit introduced a channel/custom PathPart which uses the new
custom channel expressions, but I forgot to remove CaptureArgs, so the
URL really is channel/latest/ignored-value.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Reported-by: Peter Simons <simons@cryp.to>
2015-11-02 14:51:42 +01:00
Eelco Dolstra c087472c71 Remove superfluous "has" function 2015-11-02 14:29:12 +01:00
Eelco Dolstra a5879cbeb8 Consistent terminology: "Clear" -> "Cancel" 2015-11-02 14:22:25 +01:00
Eelco Dolstra e1f9a209c5 Fix cancelling non-current builds 2015-11-02 14:20:27 +01:00
Eelco Dolstra 2d128d2a6f Don't show redundant "removing machine..." messages 2015-10-30 18:22:43 +01:00
Eelco Dolstra d8d188301d Fix division-by-zero crash
Not clear why step_->jobsets was empty...
2015-10-30 18:01:48 +01:00
Eelco Dolstra 1ff48da3d3 int2String -> std::to_string 2015-10-30 18:01:38 +01:00
Eelco Dolstra 19b52e907f Add action to restart all failed builds in an eval 2015-10-28 15:04:12 +01:00
Eelco Dolstra b9ebd38edf Fix restarting builds
Failed paths were not cleared because the $builds query becomes empty
as soon as the "finished" flag has been cleared. So this needs to be
done last.
2015-10-28 14:54:54 +01:00
Eelco Dolstra bbe1fb2f9f Ignore non-existence of IndexBuildsOnFinishedBusy
Apparently for some reason not all Hydra installations had this
index. Another -1 for non-declarative schema definition languages.
2015-10-28 14:09:45 +01:00
Eelco Dolstra 4d1816b152 Remove obsolete Builds columns and provide accurate "Running builds"
This removes the "busy", "locker" and "logfile" columns, which are no
longer used by the queue runner. The "Running builds" page now only
shows builds that have an active build step.
2015-10-27 15:37:17 +01:00
Eelco Dolstra 9fb91460e3 Restart jobset evals efficiently
Fixes

  DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: stack depth limit exceeded
  HINT: Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate. [for Statement "UPDATE Builds SET busy = ?, finished = ?, iscachedbuild = ?, locker = ? WHERE ( ( id = ? OR id = ? OR ...
2015-10-27 13:43:19 +01:00
Eelco Dolstra 53c80d9526 getQueuedBuilds(): Periodically stop to handle priority bumps
Previously, priority bumps could take a long time to get noticed if
getQueuedBuilds() was busy processing zillions of queue
additions. (This was made worse by the reintroduction of substitute
checking.)
2015-10-22 17:00:46 +02:00
Eelco Dolstra 71bf7e02d5 Use nix::willBuildLocally() 2015-10-21 15:44:29 +02:00