Commit graph

926 commits

Author SHA1 Message Date
Eelco Dolstra f6081668dc
Allow determinism checking for entire jobsets
Setting

  xxx-jobset-repeats = patchelf:master:2

will cause Hydra to perform every build step in the specified jobset 2
additional times (i.e. 3 times in total). Non-determinism is not fatal
unless the derivation has the attribute "isDeterministic = true"; we
just note the lack of determinism in the Hydra database. This will
allow us to get stats about the (lack of) reproducibility of all of
Nixpkgs.
2016-12-07 15:57:13 +01:00
Eelco Dolstra d0ad3fd806
Revert "Remove the global channel"
This reverts commit 2f6c2f5622.

This unintentionally also removed /all, which is actually somewhat
useful.
2016-12-06 14:25:09 +01:00
Eelco Dolstra 2f6c2f5622
Remove the global channel
It's useless and it makes it very easy to kill the server by fetching
/channel/latest/closure.
2016-11-17 18:17:02 +01:00
Eelco Dolstra 98e7e37832
hydra-server: Fix and simplify robots.txt 2016-11-17 18:13:57 +01:00
Eelco Dolstra 07decd6915 Also hide disabled projects on the overview page by default 2016-11-01 13:15:55 +01:00
Eelco Dolstra ef711ce845 Handle Referer not having a trailing slash
Fixes #415.
2016-11-01 11:00:59 +01:00
Eelco Dolstra a2be29377e Hide hidden jobsets for project admins by default
This unclutters project pages.

Fixes #390.
2016-10-31 11:40:36 +01:00
Eelco Dolstra 2b790ceefa Remove the pretty-printed log
It didn't work anymore due to the removal of nix-log2xml. And in any
case XSL is the work of the devil.
2016-10-27 14:43:17 +02:00
Eelco Dolstra a12e9478e7 When manually scheduling an eval, force re-instantiation of store derivations
Without this, if (failed or aborted) derivations have been
garbage-collected, there is no way to restart them, which is very
annoying. Now we set a forceEval flag in the jobset to cause it to be
re-evaluated even if none of the inputs have changed.
2016-10-24 20:20:20 +02:00
Eelco Dolstra 140cbe9302 Set utf-8 charset for raw logs
Also fix incorrect use of Content-Encoding header in View::Plain.
2016-10-24 17:14:33 +02:00
Eelco Dolstra f7ff7f741b Fix a couple of encoding issues 2016-10-24 16:49:11 +02:00
Eelco Dolstra 5d8fbc6ca1 Fix referer check 2016-10-21 17:56:34 +02:00
Eelco Dolstra c928c41ee1 Add XSRF protection for POST requests
Some Hydra API requests were vulnerable to XSRF attacks, e.g. you
could have a form on another website using http://hydra/logout as the
form action. So we now require POST requests to come from the same
origin.

Reported by Hans-Christian Esperer.
2016-10-20 16:11:33 +02:00
Eelco Dolstra bbe45ed844 Remove Persona support
Persona is no longer supported by Mozilla, so let's remove it.
2016-10-20 14:14:04 +02:00
Eelco Dolstra f2724d088d Test: Show error output from hydra-queue-runner on timeout 2016-10-06 17:18:59 +02:00
Eelco Dolstra b03e3c593c Merge pull request #338 from expipiplus1/apache-include
Enable apache style includes in the config file
2016-08-19 13:32:44 +02:00
Domen Kožar 2840a5e089 push_github: return empty response instead of a template, fixes #309 2016-08-17 00:21:18 +02:00
Joe Hermaszewski 12194447ee GitHub push suffix (#332)
* Fix api/push-github

Fixes #309

* Don't require .git suffix on jobset inputs
2016-08-17 00:00:09 +02:00
Domen Kožar 03e8538746 Merge pull request #331 from expipiplus1/github-push
Fix api/push-github
2016-08-16 23:59:12 +02:00
Domen Kožar 23bf99b8e2 Use pixz instead of bzip2 for compressing nars
According to following two benchmarks:

- https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/
- http://catchchallenger.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

xz has better compression ratio than bzip2 at lowest compression rate.

https://github.com/vasi/pixz has been chosen as it can scale compressing
over multiple cores linearly.

We're using this in snabblab for a month now and it has improved CPU
wise the main Hydra server.
2016-08-12 11:39:07 +02:00
Joe Hermaszewski 02d56dda58 Trigger ssh like urls from github pushes
URLs like "git@github.com:owner/repo.git

This commit removed the need for a forward slash between 'github.com' and
'owner'.
2016-06-21 16:34:54 +01:00
Joe Hermaszewski 02baff987a Enable apache style includes in the config file 2016-06-19 16:59:24 +01:00
Joe Hermaszewski eb9aba0ff4 Fix api/push-github
Fixes #309
2016-06-14 10:36:43 +01:00
Eelco Dolstra a55942603a Provide a plugin hook for when build steps finish
Fixes #318.
2016-05-27 14:35:32 +02:00
Eelco Dolstra f70946efca Allow public dashboards
Dashboards can now be marked as publically visible in the user
preferences. The dashboard URL has changed from /user/<name>/dashboard
to /dashboard/<name> because /user/<name> requires being logged in as
<name> or as an admin.
2016-05-27 14:35:32 +02:00
Shea Levy 4392d3e21d Enable declarative projects.
This allows fully declarative project specifications. This is best
illustrated by example:

* I create a new project, setting the declarative spec file to
  "spec.json" and the declarative input to a git repo pointing
  at git://github.com/shlevy/declarative-hydra-example.git
* hydra creates a special ".jobsets" jobset alongside the project
* Just before evaluating the ".jobsets" jobset, hydra fetches
  declarative-hydra-example.git, reads spec.json as a jobset spec,
  and updates the jobset's configuration accordingly:
{
    "enabled": 1,
    "hidden": false,
    "description": "Jobsets",
    "nixexprinput": "src",
    "nixexprpath": "default.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}
* When the "jobsets" job of the ".jobsets" jobset completes, hydra
  reads its output as a JSON representation of a dictionary of
  jobset specs and creates a jobset named "master" configured
  accordingly (In this example, this is the same configuration as
  .jobsets itself, except using release.nix instead of default.nix):
{
    "enabled": 1,
    "hidden": false,
    "description": "js",
    "nixexprinput": "src",
    "nixexprpath": "release.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}
2016-05-09 08:54:27 -04:00
Eelco Dolstra b2b978eda0 Fix JSON output for builds 2016-04-18 11:44:59 +02:00
Eelco Dolstra ef72569cc3 Merge pull request #280 from shlevy/github-status-api
Add a plugin to interact with the github status API.
2016-04-14 20:03:45 +02:00
Eelco Dolstra a7755678fe Drop unused BuildProducts.description column 2016-04-13 16:30:52 +02:00
Shea Levy ae24e7fb31 GithubStatus: Include the full job name and build ID in the context.
Build ID can be omitted by setting excludeBuildFromContext in config
2016-04-12 15:19:05 -04:00
Shea Levy 0d0b925af1 Add a plugin to interact with the github status API.
Mutliple <githubstatus> sections are possible:
* jobs: regexp for jobs to match
* inputs: the input which corresponds to the github repo/rev whose
  status we want to report. Can be repeated
* authorization: Verbatim contents of the Authorization header. See
  https://developer.github.com/v3/#authentication.
2016-04-12 14:42:01 -04:00
Shea Levy 9b37cb89ae Add buildStarted plugin hook 2016-04-12 14:42:01 -04:00
Eelco Dolstra ed88bbaac0 Set Vary to Accept
Otherwise, the browser may mix up HTML and JSON responses if it has
requested both. For example, hitting the back button to return to a
job metric page will show a JSON response, because that was the last
thing the browser fetched for that URL.

This requires Catalyst::Action::Rest >= 1.20.
2016-03-25 14:48:12 +01:00
Eelco Dolstra dab16fb26b Lazy load the metrics tab 2016-03-25 13:49:06 +01:00
Eelco Dolstra 759bd38ef2 Sort metrics by name 2016-03-25 11:56:25 +01:00
Eelco Dolstra 6fc4dc4e27 /queue-summary: Show number of queued builds by system type 2016-03-22 17:03:26 +01:00
Eelco Dolstra d5cffd4bc7 Make "Running builds" and "Machine status" pages faster 2016-03-16 15:19:18 +01:00
Eelco Dolstra 520c8a5826 Use faster query to determine number of running builds
The previous query

  select count(*) from builds b left join buildsteps s on s.build = b.id where busy = 1 and finished = 0

is suddenly taking several minutes. Probably PostgreSQL decided to use
a suboptimal query plan.
2016-03-16 13:41:43 +01:00
Eelco Dolstra 49a4639377 Add a more concise queue page
The old page didn't scale very well if you have 150K builds in the
queue, in fact it tended to make browsers hang. The new one just
shows, for each jobset, the number of queued builds. The actual builds
can be seen by going to the corresponding jobset page and looking at
the evals.
2016-03-08 19:44:51 +01:00
Eelco Dolstra 76104accda Return unique store paths 2016-03-03 11:32:30 +01:00
Eelco Dolstra e7655fdcbc Fix latest-finished 2016-03-02 18:06:20 +01:00
Eelco Dolstra e45bbfbef0 Fix .nixpkg channel uri
Fixes #274.
2016-03-02 15:38:40 +01:00
Eelco Dolstra 8b4f90b0d4 .nixpkgs: Drop obsolete manifest URI 2016-03-02 15:24:23 +01:00
Eelco Dolstra ec82bc2517 Add /eval/NNN/store-paths action to return store paths in an eval
Needed by the NixOS channel scripts since we can no longer get a
MANIFEST from Hydra.
2016-03-02 15:17:22 +01:00
Eelco Dolstra a74251af2b Disable channels on binary cached based Hydra instances 2016-03-02 15:08:53 +01:00
Shea Levy 0f5937503e SlackNotification: Use bigger images 2016-03-01 11:25:18 -05:00
Shea Levy 006ac1fc03 Add slack plugin.
Respects <slack> blocks in the hydra config, with attributes:

* jobs: a regexp matching the job name (in the format project:jobset:job)
* url: The URL to a slack incoming webhook
* force: If true, always send messages. Otherwise, only when the build status changes

Multiple <slack> blocks are allowed
2016-02-29 14:48:36 -05:00
Eelco Dolstra bc958c508b Merge branch 'binary-cache' 2016-02-29 18:29:07 +01:00
Eelco Dolstra e8cdfe5171 hydra-server: Don't barf if the binary cache public key can't be read 2016-02-26 21:14:40 +01:00
Eelco Dolstra 07e5fc5618 Hackery to make downloads work when using a binary cache 2016-02-26 17:28:26 +01:00