Commit graph

2127 commits

Author SHA1 Message Date
Eelco Dolstra 7ccec0900b Fix links to steps in the machines page 2017-12-07 16:20:23 +01:00
Eelco Dolstra e9670641ec
Distinguish build step states
The web interface now shows whether a build step is connecting,
copying inputs/outputs, building, etc.
2017-12-07 15:35:31 +01:00
Eelco Dolstra 457483ba0e
Don't lock the BuildSteps table when inserting
Instead, optimistically insert a row and retry if there is a conflict.
2017-12-07 14:41:29 +01:00
Eelco Dolstra 2660c3cc54
Remove redundant check 2017-12-05 16:00:18 +01:00
AmineChikhaoui 52e0ebcfed
hide disabled jobsets by default. 2017-12-05 14:07:58 +01:00
Eelco Dolstra 364e21919a
hydra-eval-{jobs,jobset}: Pass file name as <...> 2017-11-28 16:51:00 +01:00
Eelco Dolstra b7bc4384b7
Fix serving plain text files 2017-11-21 17:55:52 +01:00
Eelco Dolstra 5986aae704
Add option eval_substituter
When using the "build" or "sysbuild" jobset input types in conjunction
with a binary cache store, the evaluator needs to be able to fetch
store paths from the binary cache. Typical usage:

  store_uri = s3://nix-test-cache?secret-key=...
  eval_substituter = s3://nix-test-cache

Also, the public key of the binary cache must be added to
binary-cache-public-keys in nix.conf, otherwise the local nix-daemon
won't allow the store paths to be copied over.
2017-11-20 16:12:20 +01:00
Eelco Dolstra 7857f83251
Serve directories and symlinks correctly 2017-11-14 17:17:07 +01:00
Eelco Dolstra 100249c066
Use git rev-list --count 2017-11-14 16:18:54 +01:00
Eelco Dolstra 2cdc84f34f
Fix build against Nix master
Also, remove support in hydra-eval-jobs for multiple jobset input
alternatives. The web interface hasn't supported this in a long
time. Thus we can use the regular "--arg" handler.
2017-10-26 13:10:14 +02:00
Eelco Dolstra eef0d8861b
Remove test line 2017-10-19 13:17:29 +02:00
Eelco Dolstra b04dc6c76e
Fix root creation when the root already exists but is owned by another user 2017-10-19 12:28:38 +02:00
Eelco Dolstra b904f25247 Hide contents links 2017-10-18 15:57:43 +02:00
Eelco Dolstra d8d71ea4a3 hydra-server: Use server_store_uri instead of store_uri, if set
The store_uri may contain parameters that cause Nix to fail when
running under the hydra-server user (e.g. the signing key).
2017-10-18 15:33:55 +02:00
Eelco Dolstra 62c6cca937
Make one-click installs work for binary cache stores
This requires setting binary_cache_public_uri in hydra.conf.
2017-10-18 14:09:28 +02:00
Eelco Dolstra 44ef743d84
Hide some things that require a local store 2017-10-18 13:44:41 +02:00
Eelco Dolstra 40a9b70df2
Shut up warning 2017-10-18 13:39:39 +02:00
Eelco Dolstra 30943dd217
Detect MIME type 2017-10-18 13:38:34 +02:00
Eelco Dolstra 42fbde0383
Use "nix cat-store" to serve files from the Nix store
This makes downloading/viewing build results work with binary cache
stores. For good performance, this should be used in conjunction with
ca580bec35,
i.e. you should set store_uri to something like

  s3://my-cache?local-nar-cache=/tmp/nar-cache

to cache NARs between requests.
2017-10-18 12:48:31 +02:00
Eelco Dolstra bc60fccf78
Remove remaining references to store_mode etc. 2017-10-18 12:23:07 +02:00
Eelco Dolstra cc64e51f75
USER -> LOGNAME for consistency
Don't remember why we use LOGNAME. Also ensure that it's set.
2017-10-18 11:23:00 +02:00
Will Dietz c81594f470 hydra-queue-runner: ensure roots directory exists
Fixes #513
2017-10-17 13:04:56 -05:00
Eelco Dolstra 45b138373b
hydra-queue-runner: Write GC roots for outputs paths
We lost this behaviour somewhere. So build outputs could be GC'ed when
running the collector with --option gc-keep-outputs false.
2017-10-12 18:55:38 +02:00
Remy Goldschmidt eb7631fb9d hydra-create-user now has --password-hash option (#504)
When creating a Hydra user with the `hydra-create-user` command, you can now
provide a SHA1 password hash with the `--password-hash` flag. This is useful for
the upcoming work on Fully Declarative Hydra, since the end user should not have
to specify plaintext passwords in their `configuration.nix` file.
2017-10-08 11:55:51 +01:00
Rob Vermaas b645b03536
Fix string comparison 2017-09-25 11:54:55 +00:00
Eelco Dolstra 27103398c9
Make maxLogSize configurable 2017-09-22 15:23:58 +02:00
Rob Vermaas 2318baaade
Add 'readonly_ips' option, to allow readonly access for certain IPs when 'private' option is enabled. 2017-09-22 11:36:49 +00:00
Eelco Dolstra b828224fee
Periodically close RemoteStore connections
This prevents an accumulation of temproots. See
89dc62c174.
2017-09-14 18:16:33 +02:00
Eelco Dolstra 6517446c34
Update to latest nixUnstable 2017-09-14 17:22:48 +02:00
Eelco Dolstra d676c91bd5
Return Builds.drvpath in the JSON API
E.g.

  curl -H 'Accept: application/json' http://127.0.0.4/build/3772858 | jq -r .drvpath
  /nix/store/97vla79lavlcsk8milskr06rfm34bskx-patchelf-0.8pre843_f4522f4.drv
2017-09-12 18:42:18 +02:00
Eelco Dolstra 4af97c57f5
Acquire the send lock only while actually sending
Thus, we no longer hold the send lock while substituting missing paths
on the build machine. This is a good thing in particular for macOS
builders which have a tendency to hang forever in curl downloads.
2017-09-01 16:28:49 +02:00
Eelco Dolstra 50ab80caf2
Don't wait forever to acquire the send lock 2017-09-01 15:29:06 +02:00
Eelco Dolstra b405837d4d Merge pull request #474 from zraexy/patch-1
Make all content of rotated table header clickable
2017-08-22 22:37:23 +02:00
Eelco Dolstra 337a72ef4d
Fix displaying errors from JSON requests 2017-08-17 15:32:46 +02:00
montag451 48609664a5 Fixes NixOS/hydra#402 (#424) 2017-08-10 14:16:24 +02:00
Eelco Dolstra 7c976d2aec
hydra-queue-runner: Make build notification more reliable
Previously, when hydra-queue-runner was restarted, any pending "build
finished" notifications were lost. Now hydra-queue-runner marks
finished but unnotified builds in the database and uses that to run
pending notifications at startup.
2017-07-26 15:17:51 +02:00
Will Dietz 719df63190 queue-monitor: never move lastBuildId forward without processing jobs. 2017-07-25 20:05:37 -05:00
Eelco Dolstra e117d85c2a
hydra-queue-runner: Set a thread title for the builder threads
This should make debugging slightly easier.
2017-07-25 15:59:41 +02:00
Shea Levy a6d9201947 GithubStatus: Allow config to specify the integration description and context 2017-07-24 15:44:10 -04:00
Eelco Dolstra e78b9fd4ee
hydra-queue-runner: Allow concurrent notifications
The queue runner can now run up to ‘max-concurrent-notifications’ in
parallel (default is 2). This is useful when some hydra-notify
invocations can take a long time to complete (e.g. because they need
to compress a giant build log) and we don't want this to block all
other notifications.
2017-07-24 16:35:34 +02:00
Eelco Dolstra bba383bf1b
hydra-queue-runner: Keep some notification statistics 2017-07-24 16:26:44 +02:00
Eelco Dolstra f46a21e16e
Slight cleanup 2017-07-21 17:22:11 +02:00
Eelco Dolstra dc5e0b120a
Fix a race that can cause hydra-queue-runner to ignore newly added builds
As @dtzWill discovered, with the concurrent hydra-evaluator, there can
be multiple active transactions adding builds to the database. As a
result, builds can become visible in a non-monotonically increasing
order, breaking the queue monitor's assumption that build IDs only go
up.

The fix is to have hydra-eval-jobset provide the lowest build ID it
just added in the builds_added notification, and have the queue
monitor check from there.

Fixes #496.
2017-07-21 14:34:48 +02:00
Eelco Dolstra b0432c7762
Sync with nixUnstable 2017-07-21 13:09:08 +02:00
Eelco Dolstra 6fc851d376 Improve erorr message 2017-07-17 14:10:34 +02:00
Eelco Dolstra 66ae66024e Sync with latest Nix 2017-07-17 11:38:58 +02:00
Eelco Dolstra 064fd20b6f Merge pull request #494 from Chakerbh/master
Use build id for bitbucket Key
2017-07-11 17:41:22 +02:00
Chaker Benhamed c5666095a1 Use build id for bitbucket Key 2017-07-11 16:13:56 +01:00
Eelco Dolstra f95516a82b Merge pull request #466 from cyraxjoe/configurable-git-input
Allow to configure the timeout value for the GitInput plugin in different places.
2017-07-07 11:03:06 +02:00
Chaker Benhamed 8da3ca6e94 Skip the input if the emailResponsible field is disabled 2017-07-05 15:51:20 +01:00
Chaker Benhamed fd8cb39a11 Add BitBucket plugin
This plugin will post to the build status system in BitBucket. In order
to use it you need to add to ExtraConfig
<bitbucket>
username = bitbucket_username
password = bitbucket_password
</bitbucket>
You can use an application password https://blog.bitbucket.org/2016/06/06/app-passwords-bitbucket-cloud/
2017-07-04 10:07:25 +01:00
Eelco Dolstra 803833aba7 Disable the build-queued hook
This can take an excessive amount of time. For example, on
hydra.nixos.org, a call to hydra-notify takes 0.7s even if there are
no plugins. So for an eval with ~45K new builds, the calls to
hydra-notify add up to about 9 hours.

The proper fix would be to pass a list of build IDs, or an eval ID.
2017-06-22 15:09:04 +02:00
Shea Levy 207d2dd10c GithubStatus: Don't send a pending status for already-finished builds
Fixes #488
2017-06-21 14:51:09 -04:00
michael bishop 6e4a7a30f7
fix github pulls json to be deterministic 2017-06-21 09:35:26 -03:00
Eelco Dolstra f0d24af940
Fix "Evaluation pending since..." message even though the jobset has been evaluated 2017-06-12 14:15:00 +02:00
Shea Levy dfb3a52f67 Fix build-queued plugin hook 2017-05-24 11:12:48 -04:00
Shea Levy c40077ece1 Fix tests 2017-05-24 10:00:38 -04:00
Shea Levy 582c399420 Add buildQueued plugin hook 2017-05-24 09:45:31 -04:00
Shea Levy 3d044d2ec2 Github status API: Pull authorization from github_authorization by default 2017-05-24 08:53:29 -04:00
Shea Levy 08708e899a GithubPulls: Add 'revision' to the input 2017-05-24 07:33:06 -04:00
Shea Levy c6f0753435 Add GithubPulls input plugin.
This can be used with declarative projects to build PRs.

The github_authorization section should contain verbatim Authorization header contents keyed by repo owner for private repos
2017-05-24 05:31:27 -04:00
Eelco Dolstra 941665044e
/api/scmdiff: Set correct MIME type
It was set to application/json but should be text/plain.
2017-05-05 15:58:38 +02:00
Joel Rivera a39ffba042 Include the project and jobset names in the configuration blocks
to set the timeout for a specific input in the GitInput plugin.

Remove the comments about using inputs with spaces.
2017-04-28 21:43:35 -05:00
Eelco Dolstra 1f94f03699
Fix build 2017-04-26 15:11:12 +02:00
Shea Levy 86aa851d8d Merge branch 'circleci-plugin' 2017-04-24 11:28:09 -04:00
zraexy b3c4c4ef0d Make all content of rotated table header clickable 2017-04-20 15:58:17 -08:00
Shea Levy 4c26cf0c81 getHydraConfig: Allow multiple includes 2017-04-20 08:00:09 -04:00
Shea Levy f4b4ee5751 Add CircleCI plugin. 2017-04-19 15:20:09 -04:00
Eelco Dolstra cc85208fe4
Fix build 2017-04-18 20:50:18 +02:00
Joel Rivera af2f8122e3 Allow to configure the timeout value for the GitInput plugin in different places.
1. From the hydra configuration file.

    The configuration is loaded from the "git-input" block.
    Currently only the "timeout" variable is been looked up in the file.

    <git-input>
      # general timeout
      timeout = 400
      <input-name>
        # specific timeout for a particular input name
        timeout = 400
      </input-name>

      # use quotes when the input name has spaces
      <"foot with spaces">
        # specific timeout for a particular input name
        timeout = 400
      </"foo with spaces">
    </git-input>

2. As an argument in the input value after the repo url and branch (and after the deepClone if is defined)
   "timeout=<value>"

The preference on which value is used:

 1. input value
 2. Block with the name of the input in the <git-input> block
 3. "timeout" inside the <git-input> block
 4. Default value of 600 seconds. (original hard-coded value)

The code is generalized for more values to be configured, it might be too much
for a single value on a single plugin.
2017-04-13 19:10:53 -05:00
Shea Levy fed37ea640 nix-prefetch-git: Only remove .git, not e.g. .gitkeep (#454) 2017-04-12 21:08:04 +02:00
Eelco Dolstra c2c2633e50
Fix link to cached build log
Fixes #462.
2017-04-11 14:25:48 +02:00
Eelco Dolstra 433b94ee68
Fix Mercurial diff
It only showed the log messages of $rev1 and $rev2. Now it shows the
range $rev1::$rev2, excluding $rev1.
2017-04-11 13:39:54 +02:00
Eelco Dolstra 585d3274e3
Better error message if jobset doesn't evaluate to an attrset 2017-04-11 13:01:15 +02:00
Shea Levy 7e3f12fe10 Merge branch 'decl-project-remote-store' 2017-04-10 11:36:41 -04:00
Eelco Dolstra 426aea1236
hydra-queue-runner: Allow multiple concurrent daemon connections 2017-04-06 18:50:53 +02:00
Eelco Dolstra 5810042a3b
Periodically clear Store's path info cache
Otherwise the queue runner can consider paths as valid that have been
garbage-collected since the first time it queried them.
2017-04-06 17:20:23 +02:00
Eelco Dolstra 8364f4ec70
Upload log files to the right location
We were mixing up builds and steps. So for example

  https://cache.nixos.org/log/2w66a98iqbjdppc5s2b8qvhi3gprvy45-freecell-solver-4.8.0.drv

at the moment contains the log for
/nix/store/442r9d5ihbcpgq8q9dhijhvhlmplzp96-perl-namespace-autoclean-0.28.drv
because the latter is a step in http://hydra.nixos.org/build/51300420.
Oops.
2017-04-06 13:05:30 +02:00
Eelco Dolstra fec895a642
hydra-server: Support logs in S3 2017-04-05 17:55:56 +02:00
Eelco Dolstra 4f11cf45dc
Fix build cancellation
We nowadays ignore SIGINT, so the sshd child process inherited this
and ignored SIGINT as well.
2017-04-05 11:01:57 +02:00
Shea Levy a738f826e8 declarative projects: Pull jobset spec build from the remote store
Fixes #443
2017-04-01 10:38:49 -04:00
Will Dietz 90870f1416 NixExprs.pm: encode default.nix as utf-8 to fix missing chars/data. (#455)
cc #254
2017-04-01 11:18:51 +02:00
Eelco Dolstra 147ba3ca31
Set proper charset on log files 2017-03-31 18:00:08 +02:00
Shea Levy ec1c8efd43 Merge branch 'eval-no-build-hook' 2017-03-30 13:19:37 -04:00
Eelco Dolstra 8771f7f913 Merge pull request #382 from shlevy/cached-build-notifications
Send BuildFinished notifications on cached build results.
2017-03-29 18:52:20 +02:00
Eelco Dolstra 57bc0eaead
hydra-queue-runner: Limit concurrent database connections
Adding a 96-core aarch64 build machine to the build farm caused the
potential number of database connections to increase a lot, so we
started hitting the Postgres connection limit.
2017-03-21 11:53:46 +01:00
Shea Levy 3b5fd2d857 hydra-eval-jobs: Disable the build hook.
For some reason this causes an unexpected EOF when doing import-from-derivation.
2017-03-20 12:57:05 -04:00
Eelco Dolstra 150228d7de
Upload build logs to the binary cache 2017-03-15 16:59:57 +01:00
Eelco Dolstra 7e6486e694
Move log compression to a plugin 2017-03-15 16:59:57 +01:00
Eelco Dolstra 285754aff6
hydra-evaluator improvements
* The "Jobset" page now shows when evaluations are in progress (rather
  than just pending).

* Restored the ability to do a single evaluation from the command line
  by doing "hydra-evaluator <project> <jobset>".

* Fix some consistency issues between jobset status in PostgreSQL and
  in hydra-evaluator. In particular, "lastCheckedTime" was never
  updated internally.
2017-03-15 16:59:57 +01:00
Eelco Dolstra 242ea8009f
hydra-evaluator: Don't use the "timeout" command
"timeout" does not propagate PR_SET_PDEATHSIG, so it prevents
hydra-eval-jobset from being killed when hydra-evaluator quits.
2017-03-15 16:59:56 +01:00
Eelco Dolstra d1afb42f12
Supress debug message 2017-03-15 16:59:56 +01:00
Eelco Dolstra 73900e9f5f Fix std::stoi exception 2017-03-08 15:07:52 +01:00
Eelco Dolstra edebdf33f0
hydra-queue-runner: Handle SIGINT 2017-03-03 12:41:00 +01:00
Eelco Dolstra 500c27e4d5
Add hydra.conf option "nar_buffer_size" to configure memoryTokens limit
It defaults to half the physical RAM.
2017-03-03 12:37:27 +01:00
Rob Vermaas fd754d678e
Do not trigger eval on jobset change when check interval is 0 (disabled). 2017-02-23 09:15:26 +00:00
Eelco Dolstra 9f6afb3375
Lock paths in the scm cache
This is necessary now that hydra-evaluator runs multiple evaluations
in parallel, to prevent corruption of Git/Mercurial clones.
2017-02-21 18:12:52 +01:00
Eelco Dolstra 52753e49c2
hydra-evaluator: Handle SIGINT
This makes "systemctl stop hydra-evaluator" work promptly again.
2017-02-21 17:54:31 +01:00
Eelco Dolstra 81ac547d2b
Move most of AddBuilds to hydra-eval-jobset
Having this stuff in a separate module is a remnant of the time when
hydra-server could add builds to the queue directly. This is no longer
the case.
2017-02-21 17:20:48 +01:00