Commit graph

2724 commits

Author SHA1 Message Date
Eelco Dolstra 53e1294825 Fix Mercurial tests
In Nixpkgs 20.03, Mercurial fails if PYTHONPATH is set:

  $ hg
  Traceback (most recent call last):
    File "/nix/store/q7s856v6nw4dffdrm9k3w38qs35i8kr3-mercurial-5.2.2/bin/..hg-wrapped-wrapped", line 37, in <module>
      dispatch.run()
    File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/importlib/util.py", line 245, in __getattribute__
      self.__spec__.loader.exec_module(self)
    File "<frozen importlib._bootstrap_external>", line 728, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/nix/store/q7s856v6nw4dffdrm9k3w38qs35i8kr3-mercurial-5.2.2/lib/python3.7/site-packages/mercurial/dispatch.py", line 10, in <module>
      import difflib
    File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/difflib.py", line 1084, in <module>
      import re
    File "/nix/store/bffdy7q3wi3qinflnvbdkigqj39zzynd-python3-3.7.6/lib/python3.7/re.py", line 143, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'

(cherry picked from commit 4009d4295e)
2020-03-31 22:19:19 +02:00
Eelco Dolstra 9727892b61 Don't spam the journal with hydra-queue-runner status dumps
(cherry picked from commit 15ae932488)
2020-03-31 22:19:07 +02:00
Eelco Dolstra ccd046ca3d Keep track of the number of unsupported steps
(cherry picked from commit 45ffe578b6)
2020-03-31 22:19:03 +02:00
Eelco Dolstra 4417f9f260 Abort unsupported build steps
If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.

(cherry picked from commit f5cdbfe21d)
2020-03-31 22:19:01 +02:00
Eelco Dolstra 68a59f34a0 Remove tabs
(cherry picked from commit 5b731004da)
2020-03-31 22:18:58 +02:00
Eelco Dolstra 8b5900ac33 Remove outdated email address
(cherry picked from commit 22a65d0269)
2020-03-31 22:18:46 +02:00
Eelco Dolstra 082dbdb922
Merge pull request #729 from samueldr/fix/error-transmission
hydra-eval-jobs: Mirror eval errors in STDERR
2020-03-31 09:04:32 +02:00
Samuel Dionne-Riel 76299b9174 hydra-eval-jobs: Mirror eval errors in STDERR
Otherwise, errors will not be shown to end-users, which makes debugging
long evals pretty much impossible.
2020-03-30 16:11:44 -04:00
Graham Christensen 12cc46cdb3
fixup: hydra-init: correct reference to hydra-backill-ids 2020-03-24 11:22:14 -04:00
Eelco Dolstra c85ee736a4
Merge pull request #724 from Ma27/different-icons-for-timeout-and-loglimit
Use the stopsign as icon for timeouts and exceeded log-limits
2020-03-23 09:39:04 +01:00
Maximilian Bosch 73694087a0
Use the stopsign as icon for timeouts and exceeded log-limits
When I browse failed builds in a jobset-eval on Hydra, I regularly
mistake actual build-failures with temporary issues like timeouts (that
probably disappear at the next eval).

To prevent this kind of issue, I figured that using the stopsign-svg for
builds with timeouts or exceeded log-limits is a reasonable choice for
the following reasons:

* A user can now distinguish between actual build-errors (like
  compilation-failures or oversized outputs) and (usually) temporary issues
  (like a bloated log or a timeout).

* The stopsign is also used for aborted jobs that are shown in a
  different tab and can't be confused with timeouts for that reason.
2020-03-21 21:08:33 +01:00
Eelco Dolstra 5ae8b54d25 README.md: Link to the latest manual
(cherry picked from commit 455611b0b4e97290a858687142805babc9929673)
2020-03-13 21:33:22 +01:00
Eelco Dolstra be0ec2d223
Remove style.css
https://github.com/NixOS/nix/issues/3392
2020-03-13 16:17:48 +01:00
Eelco Dolstra 123bee1db5
Restore job type checking 2020-03-04 15:16:26 +01:00
Eelco Dolstra 69a6f3448a
Fix calling job functions
Fixes #718.
2020-03-04 15:16:04 +01:00
Eelco Dolstra 3cc1deb125
Merge pull request #721 from grahamc/one-by-one
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
2020-03-04 08:44:43 +01:00
Eelco Dolstra e6e4aa54c0
Merge pull request #722 from grahamc/fix-declarative-jobsets
Fix declarative jobsets due to `nix` command being hidden
2020-03-04 08:41:13 +01:00
Graham Christensen 994430b94b
treewide: allow nix command 2020-03-03 22:52:20 -05:00
Graham Christensen 117b9ecef1
Nix.pm: readNixFile: pass «--experimental-features nix-command»
Declarative jobsets were broken by the Nix update, causing
nix cat-file to break silently.

This commit restores declarative jobsets, based on top of a commit
making it easier to see what broke.
2020-03-03 22:36:21 -05:00
Graham Christensen 113a312f67
handleDeclarativeJobsetBuild: handle errors from readNixFile 2020-03-03 22:32:13 -05:00
Graham Christensen 5fae9d96a2
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
In the past, jobsets which are automatically evaluated are evaluated
regularly, on a schedule. This schedule means a new evaluation is
created every checkInterval seconds (assuming something changed.)

This model works well for architectures where our build farm can
easily keep up with demand.

This commit adds a new type of evaluation, called ONE_AT_A_TIME, which
only schedules a new evaluation if the previous evaluation of the
jobset has no unfinished builds.

This model of evaluation lets us have 'low-tier' architectures.

For example, we could now have a jobset for ARMv7l builds, where
the buildfarm only has a single, underpowered ARMv7l builder.
Configuring that jobset as ONE_AT_A_TIME will create an evaluation
and then won't schedule another evaluation until every job of
the existing evaluation is complete.

This way, the cache will have a complete collection of pre-built
software for some commits, but the underpowered architecture will
never become backlogged in ancient revisions.
2020-03-03 19:28:44 -05:00
Graham Christensen eaa65f51f4
hydra-evaluator: make the logic of the scheduler easier to read 2020-03-03 18:17:21 -05:00
Eelco Dolstra eb5873ae53
Fix build 2020-02-20 11:19:45 +01:00
Eelco Dolstra c642f787ee
hydra-eval-jobs: Parallelize
(cherry picked from commit be8eb9d00d)
2020-02-20 10:28:58 +01:00
Eelco Dolstra adf61e5cf8
Fix build
(cherry picked from commit 639c660abf)
2020-02-20 10:26:45 +01:00
Eelco Dolstra 5308e514ad
Fix nlohmann_json dependency 2020-02-20 10:26:18 +01:00
Eelco Dolstra e4f5156c41
Build against nix-master
(cherry picked from commit e7f2139e25)
2020-02-20 10:24:04 +01:00
Eelco Dolstra b790a00729
Disable deprecation warnings
(cherry picked from commit 950e8bef6c)
2020-02-20 10:22:21 +01:00
Eelco Dolstra 2a50daa377
Update aggregate handling
(cherry picked from commit cf961ac893)
2020-02-20 10:13:39 +01:00
Eelco Dolstra 15187b059b
Remove hydra-eval-guile-jobs
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.

(cherry picked from commit 23c9ca3e94)
2020-02-20 09:58:12 +01:00
Graham Christensen 027668f0db
hydra.sql: add an index for slow queries in production
These queries used to use (jobset, project) based indexes,
and the addition of jobset_id makes most of those indexes
unusable now.
2020-02-11 12:52:28 -05:00
Graham Christensen 834793468f
fixup: d'oh, make the migrations from #710 part-2 sequential 2020-02-11 08:36:14 -05:00
Eelco Dolstra 0dbf0a361e
Merge pull request #714 from grahamc/jobset-id-pgsql-part-2
Jobset id pgsql part 2
2020-02-11 13:51:17 +01:00
Graham Christensen 2637a7ad76
Builds: index literally what latest-finished queries 2020-02-11 07:06:21 -05:00
Graham Christensen f0f41eaaff
LatestSucceededForJob{,set}: Filter with jobset_id 2020-02-11 07:06:20 -05:00
Graham Christensen 66fbbd9692
Jobs.builds: Fetch via Jobs.jobset_id 2020-02-11 07:06:20 -05:00
Graham Christensen 7c71f9df28
Jobsets.builds: Fetch via Jobsets.id 2020-02-11 07:06:20 -05:00
Graham Christensen 3c392b8cd8
Jobsets.jobs: Fetch via Jobsets.id 2020-02-11 07:06:20 -05:00
Graham Christensen 8ef08f1385
Builds.jobset_id: make not-null 2020-02-11 07:06:20 -05:00
Graham Christensen 2cdcc7f188
Jobs.jobset_id: make not-null 2020-02-11 07:06:17 -05:00
Graham Christensen add4f610ce
Merge pull request #711 from grahamc/jobset-id-pgsql-part-1
jobset_id, #710 Part 1
2020-02-10 11:43:38 -05:00
Graham Christensen c4cc72f944
hydra-init: Warn about the schema version migration 2020-02-10 11:43:03 -05:00
Graham Christensen f69260118b
hydra-backfill-ids: create to add jobset_id values to Builds and Jobs
Vacuum every 10 iterations, update 10k at a time.
2020-02-10 11:43:03 -05:00
Graham Christensen f3a561aecd
Builds: populate Builds.jobset_id in hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Graham Christensen ddf00fa627
Builds: add a nullable jobset_id foreign key to Jobsets.
Also, adds an explicitly named "builds" accessor to the Jobsets
Schema object, which uses the project/jobset name.
2020-02-10 11:43:02 -05:00
Graham Christensen 624f1d8d2d
Jobs: populate Jobs.jobset_id field when writing from hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Graham Christensen efa1f1d4fb
Jobs: add a nullable jobset_id foreign key to Jobsets.
Also, adds an explicitly named "jobs" accessor to the Jobsets
Schema object, which uses the project/jobset name.
2020-02-10 11:43:02 -05:00
Graham Christensen e00030563b
Jobsets: add a SERIAL, unique, non-null id column
A postgresql column which is non-null and unique is treated with
the same optimisations as a primary key, so we have no need to
try and recreate the `id` as the primary key.

No read paths are impacted by this change, and the database will
automatically create an ID for each insert. Thus, no code needs to
change.
2020-02-10 11:42:59 -05:00
Eelco Dolstra 4a05bb36ac
Merge pull request #713 from grahamc/backport-flake-migration
Copy the flake migration from the flake branch
2020-02-10 17:41:46 +01:00
Graham Christensen 6fe57ab5fa
Copy the flake migration from the flake branch
hydra.nixos.org is already running this rev, and it should be safe to
apply to everyone else. If we make changes to this migration, we'll
need to write another migration anyway.
2020-02-09 15:21:28 -05:00