ma27/hydra
0
0
Fork 0
forked from lix-project/hydra
Code Pull requests Activity
3191 commits 4 branches 0 tags 25 MiB
Commit graph

3191 commits

This branch
This branch
All branches
Author SHA1 Message Date
Eelco Dolstra 3cc1deb125
Merge pull request #721 from grahamc/one-by-one 
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
 2020-03-04 08:44:43 +01:00
Eelco Dolstra e6e4aa54c0
Merge pull request #722 from grahamc/fix-declarative-jobsets 
Fix declarative jobsets due to `nix` command being hidden
 2020-03-04 08:41:13 +01:00
Graham Christensen 994430b94b
treewide: allow nix command 2020-03-03 22:52:20 -05:00
Graham Christensen 117b9ecef1
Nix.pm: readNixFile: pass «--experimental-features nix-command» 
Declarative jobsets were broken by the Nix update, causing
nix cat-file to break silently.

This commit restores declarative jobsets, based on top of a commit
making it easier to see what broke.
 2020-03-03 22:36:21 -05:00
Graham Christensen 113a312f67
handleDeclarativeJobsetBuild: handle errors from readNixFile 2020-03-03 22:32:13 -05:00
Graham Christensen 5fae9d96a2
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style 
In the past, jobsets which are automatically evaluated are evaluated
regularly, on a schedule. This schedule means a new evaluation is
created every checkInterval seconds (assuming something changed.)

This model works well for architectures where our build farm can
easily keep up with demand.

This commit adds a new type of evaluation, called ONE_AT_A_TIME, which
only schedules a new evaluation if the previous evaluation of the
jobset has no unfinished builds.

This model of evaluation lets us have 'low-tier' architectures.

For example, we could now have a jobset for ARMv7l builds, where
the buildfarm only has a single, underpowered ARMv7l builder.
Configuring that jobset as ONE_AT_A_TIME will create an evaluation
and then won't schedule another evaluation until every job of
the existing evaluation is complete.

This way, the cache will have a complete collection of pre-built
software for some commits, but the underpowered architecture will
never become backlogged in ancient revisions.
 2020-03-03 19:28:44 -05:00
Graham Christensen eaa65f51f4
hydra-evaluator: make the logic of the scheduler easier to read 2020-03-03 18:17:21 -05:00
Eelco Dolstra eb5873ae53
Fix build 2020-02-20 11:19:45 +01:00
Eelco Dolstra c642f787ee
hydra-eval-jobs: Parallelize 
(cherry picked from commit be8eb9d00d)
 2020-02-20 10:28:58 +01:00
Eelco Dolstra adf61e5cf8
Fix build 
(cherry picked from commit 639c660abf)
 2020-02-20 10:26:45 +01:00
Eelco Dolstra 5308e514ad
Fix nlohmann_json dependency 2020-02-20 10:26:18 +01:00
Eelco Dolstra e4f5156c41
Build against nix-master 
(cherry picked from commit e7f2139e25)
 2020-02-20 10:24:04 +01:00
Eelco Dolstra b790a00729
Disable deprecation warnings 
(cherry picked from commit 950e8bef6c)
 2020-02-20 10:22:21 +01:00
Eelco Dolstra 2a50daa377
Update aggregate handling 
(cherry picked from commit cf961ac893)
 2020-02-20 10:13:39 +01:00
Eelco Dolstra 15187b059b
Remove hydra-eval-guile-jobs 
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.

(cherry picked from commit 23c9ca3e94)
 2020-02-20 09:58:12 +01:00
Eelco Dolstra be8eb9d00d
hydra-eval-jobs: Parallelize 2020-02-19 21:10:22 +01:00
Eelco Dolstra c53cfb5234
flake.lock: Update 
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/6529490cc10018d5191e50c482ac1180b96b1a3c' -> 'github:NixOS/nix/4ad5826a182f8eeb4cbc61e27d2128b08050b0d4'
 2020-02-19 20:37:14 +01:00
Eelco Dolstra 6f1d68bda4
Revert "hydra-eval-jobs -> nix eval-hydra-jobs" 
This reverts commit 345512a6d0.
 2020-02-19 20:36:52 +01:00
Eelco Dolstra cf961ac893
Update aggregate handling 2020-02-17 16:33:25 +01:00
Eelco Dolstra 4824b0fd9b
flake.lock: Update 
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/144bb3ef7ddca6502ea3e1878f7fc0a3f9013aba' -> 'github:NixOS/nix/6529490cc10018d5191e50c482ac1180b96b1a3c'
 2020-02-17 16:23:39 +01:00
Eelco Dolstra 4b9cdc4933 Remove boehmgc dependency 
This is now propagated by Nix.
 2020-02-15 21:56:35 +01:00
Eelco Dolstra b0a5365602 flake.lock: Update 
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/eb19ff3b82240326fc0e999e09f81b6c8ed98640' -> 'github:NixOS/nix/144bb3ef7ddca6502ea3e1878f7fc0a3f9013aba'
 2020-02-15 21:55:33 +01:00
Eelco Dolstra 68bdf834ee
Fix hydra-evaluator --unlock 2020-02-15 16:40:16 +01:00
Eelco Dolstra 345512a6d0
hydra-eval-jobs -> nix eval-hydra-jobs 2020-02-15 15:59:34 +01:00
Eelco Dolstra 23c9ca3e94
Remove hydra-eval-guile-jobs 
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.
 2020-02-15 15:59:34 +01:00
Eelco Dolstra 639c660abf
Fix build 2020-02-15 14:54:35 +01:00
Eelco Dolstra 93ed7f81cb flake.lock: Update 
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/d2032edb2f86e955a8a7724a27c0c3225f386500' -> 'github:NixOS/nix/eb19ff3b82240326fc0e999e09f81b6c8ed98640'
 2020-02-14 23:26:56 +01:00
Graham Christensen ae1b9face3
Merge remote-tracking branch 'origin/master' into flake 2020-02-11 12:52:46 -05:00
Graham Christensen 027668f0db
hydra.sql: add an index for slow queries in production 
These queries used to use (jobset, project) based indexes,
and the addition of jobset_id makes most of those indexes
unusable now.
 2020-02-11 12:52:28 -05:00
Graham Christensen 53896ff19b
Merge remote-tracking branch 'origin/master' into flake 2020-02-11 08:36:40 -05:00
Graham Christensen 834793468f
fixup: d'oh, make the migrations from #710 part-2 sequential 2020-02-11 08:36:14 -05:00
Eelco Dolstra 881b7449fd Merge remote-tracking branch 'origin/master' into flake 2020-02-11 14:23:16 +01:00
Eelco Dolstra 0dbf0a361e
Merge pull request #714 from grahamc/jobset-id-pgsql-part-2 
Jobset id pgsql part 2
 2020-02-11 13:51:17 +01:00
Graham Christensen 2637a7ad76
Builds: index literally what latest-finished queries 2020-02-11 07:06:21 -05:00
Graham Christensen f0f41eaaff
LatestSucceededForJob{,set}: Filter with jobset_id 2020-02-11 07:06:20 -05:00
Graham Christensen 66fbbd9692
Jobs.builds: Fetch via Jobs.jobset_id 2020-02-11 07:06:20 -05:00
Graham Christensen 7c71f9df28
Jobsets.builds: Fetch via Jobsets.id 2020-02-11 07:06:20 -05:00
Graham Christensen 3c392b8cd8
Jobsets.jobs: Fetch via Jobsets.id 2020-02-11 07:06:20 -05:00
Graham Christensen 8ef08f1385
Builds.jobset_id: make not-null 2020-02-11 07:06:20 -05:00
Graham Christensen 2cdcc7f188
Jobs.jobset_id: make not-null 2020-02-11 07:06:17 -05:00
Eelco Dolstra 100e09a5b3
Merge remote-tracking branch 'origin/master' into flake 
Also update flake.lock
 2020-02-10 17:58:10 +01:00
Graham Christensen add4f610ce
Merge pull request #711 from grahamc/jobset-id-pgsql-part-1 
jobset_id, #710 Part 1
 2020-02-10 11:43:38 -05:00
Graham Christensen c4cc72f944
hydra-init: Warn about the schema version migration 2020-02-10 11:43:03 -05:00
Graham Christensen f69260118b
hydra-backfill-ids: create to add jobset_id values to Builds and Jobs 
Vacuum every 10 iterations, update 10k at a time.
 2020-02-10 11:43:03 -05:00
Graham Christensen f3a561aecd
Builds: populate Builds.jobset_id in hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Graham Christensen ddf00fa627
Builds: add a nullable jobset_id foreign key to Jobsets. 
Also, adds an explicitly named "builds" accessor to the Jobsets
Schema object, which uses the project/jobset name.
 2020-02-10 11:43:02 -05:00
Graham Christensen 624f1d8d2d
Jobs: populate Jobs.jobset_id field when writing from hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Graham Christensen efa1f1d4fb
Jobs: add a nullable jobset_id foreign key to Jobsets. 
Also, adds an explicitly named "jobs" accessor to the Jobsets
Schema object, which uses the project/jobset name.
 2020-02-10 11:43:02 -05:00
Graham Christensen e00030563b
Jobsets: add a SERIAL, unique, non-null id column 
A postgresql column which is non-null and unique is treated with
the same optimisations as a primary key, so we have no need to
try and recreate the `id` as the primary key.

No read paths are impacted by this change, and the database will
automatically create an ID for each insert. Thus, no code needs to
change.
 2020-02-10 11:42:59 -05:00
Eelco Dolstra 4a05bb36ac
Merge pull request #713 from grahamc/backport-flake-migration 
Copy the flake migration from the flake branch
 2020-02-10 17:41:46 +01:00