Graham Christensen
113a312f67
handleDeclarativeJobsetBuild: handle errors from readNixFile
2020-03-03 22:32:13 -05:00
Graham Christensen
5fae9d96a2
hydra-evaluator: add a 'ONE_AT_A_TIME' evaluator style
...
In the past, jobsets which are automatically evaluated are evaluated
regularly, on a schedule. This schedule means a new evaluation is
created every checkInterval seconds (assuming something changed.)
This model works well for architectures where our build farm can
easily keep up with demand.
This commit adds a new type of evaluation, called ONE_AT_A_TIME, which
only schedules a new evaluation if the previous evaluation of the
jobset has no unfinished builds.
This model of evaluation lets us have 'low-tier' architectures.
For example, we could now have a jobset for ARMv7l builds, where
the buildfarm only has a single, underpowered ARMv7l builder.
Configuring that jobset as ONE_AT_A_TIME will create an evaluation
and then won't schedule another evaluation until every job of
the existing evaluation is complete.
This way, the cache will have a complete collection of pre-built
software for some commits, but the underpowered architecture will
never become backlogged in ancient revisions.
2020-03-03 19:28:44 -05:00
Graham Christensen
eaa65f51f4
hydra-evaluator: make the logic of the scheduler easier to read
2020-03-03 18:17:21 -05:00
Eelco Dolstra
eb5873ae53
Fix build
2020-02-20 11:19:45 +01:00
Eelco Dolstra
c642f787ee
hydra-eval-jobs: Parallelize
...
(cherry picked from commit be8eb9d00d
)
2020-02-20 10:28:58 +01:00
Eelco Dolstra
adf61e5cf8
Fix build
...
(cherry picked from commit 639c660abf
)
2020-02-20 10:26:45 +01:00
Eelco Dolstra
5308e514ad
Fix nlohmann_json dependency
2020-02-20 10:26:18 +01:00
Eelco Dolstra
e4f5156c41
Build against nix-master
...
(cherry picked from commit e7f2139e25
)
2020-02-20 10:24:04 +01:00
Eelco Dolstra
b790a00729
Disable deprecation warnings
...
(cherry picked from commit 950e8bef6c
)
2020-02-20 10:22:21 +01:00
Eelco Dolstra
2a50daa377
Update aggregate handling
...
(cherry picked from commit cf961ac893
)
2020-02-20 10:13:39 +01:00
Eelco Dolstra
15187b059b
Remove hydra-eval-guile-jobs
...
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.
(cherry picked from commit 23c9ca3e94
)
2020-02-20 09:58:12 +01:00
Eelco Dolstra
be8eb9d00d
hydra-eval-jobs: Parallelize
2020-02-19 21:10:22 +01:00
Eelco Dolstra
c53cfb5234
flake.lock: Update
...
Flake input changes:
* Updated 'nix': 'github:NixOS/nix/6529490cc10018d5191e50c482ac1180b96b1a3c' -> 'github:NixOS/nix/4ad5826a182f8eeb4cbc61e27d2128b08050b0d4'
2020-02-19 20:37:14 +01:00
Eelco Dolstra
6f1d68bda4
Revert "hydra-eval-jobs -> nix eval-hydra-jobs"
...
This reverts commit 345512a6d0
.
2020-02-19 20:36:52 +01:00
Eelco Dolstra
cf961ac893
Update aggregate handling
2020-02-17 16:33:25 +01:00
Eelco Dolstra
4824b0fd9b
flake.lock: Update
...
Flake input changes:
* Updated 'nix': 'github:NixOS/nix/144bb3ef7ddca6502ea3e1878f7fc0a3f9013aba' -> 'github:NixOS/nix/6529490cc10018d5191e50c482ac1180b96b1a3c'
2020-02-17 16:23:39 +01:00
Eelco Dolstra
4b9cdc4933
Remove boehmgc dependency
...
This is now propagated by Nix.
2020-02-15 21:56:35 +01:00
Eelco Dolstra
b0a5365602
flake.lock: Update
...
Flake input changes:
* Updated 'nix': 'github:NixOS/nix/eb19ff3b82240326fc0e999e09f81b6c8ed98640' -> 'github:NixOS/nix/144bb3ef7ddca6502ea3e1878f7fc0a3f9013aba'
2020-02-15 21:55:33 +01:00
Eelco Dolstra
68bdf834ee
Fix hydra-evaluator --unlock
2020-02-15 16:40:16 +01:00
Eelco Dolstra
345512a6d0
hydra-eval-jobs -> nix eval-hydra-jobs
2020-02-15 15:59:34 +01:00
Eelco Dolstra
23c9ca3e94
Remove hydra-eval-guile-jobs
...
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.
2020-02-15 15:59:34 +01:00
Eelco Dolstra
639c660abf
Fix build
2020-02-15 14:54:35 +01:00
Eelco Dolstra
93ed7f81cb
flake.lock: Update
...
Flake input changes:
* Updated 'nix': 'github:NixOS/nix/d2032edb2f86e955a8a7724a27c0c3225f386500' -> 'github:NixOS/nix/eb19ff3b82240326fc0e999e09f81b6c8ed98640'
2020-02-14 23:26:56 +01:00
Graham Christensen
ae1b9face3
Merge remote-tracking branch 'origin/master' into flake
2020-02-11 12:52:46 -05:00
Graham Christensen
027668f0db
hydra.sql: add an index for slow queries in production
...
These queries used to use (jobset, project) based indexes,
and the addition of jobset_id makes most of those indexes
unusable now.
2020-02-11 12:52:28 -05:00
Graham Christensen
53896ff19b
Merge remote-tracking branch 'origin/master' into flake
2020-02-11 08:36:40 -05:00
Graham Christensen
834793468f
fixup: d'oh, make the migrations from #710 part-2 sequential
2020-02-11 08:36:14 -05:00
Eelco Dolstra
881b7449fd
Merge remote-tracking branch 'origin/master' into flake
2020-02-11 14:23:16 +01:00
Eelco Dolstra
0dbf0a361e
Merge pull request #714 from grahamc/jobset-id-pgsql-part-2
...
Jobset id pgsql part 2
2020-02-11 13:51:17 +01:00
Graham Christensen
2637a7ad76
Builds: index literally what latest-finished queries
2020-02-11 07:06:21 -05:00
Graham Christensen
f0f41eaaff
LatestSucceededForJob{,set}: Filter with jobset_id
2020-02-11 07:06:20 -05:00
Graham Christensen
66fbbd9692
Jobs.builds: Fetch via Jobs.jobset_id
2020-02-11 07:06:20 -05:00
Graham Christensen
7c71f9df28
Jobsets.builds: Fetch via Jobsets.id
2020-02-11 07:06:20 -05:00
Graham Christensen
3c392b8cd8
Jobsets.jobs: Fetch via Jobsets.id
2020-02-11 07:06:20 -05:00
Graham Christensen
8ef08f1385
Builds.jobset_id: make not-null
2020-02-11 07:06:20 -05:00
Graham Christensen
2cdcc7f188
Jobs.jobset_id: make not-null
2020-02-11 07:06:17 -05:00
Eelco Dolstra
100e09a5b3
Merge remote-tracking branch 'origin/master' into flake
...
Also update flake.lock
2020-02-10 17:58:10 +01:00
Graham Christensen
add4f610ce
Merge pull request #711 from grahamc/jobset-id-pgsql-part-1
...
jobset_id, #710 Part 1
2020-02-10 11:43:38 -05:00
Graham Christensen
c4cc72f944
hydra-init: Warn about the schema version migration
2020-02-10 11:43:03 -05:00
Graham Christensen
f69260118b
hydra-backfill-ids: create to add jobset_id values to Builds and Jobs
...
Vacuum every 10 iterations, update 10k at a time.
2020-02-10 11:43:03 -05:00
Graham Christensen
f3a561aecd
Builds: populate Builds.jobset_id in hydra-eval-jobset
2020-02-10 11:43:02 -05:00
Graham Christensen
ddf00fa627
Builds: add a nullable jobset_id foreign key to Jobsets.
...
Also, adds an explicitly named "builds" accessor to the Jobsets
Schema object, which uses the project/jobset name.
2020-02-10 11:43:02 -05:00
Graham Christensen
624f1d8d2d
Jobs: populate Jobs.jobset_id field when writing from hydra-eval-jobset
2020-02-10 11:43:02 -05:00
Graham Christensen
efa1f1d4fb
Jobs: add a nullable jobset_id foreign key to Jobsets.
...
Also, adds an explicitly named "jobs" accessor to the Jobsets
Schema object, which uses the project/jobset name.
2020-02-10 11:43:02 -05:00
Graham Christensen
e00030563b
Jobsets: add a SERIAL, unique, non-null id column
...
A postgresql column which is non-null and unique is treated with
the same optimisations as a primary key, so we have no need to
try and recreate the `id` as the primary key.
No read paths are impacted by this change, and the database will
automatically create an ID for each insert. Thus, no code needs to
change.
2020-02-10 11:42:59 -05:00
Eelco Dolstra
4a05bb36ac
Merge pull request #713 from grahamc/backport-flake-migration
...
Copy the flake migration from the flake branch
2020-02-10 17:41:46 +01:00
Graham Christensen
6fe57ab5fa
Copy the flake migration from the flake branch
...
hydra.nixos.org is already running this rev, and it should be safe to
apply to everyone else. If we make changes to this migration, we'll
need to write another migration anyway.
2020-02-09 15:21:28 -05:00
Eelco Dolstra
2b4f14963b
Merge pull request #709 from grahamc/generate-from-postgresql
...
sql: Generate models from postgresql
2020-02-06 20:18:31 +01:00
Graham Christensen
c2f932a7e3
sql: Generate models from postgresql
...
Lowercasing is due to postgresql not having case-sensitive table names.
It always technically workde before, but those table names never
existed literally.
The switch to generating from postgresql is to handle an upcoming
addition of an auto-incrementign ID to the Jobset table. Sqlite doesn't
seem to be able to handle the table having an auto incrementing ID
field which isn't the primary key, but we can't change the primary
key trivially.
Since hydra doesn't support sqlite and hasn't for many year anyway,
it is easier to just generate from pgsql directly.
2020-02-06 12:23:47 -05:00
Eelco Dolstra
ba8814a245
Fix build
2020-02-03 18:43:45 +01:00