lix-project/buildbot-nix
15
0
Fork 0
Code Issues 9 Pull requests 2 Packages Projects Releases Wiki Activity
295 commits 9 branches 0 tags 1.3 MiB
Commit graph

150 commits

Author SHA1 Message Date
eldritch horrors d394f35f55 use one scheduler and worker set per arch 
and an additional set for generic tasks like error reporting. this
prevents hol blocking for underutilized arches when at least one arch is
blocking, as usually happens to us with aarch64-linux.
 2024-03-15 14:47:49 +01:00
eldritch horrors 5e50a858d7 revert to stable web ui 
the react-based ui is too slow for our needs, janky, the log viewer
doesn't work quite right (breaking after ~600 lines of logs viewed),
loses updates to sub-builds, and just blanks its entire screen when a
build finishes. the old ui doesn't do that.
 2024-03-15 14:40:23 +01:00
raito 8d36ac1d90 feat: signing key 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-12 01:27:46 +01:00
raito 6118daa0a4 feat: binary cache 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-12 01:27:46 +01:00
puck e9b3b38bbf Skip scheduling cached builds; improve reporter message 2024-03-11 15:05:15 +00:00
eldritch horrors 5cdef7efb6 fix status reporting to gerrit 
also adjust labels from split verified to single verified, split labels
were only useful during the pre-ci hours
 2024-03-11 14:44:09 +01:00
eldritch horrors 51f7b52149 pre-filter drv_info into all_deps 
otherwise failure reporting is *enormous* with the entirety of a full
derivation info dump in there
 2024-03-11 13:07:35 +01:00
eldritch horrors 13a67b483a fix interrupt() 
can't interrupt with things to interrupt. this is technically duplicated
information but keeping parts of the code close to Trigger seems useful.
 2024-03-11 13:05:12 +01:00
eldritch horrors 9933971ab0 re-enable the gerrit status reporter 2024-03-11 09:06:29 +01:00
eldritch horrors 29a2ef63e2 show hydra job count in trigger step 
previously we immediately triggered all jobs, now we no longer do.
showing the total count at least somewhere is nice to have a rough
indication of how much longer a build may still need to run.
 2024-03-11 09:05:28 +01:00
puck 9a15348984 Fix up a few loose ends 2024-03-11 08:08:55 +01:00
puck 4d73275123 Add build result tracking, schedule newly available builds 2024-03-11 08:08:53 +01:00
puck 28ca39af25 WIP: Replace Trigger with custom logic 2024-03-11 08:06:37 +01:00
eldritch horrors e9874c3d98 wip: dependency-tracked build triggering 2024-03-11 07:53:56 +01:00
eldritch horrors f869b52a8d use build-local gc-root directory 
without this two builds can interfere with each other if:

  - builds 1 and 2 start
  - build 1 is starved of workers
  - build 2 finishes, removes the shared gcroots directory
  - gc runs
  - build 1 schedules more builds whose .drvs have now been removed

using a dedicated directory for each build fixes this.

we now also need to set alwaysRun on the cleanup command or we risk
littering the system with stale gc roots when a build fails.
 2024-03-11 06:48:41 +01:00
eldritch horrors 156e6e3dea remove skipped-builds builder 
run all of them on the normal build worker. this significantly
simplifies the overall scheduler/builder config and removes a
triplication of possible builds paths.
 2024-03-11 06:27:32 +01:00
eldritch horrors 753df8e340 remove cachix 
we aren't using it and it's somewhat in the way of our efforts to
improve scheduling and stuff.
 2024-03-11 06:26:39 +01:00
eldritch horrors 0b2545b036 remove unused GitWithRetry 2024-03-11 06:26:39 +01:00
eldritch horrors fdfeef8ad4 remove retry logic 
retries don't help us very much, in fact they mostly hurt by repeating
builds that failed for non-transient reasons. retries could help with
workers dropping while running a build, but those rare cases are better
to restart manually than to pend at least twice the ci time for commits
that simply do not build cleanly.
 2024-03-11 06:26:38 +01:00
puck ec2ef903ab use .#hydraJobs rather than .#checks 2024-03-08 23:28:49 +00:00
puck c487ada514 review callback: Set labels, render list of failed checks in reporter 2024-03-07 04:40:58 +00:00
raito 18d537e5d4 nix/coordinator(reporting): re-enable reviewCB, disable summaryCB, skip evaluations 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-04 20:50:12 +01:00
raito 322944f8d1 nix/coordinator(reporting): disable startCB and reviewCB 
excessive noise.

Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-04 20:37:13 +01:00
raito 9a67a1f2cc nix/coordinator: enable reporting to Gerrit 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-03 02:58:25 +01:00
raito 2bcc0ee4e9 nix/coordinator: hardcode remaining github properties to lix 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 23:22:36 +01:00
raito 3895b90493 nix/coordinator: disable reporter 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 23:07:36 +01:00
raito 670a1d1281 nix/coordinator: fix repourl 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 23:07:36 +01:00
raito 94e3d7aeda nix/coordinator: support specific private SSH keys 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 23:07:33 +01:00
raito 5a8ab145e6 buildbot: add repourl 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 22:09:56 +01:00
raito 7ace78107c buildbot: further Gerritification 
- `BuildTrigger` still depends on `github` properties.

Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 22:09:56 +01:00
raito 5f7b3e0cdb buildbot: instantiate fully LixSystemsOAuth2 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 22:09:56 +01:00
raito acfd225e6d buildbot/__init__.py: rework the file for gerrit usecase, our oauth2 instance 
Signed-off-by: Raito Bezarius <raito@lix.systems>
 2024-03-02 18:45:19 +01:00
Jörg Thalheim 3c1e2c0e76 cleanup drv gcroots after a build 2024-01-15 13:17:19 +00:00
Jörg Thalheim c9e850ff32 only refresh github projects in timer and sort them 
This way errors are more visible to users on the first running.

Co-authored-by: zimbatm <zimbatm@zimbatm.com>
 2024-01-08 11:45:01 +00:00
Jörg Thalheim f69a37a69e relax concurrent builds 
So far we cannot have builds running concurrently, whereas we only would
need to be strict about nix evaluation to stay below memory limits.
 2024-01-05 08:49:54 +01:00
Jörg Thalheim b2467c4ac5 typecheck twisted 2024-01-01 15:38:28 +00:00
Jörg Thalheim 6882c81e90 fixup logger warning call 2024-01-01 08:05:12 +01:00
Jörg Thalheim acdab58765 worker: fix basedir 2023-12-27 10:25:28 +00:00
Jörg Thalheim 645c7fd0cc fix workername formatting 2023-12-27 10:25:28 +00:00
Jörg Thalheim 8b9c060dbe fix new linting errors 2023-12-26 22:36:40 +01:00
Jörg Thalheim 35079f89e7 apply ruff 2023-12-26 19:50:34 +01:00
Jörg Thalheim f56e43267d expose cachix options explictly 2023-12-24 07:10:01 +00:00
Jörg Thalheim 232a617fe1 let nix handle the silent timeout of builds 2023-12-22 14:40:31 +01:00
Jörg Thalheim a13cd78a1c drop prometheus exporter again 
User can add this themself
 2023-12-08 11:11:48 +00:00
Jörg Thalheim 48ea2f7db4 GithubAuth: increase apiVersion to 4 2023-12-08 11:11:48 +00:00
Jörg Thalheim cbe4ce0571 move extraConfig into configurator 
extraConfig can be only set once. By moving it to the configurator we allow users to set this option themself.
 2023-12-08 11:11:48 +00:00
Jörg Thalheim dd6eacc4c4 allow all members in an org to restart/cancel/trigger builds 2023-12-08 11:11:48 +00:00
Jörg Thalheim ed8a6c0333 git: fix retry message 2023-12-04 10:20:27 +01:00
Jörg Thalheim 590f31eb6f retry git clonning errors 2023-12-03 15:10:16 +00:00
Jörg Thalheim b33d7e5a3e drop nix_update_flake_config pipeline 
This is better offloaded to tools like dependabot/renovatebot.
I will probably work on integrate the latter one.
 2023-12-03 15:10:16 +00:00
Jörg Thalheim 49f97876d5 fix status_name and virtual_builder_name 2023-11-20 09:07:45 +01:00
Jörg Thalheim 9d80f5f2c7 move skipped builds to local worker 2023-11-18 10:13:16 +01:00
Jörg Thalheim 392329ea71 fix conditionals when skipping cached builds 2023-11-18 07:02:34 +01:00
Jörg Thalheim 37372c90da skip cached builds 2023-11-17 18:04:27 +01:00
Jörg Thalheim 5b4ddd014c make evalWorkerCount configurable 2023-11-12 07:23:35 +01:00
Jörg Thalheim c477a14d1a fix github avatar authentication 2023-11-12 07:23:23 +01:00
zowoq 8dbd6a2101 prefix scheduler names with project.id 2023-11-11 15:10:13 +00:00
Jörg Thalheim 1719bf1810 make eval lock exclusive 
looks like maxCount doesn't quite work
 2023-11-11 15:10:13 +00:00
Jörg Thalheim 4d71870239
Merge pull request #27 from Mic92/fixes 
Also pass GITHUB_TOKEN to AvatarGithub
 2023-11-10 17:24:47 +01:00
Jörg Thalheim 524c8fd150 buildbot-nix: limit evaluation to one job per worker 2023-11-10 14:30:09 +01:00
zowoq 2a565a8d5a pad worker numbering 2023-11-10 13:49:24 +01:00
Jörg Thalheim abdff1d927 also pass GITHUB_TOKEN to AvatarGithub 2023-11-09 08:52:46 +01:00
Jörg Thalheim f7573d02bc make sure update-flake has a github project field set 2023-11-09 08:52:46 +01:00
Jörg Thalheim 30b8d7b090 Configurator: make sure all config values have defaults set before hand 2023-11-09 08:52:46 +01:00
Florian Klink 5b3f986a29 buildbot_nix: update comment 
This doesn't invoke nix-instantiate anymore, but
`nix-eval-jobs --flake .#checks`.
 2023-11-08 15:31:14 +02:00
Jörg Thalheim 0e52cdf81f buildbot: set treeStableTimer 
Otherwise we build every commit
 2023-11-04 12:14:53 +01:00
Jörg Thalheim ecaafe61b4 make nginx output paths optional 2023-11-04 11:21:36 +01:00
zowoq 3551cb6bef remove extra slash from webhook url 2023-11-04 13:47:55 +10:00
Jörg Thalheim 25fe8ed9ae fix jobs filter 2023-10-27 11:45:54 +02:00
Jörg Thalheim 9f2ec133e1 use rstrip to remove trailing newlines 2023-10-27 09:11:40 +00:00
Paul Haerle 9ad17c646b fmt 2023-10-27 09:11:40 +00:00
Paul Haerle c7b360eb16 pass project name for forced builds 2023-10-27 09:11:39 +00:00
Paul Haerle 2edf6860b0 allow github projects with "." in their name 2023-10-27 09:11:39 +00:00
Paul Haerle eac9acc041 only activate github auth if no other is active 2023-10-27 09:11:39 +00:00
Paul Haerle 44486f6633 skip github projects without sufficient perms 2023-10-27 09:11:39 +00:00
Paul Haerle a758cb41ca fix: also set repos if not cached 2023-10-27 09:11:38 +00:00
Paul Haerle f14619f0df strip read secrets 2023-10-27 09:11:38 +00:00
Paul Haerle 4ae3ef7ca7 make buildbot_nix a python module 2023-10-27 09:11:38 +00:00
Jörg Thalheim b042eb8011 apply nix fmt 2023-10-16 18:41:52 +01:00
Jörg Thalheim 14556fd91f disable periodic flake updates for now 2023-10-15 11:31:07 +02:00
Jörg Thalheim a553fdfa34 use nix-eval-jobs from nixpkgs 2023-10-15 11:31:07 +02:00
Jörg Thalheim 2409fc9c91 reload github projects every 12 hours 2023-10-15 11:15:54 +02:00
Jörg Thalheim 776a6ea499 update flakes with a random delay 2023-10-15 08:36:33 +02:00
Jörg Thalheim b5681b1ea0 remove incorrect skip if path exists 
the path might exists also the build failed
 2023-10-13 00:38:57 +02:00
Jörg Thalheim e82d3a3d91 add option to reload github projects 
fix mypy
 2023-10-12 23:58:59 +02:00
Jörg Thalheim 74fb30f6ff also install webhooks secrets 2023-10-12 21:05:37 +02:00
Jörg Thalheim 54bcb08fae also create webhooks automatically 2023-10-12 17:50:43 +02:00
Jörg Thalheim cb35c312c1 make topic filter configurable 2023-10-12 15:54:28 +02:00
Jörg Thalheim dd95055cb6 make per-project gcroots 2023-10-12 15:54:28 +02:00
Jörg Thalheim e90d7e151e fix doStepIf 2023-10-12 12:14:44 +02:00
Jörg Thalheim 08feafa5d4 drop automerge feature 2023-10-09 16:09:19 +02:00
Jörg Thalheim 94c75f9631 switch to using configurators option from nixpkgs fork again 2023-10-09 15:51:05 +02:00
Jörg Thalheim 2488ef4424 switch to configurators class 2023-09-18 17:01:07 +02:00
Jörg Thalheim 5915bf4ffa ruff: also lint upgrade rules 2023-09-18 15:43:51 +02:00
Jörg Thalheim cdde629ff3 drop irc notifications 2023-09-17 06:46:19 +02:00
Jörg Thalheim 2afa25ca27 master: move DB_URL to global variable 2023-09-15 09:10:35 +02:00
Jörg Thalheim aeffb167fe worker: move global variables to config class 2023-09-15 09:09:41 +02:00
Jörg Thalheim 97f1870c15 add first working configuration 2023-09-10 11:16:33 +00:00
Jörg Thalheim dc190c7635 improve master module 2023-09-10 08:53:04 +00:00
Jörg Thalheim 6406f34ebe first commit; very broken 2023-09-10 08:11:56 +00:00