608c0e5973
hydra: bump to 16 evaluation workers, we have enough RAM and cores to afford it
2024-07-22 23:13:33 +02:00
62ccc0282b
fix(ows): per-job runtime directories + proper local refspec
...
The local refspec was weird and exploiting a edge case for the nixpkgs
jobs where local and from were the same.
We are more explicit now, which fixes the sandbox jobs.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-22 15:41:47 +02:00
d84a43b781
builders: run gc 3x per day
...
We can still adjust it if the disks fill up, but currently it is too frequent
2024-07-21 19:49:21 +02:00
2dc5899660
baremetal: run hydra store gc as builder user
2024-07-20 17:00:39 +02:00
adaf4b0aef
baremetal: tmp on the same filesystem as hydra store
2024-07-20 17:00:39 +02:00
5bde7e2358
use dedicated store partition for hydra builds
2024-07-20 15:14:00 +02:00
d9809e1e78
gerrit-one-way-sync: disallow auto-merging a staging iteration into master
2024-07-20 15:14:00 +02:00
3fa4a25d87
gerrit-one-way-sync: set git user info
2024-07-20 15:14:00 +02:00
0ff5eea4ed
gerrit-one-way-sync: merge instead of rebase
2024-07-20 15:14:00 +02:00
80c4757571
gerrit01: add a one-way-sync service
...
It's basic and does not handle conflicts which needs to be manually
managed.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-19 17:52:44 +02:00
d1e64b6610
Fix eval warning here too
2024-07-19 12:06:03 +03:00
766dc4c383
Mimir also wants network-online.target
...
Thank you helpful eval warning
2024-07-19 12:03:55 +03:00
65b07a936b
Make sure Mimir starts after network is up
2024-07-19 12:00:52 +03:00
8afcf249d6
buildbot: upgrade to local machine specifications
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-18 12:18:02 +02:00
4473717e9f
gerrit: introduce buildbot checks plugin
...
It's a modified version of @puck 's Lix buildbot checks for
gerrit.lix.systems with a slight generalization in the configuration for
many repositories.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-18 10:56:46 +02:00
da7175303c
buildbot: add support for remote builders via baremetal machines
...
For now, only builder-3 is used.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-17 18:28:26 +02:00
7789e9ce75
services/buildbot: init
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-17 18:00:51 +02:00
fda59ee6c0
gerrit: factor more configuration in the NixOS module for external consumption
...
Other modules may require information to configure themselves from the
Gerrit module.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-17 15:43:35 +02:00
95b58de737
forgejo: use redis as cache and session provider
2024-07-16 20:09:15 +02:00
8b9d33d70c
forgejo: disable registrations, enable auto-registration for SSO
2024-07-16 17:14:23 +02:00
dd069c40d7
forgejo: init service
2024-07-16 15:44:06 +02:00
e3e60a5e72
services/monitoring: add scraping of Gerrit's internal metrics
2024-07-15 11:02:54 +00:00
2e86babc8a
services/gerrit: add metrics-prometheus-exporter
2024-07-15 11:02:54 +00:00
7a937e837a
Unlimit Mimir max series
2024-07-13 15:52:46 +03:00
7d9461808c
builders: configure a swapfile + zswap
2024-07-13 04:40:51 +02:00
293bc52ace
hydra: reduce number of parallel builds per builder to limit RAM consumption
2024-07-13 04:38:24 +02:00
756341ea4c
builders: tune sshd MaxStartups to avoid rate limiting Hydra
2024-07-12 21:57:04 +02:00
e6ead602f0
builders get a special treatment for dns64
2024-07-11 02:05:58 +02:00
b14f155d55
add ipmitool on vpn-gw and builders
2024-07-10 20:49:17 +02:00
d2336262fb
hydra: set allowed URIs in restricted mode for flake inputs
2024-07-10 18:52:22 +02:00
411d514ab9
hydra: user hydra-www needs nix-daemon access too
2024-07-10 17:36:39 +02:00
f74d1ca0f6
hydra: start signing paths
2024-07-10 17:34:57 +02:00
e84b362b7a
Allow 12 hour of backfill for metrics
...
This is somewhat experimental and may explode, but we'll see, I guess
2024-07-10 14:59:09 +03:00
9e7e6d42ab
Make nginx/loki/mimir go fast
2024-07-10 14:55:28 +03:00
f2c2bc5ab6
hydra: output machine host key as base64 in the generated machines.conf
2024-07-10 02:16:45 +02:00
f214da9228
hydra: add hydra to nix trusted-users
2024-07-10 02:03:33 +02:00
82db8f7f1e
gerrit01: some more tuning
...
* flip off proxy_buffering again
* enable REVWALK_USE_PRIORITY_QUEUE
* enable delta compression, because that's not a bottleneck and it's
nicer on bandwidth
2024-07-10 00:27:36 +01:00
9988811be5
hydra: unplug the EPYC
...
thank you for your testing services
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-10 01:13:10 +02:00
2308870aa5
builders: add a nice tag to deploy all of them at once
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-10 00:59:31 +02:00
645ad7d062
builders: add builder user
...
currently hardcoded to hydra's coordinator public key
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-10 00:55:25 +02:00
a30c1f7d78
hydra: wire up new builders
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-10 00:45:02 +02:00
eb21cb6916
add baremetal builders
2024-07-10 00:35:01 +02:00
3828721e4f
services/netbox: enable OIDC via Lix SSO
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-09 02:45:58 +02:00
8a9ff8c40d
services/gerrit: migrate to Gerrit from the-distro/nix-gerrit flake
2024-07-08 23:30:59 +01:00
7f46e5d9a4
services: add ofborg, currently running rabbitmq only
2024-07-08 23:55:11 +02:00
b55475c12e
Fix up the rest of the dashboards
2024-07-08 11:43:57 +03:00
9f0e601d84
Scrape grafana/loki/mimir own metrics
2024-07-08 10:25:15 +03:00
209f71c63a
Update node_exporter dashboard for new metrics structure
2024-07-08 10:16:37 +03:00
563e0685d4
Metrics fixups
...
- fix grafana-agent config format
- rekey metrics-push-password for fodwatch
2024-07-08 10:01:25 +03:00
8d2a367e92
grafana-agent: make bagel.monitoring.grafana-agent.exporters an attrset
...
This allows us to use multiple jobs, one for each additional exporter,
and set their `job_name` accordingly.
`job_name` is exported as `job` label on the resulting metrics.
This allows us to quickly get an understanding what metrics of an
exporter are actually available by simply filtering all metrics by
`{job="$jobname"}`
2024-07-08 09:34:26 +03:00
