563e0685d4
Metrics fixups
...
- fix grafana-agent config format
- rekey metrics-push-password for fodwatch
2024-07-08 10:01:25 +03:00
8d2a367e92
grafana-agent: make bagel.monitoring.grafana-agent.exporters an attrset
...
This allows us to use multiple jobs, one for each additional exporter,
and set their `job_name` accordingly.
`job_name` is exported as `job` label on the resulting metrics.
This allows us to quickly get an understanding what metrics of an
exporter are actually available by simply filtering all metrics by
`{job="$jobname"}`
2024-07-08 09:34:26 +03:00
db8c831c2f
grafana-agent: set hostname label on all metrics
...
This is handy to quickly see all metrics exported by a node, without
having to mangle with the already existing `instance` label.
`hostname` is essentially a variant of `instance` but without ports.
2024-07-08 09:34:26 +03:00
ba0d50624d
Switch to push metrics with Grafana Agent
2024-07-08 09:34:24 +03:00
40ba3c4ae7
Prepare for remote push metrics
2024-07-08 09:33:59 +03:00
346a74eabc
Wire up Grafana to Alertmanager
2024-07-08 09:33:59 +03:00
e8e262c6a4
Enable Mimir Alertmanager, add example alert
...
Still TODO: actually connect it to Matrix
2024-07-08 09:33:59 +03:00
5ebd71e4d5
tf/hydra: change Hydra URL
2024-07-08 00:01:24 +02:00
2700ac5efc
tf/dns: fix hydra CNAME
2024-07-08 00:01:14 +02:00
caa1fce74e
hydra: move to hydra.forkos.org
2024-07-07 23:53:21 +02:00
5f8228536c
bagel-box: switch to forkos.org DNS root
2024-07-07 23:52:40 +02:00
078f298b8c
tf/dns: add bagel-box and hydra
2024-07-07 23:48:23 +02:00
4b0a2cd7e5
tf: add DNS management via Gandi
2024-07-07 20:43:05 +02:00
dcd5f68545
tf: store hydra credentials in state via numtide/secret
2024-07-07 19:18:30 +02:00
7c6780a2a3
gitignore: add terraform lock file
2024-07-07 19:18:30 +02:00
dd72904bf1
flake: replace tf wrappers with a single '.#tf' command
2024-07-07 19:18:30 +02:00
2e9483936e
tf/hydra: fix project owner to use an automation account
2024-07-07 18:44:17 +02:00
30859b2872
terraform: store state on S3
2024-07-07 18:22:41 +02:00
0c68a23275
flake: fix 'nix flake check'
2024-07-07 18:02:55 +02:00
8dc7ee9864
hydra: add declarative controls via terranix
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-07 17:59:56 +02:00
e803c198c1
admins: provision jade
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-07 13:15:27 +00:00
578e24e634
systems: add fodwatch.forkos.org
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-07 13:15:27 +00:00
e1a034927c
Merge pull request 'Split node_exporter and cadvisor config, disable cadvisor for nodes that are themselves containers' ( #25 ) from cadvisor-containers into main
...
Reviewed-on: delroth/bagel-infra#25
Reviewed-by: raito <raito@noreply.git.lix.systems>
2024-07-05 17:21:27 +00:00
5b0f3c4541
Split node_exporter and cadvisor config, disable cadvisor for nodes that are themselves containers
2024-07-05 20:06:43 +03:00
b319b02f07
fix: remove custom logging format for Gerrit
...
This way, we get picked up by the LGTM stack exporter machinery.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-05 18:52:38 +02:00
75f779716d
Merge pull request 'Grafana' ( #24 ) from grafana into main
...
Reviewed-on: delroth/bagel-infra#24
2024-07-05 16:43:13 +00:00
2441d18f17
Add Loki + Promtail setup
2024-07-05 16:10:31 +00:00
03cb9c390c
Add postgres exporter
2024-07-05 16:10:31 +00:00
42f8ad8fa4
Add nginx log exporter
2024-07-05 16:10:31 +00:00
63b31e98cf
Add Grafana/Prometheus/Mimir minimal setup
...
More later, Loki also later.
2024-07-05 16:10:31 +00:00
99f715caca
Add devShell with agenix and colmena
2024-07-05 16:10:31 +00:00
3ad481c125
Clean up SSH key dupes, add Maxine
2024-07-05 16:10:31 +00:00
34a29552da
hydra: update the epyc.infra.newtype.fr public host key
2024-07-05 16:43:29 +02:00
fa1bc1ced9
Merge pull request 'gerrit01: those who finetune even further' ( #20 ) from gerrit-finetuning into main
...
Reviewed-on: delroth/bagel-infra#20
2024-07-05 12:37:43 +00:00
6b7ddbcd29
bagel-box: reuse common/ module
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-05 13:29:56 +02:00
e27f152f00
common/base-server: use ambiant stable lix by default
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-05 13:29:47 +02:00
6fb584109a
common/raito-vm: disable useDHCP
...
We are using networkd by default…
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-05 13:12:35 +02:00
0b01e9a99f
gerrit01: those who finetune even further
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-05 12:23:44 +02:00
832b0784d8
common/admins: add K900
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-04 23:57:05 +02:00
e148d54b18
Merge pull request 'gerrit01: make it go brrr on https clone' ( #17 ) from gerrit-http-clones into main
...
Reviewed-on: delroth/bagel-infra#17
2024-07-04 12:43:18 +00:00
6c237e8d40
gerrit01: make it go brrr on https clone
...
proxy_buffering was the root cause.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-04 14:42:49 +02:00
bf8fe65f9f
bagel-box: update ssh host key & rekey
2024-07-04 13:59:18 +02:00
f0647702d4
hotfix: hot bagel on secrets (netbox)
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-04 13:51:25 +02:00
e387fffd66
hydra: add i686-linux support to the remote builder because nixpkgs bootstrap relies on it, even on x86_64
2024-07-04 13:44:59 +02:00
0a747f2f3a
Merge pull request 'terranix: init' ( #14 ) from terranix-init into main
...
Reviewed-on: delroth/bagel-infra#14
2024-07-04 11:42:17 +00:00
e422da8f83
terranix: init
...
Adds simple `nix run .#apply` & `nix run .#destroy` which renders the
configuration file and let Terraform drive it.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-04 13:41:52 +02:00
357f5d8989
Merge pull request 'meta01: update DNS to *.forkos.org' ( #12 ) from meta01-update-dns into main
...
Reviewed-on: delroth/bagel-infra#12
2024-07-03 08:59:53 +00:00
182e55c35f
gerrit01: rename to cl.forkos.org
...
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-03 10:58:49 +02:00
c892e4ca70
Merge pull request 'gerrit01: init' ( #10 ) from gerrit01 into main
...
Reviewed-on: delroth/bagel-infra#10
2024-07-03 08:56:54 +00:00
c327423b46
meta01: *.nixpkgs.lahfa.xyz → *.forkos.org
...
As it should be.
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-07-01 21:29:34 +02:00
