Compare commits

...

44 commits

Author SHA1 Message Date
4ea20c9a96 chore: connect terraform to the hive
This way, we can autogenerate records for storage and builders nodes by
reading directly the configuration.

This makes evaluation of `nix run .#tf` slower, but this makes things
more safer and easier, so it's good.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-23 19:40:33 +01:00
349206de12 chore: introduce finer-grained baremetal management
for multiple roles such as storage or builders.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-23 19:40:33 +01:00
f4588aff2b feat: listen on Gerrit events and rewrite them as generic VCS events
This introduces the private SSH key for Gerrit event streaming.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-16 01:25:53 +01:00
90038e80a2 fix: do not propagate rabbitmq-password to all nodes
This was a mistake.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-16 00:25:54 +01:00
665a750e35 chore: fix vhost and username for ofborg
Username and vhost creation are out of band and manual.

$ cd /var/lib/rabbitmq
$ sudo -u rabbitmq rabbitmqctl create_user ofborg $pwd
$ sudo -u rabbitmq rabbitmqctl set_permissions ofborg '.*' '.*' '.*'

Here's a simple way to reproduce that setup on the RabbitMQ server.

Doing better will require the Vault server which will come soon anyway.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 18:19:49 +01:00
ab998c8fb9 chore: bump ofborg
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 17:25:42 +01:00
bb7d5c1c7d chore: re-encrypt rabbitmq password
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 17:25:35 +01:00
eaee10ec70 chore: bump ofborg
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
df0bd6b4eb feat: introduce statcheck worker
Status & checks RPC & event queue.

The status & checks is set by the rest of OfBorg, the web service needs
to be exposed.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
c007bbeeb9 feat: introduce ofborg gerrit streamer
This pipes events from Gerrit into the whole AMQP broker and enable all
the system to react to VCS changes.

We need a filter to transform raw Gerrit events into ofBorg specific
events that we will continue to send in the system.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
c1cb1ffcad feat: update ofborg
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
4fe922bcd0 feat: introduce ofborg mass rebuilder
With Gerrit support.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
adb78e633c feat: introduce ofborg pastebin service
The web service is not available yet.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
ebdb7c8aef fix: introduce the newest branch of ofborg
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
9051ce73c6 fix: disable IPv4 on amqp.forkos.org
Otherwise, the renew fails all the time!

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
8fa0e5abe3 feat: introduce ofborg stats
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
47b713ca58 feat: introduce ofborg builder
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:45 +01:00
436882c3eb fix(services/vault): proxy pass to the local vault server web port
Oopsie, forgot that commit.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:51:11 +01:00
14f5bc10a1 chore(pkgs/openbao): 2.0.2 -> 2.1.0
https://openbao.org/docs/release-notes/2-1-0/
https://openbao.org/docs/release-notes/2-0-0/#203
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:42:48 +01:00
a4d4ff8041 feat(build-coord): enable first Vault instance on it
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-15 16:30:21 +01:00
2c4e60760f feat: introduce a Vault module for secrets management
Via a fork of the Linux Foundation, called OpenBao.

The module supports high availability but we only have one node for now.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 21:49:44 +01:00
dc23bb7054 feat: introduce awareness module for WAN addresses
Introduce a data-only module to perform abstraction on the deployment,
we use it for WAN for now.

The usecase is service discovery for simple cases.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 21:47:53 +01:00
84899b48ea feat(channel-scripts): support push to git and automatic cleanup of failed streaming
Now, we won't pile a bunch of failed streaming attempts and this will
automatically push to git.

Credentials are left to be done for the push to actually work.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 19:06:47 +01:00
c3b1a3d1da feat(gerrit01): upgrade to Gerrit 3.10.3
And monitor the performance situation as always.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 17:13:18 +01:00
980709cc02 chore(ows): remove Raito personal sandbox branches
I am not using those branches anymore, we can remove them.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 14:42:22 +00:00
112f60afd1 feat(ows): support moving away onewaysync
We are running into too many out of disk space situations with OWS on
the main disk.

This way, we can reuse the Gerrit disk for all that data, which
hopefully, is quite shared with Gerrit.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 14:42:22 +00:00
879292aa9e chore: bump everything
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 15:17:29 +01:00
e912796992 fix(buildbot): use builder-4 for forkos buildbot instance
Since 070d97fd, we have shut down builder-4.

Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-12-14 15:17:22 +01:00
070d97fdf1 adjust builders list 2024-12-09 10:46:03 +01:00
2e7a702c28 adjust builder assignments 2024-12-09 10:28:20 +01:00
cae763e8dd ofborg: enable nginx for certs 2024-11-17 14:48:08 +03:00
4f1378937f Disable nixos-option, it breaks 2024-11-17 14:39:25 +03:00
7e3074a769 Update everything again 2024-11-17 14:39:19 +03:00
3182a036c0 chore: bump everything 2024-11-12 23:42:51 +03:00
54e8282aac fix: use promtool to verify rules, fix format 2024-11-12 23:21:30 +03:00
41be8dc170 fix: pin pyroscope to go 1.22 2024-11-12 23:16:46 +03:00
b08330c42b fix: allow insecure netbox 2024-11-12 22:52:11 +03:00
268422f653 fix: update grapevine to build with current nixpkgs 2024-11-12 22:49:33 +03:00
4e03cf2309 increase zram on build-coord 2024-11-04 08:48:40 +01:00
00a5d373f3 flake.lock: Update
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/e2f08f4d8b3ecb5cf5c9fd9cb2d53bb3c71807da' (2024-10-05)
  → 'github:NixOS/nixpkgs/7ffd9ae656aec493492b44d0ddfb28e79a1ea25d' (2024-11-02)
2024-11-04 08:46:51 +01:00
ec93c94e7e revert default shell to bash
zsh is unbearably slow on some machines
2024-10-30 13:29:27 +01:00
f56576d644 fix: add util-linux in git-gc-preserve
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-10-28 12:07:54 +01:00
1ae3d7c396 chore: move to forkos branch for buildbot
Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
2024-10-28 12:07:06 +01:00
cca8156e52 Revert "chore: move to faster-depinfo branch for Buildbot"
This reverts commit 7df7eaeb9b because
there was a missing intersection in the code and this broke Buildbot
entirely by exhausting all the resources.
2024-10-27 22:01:34 +01:00
36 changed files with 1130 additions and 315 deletions

View file

@ -7,6 +7,7 @@
nixpkgs.overlays = import ../overlays;
nix.package = lib.mkDefault pkgs.lix;
system.tools.nixos-option.enable = false;
services.openssh.enable = lib.mkForce true;
networking.nftables.enable = true;

View file

@ -1,7 +1,7 @@
{ lib, config, ... }:
{ lib, config, ... }:
let
cfg = config.bagel.hardware.raito-vm;
inherit (lib) mkEnableOption mkIf mkOption types;
inherit (lib) mkEnableOption mkIf mkOption types split toIntBase10;
in
{
options.bagel.hardware.raito-vm = {
@ -54,6 +54,17 @@ in
linkConfig.Name = "wan";
};
bagel.infra.self.wan =
let
parts = split "/" cfg.networking.wan.address;
address = builtins.elemAt parts 0;
prefixLength = toIntBase10 (builtins.elemAt 1 parts);
in
{
family = "inet6";
inherit address prefixLength;
};
boot.loader.systemd-boot.enable = true;
boot.initrd.kernelModules = [

View file

@ -8,18 +8,19 @@
fodwatch = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFRyTNfvKl5FcSyzGzw+h+bNFNOxdhvI67WdUZ2iIJ1L";
buildbot = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJgIu6ouagYqBeMLfmn1CbaDJMuZcPH9bnUhkht8GfuB";
git = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEQJcpkCUOx8+5oukMX6lxrYcIX8FyHu8Mc/3+ieKMUn";
bm-0 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBHSNcDGctvlG6BHcJuYIzW9WsBJsts2vpwSketsbXoL";
bm-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIQOGUjERK7Mx8UPM/rbOdMqVyn1sbWqYOG6CbOzH2wm";
bm-2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMKzXIqCoYElEKIYgjbSpqEcDeOvV+Wo3Agq3jba83cB";
bm-3 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq0A5233XGt34T097KaEKBUqFvaa7a6nYZRsSO0166l";
bm-4 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB9dVo2xZhgIMDgB1rUj5ApmppL39BtYu/+OFHeduvXr";
bm-5 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE7vZTBxrVHmHpv7slQ8A8XwjjbfN+ZJA0V5C3k0wNBD";
bm-6 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOt1qR/2BRtc6PABuSBulowwJVO6wBNDyEFzh0qsTeOF";
bm-7 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFinAAw1v8TJB8/wcmTVBbHHc4LCYh6z4TO6ViwUPkoh";
bm-8 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKGSWHNeqT0kF/e4yVy2ieW98X5QMyCYIYZh9WTmQDs1";
bm-9 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOhws9zGgocVY36dMtOL+CXadpvRMffxoWMkfEcTBJm7";
bm-10 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE7sgIuTSqZiZhp8TvObSbIEhcHHsL5hcmYA22uzwxth";
# bm-11 actually?
build-coord = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINpAEJP7F+XtJBpQP1jTzwXwQgJrFxwEJjPf/rnCXkJA";
builder-0 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBHSNcDGctvlG6BHcJuYIzW9WsBJsts2vpwSketsbXoL";
builder-1 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIQOGUjERK7Mx8UPM/rbOdMqVyn1sbWqYOG6CbOzH2wm";
builder-2 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMKzXIqCoYElEKIYgjbSpqEcDeOvV+Wo3Agq3jba83cB";
builder-3 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq0A5233XGt34T097KaEKBUqFvaa7a6nYZRsSO0166l";
builder-4 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB9dVo2xZhgIMDgB1rUj5ApmppL39BtYu/+OFHeduvXr";
builder-5 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE7vZTBxrVHmHpv7slQ8A8XwjjbfN+ZJA0V5C3k0wNBD";
builder-6 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOt1qR/2BRtc6PABuSBulowwJVO6wBNDyEFzh0qsTeOF";
builder-7 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFinAAw1v8TJB8/wcmTVBbHHc4LCYh6z4TO6ViwUPkoh";
builder-8 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKGSWHNeqT0kF/e4yVy2ieW98X5QMyCYIYZh9WTmQDs1";
builder-9 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOhws9zGgocVY36dMtOL+CXadpvRMffxoWMkfEcTBJm7";
builder-10 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE7sgIuTSqZiZhp8TvObSbIEhcHHsL5hcmYA22uzwxth";
wob-vpn-gw = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINVytPPW8XnXf/rD5TFzsw//CZc2lBjQLmDzlVGPZsjh";
# Lix

View file

@ -1,5 +1,4 @@
{ lib, pkgs, config, ... }: {
users.defaultUserShell = pkgs.zsh;
programs.zsh = {
enable = true;
enableCompletion = true;

View file

@ -27,16 +27,17 @@
"inputs": {
"crane": "crane",
"flake-compat": "flake-compat_2",
"flake-utils": "flake-utils_2",
"flake-parts": "flake-parts_2",
"nix-github-actions": "nix-github-actions_2",
"nixpkgs": "nixpkgs",
"nixpkgs-stable": "nixpkgs-stable"
},
"locked": {
"lastModified": 1711742460,
"narHash": "sha256-0O4v6e4a1toxXZ2gf5INhg4WPE5C5T+SVvsBt+45Mcc=",
"lastModified": 1731270564,
"narHash": "sha256-6KMC/NH/VWP5Eb+hA56hz0urel3jP6Y6cF2PX6xaTkk=",
"owner": "zhaofengli",
"repo": "attic",
"rev": "4dbdbee45728d8ce5788db6461aaaa89d98081f0",
"rev": "47752427561f1c34debb16728a210d378f0ece36",
"type": "github"
},
"original": {
@ -49,11 +50,11 @@
"bats-assert": {
"flake": false,
"locked": {
"lastModified": 1636059754,
"narHash": "sha256-ewME0l27ZqfmAwJO4h5biTALc9bDLv7Bl3ftBzBuZwk=",
"lastModified": 1692829535,
"narHash": "sha256-oDqhUQ6Xg7a3xx537SWLGRzqP3oKKeyY4UYGCdz9z/Y=",
"owner": "bats-core",
"repo": "bats-assert",
"rev": "34551b1d7f8c7b677c1a66fc0ac140d6223409e5",
"rev": "e2d855bc78619ee15b0c702b5c30fb074101159f",
"type": "github"
},
"original": {
@ -65,11 +66,11 @@
"bats-support": {
"flake": false,
"locked": {
"lastModified": 1548869839,
"narHash": "sha256-Gr4ntadr42F2Ks8Pte2D4wNDbijhujuoJi4OPZnTAZU=",
"lastModified": 1693050811,
"narHash": "sha256-PxJaH16+QrsfZqtkWVt5K6TwJB5gjIXnbGo+MB84WIU=",
"owner": "bats-core",
"repo": "bats-support",
"rev": "d140a65044b2d6810381935ae7f0c94c7023c8c3",
"rev": "9bf10e876dd6b624fe44423f0b35e064225f7556",
"type": "github"
},
"original": {
@ -87,16 +88,16 @@
"treefmt-nix": "treefmt-nix"
},
"locked": {
"lastModified": 1730050514,
"narHash": "sha256-QvzyEJQiFyj3ZjJmN0PlbhJjhYgvMjinhz1mwKbXeJs=",
"ref": "refs/heads/faster-depinfo",
"rev": "ab0767beddb020f17bbee2965232c0e941a037db",
"revCount": 304,
"lastModified": 1730064416,
"narHash": "sha256-Opbtu9hKijGkEx+GYbSu3MJms3lFxZmAGTFyckguWMM=",
"ref": "refs/heads/forkos",
"rev": "79137b14f3cb376204f739f44b05aebfc288ca89",
"revCount": 310,
"type": "git",
"url": "https://git.lix.systems/lix-project/buildbot-nix.git"
},
"original": {
"ref": "refs/heads/faster-depinfo",
"ref": "refs/heads/forkos",
"type": "git",
"url": "https://git.lix.systems/lix-project/buildbot-nix.git"
}
@ -108,11 +109,11 @@
]
},
"locked": {
"lastModified": 1725128016,
"narHash": "sha256-4TvaXELsl+1OcGNgqB/5HVXVxBvdIQkhJsY4FyiDcNU=",
"lastModified": 1734197525,
"narHash": "sha256-rb/+iJBNsfXnz+PJSdlsCViodtEHrgfz/Fixq2NXUFI=",
"ref": "refs/heads/main",
"rev": "23b6c38ed7e11417bf624f6e4fb6cde0d2be6400",
"revCount": 261,
"rev": "6e4ae567a3f872bdb90a62d588bb5cc4b3596258",
"revCount": 265,
"type": "git",
"url": "https://git.lix.systems/the-distro/channel-scripts.git"
},
@ -125,17 +126,18 @@
"inputs": {
"flake-compat": "flake-compat",
"flake-utils": "flake-utils",
"nix-github-actions": "nix-github-actions",
"nixpkgs": [
"nixpkgs"
],
"stable": "stable"
},
"locked": {
"lastModified": 1711386353,
"narHash": "sha256-gWEpb8Hybnoqb4O4tmpohGZk6+aerAbJpywKcFIiMlg=",
"lastModified": 1731527002,
"narHash": "sha256-dI9I6suECoIAmbS4xcrqF8r2pbmed8WWm5LIF1yWPw8=",
"owner": "zhaofengli",
"repo": "colmena",
"rev": "cd65ef7a25cdc75052fbd04b120aeb066c3881db",
"rev": "e3ad42138015fcdf2524518dd564a13145c72ea1",
"type": "github"
},
"original": {
@ -153,11 +155,11 @@
]
},
"locked": {
"lastModified": 1702918879,
"narHash": "sha256-tWJqzajIvYcaRWxn+cLUB9L9Pv4dQ3Bfit/YjU5ze3g=",
"lastModified": 1722960479,
"narHash": "sha256-NhCkJJQhD5GUib8zN9JrmYGMwt4lCRp6ZVNzIiYCl0Y=",
"owner": "ipetkov",
"repo": "crane",
"rev": "7195c00c272fdd92fc74e7d5a0a2844b9fadb2fb",
"rev": "4c6c77920b8d44cd6660c1621dea6b3fc4b4c4f4",
"type": "github"
},
"original": {
@ -167,18 +169,12 @@
}
},
"crane_2": {
"inputs": {
"nixpkgs": [
"grapevine",
"nixpkgs"
]
},
"locked": {
"lastModified": 1716569590,
"narHash": "sha256-5eDbq8TuXFGGO3mqJFzhUbt5zHVTf5zilQoyW5jnJwo=",
"lastModified": 1731098351,
"narHash": "sha256-HQkYvKvaLQqNa10KEFGgWHfMAbWBfFp+4cAgkut+NNE=",
"owner": "ipetkov",
"repo": "crane",
"rev": "109987da061a1bf452f435f1653c47511587d919",
"rev": "ef80ead953c1b28316cc3f8613904edc2eb90c28",
"type": "github"
},
"original": {
@ -219,11 +215,11 @@
"rust-analyzer-src": "rust-analyzer-src"
},
"locked": {
"lastModified": 1716359173,
"narHash": "sha256-pYcjP6Gy7i6jPWrjiWAVV0BCQp+DdmGaI/k65lBb/kM=",
"lastModified": 1731738660,
"narHash": "sha256-tIXhc9lX1b030v812yVJanSR37OnpTb/OY5rU3TbShA=",
"owner": "nix-community",
"repo": "fenix",
"rev": "b6fc5035b28e36a98370d0eac44f4ef3fd323df6",
"rev": "e10ba121773f754a30d31b6163919a3e404a434f",
"type": "github"
},
"original": {
@ -252,11 +248,11 @@
"flake-compat_2": {
"flake": false,
"locked": {
"lastModified": 1673956053,
"narHash": "sha256-4gtG9iQuiKITOjNQQeQIpoIB6b16fm+504Ch3sNKLd8=",
"lastModified": 1696426674,
"narHash": "sha256-kvjfFW7WAETZlt09AgDn1MrtKzP7t90Vf7vypd3OL1U=",
"owner": "edolstra",
"repo": "flake-compat",
"rev": "35bb57c0c8d8b62bbfd284272c928ceb64ddbde9",
"rev": "0f9255e01c2351cc7d116c072cb317785dd33b33",
"type": "github"
},
"original": {
@ -322,8 +318,8 @@
"flake-parts_2": {
"inputs": {
"nixpkgs-lib": [
"hydra",
"nix-eval-jobs",
"grapevine",
"attic",
"nixpkgs"
]
},
@ -341,6 +337,45 @@
"type": "github"
}
},
"flake-parts_3": {
"inputs": {
"nixpkgs-lib": [
"hydra",
"nix-eval-jobs",
"nixpkgs"
]
},
"locked": {
"lastModified": 1730504689,
"narHash": "sha256-hgmguH29K2fvs9szpq2r3pz2/8cJd2LPS+b4tfNFCwE=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "506278e768c2a08bec68eb62932193e341f55c90",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"flake-parts_4": {
"inputs": {
"nixpkgs-lib": "nixpkgs-lib"
},
"locked": {
"lastModified": 1727826117,
"narHash": "sha256-K5ZLCyfO/Zj9mPFldf3iwS6oZStJcU4tSpiXTMYaaL0=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "3d04084d54bedc3d6b8b736c70ef449225c361b1",
"type": "github"
},
"original": {
"id": "flake-parts",
"type": "indirect"
}
},
"flake-utils": {
"locked": {
"lastModified": 1659877975,
@ -357,30 +392,15 @@
}
},
"flake-utils_2": {
"locked": {
"lastModified": 1667395993,
"narHash": "sha256-nuEHfE/LcWyuSWnS8t12N1wc105Qtau+/OdUAjtQ0rA=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "5aed5285a952e0b949eb3ba02c12fa4fcfef535f",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"flake-utils_3": {
"inputs": {
"systems": "systems_2"
},
"locked": {
"lastModified": 1710146030,
"narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
"lastModified": 1731533236,
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
"type": "github"
},
"original": {
@ -390,21 +410,6 @@
"type": "github"
}
},
"flake-utils_4": {
"locked": {
"lastModified": 1634851050,
"narHash": "sha256-N83GlSGPJJdcqhUxSCS/WwW5pksYf3VP1M13cDRTSVA=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "c91f3de5adaf1de973b797ef7485e441a65b8935",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"gerrit-dashboard": {
"flake": false,
"locked": {
@ -427,20 +432,21 @@
"crane": "crane_2",
"fenix": "fenix",
"flake-compat": "flake-compat_3",
"flake-utils": "flake-utils_3",
"flake-utils": "flake-utils_2",
"nix-filter": "nix-filter",
"nixpkgs": [
"nixpkgs"
],
"rocksdb": "rocksdb",
"rust-manifest": "rust-manifest"
},
"locked": {
"host": "gitlab.computer.surgery",
"lastModified": 1727994504,
"narHash": "sha256-FC6M1KKX58HbU9LG+cG6EJRr02J9lE/o0iiDi6m1gv8=",
"lastModified": 1734138037,
"narHash": "sha256-pN/nJ9tR6ewnpVUUzcF+Z9L/0R0WmtBVePJOqx9rzTk=",
"owner": "matrix",
"repo": "grapevine-fork",
"rev": "5a490a4397f0c6a36dab1cb631dadc67a849deab",
"rev": "8537c0e8ac3eb388500587b035008e5f98204a4b",
"type": "gitlab"
},
"original": {
@ -480,11 +486,11 @@
]
},
"locked": {
"lastModified": 1728321752,
"narHash": "sha256-GbBAoBF7ZObz0IP+g0LZKxMafpMvNKjTEu9haiZbV54=",
"lastModified": 1733503045,
"narHash": "sha256-VoMam8Zzbk+X6dIYwH2f9NqItL6g9YDhQvGybzSl8xQ=",
"ref": "refs/heads/main",
"rev": "ee1234c15cdcb427dbd4828e0add09d02cd606c9",
"revCount": 4220,
"rev": "eccf01d4fef67f87b6383f96c73781bd08b686ac",
"revCount": 4230,
"type": "git",
"url": "https://git.lix.systems/lix-project/hydra.git"
},
@ -505,11 +511,11 @@
"pre-commit-hooks": "pre-commit-hooks"
},
"locked": {
"lastModified": 1728163191,
"narHash": "sha256-SW0IEBsPN1EysqzvfDT+8Kimtzy03O1BxQQm7ZB6fRY=",
"lastModified": 1732112222,
"narHash": "sha256-H7GN4++a4vE49SUNojZx+FSk4mmpb2ifJUtJMJHProI=",
"ref": "refs/heads/main",
"rev": "ed9b7f4f84fd60ad8618645cc1bae2d686ff0db6",
"revCount": 16323,
"rev": "66f6dbda32959dd5cf3a9aaba15af72d037ab7ff",
"revCount": 16513,
"type": "git",
"url": "https://git.lix.systems/lix-project/lix"
},
@ -520,12 +526,12 @@
},
"nix-eval-jobs": {
"inputs": {
"flake-parts": "flake-parts_2",
"flake-parts": "flake-parts_3",
"lix": [
"hydra",
"lix"
],
"nix-github-actions": "nix-github-actions",
"nix-github-actions": "nix-github-actions_3",
"nixpkgs": [
"hydra",
"nixpkgs"
@ -533,11 +539,11 @@
"treefmt-nix": "treefmt-nix_2"
},
"locked": {
"lastModified": 1723579251,
"narHash": "sha256-xnHtfw0gRhV+2S9U7hQwvp2klTy1Iv7FlMMO0/WiMVc=",
"lastModified": 1732351635,
"narHash": "sha256-H94CcQ3yamG5+RMxtxXllR02YIlxQ5WD/8PcolO9yEA=",
"ref": "refs/heads/main",
"rev": "42a160bce2fd9ffebc3809746bc80cc7208f9b08",
"revCount": 609,
"rev": "dfc286ca3dc49118c30d8d6205d6d6af76c62b7a",
"revCount": 617,
"type": "git",
"url": "https://git.lix.systems/lix-project/nix-eval-jobs"
},
@ -548,11 +554,11 @@
},
"nix-filter": {
"locked": {
"lastModified": 1710156097,
"narHash": "sha256-1Wvk8UP7PXdf8bCCaEoMnOT1qe5/Duqgj+rL8sRQsSM=",
"lastModified": 1731533336,
"narHash": "sha256-oRam5PS1vcrr5UPgALW0eo1m/5/pls27Z/pabHNy2Ms=",
"owner": "numtide",
"repo": "nix-filter",
"rev": "3342559a24e85fc164b295c3444e8a139924675b",
"rev": "f7653272fd234696ae94229839a99b73c9ab7de0",
"type": "github"
},
"original": {
@ -569,20 +575,64 @@
]
},
"locked": {
"lastModified": 1720891381,
"narHash": "sha256-bdZRPgnkROSejmwMOrlcqHMWmuPIVIzjk6r5FbS+fqU=",
"ref": "refs/heads/main",
"rev": "23dd318e6741ff686d3069c53ecf475eac8a0565",
"revCount": 5,
"lastModified": 1734192622,
"narHash": "sha256-AkT4QHHneyWBL9UDhvrmPnQUOfN9ETP295y6TtuW6rU=",
"ref": "refs/heads/bump-minor-3_10",
"rev": "c011f670b335b52150af5c75f21e987d166ecec2",
"revCount": 8,
"type": "git",
"url": "https://git.lix.systems/the-distro/nix-gerrit.git"
},
"original": {
"ref": "refs/heads/bump-minor-3_10",
"type": "git",
"url": "https://git.lix.systems/the-distro/nix-gerrit.git"
}
},
"nix-github-actions": {
"inputs": {
"nixpkgs": [
"colmena",
"nixpkgs"
]
},
"locked": {
"lastModified": 1729742964,
"narHash": "sha256-B4mzTcQ0FZHdpeWcpDYPERtyjJd/NIuaQ9+BV1h+MpA=",
"owner": "nix-community",
"repo": "nix-github-actions",
"rev": "e04df33f62cdcf93d73e9a04142464753a16db67",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "nix-github-actions",
"type": "github"
}
},
"nix-github-actions_2": {
"inputs": {
"nixpkgs": [
"grapevine",
"attic",
"nixpkgs"
]
},
"locked": {
"lastModified": 1729742964,
"narHash": "sha256-B4mzTcQ0FZHdpeWcpDYPERtyjJd/NIuaQ9+BV1h+MpA=",
"owner": "nix-community",
"repo": "nix-github-actions",
"rev": "e04df33f62cdcf93d73e9a04142464753a16db67",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "nix-github-actions",
"type": "github"
}
},
"nix-github-actions_3": {
"inputs": {
"nixpkgs": [
"hydra",
@ -591,11 +641,11 @@
]
},
"locked": {
"lastModified": 1720066371,
"narHash": "sha256-uPlLYH2S0ACj0IcgaK9Lsf4spmJoGejR9DotXiXSBZQ=",
"lastModified": 1731952509,
"narHash": "sha256-p4gB3Rhw8R6Ak4eMl8pqjCPOLCZRqaehZxdZ/mbFClM=",
"owner": "nix-community",
"repo": "nix-github-actions",
"rev": "622f829f5fe69310a866c8a6cd07e747c44ef820",
"rev": "7b5f051df789b6b20d259924d349a9ba3319b226",
"type": "github"
},
"original": {
@ -607,11 +657,11 @@
"nix2container": {
"flake": false,
"locked": {
"lastModified": 1720642556,
"narHash": "sha256-qsnqk13UmREKmRT7c8hEnz26X3GFFyIQrqx4EaRc1Is=",
"lastModified": 1724996935,
"narHash": "sha256-njRK9vvZ1JJsP8oV2OgkBrpJhgQezI03S7gzskCcHos=",
"owner": "nlewo",
"repo": "nix2container",
"rev": "3853e5caf9ad24103b13aa6e0e8bcebb47649fe4",
"rev": "fa6bb0a1159f55d071ba99331355955ae30b3401",
"type": "github"
},
"original": {
@ -622,11 +672,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1711401922,
"narHash": "sha256-QoQqXoj8ClGo0sqD/qWKFWezgEwUL0SUh37/vY2jNhc=",
"lastModified": 1726042813,
"narHash": "sha256-LnNKCCxnwgF+575y0pxUdlGZBO/ru1CtGHIqQVfvjlA=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "07262b18b97000d16a4bdb003418bd2fb067a932",
"rev": "159be5db480d1df880a0135ca0bfed84c2f88353",
"type": "github"
},
"original": {
@ -636,6 +686,18 @@
"type": "github"
}
},
"nixpkgs-lib": {
"locked": {
"lastModified": 1727825735,
"narHash": "sha256-0xHYkMkeLVQAMa7gvkddbPqpxph+hDzdu1XdGPJR+Os=",
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/fb192fec7cc7a4c26d51779e9bab07ce6fa5597a.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/fb192fec7cc7a4c26d51779e9bab07ce6fa5597a.tar.gz"
}
},
"nixpkgs-regression": {
"locked": {
"lastModified": 1643052045,
@ -654,44 +716,61 @@
},
"nixpkgs-stable": {
"locked": {
"lastModified": 1711460390,
"narHash": "sha256-akSgjDZL6pVHEfSE6sz1DNSXuYX6hq+P/1Z5IoYWs7E=",
"lastModified": 1724316499,
"narHash": "sha256-Qb9MhKBUTCfWg/wqqaxt89Xfi6qTD3XpTzQ9eXi3JmE=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "44733514b72e732bd49f5511bd0203dea9b9a434",
"rev": "797f7dc49e0bc7fab4b57c021cdf68f595e47841",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-23.11",
"ref": "nixos-24.05",
"repo": "nixpkgs",
"type": "github"
}
},
"nixpkgs_2": {
"locked": {
"lastModified": 1728093190,
"narHash": "sha256-CAZF2NRuHmqTtRTNAruWpHA43Gg2UvuCNEIzabP0l6M=",
"lastModified": 1733940404,
"narHash": "sha256-Pj39hSoUA86ZePPF/UXiYHHM7hMIkios8TYG29kQT4g=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "e2f08f4d8b3ecb5cf5c9fd9cb2d53bb3c71807da",
"rev": "5d67ea6b4b63378b9c13be21e2ec9d1afc921713",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixpkgs-unstable",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"ofborg": {
"flake": false,
"locked": {
"lastModified": 1734308727,
"narHash": "sha256-/bJhMZQ5VSblvgqAR9hSLwdm5pxenn/UMY8pDDVSquI=",
"ref": "refs/heads/vcs-generalization",
"rev": "7bcc8fa584c66f317923337658974c0525e5779f",
"revCount": 1495,
"type": "git",
"url": "https://git.lix.systems/the-distro/ofborg.git"
},
"original": {
"ref": "refs/heads/vcs-generalization",
"type": "git",
"url": "https://git.lix.systems/the-distro/ofborg.git"
}
},
"pre-commit-hooks": {
"flake": false,
"locked": {
"lastModified": 1721042469,
"narHash": "sha256-6FPUl7HVtvRHCCBQne7Ylp4p+dpP3P/OYuzjztZ4s70=",
"lastModified": 1726745158,
"narHash": "sha256-D5AegvGoEjt4rkKedmxlSEmC+nNLMBPWFxvmYnVLhjk=",
"owner": "cachix",
"repo": "git-hooks.nix",
"rev": "f451c19376071a90d8c58ab1a953c6e9840527fd",
"rev": "4e743a6920eab45e8ba0fbe49dc459f1423a4b74",
"type": "github"
},
"original": {
@ -700,6 +779,23 @@
"type": "github"
}
},
"rocksdb": {
"flake": false,
"locked": {
"lastModified": 1730475155,
"narHash": "sha256-u5uuShM2SxHc9/zL4UU56IhCcR/ZQbzde0LgOYS44bM=",
"owner": "facebook",
"repo": "rocksdb",
"rev": "3c27a3dde0993210c5cc30d99717093f7537916f",
"type": "github"
},
"original": {
"owner": "facebook",
"ref": "v9.7.4",
"repo": "rocksdb",
"type": "github"
}
},
"root": {
"inputs": {
"agenix": "agenix",
@ -715,6 +811,7 @@
],
"nix-gerrit": "nix-gerrit",
"nixpkgs": "nixpkgs_2",
"ofborg": "ofborg",
"stateless-uptime-kuma": "stateless-uptime-kuma",
"terranix": "terranix"
}
@ -722,11 +819,11 @@
"rust-analyzer-src": {
"flake": false,
"locked": {
"lastModified": 1716107283,
"narHash": "sha256-NJgrwLiLGHDrCia5AeIvZUHUY7xYGVryee0/9D3Ir1I=",
"lastModified": 1731693936,
"narHash": "sha256-uHUUS1WPyW6ohp5Bt3dAZczUlQ22vOn7YZF8vaPKIEw=",
"owner": "rust-lang",
"repo": "rust-analyzer",
"rev": "21ec8f523812b88418b2bfc64240c62b3dd967bd",
"rev": "1b90e979aeee8d1db7fe14603a00834052505497",
"type": "github"
},
"original": {
@ -750,16 +847,16 @@
},
"stable": {
"locked": {
"lastModified": 1696039360,
"narHash": "sha256-g7nIUV4uq1TOVeVIDEZLb005suTWCUjSY0zYOlSBsyE=",
"lastModified": 1730883749,
"narHash": "sha256-mwrFF0vElHJP8X3pFCByJR365Q2463ATp2qGIrDUdlE=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "32dcb45f66c0487e92db8303a798ebc548cadedc",
"rev": "dba414932936fde69f0606b4f1d87c5bc0003ede",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-23.05",
"ref": "nixos-24.05",
"repo": "nixpkgs",
"type": "github"
}
@ -810,22 +907,38 @@
"type": "github"
}
},
"systems_3": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
},
"terranix": {
"inputs": {
"bats-assert": "bats-assert",
"bats-support": "bats-support",
"flake-utils": "flake-utils_4",
"flake-parts": "flake-parts_4",
"nixpkgs": [
"nixpkgs"
],
"systems": "systems_3",
"terranix-examples": "terranix-examples"
},
"locked": {
"lastModified": 1695406838,
"narHash": "sha256-xiUfVD6rtsVWFotVtUW3Q1nQh4obKzgvpN1wqZuGXvM=",
"lastModified": 1728959489,
"narHash": "sha256-1Pu2j5xsBTuoyga08ZVf+rKp3FOMmJh/0fXen/idOrA=",
"owner": "terranix",
"repo": "terranix",
"rev": "fc9077ca02ab5681935dbf0ecd725c4d889b9275",
"rev": "7734e2ee6a1472807a33ce1e7da794bed2aaf91c",
"type": "github"
},
"original": {
@ -836,11 +949,11 @@
},
"terranix-examples": {
"locked": {
"lastModified": 1636300201,
"narHash": "sha256-0n1je1WpiR6XfCsvi8ZK7GrpEnMl+DpwhWaO1949Vbc=",
"lastModified": 1637156952,
"narHash": "sha256-KqvXIe1yiKOEP9BRYqNQN+LOWPCsWojh0WjEgv5jfEI=",
"owner": "terranix",
"repo": "terranix-examples",
"rev": "a934aa1cf88f6bd6c6ddb4c77b77ec6e1660bd5e",
"rev": "921680efb8af0f332d8ad73718d53907f9483e24",
"type": "github"
},
"original": {
@ -879,11 +992,11 @@
]
},
"locked": {
"lastModified": 1723454642,
"narHash": "sha256-S0Gvsenh0II7EAaoc9158ZB4vYyuycvMGKGxIbERNAM=",
"lastModified": 1732292307,
"narHash": "sha256-5WSng844vXt8uytT5djmqBCkopyle6ciFgteuA9bJpw=",
"owner": "numtide",
"repo": "treefmt-nix",
"rev": "349de7bc435bdff37785c2466f054ed1766173be",
"rev": "705df92694af7093dfbb27109ce16d828a79155f",
"type": "github"
},
"original": {

101
flake.nix
View file

@ -2,7 +2,7 @@
description = "Bagel cooking infrastructure";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
terranix.url = "github:terranix/terranix";
terranix.inputs.nixpkgs.follows = "nixpkgs";
@ -16,13 +16,16 @@
hydra.url = "git+https://git.lix.systems/lix-project/hydra.git";
hydra.inputs.nixpkgs.follows = "nixpkgs";
nix-gerrit.url = "git+https://git.lix.systems/the-distro/nix-gerrit.git";
nix-gerrit.url = "git+https://git.lix.systems/the-distro/nix-gerrit.git?ref=refs/heads/bump-minor-3_10";
nix-gerrit.inputs.nixpkgs.follows = "nixpkgs";
ofborg.url = "git+https://git.lix.systems/the-distro/ofborg.git?ref=refs/heads/vcs-generalization";
ofborg.flake = false;
gerrit-dashboard.url = "git+https://git.lix.systems/the-distro/gerrit-monitoring.git";
gerrit-dashboard.flake = false;
buildbot-nix.url = "git+https://git.lix.systems/lix-project/buildbot-nix.git?ref=refs/heads/faster-depinfo";
buildbot-nix.url = "git+https://git.lix.systems/lix-project/buildbot-nix.git?ref=refs/heads/forkos";
buildbot-nix.inputs.nixpkgs.follows = "nixpkgs";
channel-scripts.url = "git+https://git.lix.systems/the-distro/channel-scripts.git";
@ -42,7 +45,7 @@
};
};
outputs = { self, nixpkgs, terranix, colmena, ... } @ inputs:
outputs = { self, nixpkgs, terranix, colmena, ofborg, ... } @ inputs:
let
supportedSystems = [ "x86_64-linux" "aarch64-linux" ];
forEachSystem = f: builtins.listToAttrs (map (system: {
@ -58,12 +61,18 @@
inputs.lix.overlays.default
inputs.nix-gerrit.overlays.default
inputs.channel-scripts.overlays.default
(import inputs.ofborg {
pkgs = import nixpkgs { localSystem = system; };
}).overlay
(import "${inputs.stateless-uptime-kuma}/overlay.nix")
];
};
terraform = pkgs.opentofu;
terraformCfg = terranix.lib.terranixConfiguration {
inherit system;
extraArgs = {
inherit (self) nixosConfigurations;
};
modules = [
./terraform
{
@ -126,9 +135,14 @@
# Tag all machines which have local boot as local bootables.
deployment.tags = lib.mkMerge [
[ "floral" ]
(lib.mkIf (config.bagel.baremetal.builders.enable -> !config.bagel.baremetal.builders.netboot)
# All nodes that can be local booted, including baremetal nodes.
(lib.mkIf (config.bagel.baremetal.enable -> !config.bagel.baremetal.netboot)
[ "localboot" ]
)
# Only baremetal nodes that can be local booted.
(lib.mkIf (config.bagel.baremetal.enable && !config.bagel.baremetal.netboot)
[ "bm-localboot" ]
)
];
bagel.monitoring.grafana-agent.tenant = "floral";
@ -139,13 +153,59 @@
];
# These are Floral baremetal builders.
makeBuilder = i:
makeColoBaremetal = i:
let
enableNetboot = i >= 6;
in
lib.nameValuePair "builder-${toString i}" {
# bm for baremetal.
lib.nameValuePair "bm-${toString i}" {
imports = floralInfraModules;
bagel.baremetal.builders = { enable = true; num = i; netboot = enableNetboot; };
bagel.baremetal = { enable = true; num = i; netboot = enableNetboot; };
};
setFlavorForNode = flavor: i: node: {
imports = [
node
];
bagel.baremetal.${flavor} = {
enable = true;
num = i;
};
};
setFlavorForNodes = flavor: { ranges }: nodes:
let
setFlavor = setFlavorForNode flavor;
# Test if i is in [range.start, range.end[.
inRange = i: range: i >= range.start && i < range.end;
# Perform an enumeration from values to indexes.
reverseEnumerate = list: lib.listToAttrs (lib.zipListsWith (i: x: lib.nameValuePair x i) (lib.range 0 (lib.length list - 1)) list);
# Filter all nodes by the range data for this flavor.
filteredNodes = lib.filterAttrs (name: node: lib.any (r: inRange node.bagel.baremetal.num r) ranges) nodes;
# The enumeration from baremetal numbers to flavor-specific numbers.
indexes = reverseEnumerate (map (n: toString n.bagel.baremetal.num) (builtins.attrValues filteredNodes));
in
# Build a new attrset with a new second-level index for the specific flavor.
lib.mapAttrs (name: node: setFlavor indexes.${toString node.bagel.baremetal.num} node) filteredNodes;
closedOpenInterval = a: b: { start = a; end = b; };
interval = a: b: closedOpenInterval a b;
# Returns the singleton {x}.
singleton = x: interval x (x + 1);
# builders: [4, 10].
# storage: [5]
# build-coord: [11].
setStorage = setFlavorForNodes "storage" {
# List the indexes for the baremetal nodes which are supposed to be storage nodes.
ranges = [ (singleton 5) ];
};
setBuildCoordinators = setFlavorForNodes "build-coord" {
ranges = [ (singleton 11) ];
};
setBuilders = setFlavorForNodes "builders" {
# List the indexes for the baremetal nodes which are supposed to be builders nodes.
ranges = [ (singleton 4) (singleton 10) ];
};
lixInfraModules = commonModules ++ [
@ -173,7 +233,28 @@
}
];
builders = lib.listToAttrs (lib.genList makeBuilder 11);
checkForNonReuse = setOfNodes:
let
setOfNodesNames = lib.mapAttrs (name: nodes: builtins.attrNames nodes) setOfNodes;
in
lib.all (set:
lib.all (anotherSet: set != anotherSet -> lib.intersectLists setOfNodesNames.${set} setOfNodesNames.${anotherSet} == []) (builtins.attrNames setOfNodes)
) (builtins.attrNames setOfNodes);
baremetalNodes =
let
# We consider all possible baremetal systems and we filter out a subset that is activated.
# To configure the set of used machines, configure the `setXYZ` role setter selectors.
allNodes = lib.listToAttrs (lib.genList makeColoBaremetal 11);
perRoles = {
storageNodes = setStorage allNodes;
builderNodes = setBuilders allNodes;
# buildCoordinatorNodes = setBuildCoordinators allNodes;
};
in
assert (lib.assertMsg (checkForNonReuse perRoles) "A baremetal node is simultaneously storage, builder and build coordinator, please review the ranges.");
lib.foldl (a: b: a // b) { } (builtins.attrValues perRoles);
in {
meta.nixpkgs = systemBits.x86_64-linux.pkgs;
# Add any non-x86_64 native systems here.
@ -199,7 +280,7 @@
build01-aarch64-lix.imports = lixInfraModules ++ [ ./hosts/build01-aarch64-lix ];
buildbot-lix.imports = lixInfraModules ++ [ ./hosts/buildbot-lix ];
} // builders;
} // baremetalNodes;
hydraJobs = builtins.mapAttrs (n: v: v.config.system.build.netbootDir or v.config.system.build.toplevel) self.nixosConfigurations;
buildbotJobs = builtins.mapAttrs (_: v: v.config.system.build.toplevel) self.nixosConfigurations;

View file

@ -37,7 +37,21 @@
bagel.services = {
postgres.enable = true;
ofborg.enable = true;
ofborg = {
rabbitmq.enable = true;
pastebin.enable = true;
# TODO: statcheck.enable = true;
mass-rebuilder.enable = true;
# TODO: enable once ready.
builder.enable = false;
gerrit-event-streamer.enable = true;
gerrit-generic-vcs-filter.enable = true;
# FIXME: plug into our prometheus stack.
stats.enable = true;
};
};
bagel.sysadmin.enable = true;

View file

@ -9,8 +9,14 @@
bagel.services = {
hydra.enable = true;
# Takes 10 builders (0 → 9).
hydra.builders = lib.genList (i: "builder-${builtins.toString i}") 10;
hydra.builders = map (i: "builder-${builtins.toString i}") [4 5 10];
# Arguably, the build-coordinator is the most sensitive piece of our own infrastructure.
# Henceforth, it can run as well another sensitive piece of the system: the Vault.
vault = {
enable = true;
domain = "vault.forkos.org";
};
};
bagel.monitoring.exporters.hydra.enable = true;

View file

@ -38,7 +38,7 @@
zramSwap = {
enable = true;
memoryPercent = 25;
memoryPercent = 100;
};
networking.useNetworkd = true;
@ -72,6 +72,12 @@
];
networking.defaultGateway6 = { interface = "uplink"; address = "2a01:584:11::1"; };
bagel.infra.self.wan = {
family = "inet6";
address = "2a01:584:11::1:11";
prefixLength = 64;
};
services.coredns = {
enable = true;
config = ''

View file

@ -44,7 +44,7 @@
"nixpkgs"
"infra"
];
builders = [ "builder-9" ];
builders = [ "builder-4" ];
};
i18n.defaultLocale = "en_US.UTF-8";

View file

@ -61,6 +61,8 @@
{
enable = true;
stateDirectory = "/gerrit-data/ows";
pushUrl = "ssh://ows_bot@cl.forkos.org:29418/nixpkgs";
deployKeyPath = config.age.secrets.ows-deploy-key.path;
@ -107,21 +109,6 @@
timer = "hourly";
fromRefspec = "staging-23.11";
};
# Testing jobs for personal sandbox branches
branches."raito-unstable-sync" = {
fromUri = "https://github.com/NixOS/nixpkgs";
fromRefspec = "nixos-unstable-small";
localRefspec = "sandbox/raito/raito-unstable-small";
timer = "*-*-* 12:00:00";
};
branches."raito-release-sync" = {
fromUri = "https://github.com/NixOS/nixpkgs";
fromRefspec = "nixos-24.05";
localRefspec = "sandbox/raito/raito-nixos-24.05";
timer = "daily";
};
};
age.secrets.s3-channel-staging-keys.file = ../../secrets/floral/s3-channel-staging-keys.age;

View file

@ -1,9 +1,10 @@
{ lib
, buildGoModule
, buildGo122Module
, fetchFromGitHub
}:
buildGoModule rec {
# FIXME: update, remove this pin
buildGo122Module rec {
pname = "pyroscope";
version = "1.7.1";

View file

@ -46,6 +46,8 @@ let
postgres-ca-priv = [ machines.bagel-box ];
postgres-tls-priv = [ machines.bagel-box ];
rabbitmq-password = [ machines.bagel-box ];
gerrit-event-listener-ssh-key = [ machines.bagel-box ];
newsletter-secrets = [ machines.public01 ];
s3-revproxy-api-keys = [ machines.public01 ];

Binary file not shown.

Binary file not shown.

View file

@ -3,27 +3,18 @@
let
genBuilders = { offset ? 0, count, f }: builtins.genList (x: rec { name = "builder-${toString (offset + x)}"; value = f name; }) count;
in builtins.listToAttrs (
# The first 8 builders are general purpose hydra builders
genBuilders { count = 8; f = name: {
genBuilders { offset = 4; count = 2; f = name: {
cores = 8;
max-jobs = 8;
supported-features = [ "kvm" "nixos-test" ];
required-features = [ ];
}; }
++
# The last 2 builders are exclusively for big-parallel
genBuilders { offset = 8; count = 2; f = name: {
# This builder is exclusively for big-parallel
genBuilders { offset = 10; count = 1; f = name: {
cores = 20;
max-jobs = 1;
supported-features = [ "kvm" "nixos-test" "big-parallel" ];
required-features = [ "big-parallel" ];
}; }
++
# These are not currently used for hydra
genBuilders { offset = 10; count = 1; f = name: {
cores = 8;
max-jobs = 8;
supported-features = [ "kvm" "nixos-test" "big-parallel" ];
required-features = [ ];
}; }
)

View file

@ -1,15 +1,12 @@
{ pkgs, lib, config, ... }:
let
cfgParent = config.bagel.baremetal;
cfg = config.bagel.baremetal.builders;
in
{
imports = [ ./netboot.nix ];
options = {
bagel.baremetal.builders = {
enable = lib.mkEnableOption "baremetal bagel oven";
netboot = lib.mkEnableOption "netboot";
enable = lib.mkEnableOption "builder role";
num = lib.mkOption {
type = lib.types.int;
};
@ -17,9 +14,6 @@ in
};
config = lib.mkIf cfg.enable {
boot.initrd.availableKernelModules = [ "ahci" "ehci_pci" "usb_storage" "usbhid" "sd_mod" ];
boot.initrd.kernelModules = [ "dm-snapshot" ];
users.users.builder = {
isSystemUser = true;
group = "nogroup";
@ -48,45 +42,21 @@ in
inherit ((import ./assignments.nix).${config.networking.hostName}) max-jobs cores;
};
nixpkgs.hostPlatform = "x86_64-linux";
hardware.cpu.intel.updateMicrocode = true;
fileSystems = {
"/mnt" = {
device = "/dev/disk/by-label/hydra";
fsType = "xfs";
options = ["logbsize=256k"];
};
boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;
boot.initrd.systemd.enable = true;
# We want the tmp filesystem on the same filesystem as the hydra store, so that builds can use reflinks
"/tmp" = {
device = "/mnt/tmp";
options = [ "bind" ];
};
};
boot.initrd.services.lvm.enable = true;
boot.kernel.sysctl."fs.xfs.xfssyncd_centisecs" = "12000";
fileSystems = lib.mkMerge [
(lib.mkIf (!cfg.netboot) {
"/" = {
device = "/dev/disk/by-label/root";
fsType = "xfs";
};
"/boot" = {
device = "/dev/disk/by-label/BOOT";
fsType = "vfat";
options = [ "fmask=0022" "dmask=0022" ];
};
})
{
"/mnt" = {
device = "/dev/disk/by-label/hydra";
fsType = "xfs";
options = ["logbsize=256k"];
};
# We want the tmp filesystem on the same filesystem as the hydra store, so that builds can use reflinks
"/tmp" = {
device = "/mnt/tmp";
options = [ "bind" ];
};
}
];
swapDevices = lib.optionals (!cfg.netboot) [
swapDevices = lib.optionals (!cfgParent.netboot) [
{
device = "/swapfile";
size = 50 * 1024; # 50GiB
@ -103,39 +73,7 @@ in
"console=ttyS0,115200"
];
networking.useNetworkd = true;
networking.hostName = "builder-${toString cfg.num}";
networking.domain = "wob01.infra.forkos.org";
systemd.network = {
netdevs = {
"40-uplink" = {
netdevConfig = {
Kind = "bond";
Name = "uplink";
};
bondConfig = {
Mode = "802.3ad";
TransmitHashPolicy = "layer3+4";
};
};
};
networks = {
"40-eno1" = {
name = "eno1";
bond = [ "uplink" ];
};
"40-eno2" = {
name = "eno2";
bond = [ "uplink" ];
};
};
};
networking.interfaces.uplink.ipv6.addresses = [
{ address = "2a01:584:11::1:${toString cfg.num}"; prefixLength = 64; }
];
networking.defaultGateway6 = { interface = "uplink"; address = "2a01:584:11::1"; };
deployment.targetHost = "2a01:584:11::1:${toString cfg.num}";
deployment.tags = [ "builders" ];
# Why can't we have nice things? https://bugs.openjdk.org/browse/JDK-8170568
@ -179,11 +117,5 @@ in
wantedBy = [ "timers.target" ];
};
systemd.timers.hydra-gc.timerConfig.Persistent = true;
bagel.sysadmin.enable = true;
environment.systemPackages = [ pkgs.ipmitool ];
system.stateVersion = "24.05";
};
}

View file

@ -0,0 +1,12 @@
{
imports = [
# Compute nodes
./builders
# Storage nodes
./storage
# Bases
./netboot.nix
./hardware.nix
];
}

View file

@ -0,0 +1,96 @@
{ pkgs, lib, config, ... }:
let
cfg = config.bagel.baremetal;
in
{
options = {
bagel.baremetal = {
enable = lib.mkEnableOption "baremetal bagel oven";
netboot = lib.mkEnableOption "netboot";
num = lib.mkOption {
type = lib.types.int;
};
};
};
config = lib.mkIf cfg.enable {
boot.initrd.availableKernelModules = [ "ahci" "ehci_pci" "usb_storage" "usbhid" "sd_mod" ];
boot.initrd.kernelModules = [ "dm-snapshot" ];
nixpkgs.hostPlatform = "x86_64-linux";
hardware.cpu.intel.updateMicrocode = true;
boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;
boot.initrd.systemd.enable = true;
boot.initrd.services.lvm.enable = true;
boot.kernel.sysctl."fs.xfs.xfssyncd_centisecs" = "12000";
fileSystems = lib.mkIf (!cfg.netboot) {
"/" = {
device = "/dev/disk/by-label/root";
fsType = "xfs";
};
"/boot" = {
device = "/dev/disk/by-label/BOOT";
fsType = "vfat";
options = [ "fmask=0022" "dmask=0022" ];
};
};
zramSwap = {
enable = true;
memoryPercent = 25;
};
boot.kernelParams = [
"console=tty1"
"console=ttyS0,115200"
];
networking.useNetworkd = true;
networking.domain = "wob01.infra.forkos.org";
systemd.network = {
netdevs = {
"40-uplink" = {
netdevConfig = {
Kind = "bond";
Name = "uplink";
};
bondConfig = {
Mode = "802.3ad";
TransmitHashPolicy = "layer3+4";
};
};
};
networks = {
"40-eno1" = {
name = "eno1";
bond = [ "uplink" ];
};
"40-eno2" = {
name = "eno2";
bond = [ "uplink" ];
};
};
};
networking.interfaces.uplink.ipv6.addresses = [
{ address = "2a01:584:11::1:${toString cfg.num}"; prefixLength = 64; }
];
networking.defaultGateway6 = { interface = "uplink"; address = "2a01:584:11::1"; };
bagel.infra.self.wan = {
family = "inet6";
address = "2a01:584:11::1:${toString cfg.num}";
prefixLength = 64;
};
deployment.targetHost = "2a01:584:11::1:${toString cfg.num}";
bagel.sysadmin.enable = true;
environment.systemPackages = [ pkgs.ipmitool ];
system.stateVersion = "24.05";
};
}

View file

@ -1,6 +1,6 @@
{ modulesPath, pkgs, lib, config, extendModules, ... }@node:
let
cfg = config.bagel.baremetal.builders;
cfg = config.bagel.baremetal;
in
{
config = lib.mkIf (cfg.enable && cfg.netboot) {

View file

@ -0,0 +1,19 @@
{ lib, config, ... }:
let
cfg = config.bagel.baremetal.storage;
in
{
options = {
bagel.baremetal.storage = {
enable = lib.mkEnableOption "storage role";
num = lib.mkOption {
type = lib.types.int;
};
};
};
config = lib.mkIf cfg.enable {
networking.hostName = "storage-${toString cfg.num}";
deployment.tags = [ "storage" ];
};
}

View file

@ -152,7 +152,9 @@ in
hydra_uri = cfg.hydraUrl;
binary_cache_uri = cfg.binaryCacheUrl;
base_git_uri_for_revision = cfg.baseUriForGitRevisions;
nixpkgs_dir = "/var/lib/channel-scripts/nixpkgs";
# TODO: this leaks information about where channel-scripts are hosted.
# Cleanup this later with a proper module option.
repo_dir = "/gerrit-data/channel-scripts/nixpkgs";
s3_release_bucket_name = cfg.s3.release;
s3_channel_bucket_name = cfg.s3.channel;
};
@ -177,6 +179,24 @@ in
};
script = "true";
};
"cleanup-failed-streaming-prefixes" = {
description = "Cleanup all failed streaming prefixes on the channel bucket (channel-scripts)";
conflicts = map (service: "${service.name}.service") updateJobs;
after = [ "networking.target" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = false;
User = "channel-scripts";
DynamicUser = true;
StateDirectory = "channel-scripts";
EnvironmentFile = [
cfg.releaseBucketCredentialsFile
];
Environment = cfg.extraEnvironment;
LoadCredential = [ "password:${config.age.secrets.alloy-push-password.path}" ];
ExecStart = "${cfg.package}/bin/mirror-forkos -c ${configFile} ${concatStringsSep " " cfg.extraArgs} cleanup-streamed-prefixes";
};
};
};
systemd.timers."update-all-channels" = {
@ -188,5 +208,14 @@ in
AccuracySec = 300;
};
};
systemd.timers."cleanup-failed-streaming-prefixes" = {
description = "Cleanup failed streaming prefixes for channel-scripts";
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "daily";
RandomizedDelaySec = "1h";
};
};
};
}

View file

@ -7,14 +7,16 @@
./matrix
./monitoring
./uptime-kuma
./self
./netbox
./ofborg
./postgres
./forgejo
./baremetal-builder
./baremetal
./buildbot
./newsletter
./s3-revproxy
./vault
./extra-builders
];
}

View file

@ -57,6 +57,7 @@ in
name = "git-gc-preserve-${name}";
value = {
description = "Git-GC-Preserve Service - ${name}";
path = [ pkgs.util-linux ];
serviceConfig = {
WorkingDirectory = gcConfig.repoPath;
Type = "oneshot";

View file

@ -71,6 +71,12 @@ in
description = "Working directory for the service";
};
stateDirectory = mkOption {
type = types.str;
default = "/var/lib/onewaysync";
description = "State directory where the copies of nixpkgs are stored";
};
pushUrl = mkOption {
type = types.str;
example = "ssh://...";
@ -119,6 +125,10 @@ in
};
config = mkIf cfg.enable {
fileSystems."/var/lib/onewaysync" = mkIf (cfg.stateDirectory != "/var/lib/onewaysync") {
device = cfg.stateDirectory;
options = [ "bind" ];
};
systemd.timers = mapAttrs' (name: value: {
name = "ows-${name}";
value = mkSyncTimer name value;

View file

@ -1,4 +1,3 @@
namespace: forkos
groups:
- name: ForkOS automation
rules:

View file

@ -1,4 +1,3 @@
namespace: postgres
groups:
- name: PostgreSQL
rules:

View file

@ -1,4 +1,3 @@
namespace: resources
groups:
- name: Host & hardware
rules:

View file

@ -12,14 +12,11 @@ let
alerts = pkgs.runCommand "mimir-alerts-checked" {
src = ./alerts;
nativeBuildInputs = with pkgs; [ mimir ];
nativeBuildInputs = with pkgs; [ prometheus.cli ];
} ''
promtool check rules $src/*
mkdir $out
cp -R $src $out/anonymous/
chmod -R +w $out
mimirtool rules check --rule-dirs=$out/anonymous
mimirtool rules lint --rule-dirs=$out/anonymous
diff -r $src $out/anonymous
'';
in
{

View file

@ -20,6 +20,9 @@ in
};
config = mkIf cfg.enable {
# FIXME: why
nixpkgs.config.permittedInsecurePackages = [ pkgs.netbox_3_7.name ];
age.secrets.netbox-environment.file = ../../secrets/floral/netbox-environment.age;
services = {
netbox = {

View file

@ -1,35 +1,151 @@
{ config, lib, ... }:
{ pkgs, config, lib, ... }:
let
inherit (lib) mkIf mkMerge optional hasAttr;
cfg = config.bagel.services.ofborg;
amqpHost = "amqp.forkos.org";
amqpPort = 5671;
generators = pkgs.formats.json { };
configFile = generators.generate "ofborg-config.json" config.bagel.services.ofborg.settings;
mkOfborgWorker = binaryName: extra: extra // {
wantedBy = [ "multi-user.target" ];
description = "ofborg CI service - ${binaryName} worker";
after = [ "rabbitmq.service" ];
serviceConfig = {
DynamicUser = true;
ExecStart = "${cfg.package}/bin/${binaryName} ${configFile}";
# TODO: more hardening.
StateDirectory = "ofborg";
LogsDirectory = "ofborg";
RuntimeDirectory = "ofborg";
WorkingDirectory = "/var/lib/ofborg";
LoadCredential =
optional (hasAttr "rabbitmq-password" config.age.secrets) "rabbitmq-password:${config.age.secrets.rabbitmq-password.path}"
++ optional (hasAttr "gerrit-event-listener-ssh-key" config.age.secrets) "gerrit-ssh-key:${config.age.secrets.gerrit-event-listener-ssh-key.path}";
Environment = [
"XDG_STATE_HOME=/run/ofborg"
];
};
};
in {
options.bagel.services.ofborg = with lib; {
enable = mkEnableOption "ofborg coordinator";
rabbitmq.enable = mkEnableOption "ofborg AMQP queue";
builder.enable = mkEnableOption "ofborg builder worker";
pastebin.enable = mkEnableOption "ofborg pastebin service";
statcheck-worker.enable = mkEnableOption "ofborg status & checks worker";
mass-rebuilder.enable = mkEnableOption "ofborg evaluator worker for mass rebuilds jobs";
stats.enable = mkEnableOption "ofborg prometheus worker";
gerrit-event-streamer.enable = mkEnableOption "ofborg's Gerrit event streamer";
gerrit-generic-vcs-filter.enable = mkEnableOption "ofborg's Gerrit event transformer to generic VCS events";
package = mkPackageOption pkgs "ofborg" { };
settings = mkOption {
type = generators.type;
};
};
config = lib.mkIf cfg.enable {
services.rabbitmq = {
enable = true;
configItems = {
"listeners.tcp" = "none";
"listeners.ssl.default" = builtins.toString amqpPort;
config = mkMerge [
{
# TODO: move this to global.
bagel.services.ofborg.settings = {
rabbitmq = {
ssl = true;
host = "amqp.forkos.org";
virtualhost = "/";
username = "ofborg";
password_file = "$CREDENTIALS_DIRECTORY/rabbitmq-password";
};
feedback.full_logs = lib.mkDefault true;
log_storage.path = lib.mkDefault "/var/log/ofborg";
runner = {
identity = config.networking.fqdn;
repos = lib.mkDefault [
"nixpkgs"
"ofborg"
];
"ssl_options.certfile" = "${config.security.acme.certs.${amqpHost}.directory}/cert.pem";
"ssl_options.keyfile" = "${config.security.acme.certs.${amqpHost}.directory}/key.pem";
disable_trusted_users = true;
};
checkout.root = lib.mkDefault "/var/lib/ofborg/checkouts";
nix = {
system = "x86_64-linux";
remote = "daemon";
build_timeout_seconds = 3600;
initial_heap_size = "4g";
};
pastebin = {
root = "$STATE_DIRECTORY/pastebins";
db = "$STATE_DIRECTORY/db.json";
};
statcheck = {
db = "$STATE_DIRECTORY/db.sqlite";
};
# We use Gerrit.
vcs = "Gerrit";
gerrit = {
instance_uri = "cl.forkos.org";
username = "ofborg-event-listener";
ssh_private_key_file = "$CREDENTIALS_DIRECTORY/gerrit-ssh-key";
ssh_port = 29418;
};
};
}
(mkIf cfg.rabbitmq.enable {
age.secrets.rabbitmq-password.file = ../../secrets/floral/rabbitmq-password.age;
services.nginx.enable = true;
services.rabbitmq = {
enable = true;
configItems = {
"listeners.tcp" = "none";
"listeners.ssl.default" = builtins.toString amqpPort;
"ssl_options.certfile" = "${config.security.acme.certs.${amqpHost}.directory}/cert.pem";
"ssl_options.keyfile" = "${config.security.acme.certs.${amqpHost}.directory}/key.pem";
};
};
};
security.acme.certs.${amqpHost} = {
webroot = "/var/lib/acme/.challenges";
group = "rabbitmq";
};
services.nginx.virtualHosts.${amqpHost}.locations."/.well-known/acme-challenge".root =
"/var/lib/acme/.challenges";
systemd.services.rabbitmq.requires = ["acme-finished-${amqpHost}.target"];
security.acme.certs.${amqpHost} = {
webroot = "/var/lib/acme/.challenges";
group = "rabbitmq";
};
services.nginx.virtualHosts.${amqpHost}.locations."/.well-known/acme-challenge".root =
"/var/lib/acme/.challenges";
systemd.services.rabbitmq.requires = ["acme-finished-${amqpHost}.target"];
networking.firewall.allowedTCPPorts = [ amqpPort ];
};
networking.firewall.allowedTCPPorts = [ amqpPort ];
})
(mkIf cfg.pastebin.enable {
systemd.services.ofborg-pastebin = mkOfborgWorker "pastebin-worker" { };
})
(mkIf cfg.statcheck-worker.enable {
systemd.services.ofborg-statcheck-worker = mkOfborgWorker "statcheck-worker" { };
})
(mkIf cfg.gerrit-event-streamer.enable {
age.secrets.gerrit-event-listener-ssh-key.file = ../../secrets/floral/gerrit-event-listener-ssh-key.age;
systemd.services.ofborg-gerrit-event-streamer = mkOfborgWorker "gerrit-event-streamer" {
path = [ pkgs.openssh ];
};
})
(mkIf cfg.gerrit-generic-vcs-filter.enable {
systemd.services.ofborg-gerrit-generic-vcs-filter = mkOfborgWorker "gerrit-generic-vcs-filter" { };
})
(mkIf cfg.mass-rebuilder.enable {
systemd.services.ofborg-mass-rebuilder = mkOfborgWorker "mass-rebuilder" { };
})
(mkIf cfg.builder.enable {
systemd.services.ofborg-builder = mkOfborgWorker "builder" { };
})
(mkIf cfg.stats.enable {
systemd.services.ofborg-stats = mkOfborgWorker "stats" { };
})
];
# systemd.services.ofborg-log-message-collector = {};
# systemd.services.ofborg-evaluation-filter = {};
# systemd.services.ofborg-vcs-comment-filter = {};
# systemd.services.ofborg-vcs-comment-poster = {};
}

21
services/self/default.nix Normal file
View file

@ -0,0 +1,21 @@
# This is a data-only module for other modules consumption.
{ lib, ... }:
let
inherit (lib) mkOption types;
in
{
options.bagel.infra.self = {
wan = {
family = mkOption {
type = types.enum [ "inet" "inet6" ];
default = "inet6";
};
address = mkOption {
type = types.str;
};
prefixLength = mkOption {
type = types.int;
};
};
};
}

View file

@ -0,0 +1,72 @@
{ config, lib, ... }:
let
cfg = config.bagel.services.vault;
inherit (lib) mkEnableOption mkOption mkIf concatStringsSep types;
mkPeerNode = fqdn: ''
retry_join {
leader_api_addr = "https://${fqdn}"
leader_tls_servername = "${fqdn}"
}
'';
wanAddress = if config.bagel.infra.self.wan.family == "inet6" then "[${config.bagel.infra.self.wan.address}]" else "${config.bagel.infra.self.wan.address}";
in
{
options.bagel.services.vault = {
enable = mkEnableOption "the OpenBao (Vault fork) service";
domain = mkOption {
type = types.str;
default = config.networking.fqdn;
defaultText = "config.networking.fqdn";
example = "vault.infra.forkos.org";
};
peers = mkOption {
type = types.listOf types.str;
default = [ ];
description = "List of FQDN that are peers of this service";
};
};
imports = [
./module.nix
];
config = mkIf cfg.enable {
networking.firewall.allowedTCPPorts = [
# NGINX HTTP API access
80
443
# mTLS backed cluster port
8201
];
services.nginx = {
enable = true;
recommendedProxySettings = true;
virtualHosts."${cfg.domain}" = {
enableACME = true;
forceSSL = true;
locations."/" = {
proxyPass = "http://127.0.0.1:8200";
};
};
};
services.openbao = {
enable = true;
storageBackend = "raft";
listenerExtraConfig = ''
cluster_address = "${wanAddress}:8201"
'';
storageConfig = ''
node_id = "${config.networking.fqdn}"
# Other nodes of the cluster.
${concatStringsSep "\n" (map mkPeerNode cfg.peers)}
'';
extraConfig = ''
cluster_addr = "http://${config.networking.fqdn}:8201"
api_addr = "https://${config.networking.fqdn}"
'';
};
};
}

237
services/vault/module.nix Normal file
View file

@ -0,0 +1,237 @@
{ config, lib, options, pkgs, ... }:
with lib;
let
cfg = config.services.openbao;
opt = options.services.openbao;
configFile = pkgs.writeText "openbao.hcl" ''
# vault in dev mode will refuse to start if its configuration sets listener
${lib.optionalString (!cfg.dev) ''
listener "tcp" {
address = "${cfg.address}"
${if (cfg.tlsCertFile == null || cfg.tlsKeyFile == null) then ''
tls_disable = "true"
'' else ''
tls_cert_file = "${cfg.tlsCertFile}"
tls_key_file = "${cfg.tlsKeyFile}"
''}
${cfg.listenerExtraConfig}
}
''}
storage "${cfg.storageBackend}" {
${optionalString (cfg.storagePath != null) ''path = "${cfg.storagePath}"''}
${optionalString (cfg.storageConfig != null) cfg.storageConfig}
}
${optionalString (cfg.telemetryConfig != "") ''
telemetry {
${cfg.telemetryConfig}
}
''}
${cfg.extraConfig}
'';
allConfigPaths = [configFile] ++ cfg.extraSettingsPaths;
configOptions = escapeShellArgs
(lib.optional cfg.dev "-dev" ++
lib.optional (cfg.dev && cfg.devRootTokenID != null) "-dev-root-token-id=${cfg.devRootTokenID}"
++ (concatMap (p: ["-config" p]) allConfigPaths));
in
{
options = {
services.openbao = {
enable = mkEnableOption "OpenBao daemon";
package = mkPackageOption pkgs "openbao" { };
dev = mkOption {
type = types.bool;
default = false;
description = ''
In this mode, the Vault runs in-memory and starts unsealed. This option is not meant production but for development and testing i.e. for nixos tests.
'';
};
devRootTokenID = mkOption {
type = types.nullOr types.str;
default = null;
description = ''
Initial root token. This only applies when {option}`services.vault.dev` is true
'';
};
address = mkOption {
type = types.str;
default = "127.0.0.1:8200";
description = "The name of the ip interface to listen to";
};
tlsCertFile = mkOption {
type = types.nullOr types.str;
default = null;
example = "/path/to/your/cert.pem";
description = "TLS certificate file. TLS will be disabled unless this option is set";
};
tlsKeyFile = mkOption {
type = types.nullOr types.str;
default = null;
example = "/path/to/your/key.pem";
description = "TLS private key file. TLS will be disabled unless this option is set";
};
listenerExtraConfig = mkOption {
type = types.lines;
default = ''
tls_min_version = "tls12"
'';
description = "Extra text appended to the listener section.";
};
storageBackend = mkOption {
type = types.enum [ "inmem" "file" "consul" "zookeeper" "s3" "azure" "dynamodb" "etcd" "mssql" "mysql" "postgresql" "swift" "gcs" "raft" ];
default = "inmem";
description = "The name of the type of storage backend";
};
storagePath = mkOption {
type = types.nullOr types.path;
default = if cfg.storageBackend == "file" || cfg.storageBackend == "raft" then "/var/lib/vault" else null;
defaultText = literalExpression ''
if config.${opt.storageBackend} == "file" || cfg.storageBackend == "raft"
then "/var/lib/vault"
else null
'';
description = "Data directory for file backend";
};
storageConfig = mkOption {
type = types.nullOr types.lines;
default = null;
description = ''
HCL configuration to insert in the storageBackend section.
Confidential values should not be specified here because this option's
value is written to the Nix store, which is publicly readable.
Provide credentials and such in a separate file using
[](#opt-services.vault.extraSettingsPaths).
'';
};
telemetryConfig = mkOption {
type = types.lines;
default = "";
description = "Telemetry configuration";
};
extraConfig = mkOption {
type = types.lines;
default = "";
description = "Extra text appended to {file}`vault.hcl`.";
};
extraSettingsPaths = mkOption {
type = types.listOf types.path;
default = [];
description = ''
Configuration files to load besides the immutable one defined by the NixOS module.
This can be used to avoid putting credentials in the Nix store, which can be read by any user.
Each path can point to a JSON- or HCL-formatted file, or a directory
to be scanned for files with `.hcl` or
`.json` extensions.
To upload the confidential file with NixOps, use for example:
```
# https://releases.nixos.org/nixops/latest/manual/manual.html#opt-deployment.keys
deployment.keys."vault.hcl" = let db = import ./db-credentials.nix; in {
text = ${"''"}
storage "postgresql" {
connection_url = "postgres://''${db.username}:''${db.password}@host.example.com/exampledb?sslmode=verify-ca"
}
${"''"};
user = "vault";
};
services.vault.extraSettingsPaths = ["/run/keys/vault.hcl"];
services.vault.storageBackend = "postgresql";
users.users.vault.extraGroups = ["keys"];
```
'';
};
};
};
config = mkIf cfg.enable {
nixpkgs.overlays = [ (self: super: {
openbao = super.callPackage ./package.nix { };
}) ];
environment.systemPackages = [
pkgs.openbao
];
assertions = [
{
assertion = cfg.storageBackend == "inmem" -> (cfg.storagePath == null && cfg.storageConfig == null);
message = ''The "inmem" storage expects no services.vault.storagePath nor services.vault.storageConfig'';
}
{
assertion = (
(cfg.storageBackend == "file" -> (cfg.storagePath != null && cfg.storageConfig == null)) &&
(cfg.storagePath != null -> (cfg.storageBackend == "file" || cfg.storageBackend == "raft"))
);
message = ''You must set services.vault.storagePath only when using the "file" or "raft" backend'';
}
];
users.users.openbao = {
name = "openbao";
group = "openbao";
uid = config.ids.uids.vault;
description = "OpenBao daemon user";
};
users.groups.openbao.gid = config.ids.gids.vault;
systemd.tmpfiles.rules = optional (cfg.storagePath != null)
"d '${cfg.storagePath}' 0700 openbao openbao - -";
systemd.services.openbao = {
description = "OpenBao server daemon";
wantedBy = ["multi-user.target"];
after = [ "network.target" ]
++ optional (config.services.consul.enable && cfg.storageBackend == "consul") "consul.service";
restartIfChanged = false; # do not restart on "nixos-rebuild switch". It would seal the storage and disrupt the clients.
startLimitIntervalSec = 60;
startLimitBurst = 3;
serviceConfig = {
User = "openbao";
Group = "openbao";
ExecStart = "${lib.getExe cfg.package} server ${configOptions}";
ExecReload = "${pkgs.coreutils}/bin/kill -SIGHUP $MAINPID";
StateDirectory = "vault";
# In `dev` mode vault will put its token here
Environment = lib.optional (cfg.dev) "HOME=/var/lib/vault";
PrivateDevices = true;
PrivateTmp = true;
ProtectSystem = "full";
ProtectHome = "read-only";
AmbientCapabilities = "cap_ipc_lock";
NoNewPrivileges = true;
LimitCORE = 0;
KillSignal = "SIGINT";
TimeoutStopSec = "30s";
Restart = "on-failure";
};
unitConfig.RequiresMountsFor = optional (cfg.storagePath != null) cfg.storagePath;
};
};
}

View file

@ -0,0 +1,51 @@
{ stdenv, lib, fetchFromGitHub, buildGoModule, installShellFiles, nixosTests
, makeWrapper
, gawk
, glibc
}:
buildGoModule rec {
pname = "openbao";
version = "2.1.0";
src = fetchFromGitHub {
owner = "openbao";
repo = "openbao";
rev = "v${version}";
hash = "sha256-QzUNb4T9uau9bWZX6ulUDyfdInGd86iClBAG72C+7mo=";
};
vendorHash = "sha256-ghy1gjo/bXoT+m8CGbWN4IXk4ywACR586UQf9F/azF8=";
subPackages = [ "." ];
nativeBuildInputs = [ installShellFiles makeWrapper ];
tags = [ "openbao" ];
ldflags = [
"-s" "-w"
"-X github.com/openbao/openbao/sdk/version.GitCommit=${src.rev}"
"-X github.com/openbao/openbao/sdk/version.Version=${version}"
"-X github.com/openbao/openbao/sdk/version.VersionPrerelease="
];
postInstall = ''
echo "complete -C $out/bin/openbao openbao" > openbao.bash
installShellCompletion openbao.bash
'' + lib.optionalString stdenv.isLinux ''
wrapProgram $out/bin/openbao \
--prefix PATH ${lib.makeBinPath [ gawk glibc ]}
'';
# passthru.tests = { inherit (nixosTests) vault vault-postgresql vault-dev vault-agent; };
meta = with lib; {
homepage = "https://openbao.org/";
description = "Tool for managing secrets";
changelog = "https://github.com/openbao/openbao/blob/v${version}/CHANGELOG.md";
license = licenses.mpl20;
mainProgram = "openbao";
maintainers = with maintainers; [ raitobezarius ];
};
}

View file

@ -1,7 +1,10 @@
{ lib, config, ... }:
{ nixosConfigurations, lib, config, ... }:
let
inherit (lib) mkEnableOption mkIf tf genList;
inherit (lib) mkEnableOption mkIf tf genList filterAttrs;
cfg = config.bagel.dnsimple;
filterBaremetalNodes = submodule: nodes: filterAttrs (name: node: node.config.bagel.baremetal.enable && node.config.bagel.baremetal.${submodule}.enable) nodes;
builderNodes = builtins.attrValues (filterBaremetalNodes "builders" nixosConfigurations);
storageNodes = builtins.attrValues (filterBaremetalNodes "storage" nixosConfigurations);
in
{
options.bagel.dnsimple = {
@ -101,9 +104,11 @@ in
# git.p.forkos.org is the proxy variant of the Forgejo server.
(record "git" 300 "CNAME" "git.p.forkos.org")
(record "netbox" 300 "CNAME" "meta01.infra.p.forkos.org")
(record "amqp" 300 "CNAME" "bagel-box.infra.p.forkos.org")
# It's not a public service, so no IPv4 for it.
(record "amqp" 300 "CNAME" "bagel-box.infra.forkos.org")
(record "grafana" 300 "CNAME" "meta01.infra.p.forkos.org")
(record "hydra" 300 "CNAME" "build-coord.wob01.infra.p.forkos.org")
(record "vault" 300 "CNAME" "build-coord.wob01.infra.p.forkos.org")
(record "loki" 300 "CNAME" "meta01.infra.p.forkos.org")
(record "mimir" 300 "CNAME" "meta01.infra.p.forkos.org")
(record "pyroscope" 300 "CNAME" "meta01.infra.p.forkos.org")
@ -125,7 +130,9 @@ in
(dualProxyRecords "build-coord.wob01.infra" 300 "AAAA" "2a01:584:11::1:11")
# TODO: do not hardcode, just reuse the Colmena hive module outputs to generate all the required details.
]
++ (map (index: record "builder-${toString index}.wob01.infra" 300 "AAAA" "2a01:584:11::1:${toString index}") (genList lib.id 11))
++ (map (index: record "bm-${toString index}.wob01.infra" 300 "AAAA" "2a01:584:11::1:${toString index}") (genList lib.id 11))
++ (map (node: record "builder-${toString node.config.bagel.baremetal.builders.num}.wob01.infra" 300 "CNAME" "bm-${toString node.config.bagel.baremetal.num}") builderNodes)
++ (map (node: record "storage-${toString node.config.bagel.baremetal.storage.num}.wob01.infra" 300 "CNAME" "bm-${toString node.config.bagel.baremetal.num}") storageNodes)
++ (
let
# FIXME: figure out a way to poke `config.services.s3-revproxy` and