the-distro/ofborg
2
0
Fork 1
Code Issues 1 Pull requests 1 Packages Projects Releases Wiki Activity

README: "Not you: team."

Browse source 
Some of the information in the README is outdated, such as the sections
detailing the privileges of known users (everybody is now a known user).
The only distinction made is between that of trusted users and everybody
else. Trusted users gain an additional host to run builds and tests on:
x86_64-darwin. Everybody else is locked to x86_64-linux and
aarch64-linux.

With the removal of all mentions of "known users", the
`config.{extra-,}known-users.json` files have also been removed. This
change will require coordination with the infrastructure repo [1] --
namely, a PR removing references to these files (otherwise, I imagine,
deploys will fail).

Something Graham has mentioned a few times is how he wants to decouple
ofborg from his persona. Thus, `@ofborg` is now used in place of
`@GrahamcOfBorg` everywhere possible (it is still possible to use
`@GrahamcOfBorg`, so it has not been removed from the codebase at this
time).

[1] https://github.com/ofborg/infrastructure
This commit is contained in:
Cole Helbling 2020-05-06 00:13:34 -07:00
parent a2446b4ea1
commit b6862c2b49
4 changed files with 125 additions and 514 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

324
README.md
View file

@ -1,21 +1,19 @@
# grahamcofborg # ofborg
## Guidelines ## Guidelines
1. make sure you've reviewed the code before you trigger it on a PR 1. Review the code of all PRs before triggering the bot on them.
that isn't your own 2. Be gentle; try not to run mass rebuilds or massive builds (like Chromium) on
2. be gentle, preferably don't run mass rebuilds / massive builds like it.
chromium on it
## Automatic Building ## Automatic Building
Users who are _trusted_ or _known_ (see: Trusted Users vs Known Users) All users will have their PRs automatically trigger builds if their commits
will have their PRs automatically trigger builds if their commits follow the well-defined format of Nixpkgs. Specifically: prefixing the commit
follow the well-defined format of Nixpkgs. Specifically: prefixing the title with the package attribute. This includes package bumps as well as other
commit title with the package attribute. This includes package bumps changes.
as well as other changes.
Example messages and the builds: Example commit titles and the builds they will start:
| Message | Automatic Build | | Message | Automatic Build |
|-----------------------------------------------------------------------|----------------------------------------------------------| |-----------------------------------------------------------------------|----------------------------------------------------------|
@ -24,211 +22,172 @@ Example messages and the builds:
| `python36Packages.requests,python27Packages.requests: 1.0.0 -> 2.0.0` | `python36Packages.requests`, `python27Packages.requests` | | `python36Packages.requests,python27Packages.requests: 1.0.0 -> 2.0.0` | `python36Packages.requests`, `python27Packages.requests` |
| `python{2,3}Packages.requests: 1.0.0 -> 2.0.0` | _nothing_ | | `python{2,3}Packages.requests: 1.0.0 -> 2.0.0` | _nothing_ |
If a PR is opened with many commits, it will create a single build job When opening a PR with multiple commits, ofborg creates a single build job for
for all of the detected packages. If a PR is opened and many commits all detected packages. If multiple commits get pushed to a PR one-by-one, each
are pushed one by one to the open PR, many build jobs will be created. detected package will get a separate build job.
To disable automatic building of packages on a PR, add `[WIP]` to the If the title of a PR begins with `WIP:`, contains `[WIP]` anywhere, or has the
PR's title, or the `2.status: work-in-progress` label. `2.status: work-in-progress` label, its packages are not built automatically.
**Note**: Marking a PR as a draft does not prevent automatic builds.
## Commands ## Commands
The comment parser is line-based, so comments can be interleaved with The comment parser is line-based, so commentary can be interwoven with
instructions. instructions for ofborg.
1. To trigger the bot, the line _must_ start with a case 1. To trigger the bot, the line _must_ start with `@ofborg` (case insensitive).
insensitive version of `@GrahamcOfBorg`. * **Note**: GitHub will not suggest `@ofborg` to you, but it will work all
2. To use multiple commands, insert a bit of whitespace and then your the same. When in doubt, preview your comment and verify that `@ofborg`
new command. links to https://github.com/ofborg/.
2. To use multiple commands, separate them with whitespace. For examples, see
the "[Multiple Commands](#multiple-commands)" section.
Commands: ### test
### test (added: 2017-11-24)
``` ```
@grahamcofborg test list of tests @ofborg test list of tests
``` ```
This will run `nix-build ./nixos/release.nix -A tests.list -A tests.of -A tests.tests` in This will run `nix-build ./nixos/release.nix -A tests.list -A tests.of -A
the nixpkgs checkout. Note: this will only run on x86_64-linux machines. tests.tests` from the root of the Nixpkgs checkout.
Tests will run on all allowed machines. For more information, see the "[Trusted
Users](#trusted-users)" section.
### eval ### eval
``` ```
@grahamcofborg eval @ofborg eval
``` ```
Note: Every PR automatically evaluates when it is opened and when the See "[How does ofborg call
commits change. There is no reason to run eval on a PR unless the `nix-instantiate`?](#how-does-ofborg-call-nix-instantiate)" for what command(s)
evaluation has failed for weird reasons, or because master was broken this will run.
before.
**Note**: Every PR automatically evaluates both upon creation and when the
commits change. There is no reason to run eval on a PR unless the evaluation
failed for weird reasons or master was previously broken.
### build ### build
``` ```
@grahamcofborg build list of attrs @ofborg build list of attrs
``` ```
This will run `nix-build ./default.nix -A list -A of -A attrs` in This will run `nix-build ./default.nix -A list -A of -A attrs` from the root of
the nixpkgs checkout. the Nixpkgs checkout (see also "[How does ofborg call
`nix-build`?](#how-does-ofborg-call-nix-build)").
--- Builds will run on all allowed machines. For more information, see the "[Trusted
Users](#trusted-users)" section.
## Multiple Commands
Multiple Commands: You can use multiple commands in a variety ways. Here are some valid
combinations:
``` *
@grahamcofborg build list of attrs ```
@grahamcofborg eval @ofborg build list of attrs
``` @ofborg eval
```
or even: *
```
@ofborg build list of attrs @ofborg eval
```
``` *
@grahamcofborg build list of attrs @grahamcofborg eval ```
``` looks good to me!
@ofborg eval
@ofborg build list of attrs
```
This will also work: *
```
@ofborg eval
@ofborg build list of attrs
looks good to me!
```
``` *
looks good to me! ```
@grahamcofborg build list of attrs @ofborg build list of attrs
``` @ofborg test list of attrs
```
And this is fine: * This will build `list`, `of`, `attrs`, `looks`, `good`, `to`, and `me!` (which is probably not what you want):
```
@ofborg build list of attrs looks good to me!
```
``` ## Trusted Users
@grahamcofborg build list of attrs
looks good to me!
```
This is will build `list`, `of`, `attrs`, `looks`, `good`, `to`, `me!`: Trusted users have their builds and tests executed on _all_ available platforms,
including those without good sandboxing. Because this exposes the host to a
higher risk of security issues, the trusted users list consists of only
well-known, trusted members of the community.
``` At the time of writing, trusted users have their builds and tests run on these
@grahamcofborg build list of attrs looks good to me! platforms:
```
## Trusted Users vs Known Users
Known users have their builds executed on platforms with working
sandboxing. At the time of writing, that means:
- `x86_64-linux` - `x86_64-linux`
- `aarch64_linux` - `aarch64-linux`
Trusted users have their builds executed on _all_ platforms, even if
they don't have good sandboxing. This opens the host up to a higher
risk of security issues, so only well known, trusted member of the
community should be added to the trusted user list.
At the time of writing, trusted users have their builds run on the
following platforms:
- `x86_64-linux`
- `aarch64_linux`
- `x86_64-darwin` - `x86_64-darwin`
See ./config.public.json and ./config.known-users.json for a list of See [`config.public.json`](./config.public.json) for a list of all trusted users.
all the trusted and known users.
# How does OfBorg call nix-build? # How does ofborg call `nix-build`?
Builds are run like: ofborg runs builds with a command similar to the following:
> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-build ./default.nix ```shell
> --no-out-link --keep-going -A hello $ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-build ./default.nix \
> --option restrict-eval true -A hello \
> --option build-timeout 1800 --no-out-link \
> --argstr system thesystem --keep-going \
> --show-trace --option restrict-eval true \
--option build-timeout 1800 \
--argstr system thesystem \
--show-trace
```
# How does OfBorg call nix-instantiate? # How does ofborg call `nix-instantiate`?
NixOS evals are run like: ofborg runs NixOS evals with a command similar to the following:
> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate ./nixos/release.nix ```shell
> -A manual $ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-instantiate ./nixos/release.nix \
> --option restrict-eval true -A manual \
> --option build-timeout 1800 --option restrict-eval true \
> --argstr system thesystem --option build-timeout 1800 \
> --show-trace --argstr system thesystem \
--show-trace
```
Nixpkgs evals are run like: ofborg runs Nixpkgs evals with a command similar to the following:
> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate ./pkgs/top-level/release.nix
> -A manual
> --option restrict-eval true
> --option build-timeout 1800
> --argstr system thesystem
> --show-trace
```shell
$ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-instantiate ./pkgs/top-level/release.nix \
-A manual \
--option restrict-eval true \
--option build-timeout 1800 \
--argstr system thesystem \
--show-trace
```
# Running meta checks locally # Running meta checks locally
``` To run the meta checks, you will need the
[`outpaths.nix`](./ofborg/src/outpaths.nix) file. You can acquire this file and
run the checks themselves like so:
```shell
$ curl -o outpaths.nix https://raw.githubusercontent.com/NixOS/ofborg/released/ofborg/src/outpaths.nix $ curl -o outpaths.nix https://raw.githubusercontent.com/NixOS/ofborg/released/ofborg/src/outpaths.nix
$ GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true > out-paths $ GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true > out-paths
``` ```
---
# Running a builder
It is recommended to create a special user for the ofBorg operation.
This user should have git configuration for default username/email for
non-interactive merges. For example:
```
git config --global user.email "graham+cofborg@example.com"
git config --global user.name "GrahamCOfBorg"
```
```
nix-shell ./shell.nix
$ cd ofborg
$ cargo build
```
```
cargo build
```
then copy example.config.json to config.json and edit its vars. Set
`nix.remote` to an empty string if you're not using the daemon.
Run
```
./target/debug/builder ./config.json
```
Note the config.public.json for the public pieces of how I run ofborg,
which is merged with config.known-users.json and a third private
config file of credentials. These files contain some special keys like
- known users
- authorized users
- log storage
they are only used in the backend processing tasks, and there is no
need for them on builders. However, to update the list in
config.known-users.json, run `./scripts/update-known-users.sh`.
In case you have a non-trivial setup on Linux, make sure that the ofborg
user has access to `dev/kvm`, as it is needed for running tests.
If you want to run multiple builder instances on the same physical
machine please make sure they use different configs with different
instance identity (same username/password is OK) and different
repository paths. Running two builders with the same config risks data
corruption.
See also: https://github.com/NixOS/ofborg/wiki/Operating-a-Builder
# Hacking # Hacking
```shell ```shell
@ -269,40 +228,7 @@ This will override the default of `-D warnings` set in
[`shell.nix`](./shell.nix), which tells Rust to error if it detects any [`shell.nix`](./shell.nix), which tells Rust to error if it detects any
warnings. warnings.
## old php stuff... # Running a builder
Only Graham needs to do this, since I run the only remaining PHP If you want to run a builder of your own, check out the [wiki page on operating
components. a builder](https://github.com/NixOS/ofborg/wiki/Operating-a-Builder/).
```php
<?php
require_once __DIR__ . '/vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPSSLConnection;
use PhpAmqpLib\Message\AMQPMessage;
function rabbitmq_conn($timeout = 3) {
$host = 'events.nix.gsc.io';
$connection = new AMQPSSLConnection(
$host, 5671,
'eventsuser, eventspassword, '/',
array(
'verify_peer' => true,
'verify_peer_name' => true,
'peer_name' => $host,
'verify_depth' => 10,
'ca_file' => '/etc/ssl/certs/ca-certificates.crt',
), array(
'connection_timeout' => $timeout,
)
);
return $connection;
}
function gh_secret() {
return "github webhook secret";
}
```

82
config.extra-known-users.json
View file

@ -1,82 +0,0 @@
[
"0x4A6F",
"aethelz",
"aherrmann",
"anton-latukha",
"akru",
"alexarice",
"arcnmx",
"arianvp",
"artemist",
"balsoft",
"bbarker",
"bhipple",
"bignaux",
"brainrape",
"cdepillabout",
"costrouc",
"danieldk",
"delroth",
"devhell",
"d-goldin",
"Ekleog",
"ElvishJerricco",
"emilazy",
"Enzime",
"eqyiel",
"erictapen",
"erikarvstedt",
"etu",
"florianjacob",
"geistesk",
"gnidorah",
"grwlf",
"herrwiese",
"illegalprime",
"imalsogreg",
"ivan",
"jacereda",
"jfrankenau",
"jlesquembre",
"jluttine",
"johanot",
"johnazoidberg",
"kalbasit",
"knedlsepp",
"lopsided98",
"lschuermann",
"marsam",
"Moredread",
"nyanloutre",
"oxij",
"pacien",
"petabyteboy",
"pbogdan",
"pmiddend",
"pSub",
"rvolosatovs",
"risicle",
"samueldr",
"saschagrunert",
"SeTSeR",
"schmittlauch",
"Synthetica9",
"smaret",
"tadeokondrak",
"tadfisher",
"Taneb",
"teto",
"ThomasMader",
"tilpner",
"tokudan",
"tomberek",
"unode",
"va1entin",
"vdemeester",
"volth",
"woffs",
"xeji",
"xrelkd",
"yesbox",
"yorickvP"
]

193
config.known-users.json
View file

@ -1,193 +0,0 @@
{
"runner": {
"known_users": [
"7c6f434c",
"aanderse",
"abbradar",
"adisbladis",
"aforemny",
"akru",
"alyssais",
"aminechikhaoui",
"andersontorres",
"andir",
"arianvp",
"aristidb",
"armijnhemel",
"aszlig",
"aycanirican",
"basvandijk",
"bbarker",
"bendlas",
"benley",
"bennofs",
"bhipple",
"bignaux",
"bjornfor",
"bluescreen303",
"brainrape",
"c0bw3b",
"cdepillabout",
"chaoflow",
"cillianderoiste",
"cleverca22",
"copumpkin",
"costrouc",
"cpages",
"cstrahan",
"danbst",
"danieldk",
"dasJ",
"delroth",
"dezgeg",
"dguibert",
"disassembler",
"domenkozar",
"dotlambda",
"dtzwill",
"dywedir",
"d-goldin",
"edef1c",
"edolstra",
"edwtjo",
"ehmry",
"ekleog",
"ekleog",
"elvishjerricco",
"enzime",
"eqyiel",
"ericson2314",
"erictapen",
"erikarvstedt",
"etu",
"etu",
"falsifian",
"fgaz",
"flokli",
"florianjacob",
"fpletz",
"fridh",
"fuuzetsu",
"garbas",
"gebner",
"geistesk",
"gerschtli",
"globin",
"grahamc",
"grahamcofborg",
"grwlf",
"hedning",
"hrdinka",
"illegalprime",
"imalsogreg",
"infinisil",
"izorkin",
"jacereda",
"jagajaga",
"jfrankenau",
"jgeerds",
"jlesquembre",
"jluttine",
"joachifm",
"johanot",
"jokogr",
"jonringer",
"jtojnar",
"jwiegley",
"kalbasit",
"kalbasit",
"kevincox",
"knedlsepp",
"kosmikus",
"lheckemann",
"lilyball",
"lnl7",
"lovek323",
"lschuermann",
"lsix",
"ma27",
"madjar",
"maggesi",
"markuskowa",
"marsam",
"matejc",
"matthewbauer",
"mic92",
"moredread",
"mp2e",
"nbp",
"nckx",
"nequissimus",
"nh2",
"nicolaspetton",
"ninjatrappeur",
"nlewo",
"nyanloutre",
"obadz",
"ocharles",
"offlinehacker",
"orivej-nixos",
"oxij",
"pacien",
"pbogdan",
"peterhoeg",
"peti",
"phreedom",
"pikajude",
"pmiddend",
"primeos",
"profpatsch",
"psub",
"qknight",
"rasendubi",
"rbvermaa",
"rickynils",
"roberth",
"roconnor",
"rushmorem",
"rvolosatovs",
"ryantm",
"ryantrinkle",
"rycee",
"samueldr",
"samueldr",
"saschagrunert",
"shlevy",
"smaret",
"srhb",
"svanderburg",
"synthetica9",
"tadfisher",
"teto",
"the-kenny",
"thomasmader",
"thoughtpolice",
"tilpner",
"timokau",
"tomberek",
"ts468",
"ttuegel",
"unode",
"va1entin",
"vbgl",
"vcunat",
"vdemeester",
"veprbl",
"viric",
"volth",
"vrthra",
"wizeman",
"wkennington",
"wmertens",
"woffs",
"worldofpeace",
"wucke13",
"xeji",
"xeji",
"yegortimoshenko",
"yesbox",
"yurrriq",
"zimbatm"
]
}
}

40
scripts/update-known-users.sh
View file

@ -1,40 +0,0 @@
#!/usr/bin/env nix-shell
#!nix-shell -p bash -p jq -p curl -i bash
readonly token=$(jq -r '.github.token' ./config.private.json)
readonly dest=config.known-users.json
readonly scratch=user-list.scratch
readonly accumulator=user-list.accumulator
readonly result=user-list.result
function fetch_users() {
curl \
-H "Authorization: token $token" \
"https://api.github.com/orgs/NixOS/members?page=$1" \
| jq 'map(.login | ascii_downcase)'
}
jq "map(ascii_downcase)" ./config.extra-known-users.json > "$accumulator"
page=0
while true; do
page=$((page + 1))
fetch_users "$page" > "$scratch"
jq -s '.[0] + .[1] | sort' "$accumulator" "$scratch" > "$result"
mv "$result" "$accumulator"
if [ $(jq -r 'length' "$scratch") -eq 0 ]; then
break
fi
done
jq -s '{ "runner": { "known_users": .[0]}}' "$accumulator" > "$dest"
rm -f "$result" "$scratch" "$accumulator"
jq -s '.[0] * .[1] * .[2]' \
./config.public.json \
./config.known-users.json \
./config.private.json > ./config.prod.json