diff --git a/README.md b/README.md index 523318a..dad2b2d 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,19 @@ -# grahamcofborg +# ofborg ## Guidelines -1. make sure you've reviewed the code before you trigger it on a PR - that isn't your own -2. be gentle, preferably don't run mass rebuilds / massive builds like - chromium on it +1. Review the code of all PRs before triggering the bot on them. +2. Be gentle; try not to run mass rebuilds or massive builds (like Chromium) on + it. ## Automatic Building -Users who are _trusted_ or _known_ (see: Trusted Users vs Known Users) -will have their PRs automatically trigger builds if their commits -follow the well-defined format of Nixpkgs. Specifically: prefixing the -commit title with the package attribute. This includes package bumps -as well as other changes. +All users will have their PRs automatically trigger builds if their commits +follow the well-defined format of Nixpkgs. Specifically: prefixing the commit +title with the package attribute. This includes package bumps as well as other +changes. -Example messages and the builds: +Example commit titles and the builds they will start: | Message | Automatic Build | |-----------------------------------------------------------------------|----------------------------------------------------------| @@ -24,211 +22,172 @@ Example messages and the builds: | `python36Packages.requests,python27Packages.requests: 1.0.0 -> 2.0.0` | `python36Packages.requests`, `python27Packages.requests` | | `python{2,3}Packages.requests: 1.0.0 -> 2.0.0` | _nothing_ | -If a PR is opened with many commits, it will create a single build job -for all of the detected packages. If a PR is opened and many commits -are pushed one by one to the open PR, many build jobs will be created. +When opening a PR with multiple commits, ofborg creates a single build job for +all detected packages. If multiple commits get pushed to a PR one-by-one, each +detected package will get a separate build job. -To disable automatic building of packages on a PR, add `[WIP]` to the -PR's title, or the `2.status: work-in-progress` label. +If the title of a PR begins with `WIP:`, contains `[WIP]` anywhere, or has the +`2.status: work-in-progress` label, its packages are not built automatically. +**Note**: Marking a PR as a draft does not prevent automatic builds. ## Commands -The comment parser is line-based, so comments can be interleaved with -instructions. +The comment parser is line-based, so commentary can be interwoven with +instructions for ofborg. -1. To trigger the bot, the line _must_ start with a case - insensitive version of `@GrahamcOfBorg`. -2. To use multiple commands, insert a bit of whitespace and then your - new command. +1. To trigger the bot, the line _must_ start with `@ofborg` (case insensitive). + * **Note**: GitHub will not suggest `@ofborg` to you, but it will work all + the same. When in doubt, preview your comment and verify that `@ofborg` + links to https://github.com/ofborg/. +2. To use multiple commands, separate them with whitespace. For examples, see + the "[Multiple Commands](#multiple-commands)" section. -Commands: - -### test (added: 2017-11-24) +### test ``` -@grahamcofborg test list of tests +@ofborg test list of tests ``` -This will run `nix-build ./nixos/release.nix -A tests.list -A tests.of -A tests.tests` in -the nixpkgs checkout. Note: this will only run on x86_64-linux machines. +This will run `nix-build ./nixos/release.nix -A tests.list -A tests.of -A +tests.tests` from the root of the Nixpkgs checkout. + +Tests will run on all allowed machines. For more information, see the "[Trusted +Users](#trusted-users)" section. ### eval ``` -@grahamcofborg eval +@ofborg eval ``` -Note: Every PR automatically evaluates when it is opened and when the -commits change. There is no reason to run eval on a PR unless the -evaluation has failed for weird reasons, or because master was broken -before. +See "[How does ofborg call +`nix-instantiate`?](#how-does-ofborg-call-nix-instantiate)" for what command(s) +this will run. + +**Note**: Every PR automatically evaluates both upon creation and when the +commits change. There is no reason to run eval on a PR unless the evaluation +failed for weird reasons or master was previously broken. ### build ``` -@grahamcofborg build list of attrs +@ofborg build list of attrs ``` -This will run `nix-build ./default.nix -A list -A of -A attrs` in -the nixpkgs checkout. +This will run `nix-build ./default.nix -A list -A of -A attrs` from the root of +the Nixpkgs checkout (see also "[How does ofborg call +`nix-build`?](#how-does-ofborg-call-nix-build)"). ---- +Builds will run on all allowed machines. For more information, see the "[Trusted +Users](#trusted-users)" section. +## Multiple Commands -Multiple Commands: +You can use multiple commands in a variety ways. Here are some valid +combinations: -``` -@grahamcofborg build list of attrs -@grahamcofborg eval -``` +* + ``` + @ofborg build list of attrs + @ofborg eval + ``` -or even: +* + ``` + @ofborg build list of attrs @ofborg eval + ``` -``` -@grahamcofborg build list of attrs @grahamcofborg eval -``` +* + ``` + looks good to me! + @ofborg eval + @ofborg build list of attrs + ``` -This will also work: +* + ``` + @ofborg eval + @ofborg build list of attrs + looks good to me! + ``` -``` -looks good to me! -@grahamcofborg build list of attrs -``` +* + ``` + @ofborg build list of attrs + @ofborg test list of attrs + ``` -And this is fine: +* This will build `list`, `of`, `attrs`, `looks`, `good`, `to`, and `me!` (which is probably not what you want): + ``` + @ofborg build list of attrs looks good to me! + ``` -``` -@grahamcofborg build list of attrs -looks good to me! -``` +## Trusted Users -This is will build `list`, `of`, `attrs`, `looks`, `good`, `to`, `me!`: +Trusted users have their builds and tests executed on _all_ available platforms, +including those without good sandboxing. Because this exposes the host to a +higher risk of security issues, the trusted users list consists of only +well-known, trusted members of the community. -``` -@grahamcofborg build list of attrs looks good to me! -``` - -## Trusted Users vs Known Users - -Known users have their builds executed on platforms with working -sandboxing. At the time of writing, that means: +At the time of writing, trusted users have their builds and tests run on these +platforms: - `x86_64-linux` - - `aarch64_linux` - -Trusted users have their builds executed on _all_ platforms, even if -they don't have good sandboxing. This opens the host up to a higher -risk of security issues, so only well known, trusted member of the -community should be added to the trusted user list. - -At the time of writing, trusted users have their builds run on the -following platforms: - - - `x86_64-linux` - - `aarch64_linux` + - `aarch64-linux` - `x86_64-darwin` -See ./config.public.json and ./config.known-users.json for a list of -all the trusted and known users. +See [`config.public.json`](./config.public.json) for a list of all trusted users. -# How does OfBorg call nix-build? +# How does ofborg call `nix-build`? -Builds are run like: +ofborg runs builds with a command similar to the following: -> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-build ./default.nix -> --no-out-link --keep-going -A hello -> --option restrict-eval true -> --option build-timeout 1800 -> --argstr system thesystem -> --show-trace +```shell +$ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-build ./default.nix \ + -A hello \ + --no-out-link \ + --keep-going \ + --option restrict-eval true \ + --option build-timeout 1800 \ + --argstr system thesystem \ + --show-trace +``` -# How does OfBorg call nix-instantiate? +# How does ofborg call `nix-instantiate`? -NixOS evals are run like: +ofborg runs NixOS evals with a command similar to the following: -> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate ./nixos/release.nix -> -A manual -> --option restrict-eval true -> --option build-timeout 1800 -> --argstr system thesystem -> --show-trace +```shell +$ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-instantiate ./nixos/release.nix \ + -A manual \ + --option restrict-eval true \ + --option build-timeout 1800 \ + --argstr system thesystem \ + --show-trace +``` -Nixpkgs evals are run like: - -> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate ./pkgs/top-level/release.nix -> -A manual -> --option restrict-eval true -> --option build-timeout 1800 -> --argstr system thesystem -> --show-trace +ofborg runs Nixpkgs evals with a command similar to the following: +```shell +$ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-instantiate ./pkgs/top-level/release.nix \ + -A manual \ + --option restrict-eval true \ + --option build-timeout 1800 \ + --argstr system thesystem \ + --show-trace +``` # Running meta checks locally -``` +To run the meta checks, you will need the +[`outpaths.nix`](./ofborg/src/outpaths.nix) file. You can acquire this file and +run the checks themselves like so: + +```shell $ curl -o outpaths.nix https://raw.githubusercontent.com/NixOS/ofborg/released/ofborg/src/outpaths.nix $ GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true > out-paths ``` ---- - - - - -# Running a builder - -It is recommended to create a special user for the ofBorg operation. -This user should have git configuration for default username/email for -non-interactive merges. For example: -``` -git config --global user.email "graham+cofborg@example.com" -git config --global user.name "GrahamCOfBorg" -``` - - -``` -nix-shell ./shell.nix -$ cd ofborg -$ cargo build -``` - -``` -cargo build -``` - -then copy example.config.json to config.json and edit its vars. Set -`nix.remote` to an empty string if you're not using the daemon. - -Run - -``` -./target/debug/builder ./config.json -``` - - -Note the config.public.json for the public pieces of how I run ofborg, -which is merged with config.known-users.json and a third private -config file of credentials. These files contain some special keys like - - - known users - - authorized users - - log storage - -they are only used in the backend processing tasks, and there is no -need for them on builders. However, to update the list in -config.known-users.json, run `./scripts/update-known-users.sh`. - -In case you have a non-trivial setup on Linux, make sure that the ofborg -user has access to `dev/kvm`, as it is needed for running tests. - -If you want to run multiple builder instances on the same physical -machine please make sure they use different configs with different -instance identity (same username/password is OK) and different -repository paths. Running two builders with the same config risks data -corruption. - -See also: https://github.com/NixOS/ofborg/wiki/Operating-a-Builder - # Hacking ```shell @@ -269,40 +228,7 @@ This will override the default of `-D warnings` set in [`shell.nix`](./shell.nix), which tells Rust to error if it detects any warnings. -## old php stuff... +# Running a builder -Only Graham needs to do this, since I run the only remaining PHP -components. - -```php - true, - 'verify_peer_name' => true, - 'peer_name' => $host, - 'verify_depth' => 10, - 'ca_file' => '/etc/ssl/certs/ca-certificates.crt', - ), array( - 'connection_timeout' => $timeout, - ) - ); - - return $connection; -} - -function gh_secret() { - return "github webhook secret"; -} - - -``` +If you want to run a builder of your own, check out the [wiki page on operating +a builder](https://github.com/NixOS/ofborg/wiki/Operating-a-Builder/). diff --git a/config.extra-known-users.json b/config.extra-known-users.json deleted file mode 100644 index fadd2ce..0000000 --- a/config.extra-known-users.json +++ /dev/null @@ -1,82 +0,0 @@ -[ - "0x4A6F", - "aethelz", - "aherrmann", - "anton-latukha", - "akru", - "alexarice", - "arcnmx", - "arianvp", - "artemist", - "balsoft", - "bbarker", - "bhipple", - "bignaux", - "brainrape", - "cdepillabout", - "costrouc", - "danieldk", - "delroth", - "devhell", - "d-goldin", - "Ekleog", - "ElvishJerricco", - "emilazy", - "Enzime", - "eqyiel", - "erictapen", - "erikarvstedt", - "etu", - "florianjacob", - "geistesk", - "gnidorah", - "grwlf", - "herrwiese", - "illegalprime", - "imalsogreg", - "ivan", - "jacereda", - "jfrankenau", - "jlesquembre", - "jluttine", - "johanot", - "johnazoidberg", - "kalbasit", - "knedlsepp", - "lopsided98", - "lschuermann", - "marsam", - "Moredread", - "nyanloutre", - "oxij", - "pacien", - "petabyteboy", - "pbogdan", - "pmiddend", - "pSub", - "rvolosatovs", - "risicle", - "samueldr", - "saschagrunert", - "SeTSeR", - "schmittlauch", - "Synthetica9", - "smaret", - "tadeokondrak", - "tadfisher", - "Taneb", - "teto", - "ThomasMader", - "tilpner", - "tokudan", - "tomberek", - "unode", - "va1entin", - "vdemeester", - "volth", - "woffs", - "xeji", - "xrelkd", - "yesbox", - "yorickvP" -] diff --git a/config.known-users.json b/config.known-users.json deleted file mode 100644 index 0bfbdee..0000000 --- a/config.known-users.json +++ /dev/null @@ -1,193 +0,0 @@ -{ - "runner": { - "known_users": [ - "7c6f434c", - "aanderse", - "abbradar", - "adisbladis", - "aforemny", - "akru", - "alyssais", - "aminechikhaoui", - "andersontorres", - "andir", - "arianvp", - "aristidb", - "armijnhemel", - "aszlig", - "aycanirican", - "basvandijk", - "bbarker", - "bendlas", - "benley", - "bennofs", - "bhipple", - "bignaux", - "bjornfor", - "bluescreen303", - "brainrape", - "c0bw3b", - "cdepillabout", - "chaoflow", - "cillianderoiste", - "cleverca22", - "copumpkin", - "costrouc", - "cpages", - "cstrahan", - "danbst", - "danieldk", - "dasJ", - "delroth", - "dezgeg", - "dguibert", - "disassembler", - "domenkozar", - "dotlambda", - "dtzwill", - "dywedir", - "d-goldin", - "edef1c", - "edolstra", - "edwtjo", - "ehmry", - "ekleog", - "ekleog", - "elvishjerricco", - "enzime", - "eqyiel", - "ericson2314", - "erictapen", - "erikarvstedt", - "etu", - "etu", - "falsifian", - "fgaz", - "flokli", - "florianjacob", - "fpletz", - "fridh", - "fuuzetsu", - "garbas", - "gebner", - "geistesk", - "gerschtli", - "globin", - "grahamc", - "grahamcofborg", - "grwlf", - "hedning", - "hrdinka", - "illegalprime", - "imalsogreg", - "infinisil", - "izorkin", - "jacereda", - "jagajaga", - "jfrankenau", - "jgeerds", - "jlesquembre", - "jluttine", - "joachifm", - "johanot", - "jokogr", - "jonringer", - "jtojnar", - "jwiegley", - "kalbasit", - "kalbasit", - "kevincox", - "knedlsepp", - "kosmikus", - "lheckemann", - "lilyball", - "lnl7", - "lovek323", - "lschuermann", - "lsix", - "ma27", - "madjar", - "maggesi", - "markuskowa", - "marsam", - "matejc", - "matthewbauer", - "mic92", - "moredread", - "mp2e", - "nbp", - "nckx", - "nequissimus", - "nh2", - "nicolaspetton", - "ninjatrappeur", - "nlewo", - "nyanloutre", - "obadz", - "ocharles", - "offlinehacker", - "orivej-nixos", - "oxij", - "pacien", - "pbogdan", - "peterhoeg", - "peti", - "phreedom", - "pikajude", - "pmiddend", - "primeos", - "profpatsch", - "psub", - "qknight", - "rasendubi", - "rbvermaa", - "rickynils", - "roberth", - "roconnor", - "rushmorem", - "rvolosatovs", - "ryantm", - "ryantrinkle", - "rycee", - "samueldr", - "samueldr", - "saschagrunert", - "shlevy", - "smaret", - "srhb", - "svanderburg", - "synthetica9", - "tadfisher", - "teto", - "the-kenny", - "thomasmader", - "thoughtpolice", - "tilpner", - "timokau", - "tomberek", - "ts468", - "ttuegel", - "unode", - "va1entin", - "vbgl", - "vcunat", - "vdemeester", - "veprbl", - "viric", - "volth", - "vrthra", - "wizeman", - "wkennington", - "wmertens", - "woffs", - "worldofpeace", - "wucke13", - "xeji", - "xeji", - "yegortimoshenko", - "yesbox", - "yurrriq", - "zimbatm" - ] - } -} diff --git a/scripts/update-known-users.sh b/scripts/update-known-users.sh deleted file mode 100755 index 3420700..0000000 --- a/scripts/update-known-users.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/usr/bin/env nix-shell -#!nix-shell -p bash -p jq -p curl -i bash - -readonly token=$(jq -r '.github.token' ./config.private.json) - -readonly dest=config.known-users.json -readonly scratch=user-list.scratch -readonly accumulator=user-list.accumulator -readonly result=user-list.result - -function fetch_users() { - curl \ - -H "Authorization: token $token" \ - "https://api.github.com/orgs/NixOS/members?page=$1" \ - | jq 'map(.login | ascii_downcase)' -} - -jq "map(ascii_downcase)" ./config.extra-known-users.json > "$accumulator" - -page=0 -while true; do - page=$((page + 1)) - fetch_users "$page" > "$scratch" - - jq -s '.[0] + .[1] | sort' "$accumulator" "$scratch" > "$result" - mv "$result" "$accumulator" - - if [ $(jq -r 'length' "$scratch") -eq 0 ]; then - break - fi -done - -jq -s '{ "runner": { "known_users": .[0]}}' "$accumulator" > "$dest" - -rm -f "$result" "$scratch" "$accumulator" - -jq -s '.[0] * .[1] * .[2]' \ - ./config.public.json \ - ./config.known-users.json \ - ./config.private.json > ./config.prod.json