previously we immediately triggered all jobs, now we no longer do.
showing the total count at least somewhere is nice to have a rough
indication of how much longer a build may still need to run.
without this two builds can interfere with each other if:
- builds 1 and 2 start
- build 1 is starved of workers
- build 2 finishes, removes the shared gcroots directory
- gc runs
- build 1 schedules more builds whose .drvs have now been removed
using a dedicated directory for each build fixes this.
we now also need to set alwaysRun on the cleanup command or we risk
littering the system with stale gc roots when a build fails.
run all of them on the normal build worker. this significantly
simplifies the overall scheduler/builder config and removes a
triplication of possible builds paths.
retries don't help us very much, in fact they mostly hurt by repeating
builds that failed for non-transient reasons. retries could help with
workers dropping while running a build, but those rare cases are better
to restart manually than to pend at least twice the ci time for commits
that simply do not build cleanly.
eldritch horrors
puck