Otherwise we risk tripping over confusing statuses where a build is "done" and "Cached", but we were expecting to run it.
This will let us run tests in parallel, and creates a more Perl-standard test development experience.