When multiple threads try to upsert the same file, this could fail. Fixes #4667.
Broken atm
Requires a slight update to the test infra to work properly, but having the possibility to group tests that way makes the whole thing quite cleaner imho