Tests types run in non-deterministic order

[gsnedders]

I feel like I reported this sometime before, but we're once again in a state where ./wpt run --metadata infrastructure/metadata --manifest MANIFEST.json PRODUCT infrastructure runs tests in a non-deterministic order, because we run the test types in a non-deterministic order.

We used to always guarantee a consistent ordering to avoid any risk of flakiness due to ordering, and this potentially is now worse with #40556 having landed.

[KyNorthstar]

I think guaranteeing deterministic order of test execution is a detriment.

There's the chance that some unintentional state change in test C might cause behavior in test N.

If they always run in the same order, then you might see passing tests and think all is good, but those tests might only be passing because test N's state is always set by test C. Conversely, N might be failing for reasons not apparent by just looking at N, or C might only succeed because of something A and B did(n't) do.

However, if you run them in random order every time, then it's vanishingly unlikely that unintentional cross-test state dependencies like this occur, and so tests which are affected by state set by other tests are much easier to pick out (usually because they pass/fail randomly), to the point that they are more or less guaranteed to never be committed, because the developer writing the test will catch that inconsistency locally before even creating a pull request.

If a test suite is flaky due to any kind of ordering (random, consistent, rorating, or otherwise), that is simply an indication that the test suite itself is flaky and needs to be revised to improve its robustness.