Suitecrm: Test suite should be run in random order

[connorshea]

Issue

Essentially, this: https://qualitycoding.org/random-test-order/

Expected Behavior

The test suite should be run with a randomly generated seed to guarantee that tests don't rely on each other's state (essentially, if you assume A -> B -> C -> D, and C relies on something B adds to the database – unintentionally or otherwise – C will now fail when run in isolation but not when run as part of the test suite.)

Similarly, it can cause problems when introducing new tests that mess with the existent order and reveal that tests have been unintentionally reliant on state from other tests.

It also could potentially mean that a test doesn't _actually_ test the functionality, it's just been reliant on state that was introduced by a previous test (e.g. A creates a User and then doesn't get rid of it. B runs, and the test tries to create a User in a different way that doesn't work, but the database still has the User that A created, so B checks if there's a User in the db and passes despite the code being broken).

Actual Behavior

The tests are run without a seed, so they always run in the exact same order every time.

Possible Fix

The best fix would probably be to use Codeception's shuffle feature by setting shuffle: true in the codeception.yml file.

You may want to run the test suite a few times with shuffle: true to see if that brings up any existing state problems that would result in random test failures. Ultimately, I'd say this'd be a significant improvement over the current situation even if it may introduce flaky tests occasionally, since it's exposing an anti-pattern.

Steps to Reproduce

Look at Travis CI logs and note that the tests are always run in the same order, and that the test runner reports the seed as being blank.

[Dillon-Brown]

Very much agreed, I think this will take some time before we can implement it successfully though. Our Travis builds are failing pretty randomly as it is.

[connorshea]

Have you looked at the docs on risky tests? It seems like it could be useful to enable some of these to make sure the tests aren't messing with global scope and such: https://phpunit.de/manual/6.5/en/risky-tests.html

[lazka]

I'v tried to improve the phpunit tests for this in #7113.

Not motivated by random order, but because many tests failed if run separately and not as part of the whole test suite. But the solution is the same.