Infection: Perfectly passing test suite fails randomly when infection overlaid

[Bilge]

| Question | Answer
| ------------| ---------------
| Infection version | 0.13 - 0.14
| Test Framework version | PHPUnit 7.5.16
| PHP version | 7.1 - 7.3
| Platform | Ubuntu
| Github Repo | https://github.com/ScriptFUSION/Async-Throttle/tree/a70071e5728037215a928dcab86629dad98a1a91

See Travis builds [1] [2]. This test suite relies on precise timings to test behaviour. Despite this, the tests have a 100% success rate prior to overlaying infection. With infection, they fail most of the time when random timings bounds checks underflow expected limits. Although I have absolutely no idea how infection works internally, this is nevertheless surprising to me, as I would expect the overhead of running something over the top would take more time rather than less.

[sanmai]

Well, Infection tells you exactly what you need to do to work around these issues.

[Bilge]

I don't know where you got the idea that order matters. It does not. Timing matters. That's not the same thing. The timing is imperative _within each discrete test case_ but not between test cases.

[sanmai]

If you don't want or need that random behaviour of Infection (which is a good thing, and was very well discussed before), add executionOrder="default" to your phpunit.xml, and that's it.

But consider that these failures are a symptom of a greater problem, for one because Infection runs only tests that cover a specific line with a mutation. If your tests depend on each other, but that relationship is not declared with @depends, PHPUnit and Infection would not know that you also need to run other tests before running this test we need to run to test against a mutation. It is very important to have a consistently passing test suite with a random order.

[BackEndTea]

Making the following change to your project:

diff --git a/test/phpunit.xml b/test/phpunit.xml
index 0d9fc25..fb17095 100644
--- a/test/phpunit.xml
+++ b/test/phpunit.xml
@@ -1,5 +1,6 @@
 <phpunit
     beStrictAboutOutputDuringTests="true"
+    executionOrder="random"
 >
     <testsuite name="default">
         <directory>.</directory>

And then running the following:

$ vendor/bin/phpunit -c test --stop-on-failure --random-order-seed 1571380834

Results in this error:

PHPUnit 7.5.16 by Sebastian Bergmann and contributors.

Random seed:   1571380834

....F

Time: 2.09 seconds, Memory: 4.00 MB

There was 1 failure:

1) ScriptFUSIONTest\Async\Throttle\ThrottleTest::testConcurrency
Failed asserting that 1.0142247676849365 is less than 1.01.

/Users/gertdepagter/Projects/Async-Throttle/test/ThrottleTest.php:39
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Coroutine.php:105
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Internal/Placeholder.php:130
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Deferred.php:45
/Users/gertdepagter/Projects/Async-Throttle/src/Throttle.php:174
/Users/gertdepagter/Projects/Async-Throttle/src/Throttle.php:113
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Internal/Placeholder.php:130
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Delayed.php:22
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Loop/NativeDriver.php:121
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Loop/Driver.php:134
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Loop/Driver.php:72
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/amp/lib/Loop.php:84
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/phpunit-util/src/AsyncTestCase.php:64
/Users/gertdepagter/Projects/Async-Throttle/vendor/amphp/phpunit-util/src/AsyncTestCase.php:36

FAILURES!
Tests: 5, Assertions: 38, Failures: 1.

So it does look like there is an issue with the randomization of the tests. The problem does not appear to be with every seed, as the following seed ran without issues: 1571380854.

Now the problem here is with the testConcurrency test, which i think is the same one that failed in the travis infection run. Prehaps forcing that test to run in a separate process coudl fix the issue?

[maks-rafalko]

Closing as there is no issue in Infection, but in the project's tests suite.

[Bilge]

That's FALSE. Test suite passes perfectly, in any order, until infection
runs over the top. This issue is valid as originally stated.

On Wed, 13 Nov 2019, 14:45 Maks Rafalko, [e-mail] wrote:

Closing as there is no issue in Infection, but in the project's tests
suite.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/infection/infection/issues/807?email_source=notifications&email_token=AADS4YRYEEU45QUPHF545C3QTQHH5A5CNFSM4JB72CI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED6LZ7Q#issuecomment-553434366,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADS4YQTRPFD6M6TPJBJRS3QTQHH5ANCNFSM4JB72CIQ
.

[BackEndTea]

@Bilge

When i checked the project you linked, i found that the test suite did fail on random order, at least on certain seeds. see my previous comment: https://github.com/infection/infection/issues/807#issuecomment-543547480