Phpunit: Implement "previous failures first" mode

[sebastianbergmann]

The PHPUnit test runner should be enhanced to read a list of previously failed tests from an XML logfile and run these tests before the other tests.

[giorgiosironi]

Proposal: a command:
phpunit --run-failed
which reads the junit log file defined in phpunit.xml or from the passed --log-junit option.
The presence of the log file is a prerequisite for using the feature, because writing one by default would impact performance.

[sebastianbergmann]

This would have to be limited to tests that have no dependency (@depends) in the current architecture of PHPUnit.

[Petah]

Any updates?

[jstolp]

+1 any updates?

[reinholdfuereder]

IMHO starting from PHPUnit 7.3, because of https://github.com/sebastianbergmann/phpunit/pull/3147, this should be possible via --cache-result --order-by=defect

[epdenouden]

@reinholdfuereder Yes! That is exactly what the feature is for.

To help protect against unexpected breakages, you can activate the @depends checker to keep tests in as good an order as possible: --cache-result --order-by=depends,defects

[reinholdfuereder]

OK, great!

@sebastianbergmann So maybe this issue can be already resolved/closed as (mostly) fixed? (Although one or the other bug might be still around and documentation might need to be added/updated/completed; but that might be already tracked elsewhere, I guess...)

[epdenouden]

@reinholdfuereder The combi of functionality from #3147 + #3092 was explicitly aimed at providing this. I am not familiar with JUnit --run-failed and cannot find it in their docs. If anybody has a link that'd be great.

The only missing bit of functionality I still want myself is to only run failed and not just sorting defects to the front of a normal run. I'd have to implement a --filter based on 'status after the previous run(s)'. Totally up for doing that; will have a look when I'm settled at my new job.

[keradus]

I'm a bit 0/-1 on that functionality. While implement a fix for sth, it is good to not break anything else.
And if you know which exactly component is problematic and needs fixing, just filter to run tests only for it, it's already possible

[Petah]

FWIW you can pretty easily write a listener to rerun failed tests:

class OutputListener implements \PHPUnit\Framework\TestListener
{
    private $failures = [];

    public function addError(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        $this->failures[get_class($test)] = true;
    }

    public function addFailure(PHPUnit_Framework_Test $test, PHPUnit_Framework_AssertionFailedError $e, $time)
    {
        $this->failures[get_class($test)] = true;
    }

    public function endTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        $content = 'vendor/bin/phpunit --filter ' . implode('\|', array_keys($this->failures));
        $script = __DIR__ . '/../rerun-failed-tests.sh';
        file_put_contents($script, $content . PHP_EOL);
    }
}

[epdenouden]

@keradus
_As a_ software developer
_I want_ to rerun failed tests first THEN all tests
_so that_ I can have both a quick red-green cycle and the safety of full coverage

Sounds like something you'd want? :)

[keradus]

from opening message:

The PHPUnit test runner should be enhanced to read a list of previously failed tests from an XML logfile and run these tests before the other tests.

maybe not with xml, but with --order-by=defects we have this already possible.
imho, nothing left here and this feature request can be closed as done

[epdenouden]

Agreed :)

[lindt]

yes, should be solved with --cache-result --order-by=depends,defects.

[nickclasener]

Is it possible to set this in phpunit.xml?
-edit Never mind

cacheResult="true"
executionOrder="defects,reverse"

[GrahamCampbell]

And if you know which exactly component is problematic and needs fixing, just filter to run tests only for it, it's already possible

Can't filter on data providers though? I only wanna rerun number 624, and not the other two thousand...

[Petah]

@GrahamCampbell you should be able to filter on data providers:

Without:

$ ../bin/test.sh --filter "CurrencyRateTest::testNormalizeException"
~/work/project/laravel ~/work/project/laravel
Configuration cache cleared!
    PHPUnit 8.5.8 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.4.7 with Xdebug 2.9.5
Configuration: /Users/petah/work/project/laravel/phpunit.xml
Random seed:   1594163298


CurrencyRateTest testNormalizeException with data set #5.
CurrencyRateTest testNormalizeException with data set #4.
CurrencyRateTest testNormalizeException with data set #1.
CurrencyRateTest testNormalizeException with data set #3.
CurrencyRateTest testNormalizeException with data set #2.
CurrencyRateTest testNormalizeException with data set #0.                                                              6 / 6 (100%)

Time: 1.47 seconds, Memory: 38.00 MB

OK (6 tests, 12 assertions)

With:

$ ../bin/test.sh --filter "CurrencyRateTest::testNormalizeException.*2"
~/work/project/laravel ~/work/project/laravel
Configuration cache cleared!
    PHPUnit 8.5.8 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.4.7 with Xdebug 2.9.5
Configuration: /Users/petah/work/project/laravel/phpunit.xml
Random seed:   1594163303


CurrencyRateTest testNormalizeException with data set #2.                                                                   1 / 1 (100%)

Time: 1.01 seconds, Memory: 34.00 MB

OK (1 test, 2 assertions)

[GrahamCampbell]

Oh, cool. Thanks @Petah. 🍻