Codeception with codecoverage painfully slow

[grinsekatze003]

I hope to get some help here, because I've asked the same question on stackoverflow and got no feedback.

I am using codeception (with codecoverage) to check the code coverage of an application I have written using the Yii2 framework. Because the standard php installation on my mac has xcode not enabled, I activated it adding a zend_module line to my php.ini. Code coverage seems to work now but is painfully slow. Yes I know that the activated xdebug and coverage generation takes some time, but I think this is not normal: Even a simple unit test which checks only the initialization of an object takes up to 15 minutes.

I don't think that it is a cpu or ram problem rather than a configuration problem.

I start code coverage with:

codeception run unit --coverage-html

Things I detected: The first test runs always fast regardless how big it is. The second test is much slower (regardless what test it is) and the third is more slowly. How can I track this down? I want to detect the problem.

Again: I know that codecoverage slows down tests, but 15 minutes for a simple test is not normal. The test that runs approx. 15 minutes, takes 1 second without code-coverage enabled.

I am using the latest codeception version running on a macbook air.

[samdark]

It definitely should not be 15 minutes.

[PommeVerte]

I'm going to add that I have this issue as well. Tests run ok on a regular run but as soon as I add --coverage-xml it takes forever to complete. I'm currently having our CI run the tests for me, and am going to leave it running tonight and see what the results are tomorrow. (without coverage these tests take roughly 15-17 mn).
I've noticed that php (codeception) uses 100% of a cpu core while running these tests with coverage on, and basically stays there until it finishes.
What kind of a performance hit should be "expected" when generating coverage reports? Double the time?

[PommeVerte]

As an update, after running for over an hour and getting through 67% of the tests (that otherwise would take about 15-17mn to complete on lesser hardware) the task failed with an allowed memory size exhausted error.
The task was run right after codeception's installation so it should have been up to date. If you need any extra information I'll be glad to try and provide it.

EDIT : The same set of tests on our CI system took 3-4mn to run 100% of tests without coverage. So there is definitely an issue somewhere.

[rhertogh]

Edit: not relevant to this issue
I might have a related problem. Running the tests without coverage is working and fast, with coverage it's slow and it throws the following exception:

[yii\base\ErrorException]  
  Undefined index: B"

Exception trace:
 () at \app\vendor\phpunit\php-token-stream\src\Token\Stream.php:182
 PHP_Token_Stream->__construct() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage.php:653
 PHP_CodeCoverage->getLinesToBeIgnored() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage.php:536
 PHP_CodeCoverage->applyIgnoredLinesFilter() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage.php:288
 PHP_CodeCoverage->append() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage.php:592
 PHP_CodeCoverage->addUncoveredFilesFromWhitelist() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage.php:164
 PHP_CodeCoverage->getData() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage\Report\Factory.php:29
 PHP_CodeCoverage_Report_Factory->create() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage.php:130
 PHP_CodeCoverage->getReport() at \app\vendor\phpunit\php-code-coverage\src\CodeCoverage\Report\Text.php:56
 PHP_CodeCoverage_Report_Text->process() at \app\vendor\codeception\codeception\src\Codeception\Coverage\Subscriber\Printer.php:87
 Codeception\Coverage\Subscriber\Printer->printConsole() at \app\vendor\codeception\codeception\src\Codeception\Coverage\Subscriber\Printer.php:63
 Codeception\Coverage\Subscriber\Printer->printResult() at \app\vendor\symfony\event-dispatcher\Symfony\Component\EventDispatcher\EventDispatcher.php:164
 ::call_user_func:{\app\vendor\symfony\event-dispatcher\Symfony\Component\EventDispatcher\EventDispatcher.php:164}() at \app\vendor\symfony\event-dispatcher\Symfony\Component\EventDispatcher\EventDispatcher.php:164
 Symfony\Component\EventDispatcher\EventDispatcher->doDispatch() at \app\vendor\symfony\event-dispatcher\Symfony\Component\EventDispatcher\EventDispatcher.php:53
 Symfony\Component\EventDispatcher\EventDispatcher->dispatch() at \app\vendor\codeception\codeception\src\Codeception\Codecept.php:182
 Codeception\Codecept->printResult() at \app\vendor\codeception\codeception\src\Codeception\Command\Run.php:206
 Codeception\Command\Run->execute() at \app\vendor\symfony\console\Symfony\Component\Console\Command\Command.php:257
 Symfony\Component\Console\Command\Command->run() at \app\vendor\symfony\console\Symfony\Component\Console\Application.php:874
 Symfony\Component\Console\Application->doRunCommand() at \app\vendor\symfony\console\Symfony\Component\Console\Application.php:195
 Symfony\Component\Console\Application->doRun() at \app\vendor\symfony\console\Symfony\Component\Console\Application.php:126
 Symfony\Component\Console\Application->run() at \app\tests\run.php:26

[Ragazzo]

@rhertogh have you setup your code coverage as noted in yii2 - basic tests section ?

[rhertogh]

@Ragazzo Hi, thanks for pointing me there. We're using the advanced template and there the code coverage settings are not included.

[rhertogh]

I did tracked the slow code coverage down. In my case it was the coverage blacklist that was causing the slow down (and even 'Maximum execution time' exceptions).
It has probably to do with the performance of arrays in php. The class Codeception\Coverage\Filter spend more than 80% of the time in the function 'blackList' timing out at line 109 or 110:

foreach ($finder as $file) {
    $filter->addFileToBlacklist($file);
}

Workaround:
when possible don't use coverage->blacklist and/or coverage->whitelist->exclude in your codeception.yml (so basically only use coverage->whitelist->include)

[samdark]

@DavertMik looks like general issue.

[rhertogh]

@samdark Yes, it looks indeed like it. I haven't checked the full source, but wouldn't it performance wise be better to check the files from the coverage against the white- and black-list patterns in stead of first searching for all files and folders.

[friedlysol]

On one project tests run for about 18 minutes.

We added coverage-html to it, its been 5 hours and its still running. Plenty of memory and cores available. IO is minimial. The only solution I see is to run it in parallel.

Any other suggestions?

[jasny]

@friedlysol The workaround of @rhertogh works well: Add all the files that are subject to testing to coverage -> whitelist -> include and don't use a blacklist.

[sergeyklay]

@jasny +1

[neronmoon]

+1

[Arbuzov]

Get same problem when codeception try to iterate
https://github.com/Codeception/Codeception/blob/2.1/src/Codeception/Coverage/Filter.php#L79
If no whitelist and no black list so no codecoverage collected.

Maybe cache \PHP_CodeCoverage_Filter to codeception lanch?

[motin]

How many files are you including in the coverage report? There is a known issue with php-token-stream (used to generate code coverage reports) regarding memory management when many files are included: https://github.com/sebastianbergmann/php-token-stream/issues/60

[Naktibalda]

The code coverage blacklist functionality has been removed in PHPUnit 5.0 and PHP-Code-Coverage 3.0
https://github.com/sebastianbergmann/phpunit/blob/master/ChangeLog-5.0.md#500---2015-10-02

[ihorsamusenko]

Just faces the same issue, coverage takes five times slower then running tests without coverage.
And this is when I use such a config

coverage:
    enabled: true
    include:
        - /DependencyInjection/*

It means I include only couple of files.

[Naktibalda]

coverage takes five times slower then running tests without coverage. this is expected, coverage always makes tests take longer.

The original issue was about a single simple test taking 15 minutes.

[lisachenko]

Still actual... Running API tests under Docker is painfully slow, consumes a lot of CPU and each test can consume up to several seconds!

I don't know what is behind this, but this should be optimised as it brings a lot of pain and hurts CI pipeline where all these tests should run as fast as possible, not consuming too much resources...