Phpunit: PHPUnit 8 consumes much more memory comparing PHPUnit 7

[isfedorov]

| Q | A
| --------------------| ---------------
| PHPUnit version | 8.4.2
| PHP version | 7.3.11
| Installation Method | Composer

Composer info:

| package | version | description
|-------------|------------|----------------
|doctrine/instantiator | 1.2.0 | A small, lightweight utility to instantiate objects in PHP without invoking their constructors
|myclabs/deep-copy | 1.9.3 |Create deep copies (clones) of your objects
|phar-io/manifest | 1.0.3 |Component for reading phar.io manifest information from a PHP Archive (PHAR)
|phar-io/version | 2.0.1 | Library for handling version information and constraints
|phpdocumentor/reflection-common | 2.0.0 |Common reflection classes used by phpdocumentor to reflect the code structure
|phpdocumentor/reflection-docblock| 4.3.2 | With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information that is embedded in a DocBlock.
|phpdocumentor/type-resolver | 1.0.1 | A PSR-5 based resolver of Class names, Types and Structural Element Names
|phpspec/prophecy | 1.9.0 | Highly opinionated mocking framework for PHP 5.3+
|phpunit/php-code-coverage | 7.0.8 | Library that provides collection, processing, and rendering functionality for PHP code coverage information.
|phpunit/php-file-iterator | 2.0.2 | FilterIterator implementation that filters files based on a list of suffixes.
|phpunit/php-text-template | 1.2.1 | Simple template engine.
|phpunit/php-timer | 2.1.2 | Utility class for timing
|phpunit/php-token-stream | 3.1.1 | Wrapper around PHP's tokenizer extension.
|phpunit/phpunit | 8.4.2 | The PHP Unit Testing framework.
|sebastian/code-unit-reverse-lookup | 1.0.1| Looks up which function or method a line of code belongs to
|sebastian/comparator | 3.0.2 | Provides the functionality to compare PHP values for equality
|sebastian/diff | 3.0.2 | Diff implementation
|sebastian/environment | 4.2.2 | Provides functionality to handle HHVM/PHP environments
|sebastian/exporter | 3.1.2 | Provides the functionality to export PHP variables for visualization
|sebastian/global-state | 3.0.0 | Snapshotting of global state
|sebastian/object-enumerator | 3.0.3 | Traverses array structures and object graphs to enumerate all referenced objects
|sebastian/object-reflector | 1.1.1 | Allows reflection of object attributes, including inherited and non-public ones
|sebastian/recursion-context | 3.0.0 | Provides functionality to recursively process PHP variables
|sebastian/resource-operations | 2.0.1 | Provides a list of PHP built-in functions that operate on resources
|sebastian/type | 1.1.3 | Collection of value objects that represent the types of the PHP type system
|sebastian/version | 2.0.1 | Library that helps with managing the version number of Git-hosted PHP projects
|symfony/polyfill-ctype | v1.12.0 | Symfony polyfill for ctype functions
|theseer/tokenizer | 1.1.3 | A small library for converting tokenized PHP source code into XML and potentially other formats
|webmozart/assert | 1.5.0 | Assertions to validate method input/output with nice error messages.

Summary

PhpUnit 8 consumes much more memory for test run comparing with PhpUnit 7.

Current behavior

We run the same set of tests with the same amount of assertions and see that PhpUnit 8 consumes ~30MB more than PhpUnit 7.

How to reproduce

Write code

<?php

use PHPUnit\Framework\TestCase;

class SomeTest extends TestCase
{
    /**
     * @dataProvider provider
     */
    public function testOne()
    {
        $this->assertTrue(true);
    }

    public function provider()
    {
        for ($a = 0; $a < 20000; $a++){
            yield "constant a${a}" => [$a];
        }
    }
}

And run test using PhpUnit 8.4.2 and PhpUnit 7.5.17. Results will be similar to

PHPUnit 7.5.17 by Sebastian Bergmann and contributors.

Time: 831 ms, Memory: 40.00 MB

OK (20000 tests, 20000 assertions)

and

PHPUnit 8.4.2 by Sebastian Bergmann and contributors.

Time: 941 ms, Memory: 56.76 MB

OK (20000 tests, 20000 assertions)

Expected behavior

It would be nice if PhpUnit 8 consumes not so much memory comparing with PhpUnit 7.

[brendt]

It also seems like memory is leaked throughout the test suite. I opened this issue in laravel/framework at first, because I thought it had to do with how Laravel bootstrapped its tests, though it seems like the issue also appears in non-laravel projects. The issue can be found here: https://github.com/laravel/framework/issues/30736

And here's is what I wrote initially

---

• Laravel Version: v6.6.0
• PHP Version: 7.4.0 and 7.3
• Database Driver & Version: sqlite 3.24.0 and MySQL 5.7.25

Description:

Recently we noticed our testsuite throwing out of memory errors, a problem that hadn't happened before. Doing a little debugging we found that each test adds around 2 MB of memory usage. Our test suite has around 1000 tests, and to have it fully run we need around 600 MB of memory allocated.

Around 80% of our test cases are "unit" tests, though they do interact with the database, 20% are integration tests that will simulate requests to controllers, using the built-in Laravel functionality.

We're using both the CreatesApplication and RefreshDatabase traits as provided by Laravel, and use an in-memory sqlite database, however for this issue I also ran our test suite on a MySQL database, where the same leak seems to happen.

I have looked at several older issues and stackoverflow questions, though none of them seem to provide a solution that works in this case.

Steps To Reproduce:

Add this teardown function in the base testcase, and observe.

    protected function tearDown(): void
    {
        parent::tearDown();

        echo memory_get_usage() . PHP_EOL;
    }

Output:

.57536280
.59370792
.63121592
.65270952
…
.83115760
.84632896
.85978784
.82666928
…

[skyrpex]

Can't provide a reproduction repo, but my test suite uses 14MB RAM in PHP 7.3, and it crashes because of exhausted memory in PHP 7.4. I use PHPUnit 8.5.2 in both scenarios.

[sebastianbergmann]

@skyrpex That would be a PHP 7.3 vs. PHP 7.4 resource consumption issue which is outside the scope of both this ticket in particular and PHPUnit in general. This ticket is about increased resource consumption with PHPUnit 8 compared to PHPUnit 7 while using the same PHP version.

[skyrpex]

@sebastianbergmann you're right, sorry about that.

[brendt]

@sebastianbergmann can you confirm the issue is within PHPUnit?

[sebastianbergmann]

I cannot confirm that as I was not able to reproduce this issue back in October.

Unless somebody provides a minimal, self-contained, reproducing test case that demonstrates an increased resource consumption in PHPUnit 8 vs. PHPUnit 7, I cannot and will not invest time in this.

If you can reproduce this but cannot 1) share the code or 2) reduce it to a minimal example, you could at least use a profiler and try to find out where in the code the increased resource consumption happens.

[brendt]

I'll provide a minimal setup later today!

[brendt]

@sebastianbergmann This is the simplest proof of concept I can come up with:

• https://github.com/brendt/phpunit-memory-leak/tree/phpunit-7
• https://github.com/brendt/phpunit-memory-leak/tree/phpunit-8
• https://github.com/brendt/phpunit-memory-leak/tree/phpunit-9

Setup:

• Clone the repository
• Run composer install
• Run vendor/bin/phpunit

The same leak happens in 7, 8 and 9. So maybe it's a PHP issue…

[sebastianbergmann]

You should be able to simplify this even more by not having multiple test_X method. Just have one test method and invoke PHPUnit with --repeat X.

[brendt]

TIL! thanks

[bytestream]

Anyone made any headway on this? I'm seeing similar with 2100 tests:

PHPUnit 8.5.2 - PHP 7.3 (latest) - Memory: 769.00 MB
PHPUnit 8.5.2 - PHP 7.4 (latest) - Memory: 4.16 GB

[sebastianbergmann]

PHPUnit 8.5.2 - PHP 7.3 (latest) - Memory: 769.00 MB
PHPUnit 8.5.2 - PHP 7.4 (latest) - Memory: 4.16 GB

If the same PHPUnit version consumes more resources on PHP 7.4 than on PHP 7.3 then this is not a PHPUnit issue but a PHP issue.

[sebastianbergmann]

The same leak happens in 7, 8 and 9. So maybe it's a PHP issue…

Just to make sure: running the tests in those repositories with PHP 7.4 consumes more memory than running them with PHP 7.3?

[YetiCGN]

@bytestream This might also be an issue with the PHP configuration or PHP extensions. For example, do you have xDebug activated in both installations? Are all other settings equal?

[coxlr]

I am seeing similar results is two large projects, both use MySQL in the testing suite.

I tested using PHPUnit 8.5.2 on each so it does seem to be a PHP issue with 7.4

PHP 7.2 - Memory: 530.00 MB
PHP 7.3 - Memory: 528.00 MB
PHP 7.4 - Memory: 2.84 GB

PHP 7.2 - Memory: 254.00 MB
PHP 7.3 - Memory: 250.00 MB
PHP 7.4 - Memory: 752.00 MB

[sebastianbergmann]

@brendt wrote in https://github.com/sebastianbergmann/phpunit/issues/3915#issuecomment-597060705:

This is the simplest proof of concept I can come up with:
* https://github.com/brendt/phpunit-memory-leak/tree/phpunit-7
* https://github.com/brendt/phpunit-memory-leak/tree/phpunit-8
* https://github.com/brendt/phpunit-memory-leak/tree/phpunit-9

I cannot reproduce the leak:

PHPUnit 7 with PHP 7.3, PHP 7.4, and PHP 8.0

$ git clone [email protected]:brendt/phpunit-memory-leak.git phpunit-7
$ cd phpunit-7
$ git checkout phpunit-7
$ composer install

$ php-73 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 24.86 seconds, Memory: 6.00 MB

$ php-74 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 24.2 seconds, Memory: 6.00 MB

$ php-80 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 24.01 seconds, Memory: 6.00 MB

PHPUnit 8 with PHP 7.3, PHP 7.4, and PHP 8.0

$ git clone [email protected]:brendt/phpunit-memory-leak.git phpunit-8
$ cd phpunit-8
$ git checkout phpunit-8
$ composer install

$ php-73 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 25.97 seconds, Memory: 6.00 MB

$ php-74 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 25.8 seconds, Memory: 6.00 MB

$ php-80 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 24.68 seconds, Memory: 6.00 MB

PHPUnit 9 with PHP 7.3, PHP 7.4, and PHP 8.0

$ git clone [email protected]:brendt/phpunit-memory-leak.git phpunit-9
$ cd phpunit-9
$ git checkout phpunit-9
$ composer install

$ php-73 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 24.68 seconds, Memory: 6.00 MB

$ php-74 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 21 seconds, Memory: 6.00 MB

$ php-80 ./vendor/bin/phpunit --repeat 10000|grep Memory
Time: 20.49 seconds, Memory: 6.00 MB

For all these combinations of PHP and PHPUnit versions, \memory_get_peak_usage(true) -- which is what PHPUnit looks at for its Memory: 6.00 MB output -- always reports the same memory consumption.

The PHP runtimes used did not have PCOV, Xdebug, OpCache, or any other extension loaded that is not required for PHPUnit to work.

[mfn]

or any other extension loaded

Maybe this is some key.

Also: what architecture is everyone on? Linux / Ubuntu?

What would the best output of the php binary get a unified value? just dumping php -i (biiiiig output)?

[brendt]

@sebastianbergmann using the --repeat option doesn't yield the same result though. Using --repeat there's no change in memory usage throughout tests, but using separate unit tests in the same process does.

[brendt]

Regarding extensions: no xdebug or any other "exotic" extensions installed. The only ones I've got specifically installed are imagick and redis, both are not used in my tests.

[bytestream]

@brendt @coxy121 have you tried generating a memory profile (callgrind / blackfire)? I'll have try if I get a chance.

[bytestream]

https://blackfire.io/profiles/c80f648b-a778-4b3c-8acf-00a628e1630f/graph

Does this profile give anyone any clues? I'm never quite sure how to use the memory profile to find the leaks... if someone could give any tips then I would be grateful.

[bytestream]

PHP 7.3 - https://blackfire.io/profiles/6b973d98-014a-4dd2-8b13-f0cf5842221c/graph

Comparison of PHP 7.3 and 7.4 (previous message) - https://blackfire.io/profiles/compare/a2dfa36c-3203-4599-959b-95d87143776a/graph

[sebastianbergmann]

I can only quote myself:

[A] PHP 7.3 vs. PHP 7.4 resource consumption issue [...] is outside the scope of both this ticket in particular and PHPUnit in general.

[bytestream]

I can appreciate that, but given several others are experiencing the same issue this ticket could serve as a good means to allow the community to help identify the problem. I could open a PHP bug if you prefer...

[staabm]

I could open a PHP bug if you prefer...

Thats the way to go when php 7.4 regressed on mem consumption

[CyrilMazur]

When I upgraded from php7.3 to php7.4 the only part of my application that saw a mem consumption increase is phpunit, the rest of the app (laravel) maintained the same consumption. Others can relate?

[bytestream]

https://bugs.php.net/bug.php?id=79519 if anyone can offer reproducers then please do

[bytestream]

@sebastianbergmann pretty sure this can be closed. Issue is in PHP 7.4 due to fixes to anonymous functions exacerbating a previously unfixed leak.

[pmochine]

@bytestream just to make the last sentence more understandable. There won't be any fixes and the only think what we can do is increase the memory limit or splitting the test into more sections? I just saw that even php 8.0 has still the same memory issues..

[bytestream]

@bytestream just to make the last sentence more understandable. There won't be any fixes and the only think what we can do is increase the memory limit or splitting the test into more sections? I just saw that even php 8.0 has still the same memory issues..

Pretty much or try reducing anonymous function usage