Phpunit: Empty test with runInSeparateProcess fails after test dealing with output buffering

[samdark]

| Q | A
| --------------------| ---------------
| PHPUnit version | 8.5-gf2eeaa3cd, 8.4.3, 7
| PHP version | 7.2.20
| Installation Method | Composer

$ composer info | sort
doctrine/instantiator              1.3.x-dev ae466f7  A small, lightweight utility to instantiate objects in PHP without invoking their constructors
myclabs/deep-copy                  1.x-dev 9012edb    Create deep copies (clones) of your objects
phar-io/manifest                   dev-master 6008a32 Component for reading phar.io manifest information from a PHP Archive (PHAR)
phar-io/version                    2.0.1              Library for handling version information and constraints
phpdocumentor/reflection-common    2.0.0              Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock  dev-master 8fcadfe With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information that is embedded in a DocBlock.
phpdocumentor/type-resolver        0.7.x-dev 6f6f66c  A PSR-5 based resolver of Class names, Types and Structural Element Names
phpspec/prophecy                   1.9.0              Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage          dev-master f188418 Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator          dev-master ee5d93c FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template          1.2.1              Simple template engine.
phpunit/php-timer                  dev-master 190db5a Utility class for timing
phpunit/php-token-stream           dev-master 5bc2030 Wrapper around PHP's tokenizer extension.
phpunit/phpunit                    dev-master f2eeaa3 The PHP Unit Testing framework.
sebastian/code-unit-reverse-lookup dev-master 5872e37 Looks up which function or method a line of code belongs to
sebastian/comparator               dev-master c57ba51 Provides the functionality to compare PHP values for equality
sebastian/diff                     dev-master 16e54fb Diff implementation
sebastian/environment              dev-master 464c90d Provides functionality to handle HHVM/PHP environments
sebastian/exporter                 dev-master 10b761a Provides the functionality to export PHP variables for visualization
sebastian/global-state             dev-master 95e10dd Snapshotting of global state
sebastian/object-enumerator        dev-master 6096279 Traverses array structures and object graphs to enumerate all referenced objects
sebastian/object-reflector         dev-master aff2a6b Allows reflection of object attributes, including inherited and non-public ones
sebastian/recursion-context        dev-master f95dcff Provides functionality to recursively process PHP variables
sebastian/resource-operations      dev-master 8989397 Provides a list of PHP built-in functions that operate on resources
sebastian/type                     dev-master b4b5b44 Collection of value objects that represent the types of the PHP type system
sebastian/version                  2.0.1              Library that helps with managing the version number of Git-hosted PHP projects
symfony/polyfill-ctype             dev-master 550ebaa Symfony polyfill for ctype functions
theseer/tokenizer                  1.1.3              A small library for converting tokenized PHP source code into XML and potentially other formats
webmozart/assert                   1.5.0              Assertions to validate method input/output with nice error messages.

Summary

Empty test with @runInSeparateProcess fails if there was a test about output buffering before it.

Current behavior

Second empty test errors:

$ ./vendor/bin/phpunit
PHPUnit 8.5-gf2eeaa3cd by Sebastian Bergmann and contributors.

Runtime:       PHP 7.2.20 with Xdebug 2.7.2
Configuration: D:\dev\phpunit_runInSeparateProcess_bug\phpunit.xml.dist

.E                                                                  2 / 2 (100%)

Time: 274 ms, Memory: 6.00 MB

There was 1 error:

1) Samdark\Tests\BTest::testDummy
PHPUnit\Framework\Exception: Fatal error: Uncaught Error: Using $this when not in object context in D:\dev\phpunit_runInSeparateProcess_bug\src\template.php on line 2

Error: Using $this when not in object context in D:\dev\phpunit_runInSeparateProcess_bug\src\template.php on line 2

Call Stack:
    0.0011     597416   1. {main}() C:\Users\Alenxander\AppData\Local\Temp\PHP8386.tmp:0
    0.0171    1958416   2. require_once('D:\dev\phpunit_runInSeparateProcess_bug\src\template.php') C:\Users\Alenxander\AppData\Local\Temp\PHP8386.tmp:414

ERRORS!
Tests: 2, Assertions: 1, Errors: 1.

How to reproduce

1. Clone https://github.com/samdark/phpunit_runInSeparateProcess_bug
2. composer install
3. ./vendor/bin/phpunit

Expected behavior

No failure. First test should pass. Second one should be marked as risky.

[sebastianbergmann]

Does this only happen with master?

[samdark]

No. It happens with PHPUnit 7 and PHPUnit 8 stable releases as well. I've updated repository composer.json to use stable PHPUnit 8.4.3.

[samdark]

Not sure it's related directly with @runInSeparateProcess, likely not. Seems to be something about output buffering... anyway, reduced test case reproduces it in a stable way.

[samdark]

Tested more. It seems the following method of rendering a template that is called in one of the tests makes PHPUnit semi-randomly and cryptically fail in the following tests:

public function render(): string
{
        ob_start();
        require __DIR__ . '/template.php';
        return ob_get_clean();
}

Sometimes it's causing errors like I've posted before, sometimes it's

Warning: Illegal string offset 'failOnRisky' in D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\src\TextUI\TestRunner.php on line 789

Call Stack:
    0.0002     410480   1. {main}() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\phpunit:0
    0.0082     923912   2. PHPUnit\TextUI\Command::main() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\phpunit:61
    0.0082     924024   3. PHPUnit\TextUI\Command->run() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\src\TextUI\Command.php:162
    0.0275    2678552   4. PHPUnit\TextUI\TestRunner->doRun() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\src\TextUI\Command.php:206


Warning: Illegal string offset 'failOnWarning' in D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\src\TextUI\TestRunner.php on line 793

Call Stack:
    0.0002     410480   1. {main}() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\phpunit:0
    0.0082     923912   2. PHPUnit\TextUI\Command::main() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\phpunit:61
    0.0082     924024   3. PHPUnit\TextUI\Command->run() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\src\TextUI\Command.php:162
    0.0275    2678552   4. PHPUnit\TextUI\TestRunner->doRun() D:\dev\yii-dev\dev\yii-web\vendor\phpunit\phpunit\src\TextUI\Command.php:206

[tomtomsen]

I've looked into it, here is what i've found:

if a test method is marked with @runInSeparateProcess, phpunit will generate a temporary file based on a template (i.e. Util/PHP/Template/TestCaseClass.tpl).
if @preserveGlobalState is enabled (which it is, by default) phpunit will create a list of all included files (by using get_included_files) and requires all those files in the startup process.
Since your HtmlRenderer includes (require) template.php, that file will be required in the startup process of the seperated testcase as well, and therefor has no idea what $this means in that context.

If you add the annotation @preserveGlobalState disabled to the test which runs in a separate process, it should work.

i've found a blog post were someone stumbled across a similar problem (https://matthewturland.com/2010/08/19/process-isolation-in-phpunit/) - maybe @preserveGlobalState should be disabled by default if @runInSeparateProcess is set?

[samdark]

@tomtomsen good finds. Are there any other cases when templates are used? I've since then removed all @runInSeparateProcess from the real project tests but the issue is still there.

[tomtomsen]

@samdark
the annotation @runTestsInSeparateProcesses is currently present in
tests/Emitter/SapiEmitterTest:16 and
tests/Emitter/HTTPFunctionsTest:9

because of that a template gets generated... yada yada yada.

in my opinion you have two options:

1. remove @runTestsInSeparateProcesses in both tests,
2. add the annotation @preserveGlobalState disabled to both files

To be sure i've tested both and they seem to work fine.

[tomtomsen]

Added pull-request: https://github.com/yiisoft/yii-web/pull/207

I think this ticket is now resolved.

[samdark]

Thank you! I've somehow missed it...

[samdark]

@sebastianbergmann so now we know the reason for the issue in question. For me it's solved thanks to @tomtomsen but I still think that it is quite cryptic behavior. If it cannot be avoided, feel free to close the issue. Thanks.

[sebastianbergmann]

Looks to me like this is related to #3954.