Codeception: Test with invalid dataProvider not run

[okneloper]

Using the same test from #4887 if I run the test specifiying the exact test I want to run, the test is not run and no error message provided the reason, so it's hard to figure out. Would be good to have some indication why the test is not executed.

$ codecept run unit ExampleTest:testEmail

==== Redirecting to Composer-installed version in vendor/codeception ====
Codeception PHP Testing Framework v2.4.1
Powered by PHPUnit 4.8.36 by Sebastian Bergmann and contributors.

Unit Tests (0) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 5.22 seconds, Memory: 14.00MB

No tests executed!

Provide test source code if related

Details

• Codeception version: 2.4.x-dev (2.4.0 doesn't work (#4864))
• PHP Version: 5.6.24
• Operating System: Linux/Debian 8
• Installation type: Composer

[edno]

@okneloper - I've found the root cause of this issue. I am preparing a PR for tomorrow.

[DavertMik]

@edno please make a PR and I will release the next version ASAP

[edno]

@DavertMik - The issue is located within the FilterTest class of php-wrapper.
I can see 2 active branch: 6.0 and 7.0. Should the fix be applied on both?
Fix is currently visible in my fork: https://github.com/edno/phpunit-wrapper/commit/a054a345936238f5edd7bbe896443585cb1660ee

Also note that this is a kind of a workaround for supporting the WarningTestCase class introduced with PHPUnit 6.0. It seems the initial intent of this class was to deal with empty test class, but it is also triggered in the case of invalid dataprovider that is messing up with the stack trace, see output below:

1) WarningTestCase: Warning
 Test  vendor/phpunit/phpunit/src/Framework/WarningTestCase.php:Warning

  [PHPUnit\Framework\Warning] The data provider specified for ExampleTest::testProtected is invalid.
Trying to invoke protected method ExampleTest::protectedProvider() from scope ReflectionMethod  

/home/edno/Workspace/codeception-reportportal/vendor/phpunit/phpunit/src/Framework/WarningTestCase.php:74
/home/edno/Workspace/codeception-reportportal/vendor/phpunit/phpunit/src/Framework/TestCase.php:721
/home/edno/Workspace/codeception-reportportal/vendor/phpunit/phpunit/src/Framework/TestResult.php:710
/home/edno/Workspace/codeception-reportportal/vendor/phpunit/phpunit/src/Framework/TestCase.php:679
/home/edno/Workspace/codeception-reportportal/vendor/phpunit/phpunit/src/Framework/TestSuite.php:760
/home/edno/Workspace/codeception-reportportal/vendor/codeception/phpunit-wrapper/src/Runner.php:114
/home/edno/Workspace/codeception-reportportal/vendor/codeception/codeception/src/Codeception/SuiteManager.php:157
/home/edno/Workspace/codeception-reportportal/vendor/codeception/codeception/src/Codeception/Codecept.php:189
/home/edno/Workspace/codeception-reportportal/vendor/codeception/codeception/src/Codeception/Codecept.php:158
/home/edno/Workspace/codeception-reportportal/vendor/codeception/codeception/src/Codeception/Command/Run.php:466
/home/edno/Workspace/codeception-reportportal/vendor/codeception/codeception/src/Codeception/Command/Run.php:361
/home/edno/Workspace/codeception-reportportal/vendor/symfony/console/Command/Command.php:252
/home/edno/Workspace/codeception-reportportal/vendor/symfony/console/Application.php:865
/home/edno/Workspace/codeception-reportportal/vendor/symfony/console/Application.php:241
/home/edno/Workspace/codeception-reportportal/vendor/symfony/console/Application.php:143
/home/edno/Workspace/codeception-reportportal/vendor/codeception/codeception/src/Codeception/Application.php:108
/home/edno/Workspace/codeception-reportportal/vendor/codeception/codeception/package/bin:37
/home/edno/Workspace/codeception-reportportal/vendor/codeception/base/autoload.php:12
/home/edno/Workspace/codeception-reportportal/vendor/codeception/base/codecept:7

In addition to this fix, I would suggest to improve the support of the WarningTestCase class for this type of issue, and also having a report test status that is 'W' instead of success as currently. I would also recommend to add a dedicated event test.warning.
I've another PR almost ready for this support, so if okay with you I'll continue to extend support of warning testcases.

[DavertMik]

Should the fix be applied on both?

Yes, please

Also note that this is a kind of a workaround for supporting the WarningTestCase class introduced with PHPUnit 6.0.

All php-wrapper package is about workarounds for PHPUnit :|

[edno]

In addition to this fix, I would suggest to improve the support of the WarningTestCase class for this type of issue, and also having a report test status that is 'W' instead of success as currently. I would also recommend to add a dedicated event test.warning.
I've another PR almost ready for this support, so if okay with you I'll continue to extend support of warning testcases.

3 PRs available :

• #4901
• codeception/phpunit-wrapper#21
• codeception/phpunit-wrapper#22

[edno]

@okneloper - This issue should be fixed with PHPUnit Wrapper releases 6.0.9 and 7.0.6.

[okneloper]

Great, thanks for resolving this so quickly!