Phpunit: Add --list-tests option for listing all available tests (without executing them)

[ioleo]

Hi,

could someone briefly explain to me how PHPUnit works internally? I want to create a simple class, that takes standard PHPUnit input and returns an array of testcasePath => [ testOneName, testTwoName ]

For example, given I'm in /app_root and I call myscript -c phpunit.xml.dist --filter "::testAction" --group api with phpunit.xml.dist configured to look for tests in src/tests and given following files exist:

/app_root/src/tests/Controller/FirstControllerTest.php

<?php

class FirstControllerTest extends \PHPUnit_Framework_TestCase
{
    /* @group api */
    public function testActionOne() { /* the test body is not relevant */}

    /* @group api */
    public function testActionTwo() { /* the test body is not relevant */}

    /* @group functional */
    public function testActionThree() { /* the test body is not relevant */}
}

/app_root/src/tests/Service/FirstServiceTest.php

<?php

class FirstServiceTest extends \PHPUnit_Framework_TestCase
{
    /* @group api */
    public function testActionList() { /* the test body is not relevant */}

    /* @group api */
    public function testActionFilter() { /* the test body is not relevant */}

    /* @group api */
    public function testElse() { /* the test body is not relevant */}
}

I would like my script to return:

<?php

/* @var array */
$exampleReturnData = [
//  relative path to testsuite                            array of matching test names
    'src/tests/Controller/FirstControllerTest.php'   => ['testActionOne', 'testActionTwo'],
    'src/tests/Service/FirstServiceTest.php'         => ['testActionList', 'testActionFilter'],
];

Where do I start? Is there some already existing functionality I could reuse or extend? @sebastianbergmann if you could give me some hints to get me started, please do

[SvenRtbg]

I'd say that the XML result file of a test run will give you that kind of information. Probably also phpunit --testdox will output some parseable text that may give you the same.

If you don't want to run PHPUnit at all, you'd have to scan the test directory, tokenize or regex the PHP code, and grab what is interesting to you.

[ioleo]

thanks for reply

I'd like to get a list of tests without running them,
I wasn't aware of the --testdox parameter, however it does not solve my problem (it still has to run all the tests)

I need a list of tests to run them in parallell (1 process per code) becouse my intergration tests take too long. Running a test suite (1 hour) to get a list of avaliable tests is not an option.

Right now im doing this with find $APP_DIR/src/*/*Bundle/Tests -name '*Test.php' however this does not take into account --groups or --filter parameter. Also I'd like to use phpunit internals so that my "find all tests" is always in sync/the same as phpunit.

[SvenRtbg]

There are already existing projects that try to run test suites in parallel: https://github.com/brianium/paratest or https://github.com/verkkokauppacom/parallel-phpunit

They even do work to some extent according to my own experience - take this as a sign of warning: The task may look easy at first glance, but it is rather complex inside. It works for easy test suites with simple, independent test methods. It somehow fails with @depends connected tests or data providers.

What I'd do: Have your test suite run one more time in full. It can produce a test result file which includes test execution time. You can then take this as the basis for splitting tests into groups by assigning each executed test class to an execution group 1, 2, 3, 4... Parallel execution would run each group explicitly, and run another job with all groups excluded to catch newly added tests.

The more manual step would be to split your test suite into distinct blocks: Unit tests should never do any kind of I/O and be fast. Integration tests will do some kind of I/O and be slow - maybe there's more than one individual topic that is tested and might be a reason to split the slow tests again.

One other thing: You haven't reported a bug with PHPUnit, so it isn't really the right place to discuss using an outdated version here. I think you should at least close this ticket. A good idea is to ask this kind of question on StackOverflow with the phpunit tag.

[ioleo]

Yes, I am useing liuggio/fastest. Right now it works by getting a list of test suites (via find -name '*Test.php', and runs each with phpunit {here path to testsuite}).

However, it would be most awesome if it actually could read the phpunit.xml.dist config (or let phpunit read it, only "ask phpunit" for a list of testsuites/tests) and run them with phpunit --filter {here test name} {here testsuite path}.

Now, I could ofcourse develop a reader for phpunit.xml.dist file, and then iterate over directories and search for tests... but:

• this would be repeating phpunit`s job
• if phpunit changes somehow it's logic - it may need an update
• would be nicer to simply use the same logic as phpunit uses

[nikita2206]

I'm not sure why no one said that you could reuse PHPUnit's own configuration utility to get all the test files:

$configuration = \PHPUnit_Util_Configuration::getInstance(__DIR__ . '/phpunit.xml.dist');
$testSuite = $configuration->getTestSuiteConfiguration();

This will give you an instance of PHPUnit_Framework_TestSuite with tests() method returning all of the child PHPUnit_Framework_TestSuite which in turn will have their own children as well... Now you'll just need to traverse this tree and take only those test suites that have their testCase property as true - these will return class name from getName() call (to get a file name you'll have to do (new ReflectionClass($class))->getFileName()). In order to inspect testCase protected property you'll have to either extend from PHPUnit_Framework_TestSuite and make a static method isTestCase(PHPUnit_Framework_TestSuite $suite) { return $suite->testCase(); } or inspect it via reflection (or a bound closure)...

[ioleo]

Thank you @nikita2206 I will check it out! :+1:

[ghost]

I just wanted to pitch in and mention this would also be handy for developers looking to integrate PHPUnit into some kind of front end (think IDE or something like Sublime Text). In this context the front end would display a list of all available tests to user, allowing him to run all of them or specific ones by clicking them.

Doing this via a PHP script (either via scanning or PHPUnit itself) is all well and good and may work for most developers looking for a quick and dirty solution, but it seems a bit unnecessary to request the user for a PHP interpreter executable so a tool can somehow fetch this information by using a custom script. It also requires internal knowledge of how PHPUnit operates (what if PHPUnit decides to change the name of its base class in a newer version, what if methods don't start with test anymore, what if classes don't end in Test, ...).

Could this be exposed via the command line? It would not necessarily have to be a "fetch list" command, something like a "dry run" might also work, as long as the list of tests are returned in a parseable format without actually running the tests.