Phpunit: PHPUnit Exit Code is always 0

[ghost]

Specs first:

• Ubuntu 12.04.4
• PHP 5.4.29-3+deb.sury.org~precise+1 (cli)
• PHPUnit 4.5.0

And a small test:

<?php

class MyTest extends PHPUnit_Framework_TestCase
{
    public function testFailedAssertion ( )
    {
        $this->assertTrue ( FALSE );
    }

    public function testUnexpectedException ( )
    {
        throw new Exception ( );
    }
}
?>

Running it:

> phpunit phpunit.php 
PHPUnit 4.5.0 by Sebastian Bergmann and contributors.

FE

Time: 48 ms, Memory: 3.75Mb

There was 1 error:

1) MyTest::testUnexpectedException
Exception: 

/phpunit.php:12

--

There was 1 failure:

1) MyTest::testFailedAssertion
Failed asserting that false is true.

/phpunit.php:7

FAILURES!
Tests: 2, Assertions: 1, Failures: 1, Errors: 1.

Exit Code:

user@pc:/$ echo $?
0

The problem is that Jenkins (and most other programs, I'd assume) use the exit code to determine whether the build is successful, and an exit code 0 will mark the build as successful even though some tests failed.

[ghost]

note: this might be the same as https://github.com/sebastianbergmann/phpunit/issues/1638

[ghost]

invalid: this seems to be a problem with our php installation and not with phpunit itself

[popstas]

@daniel-gomes-sociomantic, what prolem with php installation you had?

I reproduced this on Mac Os with php 5.6.17 installed with brew install php56 and phpunit 5.2.10 installed with composer global require phpunit/phpunit.

PHPUnit installed with composer into project or composer.phar returns correct exit code even if code coverage enabled.

[ghost]

I don't remember what it was, I should've written it up there :\ sorry..

[popstas]

@sebastianbergmann, I reproduced zero exit codes on Mac OS (see above) and on clean Ubuntu 14.04:

composer global require phpunit/phpunit

Run from global composer:

vagrant@default:~/transmission-cli$ phpunit
PHPUnit 5.1.7 by Sebastian Bergmann and contributors.

......................................F.......................... 65 / 93 ( 69%)
............................                                      93 / 93 (100%)

Time: 8.69 seconds, Memory: 13.00Mb

There was 1 failure:

1) Popstas\Transmission\Console\Tests\ConfigTest::testDefaultConfigRead
Failed asserting that false is true.

/home/vagrant/transmission-cli/tests/ConfigTest.php:47
/home/vagrant/.config/composer/vendor/phpunit/phpunit/src/TextUI/Command.php:158
/home/vagrant/.config/composer/vendor/phpunit/phpunit/src/TextUI/Command.php:109

FAILURES!
Tests: 93, Assertions: 139, Failures: 1.

Generating code coverage report in Clover XML format ... done

Generating code coverage report in HTML format ... done
vagrant@default:~/transmission-cli$ echo $?
0

Run from project returns correct exit code.

Some more information:

vagrant@default:~/transmission-cli$ php -v
PHP 5.6.18-1+deb.sury.org~trusty+1 (cli)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.3.3, Copyright (c) 2002-2015, by Derick Rethans

vagrant@default:~/transmission-cli$ which phpunit
/home/vagrant/.config/composer/vendor/bin/phpunit

I used http://phansible.com for bootstraping clean vagrant environment with Ubuntu 14.04, php 5.6, xdebug and composer.