Codeception: Feature request: less verbose output

[dzoeteman]

We've recently moved 1593 unit tests of a legacy project from standard PHPunit to Codeception. However, we've found that the output from Codeception in this case is quite useless, as it's way too verbose for this amount of tests.

It would be nice to have an option to enable a less verbose mode, much like PHPunit does, where you only have dots, or a 'F' or 'E' in case of failure or error respectively. This could be enabled in a (suite) config option and/or argument to the run command.

[DavertMik]

I will think on best way to implement that

[dzoeteman]

@DavertMik I'm ready to pick this up, and I'd like to know your opinion on my proposed solution:

• Have a suite config option called 'verbosity', where you could go from silent, to a set-up as I've specified in this issue, all the way to the same behaviour as -vvv option.
• I could go for either of these options for the less verbose mode (or both available?):
  
  
  
  • Progess bar (with indication of failed tests)
  
  
  • As PHPUnit does it (with dots/F/E)
  
  
• It might be good to go for a refactor of the Subscriber\Console class with this, as it's already quite massive, and with this change included, certainly wouldn't get any smaller.

With that, what version of Codeception would you like to implement this in? I could put it in 2.2, but this might be large enough that it'd have to wait for 3.0.

[Naktibalda]

I think that "less verbose" should be a command line argument and not a setting.
If it works and looks well, it could become a default mode of output in 3.0 (current output would be pushed to more verbose).

PHPUnit style works everywhere and does not require any additional dependencies. Progress bar could be broken in some environments (e.g. Windows or CI servers) and it adds more dependencies, right?
So dots first, progress bar later. Output style could be controlled by another argument and/or autodetection of terminal capabilities.

Start with 2.2 branch because it is more up to date. Decision where to merge your work can be made later.

[dzoeteman]

Thanks @Naktibalda for your input, I will start with the PHPunit-style output then. But for later, the progress bar doesn't require additional depencies, it's included in Symfony Console :)

[DavertMik]

Thanks @gdscei and sorry for the late response.

There is a problem in reporters because as you discovered there is one default console reporter of Codeception and others are inherited from PHPUnit. What I'd really like to see if we could make our own configurable reporters, and easy to extend interface for making ones. This will probably require some changes into core, but probably it should not be a problem.

Currently we have reporters section in config. We can add a section for default reporter:

reporters:
   default: Codeception\Subscriber\Console

alternative syntax (choose which you like more):

reporters:
   cli: Codeception\Subscriber\Console

Once the reporter is loaded, a runner should take a decision. If it is EventSubscriberInterface then it is attached as subscriber to dispatcher, if it is PHPUnit_Framework_TestListener it is attached as result printer to PHPUnit. By doing so, all report options can be rewritten from PHPUnit style to Codeception style.

Probably for Codeception reporter we will need a special interface. And Subscriber\Console can implement that. The Codeception\Subscriber\Console is pretty complicated because it does everything to play nicely for different terminals (interactive, non-interactive), with different verbosity levels (test-by-test, step-by-step, etc) and print the result. If you feel your simplified reporter can take some part out of it you can make an abstract class and move the common functionality into it.

Once you do the initial implementation we can easily have dots or progress-bar, whatever it takes. Personally I vote for dots as well :)

[DavertMik]

Use

--ext DotReporter

in Codeception 2.3+