Testcafe: Improve reporting for concurrency

[gyopiazza]

Are you requesting a feature or reporting a bug?

Feature

What is the current behavior?

When running multiple tests with concurrency, the reporter only notifies 1 test that's being executed, not all of the concurrent tests. During this time, all the tests that failed/passed pile up and are reported altogether instead of having a constant update. This behaviour is fine without concurrency but doesn't apply when running simultaneous tests.

What is the expected behavior?

Each time a new test starts, the reporter should notify it. Each time a test fails/passes it should be notified.

How would you reproduce the current behavior (if this is a bug)?

Run multiple tests with concurrency higher than 1.

Specify your

• operating system: macOS 10.12.5
• testcafe version: 0.17.2
• node.js version: 8.4.0

[AlexanderMoskovkin]

Hi @gyopiazza.

TestCafe default reporters don't output any info when the next test is started. According to the Reporter Methods topic TestCafe reporters can output any info on the following events: the whole task is started, a fixture is started, a test is done, the whole task is done.

If TestCafe will output a message when a test is started the final report will be entangled:

fixture 1
    test1 started
    test2 started
    test3 started
    v  test2 passed
    test4 started
    x  test1failed 
...

I'd prefer to have a clear report when the concurrency option is enabled as it is now:

fixture 1
    x  test1 failed
    v  test2 passed
    v  test3 passed 
...

[gyopiazza]

Hello @AlexanderMoskovkin,

Just to clarify, I'm not asking to output the "test started" info in the console reporter, but to fire the event "reportTestStart" as part of the TestCafe API. Then it would be a reporter's responsibility to make sure the output is nice and readable. Or even ignore the event (just like the current reporter's behaviour).

If a reporter wants to output the "reportTestStart" event, maybe something like this could work (imagine it with nice colors):

Fixture 1 started
Fixture 1 - test1 started
Fixture 1 - test2 started
Fixture 1 - test3 started
Fixture 1 - v test2 passed
Fixture 1 - test4 started
Fixture 1 - x test1 failed
  Error details
...

I'm suggesting this because I've created a simple UI to run the tests in concurrency mode and it would be nice to get real time reporting of all the running tests and their statuses. Without "reportTestStart" event the UI seems stuck and it updates kind of randomly. It simply doesn't give a nice feeling of progress going on.

If this suggestion doesn't make sense or you don't think it's a good idea, please feel free to close this issue. :)

As always, thank you for the feedback!

[AlexanderMoskovkin]

It sounds reasonable. I can't provide estimates for now but we'll consider it. Meanwhile, contributing is welcome.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity for a long period. It will be closed and archived if no further activity occurs. However, we may return to this issue in the future. If it still affects you or you have any additional information regarding it, please leave a comment and we will keep it open.

[AcademyOfF]

I would also like this feature. I'm not sure why more people don't want it: as far as I can tell, you can't tell how long your tests take if you run concurrently. For me, that's important information so that I can find which tests are taking the longest and investigate.

I don't know if I should create a separate issue for this, but I think it would be necessary to to pass the fixture name to reportTestStart and reportTestDone to be able to report test time correctly. With the current APIs, I think the only way to associate a test with its fixture is by the order in which the APIs get called. That's not deterministic when running concurrently.

I also want to add that I like the way the reporter orders its output. I'm not asking for that to change. I just want accurate information when running concurrently.

[AcademyOfF]

Another thing I just noticed is testRunInfo.durationMs has incorrect information when you run concurrently. You can see some info here: https://github.com/Raynos/testcafe-reporter-spec-time/issues/1#issuecomment-723542028

[AcademyOfF]

sorry to keep bumping this, but shouldn't this be considered a bug at this point? Concurrency is implemented in a way that breaks reporters.

[miherlosev]

https://github.com/DevExpress/testcafe/issues/1816

Another thing I just noticed is testRunInfo.durationMs has incorrect information when you run concurrently. You can see some info here: Raynos/testcafe-reporter-spec-time#1 (comment)

Thank you for sharing this information. We will research it.