Karma: coverage summary always 100%

[crossjs]

Expected behavior

should report coverage summary correctly

Actual behavior

always output

Statements   : 100% ( 0/0 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 100% ( 0/0 )

Enviroment Details

• Karma version (output of karma --version): 1.1.0
• Relevant part of your karma.config.js file

coverageReporter: {
    reporters: [
        { type: 'text-summary' }, // always 100%
        { type: 'text-summary', file: 'lcov.txt' } // reported file is correct
    ]
  }

Steps to reproduce the behaviour

1.just run
2.
3.

[dignifiedquire]

this is probably an issue with karma-coverage. please file an issue there

[nicojs]

I think i have the same issue. It only occurs with karma 1.1.0, not 1.0.0 or 0.13.x. As the problem seems to be introduced in karma 1.1, i think the issue should be in the karma issue tracker

There seems to be a raise condition. I think karma is collecting the reports too soon. Usually this doesn't result in an error because the html reporter first needs to verify that the directory structure is created (with fs.stats, see reporter.js). By then, the collector is filled. However for Stryker we're using the in-memory reporter. That one tries to collect the report immediately which usually fails with karma 1.1.0.

If i surround reporter.js L290 with a setTimeout the collector is always filled.

I have been debugging this issue for hours, but my inside karma knowledge is just too small. Can someone point me in the right direction? Namely: did something in the life cycle of reporting change what could explain this raise condition?

See this build output (with karma 1.1.0): https://travis-ci.org/stryker-mutator/stryker-karma-runner/jobs/140456228

Note: I see in the bug request that the text-summary has the same issue, which makes sense as it is also reporting immediately, without the creation of the file system.

[nicojs]

I did some more research. Apparently the OnBrowserComplete (reporter.js line 228) is sometimes called _after_ onRunComplete (reporter.js line 237). The report is only created in the OnBrowserComplete so it is important that that one is called first.

Can anyone speculate why the ordering of the events might have changed in this new release?

_Note: This means that the fact that other reporters are still working is just luck. If they didn't do the call to mkdirIfNotExists and create the report in the async callback, it would break them as well._

[RIAstar]

@nicojs' analysis is correct. I had the same issue and tracked it down to the same reason.
( too bad I found this issue after the fact :( )

Perhaps I can offer some more information: I have one case where it happens systematically and another where it systematically doesn't.

_case 1_: order always correct:
Just launch Karma server with singlerun, no autowatch.

const config = {
    singleRun: true,
    autoWatch: false,
    ...
};
const server = new Server(cfg, onFinish(done));
server.start();

_case 2_: order always inversed:
Launch Karma server without singlerun or autowatch and use the runner to launch the tests.

const serve = done => {
    const config = {
        singleRun: false,
        autoWatch: false,
        ...
    };
    const server = new Server(config, onFinish(done));
    server.on('browsers_ready', () => done());
    server.start();
};

const runTests = done => runner.run({}, () => done()});

This case will always fire OnBrowserComplete after onRunComplete.

(for information: these 2 functions are in fact 2 Gulp tasks and gulp-watch will trigger runTests when source files change, but I don't think that is really relevant here)

[nicojs]

@RIAstar I've been able to reproduce your assumption. See karma-bug-in-reporter for a _really_ small test project which proves it.

Now, can anyone point us in the right direction on how to solve it?

[dignifiedquire]

I've implemented a fix, thanks for the research

[nicojs]

@dignifiedquire Thanks for your help!

I understand that the setTimeout fixes the issue, but what caused it? Maybe we can solve it without adding the overhead of a setTimeout. Even with 0ms, you'll have to wait till the next microtask loop (or whatever) before the event is triggered.