Istanbul: Cannot report on multiple coverage.json files

[potatopankakes]

I have two sub-directories with coverage outputs produced by istanbul in them. One is from running functional tests, and the second is from unit tests. I can use istanbul to report on them individually; however, reporting on them together (istanbul report --include **/coverage.json) fails.

Below shows the directory listing with 2 coverage sub-directories created from previous runs of istanbul cover:

LMDV-JEFFREYB:ns-api jburke$ ls
./              applogs/            notes/              test/
../             coverage-functionaltests/   package.json            x.istanbul.yml
README.md           coverage-unittests/     scripts/            xunit-functionaltests.xml*
app.js              node_modules/           src/                xunit-unittests.xml*

Below shows a successful report run on the coverage-functionaltests sub-directory:

LMDV-JEFFREYB:ns-api jburke$ node_modules/.bin/istanbul report --root ./ --include coverage-functionaltests/coverage.json --dir ./coverage text-summary

=============================== Coverage summary ===============================
Statements   : 71.48% ( 218/305 )
Branches     : 40% ( 36/90 )
Functions    : 65.48% ( 55/84 )
Lines        : 71.48% ( 218/305 )
================================================================================
Done

Below shows a successful report run on the coverage-unittests sub-directory:

LMDV-JEFFREYB:ns-api jburke$ node_modules/.bin/istanbul report --root ./ --include coverage-unittests/coverage.json --dir ./coverage text-summary

=============================== Coverage summary ===============================
Statements   : 52.23% ( 434/831 )
Branches     : 33.76% ( 133/394 )
Functions    : 40.11% ( 73/182 )
Lines        : 52.23% ( 434/831 )
================================================================================
Done

Below shows the error when I glob them:

LMDV-JEFFREYB:ns-api jburke$ node_modules/.bin/istanbul report --root ./ --include **/coverage.json --dir ./coverage text-summary
/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/reporter.js:58
            throw inputError.create('Invalid report format [' + fmt + ']');
                             ^
Error: Invalid report format [coverage-unittests/coverage.json]
    at Object.module.exports.create (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/util/input-error.js:7:15)
    at Object.Reporter.add (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/reporter.js:58:30)
    at /Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/reporter.js:69:18
    at Array.forEach (native)
    at Object.Reporter.addAll (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/reporter.js:68:14)
    at ReportCommand.Command.mix.run (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/command/report.js:101:18)
    at runCommand (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/cli.js:78:19)
    at runToCompletion (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/cli.js:82:5)
    at Object.<anonymous> (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/cli.js:88:5)
    at Module._compile (module.js:460:26)

Entering the following directly in node works correctly:

LMDV-JEFFREYB:ns-api jburke$ node
> process.version
'v0.12.2'
> var fs = require('fs'), istanbul = require('istanbul')
undefined
> var reporter = new istanbul.Reporter(), collector = new istanbul.Collector()
undefined
> reporter.addAll(['lcov', 'html', 'cobertura'])
undefined
> collector.add(JSON.parse(fs.readFileSync('coverage-functionaltests/coverage.json', 'utf8')))
undefined
> collector.add(JSON.parse(fs.readFileSync('coverage-unittests/coverage.json', 'utf8')))
undefined
> reporter.write(collector, true, function () { console.log('done') })

The above sequence produces a correct set of 'merged' output results in ./coverage/.

If I reverse the order of additions of the coverage.json files in the above example, the 2nd add fails. I'm guessing that this is somehow related to what happens when running from the CLI. Here is the failure on the 2nd add:

> collector2.add(JSON.parse(fs.readFileSync('coverage-unittests/coverage.json', 'utf8')))
undefined
> collector2.add(JSON.parse(fs.readFileSync('coverage-functionaltests/coverage.json', 'utf8')))
TypeError: Cannot read property 'length' of undefined
    at /Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/object-utils.js:247:37
    at Array.forEach (native)
    at Object.mergeFileCoverage (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/object-utils.js:244:31)
    at /Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/collector.js:71:44
    at Array.forEach (native)
    at Object.Collector.add (/Users/jburke/Documents/Src/NS/ns-api/node_modules/istanbul/lib/collector.js:68:31)
    at repl:1:12
    at REPLServer.defaultEval (repl.js:132:27)
    at bound (domain.js:254:14)
    at REPLServer.runBound [as eval] (domain.js:267:12)

I can mail/post the input coverage.json files if needed.

Version of istanbul is 0.3.17, and I know it also happens with 0.3.15

Running on Mac host:

LMDV-JEFFREYB:ns-api jburke$ uname -a
Darwin LMDV-JEFFREYB 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64

[StoneCypher]

I came here to ask whether multiple coverage tooling was possible. This is interesting.

@potatopankakes - If you'll share the config that leads to this, I'll pitch in on a quick hack resolution until upstream does something proper.

[doctyper]

I just ran into this issue. @potatopankakes @StoneCypher were you able to work around the problem?

[potatopankakes]

yes, i used the javascript that i provided above that runs in nodejs.

On Fri, Jan 8, 2016 at 6:19 PM, Richard Herrera [email protected]
wrote:

I just ran into this issue. @potatopankakes
https://github.com/potatopankakes @StoneCypher
https://github.com/StoneCypher were you able to work around the problem?

—
Reply to this email directly or view it on GitHub
https://github.com/gotwarlost/istanbul/issues/403#issuecomment-170175261
.

[richardm]

+1

Having the same issue. I have two json reports, and I can run text-summary against either of them individually as above, but I get the same error message as above when trying to merge them. From fiddling with this for a while, it appears that when it gets to the 2nd file, it's treating the file name as the report format which is causing the error message above:

Error: Invalid report format [dir/coverage.json]

[romansemko]

+1 any solution to this?

[alekitto]

Command seems to work as expected:

**/coverage.json should be sorrounded by quotes, otherwise it will be expanded by the shell resulting in istanbul report --include coverage-functionaltests/coverage.json coverage-unittests/coverage.json.

coverage-unittests/coverage.json is then correctly interpreted as argument, but no format is found called coverage-unittests/coverage.json

Correct command should be: istanbul report --include "**/coverage.json"

[potatopankakes]

@alekitto 's response is correct. After I surrounded the arguments to --include with quotes, the error went away and I get correct output.