Istanbul: Istanbul reports coverage on test files

[tleunen]

Is it normal that Istanbul reports the coverage on test files?

Here's my command:

> babel-node ./node_modules/istanbul/lib/cli.js cover --report lcov --report json --report text --report html $(npm bin)/_mocha -- -r tests.setup.js src/**/*-test.js

Istanbul reports the coverage on the required file, and on every *-test.js files as well.. But these files should not be covered :/

[codekirei]

It is not normal for Istanbul to report coverage on the test files themselves, no.

I notice that you're using babel-node here. In my experience, the 1.0.0-alpha.2 branch of Istanbul works better with ES6. If you aren't using it already, perhaps try both installing that and using babel-register with mocha instead of babel-cli?

$ npm install --save-dev [email protected] babel-register

in package.json:

{
  "scripts": {
    "cover": "istanbul cover --report lcov --report json --report text --report html _mocha -- -r babel-register -r tests.setup.js src/**/*-test.js "
  }
}

$ npm run cover

As a side note, you might consider using .istanbul.yml and mocha.opts for all the flags in cover. That's completely up to personal preference though — it has no bearing on the issue at hand.

Anyway, hope that helps!

[tleunen]

Thanks @codekirei. I was already using the branch 1.0 alpha 2, but was missing the babel-register hook.
I still have the same issue though. The coverage is taking the test files (*-test.js) and also tests.setup.js.

[codekirei]

Hmm... well you're definitely on the right track, but something is still not working right.

I've put together a super simple example repo here that has mocha using a file with require that doesn't affect Istanbul's coverage report.

In the example, get-foo.js exports a function that requires a document to work. So for testing, I have setup.js that uses jsdom to create a document for the test env. mocha.opts requires this file (and sets other standard test flags).

To verify locally:

• $ git clone https://github.com/codekirei/istanbul-mocha-required-coverage.git
• $ npm install for deps
• $ npm test shows the test case passes
  
  
  
  • if you remove the line from mocha.opts requiring setup.js, the test breaks as expected
  
  
• $ npm run cover shows 100% coverage (and only 1 function covered)

Note that I intentionally included a red herring function in setup.js here. If Istanbul were to include this file in its coverage report, it would not report 100% coverage and would show more than 1 function covered.

I know this example is overly simplified, but is there something that looks substantially different from what you're doing? If not, maybe you could try isolating your test and coverage setup from the rest of your project's complexity to get to a working state and gradually stack complexity back on to see what is causing the issue?

[tleunen]

Looks like the issue is there when the test files are inside the same directory as the source files.

> istanbul cover --report text _mocha -- -r babel-register -r test/setup.js src/**/get-foo.test.js



  0 passing (2ms)

------------------|----------|----------|----------|----------|----------------|
File              |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |
------------------|----------|----------|----------|----------|----------------|
All files         |      100 |      100 |        0 |      100 |                |
 src              |      100 |      100 |        0 |      100 |                |
  get-foo.js      |      100 |      100 |        0 |      100 |                |
 src/__tests__    |      100 |      100 |        0 |      100 |                |
  get-foo.test.js |      100 |      100 |        0 |      100 |                |
------------------|----------|----------|----------|----------|----------------|

If setup.js is at the root, it also appear in the list.

[codekirei]

Now we're getting somewhere :grin:

You're totally right — when test files are side-by-side with source, Istanbul picks them up. Not sure if this is working as intended or not...

For now, you'll want to manually exclude the files/globs that are tripping Istanbul up. Again, I like to use a config file for these sorts of things but from $ istanbul help cover I see that there is a cli flag for this as well:

    -x <exclude-pattern> [-x <exclude-pattern>]
            one or more fileset patterns e.g. "**/vendor/**"

I've updated my example repo:

• test case moved to /src
• test setup script moved to /scripts (this could be anywhere, up to preference)
• moved mocha.opts to /configs
• created configs/istanbul.yml to exclude test files
• updated "scripts" in package.json to reflect changes

[tleunen]

Indeed, excluding the test files from the report works. I guess that's a nice quick fix, thanks @codekirei!

[codekirei]

@tleunen Also just to clarify a bit on babel-register, the official babel docs say to use this flag:

--compilers js:babel-register

I believe it is functionally equivalent to simply --require babel-register (before any other required files), but I haven't dug into the mocha source to confirm precisely what --compilers is doing. I asked the babel slack channel about this, and they said the --compilers flag is significant if you have multiple test file types that require different compilers. Like, say you had some tests in ES6 and some in typescript — you'd want the typescript compiler for .ts files and babel for .js. If you use babel-register in --require, you lose that flexibility.

[lukas-gitl]

Conversely, I _want_ istanbul to report coverage on test files. How would I go about that?

[nokka]

@codekirei all your answers were beyond helpful, thank you.

[dland512]

FYI, for me the -x flag didn't work if I had a ./ at the beginning of the pattern. So this works

-x 'app/**/*.test.js'

but this does not

-x './app/**/*.test.js'