istanbul don't include unrequired JS files in coverage

[amoufaddel]

Hello,
In my nodeJS application, i have this package for DAO:

• api/dao/ClasseA.js
• api/dao/ClasseB.js

and this package for unit test with mocha:

• test/dao/testA.js that require just ClasseA.js.

I use this command to execute tests for testA.js and generate Cobertura report :
istanbul cover --report cobertura mocha test/dao/*.js
It's work fine, But the coverage report don't include ClasseB.js in coverage.
Any ideas?.

[jsdevel]

+1 for this. It looks like an option may be to instrument all the code in a separate directory and then run istanbul. The downside to this is that it increases complexity. It would be nice if istanbul had a --subject flag where you could pass a glob pattern of all the source files meant to be tested.

Another work around is to add a test file that requires the missing files without any tests, although this feels hackish.

[devangnegandhi]

+1

[inquire]

If you use this in angular with karma and require js, you can just do something like:

define(['angular', 'app'], function(ng, app) {
    describe('Reality is working, on the app', function() {

        it('has a truth that is true', function() {
            expect(true).toBe(true);
        });

        it('has the ability to name modules', function() {
            var module = ng.module('test', []);
            expect(module.name).toBe('test');
        });

    });
});

When you include your app it will include all the files, even those that you have not written tests yet.

[MShawnDillon]

+1

I would like to instrument all of my code, and then report on what, of the _instrumented_ code, was covered, regardless whether another piece of code (such as a test) loaded it. If there is an easy way to force istanbul to consider untouched files in its reporting mechanisms, I haven't found it.

The tip above about using requirejs with karma works (in my experience), but I'm one of several leaving requirejs behind for things like browserify (haven't tried integrating this with coverage yet, so I don't know if this would work to "require" the untouched, but instrumented, files).

In my scenario, it is far more important for me to see what hasn't been covered, rather than what has. This tells me where I need to step up my testing efforts.

[jsdevel]

@amoufaddel @devangnegandhi @inquire @MShawnDillon check out #199. Can you give it some +1s? I believe it would solve many issues related to this as well.

[emschwartz]

+1

[ninjatux]

+1

[gotwarlost]

Try v0.2.13 with the --preload-sources option passed to the cover command. This will require all files that istanbul thinks is code before running any of the tests.

Not yet closing this until I get some feedback. Note that this will only work for the require use case and not for, say, RequireJS modules.

[jsdevel]

/cc @asilvas @azweb76 @tracker1 @therealklanni

[emschwartz]

This is definitely a step in the right direction but I'm seeing some strange behavior with the --preload-sources option. If I run it in a project without any tests at all it returned an overall coverage figure of about 25%. I suppose that 25% represents statements, branches, etc that are hit when the files are required but I'm concerned that this will end up over-reporting test coverage.

[Raynos]

I'm concerned that this will end up over-reporting test coverage.

test coverage is not a number you can trust or relate to "how good is this code".

Test coverage is an analysis tool for authoring effective tests. The numbers mean nothing, the only useful analysis is inspecting the html reports

[emschwartz]

Fair enough but the html reports misrepresent the test coverage. I looked
through them line by line and it doesn't make sense that they show certain
lines have been hit "by the tests" when there are no tests at all

On Mon, Jun 30, 2014 at 4:00 PM, Raynos (Jake Verbaten) <
[email protected]> wrote:

I'm concerned that this will end up over-reporting test coverage.

test coverage is not a number you can trust or relate to "how good is this
code".

Test coverage is an analysis tool for authoring effective tests. The
numbers mean nothing, the only useful analysis is inspecting the html
reports

—
Reply to this email directly or view it on GitHub
https://github.com/gotwarlost/istanbul/issues/142#issuecomment-47598323.

[Raynos]

@emschwartz then turn --preload-sources off ;)

I think the way --preload-sources works is it just requires all of them and thus runs those files.

maybe @gotwarlost has an idea on how to require those files but to not mark all the statements in them as covered, i.e. remove the hits from the lines when you require through --preload-sources.

@emschwartz has a good point, it would be nice if --preload-sources didn't have the side effect of increasing code coverage, by resetting all the lines back to 0 after running.

[emschwartz]

I specifically do want to use something like --preload-sources. Without
it I wind up over-reporting test coverage because istanbul misses files
that are never required by the tests. Now with it it's over-reporting
coverage for a different reason. If there's a way to reset all of the lines
back to 0 right before the tests are actually run, that would be ideal.

On Mon, Jun 30, 2014 at 4:10 PM, Raynos (Jake Verbaten) <
[email protected]> wrote:

@emschwartz https://github.com/emschwartz then turn --preload-sources
off ;)

I think the way --preload-sources works is it just requires all of them
and thus runs those files.

maybe @gotwarlost https://github.com/gotwarlost has an idea on how to
require those files but to not mark all the statements in them as covered,
i.e. remove the hits from the lines when you require through
--preload-sources.

@emschwartz https://github.com/emschwartz has a good point, it would be
nice if --preload-sources didn't have the side effect of increasing code
coverage, by resetting all the lines back to 0 after running.

—
Reply to this email directly or view it on GitHub
https://github.com/gotwarlost/istanbul/issues/142#issuecomment-47599030.

[gotwarlost]

This is no different from how istanbul behaves when you do have tests. If all your test does is require the code and not actually exercise anything in it, the coverage will show you the numbers for the mainline code when the file was required.

In short those numbers represent the code that is covered the moment you decide to use a library. There should be no difference in the numbers for the case where you require a file and write no tests for it versus istanbul doing it on your behalf.

In short, it is working as expected.

[jsdevel]

@emschwartz see #199. I think what we're all looking for is a way to instrument our source files and have a reporter tell us what our code coverage is.

I've used glob.sync('../allmymodules').forEach(require); as a hack to get them into the coverage which seems inline with --preload-sources. It didn't really feel good though as there could be some dead files therein that boost coverage but never actually get used.

[gotwarlost]

It is a hard problem to do anything different than what I've done so far. Mainly because require-ing a file can also end up requiring other files and so on.

This means:

• storing the state of the require cache
• require-ing all files while keeping track of the 0 coverage object for all files
• get a diff of the require cache to see all the stuff that has been loaded
• overwrite the global coverage object with the 0 objects
• nuke all these entries from the require cache

It is _possible_ to come up with a solution that works the way you expect but it is not easy and the resulting code will be an unmaintainable mess.

For this reason, I'm not going to work on this any more.

[ghost]

What you did with --preload-sources is excellent, and all that I needed for my reporting.

Yes, the simple act of require-ing a module may execute some of that module's code to return the exports and this would result in 'covering' the executed code ... I can certainly take that into account even if it's never exercised by a test.

@gotwarlost, I really appreciate your efforts, and thank you for a wonderful product. I am instrumenting all of my server code (incl. tests) and leaving it up to karma-coverage to do the client code. With --preload-sources, I can now see end-to-end what is being used, what is being tested, what uncovered code I have in the tests themselves, and what server-side cruft has built up over time that is not being used or exercised at all anymore and may be good candidates for removal.

Thanks again.

[aggieben]

Is there a way to do --include-all-sources and have it respect the default excludes? When I do this, istanbul tries to instrument everything in node_modules and fails on esprima.js.

[juancarlosfarah]

I'm running into the same issue as @aggieben. Anyone recommend a way around this?

[awolden]

We really need the ability to include only certain files when we use --include-all-sources. I have the same issue as @aggieben and @juancarlosfarah.

./node_modules/istanbul/node_modules/esprima/esprima.js:5701
            throw e;
            ^
Error: Line 1: Unexpected token (
    at constructError (./node_modules/istanbul/node_modules/esprima/esprima.js:2406:21)
    at createError (./node_modules/istanbul/node_modules/esprima/esprima.js:2425:17)
..........

[mike-goodwin]

+1 for the suggestion from @aggieben, @juancarlosfarah and @awolden
When I use --include-all-sources it does literally ALL sources including default excludes and explicit excludes using-x.In my case this leads to a process out of memory error.

[abritinthebay]

Agreed, the same problem here. This STRONGLY needs to be reopened.

[shaikhspear16]

facing a similar problem, with ionic 2 angular 2 ts karma jasmine remap-istanbul

not able to get files for which spec is not written covered in code coverage

my karma.config.json

`module.exports = function(config) {
config.set({

    basePath: '.',

    frameworks: ['jasmine'],

    files: [
        // paths loaded by Karma
        {pattern: 'node_modules/angular2/bundles/angular2-polyfills.js', included: true, watched: true},
        {pattern: 'node_modules/systemjs/dist/system.src.js', included: true, watched: true},
        {pattern: 'node_modules/rxjs/bundles/Rx.js', included: true, watched: true},
        {pattern: 'node_modules/angular2/bundles/angular2.dev.js', included: true, watched: true},
        {pattern: 'node_modules/angular2/bundles/testing.dev.js', included: true, watched: true},
        {pattern: 'node_modules/angular2/bundles/http.dev.js', included: true, watched: true},
        {pattern: 'karma-test-shim.js', included: true, watched: true},

        // paths loaded via module imports
        {pattern: 'dist/**/*.js', included: true, watched: true},

        // paths to support debugging with source maps in dev tools
        {pattern: 'src/**/*.ts', included: false, watched: false},
        {pattern: 'dist/**/*.js.map', included: false, watched: false}
    ],

    // proxied base paths
    proxies: {
        // required for component assests fetched by Angular's compiler
        '/src/': '/base/src/'
    },

    port: 9876,

    logLevel: config.LOG_INFO,

    colors: true,

    autoWatch: true,

    browsers: ['Chrome'],

    // Karma plugins loaded
    plugins: [
        'karma-jasmine',
        'karma-coverage',
        'karma-chrome-launcher'
    ],

    // Coverage reporter generates the coverage
    reporters: ['progress', 'dots', 'coverage'],

    // Source files that you wanna generate coverage for.
    // Do not include tests or libraries (these files will be instrumented by Istanbul)
    preprocessors: {
        'dist/**/!(*spec).js': ['coverage']
    },

    coverageReporter: {
        reporters:[
            {type: 'json', subdir: '.', file: 'coverage-final.json'}
        ]
    },

    singleRun: true
})

};
`

Any Help or work around ?
I am new to this framework !!
Thank You

[sandy-adi]

using --include-all-sources option works, but it appears like having it in .istanbul.yml is not making a difference

[sandy-adi]

try this: add the following to your .istanbul.yml worked for me

instrumentation:
   include-all-sources: true

[jedwards1211]

As long as code in the untested modules runs without crashing, I'd say the extra coverage from that is valid.

[jedwards1211]

@jsdevel I don't think requiring the missing files is hackish, what you're testing in that case is that any code that runs when otherwise untested modules are required doesn't throw an error.