Karma: Optional fail if coverage threshold not met

[andrew-cunliffe]

I have recently started using Karma, and the team keeps adding code which drops the coverage below a level I am happy with, it would be great if Karma could fail the tests on CI based on a config setting like a percentage threshold e.g.

coverageReporter = {
    threshold: 80
}

I might have a look at doing this too if I can however it might be relating to the plugin used for coverage?

Wanted to add this here as a possible new feature as I cant see any alternatives to when I get to putting my code in and finding others have not been adding tests for their code :)

[vojtajina]

Sounds as a reasonable feature. Even though, I don't think forcing some test coverage is any good (usually people end up writing stupid tests just to get around; showing people the value of testing is better imho) ;-)

We should probably bring a new concept of "failing build even with successfull results" and allow plugins to fail it. We are already doing that for 0 tests executed, but it's kind of hardcoded.

[vojtajina]

You can also write additional taks that you run on the CI - read the generated coverage (there's always JSON dump, which is pretty simple to parse and analyze).

[andrew-cunliffe]

Thank you for the comments, we have put some work around checks in place which cause the build to fail (and thus not auto deploy) if certain 'other' criteria are not met and this is one of those checks now.

I agree however that developers can cheat, there is not much that can be done about that except to keep on trying to build in the processes and teach the values (as you are saying).

Either way having a feature in tools like this to fail a build based on criteria would be nice to avoid some of the custom tasks that have now been made for the project. I do appreciate also this is a plugin and the plugin does not support this kind of thing either, so you would be linking up more with the plugin to do this.

[vgchl]

:+1: +1

[vgchl]

Have a look at grunt-istanbul-coverage. It's "A simple grunt plugin for checking aggregated coverage thresholds from istanbul coverage JSON files."

[hugotox]

@teunvanvegchel Did it work for you? I tried it with no luck

[tarunc]

:+1: +1 Would love to see this implemented!

[matjaz]

@hugotox you must explicitly add json type and grunt-istanbul-coverage works great!

    coverageReporter: {
        reporters: [
            {type: 'html', dir: 'test/coverage'},
            {type: 'json', dir: 'test/coverage'}
        ]
    },

[santthosh]

Check out https://github.com/lithiumtech/karma-threshold-reporter this should also help you set thresholds for your tests

[stramel]

This could probably be closed. Options that I have found:

• Plugin: https://github.com/lithiumtech/karma-threshold-reporter
• Manual Parsing / Istanbul Check Coverage call

[shivanykshenoy]

This issue is closed because it has been addressed in karma-coverage as per the following commit: https://github.com/karma-runner/karma-coverage/commit/bc63b1583ec95abbf4c74df5f8346e0c8f048220

basically you add a check like below for the coverageReporter, in the karma.config file:

coverageReporter: {
  check: {
    global: {
       statements: 100,
       lines: 100,
       functions: 100,
       branches: 100
    }
 }
}

Thank you @Dignifiedquire! Much appreciated ;-)

[reyraa]

karma-threshold-reporter fails only if the overall threshold of the test coverage is below defined value.
I think having the same possibility as per file basis would be helpful. for instance, we have important modules merged without a single line of unit test, just because the overall coverage is above our 90% threshold.
specially for a project with over 100 components, it's so likely that leaving a component uncovered doesn't reduce the overall coverage below the required threshold.

[johnjbarton]

So suggest that to karma-threshold-reporter. Or write your own, it's not a big deal.