Angular-cli: Test Coverage Integration?

[SethDavenport]

Please provide us with the following information:

1. OS? Windows 7, 8 or 10. Linux (which distribution). Mac OSX (Yosemite? El Capitan?)

Mac OS X (El Capitan)

1. Versions. Please run ng --version. If there's nothing outputted, please
   run in a Terminal:
   node --version
   And paste the result here.

angular-cli: 1.0.0-beta.5
node: 6.0.0
os: darwin x64

(node 6.0.0)

1. Repro steps. Was this an app that wasn't created using the CLI? What change did you
   do on your code? etc.

None - feature request.

1. The log given by the failure. Normally this include a stack trace and some
   more information.

None - feature request.

1. Mention any other details that might be useful.

Angular-cli sets up Karma/Jasmine and basic unit tests which is really nice; thanks for that. However I'm wondering if there are plans to integrate code coverage reporting and threshold checking into the ng test flow (e.g. with Istanbul/Isparta, or similar).

In my company's starter kits we have a shortlist of things we consider mandatory for a production toolchain; the ability to fail CI when unit test coverage drops below a certain threshold is one of them.

Is code coverage on your radar?

[delasteve]

For fun, I decided to try and get this working in a test project. I was able to get things working with karma-coverage and remap-istanbul.

Caveats:

• There is no folder structure. There appears to be an issue already reported here: https://github.com/SitePen/remap-istanbul/issues/50
• I was not able to find a way to get the remap to auto-run after a karma run. I tried karma-remap-istanbul, but had no luck. This is probably due to user error. Also, it has an outdated version of remap-istanbul (https://github.com/marcules/karma-remap-istanbul/pull/7). I had used the pull request project instead of the npm package.
• Using remap-istanbul requires you to list specific source files, which if you add multiple browsers, will be multiple and will be fragile due to the specific browser needed. This will require a script to gather and remap each of the files to overcome the CLI commands I used.

My modified karma.config.js for reference (omitted parts that didn't change):

    plugins: [
      require('karma-coverage'),
      require('karma-remap-istanbul'),
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-safari-launcher')
    ],
    preprocessors: {
      'dist/!(vendor)/!(*spec).js': ['coverage']
    },
    reporters: ['progress', 'coverage', 'karma-remap-istanbul'],
    coverageReporter: {
      type : 'json'
    },
    browsers: ['Chrome', 'Safari']

    // This didn't work for me. Kept saying "Couldn't find any specified files, exiting without doing anything."
    remapIstanbulReporter: {
      src: [
        'coverage/Chrome 50.0.2661 (Mac OS X 10.11.4)/coverage-final.json',
        'coverage/Safari 9.1.0 (Mac OS X 10.11.4)/coverage-final.json'
      ],
      reports: {
        html: 'coverage/report'
      }
    }

To get the remap working, I used the following. You may need to change the numbers up depending on your browser.

$ ng test --watch=false
$ ./node_modules/.bin/remap-istanbul -i coverage/Chrome 50.0.2661 (Mac OS X 10.11.4)/coverage-final.json -o coverage/Chrome 50.0.2661 (Mac OS X 10.11.4) -t html
$ ./node_modules/.bin/remap-istanbul -i coverage/Safari 9.1.0 (Mac OS X 10.11.4)/coverage-final.json -o coverage/coverage/Safari 9.1.0 (Mac OS X 10.11.4) -t html

[delasteve]

After playing with it a bit more, I got this working on ng test and karma-remap-istanbul (from the PR version still). You can see the PR here: https://github.com/angular/angular-cli/pull/895

Caveats:

• There is still no file structure (same reason as above)
• I assumed only 1 browser output allowed. This will probably need to have work done to support reports for multiple. The output is now in coverage/report.

[jasonswett]

Where did we land with this? I see that the issue is closed but I can't really tell what the conclusion is.

[CristyTarantino]

+1

[filipesilva]

Code coverage was added via #1455.

[seescode]

How can I run the code coverage tool? When I run "ng help" I can't tell what command I need to run?

[delasteve]

It should run when you run ng test.

[filipesilva]

Actually it was changed to only run when doing ng test --code-coverage, because of how slow having it on all the time was.

[humpedli]

Do you know, how to enable this automatically? So defining this parameter somehow in angular-cli.json or something like that. And how to force WebStorm to create coverage when I am using Karma task with the root karma configuration?

[schechter]

Is there a way to make the tests fail if the coverage falls before a threshold?

[cexbrayat]

The CLI lets you have code coverage stats with ng test --cc.

There is no nothing builtin in the CLI for failing if the coverage is below a threshold, but that's quite easy to set up.
You'll have to use the karma plugin: karma-istanbul-threshold https://www.npmjs.com/package/karma-istanbul-threshold

Start by installing it : npm i --save-dev karma-istanbul-threshold

Edit your karma.conf.js

plugins: [
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-jasmine-html-reporter'),
      require('karma-coverage-istanbul-reporter'),
+     require('karma-istanbul-threshold'),
      require('@angular/cli/plugins/karma')
    ],
// ...
  coverageIstanbulReporter: {
-     reports: ['html'],
+     reports: ['html', 'json'],
      fixWebpackSourcePaths: true
    },
// ...
+  istanbulThresholdReporter: {
+    src: 'coverage/coverage-final.json',
+      reporters: ['text'],
+      thresholds: {
+        global: {
+          statements: 90,
+          branches: 90,
+          lines: 70,
+          functions: 90,
+        },
+        each: {
+          statements: 80,
+          branches: 80,
+          lines: 60,
+          functions: 80,
+        },
+      }
+    },
// ...
reporters: config.angularCli && config.angularCli.codeCoverage
-      ? ['progress', 'coverage-istanbul']
+      ? ['progress', 'coverage-istanbul', 'istanbul-threshold']

Then launch your test with ng test --cc --single-run, and you're set!

Low Coverage: GLOBAL 44.83% of 70% lines
Low Coverage: GLOBAL 48.6% of 90% statements
Low Coverage: GLOBAL 2.2% of 90% functions
Low Coverage: GLOBAL 51.35% of 90% branches
Low Coverage: src/app/app.component.ts 62.07% of 80% statements
Low Coverage: src/app/app.component.ts 22.22% of 80% functions

[marciomsm]

@cexbrayat I did all the step you described using Angular CLI 1.0, but I did not get the same result as you show below. Is there any configuration I need to do?

Low Coverage: GLOBAL 44.83% of 70% lines
Low Coverage: GLOBAL 48.6% of 90% statements
Low Coverage: GLOBAL 2.2% of 90% functions
Low Coverage: GLOBAL 51.35% of 90% branches
Low Coverage: src/app/app.component.ts 62.07% of 80% statements
Low Coverage: src/app/app.component.ts 22.22% of 80% functions

Thanks.

[cexbrayat]

@marciomsm The results will depend on how many unit tests you have in your application (see code coverage)

[marciomsm]

Hi @cexbrayat, what I wanted say is that I didn't get the percentage information after to configure as you explained. I mean, I didn't get the below information.

Low Coverage: GLOBAL X% of X% lines
Low Coverage: GLOBAL X% of X% statements
Low Coverage: GLOBAL X% of X% functions
Low Coverage: GLOBAL X% of X% branches

[cexbrayat]

Well, It works on several of our apps, and it seems like some people 👍 it so I guess it should work for you too.

Read carefully my previous comment, and make sure you did not missed any step. You can also read the docs of the plugin https://www.npmjs.com/package/karma-istanbul-threshold

[marciomsm]

Hi @cexbrayat, I really appreciate your help.

I did it again step by step and I got the correct information. I also set up 'json-summary' and it provided everything I needed.

Thank you very much.

[anilkhichar]

With @angular/cli: 1.0.0, there is no need to install any additional threshold plugin like karma-istanbul-threshold. As per cli documentation, just add below line and your are all set with threshold as well.

coverageIstanbulReporter: {
  reports: [ 'html', 'lcovonly' ],
  fixWebpackSourcePaths: true,
  thresholds: {
    statements: 80,
    lines: 80,
    branches: 80,
    functions: 80
  }
}

The thresholds property will enforce a minimum of 80% code coverage when the unit tests are run in the project.

Here is my output of

ng test --cc

16 04 2017 15:19:35.385:ERROR [reporter.coverage-istanbul]: Coverage for statements (66.28%) does no
t meet global threshold (80%)
16 04 2017 15:19:35.385:ERROR [reporter.coverage-istanbul]: Coverage for lines (63.8%) does not meet
 global threshold (80%)
16 04 2017 15:19:35.385:ERROR [reporter.coverage-istanbul]: Coverage for branches (44.55%) does not
meet global threshold (80%)
16 04 2017 15:19:35.385:ERROR [reporter.coverage-istanbul]: Coverage for functions (37.22%) does not
 meet global threshold (80%)

[ajdaniel]

@khichar-anil that doesn't work for me, but using karma-istanbul-threshold works. I have no idea why it didn't work, so I'm sticking with what works

[jasonswett]

I wrote a terse, but perhaps helpful, blog post about this: How to Add a Test Coverage Report to an Angular CLI Project

[yfain]

I tried using thresholds with Angular CLI 1.0.1, but my tests don't fail even if I specify high threshold numbers.

[aniruddhadas9]

ng test --code-coverage created code coverage for me.

[SURENDARREDDY]

I agree with @yfain. It is not going to fail with Angular CLI 1.0.1.
It keeps getting success even I have all threshold 100%.

[yfain]

After upgrading the karma-coverage-istanbul-reporter to 1.2.1 it works as expected.

[SURENDARREDDY]

Thank you @yfain, it is working now...

[ajdaniel]

I can confirm. I updated karma-coverage-istanbul-reporter to 1.2.1 and now I can use the threshold property in the coverageIstanbulReporter config like so:

        coverageIstanbulReporter: {
            reports: ['html', 'json'],
            fixWebpackSourcePaths: true,
            thresholds: config.angularCli.codeCoverage ? {
                each: { // thresholds per file
                    statements: 80,
                    lines: 80,
                    branches: 80,
                    functions: 80
                }
            } : {}
        },

This example I made only applies thresholds when code coverage is enabled in the build, else it will use out of date coverage reports.

[briancullinan]

Am I the only one that finds these config files to be incredibly nuanced and poorly documented over time?
What is the point of including this in angular-cli if it doesn't configure it correctly, automatically?

[angular-automatic-lock-bot[bot]]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{_This action has been performed automatically by a bot._}