Angular-cli: Add Coverage to e2e tests

[BenjaminSchubert]

Bug Report or Feature Request (mark with an x)

- [ ] bug report -> please search issues before submitting
- [X] feature request

Versions.

N.A.

Repro steps.

N.A.

The log given by the failure.

N.A.

Desired functionality.

I'd like to see end to end tests being able to also have test coverage as for normal tests.

This is useful when doing a project to make sure that nothing is out of the spec.

I'd be glad to help if this feature is accepted, but would need a bit of guidance on getting started.

Thanks a lot !

Mention any other details that might be useful.

[sumitarora]

@BenjaminSchubert AngularCLI uses protractor to run e2e tests, which doesn't support code coverage. https://github.com/angular/protractor/issues/370 Also having e2e do the code coverage won't be that much feasible as you would have to write e2e test for each and every case, branch etc. which can easily be tested using Unit Tests. E2E tests takes more time to run than unit tests so if you have lot of e2e tests covering each and every case it will take lot of time.

[BenjaminSchubert]

@sumitarora Well, there is at least a grunt plugin working for protractor and coverage, which means it is feasible.

I also know that e2e tests are taking much longer to run than unittests. But the aim would be to be able to know roughly if you ran through all your specification or if you have dead code. Moreover, simple tests that do not require unit tests could be covered easily by these. I therefore think that having both coverage is helpful.

[cmelion]

@sumitarora, FWIW we routinely get 100% coverage on our E2E tests for small to medium sized web-apps (when not using the CLI).

    plugins : [{
        path: 'node_modules/protractor-istanbul-plugin',
        outputPath: 'reports/feature/coverage'
    }]

[Stalyon]

Hi @cmelion,
I am interested in this coverage report.
Where did you add these lines? In protractor.conf.js?

[cmelion]

@Stalyon yep (protractor.feature.coverage.conf.js), we have separate conf files for coverage, mostly to avoid dealing with instrumentation obfuscating the code when debugging during normal test runs.

[nrazvan87]

@cmelion I've added this plugin and the settings in protractor.conf.js.
After that, I've ran: ng e2e. No reports are generated, do you have any clues where I should investigate? No extra errors/logs are displayed.

[cmelion]

@nrazvan87 I haven't looked at this in a while, last time I looked the CLI implementation of e2e did not pass the necessary configuration to webpack, so your code is probably not getting instrumented.

You need to have control over both protractor config and your webpack config (see the loader config below):

    if (isE2ETestEnv) {
        // instrument only testing sources with Istanbul, covers js compiled files for now :-/
        config.module.preLoaders.push({
            test: /\.jsx?$/,
            include: path.resolve('src'),
            loader: 'babel-istanbul',
            exclude: [/\.?spec\.js$/, /node_modules/, /test/]
        });
    }

[BenjaminSchubert]

@sumitarora would something around what @cmelion is showing be accepted into angular-cli?

I don't have the knowledge to contribute yet, but if it would be accepted, I can try to make an implementation of it.

[warstick]

@cmelion @BenjaminSchubert
For generating the E2E test cases report in angular4 cli, Where exactly i have to include following piece of code.

if (isE2ETestEnv) {
        // instrument only testing sources with Istanbul, covers js compiled files for now :-/
        config.module.preLoaders.push({
            test: /\.jsx?$/,
            include: path.resolve('src'),
            loader: 'babel-istanbul',
            exclude: [/\.?spec\.js$/, /node_modules/, /test/]
        });
    }

suppose if we can't do this, what is the fallback approach for to generate E2E Coverage report in angular 4 ?

[cmelion]

@warstick I'd have to look at the source for the CLI to see exactly how this could be integrated, probably worthy of a pull-request. I'll try to look into it this weekend.

The alternative is to bypass the CLI altogether which is not ideal.

[jedihacks]

I would also love to see this. Any luck on a configuration to get this supported @warstick @cmelion ?

[artem-v-shamsutdinov]

Same here. Eight now I'm manually instrumenting the bundles and can't even get the TS source mapping to work. Also the coverage information has to be manually pulled from the browser, which is a pain. It would be great if everything was just integrated and just worked.

This is very important for us because we want to tie in what scenarios are covered with what code is covered.

[hojt]

Anyone working on this? @BenjaminSchubert @sumitarora

[BenjaminSchubert]

@hojt I currently cannot work on this, and won't be able in the next months, sorry. Would still like to see it implemented though!

[chetanladdha]

Is there any plan when this will be done?

[edvlucas]

I have managed to get a working solution based on Istanbul/nyc, by pre-instrumenting the code, downloading the coverage info via protractor, saving it locally, remapping to the ts files and reporting to html. It is a bit hacky but it does the job:
https://github.com/edvlucas/angular-e2e-coverage

[divdavem]

@edvlucas Thank you for your working solution.

For info, I have opened PR #13131 to simplify the instrumentation of files, so that it is possible to use --code-coverage and --code-coverage-exclude with ng build ng serve or ng e2e.
This only instruments files as part of the build done by ng.

To get a coverage report, it is then still necessary to record the coverage, for example with an onComplete function in the protractor configuration:

async onComplete() {
  console.log('Retrieving coverage...');
  const coverage = await browser.executeScript('return JSON.stringify(window.__coverage__);');
  const coverageFile = path.join(__dirname, ".nyc_output", "out.json");
  console.log(`Coverage retrieved (${coverage.length} bytes), saving it to ${coverageFile}`);
  fs.writeFileSync(coverageFile, coverage);
  console.log("Coverage saved successfully!");
};

And then it is possible to produce both the lcov and html reports with

nyc report --reporter=lcov --report-dir=coverage-e2e

[mgechev]

We discussed this feature last week. Although we think it's out of the scope of the CLI at this point, we believe it could be beneficial for our users. I'd recommend you to look at @manfredsteyer's ngx-build-plus and consider letting folks take advantage of e2e test coverage through a plugin.

[divdavem]

For those looking for a sample to have e2e code coverage with ngx-build-plus, I have implemented it for ng-bootstrap in the following PR: https://github.com/ng-bootstrap/ng-bootstrap/pull/2935

[anudeep-mj]

Thanks a lot @divdavem 👍

[angular-automatic-lock-bot[bot]]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{_This action has been performed automatically by a bot._}