Cypress: Code Coverage

[denis-yuen]

I was wondering if Cypress had a recommended tool for calculating code coverage? and if so are there any examples?

Following up on https://github.com/cypress-io/cypress-example-recipes/issues/3 and opening an issue here for discussion as recommended.

For context, we previously used Protractor paired with Istanbul ( https://github.com/gotwarlost/istanbul ) to calculate code coverage for our application. We were wondering if any one has experience with something similar when working with Cypress

FYI @agduncan94

[brian-mann]

There is nothing currently built into Cypress to do this. Adding code coverage around e2e tests is much harder than unit and its possible it may not be feasible to do in a generic way.

If you can supply the processes / code you used to add coverage via Protractor that would be helpful.

The problem is that your typical JS src files can be written in many different ways, ES6, Typescript, Coffeescript, and generally the final JS files are transpiled, and then bundled via Browserify or Webpack.

It is possible to add code coverage to JS files served by a webserver though, so it's something we might be able to solve. IE: using something like this: https://github.com/gotwarlost/istanbul-middleware

But in doing so, the problem is that it would have to be integrated at your server level, and is specific to each way you build your src files - aka each individual application.

It may be possible for Cypress to do this at the network proxy level, but it would unlikely to be easy.

[agduncan94]

In our application we actually used Karma along with Istanbul (karma-coverage) to calculate code coverage on our unit tests.

We setup a Grunt task which uses the karma-coverage plugin to generate a coverage report on the unit tests. We have a karma config file similar to the one in the karma-coverage link above where we specify which JavaScript files to test.

What we were hoping to do as an alternate to karma and karma-coverage is to just call Istanbul and Mocha directly (or use something else if more appropriate) on the tests. This does not work because cypress has some special variables (like cy) which are not recognized by Mocha.

This is an example of how we would expect to call Istanbul and mocha if we were just using mocha directly, instead of cypress.

So I think we are most interested in coverage for unit testing.

[brian-mann]

@agduncan94 thanks for that explanation.

To clarify - doing unit testing when it comes to code coverage is fairly straightforward and absolutely can be done.

To date Cypress has been primarily used as integration and e2e testing whereby code coverage is substantially different.

With 0.18.0 we are now seeing users begin to add in unit testing and I have described the updates we need to make to Cypress to make this a first class feature..

[mike-engel]

This might be a bit naïve, but would it be possible to use the new code coverage tool in the chrome dev tools to accomplish this? Admittedly, I haven't looked into it much, and it would only work in chrome for now. https://developers.google.com/web/updates/2017/04/devtools-release-notes#coverage

[pvdyck]

There should be a way to instrument the tests to extract coverage and even export it to https://coveralls.io/ ( https://github.com/lemurheavy/coveralls-public/issues/1023 )

Any chance to see this happening ? What do we need ?

[brian-mann]

Cypress is open source, so you're welcome to take a look around to see what we'd need to do.

Code coverage during e2e tests is completely different than unit tests. There is no dependency tree, there are 3rd party libs, and there is no access to the original source (even with source maps) due to the browser build process that every JS codebase undergoes.

Cypress could theoretically instrument the code for you at the network layer, as its headed for the browser. But this has substantial challenges - it would be unidirectional, and we'd have no way of mapping the underlying built JS code back to the original (even with source maps its not enough).

So while we could provide code coverage on the final built JS files, it would be mostly garbage from babel, typescript, or whatever else you're using to transpile. It would also include all of the 3rd party code you import in.

All in all, not very useful.

We'd likely need to expose and build an API on top of Cypress's backend process that your server could send the original source files which we could map against the transpiled versions to come up with a lightweight set of instrumented files. We could then use those to determine the standard code coverage.

To be clear - this isn't a challenge as long as the code is instrumented properly, Cypress like every other tool could easily generate code coverage. The problem is that during e2e tests we receive the final built JS code where trying to go backwards to instrument is not really going to work.

Another option is that your own webserver could send the instrumented code (that you'd have to manage yourself). After you do that, it would be fairly trivial to have Cypress be able to read off the instrumented reports after test execution and then aggregate the results.

Lot of potential directions to go here, and is likely a good bit of work to champion something that's generic enough to work in most situations but easy enough so people could drop it in without writing a lot of code for their unique situation.

[denis-yuen]

Thanks for the explanation BTW!

[bahmutov]

@denis-yuen are you trying to compute code coverage for your tests (they should be executing all lines, so 100% right away...) or production code? If production code, maybe stick https://github.com/bahmutov/was-tested between server and Cypress and get code coverage from there - but we do not think code coverage is helpful in production E2E tests.

We are thinking about a different type of coverage - UI element coverage. You are probably more interested in knowing if all buttons or menu elements on the page were exercised by the Cypress end to end tests, rather than all code.

[atfzl]

I have set up coverage for Cypress if anyone wants to try.

First you need to instrument your code, for this I have used istanbul-instrumenter-loader with webpack.

Then add this code in your <project>/cypress/support/index.js

const istanbul = require('istanbul-lib-coverage');

const map = istanbul.createCoverageMap({});

Cypress.on('window:before:unload', e => {
  const coverage = e.currentTarget.__coverage__;

  if (coverage) {
    map.merge(coverage);
  }
});

after(() => {
  cy.window().then(win => {
    const coverage = win.__coverage__;

    if (coverage) {
      map.merge(coverage);
    }

    cy.writeFile('.nyc_output/out.json', JSON.stringify(map));
    cy.exec('nyc report --reporter=html');
  });
});

Here 'window:before:unload' event is used for capturing coverage which would be lost after a cy.visit() or refresh etc.

you also need to install nyc to generate reports which will be saved in <project>/coverage folder.

[bahmutov]

NICE @atfzl !!!

[9fingerdev]

Been looking at this but so far have not quite figured out how to get things working. Getting an out.json file but its empty. @atfzl can you shed some more light on what steps you took to config istanbul-instrumenter-loader?

[atfzl]

@9fingerdev this is where i used the istanbul-istrumenter-loader:

rules: [
      {
        test: /\.(jsx|js|tsx|ts)$/,
        include: path.resolve(__dirname, '../src'),
        rules: (isDebug
          ? [
              {
                loader: 'istanbul-instrumenter-loader',
                options: {
                  esModules: true,
                },
              },
            ]
          : []
        ).concat([{ loader: 'awesome-typescript-loader' }]),
      },

I'll create a recipe project on weekend showing complete process.

[9fingerdev]

@atfzl any progress on that recipe project? We are using Mithril.js + WebPack and keep running into problems.

[atfzl]

@9fingerdev, project showcasing cypress coverage: https://github.com/atfzl/react-redux-typescript-boilerplate

[atfzl]

For some reason istanbul-instrumenter-loader does not work with babel. For working with babel you can try babel-plugin-istanbul

[atfzl]

To instrument code with babel, there is no need to change the webpack config, just add the istanbul plugin.
sample .babelrc for working instrumented code:

{
  "presets": [
    "react",
    "flow",
    [
      "es2015",
      {
        "modules": false
      }
    ],
    "stage-2"
  ],
  "plugins": [
    "transform-runtime",
    "lodash",
    "transform-decorators-legacy",
    "istanbul"
  ],
  "env": {
    "test": {
      "plugins": [
        "transform-es2015-modules-commonjs"
      ]
    }
  },
  "retainLines": true
}

[gkemp94]

@atfzl This is exactly what I was looking for. We're using gulp though. Would the configuration be similar?

[atfzl]

@gkemp94 if you are using babel then you only need to add plugin in .babelrc, should work with gulp as well.

[abataub]

@atfzl Can you please highlight how can I configure it with grunt? I'm not using webpack at all. Thanks.

[robinyy]

Instrumenting the codes will do ONLY with non-compressed JS. However, a typical process of shipping the normal web application would be: Linting & UT passed (source code or transpiled) -> Packaging (minified & mangled) -> E2E/Integration testing (against compressed version of JS) -> promote to production

It would be difficult to calculate the coverage in the above pipeline unless we create a separated one only for the E2E coverage

[abataub]

@atfzl Can you please show light on to find code coverage when running tests in isolation mode?
cypress run --spec cypress/integration/*

Cypress version: 3.0.1

[atfzl]

@abataub the steps i have mentioned also works in isolation mode. Let me know what is the exact issue.
And for making it work with grunt see the comment at https://github.com/cypress-io/cypress/issues/346#issuecomment-368842351 if you are using babel.

[abataub]

Thanks, @atfzl. I just made changes and it works.

[crosscompile]

Thanks @atfzl that was super helpful!

In order to end up with cumulative coverage across multiple cypress/integration/*.js files, I had to add some code to read and merge any existing '.nyc_output/out.json file. Not sure if anyone else ran into the same issue.

[afenton90]

@atfzl have you tried this with env babel plugin? I can't get it to work in my setup.
How sensitive is the babel example you have given?

[paulfalgout]

The v3 solution I got working.. it's probably overkilling it in some form, but this allowed for the node stored map to be reused across all of the tests merging them along the way. I set the env coverage to open when running cypress open and it only runs the report for whichever file I run.

In plugins:

    if(config.env.coverage) {
        const istanbul = require('istanbul-lib-coverage');
        coverageMap = istanbul.createCoverageMap({});

        on('task', {
            'coverage'(coverage) {
                coverageMap.merge(coverage);
                return JSON.stringify(coverageMap);
            }
        });
    }

In support:

if(Cypress.env('coverage')) {
    afterEach(function() {
        const coverageFile = `${ Cypress.config('coverageFolder') }/out.json`;

        cy.window().then(win => {
            const coverage = win.__coverage__;

            if(!coverage) return;

            cy.task('coverage', coverage).then(map => {
                cy.writeFile(coverageFile, map);

                if(Cypress.env('coverage') === 'open') {
                    cy.exec('nyc report --reporter=html');
                }
            });
        });
    });
}

My npm script is essentially:
cross-env NODE_ENV=test yarn run build && cypress run --env coverage=true && nyc report --reporter=html --reporter=text

[hangvar]

I'm having problems with:

TS2339: Property '__coverage__' does not exist on type 'Window'.

so I'm guessing my instrumentation is not working. Since I'm running Cypress 3.1.0 it would be really interesting if @paulfalgout could share a recipe project.

[paulfalgout]

@hangvar ah yep, so that code above assumes the src is instrumented. I used babel-plugin-istanbul to do so my .babelrc looks like:

{
    "presets": ["@babel/preset-env"],
    "env": {
      "test": {
        "plugins": [ "istanbul" ]
      }
    }
}

which is why I needed the cross-env NODE_ENV=test in the npm script.

However any form of instrumenting is fine, if you don't use babel, but do use webpack istanbul-instrumenter-loader will also work. instrumenting will make the __coverage__ variable attached to window as the tests are run.

The code above I added takes that __coverage__ variable and merges it together along the way, writing the merged result to a file.

Then the npm command uses nyc to make a report from that file.

I might be able to whip up a repo soon with an example.

[hangvar]

I'm having problems with getting the instrumentation to work despite your clarifications so a repo would be great!

[paulfalgout]

:tada: https://github.com/paulfalgout/cypress-coverage-example

[rwwagner90]

@atfzl your coverage code works, but my dist is now full of instrumented code. How do I make sure to not ship the instrumented version?

[paulfalgout]

In his example with webpack he's setting a flag isDebug to determine whether to add the instrumenter rule or not.

so in the webpack file there'll be something like const isDebug = (process.env.NODE_ENV === 'test');

With babel you'd throw it in the babelrc like:

  "env": {
      "test": {
        "plugins": [ "istanbul" ]
      }
    }

and you'd use something like: https://www.npmjs.com/package/cross-env to set that var in the npm script. cross-env NODE_ENV=test npm run-script coverage

Then _don't_ set the NODE_ENV to test when you're building for dev or prod

[rwwagner90]

@paulfalgout that worked, thanks! Now just one more thing I need to figure out. I have a bunch of mocha tests that run first, and I want to merge the coverage results from those into the report we generate from cypress. Any ideas how to do that?

[paulfalgout]

@rwwagner90 it can likely be done.. you'd need to, after the tests are run in both cases, combine their json output files.
It would be some node similar to what's in the cypress plugin

    const istanbul = require('istanbul-lib-coverage');
    coverageMap = istanbul.createCoverageMap({});

    on('task', {
      'coverage'(coverage) {
        coverageMap.merge(coverage);
        return JSON.stringify(coverageMap);
      }
    });

where you would read in both files and createCoverageMaps out of them and merge them together and write them back out to a single file that nyc uses.

[rwwagner90]

@paulfalgout I got it working with just nyc. It appears that if you drop coverage output into the coverage folder, nyc automatically merges that report into subsequent runs. It's all working over on https://github.com/shipshapecode/shepherd

[hangvar]

Thanks for all your help @paulfalgout. I managed to get a coverage file generated and nyc producing a report, by ejecting our angular 5 webpack.config.js and configuring instrumentation there. However, I'm dumbfounded by the fact that the coverage file is huge indicating coverage in more or less all our components, which can only lead me to believe that coverage from unit tests in Angular is aggregated with the Cypress coverage. I've struggled to separate these, since I want to be able to see my Cypress coverage only, but with no success. Any advice?

The aggregation is present when retrieving the coverageMap below:

coverageMap = istanbul.createCoverageMap({});

    on('task', {
      'coverage'(coverage) {
        coverageMap.merge(coverage);
        return JSON.stringify(coverageMap);
      }
    });

[paulfalgout]

Well if you can, disable your unit tests or unit coverage and try it again to see if there are any difference.

I suspect what you're seeing is the Cypress coverage cannot tell you what is not covered. It can only tell you what is never touched. This is because the integration of modules and starting your app will touch many lines of code, even before a cypress test is run.

So coverage in integration tests isn't necessarily as useful, but I find it invaluable as it points out likely areas where I'm missing tests and more importantly it helps me prove that I can test a difficult to cover logic branch.

Prior to coverage I had tests where I thought I was covering a particular scenario, but was missing a key step for a obscure use case we wanted to test for.

[frattaro]

Has anyone been able to get the nyc output appended to the cypress run output?

[pvdyck]

Any chance to have this working with an angular6 project ? Anybody could
point me in the right direction ?

On Fri, 14 Sep 2018 at 16:02 Anton Frattaroli [e-mail]
wrote:

Has anyone been able to get the nyc output appended to the cypress run
output?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cypress-io/cypress/issues/346#issuecomment-421368205,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcHmFgYlSHiMEYb5azZT01_Bp-MB5gKks5ua7b8gaJpZM4LNc-b
.

[rwwagner90]

@frattaro yes, we run both mocha tests and cypress tests and merge the coverage at https://github.com/shipshapecode/shepherd

[najisawas]

for use with create-react-app, you will need to yarn start with react-app-rewired.

then, in a config-overrides.js file parallel to your src folder, add the following:

const path = require("path");

module.exports = {
  webpack(config) {
    const istanbul = {
      test: /\.(jsx|js)$/,
      include: path.resolve(__dirname, "./src"),
      rules: [
        {
          loader: "istanbul-instrumenter-loader",
          options: {
            esModules: true,
          },
        },
      ],
    };

    config.module.rules.unshift(istanbul);

    return config;
  },
};

this is in addition to installing istanbul-instrumenter-loader, istanbul-lib-coverage, nyc, and adding the above cypress snippets written by @atfzl

[bahmutov]

This week I have made a presentation at Øredev showing what we think is a good replacement for code coverage for _end-to-end_ tests. Look at the slides here https://slides.com/bahmutov/well-tested-software and in particular read the following blog posts:

• Element coverage for end-to-end tests
• state coverage in blog post Hyperapp state machine web app

I think both element coverage and state machine coverage are a lot more meaningful than code statement coverage, and we will develop these ideas more in the future.

[frattaro]

@bahmutov i assume state coverage would need a lib for each state management solution? Vuex, redux, etc.

[bahmutov]

Even worse/better - it refers to modeling application as a state transition machines, something like “xstate” library for example

Sent from my iPhone

On Nov 22, 2018, at 18:22, Anton Frattaroli [e-mail] wrote:

@bahmutov i assume state coverage would need a lib for each state management solution? Vuex, redux, etc.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Vandivier]

@rwwagner90 merging mocha and cypress coverage - nice idea. I notice the cited repo now uses Jest. Is that a recent change? What was the reason for using Jest instead of Cypress for UI component testing? If you're not using Cypress for UI component testing does then what coverage are you merging?

[rwwagner90]

@Vandivier we're no longer merging coverage. We're just using Jest and it does its own coverage. We no longer try to get coverage out of Cypress.

[ifiokjr]

@rwwagner90 interested to know if there's any particular reason for not getting coverage from Cypress. I've contemplated putting something together. Is it not worth the effort?

[rwwagner90]

@ifiokjr we decided that ultimately, Cypress coverage was not a good representation of real coverage. You may "cover" 80% of lines just by booting the app up and visiting a page, but you're really only testing the functionality of one function. We decided coverage only made sense from unit tests, and we would wait for Cypress to expose more meaningful metrics like element coverage or application state coverage, as they mention here https://docs.cypress.io/faq/questions/general-questions-faq.html#Is-there-code-coverage

[frattaro]

Unit testing with cypress is a thing that can be done. Info doesn't directly impact the conversation but is something to consider.

[paulfalgout]

I find the coverage reports I can get from Cypress to be quite useful. Yes it doesn't tell you what is covered or not as false positives are high, but it can tell you what is _definitely not_ covered. In some cases I discovered tests we thought were covering the use case, but weren't. However I generate coverage reports two ways. I merge a report across the entire code base when run headlessly, but I also run coverage tests isolated on particular tests from the gui. The coverage reports help me write better tests and can in some instances help catch dead code. They don't do the same job as unit test coverage, but it's useful data none-the-less.

[papb]

I am using TypeScript, I tried to adapt the code by @paulfalgout to make it work with TypeScript but got stuck. Thankfully someone kindly helped me in my question on stackoverflow, so if anyone is trying to make Cypress + TypeScript + IstanbulJS work together, it can be done as well, just check the accepted answer :grimacing:

[bahmutov]

Hi everyone, I have blogged how to instrument application code and report code coverage from end-to-end tests:

• first read the blog post "https://glebbahmutov.com/blog/code-coverage-for-e2e-tests/" where instrumentation is done as a separate build step
• demo repo https://github.com/bahmutov/demo-battery-api branch coverage-step

Then read the following posts:

• "Code Coverage by Parcel Bundler" which shows how to instrument the application on the fly using babel-plugin-istanbul

• "Combined End-to-end and Unit Test Coverage" where both application and unit tests are instrumented using same code and the combined coverage report is saved - easily getting to 100% code coverage

Recommended plugin: https://github.com/cypress-io/cypress-istanbul for merging code coverage information into correct object

[Stephen2]

THANK YOU so much @bahmutov

[bahmutov]

Closing this issue, as we have released plugin and detailed documentation how to get end-to-end, unit and full-stack code coverage

• read our Code Coverage guide
• use @cypress/code-coverage plugin