Nightwatch: Coverage with Istanbul

[munkyjunky]

I'm currently implementing coverage reporting with Istanbul and have coverage being written on a per-test basis, using a custom command.

Is there a way to hook a function into a global afterEach hook that will run with the same browser context as the test ran with, so it would have access to the same window object in the browser (to access the coverage variable)?

Are there any plans on adding coverage reporting into Nightwatch directly?

[beatfactor]

There aren't immediate plans for this, but it is somewhere on the roadmap. You can probably achieve this today by using a combination of jscoverage (to instrument the code) and a custom lcov reporter or using mocha which already has an lcov reporter built-in.

Hooking a function in the afterEach hook is also possible before .end() is being called.

[munkyjunky]

Could you give a bit more explanation of how to achieve this?

My current setup runs a functional test against pre-instrumented code, and my test is followed by an afterEach hook doing the following:

'afterEach': function(browser, done) {

        browser.execute(function() {
            return window.__coverage__;
        }, [], function(response){

            fs.writeFileSync('coverage/report.json', JSON.stringify(response.value));
            done();
        });

    }

The issue I'm having is that browser.execute doesn't seem to share the same context as the initial test, so the window.__coverage__ variable no longer exists.

[beatfactor]

Can we move the discussion on the mailing list? Could you post a message there instead? Also indicate where are you calling .end(). Thanks.

[munkyjunky]

Aha! Mentioning the placement of the call to end helped me work it out - I was called end in my test (before the afterEach hook) which would have been destroying the session. Removing the call to browser.end() from my test has solved this!

[KBPratap]

@munkyjunky
I'm trying use istanbul with nightwatch in my project.
Any inputs on how to approach this?

[KBPratap]

@beatfactor
Is there a better way to get code-coverage using nightwatch?

[munkyjunky]

@KBPratap I did manage to get it working, but it's a bit of a faff to get working.

First you need to have a version of your website with _instrumented_ code. You then need to run nightwatch against this instrumented version of your site, and before calling .end() in your tests you need to get the coverage information out of the window and write it to a file. I did this by creating a custom command, .finish, which does this, then calls .end, and stores coverage information into a data store.

In the global after in nightwatch I then write out reports to the file system from the store of data.

This isn't a great method, but works for me. It has some limitations though, as if you get a full page load at any point in your test then you lose coverage information from the previous page. This wasn't an issue for my project, but may be for you.

I haven't been working with nightwatch in any projects recently so can't say if there's a better way of doing it, bit this worked for me.

var fs = require('fs');
var store = require('path/to/coverage/store');
var yaml = require('yaml-js');

var config = '.istanbul.yml';

/**
 * Wrapper around browser.end, which gives us chance to write coverage details before
 * the browser session is closed without having to manually do it in each test.
 *
 * @param callback {function}
 * @returns {exports}
 */
exports.command = function(callback) {

    var browser = this;

    fs.access(config, fs.R_OK, function(err) {

        var options = err ? {} : yaml.load(fs.readFileSync(config));

        // Make sure there's an instrumentation object
        if (!options.instrumentation) {
            options.instrumentation = {};
        }

        // Get the coverage from the browser
        browser.execute(function(coverageVariable) {

            return window[coverageVariable];

        }, [ options.instrumentation.variable || '__coverage__' ], function(response){

            // Make sure something was returned - this will fail if browser.end() has been called
            if (response.status === 0 && response.value !== null) {
                store.addReport({
                    name: browser.currentTest.module + '/' + browser.currentTest.name.toLowerCase().replace(/\W/gi,'-').replace(/\-+/g, '-'),
                    data: response.value
                });
            }

            // End the browser session
            browser.end(callback);

        });

    });

    return this;

};

[swetha21]

Can anyone help me out in code coverage using nightwatch??

[aberonni]

FWIW I setup an example repo that does something similar to what is illustrated above
https://github.com/aberonni/nightwatch-test-coverage-example