Istanbul: Add documentation for in browser usage

[asciidisco]

Hi,

first of all, pretty awesome project :)
My problem is, that I´am trying to use it as a middleware to collect some coverage informations
from my client side javascript & i´am missing some kind of docs as a starting point how to integrate
everything properly.

I can generate & run instrumented code in the client, but I´am not really sure how to generate a report out of that information. Would be really cool, if one of you guys could add some information on how to do this.

Regards

[gotwarlost]

Thanks.

Let me give you a quick summary for now.

• So it looks like you have figured out how to send instrumented code to the browser when running tests.
• When the code runs on the browser it will update a window.__coverage__ object with stats
• Once your tests are run, you need to send (POST) this object back to a server that can collect this information.
• Let's say that your back-end handler writes a coverage<nnn>.json file every time it gets a call (i.e. coverage1.json for the first set of tests, coverage2.json for the second set of tests and so on)
• After all your tests have run you have a bunch of coverage*.json files sitting somewhere
• Use the following command line to get reports out of it.

$ istanbul report --root /path/to/dir/containing/coverage/files --dir /path/to/output/dir

• Type istanbul help report for more info

Note that there is also a way to use istanbul as a library so that you can keep accumulating the coverage information in memory without writing files etc. I'm just keeping it simple for now.

Also look at the public API for istanbul: http://gotwarlost.github.com/istanbul/public/apidocs/index.html

That may help as well. Let me know if you have specific questions.

[gotwarlost]

BTW, I should add that the YUI team is working on integrating istanbul with yeti. Once that is done you can directly use it or look at the code and replicate what it does.

[asciidisco]

Thanks for the help.
I got a prototype working with QUnit in the back.

The only "error" that i encountered was an error that occured in the

lib/object-utils.js

File between line 57 & 67.
I needed to add a check for the 'statementMap' there:

                // CHANGES HERE
                if (statementMap[st]) {
                    var line = statementMap[st].start.line,
                        count = statements[st],
                        prevVal = lineMap[line];
                    if (typeof prevVal === 'undefined' || prevVal < count) {
                        lineMap[line] = count;
                    }
                }

I don´t know what happens, but without this check 'statementMap[st]' was undefined in some situations...

[gotwarlost]

That should never happen - could you attach a sample coverage object so I can take a look?

[asciidisco]

I´am on my way home right now, I will do so tomorrow.

[gotwarlost]

Hey are you still having this problem? Would like to see the coverage object and the file that the coverage was produced for.

[fjanon]

It's great news to have such a tool. I am very interested in using it to check the code coverage in the browser, I don't run any server javascript. Would you mind adding a step by step guide to using Istanbul in the browser? I think there will be a lot of interest in the tool.

[linusthe3rd]

+1 for documentation on how to get istanbul to cover code executed in the browser.

[jsoverson]

@asciidisco, not sure if you've seen grunt-template-jasmine-istanbul (off of grunt-contrib-jasmine) but we've been using istanbul via the browser/phantomjs for a little while to good effect.

[asciidisco]

@jsoverson Yep, I've seen it & yeaterdayI ported grunt-contrib-qunit to use Istanbul for code coverage analysis ;)
https://github.com/asciidisco/grunt-qunit-istanbul

Thx for the hint.

[asciidisco]

Also, for documentation purposes, if people looking for "in browser" code coverage, they should check http://blanketjs.org/

[linusthe3rd]

@asciidisco Thanks for the pointer - Blanket.js was really really awesome.

I ended up using JSCover to run my in browser tests because I needed LCOV reporting (which blanket.js doesn't support).

[grawk]

I am unable to figure out the arguments required for the instrument command. I've tried many combinations but keep getting the same message: Need exactly one filename/ dirname argument for the instrument command!

Can you provide an example of this command syntax?

Thanks!

[gotwarlost]

$ istanbul help instrument

Usage: istanbul instrument <options> <file-or-directory>

Options are:

      --output <file-or-dir>
              The output file or directory. This is required when the input is
              a directory, defaults to standard output when input is a file

      -x <exclude-pattern> [-x <exclude-pattern>]
              one or more fileset patterns (e.g. "**/vendor/**" to ignore all
              files under a vendor directory). Also see the --default-excludes
              option

      --variable <global-coverage-variable-name>
              change the variable name of the global coverage variable from the
              default value of `__coverage__` to something else

      --embed-source
              embed source code into the coverage object, defaults to false

      --[no-]compact
              produce [non]compact output, defaults to compact

      --save-baseline
              produce a baseline coverage.json file out of all files
              instrumented

      --baseline-file <file>
              filename of baseline file, defaults to
              coverage/coverage-baseline.json

[grawk]

Thanks. Wouldn't you know? I couldn't even figure out the help syntax!!! I'm clearly a guy who needs a lot of it.

[tonytaylor]

+1 for in browser support.

I'm in an environment that does not have (and afaik, will not have) node (and not sure if this is achievable w/o it).

[gotwarlost]

@tonytaylor not sure what you mean. How would you expect istanbul to work in a completely node-less environment?

You can get coverage for browser tests by:

• using the istanbul command line (istanbul instrument) to pre-instrument your JS files
• load these instrumented JS files in the browser and run your tests
• POST the window.__coverage__ object to a server and capture it on the back-end
• Run reports on that JSON (using istanbul report) to get the HTML/ lcov output

What am I missing?

By the way, the core instrumentation library will actually work inside the browser although I haven't actually tested this on all browsers. But for reporting you still need node and the istanbul command line.

[tonytaylor]

@gotwarlost Thanks for the confirmation.

[grawk]

For my case, we are using Node to do unit testing, but we need to run our JS unit tests in a browser, as the JavaScript relies upon the global window object and other DOM properties to run correctly. So we are using the mocha-phantomjs module to open the tests in phantomjs. When trying to generate coverage reports with Istanbul, the step where you have to post the window.coverage variable and handle that with some additional server endpoint is inconvenient compared to getting coverage reports via JSCover (for this type of browser environment unit testing). I just wanted to provide some more background on our use case as I don't think it's that unique.

[gotwarlost]

@grawk I'm not sure how JSCover works, could you tell me? Also, if you are running tests under phantom, you should be able to eval a JS snippet in the context of the window to return the window.__coverage__ object to the outside world so the POST won't be required in this case?

https://github.com/ariya/phantomjs/wiki/API-Reference-WebPage#evaluatefunction-arg1-arg2--object

From the docs:

"Note: The arguments and the return value to the evaluate function must be a simple primitive object. The rule of thumb: if it can be serialized via JSON, then it is fine."

/ping @ariya for better suggestions.

[ariya]

@gotwarlost That's correct, getting any variable from the page context back to the main PhantomJS script is that easy.

A real example (getting page's title): https://github.com/ariya/phantomjs/blob/master/examples/loadspeed.js#L16.

[tonytaylor]

@gotwarlost I've tried http://tntim96.github.io/JSCover/ as well. FYI, it's basically a coverage server in a jar.

[ckaatz-here]

I see the same issues when i run the $istanbul cover cobertura on my jenkins to create the reports ->

istanbul/lib/object-utils.js:57
Object.keys(statements).forEach(function (st) {
^
TypeError: Object.keys called on non-object

& after catching statements in the if check i get

istanbul/lib/object-utils.js:105
Object.keys(stats).forEach(function (key) {
^
TypeError: Object.keys called on non-object

any idea, what could be the reason for that?

[davidlinse]

What is the browser you're running the report in?

[ckaatz-here]

not in the browser, just in jenkins trying to pick up the coverage.json

[davidlinse]

sorry, mixed up two issues..

[gotwarlost]

Looks like istanbul report is trying to process a file that is not a valid coverage JSON file

[ckaatz-here]

unfortunately the JSON is valid

[rafeca]

I'm getting the same issue when running istanbul report from TeamCity:

node_modules/istanbul/lib/object-utils.js:57
            Object.keys(statements).forEach(function (st) {
                   ^
TypeError: Object.keys called on non-object

When I run it from my machine it works correctly. I've checked the coverage.json file and it's also correct.

[lawnsea]

I will file a new issue for this, but for posterity:

I experienced the above TypeError when I ran istanbul report in a directory that had a descendant config/grunt/coverage.json file. This was not a coverage report, so Bad Things happened. When I cded to a directory that did not have that descendant, things worked again.

[lawnsea]

Actually, no need for an issue, though the documentation could be clearer on this score. As istanbul help report will tell you, the default globbing pattern for finding coverage files is **/coverage*.json, which turns out to match config/grunt/coverage.json. The solution is to specify a glob that doesn't match, such as a filename: istanbul report the-coverage-data-you-were-looking-for.json.

[pgbakker]

sort of hijacking this thread, but we're using istanbul to instrument our js files, run them somewhere, collect the stats from window.coverage and write them to .json files. Then we run istanbul report against them and that works, but when we then view the generated HTML file for each source .js file, it is using the instrumented version of the code, instead of the uninstrumented code.

Must be something really silly what we're doing wrong, but we can't seem to figure it out...

Any thoughts?

[pgbakker]

Never mind... got it sorted...

[gotwarlost]

Closing this issue because it has now become a crazy mismash of all sorts of things :)

If anyone believes there is a problem that still needs solving please raise a new issue with the specific problem.

[krashi]

I am getting "null" value for
Object coverageObject = js.executeScript("return JSON.stringify(window.coverage);")

What could be the reason for this?

[patkujawa-wf]

@pgbakker I know this is an old thread, but I'm also interested in getting coverage without the normal unit test routines (i.e. via functional tests run by selenium and/or a user). Can you point me to any resources about how you all "got it sorted"? Thanks in advance! :)

[p-bakker]

Sorry, no idea anymore how we got it 'sorted'. Must have been something silly on our part, because I forgot and never ran into it again

[johnzheng1975]

I am disappointed that no one can provide a guide:
How to do istanbul browser testing step by step? ;-)

[moos]

Agree docs are lacking in for browser support. I've found this article useful, esp. step 3 for getting coverage stats out of test-runner into a file to be consumed by istanbul report.

[realnikunj]

@patkujawa-wf Were you able to do code coverage of your browser-based JS code with tests that run on Selenium (Java) or by user ?
I really need help in setting up the same. Thanks!

[patkujawa-wf]

I was not; priorities changed and I wasn't able to get far enough to get anything useful, unfortunately.

[realnikunj]

@patkujawa-wf I was able to conquer this one after a lot of hit and trials in various directions, so let me know if you would like to get some info on it. We can get in touch !
Our target code is written in Angular JS.

[sonu-sdz]

@patkujawa-wf Hi, do you have any blogs or documentation on the steps you followed. I'm working on something similar.

[patkujawa-wf]

I'm sorry, I don't. It was a while ago, and I never got very far.

[realnikunj]

@sonu-sdz I am putting together all the stuff in a blog soon, probably this weekend. Will share the link here, hope that helps!

[realnikunj]

Here it is - http://notjusttechstuff.blogspot.com/2017/10/browser-based-js-code-coverage-using.html

Would be happy to help if anybody faces any issue(s) while implementing.

[ORESoftware]

I wrote a nice blog post about using Istanbul + Selenium + Front-end code coverage:

https://medium.com/@the1mills/front-end-javascript-test-coverage-with-istanbul-selenium-4b2be44e3e98