Istanbul: Better integration with other pre-processors

[schulzch]

When integrating Brunch, Karma and Istanbul precious information from source maps is lost. I think its a good idea to look for source maps during instrumentation (app.js.map for app.js) and display sources instead of intermediate code.

We use Brunch to compile the application (bare vendor.js and requirejs app.js) and Karma to compile tests, instrument app.js using Istanbul and run tests inside Chrome/Firefox/etc.. Unfortunately the coverage report and source view (developer tools) display a single app.js instead of source files. Random guessing what the source file was is a real pain in the *, not to mention virtually useless statistics.

This possibly relates to:

• Issue with generating lcov/html reports for traceur-compiled files #162
• Support source maps for html report #59
• Support for CoffeeScript, TypeScript
• Enterprisy tool chains

[gotwarlost]

Is there a sample project and a build script that will help me understand all the moving parts without having to learn a bunch of tools? I realize that you are asking for something more abstract but I like to get my hands dirty with your use-case before trying to come up with a solution.

[justindujardin]

@gotwarlost see pow-core for an example, and an example report.

It would be much nicer if istanbul could look at my source maps and generate reports that include the original TypeScript files instead of one composite javascript file.

Clone the repo, set coverageDebug = true at the top of karma.config.js, and npm install && npm install bower && ./node_modules/.bin/bower install. Then npm test whenever you want to generate a new report, which will show up in .coverage

+1 for this feature.

[jessaustin]

@justindujardin I agree this would be a nice enhancement for istanbul. This is just a quick note to make installation of pow-core easier. Add a postinstall property to the scripts property of package.json:

  "scripts": {
    "postinstall": "bower install"
  }

Now a simple npm install is all you need!

[chinchang]

+1
Right now I need to run tests on development environment where there is no concatenation and minification so that I get coverage for source files.

[schulzch]

@gotwarlost pow-core matches my use case more or less (its using TypeScript instead of CoffeeScript and Grunt instead of Brunch).

One way to implement this feature would be by using Mozilla's Source Map parser:

1. Parse source map in instrumenter.js:instrumentSync()
2. Annotate each uninstrumented AST node with a _source line_ and _source column_
3. Modify this.sourceMap in instrumenter.js:instrumentASTSync() so that instrumented code points to _source code_ instead of _uninstrumented code_

Viable approach?

[vvakame]

:+1:

[vvakame]

@gotwarlost I know good document for SourceMap.
I think gulp ( and browserify ) is defacto of streaming processing tools. I think that it is good when you reference the gulp of policy.

https://github.com/floridoo/gulp-sourcemaps/wiki/Plugins-with-gulp-sourcemaps-support
http://thlorenz.com/blog/browserify-sourcemaps
https://github.com/azu/multi-stage-sourcemap
https://github.com/twada/battlefield-sourcemaps

in karma-sourcemap.
https://github.com/demerzel3/karma-sourcemap-loader/blob/2c9ee367aa9fcf7cf65bf8c224d5c89b7f151d12/index.js#L9

I want to use SourceMap with TypeScript and karma ( karma-sourcemap -> karma-coverage (it uses istanbul!) ).

---

this is very useful document if you can read japanese...
http://efcl.info/2014/09/03/multi-stage-sourcemap/

[fkling]

:+1: Since transpilers became increasingly popular this year, having support for source maps would be tremendously useful.

[lo1tuma]

:+1:

[arielschiavoni]

+1

[dayyan]

:+1:

[GoulwenBZH]

+1

[simonzack]

:+1:, this really makes it hard with 6to5 or traceur. I cannot switch to istanbul due to this issue.

[jvilk]

:+1: I completely agree. It seems like this could be a post-processing step, since Istanbul already has the relevant JS source location information for covered / non-covered statements. I believe you can map those back to the original source code via the source maps.

[jtangelder]

:+1:

[Iorg]

+1

[danvk]

I wound up writing a script to apply a source map to LCOV data to get coverage data for a recent project. Built-in support from Istanbul would be fantastic.

[gotwarlost]

@danvk - this is great! This gives me a good starting point. Thanks!

[brunodunbar]

This feature is gonna be very helpful, especially when running tests on minified/uglified code.

[gotwarlost]

Working on it in the source-map branch... Code is still very raw and not ready for use.

[mysterycommand]

:+1:

[tiriana]

+1 Using source maps would increase built code coverage significantly. Are there any chances for this to happen? [typescript => js]

[wojciak]

+1 (ts -> js)

[topaxi]

+1 https://github.com/douglasduteil/isparta is doing something like this already for babel, but sadly isn't an option for me anymore

[PinkaminaDianePie]

+1 this feature more important than any other

[rattrayalex]

+1 TS

Any word on progress?

[acromarco]

+1 (ts -> js)

[splintor]

+1 (ts -> js)

[lzilioli]

+1, I'm using babel to compile es6 to es5 and then running the tests against the es5 babel output

[loicoudot]

+1 TypeScript will be very popular soon with AngularJS 2.0

[kitsonk]

@gotwarlost I noticed the source-map branch. I have been working on theintern/intern and dojo/core and being based on TypeScript, being able to map back to source for coverage is important. @bryanforbes and I have been looking at how we can include automated code coverage to our tooling chain. Right now I have hacked together kitsonk/grunt-cover-ts which tries to remap the lcov.info, but then realised that the coverage.json is better and was starting to look at remapping that back to source. I would be glad to help out with the source-map branch but would like to understand how best to contribute.

[PinkaminaDianePie]

@kitsonk some services like coveralls.io uses only lcov.info. So i think you need to remap both lcov.info and coverage.json

[bryanforbes]

@PinkaminaDianePie The process that @kitsonk is describing is simply what he has done in grunt-cover-ts (and what his thoughts were) to transform the coverage information after Istanbul outputs it. Since lcov doesn't have all the info needed to map back to the pre-transpiled source, his next steps were going to be to take use coverage.json to map back and then output whatever coverage format is needed (I would assume using Istanbul). Essentially:

1. Run tests, generate coverage.json
2. grunt-cover-ts uses coverage.json + source maps to map back to the TypeScript
3. grunt-cover-ts then uses a modified coverage object and hands it to Istanbul to generate the appropriate coverage report (lcov, etc.)

The source-map branch adds the idea of a transformer to the Istanbul API which would take the coverage object and manipulate it before handing it to a reporter (such as lcov), thus making grunt-cover-ts unnecessary.

[PinkaminaDianePie]

@bryanforbes ah, ok.

[gotwarlost]

For the (ts -> js) folks, can someone provide a git repo with some hello world code and simple tests that I can use as an example to test some source-map support I recently added on the branch?

[lukeapage]

maybe this is too much because it integrates with grunt-istanbul etc. but it might be useful as its a small project - https://github.com/lukeapage/vindinium-ai

[gotwarlost]

@lukeapage - I tried your repo and used a small subset of your code and tests to validate my ideas. Thanks for the info.

[gotwarlost]

@kitsonk - I was actually able to get "good" reports on your dojo/core library (you need to verify that they are _correct_) by doing the following:

• remove the istanbul ignore replacements in your gruntfile
• replace the intern istanbul dependency with the source-map branch on github

Result of a grunt test is that you get to see coverage for all your original files (.ts and not .js, I mean) with no "boilerplate coverage"

Could you give it a whirl and let me know if things look correct?

[bryanforbes]

@gotwarlost - @kitsonk and I are working together on coverage for dojo/core and he's on holiday so I tested your changes. This is amazing and it looks correct. I'll have some other colleagues try it out on other projects (dojo/dom, dojo/router, etc.) and let you know if we run into any problems. Thanks so much for doing this!!

[gotwarlost]

@bryanforbes - excellent, thanks for the quick test. Looking forward to more feedback.

[bryanforbes]

For anyone else trying this with Intern, you can do the second step of the instructions posted by @gotwarlost by doing the following:

cd node_modules/intern
npm install gotwarlost/istanbul#source-map

[dpogue]

I'm currently using istanbul as a two-step process:

1. Compile TypeScript to JS and add istanbul instrumentation using browserify-istanbul.
2. Run the tests and collect coverage stats using karma-coverage.

Will having those as separate steps cause problems with the source map caching? If you think it should work, I'm happy to spend a bit of time testing it

[gotwarlost]

This use case should simply work as long as you use the source-map branch. Try it out and let me know. Thanks for volunteering to test this.

[dpogue]

It seems to have problems when directories are involved, I'm getting the following error:

Post-processing using source maps
ERROR [coverage]: { [Error: ENOENT, no such file or directory '/project_dir/src/alert/src/alert/alert.ts']
  errno: -2,
  code: 'ENOENT',
  path: '/project_dir/src/alert/src/alert/alert.ts',
  syscall: 'open' }
Error: ENOENT, no such file or directory '/project_dir/src/alert/src/alert/alert.ts'
    at Error (native)
    at Object.fs.openSync (fs.js:500:18)
    at Object.fs.readFileSync (fs.js:352:15)
    at LookupStore.Store.mix.get (/project_dir/node_modules/karma-coverage/node_modules/istanbul/lib/store/fslookup.js:40:19)
    at HtmlReport.Report.mix.writeDetailPage (/project_dir/node_modules/karma-coverage/node_modules/istanbul/lib/report/html.js:406:83)
    at /project_dir/node_modules/karma-coverage/node_modules/istanbul/lib/report/html.js:484:26
    at SyncFileWriter.extend.writeFile (/project_dir/node_modules/karma-coverage/node_modules/istanbul/lib/util/file-writer.js:57:9)
    at FileWriter.extend.writeFile (/project_dir/node_modules/karma-coverage/node_modules/istanbul/lib/util/file-writer.js:147:23)
    at /project_dir/node_modules/karma-coverage/node_modules/istanbul/lib/report/html.js:483:24
    at Array.forEach (native)

All of the sources in the sourcemap are relative to the main project directory (/project_dir), but it seems to be trying to map them as if they were relative to the file being covered.

[gotwarlost]

@dpogue - thanks for the report. Could you give me some more info

• what options are you passing to the typescript compiler?
• are your source maps inlined or are they present in separate .map files?
• If it is the latter, do the .map files have a correct relative path to the .ts files?

[dpogue]

The compiler options for TypeScript are:

target: es5
module: commonjs
declaration: true
noImplicitAny: false
removeComments: false
noLib: false
outDir: dist
sourceMap: true

The sourcemaps are inline. I de-base64'ed the inline sourcemap and the relative paths are correct (relative to the project root).

[gotwarlost]

For the babel + karma folks, I have a stupid simple repo that demonstrates how to get this to work with source maps. Any takers for testing on a real project?

Thanks to @ezequiel who set this up as a simple example.

https://github.com/gotwarlost/babel-coverage

[gotwarlost]

@dpogue what's the version of typescript that you are using?

[bryanforbes]

@gotwarlost - I spoke too soon. One test run works fine (for instance grunt test or grunt test-local), however when a test run's final coverage information is output to JSON and then combined with the coverage information for a second test run, the coverage information for the first run is the only one that survives. To see what I'm talking about, follow these steps:

• Modify dojo/core/Gruntfile.js and change lcovhtml to combined in all three places
• Ensure selenium-standalone is installed (brew install selenium-standalone if you're on OS X)
• Ensure chromedriver is installed (brew install chromedriver)
• Run selenium-standalone -port 4444 in one terminal
• Run grunt clean:coverage test test-local to run the node and local selenium tests tests (this will output coverage-final.json for the node run and html-report for the browser run)

Once everything is complete, check html-report/src/queue.ts.html. You'll notice that line 173 is not covered. If you change Gruntfile.js back to lcovhtml and run grunt clean:coverage test-local again, you'll see that line 173 is covered by the code. I believe this happens because the post-processed coverage information from the first run is being combined with the pre-processed coverage information of the second run.

[gotwarlost]

@bryanforbes - this is exactly the kind of testing I was looking for. Thanks for the report. I'll look into it this evening.

[gotwarlost]

@dpogue - one more question. Does the de-base64'ed source map have a sourceRoot attribute?Currently I don't process this and assume that the source map is relative to the "place" where I got it from (path of source file that has the inline map, path of map file etc.)

[dpogue]

sourceRoot is "" (empty string)

[splintor]

I tried to used it with Karma, and got this exception:
TypeError: Cannot read property 'split' of undefined
at HtmlReport.Report.mix.writeDetailPage (WebClientnode_modules\karma-coveragenode_modulesistanbul\libreport\html.js:410:30)
at WebClientnode_modules\karma-coveragenode_modulesistanbul\libreport\html.js:487:26
at SyncFileWriter.extend.writeFile (WebClientnode_modules\karma-coveragenode_modulesistanbul\lib\util\file-writer.js:57:9)
at FileWriter.extend.writeFile (WebClientnode_modules\karma-coveragenode_modulesistanbul\lib\util\file-writer.js:147:23)
at WebClientnode_modules\karma-coveragenode_modulesistanbul\libreport\html.js:486:24
at Array.forEach (native)
at HtmlReport.Report.mix.writeFiles (WebClientnode_modules\karma-coveragenode_modulesistanbul\libreport\html.js:480:23)
at WebClientnode_modules\karma-coveragenode_modulesistanbul\libreport\html.js:482:22
at Array.forEach (native)
at HtmlReport.Report.mix.writeFiles (WebClientnode_modules\karma-coveragenode_modulesistanbul\libreport\html.js:480:23)

From what I checked by tweaking the code, fileCoverage.code is undefined, and fileCoverage.path is a path to one of my test source files, which are also written in TypeScript (could this be the reason?). sourceStore.sourceCache contains the generated *.js file as a key, but not the *.ts, so sourceText becomes undefined.

[gotwarlost]

@splintor do you have sourceMap enabled somehow in the karma config (this is the directive that tells karma to emit a source map for the instrumented code, not consume one). If so, could you turn it off and see what happens?

[splintor]

@gotwarlost No. TypeScript files are compiled in a different Grunt task. Karma files contain only *.js files.
It it helps, my *.js files all end with a line like this:
//# sourceMappingURL=myModule.js.map

The *.js and *.js.map reside next to the *.ts files, and the map file begins with:
{"version":3,"file":"myModule.js","sourceRoot":"","sources":["myModule.ts"],"names":

[gotwarlost]

@bryanforbes @dpogue - could you please retest with the changes that I pushed to the branch?

The API is now completely muddied after these changes but I'm trying to get something to work before I make it better :)

[gotwarlost]

@splintor - could you do me a favor and use git://github.com/gotwarlost/karma-coverage.git#source-lookup as your karma-coverage dependency and see if things get any better?

[bryanforbes]

@gotwarlost - Two things are wrong. First, either intern or grunt is interpreting the Post-processing using source maps as an error or a warning. Did you change what output that's going to? This isn't a huge deal. Second, I'm getting strange output in the html-report directory. I can upload the content somewhere, however, the gist of it is that only the following HTML files are generated:

• html-report/index.html
• html-report/Projects/dojo/src/index.html

The rest of the report files are missing. This happens when generating the HTML report (lcovhtml) for node, browsers, and also for combining the results of node and the browsers (combined). Let me know if you need more info.

[dpogue]

@gotwarlost It seems to be working wonderfully now! Thanks!!

[splintor]

@gotwarlost Tried. Some improvement (I guess). Now I get list of messages for all the typescript files in the project:
Unable to load source map from: <root location>\<typescript filename>.js.map

It looks for the map file in the root location (where package.json and grunfile.js exist) and not in its location next to the *.ts (and transpiled *.js) files.

[gotwarlost]

It looks like different people are seeing different ways in which relative paths to sources ought to be resolved from the source map object/ map file.

I pushed a commit to try multiple paths to find the file. This is gross because there should ideally be one way to do it but I can't seem to find a spec that talks about this.

In any case, could you guys try your tests again with the latest commit? Thanks for staying with me on this. Also @dpogue - please ensure that the current working setup has not regressed.

[bryanforbes]

@gotwarlost - The current code in the source-map branch seems to work well.

On the subject of resolution, from what I know about source maps the resolution should be along the lines of:

• If sourceMappingURL is base64 encoded
  
  
  
  • let sourceMapDirectory equal the directory of the file
  
  
  • let sourceMap equal the result of decoding sourceMappingURL
  
  
• if sourceMappingURL is a path
  
  
  
  • let sourceMapPath equal the result of resolving sourceMappingURL against the directory of the JS file
  
  
  • let sourceMapDirectory equal the directory of sourceMapPath
  
  
  • let sourceMap equal the contents of the file at sourceMapPath
  
  
• let sourceRoot be the result of resolving sourceMap.sourceRoot against sourceMapDirectory
• let i equal 0
• while i < sourceMap.sources.length
  
  
  
  • let path equal sourceMap.sources[i]
  
  
  • let sourcePath equal the result of resolving path against sourceRoot
  
  
  • if sourceMap.sourcesContent exists
  
  
  • let sourceContent equal sourceMap.sourcesContent[i]
  
  
  • if sourceMap.sourcesContent doesn't exist
  
  
  • let sourceContent equal the contents of the file at sourcePath
  
  
  • increment i by 1
  
  

If that resolution doesn't work, then I would assume whatever is creating the source maps is creating them with the wrong sourceRoot and sources.

[splintor]

@gotwarlost First of all, thanks for doing this for us. Having a coverage that will show TypeScript files would be awesome.

Tried the last commit - now the ts file is not found, as it is searched in the root location and not it is proper location in the project:

Error: ENOENT, no such file or directory '<root location>\<typescript filename>.ts'
    at Error (native)
    at Object.fs.openSync (fs.js:500:18)
    at Object.fs.readFileSync (fs.js:352:15)
    at LookupStore.Store.mix.get (<root location>\node_modules\istanbul\lib\store\fslookup.js:40:19)
    at Store.mix.get (<root location>\node_modules\karma-coverage\lib\source-cache-store.js:36:26)
    at HtmlReport.Report.mix.writeDetailPage (<root location>\node_modules\istanbul\lib\report\html.js:406:83)
    at <root location>\node_modules\istanbul\lib\report\html.js:484:26
    at SyncFileWriter.extend.writeFile (<root location>\node_modules\istanbul\lib\util\file-writer.js:57:9)
    at FileWriter.extend.writeFile (<root location>\node_modules\istanbul\lib\util\file-writer.js:147:23)
    at <root location>\node_modules\istanbul\lib\report\html.js:483:24

[bryanforbes]

@splintor What parameters are you passing to tsc or (if you're using it) what are your compilerOptions in tsconfig.json?

[splintor]

@bryanforbes Not using tsconfig.json. Using grunt-ts, with basic settings:

ts: {
        build: {
            src: ["src/modules/**/*.ts"]
        }
}

*.js and *.js.map files are generated at the same location as the *.ts files.

[bryanforbes]

@splintor Can you verify that for a TS file at src/modules/foo.ts, the following is true (substitute an actual file for foo.ts, foo.js, and foo.js.map below)?

• src/modules/foo.js exists and the last line is //# sourceMappingURL=foo.js.map
• src/modules/foo.js.map exists and the object contained within has a property sourceRoot that is set to "" and a property sources that is set to ["foo.ts"]

I tested with dojo-core and changed some of the compile settings to get it to output .js and .js.map files alongside the files in src and test and the routine in the current source-map branch seems to work. I'd be interested to see your project to look at what you're doing.

[dpogue]

Everything is still working for me with the latest changes :)

[bryanforbes]

@gotwarlost - I just looked at rawMapFromUrl() in the source-map branch and I made some improvements to it so that it works for source + source map + js, source + inline source map + js, and source + inline source map + inline sources:

function rawMapFromUrl(url, base) {
    url = (url || '').toString();
    var match = DATA_URI_RE.exec(url),
        contents,
        baseDir = base,
        file,
        obj;

    if (match) {
        contents = new Buffer(match[1], 'base64').toString();
    } else {
        file = path.resolve(baseDir, url);
        contents = loadRawMapFromFile(file);
        if (!contents) {
            return null;
        }
        baseDir = path.dirname(file); // resolve sources in JSON relative to map file
    }
    try {
        obj = JSON.parse(contents);
    } catch (ex) {
        console.error('Unable to parse JSON for source map');
        return null;
    }

    if (obj.sourceRoot) {
        baseDir = path.resolve(baseDir, obj.sourceRoot);
    }

    if (obj.sources && Array.isArray(obj.sources)) {
        obj.sources = obj.sources.map(function (s) {
            return path.resolve(baseDir, s);
        });
    }

    return obj;
}

This alleviates the need for firstFileOf().

[bryanforbes]

@splintor and @dpogue - I created a fork, a branch, and integrated the above change into it. Can you see if installing bryanforbes/istanbul#source-map-resolve works with your projects?

[gotwarlost]

@bryanforbes - thanks! waiting for others to confirm that this works.

Should we nuke the obj.sourceRoot property at the end given that we explicitly change the file names for the sources? Otherwise, someone will (rightly) try to resolve an already resolved file against the pre-existing sourceRoot

[dpogue]

@bryanforbes I'm getting that same path error with your branch :(

Post-processing using source maps
ERROR [coverage]: { [Error: ENOENT, no such file or directory '/project_dir/src/alert/src/alert/alert.ts']
  errno: -2,
  code: 'ENOENT',
  path: '/project_dir/src/alert/src/alert/alert.ts',
  syscall: 'open' }
Error: ENOENT, no such file or directory '/project_dir/src/alert/src/alert/alert.ts'

[bryanforbes]

@gotwarlost - More than likely, yes. However, I just thought about this: should this routine just be updating the sourceRoot location? It looks like SourceMapConsumer does the rest of the resolution for us.

[bryanforbes]

@dpogue Do you have your project somewhere public where I can check it out? I'd like to see what's going on.

[dpogue]

It's an internal project so it's not public, but I'll try to put together a sample repo that's set up the same way later tonight.

[bryanforbes]

@dpogue Thanks! I'm interested to see what's going on.

[splintor]

My bad. Someone on the team removed a *.ts file from source control as part of refactoring, and when I got his changes, the *.ts was removed from my file system, but the generated *.js and *.js.map files remained, and this is what caused the no such file or directory I got for the *.ts file, which really no longer existed. I really need to add a "clean" task to our grunt "build" task to remove all generated *.js and *.js.map files before building, to make sure this doesn't happen.
Once I removed the orphan js file, ts coverage worked like a charm, and I was able to see *.ts file in the coverage report - thanks!

I do have one wish, though. When compiling a typescript file like this:

module myModule {
    export interface IAuthorizationChangeHandler {
        (boolean):void;
    }
    export class MyClass {
        //...
    }
}

the generated js looks something like:

var myModule;
(function (myModule) {
    var MyClass = (function () {
        //...
    })();
    myModule.MyClass = MyClass;
})(myModule || (myModule = {}));

However, if more than one class is defined in that module, the myModule = {} part at the end is not executed for that class, and it is reported as an uncovered branch. However, I would really like this to not be reported, as it seems like an unimportant typescript compiler implementation detail.
I hoped that once we move to show ts coverage instead of js coverage, this would go away, but now the module name is highlighted with a branch not covered message.
So I guess the coverage data remains the same, and we are now just showing is differently. My question is: Is there a way to remove this "branch not coverage" message from such typescript generated code? (I'm aware that maybe here is not the right place to ask...)

[loicoudot]

Great job @bryanforbes @gotwarlost !!
With :

"istanbul": "bryanforbes/istanbul#source-map-resolve"
"karma-coverage": "gotwarlost/karma-coverage.git#source-lookup"

It's work perfectly on our internal framework (~10 000 TS LoC).

Like @splintor says, branch not covered message appear for each Typescript module. But, IMO, it's more a job for the javascript instrumentation but it have to know that Javascript came from transpiled Typescript. Other transpiled langage may have some other false alarm like this.

Thank you all

[gotwarlost]

@splintor @loicoudot - when a source map is found for an instrumented file, istanbul will throw away statements, functions, branches that do not map to anything in the source. This is how most of the boilerplate is thrown away.

Sounds like what _might_ be happening is that the typescript compiler generates a source map entry for the (myModule || (myModule = {})) section. I would need to look at a simple example to see what is going on. Could you make one up or point me to one that already exists on github?

[justindujardin]

@gotwarlost @bryanforbes This works as expected, thanks very much.

[splintor]

@gotwarlost Check out https://github.com/splintor/BaseCalc/

[blakeembrey]

@gotwarlost @bryanforbes Just tried it (gotwarlost/istanbul#source-map) with a project and it looks great :+1:

[zakhenry]

@gotwarlost @bryanforbes I too have tried (gotwarlost/istanbul#source-map & gotwarlost/karma-coverage.git#source-lookup) with both typescript and javascript files with gulp (over at https://github.com/spira/spira/tree/feature/typescript-rewrite) and it worked beautifully :)

[gotwarlost]

@splintor - thanks for the example. What I suspected is what is happening. For the engine.ts source mapping. The main || (main = {}) sequence is mapped to line 1

{
    "source": "engine.ts",
    "generatedLine": 21,
    "generatedColumn": 3,
    "originalLine": 1,
    "originalColumn": 7
}
{
    "source": "engine.ts",
    "generatedLine": 21,
    "generatedColumn": 7,
    "originalLine": 1,
    "originalColumn": 11
}
{
    "source": "engine.ts",
    "generatedLine": 21,
    "generatedColumn": 12,
    "originalLine": 1,
    "originalColumn": 7
}
{
    "source": "engine.ts",
    "generatedLine": 21,
    "generatedColumn": 16,
    "originalLine": 1,
    "originalColumn": 11
}

[PinkaminaDianePie]

I had used isparta with babel before, but it seems to be broken now (my tests are passes under mocha but some of them fails under isparta) and i can't check my coverage. So i waiting for merge source-map with istanbul's master.

[splintor]

@gotwarlost Thanks. So what can we do about it? Open an issue at https://github.com/Microsoft/TypeScript?

[bryanforbes]

@splintor Yes. Make sure you have a simple code sample that demonstrates the problem. The more straightforward, the better. It looks like the following is enough to demonstrate the problem:

main.ts

module main {
    export class A {}
}
module main {
    export class B {}
}

main.tests.ts

describe("problem", () => {
    it("should have an A class", () => {
        var a = new main.A();
        expect(a instanceof main.A).toBeTruthy();
    });
    it("should have a B class", () => {
        var b = new main.B();
        expect(b instanceof main.B).toBeTruthy();
    });
});

In the coverage report, main is highlighted in both blocks as not being covered. I haven't dug into this case, however I would assume that the initial case of main || (main = {}) isn't being covered in the first module block, and the secondary case isn't being covered in the second module block.

[bryanforbes]

@gotwarlost Is there any other work that needs to be done to get this (and possibly the work in my branch) into master?

[kitsonk]

Sorry I was on vacation, but it looks like @bryanforbes worked with it. The only issue I am seeing though with bryanforbes/istanbul (and I assume the source-map branch in general) is that html-report, lcov.info and the console output all "just work" but coverage.json is not getting remapped. I have a working repo I have been using for testing, of which grunt test will output the html-report, lcov.info and coverage-final.json as well as display the coverage in the console.

I will try to investigate why it isn't dealing with the coverage-final.json but it might be more obvious to others.

[kitsonk]

I also noticed that while the lcov.info line coverage numbers are getting remapped properly the function (FN) and branch (BRDA) line numbers are not getting remapped.

[gotwarlost]

@bryanforbes - yes, the current integration on the source-map branch is complete crap. As @kitsonk correctly says, the JSON doesn't get remapped although all other reports do. This is because intern uses the JSON to merge coverage objects as well as use it as a source of reporting. Now that we have source maps, we need to distinguish between the merged JSON (that refers to the unmapped files) and the final JSON output (that contains the result of the mapping)

[bryanforbes]

@gotwarlost Ok, so what needs to happen? I have some time this week that I can use to look at this and I'd like to help.

[gotwarlost]

@bryanforbes - sorry for the delay. I'm working with @davglass to come up with a better open-source model for istanbul so we can enable some "guts-level" hacking by the community that is sorely needed given the scope of these changes. The lone-hacker-at-the-helm model is just not scaling at this point, especially given my day job commitments :(

[gotwarlost]

Which is to say: I really need a lot of help from you and other guys.

[douglasduteil]

Hi @gotwarlost

Sorry for hacking around so much. I'm definitely quitting hacking on this without you. It's really time consuming...

So yeah feel free to ping me if you need a hand.
Right now I just curious about the state of your source-map branch it seems pretty solid already :+1:

[Sabartius]

@gotwarlost I'm currently using your istanbul#source-map and karma-coverage#source-lookup branches with an angular2 typescript application and it's working quite well.

But one thing is broken: ignoring code from coverage.
Comments like /* istanbul ignore else */ oder /* istanbul ignore next */ are not working at all!

When i switch back to using the Istanbul and Karma-Coverage Master-Branches, everything works as expected (except the source-mapping ;-) ).
Hope you can fix that :)

[adrigzr]

Hi @gotwarlost,

I'm currently using istanbul#source-map and karma-coverage#source-lookup, and I have a few issues with them.

First of all, I got the following error when I try to use any type of html reporter.

ERROR [coverage]: [TypeError: Cannot read property 'text' of undefined]
TypeError: Cannot read property 'text' of undefined
    at /project/node_modules/istanbul/lib/report/html.js:283:53
    at Array.forEach (native)
    at annotateBranches (/project/node_modules/istanbul/lib/report/html.js:250:30)
    at HtmlReport.Report.mix.writeDetailPage (/project/node_modules/istanbul/lib/report/html.js:421:9)
    at /project/node_modules/istanbul/lib/report/html.js:484:26
    at SyncFileWriter.extend.writeFile (/project/node_modules/istanbul/lib/util/file-writer.js:57:9)
    at FileWriter.extend.writeFile (/project/node_modules/istanbul/lib/util/file-writer.js:147:23)
    at /project/node_modules/istanbul/lib/report/html.js:483:24
    at Array.forEach (native)
    at HtmlReport.Report.mix.writeFiles (/project/node_modules/istanbul/lib/report/html.js:477:23)

And secondly, I'm using a bundled javascript file with handlebars templates compiled inside with sourcemaps and I can't get rid of them in the coverage report. It is really anoying. Is there any solution about it?

[kitsonk]

@bryanforbes and I worked together to build SitePen/remap-istanbul and have made it publicly available. I think it should solve most of the use cases mentioned here and doesn't require any direct changes to Istanbul. It take the JSON coverage report from Istanbul, reads any source maps and remaps the coverage and then can utilise Istanbul to write out new reports.

We will have it available on npm later today/tomorrow. @gotwarlost feel free to incorporate the logic directly into Istanbul if you feel appropriate (or we would be glad to help). It works under AMD or CommonJS as well as Grunt and Gulp.

[kitsonk]

remap-istanbul is now available on npm. Sorry for the delay.

[daniel-a-melo]

I have just successfully used remap-istanbul on my project. Kudos to @kitsonk and @bryanforbes.

[kinday]

remap-istanbul is great. I plugged it into istanbul and added --remap option in the PR above.

I tried it with my ES6 project and it’d be great to get remapped reports out of the box. Test it and you’ll love it too!

[MrOrz]

I think I found an interesting use case when remap-istanbuldoes not apply while istanbul's source-map branch does.

In my scenario, I use require('babel/register')({sourceMaps: 'inline',}) in my test specs and imports the untranspiled ES6 scripts directly from my test spec. The source maps are generated on the fly when istanbul runs. As istanbul terminates, the source maps are gone as well. remap-istanbul, however, expects the sourceMapUrl to exist in the source files, thus it always generates empty coverage reports. On the other hand, istanbul's source-map works great because it can access the sourceMapUrl generated by babel/register while istanbul is still running.

[bnolan]

I just modified that babel-coverage demo to show how it works with typescript and it works great! +1 to merge the sourcemaps branch into master!

https://github.com/bnolan/babel-coverage

[ghost]

:+1:

For my scenario a concatenated artifact is written to a ./dist directory. The concatenated file is built using gulp (which uses babel for transpiling). Along side the concatenated file is a .map source map file.

I then invoke karma separately to run my unit tests and generate coverage reports.

Using the source-map branch I'm able to generate HTML coverage reports that individually list the original source files, thereby allowing me to drill down into individual files, rather than navigating the concatenated file as a single report.

@gotwarlost though I would love to see this merged sooner than later, would you, in the meantime, consider rebasing it on master, and filing a merge request so that we can start to provide feedback on the code?

[alexgorbatchev]

I got remap-istanbul to work successfully, however the big problem for me is that instrumenting code doesn't update the sourcemaps and line number in stack traces get out of sync.

[faergeek]

@gotwarlost Do you still need help with source maps or you decided to go another way?
I would like to help with implementation, I'm using your source-map branch for now.

[gotwarlost]

Right now I'm in the middle of breaking out istanbul into several smaller components; merging the source-map branch as-is into master will lead to a Frankenstein API and difficult-to-support/debug scenarios.

I have most of the bits in place to replace the current functionality (without source-maps) and will open this up soon. After that I'll ask for help from people to properly introduce source maps in the code base.

There are already some really good PRs and the remap-istanbul module the meat of which need to be folded into istanbul for a seamless experience.

Given work pressures etc. I'm not able to commit to hard timelines but expect an update in the next week or two.

[faergeek]

Cool, now I'm curious on which components you're going to split it :-)

I looked at source-map branch and wanted to note that you need to support cases when sources are inlined in source map, seems like remap-istanbul does not support that either.

That's the main problem why webpack is not supported right now, as it inlines it's bootstrap code at least.

So, there's a need to read source maps and remap earlier to have access to sourcesContent property in which sources are inlined.

One more thing to note is that in current implementation istanbul does not allow us to exclude some files from reporting/instrumenting.
I'd like to exclude node_modules dir and some other things, I used instrumentation.excludes option usually for that, but with source maps it's not possible for now.
So, although you really instrument bundle file (all the code), I think you should exclude files based on instrumentation settings.

What do you think?

Anyway, if you'll need some help with source maps or something else ping me here ;-)

[MrOrz]

I am currently using source-map branch and it is looking great on Coveralls.

However, the HTML reporter on local machine seems to display the transpiled file with the line numbers of untranspiled code highlighted.

I have tried other solutions like remap-istanbul but it failed to grab source-map information because I am using babel register hooks, which generates source maps only in memory.

Edit: I tried running istanbul report html manually to generate a new HTML report after the initial run of istanbul. Fortunately, the new HTML report is correct!

It seems that in the first run of istanbul, HTML reports were messed up but both coverage.json and lcov.infowere correct, so that Coveralls and the subsequent instanbul report html could get the correct run statistics while reading the correct, original source file.

[JaKXz]

@MrOrz could you share your coverage command? This is mine and I'm not getting any files instrumented:

    "test": "mocha **/*.spec.ts --opts ./mocha.opts",
    "cover": "npm run test && istanbul report text-summary lcov"

[MrOrz]

@JaKXz here you go :D

https://github.com/MrOrz/hacktabl-parser/blob/master/package.json

For the mysterious "_mocha" please refer to https://coderwall.com/p/x6jfwg/running-istanbul-code-coverage-with-mocha

[alexgorbatchev]

Friends, i'm don't understand what is the current status of source-map remapping feature in Istanbul? Is it possible to get a quick update?

[faergeek]

Seems like it will be dropped in current implementation and rewritten.
See https://github.com/gotwarlost/istanbul/issues/212#issuecomment-152677921

[frapontillo]

I am very interested in this as well, can't wait for a new version of Istanbul.

[ariporad]

+1, This would be really nice to have.

[AndreyUtka]

+1 typescript

[gotwarlost]

Source map support is still being refined but I have projects in typescript and babel running tests under node.js with istanbul coverage reports on original source.

See:

• https://github.com/istanbuljs/sample-babel-node for babel
• https://github.com/typings/typings/pull/47 for typescript

[ariporad]

@gotwarlost: YAY! Quick two questions though:

1. Is the magic part of https://github.com/istanbuljs/sample-babel-node the babel-node bit, or the [email protected] bit?
2. Where can I find more information about [email protected]? Specifically, what are the important/breaking changes?

Thanks!

[iamstarkov]

yeah, where we can find changelog for 1.0 milestone /cc @gotwarlost

[ariporad]

@gotwarlost: Ok, I've done some experimenting (The [email protected] was the important bit):

I've been trying to integrate it with my project (ignore the content): emiw/redstone-dispatcher#istanbul-source-maps-2, your can run the tests with ./node_modules/.bin/gulp test:all:cov. And it's kind of working-ish. After a lot of fiddling, I found something interesting: In src/app.js, which contains an async function, istanbul logs an error, but the output (HTML report) is actually correct, but all the other files have incorrect coverage data:

Transformation error for /Users/Ari/Developer/EMIW/redstone/dispatcher/src/app.js ; return original code
Line 11: Unexpected identifier
Unable to post-instrument: /Users/Ari/Developer/EMIW/redstone/dispatcher/src/app.js

I hope that helps with perfecting source map support. Thanks so much for adding it! (Also, is there anything I appear to be doing wrong?)

[gotwarlost]

@ariporad - still working on the docs for breaking changes. There may be more breaking changes until I remove the alpha tag from the 1.x series.

[gotwarlost]

@ariporad - I'm not able to run your tests. I get the following error:

https://gist.github.com/gotwarlost/3a1e83f7cca84a601c61

[ariporad]

@gotwarlost: weird, that's really strange. What gulp version do you have?

[gotwarlost]

I just checked out your code ran npm install and npm test

$ node --version
v4.2.1

I don't have gulp installed globally.

[ariporad]

@gotwarlost: That's even stranger. I just re-installed, it's working for me. Regardless, you'll want npm run test:cov.

UPDATE: I just tried it in Vagrant, I got the error, try rm -rf node_modules && npm i -g npm && npm i && npm run test:cov (WARNING: will update you to npm3).

[gotwarlost]

Upgrading npm did the trick, thanks.

The following is the command I ran after trying to figure out what you are doing in your gulp file (I never knew JS development needed to be so complex! - anyway, I digress).

• istanbul v1 turns on the --include-all-sources by default and because of this, it is trying to load your JS files under src/ as regular JS files (which is why you see the weird transform error which is benign)
• I also exclude the test files from istanbul coverage processing
• The rest is more or less what you are running via gulp

Believe it or not, this actually works. I was surprised to see no branches in the coverage output given that you have an explicit if statement in the code but that is because of the way it gets translated into ES5 in combination with the source maps generated.

$ ./node_modules/.bin/istanbul cover --no-include-all-sources -x 'build/**/*.test.*.js' ./node_modules/.bin/_mocha --   -r test/setup.js 'build/**/*.test.*.js'


  app
    #start
      ✓ should start a server on `port`
      ✓ should throw if called more than once
    #stop
      ✓ should stop the server

  index
    ✓ should do something

  somethingElse
    ✓ should do something


  5 passing (218ms)


=============================== Coverage summary ===============================
Statements   : 100% ( 12/12 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 4/4 )
Lines        : 100% ( 12/12 )
================================================================================

[ariporad]

@gotwarlost: Thanks so much! You rock! (Sorry about the gulp setup, it's really messy.)

[trodrigues]

Hi there. In the process of moving a project from Babel 5 to Babel 6, I started having issues with isparta, so I decided to come back here and have a look at the current status of this issue. I was happy to see there was some activity so I decided to give it a try.

The project I was trying this on also uses https://github.com/speedskater/babel-plugin-rewire

Previously, the coverage of my project was around 90%, but now with these changes it dropped to around 50%.

I also decided to test this in isolation, so I forked the sample-babel-node project and set it up in a similar way: https://github.com/trodrigues/sample-babel-node

Coverage also changes after mocking out the new dependant module with the rewire plugin.

=============================== Coverage summary ===============================
Statements   : 60% ( 3/5 )
Branches     : 100% ( 0/0 )
Functions    : 0% ( 0/1 )
Lines        : 60% ( 3/5 )
================================================================================

I'm still not 100% sure if this would be a problem in the rewire plugin itself or istanbul, but I thought it could be worth mentioning this problem.

Let me know if I can provide any more information or feedback on this.

For reference, here's the other project where I was originally trying this out on https://github.com/contentful/contentful-space-sync/tree/babel6

[trodrigues]

So after fiddling around a bit with this and not getting different or better results I decided to actually look at the generated HTML coverage and realized that the results isparta was giving me were based on the transpiled code and were very misleading. Some parts of the code that my tests never touch were being reported as covered.

What that means is that the results I am getting now with istanbul, over source maps and the actual source code are the real ones, and everything is actually working fine and there's no problem with babel-plugin-rewire as I initially suspected.

Apologies for any confusion my previous comment might've caused.

[trodrigues]

Oh, and that also means that using 1.0.0-alpha.2 with babel and sourcemaps seems to be working great for me :)

[D34THWINGS]

@trodrigues how did you make it to work with alpha version ? I need to try this out !

[trodrigues]

@D34THWINGS Have a look at this project https://github.com/contentful/contentful-space-sync

It pretty much does the same stuff that @gotwarlost did on his demo project, essentially:

• Make sure your babel config has sourcemaps enabled: https://github.com/contentful/contentful-space-sync/blob/master/.babelrc#L5
• Run the istanbul cli script via babel-node so it reads the same configuration file https://github.com/contentful/contentful-space-sync/blob/master/package.json#L13

That's about it. Initially I posted here because of a babel plugin I was using, and I wasn't sure where the actual problem was, but it turned out to be mostly a matter of setting things up properly.

[shaharmor]

Is there any complete guide on how get this working with typescript/webpack/karma?

[joshma]

Is there planned support for sourceMappingURL? Bottom of my file looks like:

//# sourceMappingURL=__maps__/polyfills.js.map

and I'm getting this error:

Error returning source map for /Users/joshma/my_project/js-build/polyfills.js
"coffee/polyfills.coffee" is not in the SourceMap.
/Users/joshma/my_project/node_modules/istanbul/node_modules/istanbul-api/node_modules/istanbul-lib-source-maps/lib/map-store.js:133
            if (this.verbose) {

(And I'm using (gulp-sourcemaps](https://www.npmjs.com/package/gulp-sourcemaps), if that's relevant)

[donaldpipowitch]

Can someone help me and explain what is the current state of this issue? This issue is still open, but it seems some people had a working example in mid 2015.

To be specific: Can I get code coverage when I use Karma, TypeScript and Webpack in a similar setup like this one?

[tilgovi]

I got working code coverage, but it breaks source maps. The source maps work fine in the coverage reporter, but since the source map pass happens on the coverage output rather than during the code instrumentation, source maps are broken during the test run if you need to interactively debug the tests.

So, all my setups so far run test suites with an optional environment variable to enable coverage. I turn it on in CI and leave it off locally.

[donaldpipowitch]

Thank you. Could you please share your config?

[tilgovi]

Here's an example with browserify and browserify-istanbul:

• https://github.com/tilgovi/dom-seek/blob/master/package.json
• https://github.com/tilgovi/dom-seek/blob/master/karma.conf.js

You'll notice in the karma configuration that I only add istanbul to the pipeline if coverage is requested. This is because, as I said, with this setup the coverage report takes the intermediate source map into consideration so that line and branch coverage is accurate but the instrumentation does not so debugging in the browser is not possible at the same time coverage is enabled.

[donaldpipowitch]

Thank you.

[Izhaki]

For anyone interested in mapped coverage reports involving:

• Typescript
• Jasmine / Mocha
• tsc
• npm scripts

This may be of help:

Output

Jasmine

See https://github.com/Izhaki/Typescript-Jasmine-Istanbul-Boilerplate.

package.json (the relevant stuff):

{
  "scripts": {
    "postinstall": "typings install dt~jasmine --save --global",
    "test": "ts-node node_modules/.bin/jasmine JASMINE_CONFIG_PATH=jasmine.json",
    "test:coverage": "ts-node node_modules/istanbul/lib/cli.js cover -e .ts  -x \"*.d.ts\" -x \"*.spec.ts\" node_modules/.bin/jasmine -- JASMINE_CONFIG_PATH=jasmine.json"
  },
  "devDependencies": {
    "istanbul": "^1.1.0-alpha.1",
    "jasmine": "^2.4.1",
    "ts-node": "^0.9.3",
    "typescript": "^1.8.10",
    "typings": "^1.3.1"
  },
}

Mocha

See https://github.com/Izhaki/Typescript-Mocha-Istanbul-Boilerplate.

package.json (the relevant stuff):

{
  "scripts": {
    "postinstall": "typings install dt~mocha dt~require dt~chai --save --global",
    "test": "mocha --compilers ts:ts-node/register src/**/*.spec.ts",
    "test:coverage": "ts-node node_modules/istanbul/lib/cli.js cover -e .ts  -x \"*.d.ts\" -x \"*.spec.ts\" _mocha -- --compilers ts:ts-node/register -R spec src/**/*.spec.ts"
  },
  "devDependencies": {
    "chai": "^3.5.0",
    "istanbul": "^1.1.0-alpha.1",
    "mocha": "^2.5.3",
    "ts-node": "^0.9.3",
    "typescript": "^1.8.10",
    "typings": "^1.3.1"
  },
}

[nojvek]

What is the current status of source mapping with Istanbul?

For our project remap-istanbul failed horribly so we ended up writing our own script to merge and map the coverage json files.

I'd love to contribute back to istanbul so "istanbul report" can have a nice --sourcemap option.

[faergeek]

@nojvek I use nyc + babel-plugin-istanbul for that, it works great with ava and babel.

[Glavin001]

I second nyc!

I found https://github.com/Izhaki/Typescript-Mocha-Istanbul-Boilerplate/issues/2#issuecomment-249435253 which recommended reading http://rundef.com/typescript-code-coverage-istanbul-nyc

My final package.json which is working well for our project is:

{
  // ...
  "scripts": {
    "test:unit": "cross-env TS_NODE_PROJECT=test/unit mocha --compilers ts:ts-node/register --recursive test/unit",
    "test:coverage": "nyc -e '.ts' --r html -r lcov -r text npm run test:unit"
  },
  // ...
  "devDependencies": {
    "cross-env": "^3.1.4",
    "istanbul": "^1.1.0-alpha.1",
    "mocha": "^3.2.0",
    "nyc": "^10.1.2",
    "ts-node": "^2.0.0" 
  },
  // ...
}

Hope this helps!

[mmc41]

@Izhaki Could not get mocha support working. First I got the error "Unable to resolve file [.ts]", then if I removed the (undocumented?) "-e .ts" option, the typescript compiler began wrongly to process source files under node_module which gave redeclaration errors.

[mmc41]

@gotwarlost Would it be possible to have build-in support in istanbul for source maps (e.g. typescript etc). I tried various workarounds as suggested here but they either don't work with newest versions or they introduce other problems like making mocha not report correct source line on error or they are not portable.

[Izhaki]

Can someone please enlighten me as to the actual relationship between this project (istanbul) and nyc?

If one is to replace the other, it seems to me very reasonable to got the nyc path.

But if the two are developed in parallel (and thus somewhat competing), it is perhaps unwise to merge the nyc PR on the sample project, since it uses a 3rd party solution? It will be rather odd if this repository gets an update that solve the issue discussed without nyc.

What do people think?

[mummybot]

From Istanbul's website the relationship appears to be nyc IS the CLI for Istanbul:

Istanbul instruments your ES5 and ES2015+ JavaScript code with line counters, so that you can track how well your unit-tests exercise your codebase.
The nyc command-line-client for Istanbul works well with most JavaScript testing frameworks: tap, mocha, AVA, etc.

[Izhaki]

I've seen that, but still unclear as to the relationship between https://github.com/gotwarlost/istanbul and https://istanbul.js.org/. There's no mentioning of the latter in the former.

[BrainCrumbz]

TBH, @Izhaki is right, I mean, on https://istanbul.js.org/ web page the GitHub link does not point here, but to https://github.com/istanbuljs

[mummybot]

I agree the relationship between the projects is unclear. That would be a question for @davglass, given he has many contributions to both projects, or to @gotwarlost who wrote the original Istanbul.

Jest (Facebook's test framework) which bakes in coverage pulls in istanbul-api from https://github.com/istanbuljs. I don't think you can go wrong using any of the Istanbul.js.org repos. Looking at the number of commits to the base Istanbul recently I assume that the project is relatively mature.

[mmc41]

I also wondered about the project relationship. I have moved on to nyc - works well with typescript, sourcemaps etc.

[morsh]

FYI - This happened to me on a windows machine.
I manages to fix it by cleaning up the Temp folder: C:\Users\USERNAME\AppData\Local\Temp\jest. (delete jest temporary folder)