Jest: Use V8 builtin coverage reporting

[thymikee]

🚀 Feature Proposal

V8 provides a native way of getting code coverage. Recently a c8 project by @bcoe was released to use this feature in Node land. Would be cool to experiment with this edgy tech.

Here's a blog post about it.

Motivation

Make coverage 2-3x faster and babel-free.

[SimenB]

@aaronabramov you had tried this, or just looked at it?

[bcoe]

:wave: I'd love to help with this as best I can, there are a few outstanding issues that make the experience a little bit clunky, but shouldn't block a proof of concept:

https://github.com/nodejs/node/issues/22919

A strategy for handling source-maps also needs to be developed. I've been chatting with @jdalton about some of these issues, as they relate to ESM -- perhaps we can aim for a community effort.

[SimenB]

We talked a tiny bit internally last week, and it was the sourcemaps that we figured would be hard part. How much of that is handled in devtools? Or, do you have a write-up about it somewhere?

[bcoe]

@SimenB in nyc, we handle source-maps with a fairly tiny abstraction on top of Mozilla's source-map library.

Some of this might be able to be torn out of nyc ... it might be better to start over with fresh eyes, especially since source-map has switched to an async API (nyc is pinned on the old sync API).

[SimenB]

For coverage I suppose the async api works, but jest is also stuck on the sync api because of test errors which must be collected synchronously. Although I have hope for https://github.com/mozilla/source-map/issues/331

[kwonoj]

for https://github.com/mozilla/source-map/issues/331, I have created POC as discussion https://github.com/mozilla/source-map/pull/369 - while I have low expectation if it could be accepted due to some hacky workaround to achieve sync interfaces. /cc @SimenB for fyi.

[azz]

How can we get the ball rolling on this? Using V8 coverage and using native ES modules would remove the need for Jest to use babel out-of-the box, right?

[SimenB]

You'd still need it to get mock hoisting, but I suppose that's something people can live without.

This was actually unblocked by https://github.com/bcoe/c8/pull/85, so if anyone wants to work on this, that'd be awesome! 🙂

[bcoe]

@SimenB 👋 one issue we still have is source-maps that are injected dynamically, because c8 doesn't hook module.require; I'd love to get the ball rolling on implementing source-maps in V8/Node.js itself.

[SimenB]

Jest controls the source code and manually executes it in a sandbox, so we should be able to inject it still, right?

Built-in sourcemap support in the engine(s) would be absolutely amazing!

[bcoe]

@SimenB I think with some fiddling we should be able to figure out a way to provide the source-map, might take some work in v8-to-istanbul; the main limitation faced by c8, is that it doesn't intercept source-code requires itself (which is a deprecated API any ways).

[SimenB]

I'm in your slack, happy to try to figure out a way forwards in the not too distant future (swamped at work for the last few months and for a couple more).

I think this might be quite revolutionary if we're able to make it work

[SimenB]

it doesn't intercept source-code requires itself (which is a deprecated API any ways

Could you expand a bit on this? Intercept how?

In Jest's case, we inject our own require function which we can make do whatever we want

[SimenB]

For anyone following along:

I've experimented a bit with this, but the current approach is very FS based - you have to set an env variable before running node (we could wrap it a spawning node process, so not a biggie), but the report is only output to when the process exits. That makes it hard to properly deal with sourcemaps (since we manage them in memory, although they do exist on disk). Which means watch mode and coverage is a no-go (and we also have to do more manual process management).

Open issue request for a programmatic approach: https://github.com/nodejs/node/issues/28283