Privacybadger: track coverage statistics

[cowlicks]

It would be nice to know what is being covered by our Qunit tests, and what isn't. For posterity here is what I looked into:

blanket.js: it seemed to have easy support for Qunit. But it has several problems. It is unmaintained. It seems to be broken and unable to trace some of our source code. And we'd have to make our "Content security policy" include "unsafe-eval" and a bunch of hashes of our source files.

istanbul, is recommended by the former authors of blanket. However it only seems to work with qunit when run with PhantomJs, and I don't think that PhantomJs supports the chrome extension api's. ex here.

I also tried "sandboxing" the page. But the sandbox blocks access to extensio api's we use in our tests. So we get: "148 assertions of 182 passed, 34 failed." which isn't so bad. It might be worth it to extract the passings tests, from the failing tests. The failing tests could be refactored with the appropriate api's mocked out. Or the could just be run seperately as integration tests.

The tests are currently locked to a pretty old version of Qunit anyway, so maybe it would be worth it to refactor them.

So what are other people doing? Some projects (like Signal-Desktop) mock out the extension api's or avoid them. Looking at other projects, it looks like people just don't worry about coverage. Have y'all seen any good solutions to this?

[ghostwords]

Sounds right to me. Code coverage is already a bit niche, and here we delve deeper into niches, looking to do code coverage for browser extensions. Mocking extension APIs is probably the answer; I went down that path with Ghostery and it worked fine then. Unit tests don't actually need to exercise Chrome APIs, that's what functional/integration tests are for.

I think coverage is a potential nice-to-have, good for providing a better idea of what's not ... covered by unit tests. But I think we're fine without it too. Can we get coverage from our functional (Selenium) tests?

QUnit should indeed get upgraded sometime.

[cowlicks]

@ghostwords If there is a way to do it with selenium it would would likely depend on a javascript tracer, like the one the unit tests would use. So figuring out how to get coverage on those tests should probably come first.

I'll look into revamping the tests. Here are a few items I think should be considered when doing so. Let me know if they sound good:

• Either Upgrade Qunit, or switch to mocha, jasmine, something else?
• Mockout the extension api's when refactoring the tests.
• Try to get coverage statistics.

[ghostwords]

Current thoughts:

• Upgrade QUnit: Sounds great. The older, obsoleted API we use isn't even documented at this point.
• Switch to a different test runner: What would we gain? I used Mocha before, it was fine. Don't have any preference here beyond avoiding unnecessary changes.
• We would need to mock out the WebExtensions API if we could no longer run unit tests in the actual extension context for whatever reason, like in order to get code coverage (which maybe doesn't yet work in extension context). I wouldn't mock out the API just do it though.
• Coverage stats: Don't care about numeric stats, but would be happy to get a code coverage report to aid filling in important gaps. If we need to make a lot of changes to get it ... I don't know, having functional tests work in Firefox is way more important. But if you'd like to make this happen too, great!

[cowlicks]

Upgrading sounds like a good actionable item, but separate, so I'll make an issue.

I was thinking that changing to a different testing framework might make getting coverage possible. But after thinking about it, these coverage libraries are probably implemented using eval or new Function(). Which makes running them in the Extension context impossible because of the content security policy.

Firefox tests are on the way in #1168

[ghostwords]

QUnit got upgraded in https://github.com/EFForg/privacybadger/commit/375033a5158eeb1cf320824e63a0477933b08568?w=1#diff-5066d7764d4b8812be12fad9f37f36a9 (pull request in progress, not yet merged).

[ghostwords]

Also, I'm interested in getting mutation testing going at some point. There is at least one JavaScript option at this point, Stryker. Looks like Stryker doesn't yet work with QUnit, but we could probably adapt it (by creating a custom TestRunner using Stryker API?).

[simondel]

If you need advice on adding support for QUnit, feel free to contact us on gitter or somewhere else 👍

[ghostwords]

We just got (manual, sort of inaccurate since it's not limited to measuring unit test-triggered code only) code coverage with Chrome 59 for free! Would be cool to be able to start/stop recording and then get a report programmatically. Could then use in automation with both unit and functional tests.