Stryker: Interest in support for Intern

[janmulders]

Hi,

I am interested in using Stryker with Intern (https://theintern.io/), the JavaScript testing framework and runner created by Sitepen.
I have neither time nor expertise to do this myself - just recently learned about the existence of Stryker.
But if more people are interested... express your interest here!

Jan Mulders

[nicojs]

@janmulders thanks for reporting this feature request. I've started a conversation on their gitter chat. When i know more i'll add it to this issue.

[simondel]

Related issue: https://github.com/theintern/intern/issues/823

[nicojs]

We've just released Stryker 0.34 with the command test runner. We now support any test command (although without many performance features). Could this help you @janmulders ?

[simondel]

As mentioned, we have the command test runner. We're open to supporting intern properly in the future if someone wants to help (and if the demand is there).

[nicojs]

@mthmulders, feel free to create a PR for this :)

Plugin API is documented here: https://github.com/stryker-mutator/stryker-handbook/blob/master/stryker/api/plugins.md#plugins

[mthmulders]

Thanks for the invitation. Work is currently stalled (other priorities) but I had basic integration (i.e., running a passing test) working already. See mthmulders/stryker-intern-runner for those interested.

[simondel]

Reopened this issue due to interest from the community.

If I were to add support, I would start by getting intern up and running programmatically using my regular tests. Their docs could help with this:
https://theintern.io/docs.html#Intern/4/docs/docs%2Fhow_to.md/use-intern-programmatically

While building this support, take a close look at how you'd configure intern and how others may use intern. A good thing to override would be Interns leaveRemoteOpen property to ensure that Intern actually quits. Another nice thing is bail to ensure that you stop as quickly as possible. Overriding reporters can also be a good idea to prevent creating thousands of code coverage reports if you've got that set up :P

After you've got Intern running programmatically, you could start integrating it with Stryker by building a TestRunner plugin: https://github.com/stryker-mutator/stryker-handbook/blob/master/stryker/api/plugins.md#plugins

[mthmulders]

The default for leaveRemoteOpen is actually false, it would mean that if your tests run in a browser, that browser will quit after the test has run. I guess that is what you want - so I don't think we need to override that.
The bail option is indeed a good one, I've added that to my implementation.

A note on the status of my work. It currently does the following:

1. Detects when Intern has ran a test
2. Detects when Intern has ran a test that failed
3. Detects when Intern has ran a test that timed out
4. Detects when Intern has skipped a test

But I guess it's far from complete yet.
As @simondel suggested, adding coverage support would be a great way to make it run _faster_, and there's probably a few other things that need to be ironed out.
With that said, I'm more than happy with any contributions ;-) and I'd be happy to help where I can.

[nicojs]

The default for leaveRemoteOpen is actually false, it would mean that if your tests run in a browser, that browser will quit after the test has run. I guess that is what you want - so I don't think we need to override that.

Doesn't that mean that it closes the browser after a test run for a single mutant, needing to spawn a new browser for the next run? This is quite slow. Maybe we can make this configurable?

As @simondel suggested, adding coverage support would be a great way to make it run faster, and there's probably a few other things that need to be ironed out.

The current code coverage support will change after #1514 is implemented. However, both versions will be similar to the TestRunner's perspective. In both scenario's coverage is collected in a global object that needs to be reported back to Stryker's main process.