Spock: Question: JUnit 5/JUnit Platform support roadmap

[mkobit]

JUnit5-M1 was released on July 7th. I'm wondering, is there a plan or documentation of what needs to change in order to support JUnit 5?

[enriquezrene]

What do you mean with support JUnit 5, do you wanna mix spock and junit @mkobit?

[mkobit]

@enriquezrene I really enjoy the method of writing Specification, but sometimes extending or making modifications to the way Spock works is difficult. The extensions page of the current RC is still missing info around Writing Custom Extensions, which makes it somewhat difficult to understand where to do additional extensions. There are classes like IGlobalExtension and IAnnotationDrivenExtension can be used, but the way of bringing them in to me is awkward and unwieldy. You can also use @Rule, but IMO the new extension model is conceptually clearer to understand and implement.

It would also be interesting to see if any new Groovy features (like macros that I believe are coming in 2.5, current feature requests here, or other things that I am unaware of) could be used. Or also, if Groovy 3.0 and JUnit 5 could provide additional value.

I'd like to know if there have been any thoughts around what it would take to get there, and if the core developers of Spock have thoughts on this as well.

[enriquezrene]

I don't think that new features from groovy can bring new features for Spock, I mean, the contribution guide explains that Java is the preferred language in order to build Spock, and groovy could be used just when you have really good reasons (https://github.com/spockframework/spock/blob/master/CONTRIBUTING.md). On the other hand, I think that if we can provide tests sample written with Junit5, and show them to Spock developers can be helpful because they may show us the spock-style in order to achieve the same behaviour or it could become a future new feature

[sbrannen]

Spock and JUnit are already _mixed_ via the Sputnik JUnit 4 Runner.

JUnit 5 introduces a completely new model for running testing frameworks (like Spock) on the _JUnit Platform_ which departs from the Runner API in JUnit 4. Specifically, the JUnit Platform introduces a TestEngine API.

The JUnit team provides two such engines out of the box: the JupiterTestEngine for the JUnit 5 programming model and the VintageTestEngine for the JUnit 3 and JUnit 4 programming models. Furthermore, third parties are already implementing custom engines to support testing frameworks like Specsy, Spek (for Kotlin), etc.

For seamless integration with future versions of the JUnit Platform, better reporting, and more powerful IDE and build tool integration, the Spock team should consider implementing a custom SpockTestEngine that will eventually replace the need for the existing Sputnik runner.

If you have any questions about the features of the JUnit Platform, feel free to raise an issue in the JUnit 5 issue tracker.

Cheers,

Sam

[sgitt]

When testing code which uses frameworks like CDI or Apache Camel it is necessary to use specialized runners like @RunWith(CdiRunner.class) or @RunWith(CamelCdiRunner). And it seems to be impossible to combine them with the Spock Sputnik runner since JUnit 4 does not allow to have multiple runners per test.

Spock itself brings support for testing Spring code (spock-spring) but if we want to use Spock to test CDI code, it is necessary to re-implement functionality of libraries like cdi-unit (which provides CdiRunner) ourselves e.g. as a JUnit rule or Spock extension which is very time consuming. Alternatively for such tests we need to switch to JUnit.

JUnit 5 allows to have multiple Extension per test (see e.g. http://blog.codefx.org/design/architecture/junit-5-extension-model). We hope that if Spock supports JUnit 5 extensions it would allow us to combine it with large number of specialized test libraries and avoid the need to switch to JUnit when testing CDI, Camel or other popular frameworks.

[leonard84]

Spock has always tried to be as backwards compatible as possible, we still support Java 6. So we would still need to support JUnit 4 for the foreseeable future (way longer than Java 6). So a big question would be how we could achieve that, without having to support two separate code bases. Another issue is that much of Spock is built around the Specification base class which is then extensively modified with AST transformations, the question would be how this would translate to the JUnit 5 model and how those extensions interact with each other.

[sbrannen]

@leonard84, the best way to answer that question is to start playing around with JUnit Jupiter (i.e., JUnit 5) and see how far you get with writing extensions for Spock.

I'm not familiar with the internals of Spock (e.g., the AST transformations you mentioned and the possible repercussions of such an approach), but I'd be happy to answer any questions you may have about JUnit 5 in general. And of course... if you run into any _show stoppers_ with regard to supporting Spock in JUnit Jupiter, I (and the rest of the JUnit Team) would be grateful if you could report your findings in the JUnit 5 GitHub issue tracker.

Cheers,

Sam

[pniederw]

@leonard84 @sbrannen JUnit 4's Runner SPI has always been a big limitation for Spock. For example, it made implementing reporting and multi-threaded test execution difficult/impractical. This is a rare opportunity to break out of this box and make sure that the new (i.e., JUnit 5) box has everything Spock needs to innovate going forward. Would love to see this happen.

[leonard84]

So that would mean Spock 2.0 with JUnit 5 as new base, if we do that we could also drop Java 6 support and other outdated framework versions.

[thetric]

Any updates on the 2.0 roadmap? The 2.0 milestone only includes this issue

[tburny]

I would love to be able to run specific rows of an @Unroll-ed data driven test in IntelliJ. As far as I know this is possible with JUnit 5 :)

What needs to be done to move to JUnit 5?

[Petikoch]

I would love to have a Spock "2.0" based on a JUnit 5 "SpockTestEngine", instead based on the Sputnik JUnit 4 Runner.

We could get rid of the JUnit 4 dependency in our project, which contains a mix of Spock and JUnit tests...

Any updates regarding the roadmap?

[leonard84]

We will start once 1.2 is released. You can already switch to JUnit 5 and use the vintage engine to run Spock for now.

[tburny]

Just some brainstorming for the new version:

• [ ] Will Spock 2.0 be an extension to the Junit Jupiter API or shall we have a custom engine instead? What are the pro's and con's of each variant?
• [x] Maybe use https://github.com/ota4j-team/opentest4j because it is similar to what spock already does? It should aid better IDE Integration
• [ ] Which functionality of JUnit can be reused or even extended and what others cannot?

[marcphilipp]

I think Spock 2.0 should provide a custom engine and use exceptions from opentest4j. The engine should be based on HierarchicalTestEngine so it can support parallel test execution.

I think it would make sense to support (a subset of) the Jupiter extension API.

JUnit Jupiter no longer supports rules, so we need to decide whether Spock 2.0 should still support them or not. Personally I think it shouldn't.

[leonard84]

I would move support for jupiter-extensions and junit 4-rules into seperate modules spock-jupiter and spock-junit4.

[tburny]

@marcphilipp Using the HierarchicalTestEngine (HTE) would also be my suggestion. I did a toy project with a HTE some months ago which "executed" a Cucumber specification so that each Scenario, Feature and Given/When/Then was a part of the hierarchy. Took me half a day, worked like a charm! (Although the documentation has been a bit thin in the details.)

I would go for dumping JUnit rules in favor of the Spock or Junit Extension API completely. In most cases they can be replaced by Spock extensions (Global or annotation based), I never had to use any in my Spock tests. When in doubt, there is still the JUnit Extension mechanism.

Apropos extensions: Currently the functionality of Spock and JUnit Extensions has a quite huge intersection, am I right?

[leonard84]

HierarchicalTestEngine (HTE) is a given, I spent quite some time with @marcphilipp in order to bring parallel execution capabities to that engine. Furthermore, it will allow for a better reporting of @Unroll'd tests. Maybe even DataDriven Specs if we can figure out a good way to integrate it in the existing functionality.

Apropos extensions: Currently the functionality of Spock and JUnit Extensions has a quite huge intersection, am I right?

Yes Jupiter Extensions are rather similar to Spock. However, I would still try to add some support via a spock-jupiter module, although we won't be able to support everything I recon (param injection will be troublesome).

I would go for dumping JUnit rules in favor of the Spock or Junit Extension API completely.

We already have support for the old rules, moving the code to spock-junit4 doesn't cost us much and helps others to migrate more smoothly.

[lsikora]

Are there any plans to fix this issue? There is already an issue for @Unroll run on JUnit 5 Vintage engine #976. Test output differs significantly when running atop JUnit 4 vs JUnit5.

[pkurczyna]

@lsikora I believe the issue #976 was not on JUnit5 but also on JUnit4.

[mkopylec]

Can I use junit 5 extensions with spock 2.0-M3?
If not, are there any plans to support it?

[Vampire]

JUnit 5 extensions are for the Jupiter engine. Spock 2.0 provides an own Junit 5 engine. There is no Jupiter module yet that adds support for Jupiter extensions and I'm not aware of someone working on it yet or how good this would work. If you would contribute it, there is a fair chance of having it accepted. But maybe it would be better to provide native Spock extensions for the wanted functionality as they can integrate better with Spock.

[leonard84]

We looked at adding a jupiter extension support module, but quickly dismissed it and decided to focus on finishing spock 2.0, since we would have to emulate a large chunk of Jupiters internals to make it work.
However, as @Vampire said, we would probably accept contributions depending on the quality of the code.

[mkopylec]

Will Spock's engine introduce a similar concept to Jupiter extensions? I know there are extensions in Spock 2.0 but they are limited to annotation based extension. I would like to provide an extension in a form of class field, like @RegisterExtension in Jupiter, and use its methods in test cases.

[Vampire]

They are not. There are global extensions that just do their work and local annotation based extensions that only do work if you annotate a class, field or method. I doubt there is anything a Jupiter extension can do, a Spock extension cannot. Spock extensions are pretty powerful.

[mkopylec]

Ok, I was not aware that I can annotate a field with an extension annotation. Could you post some examples on how it works?
My case is that I'm using WireMock for stubbing an external REST service. Now, I would like to start/stop the wiremock server on setupSpec/cleanupSpec and additionally reset the server on cleanup. What's more, I'd like to use the instance of the wiremock server in my test methods to stub requests. With JUnit Jupiter I can achieve that this way:

class WireMockExtension implements BeforeAllCallback, AfterAllCallback, AfterEachCallback{
    private WireMockServer server;

    WireMockExtension(int port) {
        server = new WireMockServer(port)
    }

    void beforeAll() {
        server.start()
    }

    void afterAll() {
        server.stop()
    }

    void afterEach() {
        server.resetAll()
    }
}

class MySpec {

    @RegisterExtension
    private WireMockExtension wireMock = new WireMock(8080)

    void myTestMethod() {
        wireMock.stubRequest()
    }
}

How can I achieve it in Spock 2.0?

[Vampire]

Did you have a look at the documentation? It is quite extensive imho. On the other hand I'm biased, as I wrote it. http://spockframework.org/spock/docs/2.0-M3/all_in_one.html#_writing_custom_extensions

[jlink]

@leonard84 For an article (Oracle's Java Magazine) about writing 3rd party test engines on the JUnit platform - together with Matthias Merdes from JUnit 5 core team. It seems you are the main contributor of the spock engine. Are you interested in answering a few questions about your experience with the Test Engine SPI/API?