Cucumber-jvm: Add JUnit 5 Support

[reinhapa]

Summary

Provide a JUnit 5 extension for cucumber.

Expected Behavior

When writing a test with JUnit 5 in our environment, we need to replicate a base environment in a way that we use different Extensions in order to to run tests. So we like to have a Scenario something like this:

~~~
@ExtendWith(CucumberExtension.class)
@ExtendWith(MockitoExtension.class)
@ExtendWith(BaseBusinessEnvironmentExtension.class)
class MyBusinessTest {
@BeforeEach
void init(@Mock Person person) {
when(person.getName()).thenReturn("Dilbert");
}

@Test
void simpleTestWithInjectedMock(@Mock Person person) {
    assertEquals("Dilbert", person.getName());
}

}
~~~

Current Behavior

At the moment under JUnit 4 we needed to tweak the Cucumber runner in order to call default JUnit 4 annotations.

Possible Solution

We like to help create an CucumberExtension that handles the Cucumber specific annotations for JUnit 5 as an JUnit 5 extension, so that it can be combined with other extensions as for example mocking frameworks.

[mpkorstanje]

Awesome! I don't have any experience with JUnit5 so I can't help too much. Perhaps you can use #1103 as a starting point, see if it builds, meets your needs and review the code.

[reinhapa]

I will take a look into it

[reinhapa]

@mpkorstanje I looked into it briefly and it would be great to open a feature branch for this extension. I will try to contact @tenwit to see if I can help furter extend/test it...

[mpkorstanje]

Excellent! If you need help with anything let me know. I'll do what I can.

[reinhapa]

@mpkorstanje I just trying to merge the the current master and the existing pull request into my local repo right know. I will come back to you as soon I got it working.

[tenwit]

I'm here :) How can I help? My solution works using extensions; I see there's an alternative solution using the engine at https://github.com/signed/junit-cucumber-engine. I hadn't considered using an engine when I did my work. I think it's more idiomatic to use an engine.

[reinhapa]

@tenwit I give just started to merge your master code with actual 2.0.0-SNAPSHOT version on my repo https://github.com/reinhapa/cucumber-jvm/tree/jupiter as it seems, there was introduced a new API layer, which I was starting to switch over to by taking a look into the existing JUnit code. I'm not actually completed that work so far, but you may be take a look at the current state..

[mica16]

Is it currently recommended to use Cucumber with JUnit 4?

[aslakhellesoy]

@mica16, JUnit 4 is required until JUnit 5 support is merged and released.

[mica16]

@aslakhellesoy Ok, thanks :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[sherl0cks]

any word on this? I hope JUnit 5 async means that frameworks like vert.x will become cucumber compatible. https://developers.redhat.com/blog/2018/01/23/vertx-junit5-async-testing/

[aslakhellesoy]

Somebody needs to write the code @sherl0cks.

We've recently discussed a code generation approach, which would make it easier to support JUnit5 (and other test runners). See https://cucumberbdd.slack.com/archives/C6RLMP3C4/p1520605709000469

[sherl0cks]

looks like slack requires an invite, but I get the general idea. unfortunately I have no cycles for something like thing. I'll watch the issue

[mpkorstanje]

There was some experimentation in https://github.com/cucumber/cucumber-jvm/pull/1258 to setup cucumber as dynamic tests. It created quite a bit of boiler plate and seemed like the wrong solution and didn't allow for extensions.

After that I've done some research and I came to the following conclusion:

JUnit 5 consists of the JUnit Platform + JUnit Jupiter + JUnit Vintage.

JUnit Jupiter

Cucumber itself doesn't fit into JUnit Jupiters extension model. To be an extension it would have to yield control of test discovery, test instantiation and test execution to Jupiter. This is not feasible as cucumber follows an entirely different process for each.

While the TestInstancePostProcessor looks promising, it seems as if it could just as easily be applied to a glue class as it could be applied to Jupiter test instance, it is strongly tied to the assumption that a single class provides the context for a single test. In cucumber the test context consists of all glue classes.

This in my opinion makes it unlikely that we'll ever use JUnit Jupiter for Cucumber.

Even if we do through for example code generation, then we'll still be unable to use any extensions as Jupiter is only able to process the generated code of the test instance, not the glue classes that would benefit from the post processing.

Junit Platform

One of the problems of JUnit4 was that @RunWith required runners to generate test instances before they could enrich them. This made it impossible to combine different runners. They were all generating instances and unable to enrich those generated by others. JUnit Jupiter was designed to make these extensions possible. What remains is handled JUnit Platform.

As a generic platform for test execution it provides a common interface for test discovery, selection and filtering. This means that by implementing a Cucumber Engine for the Junit Platform any user who places it on class path can use the Junit Platform to discover and execute feature files.

Junit Vintage

Our current JUnit Runner should continue to work with JUnit Vintage. I reckon this will be the way to execute Cucumber for a long time to come.

[aslakhellesoy]

@sherl0cks sorry - slack joining instructions are here

[luolong]

As a generic platform for test execution it provides a common interface for test discovery, selection and filtering. This means that by implementing a Cucumber Engine for the Junit Platform any user who places it on class path can use the Junit Platform to discover and execute feature files.

This sounds like a proper approach to me. What are the issues with this one?

[mpkorstanje]

Until v4 test discovery, selection filtering and execution were hopelessly intertwined. This has recently been improved but I am not certain that what we have now is decoupled enough.

The other issue being that cucumber is open source and developed by volunteers. So far there has been no one who contributed a junit 5 module.

Feel free to help out!

[luolong]

Well, sure. I would love to help out at some point.

I suppose, first step would be to decouple test execution from any kind of test framework and then use framework to guide the test execution in some way.

But I suppose that this is not going to happen over a weekend?

[mpkorstanje]

Let's start with philosophy.

Cucumber was originally build with the ports-and-adaptors pattern/hexagonal architecture in mind.

Initially JUnit, TestNG, Android and the Command line would all use the runtime to execute cucumber tests.

Over time the internals of the runtime were opened up to expose more implementation details to allow better integration.

This got to the point where the runtime became superfluous for all but the command line.

This breakdown of the architecture happened because the Runtime, JUnit, ect all relied on the same concepts. Concepts the runtime tried to abstract away.

So for internal integrations and components we should use the The Unix Tools Philosophy. What these tools are should be needs driven.

I suppose, first step would be to decouple test execution from any kind of test framework

Currently the basic tools are there already. So no major refactoring should be needed. I suspect they will however need to be refined and sharpened for JUnit5.

If you're willing to spend some time, have a look at the existing Cucumber Runtime and the JUnit5 docs/implementation. There is also a rather old JUnit5 branch in git with a minimal implementation.

You'll need more then a weekend though. :D

[mohanaraosv]

Is there any update on this, If required would like to contribute on this if the roadmap is clear.

[mpkorstanje]

If you are willing you are most welcome to pick this up. There have been no offers to implement this so far.

I don't have much to offer in the way of a roadmap. There is much I don't know about JUnit 5. However the end result would be an implementation of Cucumber in the Junit Platform. In essence an implementation of TestEngine that will run feature files.

One of the challenges you will encounter is that the TestEngine can not configure the glue path and has only only limited methods to select which features are ran. So for now I think simply running all features and using all glue on the class path will be a good start. I reckon these problems can perhaps eventually be solved by the solutions developed in #1476 and a better understanding of JUnit 5.

The best way to understand how cucumber selects and runs feature files is to read the code in. Runtime.run. Note that this combines selection and execution, these concepts are separate in the TestEngine.

[aslakhellesoy]

Supporting various test runners is a lot of work. I've opened a discussion about using code generation instead, like SpecFlow does (see #1487).

[mpkorstanje]

The main advantage of JUnit 5's test engine is that it provides a clear way for IDE's and other tooling to discover, execute and report on cucumber scenarios. Code generation is not an alternative for this as it removes the connection between the feature files and the actual tests. So I do think we should implement Cucumber as a JUnit TestEngine.

[mpkorstanje]

In #1509 I have created a working prototype of JUnit 5 support. There are however allot of features still missing. Most issues are relatively minor and listed in the the ticket. There are however also two major issues as described below.

Fortunately none of these problems are unresolvable but will take time. If you have spare time to contribute feel to either pick the missing details on #1509 or help out with the major issues listed below.

IDEA

For a while I was pleased to observe that when running all tests in a package IDEA would detect the presence of the CucumberTestEngine and use it to execute feature files. It seems this functionality was removed in the latest release.

Getting this to work will need some sorting out .

Surefire

I am disappointed to learn that the JUnitPlatform provider in Maven Surefire does not use fully utilize the test discovery functionality of TestEngine. It is limited to class files discovered by Surefire. This makes it hard to use the the JUnitPlatform for Cucumber tests.

The problem is two fold:

1. AbstractSurefireMojo.execute does load any platform providers when no tests classes could be found. This makes it impossible to discover .feature files as tests - these can only be discovered by .

2. JUnitPlatformProvider.scanClasspath only scans classes provided by surefire. This should change to actually request a scan of the class path and using the filters provided in JUnitPlatformProvider.parameters to filter tests.

[mpkorstanje]

And IDEA works again. Just had to implement a few more selectors.

Image below shows the tests as run by JUnit Jupiter and Cucumber, both on the JUnit platform.

[mpkorstanje]

Gradle

Gradle has the same problem as Surefire. https://github.com/gradle/gradle/issues/4773

[mpkorstanje]

Fixing both Surefire and Gradle Test require an almost complete rework of both tools. For the time being the maven exec plugin (or the gradle equivalent) can be used to launch the junit platform console launcher.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>cucumber</groupId>
    <artifactId>cucumber-java-skeleton</artifactId>
    <version>0.0.1</version>
    <packaging>jar</packaging>
    <name>Cucumber-Java Skeleton</name>

    <properties>
        <java.version>1.8</java.version>
        <cucumber.version>4.2.1-SNAPSHOT</cucumber.version>
        <maven.compiler.version>3.3</maven.compiler.version>
        <junit.jupiter.version>5.3.1</junit.jupiter.version>
        <junit.platform.version>1.3.2</junit.platform.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-java</artifactId>
            <version>${cucumber.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-console</artifactId>
            <version>${junit.platform.version}</version>
        </dependency>
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit-platform-engine</artifactId>
            <version>${cucumber.version}</version>
        </dependency>

    </dependencies>

    <build>
        <plugins>

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.6.0</version>
                <executions>
                    <execution>
                        <phase>test</phase>
                        <goals>
                            <goal>java</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <mainClass>org.junit.platform.console.ConsoleLauncher</mainClass>
                    <classpathScope>test</classpathScope>
                    <arguments>
                        <argument>--include-engine</argument>
                        <argument>cucumber</argument>
                        <argument>--scan-classpath</argument>
                        <argument>${project.build.testOutputDirectory}</argument>
                    </arguments>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.version}</version>
                <configuration>
                    <encoding>UTF-8</encoding>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>

        </plugins>
    </build>
</project>

[mpkorstanje]

We're not done yet. Tried to selectively merge some changes to master but that misfired.

[lavcraft]

Hi @mpkorstanje
Any news about junit5 support, can I help you with this integration?

[mpkorstanje]

I've been porting small improvements and concepts learned from implementing this over to master but otherwise no notable changes. It kinda works. But I'm not happy about the implementation of quite a few things.

1. Selecting scenarios by unique id is currently broken because of changes introduced by #1544
2. Resource loading/scanning is completely duplicated from master. We should move these changes into master first as per #1526.
3. Logging is duplicated after #1577 was done but probably better to leave as is until core moves over to java 8.

You'd be most welcome to look at fixing any of these.

[chris-mcc]

Hello, I was wondering what supporting Junit5 means to parallelisation? In particular does it mean parallelisation by methods will be possible?

[mpkorstanje]

For JUnit 5 we'll be using the JUnit platform which supports parallel execution. The way it's implemented allows scenarios to be executed in parallel. This will be an improvement over JUnit 4 which only allows features to be executed in parallel.

[tauras007]

We were able to use the cucumber cli runners io.cucumber.core.cli.Main.run(..) inside a junit 5 test to execute specific features and used the exitStatus returned to determine if the feature failed or passed, i am sure this is not the ideal approach but something better than having both junit 5 and 4 in the project classpath.

Main.run(new String[]{"--glue ", "x.x.x.x.x", featurePath}, contextClassLoader)

[moshneagaOleg]

hello how can i run tests in parallel use jvm 5.4.0 ?

[mpkorstanje]

https://github.com/cucumber/cucumber-jvm/tree/master/junit-platform-engine#parallel-execution

[davidghiurco]

Has this been released with Cucumber 5.6.0? I'm not able to find any official Cucumber documentation that doesn't say I still need to include io.cucumber:cucumber-junit in my classpath, and that's JUnit 4.

What I'm trying to ask is, which, if any, cucumber artifacts should I include in my classpath to take advantage of the originally described desired functionality at the very top of this issue? e.g.

@ExtendWith(CucumberExtension.class)
class MyCucumberWithJUnit5Test

[mpkorstanje]

This seems to be a common point of confusion. It is worth reading the introduction of the documentation for JUnit 5.

But in brief Cucumber implements the TestEngine interface provided by the JUnit Platform. On the other hand @ExtendWith is a JUnit Jupiter annotation. These are two different parts of JUnit 5.

If you want to use JUnit 5 with Cucumber you can find all information you need here:

https://github.com/cucumber/cucumber-jvm/tree/master/junit-platform-engine

You will not find JUnit 5 advertised in other documentation yet. The tooling support for Junit Platform hasn't quite reached the point yet where I'd be comfortable changing that. And given that Cucumber JUnit works perfectly fine in combination with JUnit Vintage I see no reason too rush the upgrade either.

Internally Surefire and Gradle assume the old class based execution model. This means that Cucumber works less then stellar with these integrations. So I'm currently waiting for a few issues to be resolved here:

• https://issues.apache.org/jira/browse/SUREFIRE-1724
• https://github.com/gradle/gradle/issues/4773

With respect to IDEA, there is no good way to use the JUnit Platform to run individual scenarios and so I'm waiting for:

• https://youtrack.jetbrains.com/issue/IDEA-227508
• https://github.com/junit-team/junit5/issues/2146

You may want to upvote/star/+1 these issues to show that you have an interested in this feature. Or if you have the ability and capacity; contribute to a solution to these issues.

[helder-ciandt]

@mpkorstanje, do you have any example of junit-platform-engine + spring boot application?

[mpkorstanje]

• https://github.com/cucumber/cucumber-jvm/tree/main/examples/spring-txn
• https://github.com/mpkorstanje/cucumber-spring-boot-parallel

Neither is very exhaustive or explanatory but you can find all information you need here:

• https://github.com/cucumber/cucumber-jvm/tree/master/junit-platform-engine
• https://github.com/cucumber/cucumber-jvm/tree/main/spring
• https://github.com/cucumber/cucumber-jvm/tree/main/java

[viktozhu]

I can see that current cucumber version is 6.5.0 and this is 6.1.2-Snapshot. Is this part expected to be supported and released with later versions?

And second question is usage, currently proposed usage is @Cucumber while for Junit5 I would expect @ExtendWith(CucumberExtension.class) - in this case cucumber will not have to handle other possible extensions. What is planned direction?

[mpkorstanje]

@viktozhu it's in v6.5 and you seem to misunderstand what JUnit 5 is. Consider reading the discussion, I'd rather not repeat myself. See https://github.com/cucumber/cucumber-jvm/issues/1149#issuecomment-611716745.