Junit5: How to use internal custom TestEngine implementations in Maven-Surefire-Plugin?

Created on 12 Oct 2018 · 24Comments · Source: junit-team/junit5

For developing custom TestEngine classes there already is a lot of documentation on how to use them:

ServiceLoader and plugin system
LauncherConfig

Well, I tried to use a custom TestEngine with ServiceLoader (intern), it works fine in IntelliJ or using a Launcher, but I didn't get anything while running mvn test.

In Maven Surefire documentation it's mentioned that you only need to add JUnit-Jupiter-Engine in dependencies without the help of an provider. But these engines are external. What about internal engines?

Maven Surefire question

Source

performrumpf

All 24 comments

But these engines are external. What about internal engines?

What are _external_ and _internal_ engines?

Do you have a pom.xml snippet to share?

sormuras on 12 Oct 2018

Here is an integration test showing how to include three engines in a pom.xml:

https://github.com/apache/maven-surefire/blob/19006aa70f36705f399b8c105a16f636904f00f3/surefire-its/src/test/resources/junit-platform-multiple-engines/pom.xml#L36-L71

Basically, any test engine implementation available at test runtime (via class-path or module-path) that is registered via the ServiceLoader plugin system is picked up the the JUnit Platform.

sormuras on 12 Oct 2018

    <properties>
        <java.version>1.8</java.version>
        <junit.platform.version>1.3.1</junit.platform.version>
    </properties>

    <build>
        <directory>target</directory>
        <outputDirectory>target/classes</outputDirectory>
        <testOutputDirectory>target/test-classes</testOutputDirectory>
        <sourceDirectory>src/main/java</sourceDirectory>
        <testSourceDirectory>src/test/java</testSourceDirectory>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <testResources>
            <testResource>
                <directory>src/test/resources</directory>
            </testResource>
        </testResources>

        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.7.0</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-engine</artifactId>
            <version>${junit.platform.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>${junit.platform.version}</version>
        </dependency>
    </dependencies>

Project structure

src
- main
  - java
  - resources
- test
  - java
  - com.experiment.engine
    - MyTestEngine.java
    - MyTestDescription.java
  - resources
  - META-INF
    - services
    - org.junit.platform.engine.TestEngine

performrumpf on 12 Oct 2018

I overwrote discover and execute methods in a way that engineExecutionListener fires "start" and "finish" for the root and one child.

Internal = same maven project
External = using in pom

performrumpf on 12 Oct 2018

I overwrote discover and execute methods in a way that engineExecutionListener fires "start" and "finish" for the root and one child

Are you extending JupiterTestEngine? Or extending some class in junit-platform-launcher?

sormuras on 12 Oct 2018

I don't extend from JupiterTestEngine as that test engine will build tests from other resources (i.e. XML files). I use launcher only to see if LauncherConfig works or (for the future) if DiscoverySelectors work properly.

public class MyTestEngine implements TestEngine {
   // implemented methods
}

performrumpf on 12 Oct 2018

👍1

Can you please move your com.experiment.engine with its org.junit.platform.engine.TestEngine into the main source set and see if Surefire now sees your engine?

sormuras on 12 Oct 2018

What is listed in Surefire's class-paths when you invoke your build with mvn --debug ...?

sormuras on 12 Oct 2018

unfortunately moving it to main didn't help. Also I couldn't find any useful information in debug mode about class path. The only line i could find with class path was
<additionalClasspathElements>${maven.test.additionalClasspath}</additionalClasspathElements>
<classpathDependencyExcludes>${maven.test.dependency.excludes}</classpathDependencyExcludes>

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.experimental:junit5-sandbox >-------------------
[INFO] Building junit5-sandbox 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ junit5-sandbox ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ junit5-sandbox ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ junit5-sandbox ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 34 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ junit5-sandbox ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
[INFO] Compiling 1 source file to C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ junit5-sandbox ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.773 s
[INFO] Finished at: 2018-10-12T11:10:58+02:00
[INFO] ------------------------------------------------------------------------

performrumpf on 12 Oct 2018

What happens if you move META-INF/services/org.junit.platform.engine.TestEngine to src/main/resources as well?

sbrannen on 12 Oct 2018

This is not a --debug enabled printout. Seems only [INFO]- and [DEBUG]-level loggings are emitted.

sormuras on 12 Oct 2018

What happens if you move META-INF/services/org.junit.platform.engine.TestEngine to src/main/resources as well?

Already did, same result: nothing

performrumpf on 12 Oct 2018

Does this help you?

[DEBUG]   (s) additionalClasspathElements = []
[DEBUG]   (s) basedir = C:\Users\c.rumpf\Documents\perform\junit5sandbox
[DEBUG]   (s) childDelegation = false
[DEBUG]   (s) classesDirectory = C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\classes
[DEBUG]   (s) classpathDependencyExcludes = []
[DEBUG]   (s) dependenciesToScan = []
[DEBUG]   (s) disableXmlReport = false
[DEBUG]   (s) enableAssertions = true
[DEBUG]   (f) forkCount = 1
[DEBUG]   (s) forkMode = once
[DEBUG]   (s) forkedProcessExitTimeoutInSeconds = 30
[DEBUG]   (s) junitArtifactName = junit:junit
[DEBUG]   (s) junitPlatformArtifactName = org.junit.platform:junit-platform-engine
[DEBUG]   (s) localRepository =       id: local
      url: file:///C:/Users/c.rumpf/.m2/repository/
   layout: default
snapshots: [enabled => true, update => always]
 releases: [enabled => true, update => always]

[DEBUG]   (f) parallelMavenExecution = false
[DEBUG]   (s) parallelOptimized = true
[DEBUG]   (s) perCoreThreadCount = true

performrumpf on 12 Oct 2018

Below those configurations there should be line like [DEBUG] -- end configuration -- and then the interesting lines start. Look for lines starting with [DEBUG] test(compact) classpath: and [DEBUG] provider(compact) classpath:.

sormuras on 12 Oct 2018

At "maven-compiler-plugin":

[DEBUG] Classpath:
[DEBUG]  C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[DEBUG]  C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\classes
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\junit\platform\junit-platform-engine\1.3.1\junit-platform-engine-1.3.1.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\apiguardian\apiguardian-api\1.0.0\apiguardian-api-1.0.0.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\junit\platform\junit-platform-commons\1.3.1\junit-platform-commons-1.3.1.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\opentest4j\opentest4j\1.1.1\opentest4j-1.1.1.jar
[DEBUG]  C:\Users\c.rumpf\.m2\repository\org\junit\platform\junit-platform-launcher\1.3.1\junit-platform-launcher-1.3.1.jar

Surefire didn't display any classpath informations

performrumpf on 12 Oct 2018

Does C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes contain the MyEngine.class AND the META-INF/services/org.junit.platform.engine.TestEngine file?

Surefire didn't display any classpath informations

Strange...

sormuras on 12 Oct 2018

Does C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes contain the MyEngine.class AND the META-INF/services/org.junit.platform.engine.TestEngine file?

When they are under src/test/..., then yes. Otherwise they are in target\classes

performrumpf on 12 Oct 2018

Some more suggestions:

Add <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> to your <properties>
Update the maven-compiler-plugin to 3.8.0
Help Surefire to select its surefire-junit-platform provider

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.maven.surefire</groupId>
                        <artifactId>surefire-junit-platform</artifactId>
                        <version>2.22.1</version>
                    </dependency>
                </dependencies>
            </plugin>

For details see https://maven.apache.org/surefire/maven-surefire-plugin/examples/providers.html

Got it working with the ice.cream engine -- without any src/main directory.

sormuras on 12 Oct 2018

👍1

I made the suggested changes, but I don't get this last part of yours. Maybe Surefire is handling test discovery from a TestEngine differently then DefaultLauncher.

C:\Users\c.rumpf\Documents\perform\junit5sandbox>mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.experimental:junit5-sandbox >-------------------
[INFO] Building junit5-sandbox 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ junit5-sandbox ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 35 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ junit5-sandbox ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 4 source files to C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ junit5-sandbox ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.708 s
[INFO] Finished at: 2018-10-16T08:55:47+02:00
[INFO] ------------------------------------------------------------------------

Here my pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.experimental</groupId>
    <artifactId>junit5-sandbox</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
        <junit.platform.version>1.3.1</junit.platform.version>
    </properties>

    <build>

        <directory>target</directory>
        <outputDirectory>target/classes</outputDirectory>
        <testOutputDirectory>target/test-classes</testOutputDirectory>
        <sourceDirectory>src/main/java</sourceDirectory>
        <testSourceDirectory>src/test/java</testSourceDirectory>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
            </resource>
        </resources>
        <testResources>
            <testResource>
                <directory>src/test/resources</directory>
            </testResource>
        </testResources>

        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.maven.surefire</groupId>
                        <artifactId>surefire-junit-platform</artifactId>
                        <version>2.22.1</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-engine</artifactId>
            <version>${junit.platform.version}</version>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>${junit.platform.version}</version>
        </dependency>
    </dependencies>
</project>

And here is MyTestEngine.java

public class MyTestEngine implements TestEngine {

    @Override
    public String getId() {
        return "my-test-engine";
    }

    @Override
    public TestDescriptor discover(EngineDiscoveryRequest engineDiscoveryRequest, UniqueId uniqueId) {
        UniqueId rootId = uniqueId.append("container", "root#1");
        TestDescriptor root = new MyEngineDescriptor(uniqueId,"Root Description");
        TestDescriptor rootContainer = new MyTestDescriptor(rootId, "Container no. 1", TestDescriptor.Type.CONTAINER);

        rootContainer.addChild(new MyTestDescriptor(rootId.append("test", "child#1"), "Child no. 1", TestDescriptor.Type.TEST));
        rootContainer.addChild(new MyTestDescriptor(rootId.append("test", "child#2"), "Child no. 2", TestDescriptor.Type.TEST));
        root.addChild(rootContainer);
        return root;
    }

    @Override
    public void execute(ExecutionRequest executionRequest) {

        EngineExecutionListener listener = executionRequest.getEngineExecutionListener();
        TestDescriptor root = executionRequest.getRootTestDescriptor();

        execute(listener, root);
    }

    private void execute(EngineExecutionListener listener, TestDescriptor testDescriptor) {
        listener.executionStarted(testDescriptor);

        if(testDescriptor.isContainer()) {
            for(TestDescriptor child : testDescriptor.getChildren()) {
                execute(listener, child);
            }
        }

        listener.executionFinished(testDescriptor, TestExecutionResult.successful());
    }

    @Override
    public Optional<String> getGroupId() {
        return Optional.of("com.experimental");
    }

    @Override
    public Optional<String> getArtifactId() {
        return Optional.of("my-test-engine");
    }

    @Override
    public Optional<String> getVersion() {
        return Optional.of("1.0.0");
    }
}

performrumpf on 16 Oct 2018

I guess this is a maven issue and not an JUnit platform issue. I did following steps which resulted in executing my test:

Added JUnit-Jupiter 5.3.1 to dependencies
Wrote a Java JUnit test with a dummy test method (no assertions or output)

C:\Users\c.rumpf\Documents\perform\junit5sandbox>mvn test
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< com.experimental:junit5-sandbox >-------------------
[INFO] Building junit5-sandbox 1.0.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ junit5-sandbox ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ junit5-sandbox ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 35 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ junit5-sandbox ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 5 source files to C:\Users\c.rumpf\Documents\perform\junit5sandbox\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ junit5-sandbox ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running Container no. 1
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 s - in Container no. 1
[INFO] Running com.experimental.test.MyTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in com.experimental.test.MyTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.956 s
[INFO] Finished at: 2018-10-16T09:37:41+02:00
[INFO] ------------------------------------------------------------------------

Maybe Surefire only consideres other TestEngine implementations if it finds at least one from JUnit.

performrumpf on 16 Oct 2018

I'd like to close this ticket as this is definitely a feature of Surefire. Following reasons:

Surefire looks for files having a certain pattern (see documentation)
The filter can be overwritten with a simple order: mvn test -Dtest=*
The filter can be overwritten by adding <test>*</test> in plugin configuration
Not recommended: Using a Java class which follows default filter criteria without having any test methods (i.e. foo/DummyTest.java)

Maybe this should be mentioned in JUnit 5 User Guide Chapter Configuring Test Engines.

performrumpf on 16 Oct 2018

Maybe this should be mentioned in JUnit 5 User Guide Chapter Configuring Test Engines.

Maven's default patterns and filtering are mentioned in the User Guide: https://junit.org/junit5/docs/current/user-guide/#running-tests-build-maven-filter-test-class-names

That section also links to: https://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html

Are you saying the User Guide should do more than it already does in this regard?

sbrannen on 16 Oct 2018

I'd like to close this ticket as this is definitely a feature of Surefire.

OK. I'm closing it for you now.

sbrannen on 16 Oct 2018

Surefire should select the correct Framework Provider automatically: having org.junit.platform:junit-platform-engine in your test runtime dependencies should trigger the selection of its surefire-junit-platform. As that didn't work out in my example, I added the surefire-junit-platform provider to the plugin dependencies, which forces this provider to be selected and executed.

sormuras on 16 Oct 2018

Was this page helpful?

0 / 5 - 0 ratings