Junit5: Selecting one method in JUnit 4 parameterized test has no effect - test method is not executed

Created on 20 Oct 2016 · 23Comments · Source: junit-team/junit5

Example test code

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;

import java.util.Arrays;
import java.util.Collection;

/**
 * Created 20/10/16 16:51
 *
 * @author Vladimir Bogodukhov
 **/
@RunWith(Parameterized.class)
public class JUnit4ParametrizedExample {

    final boolean b;
    final long l;
    final int i;

    @Parameterized.Parameters(name = "{0} ---> : Test")
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[][]{
                {false, 1L, 21},
                {true, 21L, 11},
                {false, 10L, 21},
        });
    }

    public JUnit4ParametrizedExample(boolean b, long l, int i) {
        this.b = b;
        this.l = l;
        this.i = i;
    }


    @Test
    public void test1() {
        throw new IllegalStateException(); /* always fails */
    }

    @Test
    public void test2() {
        /* always succeed */
    }
}

Test that reproduces the bug

import org.junit.Assert;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.junit.platform.engine.TestExecutionResult;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.*;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;

import java.util.HashMap;
import java.util.Map;

/**
 * @author Vladimir Bogodukhov
 */
@Tag("DebugRun")
public class DynamicTestsHelperRun {

    enum State {
        skipped,
        failed,
        success,
        aborted
    }

    @Test
    void test1() {
        Launcher launcher = LauncherFactory.create();

        Map<State, Integer> callCounts = new HashMap<>();
        launcher.registerTestExecutionListeners(new TestExecutionListener() {
            @Override
            public void executionSkipped(TestIdentifier identifier, String reason) {

                if (identifier.isTest()) {
                    callCounts.putIfAbsent(State.skipped, 0);
                    callCounts.put(State.skipped, callCounts.get(State.skipped) + 1);
                }
            }

            @Override
            public void executionFinished(TestIdentifier identifier, TestExecutionResult result) {
                if (identifier.isTest()) {
                    State state = calcState(result);
                    callCounts.putIfAbsent(state, 0);
                    callCounts.put(state, callCounts.get(state) + 1);
                }
            }
        });


        final LauncherDiscoveryRequestBuilder builder = LauncherDiscoveryRequestBuilder.request();
        builder.selectors(DiscoverySelectors.selectJavaClass(JUnit4ParametrizedExample.class));
        LauncherDiscoveryRequest discoveryRequest = builder.build();

        TestPlan discover = launcher.discover(discoveryRequest);
        launcher.execute(discoveryRequest); 
        /* We call class with two parametrized methods - each method is called three times.  One test always fails another always succeed 
        And we get right result
        */

        Assert.assertEquals(new HashMap<State, Integer>() {
            {
                put(State.success, 3); /* Three succedd */
                put(State.failed, 3);/* Three failed */
            }
        }, callCounts);
    }


    @Test
        // Test that fails but certainly mustn't
    void test2() {
        Launcher launcher = LauncherFactory.create();

        Map<State, Integer> callCounts = new HashMap<>();
        launcher.registerTestExecutionListeners(new TestExecutionListener() {
            @Override
            public void executionSkipped(TestIdentifier identifier, String reason) {

                if (identifier.isTest()) {
                    callCounts.putIfAbsent(State.skipped, 0);
                    callCounts.put(State.skipped, callCounts.get(State.skipped) + 1);
                }
            }

            @Override
            public void executionFinished(TestIdentifier identifier, TestExecutionResult result) {
                if (identifier.isTest()) {
                    State state = calcState(result);
                    callCounts.putIfAbsent(state, 0);
                    callCounts.put(state, callCounts.get(state) + 1);
                }
            }
        });

        final LauncherDiscoveryRequestBuilder builder = LauncherDiscoveryRequestBuilder.request();
        builder.selectors(DiscoverySelectors.selectJavaMethod(JUnit4ParametrizedExample.class, "test1")); 
        LauncherDiscoveryRequest discoveryRequest = builder.build();

        TestPlan discover = launcher.discover(discoveryRequest);
        launcher.execute(discoveryRequest);
        /* we select  test1() method that always fails */



        /* if we call to test method that fails three times it is logical to get three failures as result */
        Assert.assertEquals(new HashMap<State, Integer>() {
            {
                put(State.failed, 3);/* Three failed */
            }
        }, callCounts);
        /* However it is not so -  we get actually only one call.  and we do not stop at the breakpoint  in test code  */
    }

    private static State calcState(TestExecutionResult result) {
        TestExecutionResult.Status status = result.getStatus();
        switch (status) {
            case ABORTED: {
                return State.aborted;
            }

            case FAILED: {
                return State.failed;
            }

            case SUCCESSFUL: {
                return State.success;
            }

            default: {
                throw new IllegalStateException("unknown state");
            }
        }
    }

}

Not that using DiscoverySelectors.selectJavaMethod(JUnit4ParametrizedExample.class, JUnit4ParametrizedExample.class.getMethod("test1")) instead of DiscoverySelectors.selectJavaMethod(JUnit4ParametrizedExample.class, "test1") has the same effect.

Vintage bug up-for-grabs

Source

SqAutoTestTeam

All 23 comments

I will provide more info should it be required.

SqAutoTestTeam on 20 Oct 2016

What version of the JUnit Platform are you using?

sbrannen on 28 Oct 2016

This is caused by JUnit Vintage's inability to filter for the method: junit-team/junit4#664.

marcphilipp on 1 Nov 2016

What version of the JUnit Platform are you using?

Version M3 - Built from Sources snapshot on 11-10-2016

This is caused by JUnit Vintage's inability to filter for the method: junit-team/junit4#664.

OK.

SqAutoTestTeam on 1 Nov 2016

Are there any plans when it's going to be fixed ?

SqAutoTestTeam on 9 Nov 2016

Have you tried using selectors for unique IDs instead?

marcphilipp on 16 Nov 2016

This issue is blocked by junit-team/junit4#664.

marcphilipp on 16 Nov 2016

@marcphilipp

Have you tried using selectors for unique IDs instead?

No I haven't.

SqAutoTestTeam on 2 Dec 2016

Can you try?

marcphilipp on 2 Dec 2016

Yes I think I can. But could you please show how uniuqeId is composed - I could not find any info by this link http://junit.org/junit5/docs/current/user-guide/

Composing UniqueId by experiment gave the same result


 UniqueId uniqueId =
         UniqueId.parse("[engine:junit-vintage]/[runner:com.company.junit5.bug.b2.JUnit4ParametrizedExample]/[test:test1(com.company.junit5.bug.b2.JUnit4ParametrizedExample)]");


 builder.selectors(DiscoverySelectors.selectUniqueId(uniqueId));
 LauncherDiscoveryRequest discoveryRequest = builder.build();


TestPlan discover = launcher.discover(discoveryRequest);
launcher.execute(discoveryRequest);

Tested method is still not invoked.

SqAutoTestTeam on 9 Feb 2017

You shouldn't compose the unique ID yourself. You can select the appropriate TestIdentifier from the TestPlan that is returned by discover. If you look at their TestSources you can find the one that corresponds with the method and get its unique ID.

marcphilipp on 12 Feb 2017

You can select the appropriate TestIdentifier from the TestPlan that is returned by discover.

I did it so. I got it from TestPlan. Then I saved gotten Id to a string and after it I created UniqueId using gotten data.
As I told you before it gives the same negative result. A method identified by its unique Id is not invoked.

SqAutoTestTeam on 13 Feb 2017

Hey junit team, will the fix for this issue be in milestone 5 or milestone 6?