Testng: Dataprovider is executed for ignored tests in case of the BeforeClass failure

[baflQA]

TestNG Version

7.0.0-beta3

Expected behavior

Dataprovider is skipped when the @BeforeClass configuration method fails, and all the tests are skipped.

Actual behavior

The @BeforeMethod config is skipped, Dataprovider is executed, then the tests are skipped as well.
Maybe dataprovider lacks alwaysRun=true flag?
Running dataproviders when afterClass/Methods are skipped causes some objects to be created but not cleaned up.

Is the issue reproductible on runner?

• [ ] Shell
• [ ] Maven
• [ ] Gradle
• [ ] Ant
• [ ] Eclipse
• [x] IntelliJ
• [ ] NetBeans

Test case sample

https://github.com/baflQA/testNG_parallel_debug/blob/37aa3c2822996599cc25a04b03a0cce9ac020879/src/test/java/testngparallel/Class1.java
Please run the "test1" test case.

[krmahadevan]

Quick question : why do you want the data provider to be executed when a @BeforeClass has a failure? Can you please elaborate

[baflQA]

Hi!
The thing is that I don't want to execute dataproviders in such case, but they are executed ;)
So if it's a bug, I will be glad to hear that. But if it's a feature - alwaysRun would let someone still use it. For questionable reasons.

[baflQA]

That's why I wrote:

Expected behavior
Dataprovider is skipped when the @BeforeClass configuration method fails, and all the tests are skipped.

[krmahadevan]

Ah ok! Sorry I misread it. Let me see what is going on. I agree. It's a bit counter intuitive for the data provider to still run when there is a config failure at class level.

[krmahadevan]

@baflQA - Fixing this issue is going to be tricky from TestNG side. It looks like fixing this would cause a lot of regression.

So here's an easy work-around that would ensure that the data provider doesn't get executed if there's a configuration failure.

import static org.testng.Assert.fail;

import com.rationaleemotions.github.issue2049.SampleTestClass.FailureDetecter;
import org.testng.IConfigurationListener;
import org.testng.ITestContext;
import org.testng.ITestResult;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(FailureDetecter.class)
public class SampleTestClass {
  @BeforeClass
  public void beforeClass() {
    System.err.println("BEFORE CLASS");
    fail();
  }

  @Test(dataProvider = "dp1master")
  public void test1(Class<?> clazz) {
    System.err.println("Running test for class " + clazz.getSimpleName());
  }

  @DataProvider
  public Object[][] dp1master(ITestContext ctx) {
    if (ctx.getAttribute(FailureDetecter.FAILURE) != null) {
      throw new RuntimeException("Failure");
    }
    return new Object[][] {new Object[] {Object.class}, {Object.class}};
  }

  public static class FailureDetecter implements IConfigurationListener {
    public static final String FAILURE = "failure";

    @Override
    public void onConfigurationFailure(ITestResult itr) {
      itr.getTestContext().setAttribute(FAILURE, Boolean.TRUE.toString());
    }
  }
}

Output

BEFORE CLASS

java.lang.AssertionError: null

    at org.testng.Assert.fail(Assert.java:97)
    at org.testng.Assert.fail(Assert.java:102)
    at com.rationaleemotions.github.issue2049.SampleTestClass.beforeClass(SampleTestClass.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131)
    at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61)
    at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:340)
    at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:294)
    at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:176)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:122)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

[Utils] [ERROR] [Error] java.lang.RuntimeException: Failure

    at com.rationaleemotions.github.issue2049.SampleTestClass.dp1master(SampleTestClass.java:31)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)


Test ignored.
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:131)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:76)
    at org.testng.internal.MethodInvocationHelper.invokeMethodNoCheckedException(MethodInvocationHelper.java:45)
    at org.testng.internal.MethodInvocationHelper.invokeDataProvider(MethodInvocationHelper.java:144)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:794)
    at org.testng.internal.Parameters.handleParameters(Parameters.java:739)
    at org.testng.internal.ParameterHandler.handleParameters(ParameterHandler.java:60)
    at org.testng.internal.ParameterHandler.createParameters(ParameterHandler.java:39)
    at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:757)
    at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:143)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
    at org.testng.TestRunner.privateRun(TestRunner.java:763)
    at org.testng.TestRunner.run(TestRunner.java:594)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:398)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:392)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:355)
    at org.testng.SuiteRunner.run(SuiteRunner.java:304)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1146)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1067)
    at org.testng.TestNG.runSuites(TestNG.java:997)
    at org.testng.TestNG.run(TestNG.java:965)
    at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)


===============================================
Default Suite
Total tests run: 1, Passes: 0, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 0
===============================================


Process finished with exit code 0

[baflQA]

Hi @krmahadevan ! Thx for Your effort.
I'm sad to hear that it's not a piece of cake to fix this ;)
Based on Your comment I've come up with this approach:
https://github.com/baflQA/testNG_parallel_debug/commit/321fbcc8872df5e611c788e13dc2e62add805163
Looks like it will do the same, but I won't need to implement the logic in each dataprovider I have.
So, I'm fine with this approach, and I think We can close this issue.

[krmahadevan]

@baflQA - Awesome. Good improvisation there :)

Based on your comments, I am closing this issue.