Cucumber-jvm: Maven BUILD SUCCESSFUL despite exceptions thrown in @Before

Created on 23 Jun 2018 · 11Comments · Source: cucumber/cucumber-jvm

Summary

I implemented a custom exception that extends RuntimeException and throw it. I intend the maven build to fail if it is thrown so I do not catch it anywhere. However, mvn clean install still builds successfully despite the custom exception being thrown in @Before.

~~~
// Custom exception
@SuppressWarnings("serial")
public class UnsupportedBrowserException extends RuntimeException {
public UnsupportedBrowserException(String message) {
super(message);
}
}

// Method that throws the exception
public final class BrowserFactory {

public static WebDriver getBrowser(String browser) {
WebDriver driver = null;

switch (browser.toUpperCase()) {
...
default:
    if (StringUtils.isBlank(browser)) {
        browser = "BLANK";
    }
    throw new UnsupportedBrowserException(ExceptionMessages.UNSUPPORTED_BROWSER + browser);
}

return driver;

}

// Call method
import cucumber.api.java.After;
import cucumber.api.java.Before;

public class ScenarioHook {
private WebDriver driver = null;

@Before
public void beforeScenario(Scenario scenario) throws Throwable {
    String browser = PropsLoader.readConfig("browser");
    driver = BrowserFactory.getBrowser(browser);
}

@After
public void afterScenario() {
    driver.quit();
}

}
~~~

Expected Behavior

Maven build is expected to fail if exception is thrown in @Before (or any hook).

Current Behavior

A successful build is returned because tests are being skipped when the exception is thrown within the hook. I also induced a NullPointerException to see if it was my custom exception causing the issue. It appears _any_ exception thrown in @before causes the tests to skip and therefore reports a successful build.
Exceptions thrown in non-hook methods promptly fails the build.

sshot_1
~
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 5.243 s - in jcucumberng.project.CucumberRunner
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ jcucumberng-framework ---
[INFO] Building jar: D:\dev\automation-testing\java\jCucumberNG-Framework\target\jcucumberng-framework-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-cucumber-reporting:3.17.1:generate (execution) @ jcucumberng-framework ---
[INFO] About to generate Cucumber report.
Jun 23, 2018 9:23:33 PM net.masterthought.cucumber.ReportParser parseJsonFiles
INFO: File 'D:\dev\automation-testing\java\jCucumberNG-Framework\target\cucumber-output\test-report.json' contains 2 features
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ jcucumberng-framework ---
[INFO] Installing D:\dev\automation-testing\java\jCucumberNG-Framework\target\jcucumberng-framework-0.0.1-SNAPSHOT.jar to C:\Users\Kat Rollo.m2\repository\krollo\jcucumberng\jcucumberng-framework\0.0.1-SNAPSHOT\jcucumberng-framework-0.0.1-SNAPSHOT.jar
[INFO] Installing D:\dev\automation-testing\java\jCucumberNG-Framework\pom.xml to C:\Users\Kat Rollo.m2\repository\krollo\jcucumberng\jcucumberng-framework\0.0.1-SNAPSHOT\jcucumberng-framework-0.0.1-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.227 s
[INFO] Finished at: 2018-06-23T21:23:34+08:00
[INFO] ------------------------------------------------------------------------
~

Possible Solution

JUnit @Before fails the build if an exception is thrown.

Context & Motivation

We would usually expect a build failure if an exception is thrown in the hook (since other frameworks do this). It may be worth checking if the same behavior occurs in @after, @beforestep, @afterstep.

Your Environment

Version used: 3.0.2
Operating System and version: Win7 x64

pinned

Source

kathyrollo

Most helpful comment

@mpkorstanje @mlvandijk

I think I have isolated the issue, I just had another suspicion - the AbstractTestNGCucumberTests in the runner class. I have been using cucumber-testng dependency.

I reverted to cucumber-junit, used the typical @RunWith(Cucumber.class), and removed extends AbstractTestNGCucumberTests. The <testFailureIgnore> config now promptly passes or fails the build with true or false respectively when my custom exception is thrown within @before hook which is the desired behavior.

It may be the AbstractTestNGCucumberTests. Let me know if you arrive at the same conclusion.

P.S.
Will be joining the Slack group after my holiday this week. I pushed enhance-33-junit. :)

kathyrollo on 25 Jun 2018

🎉1 👍1

All 11 comments

I can't reproduce this from the information you've provided.

Do your tests fail if you run the JUnit runner directly from your IDE? If so you may have to check your maven configuration. Note that if you are running cucumber as an integration tests using failsafe, running the integration tests and failing the build when the tests fail are separate actions.

If your tests also pass/skip when running JUnit directly from the IDE I would like to see a minimal reproducer, preferably in the form of a github repo.

mpkorstanje on 23 Jun 2018

Hi @mpkorstanje, thank you for your attention to this ticket.

Running my CucumberRunner.java from the IDE also skipped the tests. The runner extends AbstractTestNGCucumberTests.
~~~
[RemoteTestNG] detected TestNG version 6.14.3
...

SKIPPED: runScenario("Calculate Net Income Per Year", "Net Income Projector")
Runs Cucumber Scenarios
jcucumberng.framework.exceptions.UnsupportedBrowserException: See config.properties. Unsupported browser: BLANK
at jcucumberng.framework.factory.BrowserFactory.getBrowser(BrowserFactory.java:75)
at jcucumberng.project.hooks.ScenarioHook.beforeScenario(ScenarioHook.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at cucumber.runtime.Utils$1.call(Utils.java:26)
at cucumber.runtime.Timeout.timeout(Timeout.java:16)
at cucumber.runtime.Utils.invoke(Utils.java:20)
at cucumber.runtime.java.JavaHookDefinition.execute(JavaHookDefinition.java:60)
at cucumber.runtime.HookDefinitionMatch.runStep(HookDefinitionMatch.java:14)
at cucumber.runner.TestStep.executeStep(TestStep.java:55)
at cucumber.runner.TestStep.run(TestStep.java:42)
at cucumber.runner.TestCase.run(TestCase.java:41)
at cucumber.runner.Runner.runPickle(Runner.java:44)
at cucumber.api.testng.TestNGCucumberRunner.runScenario(TestNGCucumberRunner.java:56)
at cucumber.api.testng.AbstractTestNGCucumberTests.runScenario(AbstractTestNGCucumberTests.java:22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
at org.testng.TestNG.runSuites(TestNG.java:1049)
at org.testng.TestNG.run(TestNG.java:1017)
at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

===============================================
Default test

Tests run: 3, Failures: 0, Skips: 3

===============================================
Default suite

Total tests run: 3, Failures: 0, Skips: 3

~~~

This is my repo: https://github.com/kathyrollo/jcucumberng-framework
Feel free to pull branch enhance-33 and run mvn clean install without further config needed.

Below is the dir structure:
sshot_3

Let me know if there are any further details you need.

kathyrollo on 25 Jun 2018

@kathyrollo On branch enhance-33 in your pom.xml line 99, I see <testFailureIgnore>true</testFailureIgnore> -> this means that test failures are ignored. Either remove this line or set to false.

mlvandijk on 25 Jun 2018

Hi @mlvandijk, that was one of my first suspicions and setting it to false still skips the steps oddly enough.

kathyrollo on 25 Jun 2018

Yes, I noticed on SO ;) Have you tried removing it?

mlvandijk on 25 Jun 2018

Haha my SO posts are too easy to find. :) Yep, removed the entire line now, still successful.

kathyrollo on 25 Jun 2018

Ah, crap. Will see if I can have a look at your repo tonight. Also, join us on Slack for easier conversation? (If you havent already)

mlvandijk on 25 Jun 2018

@mpkorstanje @mlvandijk

I think I have isolated the issue, I just had another suspicion - the AbstractTestNGCucumberTests in the runner class. I have been using cucumber-testng dependency.

It may be the AbstractTestNGCucumberTests. Let me know if you arrive at the same conclusion.

P.S.
Will be joining the Slack group after my holiday this week. I pushed enhance-33-junit. :)

kathyrollo on 25 Jun 2018

🎉1 👍1

The TestNG reporter/formatter is listening to individual steps and
ignoring hooks.

Not sure why it was designed that way.

It should listen test case finished events.

On Mon, Jun 25, 2018, 15:36 Kat Rollo notifications@github.com wrote:

@mpkorstanje https://github.com/mpkorstanje @mlvandijk
https://github.com/mlvandijk

I think I have isolated the issue, I just had another suspicion - the
AbstractTestNGCucumberTests in the runner class. I have been using
cucumber-testng dependency.

I reverted to cucumber-junit, used the typical @RunWith(Cucumber.class),
and removed extends AbstractTestNGCucumberTests. The
config now promptly passes or fails the build with true or false
respectively when my custom exception is thrown within @before
https://github.com/before hook which is the desired behavior.

It may be the AbstractTestNGCucumberTests. Let me know if you arrive at
the same conclusion.

P.S.
Will be joining the Slack group after my holiday this week. :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cucumber/cucumber-jvm/issues/1403#issuecomment-399953873,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGoAZ9LinzeKasUdKrnaFkhrkNyKLrF4ks5uAOdEgaJpZM4U0y42
.

mpkorstanje on 25 Jun 2018

👍1

This was fixed as part of #1389.

If you want to move on with version 4.0 you can find instructions on getting the latest snapshot here: https://docs.cucumber.io/installation/java/#snapshot-releases the version number is in the pom.

mpkorstanje on 28 Aug 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.