Junit5: Test not failing if BeforeAll crashes

[hheg]

Junit 5.4.2 and 5.6.0
Attached a failing example

Rationale expected:
If a BeforeAll method crashes, the test should be reported as failed

Actual
Tests are silently ignored and the result is success.

Steps to reproduce

Create a maven project with Junit 5 as test runner
Create a test with a BeforeAll method which throws an RuntimeException
See that the tests passes. Example below

[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ junit_fail ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running junit_fail.JunitFailTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in junit_fail.JunitFailTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertTrue;

public class JunitFailTest {

    @BeforeAll
    public static void setUp () {
        throw new RuntimeException("Should be noted");
    }

    @Test
    public void test1() {
        assertTrue(false);
    }
    @Test
    public void test2() {
        assertTrue(true);
    }
}

Context

• Used versions (Jupiter/Vintage/Platform):
  5.6.0//1.6.0
• Build Tool/IDE:
  Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
  openjdk 11.0.6 2020-01-14
  OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10)
  OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.6+10, mixed mode)
  and
  openjdk version "1.8.0_242"
  OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_242-b08)
  OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.242-b08, mixed mode)

Deliverables

• [x] ...

junit_fail.zip

[sormuras]

Didn't look into the attached project, yet, but it seems that Surefire didn't launch the JUnit Platform. Thus, all methods with JUnit Jupiter annotations weren't invoked. Can you please run Maven with the -X parameter and post the Surefure part of the log here?

[pzygielo]

It works fine with surefire 3.0.0-M3, and fails as described by author in M4 indeed. Probably SUREFIRE-1688.

[hheg]

It works fine with surefire 3.0.0-M3, and fails as described by author in M4 indeed. Probably SUREFIRE-1688.

That bug fix is included in M4. Perhaps this use case broke when they fixed the assertion mechanics with SUREFIRE-1688?

[hheg]

Didn't look into the attached project, yet, but it seems that Surefire didn't launch the JUnit Platform. Thus, all methods with JUnit Jupiter annotations weren't invoked. Can you please run Maven with the -X parameter and post the Surefure part of the log here?

output.log

[hheg]

Running without the exception in the BeforeAll method does invoke the tests like this:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running junit_fail.JunitFailTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.038 s <<< FAILURE! - in junit_fail.JunitFailTest
[ERROR] junit_fail.JunitFailTest.test1  Time elapsed: 0.016 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
    at junit_fail.JunitFailTest.test1(JunitFailTest.java:16)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   JunitFailTest.test1:16 expected: <true> but was: <false>
[INFO] 
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[pzygielo]

It works fine with surefire 3.0.0-M3, and fails as described by author in M4 indeed. Probably SUREFIRE-1688.

That bug fix is included in M4.

But check comment and comment to not be misled by status & fixed version.

[sormuras]

Added another comment to the ticket and reopened it.

[sormuras]

@hheg can you please use the _latest-and-greatest_ version of Surefire, namely its SNAPSHOT build?

According to this comment and the following one, this issue already fixed.

[hheg]

I can confirm that the bug is indeed resolved with the latest 3.0.0-SNAPSHOT 873da2808

[sormuras]

Thus closing this issue as Surefire 3.0.0-M5 will be released ... soon, I guess.

If there's a regression, please report further issues at the Apache Maven issue tracker.

[ksiddu]

@sormuras ,
Is there any way to fail the test cases(methods with @Test tag) when @BeforeAll setup method fails.
I have two test cases in one test class with @BeforeAll and @AfterAll,
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] FirstTest.beforeAllFirstTest:23 expected: but was:
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

How can i mark those 2 test cases as failed if the @BeforeAll method fails

[Tibor17]

I found that the problem is not with the resolution of the artifacts in the POM. In the issue SUREFIRE-1688 we fixed very similar problem where the AssertionError was throws in BeforeAll method. Here we throw the RuntimeException in the same method and we get zero total tests which is the bug. So it looks like these two exceptions are handled differently in Surefire.

[cobbce]

Experiencing the same problem in which a runtime exception shows as 0 tests run-- any resolution on when a fix will be available. Pretty terrifying when your test runner silently fails...

[marcphilipp]

@Tibor17 If I understand you correctly, you're saying this issue might not be fixed by https://issues.apache.org/jira/browse/SUREFIRE-1688? If so, @ksiddu or @cobbce, please report a new issue on Surefire's issue tracker.

[Tibor17]

@marcphilipp To be more concrete, the analysis showed me that the sources of the IT for 1688 and the sources junit_fail.zip in this GH bug are the same except the type of thrown exception. In the IT 1688 we call fail("some text") and here we throw RuntimeException. I have to debug our code to be more concrete because i also do not know more yet.

[Tibor17]

@ksiddu @cobbce Don't open Jira in the Apache. We have it and i will debug the code which may result in closing our issue and reopening the old one or keeping it, or it is JUnit5 bug. But now nobody knows this.

[Tibor17]

This issue was handled in Apache JIRA and closed as a duplicate and not reproducible.
Not reproducible after re-testing with the latest SNAPSHOT version.
This was fixed with in SUREFIRE-1688 and SUREFIRE-1741.

[cobbce]

I don't think you should expect every developer in the world to understand
your internal process for managing bugs across bug trackers-- I just found
a recent thread discussing this issue and am asking when the next release
of surefire will be made available (also the surfire homepage lists 8
release milestones right at the top, but you have to dig a layer deeper to
actually determine the latest release version, which remains at M4). I
don't use external SNAPSHOT dependencies for production systems and am just
trying to find out when 3.0.0-M5 will be made available. In the meantime
I've worked around the bug that is still present in the latest release
version (M4). You may want to consider refining your bug tracking process
to denote not just Closed status, but also Release status. Having a bug
fixed in a snapshot that is not available in a safe artifact is not helpful
in my circumstance.

Cheers

On Tue, Apr 21, 2020 at 4:24 AM Tibor Digana [e-mail]
wrote:

This issue was handled in Apache JIRA
https://issues.apache.org/jira/browse/SUREFIRE-1750 and closed as a
duplicate an not reproducible.
Not reproducible after re-testing with the latest SNAPSHOT version.
This was fixed with in SUREFIRE-1688 and SUREFIRE-1741.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/junit-team/junit5/issues/2178#issuecomment-617118558,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDTCOHH24SJTW3L6NGFPN3RNV7ANANCNFSM4KSLYFUA
.

[noguespi]

This issue was handled in Apache JIRA and closed as a duplicate and not reproducible.
Not reproducible after re-testing with the latest SNAPSHOT version.
This was fixed with in SUREFIRE-1688 and SUREFIRE-1741.

Fixed for you, but not for us :man_shrugging:
I'm affected by this bug on the latest stable release (3.0.0-M4). Downgrading didn't fix the issue.

My ugly hack to fix the bug :
try/catch in beforeAll and System.exit(1) on exception.

[silh]

Still having the same issue with 3.7.0.