Googletest: Fatal failure in SetUpTestCase() function should skip running tests.

[GoogleCodeExporter]

I think googletest should skip running tests (or automatically mark as
failed) in a particular fixture if we call SetUpTestCase() function
and there is fatal failure in it. Suppose we create some shared object
in SetUpTestCase() function and try to access that in TEST_F. But the
failure to create that object in SetUpTestCase() cause to access NULL
inside TEST_F.

So if we make fatal assertion in SetUpTestCase() itself, then above
problem can be avoided.

Original issue reported on code.google.com by [email protected] on 11 Jan 2010 at 6:39

[GoogleCodeExporter]

Original comment by [email protected] on 11 Jan 2010 at 6:46

• Changed state: Accepted
• Added labels: OpSys-All, Type-Enhancement, Usability
• Removed labels: Type-Defect

[GoogleCodeExporter]

I am also facing this problem, when I add testing machoes (i.e. EXPECT_EQ) in 
SetUpTestCase errors are not reported on the console if there is any.

Original comment by [email protected] on 28 Mar 2012 at 4:57

[GoogleCodeExporter]

Is this problem resolved?

Original comment by [email protected] on 8 Aug 2012 at 1:42

[GoogleCodeExporter]

Please, may you give an update of the issue? is it still under investigation? 
What about the TearDownTestCase()? I think that also errors in the 
TearDownTestCase() should be reported on the console and in some way invalidate 
the "Pass" TEST_F methods involved inthe fixture.

Original comment by [email protected] on 20 Nov 2012 at 1:06

[GoogleCodeExporter]

Hi,

I'm also facing this issue now.
In my case, I'm setting a database connection so it's logical that if I can't 
set it up (e.g. if I don't have permissions to read the specified database) I 
shouldn't be able to even run my tests.
For now, I have a segmentation fault.

Original comment by [email protected] on 17 Oct 2013 at 7:18

[GoogleCodeExporter]

I'm having the issue also. We are testing code that interfaces with an external 
app and setup needs to fail properly if that app doesn't start.  

Currently, I have an ASSERT in setup that properly prevents running the tests, 
but the final output still says

[==========] 3 tests from 1 test case ran. (5004 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 0 tests, listed below:

 0 FAILED TESTS

But that is, in fact, incorrect. None of the tests ran and NONE of them would 
have passed if they did.

Original comment by [email protected] on 6 Nov 2013 at 10:51

[GoogleCodeExporter]

I am still facing this issue. 
Is it considered to be fixed?
Could you please let us know of the status?

Original comment by [email protected] on 17 Jul 2014 at 6:22

[GoogleCodeExporter]

This is also on my wish list.

Original comment by [email protected] on 12 Sep 2014 at 10:00

[hadrielk]

You can "fix" this yourself by using a test event listener, as described in the advanced user guide, by doing something like this in your derived test event listener class (which you would register in your main()):

class TestEventListener : public ::testing::EmptyTestEventListener
{
    // Fired before the test case starts and before SetUpTestCase() is invoked.
    virtual void OnTestCaseStart(const ::testing::TestCase& test_case) override
    {
        firstTestInTestCase = true;
        testCaseHadFailure = false;
    }

    // Fired before each individual test starts: before the test fixture is constructed
    // and SetUp() is invoked.
    virtual void OnTestStart(const ::testing::TestInfo& info) override
    {
        if (firstTestInTestCase) {
            size_t numFailures = numFatalFailures();

            if (numFailures > previousNumFatalFailures) {
                testCaseHadFailure = true;
                previousNumFatalFailures = numFailures;
            }

            firstTestInTestCase = false;
        }

        if (testCaseHadFailure) {
            // this will avoid running the test fixture's SetUp() and
            // TearDown() and TEST()/TEST_F(), but the test fixtures
            // constructor is still invoked (that's unavoidable)
            FAIL() << "The test case's SetUpTestCase() or executable had a fatal failure";
        }
    }

    // gets the number of fatal failures in the UnitTest's ad_hoc_test_result,
    // which is the number of fatal failures other than those inside of test
    // cases; fatal failures in SetUpTestCase() also count in this, but those
    // in SetUp(), TearDown(), and TEST()/TEST_F() do not.
    size_t numFatalFailures()
    {
        const auto& result = ::testing::UnitTest::GetInstance()->ad_hoc_test_result();
        const int numParts = result.total_part_count();
        size_t failures = 0;

        for (int i=0; i < numParts; ++i) {
            if (result.GetTestPartResult(i).fatally_failed()) {
                ++failures;
            }
        }

        return failures;
    }

    bool   firstTestInTestCase{true};
    bool   testCaseHadFailure{false};
    size_t previousNumFatalFailures{0};
};

And then register the event listener in your main(), with something like this:

    // Gets hold of the event listener list
    ::testing::TestEventListeners& listeners = ::testing::UnitTest::GetInstance()->listeners();
    // Adds a listener to the end.  Google Test takes the ownership.
    listeners.Append(new test::TestEventListener);

[behcetm]

Is there any progress on this issue? These seems like a major issue to me and should be prioritized.

[daaren-urthling]

I also have this issue, partially fixed with the suggested workaround.
My problem now is that if I re-execute immediately the tests after a fail in the SetUpTestCase (i.e., to debug), none of them run, but however the general status of the test is "FAILED" (see below).
Any workaround to this?
EDIT: I re-execute the tests without terminating the .exe, as I have an (optional) UI that helps developers to select the test to be executed. The issue won't happen if the fail occurs in a TEST_F

[==========] Running 3 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] Global test environment tear-down
[==========] 3 tests from 2 test cases ran. (4 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 0 tests, listed below:

0 FAILED TESTS

[dsciebu]

Guys! Any progress o this topic?? It's been hanging here since 2015...

[LazyWolfLin]

It's been hanging here since 2010...

I face this issue with gtest1.8, and when it will fix? I don't think test event listener is a good way to fix it by myself.

[ihsandemir]

Hello, any progress in this issue? I have the same problem.

[t-moe]

Here as well. Please take a look!

[gennadiycivil]

Done, see https://github.com/google/googletest/commit/9ed99c6c837ae1cbfcabd36959fc802ebb5ae07f

[hbchai]

@gennadiycivil, can you clarify whether that change is expected to prevent test bodies from running (be skipped) when SetUpTestSuite() assertions fails? That is the crux of the current issue, so I'd like to understand whether it is actually fixed.

[masterskipper]

wow , 10 years.