Catch2: GENERATE + conditional DYNAMIC_SECTION causes infinite loop

[Aivean]

Describe the bug

My code uses generated bool to conditionally create a dynamic section. When I try run test with filter by this generated section, catch2 seems to run test in a loop indefinitely.

Expected behavior

I'm not sure what is the correct behavior here, but I doubt that infinite loop is expected.
If this is an illegal usage of generators and sections, perhaps it should be explicitly documented and/or an error should be generated.
Ideally I'd expect it to be able to filter by this dynamic generated section.

Reproduction steps
Steps to reproduce the bug.

Test case:

TEST_CASE( "generate_cond_section" )
{
    bool fov = GENERATE( false, true );
    DYNAMIC_SECTION( "fov_" << fov ) {
    }
}

When run without filter by section:

$ tests/test  "generate_cond_section"
Starting the actual test at Tue Sep 15 17:59:40 2020
Filters: generate_cond_section
0.000 s: fov_0
0.000 s: generate_cond_section
0.000 s: fov_1
0.000 s: generate_cond_section
===============================================================================
test cases: 1 | 1 passed
assertions: - none -

When run with filter by section:

$ tests/test  "generate_cond_section" -c fov_0
Starting the actual test at Tue Sep 15 18:00:55 2020
Filters: generate_cond_section
0.000 s: fov_0
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
0.000 s: generate_cond_section
....
^C

Platform information:

• OS: MacOS 10.14
• Compiler+version: Apple LLVM version 10.0.0 (clang-1000.11.45.5)
• Catch version: v2.13.0

Additional context

I'm exploring my options to somehow "name" the values generated using the GENERATE, so I can run test for the specific combination of generated values. When I tried to use DYNAMIC_SECTION (or just SECTION) for that, I've encountered this bug.

[horenmar]

Definitely a bug, I will have to see how hard it turns out to fix.

Out of curiosity, does this also reproduce with versions before v2.12.3? (So e.g. 2.12.2)

[jslap]

I have the same kind of issue, but I do not need dynamic section.
When run with -c "Check cursor from buffer offset." -c "When: Buffer position is 8"
I have an infinite loop on 2.13.
And it works fine on 2.12.2

#define CATCH_CONFIG_MAIN  // This tells Catch to provide a main() - only do this in one cpp file
#include "catch.hpp"

#include <iostream>



TEST_CASE( "Empty App", "[app]" ) {
    SECTION("Check cursor from buffer offset.")
    {

        int bigFileLength = 44;
        auto bufPos = GENERATE_REF(range(0, bigFileLength));
        {
            std::cout << "YOYO : " << bufPos << "\n";
            WHEN("Buffer position is " << bufPos)
            {
                REQUIRE(1 == 1);
            }
        }

    }

}

[horenmar]

@jslap Thanks, the fact that 2.13 is what broke it is very useful information.

[horenmar]

Okay, found the bug. It is about interaction of generators and section filters -- it boils down to the generator tracker seeing an unvisited section underneath it, and decides that it needs to run again, so that it can be visited. However, it is not aware of section filters, so it doesn't know that the section can never be visited.

[dadocolussi]

I got curious about this issue, and I came up with a fix:

void SectionTracker::tryOpen() {
    if( !isComplete() )
        open();
    else if ( m_runState == NotStarted )
        m_runState = CompletedSuccessfully;
}

Any attempt to open a section that leaves it in NotStarted state would immediately complete the section, which has the effect of letting the generator proceed. I haven't found a scenario where this would produce unexpected results.

All tests pass, but I'm nevertheless quite unsure whether this is an appropriate fix. Just my two cents.

[horenmar]

I will keep that possibility in mind, but I was bitten by these quick fixes to tracking before, so I would rather make a proper one.

Also I have it working locally, I just need to clean up the code and make an automatic test.