Specflow: 1 Random failure counted as multiple failures in specrun 3 report

[liamharries]

SpecFlow Version:

• [x] 3.0
• [ ] 2.4
• [ ] 2.3
• [ ] 2.2
• [ ] 2.1
• [ ] 2.0
• [ ] 1.9

Used Test Runner

• [x] SpecFlow+Runner
• [ ] MSTest
• [ ] NUnit
• [ ] Xunit

Version number:

Visual Studio Version

• [ ] VS 2019
• [x] VS 2017
• [ ] VS 2015
• [ ] VS 2013

Are the latest Visual Studio updates installed?

• [x] Yes
• [ ] No

.NET Framework:

• [x] >= .NET 4.5
• [ ] before .NET 4.5
• [ ] .NET Core 2.0
• [ ] .NET Core 2.1
• [ ] .NET Core 2.2
• [ ] .NET Core 3.0

Test Execution Method:

• [x] Visual Studio Test Explorer
• [x] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
• [ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

Issue Description

When a test randomly fails the report picks it up and reports it as a random test failure however it the randomly failing test fails more than once then the number of failures reported is the number of iteration failures. the report of the failed tests should not include retries

Steps to Reproduce

run 1 test with a retry count of 2
force the first run to fail
force the second run to fail
allow the third to pass
observe the generated report displays "Result: 2 failed (1 randomly failed)"

[liamharries]

@SabotageAndi any update on this? Our tests are currently wrongly reporting failures.

[SabotageAndi]

No news yet. We have this issue in our current sprint.

[david1995]

Hi @liamharries,

I have written a small repro project:
Feature file

Feature: Feature1

Scenario: Add two numbers
    Given I fail randomly

Binding

[Binding]
public class Class1
{
    public static int Count { get; set; }

    [Given("I fail randomly")]
    public void Step()
    {
        Count++;
        if (Count < 3)
        {
            throw new Exception();
        }
    }
}

Relevant line in srprofile:

<Execution testThreadCount="1" testSchedulingMode="Sequential" retryCount="2" retryFor="Failing" />

When I generate reports, it produces the header you mentioned:

I have updated our logic. If i am understanding your issue correctly, this is what should be displayed:

[liamharries]

@david1995 yes that is correct and matches the pre 3.0 behaviour :)

[david1995]

Perfect, the fix for this issue will be included in the next release of SpecFlow+ Runner.

[liamharries]

Awesome thanks @david1995

[david1995]

Today we released SpecFlow+ Runner version 3.0.329 which fixes this bug in the report generation. Therefore I'll close this issue.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.