Specflow: ScenarioExecutionStatus displays as OK if test failed in before scenario

[liamharries]

In my hooks I have a method that takes a screenshot if the test fails.

it works if the test fails in a test step however if the test errors in the before scenario stage then the ScenarioExecutionStatus reports OK so I end up missing tests that error.

SpecFlow Version:

• [x] 3.0
• [ ] 2.4
• [ ] 2.3
• [ ] 2.2
• [ ] 2.1
• [ ] 2.0
• [ ] 1.9

Used Test Runner

• [x] SpecFlow+Runner
• [ ] MSTest
• [ ] NUnit
• [ ] Xunit

Version number:

Project Format of the SpecFlow project

• [ ] Classic project format using packages.config
• [ ] Classic project format using <PackageReference> tags
• [ ] Sdk-style project format

.feature.cs files are generated using

• [x] SpecFlow.Tools.MsBuild.Generation NuGet package
• [ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

• [ ] VS 2019
• [x] VS 2017
• [ ] VS 2015

Enable SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings

• [ ] Enabled
• [x] Disabled

Are the latest Visual Studio updates installed?

• [x] Yes
• [ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

• [x] >= .NET 4.5
• [ ] before .NET 4.5
• [ ] .NET Core 2.0
• [ ] .NET Core 2.1
• [ ] .NET Core 2.2
• [ ] .NET Core 3.0

Test Execution Method:

• [x] Visual Studio Test Explorer
• [ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
• [ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

[liamharries]

Hi @SabotageAndi any update on this one?

[SabotageAndi]

No. I had no time to look at it yet.
But we are working in the near area of the code currently. Perhaps I see something.

[liamharries]

Hi @SabotageAndi, is there any update on this?

[SabotageAndi]

I had now a quick look at it.
No idea why the status is not on TestError.

The code for it is here: https://github.com/techtalk/SpecFlow/blob/master/TechTalk.SpecFlow/Infrastructure/TestExecutionEngine.cs#L372

Could you send a PR with a failing scenario?

[liamharries]

Hi @SabotageAndi, sorry not sure how I could create a failing scenario that would then pass if it failed as expected.

Below is a better explanation of my issue.

Essentially if I have a setup step that fails like below

[BeforeScenario(Order = 20)]
public void SetUp()
{
    throw new Exception("Fail before scenario");
}

Then this hook i have fails to pick up the error as the ScenarioExecutionStatus is OK.
It works as expected if the test fails on a scenario step but not in the hooks.

        [AfterScenario(Order = 1)]
        public void ScreenshotError()
        {
            if (ScenarioContext.ScenarioExecutionStatus == ScenarioExecutionStatus.OK||
                ScenarioContext.ScenarioExecutionStatus == ScenarioExecutionStatus.StepDefinitionPending) return;
            DocumentFailedTest();
        }

[liamharries]

Hi @SabotageAndi, is there any update on this?

[liamharries]

Hi @SabotageAndi, is this on your backlog to look at?

[SabotageAndi]

It is not planned for the next 2 sprints (=4 weeks).

[liamharries]

@SabotageAndi it's taken me a while but I finally managed to get my head around how the tests in the project work. I've got a local branch with a test to demo this but I'm getting a 403 when I try to push my branch. is there some permissions that need to be set or is there a different process?

[SabotageAndi]

@liamharries Where do you want to push your branch? What is your remote? You can't push into the specflow repository. You have to push it to your fork of the project.

[liamharries]

@SabotageAndi I've added a pull request for a test to demo the issue.

One thing I've noticed with the project is I get loads of errors trying to build, dll's kept being locked by another process. Is this something you've seen before? I'm using visual studio 2019 with the latest updates installed.

[SabotageAndi]

@liamharries probably you need to set this environment variable: https://github.com/techtalk/SpecFlow/blob/master/docs/development/LocalSetup.md#set-environment-variables

[liamharries]

Ace got it working now 😃

[liamharries]

@SabotageAndi I've gone through this as much as I can and unfortunately can't figure out the issue, probably because of my lack of familiarity with the project. Could you put this on your backlog to look at.

[joelransom]

I know this is an inactive issue, I just wanted to comment because while I don't usually have a problem with ScenarioExecutionStatus, I just ran into this exact issue.

Is this happening because the error is thrown in BeforeScenario? It makes sense in a way, but it would be nice if the status were set to TestError.

Is there any mechanism to set ScenarioExecutionStatus?

[liamharries]

@joelransom yes if the test fails for any reason in hooks before the scenario is started then the status is not set. not sure if there is a suitable workaround for this, we did try handling errors in hooks then setting a flag that is picked up by the code inspecting the status but became too messy.

[VitaliiDolotov]

Hi guys, hope you are doing great, do you have any updates on this or plans to fix this bug? ScenarioExecutionStatus is still OK in case of any exceptions in BeforeScenario (SpecFlow v3.4.31)

[SabotageAndi]

@VitaliiDolotov SpecFlow is an Open Source Project and lives from the contributions of its community and users. We are only a small team with limited time, so we can't do everything on our own.

I am always happy to help new contributors with their Pull Requests to SpecFlow.

In this case, a good starting point would be to add a new scenario to our test suite (https://github.com/SpecFlowOSS/SpecFlow/tree/master/Tests/TechTalk.SpecFlow.Specs/Features/Execution) that fails.

[liamharries]

Hi @SabotageAndi, already have a PR with a failing test linked above #1781

[AndyLPK247]

Hi everyone! I just hit this problem myself today. What's the latest update? Is there anything I can do to help get a fix in place?

[SabotageAndi]

@AndyLPK247 We are working on it. @nemesv is currently putting the last changes to the fix.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.