SpecFlow 3 tests don't success when executed (works when debugging)

[temelj]

SpecFlow Version:

• [x] 3.1.76
• [x] 3.1.67
• [ ] 3.0
• [ ] 2.4
• [ ] 2.3
• [ ] 2.2
• [ ] 2.1
• [ ] 2.0
• [ ] 1.9
  Checked with both versions

Used Test Runner

• [ ] SpecFlow+Runner
• [ ] MSTest
• [x] NUnit
• [ ] Xunit

Version number:

Project Format of the SpecFlow project

• [x] Classic project format using packages.config
• [ ] Classic project format using <PackageReference> tags
• [ ] Sdk-style project format

.feature.cs files are generated using

• [x] SpecFlow.Tools.MsBuild.Generation NuGet package
• [ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

• [x] VS 2019
• [ ] VS 2017
• [ ] VS 2015

Enable SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings

• [ ] Enabled
• [X] Disabled

Are the latest Visual Studio updates installed?

• [x] Yes
• [ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

• [x] >= .NET 4.5
• [ ] before .NET 4.5
• [ ] .NET Core 2.0
• [ ] .NET Core 2.1
• [ ] .NET Core 2.2
• [ ] .NET Core 3.0

Test Execution Method:

• [x] Visual Studio Test Explorer
• [ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
• [ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

<SpecFlow> Section in app.config or content of specflow.json

Repro Project

None

Issue Description

This issue only started after migration to specflow 3.1.
With multiple Features, when running the tests in TestExplorer, the first set of steps pass successfully under the feature, however when the next feature starts it fails (and for all subsequent features).

The strange part is if I run in debug (without any breakpoints anywhere) the tests all execute successfully.

Steps to Reproduce

My issue could very well be an incompatible package that is being loaded.

packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Autofac" version="4.8.1" targetFramework="net471" />
  <package id="BoDi" version="1.4.1" targetFramework="net471" />
  <package id="BouncyCastle.Crypto.dll" version="1.8.1" targetFramework="net471" />
  <package id="Castle.Core" version="4.2.1" targetFramework="net471" />
  <package id="ChannelAdam.Core" version="1.6.0" targetFramework="net471" />
  <package id="ChannelAdam.TestFramework" version="1.7.1" targetFramework="net471" />
  <package id="Cucumber.Messages" version="6.0.1" targetFramework="net471" />
  <package id="DiffPlex" version="1.4.1" targetFramework="net471" />
  <package id="DnsClient" version="1.0.7" targetFramework="net471" />
  <package id="ExtentReports" version="4.0.3" targetFramework="net471" />
  <package id="Gherkin" version="6.0.0" targetFramework="net471" />
  <package id="Google.Protobuf" version="3.7.0" targetFramework="net471" />
  <package id="Microsoft.AspNet.Razor" version="3.0.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Configuration" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Configuration.Abstractions" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Configuration.Binder" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Configuration.FileExtensions" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Configuration.Json" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.DependencyInjection" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.DependencyInjection.Abstractions" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.FileProviders.Abstractions" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.FileProviders.Composite" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.FileProviders.Embedded" version="2.0.1" targetFramework="net471" />
  <package id="Microsoft.Extensions.FileProviders.Physical" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.FileSystemGlobbing" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Logging" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Logging.Abstractions" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Options" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.Extensions.Primitives" version="2.2.0" targetFramework="net471" />
  <package id="Microsoft.NETCore.Platforms" version="2.0.1" targetFramework="net471" />
  <package id="Microsoft.VisualStudio.SlowCheetah" version="3.2.20" targetFramework="net471" developmentDependency="true" />
  <package id="MongoDB.Bson" version="2.7.2" targetFramework="net471" />
  <package id="MongoDB.Driver" version="2.7.2" targetFramework="net471" />
  <package id="MongoDB.Driver.Core" version="2.7.2" targetFramework="net471" />
  <package id="Moq" version="4.7.145" targetFramework="net471" />
  <package id="NETStandard.Library" version="2.0.3" targetFramework="net471" />
  <package id="Newtonsoft.Json" version="12.0.1" targetFramework="net471" />
  <package id="NUnit" version="3.12.0" targetFramework="net471" />
  <package id="NUnit3TestAdapter" version="3.15.1" targetFramework="net471" />
  <package id="RazorEngine" version="3.10.0" targetFramework="net471" />
  <package id="RestSharp" version="106.6.7" targetFramework="net471" />
  <package id="ServicePlatform.Library.Apis" version="2.0.59" targetFramework="net471" />
  <package id="ServicePlatform.Library.ErrorHandling" version="2.0.39" targetFramework="net471" />
  <package id="ServicePlatform.Library.PersonalisedConsumerMessages" version="2.0.20" targetFramework="net471" />
  <package id="ServicePlatform.Library.Testing" version="2.0.18" targetFramework="net471" />
  <package id="ServicePlatform.Library.Utilities.Crypto" version="1.0.4" targetFramework="net471" />
  <package id="SimpleImpersonation" version="3.0.0" targetFramework="net471" />
  <package id="SpecFlow" version="3.1.67" targetFramework="net471" />
  <package id="SpecFlow.NUnit" version="3.1.67" targetFramework="net471" />
  <package id="SpecFlow.Tools.MsBuild.Generation" version="3.1.67" targetFramework="net471" />
  <package id="System.Buffers" version="4.4.0" targetFramework="net471" />
  <package id="System.ComponentModel.Annotations" version="4.5.0" targetFramework="net471" />
  <package id="System.IO" version="4.3.0" targetFramework="net471" />
  <package id="System.Memory" version="4.5.1" targetFramework="net471" />
  <package id="System.Net.Http" version="4.3.4" targetFramework="net471" />
  <package id="System.Numerics.Vectors" version="4.4.0" targetFramework="net471" />
  <package id="System.Reflection.Emit" version="4.3.0" targetFramework="net471" />
  <package id="System.Reflection.Emit.Lightweight" version="4.3.0" targetFramework="net471" />
  <package id="System.Runtime" version="4.3.0" targetFramework="net471" />
  <package id="System.Runtime.CompilerServices.Unsafe" version="4.5.1" targetFramework="net471" />
  <package id="System.Runtime.InteropServices.RuntimeInformation" version="4.0.0" targetFramework="net471" />
  <package id="System.Security.Cryptography.Algorithms" version="4.3.0" targetFramework="net471" />
  <package id="System.Security.Cryptography.Encoding" version="4.3.0" targetFramework="net471" />
  <package id="System.Security.Cryptography.Pkcs" version="4.5.2" targetFramework="net471" />
  <package id="System.Security.Cryptography.Primitives" version="4.3.0" targetFramework="net471" />
  <package id="System.Security.Cryptography.X509Certificates" version="4.3.0" targetFramework="net471" />
  <package id="System.Security.Principal.Windows" version="4.5.0" targetFramework="net471" />
  <package id="System.ServiceModel.Primitives" version="4.5.3" targetFramework="net471" />
  <package id="System.Threading.Tasks.Extensions" version="4.4.0" targetFramework="net471" />
  <package id="System.ValueTuple" version="4.4.0" targetFramework="net471" />
  <package id="Utf8Json" version="1.3.7" targetFramework="net471" />
  <package id="XMLUnit.Core" version="2.5.1" targetFramework="net471" />
</packages>

Exception Message

Test Name:  ErrorCodesMVP0001TestingAgainstUserStoryBP1331AcceptanceCriteria1GatewayReturnsErrorWhenARequestIsSentWithInvalidCredentials
Test FullName:  MyProject.Testing.MyProject.Testing.Features.ErrorCodesFeature.ErrorCodesMVP0001TestingAgainstUserStoryBP1331AcceptanceCriteria1GatewayReturnsErrorWhenARequestIsSentWithInvalidCredentials
Test Source:     : line -1
Test Outcome:   Failed
Test Duration:  0:00:00

Test Name:  ErrorCodesMVP0001TestingAgainstUserStoryBP1331AcceptanceCriteria1GatewayReturnsErrorWhenARequestIsSentWithInvalidCredentials
Test Outcome:   Failed
Result StackTrace:  
at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\ContextManager.cs:line 183
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioInitialize(ScenarioInfo scenarioInfo) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 192
   at TechTalk.SpecFlow.TestRunner.OnScenarioInitialize(ScenarioInfo scenarioInfo) in D:\a\1\s\TechTalk.SpecFlow\TestRunner.cs:line 49
   at MyProject.Testing.Features.ErrorCodesFeature.ScenarioInitialize(ScenarioInfo scenarioInfo)
   at MyProject.Testing.Features.ErrorCodesFeature.ErrorCodesMVP0001TestingAgainstUserStoryBP1331AcceptanceCriteria1GatewayReturnsErrorWhenARequestIsSentWithInvalidCredentials() in D:\repo\MyProject.Testing\MyProjectTesting\Features\ErrorCodes.feature:line 4
--TearDown
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd() in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 261
   at TechTalk.SpecFlow.TestRunner.OnScenarioEnd() in D:\a\1\s\TechTalk.SpecFlow\TestRunner.cs:line 64
   at MyProject.Testing.Features.ErrorCodesFeature.TestTearDown()
Result Message: 
System.NullReferenceException : Object reference not set to an instance of an object.
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.

Disclaimer

Please note that this an older specflow implementation specific to our use case. However I've cut down the whole project to two features, Feature1 and Feature2. To rule out any other strange bits such as incorrectly implemented attributes or anything. I have removed all references to other tests and any hooks. This is basically back to two barebone feature files.

[SabotageAndi]

Could you create a small example project where this issue is reproducible?

[temelj]

I've created a test project where i am still observing the same error/behavior
https://github.com/temelj/specflow3_test

[temelj]

I've stripped it down to bare minimum. I still think it is a nuget package related issues but I just can't pinpoint what it is.

[SabotageAndi]

Thanks for the project. I can reproduce it. I will have a look on it tomorrow or Wednesday.

[temelj]

any luck with this one?

[SabotageAndi]

Sorry no. Other stuff got into my way.

[temelj]

Hi Andi,
have you got any further on this issue? We're kind of stuck with migrating to VS2019 as this is a blocker for us. Thanks in advance.

[temelj]

No traction in this space?

This issue is still outstanding even when pulling the latest packages.

Would anyone be able to suggest/provide a workaround? Possibly using a different test runner maybe?

[SabotageAndi]

Sorry, this (https://specflow.org/2020/tricentis-acquires-specflow/) got in the way and occupied us.
We are looking at this issue in the next sprint, that starts tomorrow.

[SabotageAndi]

@temelj I had a look and I found the issue.
You had set the ParallelScope of NUnit to Children. This doesn't work with SpecFlow. Details are in https://github.com/SpecFlowOSS/SpecFlow/issues/1535

After I changed it to Fixture the error was gone.

[temelj]

Thank you so much! Must have been an inherited configuration in the AssemblyInfo from v2.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.