Specflow: FeatureContext throws a NullReferenceException

[richerm]

We have an issue using SpecFlow 2.1.0.0 when attempting to run features in parallel using NUnit. The last feature files always fail as it seems there is an issue during tear down and we receive a NullReferenceException with the following stack trace:

System.NullReferenceException : Object reference not set to an instance of an object.
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.GetStepMatch(StepInstance stepInstance)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
   at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
   at SoapApiIntegrationTests.Scenarios.MoveAssetFeature.ScenarioCleanup()
   at SoapApiIntegrationTests.Scenarios.MoveAssetFeature.TP_8037_WhenIMoveAnAssetIExpectAllPreviousVersionsMovedAsWell() in D:\VS Projects\V4.Development\SoapApiIntegrationTests\Scenarios\MoveAsset.feature:line 15

Pulling the source code and PDB's for 2.1.0.0 we find the error is the FeatureContext on the contextManager is null in the middle of a test.

            List<BindingMatch> candidatingMatches;
            StepDefinitionAmbiguityReason ambiguityReason;
            var match = stepDefinitionMatchService.GetBestMatch(stepInstance, contextManager.FeatureContext.BindingCulture, out ambiguityReason, out candidatingMatches);

            if (match.Success)
                return match;

As a test we changed the offending line to use CultureInfo.CurrentCulture instead an all our tests pass. Can you offer some insight as to why FeatureContext would be null mid-test when running in parallel? (Note we are using NUnitConsoleRunner and have the appropriate NUnit assembly properties to enable parallelization)

[LeonHagenaarsKeus]

Same problem here, but with both

        [AfterScenario]
        public void AfterScenario()
        {
            Console.WriteLine($@"{DateTime.Now.TimeOfDay} Ending scenario {_scenarioContext?.ScenarioInfo?.Title}");
            Shutdown(TimeSpan.FromSeconds(1));
            Console.WriteLine($@"{DateTime.Now.TimeOfDay} Scenario {_scenarioContext?.ScenarioInfo?.Title} ended");
        }

and

        [AfterScenario(UseDatabase)]
        public void AfterScenarioDatabase()
        {
            if(_scenarioContext == null) return;
            var db = _scenarioContext.Get<TestDatabase>();
            db.Dispose();
        }

It starts failing after 200+ scenario's, and then fails for all the following scenario's with the same exception:

Error : X.Y.Tests.Scenarios.SomeFeature._006SomeScenario
System.NullReferenceException : Object reference not set to an instance of an object.
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.GetStepMatch(StepInstance stepInstance)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
   at X.Y.Tests.Scenarios.SomeFeature.ScenarioCleanup()
   at X.Y.Tests.Scenarios.SomeFeature._006SomeScenario in C:\X\Y\X.Y.Tests\Scenarios\Some.feature:line 116

[SabotageAndi]

Could you try the latest pre-release of SpecFlow 2.2? There are some internal changes for the FeatureContext made. Perhaps they solve this problem.
If not, please provide us with a project to reproduce this error.

[LeonHagenaarsKeus]

Upgrading to specflow 2.2 is not helping.
Removing the 'AfterScenario' blocks is not helping.
Is there a way to get more logging out of Specflow?
Because of NDA reasons I cannot provide the actual failing project.

[richerm]

I did see this in my console output:

-> #9: done: BaseSteps.ThenTheCallShouldFail("INVALID_PARAMETER") (0.0s)
-> #10: warning: The previous FeatureContext was not disposed.

I'll check into the source code to see where/when that gets logged. I'm not even using FeatureContext anywhere explicitly and I commented out/removed all Before/After hooks.

[LeonHagenaarsKeus]

Some more details: this worked with specflow 2.1 and NUnit 3.6.1, it now doesn't work with nunit 3.7.1

[richerm]

I can confirm as well, downgrading NUnit to 3.6.1 and my errors go away. Still can't create reproducible project that I can share though, as I'm not sure what the triggering conditions are.

[SabotageAndi]

From the Release notes of NUnit 3.7 I see that they changed something with the parallel execution.
As the downgrade to 3.6.1 works for you, it is probably something in NUnit.
The question is, does SpecFlow something wrong and we notice it now, or is there a bug in NUnit.

As long as we don't have a working reproduction for this issue, I wouldn't ask the NUnit guys.

[SabotageAndi]

@richerm You get the The previous FeatureContext was not disposed. messages normally, when there is an exception in the After steps and so SpecFlow can not clean up everything.

[richerm]

@SabotageAndi
Reproducible Project! -- https://github.com/richerm/SpecFlowNUnitParallelism

"nunit3-console.exe" "D:\VS Projects\NUnitSpecFlowTest\NUnitSpecFlowTest\bin\Debug\NUnitSpecFlowTest.dll" --result=D:\TEMP\Output.nunit.xml --noheader --x86 --framework=net-4.0 --workers 5

Result:

1) Error : NUnitSpecFlowTest.FixtureJFeature.Take1Seconds
System.NullReferenceException : Object reference not set to an instance of an object.
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.GetStepMatch(StepInstance stepInstance)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance)
   at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
   at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
   at NUnitSpecFlowTest.FixtureJFeature.ScenarioCleanup()
   at NUnitSpecFlowTest.FixtureJFeature.Take1Seconds() in D:\VS Projects\NUnitSpecFlowTest\NUnitSpecFlowTest\FixtureJ.feature:line 5

Run Settings
    RuntimeFramework: net-4.0
    RunAsX86: True
    DisposeRunners: True
    WorkDirectory: D:\TEMP
    NumberOfTestWorkers: 5
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False

Test Run Summary
  Overall result: Failed
  Test Count: 22, Passed: 21, Failed: 1, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 0, Errors: 1, Invalid: 0
  Start time: 2017-06-13 12:06:34Z
    End time: 2017-06-13 12:06:44Z
    Duration: 10.053 seconds

[raveeolee]

Just want to confirm that I have similar issue with FeatureContext + specflow 2.1 + nunit 3.7.1:

BoDi.ObjectContainerException : Multiple public constructors with same maximum parameter count are not supported! System.Globalization.CultureInfo (resolution path: Hooks->TechTalk.SpecFlow.FeatureContext->TechTalk.SpecFlow.FeatureInfo)

(One of my Hooks file has a DI field of FeatureContext).
Disappeared after downgrading to the nunit 3.6.1.

[nazarenomanco]

On our CI (Visual Studio 2015 installed) we had to fall back to SpecFlow 2.1 and NUnit 2.6.4 to avoid the NullReferenceException

[SabotageAndi]

NUnit 3.8 was released last week. Could anybody try this and see if the issue is solved with it?

[richerm]

NUnit 3.8.1 resolve the issues for me (also updated to latest SpecFlow but likely unrelated). This is both for my sample test project above as well as my actual production project I'm working on.

For my part in this issue it can be closed as resolved.

[marwahaks]

@richerm Hey could you telll me how did you got past the following error when updating to nunit 3.8.1, "The type or namespace name 'TestFixtureSetUpAttribute' does not exist in the namespace 'NUnit.Framework' (are you missing an assembly reference?)",

[richerm]

@marwahaks You likely just need to re-generate your SpecFlow classes (and potential update SpecFlow) as I believe TextFixtureSetUpAttribute was removed and OneTimeSetUpAttribute added and your SpecFlow might be generating the old style NUnit code.

[marwahaks]

@richerm Cheers, I'll have a go at that. Thanks

[naeemakram]

Sometimes Specflow generator starts generating bad code all of a sudden for no good reason. Has this been fixed in later versions? I'm using Specflow 2.1.0.

[SabotageAndi]

@naeemakram Please open a separate bug.
Your comment has nothing to do with this issue. And please update to the latest version of SpecFlow. 2.1. is 2 1/2 years old.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.