Specflow: Wrong scenario context injected when running tests in parallel using NUnit (also specrun)

[adzpond]

Wrong scenario context injected when running tests in parallel using NUnit (also specrun)

SpecFlow Version:

• [x] 2.3
• [ ] 2.2
• [ ] 2.1
• [ ] 2.0
• [ ] 1.9

Used Test Runner

• [ ] SpecFlow+Runner
• [ ] MSTest
• [x] NUnit
• [ ] Xunit

Version number:
2.3.0

Visual Studio Version

• [x] VS 2017
• [ ] VS 2015
• [ ] VS 2013

Are the latest Visual Studio updates installed?

• [x] Yes
• [ ] No

.NET Framework:

• [x] >= .NET 4
• [ ] before .NET 4

Test Execution Method:

• [x] Visual Studio Test Explorer
• [ ] TFS/VSTS – Task – PLEASE SPECIFY THE NAME OF THE TASK
• [ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE

<SpecFlow> Section in app.config

  <specFlow>
    <generator allowDebugGeneratedFiles="true" />
    <!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->
    <!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->
    <unitTestProvider name="NUnit" />
  </specFlow>

Repro Project

AssemblyInfo.cs
[assembly: Parallelizable(ParallelScope.All)]
[assembly: LevelOfParallelism(2)]

Steps

Feature: Parallel Test
In order to know my illustrate a bug
As a user of specflow
I need to execute these scenarios

Scenario Outline: The first scenario example
Given I'm illustrating the issue
And Using this value
And And then this value
And I do something else
Then I will get this value
Examples:
| Title | Value1 | Value2 | Value3 |
| A | 4 | 5000 | 8 |
| B | 4 | 15049 | 8 |
| C | 4 | 15050 | 9 |
| D | 4 | 15051 | 9 |
| E | 4 | 24999 | 9 |
| F | 4 | 25000 | 9 |
| G | 5 | 5000 | 8 |
| H | 5 | 15049 | 8 |
| I | 5 | 15050 | 9 |
| J | 5 | 15051 | 9 |
| K | 5 | 24999 | 9 |
| L | 5 | 25000 | 9 |
| M | 6 | 5000 | 6 |
| N | 6 | 15049 | 6 |
| O | 6 | 15050 | 7 |
| P | 6 | 15051 | 7 |
| Q | 6 | 24999 | 7 |
| T | 6 | 25000 | 7 |
| S | 6 | 25000 | 7 |

Scenario Outline: The second scenario example
Given I'm illustrating the issue
And Using this value
And And then this value
And I do something else
Then I wont get a result
Examples:
| Title | Value1 | Value2 |
| T | 4 | 4999 |
| U | 4 | 25001 |
| V | 5 | 4999 |
| W | 5 | 25001 |
| X | 6 | 4999 |
| Y | 6 | 25001 |

Class

using System;
using System.Threading;
using NUnit.Framework;
using TechTalk.SpecFlow;

[Binding]
public class LosingTheWillToLiveSteps
{
    private readonly ScenarioContext _scenarioContext;

    public LosingTheWillToLiveSteps(ScenarioContext scenarioContext)
    {
        _scenarioContext = scenarioContext;
    }

    [BeforeScenario]
    public void BeforeScenario()
    {
        Console.Out.WriteLine("BeforeScenario");
        try
        {
            var id = Guid.NewGuid().ToString();
            _scenarioContext.Add("ID", id);
            WriteId("Before Scenario");
        }
        catch (Exception e)
        {
            Console.Out.WriteLine($"Error adding id {e.Message}");
        }
    }

    [AfterScenario]
    public void AfterScenario()
    {
        try
        {
            WriteId("After Scenario");
        }
        catch (Exception e)
        {
            Console.Out.WriteLine($"Error clearing up the Scenario {e.Message}");
        }
    }

    [Given(@"I'm illustrating the issue")]
    public void GivenImIllustratingTheIssue()
    {
        WriteId("I'm illustrating the issue");
    }

    [Given(@"Using this value (.*)")]
    public void UsingThisValue(string value)
    {
        WriteId("Using this value");
    }

    [Given(@"And then this value (.*)")]
    public void AndThenThisValue(string value)
    {
        WriteId("And then this value (.*)");
    }


    [Given(@"I do something else")]
    public void IDoSomethingElse()
    {
        WriteId("I do something else ");
    }

    [Then(@"I will get this value (.*)")]
    public void IWillGetThisValue(string value)
    {
        WriteId("I will get this value (.*)");
        Assert.IsTrue(true);
    }

    [Then(@"I wont get a result")]
    public void IWontGetAResult()
    {
        WriteId("I wont get a result");
        Assert.IsTrue(true);
    }

    private void WriteId(string caller)
    {
        try
        {
            Console.Out.WriteLine($"Context ID {_scenarioContext["ID"].ToString()} - {caller}");
            Thread.Sleep(1000);
        }
        catch (Exception e)
        {
            Console.Out.WriteLine($"Error- {caller} - {e.Message}");
        }
    }

}

Issue Description

When attempting to run the tests in parallel a number of tests fail with errors such as

Message: System.NullReferenceException : Object reference not set to an instance of an object.
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.

usually followed by
Message: System.NullReferenceException : Object reference not set to an instance of an object.

Message: System.ArgumentException : An item with the same key has already been added.
TearDown : System.ArgumentException : An item with the same key has already been added.

These errors occur outside the code of my steps and BeforeScenario/AfterScenario code

Steps to Reproduce

I have stripped all my code out of the steps to rule out any issues I may have added. I included some extra code to add a guid to the scenario context which I out put every step to make sure that it is the same and for the most part it is.

As in this example
BeforeScenario

Context ID ebd6e883-6924-4396-9130-c1571d748dae - Before Scenario
Given I'm illustrating the issue
Context ID ebd6e883-6924-4396-9130-c1571d748dae - I'm illustrating the issue
-> done: LosingTheWillToLiveSteps.GivenImIllustratingTheIssue() (1.0s)
And Using this value 15050
Context ID ebd6e883-6924-4396-9130-c1571d748dae - Using this value
-> done: LosingTheWillToLiveSteps.UsingThisValue("15050") (1.0s)
And And then this value 475
Context ID ebd6e883-6924-4396-9130-c1571d748dae - And then this value (.)
-> done: LosingTheWillToLiveSteps.AndThenThisValue("475") (1.0s)
And I do something else
Context ID ebd6e883-6924-4396-9130-c1571d748dae - I do something else
-> done: LosingTheWillToLiveSteps.IDoSomethingElse() (1.0s)
Then I will get this value 8.40
Context ID ebd6e883-6924-4396-9130-c1571d748dae - I will get this value (.)
-> done: LosingTheWillToLiveSteps.IWillGetThisValue("8.40") (1.0s)
Context ID ebd6e883-6924-4396-9130-c1571d748dae - After Scenario

You can see that the context id is the same through out as you would expect.

However, as soon as one of the tests fails because of errors the output from tests start to get strange like this

BeforeScenario
Context ID cf8fe6f6-b6a0-4a3a-bbaf-1835a468ad46 - Before Scenario
Given I'm illustrating the issue
Context ID cf8fe6f6-b6a0-4a3a-bbaf-1835a468ad46 - I'm illustrating the issue
-> done: LosingTheWillToLiveSteps.GivenImIllustratingTheIssue() (1.0s)
-> warning: The previous ScenarioStepContext was already disposed.
And Using this value 25000
Context ID 2fd808cd-24d5-48cb-86a7-c00668cb2b93 - Using this value
-> done: LosingTheWillToLiveSteps.UsingThisValue("25000") (1.0s)
And And then this value 475
Context ID 2fd808cd-24d5-48cb-86a7-c00668cb2b93 - And then this value (.)
-> done: LosingTheWillToLiveSteps.AndThenThisValue("475") (1.0s)
And I do something else
Context ID 2fd808cd-24d5-48cb-86a7-c00668cb2b93 - I do something else
-> done: LosingTheWillToLiveSteps.IDoSomethingElse() (1.0s)
Then I will get this value 8.40
Context ID 2fd808cd-24d5-48cb-86a7-c00668cb2b93 - I will get this value (.)
-> done: LosingTheWillToLiveSteps.IWillGetThisValue("8.40") (1.0s)
Context ID 2fd808cd-24d5-48cb-86a7-c00668cb2b93 - After Scenario

As you can see the guid changes on the third step, it's also preceded by a warning that the ScenarioStepContext was already disposed. This was output from a passing test.

This is the output from the first failing test. As you can see it ran all the way through.

BeforeScenario
Context ID 75075bde-4263-43cc-9a64-9e87494c7256 - Before Scenario
Given I'm illustrating the issue
Context ID 75075bde-4263-43cc-9a64-9e87494c7256 - I'm illustrating the issue
-> done: LosingTheWillToLiveSteps.GivenImIllustratingTheIssue() (1.0s)
And Using this value 25001
Context ID 75075bde-4263-43cc-9a64-9e87494c7256 - Using this value
-> done: LosingTheWillToLiveSteps.UsingThisValue("25001") (1.0s)
And And then this value 511
Context ID 75075bde-4263-43cc-9a64-9e87494c7256 - And then this value (.*)
-> done: LosingTheWillToLiveSteps.AndThenThisValue("511") (1.0s)
And I do something else
Context ID 75075bde-4263-43cc-9a64-9e87494c7256 - I do something else
-> done: LosingTheWillToLiveSteps.IDoSomethingElse() (1.0s)
Then I wont get a result
Context ID 75075bde-4263-43cc-9a64-9e87494c7256 - I wont get a result
-> done: LosingTheWillToLiveSteps.IWontGetAResult() (1.0s)

It also recorded this error

System.NullReferenceException : Object reference not set to an instance of an object.
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.

Can't remember the exact order of the next two failing tests but there was no output from one just this error.

Message: System.ArgumentException : An item with the same key has already been added.
TearDown : System.ArgumentException : An item with the same key has already been added.

The output for the other failed test was this

BeforeScenario
Context ID feb48879-b77b-46fb-8352-9486709874dd - Before Scenario
Given I'm illustrating the issue
Context ID feb48879-b77b-46fb-8352-9486709874dd - I'm illustrating the issue
-> done: LosingTheWillToLiveSteps.GivenImIllustratingTheIssue() (1.0s)
And Using this value 15051
Context ID feb48879-b77b-46fb-8352-9486709874dd - Using this value
-> done: LosingTheWillToLiveSteps.UsingThisValue("15051") (1.0s)
And And then this value 475
Context ID feb48879-b77b-46fb-8352-9486709874dd - And then this value (.)
-> done: LosingTheWillToLiveSteps.AndThenThisValue("475") (1.0s)
And I do something else
Context ID feb48879-b77b-46fb-8352-9486709874dd - I do something else
-> done: LosingTheWillToLiveSteps.IDoSomethingElse() (1.0s)
Then I will get this value 8.40
Context ID feb48879-b77b-46fb-8352-9486709874dd - I will get this value (.)
-> done: LosingTheWillToLiveSteps.IWillGetThisValue("8.40") (1.0s)
Context ID cf8fe6f6-b6a0-4a3a-bbaf-1835a468ad46 - After Scenario

As you can see it ran all the way through but still raised this error,

System.NullReferenceException : Object reference not set to an instance of an object.

It's not consistent either typically as I was putting the notes together for this bug I got three errors, I've had a lot more. The errors also occur with no code in the steps. The code I have added is just my attempt to make sense of the issues.

I would also like to add I first encountered this while using SpecRun I moved to NUnit to see if that would resolve the issue.

Resolving this issue would be really useful.

[SabotageAndi]

Thanks for the detailed description.
As you have a stripped down repro project, could you zip it and attach to the issue? So we don't have to create it and we have really all the same settings. Thanks!

[adzpond]

Hi,

Thanks for replying so quickly.

This is the file. Due to size limitiations I have cleared the packages folder and the bin folder. Hope that's of use.

ParallelTestingIssues.zip

[SabotageAndi]

I think you closed the issue accidentally.

[adzpond]

It's worth looking at the output from all the tests even ones that appear to have passed, as there seem to be issues with the scenario context with them.

[adzpond]

Hi,

is there any update on this please?

[gschultz77]

We've found this issue to be a blocker for running tests in parallel... so any progress would be greatly appreciated. Thanks.

[adzpond]

Hi,

is it possible to get an update on the progress with this issue please?

Thanks.

[adrichem]

I also see the same exceptions when using MSTest.V2 (1.3.0) with SpecFlow 2.3.2. It seems to occur when multiple scenarios from the same feature file are executed concurrently. When I run 2 scenarios from different feature files, then I don’t see a problem.

I see the following exceptions:

• System.ArgumentException: An item with the same key has already been added.
• System.ArgumentException: An item with the same key has already been added.TestCleanup method RegressieTest.Tests.ObjectsFeature.ScenarioTearDown threw exception. System.InvalidOperationException: System.InvalidOperationException: Collection was modified; enumeration operation may not execute..

The ‘Collection was modified’ message refers to a private non static member of my binding class. Its strange that this collection could be concurrently modified. It almost feels like the same instance of the binding class is shared between different concurrent scenarios.

[SabotageAndi]

I had a look at it and found the reason for this behavior with NUnit.
@adzpond you have [assembly: NUnit.Framework.Parallelizable(NUnit.Framework.ParallelScope.Fixtures)] configured, which executes also tests in the same test fixture in parallel.

When you have a look in the generated code, you see that the class we generate has a state:

[System.CodeDom.Compiler.GeneratedCodeAttribute("TechTalk.SpecFlow", "2.4.0.0")]
[System.Runtime.CompilerServices.CompilerGeneratedAttribute()]
[NUnit.Framework.TestFixtureAttribute()]
[NUnit.Framework.DescriptionAttribute("Parallel Test")]
public partial class ParallelTestFeature
{

    private TechTalk.SpecFlow.ITestRunner testRunner;

#line 1 "FeatureFileb8ab73b05ec94addb097a7e204e6d37b.feature"
#line hidden

    [NUnit.Framework.OneTimeSetUpAttribute()]
    public virtual void FeatureSetup()
    {
        testRunner = TechTalk.SpecFlow.TestRunnerManager.GetTestRunner();
        TechTalk.SpecFlow.FeatureInfo featureInfo = new TechTalk.SpecFlow.FeatureInfo(new System.Globalization.CultureInfo("en-US"), "Parallel Test", "    In order to know my illustrate a bug\r\n    As a user of specflow\r\n    I need t" +
                "o execute these scenarios", ProgrammingLanguage.CSharp, ((string[])(null)));
        testRunner.OnFeatureStart(featureInfo);
    }

NUnit doesn't create for each execution of a test a separate instance of the test fixture. So the state is shared between 2 executions. And that doesn't work.
If you change to ParallelScope.Fixtures your repro runs through.

For the SpecFlow+Runner: do you know the used settings?

@adrichem: which parallel execution settings do you use for MSTest?

[adrichem]

@SabotageAndi I used the settings below.
<?xml version="1.0" encoding="utf-8"?> <RunSettings> <MSTest> <Parallelize> <Workers>2</Workers> <Scope>MethodLevel</Scope> </Parallelize> </MSTest> </RunSettings>

[adzpond]

Thanks, I will try this. Can I add I was getting the same issue as @adrichem which is why I tried nunit.

[SabotageAndi]

@adrichem changing the scope to ClassLevel fixes the exception. But I am not sure why. I will continue looking at this issue next week.

[unickq]

Just want to subscribe to this topic. Also interested in parallel scenarios execution.
It works well with ClassLevel for MSTest and ParallelScope.Fixtures for NUnit - but it's not about parallel scenarios execution.

https://github.com/techtalk/SpecFlow/issues/894

[shrikantkaramkar]

Is there support for .runSettings in SpecRun to get parameters using TestContext?

[david1995]

@shrikantkaramkar please open a separate issue for this question, since it is about a completely different topic. Thank you!

[SabotageAndi]

Closed because of inactivity

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.