Specflow: Scenarios fail when MsTest is set to use parallel execution

[jrdodds]

MsTest can be configured to run tests in parallel. See How to: Run Unit Tests Faster Using a Computer with Multiple CPUs or Cores for details.

However SpecFlow appears not to be thread safe. The same ScenarioContext object is passed to multiple Scenarios and Scenarios fail with null reference exceptions from within the SpecFlow infrastructure.

SpecFlow 1.9
Visual Studio 2012
Unit test provider set to 'MsTest'

It's a serious issue that SpecFlow doesn't support parallel execution.

[darrencauthon]

This went on XUnit as well. There might be some progress on this in v2, but I can't remember.

That said, I want to point out that the title to the link you posted includes "How to: Run Unit Tests Faster"

SpecFlow is not for unit testing. It's used for integration testing, which is a much bigger problem for parallel execution. Even if SpecFlow (the library) were to support it with MSTest, I bet most SpecFlow tests suites would not as they're all tied to a shared state: The database. Probably others, too.

Parallel execution is never going to be a yes/no switch with integration testing. It takes a different approach to the development of the tests, and possibly the application, too. With that, great... faster tests, who doesn't like that? But without, it seems like a big price in dev time.

[darrencauthon]

But that said, since it's a SERIOUS issue...

[jrdodds]

@darrencauthon

Hi,
The point of the link is not about unit versus integration testing. The purpose is to show this is a documented feature of MsTest and to provide the detail on how to replicate the issue.

As to your other comment: I found and confirmed an issue. I'm sharing my findings. I don't have a fix at this time. If someone is already working on a fix that would be great to know.

Thanks.

[samholder]

I believe that a company have sponsored techtalk to implement thread safety
in specflow and that the work is done. It is being merged into v2 in the
next few weeks if I recall correctly. I think the pr the comment was on was

380 but can't be certain and can't check at the moment

On 3 Jun 2015 19:18, "Jonathan Dodds" [email protected] wrote:

@darrencauthon https://github.com/darrencauthon

Hi,
The point of the link is not about unit versus integration testing. The
purpose is to show this is a documented feature of MsTest and to provide
the detail on how to replicate the issue.

As to your other comment: I found and confirmed an issue. I'm sharing my
findings. I don't have a fix at this time. If someone is already working on
a fix that would be great to know.

Thanks.

—
Reply to this email directly or view it on GitHub
https://github.com/techtalk/SpecFlow/issues/448#issuecomment-108555444.

[darrencauthon]

I thought I saw a "thread safe" branch a few days ago...

[darrencauthon]

@jrdodds If it is confirmed that this work is done, or going to be done, perhaps you could provide the testing on the branch to verify it resolves your problem?

[jrdodds]

Yes. I can test.

[chassa]

We have implemented a SpecFlow extension to provide thread safety, and the package is currently under test by our customer. We expect that the feature will be merged to the SpecFlow v2 code base by end of June/mid of July.

[JoshuaKGoldberg]

@chassa Any update? :)

[chassa]

There's still feedback pending from the customer, hence we cannot merge yet. However, I'm still confident it will be included in v2.

[tmulkern]

Hi All,

I have added a pull request #600 that could provide the solution you are looking for, I have implemented for MbUnit but it easily could be expanded for MsTest

[RenuTanwar]

Hi All,

Few of the test scenarios are unexpectedly failing with NullReferenceException, i am using parallel execution with Specflow v2.2 + NUnit v3.7.1.

Is there any solution for this or any SpecFlow extension to provide thread safety?

[SabotageAndi]

@RenuTanwar You are in the wrong issue. This is about MsTest and not NUnit.
I think there is already an issue for your problem. Have a look at: https://github.com/techtalk/SpecFlow/issues/884

[Blackbaud-ChrisKessel]

I believe I'm running into the same issue. Using Specflow v2.2.1 + Specflow.MsTest 2.2.1 + MSTest V2 (MSTest.TestFramework 1.2.0-build-20171130-03) that allows in-assembly parallelism. I have set our tests to use method level parallelism using 4 worker threads.

I'm seeing allot of "Object reference not set to an instance of an object. due to the ScenarioContext being null.

Should specflow work with mstest in parallel?
If not ,it looks like there has been numerous discussions on the subject, is there any idea of what the scope of work look like to get this sorted out.

[mahendraschaudhari]

Hi All,
Is this issue planned to be fixed in near future? any update would be helpful

[Blackbaud-ChrisKessel]

@SabotageAndi can this be closed due to #997?

[SabotageAndi]

@Blackbaud-ChrisKessel tbh, I have no idea how the status of parallel execution in the various test frameworks is in combination with SpecFlow. See https://github.com/techtalk/SpecFlow/issues/1052 as an example

[SabotageAndi]

Closed because outdated issue. A lot has changed since this issue was opened in SpecFlow.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.