Specflow: Tests generated from Scenario Outline and Example tables show up as "External" in Group by Project view of VS2012 Test Explorer

[shashiprabhakar]

All tests that are created based on Scenario Outlines are grouped under "External" in Group by Project view of VS2012 Update 1 (or later) - rather than under the name of the project. On the other hand, all tests created based on Scenarios _are_ grouped under the project name as expected.

Version information: Specflow 1.9.1, VS2012 Ultimate RTM + Update 1 + Update 2 CTP

Steps to reproduce:
a) Create a new Specflow test project in VS2012 (create a Class Library project and add references to the Specflow dlls). You can also use an existing Specflow project if you have one.
b) In the feature file, create a Scenario Outline and add an Examples table with values for the parameters of the outline. Here is an example for your convenience:

Scenario Outline: Add any two numbers
Given I have entered first number and second number into the calculator
When I press add
Then the result should be on the screen

Examples:
| ScenarioName | Number1 | Number2 | Result |
| 50Plus70 | 50 | 70 | 120 |

c) Generate the step definition file from the scenario outline (you do not need to implement the stub methods).

d) Build the solution.

e) Open Test Explorer (Test menu > Windows > Test Explorer)

f) Select Group by Project view from the dropdown in the toolbar.

Expected: The test should be grouped under the name of the project.
Actual: The test is under "External".

ADDITIONAL INFORMATION:
a) If you add a Scenario to feature file and repeat steps c), d) and e), you should see that the test generated from it is grouped under the project name - while the test generated from the Scenario outline is grouped under "External".
b) Issue is only in VS2012 Test Explorer - tests are correctly grouped under project name in the Test List window of VS2012 and in all windows (Test View, Test List) in VS2010.

[gasparnagy]

Configuring SpecFlow with (in app.config) and regenerate the files is a workaround for this problem.
With 1.9.2 the "Run SpecFlow Scenarios" will also work with this setting.

[turbobobbytraykov]

Hi Gaspar,
Thanks for the comment, but I'm not sure I understand the 1st part of your suggestion.
What configuration in the app.config are you proposing?
Also, have in mind that in the context of the question, we want all scenario outlines for the project (and there can be more than one in a solution which is my situation).

I agree that right-clicking on the project in the Solution Explorer and choosing "Run SpecFlow Scenarios" is a valid workaround - Thanks for sharing it! :)

[gasparnagy]

The markdown syntax removed the configuration option I wanted to include... so the config option is:

<specFlow>
  <unitTestProvider name="..." /> <!-- this you have already -->
  <generator allowDebugGeneratedFiles="true" /> <!-- this is the important setting -->
</specFlow>

if you update your app.config with the setting above, and let VS regenerate the feature files, then the scenario outlines will not appear as "external" anymore. on the other hand, with this setting you will not be able to use the "run specflow scenarios" currently -- this is what 1.9.2 will improve.

[turbobobbytraykov]

Thanks for the snippet, Gaspar! It worked like a charm!

Honestly, I would have never thought of using the allowDebugGeneratedFiles flag in order to solve the problem.
Also, oddly enough after all scenario outlines' tests appeared correctly in the Test Explorer, I was _able_ to use the "Run SpecFlow Scenarios" option on the entire project.

One final question though - when SpecFlow 1.9.2 comes out, will we still need to set the allowDebugGeneratedFiles flag to TRUE in order to fix the problem?

[gasparnagy]

it is a bit tricky, indeed. the allowDebugGeneratedFiles controls whether there should be line pragmas (ie references to the original source lines) generated to the code behind file. this is necessary for debugging the feature files.
the new VS test framework uses the the source reference information (if there is any) to run/group/etc the tests. The line pragmas, that we generate are not perfect for the scenario outline example lines (because that was not necessary for the debugging), and therefore it confuses the test grouping.
setting allowDebugGeneratedFiles flag to true disables the line pragmas entirely, so the VS test framework stitches back to "normal" mode.
for the TRUE fix, we would need to modify the generator, that it generates line pragmas for the scenario outline examples in a way, that is ideal for the test runner. this is a bigger change, so we can do it only in 1.10 (1.9.2 is just a visual studio integration update - coming out next week).

the VS test framework is changing quite extensively still, so the behavior might be slightly different for every VS update or CTP.

[turbobobbytraykov]

Thanks for the awesome details, Gaspar! Will be looking forward to the iterations of 1.9 and definitely to 1.10! :)

As for the disabled line pragmas, that will just botch debugging .feature files using breakpoints, right?
(that is if I've understood the https://github.com/techtalk/SpecFlow/wiki/Debugging-Tests article correctly)

[gasparnagy]

yes, exactly

[eriksanders]

When I use [generator allowDebugGeneratedFiles="true" ] clicking on the test explorer navigates to the generated code instead of the feature file. To me this is not the behavior I like.

This navigation works with this setting both for scenario's and scenario outlines. Originally scenario outlines did not navigate at all

This behavior is the same in VS2012 and VS2013

[Sarithaa]

Hello,
We are running into the same issue. Parameterized tests belong to External category instead of Project. We are on SpecFlow 1.9.0 and do not see any updates. The proposed app.config workaround did not help.
Please advise.

[sdanailova]

We are running into the same issue. One of our projects doesn't have Scenario outline and all scenarios are missing when grouping at Test Explorer by project, the other one has scenario outline and only these tests are listed as external project at Test Explorer. This should be a specflow bug.

[MNF]

I am running VS 2015 and SpecFlow 2.0. I had the same issue, and have to specify
<generator allowDebugGeneratedFiles="true" />
to put tests in correct project group. It will be good to address the issue.

[ChristopherHaws]

This also seems to be causing an issue with going to the test definition by double clicking the test in the Test Explorer. I tried adding the generator element which make VS2015 bring me to the feature.cs file. Without it, double clicking the test does nothing at all.