Specflow: No coverage in feature files when testing with NCrunch

[EdouardMYOB]

SpecFlow Version:

• [x] 3.0
• [ ] 2.4
• [ ] 2.3
• [ ] 2.2
• [ ] 2.1
• [ ] 2.0
• [ ] 1.9

Used Test Runner

• [ ] SpecFlow+Runner
• [ ] MSTest
• [ ] NUnit
• [ ] Xunit
• [x] NCrunch + Xunit

Version number: 3.23.0.10

Visual Studio Version

• [x] VS 2017
• [ ] VS 2015
• [ ] VS 2013

Are the latest Visual Studio updates installed?

• [x] Yes
• [ ] No

.NET Framework:

• [ ] >= .NET 4.5
• [ ] before .NET 4.5
• [ ] .NET Core 2.0
• [x] .NET Core 2.1
• [ ] .NET Core 2.2
• [ ] .NET Core 3.0

Test Execution Method:

• [ ] Visual Studio Test Explorer
• [ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
• [ ] Command line – PLEASE SPECIFY THE FULL COMMAND LINE
• [x] NCrunch

Repro Project

https://github.com/techtalk/SpecFlow-Examples/tree/feature/netcore-examples/NETCore%20Examples/BowlingKata-SpecFlowJson-xUnit

Issue Description

When I open the SpecFlow Bowling Examples for NETCore, the tests are run automatically by NCrunch, and changing one of the scenario's (i.e. the score calculated from 110 to 111) triggers NCrunch to run the test again, now with that test failing. The failing tests coverage in Game.cs is shown correctly (as is the coverage in the BowlingSteps.cs and BowlingDriver.cs files), however there is no coverage in the feature file (e.g. Punkteberechnung.feature), or in the code behind file (e.g. Punkteberechnung.feature.cs).

I tried putting a breakpoint in one of the test methods in the autogenerated file (Punkteberechnung.feature.cs) and running that in the debugger from NCrunch, but the file is never loaded into the debugger ("No symbols have been loaded for this document").

Coverage back to the feature file would be a great outcome for users of NCrunch.

Is there something that Specflow is doing that means the coverage isn't able to be analyzed in the either the feature file or the code behind file?

Or is there something that NCrunch should be doing instead to allow it to pick up on coverage in the feature and code behind files? I'm happy to raise a request on the NCrunch forums with the developer if that is the case.

[SabotageAndi]

I don't think we can do here anything.
My assumption currently is, that NCrunch doesn't look at the #line directives (https://github.com/techtalk/SpecFlow-Examples/blob/feature/netcore-examples/NETCore%20Examples/BowlingKata-SpecFlowJson-xUnit/Bowling.SpecFlowXUnit/Features/Punkteberechnung.feature.cs#L27).

Docs: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/preprocessor-directives/preprocessor-line

With these, we make the connection of code-behind file and feature file. If you don't consider this information, you will only get the code coverage results of the generated code-behind file.

[jimdeane]

This definitely used to work, at least the integration with Ncrunch looked like this:

This is from a post to the ncrunch forum about a year ago (about a different issue) https://forum.ncrunch.net/yaf_postst2338_SpecFlow-integration---Failed-Marker-at-invalid-position.aspx

And i used this frequently.

It looks like the current code generated by SpecFLow isn't the same is some way?

Perhaps you could liaise with Ncrunch support (so that we don't end up with mixed stories) to get it resolved.

Thanks

Jim Deane

[SabotageAndi]

@jimdeane I am not aware that we changed anything with the generated code for SpecFlow 3.

[Stephen-Mc]

@jimdeane I suggest you raise this issue with NCrunch support directly. Let them know that you are a paying customer and refer to this issue to give them context. You are in a better position than we are to answer any questions they may have about the behaviour you are experiencing. Otherwise, we'd just end up playing a game of broken telephone.

SpecFlow is open source, meaning that the NCrunch team can look at the code directly, and submit a fix if the issue is indeed in SpecFlow. They can also ask us any questions they have regarding the technical side of SpecFlow, and we would be more than happy to help them determine what needs to be done to ensure the behaviour is the same as before.

Because of the open source nature of SpecFlow, there is also a limit to the resources that we can dedicate to dealing with issues, and we need to prioritise. In this case, where NCrunch is a paid product, we would see the onus as being on NCrunch to retain compatibility. In terms of overall priority for all of our users, this issue would end up taking a back seat to issues that affect more SpecFlow users and where investing our limited resources makes more sense to us. And if the issue is on NCrunch's end, there is only so much we can do anyway.

[jimdeane]

Thanks @Stephen-Mc I'll certainly do as you recommend!

[EdouardMYOB]

@jimdeane followed up on the NCrunch forum, and the the NCrunch team has fixed the issue in the latest release (3.26.0.4)!

[jimdeane]

Indeed they have. I thought I had reported this back to the forum, clearly I forgot 🙁. I reported this on ncrunch support along with a demo project and they said they would fix it. Got a message back the next day saying it would be in the next release 😀

[Stephen-Mc]

Thanks for letting us know.

[SabotageAndi]

Closed because resolved.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.