Specflow: Test results are no longer shown in the livingdoc report if generated using "feature-folder" argument

Created on 5 Jan 2021 · 4Comments · Source: SpecFlowOSS/SpecFlow

SpecFlow Version:

[ x] 3.5
[ ] 3.4
[ ] 3.3
[ ] 3.1
[ ] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[ x] SpecFlow+Runner
[ ] MSTest
[ ] NUnit
[ ] Xunit

Version number:

Project Format of the SpecFlow project

[ ] Classic project format using packages.config
[x ] Classic project format using <PackageReference> tags
[ ] Sdk-style project format

.feature.cs files are generated using

[ x] SpecFlow.Tools.MsBuild.Generation NuGet package
[ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[ x] VS 2019
[ ] VS 2017

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[ ] Enabled
[x ] Disabled

Are the latest Visual Studio updates installed?

[x ] Yes
[ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

[ ] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[ ] .NET Core 2.2
[ ] .NET Core 3.0
[ ] .NET Core 3.1
[ x] .NET 5.0

Test Execution Method:

[ ] Visual Studio Test Explorer
[ x] TFS/VSTS/Azure DevOps – Task –
SpecFlowPlus@0
[ x] Command line –
livingdoc feature-folder .\Tests.Acceptance\Features -t .\Tests.Acceptance\binDebug\net5.0\TestExecution.json

<SpecFlow> Section in app.config or content of specflow.json

{
  "language": { "feature": "en" },
  "bindingCulture": { "name": "en" },
  "livingDocGenerator": {
    "enabled": true,
    "filepath": "TestExecution.json"
  }
}

Issue Description

After the recent update test results are no longer shown in the generated LivingDoc HTML file when referencing Features folder.

This shows all scenarios and steps as "not executed":
livingdoc feature-folder .\Tests.Acceptance\Features -t .\Tests.Acceptance\binDebug\net5.0\TestExecution.json

This (assembly reference) is working OK using the same TestExecution.json file:
livingdoc test-assembly .\Tests.Acceptance\binDebug\net5.0\Tests.Acceptance.dll -t .\Tests.Acceptance\binDebug\net5.0\TestExecution.json

Presumably the same issue also affects an ADO extension. Test results were showing OK in a demo project before the Dec 17th extension's update to version 0.6.672.
Extension's "task" does not have an option to reference test assembly, which makes this issue a blocker there.

Steps to Reproduce

Use livingdoc CLI (3.5.186+g73f000b344) to create a report with test results using reference to Features folder.

Repro Project

Bug Documentation SpecFlow+ medium

Source

vkudley

All 4 comments

If you want to see the Test Execution results with the feature-folder argument you have to provide the Test Project Root folder and not a subfolder.
So in your case instead of .\Tests.Acceptance\Features you should use .\Tests.Acceptance
We are going to update our documentation and will try to provide some warnings when the configurations are not matching.

epresi on 6 Jan 2021

👍1

Confirmed working with the project's root path instead of direct path to Features!

vkudley on 6 Jan 2021

🎉1

We've added a warning message to the UI - as a first step - in the latest version 3.5.279. We plan to further improve this and improve the detection of this misconfiguration issues in the future.