Specflow: Can let SpecFlow support vstest of test result?

[yaochangyu]

Hi,
TFS 2015 already generated test result (*.trx)
I try via SpecFlow command load the *.trx, Scenraio item be empty at HTML
So,Can let SpecFlew support vstest of test result?
Like Pickles can support
http://docs.picklesdoc.com/en/latest/ArgumentsTestResultsFormat/

[SabotageAndi]

Which SpecFlow command are you using?

[yaochangyu]

Hi,
My batch file below:
`
@echo off

set currentFolder=%~dp0
echo currentFolder=%currentFolder%

set vsTest="%ProgramFiles(x86)%\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
echo vsTest=%vsTest%

set msTest="%ProgramFiles(x86)%\Microsoft Visual Studio 14.0\Common7\IDE\MsTest.exe"
echo msTest=%msTest%

set dll="%currentFolder%binDebug\THS.CI.Testing.Report.UnitTest.dll"
echo dll=%dll%

set vstestArguments=%dll% /logger:trx
echo vstestArguments=%vstestArguments%

set vsTestFolder="%currentFolder%TestResults"
echo vsTestFolder=%vsTestFolder%

set msTestResultFolder=%currentFolder%MsTestResults
echo msTestResultFolder=%msTestResultFolder%

set mstestResultFile="%msTestResultFolder%\TestResult.trx"
echo mstestResultFile=%mstestResultFile%

set mstestArguments=/testcontainer:%dll% /resultsfile:%mstestResultFile%
echo mstestArguments=%mstestArguments%

echo delete TestResults..
echo.

if exist %vsTestFolder% (
del %vsTestFolder%*.* /s /q
rmdir %vsTestFolder% /s /q )
mkdir %vsTestFolder%

if exist %msTestResultFolder% (
del %msTestResultFolder%*.* /s /q
rmdir %msTestResultFolder% /s /q )
mkdir %msTestResultFolder%

echo run mstest...
echo.
%msTest% %mstestArguments%
echo.

echo run vstest...
echo.
%vsTest% %vstestArguments%
echo.

set specFlow="..\packages\SpecFlow.2.0.0\tools\SpecFlow.exe"
set vsTestSpecFlowReportFile="%currentFolder%vsTestSpecFlowReport.html"
echo vsTestSpecFlowReportFile=%vsTestSpecFlowReportFile%

set msTestSpecFlowReportFile="%currentFolder%msTestSpecFlowReport.html"
echo msTestSpecFlowReportFile=%msTestSpecFlowReportFile%

::set projectFile="THS.CI.Testing.Report.UnitTest.csproj"

for /r . %%f in (*.csproj) do (
set projectFile="%%~f"
)
echo projectFile=%projectFile%

for /r TestResults %%f in (*.trx) do (
set testResultFile="%%~f"
)

echo testResultFile=%testResultFile%
echo.

echo run specFlow report...
echo.

%specFlow% mstestexecutionreport %projectFile% /testResult:%testResultFile% /out:%vsTestSpecFlowReportFile%
%specFlow% mstestexecutionreport %projectFile% /testResult:%mstestResultFile% /out:%msTestSpecFlowReportFile%

::%specFlow% stepdefinitionreport %projectFile% /out:%specFlowStepReportFile%
Start Chrome %vsTestSpecFlowReportFile%
Start Chrome %msTestSpecFlowReportFile%
pause

`

Html Result bellow:

[dirkrombauts]

We need another report, specific to VsTest.

You can probably derive some inspiration from Pickles, that also reads the vstest flavour of .trx files. The relevant code can be found at https://github.com/picklesdoc/pickles/tree/develop/src/Pickles/Pickles.TestFrameworks/VsTest

[Rakesh-Chauhan]

+1 for this feature. I am also trying to generate html report using MStest but the report generated is useless and cannot be shared with management. This was opened in 2016. Can we expect any update soon or just giving up on this is the way forward ?

[mmajcica]

I am facing the same issue. For now, the solution is to use a specific Logger during the invocation of vstest and not the default trx. You can get the necessary information in the following blog post https://blog.markvincze.com/how-to-fix-the-empty-specflow-html-report-problem-with-vstest-console-exe/.
However, I think a specific vstestexecutionreport action should be available and specflow should be able to read the default trx file that is made by vstest. I'll invest some time in this,

[mmajcica]

Unfortunately, it is not possible to write vstestexecutionreport action as vstest doesn't output the necessary information. TestDescriptionAttribute and TestPropertyAttributes are not rendered. I opened the following issue for vstest team https://github.com/Microsoft/vstest/issues/1541
In my opinion, the only thing is to use a different logger and @SabotageAndi this issue can be closed.
@tedvanhaalen

[yaochangyu]

Hi,
The Pickles can support vstest.console's test result
docs.picklesdoc.com/en/latest/ArgumentsTestResultsFormat/

[mmajcica]

@yaochangyu true, however, the information we are looking for to print in the report is not present in the trx file made by vstest.

[craigbrett17]

Am I correct in thinking the only way to support this is to grab some code from the internetz, build it somewhere, copy+paste it to the target machine's VS directory then use it?

That workflow is... not optimal.

[SabotageAndi]

Closed because of irrelevance for SpecFlow 3, since reports have been removed for now and will probably re-implemented differently.

For more details see #1036

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.