Vstest: `dotnet test` exit code should return number of failures

[livarcocc]

_From @sidshetye on May 30, 2018 18:41_

dotnet test is documented here but there isn't any documentation on the exit code.

With test runners like nunit3 (e.g. nunit3-console.exe) the exit code is the number of failed tests, which is indeed very handy. dotnet test simply return 0 if all pass and 1 if any failures.

_Copied from original issue: dotnet/cli#9368_

[singhsarab]

@sidshetye This is by design.
Exit codes other than 0, are used to understand what went wrong during the run and sometimes are not controlled. Using exit code for failing tests seems flawed.

Can you please help us understand the scenario. We should be able to give you some suggestions.

[SidShetye]

@singhsarab IMHO, the sole purpose of the runner is to run user demanded tests. So to have the return code tell how many tests fail seems most natural. If one is debugging the runner or the engine itself (something a vast majority of users don't and shouldn't care) then there is exception logging, tracing etc to indicate that.

To answer your question, our CI scripts (in powershell) record the detailed test results in .trx files but also capture the exit code to early terminate a test-suite run on too many failures. Without this, we'll have to manually parse the .trx files in powershell to get to it. Not insurmountable but not as user friendly as just reading $LASTEXITCODE

[SidShetye]

FYI, it was pretty easy to parse the trx for failures. Still think you folks should reconsider the exit code design but our CI is unblocked and this is not a priority for us anymore. Sharing the snippet from our powershell CI scripts for future readers.

Function ParseFailuresInTrxResultFile ( [string] $xmlInputFile )
{
    $xml = [Xml](Get-Content $xmlInputFile)

    # <ResultSummary outcome="Failed">
    #   <Counters total="652" executed="650" passed="630" failed="20" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    #   ...

    $ns = New-Object System.Xml.XmlNamespaceManager($xml.NameTable)
    # our input mstest XML is per this schema
    $xmlNamespace = "http://microsoft.com/schemas/VisualStudio/TeamTest/2010"
    $ns.AddNamespace("ns", $xmlNamespace)

    $xpath = "//ns:ResultSummary/ns:Counters" # xpath when namespaces exists
    $resultSummary = $xml.SelectSingleNode($xpath, $ns)

    return $resultSummary.failed
}

[singhsarab]

@sidshetye I am glad that you were able to enable what you wanted. Thanks for posting the code, very helpful.

With respect to our original discussion, the runner discovers the tests as well when you use /listtests option. Currently, we don't have a strong reason to justify the design change. We really appreciate your feedback, we will reconsider if there is enough ask.

[lgolding]

I just ran into this myself. I want to know if any of my tests failed, and dotnet test won't tell me.

for %%i in (Json.Pointer, Json.Schema, Json.Schema.ToDotNet, Json.Schema.Validation) DO (
    dotnet test --no-build --no-restore src\%%i.UnitTests\%%i.UnitTests.csproj
    if "%ERRORLEVEL%" NEQ "0" (
        echo %%i unit tests failed.
        goto ExitFailed
    )
)

So if you're taking votes for this feature, count me as +1.

[jikuma]

I am also facing this issue. I am running tests in docker image. Docker run command succeeds even ids the tests fail. This is because dotnet vstest return exit code as 0.