Sdk: dotnet test - Output test results

[Greg15153]

Currently there is no way to output the test results from dotnet test. Our CI requires a file in order to determine if a job has failed or not plus it allows us to gather data from our builds.

Would be useful for dotnet test to accept a output path for test results.

[blackdwarf]

@Greg15153 can you provide some more details? Wouldn't dotnet test > output.txt work? Or are the details that it prints not enough?

/cc @livarcocc

[Greg15153]

Outputting the result of the command and parsing out what I need isn't too big of a deal but it is a little lacking. Previously before switching over to dotnet core we would use VSTest.Console and had the option to output TRX files.

To be able to know what tests were run and how long in an XML output would be great. Found this example of xunit results.

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="nosetests" tests="1" errors="1" failures="0" skip="0">
    <testcase classname="path_to_test_suite.TestSomething"
              name="test_it" time="0">
        <error type="exceptions.TypeError" message="oops, wrong type">
        Traceback (most recent call last):
        ...
        TypeError: oops, wrong type
        </error>
    </testcase>
</testsuite>

[brthor]

You can pass -xml /some/path/out.xml

We use this in our Jenkins CI to determine what tests failed in a build.

[Greg15153]

Thats perfect. Wish dotnet test --help showed me that :)

[livarcocc]

It does not because that's a capability of the runner dotnet-test-xunit and not of dotnet test. Maybe we should do something to show the capabilities of your runner as well, when you do dotnet-test --help. I have created https://github.com/dotnet/cli/issues/3143 to consider this.

[bradwilson]

We updated dotnet-test-xunit so that you can do dotnet test -help. I know this is not ideal, but we're stuck behind the command line parsing above us.

[tghcastro]

@brthor which Jenkins Plugin do you use? I need to run "donet test" and publish jenkins report
Thanks

[tommed]

This doesn't seem to work now msbuild is used for testing? Or have I missed something?

$ dotnet test -xml results.xml MyProject.csproj
MSBUILD : error MSB1001: Unknown switch.
Switch: -xml

I'm guessing we'd now need to use a value for dotnet test --logger but the only value I can see that would be valid is trx?

[bradwilson]

@tommed You should not expect to be able to use runner-specific features with dotnet test any more.

In the world of project.json (that is DNX, and .NET Core SDK prior to Preview 3), dotnet test was just a thin shim to a console application written by the unit test framework author. The code for dotnet-test-xunit was extremely similar to the code for xunit.console.exe.

In the world of MSBuild (that is, .NET Core Preview 3 and later), dotnet test is just a thin shim to something very much like the Visual Studio test runner executable; in fact, the API that test framework authors use to plug into dotnet test is actually the same API that has powered the Visual Studio GUI and command line runners (for desktop and devices) since VS 2012. In this world, the test framework author has very little control over the experiences that surround the runner; they simply provide a way to discover and execute tests, with very little influence beyond that; definitely, we have no impact over any reporting systems.

That means if you're using dotnet test in a CI scenario, you will need to find tools prepared to consume and report against Visual Studio's .trx files.

[TheRealPiotrP]

It will also be possible to create custom loggers that output in another format like xunit v2 xml.

[plioi]

@piotrpMSFT But report file format is just one example of how unusable preview3's dotnet test is, now. As is, "dotnet test" supports "any test framework as long as that test framework doesn't have features".

[blackdwarf]

@plioi what would be your suggestion?

[plioi]

Passing the command line arguments through, like before. In the earlier previews, everything after the special positional "--" argument was intended for the test framework to receive. Test frameworks were therefore allowed to have features, were allowed to skip throwing resources at output file formats the user wasn't asking for, etc.

My understanding is that preview 3 does command line integration through the interfaces used to integrate with Test Explorer. That appears to be quite limiting. If those interfaces can't change to take on a string[] input, then dotnet test is only going to be usable for Hello World demos.

Even if the preview 3 approach changes to allow args to pass through those interfaces, this whole approach seems needlessly costly at runtime. I'd expect every test framework team to have to give up and provide their own dotnet-test-framework-name-actual.exe while recommending users not use "dotnet test".

[plioi]

Better yet, IDE mode test running and CLI mode test running are simply different things, and ought to be separate. Allowing CLI runs to be lean and featureful wouldn't have to cripple IDE running, and IDE running wouldn't have to cripple CLI running.

[blackdwarf]

@plioi thanks for the feedback, that is quite useful.

[plioi]

I'll write up a proposal in a separate issue first thing tomorrow, for what the imagined dotnet test CLI would be if the IDE concerns were separate.

[plioi]

@blackdwarf I've proposed a simplification in dotnet/cli#4921 that would restore this functionality.

[StrangeWill]

I'd expect every test framework team to have to give up and provide their own dotnet-test-framework-name-actual.exe while recommending users not use "dotnet test".

This is absurd and just solidifies the complaints of users that did _not_ want to move to MSBuild wrapping everything and csproj. We _had_ a well working unified ecosystem before, and now we're moving back to Microsoft vs. everyone else. It's been a huge step back going back on lessons learned from poor past performance.

[kubal5003]

This comes as a huge surprise. Today I was updating my CI to move to VS2017 & csproj files and this issue is currently preventing me from doing so. I'm not sure why this is closed, because it's very much valid!

[Romanx]

@kubal5003 You can use the trx to get the output like this dotnet test --logger "trx;LogFileName=abc.trx" If you're using something like AppVeyor it will upload in the MSTest format so be sure to set the type correctly 👍

[jderus]

Maybe I am missing some nuance , but this seems like a breaking change to me from 1.0 ->1.1.
I couldn't find it anywhere in Relnotes - I stumbled upon it while looking up CC in CI.

It would be nice to have these things be a little more documented and clear up front.

[bradwilson]

@jderus Presuming you mean 1.0 and 1.1 of .NET Core, there are no changes. The same runner binaries are used for both.

[StrangeWill]

Went ahead and opened a relevant ticket on the vstest project considering this is now part of Microsoft's new MSBuild task runners sitting in front of everything and causing unnecessary migraines.

Ticket here: https://github.com/Microsoft/vstest/issues/786