Sdk: Restore dotnet test functionality by separating it from IDE concerns

[plioi]

This is in response to the breakage described by the xUnit team here: https://github.com/dotnet/cli/issues/3114#issuecomment-264662326

Summary

By separating IDE integration from dotnet test, test frameworks' own features can be made available again to end users, and test frameworks can continue to be as lean as they always have been.

Background

Before preview 3, dotnet test allowed for both CLI mode and IDE mode test running. Test frameworks would provide an executable with a discoverable name, "dotnet-test-framework.exe".

When invoked from the command line, dotnet test would find and run the framework's exe once for each target framework, and would use the overall results of each run to decide whether the overall run succeeded or failed. dotnet test would use the special positional argument "--" to avoid any ambiguity of command line arguments, passing them along to the test framework so that the test framework's own features were all exposed to the end user.

When invoked by the IDE for Test Explorer, the same framework-provided exe would be located and used, this time in a "design time" mode. Test Explorer naturally needs a lot more information from the test framework in order to do its job. There's a discovery phase that is irrelevant in CLI mode, and a data model for the discovery phase and for the execution phase that gives the IDE enough information to show everything that happened.

Before preview 3, this all worked using a series of JSON messages passed back and forth between dotnet test and the framework-provided exe.

There were a few downsides to this approach. Test framework authors essentially had to copy and paste a lot of each other's code to do the JSON message passing correctly, and they started taking on JSON.NET dependencies that might trample the JSON.NET dependency of the system under test.

With the preview 3 tooling, that whole approach is now gone. Instead, both CLI mode and IDE mode test running happens through the tried-and-true IDE integration APIs that existed before .NET Core and the dotnet tooling. While that's just fine for Test Explorer, it means that all CLI test runs happen through all of that infrastructure. The most serious issue is that command line arguments no longer pass through, meaning that end users can't use dotnet test for any framework-specific features, even fundamental ones like the XML reporting understood by many CI systems.

Recommendation

It might be tempting to work around the command line argument pass-thru issue by extending the Test Explorer interfaces to include a string[ ] argument, but I think that just takes an earlier misstep and cements it.

Instead, let's consider that CLI test running and IDE test running are simply different things, and always have been in all of the .NET test frameworks:

1. Continue with the preview 3 tooling approach for IDE integration. The existing API for integrating with Test Explorer works, and is a natural fit for projects like NUnit and xUnit who are trying to take their existing code and adapt it to the .NET Core world. It's familiar, it works, and porting to .NET Core isn't all that difficult for this component.
2. Switch CLI mode test runs to have nothing to do with IDE integration.

Point 2 is worth clarifying. Picture the previous tooling that used JSON message passing between processes. All of that message passing was really a result of the IDE mode concerns, in which a lot of data does need to be communicated back for display in the UI. Without IDE concerns, the discovery phase and all its message passing goes away, as does all the rest of the message passing.

What would be left? What would the dotnet test CLI tool look like? It would become pretty trivial:

1. Ensure the target project(s) are built, like it already did.
2. Loop through the test project's target frameworks (net45, netstandard1.3...), like it already did.
3. For each target framework, find the dotnet-test-framework assembly, like it already did.
4. Start up a Process (passing through command line args) for that assembly and WaitForExit. No need for the previous socket and threading concerns. New Process, WaitForExit.
5. Consider the process's return code. If it's 0, success. Otherwise, it failed.

That's it. No more information needs to pass between the two processes in CLI test runs. Let the test frameworks own all other output in CLI mode, as they always have. dotnet test would save us from having to do the "target framework loop" ourselves, but wouldn't bring along any IDE baggage like detailed message passing, *trx file generation/processing, etc. All the current work to make IDE integration work with the tried and true Visual Studio interfaces could continue unharmed.

Each test framework could remain lean and featureful without having to "pay the Test Explorer tax".

[livarcocc]

@Faizan2304

[bradwilson]

To be pedantic, prior to Preview 3 there was still a single plugin that works for both the IDE and the command line; it was just architected differently than the one that's in Preview 3 and later.

[blackdwarf]

@plioi thanks for the writeup. I would like to take a slight step back and try to extract requirements from your writeup:

1. Decouple the dotnet test from any IDE concerns - this is mostly due to perceived performance impact of those concerns in purely CLI scenarios.
2. Allow access to the underlying test runner - this so that the user can access all of the specifics of the test framework she is using.

Are my extracted requirements correct?

[plioi]

That's correct. As you're getting at, my concerns are the overhead in CLI scenarios, and the ability to get test-framework-specific command line arguments through to the underlying test framework to enable their features. It's merely an opinion that the issues would be resolved by separating CLI mode from IDE mode as described.

[bradwilson]

FWIW, I don't feel like item 1 is an issue at all. Nobody complained in DNX or pre-Preview 3 SDK tooling about the fact that a single binary was used for IDE and CLI. It's a red herring. Fixing issue 2 is what you should focus on here (whether that means divorcing from IDE or not should be a secondary consequence of the design, rather than a primary goal).

[TheRealPiotrP]

@plioi @bradwilson thanks for the conversation here. @bradwilson probably remembers some conversations we had a year or so ago where I echo'd @plioi quite strongly :)

dotnet-test in the PJ CLI swung pretty far in one direction where it was simply a wrapper around the test runner. This was inconvenient for new users in that they had '0' consistency across test runners.

In the MSBuild-driven CLI we've adopted a pattern of "approachable facade over powerful tools" whereby we have consistency across the CLI product without taking away access to advanced features where desirable. You can see this approach in:

• dotnet build/publish/run being wrappers around MSBuild. The advanced scenarios are accessed through dotnet msbuild.
• dotnet restore/pack being wrappers around NuGet. The advanced scenarios are accessed through dotnet nuget
• dotnet test being a wrapper around dotnet vstest.

In the last scenario, as you're pointing out, we have an opportunity to provide stronger extensibility. Specifically, dotnet vstest is itself a uniform wrapper around a bunch of test adapters. If we follow the same model as we have elsewhere, you can imagine a world where dotnet test and therefore dotnet vstest provides approachable capabilities common to all adapters. Advanced users can then use some additional channel to light up advanced capabilities of a given test adapter.

This would be a rather natural evolution of the model we have today and @blackdwarf has been working with the folks responsible for dotnet vstest to find the right balance for such an experience.

We should be able to share something more concrete in the very near future, but for now I wanted to say that we hear you... loud and clear :)

[citizenmatt]

I just want to add that there are more IDE runners than just Visual Studio - we have both ReSharper and Rider to think about. And we want to continue offering existing functionality - running multiple or individual tests, debugging, profiling, and code coverage.

(And I have to again voice some frustration here. While we are in conversation with the responsible team, we're still waiting for details - high level overview, API specs, roadmap, capabilities, etc. It's not a good sign that an open source project can ship such a breaking change without releasing the source, documenting anything or even discussing it in the open before hand.)

[tommed]

@citizenmatt Agreed. Not currently having Resharper and NCrunch test runners for dotnet core is a real ache for our development team. The EAP releases are super unstable.

[blackdwarf]

I've been working through the recent comments about the Preview 3 "dotnet test" and how it changes one key behavior from Preview 2: accessing the underlying test runner. It seems that the new infrastructure that is based on vstest console and test adapters actually provides a mechanism by which users can pass parameters to the test adapter. The mechanism is called the run settings file. You can learn more about the settings files on https://msdn.microsoft.com/en-us/library/jj635153.aspx.

This is an XML file that contains the general data that configures the test run, but it also contains a node that is specific to the test adapter being used. Inside this node there can be directives that are passed to the adapter and that configure the adapter's behavior.

Below is an example for Nunit taken from
https://github.com/nunit/nunit3-vs-adapter/blob/master/demo/demo.runsettings. To me this seems quite clear.

<NUnit>
    <BasePath>D:\Dev\NUnit\nunit3-vs-adapter\demo\NUnitTestDemo\bin\Release</BasePath>
    <PrivateBinPath>extras;more.extras</PrivateBinPath>
    <DefaultTimeout>5000</DefaultTimeout>
    <WorkDirectory>work</WorkDirectory>
    <InternalTraceLevel>Info</InternalTraceLevel>
    <RandomSeed>1234567</RandomSeed>
  </NUnit>

Of course, this is also present for other test adapters. You can see the MSTest example on https://msdn.microsoft.com/en-us/library/jj635153.aspx.

Thoughts on this approach?

[Antaris]

So you would have a .runsettings within the scope of your solution, or per-project? What would generate this file? If this were to be hard-coded, the current .runsettings XML is quite verbose, is there anyway this can be reduced to just the <TestRunner> specific node, and would it also work with relative paths?

[blackdwarf]

@Antaris let's see what @codito and @Faizan2304 say on some of the more advance

Here is my take on some of the questions:

1. The file is "just" XML so you can generate it manually. It can also be generated from VS.
2. Yes, you can use this file to actually just specify <TestRunner> specific node, that is, settings to the test adapter.

[Antaris]

Just throwing this idea out there - instead of having a separate .runsettings file, could we not bolt this into the msbuild file:

<PropertyGroup Label="XUnit">

</PropertyGroup>

Although, I don't think Label forms a prefix for the attached properties? Maybe a Scope tag would be nice. Just an idea is all..

[codito]

@Antaris user could have runsettings at any scope. The runsettings are applied to all runs (one or more based on TargetFrameworks) triggered by dotnet test -s demo.runsettings foo.csproj.

Similarly in the IDE, runsettings chosen in Test menu are applied to entire run (includes all test projects which are part of the run).

[pvlakshm]

We've evaluating a simplified experience where additional arguments can be passed into the adapter from the command line directly.

The syntax is similar to dotnet run, where additional arguments can be passed in following --. In the above example, user will be able to pass the options for nunit as follows:

> dotnet test -- NUnit.WorkDirectory=work

Internally, test platform will compose a runsettings xml from this and hand it off to all available adapters. They will receive an xml as shown below.

<NUnit>
    <WorkDirectory>work</WorkDirectory>
</NUnit>

[derwasp]

I feel a bit lost here. Do I get it right: this was resolved in dotnet/cli#5181? If yes, how do I actually get my xUnit xml test results back?

[plioi]

@derwasp It may be that xUnit would have to add support for the XML suggested above to take advantage of it; that's unclear. However, this tweet suggests that xUnit is abandoning dotnet test outside of "Hello World" demos: https://twitter.com/xunit/status/824313810852777985 "Our long term solution is our own command for CLI (dotnet xunit) which will give full console runner fidelity vs. xunit.console."

That rework (and the inevitable rework all the test frameworks will embark on soon) is a consequence of the insistence that command line test running has any connection whatsoever to IDE running. dotnet test is probably not going to be used in practice outside of MSTest.

[derwasp]

@plioi thank you.
Well, looks I will wait with updating to new .NET core. Again.

[codito]

Just FYI, Microsoft/vstest#344 added a flag to differentiate console runs v/s IDE test runs. https://github.com/xunit/visualstudio.xunit/pull/100 and https://github.com/xunit/visualstudio.xunit/pull/103 are taking advantage of this to provide CLI specific optimizations.

[adback03]

@derwasp not exactly the same results as using the XML switch, but you can also do this:

dotnet test MyTestProject.csproj --logger trx

That will at least output a TRX report which your build agent could then use for reporting.

[derwasp]

@adback03 thank you, I know. And that's what I ended up doing for my Jenkins build. It turned out that xUnit jenkins plugin supports MSBuild's TRX format as well.

[RehanSaeed]

@adback03 Is there a way to control the output directory for the xml test result files? Is there a list of commands that can be run?

[derwasp]

@RehanSaeed try this: dotnet test myproj.csproj --logger "trx;LogFileName=c:\logfile.trx"

[codito]

Update: a xunit xml logger for dotnet test is available here: https://www.nuget.org/packages/XunitXml.TestLogger/

Source code and documentation is at https://github.com/Faizan2304/LoggerExtensions#xunit-logger. List of all available loggers are here: https://github.com/Microsoft/vstest-docs/blob/master/docs/report.md#test-loggers.

Please try it out and create an issue with any feedback.

[livarcocc]

Closing old issues.

[BrainCrumbz]

Hi there. Just as a heads up: is this closed just due to its age, or because it does not apply anymore, given some architectural changes etc? Thanks

[bradwilson]

The design of the dotnet CLI test runner changed since this issue was opened.