Vstest: Multitfm support for IDE (.net core)

[codito]

Description

When user has a test project with multiple target frameworks:

1. Discovery and execution should work for all targets
2. Investigate if we need to switch test explorer behavior on active targetframework or just (1) is good enough.

IVSProjectHierarchy doesn't support finding all project contexts, but CPS based project system already supports multiple project contexts.

[codito]

Related https://github.com/dotnet/roslyn-project-system/issues/976

[harshjain2]

@singhsarab Make sure that changing the target framework from netcoreapp1.0 to net46 and vice versa should not break test discovery in test window.

[codito]

Given the amount of changes (incl. introducing a new container discoverer), this feature will come post RTM.

[clairernovotny]

I can see at least two scenarios of how this should be presented in the IDE:

1. Grouped by project/tfm. For example:

ProjA (netcoreapp1.0)
   Test1
   Test2
ProjA(net46)
   Test1
   Test3

That gives me the ability to run all tests in a given platform/project.

1. Flat list
   If I filter on other views where the project isn't present, I'd expect something like the following:

Test1 (netcoreapp1.0)
Test1 (net46)
Test2 (netcoreapp1.0)
Test3 (net46)

That way I can easily run or debug a test against a specific platform. Of course, it probably makes sense to put an overall filter option to hide/show specific TFM's in that view as well.

[JamesNK]

Will there be a way to choose which TFM to run tests for?

For example if I have a test that is valid under multiple TFMs and I choose to debug it, under which TFM will the test be debugged? And if I right click inside a test method in the text editor and choose run tests, which TFM will be run?

[natemcmaster]

Related: if multiple frameworks are listed, no tests will be discovered in the VS Test Explorer window unless .NET Core is first.

Example:

No tests discovered by VS
<TargetFrameworks>net452;netcoreapp1.1</TargetFrameworks>

Tests discovered by VS
<TargetFrameworks>netcoreapp1.1;net452</TargetFrameworks>

[JamesNK]

There is a bug that only tests from the first framework are discovered.

[snkota]

This is being tracked external to this repo here:
https://developercommunity.visualstudio.com/content/problem/145945/multitfm-support-for-ide-net-core.html

[johnwc]

Why was this closed?

[bgavrilMS]

Why is this closed if the work is not done?

[smadala]

/cc @kendrahavens @ManishJayaswal @AbhitejJohn

[slaneyrw]

Please re-open this and fix. It's been 18 MONTHS since this has been reported and it's a real pain to set multi-targeted libraries.

[kendrahavens]

Since some folks are still commenting here I'll to repeat @snkota, this issue is being tracked here: https://developercommunity.visualstudio.com/content/problem/145945/multitfm-support-for-ide-net-core.html

It is on the backlog for Visual Studio 2019, but due to other high priority items we aren't ready to guarantee what release it will be in. Thank you for your patience and upvoting this issue if you are affected. It helps us prioritize what will improve the experience the most for all user scenarios!

[ViktorHofer]

As we just discussed a related issue in our standup, is there a tracking issue that discusses what it would need to introduce a test tfm property, decoupled from the configuration property, with a switch in VS that allows running netstandard test assemblies under a specific tfm?

That's not just related to VS. Running from command line with a test tfm switch for dotnet test would benefit from that too.

Right now in corefx we are doing crazy things to test against netstandard assemblies with different tfms without cross-targeting and recompiling.

[kendrahavens]

@ViktorHofer There is a test tfm switch in dotnet test. The UI would essentially use that command and set the --framework flag for the test run. Are you describing a different scenario?

[ViktorHofer]

There is a test tfm switch in dotnet test. The UI would essentially use that command and set the --framework flag for the test run. Are you describing a different scenario?

Documentation of the --framework commands says Looks for test binaries for a specific framework.. I was talking about having netstandard (i.e. netstandard2.0) test assemblies built and invoking them with different frameworks. The framework command sounds like it builds the project against the passed-in tfm and invokes the produced tfm specific assemblies.

[kendrahavens]

Thanks for the clarification!

Currently a specific target platform is needed for building and running tests. As in, if you only reference .NET standard in your csproj you will get errors when building. You can think of tests as essentially applications rather than classlibs. Today, if you add a new target framework, restore successfully, and then run the tests with different framework flags, it will still build the binaries of the new framework. I don't believe we will enable a separate build of the test assemblies if they target .NET Standard before building them for their target platform. This xUnit doc goes into some great detail on this.