Runtime: Support VS Test Explorer

[ViktorHofer]

To enable Visual Studio Test Explorer integration we need the following things to be done:

1. Tell VS to use our custom dotnet installation (currently under corefx/.dotnet/) https://github.com/Microsoft/vstest/issues/1749
2. Use the repo dotnet installation which we already use for building and get rid of the specialized one in the testhost folder. https://github.com/dotnet/core-setup/issues/4915
3. Update our launchSettings.json to point to the repo dotnet installation: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.CoreFxTesting/build/assets/launchSettings.json#L5 and add the DOTNET_ROOT env var to point to the testhost folder where the shared framework lives.

cc @danmosemsft @ericstj @safern @wtgodbe

[sharwell]

Update our launchSettings.json to point to the repo dotnet installation

This is separate from Test Explorer support, right?

[ViktorHofer]

This is separate from Test Explorer support, right?

It is part of the bring-up work. We currently copy the launchSettings.json file to all our projects when building from root and when incrementally building a specific project. We need to update the executablePath in it and add the DOTNET_ROOT env var, otherwise VS won't respect it.

[tmat]

This will also be an issue: https://github.com/Microsoft/vstest/issues/1764
Microsoft.Net.Test.Sdk currently adds dependencies to the test project, which include some CoreFX binaries and will certainly collide with the ones built from CoreFX.

The test team needs to make sure that the test runner's dependencies are isolated from the test project dependencies in order for this to work.

[ericstj]

@ViktorHofer is this still feasible for 3.0?

[ViktorHofer]

No definitely not. Moving to future.

[Gnbrkm41]

Would absolutely love this to come, test explorer have much better UI and... aesthetically pleasing :^)

[krwq]

cc: @drognanar

[ViktorHofer]

I got VS Test Explorer working locally but a few things need to happen until we can get the necessary changes in master. I summarized the steps in an offline thread and forgot to post them here:

1. https://github.com/dotnet/core-setup/issues/4809
2. Use the startvs.cmd script from aspnetcore to set DOTNET_ROOT and dotnet in the %PATH%: https://raw.githubusercontent.com/aspnet/AspNetCore/master/startvs.cmd.
3. Enable deps file generation for test projects:
   <GenerateDependencyFile>true</GenerateDependencyFile>
   xUnit has a strong dependency on the test app’s deps file: https://github.com/dotnet/corefx/issues/38508
4. Use VS 2019 latest build (>= preview 3, internal only atm)

The only "external" (outside of corefx) blocker is https://github.com/dotnet/core-setup/issues/4809 which got some push-back.

cc @danmosemsft @ericstj

[ericstj]

What if we used DOTNET_ROOT + hardlink created during test build to workaround lack of host feature? I know it’s a hack but it might work temporarily

[ViktorHofer]

We would still need to launch VS per tfm and per configuration additionally to the hardlinking. I'm mainly concerned about the end user experience here. Sure that would work as a first step but what if we would instead just add the necessary functionality in the host?

[krwq]

@ViktorHofer how much work would that be vs hack mentioned by @ericstj? I think the most common scenario is to run netcoreapp tests and remaining tfms can wait

[ViktorHofer]

I'm not happy with the hack mentioned by @ericstj. It's not about how much work it would involve but more if I want us to go that route. Probably the biggest chunk is to enable the deps.json file generation which should take about a day. Let me think about this over the weekend.

[krwq]

@ViktorHofer hack's not ideal but at least allows us to use it while we figure out better solution. If it's the same amount of work to do the better fix as doing the hack then there is no good reason for hack 😄

[Gnbrkm41]

Yay, finally!
Thanks ♥️

[krwq]

Thanks @ViktorHofer!