Runtime: Remove xunit.console and make VSTest the default runner both locally and in CI

[ViktorHofer]

Today we support two different test runners: xunit.console and VSTest. The former is used when invoking the test target dotnet msbuild /t:Test and when running on CI. The latter is used when using VS Test Explorer, F5, the dotnet msbuild /t:Run target or when invoking dotnet test. To leverage the new crash dump / hang dump features that @davidfowl is driving in partnership with the VSTest team (cc @nohwnd), we should switch to VSTest and remove xunit.console entirely.

This issue only tracks the libraries side for non-mobile and non-WASM configurations. Mobile and WASM use a different harness called xharness which is out-of-scope at the moment.

1. Remove xunit.console:
2. https://github.com/dotnet/runtime/blob/master/eng/testing/xunit/xunit.console.targets

3. https://github.com/dotnet/runtime/blob/master/eng/testing/xunit/xunit.targets#L17

4. Remove runner templates for non-mobile and non-wasm:

5. https://github.com/dotnet/runtime/blob/master/eng/testing/RunnerTemplate.cmd
6. https://github.com/dotnet/runtime/blob/master/eng/testing/RunnerTemplate.sh
7. https://github.com/dotnet/runtime/commit/85bfbfcf5451463d285fd1ccd0b3ac0d8be7b949#diff-e7bc39410f7a119e7adb69b17006c452

8. https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L22-L25

9. Revert https://github.com/dotnet/runtime/commit/85bfbfcf5451463d285fd1ccd0b3ac0d8be7b949#diff-e7bc39410f7a119e7adb69b17006c452

10. Condition the RunTemplate and Test target logic on TargetsMobile and WASM:

11. https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L109-L112

12. https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L40-L104

13. Use dotnet test in CI instead of RunTests.cmd/sh for non-mobile and non-wasm:

14. https://github.com/dotnet/runtime/blob/132be64c3c4781106b8da4328d39dcd0c696921d/src/libraries/sendtohelix.proj#L69-L70

15. Fix failing tests (ie Microsoft.Extensions.DependencyModel.Tests) and make sure that code coverage works.

16. Optional: use dotnet publish instead of the ArchiveTests target:

17. https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L27-L38

cc @Anipik @ericstj @safern

[msftbot[bot]]

Tagging subscribers to this area: @safern, @viktorhofer
Notify danmosemsft if you want to be subscribed.

[ViktorHofer]

Blocked until 5.0 P8 SDK is released.

[tmds]

I'm interested in this because it will make long running tests produce a coredump.
On our rhel8 arm64 CI machine, occasionally we see some long running tests, and having these coredumps will help debug them.

[tmds]

@ViktorHofer is this something you'll be working on short term? I've been ignoring failures from long running tests on our arm64 rhel8 ci machine because having coredumps from vstest will make it easier to debug them.

[ViktorHofer]

@tmds https://github.com/dotnet/runtime/pull/39923 is finally ready. It should be merged as soon as I discussed the new device bring-up scenario offline with my colleagues.