Vstest: Test Explorer discovers but fails to execute tests targeting full .NET Framework after upgrading to VS 15.8

[plioi]

Description

Using Visual Studio 15.7.4, I can successfully open a solution containing many test projects with a variety of TargetFramework/TargetFrameworks values, discovering and executing all tests without error.

After upgrading Visual Studio to 15.8.0, test projects which target net471, or test projects which multitarget but target net471 first, successfully discover but do not execute. Tests which do not execute disappear from Test Explorer.

Similar discrepancies appear when using dotnet vstest against the net471 build vs netcoreapp2.0 build of the same test projects.

Steps to reproduce

Sample solution which reproduces the issue: https://github.com/fixie/fixie.integration known to be reproducible as of commit 4e570de3

The solution has 19 projects. Fixie.Integration is not a test project, and Inclusive.Tests is a test project that has zero tests in it. The expectation, then, is for Test Explorer to show discovered tests for 17 projects.

Open the solution in Visual Studio 15.7.4 (suspect any 15.7.* would work). In Test Explorer, select "Show Test Hierarchy" to group discovered tests by project. Some unrelated issue may cause only the FSharp tests to appear upon first discovery. Rebuild the solution to provoke a full discovery. I get 17 discovered test projects, and Run All runs all of them. At the end of the run, all 17 remain in the Test Explorer tree. The "Tests" Output pane shows all the assemblies processed, including the full .NET Framework ones executed in a group as expected (full framework runs seem to be single-process historically, while .net core runs seem to be per-test-project historically).

Open the solution in Visual Studio 15.8.0. Rebuild to provoke discovery. The same 17 projects successfully discover tests. But, when I Run All, 3 of the test projects disappear from the UI rather than showing results. Rebuild All, and the 3 missing projects reappear as discovered (this time they'll stand out as the only projects with the blue "!" not-yet-run status indicator). Expand to any single test in these 3 projects and attempt to run it: the test disappears from the UI and the "Tests" Output pane shows:

[8/16/2018 9:40:50 PM Warning] Could not find test executor with URI 'executor://fixie.visualstudio/'. Make sure that the test executor is installed and supports .net runtime version .

Rebuild all again to rediscover the now-missing test. Run All one last time. The "Tests" Output pane shows the 3 problematic projects do get processed during discovery, but are silently skipped during execution.

Similarly, at the command line, execution is inconsistent depending on the target framework:

dotnet vstest src\Nested.Tests\bin\Debug\netcoreapp2.0\Nested.Tests.dll runs 2 passing tests.

dotnet vstest src\Nested.Tests\bin\Debug\net471\Nested.Tests.dll outputs a yellow warning:

No test is available in c:\dev\fixie.integration\src\Nested.Tests\bin\Debug\net471\Nested.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

There is a pattern to which of these tests fail to execute and which are honored. Every test project which successfully discovers and executes has either <TargetFramework>netcoreapp2.0</TargetFramework> or <TargetFrameworks>netcoreapp2.0;net471</TargetFrameworks>. Every test project which successfully discovers but fails to execute has either <TargetFramework>net471</TargetFramework> or <TargetFrameworks>net471;netcoreapp2.0</TargetFrameworks>

I know that in the case of multitargeting, Test Explorer is only expected to execute the first listed framework. So, it appears that execution of full framework assemblies has undergone a breaking change which is inconsistent from the discovery phase.

Expected behavior

All discovered tests should be executable.

Actual behavior

Full framework tests are discovered, but cannot be executed.

Diagnostic logs

"Tests" Output pane during a discovery phase and subsequent execution phase, under VS 15.8.0 (emphasis added, showing net471 assemblies processed during discovery but not execution):

[8/16/2018 10:29:08 PM Informational] ------ Discover test started ------
[8/16/2018 10:29:08 PM Warning] Test run will use DLL(s) built for framework Framework45 and platform X86. Following DLL(s) will not be part of run:
x64.Tests.dll is built for Framework Framework45 and Platform X64.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
[8/16/2018 10:29:08 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:08 PM Informational] Processing C:\dev\fixie.integration\src\Async.Tests\bin\Debug\netcoreapp2.0\Async.Tests.dll
[8/16/2018 10:29:09 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:09 PM Informational] Processing C:\dev\fixie.integration\src\Categories.Tests\bin\Debug\netcoreapp2.0\Categories.Tests.dll
[8/16/2018 10:29:10 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:10 PM Informational] Processing C:\dev\fixie.integration\src\DefaultConvention.Tests\bin\Debug\netcoreapp2.0\DefaultConvention.Tests.dll
[8/16/2018 10:29:11 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:11 PM Informational] Processing C:\dev\fixie.integration\src\FSharp.Tests\bin\Debug\netcoreapp2.0\FSharp.Tests.dll
[8/16/2018 10:29:12 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:12 PM Informational] Processing C:\dev\fixie.integration\src\Inclusive.Tests\bin\Debug\netcoreapp2.0\Inclusive.Tests.dll
[8/16/2018 10:29:12 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:12 PM Informational] Processing C:\dev\fixie.integration\src\IoC.Tests\bin\Debug\netcoreapp2.0\IoC.Tests.dll
[8/16/2018 10:29:13 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:13 PM Informational] Processing C:\dev\fixie.integration\src\LowCeremony.Tests\bin\Debug\netcoreapp2.0\LowCeremony.Tests.dll
[8/16/2018 10:29:14 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:14 PM Informational] Processing C:\dev\fixie.integration\src\MbUnitStyle.Tests\bin\Debug\netcoreapp2.0\MbUnitStyle.Tests.dll
[8/16/2018 10:29:15 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:15 PM Informational] Processing C:\dev\fixie.integration\src\Nested.Tests\bin\Debug\netcoreapp2.0\Nested.Tests.dll
[8/16/2018 10:29:15 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:15 PM Informational] Processing C:\dev\fixie.integration\src\NUnitStyle.Tests\bin\Debug\netcoreapp2.0\NUnitStyle.Tests.dll
[8/16/2018 10:29:16 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:16 PM Informational] Processing C:\dev\fixie.integration\src\Parameterized.Tests\bin\Debug\netcoreapp2.0\Parameterized.Tests.dll
[8/16/2018 10:29:17 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:17 PM Informational] Processing C:\dev\fixie.integration\src\Shuffle.Tests\bin\Debug\netcoreapp2.0\Shuffle.Tests.dll
[8/16/2018 10:29:18 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:18 PM Informational] Processing C:\dev\fixie.integration\src\Skipped.Tests\bin\Debug\netcoreapp2.0\Skipped.Tests.dll
[8/16/2018 10:29:18 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:18 PM Informational] Processing C:\dev\fixie.integration\src\Static.Tests\bin\Debug\netcoreapp2.0\Static.Tests.dll
[8/16/2018 10:29:19 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:19 PM Informational] Processing C:\dev\fixie.integration\src\xUnitStyle.Tests\bin\Debug\netcoreapp2.0\xUnitStyle.Tests.dll
[8/16/2018 10:29:20 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:20 PM Informational] Processing C:\dev\fixie.integration\src\CustomConvention.Tests\bin\Debug\net471\CustomConvention.Tests.dll
[8/16/2018 10:29:20 PM Informational] Skipping C:\dev\fixie.integration\src\Fixie.Integration\bin\Debug\netstandard1.3\Fixie.Integration.dll because it is not a test assembly.
[8/16/2018 10:29:20 PM Informational] Processing C:\dev\fixie.integration\src\x64.Tests\bin\Debug\net471\x64.Tests.dll
[8/16/2018 10:29:20 PM Informational] Processing C:\dev\fixie.integration\src\x86.Tests\bin\Debug\net471\x86.Tests.dll
[8/16/2018 10:29:21 PM Warning] A test with the same name 'CustomConvention.Tests.OverloadedTests.ShouldAdd' already exists. This test is not added to the test window.
[8/16/2018 10:29:21 PM Informational] ========== Discover test finished: 61 found (0:00:12.7393338) ==========
[8/16/2018 10:29:28 PM Informational] ------ Run test started ------
[8/16/2018 10:29:29 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:29 PM Informational] Processing C:\dev\fixie.integration\src\Async.Tests\bin\Debug\netcoreapp2.0\Async.Tests.dll
[8/16/2018 10:29:30 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:30 PM Informational] Processing C:\dev\fixie.integration\src\Categories.Tests\bin\Debug\netcoreapp2.0\Categories.Tests.dll
[8/16/2018 10:29:30 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:30 PM Informational] Processing C:\dev\fixie.integration\src\DefaultConvention.Tests\bin\Debug\netcoreapp2.0\DefaultConvention.Tests.dll
[8/16/2018 10:29:31 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:31 PM Informational] Processing C:\dev\fixie.integration\src\FSharp.Tests\bin\Debug\netcoreapp2.0\FSharp.Tests.dll
[8/16/2018 10:29:32 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:32 PM Informational] Processing C:\dev\fixie.integration\src\IoC.Tests\bin\Debug\netcoreapp2.0\IoC.Tests.dll
[8/16/2018 10:29:32 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:32 PM Informational] Processing C:\dev\fixie.integration\src\LowCeremony.Tests\bin\Debug\netcoreapp2.0\LowCeremony.Tests.dll
[8/16/2018 10:29:33 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:33 PM Informational] Processing C:\dev\fixie.integration\src\MbUnitStyle.Tests\bin\Debug\netcoreapp2.0\MbUnitStyle.Tests.dll
[8/16/2018 10:29:34 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:34 PM Informational] Processing C:\dev\fixie.integration\src\Nested.Tests\bin\Debug\netcoreapp2.0\Nested.Tests.dll
[8/16/2018 10:29:35 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:35 PM Informational] Processing C:\dev\fixie.integration\src\NUnitStyle.Tests\bin\Debug\netcoreapp2.0\NUnitStyle.Tests.dll
[8/16/2018 10:29:35 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:35 PM Informational] Processing C:\dev\fixie.integration\src\Parameterized.Tests\bin\Debug\netcoreapp2.0\Parameterized.Tests.dll
[8/16/2018 10:29:36 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:36 PM Informational] Processing C:\dev\fixie.integration\src\Shuffle.Tests\bin\Debug\netcoreapp2.0\Shuffle.Tests.dll
[8/16/2018 10:29:37 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:37 PM Informational] Processing C:\dev\fixie.integration\src\Skipped.Tests\bin\Debug\netcoreapp2.0\Skipped.Tests.dll
[8/16/2018 10:29:37 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:37 PM Informational] Processing C:\dev\fixie.integration\src\Static.Tests\bin\Debug\netcoreapp2.0\Static.Tests.dll
[8/16/2018 10:29:38 PM Informational] Fixie 2.0.0
[8/16/2018 10:29:38 PM Informational] Processing C:\dev\fixie.integration\src\xUnitStyle.Tests\bin\Debug\netcoreapp2.0\xUnitStyle.Tests.dll
[8/16/2018 10:29:39 PM Informational] ========== Run test finished: 50 run (0:00:10.1505517) ==========

[singhsarab]

@AbhitejJohn @ManishJayaswal As this seems to be an IDE related change.

[Shyam-Gupta]

@plioi Can you please disable following option in Visual Studio -> Tools -> Options -> Test dialog and see if it helps:

"For improved performance, only use test adapters in test assembly folder or as specified in runsettings file"

Note: Its only a workaround. We recommend having test adapter reference in each test project in the solution and that the test adapter dll should be present in test dll folder.

[tdmiller]

@plioi , @Shyam-Gupta -- I was having this same issue, but I just deselected the option indicated above and the tests are now showing up.

[plioi]

@Shyam-Gupta thank you! In the example solution above, disabling that option does seem to fix the issue.

As the creator of the test framework used by that solution, I'm curious about this change. Fixie deliberately avoids having a separate NuGet package just for the Test Adapter, because doing so would only encourage end users to get the version of it out of sync with the test framework itself. So, the single NuGet package includes both the test framework and its adapter. The test adapter dll isn't present in the test assembly folder because it's not referenced by anything there at runtime.

Is this new checkbox meant to avoid inspecting referenced NuGet packages for test adapters?

If there's a performance enhancement here, why was the discovery phase able to work anyway? Shouldn't all discovered tests be runnable?

Does this mean that all .NET test frameworks need to copy their test adapter dll into the build output folder of all test projects, even when the end user isn't using Test Explorer?

[Shyam-Gupta]

@tdmiller , @plioi
This change was done in VS 15.8 to improve test execution perf for C# and VB projects, by limiting the number of locations Test Platform has to lookup for test adapters.

More details about this change are available at:
https://docs.microsoft.com/en-us/visualstudio/releasenotes/vs2017-relnotes#testadapterextension

This change requires test adapter dll to be present along with test dll and that all test projects present in the solution should reference test adapter.

_If there's a performance enhancement here, why was the discovery phase able to work anyway? Shouldn't all discovered tests be runnable?_

Test Explorer discovers tests from source files instead of binaries for C# and VB projects. That's the reason test discovery works.

[ManishJayaswal]

Each project in the solution is expected to reference everything that it needs. Hence the test projects should reference the VS test adapter it needs for the tests to be properly discovered and executed in VS.

_Does this mean that all .NET test frameworks need to copy their test adapter dll into the build output folder of all test projects, even when the end user isn't using Test Explorer?

Yes if you would like the tests to be discovered and shown in TE windows and executed from there and you want test execution to be fast. If you do not want to use test explorer then you can ignore referencing the VS test adapters. If Test explorer window is still opened and source based discovery is on ( which it is by default) then test would still be discovered but they will not run. Our recommendation, if you plan to use test window or plan to execute tests from the editor context menu in VS, is to reference the testing framework and test adapter from every test project and make sure that they are copied over in the output folder which contains the test dll.

[plioi]

Thanks everyone for the troubleshooting and the background information!

The workaround works for me as well as coworkers who were experiencing the same issue, and now I know what changes to make to my own NuGet package to avoid the problem in the first place.