Vstest: Solution with netcoreapp3 and net472 projects

Created on 24 Jan 2020 · 9Comments · Source: microsoft/vstest

Description

My projects have both kind of projects: net472 and netcore3.
Very simple sample is here:
vstestissue.zip

Steps to reproduce

In folder with vstestissue.sln run:

msbuild /t:restore;build
vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll

Expected behavior

  √ TestMethodnetcore [2ms]
  √ TestMethodnetfw [3ms]

Test Run Successful.
Total tests: 2
     Passed: 2
 Total time: 0,5982 Seconds

All tests runes successful. vstest group all binaries by framework and run group in suitable runtime.

Actual behavior

vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.0 and platform X86. Following DLL(s) do not match framework/platform settings.
netfw.dll is built for Framework .NETFramework,Version=v4.7.2 and Platform AnyCPU.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][.....\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '.....\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

  √ TestMethodnetfw [3ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 0,3968 Seconds

If I change order:

vstest.console.exe  netcore\bin\Debug\netcoreapp3.1\netcore.dll netfw\bin\Debug\net472\netfw.dll
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.0 and platform X86. Following DLL(s) do not match framework/platform settings.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
netfw.dll is built for Framework .NETFramework,Version=v4.7.2 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][...\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '...\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

[MSTest][Discovery][...\netfw\bin\Debug\net472\netfw.dll] Failed to discover tests from assembly ...\netfw\bin\Debug\net472\netfw.dll. Reason:Could not load file or assembly 'netfw' or one of its dependencies. The system cannot find the file specified.
No test is available in ...\netcore\bin\Debug\netcoreapp3.1\netcore.dll ...\netfw\bin\Debug\net472\netfw.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

echo %errorlevel%
0

If I add framework netcore3:

vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll "/framework:.NETCoreApp,Version=v3.1"

Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETCoreApp,Version=v3.1 and platform X64. Following DLL(s) do not match framework/platform settings.
netfw.dll is built for Framework .NETFramework,Version=v4.7.2 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
Unable to find ...\netfw\bin\Debug\net472\netfw.deps.json. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk".
  √ TestMethodnetcore [2ms]

Test Run Aborted.
Total tests: Unknown
     Passed: 1
 Total time: 0,7384 Seconds

echo %errorlevel%
1

If I change framework to 4.7.2 I get another behavior:

vstest.console.exe  netcore\bin\Debug\netcoreapp3.1\netcore.dll netfw\bin\Debug\net472\netfw.dll  "/framework:.NETFramework,Version=v4.7.2"
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.7.2 and platform X86. Following DLL(s) do not match framework/platform settings.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][...\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '...\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

[MSTest][Discovery][...\netfw\bin\Debug\net472\netfw.dll] Failed to discover tests from assembly ...\netfw\bin\Debug\net472\netfw.dll. Reason:Could not load file or assembly 'netfw' or one of its dependencies. The system cannot find the file specified.
No test is available in ...\netcore\bin\Debug\netcoreapp3.1\netcore.dll ...\netfw\bin\Debug\net472\netfw.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

echo %errorlevel%
0

if I change order of files and leave .net742:

vstest.console.exe  netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll   "/framework:.NETFramework,Version=v4.7.2"
Microsoft (R) Test Execution Command Line Tool Version 16.5.0-preview-20200110-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.7.2 and platform X86. Following DLL(s) do not match framework/platform settings.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.


A total of 2 test files matched the specified pattern.
[MSTest][Discovery][...\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source '...\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

  √ TestMethodnetfw [3ms]

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 0,4156 Seconds

echo %errorlevel%
0

Now I have to somehow separate the binaries myself and take two steps during CI.
I don't want add knowledge about type of projects (and names) to the CI, so I separate by analyzing the targetframeworkAttribute of the binary files and run each of group files in vstest.console.exe with suitable framework.

Environment

OS: win10: 1803..2004, winserver 2019
vstest.console: 16.4, 16.5.0-preview-20200110-02

Source

alexey-gusarov

👍3

Most helpful comment

Could you share how you work around the problem now? For me and others to see 🙂

The workaround is as simple as running vstest twice, for each platform with filters applied. The (file) filter for the .NET Core tests currently is '*/netcoreapp3.1/Tests.dll'. This prevents vstest from breaking on libraries that target .NET Framework. It doesn't feel like a very future proof workaround.

gormac on 9 Apr 2020

👍4

All 9 comments

@alexey-gusarov we've made some fixes in the TP recently that might have fixed this. Could you try if you can still reproduce this with 16.5.0, please?

You are probably aware, but the whole TP ships in this package and there is 16.5.0 available now. https://www.nuget.org/packages/Microsoft.TestPlatform/

nohwnd on 13 Feb 2020

@alexey-gusarov we've made some fixes in the TP recently that might have fixed this. Could you try if you can still reproduce this with 16.5.0, please?

Unfortunately behavior wasn't changed.
Only one test was started.

d:\temp\vstestissue>d:\vstestissue\Microsoft.TestPlatform.16.5.0\tools\net451Common7IDE\Extensions\TestPlatform\vstest.console.exe netfw\bin\Debug\net472\netfw.dll netcore\bin\Debug\netcoreapp3.1\netcore.dll
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.0 and platform X86. Following DLL(s) do not match framework/platform settings.
netfw.dll is built for Framework .NETFramework,Version=v4.7.2 and Platform AnyCPU.
netcore.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.

A total of 2 test files matched the specified pattern.
[MSTest][Discovery][d:\temp\as_coverage_outorigffff\vstestissue\netcore\bin\Debug\netcoreapp3.1\netcore.dll] Unable to load types from the test source 'd:\temp\as_coverage_outorigffff\vstestissue\netcore\bin\Debug\netcoreapp3.1\netcore.dll'. Some or all of the tests in this source may not be discovered.
Error: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

√ TestMethodnetfw [4ms]

Test Run Successful.
Total tests: 1
Passed: 1
Total time: 0,7876 Seconds

alexey-gusarov on 13 Feb 2020

We face the exact same problem. We have a .NET Standard project that is tested with a test project that multi-targets .NET Framework 4.7.2 and .NET Core 3.1. I can only get vstest to reliably run when I filter out the assembly files per framework under test, but I don't want to be dependent on the naming of files or folders to determine what is .NET Framework and what is .NET Core.

gormac on 9 Apr 2020

👍2

The problem here is that the code determines the frameworks, figures them all out correctly, and then it looks at the list of frameworks present in sources (the list of dlls you provided), and sees that there is a "conflict" and falls back to net framework.

As you see in your examples 1 and 2 the order of the assemblies does not matter. As soon as there is conflict you will get .net

If you force netcore to be used is will fail to run the .framework. And if you specify .net framework then the net core assembly will fail when it attempts to do reflection only load. So there is no way out at the moment.

Looking at the code I think this way of running is probably an old choice because I see that the code is outputting the list of assembly+framework pairs like 5 levels up by out parameters where it writes the warnings, probably in order to be more informative.

A correct approach here would be to determine the frameworks and start test host for each compatible group, with core projects starting one test host per project (because we need process boundary to isolate them, because we don't have app domains). Which sounds easy in theory, but won't be as simple to implement.

That said this looks like a rather large change in how it internally works, and I need to discuss it with others to consider the impact and get it prioritized. And this sprint that will start next week is already very full. I am sorry but you will have to suffer a bit longer with this.

Could you share how you work around the problem now? For me and others to see 🙂

nohwnd on 9 Apr 2020

One thing that is defninetely worth mentioning is that this is problem only when you run single instance vstest.console. You can run dotnet test on the solution (inferred automatically) and it will work. And is more comfy. Pair with --no-build if you already built:

vstestissue> dotnet test --no-build
Test run for C:\Users\jajares\Downloads\vstestissue\netcore\bin\Debug\netcoreapp3.1\netcore.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
Test run for C:\Users\jajares\Downloads\vstestissue\netfw\bin\Debug\net472\netfw.dll(.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,1682 Seconds

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,2179 Seconds

This works because dotnet test will start one console per project, which by itself is problematic, but makes this work.

nohwnd on 9 Apr 2020

Could you share how you work around the problem now? For me and others to see 🙂

gormac on 9 Apr 2020

👍4

@nohwnd

Could you share how you work around the problem now? For me and others to see slightly_smiling_face

I wrote simple script fo this:
https://gist.github.com/alexey-gusarov/050dcac7d9bb9f0a1c192142db57c367

alexey-gusarov on 10 May 2020

@alexey-gusarov Thank you for such clean repro.

It turned out my problem was different than this but you really saved me some time and properly framed the problem.

jzabroski on 4 Jan 2021

Replicated with latest release of TestFramework 16.0.20630.163001

```
Microsoft (R) Coverage Collection Tool Version 16.0.20630.163001
Copyright (c) Microsoft Corporation. All rights reserved.
2.5448
4.0171
A total of 2 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (32-bit Desktop .NET 4.0.30319.42000)
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (32-bit Desktop .NET 4.0.30319.42000)
[xUnit.net 00:00:12.40] Discovering: UICManager.Framework.Tests
[xUnit.net 00:00:12.64] Skipping: UICManager.Core.Tests (could not find dependent assembly 'System.Runtime, Version=4.2.2')
[xUnit.net 00:00:12.73] Discovered: UICManager.Framework.Tests
[xUnit.net 00:00:12.73] Starting: UICManager.Framework.Tests
No test is available in D:\a1\s\Templates\UICManager.Core.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

[xUnit.net 00:00:13.46] Finished: UICManager.Framework.Tests

Microsoft (R) Coverage Collection Tool Version 16.0.20630.163001
Copyright (c) Microsoft Corporation. All rights reserved.
Passed UICManager.Framework.Tests.Tests.AssignTest [471 ms]
Passed UICManager.Framework.Tests.Tests.ImportExportTest [1 ms]
Passed UICManager.Framework.Tests.Tests.GetHelpTest [1 ms]
Results File: D:\a_temp\TestResults\TestResults\VssAdministrator_fv-az659-292_2021-01-11_10_28_43.trx