Vstest: VSTest with dotnet core in VSO Test Assemblies Task fails with No Tests but ran 73 tests

[rchouser]

I have a dotnet core built test dll which I'm invoking with a Test.runsettings file and /Framework:FrameworkCore10. Locally I get 73 test passes and everything looks good. With a VSO Test Assemblies task I see the same 73 successful tests but I get a "no test" error.

A snippet of the log

2018-08-17T00:11:31.7533163Z vstest.console.exe "C:\BA\1\s\Applications\InputIntel\InputIntel.UnitTests\bin\x64\Release\netcoreapp2.0\InputIntel.UnitTests.dll"
2018-08-17T00:11:31.7533500Z "C:\BA\1\s\Applications\InputIntel\InputIntel.UnitTests\obj\x64\Release\netcoreapp2.0\InputIntel.UnitTests.dll"
2018-08-17T00:11:31.7533767Z /Settings:"C:\Users\issblder\AppData\Local\Temp\tmpED79.runsettings"
2018-08-17T00:11:31.7533959Z /Logger:"trx"
2018-08-17T00:11:31.7534280Z /TestAdapterPath:"C:\BA\1\s"
2018-08-17T00:11:31.7534452Z /Framework:FrameworkCore10
2018-08-17T00:11:32.0961863Z Starting test execution, please wait...
2018-08-17T00:11:32.2661460Z Test run will use DLL(s) built for framework .NETCoreApp,Version=v1.0 and platform X64. Following DLL(s) do not match framework/platform settings.
2018-08-17T00:11:32.2663263Z InputIntel.UnitTests.dll is built for Framework 2.0 and Platform X64.
2018-08-17T00:11:32.2663699Z InputIntel.UnitTests.dll is built for Framework 2.0 and Platform X64.
2018-08-17T00:11:32.2664444Z Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
2018-08-17T00:11:35.4549023Z Passed APIMLoggerVerifyBasic
2018-08-17T00:11:35.4841233Z Passed APIMLoggerVerifyWithSubPath
2018-08-17T00:11:36.8873913Z Passed VerifyReliableOperationWithTimeout
2018-08-17T00:11:38.7481183Z Passed VerifyReliableOperationInvalidOperationException
2018-08-17T00:11:39.2340740Z Passed VerifyReliableOperationFabricNotPrimaryException
2018-08-17T00:11:39.2342018Z Passed CapacityUnitEqualsNonNull
... MORE TESTS RUN ...
2018-08-17T00:11:58.1772380Z ##[error]No test is available in C:\BA\1\s\Applications\InputIntel\InputIntel.UnitTests\obj\x64\Release\netcoreapp2.0\InputIntel.UnitTests.dll. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk" and framework version settings are appropriate and try again.
2018-08-17T00:11:58.1978525Z Total tests: Unknown. Passed: 73. Failed: 0. Skipped: 0.
2018-08-17T00:11:58.1979241Z ##[error]Test Run Aborted.

This is breaking our current build so I'm trying to understand how to fix/workaround it. Any ideas?

Thanks
Rob

[Silic0nS0ldier]

I had an issue with a fresh .NET Core test project too, though the specific error was different. Given the otherwise identical behaviour however, it might be caused by the same issue.

2018-08-21T02:02:35.8219848Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" @D:\a\_temp\45224fb1-a4e6-11e8-9f9f-a1ac60f153da.txt
2018-08-21T02:02:38.7806743Z Microsoft (R) Test Execution Command Line Tool Version 15.7.2
2018-08-21T02:02:38.7806991Z Copyright (c) Microsoft Corporation.  All rights reserved.
2018-08-21T02:02:38.7807069Z 
2018-08-21T02:02:38.8071464Z vstest.console.exe 
2018-08-21T02:02:38.8072466Z "D:\a\1\s\MHB.AppCore.Tests\bin\Debug\netcoreapp2.0\MHB.AppCore.Tests.dll"
2018-08-21T02:02:38.8072724Z "D:\a\1\s\MHB.AppCore.Tests\obj\Debug\netcoreapp2.0\MHB.AppCore.Tests.dll"
2018-08-21T02:02:38.8072916Z /logger:"trx"
2018-08-21T02:02:38.8073111Z /TestAdapterPath:"D:\a\1\s"
2018-08-21T02:02:40.6455017Z Starting test execution, please wait...
2018-08-21T02:02:52.5257946Z [xUnit.net 00:00:06.0336750]   Discovering: MHB.AppCore.Tests
2018-08-21T02:02:52.6066254Z [xUnit.net 00:00:06.1198838]   Discovered:  MHB.AppCore.Tests
2018-08-21T02:02:52.6067412Z [xUnit.net 00:00:06.1273282]   Starting:    MHB.AppCore.Tests
2018-08-21T02:02:52.8461312Z [xUnit.net 00:00:06.3659427]   Finished:    MHB.AppCore.Tests
2018-08-21T02:02:53.0100921Z Passed   MHB.AppCore.Tests.Presenters.DiaryNewEntryPresenterTest.Cancel_CallsViewCancel
2018-08-21T02:02:53.0102464Z Passed   MHB.AppCore.Tests.Presenters.DiaryNewEntryPresenterTest.Submit_CallsViewSubmit
2018-08-21T02:02:53.0102771Z Passed   MHB.AppCore.Tests.Presenters.DiaryNewEntryPresenterTest.Ctor_Baseline
2018-08-21T02:02:53.0108299Z System.IO.FileNotFoundException: Unable to find tests for D:\a\1\s\MHB.AppCore.Tests\obj\Debug\netcoreapp2.0\MHB.AppCore.Tests.dll. Make sure test project has a nuget reference of package "Microsoft.NET.Test.Sdk" and framework version settings are appropriate. Rerun with /diag option to diagnose further.
2018-08-21T02:02:53.0109149Z    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager.GetTestHostProcessStartInfo(IEnumerable`1 sources, IDictionary`2 environmentVariables, TestRunnerConnectionInfo connectionInfo)
2018-08-21T02:02:53.0109464Z    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources)
2018-08-21T02:02:53.0109706Z    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)
2018-08-21T02:02:53.1742759Z Results File: D:\a\1\s\TestResults\VssAdministrator_factoryvm-az47_2018-08-21_02_02_52.trx
2018-08-21T02:02:53.1763753Z 
2018-08-21T02:02:53.1768800Z Total tests: Unknown. Passed: 3. Failed: 0. Skipped: 0.
2018-08-21T02:02:53.1769724Z Test Run Aborted.
2018-08-21T02:02:53.1777873Z Test execution time: 11.8239 Seconds
2018-08-21T02:02:53.2023716Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2018-08-21T02:02:53.2595666Z ##[error]Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1
2018-08-21T02:02:53.6615390Z ##[error]VsTest task failed.

[singhsarab]

@rchouser @Silic0nS0ldier
I think I understand the issue here.
The test dll is getting picked up from 2 locations, one is the bin directory which is the actual output directory, and other from the obj directory.

TestPlatform is not able to run the one from obj directory because of missing dependencies (deps file)

What is that the pattern you are using for Test files in the task ?
You should actually ignore obj folder like

**\*test*.dll
!**\*TestAdapter.dll
!**\obj\**

[rchouser]

@singhsarab - Thanks for looking! That's a great observation, I'll try adding the filter for the obj folder. This wasn't required prior to switching to dotnet core so I'd not thought to look for this.

[Silic0nS0ldier]

@singhsarab Your on the money regarding the task picking up obj. The docs at https://docs.microsoft.com/en-us/vsts/pipelines/tasks/test/vstest hinted at this, though the shown syntax resulted in nothing being picked up (**/*Tests.dll!**/obj/**).

Solution for me was to specify that bin must be in the path with **/bin/**/*Tests.dll.

[singhsarab]

@Silic0nS0ldier Those are two patterns, **/*Tests.dll and !**/obj/**, and should separated with new line. Give that a shot.

[Silic0nS0ldier]

Bear with me, YAML is my kryptonite. I originally tried;

- task: VSTest@2
  inputs:
    testAssemblyVer2: [
      '**/*Tests.dll',
      '!**/obj/**'
    ]

However VSTS complained about it not being a scalar type. But if I tried;

- task: VSTest@2
  inputs:
    testAssemblyVer2:
      '**/*Tests.dll'
      '!**/obj/**'

It would work?

[singhsarab]

- task: VSTest@2
  inputs:
    testAssemblyVer2: |
     **\*Tests.dll
     !**\obj\**

This should.

[rchouser]

I've confirmed this fixes my issue. I also noticed that the 2.* version of VSO unit task does in fact give you this template which excludes the obj file which is very useful.