Azure-pipelines-tasks: Run Test Assemblies Test Plan doesn't discover NUnit .Net Core unit tests

[OrangeTimes]

Troubleshooting

log.txt

Environment

VSTS: https://orangetimes.visualstudio.com/Sandbox
Build definition: Sandbox-ASP.NET-CI-TestPlan-CORE
Agent hosted: Hosted VS2017

Issue Description

1. I need to be able to run unit test associated with manual test cases using NUnit (or xUnit). I was able to get this working on .Net Framework by copying NUnitTestAdapter to c:Program Files (x86)Microsoft Visual Studio2017EnterpriseCommon7IDECommonExtensionsMicrosoftTestWindowExtensions
   as mentioned here https://stackoverflow.com/questions/36622707/error-while-executing-run-functional-test-task-in-vsts. But the same approach doesn't seem to work on .net core test build
   

Steps to reproduce

1. Create .net core 2.0 project add few Nunit tests. 1 that passes 1 that fails. Link them to Manual test case (using VSTS api)
2. Create new VSTS build defintion
3. Add Install Nuget + Restore Nuget + Build solution build steps
4. Add Test assemblies option. With Test plan execution type.
5. Set Test adapter path to nuget of NUnit3TestAdapter on hosted agent(C:UsersVssAdministrator.nugetpackagesnunit3testadapter3.9.0buildnetcoreapp1.0)

I tried also to copy nunit3testadapter to c:Program Files (x86)Microsoft Visual Studio2017EnterpriseCommon7IDECommonExtensionsMicrosoftTestWindowExtensions but that doesn't seem to help

Error logs

2018-01-25T16:41:25.0292099Z Count of test sources found: 1
2018-01-25T16:41:45.2685687Z Number of testcases discovered : 0
2018-01-25T16:41:45.2865205Z [RunStatistics]This execution slice with id '25', received '2' testcases to execute out of which '0' is discovered.

2018-01-25T16:41:45.6063039Z ##[error]ExecutionTaskStateModel.ExecuteTask: Recived error while executing task: System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2018-01-25T16:41:45.6063996Z ##[debug]Processed: ##vso[task.logissue type=error;]ExecutionTaskStateModel.ExecuteTask: Recived error while executing task: System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2018-01-25T16:41:45.6064548Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(CancellationToken cancellationToken) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\Execution.cs:line 59
2018-01-25T16:41:45.6064928Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(CancellationToken cancellationToken) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\ExecuteAndPublish.cs:line 35
2018-01-25T16:41:45.6065292Z    at Microsoft.VisualStudio.TestService.VstestAdapter.VstestAdapter.<>c__DisplayClass2_0.<ExecuteAsync>b__0() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\VstestAdapter.cs:line 55
2018-01-25T16:41:45.6065550Z    at System.Threading.Tasks.Task`1.InnerInvoke()
2018-01-25T16:41:45.6065725Z    at System.Threading.Tasks.Task.Execute()
2018-01-25T16:41:45.6065909Z    --- End of inner exception stack trace ---
2018-01-25T16:41:45.6066202Z ---> (Inner Exception #0) System.InvalidOperationException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2018-01-25T16:41:45.6066589Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(CancellationToken cancellationToken) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\Execution.cs:line 59
2018-01-25T16:41:45.6066945Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(CancellationToken cancellationToken) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\ExecuteAndPublish.cs:line 35
2018-01-25T16:41:45.6067283Z    at Microsoft.VisualStudio.TestService.VstestAdapter.VstestAdapter.<>c__DisplayClass2_0.<ExecuteAsync>b__0() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\VstestAdapter.cs:line 55
2018-01-25T16:41:45.6067521Z    at System.Threading.Tasks.Task`1.InnerInvoke()
2018-01-25T16:41:45.6067710Z    at System.Threading.Tasks.Task.Execute()<---

[OrangeTimes]

Any help on how to setup it to make it work would be appreciated. I also tried using vstest.console.exe located in C:Program Files (x86)Microsoft Visual Studio2017EnterpriseCommon7IDEExtensions
mentioned here https://blogs.msdn.microsoft.com/devops/2016/11/29/evolving-the-test-platform-part-3-net-core-convergence-and-cross-plat/ but receive next error

2018-01-30T15:15:50.8284141Z ##[error]ExecutionTaskStateModel.ExecuteTask: Recived error while executing task: System.AggregateException: One or more errors occurred. ---> System.TypeLoadException: Could not load type 'Microsoft.VisualStudio.TestPlatform.Common.FilterExpressionWrapper' from assembly 'Microsoft.VisualStudio.TestPlatform.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
2018-01-30T15:15:50.8284796Z ##[debug]Processed: ##vso[task.logissue type=error;]ExecutionTaskStateModel.ExecuteTask: Recived error while executing task: System.AggregateException: One or more errors occurred. ---> System.TypeLoadException: Could not load type 'Microsoft.VisualStudio.TestPlatform.Common.FilterExpressionWrapper' from assembly 'Microsoft.VisualStudio.TestPlatform.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
2018-01-30T15:15:50.8286121Z    at MS.VS.TestService.TestPlatform.v1.TestCaseFilter.ValidateFilter(String filterString)
2018-01-30T15:15:50.8286670Z    at MS.VS.TestService.TestPlatform.v1.TestCaseFilter.Initialize(String filterString) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.TestPlatform.v1\TestCaseFilter.cs:line 33
2018-01-30T15:15:50.8287240Z    at Microsoft.VisualStudio.TestService.VstestAdapter.DiscoverTests.PerformDiscovery(TestAutomationRunSlice sliceDetails, IAddTestCasesToTcmRunHandler addTestCasesToTcmRunHandler) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\DiscoverTests.cs:line 58
2018-01-30T15:15:50.8287759Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.FetchPreviousPhaseData() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\Execution.cs:line 79
2018-01-30T15:15:50.8288211Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Initialize() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\Execution.cs:line 200
2018-01-30T15:15:50.8288706Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(CancellationToken cancellationToken) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\ExecuteAndPublish.cs:line 34
2018-01-30T15:15:50.8289597Z    at Microsoft.VisualStudio.TestService.VstestAdapter.VstestAdapter.<>c__DisplayClass2_0.<ExecuteAsync>b__0() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\VstestAdapter.cs:line 55
2018-01-30T15:15:50.8290700Z    at System.Threading.Tasks.Task`1.InnerInvoke()
2018-01-30T15:15:50.8290910Z    at System.Threading.Tasks.Task.Execute()
2018-01-30T15:15:50.8291086Z    --- End of inner exception stack trace ---
2018-01-30T15:15:50.8291400Z ---> (Inner Exception #0) System.TypeLoadException: Could not load type 'Microsoft.VisualStudio.TestPlatform.Common.FilterExpressionWrapper' from assembly 'Microsoft.VisualStudio.TestPlatform.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
2018-01-30T15:15:50.8291688Z    at MS.VS.TestService.TestPlatform.v1.TestCaseFilter.ValidateFilter(String filterString)
2018-01-30T15:15:50.8291965Z    at MS.VS.TestService.TestPlatform.v1.TestCaseFilter.Initialize(String filterString) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.TestPlatform.v1\TestCaseFilter.cs:line 33
2018-01-30T15:15:50.8292577Z    at Microsoft.VisualStudio.TestService.VstestAdapter.DiscoverTests.PerformDiscovery(TestAutomationRunSlice sliceDetails, IAddTestCasesToTcmRunHandler addTestCasesToTcmRunHandler) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\DiscoverTests.cs:line 58
2018-01-30T15:15:50.8292918Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.FetchPreviousPhaseData() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\Execution.cs:line 79
2018-01-30T15:15:50.8293228Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Initialize() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\Execution.cs:line 200
2018-01-30T15:15:50.8293614Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(CancellationToken cancellationToken) in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\Phases\ExecuteAndPublish.cs:line 34
2018-01-30T15:15:50.8293992Z    at Microsoft.VisualStudio.TestService.VstestAdapter.VstestAdapter.<>c__DisplayClass2_0.<ExecuteAsync>b__0() in E:\v2.0\A7\_work\15\s\Ta\Agent\VS.TestService.VstestAdapter\VstestAdapter.cs:line 55
2018-01-30T15:15:50.8294243Z    at System.Threading.Tasks.Task`1.InnerInvoke()
2018-01-30T15:15:50.8294416Z    at System.Threading.Tasks.Task.Execute()<---

[acesiddhu]

@OrangeTimes few things
1) we don't support associating .net core test and nunit test with a test case.
2) for running .net core test you need to use .net core task. running netcore test via vstest task is not supported currently
3) also we should not copy nunit adapter to extension folder. that folder is meant for out of the box adapters that ship as part of VS

[OrangeTimes]

@acesiddhu Do you support .Net Standard then? When the support for .net core via vstest will be available?

[acesiddhu]

No we don't support net standard as well

[matthew-horrocks]

@acesiddhu

Do you have any idea when this will be supported?

Thanks

[zpv]

@acesiddhu
It has been a year and I'm still struggling to find a solution to this problem.
Hope it's going to be implemented soon!

[mijuva]

While at it, please add support for the native VSTest C++ Unit Test Framework, too.