Vstest: unable to discover netcoreapp1.1 tests.

[TheRealPiotrP]

Moving https://github.com/dotnet/cli/issues/5548#issuecomment-277313556 on behalf of @niemyjski

---

I am trying to upgrade my project from project.json to the new csproj format and hitting roadblocks left and right... (https://github.com/exceptionless/foundatio/tree/feature/dotnet)

It appears that VS2017 RC3 (Latest) is unable to discover or run any tests that have either netcoreapp1.0 or netcoreapp1.1.

When I try to run it from the command line I get

Test run for C:\Code\Foundatio\test\Foundatio.Tests\bin\Debug\netcoreapp1.1\Foundatio.Tests.dll(.NETCoreApp,Version=v1.1)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Warning: No test is available in C:\Code\Foundatio\test\Foundatio.Tests\bin\Debug\netcoreapp1.1\Foundatio.Tests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

Even more when I try to run with diagnostics

dotnet test .\test\Foundatio.Tests\Foundatio.Tests.csproj -v
Index was outside the bounds of the array.

but

dotnet test .\test\Foundatio.Tests\Foundatio.Tests.csproj --no-build /v
:diag

works but doesn't give me any diagnostics on why it couldn't connect to test host / run tests.

---

It seems to be failing to discover net standard 1.0 tests as well :(

---

I looked at some of your csproj's and versions + refs look ok. Do you perhaps have an old CLI version?

Regardless, @dotnet/vstest needs to take a look to help us see how we can shore up the UX.

---

I shouldn't as I am inside of a fresh vm with a freshly installed VS2017RC3 .03

---

If you pull down the repo and switch to the dotnet branch, you may need to add back the netcoreapp1.0 to one or more test projects ( I need to get this working on a ci ASAP)

---

fyi i checked out the feature/dotnet branch today and added back netcoreapp1.0 to Foundatio.Tests.csproj and just changed the xunit.assert package reference to xunit and then it worked.. it then also worked for netcoreapp1.1 (only tested in command line, not VS)

---

Thanks for this, shouldn't xunit be picked up as a transitive package?

---

I couldn't find any noticeable difference in the built output except for the xunit metapackage appearing in deps.json.
Even tough xunit appears to just be a meta-package, it seems to be required in order for test discovery to work.
Since you split xunit.core and xunit.assert packages across the logger project (p2p referenced) and the test project itself, the resulting dependencies are identical to the following test project which also doesn't discover tests:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp1.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170123-02" />
    <PackageReference Include="xunit.assert" Version="2.2.0-beta5-build3474" />
    <PackageReference Include="xunit.core" Version="2.2.0-beta5-build3474" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0-beta5-build1225" />
  </ItemGroup>

</Project>

I'm unsure if this is expected or an issue with either vstest or the xunit test runner.

---

Yeah I added xunit and now it works so some kind of issue here

[TheRealPiotrP]

I've had a few experiences like this. The dotnet test experience seems to have a number of error cases where errors are ignored/captured and not passed on to the end-user. We need to brainstorm a bit regarding how we can pass errors/missing dependency information/next steps onto users to help them get test projects into a good state.

[niemyjski]

I had more luck after I imported xunit nuget package instead of xunit.core packages into my test framework. This used to work so I think something might be broken someplace.

[codito]

Marking this as blocking-release since diagnostics is key for rtw.

Two hypothesis need to be verified:

• Try this scenario with fix for #300, in case the adapter is printing relevant error messages.
• If the testhost process doesn't start due to dependency missing, fix for https://github.com/Microsoft/vstest/issues/281 may help

[codito]

Update: the tests are not discovered due to incorrect PackageReference. On dotnet-cli latest, following commands work as expected:

> dotnet new xunit
> # edit csproj to target netcoreapp1.1
> dotnet restore
> dotnet test

[jimmystridh]

This simple example however will not work:

mkdir web && mkdir tests && cd web

dotnet new webapi --framework netcoreapp1.1

cd ../tests

dotnet new xunit --framework netcoreapp1.1

dotnet add reference ../web/web.csproj

dotnet restore

dotnet test

gives me

Build started, please wait...
Build completed.

Test run for
/tests/bin/Debug/netcoreapp1.1/tests.dll(.NETCoreApp,Version=v1.1)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
Warning: No test is available in /tests/bin/Debug/netcoreapp1.1/tests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

dotnet cli version
Version : 1.1.0
Build : 928f77c4bc3f49d892459992fb6e1d5542cb5e86

framework version
1.0.0-rc4-004771

[smadala]

@jimmystridh xunit < 2.2.0-beta5 not compatible with netcoreapp1.1, and Microsoft.NET.Test.Sdk < 15.0.0-preview-20170210-02 doesn't provide error message from xunit runner to end user.

Please update PackageReferences to following:

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170210-02" />
    <PackageReference Include="xunit" Version="2.2.0-rc2-build3523" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0-rc2-build1249" />
  </ItemGroup>

[jimmystridh]

thanks a lot @smadala , that indeed helped in both the example above and my real project

[niemyjski]

We merged in the dotnet branch of foundatio and I'm going to try this today (I know we updated to rc2) but I'm having a heck of my time for my tests to be discovered still

https://github.com/exceptionless/foundatio

[ejsmith]

We are already using those package versions in Foundatio @niemyjski and the test runner is still very flakey for us.

[niemyjski]

Yep can confirm, very flakey on our solutions. Please run them and try it... it would be greatly appreciated as it doesn't work on some projects like AWS 100% of the time and others is very flakey.

[smadala]

@niemyjski Seems like testplatform dependency not added to all test projects. After adding them test getting discovered and running. Few tests are failing on my machine with RedisConnectionException.

Please apply following patch and verify.
add-testplatform-deps.patch

[niemyjski]

@smadala I shouldn't have to define those. They are transitive and defined in foundatio.tests project which is referenced

[dasMulli]

I'm facing an interesting issue where no tests are discovered on Mac OS, but it's running just fine on windows and linux containers. Attached a small repo (including restore & build output).
It is an empty console and an empty xunit project template extended with the package references I have in a real app.
testrepro.zip

.NET Command Line Tools (1.0.0-rc4-004788)

Product Information:
 Version:            1.0.0-rc4-004788
 Commit SHA-1 hash:  f2ad5a39ab

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /usr/local/share/dotnet/sdk/1.0.0-rc4-004788

[smadala]

@niemyjski Netcore test project need direct reference of Microsoft.NET.Test.Sdk package to successful run tests. Otherwise tests fail with Error: Could not find testhost.dll for source '<path_to>\Foundatio\test\Foundatio.AWS.Tests\bin\Debug\netcoreapp1.1\Foundatio.AWS.Tests.dll'. Make sure test project has a nuget reference of package "microsoft.testplatform.testhost".

Which is misleading, Issue: https://github.com/Microsoft/vstest/issues/341

[smadala]

Tried migration for following projects:
core-setup: master
ApplicationInsights-aspnetcore:develop
Foundatio:tree/78793fe646fd7f845b5bf697916275045f7dadde

Migrate Issues:
Core-setup
Compilation error: Need to add package(Microsoft.DotNet.ProjectModel) reference manually.
Test ran.

ApplicationInsights-aspnetcore
Compilation error: Add Reference to System.Net.Http to net451 framework manually.
Test ran.

Foundatio
Took clean commit having no migration changes, it has only net46 TF, Migrated successfully,
Test ran.

Runtime Environment:
Microsoft Visual Studio Enterprise 2017
Version 15.0.26212.0 D15REL
Microsoft .NET Framework
Version 4.6.01586

dotnet --info
.NET Command Line Tools (1.0.0-rc4-004823)

Product Information:
Version: 1.0.0-rc4-004823
Commit SHA-1 hash: 4634d0e120

OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Users\samadala\src\vstest\tools\dotnet\sdk\1.0.0-rc4-004823

[niemyjski]

@smadala you'll need to enable the tests for netcoreapp1.1 on all test projects but Foundatio.Azure.Tests, we disabled them because we couldn't detect our tests otherwise in vs.

So you are also saying that the test reference isn't transitive? That really sucks I have to add it to everyone one of my test projects when I already have a reference to it via Foundatio.Tests. Also, why does it currently work on some projects but not others? That doesn't make no sense if it's not transitive, it should only then work on Foundatio.Tests.

[smadala]

Currently recommend approach for adding package reference to multiple projects is using targets. I guess tests running only for net46. Can you please verify that. If not please provide respective project.

[niemyjski]

All net46 tests will run From command line but only a few are detected within VS. We have had a ton of issues with both netcoreapp1.1 and net46 both set on the test projects (we want to run the tests for both). This causes tests to not be discoverable or ran from command line (in some cases) or vs.

[niemyjski]

And yes, currently all test projects are only targeting net46 because otherwise we'd have serious test issues so we removed .netcoreapp from the test projects until this is resolved.

[smadala]

Are tests still failing after adding direct reference to Microsoft.Net.Test.Sdk? If so, can you please provide the project name.

[niemyjski]

@smadala I'll try it and get back to you, but I'm not happy about it. I don't want to have to define it again in every test project even though my referenced project has it and then have to update it in every test project every time the sdk revs.... I want clean csproj files and not a million files changing when I upgrade deps.

[smadala]

@niemyjski Totally agreed on having clean csproj, can please raise issue for it.

[smadala]

@dasMulli Able to repo tests not discovered for provided project. Looks like testhost failing with error

'/Users/vstest/Downloads/testrepro/tests/bin/Debug/netcoreapp1.1/Microsoft.DiaSymReader.Native.amd64.dll'. A device attached to the system is not functioning.
 (Exception from HRESULT: 0x8007001F)
File name: '/Users/vstest/Downloads/testrepro/tests/bin/Debug/netcoreapp1.1/Microsoft.DiaSymReader.Native.amd64.dll' ---> System.Runtime.InteropServices.COMException (0x8007001F): A device attached to the system is not functioning.
 (Exception from HRESULT: 0x8007001F)
   at System.Runtime.Loader.AssemblyLoadContext.nGetFileInformation(String s)
   at Microsoft.VisualStudio.TestPlatform.Common.Utilities.AssemblyResolver.OnResolve(AssemblyLoadContext loadContext, AssemblyName args) 
TpTrace Information: 0 : 2596, 3, 2017/02/17, 00:37:19.419, 2192148601555, testhost.dll, AssemblyResolver: Microsoft.DiaSymReader.Native.x86: Resolving assembly.
TpTrace Information: 0 : 2596, 3, 2017/02/17, 00:37:19.420, 2192149147009, testhost.dll, AssemblyResolver: Microsoft.DiaSymReader.Native.x86: Failed to load assembly. Reason:System.IO.FileLoadException: Could not load file or assembly '/Users/vstest/Downloads/testrepro/tests/bin/Debug/netcoreapp1.1/Microsoft.DiaSymReader.Native.x86.dll'. A device attached to the system is not functioning.
 (Exception from HRESULT: 0x8007001F)
File name: '/Users/vstest/Downloads/testrepro/tests/bin/Debug/netcoreapp1.1/Microsoft.DiaSymReader.Native.x86.dll' ---> System.Runtime.InteropServices.COMException (0x8007001F): A device attached to the system is not functioning.
 (Exception from HRESULT: 0x8007001F)
   at System.Runtime.Loader.AssemblyLoadContext.nGetFileInformation(String s)
   at Microsoft.VisualStudio.TestPlatform.Common.Utilities.AssemblyResolver.OnResolve(AssemblyLoadContext loadContext, AssemblyName args) 
TpTrace Information: 0 : 2596, 3, 2017/02/17, 00:37:19.420, 2192149428634, testhost.dll, AssemblyResolver: xunit.runner.reporters.netstandard11: Resolving assembly.
TpTrace Information: 0 : 2596, 3, 2017/02/17, 00:37:19.420, 2192149943982, testhost.dll, AssemblyResolver: xunit.runner.reporters.netstandard11: Failed to load assembly. Reason:System.IO.FileLoadException: Could not load file or assembly '/Users/vstest/Downloads/testrepro/tests/bin/Debug/netcoreapp1.1/xunit.runner.reporters.netstandard11.dll'. A device attached to the system is not functioning.
 (Exception from HRESULT: 0x8007001F)
File name: '/Users/vstest/Downloads/testrepro/tests/bin/Debug/netcoreapp1.1/xunit.runner.reporters.netstandard11.dll' ---> System.Runtime.InteropServices.COMException (0x8007001F): A device attached to the system is not functioning.
 (Exception from HRESULT: 0x8007001F)
   at System.Runtime.Loader.AssemblyLoadContext.nGetFileInformation(String s)
   at Microsoft.VisualStudio.TestPlatform.Common.Utilities.AssemblyResolver.OnResolve(AssemblyLoadContext loadContext, AssemblyName args) 

log.txt
log.host.17-02-17_00-37-18_20929_5.txt

Will try to find root cause.

[codito]

@niemyjski wrote:

@smadala I'll try it and get back to you, but I'm not happy about it. I don't want to have to define it again in every test project even though my referenced project has it and then have to update it in every test project every time the sdk revs

This sounds like the use case for a .props and .targets file. This file could be imported in all projects, and may provide the <PackageReference Include="Microsoft.NET.Test.Sdk" />.

An example in vstest repo:
https://github.com/Microsoft/vstest/blob/master/test/Microsoft.TestPlatform.CoreUtilities.UnitTests/Microsoft.TestPlatform.CoreUtilities.UnitTests.csproj#L7

https://github.com/Microsoft/vstest/blob/master/scripts/build/TestPlatform.Settings.targets#L78

Test platform looks for the dependencies (NET.Test.Sdk and it's dependency TestHost) in deps file before discovering/executing tests. I am not sure if meta packages flow transitively. @srivatsn @piotrpMSFT @livarcocc any idea?

[niemyjski]

@smadala so I added it to all my test projects which made them discoverable in vs but still having some issues (will log them in two separate issues on this repo). But how do I know what they are running under (net46 or netcoreapp1.1)?

[niemyjski]

Here is a link to the other issue I am seeing (which was closed by @smadala in December) https://github.com/Microsoft/vstest/issues/280

[smadala]

@dasMulli Update for tests failing on MacOS, looks like issue related Swashbuckle.AspNetCore 1.0.0-rc1 reference. Removing same reference makes tests passed. Raised issue https://github.com/Microsoft/vstest/issues/516.

[smadala]

But how do I know what they are running under (net46 or netcoreapp1.1)?

Running tests for MTF(Multiple Target Framework): https://github.com/Microsoft/vstest/issues/298, Currenlty does not support from IDE.

[smadala]

Following reference should fix the issue.

<ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170210-02" />
    <PackageReference Include="xunit" Version="2.2.0-rc2-build3523" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0-rc2-build1249" />
  </ItemGroup>

Tracking related bugs here:
Misleading error message when test project is missing test runner.
Microsoft.NET.Test.Sdk package is not transitive and is required by every test project.
On MacOS no tests discovered after adding reference to Swashbuckle.AspNetCore-1.0.0-rc1 in product code