Sdk: Coould not run xunit tests for two frameworks in one project

[alexvaluyskiy]

Steps to reproduce

I have this csproj

<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" />
  <PropertyGroup>
    <TargetFrameworks>net46;netcoreapp1.0</TargetFrameworks>
    <AssemblyName>ProjectName</AssemblyName>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="**\*.cs" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Sdk">
      <Version>1.0.0-alpha-20161104-2</Version>
      <PrivateAssets>All</PrivateAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Test.Sdk">
      <Version>15.0.0-preview-20161123-03</Version>
    </PackageReference>
    <PackageReference Include="xunit">
      <Version>2.2.0-beta4-build3444</Version>
    </PackageReference>
    <PackageReference Include="xunit.runner.visualstudio">
      <Version>2.2.0-beta4-build1194</Version>
    </PackageReference>
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' == 'net46' ">
    <Reference Include="System" />
    <Reference Include="Microsoft.CSharp" />
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">
    <PackageReference Include="Microsoft.NETCore.App">
      <Version>1.0.1</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.NETCore.Portable.Compatibility">
      <Version>1.0.1</Version>
    </PackageReference>
    <PackageReference Include="Microsoft.TestPlatform.TestHost">
      <Version>15.0.0-preview-20161123-03</Version>
    </PackageReference>
  </ItemGroup>
  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
</Project>

Expected behavior

dotnet test --framework net46
dotnet test --framework netcoreapp1.0

Actual behavior

dotnet test --framework net46 - works fine
dotnet test --framework netcoreapp1.0 - does not work. I got an error

Starting test execution, please wait...
Error: Could not find testhost.dll for source 'C:\Projects\ProjectName\bin\Debug\netcoreapp1.0\ProjectName.dll'. Make sure test project has a nuget reference of package "microsoft.testplatform.testhost".

Environment data

.NET Command Line Tools (1.0.0-preview4-004123)

Product Information:
Version: 1.0.0-preview4-004123
Commit SHA-1 hash: 3a3ab7e0b0

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\1.0.0-preview4-004123

[vcsjones]

Same.

.NET Command Line Tools (1.0.0-preview4-004124)

Product Information:
Version: 1.0.0-preview4-004124
Commit SHA-1 hash: eb8e0cfa40

Runtime Environment:
OS Name: Mac OS X
OS Version: 10.12
OS Platform: Darwin
RID: osx.10.11-x64
Base Path: /usr/local/share/dotnet/sdk/1.0.0-preview4-004124

[alexvaluyskiy]

I've added these two lines and it started working

<OutputType>Exe</OutputType>
<RuntimeIdentifier>win7-x64</RuntimeIdentifier>

[TheRealPiotrP]

@Faizan2304 can you take a look?

[caleblloyd]

For me, <OutputType>Exe</OutputType> also fixed the issue

Migration should add this when migrating a project.json with a testrunner key

[livarcocc]

Before having migration add it, I want to make sure that this is the expected contract for test projects and not a bug that is ligering there. @Faizan2304 @codito can you guys help us understand this?

[codito]

@livarcocc the observation is correct. For netcoreapp1.x a test project should have <OutputType>Exe</OutputType>. For netXY targets, a test project should ideally be a library (the default OutputType). Exe output type will also work for netXY, though it adds an unnecessary complication of providing a platform bitness for test runner; in the IDE it defaults to x86 so win7-x64 will not run tests in IDE without user action.

For multitargeted test projects, the OutputType could be under a conditional. E.g. <OutputType Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">Exe</OutputType>.

[livarcocc]

Where is this coming from? Sounds like an unnecessary complication from the point of view of the csproj.

People will start with one TFM and then when they add a second, they should just have to add the TFM and nothing more.

Today, when they add a full framework, they need to add at least one RuntimeIdentifier, but I know that the folks in the SDK repo are trying to get rid of this requirement over there.

Isn't this something that we can do here as well?

[codito]

People will start with one TFM and then when they add a second, they should just have to add the TFM and nothing more.

+1. We will fix the Microsoft.NET.Test.Sdk package to take care of conditional OutputType.

[nambrose]

I had the same issue, and this fixed it but now when I run, no tests are detected (this is with Nunit though). Tests were running before I did the migrate to preview3

[codito]

@nambrose nunit tests should run for desktop targets (net4x). I think nunit-vs-adapter doesn't support netcoreapp yet (see https://github.com/nunit/nunit3-vs-adapter/issues/242).

Can you share the test project csproj?

[nambrose]

Sorry this took me a bit (it's my evening project)
They were running on preview 2 with a dotnet test (but not visible in Visual Studio for Mac)
When I upgraded to csproj I get zero tests run

the test project

Exe
1.1.0
netcoreapp1.1
portable
unittestcore
$(PackageTargetFallback);netcoreapp1.1;portable-net45+win8

1.1.0


1.0.0-alpha-20161104-2
All


3.5.0


3.4.0-beta-3


15.0.0-preview-20161024-02

{A70A8D16-4EC9-4EE8-BD7D-742BBA9C81EC}
core

and the library it is testing

netstandard1.6
portable
core








1.0.0-alpha-20161104-2
All




1.6.1



$(DefineConstants);RELEASE

[nambrose]

@codito I checked out the linked issue (thanks) and looks like I am SOL for now (irritating but not a huge deal and looks like the NUnit people also are in the dark)

I'll go back to finding out why my project works in VSCode but gets a 500 error in VS now I guess :) Feels a bit like I am in the wild west on Mac but I like the general direction (it's where I would have loved .NET to start out from the beginning)

[alexvaluyskiy]

I have a test project with two frameworks: net451 and netcoreapp1.0. And I want to run the tests on Windows and Linux.

If I want to run the tests on Windows+netcoreapp1.0 - I don't need to add any RuntimeIdentifiers. It works fine

But for Windows+net451 I have to add RuntimeIdentifier. But I could add only this line
<RuntimeIdentifier>win7-x64</RuntimeIdentifier>
not this
<RuntimeIdentifiers>win7-x64;ubuntu.16.04-x64</RuntimeIdentifiers>
because the CLI does not support the plural form for .NET 4.5 and I get an error in this case

C:\Program Files\dotnet\sdk\1.0.0-preview4-004233\Sdks\Microsoft.NET.Sdk\build\Microsoft.NET.RuntimeIdentifierInference.targets(49,5): error : RuntimeIdentifier must be set for .NETFramework executables. Consider RuntimeIdentifier=win7-x86 or RuntimeIdentifier=win7-x64

For Linux + netcoreapp1.0 I have to remove
<RuntimeIdentifier>win7-x64</RuntimeIdentifier>
and add
<RuntimeIdentifiers>win7-x64;ubuntu.16.04-x64</RuntimeIdentifiers>

For Linux + net451 it it not possible to build and test project at the moment.

Environment data

.NET Command Line Tools (1.0.0-preview4-004233)

Product Information:
Version: 1.0.0-preview4-004233
Commit SHA-1 hash: 8cec61c6f7

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\1.0.0-preview4-004233

[gulbanana]

you can use an msbuild Condition so that the RuntimeIdentifier is different for each tfm

[codito]

As of 87e9815a25a29fbfa8b87b5757bd44075ef6df4a, Microsoft.NET.Test.Sdk can handle OutputType mismatch between netcoreapp and net4x targets. There is no need to specify OutputType in the test project csproj.

Please try build/test with latest dotnet-cli dev build.

[alexvaluyskiy]

@codito Good, the first bug was fixed. But I still have to declare the different RuntimeIdentifiers for both frameworks, like this

<RuntimeIdentifiers>win7-x64;win7-x86;osx.10.10-x64;osx.10.11-x64;ubuntu.14.04-x64;ubuntu.16.04-x64;centos.7-x64;rhel.7.2-x64;debian.8-x64;fedora.23-x64;opensuse.13.2-x64</RuntimeIdentifiers>
<RuntimeIdentifier Condition=" '$(TargetFramework)' == 'net451' ">win7-x64</RuntimeIdentifier>

[codito]

But for Windows+net451 I have to add RuntimeIdentifier. But I could add only this line
win7-x64

@alexvaluyskiy is there an error if RID is not specified for this scenario? Assume this test project is a Library output type. Can you share the diagnostic log in that case? (dotnet test -d:log.txt, attach the log.*txt files)

[TheRealPiotrP]

@codito @alexvaluyskiy it looks like this issue is not getting traction. Can you confirm that this will be understood in the next few days? Otherwise we'll need to leave it out of the 1.0 release. We need a fix early next week.

[alexvaluyskiy]

I think the bug is fixed in rc3. In the previous version I had to add these lines to each test project

<OutputType Condition=" '$(TargetFramework)' == 'netcoreapp1.0' ">Exe</OutputType>
<RuntimeIdentifiers>win7-x64;win7-x86;osx.10.10-x64;osx.10.11-x64;ubuntu.14.04-x64;ubuntu.16.04-x64;centos.7-x64;rhel.7.2-x64;debian.8-x64;fedora.23-x64;opensuse.13.2-x64</RuntimeIdentifiers>
<RuntimeIdentifier Condition=" '$(TargetFramework)' == 'net451' ">win7-x64</RuntimeIdentifier>

[TheRealPiotrP]

Confirmed with @dotnet/vstest and confirmed that this is fixed.