Vstest: `dotnet test` does not work for projects with custom output path

Created on 6 Feb 2017  ·  26Comments  ·  Source: microsoft/vstest

Moved from https://github.com/dotnet/cli/issues/5582 on behalf of @alecor191

Steps to reproduce

  • Use VS 2017 with .csproj and create a .NET Core test project using XUnit
  • Make sure to add XUnit runner package to project:
<PackageReference Include="xunit" Version="2.2.0-beta5-build3474" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.2.0-beta5-build1225" />
  • Configure the .csproj to use a custom output path:
<BaseIntermediateOutputPath>..\..\artifacts\obj\$(MSBuildProjectName)\$(Configuration)\$(ProcessorArchitecture)\</BaseIntermediateOutputPath>
<OutputPath>..\..\artifacts\test\$(MSBuildProjectName)\$(Configuration)\$(ProcessorArchitecture)\</OutputPath>
  • Run the XUnit tests from VS2017 IDE
  • Run the XUnit tests using dotnet test

Expected behavior

Both VS2017 IDE and dotnet test should execute the XUnit tests in the test project.

Actual behavior

VS2017 IDE is able to execute the tests successfully. However, dotnet test fails with the following warning:

Warning: The path '[...]\artifacts\test\<test_project_name>\Debug\net461' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.

Additional info

I noticed the following:

  • My guess is that the execution with dotnet test fails as I can't find xunit.runner.visualstudio assemblies in the output path (xunit.runner.visualstudio.testadapter.dll is missing).
  • If I _remove_ the custom output path settings (i.e. remove both <BaseIntermediateOutputPath> and <OutputPath> from the .csproj file, then dotnet test succeeds. In fact, I see in the default bin directory the files from xunit.runner.visualstudio package (xunit.runner.visualstudio.testadapter.dll).
  • Note that other NuGet package assemblies are properly copied to the custom output directory. I assume those are the dependencies that are also referenced in code; whereas the XUnit runner/testadapter assemblies are not referenced anywhere in the solution.

Environment data

dotnet --info output:

.NET Command Line Tools (1.0.0-rc3-004530)

Product Information:
Version: 1.0.0-rc3-004530
Commit SHA-1 hash: 0de3338607

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x64
Base Path: C:Program Filesdotnetsdk1.0.0-rc3-004530

enhancement cli
Source
picture TheRealPiotrP
👍4 ❤1

All 26 comments

@livarcocc and I hit this last week in the https://github.com/dotnet/sdk when trying to port their tests to dotnet test. This should be an easy fix, looking for places where $(OutputPath) and $(IntermediateOutputPath)'s values are being assumed and using the property instead. It may even be a nice issue to mark as up for grabs in case @alecor191 wants to take a shot at the fix :)

TheRealPiotrP picture TheRealPiotrP on 6 Feb 2017
👍1

This is actually not a test SDK issue. @alecor191 is actually hitting https://github.com/dotnet/sdk/issues/802. @livarcocc and I hit the same thing. That issue causes the targets included in the test SDK nuget packages to not get picked up, and thereby the Main method to not get generated [amongst other issues].

I think this is something that the in-box test components may be able to check for and provide some diagnosability, though I'm not sure if that's necessary once https://github.com/dotnet/sdk/issues/802 is fixed. I'll leave it to the team to decide once they have a chance to think through the issue.

TheRealPiotrP picture TheRealPiotrP on 8 Feb 2017
👍1

I've the same issue when using NUnit (without any custom output path setting).

Error message is like:

C:\xxx\SimMetrics.Net\tests\SimMetrics.Net.Tests>dotnet test
Build started, please wait...
Build completed.

Test run for C:\xxx\SimMetrics.Net\tests\SimMetrics.Net.Tests\bin\Debug\net45\SimMetrics.Net.Tests.dll(.NETFramework,Version=v4.5)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

The path 'C:\xxx\SimMetrics.Net\tests\SimMetrics.Net.Tests\bin\Debug\net45' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.

Starting test execution, please wait...
No test discoverer is registered to perform discovery of test cases. Register a test discoverer and try again.

The csproj looks like:

<Project Sdk="Microsoft.NET.Sdk">

  <ItemGroup Condition=" '$(TargetFramework)' == 'net45' ">
    <ProjectReference Include="..\..\src\SimMetrics.Net\SimMetrics.Net.csproj" />
  </ItemGroup>

  <ItemGroup>
    <Reference Include="System" />
    <Reference Include="Microsoft.CSharp" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="NFluent" Version="2.0.0-alpha" />
    <PackageReference Include="NUnit" Version="3.6.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.7.0" />
  </ItemGroup>

</Project>
StefH picture StefH on 20 Mar 2017

@StefH I have the exact same problem. Did you figure this out?

jaymedavis picture jaymedavis on 30 Mar 2017

@StefH @jaymedavis does test project output path contain any *.TestAdapter.dll assemblies? e.g. C:\xxx\SimMetrics.Net\tests\SimMetrics.Net.Tests\bin\Debug\net45

codito picture codito on 30 Mar 2017

I'm also experiencing the same issue with xunit on osx. 

The path '/xxx/Chessie.Hopac/tests/Chessie.Hopac.Tests/bin/Release/net45' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.

fsproj:

<Project Sdk="FSharp.NET.Sdk;Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp1.1;net45</TargetFrameworks>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="**/*.fs" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="FSharp.Core" Version="4.1.*" />
    <PackageReference Include="FSharp.NET.Sdk" Version="1.0.*" PrivateAssets="All" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
    <ProjectReference Include="../../src/Chessie.Hopac/Chessie.Hopac.fsproj" />
    <DotNetCliToolReference Include="Microsoft.DotNet.Watcher.Tools" Version="1.0.0" />
  </ItemGroup>
</Project>
TheAngryByrd picture TheAngryByrd on 31 Mar 2017

I am using xunit as well. I believe the issue is it's looking at the x86 version of the sdk. I only installed the 64bit version originally. When I installed the x86 sdk, that took care of it.

I did a lot of things working on this problem yesterday, so let me know if this solves it 👍

jaymedavis picture jaymedavis on 31 Mar 2017

I think NUnit test projects are probably hitting https://github.com/nunit/nunit3-vs-adapter/issues/296. Please try this workaround:

<PropertyGroup>
  <DebugType Condition="'$(TargetFramework)' != '' AND '$(TargetFramework)' != 'netcoreapp1.0'">Full</DebugType>
</PropertyGroup>

@TheAngryByrd XUnit tests seem to have a requirement on net452 (from https://xunit.github.io/docs/getting-started-desktop.html). Can you please try with net452?

codito picture codito on 31 Mar 2017 
The path '/xxx/Chessie.Hopac/tests/Chessie.Hopac.Tests/bin/Debug/net452' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.

<Project Sdk="FSharp.NET.Sdk;Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netcoreapp1.1;net452</TargetFrameworks>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="**/*.fs" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="FSharp.Core" Version="4.1.*" />
    <PackageReference Include="FSharp.NET.Sdk" Version="1.0.*" PrivateAssets="All" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.core" Version="2.2.0" />
    <PackageReference Include="xunit.extensibility.execution" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
    <ProjectReference Include="../../src/Chessie.Hopac/Chessie.Hopac.fsproj" />
    <DotNetCliToolReference Include="Microsoft.DotNet.Watcher.Tools" Version="1.0.0" />
  </ItemGroup>
</Project>

With some debug flags turned on:

Test run for xxx/Chessie.Hopac/tests/Chessie.Hopac.Tests/bin/Debug/net452/Chessie.Hopac.Tests.dll(.NETFramework,Version=v4.5.2)
VSTest: Starting vstest.console...
VSTest: Arguments: dotnet exec "/usr/local/share/dotnet/sdk/1.0.1/vstest.console.dll" --framework:".NETFramework,Version=v4.5.2" "xxx/Chessie.Hopac/tests/Chessie.Hopac.Tests/bin/Debug/net452/Chessie.Hopac.Tests.dll" --testAdapterPath:"xxx/Chessie.Hopac/tests/Chessie.Hopac.Tests/bin/Debug/net452"
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

The path 'xxx/Chessie.Hopac/tests/Chessie.Hopac.Tests/bin/Debug/net452' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again.

And the output of the build

bin/Debug/net452/
├── Chessie.Hopac.Tests.dll
├── Chessie.Hopac.dll
├── Chessie.Hopac.pdb
├── Chessie.dll
├── FSharp.Core.dll
├── Hopac.Core.dll
├── Hopac.Platform.dll
├── Hopac.dll
├── System.ValueTuple.dll
├── xunit.abstractions.dll
├── xunit.assert.dll
├── xunit.core.dll
├── xunit.execution.desktop.dll
├── xunit.runner.reporters.net452.dll
├── xunit.runner.utility.net35.dll
└── xunit.runner.visualstudio.testadapter.dll
TheAngryByrd picture TheAngryByrd on 31 Mar 2017

@TheAngryByrd I believe your issue on osx is net452 doesn't exist there. You should only target netcoreapp1.1, as that's compatible with your mac

jaymedavis picture jaymedavis on 31 Mar 2017

Figured it's some mono issue. I can get it to build with mono with this https://github.com/dotnet/netcorecli-fsc/wiki/.NET-Core-SDK-1.0.1#using-net-framework-as-targets-framework-the-osxunix-build-fails just fine, but can't run the tests.

TheAngryByrd picture TheAngryByrd on 31 Mar 2017

For sake of completion. I also get this error:

Unhandled Exception: System.InvalidOperationException: StandardError has not been redirected.
   at System.Diagnostics.Process.get_StandardError()
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.<>c__DisplayClass9_0.<SetupChannel>b__0(Process process)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Helpers.ProcessHelper.<>c__DisplayClass0_0.<LaunchProcess>b__0(Object sender, EventArgs args)
   at System.Diagnostics.Process.OnExited()
   at System.Diagnostics.Process.RaiseOnExited()
   at System.Diagnostics.Process.CompletionCallback(Object context, Boolean wasSignaled)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)
Abort trap: 6

I'd be willing to look into making this work for mono. Any pointers you have would be appreciated. cc: @nosami @mrward

TheAngryByrd picture TheAngryByrd on 31 Mar 2017

@TheAngryByrd the adapters are correctly getting copied to output directory. This error is coming because vstest.console.dll is trying to read stderr of testhost.*.exe process (it would have launched this process to actually run the tests). More diagnostic information should be in log*.txt files generated with dotnet test -d:log.txt.

Possibly testhost.*.exe process is not getting launched in mono (since vstest.console just invokes testhost.exe without mono prefix, since it is not aware of mono runtime)? We can confirm it from log.txt file.

Created https://github.com/Microsoft/vstest/issues/679 to track support running tests on mono.

Update: added observations on ubuntu with mono in https://github.com/Microsoft/vstest/issues/679#issuecomment-290913772.

codito picture codito on 1 Apr 2017

Closing this issue since mono support is tracked in #679.

codito picture codito on 11 Jul 2017
😕1

@codito, the error is not mono specific, why did you close it due to mono support?

michael-freidgeim-webjet picture michael-freidgeim-webjet on 11 Jul 2017

Yeah I shouldn't have hijacked this thread, similar error message different cause.

TheAngryByrd picture TheAngryByrd on 11 Jul 2017

@michael-freidgeim-webjet my bad, reopening it. Thanks for correcting me.

codito picture codito on 12 Jul 2017

@alecor191 The issue is with msbuild and sdk and here is the solution of your problem https://github.com/dotnet/sdk/issues/980#issuecomment-285957284

You have to update your csproj similar like

<Project >
  <PropertyGroup>
    <BaseIntermediateOutputPath>..\..\artifacts\obj\$(MSBuildProjectName)\$(Configuration)\$(ProcessorArchitecture)\</BaseIntermediateOutputPath>
    <OutputPath>..\..\artifacts\test\$(MSBuildProjectName)\$(Configuration)\$(ProcessorArchitecture)\</OutputPath>
  </PropertyGroup>

  <Import Project="Sdk.props" Sdk="Microsoft.NET.Sdk" />

  <PropertyGroup>
    <TargetFramework>netcoreapp1.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
    <PackageReference Include="xunit" Version="2.2.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
  </ItemGroup>

  <Import Project="Sdk.targets" Sdk="Microsoft.NET.Sdk" />
</Project>
Faizan2304 picture Faizan2304 on 12 Jul 2017
👍1

@StefH the issue is with the Nunit nuget package NUnit3TestAdapter (Version="3.7.0"). I tried locally and it has been fixed with NUnit3TestAdapter (Version="3.8.0-alpha1")

Faizan2304 picture Faizan2304 on 12 Jul 2017

@michael-freidgeim-webjet Please do let me know if your issue is not getting resolve with above two comment.

Faizan2304 picture Faizan2304 on 12 Jul 2017

For my MSTest project (.NETFramework,Version=v4.6.1)
dotnet test
caused

Warning: The path binDebug' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again

I was able to successfully run the tests after installing:

Install-Package MSTest.TestAdapter
Install-Package MSTest.TestFramework
michael-freidgeim-webjet picture michael-freidgeim-webjet on 12 Jul 2017

@michael-freidgeim-webjet : dotnet restore should do the package installation part.

Did you tried command in this order
1) dotnet restore
2) dotnet test

Faizan2304 picture Faizan2304 on 18 Jul 2017

@Faizan2304 Why was this closed? The OP reported this issue using the xunit runner. And, I've tried the latest preview, 2.4.0-beta.2.build4010, and this issue still exists. Is it an xunit issue or a general testrunner issue?

MiYanni picture MiYanni on 30 Jun 2018
👍1

@MiYanni I tried the repro steps mentioned in the original issue filed . It seems to be working as expected.

  • Use VS 2017 with .csproj and create a XUnit .NET Core test project
  • Make sure to add XUnit runner package to project:
<PackageReference Include="xunit" Version="2.4.0" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.0" />
  • Configure the .csproj to use a custom output path:
<BaseIntermediateOutputPath>..\..\artifacts\obj\$(MSBuildProjectName)\$(Configuration)\$(ProcessorArchitecture)\</BaseIntermediateOutputPath>
<OutputPath>..\..\artifacts\test\$(MSBuildProjectName)\$(Configuration)\$(ProcessorArchitecture)\</OutputPath>
  • Run the XUnit tests from VS2017 IDE
  • Run the XUnit tests using dotnet test

I was able to run the tests with both VS IDE and dotnet test verb. Can you please confirm ?

VS version : 15.9.4
dotnet version : 3.0.100-preview-010184

singhsarab picture singhsarab on 27 Feb 2019

@singhsarab At the time that I commented on this, I was working on our build process for Azure PowerShell related to using Azure DevOps. I had tried a lot of different ways to modify our build. Unfortunately, I didn't even put enough context for myself to understand what I was referring to.

That said, I simply modified our test projects' output folder. I ran our build and it worked appropriately. For now, I'd close this issue. If I get back to modifying our process and run into this again, I'd give more details.

MiYanni picture MiYanni on 27 Feb 2019
👍1

Thanks @MiYanni I am closing this issue.

singhsarab picture singhsarab on 28 Feb 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

dlemstra picture dlemstra  ·  5Comments
adamralph picture adamralph  ·  4Comments
MoiMoiDG picture MoiMoiDG  ·  5Comments
alastairtree picture alastairtree  ·  4Comments
NotMyself picture NotMyself  ·  5Comments