Vstest: 'dotnet test' in solution folder fails when non-test projects are in the solution

[livarcocc]

_From @bungeemonkee on August 15, 2017 16:1_

Description

Being able to run 'dotnet test' on a solution now is amazing! However, unless that solution contains ONLY test projects it always produces a failure result due to not finding the test sdk. In my experience it is rare for tests to be in a separate solution - they generally don't exist or are in the same solution as the code they are testing.

It seems logical to me that when running tests on a solution any projects without the test sdk should be ignored. If a project has the test sdk and no tests or is being tested in isolation and does not have the sdk that is likely an error. However if a project does not have the sdk and is being tested as part of a solution that is likely not an error.

Alternatively, if that feels too much like a breaking change then adding an option to filter by patterns on the project name could suffice. Something like dotnet test --projects "*.Tests.*". Since most test projects in my experience have some form of pattern to the name - usually involving the word 'test' - this would probably be sufficient for most cases.

Note that the existing --filter <EXPRESSION> switch is insufficient to prevent the exception, presumably because it is processed by the underlying test framework which happens after the missing sdk error is thrown.

Steps to reproduce

'''
dotnet new sln -n Solution
dotnet new console -n Program
dotnet sln Solution.sln add Program/Program.csproj
dotnet new mstest -n Tests
dotnet sln Solution.sln add Tests/Tests.csproj
dotnet test
$?
'''

Expected behavior

1) Existing tests run.
2) Projects with no test sdk reference are ignored.
3) Return code is zero.

Actual behavior

1) Exiting tests run.
2) Projects with no test sdk reference are run as test projects and fail due to an error.
3) Return code is non-zero. In this case it appears to be 1 but I'm not sure if that is consistent.

Environment data

dotnet --info output:
.NET Command Line Tools (2.0.0)

Product Information:
Version: 2.0.0
Commit SHA-1 hash: cdcd1928c9

Runtime Environment:
OS Name: ubuntu
OS Version: 16.04
OS Platform: Linux
RID: ubuntu.16.04-x64
Base Path: /usr/share/dotnet/sdk/2.0.0/

Microsoft .NET Core Shared Framework Host

Version : 2.0.0
Build : e8b8861ac7faf042c87a5c2f9f2d04c98b69f28d

What is not clear form this environment output is that I am running in bash via the WSL.

_Copied from original issue: dotnet/cli#7447_

[livarcocc]

_From @bencyoung on September 4, 2017 14:26_

We have the same issue. A filter or the ability to point it at a sub-directory would be great

dotnet test tests/

or something like that...

[livarcocc]

_From @peterichens on September 4, 2017 15:17_

Same problem here, the error thrown causes our CI to report failure. Ignoring projects without the test sdk would be good, or at least just logging a message/warning, rather than an error.

[livarcocc]

_From @kendrahavens on September 6, 2017 17:7_

@pvlakshm Could you add the dotnet-test label?

[livarcocc]

_From @johncrim on September 22, 2017 18:10_

Same issue appears to be covered here: Allow dotnet test to run on all test projects in a given directory #705.

[danjamesmay]

Same issue here. Please implement dotnet test more intuitively.

[nicklong98]

Same issue. It's causing my jenkins build to fail

[beforan]

travis failing for me for the same reason

[jcdickinson]

The workaround I'm using right now is this script.

.\GenerateTestSolution.ps1 .\Foo.sln

It simply strips out projects that don't end with .Tests from a new copy of the solution file. To run the tests, do something along the lines of:

test Foo.Tests.sln -c %CONFIGURATION% --no-build

--no-build prevents build failures due to missing projects.

HTH

[dasMulli]

I'm currently using these msbuild targets in the solution directory to work around this:

A Directory.Build.targets file makes sure that all projects have a VSTestIfTTestProject target:

<Project>
  <Target Name="VSTestIfTestProject">
    <CallTarget Targets="VSTest" Condition="'$(IsTestProject)' == 'true'" />
  </Target>
</Project>

Then file named after.MySolutionName.sln.targets adds a VSTest target to the solution itself that dotnet test will call:

<Project>
  <Target Name="VSTest">
    <MSBuild Projects="@(ProjectReference)" Targets="VSTestIfTestProject" />
  </Target>
</Project>

[josmitc]

@jcdickinson That script worked really well. I was able to get my build process to run the tests without failing. Thanks!

[Troncek]

I was just about to write the same bug.

Seems this is not fixed yet in 2.0.3 and we have to use workarounds?

[consultwithmike]

UPDATE

Even executing that command wasn't working right so I had to build this bash script to do the job. Now I just execute the bash script.

#!/bin/bash

clear
for i in Tests/* ; do
  if [[ -d "$i" && "$i" == *.Tests ]]; then
    limit=$((${#i}+20))

    echo "Executing Tests for $i"
    printf '=%.0s' $(seq 1 $limit)
    echo ""

    dotnet test "$i" --verbosity quiet --no-build

    echo ""
  fi
done

---

The best way to solve this is to place all of your Tests project into their own folder, which is relatively good practice to keep large projects manageable. So, let's assume a structure like this:

- SolutionFolder
  - Project1
  - Project2
  - Tests
    - Project1.Tests
    - Project2.Tests

Now to run all tests you just use dotnet test Tests/**.

[bungeemonkee]

@consultwithmike I don't think it's reasonable to expect every developer or organization to restructure all their solutions to work around this. Only testing projects that have some test sdk as a dependency - or supporting some project name filter - seems to me like a much more reasonable expectation.

[consultwithmike]

@bungeemonkee I agree with you holistically because when I first encountered this I expected the same thing. But, since I had to work around it, I found this a very easy workaround that also made sense structurally.

So then it seemed to make sense that this was a good approach regardless.

[bungeemonkee]

@consultwithmike I'm glad you found a solution that works for you. I'm just making the point that the work-arounds are not viable options for everyone and that this is still an active issue for many of us.

[lilasquared]

@consultwithmike when I run dotnet test Tests/** I get Only one project can be specified

[consultwithmike]

@lilasquared you're absolutely right. I just found that out myself when I moved another test project into that directory. So, I ended up having to build this bash script to do the work for me.

#!/bin/bash

clear
for i in Tests/* ; do
  if [[ -d "$i" && "$i" == *.Tests ]]; then
    limit=$((${#i}+20))

    echo "Executing Tests for $i"
    printf '=%.0s' $(seq 1 $limit)
    echo ""

    dotnet test "$i" --verbosity quiet --no-build

    echo ""
  fi
done

[lilasquared]

@consultwithmike thanks for sharing. I have done something similar in the past when using nunit test runners. it's a shame that it is so complicated to do something as simple as run tests still. Since my application is small enough perhaps i will keep all the tests in one project.

[bobvandevijver]

While the current solution for @consultwithmike works, it might be convenient to run all your tests from a single command, to ease stat parsing. This can be achieved with dotnet vstest. However, you are required to specify every dll separately.

For my project, this results in the following command (I know, the folders and casing are wrong, which is what you get when you maintain a .NET project originally created by Java developers):

dotnet vstest modules/**.tests/bin/Release/**/**.tests.dll

It wil run all test in the dll's matching the argument selector, and give some final, accumulated output:

Total tests: 799. Passed: 797. Failed: 0. Skipped: 2.
Test Run Successful.
Test execution time: 50.5416 Seconds

This method does not appear to be working from PowerShell.

[dasMulli]

fyi I wrote a blog post about my solution (enables dotnet test without needing to implicitly build projects multiple times) https://dasmulli.blog/2018/01/20/make-dotnet-test-work-on-solution-files/

[consultwithmike]

Well played @dasMulli - well played!

[bobvandevijver]

@dasMulli Your method works, but it does result in several invocations of the test runner, leading to several output files and a longer execution time. It also does not generate an overview of all tests run in the end.

Due to this, I personally prefer the dotnet vstest solution as it results in a single output file, which runs faster and generates an overview at the end of execution.

[rsenna]

In powershell I am using this:

 dotnet vstest ((ls -Recurse *.Test.Unit.dll | % FullName) -Match "\\bin\\Release\\")

Meaning "run all tests recursively in all sub-folders, where assemblies are named *.Test.Unit.dll and their paths contain \bin\Release\".

Pretty sure it could be improved (by using proper directory filtering instead of string match, for instance), but it gets the job done.

[ErikSchierboom]

It would be great if this would work out of the work!

[lucastheisen]

The dotnet pack command looks for the <IsPackable> property. It would be nice if dotnet test looked for <IsTestable>. At face value, it seems like a really simple solution that would be quite valuable.

[ninjaboy]

guys, until this is properly fixed by MS, please have a look at @dasMulli solution above which works perfectly fine for me even considering that I have multiple solutions in the same folder

[Martin-Smith2]

Creating Directory.Build.targets won't work on RHEL.

[dasMulli]

Creating Directory.Build.targets won't work on RHEL

@MohsenBzm for all projects or specific to this issue? If the Directory.Build.targets feature doesn't work at all, do consider opening an issue over at https://github.com/Microsoft/msbuild with a repro.

First thing I ran into with Directory.Build.targets on *nix was that I checked in a version with a lowercase b instead of B which then worked only on windows..

[cltshivash]

@smadala Please pick this up for resolution as discussed.

[plioi]

Is this going to work by literally looking for references to the MS Test SDK package, or instead for the flags that project sets like so:

  <ItemGroup>
    <ProjectCapability Include="TestContainer" />
  </ItemGroup>

  <PropertyGroup>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

I strongly recommend it work by looking for those flags.

If it looks for the SDK reference itself, that'd be quite limiting for non-MSTest frameworks. Non-MSTest frameworks today can avoid the need for an explicit test SDK reference by including these flags in their own nuget packages, which opens the door to important features such as those Fixie has:

• First class F# support by including the generated entry point after all present *.fs files, so the developer doens't have to.
• Truly-executable test assemblies, which obviates all deprecated AppDomain usages in a test framework and allows for solutions with a mix of 32/64 bit test assemblies.

There's been at least some talk of xUnit 3 treating test assemblies as truly-executable too, and they'll either need to do the same (avoid the SDK) or else they'll be pushing for the SDK package to become more flexible such as no longer inserting the empty Main() entry points. If issues like this one work with the flags, xUnit and friends won't break dotnet test in the future.

[hudsonmendes]

If anybody here accepts a sh solution:
find . -type f -name '*-test.csproj' -exec dotnet test "{}" \;
My tests all end in -test.csproj, you're likely to have to customise it for yourself.

[dasMulli]

I'd still prefer a solution that would treat it as a single test run. all solutions involve invoking vstest multiple times, either through sh commands or looking at all projects with $(IsTestProject) in the solution (see my blog post).
But none them can:

• Synchronize access to shared resources implemented through constructs like fixtures implemented by the testing framework or manually (un-)locking of shared resources.
• Generate a single run result filej

[plioi]

@dasMulli What would a single test run really mean? Imagine your test project multitargets net47 and netcoreapp2. Those definitely have to be separate processes.

[smadala]

@plioi As @dasMulli suggestions. Single invocation of dotnet vstest.console.dll is way to go, To achieve accumulated test-results and leverage parallelism built in vstest.console.

Currently dotnet test/vstest/vstest.console.dll lunches separate process to run tests for each test project(At least for .NET Core test project). For multi target scenario we can have multiple trx files rather than one trx for each test project.

[hudsonmendes]

I agree with @plioi on the single dotnet test. It should be smart enough and combine the test results. The sh solution is a temporary fix until this is done properly on the cli.
Nonetheless, I'm using dotnetcore with Xunit.

[TFTomSun]

@livarcocc I ran into the same issue. There's a quite good workaround for that:
create a Directory.Build.targets in the root of your repository with the following content:

<Project>
  <!--Override VSTest target when the project is no test project.-->
  <Import Project="OverrideTest.targets" Condition="'$(IsTestProject)' != 'true'"/>
</Project>

create the file OverrideTest.targets beside the Directory.Build.targets

<Project>
  <Target Name="VSTest">
  </Target>
</Project>

what happens here is that the VSTest target is overridden as empty (nothing happens) when the project is not considered as test project by msbuild. A better solution (reusable across many repositories) is to put that into an build nuget package and just reference that package in your Directory.Build.targets, but unfortunately I can't publish mine on nuget.org

[OlegKarasik]

I know this could be a strange solution but I personally do the following.

1. Create a separate solution configuration (I usually name it "Test") using Visual Studio Configuration Manager (do not create project configurations, select to copy configuration from the configuration you use for build i.e. "Release").
2. Now set build checkboxes only for Test projects.
3. Execute dotnet test -c Test --no-build --no-restore on the solution

The CLI would run tests only for test projects.

Now when new Test project is added only a solution configuration has to be modified (no CI changes involved).

Hope this helps.

[gilescope]

+1 to the requirement of 'works out of the box'.

[L-Sypniewski]

For people who can't make it work with Travis:

It started returning code 0 when I changed my travis.yml file from

language: csharp
solution: MySolution.sln
mono: none
dotnet: 2.1.402
script:
- dotnet test TestFolder/testProj.csproj

to

language: csharp
solution: MySolution.sln
mono: none
dotnet: 2.1.402
script:
- cd TestFolder
- dotnet test

[BladeWise]

I noticed that using dotnet 2.1.500 doing this

dotnet test MySolution.sln

provides information that tests are skipped if they not include IsTestProject = true

Skipping running test for project xxx\src\xxx\MyProject.csproj. To run tests with dotnet test add "<IsTestProject>true<IsTestProject>" property to project file.

Was the issue already fixed with a release?
I stumbled on this issue in a Build Pipeline on Azure DevOps using Linux Ubuntu 1604 host, running a .Net Core task

Task         : .NET Core
Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command. For package commands, supports NuGet.org and authenticated feeds like Package Management and MyGet.
Version      : 2.141.5
Author       : Microsoft Corporation
Help         : [More Information](https://go.microsoft.com/fwlink/?linkid=832194)

but the same was not happening using a Linux Preview Host, reporting the same task information.

[johnnyasantoss]

To me it is working as expected on 2.1.500!
It looks for projects with <IsTestProject>true</IsTestProject> in the solution.

Finally, after more than 1 year waiting it works without workarounds! 🙂

[singhsarab]

@johnnyasantoss Thanks for confirming. Apologies, you had to wait this long, thanks for your patience.

We are prioritizing issues like these and are committed to improving the experience. We are open to contributions from the community as well, so we can expedite resolution for such issues.