Vstest: Allow dotnet test to run on all test projects in a given directory

[livarcocc]

_From @sunsided on May 19, 2016 17:6_

For continuous integration, I want to be able to run dotnet test on all test projects in the test directory of my solution, without having to explicitly configure each subdirectory manually, by e.g. issuing

dotnet test --all

or

dotnet test --all-from test

from the solution directory.

Steps to reproduce

In the solution directory, run

dotnet test

Expected behavior

Runs all tests in test subdirectory.

Actual behavior

dotnet-test Error: 0 : System.InvalidOperationException: C:\solution-dir\project.json does not exist.
   at Microsoft.DotNet.Tools.Test.TestCommand.GetProjectPath(String projectPath)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

Environment data

.NET Command Line Tools (1.0.0-preview1-002702)

Product Information:
 Version:     1.0.0-preview1-002702
 Commit Sha:  6cde21225e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.10586
 OS Platform: Windows
 RID:         win10-x64

_Copied from original issue: dotnet/cli#3136_

[livarcocc]

_From @blackdwarf on May 21, 2016 17:35_

@livarcocc @piotrpMSFT thoughts?

[livarcocc]

_From @piotrpMSFT on May 26, 2016 7:52_

We need to think about this in the context of the new project model design...

[livarcocc]

_From @bojingo on June 30, 2016 23:24_

Is there even a reason for the --all flag? I can be in the solution folder and do a simple dotnet restore and all projects restore, based on looking for project.json files in the subfolders. I would _love_ dotnet test to work this same way. I have a number of test projects and my VSTS build definition is littered with tasks for running tests.

There could be some evaluation of the project.json file to ensure it is a valid test project, such as ensuring there is a testRunner property, such as "testRunner": "xunit".

[livarcocc]

_From @dazinator on July 5, 2016 15:14_

There could be some evaluation of the project.json file to ensure it is a valid test project, such as ensuring there is a testRunner property, such as "testRunner": "xunit".

Yes please! Just setting up a CI build, and having to CD into each test project directory to run dotnet test is a maintainance headache, as I keep having to revisit this build script as new test projects are added.

[livarcocc]

_From @unjello on October 5, 2016 13:52_

Any progress on that?

[livarcocc]

_From @MarinAtanasov on November 5, 2016 8:44_

Fully agree. If I want to run multiple projects from PowerShell, the only way I've found is:

ForEach ($folder in (Get-ChildItem -Path test -Directory)) { dotnet test $folder.FullName }

And the results are not easily readable because it runs every project separately.

Also, somewhat related, we should be able to run all projects in a single run. Even in VS, every project seems to have a separate loading time (about 2 seconds) which slows down the test run when you have multiple test projects.

[livarcocc]

_From @glucaci on December 13, 2016 10:36_

For example the dotnet build command accept agrument **/project.json and builds all the projects from subdirectories.
dotnet build **/project.json

I think this will be a good solution for dotnet test also.
dotnet test **/project.json

Actualy cannot parse the argument:

dotnet-test Error: 0 : System.InvalidOperationException: **/project.json does not exist.
   at Microsoft.DotNet.Tools.Test.TestCommand.GetProjectPath(String projectPath)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

[livarcocc]

_From @petrsvihlik on January 14, 2017 19:59_

@MarinAtanasov thanks! i've adjusted it a little bit for my project to run only in folders containing tests and to output the test results...

ForEach ($folder in (Get-ChildItem -Path .\ -Directory -Filter *.Tests)) { dotnet test $folder.FullName -xml ($folder.FullName + "\xunit-results.xml") }

it'd be great if it supported running tests for multiple projects OOTB

[livarcocc]

_From @StrangeWill on April 11, 2017 16:28_

Just ran into this today, we'd been using PowerShell for awhile but the problem comes when we want to test under dotCover and TeamCity (now supported on 2017.1 for .NET Core), to do that through PowerShell would be a pain...

If dotnet supported running all tests out of the box this would be a cakewalk.

[MarinAtanasov]

There is a way to run multiple projects with a single command which returns a single summary for the results. However, you must build the projects/solution first.

dotnet build SOLUTION.sln

After this, you can use _dotnet-vstest_ to execute the tests. In my case, all projects containing tests are under _/test_ subdir:

dotnet vstest (Get-ChildItem test | % { Join-Path $_.FullName -ChildPath ("bin/Debug/netcoreapp1.1/$($_.Name).dll") })

You don't need to use Powershell. You just need the path to all test assemblies. For example:

dotnet vstest .\Project1\bin\netcoreapp1.1\Project1.dll .\Project2\bin\netcoreapp1.1\Project2.dll

You can also add --Parallel to discover/run tests in parallel.
For more information about _dotnet-vstest_, check the official documentation.

It will be nice to have a way to run the tests using _dotnet-test_ while pointing to the solution file similar to building the solution in the first code snippet.

[aredfox]

Please supply a common dotnet test *.sln just like dotnet build *.sln - as this is really what ux-wise is what people want..

[riezebosch]

Strange thing is this works on dotnet core 2.0.2 on my mac:

dotnet test src\**.Tests
Build started, please wait...
Build completed.

Test run for
xxxx/src/Service.Tests/bin/Debug/netcoreapp2.0/Service.Tests.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.5794770]   Discovering: Service.Tests
[xUnit.net 00:00:00.6828850]   Discovered:  Service.Tests
[xUnit.net 00:00:00.7291560]   Starting:    Service.Tests
[xUnit.net 00:00:05.8853170]   Finished:    Service.Tests

Total tests: 8. Passed: 8. Failed: 0. Skipped: 0.

And in also in PowerShell Core on my macbook:

dotnet test src\**.Tests

But this doesn't work on dotnet core 2.0.3 in PowerShell on my Windows machine:

dotnet test src\**.Tests
MSBUILD : error MSB1009: Project file does not exist.
Switch: src\**.Tests

[Kantuz001]

Seriously, we have Linux (RHEL) ininfrastructure (bitbucket, jira, docker, jenkins) and basic commands of core cli (dotnet test) don't work in a proper way. We have no opportunity to get code coverage of solution on Linux (https://github.com/Microsoft/vstest/issues/981, https://github.com/dotnet/corefx/issues/17107). So, dotnet core is cross platform only on paper.

[EChaffraix]

@Kantuz001 I had the same problem with teamcity and found a solution. I run dotnet test --logger:trx

On linux, to run only tests projects, I use the following command :

ls *Tests*/*.csproj | xargs -L1 dotnet test --logger:trx

[VictorioBerra]

It would be nice to create a solution just for test projects and do dotnet test tests.sln

[Kantuz001]

@EChaffraix our workaround, we use groovy

case Constant.BUILD_TOOL_DOTNET:
                List csprojTests = this.getShellStdout('ls -d $(pwd)/test/*').split('\n')
                csprojTests.each { csprojTest ->
                    script.sh "dotnet test $csprojTest"
                }
                break

[VictorioBerra]

Does this append test results to a final result file?

On Jan 9, 2018 5:13 AM, "Dmitriy Voznesenskiy" [e-mail]
wrote:

@EChaffraix https://github.com/echaffraix our workaround, we use groovy
case Constant.BUILD_TOOL_DOTNET:
List csprojTests = this.getShellStdout('ls -d $(pwd)/test/*').split('\n')
csprojTests.each { csprojTest ->
script.sh "dotnet test $csprojTest"
}
break

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vstest/issues/705#issuecomment-356254840,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACzG69rjysF8TWvEvPLKknc2RJ0eKOJTks5tI0nBgaJpZM4M6XON
.

[altinoren]

As a workaround, this PS snippet works for us:

Get-ChildItem ".\test\" -recurse | where {$_.extension -eq ".csproj"} | % { dotnet test $_.FullName }

[StrangeWill]

Yeah, I'm not a fan of having to piggy-back on various scripting platforms (bash/powershell) because they're not portable (outside of Powershell Core, and I'm not a fan of pushing Powershell as a must on Linux for proper dotnet support).

This concept should really be supported by our tool chain and not hacked around.

Also not a fan of leaving it up to dotnet vstest because this needs to be supported by all test runners.

Many other actions can be taken at the solution level (restores, builds), I don't see why tests should be any different (from a UX standpoint, I understand the logistical differences on compiling tests into a larger unit of work).

[cfletcher]

@altinoren the problem with that script is that you're not handling the failure of tests in an individual project. You're running the tests, displaying the output and ignoring the result of the run to continue onto the next test project.
In something like a docker build a unit test failure will not result in a build failure.
We switched to this:
RUN Get-ChildItem -recurse *tests.csproj | % { dotnet test $_.FullName;if($? -ne $TRUE) {throw 'Unit Test Failure.'} }

The marshalling of failures in individual projects is something that should be handled as part of a solution test option in my opinion.

[VictorioBerra]

We worked around the issue by running "dotnet test" in a loop. We use CAKE and Bamboo. Bamboo is capable of parsing as many .trx files that it finds via the MSTest Parser task and I would wager most other CI/CD solutions can do this as well. If Bamboo finds failed tests, the Bamboo build fails, so we do not suffer from the issue @cfletcher has cautioned.

BTW, xunit.console.runner is now depreciated.

CAKE

///////////////////////////////////////////////////////////////////////////////
// Test
///////////////////////////////////////////////////////////////////////////////
Task("Test")
    .IsDependentOn("Build")
    .Does(() =>
{
    // Clear out the old test results
    CleanDirectories(new DirectoryPath[] { testArtifacts });

    // Build up assemblies
    var projects = GetFiles($"./tests/**/*.csproj");
    foreach(var project in projects)
    {
        Information($"Preparing {project.GetFilename()}");

        var settings = new DotNetCoreTestSettings()
        {
            Logger = $"trx;logfilename={project.GetFilenameWithoutExtension()}.trx",
            ResultsDirectory = testArtifacts
        };

        DotNetCoreTest(project.FullPath, settings);
    }

});

[TomFreeman]

This would also be of benefit when using dotnet watch. At the moment it can run all the tests but it reports the run as failed because of the non-test projects that get run.

[nzandy]

Once again, another issue our dev team has ran into when attempting to slowly shift to using .NET core / standard.

These problems are not obvious when playing around with a new technology. But when you start to migrate an enterprise solution you realise that there are still a lot of rough edges in the tooling that makes life very annoying when trying to configure things like your CI pipeline.

We currently use NUnit for our .NET framework test projects. We have a step in TeamCity which uses the NUnit command line to run the tests in the solution. Low and behold, the command line doesn't work for .NET core projects.

Do some more googling - OK cool the solution is to use the inbuilt "dotnet test" command. That actually sounds great! OK how do I call "dotnet test" on a solution instead of an individual project... that doesn't exist? Hmph..

So now I have to explicitly list each .net core test project in our solution on Teamcity... and then when a coworker adds another test project a month from now we have to remember to add that project in TC.

I was originally surprised I hadn't seen hundreds of "arms waving" blog posts from dev teams shifting enterprise solutions to .NET core - but now am thinking they are too busy hacking workarounds to actually get time to write a blog post.

[VictorioBerra]

@nzandy Core is still maturing and they are using feedback from people like us to constantly improve it. I will say, before you write your blog post take the time here to read some of the solutions that others have come up with for solving this problem. If you are moving to core, you should follow best practices and put your test projects in a "tests" folder which will make them incredibly easy to glob in either bash, powershell or CAKE, PSAKE, whatever you want to use. I personally recommend CAKE with core.

You should never have to do this, if you do, your structure is wrong:

So now I have to explicitly list each .net core test project in our solution on Teamcity... and then when a coworker adds another test project a month from now we have to remember to add that project in TC.

[nzandy]

Appreciate the comment @VictorioBerra - Currently our solution is structured in a way that the test projects sit alongside the projects they test (for example /Customers/Customers.Domain and /Customer/Customer.Domain.Tests folders). So it would be easy to iterate over all folders ending with *.Tests/ in powershell. However because currently we use the NUnit command line for running the .NET framework tests I would not want to run these again by doing a broad iteration like that.

For now because the number of projects is small I am going to explicitly refer to each .NET core tests project - I think the end goal would be just using "dotnet test" for all test projects (.NET framework included) but that (may?) require some further changes to the .NET framework test projects first.

[shanselman]

@livarcocc ya this sucks. I thought this was a good solution but the Globbing works everywhere but windows. We should fix this. @DamianEdwards

C:\Force.com-Toolkit-for-NET\src [(7b0d1fc...) +3 ~23 -8 !]> dotnet test **Tests
MSBUILD : error MSB1009: Project file does not exist.
Switch: **Tests

[dazinator]

Globbing works everywhere but windows

I am so glad this is being highlighted. Consistent usage of globbing capability in CLI commands for the win! :-)

[vikasillumina]

@livarcocc this is quite a long thread and after reading all the comments I unfortunately don't see the solution for running all the tests (at least at the test class level) for a given list of test projects using the dotnet test cli. :(

Do you or anyone else have a suggestion?
I am currently running all the tests sequentially like this:
dotnet test test.proj
where test.proj is a msbuild style file containing targets and also finds all the test projects inside a given folder, something like this

<Target Name="VSTest" DependsOnTargets="Build"> <ItemGroup> <TestProjects Include="tests\**\*Tests.csproj" /> </ItemGroup> <MSBuild Projects="@(TestProjects)" Targets="VSTest" ContinueOnError="ErrorAndContinue"/> </Target>

Since I noticed during my testing dotnet test still calls VSTest as the target (took time to figure this out too) I tried to pass in the parallel parameter like this:

dotnet test build.proj --parallel
or
dotnet test build.proj /parallel

but it fails with error message
`MSBUILD : error MSB1001: Unknown switch.
Switch: /parallel

For switch syntax, type "MSBuild /help"`

[cltshivash]

@vikasillumina We are working on fixing the issue of running tests from different projects in a solution and should have an update soon. If there's a repro for your issue, please do share it so that it is validated as well.

[VictorioBerra]

I'm confused why most of you can't just parse all the resulting test result files in your CI/CD environment like Bamboo, Jenkins, VSTS/TFS, TeamCity, you name it. All of those support parsing a variety of test result files and as many as you need.

[nzandy]

@VictorioBerra using your solution means every time you add a new test project to your solution, you then have to remember to go and update the static list of test files in your CI server, Alternatively you can create a hacky bit of powershell that iterates over some pre-defined folder structure convention, which is not viable in many use cases.

In-built support of running all test projects in a solution is the only legitimate way for this to be resolved.

[mjamro]

@VictorioBerra that's what most of us do. It means additional work. We don't wont to do that. It adds no value.

Having test and non-test projects in one solution is a common approach. Running all tests in a solution is a common use case.

[VictorioBerra]

@nzandy that's not true at all. Most test result parsers can glob. Just output your test results to the same folder. That's not even remotely hacky. If you keep all your tests in a folder structure then you can loop and run your tests. See my cake example above. Literally the only difference between my solution and the solution everyone is asking for is one single loop to run tests.

[vyadh]

@VictorioBerra for me, the argument doesn't involve a CI system. I just want to go into a solution and type 'dotnet test' pre-commit. I was pretty surprised that didn't work.

[VictorioBerra]

I can sympathize. Considering commands like dotnet sln My.sln src/**/*.csproj seem to work you would assume that could work for testing. Who knows, maybe it does?

[noelmcgrath]

any progress on this?

[jeffdoolittle]

This is a bit hacky but is working for me for now:

Get-ChildItem -recurse *tests.csproj | % { Start-Process -FilePath 'dotnet' -NoNewWindow -ArgumentList "watch", "--project="$($_.FullName)"", "test";if($? -ne $TRUE) {throw 'Unit Test Failure.'} }

This watches changes and also filters down to test projects only. Not ideal, but it does allow me to keep all my tests rebuilding and running the background across multiple test projects.

[riezebosch]

Seems like they answered our requests and slipped in a new feature:

Skipping running test for project /Users/***/ConsoleApp/ConsoleApp.csproj. To run tests with dotnet test add "<IsTestProject>true<IsTestProject>" property to project file.

dotnet --version: 2.1.500

[singhsarab]

Indeed it is. With dotnet version 2.1.500, one can now run tests at the solution level without having to create another Testsln only for tests.
dotnet test [*sln]

Thanks for confirming @riezebosch.

[jmck-contessa]

@contessa-zoey

[commonsensesoftware]

The IsTestProject property is indeed nice, but is still incomplete IMO. The filtering of test projects is great, but now there is a bunch of noise for projects that are skipped due to:

<!-- Microsoft.TestPlatform.targets, Line 55 -->
<Target Name="ShowInfoMessageIfProjectHasNoIsTestProjectProperty"
        Condition="'$(IsTestProject)' != 'true'">
    <Microsoft.TestPlatform.Build.Tasks.VSTestLogsTask
        LogType="NoIsTestProjectProperty"
        ProjectFilePath="$(MSBuildProjectFullPath)" />
</Target>

Arguably, the Condition should be something like:

<Target Name="ShowInfoMessageIfProjectHasNoIsTestProjectProperty"
        Condition="'$(IsTestProject)' != 'true' AND '$(IsTestProject)' != 'false'">
    <Microsoft.TestPlatform.Build.Tasks.VSTestLogsTask
        LogType="NoIsTestProjectProperty"
        ProjectFilePath="$(MSBuildProjectFullPath)" />
</Target>

This would still allow the filtering of test projects, but can suppress emitting messages about skipped projects when the IsTestProject is explicitly set to false. Effectively any unset or non-Boolean value will continue to emit a message about a skipped project.

Not sure if that warrants an entirely new issue, but that would be nice. :)

[MarinAtanasov]

I agree with the previous comment.

Additionally, it would be better if dotnet test [*.sln] can group the output like dotnet vstest instead of printing the results of each test project separately. There is a lot of noise for solutions with multiple test projects.

[EddyP23]

Any way of running separate project in parallel with dotnet test, something like:
dotnet test --run-seprate-projects-in-parallel?

[singhsarab]

@commonsensesoftware I tried fixing that issue differently based on another feedback. If IsTestProject is defined, skip the message. https://github.com/Microsoft/vstest/pull/1867
It will be available in the next release. Let me know if you have concerns.

[singhsarab]

I would request folks to create a new issue for any asks post this. Thanks.