Sdk: Allow dotnet test to run on all test projects in a given directory

[sunsided]

For continuous integration, I want to be able to run dotnet test on all test projects in the test directory of my solution, without having to explicitly configure each subdirectory manually, by e.g. issuing

dotnet test --all

or

dotnet test --all-from test

from the solution directory.

Steps to reproduce

In the solution directory, run

dotnet test

Expected behavior

Runs all tests in test subdirectory.

Actual behavior

dotnet-test Error: 0 : System.InvalidOperationException: C:\solution-dir\project.json does not exist.
   at Microsoft.DotNet.Tools.Test.TestCommand.GetProjectPath(String projectPath)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

Environment data

.NET Command Line Tools (1.0.0-preview1-002702)

Product Information:
 Version:     1.0.0-preview1-002702
 Commit Sha:  6cde21225e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.10586
 OS Platform: Windows
 RID:         win10-x64

[blackdwarf]

@livarcocc @piotrpMSFT thoughts?

[TheRealPiotrP]

We need to think about this in the context of the new project model design...

[bojingo]

Is there even a reason for the --all flag? I can be in the solution folder and do a simple dotnet restore and all projects restore, based on looking for project.json files in the subfolders. I would _love_ dotnet test to work this same way. I have a number of test projects and my VSTS build definition is littered with tasks for running tests.

There could be some evaluation of the project.json file to ensure it is a valid test project, such as ensuring there is a testRunner property, such as "testRunner": "xunit".

[dazinator]

There could be some evaluation of the project.json file to ensure it is a valid test project, such as ensuring there is a testRunner property, such as "testRunner": "xunit".

Yes please! Just setting up a CI build, and having to CD into each test project directory to run dotnet test is a maintainance headache, as I keep having to revisit this build script as new test projects are added.

[unjello]

Any progress on that?

[MarinAtanasov]

Fully agree. If I want to run multiple projects from PowerShell, the only way I've found is:

ForEach ($folder in (Get-ChildItem -Path test -Directory)) { dotnet test $folder.FullName }

And the results are not easily readable because it runs every project separately.

Also, somewhat related, we should be able to run all projects in a single run. Even in VS, every project seems to have a separate loading time (about 2 seconds) which slows down the test run when you have multiple test projects.

[glucaci]

For example the dotnet build command accept agrument **/project.json and builds all the projects from subdirectories.
dotnet build **/project.json

I think this will be a good solution for dotnet test also.
dotnet test **/project.json

Actualy cannot parse the argument:

dotnet-test Error: 0 : System.InvalidOperationException: **/project.json does not exist.
   at Microsoft.DotNet.Tools.Test.TestCommand.GetProjectPath(String projectPath)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)

[petrsvihlik]

@MarinAtanasov thanks! i've adjusted it a little bit for my project to run only in folders containing tests and to output the test results...

ForEach ($folder in (Get-ChildItem -Path .\ -Directory -Filter *.Tests)) { dotnet test $folder.FullName -xml ($folder.FullName + "\xunit-results.xml") }

it'd be great if it supported running tests for multiple projects OOTB

[StrangeWill]

Just ran into this today, we'd been using PowerShell for awhile but the problem comes when we want to test under dotCover and TeamCity (now supported on 2017.1 for .NET Core), to do that through PowerShell would be a pain...

If dotnet supported running all tests out of the box this would be a cakewalk.

[livarcocc]

This issue was moved to Microsoft/vstest#705