Vstest: add support for `dotnet test path/to/my.sln`

[TheRealPiotrP]

As folks are spending more time with the new command line experience, the desire for a sln-level test command is growing.

Some desirable characteristics:

• parallelism! allow MSBuild's /m to control the number of projects to test in parallel
• Unified output. Provide a summary of how the overall test run did, not just individual projects

[ErikSchierboom]

I would love to see this implemented! It will makes build scripts just a little bit shorter and easier.

[bernardbr]

In my case, I solved it with a python script. I've tried to do it with a PowerShell script, but I got restriction error.

Python solution:

import subprocess
from argparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument("-f", "--files", nargs="+", dest="files", help="Define a .csproj file list.")

args = parser.parse_args()

for f in args.files:
    subprocess.call(r"C:\Program Files\dotnet\dotnet.exe" +  " test " + f)

PowerShell solution:

param(
    [Parameter(Mandatory = $true, Position = 0)]
    [string[]]$csProjFiles
)    

foreach ($csProjFile in $csProjFiles)
{
    dotnet test $csProjFile
}

My task.json (VsCode):

{
    // See https://go.microsoft.com/fwlink/?LinkId=733558
    // for the documentation about the tasks.json format
    "version": "0.1.0",
    "isShellCommand": true,
    "args": [],
    "tasks": [
        {
            "taskName": "build",
            "command": "dotnet",
            "args": [
                "build",
                "solution.sln"
            ],
            "isBuildCommand": true,
            "showOutput": "silent",
            "problemMatcher": "$msCompile"
        },
        {
            "taskName": "test",
            "command": "py",
            "args": [
                "${cwd}/run_tests.py",
                "-f",
                "${cwd}/test/unit/model/model.csproj",
                "${cwd}/test/unit/repository/repository.csproj"
            ],
            "isTestCommand": true,
            "showOutput": "silent"
        }
    ]
}

[tomasaschan]

Unified output would be really nice, but could prove quite complex given the wide range of testing frameworks and how they present their output. However, there's another killer reason for me to want this: reduced test running time. It's quite easy to find the project files and pass each of them to dotnet test, but doing that triggers a new build for each project, meaning that in a larger solution with many test projects, I have to wait for many builds just to run the tests.

[tverboon]

It's quite easy to find the project files and pass each of them to dotnet test, but doing that triggers a new build for each project, meaning that in a larger solution with many test projects, I have to wait for many builds just to run the tests.

@tlycken There is already a --no-build option. We build our solution first and then run the tests parallel. It's much faster that way. Only negative, is that you need to find a failing test in the output, so this doesn't scale well.

#!/usr/bin/env bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CD $DIR/../
dotnet build

find $DIR/../test -not -path '*.TestCommon*' -name '*.csproj' -print0 | xargs -L1 -0 -P 8 dotnet test --no-build

if [ "$?" !=  "0" ] ; then
    echo -e "######### TEST FAILED #########"
fi

[bromanko]

This suggestion worked well for me.

[Kritner]

I liked the loop solutions to getting it accomplished however, when i was doing it with powershell for my teamcity builds, it seems that team city doesn't really understand "pass vs failure" on a test when run via powershell

i think it only understand when running as a "dotnet test" task, rather than a "powershell" task that just happens to invoke dotnet test

So while I find the loops clever, I can really do the same thing with a GUI test runner - I feel it's more important on the solution level so that my dotnet test task can work. Is there another option in this regard?

[tomasaschan]

@Kritner TeamCity (and other build systems) usually infer the success or failure state of script steps from their exit codes; if you amend the PS script to return an exit code that's 0 if all tests passed and something else otherwise, you could probably get it working.

That said, I still want this feature :)

[Kritner]

Thanks @tlycken I'll have to give that a go

[ErikSchierboom]

Although the scripting solution is a workaround, it is precisely that. To see the problem with this approach, consider doing the same for building a set of projects. Building the projects individually instead of as part of a solution is much, much slower. This is the main reason I'd like to be do dotnet test <name>.sln: improved performance.

[thenathanjones]

+1. I'm trying to do an automated build on a CI server, and while the loops "work" they are indeed a workaround. If I'm grouping them in a solution, it's logical that I should be able to run all the projects within that group. I'll work around it, but I feel that some other people may just use it as an excuse to just create a mega project and not deal with the problem.

[tverboon]

My script was intended for local use, as it runs all tests in parallel. On the CI server you can call it with MSBuild. Put the following target in a MSBuild project file.

<Target Name="TestAll">
    <ItemGroup>
        <TestProjects Include="$(MSBuildThisFileDirectory)test\**\*.csproj" Exclude="$(MSBuildThisFileDirectory)test\TestCommon\**\*.csproj" />
    </ItemGroup>
    <MSBuild Projects="@(TestProjects)" Targets="vstest" Properties="VSTestNoBuild=true" ContinueOnError="ErrorAndContinue"/>
</Target>

The VSTestNoBuild property will prevent a build.

You can call it like this: dotnet msbuild YourProject.build /t:TestAll to test it.

Only thing is that running all test assemblies in parallel doesn't seem to work.

[hizhubo]

@tverboon Error "error MSB4057: The target "vstest" does not exist in the project." occurred after applied your msbuild approach. Is it necessary to manually add target "vstest" in all test projects?

[tverboon]

@hizhubo It is part of Microsoft Test platform: https://github.com/Microsoft/vstest/blob/master/src/Microsoft.TestPlatform.Build/Microsoft.TestPlatform.targets.

My test projects target netcoreapp2.0 and have a package reference to Microsoft.NET.Test.Sdk.

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />

[hizhubo]

@tverboon
My project has the same reference (as shown below) but the issue error MSB4057: The target "vstest" does not exist in the project. still exists. Would you please help me to check if there is any issue in my project, batch script or MSBuild script? Thanks a lot.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
    <PackageReference Include="Microsoft.TestPlatform.Build" Version="15.3.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.1.18" />
    <PackageReference Include="MSTest.TestFramework" Version="1.1.18" />
    <PackageReference Include="NETStandard.Library" Version="2.0.0" />
    <PackageReference Include="System.Runtime" Version="4.3.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="ProjectToBeTested.csproj" />
  </ItemGroup>

</Project>

The command line to invoke MSBuild

"C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" "%CurrentDir%NugetPackage.sln.Build.proj"

The MSBuild script snippet

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Build">
...
  <Target Name="RunTests">
    <ItemGroup>
      <TestProjects Include="$(MSBuildThisFileDirectory)Test\**\*.Test.csproj" />
    </ItemGroup>
    <MSBuild Projects="@(TestProjects)" Targets="VSTest" Properties="VSTestNoBuild=true" ContinueOnError="ErrorAndContinue"/>
  </Target>
...
</Project>

[tverboon]

@hizhubo Your MSBuild files look good to me. Could you try to run it with the dotnet 2.0 cli toolchain?

First build your solution, packages will be restored automatically:
dotnet build NugetPackage.sln

Then run your test target:
dotnet msbuild NugetPackage.sln.Build.proj /t: RunTests

[hizhubo]

@tverboon dotnet msbuild NugetPackage.sln.Build.proj /t: RunTests works! Whereas "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe" NugetPackage.sln.Build.proj /t:RunTests doesn't

I guess it's because that MSBuild.exe 15.0 is not dotnet cli toolchain and target VSTest somehow needs to be running under dotnet 2.0 cli.

I changed my command line as shown below and now it works like a charm. Thanks a lot!

dotnet msbuild "%CurrentDir%NugetPackage.sln.Build.proj"

[dasMulli]

@Faizan2304 @codito Since the Test SDK NuGet package sets $(IsTestProject) to true, the VSTest target could condition on that:
https://github.com/Microsoft/vstest/blob/3983a3c97a7f13b7cb7f0b6b681b6dacfbc81115/src/Microsoft.TestPlatform.Build/Microsoft.TestPlatform.targets#L28

=>

<Target Name="VSTest" Condition="'$(IsTestProject)' == 'true'">

This works quite well in my manual testing, are there any drawbacks this would introduce?

[brunolm]

Looks like specifying a solution is already supported in 2.1.2

> dotnet test .\dotnetcore-how-to.Tests.sln
Build started, please wait...
Build completed.

Test run for D:\BrunoLM\Projects\dotnetcore-how-to\dotnetcore-how-to.Tests\bin\Debug\netcoreapp2.0\dotnetcore-how-to.Tests.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 0.8438 Seconds

> dotnet --version
2.1.2

[tomasaschan]

@brunolm does that solution contain any projects that are not test projects? I have 2.1.300, and it does allow dotnet test targeting a solution, but attempts to run tests in _all_ projects, not just the test projects, and then spews out ugly errors for all non-test projects.

[Kritner]

@tlycken Yeah, still same issue here. If you add Microsoft.NET.Test.Sdk to all projects, not just test projects, this seems to be enough to get the solution to report an overall "success". It's kinda gross putting a Test.SDK on a non test project, but it's a workaround for the moment.

Be prepared if you go this route to get lots of localization stuff in your build if you didn't already have it (at least I think it's from this nuget package)

[brunolm]

@tlycken I have 2 .sln files

dotnetcore-how-to.sln          // all
dotnetcore-how-to.Tests.sln    // tests only

When I run the test sln only my test project is run.

Not as good as having 1 sln and having dotnet automatically figure out what to do, but it works

[toqueteos]

@brunolm does that solution contain any projects that are not test projects? I have 2.1.300, and it does allow dotnet test targeting a solution, but attempts to run tests in _all_ projects, not just the test projects, and then spews out ugly errors for all non-test projects.

This has been solved by https://github.com/Microsoft/vstest/pull/1745

[slang25]

Just to echo the previous comment (https://github.com/microsoft/vstest/issues/411#issuecomment-463998490), this can be closed now :tada:

[romantitov]

The issue still happens with dontet version 3.1.403