vstest.console does not change the errorlevel when one of the test fails.

[dlemstra]

Description

I am using vstest.console on the command line to run my unit tests. When of the unit tests fails I would expect the exit code to change to one but with VS2017 the exit code remains zero. This issue can also be reproduced with mstest.exe.

Steps to reproduce

I have created a project that can be used to demonstrate the issue: https://github.com/dlemstra/mstest-exit-code. Running the batch file run-test.cmd will demonstrate the issue. In the initial commit of the project mstest.exe was used and running run-test.cmd with that commit demonstrates the issue with mstest.exe.

Expected behavior

The last two lines when running run-test.cmd should be:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
"ERRORLEVEL=1"

Actual behavior

The last two lines when running run-test.cmd are:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
"ERRORLEVEL=0"

Environment

Windows 10, Visual Studio 2017 15.3.5, vstext.console (15.0.26720.2)

[Faizan2304]

@dlemstra : Can you please try running test using vstest.console.exe from path C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe
I hope this will solve your issue.

This is the new vstest.console.exe which we will be default in coming updates of VS 2017 and old vstest.console.exe (which you are using) is on depreciation path.
Please do let us know if you have any concern or query.

[dlemstra]

I tried to use the other executable but that is unable to discover my tests. I am getting the following error:

No test is available in D:\PullRequests\mstest-exit-code\UnitTestProject\bin\Debug\UnitTestProject.dll.
Make sure that test discoverer & executors are registered and platform & framework version
settings are appropriate and try again.

Am I missing some kind of option? I am using the following options:

/framework:.NETFramework,Version=v4.6 /platform:x64

The diag output contains some assembly loading errors but I have no clue why that is happening. The output can be found here: https://github.com/dlemstra/mstest-exit-code/blob/master/diag.txt and here: https://github.com/dlemstra/mstest-exit-code/blob/master/diag.host.17-09-21_07-30-27_01743_1.txt

[Faizan2304]

You have to add testAdapterPath as well. So the full command will be

"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" D:\PullRequests\mstest-exit-code\UnitTestProject\bin\Debug\UnitTestProject.dll /framework:.NETFramework,Version=v4.6 /platform:x64 /TestAdapterPath:D:\PullRequests\mstest-exit-code\UnitTestProject\bin\Debug

In VS 2017 update5 we will automatically pick adapter from source directory and then you will no need to provide TestAdapterPath. For now you have to provide TestAdapterPath.

[dlemstra]

Thanks for your help. Specifying the TestAdapterPath does indeed solve the issue and produces a correct exit code.

[derekgoodwin]

Hi where can i get a copy of this latest version of vstest.console.exe I don't seem to have it in the path specified. We are using professional and not enterprise is that why