vstest.console.exe returns a zero exit code even if no tests were found/executed

[icnocop]

Description

vstest.console.exe returns a zero exit code even if no tests were found/executed.
A zero exit code usually means success, but since no tests were found/executed, I expected a non-zero exit code meaning a failure occurred.

Steps to reproduce

Create and build a new Unit Test Project (.NET Framework 4.6.1) in Visual Studio 2017 v15.9.17

Create a `Local.testsettings' file:

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local" id="068518c9-0930-453f-bf19-5e97cdf5545e" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
  <Execution>
    <Timeouts runTimeout="9000000" testTimeout="9000000" />
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution>
          <TestDirectory useLoadContext="true" />
        </AssemblyResolution>
      </UnitTestRunConfig>
      <WebTestRunConfiguration testTypeId="4e7599fa-5ecb-43e9-a887-cd63cf72d207">
        <Browser name="Internet Explorer 9.0" MaxConnections="6">
          <Headers>
            <Header name="User-Agent" value="Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" />
            <Header name="Accept" value="*/*" />
            <Header name="Accept-Language" value="{{$IEAcceptLanguage}}" />
            <Header name="Accept-Encoding" value="GZIP" />
          </Headers>
        </Browser>
      </WebTestRunConfiguration>
    </TestTypeSpecific>
    <AgentRule name="LocalMachineDefaultRole">
    </AgentRule>
  </Execution>
  <Properties />
</TestSettings>

Run "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "c:\unittests\MyUnitTestProject\MyUnitTestProject.dll" /tests:MyUnitTestProject.UnitTest1.TestMethod1 /logger:trx /settings:"c:\unittests\Local.testsettings"

Another way to reproduce the issue is to just pass a test method that doesn't exist.

Ex., run "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" "c:\unittests\MyUnitTestProject\MyUnitTestProject.dll" /tests:MyUnitTestProject.UnitTest1.TestMethod2

Expected behavior

I expected a non-zero exit code

Actual behavior

vstest.console.exe exited with a zero exit code.

Console Output:

Microsoft (R) Test Execution Command Line Tool Version 15.9.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test discovery, please wait...
Test run will use DLL(s) built for framework .NETFramework,Version=v4.5 and platform X86. Following DLL(s) do not match framework/platform settings.
MyUnitTestProject.dll is built for Framework 4.6.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.

TestSettings will soon be deprecated for automated unit and functional testing scenarios. It is recommended that you use RunSettings. To learn more, see http://aka.ms/runsettings
Warning : A testsettings file, a runsettings file with a ForcedLegacyMode set to true or a vsmdi file is not supported with the MSTest V2 Adapter.
No test is available in c:\unittests\MyUnitTestProject\MyUnitTestProject.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
No test is available in c:\unittests\MyUnitTestProject\MyUnitTestProject.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again. Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.

In the case where the test method doesn't exist:

Microsoft (R) Test Execution Command Line Tool Version 15.9.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test discovery, please wait...
A total of 1 tests were discovered but no test matches the specified selection criteria(MyUnitTestProject.UnitTest1.TestMethod2). Use right value(s) and try again.

Environment

Windows Server 2016 64-bit
Visual Studio Test Agent 2017 15.9.28307.858
MSTest.TestAdapter v1.3.2
MSTest.TestFramework v1.3.2

[singhsarab]

@icnocop This is by design. Any change in this behavior is potential breaking change for our customers.

[icnocop]

I'm not exactly sure the scenario where returning a zero exit code is expected.

In any case, I believe this "feature" can still be implemented without breaking changes.

For example, an additional setting and command line parameter can be passed to vstest.console.exe to require that at least one test is run or else exit with a non-zero exit code:

1. The additional setting in the runsettings file:

<TreatNoTestsAsError>true</TreatNoTestsAsError>

1. Passing runsetting parameters through the command line:

vstest.console.exe -- MSTest.TreatNoTestsAsError=true

Thank you.

[icnocop]

@singhsarab, please consider re-opening this issue.

Thank you.

[Volkmire]

@singhsarab Please re-open the issue. We constantly encounter it during automated pipeline setups. A couple of examples:

• no test.dlls published -> no tests run -> visible result is success;
• incorrect folder specified during publishing/execution -> no tests run -> visible result is success;
• failure to download test.dlls -> no tests run -> visible result is success;

In all of those cases we expect the runner to fail, because it had 1 job, to run tests and didn't do it.
If direct implementation would be breaking, considering adding a custom flag to toggle this behavior.

[icnocop]

For reference, see https://github.com/microsoft/vstest/issues/2262

[nohwnd]

@jakubch1 ( @pavelhorak ) should we add this to 16.9.0?

[jakubch1]

We will have meeting this week regarding bugs and we will discuss it.

[pavelhorak]

Expected behaviour is that vstest.console.exe command, like any other external command, exits with status code 0 if it terminates normally. This also applies to a situation where no tests meet the selection criteria. Now, we won't change this as a DEFAULT behaviour, however, to satisfy the needs, we will introduce an option (new .runsettings parameter and command line switch like suggested by @icnocop) to override this behaviour and exit vstest.console.exe with non zero code if no test was executed.

[Volkmire]

Thanks for an update, @pavelhorak .
On a side note, I urge you to check what exit code is returned if you try to use cp or copy commands if you specify a non-existent file to them or a pattern that matches 0 elements. I guess, it's actually a developer who defines what is considered a normal termination after all. :)