Testfx: Unable to run unit tests remotely on Test Agent using MSTest V2

[maxinfet]

Description

I am trying to run my unit tests with UI automation on a Visual Studio Test Controller and Visual Studio Test Agent but I get the following warning Warning : A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter. and none of my tests load.

Steps to reproduce

• Add a .testsettings file to your solution
• Add a .runsettings file to your solution
• Add a SettingsFile element to the MSTest element in the .runsettings
• Add a ForceLegacyMode element to the MSTest element in the .runsettings
• Select the .runsettings as the active test settings (Test -> Test Settings -> Select Test Settings File)
• Rebuild Solution

Expected behavior

Tests are loaded and I am able to run my unit tests on the Test Controller. I understand the legacy mode not being supported but it seems like the .runsettings should also have a means for me to specify the host for the test controller.

Actual behavior

Tests are not loaded and I get the following warnings. When I remove loading the SettingsFile and the ForceLegacyMode elements my tests load but then I won't be able to connect to the controller.

[11/22/2017 9:54:12 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:13 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:13 AM Warning] Index was outside the bounds of the array.
[11/22/2017 9:54:14 AM Warning] Warning : A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter.

Environment

Just to clarify I am not doing coded UI tests, I am running unit tests that are using Microsoft's UIA through a open source project called FlaUI. I am trying to use the test controller as part of our CD CI pipeline to run out UI automation on producing builds.

Here are my .runsettings and .testsettings

.runsettings

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <!-- Configurations that affect the Test Framework -->
  <RunConfiguration>
    <MaxCpuCount>1</MaxCpuCount>
    <!-- Path relative to solution directory -->
    <ResultsDirectory>.\TestResults</ResultsDirectory>

    <!-- [x86] | x64    
      - You can also change it from menu Test, Test Settings, Default Processor Architecture -->
    <TargetPlatform>x86</TargetPlatform>

    <!-- Framework35 | [Framework40] | Framework45 -->
    <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
  </RunConfiguration>

  <!-- Configurations for data collectors -->
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
              </Exclude>
            </ModulePaths>

            <!-- We recommend you do not change the following values: -->
            <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>

          </CodeCoverage>
        </Configuration>
      </DataCollector>

    </DataCollectors>
  </DataCollectionRunSettings>

  <!-- Parameters used by tests at runtime -->
  <TestRunParameters>
  </TestRunParameters>

  <!-- Adapter Specific sections -->

  <!-- MSTest adapter -->
  <MSTest>
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <CaptureTraceOutput>false</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>False</DeploymentEnabled>
    <SettingsFile>test.testsettings</SettingsFile>
    <ForcedLegacyMode>true</ForcedLegacyMode>
    <AssemblyResolution>
      <Directory Path=".\Tests" includeSubDirectories="true"/>
    </AssemblyResolution>
  </MSTest>

</RunSettings>

.testsettings

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="rmoqa01" id="076be28c-d18b-46bf-ad20-4d43ec821ea4" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <RemoteController name="10.2.0.101" />
  <Execution location="Remote">
    <Hosts skipUnhostableTests="false">
      <VSSDKTestHostRunConfig name="VS IDE" HiveKind="DevEnv" HiveName="15.0_c9b36733" xmlns="http://microsoft.com/schemas/VisualStudio/SDK/Tools/IdeHostAdapter/2006/06" />
    </Hosts>
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution>
          <TestDirectory useLoadContext="true" />
        </AssemblyResolution>
      </UnitTestRunConfig>
    </TestTypeSpecific>
    <AgentRule name="AllAgentsDefaultRole">
    </AgentRule>
  </Execution>
  <Properties />
</TestSettings>

[maxinfet]

Also is the API for the Test Controller available any where so I could possibly contribute to this?

[jayaranigarg]

@maxinfet : Remote execution via testsettings is currently not supported with MStestV2. An issue is already filed for the same. Tagging @pvlakshm @cltshivash for prioritizing this from the backlog.

[jayaranigarg]

MSTest V2 will not support .testsettings.
Getting .runsettings to parity will be taken up as a separate item.

[joltcoder]

Are there plans to support deployment items in the runsettings file as they are/were with the testsettings file?
Partial xml below:
<TestSettings name="TestSettings1" id="245d6fc9-99ae-44e7-8e9b-68c737b2da21" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>These are default test settings for a local test run.</Description>
<Deployment>
<DeploymentItem filename="TestData\US_TestData.xml" />
<DeploymentItem filename="TestData\" /> `

[maxinfet]

@jayaranigarg Since the issue you mentioned previously was closed could this one be reopened? I am not asking for ,testsettings to work I am asking for .runsettings to support remote execution.

[jayaranigarg]

@joltcoder : Plans to support deployment items are in flight. They will be delivered soon.

[jayaranigarg]

Tagging @pvlakshm @cltshivash for .runsettings to support remote execution prioritization.

[jayaranigarg]

@joltcoder: We have [DeploymentItem] attribute which can be used over a testmethod or test class. Here is the documentation : https://docs.microsoft.com/en-us/dotnet/api/microsoft.visualstudio.testtools.unittesting.deploymentitemattribute?view=mstest-net-1.2.0

Can you try using that and see if that helps?

[Tyale-cpu]

Hi, Is there any roadmap that might show when/if the issue of remote execution might have any chance of being rolled into .runsettings? Our entire automation environment uses Test Controller / Test Agents to execute all of our automation and we want to use some of the new features of MSTest V2. I am just not sure if we need to look into switching over to another execution framework for our tests. Thanks Tom!

[NIBShura]

Running the test on remote machine from within Visual Studio was a nice feature in MSTest (V1?), through ".testsettings". MSTest V2 uses ".runsettings" which does not have this feature of running test on remote machine from within Visual Studio, or atleast I dont know how this can be done using ".runsettings".

Allowing the test to run on remote machine (thru test controller/test agent) is a very helpful features for test driven development. We write a .NET Class Libraries (API) on top of unmanaged C libraries. And in order to run our test for these library while development, we need to install lot many thing as a dependencies. It is not advisable for us to install these dependencies on our development machine, and hence we want to make use of Agents for Visual Studio (Test Controller/Test Agent), which allow us to write test on our development machine (test controller) and run it on remote machine (test agent - where required dependencies are already installed). This missing feature is currently preventing us from using MSTest V2.

Please add this feature to MSTest V2.

[VanKrock]

How I can downgrade to MSTest v1?

[cltshivash]

@NIBShura @Tyale-cpu There are no plans to have remote execution capability through MSTVest2. You can use the test task in VSTS which supports distributed execution on multiple agents. https://docs.microsoft.com/en-us/vsts/pipelines/test/set-up-continuous-test-environments-builds?view=vsts

[Tyale-cpu]

I have to say that is terrible news for us. The assumption there is that we are using VSTS as part of our CI process, which we don't. We use TeamCity to build our products and Jenkins to execute our tests and I can't believe we are the only ones not using VSTS. It really is a shame as we had high hopes for the new framework.

I do appreciate the follow up, just disappointed.

Tom

[maxinfet]

@Tyale-cpu We are using teamcity and octopus deploy so we are in the same boat.

[PablitoPaul]

@jayaranigarg : "_Getting .runsettings to parity will be taken up as a separate item_"
Which item?

[jayaranigarg]

@PablitoPaul : We are trying to minimize the gap between runsettings and testsettings. Is there anything specific that you are looking for?

[PablitoPaul]

@jayaranigarg
The _testsettings_ files in the codebase I inherited (and that I should migrate to the new framework, if I manage to avoid the other problems I'm facing) contain the following options:

• [Timeouts runTimeout="36000000" testTimeout="10800000" /] (I think testTimeout can be replaced by the Timeout attribute in single tests, and for runTimeout, probably the TFS Release timeout "wins")
• [AgentRule name="LocalMachineDefaultRole"]
• [TestDirectory useLoadContext="true" /]
• [Execution hostProcessPlatform="MSIL"] (I can obtain the same effect in runsettings using [TargetPlatform]x64[/TargetPlatform])

(of course, I replaced angled brackets with square ones)
Thank you very much!

[jayaranigarg]

@PablitoPaul : We have introduced a new node <LegacySettings>inside runsettings which should have pretty much everything which testsettings used to have.
Refer this doc : https://github.com/Microsoft/vstest-docs/blob/master/docs/configure.md#other-sections.

[PablitoPaul]

@jayaranigarg Thank you, but the document you linked says that "legacy setting are honored only when this flag is set to true", where "this flag" is "ForcedLegacyMode".
And the opening post by @maxinfet says that "_A testsettings file or a runsettings with a ForcedLegacyMode set to true is not supported with the MSTest V2 Adapter_" ...

[jayaranigarg]

@PablitoPaul : Yes earlier that was the case. At the time we inserted legacy settings, we changed the behavior and now you can set ForcedLegacyMode to true and MSTestV2 will honour it. Refer the new warning message : https://github.com/Microsoft/testfx/blob/f3417a8577e2ab900b91cae07602d7d43fec82eb/src/Adapter/MSTest.CoreAdapter/Resources/Resource.resx#L288

[omghb]

@cltshivash Please reconsider the decision to not support remote execution capability through MSTest V2. This is a blocking issue for us to migrate from MSTest V1 to MSTest V2.

[cltshivash]

@omghb The capability to distribute tests has been removed from the test platform and is part of the Azure Pipelines alone. We recommend users to have pipelines setup targeting user branches as well if there's a need for distributed testing. Please do open a separate issue to log this feedback as an ask for prioritization.