Azure-pipelines-tasks: Version 2.157.4 forcing settings file to be "runsettings"

[aapalm]

Question, Bug, or Feature?
Type: Bug

Environment

Azure Pipelines

Issue Description

Version 2.157.4 is forcing test runs to use "runsettings". Since it does not have the ability to run startup / cleanup batch scripts, this is a breaking change for our test runs.

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

version 2.157.3
2019-09-24T10:54:06.6063152Z [command]E:\agent_work_tool\VsTest\16.3.0\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe @E:\agent_work_temp\a10d7300-deb9-11e9-85d8-e5dff3c83a44.txt
2019-09-24T10:54:07.0644316Z Microsoft (R) Test Execution Command Line Tool Version 16.3.0
2019-09-24T10:54:07.0647178Z Copyright (c) Microsoft Corporation. All rights reserved.
2019-09-24T10:54:07.0647347Z
2019-09-24T10:54:07.3414982Z vstest.console.exe
2019-09-24T10:54:07.3415450Z "E:\agent_work\r1\a\drop\src\tests.dll"
2019-09-24T10:54:07.3415714Z /Settings:"E:\agent_work\r1\a\drop\src\RemoteTestRun.testsettings"

version 2.157.4
2019-10-01T11:44:38.2066356Z [command]E:\agent_work_tool\VsTest\16.3.0\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@C:\Users\localaccount\AppData\Local\Temp\tmpAC70.tmp"
2019-10-01T11:44:38.8124370Z Microsoft (R) Test Execution Command Line Tool Version 16.3.0
2019-10-01T11:44:38.8124898Z Copyright (c) Microsoft Corporation. All rights reserved.
2019-10-01T11:44:39.1356359Z vstest.console.exe "E:\agent_work\r2\a\drop\src\tests.dll"
2019-10-01T11:44:39.1356883Z /Settings:"E:\agent_work\r2\a\drop\src\brvl4smzbqx.runsettings"

[kaadhina]

Hi, if no settings file is specified, we do create a runsettings. But if a testsettings is specified, it will be embedded in the runsettings and honoured. Please let us know if that does not work for you. (SHare full debug logs in case this does not work)
Also on another note, runsettings now supports batch scripts under a 'LegacySettings' node: https://github.com/microsoft/vstest-docs/blob/master/docs/configure.md (Please look for LegacySettings node)

[ShreyasRmsft]

Please try out the suggestions above. And if you are still facing issues please provide the full zip logs with system.debug = true and we will re-open the issue

[aapalm]

Attached are the logs, which shows it converting our "testsettings" file to a "runsettings" file and breaking our test run. @kaadhina , are you suggesting that this has been fixed in latest version (2.157.5+)?

13_VsTest - testAssemblies.log

[ShreyasRmsft]

@aapalm can you add a runsettings similar to below

<RunSettings> 
  <MSTest>
    <SettingsFile>relative path to your test settings file</SettingsFile>
    <DeploymentEnabled>False</DeploymentEnabled>
    <ForcedLegacyMode>true</ForcedLegacyMode>
  </MSTest>
</RunSettings>

<DeploymentEnabled>False</DeploymentEnabled>

This being the node of importance. Basically mstest framework uses a temporary folder E:\agent_work_temp\TestResults\localaccount_AZRFL2W007 2019-10-01 11_44_56\Out\ to run the tests from where your dependent items are not getting copied.

Let me know if this works for you.

[aapalm]

ack, will give this try and report back soon. Thanks @ShreyasRmsft !

[ShreyasRmsft]

Hi, small update: remove the results directory for the above

[ShreyasRmsft]

Also if the above workaround doesn't work then try using the one below:

<RunSettings> 
  <MSTest>
    <SettingsFile>relative path to your test settings file</SettingsFile>
    <ForcedLegacyMode>true</ForcedLegacyMode>
  </MSTest>
  <RunConfiguration>
    <ResultsDirectory>%BUILD_SOURCESDIRECTORY%\TestResults</ResultsDirectory>
  </RunConfiguration>
</RunSettings>

[aapalm]

@ShreyasRmsft, I've tried both runsettings configurations (with and without ResultsDirectory), both gave the same behavior when running locally:

vstest.console.exe /settings:D:\repos\ust\aloha\src\Test\Signup.SystemTests\bin\debug\LocalTestRun.runsettings /Logger:trx D:\repos\ust\aloha\src\Test\Signup.SystemTests\bin\debug\Microsoft.O365.Signup.SystemTests.dll
Microsoft (R) Test Execution Command Line Tool Version 15.9.1
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
TestSettings will soon be deprecated for automated unit and functional testing scenarios. It is recommended that you use RunSettings. To learn more, see http://aka.ms/runsettings
Skipped InstantOnAPI_SuccessTest
Skipped IWTrialNonprofitSignupE2E
Skipped SignupForETrialOnCustomization_EmailFirst
Skipped VerifyRequiredFieldValidationForETrialOnCustomization_EmailFirst
Skipped AzureQuoteSignup_NewUser_ShouldSignupAndRedirectToStoreForBusiness
Failed Website_ValidateVersionEndpoint_Success
Error Message:
Unable to create instance of class Microsoft.O365.Signup.Test.Website.WebsiteTests. Error: System.IO.FileNotFoundException: Could not find file 'D:\repos\ust\aloha\src\Test\Signup.SystemTests\bin\debug\TestResults\aapalm_AAPALM-SURFACE 2019-10-11 16_02_21\Out\Base.SBTest.Config'..
...
Total tests: 116. Passed: 0. Failed: 111. Skipped: 5.
Test Run Failed.

Do we need to upgrade to the new MSTest.TestAdapter in MSTest 2?

[ShreyasRmsft]

@aapalm updating to mstest v2 will not help in your case since you are running visual studio webtests. (correct me in case you not running webtests- i inferred this from the test settings file).

[aapalm]

Hey @ShreyasRmsft, these are being ran as normal unit tests. I removed the webtest configuration from testsettings to verify it wasn't being used.

[ShreyasRmsft]

@aapalm closing this here as the decision to change the default TestResults directory from $(Agent.DefaultWorkingDirectory) to $(Agent.TempDirectory) was intentional and more of a bug fix as $(Agent.DefaultWorkingDirectory) being the sources directory was never the right place to put the test results.

The test results directory can be overidden to the old location using a runsettings file and the agent env variables as described above. But ideally the tests should be modified to not depend on these agent paths.