Azure-pipelines-tasks: [VsTest] .testsettings not taken into account

[wbuchwalter]

Hey!
The Visual Studio Test task's documentation for the Run Settings File parameter specifies:

Run Settings File: Path to a runsettings or testsettings file can be specified here.

But any .testsettings file passed will be ignored.

I guess this is just an outdated documentation, but I want to make sure that .testsettings are indeed not supported and that this is not a bug.

Thanks!

[atinb]

Testsettings file is supported. Why do you think it is ignored?
Thanks!

[wbuchwalter]

In that case I'll try to reproduce the issue and get you detailed infos.

As a side note, would be nice to update the tooltip in the task to reflect this, since only runsettings is specified:

Thanks!

[Ryanman]

@wbuchwalter I may be able to help you with this - the .testsettings file that I use in my application correctly deploys additional test artifacts when used locally, but does not deploy any of the supporting files when using the "Run Functional Tests" Task through TFS 2015. I know this is different than you executing with the VS Test task, but it may be similar.

You can find more info HERE.

[Ryanman]

Team,

I want to update this issue rather than opening a new one - I'm not sure if @wbuchwalter was having similar problems with .testsettings files as I am right now.

The file I specify in the build configuration screen is being correctly pushed to the member of the Machine Group that's intended to run the test:

The directory containing these files on the member of the machine group is identical to how my solution looks on my dev box:

And yet, you can see that no files are deployed, using my kung-fu-super-fast-copy-paste skills from the byzantine App Data folder:

What gives here? One of the key components of the .testsettings file is the deployment functionality that works locally. I can't tell if it's being ignored or not.

[wbuchwalter]

Hey,

So I spent a little bit of time trying to reproduce the issue.
Here is a simple .testsettings file that I'm using (I removed some non-relevant stuff to keep it short):

<TestSettings name="Local">
  <Deployment>
    <DeploymentItem filename="..\..\scripts\echo.ps1" />
  </Deployment>
</TestSettings>

If I don't change any settings from the default Visual Studio Test task except the Run Settings param, everything seems to work fine (log from build agent):

Executing [...]\vstest.console.exe [paths to all my assemblies] /Settings:"C:\a\1\s\tests\PartsUnlimitedWebsiteTests\local.testsettings"

But once I change another parameter, for example by checking Run In Parallel :

Executing [...]\vstest.console.exe [test assemblies] /Settings:"C:\Users\buildguest\AppData\Local\Temp\tmp1B2E.tmp"

And if I take a look at tmp1B2E.tmp:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>0</MaxCpuCount>
  </RunConfiguration>
</RunSettings>

So basically, since the vsts task itself will store it's configuration in a runsettings file, everything will work fine if we pass a runsettings file as well for the Run Settings param, since both settings will be merge in something.tmp.

But this cannot work with a .testsettings file, since the merge will fail, and the runsettings of the task itself will take prescendence. Basically the testsettings file will be ignored unless not changing any parameter from the original task.
(lot of settings in this sentence :) ).

At the end of the day, it may be the expected behavior, but it would be cool to give a hint about this in the UI, like disabling every option if we pass a testsettings file for example.

Let me know if I missed something!

[Ryanman]

Ahhh okay. So, in summary: it looks like this may be specific to the VS Test Task, as I have parameters defined in the Functional Test Task (Test Filter Criteria) and my .testsettings file at least is copied into my temp directory rather than a blank slate .runsettings file.

I'll open my own issue here - thanks for the clarification!

[nzthiago]

@atinb - from @wbuchwalter's investigation it seems like it's a problem with the VS Test Task? Can the team check it out? We are tracking this as part of a partner feedback

[atinb]

We have filed a bug and will be fixing it in the next sprint. Run in Parallel should not work with testsettings file. As a workaround, please uncheck the "Run in Parallel" check box. Thanks!

[prawalagarwal]

This is fixed now. Closing the issue.