Vstest: dotnet test command with TestRunParameters parameter, fails with invalid format error

[sureshtadisetty]

Description

dotnet test command with fails with invalid format error, was working 1 week back, didn't build last week, so don't know exactly when did this issue start.
I tried the format suggested in the error message, still get the same error.

Steps to reproduce

What steps can reproduce the defect?
"dotnet test" task Arguments in ADO

--no-restore --settings $(System.DefaultWorkingDirectory)/_Office-Experimentation-Repo/src/oexptest.runsettings -- TestRunParameters/Parameter[contains(@name,'Name1')]/@value=$(Secret1) -- TestRunParameters/Parameter[contains(@name,'Name2')]/@value=$(Secret2)

command I see in log file:
[command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\r1\asrc\Tests\Tests.csproj --logger trx --results-directory D:\a_temp --no-restore --settings D:\a\r1\a/src/test.runsettings -- TestRunParameters/Parameter[contains(@name,'Name1')]/@value=* -- TestRunParameters/Parameter[contains(@name,'Name2')]/@value=*

Expected behavior

should run tests

Actual behavior

Fail with below error:
The test run parameter argument 'TestRunParameters/Parameter[contains(@name,'Name1')]/@value=*' is invalid. Please use the format below.
Format: TestRunParameters.Parameter(name=\"\", value=\"\")

[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

Diagnostic logs

TpTrace Verbose: 0 : 15944, 1, 2020/04/08, 02:59:08.031, 1695795794784, vstest.console.dll, Using effective platform:X86 effective framework:.NETCoreApp,Version=v1.0
TpTrace Information: 0 : 15944, 1, 2020/04/08, 02:59:08.037, 1695795829501, vstest.console.dll, Using .Net Framework version:.NETCoreApp,Version=v2.2
TpTrace Error: 0 : 15944, 1, 2020/04/08, 02:59:08.064, 1695796098990, vstest.console.dll, ArgumentProcessorFactory.WrapLazyProcessorToInitializeOnInstantiation: Exception initializing argument processor: Microsoft.VisualStudio.TestPlatform.CommandLine.CommandLineException: The test run parameter argument 'TestRunParameters/Parameter[contains(@name,'Name1')]/@value=Secret1' is invalid. Please use the format below.
Format: TestRunParameters.Parameter(name=\"\", value=\"\")
at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.CLIRunSettingsArgumentExecutor.UpdateTestRunParameterNode(IRunSettingsProvider runSettingsProvider, String node)
at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.CLIRunSettingsArgumentExecutor.CreateOrOverwriteRunSettings(IRunSettingsProvider runSettingsProvider, String[] args)
at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.CLIRunSettingsArgumentExecutor.Initialize(String[] arguments)
at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass21_0.b__0()

Environment

Please share additional details about the test environment.
ADO Task : .NET Core
Version : 2.166.2

[sureshtadisetty]

I tried format suggested in the error message:
--no-restore --settings $(System.DefaultWorkingDirectory)/src/test.runsettings -- TestRunParameters.Parameter(name=\"Name1\",value=\"$(Secret1)\") -- TestRunParameters.Parameter(name=\"Name2\",value=\"$(Secret2)\")

Below is the new command I see in the log:
(didn't escape characters well)
[command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\r1\asrc\Tests\Tests.csproj --logger trx --results-directory D:\a_temp --no-restore --settings D:\a\r1\a/src/test.runsettings -- "TestRunParameters.Parameter(name=\Name1\",value=\"\") -- TestRunParameters.Parameter(name=\"Name2\",value=\"\")"

Error message:
The test run parameter argument 'TestRunParameters.Parameter(name=\Name1' is invalid. Please use the format below.
Format: TestRunParameters.Parameter(name=\"\", value=\"\")

[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

[nohwnd]

Please go to https://github.com/microsoft/vstest/issues/862#issuecomment-603894275 for options how to solve this. From dotnet test you need to escape it twice at the moment, unlike what the syntax suggests.

Duplicate #862

[sureshtadisetty]

@nohwnd , no I tried double and tripple escape it doesn't work.

Arguments to dotnet task in ADO:

--no-restore --settings $(System.DefaultWorkingDirectory)/_Office-Experimentation-Repo/src/oexptest.runsettings -- TestRunParameters.Parameter(name=\\"EdiStagingFunctionsKey\\",value=\\"$(EdiStagingFunctionsKey)\\") -- TestRunParameters.Parameter(name=\\"EdiAADKey\\",value=\\"$(EdiAADKey)\\") 

Command I see in build log (didn't escape characters well) :

[command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\r1\a\src\Tests.csproj --logger trx --results-directory D:\a\_temp --no-restore --settings D:\a\r1\a/src/oexptest.runsettings -- TestRunParameters.Parameter(name=\\Name1\\,value=\\***\\) -- TestRunParameters.Parameter(name=\\Name2\\,value=\\***\\)

Error message from build log:

The test run parameter argument 'TestRunParameters.Parameter(name=\\EdiStagingFunctionsKey\\,value=\\***\\)' is invalid. Please use the format below.
Format: TestRunParameters.Parameter(name=\"<name>\", value=\"<value>\")

[nohwnd]

Did you try the solution in the other thread that is escaping every double quote with three backslashes? \\\" In your first command above I only see two backslashes.

[sureshtadisetty]

@nohwnd yes I tried three backslashes, still doesn't work.

Arguments to dotnet task in ADO:
--no-restore --settings $(System.DefaultWorkingDirectory)/src/test.runsettings -- TestRunParameters.Parameter(name=\\\"Name1\\\",value=\\\"$(Secret1)\\\") -- TestRunParameters.Parameter(name=\\\"Name2\\\",value=\\\"$(Secret2)\\\") 

Command I see in build log (see how the double quotes endup):

[command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\r1\a\src\Tests\Tests.csproj --logger trx --results-directory D:\a\_temp --no-restore --settings D:\a\r1\a/src/test.runsettings -- "TestRunParameters.Parameter(name=\\\Name1\\\\\",value=\\\\\"***\\\\\") -- TestRunParameters.Parameter(name=\\\\\"Name2\\\\\",value=\\\\\"***\\\\\")"

Error message from build log:

The test run parameter argument 'TestRunParameters.Parameter(name=\\\Name1\' is invalid. Please use the format below.
Format: TestRunParameters.Parameter(name=\"<name>\", value=\"<value>\")
##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

[nohwnd]

Seems like it escapes your sequence one more time but only part of it.

[sureshtadisetty]

@nohwnd is there any workaround for this?
We're unable to run tests in build pipeline because of this issue.

[nohwnd]

https://docs.microsoft.com/cs-cz/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019#testrunparameters would you be able to pass the parameters using runsettings?

[sureshtadisetty]

@nohwnd no these are secrets, otherwise would've done that already

[bfinagraph]

@nohwnd i'm running into the similar issue. Are there any updates ?
@sureshtadisetty did you find any workarounds to get yourself unblocked?

If it helps, i'm using
ADO Task : .NET Core
Version : 2.168.0

[bfinagraph]

FYI:
Ended up writing a custom powershell script to create a temp runsettings file. Used this as inspiration:
https://blog.magnusmontin.net/2019/05/30/override-testrunparameters-in-net-core/

[nohwnd]

The fix for #862 was merged into master of vstest and dotnet/sdk, I don't think there was any preview release in the meantime, but it will end up in 16.7.0-previewxxx and dotnet/sdk 5.0.0-previewxxx.

[TroyWalshProf]

@nohwnd - Any chance this could include?
https://github.com/microsoft/vstest/issues/862#issuecomment-606005714

[nohwnd]

@TroyWalshProf so far I am not sure if enabling this would not cause problems. Moving to a new issue and adding to 16.7.0 16.8.0 milestone. Right now I am working on improving crash analysis so hopefully this can come next and be a simple change.

[nohwnd]

I was able to repro, and this is most likely issue of the task itself because it changes the string. You can use %22 to provide the quotes, which is not ideal, but works.

-- TestRunParameters.Parameter(name=%22EdiStagingFunctionsKey%22,value=%22key2%22) TestRunParameters.Parameter(name=%22EdiAADKey%22,value=%22key1%22)

[nohwnd]

Please refer here https://github.com/microsoft/azure-pipelines-tasks/issues/13130

[sudgarre]

Thanks @nohwnd for the work around !

[ALM-at-RZL]

WORKAROUND
Will work with ".NET Core"-Build-Task with Command "test" on on-premise server 2019.0.1. Later not tested. But maybe...

-- "TestRunParameters.Parameter(name=\\\"<name1>\\\", value=\\\"<value1>\\\")" "TestRunParameters.Parameter(name=\\\"<name2>\\\", value=\\\"<value2>\\\")"

[ALM-at-RZL]

After Update from 219.0.1 to 2020.0.1 the behavior has changed.
Will work with ".NET Core"-Build-Task with Command "test" on on-premise server 2020.0.1

-- "TestRunParameters.Parameter(name=\"<name1>\", value=\"<value1>\")" "TestRunParameters.Parameter(name=\"<name2>\", value=\"<value2>\")"