Vstest: Allow dotnet test to override values in runsettings files

[livarcocc]

_From @CreepyGnome on September 13, 2017 22:41_

It would be great to be able to have the same functionality that MSTest/VSTest has when it comes ot overriding values at runtime in a runsettings file.

Example of what you can do in normal .NET Framework is here:

https://blogs.msdn.microsoft.com/devops/2015/09/04/supplying-run-time-parameters-to-tests/#comment-132205

That functionality in .NET Core would be great as it is a requirement to secure keys, secrets, passwords etc so they are not checked in to source control with integration test data that is required to run the integration tests in all environments including production.

_Copied from original issue: dotnet/cli#7644_

[livarcocc]

@kendrahavens

[CreepyGnome]

Just wondering why this was moved to VSTest? Is VSTest responsible for the "dotnet test" CLI and the features it supports?

[kendrahavens]

@CreepyGnome thank you for posting this.
@pvlakshm Can probably explain it better, but currently anything to do with "dotnet test" is mostly tracked here. This is because of our current team organization and not as much the product organization. You definitely filed it in the correct repo. Moving it here just helps us define team ownership.

Fair enough, Pratap?

[CreepyGnome]

@kendrahavens thanks, so if I understand correctly issues sometimes may not be worked in the repo the code lives in due to organizational separations within Microsoft which can make sense in a company as large as Microsoft.

[kendrahavens]

@CreepyGnome Yes, I believe that's currently how this specific case works. Thank you for understanding. Pratap (pvlakshm), might correct me, but that is why a lot of the bugs in vstest are "dotnet test" specific.

[harshjain2]

@CreepyGnome @kendrahavens @livarcocc vstest.console and dotnet test has the capability to pass runsettings values through command line.

Runsettings arguments takes precedence over runsettings.

More info :

1. https://github.com/Microsoft/vstest-docs/blob/master/docs/RunSettingsArguments.md.
2. Refer "RunSettings arguments" section in dotnet test help ( dotnet test --help ).

[CreepyGnome]

Thanks for the link @harshjain2. That is very confusing what is listed in --help, and not listed on the "official" documentation page at all for dotnet test:

https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test

Does that work on DotNet 1.x? I assume since this only exists in Github docs which was created on April 21, 2017 that this is most likely a new feature of 2.0.

I may have missed it but don't remember seeing anything about arguments when I did dotnet test --help at work using 1.x. Also did a lot of "googling" and never found that GitHub page, only found the VsTest/MsTest information which is all GUI based in TFS and VSTS, not DotNet core information.

At work I have to use 1.x and mostly because of VSTS the build/release hosted agent is stuck way in the past on VS 2017 Update 1, so it also only has the older version of DotNet core. Even using the Task that installs SDK 2.0 before we run any dotnet commands doesn't use the 2.0 that was installed it uses the 1.x that is part of the hosted build agent. Basically, if Microsoft could keep their VSTS up to date with build agents that support their releases of DotNet core it would remove the only blocker currently we have to upgrading to 2.0. Basically if VS2017 gets an update there should be a new hosted build agent that supports that update. We are on update 3.4 and its still on update 1.

I would love to see that documentation added to the official doc page under at least the 2.x tab and if it does work with 1.x love to see it there. Since when we don't see what we want with --help or want more info about something their our next stop is docs.microsoft.com not github. After that, we start googling.

[nigurr]

Hi,

Update on this thread. Work is in progress to support .NET core tests in Visual Studio test task which provides the option to override parameters.
However, we don't have exact timeline for roll out. Assured that we are actively working on this.

Thanks for your support. Please do raise your feature requests/voice here: https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services

[CreepyGnome]

Hi @nigurr if what you say is true why would you close the issue that is listed in the backlog unless it was completed? Seems closed backlog issues are either done or are considered not going to do them anytime in the foreseeable future. You state work is in progress, but you closed the issue, so it seems it's not done nor is it not going to get done as it's in progress. I am a little confused as to why the issue is closed.

[nigurr]

@CreepyGnome Currently we do have internal tracking system where we track the features/user stories. It would be hard for us to maintaining them separately (github/internal).
We do sincerely understand your concerns/feature requests and update our backlog. We keep informing about feature deliveries in VSTS release notes which gets published with every update.
Please let us know if there's better approach for the same.

[CreepyGnome]

@nigurr seems like your teams can learn a lot from the .NET Core teams. If you are going to have a public facing issue tracking system, why not just use it and do it all in the public like other teams at Microsoft.

Also VSTS issues like source, if I remember correctly, can be made public facing as well and you could just use that instead of GitHub if you have no source here. Also you could put a URL in the comments or description of each that says where each is.

Better yet, I am sure a with all the VSTS teams of developers who have already integrated VSTS with GitHub for a number of cool features could find a way to keep internal and public issues in sync with each other.

I know other companies who want to keep a public and private system may like that feature, as it would be useful to various MS teams.

[nigurr]

@CreepyGnome Thanks for feedback. Really appreciate it. We will discuss this feedback and will be more transparent to community.

[matthew-horrocks]

@nigurr

Is there an update available on the status of getting .net core 2 tests running in the Visual Studio Test Task?

Is it possible for my company's DevOps subscription to be added to a beta version of this test task that offers this functionality?

Thanks,
Matt

[nigurr]

@matthew-horrocks Currently we do support .net core tests in Visual Studio test task with limited functionality (Advance options/Distribution work is in progress).

[tourdedave]

Is there an update on this issue? This feature would benefit a lot of people writing end-to-end tests with Selenium & C#.

[borgewi]

@nigurr

Is there any update on overriding .net core test run parameters using the vs test task?

I am able to run the test task, however when the task is running, the updated runsettings is leaving out the TestRunParameters, as such:

Thanks!
Børge

[xamadev]

Any news on this? Would like to be able to override TestRunParamteres defined in .runsettings file by passing it to CLI command in my CI-pipeline:

dotnet test -- myrunparametertobeoverridden=newvalue

[sudgarre]

@xamadev : Did you try workaround