Vstest: Proposal - ability to set runsettings in project file

[am11]

Issue

When we build a solution in VS2017, the test explorer require us to select .runsettings file (where we have <TargetPlatform>x64</TargetPlatform> defined). With x86 being default, our tests fail to run in the test explorer. There are other settings in that file as well which are required to be altered for our test runs.

However, in git paradigm, when we do a fresh clone or cleaning the repo directory with git clean -dfx, VS2017 does not persist the runsettings selection. Therefore, we have to specify in the repo's README for contributors to select the .runsettings file from the Test menu. This is a slight inconvenience for consumers and contributors.

Proposal

It would be great if we could specify the default .runsettings in the test project file (UnitTests.csproj etc.) as follow:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <RunSettings>.\.runsettings</RunSettings> <!-- relative to this [cs]proj file -->
  </PropertyGroup>
  ..
</Project>

[codito]

Thanks @am11 for the proposal. This is a good value add in my opinion. Users can specify settings with dotnet test -s <runsettings> too. The order in which they're picked up could be:

• Use the runsettings provided with -s <runsettings>
• Look for runsettings available in the csproj

If user wants to override a particular setting in the runsettings, they could pass in dotnet test -- RunConfiguration.TargetPlatform=x64, this should merge with provided runsettings file.

(Marked as question to discuss and close on the behavior)

[AbhitejJohn]

Sounds great. This was something we heard from a bunch of folks. This is also solution level now. From this issue, it looks like we want to make it a per-project level setting as well. Correct me if that is not true. /cc @pvlakshm

[am11]

@codito, extending on your feedback and deriving form some projects of past, the precedence order could be:

• Command line argument -s <runsettings>
• If missing the previous: read <RunSettings> from csproj
• If missing the previous: per @AbhitejJohn's comment, read the solution level (not sure would that be in the SLN file? that would be 🆒 ❗️ 😎)
• If missing the previous: read from the environment variable, say DOTNE_RUNSETTINGS
• If missing the previous: do the directory scan from csproj directory up to the volume root and then fallback to platform's %HOME% directory (this kind of algorithm: https://github.com/maxmind/MaxMind-DB-Reader-dotnet/blob/9d247d7/MaxMind.Db.Test/Helper/TestUtils.cs#L33-L44 with additional Environment.SpecialFolder.UserProfile lookup)
• If all of the above are missing, use default settings

Doesn't have to be that invasive, but for example some linters (ESLint foremerly JSHint) has such fallback for their configuration files. 8-)

[ekumlin]

This feature is something we are interested in for our current projects. We would also be satisfied if we could set this at the solution level (i.e. in the sln file).

@AbhitejJohn, above you said:

This is also solution level now.

Does this mean this is already possible in the sln file? How can we set the runsettings file association (without using the transient .suo file), and retain it between users in source control?

Thanks!

[AbhitejJohn]

@ekumlin : This is actually in the suo file today, not part of the sln unfortunately. So this isn't actually persisted in an sln pulled from source control. This at least does seem like a basic workflow that needs to fixed.

[ekumlin]

Agreed. Is sln or *proj support for runsettings association on the backlog, then?

[commonsensesoftware]

Explicitly configuring the *.runsettings file would be great, but would be even better is if the feature allows for _Convention over Configuration_. There are arguably only a few possible use cases for a *.runsettings file:

1. Explicit build property (as already proposed)
2. Per project
3. Per solution

Where #1 trumps #2 and #2 trumps #3. In terms of implementing a convention, this could be realized a couple of ways.

Convention - By Project Name

If the <RunSettings> property hasn't already been set, then infer the value from the presence of a *.runsettings file that has the same name as the target project.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <RunSettings Condition=" '$(RunSettings)' == '' AND EXISTS('$(MSBuildProjectName).runsettings') ">$(MSBuildProjectName).runsettings</RunSettings>
  </PropertyGroup>
</Project>

Convention - In Folder Hierarchy

Another useful method would be have *.runsettings detected by folder hierarchy. While it's technically possible to have multiple *.runsettings in a folder, this seems an unlikely use case. Should multiple files exist, picking the first available file won't break anything and this behavior can always be overridden by a more explicit setting. I'm not sure whether this can be realized without a custom task, but it would be useful.

[KevinH-MS]

@codito @AbhitejJohn any idea when this might make it onto the schedule? It's a highly sought-after feature for people who are trying to update large solutions to enable Live Unit Testing.

[pvlakshm]

Closing. Please file using VSFeedback if still required.

[ghost]

Closing. Please file using VSFeedback if still required.

Seriously?? Have you read the conversation in this thread? First we were given the impression that it is being worked on, and now all of the sudden closed that issue. No, that's not cool!

[pvlakshm]

Apologies for not providing context.

Yes, I read through the issue.
We are separating out IDE related issues from the core test platform related issues. We a tracking core test platform related issues in this repo, and are requesting that issues related to Test Explorer, sln/proj file support, and other IDE related issues be filed via VSFeedback. We just updated the issue template as well to this effect.

I have since copied over this issue via VSFeedback here: https://developercommunity.visualstudio.com/content/problem/151608/ability-to-set-runsettings-in-project-file.html, and informed the relevant folks.

Hope this clarifies.

[6bee]

@pvlakshm
Can we re-open this one? Seems the item coped over to VSFeedback got just stale too!?

[AbhitejJohn]

@6bee : The VS Feedback issue above is still around. We haven't gotten to prioritizing that in just yet but we have heard of a fair amount of feedback for that ask.
/cc: @ManishJayaswal

[am11]

Thanks! This is implemented and released in VS 16.4 Preview 2.

we can specify a property in csproj, with absolute path:

<RunSettingsFilePath>$(MSBuildThisFileDirectory)/.runsettings</RunSettingsFilePath>

another example: https://github.com/dotnet/project-system/blob/bd01722157391c78ab8d74e72b5e27099804c812/build/import/IntegrationTests.targets#L20

The other approach is to place .runsettings file next to .sln, but in my experiments, VS 16.4.2 did not detect the .runsettings file that way (only <RunSettingsFilePath> in csproj approach worked for me).

---

Some noticeable limitations:

• when runsettings is loaded via <RunSettingsFilePath>, the UI does not reflect the current state. For example, the loaded .runsettings file has <RunConfiguration><TargetPlatform>x64</TargetPlatform></RunConfiguration> and UI still points to x86:
  

• it seems that the only way to know whether .runsettings is loaded is a Using automatically detected ... log entry in Test Ouput Window:

  [10/27/2019 10:05:17.131 AM Informational] Store opened in 0.075 sec.
  [10/27/2019 10:05:18.295 AM Informational] ---------- Discovery started ----------
  [10/27/2019 10:05:18.461 AM Informational] Using automatically detected runsettings file(s). To learn more visit https://aka.ms/vs-runsettings.
  [10/27/2019 10:05:20.522 AM Informational] ========== Discovery finished: 21 tests found (0:00:02.1363493) ==========
  

  this log could be improved by including path to runsettings file, which it has detected.

• <RunSettingsFilePath> property only accepts absolute paths, unlike other path-like properties by SDK which resolve relative path with respect to MSBuildThisFileDirectory.

[6bee]

Thanks for your effort - it's much appreciated!
I've successfully tested '.runsettings' file on solution level with 'auto detect runsettings files' enabled in VS 16.4 preview 2.

[AbhitejJohn]

Thanks for trying this out and providing feedback.
Tagging @vritant24 to help with the issues @am11 detailed. @am11 : Would you mind filing this on developer community too please? Do share the link here so we can follow up.

[vritant24]

@am11 I greatly appreciate that you took the time to try out the feature and provided feedback.

The other approach is to place .runsettings file next to .sln, but in my experiments, VS 16.4.2 did not detect the .runsettings file that way (only in csproj approach worked for me).

Did you enable auto detection of .runsettings file in Test > Options > Auto Detect runsettings Files?

when runsettings is loaded via , the UI does not reflect the current state. For example, the loaded .runsettings file has x64 and UI still points to x86:

This is by design as multiple projects can have separate .runsettings files with separate target platforms, and the UI represents default settings for the entire solution. Although, when it comes to selecting a solution level runsettings file (".runsettings" at the root of the solution), reflecting that in the UI would be appropriate. I recommend creating a post on developer community for this.

this log could be improved by including path to runsettings file, which it has detected

This was by design as keeping the informational log output less noisy is ideal (multiple projects targeting their own runsettings files can cause a lot of logs).
If you change the logging level to Diagnostic in Test > Options > Logging, the Test output pane will have logs that show the runsettings files used mapped to the projects that use them.

property only accepts absolute paths, unlike other path-like properties by SDK which resolve relative path with respect to MSBuildThisFileDirectory.

That is right, this is something we are currently unable to support.
The recommended way is using MSBuild macros such as $(SolutionDir) should aid in setting the runsettingsfile paths
eg: <RunSettingsFilePath>$(SolutionDir)folder\file.runsettings</RunSettingsFilePath>

Please do post in the developer community with any other feedback you may have.