Vstest: Remove dependencies from Microsoft.Net.Test.Sdk

[tmat]

The Microsoft.Net.Test.Sdk package depends on the following packages, which causes these dependencies to flow to unit tests, potentially causing conflicts with other dependencies that the tests have. The test runner should not leak its implementation details. Please remove all these dependencies from the package nuspec.

      <group targetFramework="UAP10.0">
        <dependency id="System.ComponentModel.Primitives" version="4.1.0" />
        <dependency id="System.ComponentModel.TypeConverter" version="4.1.0" />
        <dependency id="System.Runtime.InteropServices.RuntimeInformation" version="4.0.0" />
        <dependency id="Newtonsoft.Json" version="9.0.1" />
      </group>
      <group targetFramework=".NETCoreApp1.0">
        <dependency id="Microsoft.TestPlatform.TestHost" version="15.8.0" />
        <dependency id="Microsoft.CodeCoverage" version="15.8.0" />
      </group>
      <group targetFramework=".NETFramework4.5">
        <dependency id="Microsoft.CodeCoverage" version="15.8.0" />
      </group>

[ShreyasRmsft]

@tmat those are necessary dependencies for the testplatform to function. This is a meta package and the purpose it to bring in these transitive dependencies via the meta package.

Please use the following option to instruct the testplatform to load the dll.config of the test sources and thus picking up the versions of these dlls that your tests are specifying in their dll.config files

False

[tmat]

@ShreyasRmsft Why are they necessary? Why does TestPlatform need to add these dependencies to test projects?

[ShreyasRmsft]

@tmat they are required by the testhost process and the way we leverage these dependencies in the dotnet core world is via the meta package.

[ShreyasRmsft]

@singhsarab to provide a more detailed explanation

[singhsarab]

@tmat

• For UWP projects, we need those dependencies to be part of the appx container along with the test dll.
• For NetCoreApp, TestHost dependency is required to find the path to the testhost.dll which is the actual host where the adapter gets loaded and tests run. Also, in case of publish of test project, we want testhost.dll to get copied next to the source.
• Code coverage nuget is for bringing code coverage related implementation.

Just curious, did you get into issues where there were conflicts because of these dependencies? If yes, could you please share a repro project with us?

[tmat]

did you get into issues where there were conflicts because of these dependencies?

Indeed.

To repro, clone repository https://github.com/tmat/roslyn and checkout branch NS20-VSTest.
Run restore from the root of the repository.

You'll see the following errors:

D:\R2\src\CodeStyle\VisualBasic\Tests\Microsoft.CodeAnalysis.VisualBasic.CodeStyle.UnitTests.vbproj : error NU1605: Detected package downgrade: System.Runtime.InteropServices from 4.3.0 to 4.1.0. Reference the package directly from the project to select a different version.  [D:\R2\Roslyn.sln]
D:\R2\src\CodeStyle\VisualBasic\Tests\Microsoft.CodeAnalysis.VisualBasic.CodeStyle.UnitTests.vbproj : error NU1605:  Microsoft.CodeAnalysis.VisualBasic.CodeStyle.UnitTests -> Microsoft.NET.Test.Sdk 15.8.0 -> Microsoft.TestPlatform.TestHost 15.8.0 -> Microsoft.TestPlatform.ObjectModel 15.8.0 -> System.Diagnostics.Process 4.1.0 -> Microsoft.Win32.Primitives 4.0.1 -> runtime.win.Microsoft.Win32.Primitives 4.3.0 -> System.Runtime.InteropServices (>= 4.3.0)  [D:\R2\Roslyn.sln]
D:\R2\src\CodeStyle\VisualBasic\Tests\Microsoft.CodeAnalysis.VisualBasic.CodeStyle.UnitTests.vbproj : error NU1605:  Microsoft.CodeAnalysis.VisualBasic.CodeStyle.UnitTests -> Microsoft.NET.Test.Sdk 15.8.0 -> Microsoft.TestPlatform.TestHost 15.8.0 -> Microsoft.TestPlatform.ObjectModel 15.8.0 -> System.Diagnostics.Process 4.1.0 -> System.Runtime.InteropServices (>= 4.1.0) [D:\R2\Roslyn.sln]
D:\R2\src\CodeStyle\CSharp\Tests\Microsoft.CodeAnalysis.CSharp.CodeStyle.UnitTests.csproj : error NU1605: Detected package downgrade: System.Runtime.InteropServices from 4.3.0 to 4.1.0. Reference the package directly from the project to select a different version.  [D:\R2\Roslyn.sln]

If you change this line in build/Targets/Packages.props:

<MicrosoftNETTestSdkVersion>15.8.0</MicrosoftNETTestSdkVersion>

to

<MicrosoftNETTestSdkVersion>15.9.0-dev2</MicrosoftNETTestSdkVersion>

The restore completes successfully. The version 15.9.0-dev2 of the package is a custom build that I produced from vstest repo with the nuspec modified to not list any dependencies.

[tmat]

If you need some files to be copied to the output directory of the test project, you can do so without adding nuget dependencies. Include these files in tools directory of the Test.Sdk and simply add an item group to the props file listing the files and CopyToOutputDirectory="PreserveNewest" attribute.

[singhsarab]

@tmat We will be looking into the options we have based on your suggestions. Shreyas and I will keep you posted.

[ShreyasRmsft]

@tmat

The Microsoft.NET.Test.Sdk package is a metapackage.

Its job is to bring in all the dependencies we need to run tests. If you go expand on the transitive dependencies of this package you see that it depends on certain commonly used nuget packages. For instance

Newtonsoft.Json
System.Runtime.InteropServices

Now the Microsoft.NET.Test.Sdk specifies in it’s nuspec file it needs a minimum version of its dependencies and this is applied transitively to all sub dependencies. Anything lesser could destabilize or cause the testplatform to stop working.

Ideally any user code that references an older version ends up using the version the platform mandates as minimum unless a separate appdomain is created for running tests. Now using a newer version should be acceptable as compat will generally be maintained with newer versions.

The problem we believe you are facing is that your test project is trying to take a dependency on a lower version of one of these dependencies and the nuget restore fails with the following message:

D:\R2\src\CodeStyle\CSharp\Tests\Microsoft.CodeAnalysis.CSharp.CodeStyle.UnitTests.csproj : error NU1605: Detected package downgrade: System.Runtime.InteropServices from 4.3.0 to 4.1.0. Reference the package directly from the project to select a different version.

Now let’s say we get past this and somehow allow you to take a dependency on an older version than the minimum mandated by the Microsoft.NET.Test.Sdk package. When you run your tests without the testplatform creating a new appdomain (We have seen multiple issues with appdomains and will soon be making the default execution flow to be without creating a new appdomain) the older version will get loaded and the testplatform execution is likely to fail.

I have a sample project here that you can try running to see the consequences.

Now let’s say somehow we do allow you to run your tests in a separate appdomain, you will still end up with a problem running your tests after publishing the test project.
The common route for running tests in dotnetcore after build is to publish the test project and then run the tests from there. When dotnet publishes the test project all dependencies are laid flat in the publish folder thus causing the version referenced by the test project to override the one brought in by Microsoft.NET.Test.Sdk. This will cause the test run to fail regardless of whether we choose to use an appdomain or not.

Therefore it is best for us to warn the user upfront about this if ever a lower versioned dependency is added.

Can you please either share this custom 15.9.0-dev2 package you have created or yourself try it on the sample project I have created on GitHub and share the results?

[tmat]

@ShreyasRmsft The links to the project that you listed above actually point to the metapackage definition.

The package is here: https://dotnet.myget.org/feed/roslyn-tools/package/nuget/Microsoft.NET.Test.Sdk

I have simply removed all its dependencies, so none are copied to the output directory.

What functionality in test platform needs these dependencies? Running tests from Test Explorer in Visual Studio works just fine without the dependencies.

[ShreyasRmsft]

@tmat apologies, fixing the links in a minute

I have fixed the links. please take a look

[tmat]

The test project references Newtonsoft.Json that's not compatible with the TFM (netcoreapp2.1). So that's not really a valid scenario. That said I get your point that if the Test Platform needs newer version of Newtonsoft.Json than the test project specifies, you might run into issues.

That's why the Test Platform component that needs to be loaded into the process that runs the tests needs to have minimal set of dependencies and certainly not on any particular version of Newtonsoft.Json. You can't be forcing test projects to auto upgrade their dependencies. By doing so you are subverting the customer's intent to test their code against exactly the versions of these dependencies they want to and effectively making it possible for a bug to slip thru their tests.

Hence my question above that you haven't answered: "What functionality in test platform needs these dependencies?"

[ShreyasRmsft]

@tmat Totally agree with you on

Test Platform component that needs to be loaded into the process that runs the tests needs to have minimal set of dependencies

Newtonsoft is for serializing and deserializing messages used for communication between the testplatform/datacollector/testhost processes.

The others are transitive dependencies of packages we use. You can walk through this dependency tree on the nuget package dependencies section here

[ShreyasRmsft]

@tmat we may have found a way to reduce the minimum version of system.runtime.interopservices required. I'm testing out the changes. Will get back to you once I've validated the solution

[tmat]

I attached to one of our test projects, running tests from VS 15.9 Test Explorer on .NET Framework. The debugger shows 3 AppDomains in the Modules window:

• Testhost.x86.exe
• Microsoft.Build.Tasks.Git.UnitTests
• xunit.runner.utility.net452

Not sure why the last one is needed, but it seems that Test Platform is only loaded into the first one, while the test code is loaded into the second one. This is what I’d expect as it allows to isolate the dependencies of the TP from the dependencies of the tests being run and thus we do not need them to be compatible with each other. To me this design is sound and allows us to remove all the package dependencies from Microsoft.Test.Sdk.

The same isolation can be implement on .NET Core using AssemblyLoadContext. You can load the test assembly and its dependencies to one context and the Test Platoform assemblies to another one, isolating their dependencies.
Is it what the Test Plarform already does on .NET Core? (I can’t tell without looking at the source if that’s what you’re already doing as the debugger doesn’t show which context are assemblies loaded to).

One oddity that we should also look at is the xunit.runner.visualstudio.testadapter.dll dependency on Microsoft.VisualStudio.TestPlatform.ObjectModel.dll (the assembly has a reference). Seems like that shouldn’t be there. It doesn’t seem that this assembly reference is actually used at runtime in the common test run scenarip as Microsoft.VisualStudio.TestPlatform.ObjectModel is not loaded to Microsoft.Build.Tasks.Git.UnitTests AppDomain. It would be better if the reference wasn’t even there.

The list of modules by AppDomain:

Testhost.x86.exe

C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\CMAKE\TESTEXPLORER\libcmake.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\CMAKE\TESTEXPLORER\Microsoft.VisualStudio.CMake.TestProvider.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.TestPlatform.Extensions.BlameDataCollector.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.TestPlatform.Extensions.EventLogCollector.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.TestPlatform.TestHostRuntimeProvider.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.CodedWebTestAdapter.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.TmiAdapter.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.Trx.TestLogger.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.WebTestAdapter.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestTools.CppUnitTestFramework.ComInterfaces.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestTools.DataCollection.MediaRecorder.Model.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorderCollector.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Extensions\Microsoft.VisualStudio.TraceDataCollector.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.IntelliTrace.Core.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.TestPlatform.CommunicationUtilities.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.TestPlatform.CoreUtilities.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.TestPlatform.CrossPlatEngine.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.TestPlatform.PlatformAbstractions.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.VisualStudio.ArchitectureTools.PEReader.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.VisualStudio.QualityTools.Common.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.VisualStudio.QualityTools.Resource.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.VisualStudio.TestPlatform.Common.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Microsoft.VisualStudio.TestPlatform.ObjectModel.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\Newtonsoft.Json.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\System.Collections.Immutable.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\System.Reflection.Metadata.dll
C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO\PREVIEW\ENTERPRISE\COMMON7\IDE\EXTENSIONS\TESTPLATFORM\testhost.x86.exe
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\_common\xunit.abstractions.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\_common\xunit.runner.reporters.net452.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\_common\xunit.runner.utility.net452.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\_common\xunit.runner.visualstudio.testadapter.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\netcoreapp1.0\xunit.abstractions.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\netcoreapp1.0\xunit.runner.visualstudio.dotnetcore.testadapter.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\uap10.0\xunit.runner.utility.uwp10.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\uap10.0\xunit.runner.visualstudio.uwp.testadapter.dll
C:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.UnitTestFramework\10.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
C:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.WebTestFramework\10.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.QualityTools.WebTestFramework.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_32\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Collections\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Collections.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.ComponentModel.Composition\v4.0_4.0.0.0__b77a5c561934e089\System.ComponentModel.Composition.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Globalization\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Globalization.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.IO.FileSystem.Primitives\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.IO.FileSystem.Primitives.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.IO.FileSystem\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.IO.FileSystem.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.IO.MemoryMappedFiles\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.IO.MemoryMappedFiles.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.IO\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.IO.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Linq\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Linq.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Net.Http\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Net.Http.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Numerics\v4.0_4.0.0.0__b77a5c561934e089\System.Numerics.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Reflection.Extensions\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Reflection.Extensions.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Reflection\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Reflection.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Runtime.Extensions\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Runtime.Extensions.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Runtime.Handles\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Runtime.Handles.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Runtime.Serialization\v4.0_4.0.0.0__b77a5c561934e089\System.Runtime.Serialization.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Runtime\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Runtime.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Text.Encoding\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Text.Encoding.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Threading\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Threading.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.ValueTuple\v4.0_4.0.0.0__cc7b13ffcd2ddd51\System.ValueTuple.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Xml.Linq\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.Linq.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Xml.XDocument\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Xml.XDocument.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
C:\WINDOWS\system32\WinMetadata\Windows.ApplicationModel.winmd
C:\WINDOWS\system32\WinMetadata\Windows.Storage.winmd

xunit.runner.utility.net452

C:\Users\tomat\AppData\Local\Temp\7039b19a-78a3-4e23-8061-a19e20140d41\7039b19a-78a3-4e23-8061-a19e20140d41\assembly\dl3\9ba790af\00596be3_356fd401\xunit.runner.utility.net452.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll

Microsoft.Build.Tasks.Git.UnitTests

C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\LibGit2Sharp.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\Microsoft.Build.Tasks.Git.Operations.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\Microsoft.Build.Tasks.Git.UnitTests.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\TestUtilities.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\xunit.abstractions.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\xunit.assert.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\xunit.core.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\xunit.execution.desktop.dll
C:\sourcelink\artifacts\bin\Microsoft.Build.Tasks.Git.UnitTests\Debug\net461\xunit.runner.visualstudio.testadapter.dll
C:\Users\tomat\.nuget\packages\xunit.runner.visualstudio\2.4.1\build\_common\xunit.runner.utility.net452.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Collections\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Collections.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Linq\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Linq.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Reflection\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Reflection.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Runtime\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Runtime.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System.Threading.Tasks\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Threading.Tasks.dll
C:\WINDOWS\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll

[tmat]

We have seen multiple issues with appdomains and will soon be making the default execution flow to be without creating a new appdomain)

I think that would be a mistake. What issues are there with AppDomains?

[ShreyasRmsft]

@tmat the test sdk package released after the above checkin should solve you issue.

Can you generate a local package and verify if the fix works for you?

[tmat]

Simply changing the version of one dependency does not resolve this issue. It might address the problem with the specific snapshot of the Roslyn repo, but that's just one problem. The package still has the same dependencies after that change.

[singhsarab]

Summary

Main concern: The ability to be able to test any version of a dll or dll having dependency, via TestPlatform v2.

Problem: Testhost has a bunch of dependencies, targeting specific versions, for example NewtonSoft.Json (currently 9.0.0.0). If the user wants to test NewtonSoft.Json 8.0.0.0, it is not possible given Test host process requires a higher version.

There are 2 scenarios and possible solutions that need to be tested:

1. Full desktop : Considering DisableAppDomain scenario, both these dlls, one which the test host depends on and other required by the test project need to be loaded in the same app domain.
   Possible solution: Full clr allows load for different versions for strong named dlls. Note, these dlls need to be loaded from different locations, so scenarios like publish need to be considered.
2. Net core: Assemblies can be loaded in AssemblyLoadContext, so if the adapter dll is loaded in a different AssemblyLoadContext eventually the test project dependencies also get loaded in the same context.

The solution discussed above suggest that testhost dependencies should be isolated from the test project’s dependencies. We will try to experiment using these solutions and see if we can make changes without impacting performance.

Test platform currently has dependency on relatively older or min versions for all the dependencies, which allows users to use higher versions of these dependencies. Hence, there is no immediate need to react and expedite the changes based on our hypothesis unless there is real customer ask.

That said, if any of dependencies in question, need/are to be upgraded to a higher version, we will run into issues. Hence long term, we will try to isolate, given the possible solutions discussed above.

[ViktorHofer]

@tmat we are mainly concerned about the .NET Core scenario here, right?

[tmat]

@ViktorHofer We are concerned about both desktop and core equally. Many of our repos target desktop.

[ViktorHofer]

I'm hitting this now too in corefx. We recently enabled project restore (in a separate branch yet) and when referencing Microrosft.Net.Test.Sdk from the test project, it adds the dependencies as app-locals and therefore overrides the runtime assemblies from the shared framework. We need a fix for this asap and I'm willing to help here if possible.

Have you thought about @tmat's initial proposal to copy dependencies from the package's tools directory to the output directory via an msbuild file? In example: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.RemoteExecutor/src/Microsoft.DotNet.RemoteExecutor/build/Microsoft.DotNet.RemoteExecutor.targets#L5-L6

[ViktorHofer]

Also after reading the full conversation I assume that work hasn't yet started to isolate the testhost from test assemblies and their dependencies?

@cltshivash I believe this is a bug and should be marked as such. Whenever dependencies are needed app-local which conflict with what is used by the testhost, you will most likely hit issues.

[singhsarab]

@ViktorHofer @tmat I am closing this one as the current set of dependencies point to the minimum versions.
We already plan to move to netstandard2.0 in the next couple of weeks so some of explicit dependencies will away.
Also, I have already gotten rid of the dependency on the Microsoft.Extensions.DependencyModel so that should be relief.

I have created another issue to track issues/improvements while trying integrate with Arcade
here is the link to that https://github.com/microsoft/vstest/issues/2065

I am closing this issue, please let me know if you have any concerns. Thanks,

[MartyIX]

Can anyone share status of this?

I believe it is not fixed as my package.lock.json seems to indicate that Newtonsoft.Json is accessible in my unit test project thanks to Microsoft.NET.Test.Sdk dependency:

      "Microsoft.NET.Test.Sdk": {
        "type": "Direct",
        "requested": "[16.9.1, )",
        "resolved": "16.9.1",
        "contentHash": "9acz3fExifstaoKTAvHVwRGMFrtb1QLlT6KfxOFTYM4dksuzwpkApjt0xP+yJcuRsPcf14F1b0Du3GgfKZWHJw==",
        "dependencies": {
          "Microsoft.CodeCoverage": "16.9.1",
          "Microsoft.TestPlatform.TestHost": "16.9.1"
        }
      },
// ...
      "Microsoft.TestPlatform.TestHost": {
        "type": "Transitive",
        "resolved": "16.9.1",
        "contentHash": "j/lZDlkuoUJ+lRJXOXLJpwUGXmko5/woAPo/hN6QhFRo0J5wscQPoPJ1isvXpB4Iw7x7A3jYllxR5QjV3cMlRw==",
        "dependencies": {
          "Microsoft.TestPlatform.ObjectModel": "16.9.1",
          "Newtonsoft.Json": "9.0.1"
        }
      },
// ...
      "Newtonsoft.Json": {
        "type": "Transitive",
        "resolved": "9.0.1",
        "contentHash": "U82mHQSKaIk+lpSVCbWYKNavmNH1i5xrExDEquU1i6I5pV6UMOqRnJRSlKO3cMPfcpp0RgDY+8jUXHdQ4IfXvw==",
        "dependencies": {
          "Microsoft.CSharp": "4.0.1",
          "System.Collections": "4.0.11",
          "System.Diagnostics.Debug": "4.0.11",
          "System.Dynamic.Runtime": "4.0.11",
          "System.Globalization": "4.0.11",
          "System.IO": "4.1.0",
          "System.Linq": "4.1.0",
          "System.Linq.Expressions": "4.1.0",
          "System.ObjectModel": "4.0.12",
          "System.Reflection": "4.1.0",
          "System.Reflection.Extensions": "4.0.1",
          "System.Resources.ResourceManager": "4.0.1",
          "System.Runtime": "4.1.0",
          "System.Runtime.Extensions": "4.1.0",
          "System.Runtime.Serialization.Primitives": "4.1.1",
          "System.Text.Encoding": "4.0.11",
          "System.Text.Encoding.Extensions": "4.0.11",
          "System.Text.RegularExpressions": "4.1.0",
          "System.Threading": "4.0.11",
          "System.Threading.Tasks": "4.0.11",
          "System.Xml.ReaderWriter": "4.0.11",
          "System.Xml.XDocument": "4.0.11"
        }
      },
// ...

This is very inconvenient because we don't use Newtonsoft.Json in our project and the intellisense in my MSVS returns me Newtonsoft.Json C# types (which are very similarly named as in System.Text.Json) and thus I introduced a bug in my test thanks to this.

Hopefully, I'm not missing something. I have only limited knowledge of NuGet.

Edit: Is there possibly a workaround for this?

I have tried changing:

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.1" />

to

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.1">
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>

but that does not seem to help.