Testfx: TestFx 2.2.0-preview won't discover or run tests on VS16.8

[Haplois]

Description

TestFx won't discover or run tests on .Net Framework 4.7 when using Visual Studio 16.8.

Steps to reproduce

1. Create a Unit Test Project on .Net Framework 4.7.
2. Verify that tests are getting discovered and run successfully.
3. Update MSTest.TestAdapter and MSTest.TestFramework to version 2.2.0-preview-20201126-03.
4. Run the tests using the Test Explorer Window.

Expected behavior

Tests are discovered and ran.

Actual behavior

---------- Starting test discovery for requested test run ----------
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Could not load type 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Utilities.TypesToLoadAttribute' from assembly 'Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
No test is available in C:\...\UnitTestProject9\UnitTestProject9\bin\Debug\UnitTestProject9.dll. Make sure that test discoverer & executors are registered, and platform & framework version settings are appropriate and try again.
========== Test discovery finished: 0 Tests found in 8.1 sec ==========
---------- Starting test run ----------
An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load type 'Microsoft.VisualStudio.TestPlatform.ObjectModel.Utilities.TypesToLoadAttribute' from assembly 'Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
========== Test run finished: 0 Tests run in 7.4 sec (0 Passed, 0 Failed, 0 Skipped) ==========

Environment

Visual Studio v16.8.0

[nohwnd]

@Sanan07 Imho this is what you added. Are you protecting somehow against not having that type available?

[Sanan07]

I am aware of this bug. I have protection on Test Platform side, where it checks if the assembly has the attribute, so it loads specific types, otherwise load all types.

Problem here with testfx. It uses old 'Microsoft.VisualStudio.TestPlatform.ObjectModel, Version=15.0.0.0, which does not contain new types. It gets version of TP from Test Window which does not have our changes, so it breaks.

My suggestion to @Haplois was to update Test Platform on older VS versions, but he wants to find more elegant solution.

[nohwnd]

Well you introduced that change into testfx to make loading it faster in the new version, I don't see why it would be expected to break this when used with older version, so imho this is a bug that should be solved. We should in the end implement the same in xunti and nunit and I don't think they would be happy about this breaking.

[Sanan07]

I did changes inside ObjectModel in TP and use it in testfx. In older VS version we do not have that new classes introduced in TP, that why it is breaking.
We get the exception not only from my change, also recent @Haplois changes also get the same exception. Because he also introduced new types inside TP and uses them in testfx and when in VS 16.8 he updates MSTestAdapter he gets the same exception as you can see above.
So this is not bug related to specific change, it is related to miss configuration of dlls used inside testfx.

[Sanan07]

We need think how can we release testfx to avoid this type of exception. Problem occurs when you use VS which does not have TP version which contains specific logic which we use inside testfx.
If you use VS 16.8, it does not have our recent changes, and if you update MSTestAdapter inside your project to newer version which has recent changes it will break. Because MSTestAdapter has new types inside it, but TP in VS does not have it, that's why it breaks.
For instance in VS 16.9 Preview 2 everything works fine.

[Haplois]

This problem is not about one simple change. Since we started updating the TestPlatform and referencing it from TestFx, any change can be breaking. If we want to be able to update TestPlatform without breaking anything, this needs to be addressed. I was chatting with @jfleisher yesterday; he is looking into it from TestWindow'ss perspective; maybe something can be done there and retroactively fixed in older versions.

Also, I like to point out that this doesn't fail when using NETCoreApp, since we're referencing Microsoft.NET.Test.Sdk and using the referenced version. Is it possible to do that for NetFramework versions as well, @shyamnamboodiripad?

[Haplois]

We decided to start by altering the changes introduced with https://github.com/microsoft/testfx/pull/746 to make the changes non-breaking in older versions of Visual Studio and Test Platform. We'll start by removing the assembly level attribute. After that, we'll move the FQN changes out of the "ObjectModel" assembly.

[Haplois]

This partially fixed with https://github.com/microsoft/testfx/pull/753 and https://github.com/microsoft/vstest/pull/2674.